| *** mmaglana has quit IRC | 00:00 | |
| *** mmaglana has joined #openstack-infra | 00:01 | |
| *** tnovacik has quit IRC | 00:08 | |
| *** rkukura has quit IRC | 00:12 | |
| *** mmaglana has quit IRC | 00:18 | |
| *** Ryan_Lane has joined #openstack-infra | 00:20 | |
| *** banix has joined #openstack-infra | 00:23 | |
| *** armax has joined #openstack-infra | 00:24 | |
| *** sarob has quit IRC | 00:28 | |
| *** salv-orlando has quit IRC | 00:29 | |
| *** emagana has joined #openstack-infra | 00:29 | |
| *** ZZelle_ has quit IRC | 00:34 | |
| *** emagana has quit IRC | 00:34 | |
| *** dimsum__ has joined #openstack-infra | 00:35 | |
| *** harlowja_at_home has joined #openstack-infra | 00:37 | |
| *** banix has quit IRC | 00:38 | |
| *** armax has quit IRC | 00:42 | |
| *** banix has joined #openstack-infra | 00:44 | |
| *** Ryan_Lane has quit IRC | 00:50 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/elastic-recheck: Add stable/icehouse query for bug 1395368 https://review.openstack.org/136657 | 00:52 |
|---|---|---|
| uvirtbot | Launchpad bug 1395368 in tempest "ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip (icehouse) failures" [Undecided,New] https://launchpad.net/bugs/1395368 | 00:52 |
| *** Ryan_Lane has joined #openstack-infra | 00:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 01:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add debootstrap and rinse to nodepool https://review.openstack.org/136598 | 01:05 |
| mordred | clarkb, fungi, jhesketh: ^^ those at least build something. next step, try booting the to see if they are operable | 01:06 |
| mordred | oh - and I should add the root ssh key :) | 01:06 |
| clarkb | that is an important step | 01:06 |
| mordred | yah | 01:07 |
| mordred | ooh. new bug found on the ubuntu side ... no ssl certs ... anybody remember what the ubuntu package is to get the certs? ca-certificates right? | 01:07 |
| *** koolhead17 has quit IRC | 01:08 | |
| *** banix has quit IRC | 01:09 | |
| zxiiro | Does anyone know if "git review" and submit drafts? I can't see any docs that provide the arguments for that | 01:09 |
| clarkb | it can git review -d | 01:10 |
| clarkb | should be in the man page | 01:10 |
| zxiiro | ah ok thanks (I was googling, should have thought to check the man page...) | 01:10 |
| mordred | clarkb: speaking of manpages - I made sure that the centos elements above install both man-pages and lsof | 01:11 |
| mordred | :) | 01:11 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/elastic-recheck: Add stable/icehouse query for bug 1395368 https://review.openstack.org/136657 | 01:12 |
| uvirtbot | Launchpad bug 1395368 in tempest "ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip (icehouse) failures" [Undecided,New] https://launchpad.net/bugs/1395368 | 01:12 |
| *** Ark has joined #openstack-infra | 01:15 | |
| *** Ark is now known as Guest81236 | 01:15 | |
| openstackgerrit | Zhidong Yu proposed openstack/requirements: Add cm-api to global requirements. https://review.openstack.org/130153 | 01:15 |
| *** yaguang has joined #openstack-infra | 01:17 | |
| *** superdan is now known as dansmith | 01:23 | |
| *** emagana has joined #openstack-infra | 01:23 | |
| *** bhunter71 has joined #openstack-infra | 01:25 | |
| *** emagana has quit IRC | 01:28 | |
| *** dimsum__ has quit IRC | 01:31 | |
| *** dimsum__ has joined #openstack-infra | 01:31 | |
| *** dimsum__ has quit IRC | 01:36 | |
| *** adalbas has quit IRC | 01:38 | |
| *** harlowja_at_home has quit IRC | 01:42 | |
| *** stevemar has joined #openstack-infra | 01:44 | |
| *** Ryan_Lane has quit IRC | 01:44 | |
| *** MaxV has joined #openstack-infra | 01:46 | |
| *** wuhg has joined #openstack-infra | 01:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 01:51 |
| *** MaxV has quit IRC | 01:51 | |
| mordred | ok. that fixes the ubuntu build | 01:52 |
| *** dimsum__ has joined #openstack-infra | 01:54 | |
| *** banix has joined #openstack-infra | 01:56 | |
| *** lttrl has joined #openstack-infra | 01:57 | |
| *** achuprin_ has quit IRC | 02:01 | |
| *** fandi has joined #openstack-infra | 02:02 | |
| *** xchu has joined #openstack-infra | 02:05 | |
| *** Guest81236 has quit IRC | 02:05 | |
| *** rkukura has joined #openstack-infra | 02:06 | |
| *** xchu has quit IRC | 02:06 | |
| *** stevemar has quit IRC | 02:10 | |
| *** emagana has joined #openstack-infra | 02:17 | |
| *** achuprin_ has joined #openstack-infra | 02:21 | |
| *** emagana has quit IRC | 02:22 | |
| *** pcrews has joined #openstack-infra | 02:28 | |
| *** yongli has quit IRC | 02:29 | |
| *** camunoz is now known as camunoz_away | 02:30 | |
| jogo | clarkb: any ideas why this didn't work? https://review.openstack.org/#/c/136596/ I am stumped | 02:32 |
| *** weshay has quit IRC | 02:33 | |
| *** armax has joined #openstack-infra | 02:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 02:44 |
| *** unicell has joined #openstack-infra | 02:46 | |
| *** Ryan_Lane has joined #openstack-infra | 02:47 | |
| anteaya | been lurking to see if Yongli He and Shane Wang make an appearance | 02:55 |
| anteaya | thinking about heading offline | 02:55 |
| *** harlowja_at_home has joined #openstack-infra | 02:57 | |
| anteaya | yep, cat is sleeping in my lap and I can't stay awake | 02:58 |
| *** mase_x200 has joined #openstack-infra | 02:58 | |
| *** MaxV has joined #openstack-infra | 02:58 | |
| *** shashankhegde has joined #openstack-infra | 03:00 | |
| *** MaxV has quit IRC | 03:02 | |
| *** KanagarajM has joined #openstack-infra | 03:04 | |
| *** pcrews has quit IRC | 03:07 | |
| *** emagana has joined #openstack-infra | 03:12 | |
| *** harlowja_at_home has quit IRC | 03:15 | |
| *** Ark has joined #openstack-infra | 03:16 | |
| *** Ark is now known as Guest47983 | 03:16 | |
| *** emagana has quit IRC | 03:16 | |
| *** fifieldt has joined #openstack-infra | 03:19 | |
| *** Guest47983 has quit IRC | 03:20 | |
| *** annegentle has quit IRC | 03:21 | |
| *** davideagnello has joined #openstack-infra | 03:22 | |
| *** dimsum__ has quit IRC | 03:27 | |
| *** davideagnello has quit IRC | 03:27 | |
| *** sarob has joined #openstack-infra | 03:29 | |
| *** sarob has quit IRC | 03:33 | |
| *** ddieterly has quit IRC | 03:35 | |
| *** baoli has quit IRC | 03:37 | |
| *** baoli has joined #openstack-infra | 03:38 | |
| *** banix has quit IRC | 03:39 | |
| *** hdd has joined #openstack-infra | 03:45 | |
| *** banix has joined #openstack-infra | 03:46 | |
| *** camunoz_away is now known as camunoz | 03:48 | |
| *** Ryan_Lane has quit IRC | 03:57 | |
| *** boris-42 has quit IRC | 03:57 | |
| *** armax has quit IRC | 03:58 | |
| *** bhunter71 has quit IRC | 03:59 | |
| *** armax has joined #openstack-infra | 03:59 | |
| *** armax has quit IRC | 04:00 | |
| *** hdd has quit IRC | 04:00 | |
| *** Ryan_Lane has joined #openstack-infra | 04:00 | |
| *** mase_x200 has quit IRC | 04:03 | |
| *** emagana has joined #openstack-infra | 04:06 | |
| *** emagana has quit IRC | 04:10 | |
| *** Ryan_Lane has quit IRC | 04:16 | |
| *** ryanpetrello has joined #openstack-infra | 04:20 | |
| *** armax has joined #openstack-infra | 04:20 | |
| *** banix has quit IRC | 04:20 | |
| *** zz_dimtruck is now known as dimtruck | 04:21 | |
| *** Hal_ has joined #openstack-infra | 04:24 | |
| *** dimsum__ has joined #openstack-infra | 04:27 | |
| *** ddieterly has joined #openstack-infra | 04:29 | |
| *** ddieterly has quit IRC | 04:31 | |
| *** ddieterly has joined #openstack-infra | 04:31 | |
| *** dimsum__ has quit IRC | 04:32 | |
| *** shashankhegde has quit IRC | 04:34 | |
| *** ddieterly has quit IRC | 04:36 | |
| *** armax has quit IRC | 04:36 | |
| *** Hal_ has quit IRC | 04:36 | |
| *** chandankumar has joined #openstack-infra | 04:40 | |
| *** boris-42 has joined #openstack-infra | 04:42 | |
| *** koolhead17 has joined #openstack-infra | 04:47 | |
| *** yfried_ has joined #openstack-infra | 04:52 | |
| *** chandankumar has quit IRC | 04:55 | |
| *** baoli has quit IRC | 05:06 | |
| *** dimtruck is now known as zz_dimtruck | 05:07 | |
| *** otter768 has quit IRC | 05:14 | |
| *** shashankhegde has joined #openstack-infra | 05:14 | |
| *** armax has joined #openstack-infra | 05:16 | |
| *** Longgeek has joined #openstack-infra | 05:21 | |
| *** Longgeek has quit IRC | 05:27 | |
| *** armax has quit IRC | 05:28 | |
| *** ddieterly has joined #openstack-infra | 05:30 | |
| *** teran has quit IRC | 05:31 | |
| *** viglesias has quit IRC | 05:35 | |
| *** ddieterly has quit IRC | 05:35 | |
| *** viglesias has joined #openstack-infra | 05:41 | |
| *** stevemar has joined #openstack-infra | 05:42 | |
| *** rushiagr_away is now known as rushiagr | 05:45 | |
| *** yongli has joined #openstack-infra | 05:46 | |
| *** yfried_ has quit IRC | 05:46 | |
| *** ryanpetrello has quit IRC | 05:49 | |
| *** k4n0 has joined #openstack-infra | 06:00 | |
| *** yongli has quit IRC | 06:03 | |
| *** BharatK has joined #openstack-infra | 06:12 | |
| *** chandankumar has joined #openstack-infra | 06:14 | |
| *** Hal_ has joined #openstack-infra | 06:17 | |
| *** Hal_ has quit IRC | 06:18 | |
| *** shashankhegde has quit IRC | 06:21 | |
| *** ddieterly has joined #openstack-infra | 06:30 | |
| *** teran has joined #openstack-infra | 06:31 | |
| *** ddieterly has quit IRC | 06:35 | |
| *** teran has quit IRC | 06:36 | |
| *** ildikov has quit IRC | 06:42 | |
| *** camunoz is now known as camunoz_gone | 06:44 | |
| *** boris-42 has quit IRC | 06:47 | |
| *** patrickeast has joined #openstack-infra | 06:48 | |
| *** koolhead17 has quit IRC | 06:48 | |
| *** yfried_ has joined #openstack-infra | 06:48 | |
| *** michchap_ has quit IRC | 06:51 | |
| *** patrickeast has quit IRC | 06:51 | |
| *** michchap has joined #openstack-infra | 06:52 | |
| *** talluri has joined #openstack-infra | 06:53 | |
| *** talluri_ has joined #openstack-infra | 06:56 | |
| *** talluri has quit IRC | 06:58 | |
| *** davideagnello has joined #openstack-infra | 07:00 | |
| *** viglesias has quit IRC | 07:01 | |
| *** talluri has joined #openstack-infra | 07:01 | |
| *** afazekas has joined #openstack-infra | 07:01 | |
| *** talluri_ has quit IRC | 07:03 | |
| *** AlexF has joined #openstack-infra | 07:04 | |
| *** davideagnello has quit IRC | 07:04 | |
| *** Longgeek has joined #openstack-infra | 07:05 | |
| *** Hefeweizen has quit IRC | 07:09 | |
| *** viglesias has joined #openstack-infra | 07:12 | |
| *** otter768 has joined #openstack-infra | 07:15 | |
| *** AlexF has quit IRC | 07:15 | |
| *** stevemar has quit IRC | 07:16 | |
| *** achanda has joined #openstack-infra | 07:17 | |
| *** belmoreira has joined #openstack-infra | 07:19 | |
| *** Murad has joined #openstack-infra | 07:19 | |
| *** otter768 has quit IRC | 07:20 | |
| *** achanda has quit IRC | 07:20 | |
| *** achanda has joined #openstack-infra | 07:20 | |
| *** ildikov has joined #openstack-infra | 07:20 | |
| *** koolhead17 has joined #openstack-infra | 07:20 | |
| *** koolhead17 has joined #openstack-infra | 07:20 | |
| *** yfried_ is now known as yfried|afk | 07:21 | |
| *** koolhead17 has quit IRC | 07:23 | |
| *** viglesias has quit IRC | 07:24 | |
| *** viglesias has joined #openstack-infra | 07:30 | |
| *** ddieterly has joined #openstack-infra | 07:30 | |
| *** yfried|afk is now known as yfried_ | 07:32 | |
| *** teran has joined #openstack-infra | 07:32 | |
| *** talluri has quit IRC | 07:33 | |
| *** ddieterly has quit IRC | 07:34 | |
| *** ddieterly has joined #openstack-infra | 07:36 | |
| *** teran has quit IRC | 07:37 | |
| *** mrmartin has joined #openstack-infra | 07:38 | |
| *** jgallard_ has joined #openstack-infra | 07:40 | |
| *** ddieterly has quit IRC | 07:40 | |
| *** belmoreira has quit IRC | 07:42 | |
| *** salv-orlando has joined #openstack-infra | 07:42 | |
| *** emagana has joined #openstack-infra | 07:44 | |
| *** belmoreira has joined #openstack-infra | 07:45 | |
| *** ivar-lazzaro has quit IRC | 07:46 | |
| *** koolhead17 has joined #openstack-infra | 07:47 | |
| openstackgerrit | Mate Lakat proposed openstack-infra/project-config: XenServer: Use nodepool to inject XVA and ISO url https://review.openstack.org/136700 | 07:48 |
| *** emagana has quit IRC | 07:49 | |
| openstackgerrit | Mate Lakat proposed openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/97787 | 07:51 |
| openstackgerrit | Mate Lakat proposed openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/97798 | 07:51 |
| *** Daisy has joined #openstack-infra | 07:57 | |
| *** jyuso has joined #openstack-infra | 07:57 | |
| *** achanda has quit IRC | 07:59 | |
| *** amuller has joined #openstack-infra | 08:02 | |
| *** ZZelle has quit IRC | 08:02 | |
| *** ZZelle has joined #openstack-infra | 08:02 | |
| *** miqui_ has quit IRC | 08:05 | |
| *** rcarrillocruz has quit IRC | 08:09 | |
| *** rcarrillocruz has joined #openstack-infra | 08:09 | |
| *** teran has joined #openstack-infra | 08:12 | |
| *** talluri has joined #openstack-infra | 08:13 | |
| *** skolekonov has joined #openstack-infra | 08:15 | |
| *** HeOS has quit IRC | 08:18 | |
| *** e0ne has joined #openstack-infra | 08:18 | |
| *** KanagarajM has quit IRC | 08:19 | |
| *** talluri has quit IRC | 08:22 | |
| *** doude has quit IRC | 08:25 | |
| *** achuprin_ has quit IRC | 08:28 | |
| *** jcoufal has joined #openstack-infra | 08:31 | |
| openstackgerrit | Mate Lakat proposed openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/97798 | 08:36 |
| *** jerryz has joined #openstack-infra | 08:38 | |
| *** jlibosva has joined #openstack-infra | 08:38 | |
| *** emagana has joined #openstack-infra | 08:38 | |
| *** arxcruz has joined #openstack-infra | 08:39 | |
| *** achuprin_ has joined #openstack-infra | 08:40 | |
| *** MaxV has joined #openstack-infra | 08:41 | |
| *** emagana has quit IRC | 08:43 | |
| *** bo_sh has joined #openstack-infra | 08:45 | |
| *** nadya has joined #openstack-infra | 08:47 | |
| *** nadya is now known as Guest36645 | 08:48 | |
| *** berendt has joined #openstack-infra | 08:49 | |
| *** Guest36645 has quit IRC | 08:49 | |
| *** jistr has joined #openstack-infra | 08:55 | |
| *** jlibosva has quit IRC | 08:56 | |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** ala_ has joined #openstack-infra | 09:00 | |
| *** nfedotov has joined #openstack-infra | 09:02 | |
| *** jlibosva has joined #openstack-infra | 09:03 | |
| *** teran has quit IRC | 09:06 | |
| *** derekh has joined #openstack-infra | 09:11 | |
| *** jedimike has joined #openstack-infra | 09:13 | |
| *** Murad has quit IRC | 09:15 | |
| *** otter768 has joined #openstack-infra | 09:16 | |
| *** andreykurilin_ has joined #openstack-infra | 09:18 | |
| *** tnovacik has joined #openstack-infra | 09:18 | |
| *** jlibosva has quit IRC | 09:20 | |
| *** otter768 has quit IRC | 09:20 | |
| openstackgerrit | Claudiu Popa proposed openstack-dev/pbr: Support platform-specific requirements files https://review.openstack.org/136707 | 09:21 |
| *** jlibosva has joined #openstack-infra | 09:21 | |
| *** IvanBerezovskiy has joined #openstack-infra | 09:22 | |
| *** HeOS has joined #openstack-infra | 09:23 | |
| *** bo_sh has left #openstack-infra | 09:25 | |
| *** teran has joined #openstack-infra | 09:26 | |
| *** andreykurilin_ has quit IRC | 09:26 | |
| *** amuller_ has joined #openstack-infra | 09:26 | |
| *** amuller__ has joined #openstack-infra | 09:28 | |
| *** nadya has joined #openstack-infra | 09:29 | |
| *** nadya is now known as Guest24890 | 09:29 | |
| *** maishsk has joined #openstack-infra | 09:29 | |
| * maishsk says hi and good morning - anyone awake? | 09:30 | |
| *** amuller has quit IRC | 09:30 | |
| *** mpaolino has joined #openstack-infra | 09:31 | |
| *** hashar has joined #openstack-infra | 09:31 | |
| *** amuller_ has quit IRC | 09:31 | |
| *** emagana has joined #openstack-infra | 09:33 | |
| *** talluri has joined #openstack-infra | 09:33 | |
| *** Longgeek has quit IRC | 09:35 | |
| *** emagana has quit IRC | 09:37 | |
| *** talluri has quit IRC | 09:38 | |
| *** yamamoto has joined #openstack-infra | 09:39 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:42 | |
| *** bo_sh has joined #openstack-infra | 09:42 | |
| *** hashar has quit IRC | 09:43 | |
| *** teran has quit IRC | 09:44 | |
| *** maishsk has quit IRC | 09:46 | |
| *** jp_at_hp has joined #openstack-infra | 09:47 | |
| *** Longgeek has joined #openstack-infra | 09:51 | |
| *** jcoufal has quit IRC | 09:51 | |
| *** hashar has joined #openstack-infra | 09:52 | |
| *** bo_sh has left #openstack-infra | 09:53 | |
| *** deepakcs has joined #openstack-infra | 09:53 | |
| *** hashar has quit IRC | 09:53 | |
| *** maishsk has joined #openstack-infra | 09:53 | |
| maishsk | Anyone awake yet? | 09:53 |
| *** hashar has joined #openstack-infra | 09:53 | |
| *** dimsum__ has joined #openstack-infra | 09:54 | |
| *** belmoreira has quit IRC | 09:55 | |
| *** Guest24890 has quit IRC | 09:58 | |
| *** dimsum__ has quit IRC | 09:59 | |
| *** cnesa has joined #openstack-infra | 10:00 | |
| *** boris-42 has joined #openstack-infra | 10:01 | |
| *** yaguang has quit IRC | 10:02 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 10:03 | |
| *** rlandy has joined #openstack-infra | 10:04 | |
| *** cnesa has quit IRC | 10:04 | |
| *** cnesa has joined #openstack-infra | 10:05 | |
| *** marcusvrn has joined #openstack-infra | 10:07 | |
| *** amuller__ has quit IRC | 10:10 | |
| *** dmelladol is now known as dmellado|afk | 10:12 | |
| *** Daisy has quit IRC | 10:13 | |
| *** Daisy has joined #openstack-infra | 10:13 | |
| *** mase_x200 has joined #openstack-infra | 10:20 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 10:21 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 10:24 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 10:24 | |
| *** dmellado|afk has quit IRC | 10:25 | |
| *** belmoreira has joined #openstack-infra | 10:26 | |
| *** dmellado has joined #openstack-infra | 10:27 | |
| *** emagana has joined #openstack-infra | 10:27 | |
| *** fandi has quit IRC | 10:28 | |
| *** mase_x200 has quit IRC | 10:29 | |
| *** Daisy has quit IRC | 10:30 | |
| *** emagana has quit IRC | 10:31 | |
| *** BharatK has quit IRC | 10:32 | |
| *** mase_x200 has joined #openstack-infra | 10:32 | |
| *** vdo has joined #openstack-infra | 10:33 | |
| *** hdd has joined #openstack-infra | 10:34 | |
| *** mpaolino has quit IRC | 10:35 | |
| *** yamamoto has quit IRC | 10:36 | |
| *** davideagnello has joined #openstack-infra | 10:37 | |
| *** mase_x200 has quit IRC | 10:38 | |
| *** ldnunes has joined #openstack-infra | 10:39 | |
| *** davideagnello has quit IRC | 10:42 | |
| *** BharatK has joined #openstack-infra | 10:43 | |
| *** akuznetsova has joined #openstack-infra | 10:46 | |
| *** mpaolino has joined #openstack-infra | 10:48 | |
| *** yamamoto has joined #openstack-infra | 10:49 | |
| *** jcoufal has joined #openstack-infra | 10:53 | |
| *** yfried_ is now known as yfried|afk | 10:56 | |
| *** maishsk has quit IRC | 11:05 | |
| *** ldnunes has quit IRC | 11:07 | |
| *** yamamoto has quit IRC | 11:07 | |
| *** MaxV has quit IRC | 11:08 | |
| *** teran has joined #openstack-infra | 11:08 | |
| *** ldnunes has joined #openstack-infra | 11:09 | |
| *** yamamoto has joined #openstack-infra | 11:10 | |
| *** MaxV has joined #openstack-infra | 11:12 | |
| *** sergsh has joined #openstack-infra | 11:13 | |
| *** hdd has quit IRC | 11:15 | |
| *** jgallard_ has quit IRC | 11:15 | |
| *** yfried|afk is now known as yfried_ | 11:16 | |
| *** otter768 has joined #openstack-infra | 11:17 | |
| *** nadya has joined #openstack-infra | 11:17 | |
| *** nadya is now known as Guest86919 | 11:18 | |
| *** emagana has joined #openstack-infra | 11:21 | |
| *** hdd has joined #openstack-infra | 11:21 | |
| *** otter768 has quit IRC | 11:21 | |
| *** amuller__ has joined #openstack-infra | 11:23 | |
| *** rfolco has joined #openstack-infra | 11:24 | |
| *** emagana has quit IRC | 11:25 | |
| *** yfried_ is now known as yfried|afk | 11:26 | |
| *** MaxV has quit IRC | 11:27 | |
| *** maishsk has joined #openstack-infra | 11:28 | |
| *** rfolco has quit IRC | 11:29 | |
| *** yfried|afk is now known as yfried_ | 11:31 | |
| *** groknix has quit IRC | 11:31 | |
| *** teran has quit IRC | 11:31 | |
| *** teran has joined #openstack-infra | 11:31 | |
| *** groknix has joined #openstack-infra | 11:31 | |
| *** ldnunes_ has joined #openstack-infra | 11:32 | |
| *** ldnunes has quit IRC | 11:32 | |
| *** hdd has quit IRC | 11:33 | |
| *** groknix has quit IRC | 11:34 | |
| maishsk | Anyone around? | 11:34 |
| *** groknix has joined #openstack-infra | 11:34 | |
| *** pblaho has joined #openstack-infra | 11:35 | |
| *** ashaeron has joined #openstack-infra | 11:35 | |
| *** isaacb has joined #openstack-infra | 11:36 | |
| *** koolhead17 has quit IRC | 11:36 | |
| *** koolhead17 has joined #openstack-infra | 11:36 | |
| *** amuller__ is now known as amuller | 11:36 | |
| *** koolhead17 has quit IRC | 11:37 | |
| *** aysyd has joined #openstack-infra | 11:37 | |
| *** marcusvrn has quit IRC | 11:37 | |
| *** adalbas has joined #openstack-infra | 11:40 | |
| *** yfried_ is now known as yfried|afk | 11:41 | |
| *** yfried|afk is now known as yfried_ | 11:42 | |
| *** pblaho has quit IRC | 11:45 | |
| *** rcarrillocruz has quit IRC | 11:55 | |
| *** rcarrillocruz has joined #openstack-infra | 11:55 | |
| *** marcusvrn has joined #openstack-infra | 11:56 | |
| *** dkehn has quit IRC | 11:56 | |
| *** chandankumar has quit IRC | 11:56 | |
| *** dkehn has joined #openstack-infra | 11:57 | |
| *** yfried_ is now known as yfried|afk | 12:01 | |
| *** chandankumar has joined #openstack-infra | 12:01 | |
| *** dimsum__ has joined #openstack-infra | 12:03 | |
| *** unicell has quit IRC | 12:10 | |
| *** hashar has quit IRC | 12:14 | |
| *** BharatK has quit IRC | 12:16 | |
| *** pblaho has joined #openstack-infra | 12:21 | |
| *** koolhead17 has joined #openstack-infra | 12:21 | |
| *** yfried|afk is now known as yfried_ | 12:21 | |
| *** mase_x200 has joined #openstack-infra | 12:22 | |
| *** MaxV has joined #openstack-infra | 12:27 | |
| *** mase_x200 has quit IRC | 12:29 | |
| *** mase_x200 has joined #openstack-infra | 12:30 | |
| *** weshay has joined #openstack-infra | 12:32 | |
| *** amuller_ has joined #openstack-infra | 12:37 | |
| *** amuller has quit IRC | 12:37 | |
| *** vdo has quit IRC | 12:39 | |
| *** amuller__ has joined #openstack-infra | 12:41 | |
| openstackgerrit | Merged openstack-infra/storyboard: setup for running as a stand alone application. https://review.openstack.org/131870 | 12:43 |
| *** amuller_ has quit IRC | 12:44 | |
| *** amuller has joined #openstack-infra | 12:45 | |
| *** amuller__ has quit IRC | 12:49 | |
| openstackgerrit | Merged openstack-infra/storyboard: Split Token DB API into separate file https://review.openstack.org/134408 | 12:50 |
| *** jcoufal_ has joined #openstack-infra | 12:51 | |
| openstackgerrit | Merged openstack-infra/storyboard: User token API https://review.openstack.org/134409 | 12:53 |
| *** jcoufal has quit IRC | 12:54 | |
| *** marcusvrn1 has joined #openstack-infra | 12:56 | |
| *** marcusvrn has quit IRC | 12:57 | |
| *** deepakcs has quit IRC | 12:59 | |
| *** k4n0 has quit IRC | 13:00 | |
| *** ddieterly has joined #openstack-infra | 13:00 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add timeout to the blur event of tag-complete https://review.openstack.org/135334 | 13:02 |
| *** maishsk has quit IRC | 13:04 | |
| *** yolanda has joined #openstack-infra | 13:04 | |
| *** sandywalsh has joined #openstack-infra | 13:04 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Switched use of "Resource.read()" to "Resource.get()" https://review.openstack.org/136148 | 13:04 |
| *** jaypipes has joined #openstack-infra | 13:05 | |
| *** emagana has joined #openstack-infra | 13:09 | |
| *** mbacchi has joined #openstack-infra | 13:09 | |
| *** emagana has quit IRC | 13:13 | |
| *** maishsk has joined #openstack-infra | 13:16 | |
| openstackgerrit | Merged openstack-infra/storyboard: Added project group title to loader. https://review.openstack.org/133248 | 13:17 |
| *** otter768 has joined #openstack-infra | 13:18 | |
| *** dprince has joined #openstack-infra | 13:21 | |
| *** bswartz has quit IRC | 13:22 | |
| *** baoli has joined #openstack-infra | 13:22 | |
| *** baoli has quit IRC | 13:22 | |
| *** otter768 has quit IRC | 13:22 | |
| *** eharney has joined #openstack-infra | 13:23 | |
| *** baoli has joined #openstack-infra | 13:23 | |
| *** Ng has quit IRC | 13:24 | |
| *** jraim has quit IRC | 13:24 | |
| *** tchaypo has quit IRC | 13:24 | |
| *** serverascode___ has quit IRC | 13:24 | |
| *** simonmcc has quit IRC | 13:24 | |
| *** Ng has joined #openstack-infra | 13:24 | |
| *** zhiyan has quit IRC | 13:24 | |
| *** jraim has joined #openstack-infra | 13:25 | |
| *** boris-42 has quit IRC | 13:25 | |
| *** sweston_ has quit IRC | 13:25 | |
| *** sweston_ has joined #openstack-infra | 13:25 | |
| *** rainya has quit IRC | 13:26 | |
| *** boris-42 has joined #openstack-infra | 13:26 | |
| *** tchaypo has joined #openstack-infra | 13:26 | |
| *** serverascode___ has joined #openstack-infra | 13:26 | |
| *** rainya has joined #openstack-infra | 13:27 | |
| *** zhiyan has joined #openstack-infra | 13:27 | |
| *** simonmcc_ has joined #openstack-infra | 13:28 | |
| nibalizer | good morning | 13:29 |
| *** koolhead17 has quit IRC | 13:30 | |
| *** koolhead17 has joined #openstack-infra | 13:30 | |
| maishsk | hi | 13:32 |
| *** julim has joined #openstack-infra | 13:33 | |
| *** jerryz1 has joined #openstack-infra | 13:33 | |
| *** che-arne has joined #openstack-infra | 13:34 | |
| *** pc_m has joined #openstack-infra | 13:35 | |
| *** pc_m has quit IRC | 13:35 | |
| *** jerryz has quit IRC | 13:35 | |
| *** koolhead17 has quit IRC | 13:35 | |
| *** pc_m has joined #openstack-infra | 13:35 | |
| *** NithyaG has joined #openstack-infra | 13:35 | |
| *** mase_x200 has quit IRC | 13:36 | |
| *** ayoung has joined #openstack-infra | 13:41 | |
| *** alexpilotti has joined #openstack-infra | 13:43 | |
| *** koolhead17 has joined #openstack-infra | 13:44 | |
| *** e0ne has quit IRC | 13:50 | |
| *** jgallard_ has joined #openstack-infra | 13:50 | |
| *** jamespage_ has joined #openstack-infra | 13:51 | |
| *** e0ne has joined #openstack-infra | 13:51 | |
| *** alexpilotti has quit IRC | 13:52 | |
| *** jamespage_ has quit IRC | 13:52 | |
| *** jgallard_ has quit IRC | 13:53 | |
| *** ddieterly has quit IRC | 13:53 | |
| *** jgallard_ has joined #openstack-infra | 13:53 | |
| *** jgallard_ has quit IRC | 13:53 | |
| *** koolhead17 has quit IRC | 13:56 | |
| *** koolhead17 has joined #openstack-infra | 13:57 | |
| *** dustins has joined #openstack-infra | 13:58 | |
| *** groknix has quit IRC | 13:58 | |
| *** groknix has joined #openstack-infra | 13:59 | |
| *** bswartz has joined #openstack-infra | 13:59 | |
| *** cpowell has joined #openstack-infra | 14:00 | |
| *** dimsum__ has quit IRC | 14:01 | |
| *** koolhead17 has quit IRC | 14:02 | |
| *** dimsum__ has joined #openstack-infra | 14:02 | |
| *** otherwiseguy has joined #openstack-infra | 14:03 | |
| *** dkliban_afk is now known as dkliban | 14:03 | |
| *** emagana has joined #openstack-infra | 14:04 | |
| *** emagana has quit IRC | 14:09 | |
| *** chandankumar has quit IRC | 14:12 | |
| *** ryanpetrello has joined #openstack-infra | 14:12 | |
| *** esker has joined #openstack-infra | 14:12 | |
| *** esker has quit IRC | 14:13 | |
| *** esker has joined #openstack-infra | 14:13 | |
| *** davideagnello has joined #openstack-infra | 14:15 | |
| *** esker has quit IRC | 14:16 | |
| *** davideagnello has quit IRC | 14:20 | |
| *** dkranz has joined #openstack-infra | 14:26 | |
| *** ddieterly has joined #openstack-infra | 14:26 | |
| *** Sincler has joined #openstack-infra | 14:30 | |
| *** belmoreira has quit IRC | 14:31 | |
| *** jerryz has joined #openstack-infra | 14:32 | |
| *** jungleboyj has quit IRC | 14:33 | |
| *** BharatK has joined #openstack-infra | 14:34 | |
| *** jerryz1 has quit IRC | 14:35 | |
| *** amitgandhinz has joined #openstack-infra | 14:35 | |
| *** koolhead17 has joined #openstack-infra | 14:36 | |
| maishsk | nibalizer: | 14:36 |
| maishsk | ? | 14:36 |
| nibalizer | maishsk: yes? | 14:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 14:37 |
| maishsk | I am trying to understand how the CI works and uses Openstack compute for resource provisioning | 14:38 |
| maishsk | Is it done through Jenkins? | 14:38 |
| maishsk | Is there a Jenkins plugin that plugs into OpenStack? like the Vmware plugin? | 14:38 |
| jkt | maishsk: http://ci.openstack.org/nodepool.html | 14:39 |
| maishsk | This is what I was looking for.. - http://ci.openstack.org/nodepool/configuration.html#providers | 14:41 |
| maishsk | thanks jkt ! | 14:42 |
| *** mjturek has joined #openstack-infra | 14:42 | |
| *** jklare_ is now known as jklare | 14:44 | |
| fungi | maishsk: also you can see the template of our current production nodepool configuration file at http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 14:44 |
| *** Hal_ has joined #openstack-infra | 14:45 | |
| maishsk | thanks fungi | 14:45 |
| *** Hal_ has quit IRC | 14:45 | |
| *** maishsk has quit IRC | 14:45 | |
| *** mriedem has joined #openstack-infra | 14:46 | |
| *** signed8bit has joined #openstack-infra | 14:48 | |
| *** thedodd has joined #openstack-infra | 14:50 | |
| *** esker has joined #openstack-infra | 14:51 | |
| *** emagana has joined #openstack-infra | 14:53 | |
| *** kgiusti has joined #openstack-infra | 14:53 | |
| *** wuhg has quit IRC | 14:53 | |
| *** dangers_away is now known as dangers | 14:56 | |
| *** bhunter71 has joined #openstack-infra | 15:00 | |
| mtreinish | jogo: I am now | 15:00 |
| *** unicell has joined #openstack-infra | 15:05 | |
| *** esker has quit IRC | 15:10 | |
| *** esker has joined #openstack-infra | 15:10 | |
| *** rushiagr is now known as rushiagr_away | 15:11 | |
| *** xyang0 has joined #openstack-infra | 15:12 | |
| *** doug-fish has joined #openstack-infra | 15:12 | |
| *** prad has joined #openstack-infra | 15:13 | |
| *** erikwilson has joined #openstack-infra | 15:15 | |
| *** zz_jgrimm is now known as jgrimm | 15:15 | |
| *** esker has quit IRC | 15:18 | |
| *** otter768 has joined #openstack-infra | 15:18 | |
| *** koolhead17 has quit IRC | 15:19 | |
| *** koolhead17 has joined #openstack-infra | 15:19 | |
| *** beekneemech is now known as bnemec | 15:20 | |
| *** AlexF has joined #openstack-infra | 15:21 | |
| *** ayoung is now known as ayoung-afk | 15:22 | |
| *** r-daneel has joined #openstack-infra | 15:22 | |
| *** thedodd has quit IRC | 15:22 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Revert "move nova-tox-functional to experimental until there is content" https://review.openstack.org/136795 | 15:22 |
| mtreinish | fungi: so for subunit2sql bugs I go through storyboard now? | 15:23 |
| fungi | mtreinish: yep | 15:23 |
| mtreinish | ok, time to give this a try... | 15:24 |
| *** otter768 has quit IRC | 15:24 | |
| *** koolhead17 has quit IRC | 15:24 | |
| *** jerryz has quit IRC | 15:25 | |
| *** stevemar has joined #openstack-infra | 15:25 | |
| fungi | sdague: what's the story behind 136795? the commit that reverts is from yesterday, right? | 15:25 |
| *** funzo_ is now known as funzo | 15:26 | |
| *** jungleboyj has joined #openstack-infra | 15:28 | |
| sdague | fungi: yeh, basically, I actually got working on it today | 15:28 |
| fungi | sdague: wow, you're speedy | 15:29 |
| *** teran has quit IRC | 15:31 | |
| *** teran has joined #openstack-infra | 15:31 | |
| *** pcrews has joined #openstack-infra | 15:32 | |
| *** dimsum__ is now known as dims | 15:32 | |
| *** atiwari has joined #openstack-infra | 15:35 | |
| sdague | fungi: so what's the overall nodepool time for both the build up and tear down of a server? | 15:35 |
| tchaypo | ./toci_devtest.sh: line 173: cd: /opt/stack/new//os-net-config: No such file or directory | 15:35 |
| tchaypo | wheee | 15:36 |
| tchaypo | oh, here we go | 15:36 |
| tchaypo | https://www.irccloud.com/pastebin/sak6KepY | 15:36 |
| fungi | sdague: it varies extremely depending on provider and load | 15:37 |
| sdague | fungi: ranges? | 15:37 |
| fungi | sdague: for example, in rax dfw it can take up to an hour for nova to assign an ip address because they're severely constrained on address turn-over | 15:37 |
| fungi | whereas i've seen some nodes booted in ~5 minutes | 15:38 |
| sdague | gotcha | 15:38 |
| sdague | man... that's kind of special :( | 15:38 |
| fungi | nova instance reuse might help us there. i know mordred was looking at it at one point | 15:38 |
| sdague | so, as I was looking at the job definitions, we seem to be exploding on project tests that run in a couple of minutes. For instance, the docs team run 4 tests that individually take < 2m each | 15:39 |
| sdague | on every manuals change | 15:39 |
| mordred | fungi: yes, it's next on my list | 15:39 |
| sdague | and if we have substantial setup / teardown time, it seems like it would behoove us to be a little smarter there | 15:39 |
| *** rushiagr_away is now known as rushiagr | 15:40 | |
| mordred | sdague: ++ | 15:40 |
| fungi | sdague: also dibification may help. we've seen some providers exhibit much worse boot time from snapshots than from glance images, due to the storage backends in use | 15:40 |
| fungi | or more likely, due to the network characteristics between nova compute hosts and wherever snapshots are being stashed | 15:41 |
| zul | are you guys planning to do a release of pbr soon? | 15:41 |
| fungi | dhellmann: ^ ? | 15:41 |
| *** zz_dimtruck is now known as dimtruck | 15:41 | |
| clarkb | and dibification can continue now. I want to remove f21 because its noisy | 15:41 |
| mrmartin | re | 15:41 |
| clarkb | then get my nodepool change in for vpc images | 15:42 |
| clarkb | at that point hopefully we can focus on images | 15:42 |
| fungi | mrmartin: i have the groups.openstack.org cert purchased now. where's the change you had to reference it? | 15:42 |
| mrmartin | fungi, thats great :) we need some cert for groups-dev.openstack.org first | 15:43 |
| mrmartin | https://review.openstack.org/#/c/135708/ | 15:43 |
| fungi | mrmartin: looking | 15:43 |
| mrmartin | but this patch enabling ssl for groups-dev first, for groups.openstack.org we need another one | 15:44 |
| mrmartin | based on this. | 15:44 |
| fungi | k | 15:44 |
| *** bhunter71 has quit IRC | 15:44 | |
| mrmartin | so I suggest, to do the groups-dev cert deployment first, and if it works, go with the prod site. | 15:44 |
| *** matel has joined #openstack-infra | 15:44 | |
| *** ildikov has quit IRC | 15:45 | |
| mrmartin | and something else. :) I'm working on this askbot migration, and it is using postgresql as a db backend. do we have pgsql in rackspace dbaas ? | 15:45 |
| fungi | mrmartin: yep, i'll review in a bit to compare it against how we're doing self-signed certs for other dev sites | 15:45 |
| *** bhunter71 has joined #openstack-infra | 15:45 | |
| *** banix has joined #openstack-infra | 15:47 | |
| matel | Hi guys, I'm looking for some nodepool expertise, anyone around? | 15:47 |
| *** krtaylor has quit IRC | 15:47 | |
| *** emagana has quit IRC | 15:47 | |
| *** koolhead17 has joined #openstack-infra | 15:47 | |
| *** emagana has joined #openstack-infra | 15:47 | |
| *** jerryz has joined #openstack-infra | 15:47 | |
| clarkb | oh there is also a change to allow mixed dib and snapshot images against the same label | 15:48 |
| clarkb | that one is important for migrating to dib | 15:48 |
| clarkb | I will look them over and rebase as necessary today | 15:48 |
| sdague | fungi / clarkb: +A? - https://review.openstack.org/#/c/134620/ | 15:50 |
| clarkb | matel its usually best to just ask your question | 15:51 |
| matel | I have two changes to generate snapshot images | 15:51 |
| matel | https://review.openstack.org/97787 | 15:51 |
| matel | And https://review.openstack.org/97798 | 15:52 |
| *** mattfarina has joined #openstack-infra | 15:52 | |
| *** esker has joined #openstack-infra | 15:52 | |
| matel | These changes are required for XenServer CI. | 15:52 |
| fungi | skimming those, i wonder if we couldn't just build those via dib now rather than trying to shoehorn something like that into the snapshot style image generation | 15:53 |
| clarkb | fungi ++ | 15:53 |
| *** ayoung-afk is now known as ayoung | 15:53 | |
| matel | fungi: Can you run a "VM" inside DIB? | 15:54 |
| clarkb | though would not be surprised if xenserver needs more than achroot to build | 15:54 |
| matel | It does need more | 15:54 |
| matel | That's why I rebased these changes. | 15:54 |
| fungi | matel: oh, you can't just assemble that by downloading/installing things into a loopback-mounted filesystem in a chroot? | 15:54 |
| *** krtaylor has joined #openstack-infra | 15:54 | |
| matel | fungi: You can't do that. You have to run the XenServer installer. | 15:54 |
| jeblair | matel: what does the installer do? | 15:55 |
| matel | jeblair: It's a custom installer script, inside an initrd. | 15:55 |
| * fungi notes that's an answer to a different question | 15:57 | |
| matel | jeblair: I don't know exactly what it does. | 15:57 |
| matel | jeblair: I can dig out the sources, and reverse engineer that, but that would be a much bigger project | 15:57 |
| *** BharatK has quit IRC | 15:58 | |
| openstackgerrit | Bradley Klein proposed openstack-infra/project-config: Add puppet-monasca acls and review group. https://review.openstack.org/136432 | 15:58 |
| jeblair | matel: the current way of doing xenserver installs is very complicated and fragile -- the fact that we needed to do that kind of work to get the images we wanted was a big part of why we wanted to move nodepool to dib | 15:58 |
| *** kgiusti has quit IRC | 15:58 | |
| *** kgiusti has joined #openstack-infra | 15:59 | |
| fungi | just worth noting that debian/ubuntu, rhel/centos, et cetera installers similarly run from an in-memory virtual filesystem, so that's not unusual, but people have also written tools to bootstrap them from running operating systems as well | 15:59 |
| krotscheck | Storyboard meeting in #openstack-meeting-3 | 16:00 |
| matel | fungi: we don't have such installer for XenServer | 16:00 |
| jeblair | matel: i believe in the long run, we don't want to have nodepool using running vms to create images, and only use dib in the future; i think it would be worth spending some time thinking about what it would take to get a xenserver image with dib | 16:00 |
| *** mpavlase has joined #openstack-infra | 16:01 | |
| matel | jeblair: I completely agree with that, and I think that's the correct way of doing it, however, I doubt that we'll have the resources to mimic the behavior of the installer inside a chroot any time soon. | 16:01 |
| fungi | so, can't we do this with a downloaded xenserver base image and a very thin dib element set to customize it? | 16:02 |
| jeblair | fungi: that seems reasonable -- matel: that's another advantage of dib -- we don't have to start with what our cloud providers give us, we can start with something that already exists | 16:03 |
| matel | fungi: looks like "sysprep" ing a XenServer, right? | 16:03 |
| jkt | hmm, anyone got success with using gertty to connect to gerrit-review.googlesource.com? | 16:03 |
| fungi | matel: more like retrieve a xenserver image, mount it on a loopback, modify files present in it, then repack the image and use that | 16:04 |
| jkt | their web UI doesn't show me my HTTP password, just some, er, crap straight for git | 16:04 |
| *** davideagnello has joined #openstack-infra | 16:04 | |
| *** enikanorov has quit IRC | 16:04 | |
| *** juice has quit IRC | 16:04 | |
| matel | fungi: That still leaves us with one problem: launching that instance. XenServer itself does not know about the cloud - it can't communicate with the agent. | 16:05 |
| jkt | eh, auth-type: basic | 16:05 |
| jkt | another PEBKAC on my side today :( | 16:05 |
| *** bhunter71 has quit IRC | 16:05 | |
| fungi | matel: that modified xenserver image gets uploaded to glance and we nova-boot it. what else does it need to know? | 16:05 |
| sdague | hmmmm.... why are we doing a puppet apply on centos6 on zuul layout changes? | 16:06 |
| openstackgerrit | Merged openstack-infra/project-config: Remove python26 jobs from various projects https://review.openstack.org/129435 | 16:06 |
| matel | fungi: first, the image boots an Ubuntu, that has an agent inside. This agent will know the instance's address, and sets these parameters to XenServer, and re-boots the VM to XenServer. | 16:06 |
| openstackgerrit | Merged openstack-infra/project-config: Update ironic pxe job names to reflect voting status https://review.openstack.org/134620 | 16:06 |
| *** juice has joined #openstack-infra | 16:07 | |
| matel | johnthetubaguy: ping | 16:07 |
| jeblair | fungi, clarkb, pleia2: any storyboard feedback? | 16:08 |
| *** davideagnello has quit IRC | 16:08 | |
| fungi | matel: why not just make an image that boots directly into xenserver? | 16:08 |
| *** garyh has joined #openstack-infra | 16:09 | |
| matel | fungi: You don't know your IP address - XenServer is not cloud aware, it's HVM | 16:09 |
| matel | fungi: the alternate route would be to use config drive. | 16:09 |
| fungi | matel: you're saying the ip address of the instance has to be embedded somewhere in teh xenserver filesystem before the instance boots xenserver? | 16:11 |
| matel | fungi: exactly | 16:11 |
| *** enikanorov has joined #openstack-infra | 16:11 | |
| matel | fungi: that's why we first boot the ubuntu, inject the IP, and reboot the box, and on next reboot XenServer picks that up. | 16:12 |
| fungi | matel: that suggests that if you want to change the ip address of a xenserver in production you have to rerun the installer on your server? sounds unbelievably painful | 16:12 |
| matel | You are not running the full installer at that point, you just re-configure the IP address. | 16:12 |
| fungi | matel: but it needs a reboot to change its ip address? | 16:13 |
| *** yfried_ has quit IRC | 16:14 | |
| *** tonytan4ever has joined #openstack-infra | 16:14 | |
| matel | fungi: No, it doesn't. I need to re-boot it, because In my image I have two operating systems: A cloud aware ubuntu and a XenServer. First the partition with Ubuntu is active. That has the agent, gets the IP, modifies the boot loader, and reboots | 16:14 |
| fungi | matel: so this is a multi-partition block device? or a virtual block device inside another block device? | 16:15 |
| *** amitgandhinz has quit IRC | 16:16 | |
| *** achanda has joined #openstack-infra | 16:16 | |
| fungi | matel: i'm mostly asking why we can't just boot the xenserver and change its ip address configuration once it's up, and skip the ubuntu partition, the agent therein, and the extra reboot | 16:16 |
| matel | The layout of the image is a multi-partition block device with 3 partitions. One for Ubuntu, one for XenServer, and one for XenServer's storage, where the VMs live (that is an embedded block device, this is where devstack bits live) | 16:17 |
| sdague | hmmm... there are a relatively small number of nodes actually running tests right now, which seems quite odd. | 16:17 |
| matel | fungi: the image runs xen and that prevents us from communicating with the xen under that one. | 16:18 |
| matel | fungi: And xenstore is used for communication | 16:19 |
| *** nfedotov has quit IRC | 16:19 | |
| fungi | matel: it sounds like you're describing the current design while i'm asking why we can't change the design. why can't you use a statically-assigned loopback address on the xenserver for communicating with the xen instance it's managing? why does it have to be the ever-changing nova instance i address from the service provider instead? | 16:19 |
| mordred | matel, fungi I'm hacking in dib ... maybe I take a look this afternoon | 16:19 |
| *** amitgandhinz has joined #openstack-infra | 16:20 | |
| fungi | matel: if you can design it so that it's not dependent on the nova-assigned interface ip address then it sounds like all this other bootstrapping complexity goes away? | 16:21 |
| matel | fungi: Rackspace agent uses xenstore to communicate the IP, how can you work around that? | 16:21 |
| matel | fungi: How can I reach the instance from the outside? | 16:22 |
| *** dannywilson has joined #openstack-infra | 16:22 | |
| fungi | matel: oh, so the problem is not that you need that interface for internal communication between the cubcomponents you're testing, but rather that you need some mechanism to set a static ip address in the system so that it can configure the interface? | 16:23 |
| *** Longgeek has quit IRC | 16:23 | |
| fungi | s/cubcomponents/subcomponents/ | 16:24 |
| matel | Yes, I need to know that IP to configure the interface - so that the system is accessible. | 16:24 |
| fungi | so this takes us back to rackspace's file injection or dhcp in hpcloud (hpcloud is dhcp right?) | 16:24 |
| matel | fungi: DHCP or config drive would be nice. | 16:25 |
| clarkb | yes hpcloud is dhcp | 16:25 |
| *** timrc-afk is now known as timrc | 16:26 | |
| mordred | isn't rax also dhcp now? | 16:27 |
| mordred | if you set it properly? | 16:27 |
| mordred | like, if you set the image to be a non-agent instance | 16:27 |
| mordred | like we are doing for our other dib instances | 16:27 |
| clarkb | you can do that? | 16:28 |
| mordred | those glance meta params | 16:28 |
| matel | That would be excellent. | 16:28 |
| clarkb | we havent dibed anything in rax yet | 16:28 |
| mordred | one of them informs rackspace that you are not going to be running the rackspace agent on this image | 16:28 |
| clarkb | mordred I think theory is yes maybe | 16:28 |
| clarkb | reality is who knows :) | 16:28 |
| mordred | so - at that point, the choices would be config-drive or dhcp | 16:28 |
| matel | Anyone from rax here? | 16:28 |
| mordred | both of which are available to us | 16:28 |
| mordred | so - I think this is simply going to be a matter of trying some things | 16:29 |
| mordred | which I'm willing to do | 16:29 |
| johnthetubaguy | matel: you called? | 16:29 |
| *** achanda has quit IRC | 16:29 | |
| *** armax has joined #openstack-infra | 16:29 | |
| matel | johnthetubaguy: does rax support DHCP / config drive? | 16:29 |
| clarkb | mordred but we are close to rax dib so should know soon | 16:29 |
| mordred | clarkb: ++ | 16:29 |
| johnthetubaguy | matel: you can't inject networking via config drive right now, certainly no DCHP support | 16:29 |
| matel | johnthetubaguy: thanks | 16:29 |
| mordred | johnthetubaguy: so the _only_ way to inject networking is via nova-agent? | 16:29 |
| fungi | matel: it does support config drive... i just mounted /dev/xvdd on one of my rax instances and am poking around inside it out of curiosity | 16:29 |
| openstackgerrit | James Polley proposed openstack-infra/devstack-gate: Add os-net-config to the list of packages we clone https://review.openstack.org/136811 | 16:30 |
| johnthetubaguy | matel: the plan is to do config drive network injection soon ish, but I can't promise anything, its used by on metal, but not VMs right now | 16:30 |
| *** mudassirlatif has joined #openstack-infra | 16:30 | |
| mordred | johnthetubaguy: but what about for custom images? | 16:30 |
| johnthetubaguy | fungi: it does config drive, but we don't but the correct network config in there right now | 16:30 |
| fungi | matel: any way we could install nova-agent into the xenserver image? | 16:30 |
| johnthetubaguy | mordred: nope, its to do with how we set the OVS rules, sadly | 16:31 |
| mordred | zomg | 16:31 |
| johnthetubaguy | quite | 16:31 |
| mordred | so we're going to have to install nova-agent into our dib instances? | 16:31 |
| clarkb | apparently | 16:31 |
| * mordred smashes head against wall | 16:31 | |
| * mordred screams | 16:31 | |
| * mordred throws things | 16:31 | |
| * mordred cries | 16:31 | |
| matel | fungi: I can install something that read config drive, yes | 16:32 |
| johnthetubaguy | yeah, I mean you can try without the agent, using the image property use_xenapi_agent = False | 16:32 |
| * mordred resigns himself to a world where he can't have nice things | 16:32 | |
| johnthetubaguy | and we do attempt to inject the networking | 16:32 |
| johnthetubaguy | but I am told it doesn't work, but I have not tested it myself | 16:32 |
| mordred | johnthetubaguy: so you do file injection when we do use_xenai_agent = False | 16:32 |
| fungi | clarkb: so it's working for a variety of our images now... do we explicitly install nova-agent into them, or do ubuntu et al include it preinstalled and running in their official base images? | 16:32 |
| johnthetubaguy | mordred: no file injection, its just injecting network data into config drive | 16:32 |
| matel | mordred: config drive has all the details, why would we go for file inject? | 16:32 |
| clarkb | fungi it would be explicut likely a dib element | 16:33 |
| mordred | fungi: it's installed into the rax base images by default | 16:33 |
| johnthetubaguy | mordred: I have a feeling those compute nodes don't have the template file in place | 16:33 |
| *** emagana has quit IRC | 16:33 | |
| johnthetubaguy | the one to generate the interfaces file | 16:33 |
| fungi | mordred: though we're not necessarily using the rax images when we dib | 16:33 |
| mordred | fungi: we arent' dib-ing on rax yet | 16:33 |
| clarkb | fungi we dont dib rax yet | 16:33 |
| fungi | ohhhhh | 16:33 |
| clarkb | because there are about 100 things broken about it | 16:33 |
| fungi | for some reason i thought we were booting from dib on rax already | 16:33 |
| *** emagana has joined #openstack-infra | 16:33 | |
| mordred | not yet | 16:33 |
| fungi | but right, there's also the glance issue | 16:33 |
| mordred | so - this problem is quite literally the next one on my plate | 16:34 |
| mordred | so I'll poke at options | 16:34 |
| mordred | and let everyone know | 16:34 |
| clarkb | mordred can you review my nodepool changes then? | 16:34 |
| mordred | yup | 16:34 |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Add API call to return task statuses https://review.openstack.org/135221 | 16:34 |
| johnthetubaguy | fungi: whats the glance issue? | 16:35 |
| fungi | matel: so, yes, this is something we basically need to solve for anything we dib on rax, and perhaps as a result a much simpler and less fragile xenserver image build process could leverage the same solution | 16:35 |
| *** mrmartin has quit IRC | 16:35 | |
| *** mrmartin has joined #openstack-infra | 16:35 | |
| *** ashaeron has quit IRC | 16:35 | |
| matel | fungi: We can avoid the reboot, if I use the config drive to configure the xenserver image. Question is: can I build / re-shape a xenserver image with DIB? | 16:36 |
| matel | fungi: The image layout of a typical installation looks like: first partition with dom0's filesystem, second partition is an ext3, and the disk images are on that. | 16:36 |
| johnthetubaguy | matel: ah, I forgot about XenServer not having access to xenstore, I remember now, yuck | 16:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 16:37 |
| clarkb | glance endpoint is wrong iirc | 16:37 |
| fungi | johnthetubaguy: i think it had to do with the | 16:37 |
| mordred | clarkb: I'll review yours if you review mine :) | 16:37 |
| fungi | yeah what clarkb said | 16:37 |
| clarkb | and vpv images are bad | 16:37 |
| mordred | clarkb: it's not glance endpoint | 16:37 |
| mordred | clarkb: it's version | 16:37 |
| clarkb | *vpc | 16:37 |
| fungi | v1 vs v2 discrepancy. it reports as one but uses the other, right? | 16:37 |
| matel | johnthetubaguy: that's it | 16:37 |
| mordred | fungi: it reports NEITHER | 16:37 |
| *** emagana has quit IRC | 16:37 | |
| mordred | and glanceclient defaults to the other value | 16:37 |
| mordred | this is because glance does not report versions | 16:38 |
| fungi | oh, and fallback is to the wrong api version | 16:38 |
| mordred | and expects the user to explicitly tell it | 16:38 |
| mordred | yah | 16:38 |
| johnthetubaguy | fungi: hmm, that was news to me... | 16:38 |
| johnthetubaguy | fungi: sounds like an upstream glance bug? or am I missing something? | 16:38 |
| mordred | I've got the overrides in shade for reference if we need it | 16:38 |
| mordred | johnthetubaguy: it's that upstream glance does an evil thing - in this case, it is not rax fault | 16:38 |
| sdague | mordred: flaper87 is fixing some of those glance client terribles | 16:38 |
| *** tsg_ has joined #openstack-infra | 16:38 | |
| mordred | sdague: yes, thank god | 16:38 |
| flaper87 | o/ | 16:39 |
| mordred | but it still won't fully help if glance doesn't put a version in the service catalog | 16:39 |
| johnthetubaguy | mordred: OK, cool, needs fixing either way, but OK | 16:39 |
| flaper87 | mordred: sdague please, send them all my way | 16:39 |
| * mordred hands flaper87 a beer | 16:39 | |
| * mordred stands on mountain top and shouts "I should not have to know the API version the server is running in advance" | 16:39 | |
| flaper87 | and btw, in name of all the people that haven't done this (and probably won't ever do it), I want to say I'M SO FUCKING SORRY FOR SUCH A TERRIBLE AND INCONSISTENT API | 16:39 |
| mordred | flaper87: thank you for working on it | 16:40 |
| fungi | out of curiosity how does hpcloud work around it? | 16:40 |
| mordred | fungi: what do you mean? | 16:40 |
| clarkb | they accpet the fallback version likely | 16:40 |
| *** vryzhenkin has quit IRC | 16:40 | |
| mordred | fungi: hpcloud is running the version that glance defaults to | 16:40 |
| fungi | aha | 16:40 |
| mordred | so it's the same problem | 16:40 |
| fungi | right, that | 16:40 |
| mordred | it just happens to fail open | 16:40 |
| clarkb | so rax could fix this | 16:40 |
| clarkb | but its still a bug | 16:41 |
| mordred | yah | 16:41 |
| fungi | "fix" it by using v1 i guess? | 16:41 |
| clarkb | ya but as a client I dont care :) | 16:41 |
| clarkb | I just was images | 16:41 |
| *** emagana has joined #openstack-infra | 16:41 | |
| johnthetubaguy | fungi: oh, right we only expose v2 as we need protected properties | 16:41 |
| mordred | what clarkb said | 16:41 |
| *** david-lyle_afk is now known as david-lyle | 16:41 | |
| mordred | johnthetubaguy: which is fine - it's that this version should be in the service catalog - which for bonghits reasons it's not | 16:42 |
| clarkb | but in addition to that we need vpc images that work | 16:42 |
| mordred | clarkb: what's the vpc issue? | 16:42 |
| clarkb | mordred we have to support all that in nodepool and I had to patch dib | 16:43 |
| fungi | oh, right, so in my personal scripts i shadow openstackclient's image calls and pass them off to glanceclient with OS_IMAGE_API_VERSION and OS_IMAGE_URL overridden to work around it | 16:43 |
| clarkb | and we may not get images that resize properly | 16:43 |
| matel | fungi: The image layout of a typical installation looks like: first partition with dom0's filesystem, second partition is an ext3, and the disk images are on that in vhd format - can we build such images with DIB? | 16:43 |
| clarkb | mordred it isnt clear to me if qemu-img does the right thing there | 16:44 |
| clarkb | but one step at a time | 16:44 |
| clarkb | also those images are massive... | 16:44 |
| clarkb | ~5GB | 16:44 |
| fungi | matel: i'm wondering if that would need multiple dib runs (one to manage the dom0 filesystems and one for modifying the inner vhd images) | 16:45 |
| mordred | clarkb: questions ... a) do we care about resize b) does it help if we go AMI/AKI/ARI format instead of all-in-one? | 16:45 |
| fungi | matel: the vhd images are the ones which need altering before dom0 boots? | 16:46 |
| clarkb | mordred a-yes because we use / b-no idea | 16:46 |
| clarkb | mordred rax docs say use vpc | 16:46 |
| fungi | matel: or are they something which could be provided as-is? | 16:46 |
| clarkb | unsure if we have any other options | 16:46 |
| ddieterly | not sure if this is the right place to ask this... is zuul sick? | 16:46 |
| mordred | ddieterly: it is ALWAYS the right place to ask that question | 16:46 |
| ddieterly | jobs seem to be stacking up | 16:46 |
| ddieterly | mordred: oh, good | 16:46 |
| matel | fungi: no, dom0 (the first partition), but I guess we want to have a basic ubuntu/centos for devstack, and that goes to the vhd. | 16:47 |
| mordred | ddieterly: we may just be busy - looking | 16:47 |
| mordred | ddieterly: we seem to just be busy | 16:47 |
| *** alexpilotti has joined #openstack-infra | 16:47 | |
| ddieterly | mordred: ok, thanks for checking | 16:47 |
| fungi | matel: but we wouldn't need to create/modify the vhd images during the dom0 image creation? in which case we can just treat them as normal files and reuse them if they're already present on the base image we're starting from or download them and add them if necessary | 16:48 |
| *** mudassirlatif has quit IRC | 16:48 | |
| mordred | clarkb: ^^ we do have a rather small number of nodes in nodepool - known problem? | 16:48 |
| jogo | clarkb: if you have a few minutes I can use some help debugging https://review.openstack.org/#/c/136596/4 | 16:48 |
| * tchaypo wonders what gymnastics we have to do in order to check things like https://review.openstack.org/#/c/136811/ | 16:48 | |
| jogo | not sure what went wrong | 16:48 |
| clarkb | mordred not known to me | 16:48 |
| fungi | mordred: clarkb: ddieterly: zuul says it's aware of around 250-300 jobs currently running in progress | 16:49 |
| matel | fungi: yes, you can do that. | 16:49 |
| fungi | mordred: clarkb: ddieterly: with ~1000 pending | 16:49 |
| ddieterly | fungi: ok | 16:50 |
| fungi | also nodepool seems to have decided it should boot a bunch of additional nodes to help with the demand, but they're not done being added yet | 16:50 |
| *** davideagnello has joined #openstack-infra | 16:51 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Add autokick.py https://review.openstack.org/136820 | 16:51 |
| mtreinish | jogo: what exactly did you think you broke there? Because it looks like a couple of weird things happened, like largeops was running all the tests | 16:51 |
| jogo | mtreinish: oh that is intentional | 16:52 |
| jogo | mtreinish: https://review.openstack.org/#/c/136596/4/devstack-vm-gate.sh,cm | 16:52 |
| fungi | also the pending changes in the post pipeline (as well as the sparkline for the merge-check pipeline) show quite a number of changes getting merged out of the gate as well | 16:52 |
| mtreinish | and stable failures, although i'm not sure why from a quick glance | 16:52 |
| *** arxcruz has quit IRC | 16:52 | |
| mtreinish | jogo: oh, yeah it's because this is on top of the test patch | 16:52 |
| jogo | so the patch should be adding ssh-hostkeys by hostname | 16:53 |
| sdague | clarkb: can I get a +A on this - https://review.openstack.org/#/c/136795/ ? | 16:53 |
| jogo | mtreinish: but in http://logs.openstack.org/96/136596/4/experimental/check-tempest-dsvm-aiopcpu/7def73d/console.html#_2014-11-23_02_37_26_672 it isn't working | 16:53 |
| *** AlexF has quit IRC | 16:54 | |
| *** afazekas has quit IRC | 16:54 | |
| *** JayJ has joined #openstack-infra | 16:54 | |
| *** e0ne has quit IRC | 16:55 | |
| jogo | mtreinish: looks like its not able to resolve the name of the second node (slave) | 16:55 |
| jogo | even though /etc/hosts contains that information | 16:56 |
| mordred | clarkb: it' nodepool patches from you I need to look like? | 16:56 |
| mordred | look at | 16:56 |
| *** otherwiseguy has quit IRC | 16:57 | |
| *** bhunter71 has joined #openstack-infra | 16:58 | |
| clarkb | mordred: ya let me dig them up | 16:58 |
| clarkb | they are older and may need rebasing or other love | 16:59 |
| *** isaacb has quit IRC | 16:59 | |
| clarkb | mordred: https://review.openstack.org/#/c/130878/ definitely needs a new commit message. it tests and fixes that behavior not just tests it. https://review.openstack.org/#/c/126747/ is the change to allow us to use both qcow2 and vpc images | 16:59 |
| *** nikhil_k is now known as nikhil_k|vacay | 17:00 | |
| mordred | clarkb:126747 lgtm - rebase and I'll get the +2 on there | 17:00 |
| clarkb | cool doing that now | 17:00 |
| * clarkb notes the error rates as reported by nodepool are non trivial right now | 17:01 | |
| clarkb | may explain the lack of test nodes | 17:01 |
| clarkb | *cloud error rates | 17:01 |
| *** sarob has joined #openstack-infra | 17:01 | |
| mordred | clarkb: has_snapshot and has_image to me read like they should return boolean | 17:02 |
| pleia2 | good morning | 17:02 |
| *** MaxV has quit IRC | 17:02 | |
| mordred | just, fwiw | 17:03 |
| mordred | morning pleia2 ! | 17:03 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Support multiple image formats in a diskimage https://review.openstack.org/126747 | 17:03 |
| clarkb | mordred: ^ | 17:03 |
| clarkb | mordred: ya I think renaming those vars was suggestined | 17:03 |
| clarkb | *suggested | 17:03 |
| clarkb | mordred: something like snapshot_list and image_list? | 17:03 |
| clarkb | I will refresh my memory on that code and can update that | 17:03 |
| *** chandankumar has joined #openstack-infra | 17:04 | |
| mordred | clarkb: yeah | 17:04 |
| mordred | or just snapshots and images | 17:04 |
| mordred | but I dont' _really_ care strongly | 17:04 |
| clarkb | well I need a new patchset regardless so can rename to something better | 17:04 |
| *** MaxV has joined #openstack-infra | 17:05 | |
| *** teran has quit IRC | 17:05 | |
| clarkb | but with these two changes on top of nodepool trusty server we should be able to start rolling on dib again | 17:05 |
| clarkb | there was also ianw's change to support glance meta vars but I think that merged | 17:05 |
| clarkb | yup 34335b5fbff40c0129ac641aac4179a2275ee338 | 17:06 |
| mordred | clarkb: did the dib changes land then? | 17:06 |
| clarkb | mordred: yes latest dib has my change and ghe's change in it | 17:06 |
| clarkb | and that is what is installed on new nodepool | 17:06 |
| mordred | cool | 17:06 |
| clarkb | I might've helped greghaynes do that release in a bar >_> | 17:06 |
| greghaynes | :) | 17:06 |
| mordred | perfect. what could possibly go wrong | 17:07 |
| *** davideagnello has quit IRC | 17:08 | |
| mordred | clarkb, fungi: btw - if you want to laugh - look at the section starting at line 90 here: https://review.openstack.org/#/c/136597/7/elements/centos-minimal/root.d/08-rinse | 17:10 |
| mordred | dtroyer: fwiw, this ^^ fixed my problem with centos yesterday | 17:10 |
| * dtroyer is again glad OpenWRT exists… | 17:12 | |
| mordred | clarkb: dib_image.filename + image_type | 17:12 |
| mordred | clarkb: that seems like it's going to make a strangely named file | 17:12 |
| fungi | dtroyer: because ddwrt is so awfully assembled from entirely non-free blobs? | 17:13 |
| *** radez_g0n3 is now known as radez | 17:13 | |
| dtroyer | fungi: because it doesn't have crowd-following Insanity as a Service | 17:13 |
| mordred | clarkb: also, I'm not sure I see where you compose a filename from filename and image_type | 17:14 |
| fungi | dtroyer: that, definitely | 17:14 |
| mordred | dtroyer: maybe we shoudl start using openwrt as our base os | 17:14 |
| clarkb | mordred: on the has_diskimage front I think renaming it has helped me find a bug \o/ | 17:14 |
| dtroyer | mordred: don't think I haven't started that already | 17:14 |
| *** emagana has quit IRC | 17:15 | |
| clarkb | mordred: dib will take a non suffixed file name then add the file suffix for each file type it writes out | 17:15 |
| sdague | mordred: so... there is a reason why the project I wrote years ago to do the same thing as dib is something that got abandoned :) | 17:15 |
| clarkb | mordred: so when we call dib we use it without a suffix | 17:15 |
| *** emagana has joined #openstack-infra | 17:15 | |
| clarkb | so filename there is dibs concept of a filename | 17:16 |
| mordred | clarkb: ah, ok. cool. thanks | 17:17 |
| mordred | sdague: :) | 17:17 |
| dhellmann | it looks like the oslo-messaging-release group is empty in gerrit, could I get someone to add me, please? https://review.openstack.org/#/admin/groups/463,members | 17:17 |
| *** amuller has quit IRC | 17:18 | |
| sdague | ... if zuul remains with only 150 active nodes all day... it's going to be a long day. I wonder why it can't seem to get beyond that | 17:18 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 17:18 |
| sdague | did we lose a cloud? | 17:19 |
| fungi | Failed to fetch http://security.ubuntu.com/ubuntu/dists/trusty-security/main/i18n/Translation-en Hash Sum mismatch | 17:19 |
| mordred | fungi: that's the saddest thing I've ever heard | 17:19 |
| jogo | mtreinish: any ideas? | 17:19 |
| *** tonytan4ever has quit IRC | 17:19 | |
| fungi | this is not shaping up to be a high-throughput day | 17:19 |
| *** otter768 has joined #openstack-infra | 17:20 | |
| mordred | fungi, jeblair: oh - btw - I found a cantrip for telling apt to not grab translations files | 17:20 |
| *** emagana has quit IRC | 17:20 | |
| clarkb | sdague: cloud error rates are high | 17:20 |
| mordred | maybe we should put it on our stuff, since those are always goign to just be overhead | 17:20 |
| sdague | it's like there are no bare nodes in play | 17:20 |
| *** ala_ has quit IRC | 17:20 | |
| clarkb | sdague: across the board | 17:20 |
| *** winston-d_ has joined #openstack-infra | 17:20 | |
| sdague | clarkb: ok, fun | 17:20 |
| *** alexpilotti has quit IRC | 17:20 | |
| fungi | nodepool seems to be aware of 8 bare nodes in use | 17:22 |
| fungi | with 4 building | 17:22 |
| jeblair | clarkb: both clouds? | 17:22 |
| fungi | oh, and 8 ready | 17:22 |
| clarkb | jeblair: ya the graphs mordred had made seem to imply that | 17:22 |
| clarkb | I haven't looked at nodepool logs yet | 17:22 |
| winston-d_ | jeblair: hi, a quick question about gertty search & local check-out/cherry-pick functions. these don't seem to work on my Mac. | 17:23 |
| sdague | fungi: are there any bare-trusty? | 17:23 |
| fungi | 205 devstack nodes in use, 93 ready, 61 being deleted and 1 building | 17:23 |
| sdague | the only ones I can see look like bare-precise on chef jobs | 17:23 |
| *** koolhead17 has quit IRC | 17:24 | |
| fungi | sdague: for bare-trusty we have 2 building, 4 ready and 5 in use | 17:24 |
| mtreinish | jogo: hmm, not sure I do. I see where it adds the hostname to /etc/hosts right above ssh-keyscan failure | 17:24 |
| *** otter768 has quit IRC | 17:24 | |
| *** koolhead17 has joined #openstack-infra | 17:24 | |
| clarkb | fungi: hrm we should have bare-trusty images available but maybe we don't and are hitting quota issues for that type? | 17:24 |
| fungi | clarkb: i'm hunting in logs | 17:25 |
| mtreinish | jogo: I guess I would suggest catting things (like /etc/hosts) and adding status checks around the failed call to make sure the system state is what you think it is | 17:25 |
| fungi | clarkb: but usually quota issues cause huge numbers to show up in building | 17:25 |
| jeblair | winston-d_: that's strange. i don't have a mac to test with. i know there was recently a new release of gitpython, if you installed it within the last week or so, perhaps it is behaving differently. (i haven't tested that) | 17:25 |
| *** ivar-lazzaro has joined #openstack-infra | 17:25 | |
| *** mudassirlatif has joined #openstack-infra | 17:25 | |
| fungi | clarkb: we've got tracebacks from the DiskImageBuilderThread by the way | 17:26 |
| *** jpich has quit IRC | 17:26 | |
| fungi | likely not related to our node starvation issues however | 17:26 |
| *** ivar-lazzaro has quit IRC | 17:26 | |
| *** koolhead_ has joined #openstack-infra | 17:26 | |
| *** koolhead17 has quit IRC | 17:27 | |
| winston-d_ | jeblair: hmm, this version of gertty has been installed for more than one month, works pretty well by the way. :) let me grab a Linux VM and test those functions. | 17:27 |
| *** ivar-lazzaro has joined #openstack-infra | 17:27 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Allow labels to have snapshot and dib images https://review.openstack.org/130878 | 17:27 |
| clarkb | mordred: ^ renamed the bug I thought was a bug wasn't really and just needed a better comment so it has that now | 17:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 17:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add debootstrap and rinse to nodepool https://review.openstack.org/136598 | 17:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Make apt skip grabbing translations https://review.openstack.org/136837 | 17:28 |
| clarkb | fungi: yes building f21 images | 17:28 |
| mordred | clarkb: cool | 17:28 |
| clarkb | fungi: which is why I just want to rip out f21. https://review.openstack.org/#/c/136534/ does that match up with what you are seeing? | 17:28 |
| mordred | clarkb, fungi, jeblair: may or may not help, but https://review.openstack.org/136837 may cut down on the number of files we try to grab from the internets | 17:28 |
| fungi | clarkb: oh, fun! looks like maybe we have issues with at least one image in ord... it tried to nova boot with an image there that returned http 400 (image is not active) | 17:28 |
| mordred | fungi: whee! | 17:28 |
| jeblair | mordred: i think you forgot a file in the commit | 17:29 |
| fungi | clarkb: ahh, yes, f21 is the cause for the diskimage tracebacks | 17:30 |
| fungi | clarkb: too bad i'm the only +2 on that change so far | 17:31 |
| *** isaacb has joined #openstack-infra | 17:31 | |
| jeblair | i'm reviewing | 17:31 |
| fungi | clarkb: i'm going to delete image ba4c3302-1e18-460b-841a-c7cdbd5ea8d3 in rax-ord (it's the only one throwing this http 400) | 17:32 |
| clarkb | fungi: ok | 17:32 |
| clarkb | fungi: is that a bare-trusty image? | 17:32 |
| fungi | seems to be a devstack-trusty node, so probably not responsible for the bare images shortage | 17:32 |
| jeblair | clarkb: 136534 looks like a nodepool bug in the traceback; do you understand it? | 17:32 |
| clarkb | jeblair: sort of. exec is complaining that the data being passed in the env var is not of a byte type | 17:33 |
| clarkb | or string in this case because python2 | 17:33 |
| mordred | jeblair: BAH | 17:33 |
| clarkb | jeblair: I do not know why the vars used for f21 are different than the vars used to override tmpdir and cachedir locations | 17:33 |
| jeblair | clarkb: it looks like yeah.... that :) | 17:33 |
| clarkb | jeblair: likely has to do with yaml and its string types | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Make apt skip grabbing translations https://review.openstack.org/136837 | 17:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 17:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add debootstrap and rinse to nodepool https://review.openstack.org/136598 | 17:34 |
| mordred | jeblair: sorry. there you go | 17:34 |
| *** dims has quit IRC | 17:35 | |
| mordred | clarkb: btw - my ubuntu-minimal images are 245M | 17:35 |
| *** dims has joined #openstack-infra | 17:35 | |
| *** rushiagr is now known as rushiagr_away | 17:36 | |
| mordred | clarkb: I have not yet tried running the nodepool elements on top of it - I imagine most of the 5G is actually the data cache | 17:36 |
| fungi | clarkb: jeblair: also we seem to have maybe broken the nodepool cli's image-delete subcommand | 17:36 |
| fungi | taking a closer look | 17:37 |
| *** AlexF has joined #openstack-infra | 17:37 | |
| *** AlexF has quit IRC | 17:37 | |
| jeblair | clarkb: repr(d['diskimages'][-1]['env-vars']) | 17:37 |
| jeblair | "{'BASE_IMAGE_FILE': 'Fedora-Cloud-Base-20141029-21_Beta.x86_64.qcow2', 'DIB_IMAGE_CACHE': '/opt/dib_cache', 'DIB_CLOUD_IMAGES': 'http://download.fedoraproject.org/pub/fedora/linux/releases/test/21-Beta/Cloud/Images/x86_64/', 'TMPDIR': '/opt/dib_tmp'}" | 17:37 |
| jeblair | that looks sane to me | 17:37 |
| clarkb | mordred: it is, but the big annoyance is qcow2 is compressed and comes in under 3GB. vpc is not and is just bad | 17:37 |
| clarkb | jeblair: ya | 17:37 |
| *** emagana has joined #openstack-infra | 17:38 | |
| jeblair | clarkb: i agree this needs further offline debugging. +3ing. | 17:38 |
| *** dims has quit IRC | 17:40 | |
| jogo | mtreinish: hmm good idea, I tried things locally. but adding status checks is a good idea | 17:40 |
| fungi | ahh, nope, image-delete still works. for some reason it's just this image uuid being a pain | 17:40 |
| clarkb | fungi: as a temporary measure maybe start building a new image of that type in ord | 17:41 |
| clarkb | nodepool should seamlessly switch to using it once it is done building | 17:41 |
| fungi | for some reason snap_image in _deleteImage ends up being None for this one | 17:41 |
| fungi | so maybe nodepool got a success result for the snapshot action but then rax disappeared it out from under us and the nova image list is now lacking it | 17:42 |
| fungi | checking that theory now | 17:42 |
| *** shashankhegde has joined #openstack-infra | 17:43 | |
| fungi | and as suggested i've started an image-update of the same label+provider now in case this takes longer to clean up | 17:43 |
| *** otherwiseguy has joined #openstack-infra | 17:43 | |
| *** mikedillion has joined #openstack-infra | 17:44 | |
| winston-d_ | jeblair: well, tried on Linux, search/check out worked. And I did a uninstall/install gertty on Mac, still the same. | 17:45 |
| fungi | fwiw, looks like it retried a few dozen times before finally getting a response from the api endpoint | 17:45 |
| jeblair | winston-d_: can you file a bug about that on storyboard.openstack.org ? | 17:46 |
| *** andreykurilin_ has joined #openstack-infra | 17:46 | |
| jeblair | winston-d_: and any help you can provide toward diagnosing it or fixing it would be great :) | 17:47 |
| winston-d_ | jeblair: sure, but let me confirm with someone else is using gertty on Mac first. | 17:47 |
| clarkb | ok I think https://review.openstack.org/#/c/130878 and https://review.openstack.org/#/c/126747 are ready for review now (though still waiting on zuul to +1 each of them) | 17:48 |
| clarkb | but those are next step in dibification | 17:48 |
| *** mpaolino has quit IRC | 17:48 | |
| *** dims has joined #openstack-infra | 17:48 | |
| *** rushiagr_away is now known as rushiagr | 17:50 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove nova pylint https://review.openstack.org/136846 | 17:51 |
| jeblair | clarkb: ack will review | 17:51 |
| *** AlexF has joined #openstack-infra | 17:51 | |
| *** harlowja_away is now known as harlowja | 17:52 | |
| dhellmann | fungi, jeblair : it looked like you were in fire-fighting mode when I asked before. When things calm down, could one of you add me to the oslo-messaging-release group in gerrit, please? It's completely empty somehow, which will prevent us from releasing next week. | 17:53 |
| ddieterly | i'm still unable to see any progress with jobs in zuul | 17:53 |
| fungi | okay, after confirming nova image-list did not know about the offending snapshot, i removed its row from the snapshot_images table | 17:53 |
| *** mikedillion has quit IRC | 17:54 | |
| jeblair | dhellmann: on it | 17:54 |
| *** shashankhegde has quit IRC | 17:54 | |
| fungi | dhellmann: done | 17:54 |
| dhellmann | jeblair: thanks, no rush if you guys still have an issue | 17:54 |
| jeblair | drat :) | 17:54 |
| dhellmann | fungi: thanks! | 17:54 |
| dhellmann | heh | 17:54 |
| jeblair | dhellmann: confirmed that fungi did it! :P | 17:55 |
| dhellmann | jeblair, fungi : thanks! I've added the rest of the folks we need in that group so we're all set now. | 17:55 |
| *** mpavlase has quit IRC | 17:55 | |
| ddieterly | looks like jobs just keep piling up and jobs launched per hour is down | 17:55 |
| fungi | we're getting lots of node launches in error state in rax-dfw | 17:55 |
| fungi | though https://status.rackspace.com/ suggests it should be smooth sailing | 17:56 |
| fungi | ooh, here's a new one... ERROR nodepool.GearmanClient: Exception while listing functions [...] TimeoutError | 17:57 |
| fungi | maybe it's having trouble talking to zuul to determine demand? | 17:57 |
| *** tsg_ has quit IRC | 17:57 | |
| *** bhunter71 has quit IRC | 17:59 | |
| jeblair | oh | 18:00 |
| jeblair | is the old nodepool server still around? | 18:01 |
| *** sarob has quit IRC | 18:01 | |
| *** mpaolino has joined #openstack-infra | 18:01 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 18:01 |
| clarkb | jeblair: yes but nodepool shouldn't be running on it | 18:01 |
| jeblair | anyone have an ip handy? | 18:01 |
| clarkb | ya one sec | 18:01 |
| fungi | getting it | 18:01 |
| *** emagana has quit IRC | 18:01 | |
| clarkb | 192.237.211.91 | 18:01 |
| mordred | 192.237.211.91 | 18:02 |
| mordred | blast | 18:02 |
| mordred | clarkb beat me | 18:02 |
| fungi | that matches what i looked up too | 18:02 |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Initial Fedora 21 nodepool disk-image creation" https://review.openstack.org/136534 | 18:02 |
| mordred | 2001:4800:7813:0516:3bc3:d7f6:ff04:b863 | 18:02 |
| mordred | if you want ipv6 | 18:02 |
| *** sweston_ is now known as sweston | 18:02 | |
| *** bhunter71 has joined #openstack-infra | 18:02 | |
| *** emagana has joined #openstack-infra | 18:02 | |
| fungi | nodepoold is definitely not active on iot | 18:02 |
| fungi | it | 18:02 |
| clarkb | oh hrm did iptables not apply like I thought they would on zuul.o.o? | 18:02 |
| *** tonytan4ever has joined #openstack-infra | 18:02 | |
| *** prad has quit IRC | 18:03 | |
| clarkb | they must've otherwise nodepoold would've hung at startup like it was doing previously | 18:03 |
| clarkb | I can telnet from new nodepool to zuul over 4730 so that isn't the issue | 18:03 |
| *** e0ne has joined #openstack-infra | 18:04 | |
| *** derekh has quit IRC | 18:05 | |
| *** davideagnello has joined #openstack-infra | 18:05 | |
| jeblair | fungi: does that timeout happen a lot? | 18:05 |
| *** jlibosva has quit IRC | 18:06 | |
| *** emagana has quit IRC | 18:06 | |
| *** tsg_ has joined #openstack-infra | 18:07 | |
| fungi | jeblair: 3 times today since the log was rotated | 18:07 |
| *** ci-testing_ has quit IRC | 18:07 | |
| *** cpowell has quit IRC | 18:07 | |
| fungi | oh, the log was rotated at 17:39 | 18:07 |
| fungi | so that's 3 times in half an hour | 18:08 |
| fungi | or about every 10 minutes | 18:08 |
| *** isaacb has quit IRC | 18:08 | |
| *** mpaolino has quit IRC | 18:08 | |
| fungi | also i think we need to roll back the change that increased rotation frequency | 18:09 |
| *** cpowell has joined #openstack-infra | 18:09 | |
| fungi | oh, nevermind | 18:09 |
| fungi | the 3 days of history is because this server is 3 days old ;) | 18:10 |
| ddieterly | the check queue on zuul is continuing to grow. is anyone looking into that? | 18:10 |
| fungi | ddieterly: it's the entirety of what we're discussing in here | 18:10 |
| ddieterly | great, thanks | 18:11 |
| jeblair | i'm trying to figure out if this is related to the async io problems in geard (we raised the timeout on the zuul server, and there's a patch up to improve geard) | 18:11 |
| jeblair | fungi: however, as long as it isn't failing all the time, it should work well enough | 18:11 |
| matel | fungi: So with DIB in nodepool, do you expect DIB to build a node, that has repos cached, etc, or just a base operating system? | 18:12 |
| fungi | matel: build a node with cached repos et cetera, starting from a base operating system image of whatever origin we want | 18:13 |
| jeblair | huh | 18:13 |
| matel | fungi: so the starting point could be a xenserver image, which is presented in a qcow2 image format? | 18:13 |
| jeblair | the old nodepool server has an older version of gear, though i doubt the differences would account for that error | 18:14 |
| fungi | matel: that's what i was suggesting, yes | 18:14 |
| *** groknix has quit IRC | 18:14 | |
| *** groknix has joined #openstack-infra | 18:15 | |
| *** bhunter71 has quit IRC | 18:15 | |
| fungi | the node graph is showing an interestingly regular building hysteresis | 18:15 |
| matel | fungi: I can provide such an image, but the node installation has to happen "inside" the vhd file, which is sitting in the image's filesystem. | 18:16 |
| *** bhunter71 has joined #openstack-infra | 18:16 | |
| *** Bobba is now known as BobBall_AWOL | 18:16 | |
| fungi | matel: what is "node installation" in this sense? | 18:16 |
| jogo | both aiopcpu tests for a patch (nova-network and neutron) just failed with the same error: https://jenkins02.openstack.org/job/check-tempest-dsvm-neutron-aiopcpu/32/console | 18:16 |
| matel | fungi: all the openstacky stuff, cached repos, etc | 18:17 |
| winston-d_ | jeblair: ok, jgriffith helped me confirm that 'search' doesn't work for him on Mac neither. I'll create a bug on storyboard. | 18:17 |
| jogo | https://jenkins06.openstack.org/job/check-tempest-dsvm-aiopcpu/26/console | 18:17 |
| matel | fungi: so what could be possible is to have DIB build a VM - as if it was not inside xenserver, and produce say a vhd. This image, let's call it domU image would have all the openstacky bits | 18:17 |
| fungi | matel: ahh, any way that could be presented from the dom0? e.g. via hostfs or something? | 18:17 |
| matel | fungi: so what you are looking for is a way to mount domU's filesystem, given a full xenserver image, right? | 18:18 |
| jogo | clarkb fungi:^ | 18:19 |
| fungi | matel: as a possible workaround, yes | 18:19 |
| fungi | okay, the image rebuild in rax-ord finally completed | 18:19 |
| sdague | matel: hey, xenserver logs, where do you put your console.html when things fail, because I don't see that in the reports it's posting | 18:19 |
| matel | sdague: could you give me a log dir url? | 18:20 |
| sdague | http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/22/136822/1/32840/results.html | 18:20 |
| jeblair | fungi: i think we're seeing the periodicity of the nodepool main loop | 18:20 |
| fungi | jeblair: that's what i wondered | 18:20 |
| *** patrickeast has joined #openstack-infra | 18:20 | |
| jeblair | fungi: https://etherpad.openstack.org/p/magzsGkQrX | 18:20 |
| *** ci-testing_ has joined #openstack-infra | 18:21 | |
| fungi | jeblair: that looks like about the same frequency, yes | 18:21 |
| matel | fungi: In theory, you can mount the xenserver image's second partition as an ext3 to your system, and mount the vhd file from that. - I need to look at how to properly mount vhd files though - would that work? | 18:21 |
| winston-d_ | jeblair: not sure if this is the right format, but here's the story for gertty 'search' bug: https://storyboard.openstack.org/#!/story/2000024 | 18:21 |
| fungi | matel: yeah, that would be another possible solution | 18:21 |
| matel | sdague:looking at it | 18:22 |
| jeblair | fungi: updated etherpad | 18:22 |
| *** berendt has quit IRC | 18:23 | |
| fungi | jeblair: Demand from gearman: bare-trusty: 542 | 18:23 |
| fungi | et cetera | 18:23 |
| fungi | looking in the debug log | 18:23 |
| clarkb | jeblair: so its hanging in the mail loop somewhere? | 18:23 |
| matel | sdague: I guess this job did not run any tests, and you're interested why is that? | 18:23 |
| fungi | so it does seem to be finding the demand with numbers which match what we're seeing in the zuul status | 18:24 |
| jeblair | winston-d_: updated | 18:24 |
| fungi | devstack-trusty demand seems to be roughly equal to bare-trusty demand at this point | 18:25 |
| *** AJaeger has joined #openstack-infra | 18:25 | |
| fungi | according to the log | 18:25 |
| jeblair | fungi: yep. though we went from 17:44 to 18:00 with no demand info | 18:25 |
| jeblair | because of the timeouts | 18:25 |
| matel | sdague: I would expect run_tests.log to contain those bits, see this: http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/98/134598/1/31652/run_tests.log | 18:26 |
| AJaeger | Hi!one more strange thing in case you haven't noticed: we have a periodic-stable job since 36 hours in the queue ;( | 18:26 |
| *** andreykurilin_ has quit IRC | 18:26 | |
| fungi | but also the number of devstack-trusty nodes reported in existence is ~20x the number of bare-trusty nodes | 18:26 |
| jeblair | fungi, clarkb: should we try downgrading gear to the old version? | 18:26 |
| *** andreykurilin_ has joined #openstack-infra | 18:26 | |
| matel | fungi: Let me find a way to mount that partition - will get back to you tomorrow. | 18:26 |
| jeblair | fungi: i think that running without demand for a cycle will mess up the round robin allocator | 18:26 |
| *** MaxV has quit IRC | 18:27 | |
| fungi | ahh | 18:27 |
| fungi | this is likely the case | 18:27 |
| jeblair | the old allocator did not have that problem, but the new one maintains state | 18:27 |
| clarkb | jeblair: its a reasonably simple thing to try so +2 from me | 18:27 |
| *** chandankumar has quit IRC | 18:27 | |
| jeblair | it was 0.5.2 | 18:27 |
| fungi | yeah, downgrade gear and restart nodepool i guess | 18:27 |
| fungi | worth a shot | 18:27 |
| jeblair | clarkb: can you do that? i'm trying to dig into it further from another angle | 18:27 |
| *** melwitt has joined #openstack-infra | 18:28 | |
| clarkb | ya I can do that | 18:28 |
| * clarkb starts now | 18:28 | |
| jeblair | ianw: ping | 18:28 |
| matel | sdague: did that help? | 18:28 |
| jeblair | in the long run, we can't depend on things always working, so we need the allocator to not behave that way | 18:28 |
| clarkb | gear is downgraded. restarting nodepool now | 18:29 |
| viscious | has anyone been looking at why the postgres tests are failing in stable jobs? | 18:29 |
| *** viscious is now known as vishy | 18:29 | |
| vishy | database "openstack_citest" is being accessed by other users | 18:29 |
| clarkb | done | 18:29 |
| *** teran has joined #openstack-infra | 18:29 | |
| fungi | jeblair: clarkb: i suppose in the long term a failure to query for demand should just no-op for that cycle rather than assuming all zero?" | 18:30 |
| jeblair | fungi: then we'll never build anything, even the min-ready | 18:30 |
| *** mpavlase has joined #openstack-infra | 18:30 | |
| jeblair | (if the network connection breaks) | 18:30 |
| clarkb | so we probably want to build to min ready then until we get data | 18:30 |
| jeblair | clarkb: that's what we do | 18:31 |
| sdague | matel: well run_tests.log is completely missing anything useful here | 18:31 |
| jeblair | clarkb: the problem is that the current allocator says everything is satisfied for that round, and so the next round, with actual demand data, doesn't build as much as what you would expect | 18:31 |
| *** shashankhegde has joined #openstack-infra | 18:31 | |
| jeblair | at least, the proportion is wrong | 18:31 |
| matel | sdague: yes, it's strange. | 18:31 |
| clarkb | jeblair: oh because of state | 18:31 |
| clarkb | gotcha | 18:31 |
| matel | sdague: Do you have a change ref? | 18:32 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add networking-odl project to StackForge https://review.openstack.org/136854 | 18:32 |
| matel | sdague: refs/changes/22/136822/1 i guess | 18:32 |
| fungi | following the nodepool restart, we've got more than 100 bare-trusty nodes building now | 18:33 |
| *** pc_m has quit IRC | 18:34 | |
| *** mriedem has quit IRC | 18:34 | |
| *** erikwilson has quit IRC | 18:34 | |
| fungi | i guess if we stop seeing "Exception while listing functions" in the log after the 18:29 restart, then it's probably new gear | 18:34 |
| *** erikwilson has joined #openstack-infra | 18:35 | |
| *** wenlock has joined #openstack-infra | 18:35 | |
| jeblair | yeah. i'm running parallel tests to see if i can reproduce that behavior in each version | 18:35 |
| *** zaro has joined #openstack-infra | 18:36 | |
| *** Ryan_Lane has joined #openstack-infra | 18:37 | |
| *** unicell has quit IRC | 18:37 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:39 | |
| *** tgohad has joined #openstack-infra | 18:40 | |
| *** signed8bit_ZZZzz has quit IRC | 18:40 | |
| *** tsg_ has quit IRC | 18:40 | |
| *** marcusvrn1 has quit IRC | 18:40 | |
| fungi | no dice | 18:41 |
| fungi | 2014-11-24 18:37:53,732 ERROR nodepool.GearmanClient: Exception while listing functions | 18:41 |
| jeblair | huh | 18:41 |
| jeblair | so right around that time, both of my tests took 29.x seconds to return from a listing | 18:42 |
| fungi | with the same traceback as before | 18:42 |
| jeblair | Mon Nov 24 18:36:54 2014 29.1994440556 | 18:42 |
| *** pblaho has quit IRC | 18:42 | |
| jeblair | though a timeout for an admin request should be 90 seconds | 18:42 |
| fungi | does it time out at 30s? | 18:42 |
| fungi | oh | 18:42 |
| jeblair | and they did not actually timeout | 18:42 |
| *** erlon has joined #openstack-infra | 18:42 | |
| winston-d_ | jeblair: I did tried some other keybindings for gertty search, but no luck. | 18:43 |
| *** emagana has joined #openstack-infra | 18:43 | |
| matel | sdague: It looks as if gate_hook terminated the whole run - although I don't really understand why we don't have any output at all. | 18:43 |
| clarkb | jeblair: are you running your tests on a different host too? | 18:43 |
| jeblair | winston-d_: what did you do? | 18:43 |
| openstackgerrit | Merged openstack-infra/infra-manual: Added some initial content in Peer Review before ReviewChecklist link https://review.openstack.org/107588 | 18:43 |
| jeblair | clarkb: no, on the new nodepool.o.o | 18:43 |
| *** signed8bit has joined #openstack-infra | 18:44 | |
| jeblair | i can run it on the old nodepool as well | 18:44 |
| clarkb | jeblair: ya it may be worthwhile just to see if its isolated to new nodepool.o.o | 18:44 |
| *** amcrn has joined #openstack-infra | 18:44 | |
| jeblair | clarkb: we'll need to open the firewall on zuul i think | 18:44 |
| clarkb | jeblair: ya | 18:44 |
| jeblair | i'll do that manually real quick | 18:44 |
| *** AlexF has quit IRC | 18:45 | |
| winston-d_ | jeblair: i changed '~/.gertty.yaml' and added a entry under 'keymap' with "change-search: 'ctrl i'" | 18:45 |
| ekarlso- | https://review.openstack.org/#/c/136624/ < anyone wanna sign off on that ? | 18:45 |
| *** e0ne has quit IRC | 18:46 | |
| *** achanda has joined #openstack-infra | 18:46 | |
| fungi | or test from zm0X or jenkins0X? | 18:47 |
| *** otherwiseguy has quit IRC | 18:47 | |
| sdague | matel: my guess is that your system isn't being careful with output buffering. We used to lose stuff like that in devstack | 18:47 |
| *** mudassirlatif has quit IRC | 18:48 | |
| jeblair | fungi: did the firewall thing | 18:48 |
| jeblair | there's a bit of a heisenberg thing going on here; polling from a 2nd host slows it down considerably (probably because of the lack of async io handling) | 18:50 |
| *** bhunter71 has quit IRC | 18:51 | |
| matel | sdague: will look into that. Let's see if it does the same with the second patchset as well. | 18:51 |
| *** bhunter71 has joined #openstack-infra | 18:51 | |
| clarkb | the gearman server log on zuul is empty | 18:51 |
| jeblair | fungi: i think we may be about to hit a timeout | 18:51 |
| *** signed8bit has quit IRC | 18:52 | |
| jeblair | there it is | 18:52 |
| *** tgohad has quit IRC | 18:53 | |
| *** marun has joined #openstack-infra | 18:53 | |
| *** HeOS has quit IRC | 18:53 | |
| *** achanda has quit IRC | 18:53 | |
| matel | sdague: Failed at the same place... | 18:53 |
| *** tsg_ has joined #openstack-infra | 18:53 | |
| *** achanda has joined #openstack-infra | 18:54 | |
| jeblair | clarkb, fungi: my test script on both the old and new servers saw that | 18:54 |
| fungi | clarkb: we only temporarily enable that when we're trying to track down gearman-related issues because of verbosity, right? | 18:54 |
| *** andreykurilin_ has quit IRC | 18:54 | |
| clarkb | fungi: iirc its >DEBUG most of the time | 18:54 |
| *** yfried_ has joined #openstack-infra | 18:54 | |
| fungi | ahh | 18:54 |
| *** andreykurilin_ has joined #openstack-infra | 18:55 | |
| clarkb | jeblair: I guess thats good news. | 18:55 |
| clarkb | at least doesn't point to new server being the only thing at play her | 18:55 |
| jeblair | i ran tcpdump on zuul for the last part of that; there seemed to be gearman related traffic in both directions | 18:55 |
| *** winston-d_ has quit IRC | 18:58 | |
| jeblair | the server is currently logging at warning level | 19:00 |
| openstackgerrit | Devananda van der Veen proposed openstack-infra/project-config: Update Ironic jobs post-graduation https://review.openstack.org/126627 | 19:00 |
| *** mudassirlatif has joined #openstack-infra | 19:01 | |
| jeblair | right now if i telnet to 4730, i can't run anything :/ | 19:01 |
| *** davideagnello has quit IRC | 19:01 | |
| *** davideagnello has joined #openstack-infra | 19:02 | |
| *** koolhead_ has quit IRC | 19:02 | |
| *** koolhead17 has joined #openstack-infra | 19:02 | |
| jeblair | and zuul just dropped its gearman connection too | 19:03 |
| *** emagana has quit IRC | 19:03 | |
| *** afazekas has joined #openstack-infra | 19:05 | |
| fungi | the cacti graphs for zuul don't (or at least didn't a few minutes ago) look too bad | 19:05 |
| openstackgerrit | Surojit Pathak proposed openstack-dev/hacking: Fixing broken while loop in imports.py https://review.openstack.org/136517 | 19:06 |
| *** emagana_ has joined #openstack-infra | 19:06 | |
| jeblair | [pid 23955] sendto(124, "build:gate-oslo.config-requireme"..., 55, 0, NULL, 0 | 19:06 |
| jeblair | zuul-serv 23954 zuul 124u IPv4 58579682 0t0 TCP zuul.openstack.org:4730->nodepool.openstack.org:44312 (CLOSE_WAIT) | 19:06 |
| *** mudassirlatif has quit IRC | 19:06 | |
| jeblair | that doesn't look great | 19:06 |
| *** davideagnello has quit IRC | 19:07 | |
| *** mriedem has joined #openstack-infra | 19:07 | |
| *** koolhead_ has joined #openstack-infra | 19:07 | |
| *** koolhead17 has quit IRC | 19:07 | |
| *** bhunter71 has quit IRC | 19:07 | |
| *** jp_at_hp has quit IRC | 19:07 | |
| jeblair | fungi: i'm wondering if one of the closing packets for that connection was dropped | 19:07 |
| *** smoser has quit IRC | 19:07 | |
| *** rkukura has quit IRC | 19:07 | |
| *** rfolco has joined #openstack-infra | 19:08 | |
| *** davideagnello has joined #openstack-infra | 19:08 | |
| fungi | jeblair: entirely possible we've got some sort of packet loss, but fin should retry if no fin/ack comes back | 19:08 |
| fungi | er, should retransmit | 19:08 |
| * fungi is starting to lose his network engineering vocabulary | 19:09 | |
| *** emagana_ has quit IRC | 19:09 | |
| *** emagana has joined #openstack-infra | 19:09 | |
| jeblair | fungi: there's no corresponding connection on the nodepool.o.o side | 19:09 |
| clarkb | ya I just checked that | 19:09 |
| *** bhunter71 has joined #openstack-infra | 19:10 | |
| clarkb | 104.130.155.213:41955 is the port nodepool claims to be connected from | 19:10 |
| *** afazekas has quit IRC | 19:10 | |
| jeblair | that's established on both sides | 19:10 |
| fungi | ideally if the fin is received but the fin/ack never arrives, then when the fin is retransmitted the receiving end should send an rst | 19:10 |
| jeblair | there are 4 connections on zuul.o.o that are in close_wait | 19:10 |
| *** signed8bit has joined #openstack-infra | 19:12 | |
| *** packet has joined #openstack-infra | 19:12 | |
| *** rfolco has quit IRC | 19:12 | |
| fungi | hrm. if the kernel on the zuul end is reporting close_wait then i believe that means it thinks the nodepool end may already be closed but the process at the zuul end is still holding the fd of the associated socket open | 19:12 |
| *** andreykurilin_ has quit IRC | 19:13 | |
| *** sarob has joined #openstack-infra | 19:14 | |
| *** smcginnis has joined #openstack-infra | 19:14 | |
| jeblair | 19:11:26.735845 IP 162.242.150.96.4730 > 104.130.155.213.43882: Flags [P.], seq 4222912087:4222912144, ack 3780423436, win 114, options [nop,nop,TS val 1228423088 ecr 62825158], length 57 | 19:14 |
| jeblair | 19:12:21.071842 IP 162.242.150.96.4730 > 104.130.155.213.43882: Flags [P.], seq 0:57, ack 1, win 114, options [nop,nop,TS val 1228436672 ecr 62825158], length 57 | 19:14 |
| jeblair | 19:14:09.871860 IP 162.242.150.96.4730 > 104.130.155.213.43882: Flags [P.], seq 0:57, ack 1, win 114, options [nop,nop,TS val 1228463872 ecr 62825158], length 57 | 19:14 |
| jeblair | that's on the nodepool side | 19:14 |
| jeblair | that's one of the close_wait connections on the zuul side | 19:14 |
| fungi | oh, it's old nodepool | 19:15 |
| fungi | er, nevermind, that is the new one | 19:15 |
| *** gyee_ has joined #openstack-infra | 19:15 | |
| *** jcoufal_ has quit IRC | 19:15 | |
| clarkb | 104.130.155.213 is new nodepool | 19:15 |
| jeblair | whew | 19:15 |
| *** alexpilotti has joined #openstack-infra | 19:15 | |
| *** MaxV has joined #openstack-infra | 19:16 | |
| fungi | so it's receiving packets from zuul for a connection zuul lists as being in close_wait except those don't look like they're associated with trying to close the socket | 19:17 |
| jeblair | yep, and they are continuing to trickle in | 19:18 |
| *** ci-testing_ has quit IRC | 19:18 | |
| fungi | also i had to try four times to load https://status.rackspace.com/ | 19:18 |
| clarkb | hrm, could that be a bug in newer gear so restarting nodepool with older gear was a step in the right direction but not sufficient | 19:18 |
| fungi | but it's still showing all green | 19:18 |
| clarkb | except it is showing as close wait on zuul | 19:19 |
| clarkb | which implies it knows that connection went away? | 19:19 |
| *** gothicmindfood has quit IRC | 19:19 | |
| *** rushiagr is now known as rushiagr_away | 19:19 | |
| *** koolhead_ has quit IRC | 19:19 | |
| *** prad has joined #openstack-infra | 19:20 | |
| *** koolhead17 has joined #openstack-infra | 19:20 | |
| *** otter768 has joined #openstack-infra | 19:20 | |
| *** gothicmindfood has joined #openstack-infra | 19:21 | |
| jeblair | 19:22:11.149857 IP zuul.openstack.org.4730 > nodepool.openstack.org.43882: Flags [P.], seq 4222912087:4222912144, ack 3780423436, win 114, options [nop,nop,TS val 1228584192 ecr 62825158], length 57 | 19:22 |
| jeblair | 19:22:11.150553 IP nodepool.openstack.org > zuul.openstack.org: ICMP host nodepool.openstack.org unreachable - admin prohibited, length 117 | 19:22 |
| jeblair | that's from the zuul side | 19:22 |
| jeblair | so zuul keeps sending that packet, and nodepool rejects it due to the iptables rules | 19:22 |
| *** teran has quit IRC | 19:23 | |
| *** ci-testing has joined #openstack-infra | 19:23 | |
| *** amitgandhinz has quit IRC | 19:23 | |
| clarkb | because it knows of no such connection | 19:23 |
| jeblair | yep | 19:23 |
| *** amitgandhinz has joined #openstack-infra | 19:24 | |
| *** koolhead17 has quit IRC | 19:24 | |
| *** otter768 has quit IRC | 19:25 | |
| jeblair | oh, is the problem that geard is not closing those sockets? | 19:25 |
| fungi | that's one way that you can end up with hung close_wait, yes | 19:25 |
| *** timrc is now known as timrc-afk | 19:26 | |
| fungi | so maybe geard is continuing to try to send on that socket even though the kernel thinks the other end has closed it | 19:26 |
| jeblair | oh | 19:26 |
| jeblair | one of the close_waits just went away | 19:26 |
| fungi | and there's the gearman log entry now | 19:27 |
| *** alexpilotti_ has joined #openstack-infra | 19:27 | |
| fungi | the file is no longer empty. plenty of tracebacks | 19:27 |
| *** alexpilotti has quit IRC | 19:27 | |
| *** Ryan_Lane1 has joined #openstack-infra | 19:27 | |
| *** alexpilotti_ is now known as alexpilotti | 19:27 | |
| fungi | lots of broken pipes hit while sending | 19:27 |
| *** Ryan_Lane has quit IRC | 19:27 | |
| clarkb | zaro: the webui for editing acls is really broken on review-dev | 19:27 |
| clarkb | zaro: not a huge issue but worth pointing out | 19:28 |
| fungi | ahh, that was from the incident where zuul had trouble communicating with its gearman service | 19:28 |
| fungi | all from around 19:10 | 19:28 |
| jeblair | yep | 19:28 |
| fungi | so nothing reflecting the more recent sockets going away | 19:29 |
| *** amitgandhinz has quit IRC | 19:29 | |
| *** amitgandhinz has joined #openstack-infra | 19:30 | |
| *** cpowell has quit IRC | 19:31 | |
| *** jistr has quit IRC | 19:31 | |
| jeblair | so i think it recovered from that | 19:32 |
| jeblair | seems to be picking up demand again | 19:32 |
| *** MarkAtwood has joined #openstack-infra | 19:33 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 19:34 | |
| jeblair | nodepool seems to have a lot in 'ready' according to the graph | 19:34 |
| jeblair | i think maybe now we're waiting on zuul to catch up? | 19:35 |
| clarkb | ya zuul has a ton of results to get through | 19:35 |
| *** iax7 has joined #openstack-infra | 19:36 | |
| *** iax7 has quit IRC | 19:36 | |
| *** bo_sh has joined #openstack-infra | 19:37 | |
| *** bo_sh has left #openstack-infra | 19:38 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/publications: Update tools and review purposes. https://review.openstack.org/128722 | 19:38 |
| *** koolhead17 has joined #openstack-infra | 19:38 | |
| jeblair | okay, now geard has decided to try to send a bunch of data to another close_wait | 19:39 |
| jeblair | (it's still one of the 4 from earlier) | 19:39 |
| jeblair | i suspect everything will stop again until it works through that | 19:39 |
| *** SumitNaiksatam has quit IRC | 19:40 | |
| *** emagana has quit IRC | 19:40 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:40 | |
| *** emagana has joined #openstack-infra | 19:41 | |
| jeblair | clarkb, fungi: i think we should restart zuul to reset the state | 19:41 |
| jeblair | and see if we accumulate more close_wait sockets | 19:41 |
| clarkb | jeblair: ok | 19:42 |
| fungi | sounds fair | 19:42 |
| clarkb | sounds reasonable to me. anything I can do to help with that? | 19:42 |
| *** tkelsey has joined #openstack-infra | 19:42 | |
| jeblair | i'll just save the queues and restart/re-enqueue | 19:42 |
| *** cpowell has joined #openstack-infra | 19:42 | |
| fungi | it's taken >2mos to get into this state since its last restart | 19:43 |
| clarkb | though that may have partially been triggered by the nodepool move? | 19:43 |
| clarkb | btw https://etherpad.openstack.org/p/third-party-openid-accounts is a thing now | 19:43 |
| fungi | yeah, i'm trying to think of what things have gone on in that span of time | 19:43 |
| clarkb | I think it would be relatively simple to get third party accounts as lp openid things going. just need to create some groups | 19:43 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/publications: Update tools and review purposes. https://review.openstack.org/128722 | 19:44 |
| clarkb | fungi: though looking at the graphs it wasn't until today that it went sideways | 19:44 |
| fungi | right, which is the first real load we've had on it since the nodepool replacement | 19:44 |
| dvorak | is there a way to tie matrix jobs parent job without using the deprecated tie matrix job parent plugin? | 19:44 |
| clarkb | anteaya: if you are around https://etherpad.openstack.org/p/third-party-openid-accounts probabl interests you | 19:44 |
| fungi | especially since i had stuff offline for much of the weekend dealing with the log vg | 19:45 |
| clarkb | dvorak: I don't know that many of us would know. we avoid matrix jobs and rely on zuul + jjb to provide that sort of job explosion feature | 19:45 |
| harlowja | did zuul just restart | 19:45 |
| *** e0ne has joined #openstack-infra | 19:45 | |
| dvorak | fair enough | 19:45 |
| *** emagana has quit IRC | 19:45 | |
| clarkb | harlowja: yes | 19:45 |
| fungi | harlowja: yes, jeblair's restarting it to see if we clear a misbehavior we've been observing | 19:45 |
| harlowja | kk | 19:45 |
| jedimike | pleia2, maybe I'm missing something, on https://review.openstack.org/128722 the commit messages says we note our use of review for translations and our use of Storyboard, but I can't see us saying that in the diff anywhere. Is that info just for the commit message? | 19:46 |
| dvorak | clarkb: I could do that, but I actually use the jenkins UI, so that'd be a lot of extra jobs :) | 19:46 |
| jeblair | clarkb, fungi: if you feel like reviewing the non-blocking io patch for gear: 128754 | 19:47 |
| * clarkb pulls that up now | 19:47 | |
| dvorak | oddly, the JJB docs for the tie matrix job support explicitly mention that it's deprecated, and that comment was added as part of the initial implementation. | 19:47 |
| *** mrmartin has quit IRC | 19:47 | |
| clarkb | dvorak: ya it definitely doesn't fit everyones needs, but the alternatives are not something we are very familiar with | 19:48 |
| jeblair | i think that if we decide we need to dig deeper into this, we should use that patch if we think it's the way we want to go (otherwise we may waste some effort) | 19:48 |
| dvorak | yeap, understood :) | 19:48 |
| clarkb | jeblair: iirc that patch existed as a different change at one point? or am I misremembering? | 19:49 |
| jeblair | clarkb: there is also 96294, which is 'use non-blocking io' (everywhere) | 19:49 |
| jeblair | clarkb: the new one is only use it in the server | 19:49 |
| clarkb | gotcha | 19:49 |
| *** luqas has joined #openstack-infra | 19:50 | |
| jeblair | clarkb: i'm leaning toward that in order to keep the client simple and more predictable | 19:50 |
| *** otherwiseguy has joined #openstack-infra | 19:50 | |
| clarkb | ++ | 19:50 |
| pleia2 | jedimike: it's in there... | 19:50 |
| pleia2 | jedimike: Storyboard is line 64, translations & specs lines 94-95 | 19:51 |
| dvorak | oh, I see. it just uses the normal node restriction field. that wasn't clear at all. | 19:51 |
| adam_g | anyone know why installation of oslo.vmware here wouldn't be pulling up the system's eventlet depedency to match oslo.vmware's requirements.txt? http://logs.openstack.org/73/135673/2/check/check-grenade-dsvm/23ba48c/logs/old/devstacklog.txt.gz#_2014-11-24_10_27_46_144 | 19:51 |
| jedimike | pleia2, that's a sign i need to eat. Of course it's there :) | 19:52 |
| *** mmaglana has joined #openstack-infra | 19:52 | |
| *** rcarrillocruz has quit IRC | 19:52 | |
| *** Ryan_Lane1 is now known as Ryan_Lane | 19:52 | |
| *** Ryan_Lane has joined #openstack-infra | 19:52 | |
| pleia2 | jedimike: eating is good! (also, I hope you're feeling better) | 19:52 |
| mtreinish | adam_g: maurosr was looking at this morning, IIRC it's because transitive deps don't really work | 19:53 |
| jedimike | pleia2, yeah, feeling much more alert this week :) was so jealous of your vacation photos :p | 19:53 |
| *** rcarrillocruz has joined #openstack-infra | 19:53 | |
| pleia2 | jedimike: glad to hear it, and it was a wonderful vacation, much needed :) | 19:54 |
| adam_g | mtreinish, curious as to whats changed? other than the devstack backports that went in last week around libs from git/release | 19:54 |
| fungi | mtreinish: adam_g: that could be the order-dependent issue dstufft was describing to clarkb over the weekend | 19:54 |
| fungi | or was that friday | 19:54 |
| * adam_g has some backscroll to read | 19:54 | |
| mtreinish | adam_g: that would do it, because when oslo.vmware was installed from git the eventlet dep was at the higher version | 19:55 |
| *** smcginnis has left #openstack-infra | 19:55 | |
| jeblair | fungi: the nodepool main loop interval seems to be about 30 seconds now, which seems more normal | 19:55 |
| clarkb | it was friday | 19:56 |
| fungi | basically if you have package A already installed at version 1.2.3 and then feed pip a list of packages to install (directly or transitively) if the first dependency it sees listed on package A doesn't require newer than 1.2.3 then a later listed dependency on A>=2.3.4 will basically be ignored and won't trigger an upgrade | 19:56 |
| clarkb | tl;dr is "highest" req wins | 19:56 |
| mtreinish | adam_g: I expect that being installed from pypi on icehouse doesn't pull in the newer dep initially it doesn't work now | 19:56 |
| clarkb | so if you have a top level req that is >=1.0 and a lower level req that is >=1.5 but you already have 1.3 installed then pip does nothing | 19:56 |
| clarkb | because 1.0 wins and is satisfied | 19:56 |
| adam_g | fungi, oh, i seem to have overestimated pip | 19:56 |
| fungi | adam_g: yes, i do that all the time :/ | 19:56 |
| clarkb | jeblair: that is good news | 19:56 |
| fungi | jeblair: so whatever had it gummed up is no longer present since the zuul restart i guess | 19:57 |
| *** dprince has quit IRC | 19:57 | |
| jeblair | seems like it | 19:57 |
| *** Guest86919 has quit IRC | 19:58 | |
| *** koolhead17 has quit IRC | 19:58 | |
| *** sarob has quit IRC | 19:59 | |
| *** koolhead17 has joined #openstack-infra | 19:59 | |
| *** MaxV has quit IRC | 19:59 | |
| *** sarob has joined #openstack-infra | 20:00 | |
| *** davideagnello has quit IRC | 20:01 | |
| ekarlso- | https://review.openstack.org/#/c/136624/ < can we get a +A there ? | 20:02 |
| jeblair | clarkb, fungi: geard (and therefore nodepoold) seems to be sluggish again | 20:02 |
| fungi | that was quick | 20:02 |
| *** nadya has joined #openstack-infra | 20:03 | |
| *** nadya is now known as Guest82190 | 20:03 | |
| ekarlso- | pretty please ? :) it's a simple change :D | 20:03 |
| *** AJaeger has quit IRC | 20:03 | |
| *** koolhead17 has quit IRC | 20:03 | |
| fungi | and indeed we got a gearmanclient timeout in nodepool as recently as 4 minutes ago | 20:04 |
| *** mjturek has quit IRC | 20:05 | |
| *** bhunter71 has quit IRC | 20:05 | |
| *** luqas has quit IRC | 20:06 | |
| *** Hal_ has joined #openstack-infra | 20:06 | |
| *** luqas has joined #openstack-infra | 20:06 | |
| *** bhunter71 has joined #openstack-infra | 20:06 | |
| *** banix has quit IRC | 20:07 | |
| jeblair | huh, so the connection nodepool was using is still in use | 20:07 |
| jeblair | that is, even after the timeout, it did not close/reopen the connection | 20:07 |
| jeblair | it just started working again | 20:07 |
| *** Sincler has quit IRC | 20:07 | |
| *** MaxV has joined #openstack-infra | 20:07 | |
| *** HeOS has joined #openstack-infra | 20:09 | |
| *** davideagnello has joined #openstack-infra | 20:09 | |
| jeblair | fungi, clarkb: oh, that would be because there's nothing in either gear or nodepool to drop/reset a connection when an admin request times out | 20:10 |
| *** luqas has quit IRC | 20:10 | |
| jeblair | so it seems like it just resumed using the connection and that worked :/ | 20:10 |
| *** gyee_ has quit IRC | 20:12 | |
| *** luqas has joined #openstack-infra | 20:12 | |
| *** ldnunes_ has quit IRC | 20:16 | |
| fungi | profiling the packet rates from gearman clients connected to zuul has yielded little of interest other than zuul's receiving a lot of gearman packets from the jenkins masters, an order of magnitude less from the zuul mergers and basically none from nodepool | 20:16 |
| *** thedodd has joined #openstack-infra | 20:16 | |
| * mordred back from lunch ... from what I can tell, some stuff started working but some stuff didn't and we still dont' absolutely know the issue, yeah? | 20:16 | |
| fungi | mordred: that's every day | 20:16 |
| mordred | fungi: yup. just thought I'd express it out loud | 20:17 |
| adam_g | fungi, curious if this is the proper fix for that dependency issue or if i've oversimplified the problem, https://review.openstack.org/#/c/136879/ | 20:17 |
| jeblair | fungi, mordred: i suspect that not dropping the connection after the admin request timeout is a problem | 20:17 |
| *** smcginnis has joined #openstack-infra | 20:17 | |
| *** sarob has quit IRC | 20:17 | |
| jeblair | it's sent another request, some data went over the network, but it's still sitting waiting for a response | 20:18 |
| jeblair | so i think it's gotten out of sync | 20:18 |
| *** mjturek has joined #openstack-infra | 20:18 | |
| jeblair | (part of why we're generally very agressive about dropping connections is because it's pretty easy to get into that state with the gearman protocol(s)) | 20:18 |
| jeblair | so i think the first time geard gets stuck, nodepool goes into a state where it is difficult to recover | 20:19 |
| *** mjturek has left #openstack-infra | 20:19 | |
| jeblair | (and subsequent demand checks simply may or may not work) | 20:19 |
| clarkb | jeblair: ok commented on gear change | 20:19 |
| clarkb | hrm it started again :/ | 20:19 |
| fungi | though that's part of a failure pattern which starts with something causing admin requests to timeout, and we don't know why that's the case either (but suspect nonblocking i/o in the server will help)? | 20:20 |
| *** AlexF has joined #openstack-infra | 20:20 | |
| jeblair | fungi: yes | 20:20 |
| *** shashankhegde has quit IRC | 20:20 | |
| *** ilyashakhat has quit IRC | 20:21 | |
| *** banix has joined #openstack-infra | 20:21 | |
| *** mmaglana has quit IRC | 20:21 | |
| *** bhunter71 has quit IRC | 20:21 | |
| *** ilyashakhat has joined #openstack-infra | 20:22 | |
| mordred | makes sense to me | 20:22 |
| clarkb | jeblair: let me know if my comments are just wrong. I will sit down wth more caffeine :) | 20:23 |
| *** otherwiseguy has quit IRC | 20:23 | |
| *** bhunter71 has joined #openstack-infra | 20:24 | |
| *** luqas has quit IRC | 20:24 | |
| *** otherwiseguy has joined #openstack-infra | 20:24 | |
| *** baoli has quit IRC | 20:24 | |
| *** alexpilotti has quit IRC | 20:24 | |
| *** amitgandhinz has quit IRC | 20:25 | |
| *** emagana has joined #openstack-infra | 20:25 | |
| *** gyee_ has joined #openstack-infra | 20:25 | |
| mordred | just in case anyone was curious, there is no /etc/apt/sources.list file on centos | 20:26 |
| mtreinish | mordred: heh, you should add it then :) | 20:26 |
| mordred | mtreinish: I really should ... | 20:27 |
| *** amitgandhinz has joined #openstack-infra | 20:28 | |
| *** yfried_ has quit IRC | 20:28 | |
| *** amitgandhinz has quit IRC | 20:29 | |
| fungi | mordred: https://admin.fedoraproject.org/pkgdb/package/fedora-package-config-apt/ | 20:30 |
| *** amitgandhinz has joined #openstack-infra | 20:30 | |
| mordred | the other choice is that i can put sources.list manipulation somewhere that's not run on centos | 20:30 |
| fungi | adam_g: you could reorder the requirements.txt for ceilo and glance to list oslo.vmware before eventlet | 20:32 |
| fungi | adam_g: i think pinning oslo libs in stable reqs doesn't yet have the desired effect | 20:32 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/nodepool: Add REST API to Nodepool https://review.openstack.org/136884 | 20:33 |
| adam_g | fungi, ah, good idea | 20:33 |
| *** Ryan_Lane has quit IRC | 20:33 | |
| dstanek | i'm working on some changes to add functional testing into Keystone. is there a good way for me to test everything out before i submit changesets? | 20:33 |
| *** AlexF has quit IRC | 20:35 | |
| clarkb | dstanek: the best thing is to run the tests as they will be run. this depends on how you are setting them up. might be with tox or with devstack-gate | 20:35 |
| clarkb | dstanek: either way, executing it locally against master is a good idea | 20:35 |
| clarkb | dstanek: if using devstack gate there are directions in that repo on how to run it locally | 20:35 |
| clarkb | in the readme iirc | 20:35 |
| dstanek | clarkb: ok, i'll check that out | 20:35 |
| dstanek | clarkb: i also heard there may be a way to tell devstack to install software without actually making changes to devstack itself. i have to potentially install apache modules and my own configs for those. | 20:37 |
| *** achanda has quit IRC | 20:38 | |
| openstackgerrit | Clayton O'Neill proposed openstack-infra/jenkins-job-builder: Document node parameter usage with matrix projects https://review.openstack.org/136886 | 20:38 |
| *** AlexF has joined #openstack-infra | 20:39 | |
| *** achanda has joined #openstack-infra | 20:40 | |
| *** aysyd has quit IRC | 20:40 | |
| *** teran has joined #openstack-infra | 20:40 | |
| *** jedimike has quit IRC | 20:41 | |
| jeblair | clarkb: i can not answer your second question yet. i'm doing some testing. thanks. :) | 20:42 |
| clarkb | dstanek ya devstack has plugin points where you drop files in and they are run | 20:42 |
| dstanek | clarkb: thanks, i'll look into that too | 20:42 |
| openstackgerrit | Allison Randal proposed stackforge/gertty: Sample color palette for inline review comments https://review.openstack.org/135799 | 20:43 |
| dvorak | Could I get someone to take a look at this? https://review.openstack.org/#/c/116704/ I think it's pretty close to final form and it has one +2 already | 20:43 |
| *** baoli has joined #openstack-infra | 20:45 | |
| *** baoli has quit IRC | 20:46 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add networking-odl project to StackForge https://review.openstack.org/136854 | 20:47 |
| *** baoli has joined #openstack-infra | 20:47 | |
| *** lttrl has quit IRC | 20:48 | |
| *** esker has quit IRC | 20:49 | |
| *** ddieterly has quit IRC | 20:49 | |
| *** dolphm has joined #openstack-infra | 20:50 | |
| *** shashankhegde has joined #openstack-infra | 20:50 | |
| *** smcginnis has left #openstack-infra | 20:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "move nova-tox-functional to experimental until there is content" https://review.openstack.org/136795 | 20:51 |
| dolphm | where can i find notify_impact config as referred to here?: https://bugs.launchpad.net/keystonemiddleware/+bug/1393920 | 20:51 |
| uvirtbot | Launchpad bug 1393920 in keystonemiddleware "I18n" [Medium,Confirmed] | 20:51 |
| mordred | dolphm: ./gerrit/notify_impact.yaml in openstack-infra/project-config | 20:52 |
| dolphm | mordred: thanks! | 20:52 |
| mordred | dolphm: I think | 20:52 |
| mordred | dolphm: nope. I think I'm wrong | 20:53 |
| *** Guest82190 has quit IRC | 20:53 | |
| *** lttrl has joined #openstack-infra | 20:53 | |
| dolphm | mordred: dev/gerrit/notify_impact.yaml ? | 20:54 |
| mordred | dolphm: gerrit/projects.yaml | 20:54 |
| mordred | dolphm: you'll see docimpact-group listings | 20:54 |
| dolphm | ah that makes more sense | 20:54 |
| mordred | I believe that the notify_impact verbage there is misleading | 20:54 |
| dolphm | i was planning to update the verbiage once i understood it | 20:55 |
| *** AlexF has quit IRC | 20:55 | |
| *** achanda has quit IRC | 20:55 | |
| *** rkukura has joined #openstack-infra | 20:57 | |
| *** weshay has quit IRC | 20:58 | |
| *** matel has quit IRC | 21:00 | |
| *** amotoki has joined #openstack-infra | 21:00 | |
| *** MaxV has quit IRC | 21:00 | |
| *** tkelsey has quit IRC | 21:01 | |
| *** melwitt has quit IRC | 21:01 | |
| mtreinish | fungi: are things calm enough now to give the mysql_proxy stuff a shot? | 21:01 |
| *** melwitt has joined #openstack-infra | 21:02 | |
| *** cpowell has quit IRC | 21:02 | |
| *** Hal_ has quit IRC | 21:05 | |
| *** shashankhegde has quit IRC | 21:05 | |
| fungi | mtreinish: not really. my to do list from this morning is basically now a mostly untouched to do clog as evening approaches | 21:06 |
| mtreinish | fungi: ok sure, anything I can do to help lighten the load? | 21:07 |
| fungi | mtreinish: not really. need to ping devs on some security bugs, finish going over the groups portal ssl cert stuff, evaluate impact of big-tent and options for retuning on our free summit pass numbers | 21:10 |
| *** aysyd has joined #openstack-infra | 21:11 | |
| *** melwitt has quit IRC | 21:11 | |
| *** melwitt has joined #openstack-infra | 21:11 | |
| *** Ryan_Lane has joined #openstack-infra | 21:11 | |
| fungi | trying to get through a review of the gear non-blocking i/o change but i think my mind is not fresh enough to do so quickly | 21:11 |
| *** sarob has joined #openstack-infra | 21:12 | |
| *** weshay has joined #openstack-infra | 21:13 | |
| *** nadya has joined #openstack-infra | 21:14 | |
| *** nadya is now known as Guest33095 | 21:14 | |
| *** kgiusti has quit IRC | 21:18 | |
| *** shashankhegde has joined #openstack-infra | 21:20 | |
| *** otter768 has joined #openstack-infra | 21:21 | |
| fungi | spending too much time rereading the docs on select | 21:24 |
| *** baoli has quit IRC | 21:24 | |
| adam_g | hmm. is the 'check experimental' trigger expected to work against stable branches, or only master? | 21:26 |
| *** otter768 has quit IRC | 21:26 | |
| *** ildikov has joined #openstack-infra | 21:27 | |
| *** bhunter71 has quit IRC | 21:27 | |
| *** bhunter71 has joined #openstack-infra | 21:27 | |
| *** tsg_ has quit IRC | 21:29 | |
| *** mudassirlatif has joined #openstack-infra | 21:29 | |
| *** erlon has quit IRC | 21:29 | |
| clarkb | predominantly master. we dont backport jobs usually | 21:29 |
| *** Ryan_Lane has quit IRC | 21:30 | |
| *** Sukhdev has joined #openstack-infra | 21:31 | |
| *** mmaglana has joined #openstack-infra | 21:32 | |
| *** bhunter71 has quit IRC | 21:33 | |
| *** tsg has joined #openstack-infra | 21:33 | |
| *** bhunter71 has joined #openstack-infra | 21:34 | |
| adam_g | clarkb, trying to trigger the grenade forward jobs that are listed in most project's experimental, was hoping to test an I -> J forward on stable/icehouse | 21:34 |
| adam_g | clarkb, nvm, they're running now | 21:34 |
| *** jerryz has quit IRC | 21:34 | |
| clarkb | ya I wouldn't expect those to be branch restricted but I know many experimental jobns are | 21:35 |
| *** tsg_ has joined #openstack-infra | 21:36 | |
| *** mmaglana has quit IRC | 21:36 | |
| *** Ryan_Lane1 has joined #openstack-infra | 21:36 | |
| *** achanda has joined #openstack-infra | 21:37 | |
| *** smoser has joined #openstack-infra | 21:37 | |
| *** Ryan_Lane1 is now known as Ryan_Lane | 21:38 | |
| *** Ryan_Lane has joined #openstack-infra | 21:38 | |
| *** tsg has quit IRC | 21:39 | |
| clarkb | ok lunch consumed /me dives back into stuff | 21:39 |
| clarkb | puppet doesn't appear to be running on nodepool.o.o for some reason. going to look into that | 21:41 |
| fungi | clarkb: sudo ssh nodepool.openstack.org from puppetmaster.o.o | 21:42 |
| fungi | i'm betting you need to replace its known_hosts entry there | 21:42 |
| fungi | i always forget that until i notice my replaced server isn't puppeting at all | 21:43 |
| clarkb | danke | 21:43 |
| *** Sincler has joined #openstack-infra | 21:43 | |
| clarkb | thats it | 21:43 |
| *** dizquierdo has joined #openstack-infra | 21:44 | |
| * mordred is trying to make the new node launching stuff DTRT WRT known_hosts, fwiw | 21:44 | |
| mordred | also, I poked sdague about the idea that perhaps nova should be able to tell you a servers host cert | 21:45 |
| greghaynes | mordred: are you going to take a swing at preserving ssh host keys across boots of dib images? | 21:45 |
| clarkb | also I note that the current puppetmaster key is not restricted to running puppet. Ithink that is intentional so that we can run other ansible things? | 21:45 |
| mordred | clarkb: yes | 21:45 |
| greghaynes | mordred: because I think no less than 3 different people have taken a swing at that :) | 21:45 |
| mordred | clarkb: ansible needs to be able to do all the things | 21:45 |
| clarkb | mordred: rgr | 21:45 |
| mordred | greghaynes: I was not intending on it - but I may not even understand the problem space you're referring to | 21:46 |
| *** Guest33095 has quit IRC | 21:46 | |
| mordred | the main thing I want to add to nova is to have nova poke vms locally to find out their ssh host key | 21:46 |
| mordred | because I have a trusted relationship with nova, and nova has an under-the-covers relationship with the vm | 21:47 |
| greghaynes | oh, well we have been wanting to preserve the ssh host key gen'd on first boot to solve a similar problem, but your problem might be a bit different | 21:47 |
| mordred | so I should be able to ask the API for the host key of a host, so that I can verify that I'm talking to the right thing | 21:47 |
| greghaynes | ah, we could use that | 21:47 |
| mordred | greghaynes: yeah - mine is a bit different, I want to be able to register the host key on the host I use to run automation when I spin up a new host | 21:47 |
| greghaynes | as is the only solution involves ansible putting the host key on the persistent partition but that would be a much cleaner solution | 21:48 |
| *** ddieterly has joined #openstack-infra | 21:48 | |
| mordred | but - honestly I don't have a good way right now to know that I haven't been MITM'd between the time I created the vm and the first time I talk to it | 21:48 |
| fungi | mordred: that would be dependent on nova agent though, right? i mean, some implementations just start with no ssh keys and then the initscript/whatever for sshd creates them the first time it's started | 21:48 |
| JayF | mordred: someone here was talking about that exact same problem on Friday | 21:48 |
| mordred | fungi: nope | 21:48 |
| mordred | fungi: nova could totally hit the ssh port of the vm - just do it on the local bridge | 21:48 |
| *** emagana has quit IRC | 21:48 | |
| fungi | mordred: ahh, got it | 21:49 |
| mordred | fungi: it would also let nova report "ssh is running" | 21:49 |
| clarkb | ok puppet is puppeting nodepool | 21:49 |
| *** emagana has joined #openstack-infra | 21:49 | |
| fungi | mordred: right, thus not relevant for systems with no ssh at all | 21:49 |
| clarkb | I will not clean up old nodepool + images quite yet just in case we feel we need to rollback due to zuul, gearman nodepool weirdness | 21:49 |
| nibalizer | clarkb: play the power rangers theme while puppet runs | 21:49 |
| JayF | fungi: mordred: You gotta be careful not to assume state on the machine that's being deployed; relying or wnating to rely on SSH blocks out windows use | 21:50 |
| asselin | krtaylor, mmedvede, sweston, nibalizer I have a script running that is sub-treeing all the puppet modules. https://github.com/rasselin?tab=activity | 21:50 |
| fungi | mordred: implementation would probably also depend on sshd being configured to run on the standard port and firewall rules not blocking ssh connections from nova | 21:50 |
| JayF | fungi: mordred: Not to mention in the bare metal world there's not always a way to get that level of assuredness that you're connecting to *that machine* | 21:50 |
| clarkb | JayF: windows should just run an sshd | 21:50 |
| fungi | clarkb: and a gnu userspace and a linux/bsd kernel | 21:51 |
| sweston | asselin: in the network meeting right now, just a moment | 21:51 |
| mordred | JayF: yah | 21:51 |
| clarkb | fungi: thats the spirit | 21:51 |
| JayF | clarkb: for clouds that allow customer images; that's not a solution at all. I'm highly skeptical any inside-instance thing could interact reliably with the nova control plane | 21:51 |
| JayF | regardless of the direction the chat is going | 21:51 |
| mordred | fungi: well, if you boot an image that doesn't have ssh - then you probably won't run "nova check-ssh" | 21:51 |
| nibalizer | asselin: okay | 21:51 |
| clarkb | JayF: its mostly me being mean to windows | 21:51 |
| *** dimtruck is now known as zz_dimtruck | 21:52 | |
| fungi | mordred: this is true. though if you run it on a nonstandard port or get really restrictive with your iptables rules, you might be confused as to why it thinks no ssh is running | 21:52 |
| ekarlso- | https://review.openstack.org/#/c/136624/ < can we get a +A there ? | 21:52 |
| *** SumitNaiksatam has quit IRC | 21:52 | |
| ekarlso- | fungi: ? :D | 21:52 |
| nibalizer | asselin: i worry that you won't get reviews fast enough to land those changes all at once | 21:52 |
| nibalizer | whereas one at a time you can keep that in your head | 21:53 |
| nibalizer | but okay, work with the cores | 21:53 |
| *** SumitNaiksatam has joined #openstack-infra | 21:53 | |
| asselin | nibalizer, no need to merge all at once. I will remain up-to-date, so ready to go when you are. | 21:53 |
| *** emagana has quit IRC | 21:53 | |
| mordred | fungi: I think that's a strange enough use case that nova can be forgiven for not handling it | 21:53 |
| nibalizer | asselin: oh you have it set to track all changes? | 21:53 |
| mordred | fungi: although "nova check-ssh --port=2222" should be easy enough to deal with | 21:54 |
| asselin | I'm putting it in a post-merge-job now | 21:54 |
| JayF | clarkb: I know; we've just put a lot of thought in it for Rackspace OnMetal, because it's crappy UX for the instance to go active before POST is completed, but it doesn't seem like there's a way to know when it's UP without making assumptions you shouldn't make about network configuration or image configuration | 21:54 |
| clarkb | mordred: right I Think an important aspect of this is its best effort and it fails gracefully | 21:54 |
| *** emagana has joined #openstack-infra | 21:54 | |
| clarkb | eg it won't give you the wrong key it will only give you no ke | 21:54 |
| mordred | fungi: how could you firewall off connections from the nova compute host? | 21:54 |
| mordred | clarkb: ++ | 21:54 |
| *** tonytan4ever has quit IRC | 21:54 | |
| mordred | JayF: so - when I launch things with ansible, there are two different statuses I care about | 21:54 |
| asselin | nibalizer, no, it retree's on every merge. not very smart, but that can be added next. | 21:55 |
| mordred | JayF: "active" - which means that no more cloud API things are needed, and when does ssh port become active | 21:55 |
| mordred | I think the biggest problem is the term "Active" | 21:55 |
| JayF | mordred: that's basically the pattern we tell our customers to follow; but it's still not super friendly compared to telling the customer what they actually care about: when the compute resource is usable | 21:55 |
| mordred | because what has finished is the nova internal communication needed to allocate this thing fully | 21:55 |
| *** MarkAtwood has quit IRC | 21:55 | |
| mordred | yup | 21:55 |
| *** packet has quit IRC | 21:57 | |
| *** bhunter71 has quit IRC | 21:57 | |
| *** atiwari has quit IRC | 21:58 | |
| *** MarkAtwood has joined #openstack-infra | 21:58 | |
| *** erikwilson has quit IRC | 21:58 | |
| *** e0ne has quit IRC | 22:00 | |
| *** dkranz has quit IRC | 22:01 | |
| *** tonytan4ever has joined #openstack-infra | 22:02 | |
| jeblair | clarkb: i believe your comment #2 is correct | 22:03 |
| *** dustins has quit IRC | 22:03 | |
| clarkb | woot | 22:03 |
| jeblair | clarkb: i responded to comments now; let me know if my justification for not changing the thing from your #1 comment is insufficient :) | 22:04 |
| *** e0ne has joined #openstack-infra | 22:04 | |
| clarkb | looking | 22:04 |
| *** mriedem has quit IRC | 22:05 | |
| *** harlowja is now known as harlowja_away | 22:05 | |
| *** mikedillion has joined #openstack-infra | 22:06 | |
| *** emagana has quit IRC | 22:06 | |
| clarkb | jeblair: so the problem with that first thing is that we modify self.conn but we don't have a connection at that point aiui | 22:06 |
| clarkb | so I think you can do what ou have there if you add a connect method? | 22:06 |
| *** emagana has joined #openstack-infra | 22:07 | |
| *** dkliban is now known as dkliban_afk | 22:07 | |
| *** ddieterly has quit IRC | 22:07 | |
| clarkb | actually you don't need to add a connect method you just need to not try to set nonblocking in server connection init | 22:07 |
| *** emagana has quit IRC | 22:09 | |
| jeblair | clarkb: i mean, my real answer was that i didn't want to change the existing code more than necessary | 22:09 |
| clarkb | jeblair: oh wait | 22:09 |
| clarkb | server connection expects something else to connect for it? | 22:09 |
| *** emagana has joined #openstack-infra | 22:09 | |
| jeblair | clarkb: yes | 22:09 |
| clarkb | nevermind I think my concerns are not valid (particularly if you want to avoid a lot of changes) | 22:10 |
| clarkb | so I am good with what you have there | 22:10 |
| jeblair | ok. yeah, the connection process is _very_ different, it's just they share a lot in common once things get going | 22:10 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 22:10 |
| jeblair | the object model is probably all wrong for that :) | 22:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear: Use non-blocking IO in server https://review.openstack.org/128754 | 22:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 22:11 |
| *** imcsk8 has quit IRC | 22:11 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 22:11 |
| *** SumitNaiksatam has quit IRC | 22:11 | |
| *** imcsk8 has joined #openstack-infra | 22:11 | |
| jeblair | clarkb, fungi: ^ there's clarkb's point addressed, and also a bonus bug fix i noticed when testing that (it could truncate data it was sending if it blocked) | 22:11 |
| clarkb | so one of the things I said I would do is coming up, deprecating py26 across a larger set of projects. I didn't hear any complaining but will send a reminder this week | 22:12 |
| *** SumitNaiksatam has joined #openstack-infra | 22:12 | |
| clarkb | please do complain if appropriate :) | 22:12 |
| *** aysyd has quit IRC | 22:12 | |
| clarkb | mordred: we have been putting elements in project-config for nodepool. any reason to not continue doing that? | 22:12 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 22:12 |
| *** mattfarina has quit IRC | 22:12 | |
| mordred | clarkb: yeah. these are not nodepool elements | 22:12 |
| mordred | clarkb: these are elements for the servers in project_config | 22:13 |
| mordred | gah | 22:13 |
| mordred | clarkb: these are elements for the servers in system-config | 22:13 |
| clarkb | mordred: right but then we have to duplicate documentation and helper scripts? | 22:13 |
| *** jungleboyj has quit IRC | 22:13 | |
| mordred | clarkb: I believe the helper scripts for these are going to be slightly different | 22:14 |
| clarkb | ok | 22:14 |
| fungi | jeblair: clarkb: i think maybe i'm just turned around after staring at it for too long, but in the readPacket() method why append to input_buffer when testing for a byte present? shouldn't that byte be prepended back instead? | 22:14 |
| fungi | jeblair: clarkb: nevermind | 22:14 |
| fungi | i was looking at it wrong. grrr | 22:14 |
| mordred | clarkb: also - actually, I've submitted all of this other than the infra element to dib upstream | 22:14 |
| mordred | clarkb: so the patch should get much smaller before it lands | 22:15 |
| mordred | and that way if we want to use ubuntu-minimal for nodepool, we can without it being weird | 22:15 |
| clarkb | jeblair: one question about the bugfix you added. when the ssl exceptions are thrown we set r to 0. is that correct? we could have written >0 bytes then hit the exception ya? | 22:15 |
| clarkb | jeblair: might be better to have your outer r=0 then then r update as appropriate and fall through? | 22:15 |
| *** harlowja_away is now known as harlowja | 22:16 | |
| clarkb | I am leaving that as a ocmment on the review | 22:17 |
| *** shashankhegde has quit IRC | 22:17 | |
| clarkb | jeblair: left | 22:17 |
| *** andreykurilin_ has joined #openstack-infra | 22:17 | |
| jeblair | clarkb: oh sorry, back now (was looking at the other problem -- closing the connection after timeout) | 22:18 |
| jeblair | SpamapS: can you look at that? | 22:19 |
| *** sputnik13 has joined #openstack-infra | 22:19 | |
| jeblair | SpamapS: clarkb's comment on https://review.openstack.org/#/c/128754/ | 22:19 |
| sweston | asselin: separate repositories for each subtree? | 22:20 |
| *** bswartz has quit IRC | 22:20 | |
| asselin | sweston, yes | 22:20 |
| jeblair | clarkb: i'm not certain i understand all the implications, but i think you are right | 22:21 |
| asselin | sweston, steps 2, 3, 4 here: http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html | 22:21 |
| sweston | asselin: nice. looks like you can also merge from upstream without losing history | 22:21 |
| clarkb | jeblair: ya I don't either which is why I didn't -1, but I think my suggestion is a bit more defensive | 22:21 |
| jeblair | clarkb: (in particular, i do not know how to generate that error for testing :/) | 22:21 |
| asselin | sweston, how? | 22:21 |
| clarkb | my suggsetion shouldn't be wrong but may be more correct :) | 22:21 |
| jeblair | yp | 22:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear: Use non-blocking IO in server https://review.openstack.org/128754 | 22:22 |
| sweston | asselin: switch to the upstream, checkout, pull, switch to subtree master, then merge | 22:23 |
| *** emagana has quit IRC | 22:23 | |
| *** banix has quit IRC | 22:23 | |
| *** rlandy has quit IRC | 22:24 | |
| *** emagana has joined #openstack-infra | 22:24 | |
| *** emagana has quit IRC | 22:24 | |
| *** emagana has joined #openstack-infra | 22:25 | |
| *** baoli has joined #openstack-infra | 22:25 | |
| *** tonytan4ever has quit IRC | 22:25 | |
| asselin | sweston, so when you do the merge on the subtree, it will only take what's new? | 22:25 |
| clarkb | jeblair: +2 | 22:25 |
| *** dims has quit IRC | 22:25 | |
| *** dims has joined #openstack-infra | 22:26 | |
| sweston | asselin: it's supposed to work with git merge -s subtree | 22:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Reconnect to gearman on error https://review.openstack.org/136910 | 22:26 |
| jeblair | clarkb: ^ there's the other part of this | 22:26 |
| *** dims has quit IRC | 22:26 | |
| sweston | asselin: I have not tested this though, admittedly .. never this particular use case, hehe | 22:27 |
| jeblair | clarkb: what do you think about cowboy running those in production now, since things are still in flames? | 22:27 |
| clarkb | jeblair: I am cool with it | 22:27 |
| *** dims_ has joined #openstack-infra | 22:27 | |
| clarkb | let me review that second change | 22:27 |
| asselin | sweston, I can play around with it. It would certainly be faster than re-sub-treeing | 22:27 |
| *** dims_ has quit IRC | 22:28 | |
| sweston | asselin: cool, let me know how it goes ;-) | 22:28 |
| fungi | jeblair: yeah, no obvious bugs jumped out at me in 128754 anyway | 22:29 |
| clarkb | jeblair: lgtm. Did you want me to help cowboy that in? | 22:29 |
| jeblair | clarkb: i think i can do it real quick | 22:29 |
| fungi | and 136910 lgtm | 22:29 |
| jeblair | i've installed that gear on zuul.o.o. note that this is replacing the previously cowboy'd local patch to set the submit job timeout to 300 seconds | 22:31 |
| clarkb | noted | 22:31 |
| jeblair | i'm not happy about how long that stayed there :( | 22:31 |
| jeblair | i will also use that gear on nodepool; shouldn't make a difference, but just to be consistent | 22:32 |
| *** camunoz_gone is now known as camunoz | 22:32 | |
| *** achanda has quit IRC | 22:32 | |
| clarkb | sounds good | 22:33 |
| jeblair | i'm setting the geard log to INFO | 22:34 |
| fungi | long enough that i don't even remember why we'd overridden the submit job timeout | 22:34 |
| jeblair | fungi: because of this problem :) | 22:34 |
| jeblair | fungi: big response packets to nodepool with blocking io could freeze geard for > 30 seconds, especially if there was a network problem between zuul and nodepool | 22:35 |
| fungi | d'oh | 22:35 |
| fungi | right, that | 22:35 |
| jeblair | fungi: so zuul would drop/reconnect | 22:35 |
| fungi | so when we saw zuul timeout on it today, that was a full five minutes with no response | 22:36 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers. https://review.openstack.org/129609 | 22:36 |
| fungi | pretty bad | 22:36 |
| jeblair | yep | 22:37 |
| *** xyang0 has quit IRC | 22:37 | |
| jeblair | i stopped zuul and restarted nodepool | 22:37 |
| jeblair | the nodepool restart does not look good | 22:37 |
| jeblair | i don't understand the traceback in the debug log there | 22:38 |
| *** dims has joined #openstack-infra | 22:39 | |
| jeblair | oh! | 22:39 |
| jeblair | that's what happens when zuul is not running | 22:39 |
| jeblair | doh | 22:39 |
| fungi | right, clarkb noticed friday that it blocks waiting to connect to a gearman server | 22:40 |
| clarkb | ya took a while to figure out | 22:40 |
| clarkb | dove in with pdb too | 22:40 |
| *** e0ne has quit IRC | 22:40 | |
| jeblair | we should fix that | 22:40 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers. https://review.openstack.org/129609 | 22:40 |
| jeblair | it's just on the initial startup | 22:40 |
| *** ZZelle_ has joined #openstack-infra | 22:41 | |
| *** tsg_ has quit IRC | 22:42 | |
| jeblair | oh, still can't use info logging; too verbose. set it back to warning. | 22:43 |
| *** amitgandhinz has quit IRC | 22:47 | |
| *** ChuckC has quit IRC | 22:47 | |
| *** JayJ has quit IRC | 22:47 | |
| *** JayJ has joined #openstack-infra | 22:48 | |
| openstackgerrit | Eduardo Costa proposed openstack-infra/elastic-recheck: Add e-r query for bug 1372670 https://review.openstack.org/136915 | 22:49 |
| uvirtbot | Launchpad bug 1372670 in nova "libvirtError: operation failed: cannot read cputime for domain" [High,Confirmed] https://launchpad.net/bugs/1372670 | 22:49 |
| asselin | sweston, seems like it's going to work. will reconfirm when I know for sure. | 22:52 |
| sweston | asselin: sweet! | 22:53 |
| jeblair | RuntimeError: Set changed size during iteration | 22:54 |
| jeblair | that's showing up a lot in the log, but i believe it's not harmful | 22:55 |
| *** shakamunyi_ has joined #openstack-infra | 22:55 | |
| clarkb | jeblair: should probably copy the reader and writer sets before iterating on them? | 22:55 |
| jeblair | (i mean, we should fix it, but the exception handlers will be retrying it, so we're not losing data or anything) | 22:55 |
| mmedvede | asselin: be aware some history can be lost, e.g. compare your elasticsearch module with this https://github.com/mmedvede/puppet-elasticsearch | 22:56 |
| mmedvede | asselin: otherwise, this is rather good :) | 22:56 |
| jeblair | clarkb: yeah, i think so; not sure what we can do there atomically | 22:56 |
| * mordred knows some excellent atomic operators in C++ | 22:56 | |
| asselin | mmedvede, yea, I know. testing on the automation. We can redo then with a smarter script. | 22:56 |
| clarkb | jeblair: ya I would just be worried about starvation if sort order is stable during iteration | 22:57 |
| *** banix has joined #openstack-infra | 22:57 | |
| *** weshay has quit IRC | 22:57 | |
| jeblair | okay, so that started spewing out NOT_REGISTERED results | 22:58 |
| jeblair | i stopped zuul | 22:59 |
| *** ChuckC has joined #openstack-infra | 23:00 | |
| jeblair | i'm guessing because the nodepool is in such bad shape | 23:00 |
| *** camunoz has quit IRC | 23:00 | |
| *** sarob has quit IRC | 23:01 | |
| jeblair | i'm doing a mass delete in nodepool | 23:01 |
| clarkb | ok | 23:02 |
| *** shakamunyi_ has quit IRC | 23:04 | |
| *** dizquierdo has quit IRC | 23:04 | |
| *** otherwiseguy has quit IRC | 23:05 | |
| *** ecosta has joined #openstack-infra | 23:06 | |
| ci-testing | Hi, I am setting up my CI testbed, but am running into problem with running tempest test from the dvsm-tempest-full jenkins job. Running "sudo -H -u tempest tox -esmoke -- --concurrency=2" would produce this error: "InvocationError: '/bin/bash tools/pretty_tox.sh ...". | 23:08 |
| clarkb | ci-testing: without more of that error its hard to understand what went wrong. However that sounds like a tempest issue. you may have more luck debugging in #openstack-qa | 23:10 |
| *** banix has quit IRC | 23:11 | |
| *** achanda has joined #openstack-infra | 23:11 | |
| *** shashankhegde has joined #openstack-infra | 23:12 | |
| ci-testing | clarkb: ok will try that forum. I am following jaypipes guide on joinfu.com for setting up a openstack CI testing system, it 's sort of outdated. Do you know of any other good reference site,messages other then openstack-qa and openstack-infra? | 23:12 |
| *** camunoz has joined #openstack-infra | 23:13 | |
| clarkb | ci-testing: http://ci.openstack.org is a good place | 23:13 |
| ci-testing | Clarkb: ok thanks! | 23:14 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Loosen name restrictions for dedicated 3rd-parties https://review.openstack.org/135050 | 23:14 |
| asselin | ci-testing, you can try my fork of jaypipes's repo: https://github.com/rasselin/os-ext-testing | 23:15 |
| ci-testing | asselin: thanks will go over and retry/setup if needed. | 23:17 |
| *** emagana has quit IRC | 23:19 | |
| *** jgrimm is now known as zz_jgrimm | 23:20 | |
| *** emagana has joined #openstack-infra | 23:20 | |
| jeblair | mordred: how do i disable puppet on a host? | 23:20 |
| *** emagana has quit IRC | 23:21 | |
| *** AlexF has joined #openstack-infra | 23:21 | |
| fungi | 'puppet agent --disable' as root is what i've been doing | 23:21 |
| jeblair | fungi: thx | 23:21 |
| *** emagana has joined #openstack-infra | 23:21 | |
| mordred | yah | 23:21 |
| mordred | ansible does the right thing | 23:21 |
| fungi | alternatively, remove it temporarily from /etc/ansible/hostlist on puppetmaster | 23:21 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Precreate temp holding dir in static publish job https://review.openstack.org/136921 | 23:22 |
| *** otter768 has joined #openstack-infra | 23:22 | |
| *** dkranz has joined #openstack-infra | 23:23 | |
| clarkb | fungi: curious of what you think about https://etherpad.openstack.org/p/third-party-openid-accounts considering ^ | 23:23 |
| *** MaxV has joined #openstack-infra | 23:24 | |
| fungi | clarkb: i'm in favor, though i wonder what implications that has on account naming | 23:25 |
| clarkb | fungi: I think it would require individuals to edit that stuff themselves | 23:25 |
| clarkb | you can set it under contact info | 23:26 |
| fungi | clarkb: yep | 23:26 |
| fungi | clarkb: so i suppose those managing the allowed voting groups would only add accounts with conforming name patterns, and would remove them on report of name changes which brought them out of conformance | 23:26 |
| clarkb | ya | 23:27 |
| fungi | that'd be workable i think | 23:27 |
| clarkb | and I think we would still end up disabling some accounts, but a bulk of the work would be pushed onto account owners | 23:27 |
| *** otter768 has quit IRC | 23:27 | |
| clarkb | ssh key and email updates | 23:27 |
| clarkb | and account creation itself | 23:27 |
| fungi | how easy is it to juggle multiple lp accounts? | 23:28 |
| fungi | i've never tried | 23:28 |
| clarkb | fungi: its not too bad (I have had to juggle to gerrig before, worst thing is when you forget to switch and file a bug with wrong account >_>) | 23:28 |
| fungi | oh, wait, i have | 23:28 |
| clarkb | you basically log out then back in with other credentials | 23:28 |
| clarkb | in the future I will use browser privacy modes to avoid the bug thing | 23:29 |
| fungi | yeah, logout of lp, log into lp with other account, then relogin to gerrit via openid | 23:29 |
| fungi | did we also want to have a way to prevent these accounts from leaving code review votes? | 23:30 |
| jeblair | fungi: i left a -1 on 135050. | 23:31 |
| clarkb | fungi: I don't think we do that with existing accounts | 23:31 |
| *** Sukhdev has quit IRC | 23:31 | |
| jeblair | i don't feel that what we're doing to zuul and nodepool is going very well | 23:31 |
| *** dkranz has quit IRC | 23:31 | |
| *** thedodd has quit IRC | 23:31 | |
| clarkb | fungi: so I am not worried about it. but we could add DENY rules for those groups with +/-1 code review label | 23:31 |
| clarkb | jeblair: :/ | 23:31 |
| jeblair | if anyone wants to help me figure out what's going on, that'd be swell | 23:31 |
| clarkb | jeblair: can do. where should I look? | 23:31 |
| jeblair | i'm getting really close to the point of saying we should roll everything back to where we were last thurdays | 23:32 |
| jeblair | clarkb: i'm not sure what is or is not working at this point | 23:32 |
| fungi | jeblair: as in switch back to the old nodepool server? | 23:32 |
| jeblair | fungi: yeah | 23:32 |
| jeblair | clarkb: if you look at the status page, there are some NOT_REGISTERED entries there | 23:33 |
| jeblair | i think the zuul mergers are idle | 23:33 |
| *** banix has joined #openstack-infra | 23:34 | |
| fungi | oh, weird yeah... for multiple job names on 136798,1 where the same jobs are still pending or running on other changes ahead of it | 23:35 |
| clarkb | it looks like nodepools allocations are still off | 23:35 |
| jeblair | clarkb: is it because it's the icehouse branch? | 23:35 |
| clarkb | jeblair: oh ya that could do it since those are precise nodes | 23:35 |
| jeblair | i mean, do we need to shut the whole thing down for like an hour until we can finally build a precise node or something? | 23:35 |
| clarkb | and will register independently of the trusty nodes | 23:35 |
| fungi | jeblair: it's possible nodepool hasn't built any new devstack-precise nodes for that yet | 23:35 |
| jeblair | clarkb: can you verify that? | 23:36 |
| fungi | two building and a bunch delete | 23:36 |
| fungi | no ready/used | 23:36 |
| clarkb | jeblair: yes, will look in zuul logs to confirm | 23:36 |
| clarkb | nodepool did just start launching a devstack-precise node too | 23:36 |
| fungi | one has been building for about 5 minutes | 23:37 |
| *** marun has quit IRC | 23:37 | |
| *** MaxV has quit IRC | 23:39 | |
| clarkb | 'ZUUL_NODE': 'devstack-precise' thats in the job parameter list | 23:39 |
| clarkb | 2014-11-24 23:34:02,871 ERROR zuul.Gearman: Job <gear.Job 0x7f70f572f310 handle: None name: build:gate-devstack-dsvm-cells:devstack-precise unique: 87b12bcea2e94fd3b52e7c9c6b930203> is not registered with Gearman | 23:40 |
| clarkb | jeblair: I think that confirms it | 23:40 |
| jeblair | okay, so that's just a matter of waiting... | 23:40 |
| *** rcarrillocruz has quit IRC | 23:40 | |
| jeblair | what could be the problem with the mergers? | 23:41 |
| fungi | and definitely still seeing "GearmanClient: Exception while listing functions" in the nodepool logs since the restart | 23:41 |
| *** rcarrillocruz has joined #openstack-infra | 23:41 | |
| jeblair | fungi: i've restarted it a lot | 23:41 |
| clarkb | jeblair: maybe they have CLOSE WAIT connections too? | 23:41 |
| *** banix has quit IRC | 23:41 | |
| * clarkb hops on a merger | 23:41 | |
| fungi | most recent log entry for that was from 1 minute ago | 23:41 |
| jeblair | fungi: what's the exception? | 23:41 |
| fungi | jeblair: TimeoutError | 23:42 |
| jeblair | clarkb: they seem to each run one merge operation after the restart | 23:42 |
| fungi | 2014-11-24 23:40:04 | 23:42 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Add glance_store, kite, python-kiteclient to projects.txt https://review.openstack.org/135603 | 23:42 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: Add oslo.context to devstack-vm-gate-wrap.sh https://review.openstack.org/135093 | 23:42 |
| clarkb | ya nodepool's allocation numbers don't look right to me (seem to be min ready) | 23:43 |
| jeblair | clarkb: there's very little load | 23:43 |
| jeblair | clarkb: because zuul is stuck waiting on merges | 23:43 |
| clarkb | oh I see | 23:43 |
| *** rhe00 has joined #openstack-infra | 23:44 | |
| *** ddieterly has joined #openstack-infra | 23:44 | |
| mordred | every time I think I have an idea someone else says something which tells me I was wrong | 23:44 |
| clarkb | zm01 definitely isn't doing much but its got a connection that looks good from both sides | 23:44 |
| clarkb | jeblair: is it possible that the set() modifications are starving merger connections? | 23:44 |
| clarkb | jeblair: we don't service them because the set is modified quickly enough to short circuit iteration | 23:44 |
| jeblair | clarkb: i restarted that with .copy() | 23:44 |
| fungi | jeblair: though gearman status reports merger:update and merger:merge with 8 workers and no waiting tasks | 23:44 |
| clarkb | zuul merger is sitting on a pause() call according to strace | 23:45 |
| jeblair | something just changed | 23:45 |
| *** armax has quit IRC | 23:46 | |
| jeblair | i think some mergers ran some jobs | 23:46 |
| clarkb | ya at least zm01 did | 23:46 |
| clarkb | all I did was strace the merger process which shouldn't interfer | 23:46 |
| jeblair | they all did | 23:46 |
| clarkb | http://paste.openstack.org/show/137770/ is from the log | 23:47 |
| fungi | yeah, another stable/icehouse change popped not_registered for stuff and then the gate queue changes after it got recalculated | 23:47 |
| fungi | which would explain sudden activity for merge workers | 23:47 |
| jeblair | the enqueue calls haven't finished yet | 23:48 |
| fungi | oh | 23:48 |
| *** oomichi has joined #openstack-infra | 23:48 | |
| jeblair | i assume they are waiting on mergers | 23:48 |
| jeblair | i think there's far too much going wrong right now to debug and am burning out | 23:50 |
| *** MaxV has joined #openstack-infra | 23:50 | |
| jeblair | if we want to roll back to the old nodepool server, i think we should do so soon | 23:50 |
| clarkb | jeblair: looking at the zuul process list I don't see what I expect | 23:50 |
| fungi | of the two devstack-precise nodes which were booting, the one in rax seems to have errored and the other has been building in hpcloud for about 20 minutes now | 23:50 |
| clarkb | usually you get zuuld and a child which is geard | 23:50 |
| jeblair | clarkb: i'm running geard separately | 23:50 |
| clarkb | ok | 23:50 |
| jeblair | clarkb: so that i can restart it and zuul separately, so that it might accumulate function registrations to avoid spurious not_registered | 23:51 |
| mordred | jeblair: I think rolling back to old nodepool is at least a thing that's worth trying | 23:51 |
| mordred | jeblair: no idea why that would affect zuul mergers - but its at least _one_ variable that could be removed | 23:51 |
| clarkb | stracing that geard its sitting on a select | 23:52 |
| clarkb | so I think the epoll edges are not being tripped? | 23:52 |
| jeblair | mordred: i strongly suspect the async io is broken | 23:52 |
| clarkb | ya | 23:52 |
| jeblair | mordred: which is what's wrong with the zuul mergers | 23:52 |
| *** AlexF has quit IRC | 23:52 | |
| mordred | nod | 23:52 |
| jeblair | but the reason we're doing anything with async io today is because something was broken between zuul and nodepool this morning | 23:52 |
| jeblair | we thought it might make things better, but it seems to only be adding problems | 23:53 |
| mordred | yah. so you're saying "rollback nodepool, and also rollback async io" | 23:53 |
| jeblair | yes | 23:53 |
| *** JayJ has quit IRC | 23:54 | |
| mordred | I think that sounds like the sane thing | 23:54 |
| *** banix has joined #openstack-infra | 23:54 | |
| * clarkb notes that the read fds list is really small | 23:54 | |
| jeblair | clarkb: how can you see it? | 23:54 |
| clarkb | jeblair: strace | 23:54 |
| jeblair | i think i don't understand what you're referring to | 23:55 |
| clarkb | select(6, [3 5], [], NULL, NULL | 23:55 |
| jeblair | oh | 23:55 |
| clarkb | jeblair: I am stracing geard | 23:55 |
| jeblair | clarkb: edge triggering | 23:55 |
| asselin | sweston, actually it didn't work. I'm trying this instead: --rejoin to 'save state' git subtree -q split --prefix=modules/$module --branch=$module --rejoin | 23:55 |
| jeblair | clarkb: only shows up there when one of them changes | 23:55 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 23:55 |
| jeblair | clarkb, mordred, fungi: i have a hard stop in 1 hour. how would you like to proceed? | 23:56 |
| fungi | i'm trying to confirm the images in old nodepool are still around | 23:56 |
| jeblair | i'm happy to continue debugging, but i don't know if we'll be where we want in an hour | 23:56 |
| clarkb | fungi: they should be, I didn't delete any of them and nodepool was turned off. | 23:56 |
| clarkb | jeblair: but that is a select call | 23:57 |
| clarkb | jeblair: or is that coming from some unrelated portion of geard? | 23:58 |
| *** amitgandhinz has joined #openstack-infra | 23:58 | |
| *** camunoz has quit IRC | 23:58 | |
| *** andreykurilin_ has quit IRC | 23:58 | |
| clarkb | anyways I am ok with rolling back | 23:58 |
| *** sarob has joined #openstack-infra | 23:58 | |
| mordred | jeblair: steps should be "turn off new nodepool, delete a bunch of nodes, turn on old nodepool" - yah? | 23:58 |
| clarkb | it is why I was conservative | 23:59 |
| clarkb | mordred: also update dns and iptables | 23:59 |
| mordred | ++ | 23:59 |
| jeblair | and re-install old version of gear (with 300 second patch) to zuul | 23:59 |
| fungi | yep, images look like they're still around | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!