| *** dstanek_afk has quit IRC | 00:02 | |
| *** tma996 has quit IRC | 00:02 | |
| *** yamahata has joined #openstack-infra | 00:19 | |
| *** dstanek_afk has joined #openstack-infra | 00:19 | |
| *** yamahata__ has quit IRC | 00:20 | |
| *** rakhmerov has joined #openstack-infra | 00:20 | |
| *** tma996 has joined #openstack-infra | 00:23 | |
| *** rakhmerov has quit IRC | 00:24 | |
| *** sarob has joined #openstack-infra | 00:26 | |
| *** sarob has quit IRC | 00:31 | |
| *** yamahata_ has joined #openstack-infra | 00:42 | |
| *** rongze has joined #openstack-infra | 00:44 | |
| *** rongze has quit IRC | 00:49 | |
| *** yamahata has quit IRC | 00:54 | |
| *** tma996_ has joined #openstack-infra | 00:54 | |
| *** tma996 has quit IRC | 00:54 | |
| *** Guest90126 has joined #openstack-infra | 01:00 | |
| *** sarob has joined #openstack-infra | 01:00 | |
| *** sarob has quit IRC | 01:05 | |
| *** nosnos has joined #openstack-infra | 01:06 | |
| *** fallenpegasus has joined #openstack-infra | 01:08 | |
| *** dkranz has quit IRC | 01:13 | |
| *** dstufft__ is now known as dstufft | 01:14 | |
| *** senk has joined #openstack-infra | 01:18 | |
| *** rakhmerov has joined #openstack-infra | 01:20 | |
| *** senk has quit IRC | 01:24 | |
| *** rakhmerov has quit IRC | 01:25 | |
| *** yaguang has joined #openstack-infra | 01:25 | |
| *** tma996_ has quit IRC | 01:28 | |
| *** yaguang has quit IRC | 01:28 | |
| *** fifieldt has joined #openstack-infra | 01:32 | |
| *** senk has joined #openstack-infra | 01:45 | |
| *** yaguang has joined #openstack-infra | 01:46 | |
| *** herndon has joined #openstack-infra | 01:48 | |
| *** dstanek_afk has quit IRC | 01:49 | |
| *** herndon has quit IRC | 01:51 | |
| *** tma996 has joined #openstack-infra | 01:53 | |
| *** AaronGr_Zzz is now known as AaronGr | 01:54 | |
| *** mlipchuk has joined #openstack-infra | 01:54 | |
| mlipchuk | hi all, can someone help me regarding adding adding contact information in review.openstack.org | 01:57 |
|---|---|---|
| *** fallenpegasus has quit IRC | 01:57 | |
| fungi | mlipchuk: have you read https://wiki.openstack.org/wiki/CLA-FAQ | 01:59 |
| *** rongze has joined #openstack-infra | 02:00 | |
| fungi | i tried to answer most of the common questions we get in that article | 02:00 |
| fungi | (cla/contact info related questions anyway) | 02:00 |
| mlipchuk | fungi: thanks | 02:01 |
| mlipchuk | fungi: I will try to look into it | 02:01 |
| *** fallenpegasus has joined #openstack-infra | 02:02 | |
| *** fallenpegasus has quit IRC | 02:04 | |
| *** dstanek_afk has joined #openstack-infra | 02:09 | |
| *** senk has quit IRC | 02:10 | |
| *** fallenpegasus has joined #openstack-infra | 02:14 | |
| *** dcramer_ has joined #openstack-infra | 02:17 | |
| *** fallenpegasus has quit IRC | 02:21 | |
| *** rakhmerov has joined #openstack-infra | 02:21 | |
| *** fallenpegasus has joined #openstack-infra | 02:22 | |
| *** rakhmerov has quit IRC | 02:26 | |
| *** sarob has joined #openstack-infra | 02:26 | |
| *** tma996 has quit IRC | 02:27 | |
| *** fallenpegasus has quit IRC | 02:28 | |
| *** sarob has quit IRC | 02:31 | |
| *** fallenpegasus has joined #openstack-infra | 02:33 | |
| *** senk has joined #openstack-infra | 02:34 | |
| *** yamahata has joined #openstack-infra | 02:36 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: [WIP] Add Storyboard puppet module https://review.openstack.org/65017 | 02:36 |
| *** sdake_ has quit IRC | 02:39 | |
| *** sdake has joined #openstack-infra | 02:39 | |
| *** sdake has quit IRC | 02:39 | |
| *** sdake has joined #openstack-infra | 02:39 | |
| *** _ruhe is now known as ruhe | 02:40 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: [WIP] Add Storyboard puppet module https://review.openstack.org/65017 | 02:46 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/elastic-recheck: Tox allow install of lazr.authentication https://review.openstack.org/64898 | 02:51 |
| *** fallenpegasus has quit IRC | 02:52 | |
| *** banix has quit IRC | 02:54 | |
| *** fallenpegasus has joined #openstack-infra | 02:54 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: [WIP] Add Storyboard puppet module https://review.openstack.org/65017 | 02:55 |
| *** fallenpegasus has quit IRC | 02:57 | |
| *** fallenpegasus has joined #openstack-infra | 02:57 | |
| *** ruhe is now known as ruhe_away | 03:01 | |
| *** mlipchuk has quit IRC | 03:09 | |
| *** ruhe_away is now known as _ruhe | 03:20 | |
| *** rakhmerov has joined #openstack-infra | 03:22 | |
| openstackgerrit | Zhiwei Chen proposed a change to openstack-infra/config: Rename a StackForge project https://review.openstack.org/64788 | 03:23 |
| *** tma996 has joined #openstack-infra | 03:24 | |
| *** CaptTofu has quit IRC | 03:24 | |
| *** CaptTofu has joined #openstack-infra | 03:24 | |
| *** rakhmerov has quit IRC | 03:25 | |
| *** loq_mac has joined #openstack-infra | 03:26 | |
| *** sarob has joined #openstack-infra | 03:26 | |
| *** fallenpegasus has quit IRC | 03:28 | |
| *** senk has quit IRC | 03:28 | |
| *** tma996 has quit IRC | 03:28 | |
| *** sarob has quit IRC | 03:31 | |
| *** fallenpegasus has joined #openstack-infra | 03:33 | |
| *** rcleere has joined #openstack-infra | 03:37 | |
| *** mriedem1 has quit IRC | 03:42 | |
| *** slong_ has joined #openstack-infra | 03:48 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/reviewday: Whitelist external lazr.authentication requirement https://review.openstack.org/65026 | 03:48 |
| *** slong has quit IRC | 03:48 | |
| *** CaptTofu has quit IRC | 03:55 | |
| *** CaptTofu has joined #openstack-infra | 03:56 | |
| *** banix has joined #openstack-infra | 03:56 | |
| *** fallenpegasus has quit IRC | 03:57 | |
| *** fallenpegasus has joined #openstack-infra | 03:58 | |
| *** dcramer_ has quit IRC | 04:04 | |
| *** senk has joined #openstack-infra | 04:14 | |
| *** fallenpegasus has quit IRC | 04:20 | |
| *** coolsvap has joined #openstack-infra | 04:20 | |
| *** tma996 has joined #openstack-infra | 04:24 | |
| *** senk has quit IRC | 04:25 | |
| *** sarob has joined #openstack-infra | 04:26 | |
| *** loq_mac has quit IRC | 04:26 | |
| *** sarob has quit IRC | 04:31 | |
| *** slong_ has quit IRC | 04:37 | |
| *** Guest90126 has quit IRC | 04:39 | |
| *** rongze has quit IRC | 04:40 | |
| *** medberry has joined #openstack-infra | 04:47 | |
| *** medberry has joined #openstack-infra | 04:47 | |
| *** chandankumar has joined #openstack-infra | 04:48 | |
| *** tma996 has quit IRC | 04:58 | |
| *** fallenpegasus has joined #openstack-infra | 05:02 | |
| *** fallenpegasus has quit IRC | 05:12 | |
| *** dstanek_afk has quit IRC | 05:12 | |
| *** chandankumar has quit IRC | 05:16 | |
| *** fallenpegasus has joined #openstack-infra | 05:16 | |
| *** sarob has joined #openstack-infra | 05:18 | |
| *** dstanek_afk has joined #openstack-infra | 05:18 | |
| *** moted has quit IRC | 05:20 | |
| *** chandankumar has joined #openstack-infra | 05:23 | |
| *** sarob has quit IRC | 05:23 | |
| *** rakhmerov has joined #openstack-infra | 05:23 | |
| *** ryanpetrello has joined #openstack-infra | 05:25 | |
| *** talluri has joined #openstack-infra | 05:27 | |
| *** rakhmerov has quit IRC | 05:28 | |
| *** sarob has joined #openstack-infra | 05:30 | |
| *** sarob has quit IRC | 05:33 | |
| *** sarob has joined #openstack-infra | 05:33 | |
| *** nosnos_ has joined #openstack-infra | 05:34 | |
| *** nosnos__ has joined #openstack-infra | 05:36 | |
| *** sarob has quit IRC | 05:37 | |
| *** nosnos_ has quit IRC | 05:37 | |
| *** sarob has joined #openstack-infra | 05:37 | |
| *** nosnos has quit IRC | 05:38 | |
| *** sarob has quit IRC | 05:39 | |
| *** sarob has joined #openstack-infra | 05:42 | |
| *** loq_mac has joined #openstack-infra | 05:42 | |
| *** ryanpetrello has quit IRC | 05:42 | |
| *** fallenpegasus has quit IRC | 05:43 | |
| *** fallenpegasus has joined #openstack-infra | 05:45 | |
| *** garyk has quit IRC | 05:48 | |
| *** dstanek_afk has quit IRC | 05:50 | |
| openstackgerrit | Christopher Yeoh proposed a change to openstack-infra/elastic-recheck: Add query for bug 1265672 https://review.openstack.org/64885 | 05:51 |
| *** tma996 has joined #openstack-infra | 05:55 | |
| *** mlipchuk has joined #openstack-infra | 05:58 | |
| *** boris-42 has joined #openstack-infra | 06:03 | |
| *** nati_ueno has joined #openstack-infra | 06:05 | |
| *** fallenpegasus2 has joined #openstack-infra | 06:14 | |
| *** fallenpegasus has quit IRC | 06:14 | |
| *** zhiyan has joined #openstack-infra | 06:15 | |
| *** rongze has joined #openstack-infra | 06:18 | |
| *** rongze_ has joined #openstack-infra | 06:21 | |
| *** rakhmerov has joined #openstack-infra | 06:24 | |
| *** rongze has quit IRC | 06:25 | |
| *** rakhmerov has quit IRC | 06:29 | |
| *** tma996 has quit IRC | 06:29 | |
| *** fallenpegasus2 has quit IRC | 06:34 | |
| *** fallenpegasus has joined #openstack-infra | 06:38 | |
| *** jcoufal has joined #openstack-infra | 06:39 | |
| *** fallenpegasus has quit IRC | 06:43 | |
| *** fallenpegasus has joined #openstack-infra | 06:44 | |
| *** mkoderer_ is now known as mkoderer | 06:57 | |
| *** banix has quit IRC | 07:03 | |
| *** fallenpegasus has quit IRC | 07:05 | |
| *** nosnos__ has quit IRC | 07:06 | |
| *** nosnos has joined #openstack-infra | 07:06 | |
| *** garyk has joined #openstack-infra | 07:14 | |
| *** loq_mac has quit IRC | 07:15 | |
| *** sarob has quit IRC | 07:16 | |
| *** persia has joined #openstack-infra | 07:17 | |
| *** sarob has joined #openstack-infra | 07:18 | |
| *** rcleere has quit IRC | 07:21 | |
| *** sarob has quit IRC | 07:22 | |
| *** tma996 has joined #openstack-infra | 07:24 | |
| *** zhiyan has quit IRC | 07:24 | |
| *** rakhmerov has joined #openstack-infra | 07:25 | |
| *** rakhmerov has quit IRC | 07:29 | |
| *** zhiyan has joined #openstack-infra | 07:31 | |
| *** sarob has joined #openstack-infra | 07:32 | |
| *** sarob has quit IRC | 07:35 | |
| *** loq_mac has joined #openstack-infra | 07:35 | |
| *** fallenpegasus has joined #openstack-infra | 07:37 | |
| *** yolanda_ has joined #openstack-infra | 07:39 | |
| *** afazekas has joined #openstack-infra | 07:42 | |
| *** NikitaKonovalov has joined #openstack-infra | 07:45 | |
| *** NikitaKonovalov has quit IRC | 07:45 | |
| clarkb2 | notmyname: setup.py test not setup.py testr | 07:45 |
| notmyname | clarkb2: error: option --slowest not recognized | 07:46 |
| clarkb2 | gah | 07:46 |
| * clarkb2 thinks more | 07:46 | |
| *** afazekas has quit IRC | 07:47 | |
| *** persia_ has joined #openstack-infra | 07:48 | |
| *** cyeoh has joined #openstack-infra | 07:48 | |
| *** briancline has joined #openstack-infra | 07:48 | |
| *** Alex_Gaynor has joined #openstack-infra | 07:48 | |
| *** zigo has joined #openstack-infra | 07:48 | |
| *** morganfainberg has joined #openstack-infra | 07:48 | |
| *** persia_ is now known as Guest60665 | 07:49 | |
| clarkb2 | notmyname: is testrepository installed? | 07:49 |
| notmyname | $ testr list-tests | wc -l | 07:50 |
| notmyname | 2046 | 07:50 |
| *** tma996 has quit IRC | 07:59 | |
| *** flaper87|afk is now known as flaper87 | 08:00 | |
| *** afazekas has joined #openstack-infra | 08:01 | |
| *** yolanda_ has quit IRC | 08:02 | |
| *** yolanda_ has joined #openstack-infra | 08:02 | |
| *** Guest60665 has quit IRC | 08:02 | |
| *** Guest60665 has joined #openstack-infra | 08:02 | |
| *** persia is now known as Guest26499 | 08:03 | |
| *** Guest60665 is now known as persia | 08:03 | |
| *** persia__ has joined #openstack-infra | 08:03 | |
| *** persia__ is now known as persia_ | 08:03 | |
| *** persia_ has quit IRC | 08:04 | |
| *** persia_ has joined #openstack-infra | 08:04 | |
| *** fallenpegasus has quit IRC | 08:07 | |
| *** persia_ has left #openstack-infra | 08:07 | |
| *** fallenpegasus has joined #openstack-infra | 08:08 | |
| *** mlipchuk has quit IRC | 08:10 | |
| *** jroovers has joined #openstack-infra | 08:11 | |
| *** fbo is now known as fbo_away | 08:12 | |
| openstackgerrit | SlickNik proposed a change to openstack-infra/devstack-gate: Add Trove testing support https://review.openstack.org/65040 | 08:13 |
| *** yolanda_ has quit IRC | 08:15 | |
| *** yolanda_ has joined #openstack-infra | 08:16 | |
| *** yolanda_ has quit IRC | 08:16 | |
| *** Adri2000 has joined #openstack-infra | 08:17 | |
| *** Adri2000 has joined #openstack-infra | 08:17 | |
| *** yolanda has joined #openstack-infra | 08:17 | |
| *** yolanda has joined #openstack-infra | 08:19 | |
| *** pblaho has joined #openstack-infra | 08:19 | |
| *** loq_mac has quit IRC | 08:21 | |
| *** loq_mac has joined #openstack-infra | 08:23 | |
| *** zhiyan has quit IRC | 08:23 | |
| *** jroovers has quit IRC | 08:24 | |
| *** zhiyan has joined #openstack-infra | 08:25 | |
| *** rakhmerov has joined #openstack-infra | 08:26 | |
| *** jooools has joined #openstack-infra | 08:28 | |
| *** yolanda has quit IRC | 08:29 | |
| *** mlipchuk has joined #openstack-infra | 08:29 | |
| *** yolanda has joined #openstack-infra | 08:29 | |
| *** rakhmerov has quit IRC | 08:30 | |
| *** zehicle has quit IRC | 08:35 | |
| *** zehicle_at_dell has quit IRC | 08:35 | |
| *** fallenpegasus has quit IRC | 08:37 | |
| *** fallenpegasus has joined #openstack-infra | 08:38 | |
| *** zehicle_at_dell has joined #openstack-infra | 08:40 | |
| *** zehicle has joined #openstack-infra | 08:40 | |
| *** mtreinish has quit IRC | 08:52 | |
| *** tma996 has joined #openstack-infra | 08:54 | |
| *** mtreinish has joined #openstack-infra | 08:55 | |
| *** yassine has joined #openstack-infra | 08:56 | |
| *** yolanda has quit IRC | 08:58 | |
| *** yolanda has joined #openstack-infra | 08:59 | |
| *** tma996 has quit IRC | 08:59 | |
| *** jroovers has joined #openstack-infra | 09:02 | |
| *** jroovers has quit IRC | 09:02 | |
| *** loq_mac has quit IRC | 09:02 | |
| *** loq_mac has joined #openstack-infra | 09:03 | |
| *** jpich has joined #openstack-infra | 09:04 | |
| *** loq_mac has quit IRC | 09:04 | |
| *** pblaho has quit IRC | 09:05 | |
| *** loq_mac has joined #openstack-infra | 09:05 | |
| *** fallenpegasus has quit IRC | 09:07 | |
| *** fallenpegasus has joined #openstack-infra | 09:08 | |
| *** derekh has joined #openstack-infra | 09:10 | |
| *** praneshp has quit IRC | 09:10 | |
| *** yolanda has quit IRC | 09:15 | |
| *** yolanda has joined #openstack-infra | 09:18 | |
| *** mlipchuk1 has joined #openstack-infra | 09:19 | |
| *** andreaf has joined #openstack-infra | 09:20 | |
| *** mlipchuk has quit IRC | 09:23 | |
| *** rongze_ has quit IRC | 09:23 | |
| *** pblaho has joined #openstack-infra | 09:23 | |
| *** talluri has quit IRC | 09:24 | |
| *** rakhmerov has joined #openstack-infra | 09:27 | |
| *** yolanda has quit IRC | 09:29 | |
| *** yolanda has joined #openstack-infra | 09:29 | |
| *** rakhmerov has quit IRC | 09:31 | |
| *** yolanda has quit IRC | 09:32 | |
| *** johnthetubaguy has joined #openstack-infra | 09:32 | |
| *** yolanda has joined #openstack-infra | 09:33 | |
| *** fallenpegasus has quit IRC | 09:37 | |
| *** fallenpegasus has joined #openstack-infra | 09:38 | |
| *** fbo_away is now known as fbo | 09:40 | |
| *** loq_mac has quit IRC | 09:43 | |
| *** rongze has joined #openstack-infra | 09:43 | |
| *** yolanda has quit IRC | 09:43 | |
| *** fallenpegasus has quit IRC | 09:45 | |
| *** zhiyan has quit IRC | 09:46 | |
| *** zhiyan has joined #openstack-infra | 09:46 | |
| *** fallenpegasus has joined #openstack-infra | 09:48 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:48 | |
| *** johnthetubaguy has quit IRC | 09:51 | |
| *** nati_ueno has quit IRC | 09:54 | |
| *** tma996 has joined #openstack-infra | 09:55 | |
| *** fallenpegasus has quit IRC | 09:57 | |
| *** enikanorov has joined #openstack-infra | 09:58 | |
| *** loq_mac has joined #openstack-infra | 09:59 | |
| *** tma996 has quit IRC | 10:00 | |
| *** yolanda has joined #openstack-infra | 10:01 | |
| *** nati_ueno has joined #openstack-infra | 10:01 | |
| *** yolanda has quit IRC | 10:02 | |
| *** yolanda has joined #openstack-infra | 10:03 | |
| *** loq_mac has quit IRC | 10:04 | |
| *** loq_mac has joined #openstack-infra | 10:04 | |
| *** zhiyan has quit IRC | 10:07 | |
| *** mkerrin has joined #openstack-infra | 10:09 | |
| *** hashar has joined #openstack-infra | 10:09 | |
| *** che-arne has joined #openstack-infra | 10:13 | |
| *** yassine has quit IRC | 10:13 | |
| *** yassine has joined #openstack-infra | 10:14 | |
| *** mancdaz_away is now known as mancdaz | 10:18 | |
| *** yolanda has quit IRC | 10:18 | |
| *** yolanda has joined #openstack-infra | 10:19 | |
| *** yolanda has quit IRC | 10:19 | |
| *** yolanda has joined #openstack-infra | 10:20 | |
| *** rakhmerov has joined #openstack-infra | 10:28 | |
| *** rakhmerov has quit IRC | 10:32 | |
| *** yolanda has quit IRC | 10:35 | |
| *** yolanda has joined #openstack-infra | 10:36 | |
| *** loq_mac has quit IRC | 10:38 | |
| *** johnthetubaguy has joined #openstack-infra | 10:40 | |
| *** yassine has quit IRC | 10:40 | |
| *** johnthetubaguy1 has quit IRC | 10:42 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/config: Initial commit of a XenServer builder script https://review.openstack.org/60581 | 10:45 |
| *** yolanda has quit IRC | 10:51 | |
| *** yassine has joined #openstack-infra | 10:52 | |
| *** coolsvap has quit IRC | 10:54 | |
| *** tma996 has joined #openstack-infra | 10:56 | |
| *** nati_ueno has quit IRC | 10:58 | |
| *** rfolco has joined #openstack-infra | 10:59 | |
| *** rongze has quit IRC | 11:04 | |
| *** rongze has joined #openstack-infra | 11:05 | |
| *** yolanda has joined #openstack-infra | 11:07 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/61463 | 11:07 |
| *** yolanda has quit IRC | 11:07 | |
| *** yamahata has quit IRC | 11:08 | |
| *** yolanda has joined #openstack-infra | 11:09 | |
| *** rongze has quit IRC | 11:09 | |
| *** ihrachys has quit IRC | 11:12 | |
| *** yaguang has quit IRC | 11:16 | |
| *** rakhmerov has joined #openstack-infra | 11:28 | |
| *** yolanda has quit IRC | 11:29 | |
| *** yolanda has joined #openstack-infra | 11:30 | |
| *** tma996 has quit IRC | 11:30 | |
| *** tma996 has joined #openstack-infra | 11:30 | |
| *** rakhmerov has quit IRC | 11:32 | |
| *** rongze has joined #openstack-infra | 11:35 | |
| *** coolsvap has joined #openstack-infra | 11:36 | |
| *** ArxCruz has joined #openstack-infra | 11:37 | |
| *** rongze has quit IRC | 11:40 | |
| *** hashar has quit IRC | 11:40 | |
| openstackgerrit | SlickNik proposed a change to openstack-infra/config: Update devstack-gate jobs for Trove tempest tests https://review.openstack.org/65065 | 11:44 |
| *** yassine has quit IRC | 11:46 | |
| *** CaptTofu has quit IRC | 11:52 | |
| *** CaptTofu has joined #openstack-infra | 11:52 | |
| *** hashar has joined #openstack-infra | 11:54 | |
| *** coolsvap has quit IRC | 11:55 | |
| *** dims_ is now known as dims | 11:59 | |
| *** talluri has joined #openstack-infra | 12:06 | |
| *** yidclare has quit IRC | 12:06 | |
| *** yidclare has joined #openstack-infra | 12:10 | |
| *** yamahata has joined #openstack-infra | 12:11 | |
| *** mlipchuk1 has quit IRC | 12:17 | |
| *** smarcet has joined #openstack-infra | 12:19 | |
| *** rongze has joined #openstack-infra | 12:35 | |
| *** nosnos has quit IRC | 12:36 | |
| *** rongze has quit IRC | 12:39 | |
| *** banix has joined #openstack-infra | 12:46 | |
| *** mlipchuk has joined #openstack-infra | 12:55 | |
| *** banix has quit IRC | 12:56 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Tox allow install of lazr.authentication https://review.openstack.org/64898 | 12:56 |
| *** yassine has joined #openstack-infra | 12:57 | |
| *** dkliban has quit IRC | 13:00 | |
| ArxCruz | does anyone having this problem: Could not find a version that satisfies the requirement netaddr>=0.7.6 (from -r /opt/stack/tempest/requirements.txt (line 10)) (from versions: 0.3.1, 0.3.1, 0.4, 0.4, 0.5.1, 0.5.1, 0.5.2, 0.5.2, 0.5, 0.5, 0.6.1, 0.6.1, 0.6.2, 0.6.2, 0.6.3, 0.6.3, 0.6.4, 0.6.4, 0.6, 0.6, 0.7.1, 0.7.1, 0.7.2, 0.7.2, 0.7.3, 0.7.3, 0.7, 0.7) | 13:08 |
| maurosr | ArxCruz: pip 1.5 http://lists.openstack.org/pipermail/openstack-dev/2014-January/023336.html | 13:11 |
| *** coolsvap has joined #openstack-infra | 13:12 | |
| *** mugsie_ is now known as mugsie | 13:13 | |
| *** mugsie has joined #openstack-infra | 13:13 | |
| *** weshay has joined #openstack-infra | 13:16 | |
| *** zul has quit IRC | 13:16 | |
| *** juice has quit IRC | 13:17 | |
| *** sandywalsh has joined #openstack-infra | 13:18 | |
| ArxCruz | maurosr: ty | 13:19 |
| *** juice has joined #openstack-infra | 13:19 | |
| *** zul has joined #openstack-infra | 13:20 | |
| ArxCruz | ALL: When I try run tempest with run_tests.sh I got a huge output with a list of tests and it says 0 tests were run does anyone had this problem before? and how it was fixed ? | 13:20 |
| *** rongze has joined #openstack-infra | 13:25 | |
| *** CaptTofu has quit IRC | 13:29 | |
| *** CaptTofu has joined #openstack-infra | 13:29 | |
| *** coolsvap has quit IRC | 13:42 | |
| clarkb2 | ArxCruz: try tox instead? | 13:43 |
| ArxCruz | clarkb2: tox works, but I want to run one single test | 13:43 |
| clarkb2 | ArxCruz you can do that with tox but must use the full env. so tox -efull nameoftest | 13:44 |
| ArxCruz | clarkb2: okay, I will give a try, so, can I assume run_tests.sh is broken somehow ? | 13:45 |
| clarkb2 | thats my guess as it isnt gated | 13:45 |
| ArxCruz | clarkb2: funny part is run_test.sh calls testr to run | 13:47 |
| ArxCruz | if I call testr I'm able to run the test | 13:47 |
| *** chandankumar has quit IRC | 13:47 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add support for Python 3 requirements https://review.openstack.org/58770 | 13:48 |
| *** CaptTofu has quit IRC | 13:49 | |
| *** CaptTofu has joined #openstack-infra | 13:50 | |
| *** yamahata has quit IRC | 13:50 | |
| *** herndon has joined #openstack-infra | 13:51 | |
| derekh | Hi, I'm setting up the tripleo-ci hosts, the jenkins nodes from nodepool will need to ssh to these hosts, any idea what would be the best way to distribute keys for ssh, I would need a public key for the jenkins node on the test env host. | 13:57 |
| *** dkranz has joined #openstack-infra | 14:04 | |
| sdague | ArxCruz: you actually need tox -eall nameoftest | 14:05 |
| sdague | because testr doesn't have an exclude option | 14:05 |
| sdague | and treats all regex as or | 14:06 |
| *** jasondotstar has joined #openstack-infra | 14:06 | |
| ArxCruz | sdague: tyvm | 14:06 |
| *** eharney has quit IRC | 14:07 | |
| sdague | so netaddr is on pypi here - https://pypi.python.org/pypi/netaddr | 14:09 |
| sdague | why isn't that getting picked up? | 14:09 |
| *** dstanek_afk has joined #openstack-infra | 14:10 | |
| sdague | oh, the downloads go offsite | 14:10 |
| *** dstanek_afk is now known as dstaek | 14:12 | |
| *** dstaek is now known as dstanek | 14:12 | |
| ttx | fungi: around ? Now is a good time to approve https://review.openstack.org/#/c/64683/ -- while no other meeting is running | 14:15 |
| ttx | clarkb2: ^ | 14:15 |
| ttx | (need to have it logged before starting using it tomorrow) | 14:16 |
| clarkb2 | ttx looking but on bad connection | 14:16 |
| ttx | clarkb2: thx | 14:16 |
| *** mriedem has joined #openstack-infra | 14:17 | |
| *** dprince has joined #openstack-infra | 14:18 | |
| clarkb2 | approved | 14:19 |
| *** herndon has quit IRC | 14:19 | |
| ttx | clarkb2: coolthx. Shouldn't you be sleeping ? | 14:19 |
| clarkb2 | soon, currently in perth attending lca | 14:20 |
| ttx | it's late there too. | 14:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Add logging to #openstack-relmgr-office https://review.openstack.org/64683 | 14:22 |
| clarkb2 | almost 2230, sleep will happen soon. trying to stay up to fight jet lag | 14:22 |
| ttx | now let's see if the bot reloads before the next meeting starts. | 14:22 |
| clarkb2 | should reload by 2240 perth time | 14:23 |
| ttx | clarkb2: that would work. Cool. | 14:23 |
| *** dkliban has joined #openstack-infra | 14:25 | |
| *** coolsvap has joined #openstack-infra | 14:31 | |
| *** annegentle_ has joined #openstack-infra | 14:31 | |
| *** zehicle_at_dell has quit IRC | 14:32 | |
| fungi | ttx: sure, i can do that | 14:32 |
| *** zehicle has quit IRC | 14:32 | |
| *** mrodden has joined #openstack-infra | 14:32 | |
| *** blamar has joined #openstack-infra | 14:33 | |
| *** openstack has joined #openstack-infra | 14:33 | |
| *** ryanpetrello has joined #openstack-infra | 14:34 | |
| *** prad has joined #openstack-infra | 14:34 | |
| fungi | ttx: i see that now. trying to read nick highlights in reverse-chronological order but teh rest of scrollback chronologically is also not working so well | 14:35 |
| *** mlipchuk has quit IRC | 14:36 | |
| fungi | sdague: since you were also apparently surprised by the netaddr thing, https://review.openstack.org/65019 (i'm still building up a list of the ones we need in https://etherpad.openstack.org/p/pip1.5Upgrade section 3) | 14:37 |
| *** julim has joined #openstack-infra | 14:38 | |
| *** jaypipes has joined #openstack-infra | 14:39 | |
| *** jaypipes has quit IRC | 14:39 | |
| *** senk has joined #openstack-infra | 14:39 | |
| *** jaypipes has joined #openstack-infra | 14:39 | |
| *** jgrimm has quit IRC | 14:39 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/61463 | 14:41 |
| *** homeless has joined #openstack-infra | 14:42 | |
| *** mfer has joined #openstack-infra | 14:44 | |
| *** bknudson has joined #openstack-infra | 14:46 | |
| *** eharney has joined #openstack-infra | 14:47 | |
| *** dcramer_ has joined #openstack-infra | 14:47 | |
| *** coolsvap is now known as coolsvap_away | 14:48 | |
| *** dstufft has quit IRC | 14:49 | |
| *** dstufft has joined #openstack-infra | 14:50 | |
| *** yamahata has joined #openstack-infra | 14:51 | |
| *** beekneemech is now known as bnemec | 14:54 | |
| *** senk has quit IRC | 14:54 | |
| *** rongze has quit IRC | 14:58 | |
| *** rongze has joined #openstack-infra | 14:59 | |
| *** hashar has quit IRC | 14:59 | |
| *** CaptTofu has quit IRC | 15:00 | |
| *** senk has joined #openstack-infra | 15:01 | |
| *** banix has joined #openstack-infra | 15:03 | |
| *** rongze has quit IRC | 15:03 | |
| *** mgagne has joined #openstack-infra | 15:03 | |
| *** senk1 has joined #openstack-infra | 15:04 | |
| *** senk has quit IRC | 15:06 | |
| *** mgagne1 has joined #openstack-infra | 15:06 | |
| *** luisg has quit IRC | 15:07 | |
| *** luisg has joined #openstack-infra | 15:07 | |
| *** odyi has quit IRC | 15:07 | |
| *** mgagne has quit IRC | 15:08 | |
| *** _ruhe has quit IRC | 15:09 | |
| *** salv-orlando_ has joined #openstack-infra | 15:09 | |
| *** _ruhe has joined #openstack-infra | 15:10 | |
| *** Guest97522 has quit IRC | 15:11 | |
| *** krotscheck has joined #openstack-infra | 15:11 | |
| *** notmyname has quit IRC | 15:12 | |
| *** odyi has joined #openstack-infra | 15:12 | |
| *** jog0 has joined #openstack-infra | 15:12 | |
| *** dkranz has quit IRC | 15:12 | |
| *** rwsu has joined #openstack-infra | 15:12 | |
| *** salv-orlando has quit IRC | 15:13 | |
| *** salv-orlando_ is now known as salv-orlando | 15:13 | |
| *** notmyname has joined #openstack-infra | 15:13 | |
| *** belliott has quit IRC | 15:14 | |
| *** che-arne has quit IRC | 15:14 | |
| *** hashar has joined #openstack-infra | 15:15 | |
| *** belliott has joined #openstack-infra | 15:16 | |
| *** fifieldt has quit IRC | 15:17 | |
| *** banix has quit IRC | 15:18 | |
| *** CaptTofu has joined #openstack-infra | 15:20 | |
| *** herndon has joined #openstack-infra | 15:20 | |
| *** rongze has joined #openstack-infra | 15:20 | |
| *** che-arne has joined #openstack-infra | 15:26 | |
| sdague | fungi: yeh, I sort of knew netaddr was still an issue | 15:30 |
| *** krotscheck has quit IRC | 15:30 | |
| fungi | sdague: and netifaces (which swift uses). also lazr.restful seems to be needed by elastic-recheck and reviewday | 15:31 |
| fungi | i'm spinning up a throwaway vm now to start testing pypi-mirror with pip 1.5 and a patched virtualenv 1.11 to see what else breaks | 15:32 |
| sdague | lazr.restful is a shadow dependency of launchpadlib | 15:33 |
| fungi | i think we may want to come up with some sort of notation in the global-requirements.txt for packages which need whitelisting, so that we can deal with them programmatically in requirements jobs and mirror updates | 15:33 |
| sdague | so we can elimintate that once story board is live :) | 15:33 |
| ArxCruz | fungi: tyvm | 15:35 |
| *** saschpe has quit IRC | 15:38 | |
| *** saschpe has joined #openstack-infra | 15:38 | |
| *** wenlock has joined #openstack-infra | 15:38 | |
| *** jgrimm has joined #openstack-infra | 15:39 | |
| *** MarkAtwood has joined #openstack-infra | 15:42 | |
| *** b3nt_pin has joined #openstack-infra | 15:43 | |
| *** senk1 has quit IRC | 15:44 | |
| *** tma996 has quit IRC | 15:45 | |
| *** thedodd has joined #openstack-infra | 15:47 | |
| *** sc68cal has joined #openstack-infra | 15:47 | |
| sc68cal | Who do I need to contact about updating a meeting on the calendar ? | 15:49 |
| fungi | sc68cal: if you update the wiki page, ttx gets a notification and makes corresponding updates to the calendar feed | 15:51 |
| *** burt has joined #openstack-infra | 15:51 | |
| fungi | but it's a manual process, so expect it to possibly take a few days to be reflected there | 15:51 |
| sc68cal | perfect! | 15:51 |
| ArxCruz | mriedem: so, regarding the https://review.openstack.org/#/c/65019/ the big output we are seeing with run_tests is a oslo.config problem right? | 15:55 |
| chmouel | fun times gerrit is borking links like https://bugs.launchpad.net/openstack-ci/+bug/1242569 removing the 9 at the end, i.e: review https://review.openstack.org/#/c/62377/ | 15:56 |
| mriedem | ArxCruz: i don't know the history on this really, but sdague said oslo requires a fix before we can fix run_tests in nova for that patch | 15:56 |
| ArxCruz | mriedem: remember some time ago you told me that the big output from tox -efull was because oslo.config wasn't installed, then I had to run setup.py again ? | 15:57 |
| ArxCruz | I'm having the same problem, that big output but oslo.config is properly installed | 15:57 |
| mriedem | ArxCruz: i don't remember that really, are you thinking mtreinish? | 15:58 |
| ArxCruz | oh, damn... | 15:58 |
| ArxCruz | probably yes | 15:58 |
| ArxCruz | :D | 15:58 |
| mtreinish | mriedem: it'll never end | 15:58 |
| ArxCruz | I always confusing you two :) | 15:59 |
| mriedem | there can be only one... | 15:59 |
| sdague | mriedem: the issue is pip 1.5 won't install things not from pypi by default any more | 15:59 |
| sdague | as a security measure | 15:59 |
| sdague | you have to whitelist | 15:59 |
| sdague | so netaddr still isn't hosting downloads on pypi | 15:59 |
| mriedem | mtreinish: http://www.gagdonkey.com/photos/There-can-be-only-one.jpg | 16:00 |
| mriedem | sdague: yeah, i remember seeing some of that conversation, i'm not against the workaround, just like to see things tracked so we can point people at a central thing when they come asking in -nova, which has been happening since last week | 16:01 |
| mriedem | lp not scaling isn't cool | 16:01 |
| fungi | chmouel: i'll see if i can figure out what's going on with the gerrit hyperlinks there. we just reordered them to fix another link pattern breakage, so it's apparently like a game of jenga to get these to not step on one another | 16:02 |
| chmouel | fungi: oh fun times, good luck. | 16:03 |
| *** markvan has joined #openstack-infra | 16:03 | |
| *** UtahDave has joined #openstack-infra | 16:03 | |
| *** david-lyle_ has joined #openstack-infra | 16:04 | |
| *** rnirmal has joined #openstack-infra | 16:09 | |
| *** gyee_ has joined #openstack-infra | 16:14 | |
| *** jroovers has joined #openstack-infra | 16:14 | |
| jroovers | hi fungi, are you there? | 16:15 |
| fungi | jroovers: i think so | 16:15 |
| jroovers | fungi, haha :-) Happy New Year! | 16:15 |
| jroovers | fungi, time for a question or 2 about openstack infra? | 16:15 |
| fungi | jroovers: sure, and happy new year to you too! | 16:15 |
| jroovers | I'm trying to better understand the test and gating process | 16:16 |
| *** rongze has quit IRC | 16:16 | |
| jroovers | it is a little confusing, as there is both a gate queue in zuul as well as the devstack-gate scripts | 16:16 |
| jroovers | so, first things first | 16:17 |
| *** rongze has joined #openstack-infra | 16:17 | |
| jroovers | when people talk about "the gate", they mean the job in the gate queue in zuul right? | 16:17 |
| *** rongze has quit IRC | 16:17 | |
| fungi | jroovers: have you seen the overview presentation at http://docs.openstack.org/infra/publications/ yet? there are some nice diagrams in there too | 16:17 |
| jroovers | I have | 16:18 |
| fungi | jroovers: the term "gate" tends to be nebulous/overloaded, so can depend a lot on context. in think in general the reference is usually to zuul's "gate" pipeline | 16:18 |
| jroovers | fungi, just want to clarify a thing or 2 | 16:18 |
| jroovers | fungi, ok that is what I figured | 16:18 |
| jroovers | now, when you look at a particular patch | 16:19 |
| *** rongze has joined #openstack-infra | 16:19 | |
| jroovers | there are a bunch of tests that are run right | 16:19 |
| fungi | or, more specifically, the main integrated projects gate queue within zuul's gate pipeline, which mostly hinges on the jobs shared by those projects *which in turn are mostly devstack-gate jobs) | 16:19 |
| *** rongze has quit IRC | 16:19 | |
| jroovers | fungi, indeed | 16:19 |
| jroovers | so, I'm wondering | 16:19 |
| *** rongze has joined #openstack-infra | 16:20 | |
| jroovers | if check-tempest-dsvm-full runs all tests | 16:20 |
| jroovers | why is it then neccessary to run some of the gate tests again | 16:20 |
| *** rongze has quit IRC | 16:20 | |
| jroovers | as they basically test something that is already done by check-tempest-dsvm-full | 16:20 |
| *** rongze has joined #openstack-infra | 16:20 | |
| jroovers | is this to make sure that tests don't interfere with the gate tests? | 16:20 |
| *** thedodd has quit IRC | 16:21 | |
| fungi | it doesn't actually run all tests under all scenarios | 16:21 |
| *** rongze has quit IRC | 16:21 | |
| *** smarcet has quit IRC | 16:21 | |
| fungi | that job, for example, runs the tests using a typical configuration employing mysql | 16:21 |
| jroovers | fungi, yes | 16:21 |
| *** rongze has joined #openstack-infra | 16:21 | |
| fungi | so the postgres variant does the same using postgres | 16:22 |
| jroovers | agreed | 16:22 |
| jroovers | I understand that | 16:22 |
| jroovers | but then there is a test for example | 16:22 |
| *** smarcet has joined #openstack-infra | 16:22 | |
| *** rongze has quit IRC | 16:22 | |
| jroovers | gate-tempest-dsvm-large-ops | 16:22 |
| *** dkranz has joined #openstack-infra | 16:22 | |
| fungi | the full job, similarly, uses nova network. the neutron variant runs mostly the same tests, but with neutron instead | 16:22 |
| fungi | and so on | 16:22 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/config: Support proposed/* branches for milestone-proposed https://review.openstack.org/65103 | 16:22 |
| *** rongze has joined #openstack-infra | 16:22 | |
| fungi | the large-ops job runs scalability tests which are not run as part of the "full" job | 16:22 |
| fungi | basically re-running a subset of tests, but with different configuration making it use a lot more resources | 16:23 |
| *** rongze has quit IRC | 16:23 | |
| jroovers | fungi, ah ok | 16:23 |
| jroovers | fungi, that makes sense | 16:23 |
| *** rongze has joined #openstack-infra | 16:23 | |
| jroovers | that I didn't think of that | 16:23 |
| *** thedodd has joined #openstack-infra | 16:23 | |
| jroovers | ok, so my next question | 16:24 |
| *** pmathews has joined #openstack-infra | 16:24 | |
| *** rongze has quit IRC | 16:24 | |
| fungi | in essence, while we do have some overlap in coverage between some jobs, they run under different scenarios (different configurations, different combinations of underlying components, and so on, in parallel) | 16:24 |
| *** rongze has joined #openstack-infra | 16:24 | |
| jroovers | fungi, makes sense. I just hadn't figured it out for the large ops | 16:24 |
| jroovers | :-) | 16:24 |
| jroovers | fungi, it looks like sometimes different gate tests are ran for different patch sets, how does that work? | 16:24 |
| *** rongze has quit IRC | 16:25 | |
| fungi | it's determined by zuul's configuration, http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml | 16:25 |
| zul | whee libvirt-python should be in the cloud archive today | 16:25 |
| fungi | zul: awesome. did we ever get anywhere with the consistent tempest failures using newer libvirt? | 16:26 |
| zul | fungi: nope | 16:26 |
| *** rongze has joined #openstack-infra | 16:26 | |
| *** rongze has quit IRC | 16:26 | |
| fungi | jroovers: so basically, each project has a list of jobs which run for each pipeline (check, gate, post and so on) | 16:27 |
| *** rongze has joined #openstack-infra | 16:27 | |
| *** rongze has quit IRC | 16:27 | |
| fungi | jroovers: then those jobs are further filtered by the jobs list's patterns which can decide what jobs to run by branch name, files touched by a change, et cetera | 16:27 |
| *** rongze has joined #openstack-infra | 16:27 | |
| *** rongze has quit IRC | 16:28 | |
| jroovers | fungi, ok, that is how I understood it. Those prezo's you pointed me too are very good, but it's good to verify :-) | 16:28 |
| *** rongze has joined #openstack-infra | 16:28 | |
| fungi | jroovers: also, a lot of it is documented at http://ci.openstack.org/ but it's a lot of varyied information to absorb all at once, so can be quite daunting at first | 16:29 |
| *** rongze has quit IRC | 16:29 | |
| jroovers | yeah, I've been at it for 2 hours or so, very interesting | 16:29 |
| *** ^d has joined #openstack-infra | 16:29 | |
| *** rongze has joined #openstack-infra | 16:29 | |
| *** rongze has quit IRC | 16:29 | |
| jroovers | fungi, sometimes you just need someone to confirm what you are reading :-) | 16:29 |
| fungi | i've been at it for 16 months, and am still fuzzy on a few points ;) | 16:29 |
| jroovers | fungi :-) | 16:30 |
| *** rongze has joined #openstack-infra | 16:30 | |
| *** rongze has quit IRC | 16:30 | |
| jroovers | fungi, so another question, related to tempest test tagging | 16:30 |
| jroovers | some of these tests have tags like "gate" or "smoke" | 16:30 |
| *** rongze has joined #openstack-infra | 16:30 | |
| fungi | that may get into a topic better covered in #openstack-qa. i don't know a lot about tempest internals | 16:30 |
| *** rongze has quit IRC | 16:30 | |
| *** b3nt_pin is now known as beagles | 16:31 | |
| fungi | but those tags are used to determine the context under which they should be run, i do know that much | 16:31 |
| jroovers | fungi, I thought you might say that, still want to ask as I think you might know the answer | 16:31 |
| *** rongze has joined #openstack-infra | 16:31 | |
| fungi | so when you run tempest smoke tests, just the tests marked smoke get run, i think | 16:31 |
| fungi | whereas i believe tempest full runs the tests tagged gate and smoke | 16:32 |
| fungi | or something like that | 16:32 |
| fungi | but the #-qa peeps are more knowledgeable about that since they're the ones mainly handling it | 16:33 |
| jroovers | fungi, ok thanks, much appreciated! | 16:33 |
| fungi | any time! | 16:33 |
| jroovers | fungi, you took away some of the vagueness, thanks! | 16:33 |
| zaro | fungi: https://review-dev.openstack.org/monitoring | 16:34 |
| fungi | zaro: neat! that will come in handy particularly once we cut over to new gerrit and start running into unexpected scalability bottlenecks | 16:34 |
| *** afazekas has quit IRC | 16:35 | |
| *** yolanda has quit IRC | 16:38 | |
| sdague | all the definitions of what is run is in the tox.ini | 16:46 |
| *** dangers_away is now known as dangers | 16:47 | |
| *** oubiwann has joined #openstack-infra | 16:47 | |
| *** pcrews_ has quit IRC | 16:48 | |
| zaro | fungi: i'de like to start adding patches to our gerrit openstack/2.8 branch. however the following needs merging for the build and test jobs to work, in order 60984, 60985, 60348. | 16:50 |
| zaro | fungi: could you please take a quick look? i think i've already reviewed. | 16:51 |
| zaro | fungi: actually order is wrong. correct order is 60348, 60984, 60985 | 16:52 |
| fungi | zaro: i'll bump those to the top of my review pile | 16:52 |
| zaro | fungi: thanks. | 16:52 |
| *** luis_ has joined #openstack-infra | 16:52 | |
| *** luisg has quit IRC | 16:53 | |
| *** dstanek has quit IRC | 16:54 | |
| fungi | mriedem: bug 1266513 | 16:54 |
| fungi | i guess we still don't have uvirtbot back | 16:54 |
| fungi | mriedem: https://launchpad.net/bugs/1266513 | 16:54 |
| *** alexpilotti has joined #openstack-infra | 16:56 | |
| *** rcleere has joined #openstack-infra | 16:56 | |
| mriedem | fungi: cool, thanks, are you going to update the nova patch with a Partial-Bug: #1266513 line? | 16:56 |
| fungi | mriedem: i can. it's 8 patches so far (and counting) | 16:57 |
| fungi | so it'll take me a bit to amend and push them all | 16:58 |
| fungi | should get to it shortly | 16:58 |
| *** ociuhandu has joined #openstack-infra | 16:58 | |
| fungi | mriedem: also, nova and swift need havana and grizzly series bugtasks for that | 16:59 |
| fungi | not sure if you're a nova driver, but if so that would be appreciated when you triage it | 16:59 |
| mriedem | fungi: we can just mark the bug as havana-backport-potential and grizzly-backport-potential | 17:00 |
| fungi | i can add those tags for now, sure | 17:00 |
| mriedem | alreayd did | 17:00 |
| fungi | okay, cool | 17:00 |
| fungi | i assume the tags shouldn't be removed for now since we still don't know how many other projects are going to get tacked onto this report | 17:01 |
| mriedem | right | 17:01 |
| fungi | it's basically backport fodder to all supported branches for all projects which have "external" requirements | 17:01 |
| *** sarob has joined #openstack-infra | 17:03 | |
| *** dhellmann is now known as dhellmann_ | 17:03 | |
| *** garyk has quit IRC | 17:03 | |
| *** rockstar` is now known as rockstar | 17:04 | |
| *** praneshp has joined #openstack-infra | 17:06 | |
| chmouel | fungi: https://review.openstack.org/#/c/65025/ is that something that has to do with swift? | 17:07 |
| chmouel | fungi: it seems to be failing with the ec2 tests | 17:09 |
| fungi | chmouel: oh, maybe. i've been too busy with other broken stuff to dive into it yet, so if you have ideas as to what happened then great! | 17:12 |
| fungi | chmouel: if it is ec2 stuff, it may be related to the reason sdague disabled devstack ec2 exercises in grenade | 17:13 |
| fungi | (so he might have more details) | 17:13 |
| chmouel | fungi: yep probably, the failures we had over christmas | 17:14 |
| *** pblaho has quit IRC | 17:17 | |
| *** yassine has quit IRC | 17:19 | |
| *** reed has joined #openstack-infra | 17:20 | |
| *** herndon has quit IRC | 17:21 | |
| *** MIDENN_ has quit IRC | 17:23 | |
| *** MIDENN_ has joined #openstack-infra | 17:23 | |
| *** praneshp has quit IRC | 17:23 | |
| *** branen__ has joined #openstack-infra | 17:23 | |
| *** herndon has joined #openstack-infra | 17:25 | |
| *** jpich has quit IRC | 17:25 | |
| *** herndon has quit IRC | 17:26 | |
| *** dkliban has quit IRC | 17:26 | |
| *** branen_ has quit IRC | 17:27 | |
| *** sc68cal has left #openstack-infra | 17:29 | |
| *** oubiwann has quit IRC | 17:30 | |
| derekh | I'm setting up the tripleo-ci hosts, the jenkins nodes from nodepool will need to ssh to these hosts, any idea what would be the best way to distribute keys for ssh, I would need a public key for the jenkins node on the test env host. | 17:31 |
| *** jroovers has quit IRC | 17:31 | |
| *** _david_ has joined #openstack-infra | 17:31 | |
| fungi | derekh: oh, right, i saw your question in scrollback and meant to answer it, but then got sidetracked by other shiny broken things | 17:31 |
| *** MarkAtwood has quit IRC | 17:32 | |
| derekh | fungi: no prob, if things are still broken I can mail the list instead | 17:32 |
| *** hashar has quit IRC | 17:32 | |
| derekh | fungi: and you can reply at your leasure | 17:32 |
| fungi | derekh: what do you currently use to configure the nodepool nodes... just the nodepool prep scripts? | 17:32 |
| *** fbo is now known as fbo_away | 17:32 | |
| fungi | oh, wait, these are the nodepool nodes we're building | 17:33 |
| fungi | right, so i think we (infra root) needs to create a keypair for it and then give you the public component so you can puppet/dib/whatever that onto the test env hosts? | 17:33 |
| derekh | fungi: pleia2 was the one involved in setting up those node but I think they are standard gate nodes from nodepoll | 17:33 |
| derekh | fungi: yup, that sounds about right | 17:34 |
| fungi | derekh: that's pretty simple then. we'd generate the keypair, install it in our hiera db so that nodepool.o.o knows it from its puppet config and can inject it into the toci nodepool nodes | 17:35 |
| *** yidclare has quit IRC | 17:35 | |
| fungi | derekh: if you can just open a bug against openstack-ci summarizing that, and assign it to me, i'll try to knock it out in the next couple days | 17:35 |
| derekh | fungi: will do, thanks | 17:36 |
| fungi | "Jeremy Stanley (fungi)" is my lp account, unsurprisingly | 17:36 |
| derekh | fungi: k | 17:36 |
| *** cody-somerville has joined #openstack-infra | 17:36 | |
| *** mancdaz is now known as mancdaz_away | 17:39 | |
| *** thedodd has quit IRC | 17:40 | |
| *** alexpilotti has quit IRC | 17:44 | |
| *** thedodd has joined #openstack-infra | 17:44 | |
| derekh | fungi: done, I can't assign it to you for some reason, https://bugs.launchpad.net/openstack-ci/+bug/1266530 | 17:45 |
| fungi | derekh: oh, sorry, i'm too used to being on every bug team in openstack i think :/ | 17:46 |
| fungi | thanks! | 17:46 |
| derekh | fungi: no prob | 17:46 |
| *** rongze has quit IRC | 17:48 | |
| *** harlowja has joined #openstack-infra | 17:49 | |
| *** senk has joined #openstack-infra | 17:49 | |
| *** dhellmann_ is now known as dhellmann | 17:50 | |
| *** rongze has joined #openstack-infra | 17:50 | |
| *** rongze has quit IRC | 17:51 | |
| *** rongze has joined #openstack-infra | 17:51 | |
| *** rongze has quit IRC | 17:52 | |
| *** rongze has joined #openstack-infra | 17:52 | |
| *** rongze has quit IRC | 17:53 | |
| *** salv-orlando_ has joined #openstack-infra | 17:53 | |
| *** mgagne1 is now known as mgagne | 17:53 | |
| *** rongze has joined #openstack-infra | 17:53 | |
| *** derekh has quit IRC | 17:53 | |
| *** rongze has quit IRC | 17:54 | |
| *** rongze has joined #openstack-infra | 17:54 | |
| *** rongze has quit IRC | 17:54 | |
| *** rongze has joined #openstack-infra | 17:55 | |
| *** rongze has quit IRC | 17:55 | |
| *** rongze has joined #openstack-infra | 17:56 | |
| *** rongze has quit IRC | 17:56 | |
| *** rongze has joined #openstack-infra | 17:56 | |
| *** salv-orlando has quit IRC | 17:57 | |
| *** salv-orlando_ is now known as salv-orlando | 17:57 | |
| *** pcrews_ has joined #openstack-infra | 17:57 | |
| *** rongze has quit IRC | 17:57 | |
| *** rongze has joined #openstack-infra | 17:57 | |
| *** dcramer_ has quit IRC | 17:57 | |
| *** rongze has quit IRC | 17:58 | |
| *** rongze has joined #openstack-infra | 17:58 | |
| *** rongze has quit IRC | 17:58 | |
| *** nati_ueno has joined #openstack-infra | 17:59 | |
| *** rongze has joined #openstack-infra | 17:59 | |
| *** rongze has quit IRC | 17:59 | |
| *** rongze has joined #openstack-infra | 18:01 | |
| *** dcramer_ has joined #openstack-infra | 18:01 | |
| *** rongze has quit IRC | 18:01 | |
| *** rongze has joined #openstack-infra | 18:02 | |
| *** rongze has quit IRC | 18:02 | |
| *** herndon has joined #openstack-infra | 18:04 | |
| *** che-arne has quit IRC | 18:04 | |
| *** yidclare has joined #openstack-infra | 18:04 | |
| *** rongze_ has joined #openstack-infra | 18:05 | |
| *** rongze_ has quit IRC | 18:05 | |
| *** rongze has joined #openstack-infra | 18:05 | |
| *** malini has joined #openstack-infra | 18:05 | |
| *** rongze has quit IRC | 18:06 | |
| *** rongze has joined #openstack-infra | 18:06 | |
| *** rongze has quit IRC | 18:06 | |
| *** sarob has quit IRC | 18:07 | |
| *** sarob has joined #openstack-infra | 18:07 | |
| *** dcramer_ has quit IRC | 18:08 | |
| *** rongze has joined #openstack-infra | 18:09 | |
| *** rongze_ has joined #openstack-infra | 18:10 | |
| *** rongze_ has quit IRC | 18:10 | |
| *** Ajaeger has joined #openstack-infra | 18:11 | |
| *** rongze_ has joined #openstack-infra | 18:11 | |
| *** sarob has quit IRC | 18:11 | |
| *** rongze_ has quit IRC | 18:11 | |
| *** marun has joined #openstack-infra | 18:13 | |
| *** rongze has quit IRC | 18:14 | |
| *** rongze has joined #openstack-infra | 18:14 | |
| *** gothicmindfood has joined #openstack-infra | 18:16 | |
| *** rongze has quit IRC | 18:19 | |
| *** sarob has joined #openstack-infra | 18:21 | |
| *** ArxCruz has quit IRC | 18:22 | |
| *** _david_ has quit IRC | 18:23 | |
| *** fbo_away is now known as fbo | 18:23 | |
| *** oubiwann has joined #openstack-infra | 18:23 | |
| *** mriedem has quit IRC | 18:24 | |
| *** sarob has quit IRC | 18:25 | |
| *** gothicmindfood has quit IRC | 18:26 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:26 | |
| *** rfolco has quit IRC | 18:27 | |
| *** oubiwann has quit IRC | 18:30 | |
| *** oubiwann has joined #openstack-infra | 18:30 | |
| *** mriedem has joined #openstack-infra | 18:30 | |
| *** jerryz has joined #openstack-infra | 18:31 | |
| SpamapS | fungi: are you the lone soldier while everybody else heads down under? | 18:31 |
| *** praneshp has joined #openstack-infra | 18:32 | |
| *** jasondotstar has quit IRC | 18:34 | |
| fungi | SpamapS: yes, and unfortunately i think we have a nodepool node starvation issue. starting to look into it now | 18:34 |
| SpamapS | fungi: suck. We're finding that pip 1.5 is breaking Tripleo badly.. need to get some background before I dive into it. :-P | 18:35 |
| *** jooools has quit IRC | 18:35 | |
| fungi | SpamapS: probably you want to see https://launchpad.net/bugs/1266530 | 18:35 |
| fungi | er, no, not that one | 18:36 |
| fungi | SpamapS: https://launchpad.net/bugs/1266513 | 18:36 |
| * fungi is swimming in bugs | 18:36 | |
| *** rfolco has joined #openstack-infra | 18:43 | |
| SpamapS | fungi: ty | 18:44 |
| *** rongze has joined #openstack-infra | 18:45 | |
| *** rongze has quit IRC | 18:45 | |
| *** rongze has joined #openstack-infra | 18:46 | |
| *** rongze has quit IRC | 18:46 | |
| *** rongze has joined #openstack-infra | 18:47 | |
| *** rongze has quit IRC | 18:47 | |
| *** rongze has joined #openstack-infra | 18:47 | |
| *** rongze has quit IRC | 18:48 | |
| *** rongze has joined #openstack-infra | 18:48 | |
| *** rcarrillocruz has joined #openstack-infra | 18:49 | |
| *** rongze has quit IRC | 18:49 | |
| *** rongze has joined #openstack-infra | 18:49 | |
| *** rongze has quit IRC | 18:50 | |
| *** rongze has joined #openstack-infra | 18:50 | |
| *** rongze has quit IRC | 18:51 | |
| *** rongze_ has joined #openstack-infra | 18:51 | |
| *** rongze_ has quit IRC | 18:52 | |
| *** rongze has joined #openstack-infra | 18:52 | |
| *** weshay is now known as weshay_brb | 18:52 | |
| *** rongze has quit IRC | 18:52 | |
| *** rongze has joined #openstack-infra | 18:53 | |
| *** rongze has quit IRC | 18:53 | |
| *** rongze has joined #openstack-infra | 18:54 | |
| *** rongze has quit IRC | 18:54 | |
| *** rongze has joined #openstack-infra | 18:55 | |
| *** rongze has quit IRC | 18:56 | |
| *** rongze has joined #openstack-infra | 18:56 | |
| *** herndon has quit IRC | 18:56 | |
| *** sandywalsh has quit IRC | 18:57 | |
| *** rongze has quit IRC | 18:57 | |
| *** rongze has joined #openstack-infra | 18:57 | |
| *** rongze has quit IRC | 18:57 | |
| *** rongze has joined #openstack-infra | 18:58 | |
| *** johnthetubaguy has quit IRC | 18:58 | |
| malini | hello..I am adding a new job in https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L1817l - What is the difference between a check & a gate job ? | 18:58 |
| *** praneshp has quit IRC | 18:58 | |
| *** rongze has quit IRC | 18:58 | |
| *** rongze has joined #openstack-infra | 18:59 | |
| *** rongze has quit IRC | 18:59 | |
| Ajaeger | malini: When you submit a patch with "git review", the check jobs are run | 18:59 |
| mgagne | malini: check is when a change is created, gate is when it's approved | 18:59 |
| Ajaeger | malini: When a patch is approved, the gate ones | 18:59 |
| *** rongze has joined #openstack-infra | 19:00 | |
| mgagne | malini: see description at top of file =) | 19:00 |
| *** rongze has quit IRC | 19:00 | |
| fungi | malini: also, descriptions at top of http://status.openstack.org/zuul/ | 19:00 |
| *** rongze has joined #openstack-infra | 19:00 | |
| malini | Ajaeger, mgagne, fungi: thanks a lot ! | 19:01 |
| malini | mgagne: next time, I'll look at the top of the file before asking ;) | 19:01 |
| *** rongze has quit IRC | 19:01 | |
| *** rongze has joined #openstack-infra | 19:01 | |
| fungi | we tend to put lots of neat information at the tops of things, so they are more likely to catch your eye ;) | 19:01 |
| fungi | (i guess it doesn't always work) | 19:02 |
| *** saschpe has quit IRC | 19:02 | |
| *** rongze has quit IRC | 19:02 | |
| *** saschpe has joined #openstack-infra | 19:02 | |
| malini | fungi: now that I know this, will work the next time :) | 19:02 |
| *** rongze has joined #openstack-infra | 19:02 | |
| *** praneshp has joined #openstack-infra | 19:02 | |
| *** rongze has quit IRC | 19:03 | |
| *** rongze has joined #openstack-infra | 19:03 | |
| *** rongze has quit IRC | 19:04 | |
| *** rongze has joined #openstack-infra | 19:04 | |
| *** rongze has quit IRC | 19:05 | |
| *** rongze has joined #openstack-infra | 19:06 | |
| *** rongze has quit IRC | 19:06 | |
| *** rongze has joined #openstack-infra | 19:07 | |
| *** rongze has quit IRC | 19:07 | |
| *** hashar has joined #openstack-infra | 19:08 | |
| *** rongze has joined #openstack-infra | 19:08 | |
| *** rongze has quit IRC | 19:08 | |
| *** rongze has joined #openstack-infra | 19:09 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add different envs for different sqlalchemy versions https://review.openstack.org/65135 | 19:09 |
| *** rongze has quit IRC | 19:09 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add different envs for different sqlalchemy versions https://review.openstack.org/65135 | 19:10 |
| *** rongze has joined #openstack-infra | 19:10 | |
| *** rongze has quit IRC | 19:10 | |
| harlowja | if i messed that up guys let me know :-P | 19:10 |
| *** rongze has joined #openstack-infra | 19:11 | |
| *** rongze has quit IRC | 19:11 | |
| *** rongze has joined #openstack-infra | 19:11 | |
| *** rongze has quit IRC | 19:12 | |
| *** rongze has joined #openstack-infra | 19:12 | |
| *** rongze has quit IRC | 19:13 | |
| *** hashar has quit IRC | 19:13 | |
| *** wenlock has quit IRC | 19:13 | |
| Ajaeger | harlowja: you just added the jenkins site - if those should be run, you need to configure them for Zuul as well. | 19:13 |
| *** rongze has joined #openstack-infra | 19:13 | |
| harlowja | Ajaeger k | 19:13 |
| *** wenlock has joined #openstack-infra | 19:14 | |
| harlowja | Ajaeger i should probably do that in secondary review i think, since need to make those exist in taskflow tox.ini before that | 19:14 |
| *** rongze has quit IRC | 19:14 | |
| *** sandywalsh has joined #openstack-infra | 19:14 | |
| harlowja | *which is in progress (making them exixst there) | 19:14 |
| Ajaeger | harlowja: see how I did it: https://review.openstack.org/#/c/64567/ - just two see both parts of the change | 19:14 |
| harlowja | k | 19:14 |
| harlowja | thx | 19:15 |
| Ajaeger | harlowja: my preference would be one patch that does both steps - so, change now tox.ini and set this patch to Work-in-progress so that it does not get merged before you are ready... | 19:15 |
| harlowja | Ajaeger sounds good to me | 19:15 |
| Ajaeger | but I'm not part of the infra team, not sure whether this is best practice - fungi might disagree ;) | 19:16 |
| harlowja | ;) | 19:16 |
| *** rongze has joined #openstack-infra | 19:17 | |
| * fungi will have to evaluate once he's freed up enough for code review | 19:17 | |
| *** rongze has quit IRC | 19:17 | |
| *** rongze has joined #openstack-infra | 19:18 | |
| *** rongze has quit IRC | 19:18 | |
| *** dstanek has joined #openstack-infra | 19:18 | |
| *** sandywalsh has quit IRC | 19:18 | |
| *** rongze has joined #openstack-infra | 19:18 | |
| *** rongze has quit IRC | 19:18 | |
| *** senk has quit IRC | 19:20 | |
| *** rongze has joined #openstack-infra | 19:20 | |
| *** rongze has quit IRC | 19:20 | |
| Ajaeger | fungi: please include my patch mentioned above in that code review pile if possible | 19:21 |
| *** dripton__ has quit IRC | 19:22 | |
| fungi | Ajaeger: yep, i usually try to work through the oldest/most neglected changes first. we're just very short-handed coming out of the holidays, and with so many people at lca | 19:23 |
| *** rongze_ has joined #openstack-infra | 19:23 | |
| *** rongze_ has quit IRC | 19:23 | |
| fungi | doesn't help that stuff keeps breaking either | 19:23 |
| *** rongze_ has joined #openstack-infra | 19:25 | |
| Ajaeger | fungi: yeah ;( | 19:25 |
| *** rongze_ has quit IRC | 19:25 | |
| *** dripton__ has joined #openstack-infra | 19:25 | |
| *** dkliban has joined #openstack-infra | 19:26 | |
| *** rongze has joined #openstack-infra | 19:26 | |
| *** rongze has quit IRC | 19:26 | |
| *** rongze has joined #openstack-infra | 19:27 | |
| *** rongze has quit IRC | 19:27 | |
| *** rongze has joined #openstack-infra | 19:27 | |
| *** rongze has quit IRC | 19:28 | |
| *** kraman has joined #openstack-infra | 19:28 | |
| *** rongze has joined #openstack-infra | 19:28 | |
| *** rongze has quit IRC | 19:28 | |
| *** yolanda has joined #openstack-infra | 19:29 | |
| *** ^d has quit IRC | 19:29 | |
| fungi | okay, i think i see the crux nodepool starvation problem... nodepool says jenkins02 has ~75 ready nodes, but it's not picking up any jobs | 19:29 |
| *** rongze has joined #openstack-infra | 19:29 | |
| *** rongze has quit IRC | 19:30 | |
| fungi | and now i'm getting proxy errors from the jenkins02 webui | 19:30 |
| *** oubiwann has quit IRC | 19:30 | |
| fungi | (it was responding as recently as an hour ago, so must be a progressive issue) | 19:31 |
| *** rongze_ has joined #openstack-infra | 19:31 | |
| *** rongze_ has quit IRC | 19:31 | |
| kraman | fungi: clarkb2 mordred: Hi, I was looking into using Zuul within Solum and had a few questions. got a few minutes? | 19:32 |
| fungi | well, it came up after a few tries, but it's got a ton of offline nodes and is running only a handful of jobs. i'm putting it in shutdown mode to gracefully restart it | 19:32 |
| *** sandywalsh has joined #openstack-infra | 19:32 | |
| fungi | kraman: kinda swamped... the others are on australia time this week | 19:32 |
| *** rongze has joined #openstack-infra | 19:33 | |
| *** rongze has quit IRC | 19:33 | |
| kraman | ah i see. ok ill come back later. hopefully when they are online | 19:33 |
| fungi | so it's mostly just me trying to shore up the timbers | 19:33 |
| mriedem | fungi: can i help you out with https://review.openstack.org/#/c/65019/ by adding that bug line? | 19:34 |
| fungi | mriedem: if you like, but it won't take me long to do once i'm free'd up | 19:34 |
| *** rongze_ has joined #openstack-infra | 19:34 | |
| *** rongze_ has quit IRC | 19:35 | |
| *** rongze_ has joined #openstack-infra | 19:35 | |
| *** rongze_ has quit IRC | 19:35 | |
| fungi | #status notice gating is operating at reduced capacity while we work through a systems problem (ETA 21:00 UTC) | 19:36 |
| openstackstatus | NOTICE: gating is operating at reduced capacity while we work through a systems problem (ETA 21:00 UTC) | 19:36 |
| *** oubiwann has joined #openstack-infra | 19:36 | |
| fungi | #status log gracefully stopping jenkins02 now. it has many nodes which are offline status and only a handful online, while nodepool thinks it has ~75 available to run jobs | 19:37 |
| *** hashar has joined #openstack-infra | 19:38 | |
| *** rongze has joined #openstack-infra | 19:38 | |
| *** rongze has quit IRC | 19:39 | |
| *** rongze has joined #openstack-infra | 19:39 | |
| *** rongze has quit IRC | 19:39 | |
| *** flaper87 is now known as flaper87|afk | 19:40 | |
| *** rongze has joined #openstack-infra | 19:40 | |
| *** rongze has quit IRC | 19:40 | |
| *** rongze has joined #openstack-infra | 19:40 | |
| *** rongze has quit IRC | 19:41 | |
| *** rongze has joined #openstack-infra | 19:41 | |
| *** rongze has quit IRC | 19:41 | |
| *** Ajaeger has quit IRC | 19:41 | |
| *** rongze has joined #openstack-infra | 19:42 | |
| *** praneshp has quit IRC | 19:42 | |
| *** rongze has quit IRC | 19:42 | |
| *** rongze has joined #openstack-infra | 19:42 | |
| *** rongze has quit IRC | 19:42 | |
| *** mlipchuk has joined #openstack-infra | 19:45 | |
| *** rongze has joined #openstack-infra | 19:45 | |
| *** ArxCruz has joined #openstack-infra | 19:46 | |
| *** rongze has quit IRC | 19:46 | |
| *** praneshp has joined #openstack-infra | 19:46 | |
| *** rongze has joined #openstack-infra | 19:46 | |
| *** afazekas has joined #openstack-infra | 19:46 | |
| *** rongze has quit IRC | 19:46 | |
| SpamapS | fungi: IIRC you have IRC rights powerful enough to ban somebody | 19:47 |
| SpamapS | fungi: ^^ rongze ... join/disconnecting constantly for the last half hour | 19:47 |
| fungi | SpamapS: probably not, but i can find login information for the bot account we set up if needed | 19:47 |
| fungi | oh, actually, yes i can | 19:47 |
| fungi | which channel? | 19:47 |
| SpamapS | this one | 19:48 |
| SpamapS | #tripleo | 19:48 |
| SpamapS | fungi: #openstack-dev too | 19:48 |
| *** rongze has joined #openstack-infra | 19:48 | |
| SpamapS | fungi: and #heat | 19:48 |
| SpamapS | fungi: I also reported it on #freenode | 19:48 |
| SpamapS | two different IPs though | 19:48 |
| *** rongze has quit IRC | 19:48 | |
| SpamapS | so may be something sinister going on | 19:48 |
| *** ChanServ sets mode: +o fungi | 19:48 | |
| *** fungi sets mode: +b #openstack-infra!*@* | 19:49 | |
| * SpamapS shields his eyes from the shining +o halo atop fungi's head | 19:49 | |
| *** fungi sets mode: -b #openstack-infra!*@* | 19:49 | |
| *** ivar-lazzaro has quit IRC | 19:49 | |
| SpamapS | haha | 19:49 |
| *** fungi sets mode: +b *!~rongze@211.155.113.189 | 19:49 | |
| *** rongze has joined #openstack-infra | 19:49 | |
| SpamapS | did you just ban everybody? ;) | 19:50 |
| *** rongze has quit IRC | 19:50 | |
| *** fungi sets mode: +b *!~rongze@124.237.96.45 | 19:50 | |
| SpamapS | fungi: looks like *!~rongze@* is in order | 19:50 |
| SpamapS | fungi: IP has changed rapidly | 19:50 |
| *** markvan has quit IRC | 19:50 | |
| fungi | agreed | 19:50 |
| *** fungi sets mode: -b *!~rongze@124.237.96.45 | 19:51 | |
| *** weshay_brb is now known as weshay | 19:51 | |
| *** ^d has joined #openstack-infra | 19:51 | |
| *** fungi sets mode: -b *!~rongze@211.155.113.189 | 19:51 | |
| jog0 | sdague: if you have a moment https://review.openstack.org/#/q/status:open+project:openstack-dev/hacking+branch:master+topic:docs2,n,z | 19:51 |
| *** fungi sets mode: +b *!~rongze@* | 19:51 | |
| *** ^d has quit IRC | 19:51 | |
| *** fungi sets mode: -o fungi | 19:51 | |
| *** ^d has joined #openstack-infra | 19:52 | |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration https://review.openstack.org/65140 | 19:54 |
| SpamapS | fungi: btw 11:54 [Freenode] ... channels : #heat #tripleo #openstack-ironic #openstack-meeting-alt #openstack-meeting | 19:54 |
| fungi | SpamapS: thanks, i'll hit those as well | 19:55 |
| SpamapS | fungi: cool. Seems like there should be one big "openstack ban" button built into chanserv. | 19:55 |
| *** afazekas has quit IRC | 19:55 | |
| *** rhsu has joined #openstack-infra | 19:56 | |
| fungi | we might be able to do something along those lines once we get our channel config bot online | 19:56 |
| SpamapS | fungi: do all the things | 19:57 |
| SpamapS | now | 19:58 |
| SpamapS | now? | 19:58 |
| * SpamapS resists the disappointment | 19:58 | |
| fungi | not now, no. someone still needs to code it | 19:58 |
| zaro | fungi: there are no repositories for pre-built gerrit plugins. what would you recommend we do? | 19:58 |
| zaro | fungi: i guess there are 2 options. | 19:58 |
| *** rcarrillocruz has quit IRC | 19:59 | |
| zaro | fungi: build with our forked gerrit. | 19:59 |
| *** julim has quit IRC | 20:00 | |
| zaro | fungi: or create a pre-built one and put on tarballs. then just install when needed. | 20:00 |
| fungi | what are pre-built gerrit plugins? | 20:00 |
| zaro | pre-built means they are in jar form ready for install. | 20:01 |
| fungi | and you're saying those don't exist for download, or that there are no source code repositories for them? | 20:01 |
| * fungi is slightly confused | 20:01 | |
| fungi | also trying to fix too many things at once | 20:02 |
| zaro | there is source but no jars. you need to build them | 20:02 |
| *** jasondotstar has joined #openstack-infra | 20:02 | |
| *** nati_uen_ has joined #openstack-infra | 20:02 | |
| *** julim has joined #openstack-infra | 20:02 | |
| fungi | ahh, okay. so we probably need jobs to build those (so that we can easily recreate them when new versions are released to fix security bugs) | 20:03 |
| zaro | so gerrit monitoring is a non-core plugin to gerrit. we would need to checkout source and build it. you can't just download it from somewhere. | 20:03 |
| *** nati_uen_ has quit IRC | 20:04 | |
| *** nati_uen_ has joined #openstack-infra | 20:05 | |
| fungi | SpamapS: okay... i banned that account on all the channels you mentioned except for #heat and #openstack-ironic since i'm not in the access list for those channels | 20:05 |
| *** nati_ueno has quit IRC | 20:05 | |
| jerryz | fungi: just give you an update on the virutalenv issue i had last friday. I tried using a totally clean precise base image as the base for nodepool and then correct virutalenv version could be installed. | 20:06 |
| jerryz | fungi: before that, i used a base image with pip and virtualenv 1.10.1 already installed | 20:07 |
| fungi | jerryz: i saw your update--good to know | 20:07 |
| fungi | jerryz: probably the reason we hadn't run into it is that we start from provider base images, which have basically nothing installed except the bare minimums | 20:07 |
| SpamapS | fungi: werd | 20:08 |
| fungi | fewer surprises that way | 20:08 |
| jerryz | fungi: still curious about why. | 20:08 |
| sdague | jog0: I'll spin through those tomorrow | 20:08 |
| jerryz | fungi: because rerun puppet apply could make it work | 20:08 |
| fungi | jerryz: seems like it probably had trouble downgrading those during the install--we haven't really tested those scripts on base images with pip and virtualenv already installed | 20:09 |
| jog0 | sdague: thanks I am trying to get some of my 22 open patches merged | 20:09 |
| zaro | fungi: so would you just clone the plugin from upstream in the gerrit build job OR create an openstack project that tracks the plugin upstream and clone the from openstack project? | 20:09 |
| fungi | zaro: probably the latter, since we eventually want to be able to consume upstream's prebuilt gerrit warfiles | 20:10 |
| fungi | so something like an openstack-infra/gerrit-monitoring project which tracks that upstream source | 20:11 |
| *** herndon has joined #openstack-infra | 20:12 | |
| sdague | jog0: shouldn't you be asleep now? :) | 20:12 |
| jog0 | sdague: I didn't make it to LCA, got sick and had to stay home with dual ear infections | 20:13 |
| *** rfolco has quit IRC | 20:13 | |
| *** yolanda has quit IRC | 20:14 | |
| zaro | fungi: ok. when you have some free time would you be able to add an openstack-infra/javamelody project to track https://gerrit.googlesource.com/plugins/javamelody (this is the monitoring plugin). | 20:15 |
| sdague | jog0: oh, bummer | 20:15 |
| sdague | so I want to make headway on the first batch of ER patches right now, add me to review on anything you want me to look at, and I'll do it in the morning | 20:16 |
| sdague | also, commentary on the ER next steps thread would be appreciated | 20:16 |
| zaro | fungi: hopefully the gerrit guys will setup a maven repo to so we can consume the binaries soon. | 20:16 |
| jog0 | sdague: yeah I am still catching up from being sick that is on my list | 20:16 |
| fungi | zaro: let's make sure that gets added to the gerrit upgrade etherpad and then i or someone else can write up the puppet config change to add that project | 20:17 |
| zaro | fungi: ohh yeah forgot, puppet does this. i'll update the etherpad and create a patch for it. thanks. | 20:18 |
| fungi | thanks. right now my isp is severely breaking a bunch of my network traffic, which is making it hard to just keep on top of the nodepool/jenkins cleanup | 20:19 |
| *** burt has quit IRC | 20:23 | |
| *** sarob has joined #openstack-infra | 20:23 | |
| sdague | jog0: does that mean you might decide to join us in montreal? | 20:24 |
| jog0 | sdague: not sure but I think I can make that work yeah | 20:28 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/devstack-gate: Add Support for Marconi https://review.openstack.org/65145 | 20:28 |
| *** beagles has quit IRC | 20:28 | |
| *** jasondotstar has quit IRC | 20:29 | |
| *** jpeeler1 has quit IRC | 20:29 | |
| *** AaronGr is now known as AaronGr_Lunch | 20:30 | |
| *** jasondotstar has joined #openstack-infra | 20:30 | |
| portante | folks: had a job fail with DEVSTACK_GATE_TIMEOUT killed script: http://logs.openstack.org/22/61822/4/check/check-tempest-dsvm-full/7e86c15/console.html | 20:39 |
| portante | I quickly search nova and openstack-ci bug databases but did not see anything bugs covering this | 20:40 |
| *** thomasem has joined #openstack-infra | 20:40 | |
| portante | note that the .html file does not contain the analysis phase of the failures | 20:40 |
| portante | Logstash shows this has happened a bit: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiJHtERVZTVEFDS19HQVRFX1RJTUVPVVR9bVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiJhbGwiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg5MDQwMjMzMDMyfQ== | 20:41 |
| portante | just not very often | 20:41 |
| fungi | portante: we recently implemented a timeout capability inside the devstack-gate scripts so that we could still collect logs from jobs which run too long and timeout (the jenkins timeouts kill the log collection) | 20:41 |
| hashar | sounds like some of your folks think I am part of -infra :-D I am getting review requests! | 20:41 |
| portante | okay | 20:41 |
| fungi | we set the devstack-gate internal timeout to the old jenkins timeout values and then bumped the jenkins timeouts by an additional 5 minutes to cover script overhead | 20:42 |
| *** _ruhe is now known as ruhe | 20:42 | |
| *** dcramer_ has joined #openstack-infra | 20:42 | |
| fungi | hashar: welcome aboard! ;D | 20:42 |
| *** coolsvap_away has quit IRC | 20:42 | |
| hashar | I had folks at wikimedia asking me to conduct some root tasks. Had to publicly make it known that I have no root access :D | 20:42 |
| portante | fungi: so in this case, how best for me to set about rechecking this job? should I file a new bug, or is there an existing one to use? | 20:42 |
| bknudson | is netaddr>=0.7.6 not installing a known problem? | 20:43 |
| fungi | portante: the timeout isn't the problem, it's a symptom. you'll want to analyze why the job ran so long | 20:43 |
| fungi | bknudson: https://launchpad.net/bugs/1266513 | 20:43 |
| portante | okay, but can't we kill the job in such a way that it still provides the failure analysis for the tests that actually failed? | 20:44 |
| fungi | portante: did it not collect the logs after all? | 20:44 |
| *** melwitt has joined #openstack-infra | 20:44 | |
| bknudson | fungi: does every project that uses netaddr need a change like https://review.openstack.org/#/c/65019/ ? | 20:45 |
| portante | it collected, logs, its just that the end of the test run where it reports the stack traces for the individual failures does not happen | 20:45 |
| fungi | bknudson: yep | 20:45 |
| portante | if we are going to kill timeouts, can we also change testr to emit the FAIL report at the time the test failed? | 20:45 |
| fungi | portante: sounds like there's room for improvement then (previously there would have been no logs collected at all for a job which ran longer than the timeout) | 20:46 |
| portante | I agree this is a good step forward to not have jobs running for ever | 20:46 |
| *** AaronGr_Lunch is now known as AaronGr_Zzz | 20:46 | |
| portante | it sounds like testr summary can be nixed in favor of reporting failures as they happen given timeouts killing the script | 20:47 |
| *** sarob has quit IRC | 20:52 | |
| *** nati_ueno has joined #openstack-infra | 20:52 | |
| *** sarob has joined #openstack-infra | 20:52 | |
| harlowja | hmmm, is http://logs.openstack.org/23/65123/5/check/gate-taskflow-python26/229480e/console.html a known bug? | 20:53 |
| harlowja | 'Could not initialize class jenkins.model.Jenkins$MasterComputer' | 20:53 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add grenade-dsvm-rolling test https://review.openstack.org/64200 | 20:53 |
| fungi | harlowja: looks like a jenkins issue. i'll see if that slave is busted | 20:54 |
| harlowja | thx fungi | 20:54 |
| *** nati_uen_ has quit IRC | 20:55 | |
| fungi | seems to be okay, but i'll keep an eye on it | 20:56 |
| *** sarob has quit IRC | 20:57 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Add H904: don't wrap lines using a backslash https://review.openstack.org/64584 | 20:57 |
| bknudson | fungi: I proposed the change to python-keystoneclient: https://review.openstack.org/#/c/65149/ -- seems to work for me. | 20:57 |
| fungi | bknudson: awesome--thanks! | 20:58 |
| *** markmcclain1 has quit IRC | 20:58 | |
| *** markmcclain has joined #openstack-infra | 20:58 | |
| *** ruhe is now known as ruhe_away | 20:58 | |
| *** jpeeler has joined #openstack-infra | 20:59 | |
| *** jpeeler has quit IRC | 20:59 | |
| *** jpeeler has joined #openstack-infra | 20:59 | |
| *** ruhe_away is now known as ruhe | 20:59 | |
| *** sarob has joined #openstack-infra | 20:59 | |
| portante | fungi: not sure where to begin trying to find this problem with tests taking too long, since there is no recorded indication of what the failure mode of the test was | 21:00 |
| portante | if you have some ideas on where to start looking, that would be great | 21:00 |
| portante | I'd like to not "recheck no bug" this job, but neither can I afford to spent much more time on this | 21:01 |
| fungi | ick, looks like centos-6-3 is actually rapid-fire failing py26 unit test jobs | 21:01 |
| fungi | taking it offline | 21:01 |
| *** amotoki has joined #openstack-infra | 21:02 | |
| fungi | portante: maybe retrieve the subunit stream file and process it yourself? i'm not sure how to do that, but i've heard it suggested before | 21:02 |
| *** saschpe has quit IRC | 21:02 | |
| *** saschpe has joined #openstack-infra | 21:05 | |
| portante | fungi: thanks | 21:08 |
| *** saschpe has quit IRC | 21:09 | |
| *** saschpe has joined #openstack-infra | 21:11 | |
| *** kraman is now known as kraman_lunch | 21:12 | |
| *** krtaylor has quit IRC | 21:15 | |
| *** thomasem has quit IRC | 21:20 | |
| fungi | #status log gracefully stopping jenkins01 now. it has many nodes which are offline status and only a handful online, while nodepool thinks it has ~90 available to run jobs | 21:20 |
| *** _david_ has joined #openstack-infra | 21:21 | |
| *** dims has quit IRC | 21:22 | |
| *** AaronGr_Zzz is now known as AaronGr | 21:25 | |
| *** jcoufal has quit IRC | 21:26 | |
| Guest796 | fungi: what do you need help with? | 21:29 |
| Guest796 | gah | 21:29 |
| *** Guest796 is now known as jeblair | 21:29 | |
| jeblair | fungi: what do you need help with? | 21:29 |
| fungi | jeblair: nodepool/jenkins got themselves into a situation... i think it's on track to recovery now | 21:30 |
| *** jeblair is now known as Guest45132 | 21:30 | |
| fungi | nodepool seemed to be assigning lots of ready nodes to jenkins02 even though it was running basically no jobs and had nearly all its nodepool nodes marked offline in the jenkins webui | 21:31 |
| *** vipul is now known as vipul-away | 21:31 | |
| *** vipul-away is now known as vipul | 21:31 | |
| fungi | i gracefully stopped jenkins on jenkins02, then restarted the service | 21:31 |
| *** Guest45132 is now known as corvus | 21:32 | |
| fungi | it's got jobs running now, but nodepool also seems to think jenkins01 has way too many ready nodes which aren't getting used for the numerous backlogged jobs | 21:32 |
| fungi | so i've put jenkins01 on the path to a graceful shutdown for a graceful restart as well | 21:32 |
| *** dangers is now known as dangers_away | 21:33 | |
| fungi | then i'll see if we have any aliens reported by nodepool which look like nodes it created and clean those up too | 21:33 |
| fungi | just in case those are further reducing our available quota in places | 21:33 |
| fungi | also centos6-3 went into rapid-file job failures from jenkins agent problems, so i've offlined it and am hammering on it as i get time | 21:34 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add different envs for different sqlalchemy versions https://review.openstack.org/65135 | 21:34 |
| fungi | corvus: how's the conference? | 21:34 |
| corvus | fungi: warm. :) | 21:35 |
| fungi | i can imagine | 21:35 |
| corvus | fungi: the nodepool cleanup thread seems to be operational | 21:35 |
| fungi | yeah, i didn't really see much in the way of obviously stale nodes in the nodepool list | 21:36 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Mark resolved bugs https://review.openstack.org/65157 | 21:36 |
| *** dprince has quit IRC | 21:36 | |
| corvus | 2014-01-06 21:07:21,914 ERROR nodepool.NodeCompleteThread: Exception handling event for devstack-precise-hpcloud-az2-996323: | 21:37 |
| corvus | JenkinsException: node[devstack-precise-hpcloud-az2-996323] does not exist | 21:37 |
| corvus | that's interesting | 21:37 |
| fungi | that was probably me | 21:37 |
| *** dims has joined #openstack-infra | 21:37 | |
| corvus | fungi: i just picked an offline node from jenkins02 at random | 21:38 |
| *** dims is now known as Guest46822 | 21:38 | |
| corvus | fungi: also worth remembering that jenkins02 is running the newer version of jenkins | 21:38 |
| fungi | i started using nodepool delete on the jenkins02 nodes once it was shutdown, but then realized 1) it probably couldn't remove them from jenkins while in shutdown and 2) the cleanup thread ought to take care of those anyway | 21:38 |
| harlowja | fungi for that change releated to ' jenkins.model.Jenkins$MasterComputer' should i recheck against any bug? | 21:38 |
| fungi | corvus: yes, that crossed my mind | 21:38 |
| fungi | harlowja: can you open one? looks like there were quite a few changes which might end up getting failed out on py26 tests which ran on centos6 before i unplugged and reconditioned it | 21:39 |
| corvus | fungi: ok, yeah, based on the timestamp, that probably was you | 21:39 |
| harlowja | fungi sure | 21:39 |
| *** _david_ has left #openstack-infra | 21:39 | |
| fungi | harlowja: er, which ran on centos6-3 | 21:39 |
| corvus | fungi: i don't see any log messages after nodepool got the complete event for that node and before you ran delete... | 21:40 |
| *** flaper87|afk is now known as flaper87 | 21:41 | |
| fungi | corvus: i started shutting down jenkins02 shortly after 19:30 utc, so anything before that was not likely me | 21:41 |
| ubuntu1 | I have a request for the fungi work queue, when you can get to it: please disable verification permissions on this account: https://review.openstack.org/#/dashboard/9695 | 21:41 |
| *** ubuntu1 is now known as anteaya | 21:41 | |
| anteaya | sorry I didn't see my nick had changed | 21:42 |
| anteaya | I also am of the mind future 3rd party test accounts should not have verification permissions by default, but I have no group consense on that | 21:42 |
| fungi | corvus: and in hindsight, it looked like it had worked itself into an unpleasant state at least by at least 08:00 utc looking at the test nodes graph pattern | 21:43 |
| corvus | anteaya: i believe we have consensus on that; it's one of the undisputed points in the proposed change to the 3rd party test docs | 21:43 |
| anteaya | sorry, great | 21:43 |
| anteaya | let's do that then | 21:43 |
| fungi | anteaya: done | 21:44 |
| anteaya | fungi: thanks | 21:44 |
| * anteaya posts to ml | 21:44 | |
| *** thedodd has quit IRC | 21:44 | |
| fungi | i can add it back to https://review.openstack.org/#/admin/groups/91,members once they're confident it's working correctly | 21:44 |
| corvus | anteaya: (i don't think we've entirely worked out the criteria for _granting_ that access other than a vague requirement that the system have proved itself basically competent) | 21:44 |
| *** ociuhandu has left #openstack-infra | 21:46 | |
| fungi | worth noting, we're at roughly 30 service accounts for third-party systems now (a few of which we've disabled voting for by removing them from that group) | 21:46 |
| *** ociuhandu has joined #openstack-infra | 21:46 | |
| fungi | requests have been coming in at a steady rate of a few a week for a couple of months | 21:47 |
| *** mlipchuk has quit IRC | 21:47 | |
| *** Guest46822 has quit IRC | 21:48 | |
| *** CaptTofu has quit IRC | 21:48 | |
| *** CaptTofu has joined #openstack-infra | 21:49 | |
| harlowja | when u guys get some more time, was wondering if i could also work on seeing about how to do the mysql testing for taskflow, if i put a script in https://github.com/openstack-infra/config/tree/master/modules/openstack_project/files/jenkins_job_builder/config that installs & starts mysql, would that be enough | 21:50 |
| corvus | harlowja: mysql is already running on the unit test slaves | 21:50 |
| anteaya | corvus: yes, I conveyed as such in my email: http://lists.openstack.org/pipermail/openstack-dev/2014-January/023536.html | 21:51 |
| harlowja | corvus ok, so if i just connect to localhost:3306 (i think thats the port) all will be automatic | 21:51 |
| *** CaptTofu has quit IRC | 21:51 | |
| *** CaptTofu has joined #openstack-infra | 21:51 | |
| corvus | harlowja: yeah; check modules/jenkins/manifests/slave.pp; i think the user and pw are 'openstack_citest' | 21:52 |
| harlowja | k | 21:52 |
| harlowja | so i guess its typical for people to have a custom env in there tox.ini that has like 'use_mysql=1' | 21:52 |
| corvus | harlowja: (you're responsible for database cleanup) | 21:52 |
| *** senk has joined #openstack-infra | 21:52 | |
| harlowja | corvus interesting, doesn't reset on each build? | 21:52 |
| corvus | nope | 21:53 |
| harlowja | k | 21:53 |
| *** dizquierdo has joined #openstack-infra | 21:53 | |
| *** jaypipes has quit IRC | 21:53 | |
| corvus | harlowja: i think nova may have some test fixtures to handle creating and destroying a schema for each test | 21:53 |
| harlowja | ya, i can make something like that to, shouldn't be to bad | 21:53 |
| corvus | harlowja: check in with clarkb on that; i'll ask him when i see him | 21:53 |
| harlowja | k, thx | 21:53 |
| corvus | harlowja: (doing it for each test rather than job means testr can still run multiple tests in parallel) | 21:54 |
| harlowja | if each test does it, doesn't that mean each test would create/delete the db, even if another test is still using it? | 21:55 |
| *** jaypipes has joined #openstack-infra | 21:55 | |
| corvus | harlowja: i think the idea is that each one gets a unique db (so there are uniquely generated schema names involved) | 21:56 |
| harlowja | k | 21:56 |
| harlowja | that makes sense | 21:56 |
| *** senk has quit IRC | 21:56 | |
| fungi | corvus: got any hints on why nodepool would claim jenkins02 has 75 ready devstack nodes in hpcloud, and yet there are numerous devstack jobs backlogged in the gate pipeline? | 21:59 |
| fungi | (and this is after restarting jenkins02) | 21:59 |
| fungi | i guess jenkins02 isn't assigning jobs to them, just curious why | 22:00 |
| fungi | or is there a lag between nodepool-ready and jenkins-ready? | 22:00 |
| corvus | fungi: no, sholud be close, and i think nodepool-ready is only after the node at least exists in jenkins | 22:01 |
| *** sarob has quit IRC | 22:01 | |
| fungi | okay | 22:01 |
| corvus | fungi: i just picked one from the list: 997268 | 22:01 |
| corvus | 2014-01-06 21:46:48,019 DEBUG nodepool.NodeLauncher: Adding node id: 997268 to jenkins | 22:01 |
| corvus | fungi: but i don't see that on jenkins02 | 22:02 |
| *** ArxCruz has quit IRC | 22:02 | |
| corvus | fungi: ah, it never completed that... | 22:03 |
| corvus | fungi: oops, i was wrong; nodepool-ready is before jenkins-ready | 22:03 |
| *** ArxCruz has joined #openstack-infra | 22:03 | |
| corvus | fungi: so i think that explains the mismatch -- nodepool failed to add a bunch of nodes to jenkins but marked them ready anyway | 22:04 |
| fungi | well, the ready count in nodepool is dropping for those, so i guess they are making it into the fold, just with some lag | 22:04 |
| corvus | 2014-01-06 21:45:53,705 DEBUG nodepool.NodeLauncher: Adding node id: 997223 to jenkins | 22:06 |
| corvus | 2014-01-06 22:05:13,743 INFO nodepool.NodeLauncher: Node id: 997223 added to jenkins | 22:06 |
| corvus | fungi: that is some serious lag | 22:06 |
| *** mfer has quit IRC | 22:06 | |
| fungi | oh wow | 22:07 |
| corvus | fungi: it looks like the most recent node took 75 seconds to create on jenkins02 (that's after it started working on that task) | 22:09 |
| fungi | i'm looking at the thread list on 02 now, hoping something will jump out at me | 22:10 |
| corvus | jenkins01 added a node in 23 seconds | 22:11 |
| *** dims has joined #openstack-infra | 22:12 | |
| *** dims is now known as Guest15139 | 22:12 | |
| *** smarcet has left #openstack-infra | 22:12 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add new javamelody-plugin project https://review.openstack.org/65162 | 22:22 |
| *** Guest15139 has quit IRC | 22:23 | |
| SpamapS | has anyone seen issues with the new virtualenv where it doesn't install pip even though --no-pip was not specified? | 22:24 |
| SpamapS | seems to only happen with --system-site-packages | 22:24 |
| fungi | SpamapS: yeah, that's related to the fix i mentioned in that bug i linked (the one which should get fixed in 1.11.1) | 22:25 |
| *** homeless_ has joined #openstack-infra | 22:25 | |
| *** homeless_ has quit IRC | 22:26 | |
| *** homeless has quit IRC | 22:26 | |
| corvus | fungi: i'm wondering if the problem isn't just that the two jenkins masters are slow and nodepool is backlogged, but otherwise there isn't actually a problem (at least not of the kind where something definitively breaks) | 22:26 |
| *** homeless has joined #openstack-infra | 22:26 | |
| SpamapS | fungi: AHHHHHH | 22:27 |
| SpamapS | fungi: so maybe we should 'pip install virtualenv <1.11' until 1.11.1 is out? | 22:27 |
| fungi | corvus: that's about the only conclusion i've been able to arrive at so far. nothing is jumping out at me as obviously broken | 22:27 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add new javamelody-plugin project https://review.openstack.org/65162 | 22:27 |
| fungi | SpamapS: that's how we solved it for infra (pinned to 1.10.1 in puppet manifest) | 22:27 |
| corvus | fungi: your initial problem statement suggested that the issue was that jenkins wasn't running backlogged jobs; and i think at this point after looking at the code and logs, it seems like since nodepool marks it as ready before it's actually added to jenkins... | 22:28 |
| corvus | fungi: it would look like what we're seeing | 22:28 |
| SpamapS | lifeless: ^^ looks like virtualenv 1.11 is borken | 22:28 |
| *** ruhe is now known as _ruhe | 22:28 | |
| corvus | fungi: further, i note that nodepool is _still_ adding nodes to jenkins01, even though it should have stopped that because it's in shutdown, leading me to believe that nodepool's task queue for adding nodes to jenkins is very long. | 22:29 |
| SpamapS | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1266513 this bug right? | 22:29 |
| corvus | fungi: (unfortunately, task queue length is not logged) | 22:29 |
| fungi | corvus: do you think maybe we have some slowdown impacting us from one of our providers? | 22:29 |
| corvus | fungi: possibly (rax is throwing a bunch of http 500s right now), but that shouldn't cause a delay adding nodes to jenkins | 22:30 |
| *** dkranz has quit IRC | 22:30 | |
| fungi | SpamapS: probably. i'm having serious network problems with my isp right now, so can't verify. the external packages bug mentions the site packages issue, but the two are unrelated other than both cropping up due to new virtualenv/pip | 22:30 |
| SpamapS | fungi: roger, I'll just address it in TripleO and we can figure out what bug to hang it off of when you're not overloaded :) | 22:31 |
| fungi | SpamapS: i think there's an issue linked from https://github.com/pypa/virtualenv/commit/a6ca6f4 | 22:32 |
| corvus | fungi: some jenkins nodes ran multiple jobs though; i don't know what would cause that. | 22:32 |
| fungi | corvus: i wonder if jenkins02 came back up and reused nodes which hadn't been deleted | 22:32 |
| corvus | fungi: oh, yeah, that's probably it; because jenkins doesn't remember that they are offline | 22:33 |
| *** krtaylor has joined #openstack-infra | 22:33 | |
| * fungi will make sure to avoid that when bringing 01 back up after the restart | 22:33 | |
| corvus | fungi: still want to restart? | 22:33 |
| fungi | at this point, it seems safe to not restart it i guess | 22:34 |
| fungi | grah... about 50% of my outbound tcp sessions end in socket timeouts. this is not a good day for this | 22:34 |
| corvus | fungi: don't do anything to jenkins01 yet | 22:34 |
| corvus | fungi: eg, don't stop it but also don't remove it from shutdown mode, please | 22:35 |
| corvus | fungi: i want to poke a bit | 22:35 |
| fungi | corvus: not touching it | 22:35 |
| fungi | corvus: btw, i saw tempest tests running in hpcloud taking 2 hours to complete while waiting for 02 to shut down, which is what made me wonder if there's something external going on | 22:36 |
| corvus | fungi: yikes! | 22:37 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: write gerrit openstack/2.8 branch activity to #openstack-infra channel https://review.openstack.org/65165 | 22:37 |
| corvus | fungi: jenkins01 is deleting nodes in a reasonable time (a few seconds) | 22:37 |
| fungi | fast, but it's also not doing much at the moment, so little contention | 22:38 |
| corvus | fungi: yeah; i wanted to get a handle on why it was taking so long to add nodes | 22:38 |
| corvus | but i think it worked through that part of the task queue and probably just has node delete tasks now | 22:38 |
| corvus | fungi: it may be (past) time to spin up jenkins03 and 04 | 22:39 |
| *** kraman_lunch is now known as kraman | 22:39 | |
| *** dims_ has joined #openstack-infra | 22:41 | |
| fungi | one of the 2-hour tempest examples was http://logs.openstack.org/33/59033/17/check/check-tempest-dsvm-full/2dcdf08/console.html but it looks like that change had lots of tempest runs which didn't take that long | 22:42 |
| fungi | so probably not widespread | 22:42 |
| *** julim has quit IRC | 22:43 | |
| *** salv-orlando_ has joined #openstack-infra | 22:43 | |
| corvus | fungi: i think the task queue for jenknis01 is empty now | 22:44 |
| corvus | fungi: if you want to restart it, we should 'nodepool delete' the remaining offline nodes | 22:44 |
| *** amotoki has quit IRC | 22:45 | |
| fungi | i can do that--there aren't many at this point | 22:45 |
| *** david-lyle_ has quit IRC | 22:46 | |
| *** jgrimm has quit IRC | 22:46 | |
| corvus | fungi: actually, grep for them in nodepool list first; i think they may already be deleted, just not from jenknis | 22:46 |
| corvus | that's the case for 729800-845592 | 22:46 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1265672 https://review.openstack.org/64885 | 22:46 |
| fungi | will do. if so, i'll just delete them from jenkins | 22:47 |
| corvus | (i just checked the log for one of those; nodepool thought jenkins deleted it. sigh. | 22:47 |
| *** salv-orlando has quit IRC | 22:47 | |
| *** salv-orlando_ is now known as salv-orlando | 22:47 | |
| corvus | i guess nodepool will have to verify that in the future) | 22:47 |
| fungi | none of them are in nodepool's list | 22:47 |
| fungi | 8 in total | 22:48 |
| fungi | i'll start deleting them in jenkins | 22:48 |
| corvus | fungi: i'll pitch in | 22:48 |
| fungi | yep, all gone now | 22:49 |
| corvus | fungi: cool, i'm ready for you to restart it | 22:49 |
| fungi | i'll restart the jenkins service since we can without impact | 22:49 |
| *** mlipchuk has joined #openstack-infra | 22:50 | |
| fungi | stopped cleanly, so i've restarted it now. should be on its way back up | 22:50 |
| fungi | watching the job it's running on centos6-3 to make sure it's not gone off the rails again | 22:52 |
| corvus | fungi: cool. so how about i spin up jenkins03 and jenkins04 on 16gb performance nodes? | 22:54 |
| fungi | corvus: sounds great--the the necessary bits for them are already in hiera at this point, so should go quickly | 22:55 |
| fungi | i would do it, but i don't remember what you carted over from jenkins.o.o when you built 01 and 02 | 22:55 |
| fungi | since i recall you saying there were bits which needed to be manually copied | 22:56 |
| corvus | fungi: yeah, there's a tarball on jenkins.o.o that should take care of most of it. | 22:56 |
| fungi | was it just the jenkins ssh key, or also the main config.xml stream-edited? | 22:56 |
| fungi | i'll take a look through it so i know for next time | 22:57 |
| *** rnirmal_ has joined #openstack-infra | 22:57 | |
| *** rnirmal_ has quit IRC | 22:57 | |
| *** dims_ has quit IRC | 22:58 | |
| fungi | not sure why, but zuul seems to be preferring check pipeline jobs over gate pipeline for the devstack jobs. that keystone job has been waiting a long while for runners and only just got its first | 22:58 |
| *** praneshp has quit IRC | 22:58 | |
| corvus | fungi: because of a gate reset | 22:59 |
| *** rnirmal has quit IRC | 22:59 | |
| corvus | fungi: they share the same priority, so when a gate reset happens, all the gate jobs end up after the check jobs | 22:59 |
| fungi | yeah, i guess the reset was done and gone for a while and zuul had already decided what needed processing next | 22:59 |
| fungi | makes sense | 22:59 |
| ^d | Does mcuadros hang around IRC? | 23:00 |
| ^d | Whoops, wrong channel. | 23:00 |
| fungi | the test nodes graph looks like it may be sane again for the first time all day | 23:00 |
| *** dcramer_ has quit IRC | 23:01 | |
| corvus | fungi: looking at http://cacti.openstack.org/cacti/graph_image.php?local_graph_id=819&rra_id=0&view_type=tree&graph_start=1388962638&graph_end=1389049038 | 23:01 |
| corvus | fungi: does 15g look ok to you? | 23:01 |
| corvus | fungi: oh, wait | 23:02 |
| corvus | fungi: 15g drops back down to 4 vcpus | 23:02 |
| fungi | corvus: yeah, it looks like it could benefit from additional cache memory above that, but the cpu count is what's going to hit us harder i suspect | 23:03 |
| corvus | fungi: hrm, looking at the cpu graphs, i wonder if we're getting much benefit from 8 now.. | 23:04 |
| *** dstanek has quit IRC | 23:04 | |
| *** praneshp has joined #openstack-infra | 23:04 | |
| corvus | load average of 3 and <40% cpu usage under stress | 23:05 |
| fungi | mmm, yeah 01 looks underutilized | 23:05 |
| corvus | fungi: 02 doesn't look much different | 23:05 |
| corvus | i expect it's related to all the thread locks | 23:05 |
| fungi | 02 as well, so new thread manager doesn't seem to let it push any further | 23:06 |
| fungi | yeah, agree | 23:06 |
| fungi | so maybe the 15g performance won't be that bad after all | 23:06 |
| corvus | fungi: let's try it | 23:06 |
| fungi | the plan is also to start using only nodepool nodes on 03 and 04, and up the number of jobs which are using bare nodes to make more use of the new masters, right? | 23:07 |
| corvus | fungi: yeah, let's leave the long-running slaves just on 01 and 02 for now | 23:07 |
| *** rnirmal has joined #openstack-infra | 23:07 | |
| fungi | sounds good--saves us a migration | 23:08 |
| reed | oh, I better don't launch on a rant against mediawiki | 23:09 |
| reed | what can I rant about instead? | 23:09 |
| corvus | we should see if java is really using all 8g of heap; if we can shrink that, we might be able to use an 8g/8vcpu node | 23:10 |
| *** dims_ has joined #openstack-infra | 23:10 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: use pandas to handle all_fails list https://review.openstack.org/65173 | 23:10 |
| corvus | fungi: biab | 23:10 |
| fungi | corvus: k | 23:11 |
| corvus | fungi: (nodes launching) | 23:11 |
| *** UtahDave has quit IRC | 23:11 | |
| reed | let's rant about java | 23:12 |
| reed | corvus, good morning Australia! | 23:12 |
| *** CaptTofu has quit IRC | 23:13 | |
| *** CaptTofu has joined #openstack-infra | 23:13 | |
| *** UtahDave has joined #openstack-infra | 23:13 | |
| *** hashar has quit IRC | 23:14 | |
| fungi | corvus: out of curiosity, what does nodepool use most of its memory for? it's got about 0.85gib active according to the process list. i guess that's tons of novaclient sessions? | 23:15 |
| fungi | seems like a lot of ram for a very lightweight python daemon, but i guess the novaclient lib makes it much less lightweight | 23:16 |
| rhsu | i noticed the "Devstack nodes" jenkins jobs have been disabled, have they been replaced with some other mechanism? | 23:19 |
| *** ryanpetrello has quit IRC | 23:20 | |
| portante | corvus: on a gate reset, does the lead gate job get resources assigned to it before the others behind it in the queue? | 23:20 |
| *** mrodden has quit IRC | 23:21 | |
| fungi | rhsu: they were renamed to "dsvm" so we could shorten the job names | 23:22 |
| *** pcrews_ has quit IRC | 23:22 | |
| fungi | rhsu: oh, wait, you mean the ones which used to spin up and tear down devstack-gate nodes. that got replaced by the openstack-infra/nodepool project | 23:22 |
| fungi | portante: yes, they get resources assigned in series based on their position in the gate, so the ones at the head get available nodes before the ones behind them | 23:23 |
| portante | k thanks | 23:24 |
| rhsu | fungi: does nodepool also handle updating the vms? Just curious because I was reading something that referenced "devstack-vm-update-image.sh" which doesn't seem to exist anymore. | 23:25 |
| jog0 | sdague: reading your e-r email now | 23:25 |
| *** dstanek has joined #openstack-infra | 23:26 | |
| *** david-lyle_ has joined #openstack-infra | 23:26 | |
| *** oubiwann has quit IRC | 23:27 | |
| fungi | rhsu: yes. the nodepoold daemon builds template vms to create preconfigured images, then caches those snapshots and launches jenkins slaves from them | 23:27 |
| fungi | rhsu: has its own built-in scheduler, predictive heuristics based on load and other neat features | 23:27 |
| rhsu | awesome, exactly what i was looking for! | 23:27 |
| rhsu | many thanks | 23:28 |
| fungi | rhsu: several groups outside openstack-infra are already using it to some success, so hopefully it shouldn't be hard to figure out, but we're here if you have questions | 23:28 |
| *** dims_ has quit IRC | 23:31 | |
| *** dizquierdo has quit IRC | 23:31 | |
| *** kraman has quit IRC | 23:32 | |
| corvus | rhsu: it doesn't have docs yet; i've been writing them in my spare time on this trip i'm on, but they aren't done yet. sorry. | 23:33 |
| *** wenlock has quit IRC | 23:33 | |
| rhsu | corvus: ah no worries, i'm just poking through the code right now :) | 23:33 |
| fungi | corvus: rhsu: though as i said, it's in use more places than just here, so apparently the code is well-commented enough to suss out how to get it up and running | 23:34 |
| *** david-lyle_ has quit IRC | 23:35 | |
| *** senk has joined #openstack-infra | 23:35 | |
| *** rwsu has quit IRC | 23:35 | |
| *** wenlock has joined #openstack-infra | 23:38 | |
| fungi | (with some gnashing of teeth and the occasional question in irc, perhaps) | 23:38 |
| *** pcrews has joined #openstack-infra | 23:39 | |
| *** homeless has quit IRC | 23:40 | |
| corvus | wow, took 3 tries to add the A record for jenkins04 | 23:41 |
| fungi | sounds like rackspace may be taking a beating? i'll check their status page | 23:41 |
| fungi | nope, all green, so at least nothing they want anybody to know about | 23:42 |
| corvus | fungi: can you install the newer of our two jenkins versions on 03 and 04? | 23:43 |
| fungi | cloud files apparently degraded in ord, but i can't seem to expand teh detail note | 23:43 |
| fungi | corvus: sure. you want it to match -dev and 02 presumably | 23:43 |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add compass project to stackforge https://review.openstack.org/65178 | 23:44 |
| corvus | fungi: yep | 23:44 |
| fungi | working on that now | 23:44 |
| *** flaper87 is now known as flaper87|afk | 23:44 | |
| fungi | corvus: out of curiosity, what does nodepool use most of its memory for? it's got about 0.85gib active according to the process list. i guess that's tons of novaclient sessions? | 23:44 |
| corvus | fungi: oh, sorry. hrm. it should only have 1 session per provider | 23:45 |
| corvus | fungi: worth looking into. | 23:45 |
| *** vipul is now known as vipul-away | 23:47 | |
| clarkb2 | corvus: you have been productive. /me hangs head in shame | 23:47 |
| fungi | looks like we want http://pkg.jenkins-ci.org/debian/binary/jenkins_1.543_all.deb | 23:48 |
| corvus | clarkb2: i was really looking forward to that walk through the park this morning. :( | 23:48 |
| clarkb2 | :( tomorrow? | 23:48 |
| *** dims_ has joined #openstack-infra | 23:48 | |
| clarkb2 | tomorrow will be cool too | 23:48 |
| corvus | clarkb2: hope so; assuming i'm still getting up at 5am. | 23:48 |
| fungi | corvus: jenkins has been downgraded from 1.546 to 1.543 on 03 and 04 | 23:49 |
| openstackgerrit | Jim Branen proposed a change to openstack/requirements: Use new hplefthandclient https://review.openstack.org/65179 | 23:49 |
| *** krotscheck has joined #openstack-infra | 23:50 | |
| corvus | hrm | 23:50 |
| corvus | jenkins user was created with wrong homedir location | 23:50 |
| *** rwsu has joined #openstack-infra | 23:50 | |
| corvus | passwd says /home/jenkins instead of /var/lib/jenkins | 23:50 |
| fungi | No directory, logging in with HOME=/ | 23:50 |
| *** vipul-away is now known as vipul | 23:50 | |
| fungi | (seen in the dpkg output) | 23:50 |
| fungi | i wonder if the debs got installed by puppet before the user creation was triggered by puppet (such that the package maintainer scripts created the jenkins user instead with defaults)? | 23:51 |
| *** jasondotstar has quit IRC | 23:51 | |
| corvus | fungi: i thought the package created it with homedir in var | 23:51 |
| corvus | maybe that changed | 23:52 |
| corvus | fungi: and then what you said happened | 23:52 |
| fungi | might be that we have it the other way around in puppet too | 23:52 |
| fungi | maybe puppet went and created the user first and doesn't specify the correct homedir in the manifest... checking now | 23:52 |
| corvus | i ran usermod -d /var/lib/jenkins jenkins | 23:52 |
| fungi | you may want to restart the daemon too | 23:53 |
| corvus | well, i had to stop it to run that | 23:53 |
| fungi | ahh, fair enough | 23:53 |
| fungi | directory was in use | 23:53 |
| corvus | also chsh -s /bin/bash jenkins | 23:54 |
| *** oubiwann has joined #openstack-infra | 23:55 | |
| corvus | untarred the config tarball | 23:55 |
| corvus | clarkb2: want a ride? | 23:55 |
| fungi | looks like we don't create the jenkins user in the puppet manifest (except on slaves), so maybe the package did change it | 23:55 |
| clarkb2 | corvus: pleia2 anteaya larissa and I are meeting in the lobby of hilton at 8am and going to bus over. happy to ask if they want a ride instead. will you have room or should we just bus? | 23:56 |
| corvus | clarkb2: that's 1 too many, sorry | 23:57 |
| *** david-lyle_ has joined #openstack-infra | 23:57 | |
| pleia2 | I can take a bus with anteaya | 23:57 |
| *** fifieldt has joined #openstack-infra | 23:58 | |
| clarkb2 | np we'll bus thanks though | 23:58 |
| *** rcleere_ has joined #openstack-infra | 23:58 | |
| *** yidclare1 has joined #openstack-infra | 23:58 | |
| *** nati_uen_ has joined #openstack-infra | 23:58 | |
| corvus | fungi: can you install the zmq-event-publisher and scp plugins by copying from one of the other masters? | 23:58 |
| fungi | nothing obvious in the jenkins changelog nor the package changelog suggesting the homedir moved | 23:59 |
| *** salv-orlando_ has joined #openstack-infra | 23:59 | |
| fungi | corvus: sure thing | 23:59 |
| corvus | fungi: i'm going to finish getting ready and head over | 23:59 |
| fungi | corvus: have fun | 23:59 |
| corvus | fungi: i'll rejoin you shortly | 23:59 |
| *** branen_ has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
