| *** salv-orlando has joined #openstack-infra | 00:02 | |
| *** asettle has quit IRC | 00:03 | |
| *** zz_dimtruck is now known as dimtruck | 00:04 | |
| *** baoli has joined #openstack-infra | 00:05 | |
| *** asettle has joined #openstack-infra | 00:06 | |
| *** baoli has quit IRC | 00:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Update contributing resources doc benefits section https://review.openstack.org/300912 | 00:12 |
|---|---|---|
| fungi | jhesketh: ^ any chance you could give that a once-over? about to reply to someone at a resource donor about that | 00:13 |
| jhesketh | fungi: of course, looking | 00:13 |
| fungi | appreciated! | 00:13 |
| fungi | jeblair: ^ you too, if you happen to be around (since you were the original author, so might have omitted those things intentionally) | 00:14 |
| * fungi won't promise it's exciting weekend reading | 00:15 | |
| jhesketh | fungi: +2 | 00:16 |
| *** asettle has quit IRC | 00:18 | |
| *** achanda has quit IRC | 00:18 | |
| *** asettle has joined #openstack-infra | 00:19 | |
| *** asettle has quit IRC | 00:23 | |
| *** asettle has joined #openstack-infra | 00:24 | |
| *** aeng has quit IRC | 00:24 | |
| *** darrenc is now known as darrenc_afk | 00:26 | |
| *** asettle has quit IRC | 00:27 | |
| *** dimtruck is now known as zz_dimtruck | 00:27 | |
| *** salv-orlando has quit IRC | 00:28 | |
| *** asettle has joined #openstack-infra | 00:28 | |
| *** sflanigan has joined #openstack-infra | 00:28 | |
| *** sflanigan has joined #openstack-infra | 00:28 | |
| *** sdake has joined #openstack-infra | 00:31 | |
| *** sdake_ has quit IRC | 00:34 | |
| *** thorst has joined #openstack-infra | 00:37 | |
| *** thorst has quit IRC | 00:45 | |
| *** markvoelker has joined #openstack-infra | 00:47 | |
| *** EricGonczer_ has joined #openstack-infra | 00:49 | |
| *** aeng has joined #openstack-infra | 00:51 | |
| *** darrenc_afk is now known as darrenc | 00:53 | |
| *** achanda has joined #openstack-infra | 00:57 | |
| *** thorst has joined #openstack-infra | 00:58 | |
| *** hichihara has joined #openstack-infra | 00:58 | |
| *** nadya has joined #openstack-infra | 00:59 | |
| *** thorst has quit IRC | 01:00 | |
| *** tphummel has quit IRC | 01:02 | |
| *** nadya has quit IRC | 01:03 | |
| *** tphummel has joined #openstack-infra | 01:04 | |
| *** aeng has quit IRC | 01:06 | |
| *** achanda has quit IRC | 01:07 | |
| *** mriedem has quit IRC | 01:11 | |
| *** sdake_ has joined #openstack-infra | 01:16 | |
| *** sdake has quit IRC | 01:18 | |
| *** aeng has joined #openstack-infra | 01:19 | |
| *** Goneri has quit IRC | 01:20 | |
| *** mylu has quit IRC | 01:21 | |
| *** salv-orlando has joined #openstack-infra | 01:27 | |
| *** mylu has joined #openstack-infra | 01:29 | |
| *** mylu has quit IRC | 01:30 | |
| *** mylu has joined #openstack-infra | 01:32 | |
| *** EricGonczer_ has quit IRC | 01:37 | |
| *** asettle has quit IRC | 01:41 | |
| *** asettle has joined #openstack-infra | 01:41 | |
| *** thorst has joined #openstack-infra | 01:44 | |
| *** salv-orlando has quit IRC | 01:44 | |
| *** asettle has quit IRC | 01:45 | |
| *** asettle has joined #openstack-infra | 01:45 | |
| *** baoli has joined #openstack-infra | 01:46 | |
| *** bapalm has joined #openstack-infra | 01:51 | |
| *** sdake has joined #openstack-infra | 01:53 | |
| *** thorst has quit IRC | 01:53 | |
| *** thorst has joined #openstack-infra | 01:53 | |
| *** sdake_ has quit IRC | 01:54 | |
| *** thorst has quit IRC | 02:02 | |
| *** mylu has quit IRC | 02:02 | |
| *** asettle has quit IRC | 02:02 | |
| *** asettle has joined #openstack-infra | 02:02 | |
| *** mylu has joined #openstack-infra | 02:03 | |
| *** sdake has quit IRC | 02:03 | |
| *** ramishra_ has joined #openstack-infra | 02:04 | |
| *** ramishra_ has quit IRC | 02:05 | |
| *** kengo_sakai has joined #openstack-infra | 02:05 | |
| *** mylu has quit IRC | 02:05 | |
| *** sdake has joined #openstack-infra | 02:06 | |
| *** mylu has joined #openstack-infra | 02:06 | |
| *** achanda has joined #openstack-infra | 02:07 | |
| *** ramishra has quit IRC | 02:09 | |
| *** ramishra has joined #openstack-infra | 02:09 | |
| *** ramishra has quit IRC | 02:12 | |
| *** achanda has quit IRC | 02:13 | |
| *** gildub has joined #openstack-infra | 02:14 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Update TODO.rst to point the StoryBoard https://review.openstack.org/297521 | 02:19 |
| *** pcrews_ has quit IRC | 02:26 | |
| *** pcrews has joined #openstack-infra | 02:26 | |
| *** yamamoto has quit IRC | 02:27 | |
| *** roxanaghe has joined #openstack-infra | 02:27 | |
| *** baoli has quit IRC | 02:28 | |
| *** denisra has quit IRC | 02:28 | |
| *** denisra has joined #openstack-infra | 02:29 | |
| *** roxanaghe has quit IRC | 02:31 | |
| *** yamamoto has joined #openstack-infra | 02:31 | |
| *** sridhar_ram has joined #openstack-infra | 02:35 | |
| *** sridhar_ram1 has joined #openstack-infra | 02:38 | |
| *** sridhar_ram has quit IRC | 02:40 | |
| *** links has joined #openstack-infra | 02:46 | |
| *** tphummel has quit IRC | 02:46 | |
| *** tphummel has joined #openstack-infra | 02:46 | |
| *** yamamoto has quit IRC | 02:50 | |
| *** thorst has joined #openstack-infra | 02:59 | |
| *** nadya has joined #openstack-infra | 02:59 | |
| *** kengo_sakai has quit IRC | 03:00 | |
| *** kengo_sakai has joined #openstack-infra | 03:00 | |
| *** Qiming has quit IRC | 03:00 | |
| *** kengo_sakai has quit IRC | 03:00 | |
| *** kengo_sakai has joined #openstack-infra | 03:01 | |
| *** salv-orlando has joined #openstack-infra | 03:03 | |
| *** nadya has quit IRC | 03:04 | |
| *** thorst has quit IRC | 03:07 | |
| *** achanda has joined #openstack-infra | 03:09 | |
| *** salv-orlando has quit IRC | 03:13 | |
| *** markvoelker has quit IRC | 03:14 | |
| *** achanda has quit IRC | 03:16 | |
| *** [1]Thelo has joined #openstack-infra | 03:20 | |
| *** salv-orlando has joined #openstack-infra | 03:22 | |
| *** Thelo has quit IRC | 03:22 | |
| *** [1]Thelo is now known as Thelo | 03:22 | |
| openstackgerrit | KATO Tomoyuki proposed openstack-infra/project-config: Enable release notes translation for python projects https://review.openstack.org/300923 | 03:24 |
| *** jamielennox is now known as jamielennox|away | 03:27 | |
| *** salv-orlando has quit IRC | 03:27 | |
| *** sdake_ has joined #openstack-infra | 03:29 | |
| *** fedexo has joined #openstack-infra | 03:30 | |
| *** sdake has quit IRC | 03:32 | |
| *** jamielennox|away is now known as jamielennox | 03:37 | |
| *** abregman has quit IRC | 03:40 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Register connections when testing configuration https://review.openstack.org/300924 | 03:40 |
| *** nwkarsten has joined #openstack-infra | 03:53 | |
| *** asettle has quit IRC | 03:54 | |
| *** asettle has joined #openstack-infra | 03:54 | |
| *** mylu has quit IRC | 03:56 | |
| *** nwkarsten has quit IRC | 03:57 | |
| *** yamamoto has joined #openstack-infra | 04:01 | |
| *** asettle has quit IRC | 04:02 | |
| *** asettle has joined #openstack-infra | 04:02 | |
| *** akshai_ has joined #openstack-infra | 04:03 | |
| *** thorst has joined #openstack-infra | 04:05 | |
| *** mylu has joined #openstack-infra | 04:05 | |
| *** akshai has quit IRC | 04:06 | |
| *** roxanaghe has joined #openstack-infra | 04:08 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Add the log of package list for pip3 https://review.openstack.org/246017 | 04:11 |
| *** sdake_ is now known as sdake | 04:12 | |
| *** thorst has quit IRC | 04:12 | |
| *** roxanaghe has quit IRC | 04:12 | |
| *** achanda has joined #openstack-infra | 04:13 | |
| *** markvoelker has joined #openstack-infra | 04:15 | |
| *** nadya has joined #openstack-infra | 04:15 | |
| *** achanda has quit IRC | 04:18 | |
| *** nadya has quit IRC | 04:20 | |
| *** markvoelker has quit IRC | 04:21 | |
| *** maishsk has quit IRC | 04:23 | |
| *** mylu_ has joined #openstack-infra | 04:25 | |
| *** mylu has quit IRC | 04:27 | |
| *** maishsk has joined #openstack-infra | 04:35 | |
| *** sridhar_ram1 has quit IRC | 04:36 | |
| *** asselin__ has quit IRC | 04:37 | |
| *** netsin has quit IRC | 04:38 | |
| *** maishsk has quit IRC | 04:39 | |
| *** baoli has joined #openstack-infra | 04:40 | |
| *** sshnaidm has quit IRC | 04:41 | |
| *** tphummel has quit IRC | 04:41 | |
| *** salv-orlando has joined #openstack-infra | 04:42 | |
| *** baoli has quit IRC | 04:45 | |
| *** salv-orlando has quit IRC | 04:45 | |
| *** elo has quit IRC | 04:45 | |
| *** achanda has joined #openstack-infra | 04:46 | |
| *** sshnaidm has joined #openstack-infra | 04:49 | |
| *** elo has joined #openstack-infra | 04:51 | |
| *** abregman has joined #openstack-infra | 04:54 | |
| *** akshai_ has quit IRC | 04:58 | |
| *** kengo_sa_ has joined #openstack-infra | 04:59 | |
| *** maishsk has joined #openstack-infra | 04:59 | |
| *** thorst has joined #openstack-infra | 05:10 | |
| *** amotoki has joined #openstack-infra | 05:11 | |
| *** asettle has quit IRC | 05:12 | |
| *** asettle has joined #openstack-infra | 05:13 | |
| *** thorst has quit IRC | 05:17 | |
| *** amotoki has quit IRC | 05:18 | |
| *** mylu_ has quit IRC | 05:19 | |
| *** oanson has joined #openstack-infra | 05:23 | |
| *** akaszuba has joined #openstack-infra | 05:34 | |
| *** netsin has joined #openstack-infra | 05:34 | |
| *** amotoki has joined #openstack-infra | 05:37 | |
| *** jcoufal has joined #openstack-infra | 05:38 | |
| *** nadya has joined #openstack-infra | 05:40 | |
| *** cody-somerville has quit IRC | 05:41 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr: Remove openstack-common.conf https://review.openstack.org/300943 | 05:47 |
| *** maishsk_ has joined #openstack-infra | 05:48 | |
| *** maishsk has quit IRC | 05:49 | |
| *** maishsk_ is now known as maishsk | 05:49 | |
| *** kengo_sa_ has quit IRC | 05:52 | |
| *** asettle has quit IRC | 05:55 | |
| *** roxanaghe has joined #openstack-infra | 05:56 | |
| *** asettle has joined #openstack-infra | 05:57 | |
| *** asettle has quit IRC | 05:57 | |
| *** roxanaghe has quit IRC | 06:01 | |
| *** nadya has quit IRC | 06:05 | |
| *** asettle has joined #openstack-infra | 06:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove unused magnum-ui mitaka translation job https://review.openstack.org/294286 | 06:13 |
| *** salv-orlando has joined #openstack-infra | 06:13 | |
| *** asettle has quit IRC | 06:14 | |
| *** thorst has joined #openstack-infra | 06:15 | |
| *** netsin has quit IRC | 06:15 | |
| *** salv-orlando has quit IRC | 06:16 | |
| *** asettle has joined #openstack-infra | 06:16 | |
| *** jtomasek has joined #openstack-infra | 06:20 | |
| *** ihrachys has joined #openstack-infra | 06:21 | |
| *** ihrachys has quit IRC | 06:21 | |
| *** thorst has quit IRC | 06:21 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr: Add version option for CLI https://review.openstack.org/300952 | 06:23 |
| *** salv-orlando has joined #openstack-infra | 06:24 | |
| *** daemontool has joined #openstack-infra | 06:24 | |
| *** salv-orlando has quit IRC | 06:28 | |
| *** scheuran has joined #openstack-infra | 06:28 | |
| *** flepied has quit IRC | 06:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/300845 | 06:43 |
| *** achanda has quit IRC | 06:48 | |
| *** sdake has quit IRC | 06:48 | |
| *** sdake has joined #openstack-infra | 06:50 | |
| yolanda | good morning | 06:52 |
| *** sdake_ has joined #openstack-infra | 06:53 | |
| *** maishsk has quit IRC | 06:54 | |
| *** kengo_sa_ has joined #openstack-infra | 06:55 | |
| *** sdake has quit IRC | 06:56 | |
| *** yamahata has quit IRC | 06:58 | |
| *** nmagnezi has joined #openstack-infra | 06:58 | |
| *** sdake has joined #openstack-infra | 06:59 | |
| *** sdake_ has quit IRC | 07:00 | |
| *** ihrachys has joined #openstack-infra | 07:03 | |
| *** achanda has joined #openstack-infra | 07:03 | |
| *** kengo_sa_ has quit IRC | 07:03 | |
| *** nadya has joined #openstack-infra | 07:03 | |
| *** mikelk has joined #openstack-infra | 07:04 | |
| *** yamamoto has quit IRC | 07:04 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr: Add version option for ostestr and subunit-trace https://review.openstack.org/300952 | 07:05 |
| *** yamahata has joined #openstack-infra | 07:05 | |
| *** yamamoto has joined #openstack-infra | 07:06 | |
| *** fedexo has quit IRC | 07:08 | |
| *** nadya has quit IRC | 07:08 | |
| *** asettle has quit IRC | 07:08 | |
| yolanda | #status notice Gerrit is going to be restarted due to bad performance | 07:12 |
| openstackstatus | yolanda: sending notice | 07:12 |
| *** flepied has joined #openstack-infra | 07:12 | |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to bad performance | 07:14 | |
| openstackstatus | yolanda: finished sending notice | 07:15 |
| *** arxcruz has joined #openstack-infra | 07:16 | |
| *** salv-orlando has joined #openstack-infra | 07:17 | |
| openstackgerrit | Merged openstack-infra/system-config: Update contributing resources doc benefits section https://review.openstack.org/300912 | 07:18 |
| *** thorst has joined #openstack-infra | 07:19 | |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 07:23 |
| *** achanda has quit IRC | 07:24 | |
| *** achanda has joined #openstack-infra | 07:24 | |
| *** jlanoux has joined #openstack-infra | 07:26 | |
| *** yamahata has quit IRC | 07:27 | |
| *** thorst has quit IRC | 07:27 | |
| *** tojuvone_ is now known as tojuvone | 07:28 | |
| *** yamamoto has quit IRC | 07:29 | |
| *** yamamoto has joined #openstack-infra | 07:30 | |
| *** |-paul-| has joined #openstack-infra | 07:30 | |
| *** achanda has quit IRC | 07:32 | |
| *** fhubik has joined #openstack-infra | 07:33 | |
| *** vincentll has joined #openstack-infra | 07:34 | |
| openstackgerrit | Merged openstack-infra/system-config: Add Zuul restart documentation https://review.openstack.org/300223 | 07:34 |
| *** Qiming has joined #openstack-infra | 07:34 | |
| *** nelsnels_ has joined #openstack-infra | 07:34 | |
| *** davideag_ has joined #openstack-infra | 07:34 | |
| *** nelsnelson has quit IRC | 07:34 | |
| *** maishsk has joined #openstack-infra | 07:36 | |
| *** davideagnello has quit IRC | 07:37 | |
| *** eliqiao has quit IRC | 07:37 | |
| *** charz has quit IRC | 07:37 | |
| *** mrunge has quit IRC | 07:37 | |
| *** eliqiao has joined #openstack-infra | 07:39 | |
| *** adreznec has quit IRC | 07:39 | |
| *** taron has quit IRC | 07:39 | |
| *** nibalizer has quit IRC | 07:39 | |
| *** nibalizer has joined #openstack-infra | 07:40 | |
| *** Jeffrey4l__ has quit IRC | 07:40 | |
| *** bswartz has quit IRC | 07:40 | |
| *** kevinshort has quit IRC | 07:40 | |
| *** kevinshort has joined #openstack-infra | 07:40 | |
| *** grassass has quit IRC | 07:40 | |
| *** sflanigan has quit IRC | 07:40 | |
| *** sflanigan has joined #openstack-infra | 07:40 | |
| *** sflanigan has joined #openstack-infra | 07:40 | |
| *** adreznec has joined #openstack-infra | 07:41 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump keystoneauth to 2.5.0 https://review.openstack.org/300967 | 07:42 |
| *** Jeffrey4l__ has joined #openstack-infra | 07:43 | |
| *** bigjools has quit IRC | 07:43 | |
| *** rcernin has joined #openstack-infra | 07:43 | |
| *** grassass has joined #openstack-infra | 07:43 | |
| *** charz has joined #openstack-infra | 07:43 | |
| *** bigjools has joined #openstack-infra | 07:43 | |
| *** bigjools has quit IRC | 07:43 | |
| *** bigjools has joined #openstack-infra | 07:43 | |
| *** taron has joined #openstack-infra | 07:44 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:44 | |
| *** mrmartin has joined #openstack-infra | 07:45 | |
| *** ifarkas has joined #openstack-infra | 07:47 | |
| *** isaacb has joined #openstack-infra | 07:48 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add experimental job to build atomic images https://review.openstack.org/300968 | 07:52 |
| *** zeih has joined #openstack-infra | 07:52 | |
| *** salv-orlando has quit IRC | 07:53 | |
| *** sdake has quit IRC | 07:53 | |
| *** fawadkhaliq has joined #openstack-infra | 07:54 | |
| *** maishsk has quit IRC | 07:54 | |
| *** fhubik has quit IRC | 07:56 | |
| *** salv-orlando has joined #openstack-infra | 07:56 | |
| *** fhubik has joined #openstack-infra | 07:56 | |
| *** kengo_sa_ has joined #openstack-infra | 08:00 | |
| *** jamielennox is now known as jamielennox|away | 08:01 | |
| *** gildub has quit IRC | 08:02 | |
| *** yaume has joined #openstack-infra | 08:02 | |
| *** e0ne has joined #openstack-infra | 08:02 | |
| *** kengo_sa_ has quit IRC | 08:05 | |
| *** jed56 has joined #openstack-infra | 08:06 | |
| mrmartin | yolanda, morning | 08:07 |
| yolanda | hi | 08:08 |
| *** shashank_hegde has quit IRC | 08:08 | |
| *** pblaho has joined #openstack-infra | 08:08 | |
| mrmartin | may i ask you to switch the DNS records of groups.o.o to the new instance? | 08:09 |
| yolanda | sure, everything is fine? | 08:09 |
| mrmartin | the puppet run is ok now, I've finished with the backup / restore | 08:09 |
| mrmartin | yeap | 08:09 |
| yolanda | ok will do | 08:09 |
| mrmartin | and I also added the bup ssh keys from old instance | 08:09 |
| mrmartin | thnx | 08:09 |
| *** fawadkhaliq has quit IRC | 08:10 | |
| mrmartin | we need to check the bup keys for the groups-dev, but we can live without backup there | 08:10 |
| mrmartin | so it is not mission critical | 08:10 |
| yolanda | mrmartin, ok, first i'll change the dns entries and then i can help you with the info you need from bup | 08:11 |
| *** daemontool_ has joined #openstack-infra | 08:13 | |
| *** jaosorior has joined #openstack-infra | 08:14 | |
| mrmartin | sure, this one: 23.253.230.177 | 08:17 |
| *** daemontool has quit IRC | 08:17 | |
| *** hashar has joined #openstack-infra | 08:17 | |
| *** rossella_s has joined #openstack-infra | 08:18 | |
| *** wfoster_ is now known as wfoster | 08:19 | |
| *** jistr has joined #openstack-infra | 08:19 | |
| *** zeih has quit IRC | 08:21 | |
| *** thorst has joined #openstack-infra | 08:25 | |
| yolanda | mrmartin, so dns updated on groups.openstack.org, and i also removed old groups-dev | 08:29 |
| yolanda | we can let some time for old groups instance to live, and then delete as well | 08:30 |
| mrmartin | yeah, give a few days for old groups.o.o | 08:30 |
| mrmartin | the next one will be the openstackid-dev.o.o, and if it works well, then openstackid.org, I think it is better to upgrade the production openstackid.org to trusty after the summit. | 08:31 |
| *** nadya has joined #openstack-infra | 08:31 | |
| *** thorst has quit IRC | 08:31 | |
| yolanda | yep, makes sense | 08:32 |
| *** achanda has joined #openstack-infra | 08:32 | |
| *** zeih has joined #openstack-infra | 08:33 | |
| *** achanda has quit IRC | 08:36 | |
| *** rpodolyaka_afk is now known as rpodolyaka | 08:36 | |
| *** lezbar has joined #openstack-infra | 08:38 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 08:38 |
| *** sfinucan has joined #openstack-infra | 08:38 | |
| *** lezbar has quit IRC | 08:38 | |
| *** lezbar has joined #openstack-infra | 08:39 | |
| *** derekh has joined #openstack-infra | 08:41 | |
| *** sfinucan has quit IRC | 08:41 | |
| *** sflanigan has quit IRC | 08:44 | |
| *** ggillies has quit IRC | 08:44 | |
| *** sflanigan has joined #openstack-infra | 08:44 | |
| *** sflanigan has joined #openstack-infra | 08:44 | |
| *** ggillies has joined #openstack-infra | 08:45 | |
| *** sfinucan has joined #openstack-infra | 08:48 | |
| *** amotoki has quit IRC | 08:49 | |
| *** amotoki has joined #openstack-infra | 08:49 | |
| *** cdent has joined #openstack-infra | 08:51 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: tripleo.sh: check if stack-delete supports -y option https://review.openstack.org/300976 | 08:51 |
| *** dtantsur|afk is now known as dtantsur | 08:51 | |
| *** jaosorior has quit IRC | 08:57 | |
| *** esikachev has joined #openstack-infra | 08:58 | |
| *** maishsk has joined #openstack-infra | 08:59 | |
| *** _degorenko|afk is now known as degorenko | 08:59 | |
| *** wznoinsk_ has joined #openstack-infra | 09:03 | |
| *** wznoinsk_ is now known as wznoinsk | 09:04 | |
| *** jaosorior has joined #openstack-infra | 09:05 | |
| *** claudiub has joined #openstack-infra | 09:06 | |
| *** electrofelix has joined #openstack-infra | 09:06 | |
| wznoinsk | hi infra | 09:09 |
| wznoinsk | what's the rationale behind always have 8gb of swapfile in devstack gate's vm? | 09:10 |
| wznoinsk | ... over configurable/parametrized amount? | 09:11 |
| *** sputnik13 has joined #openstack-infra | 09:13 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: tripleo.sh: fix stack-delete when -y option isn't supported yet https://review.openstack.org/300976 | 09:13 |
| *** sambetts|afk is now known as sambetts | 09:14 | |
| wznoinsk | I guess the RAM limit imposed at kernel CMD line is why we have it at 8gb but I'd rather have it no/smaller swapfile in my environment - but I'd like to hear your side first | 09:14 |
| *** skolekonov|away is now known as skolekonov | 09:15 | |
| *** amotoki has quit IRC | 09:17 | |
| *** amotoki has joined #openstack-infra | 09:18 | |
| *** sputnik13 has quit IRC | 09:19 | |
| *** jordanP has joined #openstack-infra | 09:20 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: tripleo.sh: fix stack-delete when -y option isn't supported yet https://review.openstack.org/300976 | 09:21 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: ironic-inspector: make the discovery job voting https://review.openstack.org/300989 | 09:27 |
| dtantsur | aarefiev, ^^^ | 09:27 |
| *** exploreshaifali has joined #openstack-infra | 09:28 | |
| jhesketh | AJaeger: please see my replies to https://review.openstack.org/#/c/292676/3 and https://review.openstack.org/#/c/292675/2 | 09:29 |
| openstackgerrit | Julien Danjou proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config https://review.openstack.org/296061 | 09:29 |
| *** thorst has joined #openstack-infra | 09:30 | |
| *** roxanaghe has joined #openstack-infra | 09:32 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: tripleo.sh: refactor stack-delete -y https://review.openstack.org/300976 | 09:33 |
| *** zeih has quit IRC | 09:33 | |
| AJaeger | jhesketh: why do it this way? | 09:36 |
| *** roxanaghe has quit IRC | 09:36 | |
| AJaeger | You can publish directly to the subdirectory... | 09:37 |
| *** thorst has quit IRC | 09:37 | |
| jhesketh | AJaeger: the election site to begin with is very basic (ie just the rendered sphinx) but there may be scope in the future to change it.. this will make switching over to its own vhost easier | 09:39 |
| jhesketh | AJaeger: it also means if the governance repo somehow creates an election folder/file it won't be overwritten | 09:39 |
| jhesketh | AJaeger: it just seemed neater when I was doing it, but I agree it isn't necessary | 09:39 |
| *** tosky has joined #openstack-infra | 09:39 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add post job for building and uploading tinyipa https://review.openstack.org/299485 | 09:42 |
| *** links has quit IRC | 09:53 | |
| *** sfinucan has quit IRC | 09:53 | |
| *** sfinucan has joined #openstack-infra | 09:54 | |
| *** links has joined #openstack-infra | 09:54 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: ironic-inspector: make the discovery job voting https://review.openstack.org/300989 | 09:57 |
| *** ociuhandu has quit IRC | 09:59 | |
| *** vgridnev has joined #openstack-infra | 09:59 | |
| AJaeger | jhesketh: Ok, understood now what you do. Let's see whether fungi and jeblair have any preferences for 292676 and 292675, I'll remove my -1. | 09:59 |
| jhesketh | AJaeger: cool, thanks | 10:00 |
| *** maishsk has quit IRC | 10:00 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Only reload layout on HUP https://review.openstack.org/281166 | 10:01 |
| *** SotK has quit IRC | 10:01 | |
| *** amrith is now known as _amrith_ | 10:02 | |
| *** maishsk has joined #openstack-infra | 10:02 | |
| jhesketh | jeblair: I believe ^ may help with zuul's current memory issue. I haven't got enough data to say with certainty, but I believe it's a useful fix either way. | 10:02 |
| *** yamamoto has quit IRC | 10:03 | |
| *** SotK has joined #openstack-infra | 10:03 | |
| *** yamamoto has joined #openstack-infra | 10:04 | |
| *** achanda has joined #openstack-infra | 10:05 | |
| *** achanda has quit IRC | 10:11 | |
| *** sdague has joined #openstack-infra | 10:12 | |
| *** zeih has joined #openstack-infra | 10:12 | |
| *** thorst has joined #openstack-infra | 10:14 | |
| *** yamamoto has quit IRC | 10:16 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Upgrade groups.o.o to trusty https://review.openstack.org/301011 | 10:16 |
| *** yamamoto has joined #openstack-infra | 10:17 | |
| *** thorst has quit IRC | 10:21 | |
| *** yamamoto has quit IRC | 10:26 | |
| *** zeih has quit IRC | 10:27 | |
| *** salv-orl_ has joined #openstack-infra | 10:30 | |
| *** salv-orlando has quit IRC | 10:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Add bareon-image-elements project https://review.openstack.org/295167 | 10:33 |
| rm_work | anyone here happen to be familiar with running ansible playbooks from within python? not relevant to channel, just curious since I figure probably a lot of people here use ansible and I'm having trouble finding docs (or confirming whether anyone actually DOES this, maybe I am crazy) | 10:34 |
| *** mattt has left #openstack-infra | 10:35 | |
| *** mikelk has quit IRC | 10:38 | |
| *** ociuhandu has joined #openstack-infra | 10:40 | |
| *** vgridnev has quit IRC | 10:41 | |
| *** vgridnev has joined #openstack-infra | 10:42 | |
| *** jlanoux has quit IRC | 10:46 | |
| odyssey4me | rm_work pabelanger or mordred may have tried - I vaguely recall seeing some chatter about it some time ago... doing it for v1.9x was not a happy place... maybe with v2.x it's better | 10:47 |
| *** openstackgerrit has quit IRC | 10:48 | |
| *** openstackgerrit has joined #openstack-infra | 10:48 | |
| rm_work | hmm yeah there was like one or two guides total for 1.x | 10:48 |
| rm_work | I'm trying with 2.x and it seems a bit ugly right now, but i figure it's because I don't know what i'm doing :P and there are ZERO examples, else my google-fu is weak | 10:49 |
| rm_work | thanks odyssey4me, i'll poke at them later mayhaps | 10:49 |
| odyssey4me | rm_work yeah, Ansible's still a young project and it's primarily catering for CLI consumers at this point | 10:51 |
| rm_work | yeah | 10:51 |
| rm_work | I'm actually wondering if it's realistic to propose it for inclusion in global-reqs :/ | 10:51 |
| rm_work | we'd like to hook an ansible playbook as part of our taskflow pipeline | 10:52 |
| *** zeih has joined #openstack-infra | 10:54 | |
| *** exploreshaifali has quit IRC | 10:55 | |
| *** nmagnezi has quit IRC | 10:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Request for new repo - python-k8sclient https://review.openstack.org/300729 | 10:58 |
| *** swat30 has quit IRC | 10:58 | |
| *** acruz has joined #openstack-infra | 11:01 | |
| *** maishsk has quit IRC | 11:01 | |
| *** hichihara has quit IRC | 11:02 | |
| *** fhubik has quit IRC | 11:02 | |
| *** arxcruz has quit IRC | 11:02 | |
| rossella_s | AJaeger, hi Adrenas | 11:09 |
| rossella_s | Andreas | 11:09 |
| *** nmagnezi has joined #openstack-infra | 11:10 | |
| *** krotscheck_vaca is now known as krotscheck | 11:13 | |
| *** yamamoto has joined #openstack-infra | 11:14 | |
| *** maishsk has joined #openstack-infra | 11:14 | |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/reviewday: Add a link for Neutron Gerrit dashboard https://review.openstack.org/298779 | 11:15 |
| *** yamamoto_ has joined #openstack-infra | 11:15 | |
| *** maishsk has quit IRC | 11:15 | |
| *** chem has joined #openstack-infra | 11:18 | |
| *** yamamoto has quit IRC | 11:19 | |
| *** roxanaghe has joined #openstack-infra | 11:20 | |
| *** exploreshaifali has joined #openstack-infra | 11:24 | |
| *** roxanaghe has quit IRC | 11:25 | |
| openstackgerrit | Julien Danjou proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config https://review.openstack.org/296061 | 11:27 |
| *** ldnunes has joined #openstack-infra | 11:28 | |
| *** ldnunes has quit IRC | 11:32 | |
| *** ldnunes has joined #openstack-infra | 11:33 | |
| *** weshay has quit IRC | 11:33 | |
| *** sdake has joined #openstack-infra | 11:34 | |
| *** zz_dimtruck is now known as dimtruck | 11:34 | |
| *** sdake_ has joined #openstack-infra | 11:36 | |
| *** gordc has joined #openstack-infra | 11:36 | |
| *** jtomasek_ has joined #openstack-infra | 11:36 | |
| *** electrofelix has quit IRC | 11:37 | |
| *** sdake has quit IRC | 11:38 | |
| *** claudiub has quit IRC | 11:39 | |
| *** thorst has joined #openstack-infra | 11:41 | |
| *** jaypipes has joined #openstack-infra | 11:42 | |
| *** baoli has joined #openstack-infra | 11:42 | |
| *** _amrith_ is now known as amrith | 11:44 | |
| *** baoli_ has joined #openstack-infra | 11:44 | |
| *** jlanoux has joined #openstack-infra | 11:45 | |
| *** claudiub has joined #openstack-infra | 11:45 | |
| *** fhubik has joined #openstack-infra | 11:47 | |
| *** baoli has quit IRC | 11:48 | |
| *** jtomasek_ has quit IRC | 11:50 | |
| *** maishsk has joined #openstack-infra | 11:51 | |
| *** Hal has joined #openstack-infra | 11:52 | |
| *** Hal is now known as Guest48503 | 11:53 | |
| *** marcusvrn_ has joined #openstack-infra | 11:56 | |
| *** cdent has left #openstack-infra | 11:56 | |
| *** jordanP has quit IRC | 12:00 | |
| *** zeih has quit IRC | 12:02 | |
| dims | AJaeger : sdague : is initial import of a repo from github a separate step after the repo is created? (python-k8sclient seems empty) | 12:02 |
| *** HeOS has quit IRC | 12:03 | |
| *** HeOS has joined #openstack-infra | 12:03 | |
| *** amrith is now known as _amrith_ | 12:05 | |
| *** keedya has joined #openstack-infra | 12:06 | |
| *** EricGonczer_ has joined #openstack-infra | 12:07 | |
| sdague | dims: it should import | 12:07 |
| *** kengo_sakai has quit IRC | 12:08 | |
| *** nmagnezi_ has joined #openstack-infra | 12:08 | |
| *** achanda has joined #openstack-infra | 12:09 | |
| *** esikachev has quit IRC | 12:09 | |
| dims | sdague : ack, thanks, i'll see if i can find logs why it failed | 12:09 |
| *** kengo_sakai has joined #openstack-infra | 12:09 | |
| *** weshay has joined #openstack-infra | 12:11 | |
| *** dprince has joined #openstack-infra | 12:11 | |
| *** nmagnezi has quit IRC | 12:12 | |
| *** zeih has joined #openstack-infra | 12:13 | |
| *** erlon has joined #openstack-infra | 12:13 | |
| *** kengo_sakai has quit IRC | 12:13 | |
| *** achanda has quit IRC | 12:14 | |
| *** keedya has quit IRC | 12:14 | |
| *** keedya has joined #openstack-infra | 12:14 | |
| *** markvoelker has joined #openstack-infra | 12:15 | |
| *** _amrith_ is now known as amrith | 12:16 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new repos for fuel plugins https://review.openstack.org/301080 | 12:19 |
| *** eharney has quit IRC | 12:21 | |
| tosky | is there a reason why there are two accounts with my name, same email, and so people can't add me to reviews? (ltoscano at redhat.com) | 12:21 |
| *** Goneri has joined #openstack-infra | 12:21 | |
| tosky | I remember it started happening some time ago after some outage with the launchpad authentication | 12:21 |
| *** pradk has quit IRC | 12:21 | |
| tosky | so I guess the real question is: can it be fixed? | 12:22 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new repos for fuel plugins https://review.openstack.org/301080 | 12:23 |
| *** edmondsw has joined #openstack-infra | 12:24 | |
| *** rfolco has joined #openstack-infra | 12:25 | |
| *** EricGonczer_ has quit IRC | 12:26 | |
| *** maishsk has quit IRC | 12:27 | |
| *** pradk has joined #openstack-infra | 12:28 | |
| vgridnev | AJaeger, what do you think about tosky question? I'm trying to add him to the group, but I failed | 12:28 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Remove redundant 'code' https://review.openstack.org/301086 | 12:30 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh https://review.openstack.org/301087 | 12:30 |
| *** sdake_ has quit IRC | 12:30 | |
| *** maishsk has joined #openstack-infra | 12:31 | |
| *** salv-orl_ has quit IRC | 12:33 | |
| *** sdake has joined #openstack-infra | 12:34 | |
| *** salv-orlando has joined #openstack-infra | 12:34 | |
| *** rodrigods has joined #openstack-infra | 12:36 | |
| rodrigods | hi... can someone give some feedback about this patch https://review.openstack.org/#/c/298696/ ? | 12:37 |
| rodrigods | the change that adds the tempest plugin in keystone was merged already | 12:37 |
| *** rlandy has joined #openstack-infra | 12:38 | |
| *** salv-orlando has quit IRC | 12:38 | |
| marcusvrn_ | patrickeast: hi! do you know how to get the lost queue from CI? I mean, when CI machine get disconnected/turned off, all builds that were in the queue are lost... There is some tool to retrieve those builds or will I have to develop it? | 12:38 |
| *** bswartz has joined #openstack-infra | 12:40 | |
| *** xyang1 has joined #openstack-infra | 12:40 | |
| *** kgiusti has joined #openstack-infra | 12:41 | |
| openstackgerrit | Richard Theis proposed openstack/requirements: Bump openstacksdk version to 0.8.4 https://review.openstack.org/301094 | 12:43 |
| AJaeger | dims: it should import, if this failed, ask one of the infra-roots, please | 12:46 |
| dims | AJaeger : It took a while, looks like it imported fine. thanks AJaeger and sdague | 12:47 |
| AJaeger | vgridnev, tosky : Please talk with one of the admin, I don't have those permissions. This can happen if you change your launchpad email address AFAIK | 12:47 |
| tosky | AJaeger: I never did that - who are the admins? | 12:47 |
| AJaeger | dims: changes to project-config in general need until next puppet run to take effect - so always wait an hour ;) | 12:48 |
| AJaeger | tosky: infra-root will address them here;) | 12:48 |
| dims | AJaeger : ah i see. thanks | 12:48 |
| *** zul has quit IRC | 12:48 | |
| dims | AJaeger : wasn't trying to rush it. just trying to understand the flow :) | 12:48 |
| *** zul has joined #openstack-infra | 12:50 | |
| AJaeger | team, I have a few errands to run today, won't be online further today. | 12:50 |
| *** kzaitsev_mb has quit IRC | 12:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:52 | |
| *** amrith is now known as _amrith_ | 12:55 | |
| dgonzalez | hi sdague: if you have time, could you have a look at https://review.openstack.org/#/c/300369/? I'm trying to set the user/project domain id for devstack in openrc, but some grenade tests keep failing and i can't figure out why | 12:57 |
| *** _amrith_ is now known as amrith | 12:57 | |
| trash | SpamapS: https://review.openstack.org/#/c/280178/13 remains broken, how do we continue with that? :) | 13:00 |
| sdague | dgonzalez: http://logs.openstack.org/69/300369/3/check/gate-grenade-dsvm/6e48097/logs/grenade.sh.txt.gz#_2016-04-04_07_11_11_593 is the failure | 13:01 |
| openstackgerrit | Rodion Promyshlennikov proposed openstack/requirements: Bump alembic version to 0.8.4 https://review.openstack.org/301108 | 13:02 |
| dgonzalez | sdague: yes, but i do not understand how the two env variables i added to the openrc can cause this failure... any idea? | 13:03 |
| sdague | dgonzalez: I don't atm, it appears to be changing how openstack client is communicating with services | 13:04 |
| sdague | it is probably worth asking a keystone or osc person | 13:04 |
| odyssey4me | hmm, it would seem that zuul is dropping events? We had a few in the queue and they seem to have gotten lost. rechecks and rebases aren't making a difference. | 13:05 |
| dgonzalez | sdague: ok thanks, i will ask them | 13:06 |
| sdague | odyssey4me: do you have specific reviews? it's helpful for people tracking them down | 13:08 |
| *** roxanaghe has joined #openstack-infra | 13:08 | |
| *** Guest48503 has quit IRC | 13:09 | |
| *** Hal has joined #openstack-infra | 13:10 | |
| *** Hal is now known as Guest12623 | 13:10 | |
| *** salv-orlando has joined #openstack-infra | 13:11 | |
| *** smarcet has joined #openstack-infra | 13:11 | |
| amrith | hello, what is the current plan for opening newton for tagging releases? I would like to restore https://review.openstack.org/#/c/292650/ for newton and get a new trove client tagged and on pypi. Thx! | 13:12 |
| *** dtantsur is now known as dtantsur|brb | 13:12 | |
| *** roxanaghe has quit IRC | 13:13 | |
| *** sfinucan_ has joined #openstack-infra | 13:14 | |
| *** jaosorior has quit IRC | 13:14 | |
| *** jaosorior has joined #openstack-infra | 13:14 | |
| smarcet | fungi: morning, we are good to move trusty to production whenever u want | 13:15 |
| *** matt-borland has joined #openstack-infra | 13:15 | |
| *** nwkarsten has joined #openstack-infra | 13:16 | |
| *** sfinucan has quit IRC | 13:17 | |
| *** jaosorior has quit IRC | 13:17 | |
| *** jaosorior has joined #openstack-infra | 13:18 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn-bagpipe jobs https://review.openstack.org/258522 | 13:18 |
| *** [1]Thelo has joined #openstack-infra | 13:20 | |
| *** nelsnels_ has quit IRC | 13:21 | |
| *** _ari_ has joined #openstack-infra | 13:21 | |
| *** links has quit IRC | 13:21 | |
| *** akshai has joined #openstack-infra | 13:21 | |
| odyssey4me | sdague ah, it appears that either my browser was acting up - or everything's back to normal | 13:22 |
| *** Thelo has quit IRC | 13:22 | |
| *** [1]Thelo is now known as Thelo | 13:22 | |
| pabelanger | infra-root: I've just deleted 9292911 in nodepool, as it was hung for 6 hours trying to upload to swift | 13:23 |
| *** sfinucan__ has joined #openstack-infra | 13:23 | |
| *** tiswanso has joined #openstack-infra | 13:23 | |
| *** mtanino has joined #openstack-infra | 13:24 | |
| openstackgerrit | Russell Bryant proposed openstack/requirements: Add ovs with Python 3 support. https://review.openstack.org/291775 | 13:25 |
| openstackgerrit | Russell Bryant proposed openstack/requirements: Update ovs to 2.5.0. https://review.openstack.org/291769 | 13:25 |
| *** amitgandhinz has joined #openstack-infra | 13:25 | |
| *** sfinucan_ has quit IRC | 13:25 | |
| Shrews | trash: https://review.openstack.org/298789 should be the fix | 13:27 |
| *** dalgaaf has joined #openstack-infra | 13:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix consent screen OAUTH2.0 flow * updated css on ui consent screen to better mobile UX * added autocomplete off to login form https://review.openstack.org/301125 | 13:27 |
| *** jsavak has joined #openstack-infra | 13:28 | |
| trash | Shrews: Ah | 13:29 |
| *** fhubik has quit IRC | 13:30 | |
| *** esker has quit IRC | 13:30 | |
| trash | Shrews: thanks, seems to work with 1.7.0 now | 13:35 |
| EmilienM | sdague: I have the same problem as odyssey4me | 13:35 |
| EmilienM | sdague: 300786 | 13:35 |
| *** bizarrodan is now known as dansmith | 13:35 | |
| EmilienM | we tried to +2 +A again, recheck, etc. Patch is not going in any queue | 13:35 |
| *** woodster_ has joined #openstack-infra | 13:36 | |
| EmilienM | and I tried 2 browsers | 13:37 |
| *** mikelk has joined #openstack-infra | 13:38 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Move tox-bandit to check as non voting https://review.openstack.org/300564 | 13:38 |
| EmilienM | even if I push a new patchset, nothing goes in zuul | 13:39 |
| *** |-paul-| has quit IRC | 13:39 | |
| *** ddieterly has joined #openstack-infra | 13:41 | |
| *** ddieterly has quit IRC | 13:42 | |
| *** mriedem has joined #openstack-infra | 13:43 | |
| *** ddieterly has joined #openstack-infra | 13:45 | |
| *** pradk_ has joined #openstack-infra | 13:45 | |
| *** pradk has quit IRC | 13:45 | |
| *** ddieterly has quit IRC | 13:46 | |
| *** pradk_ is now known as pradk | 13:46 | |
| *** ddieterly has joined #openstack-infra | 13:46 | |
| *** eharney has joined #openstack-infra | 13:48 | |
| AJaeger | EmilienM: is there another patch that has a depends-on on that one - and does this some leat to a cyclic dependency? That would explain it.... | 13:48 |
| AJaeger | EmilienM: otherwise an admin needs to check zuul debug logs... | 13:49 |
| EmilienM | AJaeger: https://review.openstack.org/#/c/300786/ depends on https://review.openstack.org/#/c/300786/ | 13:51 |
| dansmith | I see the same behavior on new patches I have pushed | 13:52 |
| mriedem | did something blow up with openstack/requirements on stable/liberty recently? | 13:53 |
| *** mrunge has joined #openstack-infra | 13:53 | |
| *** ddieterly is now known as ddieterly[away] | 13:54 | |
| *** yamamoto_ has quit IRC | 13:56 | |
| *** yamamoto has joined #openstack-infra | 13:56 | |
| *** eharney has quit IRC | 13:57 | |
| YorikSar | Hello, #openstack-infra. | 13:57 |
| YorikSar | I have a CR that is being ignored by Zuul. | 13:57 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again https://review.openstack.org/288915 | 13:57 |
| YorikSar | No new patchsets appear on Zuul status page. | 13:57 |
| *** ddieterly[away] is now known as ddieterly | 13:57 | |
| YorikSar | The CR is https://review.openstack.org/301082 | 13:57 |
| *** aysyd has joined #openstack-infra | 13:58 | |
| mattymo | I'm having issues getting https://review.openstack.org/#/c/299943/ to merge. It has wf+1 for several hours, gate jobs succeeded. I tried recheck and wf-0,wf+1, but no luck | 13:58 |
| *** dtantsur|brb is now known as dtantsur | 13:58 | |
| mattymo | similar to the esteemed YorikSar's issue | 13:58 |
| YorikSar | Have you tried publishing new patchset? | 13:58 |
| *** aysyd has quit IRC | 13:59 | |
| fungi | mrmartin: openstackid-dev is already running trusty (after some additional patches), and smarcet has given the go-ahead for us to update production as soon as we're ready | 13:59 |
| mattymo | YorikSar, not yet | 13:59 |
| *** aysyd has joined #openstack-infra | 13:59 | |
| *** eharney has joined #openstack-infra | 14:00 | |
| odyssey4me | yeah, ok - so here's one that just got ignored - https://review.openstack.org/300936 it has no deps on any other changes | 14:00 |
| fungi | wznoinsk: i think the swap configuration in devstack-gate is probably something we'd be open to letting jobs adjust on their own (in theory they already can, just swapoff, make different device, swapon again, but could in theory make the setup function reusable and parameterized instead) | 14:00 |
| mattymo | odyssey4me, at least I'm not alone | 14:01 |
| *** isaacb has quit IRC | 14:01 | |
| YorikSar | Yeah, odyssey4me, welcome to the club :) | 14:01 |
| *** fzhadaev has joined #openstack-infra | 14:01 | |
| mrmartin | fungi: good to know that | 14:01 |
| *** dimtruck is now known as zz_dimtruck | 14:02 | |
| *** ddieterly is now known as ddieterly[away] | 14:03 | |
| *** nelsnelson has joined #openstack-infra | 14:05 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 14:05 |
| yolanda | infra-root, i just received a confirmation that all infra-cloud servers in east/west have been setup properly, and we are waiting for network team to finish the config | 14:06 |
| fungi | dims: i'll look into python-k8sclient shortly if nobody has yet | 14:07 |
| fungi | dims: oh, good, i guess it worked out after all | 14:07 |
| jpmaxman | mrmartin fungi yes we have a ux improvement for the login screen that is holding up the app being published - so we kind of need this to move super quick | 14:07 |
| jpmaxman | Apple tends to take their time, but the quicker we can turn it around to them the better | 14:08 |
| dims | fungi : y, i was not sure where/when the imports would happen, it finally did :) | 14:08 |
| *** abregman has quit IRC | 14:08 | |
| dims | fungi : thanks! looks like the first review has the CI run jobs as well. So i am good (thanks to sdague and AJaeger as well) | 14:08 |
| EmilienM | is someone from infra-root investigating why zuul isn't working well at this time? | 14:08 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:09 | |
| *** knikolla has joined #openstack-infra | 14:09 | |
| *** burgerk has joined #openstack-infra | 14:09 | |
| fungi | tosky: yes, chances are you have more than one gerrit account with different launchpad openids. i'll look in a bit to see what their respective states are and try to work out a good way to condense them/deactivate one | 14:09 |
| *** aysyd has quit IRC | 14:10 | |
| tosky | fungi: thanks | 14:10 |
| *** xarses has quit IRC | 14:10 | |
| fungi | EmilienM: i'm in a meeting and also just finished catching up on scrollback. will try to look into it as quickly as i can | 14:10 |
| *** jamesmcarthur has joined #openstack-infra | 14:10 | |
| *** aysyd has joined #openstack-infra | 14:10 | |
| EmilienM | cool, thanks | 14:10 |
| *** ddieterly[away] is now known as ddieterly | 14:11 | |
| fungi | doesn't appear to be a problem with stuck stream-events worker threads on gerrit at least | 14:11 |
| *** achanda has joined #openstack-infra | 14:12 | |
| yolanda | i see entries in BUILD in nodepool log | 14:12 |
| fungi | i see it adding/updating changes | 14:12 |
| fungi | i'll check into one of the specific examples | 14:12 |
| yolanda | fungi, i can help with tosky's problem | 14:12 |
| fungi | thanks yolanda | 14:13 |
| tosky | thanks both :) | 14:13 |
| *** doug-fish has joined #openstack-infra | 14:15 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 14:15 |
| *** ajmiller has joined #openstack-infra | 14:16 | |
| *** achanda has quit IRC | 14:17 | |
| *** oanson has quit IRC | 14:17 | |
| *** xyang1 has quit IRC | 14:17 | |
| *** xyang1_ has joined #openstack-infra | 14:17 | |
| fungi | EmilienM: on your 300786 example, the last thing zuul knew about that change is that it reported job results at 13:13 but it doesn't even seem to have seen the event resulting from its own comment, so something does seem to be amiss here | 14:18 |
| *** krtaylor has quit IRC | 14:18 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/irc-meetings: Move Heat meeting https://review.openstack.org/301151 | 14:18 |
| *** regXboi has joined #openstack-infra | 14:20 | |
| fungi | EmilienM: now it saw a new workflow +1 event for 300786 at 14:18:36 | 14:20 |
| EmilienM | right but nothing in gate gueue | 14:21 |
| odyssey4me | fungi it would seem that some of the jobs don't show in the zuul status page, but later post results | 14:21 |
| odyssey4me | so it might have nothing to do with the event queue and everything to do with whatever posts data to the status page | 14:22 |
| fungi | wait, this is a verify +1 not a workflow +1 | 14:22 |
| *** rbrndt_ has joined #openstack-infra | 14:22 | |
| *** esker has joined #openstack-infra | 14:22 | |
| *** claudiub has quit IRC | 14:22 | |
| fungi | timeline seems way off | 14:23 |
| *** jamesmcarthur has quit IRC | 14:23 | |
| fungi | 2016-04-04 14:18:36,495 DEBUG zuul.DependentPipelineManager: Event <TriggerEvent comment-added openstack/puppet-tempest master 300786,1 Verified:1> for change <Change 0x7ff142750d50 300786,1> matched <EventFilter types: comment-added ignore_deletes: True event_approvals: verified:1 username_filters: jenkins> in pipeline <DependentPipelineManager gate> | 14:23 |
| *** esker has quit IRC | 14:23 | |
| *** mrmartin has quit IRC | 14:24 | |
| *** jamesmcarthur has joined #openstack-infra | 14:24 | |
| fungi | i think that's the verified +1 from the comment it left at 13:13 | 14:24 |
| *** esker has joined #openstack-infra | 14:24 | |
| *** hieulq has joined #openstack-infra | 14:24 | |
| openstackgerrit | Pradeep Kilambi proposed openstack-infra/tripleo-ci: Include Gnocchi endpoints in enable-tls https://review.openstack.org/301157 | 14:25 |
| *** mrmartin has joined #openstack-infra | 14:25 | |
| fungi | i'm finding tracebacks in the debug log which are infinite loops between _getChange() and _updateChange() | 14:26 |
| *** edmondsw has quit IRC | 14:27 | |
| openstackgerrit | Nikolas Hermanns proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn-bagpipe jobs https://review.openstack.org/258522 | 14:28 |
| *** isaacb has joined #openstack-infra | 14:29 | |
| fungi | RuntimeError: maximum recursion depth exceeded | 14:29 |
| *** exploreshaifali has quit IRC | 14:30 | |
| *** zz_dimtruck is now known as dimtruck | 14:30 | |
| yolanda | tosky, can you verify if your account issue is fixed now? | 14:30 |
| tosky | yolanda: let me check | 14:30 |
| tosky | vgridnev: ^^ | 14:30 |
| *** achanda has joined #openstack-infra | 14:31 | |
| EmilienM | fungi: same for new patches, I just sent 301162 and nothing in Zuul | 14:31 |
| vgridnev | yolanda, looks like issue is fixed | 14:31 |
| vgridnev | thanks | 14:31 |
| yolanda | nice | 14:31 |
| *** kushal has quit IRC | 14:31 | |
| vgridnev | tosky, ^^ | 14:32 |
| fungi | EmilienM: yeah, i think zuul is getting them but is waaaaay backed up on the event stream because of these recursive calls | 14:32 |
| *** nadya has quit IRC | 14:32 | |
| tosky | yolanda: thanks! | 14:32 |
| odyssey4me | EmilienM fungi yeah, same with https://review.openstack.org/301134 - nothing | 14:32 |
| fungi | which would explain the lengthy delays between log actual events in gerrit and log messages | 14:32 |
| *** piet has joined #openstack-infra | 14:32 | |
| *** jamesmcarthur has quit IRC | 14:33 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 14:33 |
| *** Jeffrey4l__ has quit IRC | 14:33 | |
| jeblair | fungi: perhaps a bug introduced in the temporary foreign projects revert | 14:34 |
| fungi | this is still the same build of zuul with "Dependencies from undefined projects" reverted, so there might be a bug in that revert | 14:34 |
| fungi | jeblair: indeed! | 14:34 |
| *** Thelo has quit IRC | 14:34 | |
| *** amotoki has quit IRC | 14:35 | |
| fungi | possibly exposed by yolanda's gerrit restart around 07:15z? but the exceptions aren't showing up until after 10:30z | 14:36 |
| *** tonytan4ever has joined #openstack-infra | 14:36 | |
| fungi | first change to hit it was 300475 | 14:37 |
| *** esikachev has joined #openstack-infra | 14:37 | |
| fungi | which has a depends-on loop | 14:38 |
| fungi | so maybe that's the trigger | 14:38 |
| *** andymaier has joined #openstack-infra | 14:38 | |
| *** kushal has joined #openstack-infra | 14:38 | |
| *** achanda has quit IRC | 14:38 | |
| fungi | jeblair: what are your thoughts at this point, reupgrade zuul to normal master branch tip and perform a restart? | 14:39 |
| jeblair | fungi: without a manual dequeue command (wasn't someone writing that?) i don't think this instance can recover. | 14:39 |
| *** Thelo has joined #openstack-infra | 14:39 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 14:39 |
| fungi | jeblair: on a positive note, memory usage seems to have been far more stable with the revert in http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 14:40 |
| jeblair | fungi: yeah. let me see if i can spot the error quickly before we restart | 14:40 |
| fungi | okay, holding pattern | 14:40 |
| jeblair | wow, that's actually a complex cycle | 14:42 |
| jeblair | there are 4 different _updateChange calls | 14:42 |
| jeblair | fungi: oh, i think this is all happening before a change is being enqueued so this might be recoverable -- try breaking the dependency cycle in gerrit by modifying the commit messages as needed. but remember the structure in case we need it for debugging later. | 14:47 |
| *** andymaier has quit IRC | 14:47 | |
| odyssey4me | jeblair so one particular change is holding up the queue somehow? | 14:48 |
| *** burgerk has quit IRC | 14:49 | |
| fungi | jeblair: yeah, i'll give that a shot | 14:50 |
| *** burgerk has joined #openstack-infra | 14:50 | |
| *** asselin has joined #openstack-infra | 14:50 | |
| *** maishsk has quit IRC | 14:50 | |
| fungi | for the record, 300475 in fuel-library has a depends-on to 299838 in fuel-noop-fixtures which has a depends-on back to 300475 | 14:51 |
| fungi | i'm going to remove the depends-on header in the 300475 commit message | 14:51 |
| *** annegentle has joined #openstack-infra | 14:53 | |
| fungi | okay, that particular dependency loop should be broken now | 14:56 |
| *** eharney has quit IRC | 14:56 | |
| *** mixos has joined #openstack-infra | 14:59 | |
| fungi | haven't seen another "maximum recursion depth exceeded" in the log now since 14:53:50z | 15:00 |
| *** burgerk has quit IRC | 15:00 | |
| *** rhochmuth has joined #openstack-infra | 15:01 | |
| *** mriedem is now known as mriedem_meeting | 15:01 | |
| *** mrmartin has quit IRC | 15:01 | |
| *** edmondsw has joined #openstack-infra | 15:02 | |
| *** krtaylor has joined #openstack-infra | 15:02 | |
| fungi | odd that other dependency cycles aren't triggering this. for example, we're still logging this one zuul skipped intentionally: Dependency cycle detected: 300475 in [u'301107', u'300475', u'299838'] | 15:02 |
| *** mrmartin has joined #openstack-infra | 15:03 | |
| jeblair | odyssey4me: yes, due to a bug | 15:03 |
| jeblair | fungi: yeah, i think this may be a legit zuul bug exposed by a complex dependency graph. | 15:03 |
| odyssey4me | jeblair ah, thanks for figuring it out - it looks like we're seeing a ton of jobs jump into the queue now | 15:03 |
| fungi | and possibly unrelated to the revert | 15:03 |
| jeblair | fungi: yes, at this point i think it's likely unrelated to the revert | 15:03 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 15:04 | |
| *** mrmartin has quit IRC | 15:04 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:04 | |
| *** EricGonczer_ has joined #openstack-infra | 15:04 | |
| *** jsavak has quit IRC | 15:04 | |
| *** mriedem_meeting is now known as mriedem | 15:05 | |
| YorikSar | fungi, jeblair: Thanks! | 15:06 |
| YorikSar | Now we have lots of changes in Zuul, but at least they should go through eventually. | 15:07 |
| *** cody-somerville has joined #openstack-infra | 15:07 | |
| *** cody-somerville has quit IRC | 15:07 | |
| *** cody-somerville has joined #openstack-infra | 15:07 | |
| *** jsavak has joined #openstack-infra | 15:08 | |
| YorikSar | I guess it would be great if CRs didn't affect each other in Zuul... | 15:08 |
| openstackgerrit | Adam Reznechek proposed openstack-infra/system-config: Add openstack-powervm to meetbot and statusbot channels https://review.openstack.org/264307 | 15:09 |
| fungi | YorikSar: well, normally zuul silently skips testing on changes which are involved in dependency loops... this particular one seemed not to be detected by the normal dependency detection though | 15:09 |
| *** ddieterly is now known as ddieterly[away] | 15:10 | |
| *** acruz has quit IRC | 15:10 | |
| YorikSar | fungi: I mean like make failure domain for such issues very small. | 15:10 |
| fungi | YorikSar: i have no idea what that means in this context | 15:10 |
| fungi | given that the reason zuul exists is to be able to test changes together, they need to affect each other | 15:11 |
| YorikSar | fungi: For example, handle every CR in a separate (green) thread so that if it fails, it doesn't affect other CRs. | 15:11 |
| *** eharney has joined #openstack-infra | 15:12 | |
| *** isaacb has quit IRC | 15:13 | |
| *** ddieterly[away] is now known as ddieterly | 15:13 | |
| anteaya | YorikSar: that is the description of the check queue | 15:15 |
| anteaya | however zuul needs to have time to get to them | 15:15 |
| anteaya | and when looping it is sort of stuck | 15:15 |
| anteaya | which I believe is what happened here | 15:15 |
| *** jamesmcarthur has joined #openstack-infra | 15:16 | |
| *** swat30 has joined #openstack-infra | 15:16 | |
| pabelanger | yolanda: excellent news | 15:17 |
| mriedem | is vexxhost == osic-cloud? | 15:18 |
| mordred | mriedem: nope | 15:18 |
| anteaya | no | 15:18 |
| anteaya | separate | 15:18 |
| mordred | mriedem: vexxhost is a public cloud based in canada | 15:18 |
| fungi | mriedem: no, vexxhost is a public cloud provider. osic is the "openstack innovation center" (a joing venture between intel and rackspace) | 15:19 |
| mordred | mriedem: osic is the OpenStack Innovaction Center cloud run by rackspace/intel | 15:19 |
| rcarrillocruz | we don't have auro, do we? | 15:19 |
| rcarrillocruz | speaking of canadians | 15:19 |
| mordred | rcarrillocruz: we do not have an account on auro | 15:19 |
| anteaya | rodrigods: vexxhost doesn't have a grafana dashboard yet? http://grafana.openstack.org/ | 15:19 |
| mordred | rcarrillocruz: although I have an auro account and do test that shade/occ works against it | 15:19 |
| mriedem | ok, trying to write a logstash query for the multitude of vexxhost job timeouts | 15:19 |
| *** yamahata has joined #openstack-infra | 15:19 | |
| rcarrillocruz | nice | 15:19 |
| mriedem | but not seeing vexxhost showing up as a node_provider | 15:19 |
| anteaya | mriedem: logstash doesn't see vexxhost? | 15:20 |
| mriedem | it does for build_node | 15:20 |
| mriedem | node_provider is apparently ymq-21 | 15:20 |
| mriedem | *ymq-1 | 15:20 |
| pabelanger | also see http://grafana.openstack.org/dashboard/db/nodepool-vexxhost | 15:21 |
| anteaya | mriedem: that is vexxhost: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n43 | 15:21 |
| openstackgerrit | Merged openstack-infra/openstackid: Fix consent screen OAUTH2.0 flow * updated css on ui consent screen to better mobile UX * added autocomplete off to login form https://review.openstack.org/301125 | 15:21 |
| mriedem | yeah, just wasn't obvious | 15:21 |
| anteaya | pabelanger: I don't see vexxhost show up in the main view: http://grafana.openstack.org/ | 15:21 |
| anteaya | mriedem: I agree, ymq-1 isn't obviously vexxhost | 15:22 |
| *** vgridnev has quit IRC | 15:22 | |
| *** vgridnev has joined #openstack-infra | 15:22 | |
| mtreinish | mriedem: it's the region name | 15:22 |
| mtreinish | I only knew that because my personal stuff is in the same region | 15:23 |
| pabelanger | anteaya: the default list of dashboard is 10, so it is a truncated list now | 15:23 |
| anteaya | mtreinish: if you look at node_provider for the rest of them they are hostname-region | 15:23 |
| *** smarcet has quit IRC | 15:24 | |
| anteaya | pabelanger: ah, is there pagination? | 15:24 |
| mriedem | mtreinish: ok, that's fine. i'll just use ymq-1. point is, 79% of build timeouts are on vexxhost | 15:24 |
| *** vgridnev has quit IRC | 15:24 | |
| mriedem | sdague: ^ | 15:24 |
| *** mixos has quit IRC | 15:24 | |
| pabelanger | anteaya: not on the default. However, is is a dashboard which we can customize. Somebody just needs to propose a new default page | 15:25 |
| anteaya | okay | 15:25 |
| anteaya | thanks | 15:25 |
| *** Thelo has quit IRC | 15:25 | |
| *** bkeller` has joined #openstack-infra | 15:25 | |
| hashar | ianw: thank you for the blog post https://www.technovelty.org/openstack/image-building-for-openstack-ci-minimal-images-big-effort.html . Quite informative, kudos! | 15:25 |
| *** mikelk has quit IRC | 15:25 | |
| *** Thelo has joined #openstack-infra | 15:26 | |
| *** akaszuba has quit IRC | 15:26 | |
| *** ddieterly is now known as ddieterly[away] | 15:26 | |
| fungi | jeblair: wondering if you get a moment whether you could quickly review 300912 (even though it's merged now)? i should have left a comment on it that i was going to get your input as the original author of that section, just making sure you don't object and any of that hadn't been omitted intentionally | 15:26 |
| anteaya | pabelanger: can we prioritize the nodepool pages on the index page for the time being? | 15:26 |
| *** vgridnev has joined #openstack-infra | 15:26 | |
| sdague | mriedem: is there a question? | 15:26 |
| fungi | jeblair: i'm planning to link to that page when replying to the internap marketing thread | 15:26 |
| *** vgridnev has quit IRC | 15:26 | |
| *** mylu has joined #openstack-infra | 15:26 | |
| mriedem | sdague: no | 15:26 |
| mriedem | i'm going to put up an e-r query for that specifically though | 15:27 |
| *** mixos has joined #openstack-infra | 15:27 | |
| mriedem | because i'm tired of digging into logs to just see build timeouts on vexxhost | 15:27 |
| sdague | mriedem: yep, that's fair | 15:27 |
| fungi | mriedem: we're still seeing slow performance on vexxhost workers? mnaser seemed to think he had worked out and solved the cause, but i guess if we can collect hostnames of more examples that will help him narrow it down further | 15:27 |
| pabelanger | anteaya: it requires somebody to add a new grafyaml template for it, so about the same amount of work to update the page | 15:27 |
| sdague | is there a query up for that | 15:27 |
| *** tbachman has joined #openstack-infra | 15:28 | |
| anteaya | okay | 15:28 |
| mordred | a logstash query will certainly help to provide hostnames | 15:28 |
| mriedem | http://tinyurl.com/zmwphpz | 15:28 |
| * fungi is concerned mriedem may be rickrolling him | 15:28 | |
| *** ddieterly[away] is now known as ddieterly | 15:28 | |
| *** abregman has joined #openstack-infra | 15:29 | |
| mordred | fungi: mriedem is always rickrolling you, even when he's not | 15:29 |
| anteaya | mriedem: I see 60 hits in the last 2 days | 15:30 |
| fungi | indeed, extending that query to the past week, it's still pretty constant | 15:30 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for build timeouts on vexxhost bug 1565850 https://review.openstack.org/301187 | 15:31 |
| *** ddieterly is now known as ddieterly[away] | 15:31 | |
| openstack | bug 1565850 in OpenStack-Gate "79% of build timeouts are on vexxhost ymq-1 nodes" [Undecided,New] https://launchpad.net/bugs/1565850 | 15:31 |
| mriedem | yeah 376 in 7 days, | 15:31 |
| mriedem | check and gate and voting jobs | 15:31 |
| anteaya | :( | 15:31 |
| anteaya | I can see your point | 15:31 |
| mriedem | the pie chart on node providers makes it pretty obvious | 15:31 |
| anteaya | I believed you before, but now I can see it | 15:31 |
| mriedem | once you remove node_provider from teh query | 15:31 |
| anteaya | pie chart? | 15:32 |
| fungi | removing the provider-specific filter from that query, it's still mostly vexxhost (the ovh hits are less likely performance related and merely because ovh provides an order of magnitude more quota) | 15:32 |
| mriedem | do a trend on node_provider | 15:32 |
| mriedem | anteaya: sorry, remove node_provider from the query, then click on node_provider on the left, do terms > pie | 15:32 |
| *** nadya has joined #openstack-infra | 15:32 | |
| mriedem | over 7 days | 15:33 |
| anteaya | mriedem: I will try that, thank you | 15:33 |
| fungi | for only being around 5% of our aggregate quota, the quantity we're seeing time out there are definitely disproportionately high | 15:33 |
| mordred | also, I _am_ confused as to why node_provider for vexxhost is showing up at ymq-1 and not vexxhost-ymq-1 | 15:34 |
| mordred | I wonder if somewhere we're splitting on - and doing it wrong | 15:34 |
| fungi | mordred: likely a regex issue due to the extra - in it | 15:34 |
| sdague | mordred: yes | 15:34 |
| *** jsavak has quit IRC | 15:35 | |
| sdague | no other cloud uses '-' in region | 15:35 |
| sdague | the vexxhost entries have 2 dashes | 15:35 |
| anteaya | mriedem: look at that! a pie chart | 15:35 |
| *** jsavak has joined #openstack-infra | 15:35 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: rename vexxhost to match existing conventions https://review.openstack.org/301191 | 15:35 |
| jeblair | yeah, this is a bug in the logstash processor | 15:35 |
| sdague | I kind of wonder if that will help | 15:35 |
| sdague | or if it uses deeper things | 15:36 |
| jeblair | it hardcodes a mere convention. it splits on '-' and assumes a certain structure | 15:36 |
| sdague | sure, but there is no reason we can't enforce that convention, right? | 15:36 |
| jeblair | sdague: no, other than we didn't. | 15:36 |
| *** fedexo has joined #openstack-infra | 15:37 | |
| jeblair | (this has been discussed before, and the possibility this happened was raised, and yet it happened) | 15:37 |
| jeblair | (sorry, possibility it could happen) | 15:37 |
| *** nadya has quit IRC | 15:37 | |
| *** mixos has quit IRC | 15:37 | |
| mordred | well, I mean, also we can't enforce anything about vexxhost's region name having a - in it | 15:37 |
| mordred | but we can fix the log processor | 15:37 |
| jeblair | yeah, i'd really prefer a robust fix | 15:38 |
| sdague | I guess the question is, are we exposing nodepools ideas of naming, or the providers ideas of naming in the final result? | 15:38 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/300845 | 15:39 |
| mordred | sdague: the providers | 15:39 |
| sdague | mordred: ah, bummer | 15:39 |
| fungi | i think nodepool's idea of naming exposes the provider's ideas of naming | 15:39 |
| jeblair | and the region name is actually 'ca-ymq-1' | 15:39 |
| sdague | too bad we don't have an abstraction there | 15:39 |
| jeblair | i don't want one | 15:39 |
| *** Swami has joined #openstack-infra | 15:40 | |
| mordred | yah. I'd really like this to explicitly be {cloud_name}-{region_name} and not something else | 15:40 |
| jeblair | the cloud is 'vexxhost' and the region is 'ca-ymq-1'. and when we debug nodepool issues, i don't want a translation table. | 15:40 |
| mordred | otherwise tracking things down gets harder | 15:40 |
| mordred | ++ | 15:40 |
| mtreinish | jeblair: oh, that thing I added to try and extract the provider. I don't know if we thought about a double - | 15:40 |
| mtreinish | I know I only tested it with the existing providers at the time | 15:40 |
| *** catherineD has left #openstack-infra | 15:40 | |
| rcarrillocruz | mordred: played with ansible-openstack-cloud-launcher yet? I expect you to break it so I can fix missing things :-) | 15:40 |
| mordred | rcarrillocruz: not yet - it's on my list | 15:41 |
| fungi | it _does_ however mean that a human needs to involve judgment since we're encoding a separator which can also appear in the fields themselves without escaping, so makes it not easily machine-parsed | 15:41 |
| rcarrillocruz | ++ | 15:41 |
| *** jistr has quit IRC | 15:41 | |
| mordred | so - how about this | 15:41 |
| mordred | how about we track 3 fields | 15:41 |
| jeblair | fungi: the *problem* is that we are *parsing* it. | 15:41 |
| mordred | nodepool_provider - nodepool_cloud and nodepool_region_name | 15:41 |
| fungi | jeblair: yep, that i agree with | 15:41 |
| mordred | they are three different pieces of information and all of them should be directly available | 15:42 |
| *** salv-orlando has quit IRC | 15:42 | |
| *** mixos has joined #openstack-infra | 15:42 | |
| rcarrillocruz | i'll try to write up an example geared to openstack infra conventions | 15:42 |
| mordred | and clearly all of them are useful | 15:42 |
| mordred | rcarrillocruz: it was immediately clear to me how it would map to infra in my brain - so good job | 15:42 |
| fungi | i don't think they're actually independently useful at the logstash level | 15:42 |
| fungi | at least for now, the most regions we have at a single provider is 3 | 15:43 |
| rcarrillocruz | indeed, as a matter of fact i started the example with 'infracloud' and 'rackspace', and with projects 'openstack-ci'/'openstack-jenkins' | 15:43 |
| fungi | it's easier to just treat the provider and region together as a single identifier and be done with it | 15:43 |
| *** Sukhdev has joined #openstack-infra | 15:43 | |
| rcarrillocruz | but decided to change it to typical ops/qa/rnd scenarios for the blogpost | 15:43 |
| mordred | fungi: I could see that too | 15:43 |
| sdague | provider + region is usually the correct error bucket | 15:43 |
| mordred | fungi: but we actually already write all three pieces of information to the node | 15:43 |
| sdague | things go wrong in a region most of them | 15:44 |
| jeblair | sdague: cloud+region == provider | 15:44 |
| sdague | the time | 15:44 |
| mordred | so it still goes back to - there is no need to parse something | 15:44 |
| clarkb | you have to parse the hostname, this was always a best effort thing to begin with | 15:44 |
| jeblair | fungi: cloud+region == provider | 15:44 |
| *** sputnik13 has joined #openstack-infra | 15:44 | |
| clarkb | I am not sure what the issue is | 15:44 |
| fungi | jeblair: yep, that's what i meant | 15:44 |
| mtreinish | mordred: there is because the log processor doesn't get the provider as a field over zmq | 15:44 |
| *** shashank_hegde has joined #openstack-infra | 15:44 | |
| mtreinish | just hostname | 15:44 |
| clarkb | the information provided is accurate | 15:44 |
| fungi | cloud+region not provider+region | 15:44 |
| mtreinish | that's why we added the parsing | 15:45 |
| clarkb | mtreinish: right | 15:45 |
| jeblair | yes, if this were easy, it would have been done :) | 15:45 |
| jeblair | (it will be easy in zuulv3) | 15:45 |
| jeblair | but at the moment, we don't have a good way to get that metadata from nodepool to zuul | 15:45 |
| jeblair | it is disintermediated by jenkins at the moment | 15:45 |
| sdague | well, the regex can be fixed to be made a bit better at least, mtreinish you have a pointer to where it is? | 15:46 |
| *** tbachman has quit IRC | 15:46 | |
| *** apoorvad has joined #openstack-infra | 15:46 | |
| clarkb | the problem is you can have an aritrary number of -s in the hostname too | 15:46 |
| *** bpokorny has joined #openstack-infra | 15:46 | |
| *** jaosorior has quit IRC | 15:46 | |
| clarkb | we may have since cleaned up any with 2 or more though | 15:46 |
| mtreinish | sdague: http://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-client.py#n130 | 15:47 |
| fungi | true. dns is pretty limited in available characters to provide separators, so no way to tell where parts of the node type end and parts of the provider name begin | 15:47 |
| sdague | regardless, we should really pull vexxhost out of use. It's still randomly killing things by being super slow | 15:47 |
| jeblair | right, we can flip it so the potential failure is in the other direction | 15:47 |
| *** jaosorior has joined #openstack-infra | 15:47 | |
| *** harlowja_at_home has joined #openstack-infra | 15:47 | |
| *** MarkAtwood has joined #openstack-infra | 15:47 | |
| jeblair | we also have a proposal on the table to make hostnames meaningless | 15:47 |
| clarkb | we do? | 15:47 |
| *** jaosorior has quit IRC | 15:48 | |
| *** mixos has quit IRC | 15:48 | |
| *** david-lyle_ has quit IRC | 15:48 | |
| fungi | sdague: agreed. i'll put together the list of impacted nodes for mnaser and we can lower max-servers to 0 there for now. still need to retry the traceroute bit so we can provide more feedback to rackspace and hopefully turn ord back on though | 15:48 |
| *** jaosorior has joined #openstack-infra | 15:48 | |
| *** david-lyle has joined #openstack-infra | 15:48 | |
| *** sputnik13 has quit IRC | 15:48 | |
| clarkb | fungi: you can add traceroute to the package listing that i updated for uuidgen and friends | 15:48 |
| jeblair | clarkb: yeah, there was talk of that with the nova launch count optimization. i don't think we're going to head in that direction. just saying there may in general be good reasons to avoid 'hostname' as a data communication channel. | 15:48 |
| *** amotoki has joined #openstack-infra | 15:48 | |
| clarkb | fungi: you will have to separate add it for bare-* if that isn't already done | 15:49 |
| mordred | also, we could do ubuntu-trusty-2-node-9298435.bluebox-sjc1 instead of ubuntu-trusty-2-node-bluebox-sjc1-9298435 | 15:49 |
| fungi | jeblair: that likely means i should reconsider part of 300912 | 15:49 |
| jeblair | fungi: well, it's correct now. | 15:49 |
| mordred | or even ubuntu-trusty-2-node-9298435.sjc1.bluebox to match the mirror pattern | 15:49 |
| jeblair | fungi: and like i said, i don't think it's going to change any time soon. | 15:50 |
| fungi | clarkb: yep, i was leaning toward adding it to the infra-package-needs element | 15:50 |
| *** greghaynes has quit IRC | 15:50 | |
| fungi | jeblair: okay. in that case if you're not concerned about any changes i made in 300912 i'll go ahead and fire off my reply | 15:51 |
| jeblair | maybe someone should look into whether we can attach more metadata to jenkins slaves. | 15:51 |
| jeblair | fungi: lgtm | 15:51 |
| fungi | thanks! | 15:51 |
| *** ddieterly[away] is now known as ddieterly | 15:51 | |
| jeblair | fungi: i think the spirit ("developers can/will know jobs are running on your cloud") will persist beyond the particular implementation of encoding that in hostnames anyway. | 15:52 |
| fungi | jeblair: i agree, though that metadata may surface in different places in the future | 15:52 |
| *** tosky has quit IRC | 15:52 | |
| *** Qiming has quit IRC | 15:52 | |
| fungi | i still expect it to hold true aside from the current implementation detail of it being in hostnames | 15:52 |
| *** mixos has joined #openstack-infra | 15:53 | |
| *** piet has quit IRC | 15:53 | |
| *** tbachman has joined #openstack-infra | 15:54 | |
| jeblair | okay, i have a way to fix this problem correctly | 15:54 |
| clarkb | we also need to be careful about hostname limits, mordreds suggestion should neatly deal with that | 15:54 |
| *** zeih has quit IRC | 15:54 | |
| jeblair | oh | 15:54 |
| *** Guest12623 has quit IRC | 15:54 | |
| jeblair | by correctly, i mean, "not by parsing hostname" | 15:55 |
| *** vincentll has quit IRC | 15:55 | |
| jeblair | it will involve a nodepool change to pass some extra metadata to jenkins and possibly a zmq-plugin to extract that metadata | 15:56 |
| jeblair | is anyone interested in working on it? | 15:56 |
| *** tbachman has quit IRC | 15:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Use . between node and provider info in hostnames https://review.openstack.org/301202 | 15:56 |
| jeblair | no? all we're going to do is just parse hostnames? | 15:57 |
| mordred | jeblair: I think doing the zmq/nodepool work is an excellent idea | 15:57 |
| mordred | I do not want to offer to work on it right now becaue I'm a little bit slammed | 15:57 |
| jeblair | me too. | 15:57 |
| openstackgerrit | Sean Dague proposed openstack-infra/puppet-log_processor: a better heuristic for figuring out node provider https://review.openstack.org/301204 | 15:58 |
| jeblair | more parsing. | 15:58 |
| sdague | how about that heuristic change | 15:58 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Turn off vexxhost in nodepool https://review.openstack.org/301206 | 15:59 |
| jeblair | sdague, mtreinish: could i convince one of you to spend just a little time in nodepool so we can pass the metadata via a channel other than hostname? | 15:59 |
| sdague | jeblair: that's not really in the cards for me atm given the contexts I need to keep in my head. I can propose the narrow fixes, but the broader stuff is going to have to be left to someone that has nodepool in their head. | 16:00 |
| mtreinish | jeblair: I doubt I'll have the bandwidth for that | 16:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add experimental centos-7 beaker job for testing https://review.openstack.org/301207 | 16:01 |
| *** mixos has quit IRC | 16:01 | |
| *** hieulq has quit IRC | 16:01 | |
| *** mixos has joined #openstack-infra | 16:02 | |
| zaro | morning | 16:02 |
| *** greghaynes has joined #openstack-infra | 16:02 | |
| pabelanger | need to step out for 30mins, but if project-config-core would mind reviewing ^. Adds experimental job for beaker on centos-7 | 16:02 |
| mnaser | fungi please let me know if you have a list | 16:02 |
| mnaser | or just one | 16:02 |
| *** mixos has quit IRC | 16:02 | |
| sdague | mnaser: https://jenkins07.openstack.org/job/gate-grenade-dsvm/31594/console is currently running | 16:02 |
| sdague | but will be of the same fail class | 16:02 |
| sdague | ubuntu-trusty-vexxhost-ca-ymq-1-9298730 | 16:02 |
| sdague | if fungi can translate that to uuid, you could see it in action | 16:03 |
| mnaser | i can do that | 16:03 |
| anteaya | morning zaro | 16:03 |
| fungi | mnaser: want me to hold that node so it's not deleted when the job completes? | 16:03 |
| openstackgerrit | Merged openstack-infra/project-config: Move tox-bandit to check as non voting https://review.openstack.org/300564 | 16:04 |
| jeblair | this is the sort of low-hanging-fruit project that i really wish people would work on rather than tab alignment | 16:04 |
| *** kzaitsev_mb has quit IRC | 16:04 | |
| mordred | jeblair: ++ | 16:04 |
| *** kushal has quit IRC | 16:04 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 16:04 |
| jeblair | it is absolutely something that anyone in this channel can work on | 16:04 |
| anteaya | mnaser: we are discussing not having nodes spin up in vexxhost for a bit until this is addressed: https://review.openstack.org/#/c/301206/1 | 16:04 |
| *** ddieterly is now known as ddieterly[away] | 16:05 | |
| anteaya | jeblair: I was going to try to get the puppet-mediawiki module up on a vm | 16:05 |
| anteaya | jeblair: but I can do this first if I am given some guidance | 16:05 |
| anteaya | if you think I can do the work | 16:05 |
| mnaser | fungi: yes please, and can i get access to it? | 16:05 |
| mnaser | the vm is using ~46% cpu usage | 16:05 |
| anteaya | otherwise I'll default to my first plan | 16:05 |
| mnaser | the hypervisor is running at 0% iowait, 40% idle cpu | 16:06 |
| fungi | mnaser: i've held it now. what's your ssh key again? | 16:06 |
| *** kushal has joined #openstack-infra | 16:06 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/irc-meetings: Move Heat meeting https://review.openstack.org/301151 | 16:06 |
| jeblair | and i find it very frustrating that we would add a known-broken feature like this without folks willing to step up to fix it | 16:06 |
| *** tiswanso has quit IRC | 16:06 | |
| mnaser | fungi: http://paste.openstack.org/show/492895/ | 16:07 |
| *** e0ne has quit IRC | 16:07 | |
| * anteaya defaults to the wiki thing | 16:07 | |
| fungi | mnaser: looks like the current process eating an entire cpu is running "nova-manage db sync" | 16:07 |
| mnaser | im getting quite suspicious this is a kernel issue | 16:07 |
| mnaser | hmm | 16:07 |
| mnaser | single-threaded process | 16:07 |
| *** ddieterly[away] is now known as ddieterly | 16:07 | |
| *** ddieterly is now known as ddieterly[away] | 16:08 | |
| *** ddieterly[away] is now known as ddieterly | 16:08 | |
| mnaser | that actually explains a lot | 16:08 |
| *** kushal has quit IRC | 16:08 | |
| fungi | mnaser: ssh jenkins@162.253.53.209 | 16:08 |
| mnaser | so libvirt applies limits per vcpu instead of entire instance, so single threaded processes become slower instead of bursting up into an entire cpu | 16:09 |
| *** mixos has joined #openstack-infra | 16:09 | |
| fungi | yeah, now apache2 is eating an entire cpu on that worker | 16:10 |
| fungi | that was short-lived though | 16:10 |
| mnaser | i'm fairly certain i have an idea of where the problem is now | 16:10 |
| mnaser | well, at least our initial resolution fixed it but in turn limited things a bit more than they should be, for single threaded work | 16:11 |
| fungi | mnaser: still want a list of instances where we saw similar performance issues in the past half hour? | 16:11 |
| fungi | er, past 24 hours i mean | 16:11 |
| mnaser | no, this should be enough to help us find the problem | 16:11 |
| *** claudiub has joined #openstack-infra | 16:11 | |
| mordred | jeblair: I'm totally willing to eventually step up and fix it - it'll just realistically be a few weeks until I'm likely to be able to do it - so I was putting up hacky-workaround related patches to help alleviate the current situation and give space for getting the proper fix done | 16:11 |
| *** scheuran has quit IRC | 16:11 | |
| fungi | mnaser: okay, thanks for digging into it! | 16:11 |
| clarkb | I am still not sure I understand what the issue is | 16:12 |
| clarkb | the data in elasticserach is accurate | 16:12 |
| clarkb | you can query on it reliably | 16:12 |
| *** tiswanso has joined #openstack-infra | 16:12 | |
| mordred | clarkb: yes. it's just not obvious | 16:12 |
| clarkb | and I think people might be more willing to volunteer if the individuals that wrote the existing code aren't made to feel like they shouldn't have bothered | 16:12 |
| fungi | clarkb: i think the complaint is that the "node_provider" parameter doesn't reflect reality | 16:13 |
| mordred | clarkb: that you need to search on ymq-1 to get vexxhost | 16:13 |
| clarkb | fungi: yes I agree it can be better | 16:13 |
| *** EricGonczer_ has quit IRC | 16:14 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Create openstack/designate-tempest repository https://review.openstack.org/301219 | 16:14 |
| clarkb | but the current info is complete and accurate | 16:14 |
| mordred | clarkb: I also agree - the tone on this was entirely wrong, and I apologize for my part in that | 16:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-log_processor: Update provider regex to deal with provider subdomains https://review.openstack.org/301220 | 16:15 |
| jeblair | clarkb: when the original code was written, this problem was raised as a potential issue but ignored. it has happened, and yet, no one in stepping up to fix it properly. | 16:15 |
| *** MarkAtwood has quit IRC | 16:15 | |
| mordred | there is the other half of my thing - based on top of sdague's thing - I will not be offended if those patches are not used, but there they are | 16:15 |
| jeblair | i'm going to write the fix to this | 16:17 |
| *** esikachev has quit IRC | 16:17 | |
| andreaf | hi - I'm looking for an API spec for http://codesearch.openstack.org/api/v1/search | 16:18 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/gear: Bump version requirement for pbr https://review.openstack.org/301222 | 16:19 |
| clarkb | andreaf: https://github.com/etsy/hound is the tool used | 16:19 |
| *** yaume has quit IRC | 16:19 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: (WIP) Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 16:20 |
| *** ramishra has joined #openstack-infra | 16:20 | |
| andreaf | clarkb: thanks! | 16:21 |
| dims | fungi : can you please add me to the new python-k8sclient-core and -release groups? | 16:21 |
| morgan | mordred, fu | 16:21 |
| morgan | Ugh | 16:21 |
| *** esikachev has joined #openstack-infra | 16:21 | |
| morgan | mordred, fungi, clarkb: what are the amounts of resources wanted if a company is donating to infra? | 16:21 |
| morgan | Cloud resources that is | 16:22 |
| anteaya | morgan: right now we accept public cloud | 16:22 |
| jeblair | morgan: http://docs.openstack.org/infra/system-config/contribute-cloud.html | 16:22 |
| morgan | mordred: I tab fail | 16:22 |
| *** fedexo has quit IRC | 16:22 | |
| anteaya | morgan: the page jeblair linked | 16:22 |
| morgan | anteaya: correct. Was looking for the volume | 16:22 |
| anteaya | morgan: 100 vms | 16:22 |
| morgan | Perfect | 16:23 |
| anteaya | glad to hear you say so | 16:23 |
| jeblair | (which is on that page) | 16:23 |
| morgan | jeblair: yep. Looking at it. Thanks! Important any for $reasons$ | 16:23 |
| *** tbachman has joined #openstack-infra | 16:24 | |
| mnaser | question: do jobs require 8 cores specifically? if we provided an instance type with 2 vcpus @ 2.93ghz (but fully dedicated to the instance, not shared), would that work? | 16:24 |
| morgan | mnaser: ooh yay vexxhost things. ;) | 16:24 |
| mnaser | :) | 16:25 |
| mordred | mnaser: the main thing is "can devstack gate jobs complete in the amount of time that is withing acceptable tolerances" | 16:25 |
| *** mylu has quit IRC | 16:25 | |
| morgan | mnaser: the jobs are mostly budgeted for speed of devstack/them..what mordred said | 16:25 |
| mnaser | hmm | 16:25 |
| clarkb | and the extra cpus do help when you are running mysql and opentsack and rabbit all together | 16:25 |
| mordred | yah | 16:25 |
| fungi | also number of vcpus is used to determine some job parallelism | 16:25 |
| mordred | devstack/tempest have ability to do multi-threaded too - but yah, honestly, if it was one REALLY FAST vcpu I don't think that would be a problem | 16:26 |
| clarkb | osic jobs run on 4vcpu though | 16:26 |
| mnaser | so basically the idea to be transparent here is that 8gb instances get 2 physical cores from the system as a limit overall | 16:26 |
| mnaser | so it's 8vcpus, but not going more than 2 physical cores on the system | 16:26 |
| mnaser | the problem is that libvirt currently limits each core instead of the entire set of cores combined | 16:26 |
| *** tphummel has joined #openstack-infra | 16:27 | |
| *** mylu has joined #openstack-infra | 16:27 | |
| mnaser | this is why instances that are doing single-threaded work are struggling, because they're getting 50% of a single core | 16:27 |
| mordred | ah - so the budgeted amount of real cpu space is going unused if we have one single vcpu pegged | 16:27 |
| mordred | yah | 16:27 |
| mnaser | ideally we'd use the global bandwidth settings but that is to come in libvirt 1.3.3 | 16:27 |
| mnaser | https://github.com/libvirt/libvirt/commit/ef1fa55e469088a3819024ae86d6b8cd27e4e919 | 16:27 |
| mnaser | this would let us limit all 8 cores to 2 physical, therefore one or two cores can burst up to an entire physical core, or the combination of the 8 will combine to 2 in total | 16:28 |
| *** EricGonczer_ has joined #openstack-infra | 16:28 | |
| mnaser | so the alternative is to either: a) present 2 fully dedicated cores to the OS, b) patch nova to manually set limits until that patch lands in libvirt and we move to 1.3.3 .. can't think of much more | 16:28 |
| clarkb | mnaser: is this a relatively new change? the runtimes we had when we first put vexxhost in place were in line with the other clouds | 16:30 |
| clarkb | between 50 minutes and an hour iirc | 16:30 |
| *** sfinucan__ has quit IRC | 16:31 | |
| *** nmagnezi_ has quit IRC | 16:31 | |
| mnaser | clarkb: yes, we didn't have any limits initially but the scheduler was sending instances to already-busy compute nodes | 16:32 |
| mnaser | so the foundation was being it's own noisy neighbor as well as a few other tenants that are very cpu hungry | 16:32 |
| mordred | ah. yeah. we can definitely be our own noisy neighbor :) | 16:33 |
| mnaser | we have the utilization based scheduler in place so it does aim to throw instances at least-busy-compute-nodes now instead of just based on free memory | 16:33 |
| *** ramishra has quit IRC | 16:33 | |
| *** sputnik13 has joined #openstack-infra | 16:33 | |
| mnaser | that helped in sending instances to not-so-busy compute nodes, but the problem is that you can have 3 instances launched, places on the same node because it's not loaded, they start running their tests and now it's overloaded and you can't do much | 16:34 |
| *** oanson has joined #openstack-infra | 16:34 | |
| *** rossella_s has quit IRC | 16:34 | |
| *** ramishra has joined #openstack-infra | 16:35 | |
| *** rossella_s has joined #openstack-infra | 16:35 | |
| mnaser | is there a way i could a test suite on an independent instance (or to try it out on a different flavor/instance_type?) | 16:37 |
| clarkb | mnaser: yes, we have a reproduce.sh script written out by each job that you can run | 16:38 |
| clarkb | mnaser: let me get you a link to one | 16:38 |
| mnaser | awesome, thanks clarkb | 16:38 |
| *** esker has quit IRC | 16:39 | |
| *** harlowja_at_home has quit IRC | 16:39 | |
| *** shashank_hegde has quit IRC | 16:39 | |
| clarkb | mnaser: http://logs.openstack.org/24/297324/1/gate/gate-tempest-dsvm-full/7e3f687/logs/reproduce.sh | 16:39 |
| *** jsavak has quit IRC | 16:40 | |
| mnaser | clarkb: perfect, let me do some checking | 16:40 |
| *** jsavak has joined #openstack-infra | 16:41 | |
| clarkb | mnaser: the comment header should give you a quick rundown of what is required ot run that | 16:41 |
| *** EricGonczer_ has quit IRC | 16:42 | |
| *** EricGonczer_ has joined #openstack-infra | 16:43 | |
| *** salv-orlando has joined #openstack-infra | 16:43 | |
| clarkb | mnaser: one of thoes files should be written out for every devstack-gate job that is run. I picked a recent one for a change that merged, but you can use that to reproduce any specific change's tests | 16:44 |
| SpamapS | trash: huh? | 16:45 |
| SpamapS | trash: it merged on March 22? | 16:45 |
| mnaser | clarkb: trying it now | 16:45 |
| *** esikachev has quit IRC | 16:45 | |
| *** ramishra has quit IRC | 16:46 | |
| *** nmagnezi_ has joined #openstack-infra | 16:47 | |
| *** smarcet has joined #openstack-infra | 16:47 | |
| *** ddieterly is now known as ddieterly[away] | 16:47 | |
| mnaser | clarkb: trying to run it now on a dedicated 2 core machine and will see time | 16:47 |
| *** rcernin has quit IRC | 16:48 | |
| clarkb | mnaser: note that you don't want to compare the jobs absolute time because you won't have the mirrors and git cache, but you can compare the tempest run time which should be comparable to the other jobs | 16:48 |
| mnaser | btw | 16:49 |
| mnaser | dd if=/dev/zero of=/root/swapfile bs=1M count=8192 | 16:49 |
| mnaser | why not replace this with fallocate | 16:49 |
| mnaser | waaaaay faster and easier on disk io | 16:49 |
| *** eharney has quit IRC | 16:49 | |
| clarkb | mnaser: because centos in its great wisdom uses xfs | 16:49 |
| mnaser | :( | 16:49 |
| clarkb | mnaser: it is possible that once we are 100% building our own images we can switch back to fallocate | 16:50 |
| clarkb | since we can force everything to be ext4 | 16:50 |
| mordred | ++ | 16:50 |
| *** ifarkas has quit IRC | 16:50 | |
| mordred | clarkb: we're still snapshotting centos right? | 16:50 |
| clarkb | mordred: not snapshotting, but in the middle of switching form using upstream base images to the minimal element | 16:50 |
| clarkb | mordred: the upstream base images start with an xfs / | 16:51 |
| *** tbachman has left #openstack-infra | 16:51 | |
| mordred | clarkb: ah. cool | 16:51 |
| *** esikachev has joined #openstack-infra | 16:51 | |
| clarkb | but dib defaults to ext4 so as long as we are formatting our own disk setup we should be fine | 16:51 |
| *** ddieterly[away] is now known as ddieterly | 16:52 | |
| *** bkeller` has quit IRC | 16:52 | |
| clarkb | mnaser: we had also tried a sparse file loopback mounted but that caused really odd behaviors of instances basically going away | 16:52 |
| *** salv-orlando has quit IRC | 16:52 | |
| clarkb | and never coming back (I left one running overnight to see if it would recover) | 16:52 |
| mnaser | that's strange behaviour | 16:52 |
| clarkb | mnaser: theory was that we were running low on disk and memory and then trying to swap | 16:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Encode nodepool info in jenkins description https://review.openstack.org/301236 | 16:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zmq-event-publisher: Add node description to event data https://review.openstack.org/301237 | 16:53 |
| mnaser | should this be logging things somewhere | 16:53 |
| clarkb | and since neither end of that could be satisfied the instance was hosed | 16:53 |
| clarkb | mnaser: in /opt/stack/new/ | 16:53 |
| jeblair | clarkb: can you take a quick look at 301326 and 301327 and tell me if you are okay with the approach? | 16:53 |
| clarkb | jeblair: sure | 16:53 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 16:54 |
| *** oanson has quit IRC | 16:54 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch gate-infra-puppet-apply-centos-7 voting https://review.openstack.org/301239 | 16:54 |
| mnaser | it's still just cloning things so unsure if there should be any output or not yet during clone | 16:55 |
| *** eharney has joined #openstack-infra | 16:55 | |
| *** oanson has joined #openstack-infra | 16:55 | |
| clarkb | jeblair: that seems fine, we aren't really using that description otherwise and the slave name itself encodes the details that will no longer be in the description | 16:55 |
| jeblair | clarkb: yeah, and we can add more to that dict in the future | 16:55 |
| clarkb | mnaser: during the clones there isn't output, it is logged to hte setup workspace log file | 16:55 |
| mordred | jeblair: love it | 16:56 |
| pabelanger | if anybody has time to review https://review.openstack.org/#/q/topic:centos-7+status:open that will help with our devstack-centos7 to centos-7 migration | 16:56 |
| jeblair | clarkb: an alternative approach i considered was -- apparently you can add node-specific environment variables. i do not know if those would show up in the parameters/env vars that currently end up in the zmq stream. | 16:56 |
| clarkb | mordred: if you are revieiwng nodepool changes https://review.openstack.org/#/c/300678/ should get in before next restart so that we don't have to manually clenaup the db again | 16:57 |
| *** mylu has quit IRC | 16:57 | |
| jeblair | but if we don't like using the description field, we could see if those would work. | 16:58 |
| mordred | clarkb: looks generally good - any reason to remove that print? | 16:58 |
| *** exploreshaifali has joined #openstack-infra | 16:58 | |
| mordred | jeblair: I think using the description field sounds like the right amount of effort to fix this until we're on v3 | 16:58 |
| clarkb | mordred: ya it was debugging code that had snuck in >_> | 16:58 |
| mordred | clarkb: so, I'm going to +2 but not +A since we're still rolled-back on nodepool.o.o | 16:59 |
| *** aysyd has quit IRC | 16:59 | |
| clarkb | oh I approved jeblair's nodepool chnag etoo, should I remove the approval and just leave the +2/ | 16:59 |
| mordred | clarkb: well, we just need to not restart nodepool, or re-revert the git repo state locally | 17:00 |
| clarkb | I will do that | 17:00 |
| mordred | or just remove the +A for now - the occ change we need is in the midst of landing, so we should be out of weird-zone soon | 17:00 |
| anteaya | never | 17:00 |
| anteaya | I have never seen us out of weird-zone | 17:00 |
| clarkb | mordred: ya I removed hte +A and left my +2 | 17:00 |
| *** mylu has joined #openstack-infra | 17:01 | |
| mordred | anteaya: well, this specific zone of weird where we're running on a reverted nodepool | 17:03 |
| *** rossella_s has quit IRC | 17:03 | |
| *** oanson has quit IRC | 17:04 | |
| *** rossella_s has joined #openstack-infra | 17:04 | |
| jasondotstar | jeblair: just read back... can you take a moment to talk more about this nodepool issue that you were discussing earlier? | 17:04 |
| jasondotstar | or if there is somewhere I can read up on it, I'll that that. | 17:05 |
| *** tqtran has joined #openstack-infra | 17:06 | |
| anteaya | mordred: okay this specific zone | 17:06 |
| cody-somerville | Happy Monday Folks. \o_ | 17:06 |
| *** hashar has quit IRC | 17:06 | |
| *** armax has joined #openstack-infra | 17:07 | |
| *** EricGonczer_ has quit IRC | 17:07 | |
| *** mylu has quit IRC | 17:07 | |
| *** jamesmcarthur has quit IRC | 17:08 | |
| *** mylu has joined #openstack-infra | 17:08 | |
| pabelanger | clarkb: +2 on 300678, hit +A by mistake | 17:09 |
| *** mylu has quit IRC | 17:10 | |
| *** shashank_hegde has joined #openstack-infra | 17:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate puppet jobs to centos-7 dib https://review.openstack.org/301251 | 17:12 |
| *** esikachev has quit IRC | 17:13 | |
| pabelanger | EmilienM: migration from devstack-centos7 to centos-7 dib ^, if you want to review it. Doubt it will land today, but will take some time to reload JJB before of it | 17:13 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for build timeouts on vexxhost bug 1565850 https://review.openstack.org/301187 | 17:14 |
| openstack | bug 1565850 in OpenStack-Gate "79% of build timeouts are on vexxhost ymq-1 nodes" [Undecided,New] https://launchpad.net/bugs/1565850 | 17:14 |
| *** aysyd has joined #openstack-infra | 17:15 | |
| *** xarses has joined #openstack-infra | 17:16 | |
| *** Sukhdev has quit IRC | 17:16 | |
| *** ddieterly is now known as ddieterly[away] | 17:17 | |
| *** shashank_hegde has quit IRC | 17:19 | |
| *** nadya has joined #openstack-infra | 17:19 | |
| *** amrith is now known as _amrith_ | 17:19 | |
| *** trown is now known as trown|lunch | 17:20 | |
| *** claudiub|2 has joined #openstack-infra | 17:21 | |
| *** jlanoux has quit IRC | 17:22 | |
| *** nmagnezi_ has quit IRC | 17:23 | |
| *** amotoki has quit IRC | 17:23 | |
| *** claudiub has quit IRC | 17:24 | |
| *** ihrachys has quit IRC | 17:24 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:25 | |
| *** mrmartin has joined #openstack-infra | 17:26 | |
| *** sridhar_ram has joined #openstack-infra | 17:26 | |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/puppet-reviewday: Run pip install to install the dependencies https://review.openstack.org/301255 | 17:29 |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/reviewday: Add a link for Neutron Gerrit dashboard https://review.openstack.org/298779 | 17:30 |
| mnaser | clarkb: how much of a speedup do the mirrors give when running devstak | 17:31 |
| mnaser | *devstack | 17:31 |
| clarkb | mnaser: it drops the git clone setup time to about 5 minutes and devstack setup time to about 10 minutes | 17:31 |
| clarkb | mnaser: you can compare the tempest runtime numbers which it outputs and that should be roughly comparable | 17:32 |
| mnaser | oh it will output the tempest runtime at the end? thats good | 17:32 |
| clarkb | yup | 17:32 |
| clarkb | testr does that for us | 17:32 |
| clarkb | or the testr wrapper | 17:32 |
| rossella_s | clarkb, hi! I've followed your suggestions in https://review.openstack.org/#/c/298779/ , I'd appreciate if you could have another look | 17:32 |
| clarkb | rossella_s: will do | 17:32 |
| rossella_s | clarkb, thanks a lot, I've also created a patch to run 'pip install' in puppet-reviewday...not sure if the code is right | 17:33 |
| *** tiswanso has quit IRC | 17:33 | |
| dims | clarkb: can you please add me to the new python-k8sclient-core and -release groups? | 17:33 |
| *** tiswanso has joined #openstack-infra | 17:33 | |
| *** claudiub|2 has quit IRC | 17:33 | |
| *** oanson has joined #openstack-infra | 17:33 | |
| clarkb | dims: sure | 17:34 |
| dims | thanks! | 17:34 |
| clarkb | dims: done | 17:34 |
| dims | awesome clarkb thanks | 17:34 |
| *** claudiub has joined #openstack-infra | 17:35 | |
| clarkb | rossella_s: where is the puppet change? | 17:38 |
| EmilienM | pabelanger: ok - in case you're landing it tomorrow / wednesday, note I'll be afk. Your CI contact will be degorenko or mwhahaha | 17:38 |
| degorenko | hey :) | 17:38 |
| rossella_s | clarkb, https://review.openstack.org/#/c/301255/ , I put a the depends-on tag on the dashboard patch | 17:38 |
| pabelanger | EmilienM: ack' | 17:38 |
| *** mylu has joined #openstack-infra | 17:40 | |
| *** rhochmuth has quit IRC | 17:41 | |
| *** rhochmuth has joined #openstack-infra | 17:41 | |
| *** rhochmuth has quit IRC | 17:41 | |
| *** piet has joined #openstack-infra | 17:42 | |
| clarkb | pabelanger: EmilienM so we should hurry up and do https://review.openstack.org/#/c/301251/ ? | 17:43 |
| EmilienM | clarkb: I'm ready | 17:43 |
| EmilienM | clarkb: I was reviewing the zuul/layout | 17:43 |
| EmilienM | but lgtm | 17:43 |
| *** shashank_hegde has joined #openstack-infra | 17:44 | |
| *** pvaneck has joined #openstack-infra | 17:45 | |
| clarkb | pabelanger: and you are good to watch the solum and other smaller hcanges if I approve them? | 17:45 |
| pabelanger | clarkb: yes | 17:46 |
| clarkb | eh I can do that too so will go ahead | 17:46 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on expand for Summit Event Feeback https://review.openstack.org/301261 | 17:46 |
| *** bkeller` has joined #openstack-infra | 17:46 | |
| clarkb | pabelanger: do you need https://review.openstack.org/#/c/301207/1 first? | 17:48 |
| pabelanger | clarkb: don't need. That's to test our infra CI | 17:48 |
| *** oanson has quit IRC | 17:48 | |
| pabelanger | Can rebase it out if needed | 17:48 |
| openstackgerrit | Nikhil Komawar proposed openstack-infra/irc-meetings: Update the Glare team meeting chair https://review.openstack.org/301263 | 17:49 |
| clarkb | no rebasing needed just trying ot understand what oes where | 17:49 |
| pabelanger | I just wanted to make sure out puppet-openstackci beaker job worked before moving infra to centos-7 | 17:49 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on expand for Summit Event Feeback https://review.openstack.org/301261 | 17:49 |
| pabelanger | pretty sure it will | 17:49 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on expand for Summit Event Feeback https://review.openstack.org/301261 | 17:50 |
| *** weshay has quit IRC | 17:51 | |
| clarkb | rossella_s: reviewed | 17:52 |
| *** nwkarsten has quit IRC | 17:53 | |
| *** nwkarsten has joined #openstack-infra | 17:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Move gate-tempest-dsvm-docker-centos7 to centos-7 https://review.openstack.org/300119 | 17:55 |
| *** jamesmcarthur has joined #openstack-infra | 17:56 | |
| *** nwkarsten has quit IRC | 17:57 | |
| *** mrmartin has quit IRC | 17:58 | |
| *** angdraug has joined #openstack-infra | 17:58 | |
| pabelanger | before we land 301206 (turn off vexxhost) did we want to check if we can re-enable rax-ord? | 17:58 |
| pabelanger | since we'll be down 2 regions | 17:58 |
| clarkb | pabelanger: I think fungi wanted to have traceroute working before turning rax-ord back on | 17:58 |
| pabelanger | Ah | 17:59 |
| *** bpokorny has quit IRC | 17:59 | |
| clarkb | pabelanger: I htink its fine to stop using vexxhost now regardless due to the error rate | 17:59 |
| pabelanger | okay | 17:59 |
| clarkb | and mnaser is looking into it on their side so we shouldn't need more data | 17:59 |
| fungi | yep, just about caught up on other obligations and putting that change to add traceroute in now | 17:59 |
| fungi | most of the work is figuring out what the packages for it are called on all our different platforms | 18:00 |
| clarkb | fungi: its 'traceroute' on ubuntu | 18:00 |
| *** sambetts is now known as sambetts|afk | 18:00 | |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| *** yamamoto has quit IRC | 18:01 | |
| *** yamamoto has joined #openstack-infra | 18:01 | |
| fungi | clarkb: yep, that one was easy ;) | 18:01 |
| clarkb | fungi: and on fedora too https://apps.fedoraproject.org/packages/traceroute | 18:02 |
| fungi | mnaser: do you still need ubuntu-trusty-vexxhost-ca-ymq-1-9298730 held or am i clear to release it for deletion now? | 18:02 |
| *** flepied has quit IRC | 18:02 | |
| *** rhochmuth has joined #openstack-infra | 18:02 | |
| mnaser | sorry, no need for it anymore fungi | 18:02 |
| openstackgerrit | Merged openstack-infra/project-config: Move gate-solum-devstack-dsvm-centos7 to centos-7 https://review.openstack.org/300117 | 18:02 |
| fungi | clarkb: i'm hoping centos 6 is similar | 18:02 |
| fungi | and precise | 18:02 |
| fungi | mnaser: thanks! | 18:02 |
| clarkb | fungi: seems to be for precise http://packages.ubuntu.com/search?keywords=traceroute | 18:03 |
| *** doug-fish has quit IRC | 18:03 | |
| clarkb | centos is harder because they don't seem ot have a fancy package search anywhere I can find | 18:03 |
| *** doug-fish has joined #openstack-infra | 18:03 | |
| *** doug-fish has quit IRC | 18:03 | |
| *** mixos has quit IRC | 18:04 | |
| *** davideag_ has quit IRC | 18:04 | |
| fungi | our bare-trusty workers have traceroute6 but not traceroute | 18:04 |
| fungi | er, bare-precise | 18:04 |
| *** jsavak has quit IRC | 18:04 | |
| *** jsavak has joined #openstack-infra | 18:05 | |
| fungi | oh, also package installation in jobs for precise is broken anyway, because we're pointing it at our package mirrors | 18:06 |
| *** degorenko is now known as _degorenko|afk | 18:06 | |
| fungi | we should probably not update sources.list for precise | 18:06 |
| *** e0ne has joined #openstack-infra | 18:06 | |
| fungi | or else we should start mirroring precise | 18:06 |
| fungi | but given we only use it in rackspace, having it on our distributed mirrors is of questionable value | 18:07 |
| fungi | especially since we're working on getting rid of it anyway | 18:07 |
| clarkb | I vote for not mirroring precise and just switching on release in the setup script | 18:07 |
| clarkb | fungi: http://mirror.centos.org/centos/6/os/x86_64/Packages/traceroute-2.0.14-2.el6.x86_64.rpm | 18:08 |
| *** ddieterly[away] is now known as ddieterly | 18:08 | |
| clarkb | fungi: and http://mirror.centos.org/centos/7/os/x86_64/Packages/traceroute-2.0.19-5.el7.x86_64.rpm so I think it is universally traceroute, not sure if traceroute6 comes in a different package on different distros though | 18:09 |
| *** kushal has joined #openstack-infra | 18:09 | |
| fungi | that's the part i'm worried about and currently validating | 18:10 |
| *** jsavak has quit IRC | 18:10 | |
| *** jsavak has joined #openstack-infra | 18:11 | |
| *** salv-orlando has joined #openstack-infra | 18:11 | |
| fungi | we seem to already on centos7 they're in the same package at least | 18:11 |
| fungi | grr. composing two independent thoughts there ;) | 18:12 |
| fungi | on centos7 they're in the same package at least | 18:12 |
| clarkb | fungi: https://bugs.launchpad.net/ubuntu/+source/ubuntu-meta/+bug/57167 | 18:13 |
| openstack | Launchpad bug 57167 in ubuntu-meta (Ubuntu) "traceroute6 installed by default but not traceroute" [Undecided,Confirmed] | 18:13 |
| fungi | hah | 18:13 |
| clarkb | seems yhat at least in precise that behavior persisted | 18:13 |
| clarkb | but we should make sure our debootstrap is installing that iputils package | 18:14 |
| fungi | looks like it's "traceroute" everywhere | 18:14 |
| *** mixos has joined #openstack-infra | 18:14 | |
| pabelanger | fungi: should be easy to test with a bindep job. we gate on all our OSes now | 18:14 |
| pabelanger | minus bare-precise | 18:15 |
| clarkb | maybr I misread the bug but traceroute6 does seem special on ubuntu | 18:15 |
| clarkb | fungi pabelanger ehat I eould do is guard the traceroutes with a which check | 18:15 |
| clarkb | get that in now, then work on getting images updated | 18:15 |
| clarkb | then remove the guard at some point in future when all images are happy | 18:16 |
| fungi | clarkb: on trusty you don't get traceroute6 until you install the traceroute package, so seems to be limited to <=precise for the weirdness | 18:16 |
| *** esikachev has joined #openstack-infra | 18:17 | |
| fungi | clarkb: agreed, that's what i was planning to fix. only run traceroute commands if they exist (or optionally, just `|| true` on the end, which my original job config change had and then i spent some time testing various traceroute{,6} exit code behaviors and determined it was "unneeded") | 18:17 |
| fungi | if i'd left ||true on the end, wouldn't have broken the world | 18:18 |
| *** daemontool_ has quit IRC | 18:20 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add traceroute to default preinstalled packageset https://review.openstack.org/301272 | 18:21 |
| *** samueldmq has joined #openstack-infra | 18:21 | |
| *** e0ne has quit IRC | 18:22 | |
| *** trown|lunch is now known as trown | 18:23 | |
| scottda | Hi Infra. We've had a couple +2's on this patch. Could anyone else have a look: https://review.openstack.org/#/c/265925 | 18:23 |
| scottda | Add functional test job for python-brick-cinderclient-ext | 18:23 |
| dims | clarkb : fungi : is one hour a good time to wait between adding to ACL and seeing the +2 option in a review? | 18:25 |
| fungi | clarkb: the ||true solution is beginning to grow on me. traceroute even if it fails to reach its destination still exits 0, so this is really only guarding against the executable not existing at all. the only down-side i see is possibly misleading noise in the console log about "-bash: traceroute: command not found" et cetera | 18:25 |
| fungi | clarkb: oh, it does exit 2 if you give it an invalid/unresolvable target hostname | 18:26 |
| *** salv-orlando has quit IRC | 18:27 | |
| fungi | dims: yeah, should be plenty. but gerrit caches weird state in your browser. on the page with the voting dialogue, try a forced refresh | 18:27 |
| clarkb | dims: it has to do with gerrit caching, it might be quicker if you log out and in? | 18:27 |
| *** EricGonczer_ has joined #openstack-infra | 18:27 | |
| dims | clarkb : fungi : ack will try | 18:27 |
| *** mixos has quit IRC | 18:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental centos-7 beaker job for testing https://review.openstack.org/301207 | 18:27 |
| *** mixos has joined #openstack-infra | 18:28 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate puppet jobs to centos-7 dib https://review.openstack.org/301251 | 18:28 |
| dims | clarkb : fungi : worked like a charm. thanks | 18:29 |
| *** angdraug has quit IRC | 18:29 | |
| clarkb | fungi: note that won't affect bare-precise | 18:30 |
| clarkb | fungi: but I approved it anyways and we can just not do tracerouts on bare-precise if you || true | 18:30 |
| *** bpokorny has joined #openstack-infra | 18:30 | |
| *** mixos has quit IRC | 18:31 | |
| *** esikachev has quit IRC | 18:31 | |
| rossella_s | clarkb, thanks a lot! | 18:31 |
| *** salv-orlando has joined #openstack-infra | 18:32 | |
| fungi | clarkb: i also think that job macro is probably not used on jobs running on bare-precise | 18:32 |
| fungi | and we don't need to keep it around indefinitely either if we don't want | 18:32 |
| clarkb | ya | 18:33 |
| *** davideagnello has joined #openstack-infra | 18:33 | |
| *** kushal has quit IRC | 18:35 | |
| *** ddieterly is now known as ddieterly[away] | 18:35 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Traceroute to git.o.o in net-info macro https://review.openstack.org/301281 | 18:35 |
| jroll | it appears patches with a depends-on line, that are approved before the dependency is approved, don't get into the gate queue, is that intentional? https://review.openstack.org/#/c/300052/ | 18:36 |
| clarkb | jroll: if they do not share a gate queue in zuul yes | 18:36 |
| *** mixos has joined #openstack-infra | 18:36 | |
| fungi | jroll: _if_ they don't share any jobs in common with the dependency, then yes | 18:36 |
| anteaya | well less intentional ad more how the system works | 18:36 |
| anteaya | jroll: zuul doesn't have any memory, it only understands now | 18:36 |
| jroll | it seems they do not, in which case, how do I kick these into the gate? | 18:36 |
| clarkb | jroll: reapprove | 18:37 |
| clarkb | if you have an existing approval vote you will have to remove it and add it | 18:37 |
| clarkb | if another individual has an existing approval you cna just add yours | 18:37 |
| jroll | or neither if I am not core :) | 18:37 |
| fungi | jroll: basically since you can't integration-test the dependencies together (because of not sharing a job) it's waiting to test in the gate until the stated dependency has completely merged | 18:37 |
| *** nwkarsten has joined #openstack-infra | 18:37 | |
| jroll | dtroyer: sdague: could you please re-approve https://review.openstack.org/#/c/300054/ and https://review.openstack.org/#/c/300052/ ? | 18:37 |
| jroll | fungi: right, though the dep here is completely merged | 18:38 |
| clarkb | mordred: you probably want https://review.openstack.org/#/c/300745/ in before any release happens right? | 18:38 |
| clarkb | it claims to depend on a change that failed to merge | 18:39 |
| *** Sukhdev has joined #openstack-infra | 18:39 | |
| clarkb | but the change it depends on merged so it may just need rebase | 18:39 |
| *** mylu has quit IRC | 18:40 | |
| fungi | jroll: the other challenge (and reason it doesn't auto-enqueue) is because the depends-on in the commit message isn't easily queryable through gerrit, so there's no clean way to make the merger of the dependency trigger enqueuing the dependent | 18:41 |
| *** mixos has quit IRC | 18:41 | |
| jroll | fungi: yeah, makes sense | 18:41 |
| jroll | fungi: though, quoting the entire depends-on line in the search seems to work ok? https://review.openstack.org/#/q/%22Depends-On:+I3ec6dafb9522b5fe79e0c6681c2578d2a723380a%22 | 18:41 |
| fungi | or at least it used to not be easily queryable. you can now | 18:42 |
| *** _amrith_ is now known as amrith | 18:42 | |
| jroll | maybe api behaves differently | 18:42 |
| fungi | hah, i was just about to post taht link | 18:42 |
| jroll | aha, yeah | 18:42 |
| fungi | well, similar one anyway | 18:42 |
| fungi | https://review.openstack.org/#/q/message:%22Depends-On:+I3ec6dafb9522b5fe79e0c6681c2578d2a723380a%22 | 18:42 |
| anteaya | AJaeger: thanks | 18:42 |
| anteaya | any objection to me turning off vexxhost in nodepool by self-approving 301206? | 18:42 |
| fungi | jeblair: but yeah, the message querying in gerrit was pretty terrible until recent releases | 18:42 |
| fungi | er, jroll ^ | 18:43 |
| *** salv-orlando has quit IRC | 18:43 | |
| clarkb | anteaya: no, I didn't realize that was still not approved | 18:43 |
| fungi | sorry jeblair | 18:43 |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/puppet-reviewday: Run pip install to install the dependencies https://review.openstack.org/301255 | 18:43 |
| anteaya | clarkb: you are welcome to approve it you like | 18:43 |
| jroll | fungi: yeah, makes sense | 18:43 |
| *** mylu has joined #openstack-infra | 18:43 | |
| clarkb | anteaya: can do | 18:43 |
| anteaya | clarkb: would make me feel better | 18:43 |
| anteaya | thanks | 18:43 |
| *** e0ne has joined #openstack-infra | 18:43 | |
| *** tosky has joined #openstack-infra | 18:44 | |
| rossella_s | clarkb, I've just updated https://review.openstack.org/#/c/301255 according to your suggestions. Thanks a lot again for your help. | 18:45 |
| *** mylu has quit IRC | 18:46 | |
| *** EricGonczer_ has quit IRC | 18:48 | |
| *** EricGonczer_ has joined #openstack-infra | 18:48 | |
| *** matrohon has joined #openstack-infra | 18:49 | |
| *** nadya has quit IRC | 18:50 | |
| *** mylu has joined #openstack-infra | 18:52 | |
| mordred | clarkb: yes | 18:52 |
| mordred | clarkb: I have rechecked it | 18:53 |
| mordred | clarkb: once that lands, I'll put in a release request | 18:53 |
| clarkb | mordred: which of those changes does the find port with router stuff? | 18:53 |
| *** mylu has quit IRC | 18:53 | |
| clarkb | mordred: or did you decide to solve another way? | 18:53 |
| *** mylu has joined #openstack-infra | 18:53 | |
| clarkb | I guess that would be in shade? | 18:53 |
| *** kzaitsev_mb has quit IRC | 18:53 | |
| mordred | clarkb: https://review.openstack.org/#/c/300598/ - but I need to rework it to use the API I just introduced in the occ change | 18:54 |
| mordred | (and to address jeblair review comments) | 18:54 |
| mordred | clarkb: I'm about to get on a plane, so I plan to do that patch rework when I become airborn | 18:54 |
| clarkb | ok | 18:54 |
| *** mylu has quit IRC | 18:55 | |
| *** mixos has joined #openstack-infra | 18:55 | |
| *** ddieterly[away] is now known as ddieterly | 18:56 | |
| *** mixos has quit IRC | 18:56 | |
| anteaya | mordred: happy planing | 18:57 |
| *** annegentle has quit IRC | 18:57 | |
| *** angdraug has joined #openstack-infra | 18:57 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:58 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:58 | |
| *** bkeller` has quit IRC | 18:59 | |
| *** bkeller` has joined #openstack-infra | 19:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Turn off vexxhost in nodepool https://review.openstack.org/301206 | 19:02 |
| *** vgridnev has joined #openstack-infra | 19:02 | |
| *** hashar has joined #openstack-infra | 19:02 | |
| *** mylu has joined #openstack-infra | 19:03 | |
| rossella_s | clarkb, thanks a lot for reviewing! | 19:07 |
| *** kzaitsev_mb has joined #openstack-infra | 19:08 | |
| clarkb | anteaya: also vexxhost is in grafana don't know if you found it or not | 19:08 |
| clarkb | you have to use the drop down menu to get it | 19:09 |
| *** salv-orlando has joined #openstack-infra | 19:09 | |
| anteaya | clarkb: drop down menu, thank you | 19:09 |
| * anteaya looks again | 19:09 | |
| anteaya | ah there it is, thank you | 19:09 |
| anteaya | that helps me | 19:10 |
| *** mdenny has quit IRC | 19:10 | |
| anteaya | until either me or someone else can offer a patch to the home page | 19:10 |
| *** flepied has joined #openstack-infra | 19:11 | |
| *** sdake_ has joined #openstack-infra | 19:12 | |
| *** annegentle has joined #openstack-infra | 19:13 | |
| *** salv-orlando has quit IRC | 19:14 | |
| *** salv-orlando has joined #openstack-infra | 19:14 | |
| *** sdake has quit IRC | 19:15 | |
| *** imcsk8 is now known as imcsk8|lunch | 19:15 | |
| *** salv-orlando has quit IRC | 19:16 | |
| *** netsin has joined #openstack-infra | 19:16 | |
| *** salv-orlando has joined #openstack-infra | 19:16 | |
| *** salv-orlando has quit IRC | 19:17 | |
| *** ociuhandu has quit IRC | 19:17 | |
| *** salv-orlando has joined #openstack-infra | 19:17 | |
| *** EricGonczer_ has quit IRC | 19:18 | |
| clarkb | fungi: the traceroute package change is about to merge I can work on new image builds if you like (or we can just wait for tomorrow to build them) | 19:18 |
| clarkb | I guess start with rax-ord | 19:19 |
| openstackgerrit | Merged openstack-infra/project-config: Add traceroute to default preinstalled packageset https://review.openstack.org/301272 | 19:19 |
| pabelanger | clarkb: actually, looks like we are out of HDD for dibs | 19:20 |
| clarkb | pabelanger: oh fun, not entirely surprising | 19:20 |
| pabelanger | failures both today and yesterday | 19:20 |
| *** nadya has joined #openstack-infra | 19:20 | |
| pabelanger | /me nods | 19:20 |
| clarkb | fungi: I will +2 the macro update but not approve it since I am about to eat lunch | 19:20 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:21 | |
| tosky | clarkb: hi! Mind a question about my quest for merging-with-history, if you remember? | 19:21 |
| clarkb | pabelanger: removing devstack-trusty will help quite a bit there as each image is about 50GB of disk | 19:21 |
| clarkb | tosky: sure | 19:21 |
| pabelanger | clarkb: Oh, I thought we had devstack-trusty all cleaned up | 19:22 |
| clarkb | pabelanger: 6GB qcow2, 10GB raw, 17GB vhd * 2 | 19:22 |
| clarkb | pabelanger: we did | 19:22 |
| *** mixos has joined #openstack-infra | 19:22 | |
| tosky | clarkb: so, sahara-tests-release (which I'm now part of) has the "create" permission, which I thought was enough, but I didn't find a way to create this new orphan branch remotely | 19:23 |
| clarkb | pabelanger: but that doens't remove trusty stuff from the cache just the disk images, so when we added in xenail the cache got bigger overall | 19:23 |
| clarkb | tosky: you have to use the gerrit ui | 19:23 |
| tosky | clarkb: also for orphan branches? | 19:23 |
| clarkb | tosky: for any brnach | 19:23 |
| clarkb | (you need force push to use git directly and we don't give that out) | 19:23 |
| tosky | clarkb: oh, I see, let me check (I was granted the permissions few hours ago) | 19:24 |
| *** yamahata has quit IRC | 19:24 | |
| *** mrmartin has joined #openstack-infra | 19:25 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: New library split from Magnum - python-k8sclient https://review.openstack.org/301302 | 19:25 |
| *** mylu has quit IRC | 19:28 | |
| tosky | clarkb: what happens if I don't specify the "Initial Revision:"? Will master be used, as it seems it happened earlier to the PTL trying to do the same? | 19:30 |
| tosky | from https://gerrit-review.googlesource.com/Documentation/cmd-create-branch.html (which I guess maps to the web ui command) it seems you need a base branch somehow | 19:30 |
| clarkb | you should be able to use a sha1 | 19:31 |
| clarkb | like 000000 | 19:31 |
| clarkb | or another base commit that makes more sense | 19:31 |
| *** Hal has joined #openstack-infra | 19:32 | |
| *** Hal is now known as Guest6437 | 19:32 | |
| tosky | the idea of an orphan branch is no parent commit - but let's 00000 | 19:32 |
| clarkb | yes the first commit in any git repo has a parent of 000000 | 19:33 |
| *** slogan has joined #openstack-infra | 19:33 | |
| tosky | 400 invalid revision "000000" | 19:33 |
| clarkb | bah gerrit why do you hate me | 19:33 |
| *** derekh has quit IRC | 19:34 | |
| *** mylu has joined #openstack-infra | 19:34 | |
| clarkb | will it take a full sha1 of zeros? it might be conservative with the short sha | 19:34 |
| slogan | quick question: How do I contribute code to a project I'm not a part of, git clone, make changes, git add/commit, git review enogut, or do I have to have myself added to a list for that project? I'm a driver on another project, and considering contributing elsewhere, but not sure what the process is - not super clear in the docs. | 19:35 |
| openstackgerrit | Merged openstack-infra/project-config: Switch gate-infra-puppet-apply-centos-7 voting https://review.openstack.org/301239 | 19:35 |
| openstackgerrit | Merged openstack/diskimage-builder: ironic-agent postinst fails on systemd with no iptables https://review.openstack.org/295597 | 19:35 |
| clarkb | slogan: you just write and push the code as you have described | 19:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate to gate-openstackci-beaker-centos-7 https://review.openstack.org/301304 | 19:36 |
| tosky | clarkb: tried 0000000000000000000000000000000000000000, which should be the right size, no luck | 19:36 |
| fungi | clarkb: thanks, i'll go ahead and approve the macro change and then babysit jjb updates since it looks like it will change 490 jobs | 19:36 |
| slogan | clarkb: thanks | 19:36 |
| clarkb | tosky: zaro may have better ideas but sounds like gerrit may not do what we need :( | 19:37 |
| tosky | clarkb: and wouldn't one of the almighty admin do it for me? *puppy eyes* | 19:37 |
| clarkb | tosky may need a gerrit admin yo force push an initial commit :( | 19:38 |
| tosky | clarkb: how do I identify the gerrit admins? | 19:38 |
| *** matrohon has quit IRC | 19:38 | |
| *** Sukhdev has quit IRC | 19:38 | |
| *** cloudtrainme has joined #openstack-infra | 19:39 | |
| clarkb | tosky: there us a gerrit group called administrators which you can lost membership on iirc | 19:39 |
| clarkb | it should be the samr as infra root | 19:39 |
| tosky | lost membership? | 19:39 |
| clarkb | list I mean | 19:40 |
| tosky | ah! | 19:40 |
| anteaya | I have to go uptown for a meeting, probably be offline the rest of the day | 19:40 |
| fungi | have a good afernoon, anteaya | 19:40 |
| *** weshay has joined #openstack-infra | 19:40 | |
| *** nikhil_k is now known as nikhil | 19:41 | |
| *** ams_ has joined #openstack-infra | 19:41 | |
| *** piet has quit IRC | 19:41 | |
| tosky | clarkb: there is no administrators, but I guess I can check infra-core | 19:41 |
| clarkb | zaro: do you know if gerrit supports creating an orphan branch via the web ui? | 19:43 |
| *** exploreshaifali has quit IRC | 19:44 | |
| fungi | wow, getting massive packet loss on my v6 tunnel suddenly | 19:44 |
| *** mylu has quit IRC | 19:44 | |
| fungi | massive packet loss to the v4 endpoint in he | 19:46 |
| fungi | so that prolly 'splains it | 19:46 |
| zaro | sorry, i do not. i think only way is to force push a new branch. | 19:46 |
| smcginnis | Is it possible to get rights to a channel to do things like ban folks? | 19:47 |
| *** nt has quit IRC | 19:47 | |
| *** mylu has joined #openstack-infra | 19:48 | |
| clarkb | pabelanger: "Job gate-puppet-openstackci-puppet-beaker-rspec-devstack-centos7 not defined" from the puppet switch change | 19:48 |
| *** EricGonczer_ has joined #openstack-infra | 19:48 | |
| clarkb | pabelanger: I think the streams crossed between that change and the other change | 19:48 |
| clarkb | smcginnis: yes we can and do give individuals op perms in the various channels | 19:49 |
| clarkb | I believe the statusbot is managing that? let me dig up the config file | 19:49 |
| smcginnis | clarkb: Cool. Any chance I can get that for #openstack-cinder? | 19:49 |
| tosky | zaro: which is something not allowed, I guess? | 19:49 |
| clarkb | accessbot | 19:49 |
| *** mylu has quit IRC | 19:49 | |
| fungi | clarkb: smcginnis: for individual chanops on specific channels we just add you via chanserv | 19:50 |
| openstackgerrit | Merged openstack-infra/project-config: Traceroute to git.o.o in net-info macro https://review.openstack.org/301281 | 19:50 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/accessbot/channels.yaml is the config file though doesn't look like we manage the individual channel opers there, I guess that others have ya ok that | 19:50 |
| clarkb | fungi: is there a reason it isn't done via the config that does it anywyas? | 19:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate puppet jobs to centos-7 dib https://review.openstack.org/301251 | 19:50 |
| pabelanger | clarkb: Ya, that should fix it^ | 19:50 |
| fungi | clarkb: precedent i guess? i don't think anyone spent the time to look them all up and add them to the config | 19:50 |
| pabelanger | clarkb: it was a missed puppet job | 19:51 |
| fungi | clarkb: but also that config file might get a bunch more complex | 19:51 |
| *** jcoufal has quit IRC | 19:51 | |
| smcginnis | clarkb, fungi: So it's just a manual admin change for now? | 19:51 |
| fungi | given just about ever channel has at least one additional operator (the founder) if we didn't create it | 19:51 |
| fungi | smcginnis: right, one of us with admin access to the channel just adds you as a channel operator on that channel | 19:52 |
| fungi | clarkb: were you doing that, or shall i? | 19:52 |
| clarkb | fungi: go for it, trying to review centos-7 things | 19:52 |
| fungi | k | 19:52 |
| fungi | on it | 19:52 |
| smcginnis | clarkb, fungi: Thanks! | 19:53 |
| openstackgerrit | Roland Hochmuth proposed openstack-infra/project-config: Add openstack/monasca-transform project https://review.openstack.org/300742 | 19:53 |
| clarkb | looks like jgriffith is the original founder? | 19:54 |
| clarkb | pabelanger: are there any other jobs that need switching that you are aware of? or is the proposed change set complete? | 19:54 |
| zaro | tosky: actually. i don't think you can do it at all. i see the create-branch rest endpoint requires a valid revision. | 19:54 |
| clarkb | zaro: 000000 is a valid revsion | 19:54 |
| clarkb | zaro: it is the parent revision of the first commit on a branch | 19:55 |
| marcusvrn_ | Hi all! I want to setup a zuul monitoring dashboard (like openstack zuul status) for my CI. Can someone help me? | 19:55 |
| *** nt has joined #openstack-infra | 19:55 | |
| *** amitgandhinz has quit IRC | 19:55 | |
| fungi | smcginnis: should be all set now | 19:55 |
| pabelanger | clarkb: kolla is the last big change. Then a few 1 lines we can lump together | 19:56 |
| zaro | tosky, clarkb : so have you tried with ssh or rest endpoint? | 19:56 |
| clarkb | zaro: just the web ui | 19:56 |
| *** _ari_ has quit IRC | 19:56 | |
| pabelanger | clarkb: I was holding off on that since I wasn't sure if they did their mitaka release yet | 19:57 |
| clarkb | pabelanger: that seems reasonable, I think there is mail about that to the list let me catch up | 19:57 |
| *** jaosorior has quit IRC | 19:57 | |
| smcginnis | fungi: Thanks! | 19:57 |
| pabelanger | clarkb: I can prep the change now | 19:57 |
| *** jsavak has quit IRC | 19:57 | |
| zaro | tosky: i would try 0000000 with ssh command https://gerrit-review.googlesource.com/Documentation/cmd-create-branch.html | 19:57 |
| clarkb | pabelanger: looks like rc3 is recent | 19:57 |
| tosky | zaro: let me try | 19:58 |
| clarkb | pabelanger: but no proper release yet so holding off seems like a good idea for now | 19:58 |
| *** tonytan4ever has quit IRC | 19:58 | |
| fungi | clarkb: any reason to keep the "old" precise-based etherpad.openstack.org around? | 19:59 |
| fungi | seeing what i can clean up to free up some quota for us | 19:59 |
| *** amitgandhinz has joined #openstack-infra | 19:59 | |
| *** mylu has joined #openstack-infra | 19:59 | |
| jgriffith | clarkb: yeah, but I think my perms were turned off when for channel at some point... anyway; sounds like smcginnis has it worked out | 19:59 |
| clarkb | fungi: not that I can think of | 20:00 |
| openstackgerrit | Roland Hochmuth proposed openstack-infra/project-config: Add openstack/monasca-transform project https://review.openstack.org/300742 | 20:00 |
| *** tqtran is now known as tqtran-afk | 20:00 | |
| fungi | clarkb: thanks, BALEETED! | 20:00 |
| tosky | zaro: fatal: invalid revision "0000000" (same with 0000000000000000000000000000000000000000) | 20:00 |
| *** bkeller` has quit IRC | 20:00 | |
| *** christiana has quit IRC | 20:02 | |
| zaro | hmm, clarkb : how can we verify that 000000 is a valid revision? | 20:03 |
| clarkb | zaro: look at the first commit in any repo | 20:03 |
| clarkb | zaro: the parent of that ocmmit should be 000000 | 20:03 |
| clarkb | zaro: so it isn't a normal revision but it is valid for the purposes of seeding branches | 20:04 |
| tosky | but it's not an existing commit, so maybe git handles this in a special way | 20:04 |
| tosky | but if gerrit looks into the list of objects which are commits, it won't find it | 20:04 |
| clarkb | (because every commit in gerrit must hvae a parent you need to do something) | 20:04 |
| fungi | yeah, for example git show on system-config shows the first commit with a diff between revisions at index 0000000..a908c6e | 20:05 |
| *** bkeller` has joined #openstack-infra | 20:05 | |
| fungi | et cetera | 20:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate kolla jobs to centos-7 dib https://review.openstack.org/301320 | 20:05 |
| fungi | huh, though actually git rev-list --parents shows the first commit as its own parent | 20:06 |
| fungi | also git cat-file -p on the initial revision doesn't show a parent | 20:08 |
| clarkb | huh git log does though | 20:08 |
| fungi | so the 0 commit in git is indeed so very special you can't really get git tools to show you it's there, i guess | 20:08 |
| clarkb | sounds like gerrit just needs to spceial case it too and probably doesn't today | 20:09 |
| clarkb | "create a new branch with no parent" | 20:09 |
| *** esikachev has joined #openstack-infra | 20:09 | |
| *** jsavak has joined #openstack-infra | 20:10 | |
| *** mrmartin has quit IRC | 20:10 | |
| *** e0ne has quit IRC | 20:11 | |
| tosky | but then how are new projects created? Different code? | 20:11 |
| clarkb | yes gerrit has a create project command | 20:11 |
| clarkb | which likely special cases this | 20:11 |
| tosky | uhm | 20:12 |
| tosky | so back to square one, if gerrit can't do it with its command, could this still be handled by force pushing for people who have the access? | 20:13 |
| *** Goneri has quit IRC | 20:13 | |
| *** sdake_ is now known as sdake | 20:14 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Remove group regex on .testr.conf https://review.openstack.org/301324 | 20:14 |
| tosky | or should I simply forget about this (well, I could use the first commit of the project, which is empty) | 20:14 |
| *** vgridnev has quit IRC | 20:15 | |
| clarkb | if you point infra root at the code that should be pushed I am sure we can get to it at some point, but I don't think it will be a priority. Can SergeyLukjanov do it (covers both sides of the equation?) | 20:16 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:17 | |
| *** shashank_hegde has quit IRC | 20:18 | |
| *** shashank_hegde has joined #openstack-infra | 20:18 | |
| tosky | maybe yes - let's see if someone else would like to comment too | 20:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate diskimage-builder tests to centos-7 https://review.openstack.org/301328 | 20:19 |
| *** esikachev has quit IRC | 20:22 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate anvil to centos-7 https://review.openstack.org/301329 | 20:22 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Disable openstackid.org by UUID before rebuilding https://review.openstack.org/301330 | 20:22 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Declare redis 2.8.4 on openstackid.org https://review.openstack.org/301331 | 20:22 |
| *** baoli_ has quit IRC | 20:23 | |
| *** ZZelle_ has joined #openstack-infra | 20:24 | |
| pabelanger | clarkb: fungi: So, rally is still using bare-precise. We should find out why and move them to ubuntu-trusty if possible | 20:24 |
| pabelanger | seems like a basic install check | 20:24 |
| fungi | mordred: does 301330 work the way i'm trying to do that? | 20:24 |
| *** akshai has quit IRC | 20:25 | |
| SergeyLukjanov | clarkb, tosky yeah, i can handle it as infra-root for sure. | 20:26 |
| SergeyLukjanov | tosky what's the current approach selected? | 20:26 |
| *** edmondsw has quit IRC | 20:26 | |
| tosky | SergeyLukjanov: something like: | 20:26 |
| tosky | (moment) | 20:26 |
| fungi | jgriffith: fwiw i see you as having channel operator permissions on #openstack-cinder too, so it should have been working for you | 20:27 |
| tosky | SergeyLukjanov: http://paste.openstack.org/show/492923/ | 20:28 |
| fungi | jgriffith: /msg chanserv access #openstack-cinder list | 20:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate rally to centos-7 https://review.openstack.org/301334 | 20:28 |
| pabelanger | fungi: nibalizer: So, to late for today, but I'd like to see about migrating puppetdb.o.o to ubuntu-trusty this week. Just making sure you're both cool with it, and it will be safe to have 2 puppetboard.o.o servers online when we start | 20:28 |
| fungi | pabelanger: seems fine, i don't see an issue with it | 20:29 |
| SergeyLukjanov | tosky and then create a merge commit from orphan branch to master? | 20:29 |
| *** ddieterly is now known as ddieterly[away] | 20:30 | |
| tosky | SergeyLukjanov: first I need to push the commits - but that I should be able to handle, sahara-tests-release has the permissions | 20:30 |
| clarkb | pabelanger: I notice that our raw images are no longer being smaller due to the block freeing | 20:31 |
| SergeyLukjanov | clarkb if you're ok with http://paste.openstack.org/show/492923/ I'll work with tosky to make it happen | 20:31 |
| clarkb | or wait I have to ocmpare du -hs to ls -lh | 20:31 |
| SamYaple | clarkb: you mean the fstrim stuff? | 20:31 |
| *** Sukhdev has joined #openstack-infra | 20:31 | |
| clarkb | I take that back they are still smaller on disk | 20:31 |
| SergeyLukjanov | tosky I think we can push the branch with all needed code | 20:31 |
| fungi | pabelanger: the main challenge right now is around quota. we're mid-replacement on multiple large servers at the moment so memory quota is getting tight | 20:31 |
| clarkb | SamYaple: ya its working I just forgot that ls shows you the apparent size and du shows you disk use | 20:31 |
| clarkb | SamYaple: du is showing 11GB vs 17/18GB | 20:31 |
| tosky | SergeyLukjanov: oh, then you can see it from my work github repository | 20:31 |
| jgriffith | fungi: hmm... I was mistaken, thanks! | 20:31 |
| SergeyLukjanov | tosky yup | 20:31 |
| pabelanger | fungi: I don't have to start now, I figured it would be a trivial upgrade to do. | 20:32 |
| *** Sukhdev has quit IRC | 20:32 | |
| tosky | SergeyLukjanov: my master branch here: https://github.com/ltoscano-rh/sahara-tests - you can check from the history graph | 20:32 |
| *** elo has quit IRC | 20:32 | |
| fungi | pabelanger: also that one's pretty small, so might not pose a challenge | 20:33 |
| *** elo has joined #openstack-infra | 20:33 | |
| SamYaple | clarkb: btw i figured out how to do the multinode with the current infrastructure where the vms get internet access | 20:33 |
| pabelanger | clarkb: how do we fix that? | 20:33 |
| *** tphummel has quit IRC | 20:34 | |
| tosky | SergeyLukjanov: but my master branch was outdated, so either I should update it or you can take the content of that tempest-sahara-api and do the merge | 20:34 |
| tosky | you know :) | 20:34 |
| *** sdake_ has joined #openstack-infra | 20:34 | |
| *** sdake has quit IRC | 20:34 | |
| SergeyLukjanov | so, I think we can push content of tempest-sahara-api branch (just extraction of code from sahara master) to the sahara-tests branch tempest-sahara-api and then do appropriate work on this branch as CR and then create a merge commit to master. what do you think? | 20:35 |
| openstackgerrit | Steve Lewis (stevelle) proposed openstack-infra/project-config: Add os_gnocchi role project for OSA https://review.openstack.org/301340 | 20:35 |
| *** ddieterly[away] is now known as ddieterly | 20:35 | |
| *** tqtran-afk is now known as tqtran | 20:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Experimental gate-rally-install-ubuntu-trusty job https://review.openstack.org/301341 | 20:37 |
| *** tphummel has joined #openstack-infra | 20:37 | |
| tosky | SergeyLukjanov: I was thinking about merging the code as is (it does not conflict with the existing code), then move back to the normal working mode and do the normal reviews which adapts it | 20:37 |
| SergeyLukjanov | tosky I agree, it's even better | 20:37 |
| tosky | just remember that the available tempest-sahara-api is not an orphan branch, so it should be removed first | 20:38 |
| tosky | or another name should be used | 20:38 |
| SergeyLukjanov | tosky, so, push tempest-sahara-api (with just tempest/api/data_processing folder) from your repo to sahara-tests, create merge CR to master, merge it, delete branch tempest-sahara-api | 20:39 |
| tosky | SergeyLukjanov: yes | 20:39 |
| SergeyLukjanov | tosky cool, I think it's safe enough | 20:39 |
| tosky | yeah, I think so :) | 20:39 |
| SergeyLukjanov | tosky is tempest-sahara-api branch in your repo up to date? | 20:40 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build openstack-pkg-tools https://review.openstack.org/294022 | 20:40 |
| clarkb | pabelanger: I have freed 63GB on nodepool.o.o:/opt. I see part of the problem is that the vhd conversion requires 2x the image size of free space | 20:40 |
| SergeyLukjanov | tosky and let's switch back to openstack-sahara to not spam infra channel ;) | 20:40 |
| tosky | SergeyLukjanov: I don't think there have been other commits, let me quickly check | 20:40 |
| clarkb | pabelanger: we may want to optimize that and always do vhd conversion first so that we don't have a qcow2 floating around taking up the extra disk space | 20:41 |
| Shrews | ~/b 7 | 20:41 |
| Shrews | doh | 20:41 |
| clarkb | pabelanger: there was a stale old image that nodepool didn't know about and a failed vhd conversion on the most recent image on disk | 20:42 |
| pabelanger | clarkb: great | 20:42 |
| *** mylu has quit IRC | 20:43 | |
| jeblair | oh wow | 20:43 |
| jeblair | the summit party is a lead generation engine | 20:43 |
| jeblair | "The sponsors of StackCity Austin will receive all of the info you provide when registering." | 20:43 |
| fungi | that's frightening | 20:43 |
| jeblair | "All information entered into the StackCity Austin event registration page (Eventbrite) will be provided to the event sponsors. You can view a list of StackCity Austin sponsors here: https://www.openstack.org/summit/austin-2016/sponsors/#sponsors. You are not required to attend StackCity Austin. If you do not want your information to be shared with the event sponsors, then please do not register. " | 20:44 |
| jeblair | that's in the waiver you must complete to register | 20:44 |
| fungi | yep, i certainly won't be registering for that one | 20:44 |
| *** mylu has joined #openstack-infra | 20:44 | |
| jeblair | in unrelated news, my Tuesday, April 26 is free. | 20:44 |
| fungi | :/ | 20:44 |
| jeblair | fungi: grab some beer then? :) | 20:44 |
| fungi | jeblair: i'm kinda planning to give most of the vendor parties a miss anyway. there are apparently awesome bars/bbq all over the place anyway | 20:45 |
| clarkb | I registered and seem to recall it was optional to do that? | 20:45 |
| clarkb | maybe I misread it | 20:45 |
| *** dprince has quit IRC | 20:46 | |
| jeblair | clarkb: name, email, title, org, and waiver (including the above statement) are all required fields according to the little red asterisks | 20:46 |
| *** tphummel has quit IRC | 20:47 | |
| fungi | of course, i suppose you could just register with a temporary throwaway address and then delete it as soon as you have your confirmation | 20:47 |
| clarkb | I might also have mixed up various things | 20:47 |
| clarkb | I swear one of them was "share contact info with organizers" and acheck box | 20:47 |
| jeblair | clarkb: maybe they changed it? | 20:48 |
| jeblair | ("hey, no one is choosing to share contact information!" "i know how to fix that!") | 20:48 |
| clarkb | ha maybe | 20:48 |
| fungi | and also i can't imagine they're closely vetting titles and whatnot. i usually put something like "sentient lifeform" or "earthling" | 20:48 |
| clarkb | pabelanger: then when we rmeove devstack-centos7 we should free up another 100GB or so | 20:49 |
| clarkb | er thats bad math | 20:50 |
| *** nadya has quit IRC | 20:50 | |
| clarkb | 60GB orso | 20:50 |
| *** annegentle has quit IRC | 20:50 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Revert "[sahara-tests] create and merge, disable tests" https://review.openstack.org/299652 | 20:51 |
| fungi | bad math, no mantissa! | 20:51 |
| clarkb | fungi: should I start an ubuntu-trusty build to pick up the traceroute changes? this way I/we can omonitor the build and see if 63GB is plenty for it to complete | 20:51 |
| clarkb | (I think it iwll be very very close) | 20:51 |
| *** sc68cal has joined #openstack-infra | 20:51 | |
| clarkb | due to the vdh conversion overhead | 20:52 |
| clarkb | its 17gb * 3 (raw copy that dib is using as source, vhd intermediate, vhd actual) + qcow 2 | 20:52 |
| fungi | clarkb: sounds good to me | 20:52 |
| clarkb | it is building | 20:52 |
| *** hichihara has joined #openstack-infra | 20:53 | |
| *** sc68cal has quit IRC | 20:53 | |
| *** tphummel has joined #openstack-infra | 20:55 | |
| *** tphummel has quit IRC | 20:56 | |
| *** sc68cal has joined #openstack-infra | 20:57 | |
| *** annegentle has joined #openstack-infra | 20:57 | |
| *** ihrachys has joined #openstack-infra | 20:57 | |
| clarkb | pabelanger: looks like the newer centos-7 changes are failing | 20:58 |
| *** akshai has joined #openstack-infra | 20:58 | |
| *** esikachev has joined #openstack-infra | 20:59 | |
| *** tphummel has joined #openstack-infra | 20:59 | |
| clarkb | fungi: pabelanger if this fails the other thing we can try doing is deleting our dib cache and rebuilding it from scratch since that cache is not cleaned up at all | 20:59 |
| clarkb | meaning it just grows and grows and we likely have old stuff in there that can go away | 20:59 |
| fungi | yeah, we can "expire" it, but that will make the next builds probably a good deal slower | 21:00 |
| clarkb | yup but thats mostly fine | 21:00 |
| *** hashar has quit IRC | 21:00 | |
| *** jsavak has quit IRC | 21:01 | |
| clarkb | I am guessing there is old fedora release stuff and precise stuff in there that can just go away | 21:01 |
| *** kgiusti has left #openstack-infra | 21:01 | |
| clarkb | not sure how hard to do manual surgery on it would be but deleting it entirely should be fine | 21:01 |
| *** jsavak has joined #openstack-infra | 21:01 | |
| *** Sukhdev has joined #openstack-infra | 21:02 | |
| clarkb | ya there are precise and fedora 21/22 things I see | 21:02 |
| clarkb | and centos6 | 21:02 |
| *** esikachev has quit IRC | 21:02 | |
| *** rossella_s has quit IRC | 21:02 | |
| *** vgridnev has joined #openstack-infra | 21:03 | |
| clarkb | I am running a du to get a feel for distribution of data in the cache then can attempt surgery on the ase images at least | 21:03 |
| *** rossella_s has joined #openstack-infra | 21:03 | |
| *** amitgandhinz has quit IRC | 21:03 | |
| clarkb | CentOS-6-x86_64-GenericCloud-20141129_01.qcow2 can go away | 21:03 |
| *** esikachev has joined #openstack-infra | 21:03 | |
| clarkb | it is 1.GB | 21:03 |
| *** asettle has joined #openstack-infra | 21:04 | |
| clarkb | pabelanger: Job gate-dib-dsvm-functests-devstack-centos7 not defined I think you didn't rename all of them in the zuul layout file | 21:04 |
| *** thorst has quit IRC | 21:04 | |
| *** jamesmcarthur has quit IRC | 21:04 | |
| *** thorst has joined #openstack-infra | 21:05 | |
| clarkb | http://paste.openstack.org/show/492930/ | 21:05 |
| clarkb | maybe we should gc the git repos in that cache? | 21:05 |
| nibalizer | pabelanger: ya i like it | 21:06 |
| *** ociuhandu has joined #openstack-infra | 21:06 | |
| fungi | clarkb: i wonder if that will make our images smaller too (or do we pull into the cache and then clone from the cache into the image?) | 21:06 |
| clarkb | fungi: I think it does a clone into the image | 21:06 |
| *** thorst_ has joined #openstack-infra | 21:07 | |
| fungi | okay, so image sizes likely won't improve, but yeah might help performance and some space on the server | 21:07 |
| clarkb | its probably worth removing the cache and having it rebuild just to see (and to fliush out things we know we don't need) | 21:07 |
| clarkb | but the vast majority of our disk use here is the resulting images | 21:08 |
| *** thiagop has quit IRC | 21:09 | |
| pabelanger | clarkb: ya, working on it now | 21:09 |
| zigo | Do I need special rights to be able to edit the -infra agenda at https://wiki.openstack.org/w/index.php?title=Meetings/InfraTeamMeeting? | 21:10 |
| pleia2 | sorry for being absent today, sick, and going back to tend to that | 21:10 |
| pleia2 | zigo: nope | 21:10 |
| zigo | :/ | 21:10 |
| clarkb | pleia2: get better | 21:10 |
| *** thorst has quit IRC | 21:10 | |
| zigo | When I click on "save", it doesn't seem to be saving anything. | 21:10 |
| pleia2 | zigo: did you fill out the captcha? | 21:10 |
| pabelanger | pleia2: np, rest up | 21:10 |
| zigo | Oh ! | 21:10 |
| zigo | pleia2: Cheers. | 21:10 |
| zigo | pleia2: If the captcha was next to the save button, I'd have notice... | 21:11 |
| *** thorst_ has quit IRC | 21:11 | |
| pleia2 | zigo: yeah :\ | 21:11 |
| fungi | zigo: yeah, hoping to upgrade to a newer mediawiki soon and use better captcha plugin and other spam mitigation solutions available at that point | 21:11 |
| zigo | I wasn't complaining, I was trying to find myself an excuse for being lame! :) | 21:13 |
| *** jamesmcarthur has joined #openstack-infra | 21:13 | |
| *** daemontool has joined #openstack-infra | 21:13 | |
| fungi | zigo: i didn't think you were complaining, i just wanted to explain the reason for the sub-optimal captcha implementation we're on now | 21:13 |
| zigo | fungi: Any chance I may bother you for a quick look? https://review.openstack.org/#/c/294022/ | 21:14 |
| *** sdake_ has quit IRC | 21:14 | |
| *** dimtruck has quit IRC | 21:16 | |
| *** ldnunes has quit IRC | 21:16 | |
| clarkb | fungi: it is interesting how much smaller the apt cache is htan the yum cache | 21:16 |
| clarkb | jessie + precise + trusty + xenial vs fedora 21/22/23 + centos 6/7 | 21:16 |
| zigo | clarkb: You pre-populate /var/cache/apt with all of that? | 21:16 |
| *** |-paul-| has joined #openstack-infra | 21:16 | |
| clarkb | oh doesn't look like jessie is caching | 21:16 |
| pabelanger | clarkb: I'm gate for deleting cache. | 21:16 |
| clarkb | zigo: yes | 21:16 |
| zigo | I'd use approx for doing it. | 21:16 |
| clarkb | zigo: its built into dib so we use whatever they use | 21:16 |
| zigo | Ok. | 21:16 |
| *** trown is now known as trown|outtypewww | 21:16 | |
| clarkb | I am just comparing the size of that cache against the size of the corresponding yum cache | 21:16 |
| *** dimtruck has joined #openstack-infra | 21:16 | |
| * zigo will think about implementing an element for approx then. | 21:16 | |
| clarkb | and ya jesie doesn't seem to be caching | 21:16 |
| pabelanger | Ya, yum was 3.7GB when I checked | 21:17 |
| openstackgerrit | Boris Pavlovic proposed openstack/requirements: Bump alembic version to 0.8.4 https://review.openstack.org/301108 | 21:18 |
| clarkb | pabelanger: boris-42 ^ is the person to ask about rally and bare-precise | 21:19 |
| pabelanger | clarkb: Ya, I posted my questions in openstack-rally too | 21:19 |
| boris-42 | clarkb: hi | 21:19 |
| boris-42 | pabelanger: hi | 21:19 |
| pabelanger | boris-42: see my comments in #openstack-rally for context | 21:19 |
| nibalizer | pabelanger: so you think we just boot a trusty node, run puppet on it, and see where we get? | 21:20 |
| nibalizer | i'm wondering if we specify our puppetdb version or not | 21:20 |
| boris-42 | pabelanger: thanks for work | 21:20 |
| pabelanger | nibalizer: Ya, I tested it at midcycle at it worked then | 21:20 |
| boris-42 | pabelanger: there are no reasons | 21:20 |
| boris-42 | pabelanger: we just want to test that installation work on centos and ubuntu | 21:21 |
| pabelanger | nibalizer: so, the plan I think is, boot new node, validate puppet run | 21:21 |
| pabelanger | nibalizer: then run puppet again, and post to localhost db, check results.. .if they look good, update DNS records for new server | 21:21 |
| pabelanger | boris-42: okay, great. I added an experimental job first, so if that passes I'll change to ubuntu-trusty in a follow up | 21:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate rally to centos-7 https://review.openstack.org/301334 | 21:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate diskimage-builder tests to centos-7 https://review.openstack.org/301328 | 21:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate anvil to centos-7 https://review.openstack.org/301329 | 21:22 |
| nibalizer | pabelanger: yea i think we'll need to pin puppetboard at least | 21:22 |
| nibalizer | puppetdb can probably float | 21:22 |
| pabelanger | nibalizer: Yup, pretty sure I did that... let me check review | 21:22 |
| *** aysyd has quit IRC | 21:23 | |
| pabelanger | https://review.openstack.org/#/c/284874/ | 21:23 |
| pabelanger | nibalizer: ^ | 21:23 |
| *** claudiub|2 has joined #openstack-infra | 21:26 | |
| *** jamesmcarthur has quit IRC | 21:27 | |
| *** ddieterly is now known as ddieterly[away] | 21:28 | |
| *** thorst has joined #openstack-infra | 21:29 | |
| *** claudiub has quit IRC | 21:29 | |
| *** ddieterly[away] is now known as ddieterly | 21:30 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Create openstack/designate-tempest repository https://review.openstack.org/301219 | 21:32 |
| *** thorst has quit IRC | 21:33 | |
| *** bpokorny has quit IRC | 21:35 | |
| *** bkeller` has quit IRC | 21:35 | |
| nibalizer | pabelanger: i think we should just tag on this https://github.com/voxpupuli/puppetboard/tree/v0.0.5 | 21:36 |
| nibalizer | since thats the last version that doesn't require 3 | 21:36 |
| *** vgridnev has quit IRC | 21:36 | |
| nibalizer | puppetdb 3, i mean | 21:36 |
| pabelanger | nibalizer: sure, if you think we are safe | 21:36 |
| *** jsavak has quit IRC | 21:37 | |
| nibalizer | ya we are | 21:37 |
| pabelanger | okay, I'll update the review in a bit | 21:38 |
| nibalizer | cool | 21:38 |
| *** jsavak has joined #openstack-infra | 21:38 | |
| *** alex_xu has quit IRC | 21:39 | |
| *** eliqiao has quit IRC | 21:39 | |
| *** tiswanso has quit IRC | 21:41 | |
| *** bkeller` has joined #openstack-infra | 21:42 | |
| *** regXboi has quit IRC | 21:44 | |
| *** daemontool has quit IRC | 21:45 | |
| *** imcsk8|lunch is now known as imcsk8 | 21:45 | |
| *** eliqiao has joined #openstack-infra | 21:45 | |
| *** alex_xu has joined #openstack-infra | 21:47 | |
| *** sputnik13 has quit IRC | 21:47 | |
| pabelanger | clarkb: fungi: we should land https://review.openstack.org/#/c/299592/ just to be safe for centos-7 ansible jobs | 21:47 |
| clarkb | pabelanger: is it still easier at this point to add that to fallback rather than just to the repo in question? | 21:48 |
| clarkb | I guess each repo will need to untangle what they need out of the list but it is self testing | 21:48 |
| *** sputnik13 has joined #openstack-infra | 21:48 | |
| *** nwkarsten has quit IRC | 21:49 | |
| clarkb | fungi: ubuntu-trusty image build succeeded | 21:50 |
| pabelanger | clarkb: Ya, the openstack ansible team is fine, same with the roles I've done. It this is more for kolla I figure. I haven't looked to see what is needed to migrate them to other-requirements.txt | 21:50 |
| *** weshay has quit IRC | 21:50 | |
| clarkb | it seems that 176782 has leaked | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Bump upper constraint on os-client-config https://review.openstack.org/301364 | 21:52 |
| clarkb | I take that back, there was just lag between my df ing and the files actually deleting | 21:54 |
| clarkb | I will start uploads to osic, internap, bluebox and rax-ord | 21:54 |
| clarkb | the first 3 because they are qcow2 consumers and the last because that is where we had network issues | 21:54 |
| *** sdake has joined #openstack-infra | 21:54 | |
| *** mixos has quit IRC | 21:55 | |
| clarkb | fungi: this typically takes and hour or two for a rax upload | 21:55 |
| clarkb | fungi: fingers crossed we can enable rax-ord late today or early tomorrow | 21:56 |
| *** matt-borland has quit IRC | 21:58 | |
| *** mixos has joined #openstack-infra | 21:59 | |
| *** david-lyle_ has joined #openstack-infra | 22:02 | |
| *** david-lyle has quit IRC | 22:03 | |
| *** sdague has quit IRC | 22:03 | |
| *** david-lyle_ is now known as david-lyle | 22:03 | |
| *** jsavak has quit IRC | 22:04 | |
| *** mixos has quit IRC | 22:05 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration to move attrs and tags into columns https://review.openstack.org/298069 | 22:05 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Rework migration 1679b5bc102c to not use the db api https://review.openstack.org/301369 | 22:05 |
| clarkb | fungi: osic is done so you should start seeing traceroutes there real soon now | 22:06 |
| *** xyang1_ has quit IRC | 22:07 | |
| *** bkeller` has quit IRC | 22:07 | |
| jeblair | fungi: i think i have reproduced the situation from this morning | 22:09 |
| jeblair | (with zuul and dependency loops) | 22:09 |
| fungi | clarkb: pabelanger: is libselinux-python a python library? is it going to drag in python lib dependencies? | 22:10 |
| clarkb | fungi: https://jenkins04.openstack.org/job/gate-tempest-dsvm-neutron-full/17400/console there is a traceroute | 22:10 |
| *** eharney has quit IRC | 22:10 | |
| clarkb | fungi: oh thats a good question | 22:10 |
| openstackgerrit | Merged openstack-infra/project-config: Add libselinux-python for ansible jobs https://review.openstack.org/299592 | 22:10 |
| fungi | jeblair: wow--what turned out to be the corner case that triggers it separate from a normal loop? | 22:10 |
| jeblair | fungi: the way the loop was constructed (in terms of patchset updates) | 22:11 |
| jeblair | fungi: possibly also the fact that other dependencies were involved | 22:11 |
| jeblair | i don't fully understand it yet :) | 22:12 |
| jeblair | just managed to add in enough details to get it to break reliably | 22:12 |
| *** sc68cal has quit IRC | 22:12 | |
| *** esikachev has quit IRC | 22:13 | |
| fungi | it was certainly a strange scenario | 22:13 |
| fungi | glad it didn't entail a restart in the end | 22:14 |
| *** eric_lopez has joined #openstack-infra | 22:15 | |
| *** mixos has joined #openstack-infra | 22:15 | |
| *** elo has quit IRC | 22:16 | |
| *** nelsnelson has quit IRC | 22:16 | |
| fungi | pabelanger: clarkb: i think this means it's not depending on any other python libs so probably safe-ish https://www.rpmfind.net/linux/RPM/centos/7.2.1511/x86_64/Packages/libselinux-python-2.2.2-6.el7.x86_64.html | 22:16 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints https://review.openstack.org/301375 | 22:16 |
| fungi | the upstream url listed there for its source tracker goes to a 404 | 22:17 |
| *** mriedem is now known as mriedem_away | 22:17 | |
| pabelanger | fungi: Ya, it shouldn't. It is a weird lib that depends on rpm C library | 22:18 |
| smcginnis | I'm seeing check failures in stacking, timing out to https://git.openstack.org/openstack-dev/devstack/ | 22:18 |
| *** smarcet has quit IRC | 22:18 | |
| smcginnis | I get a blank page if I try to hit it directly/ | 22:18 |
| smcginnis | Known issue? | 22:18 |
| fungi | smcginnis: that's not a browseable url, so getting a blank page is not surprising | 22:18 |
| smcginnis | Ah | 22:19 |
| fungi | smcginnis: it's a git clone url | 22:19 |
| fungi | smcginnis: have a link to the job log? | 22:19 |
| pabelanger | https://git.openstack.org/cgit/openstack-dev/devstack/ | 22:19 |
| pabelanger | for web interface | 22:19 |
| fungi | smcginnis: there seem to be network connectivity issues in rackspace we're trying to help hunt down | 22:19 |
| smcginnis | fungi: http://logs.openstack.org/93/301293/1/check/gate-tempest-dsvm-full/b6e3035/ | 22:19 |
| smcginnis | fungi: That's probably it then. | 22:19 |
| fungi | just now in the process of adding some additional diagnostic details to the console logs | 22:19 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:20 | |
| fungi | yep, happened in rackspace | 22:20 |
| smcginnis | fungi: Cool, not critical on my end, so I'll just check back later. | 22:20 |
| fungi | but it ran too soon to add traceroute outpur | 22:20 |
| fungi | output | 22:20 |
| fungi | we've still got some image updates and job config updates in progress which are adding those | 22:20 |
| *** mixos has quit IRC | 22:21 | |
| fungi | job updates for that have completed everywhere except jenkins01, which hit a proxy error after 51 minutes. restarting the update there now | 22:21 |
| EmilienM | mhh, when ubuntu-trusty-rax was updated last time? | 22:22 |
| fungi | EmilienM: it's in the process of updating now i think--what are you seeing? | 22:22 |
| *** Goneri has joined #openstack-infra | 22:23 | |
| EmilienM | fungi: this patch for ubuntu (puppet) jobs is not in the rax image https://review.openstack.org/#/c/300700/ (but it's in some other cloud providers, ie bluebox) | 22:23 |
| clarkb | EmilienM: there are 3 rax images | 22:24 |
| clarkb | because we hvae 3 regions | 22:24 |
| EmilienM | ubuntu-trusty-rax-dfw-9312476 | 22:24 |
| clarkb | I think they are all roughly 3 days old which is when we ran out of disk | 22:24 |
| fungi | or rather, 3 rackspace regions to which we try daily to upload consistent images into | 22:24 |
| *** asettle has quit IRC | 22:24 | |
| clarkb | I have all 3 rax region uploads for ubuntu-trusty in progress now | 22:25 |
| EmilienM | that's a good news! thanks clarkb | 22:25 |
| clarkb | osic and bluebox are done | 22:25 |
| EmilienM | yeah, no failure to report on them | 22:25 |
| fungi | clarkb: in good news, the traceroute job addition this time isn't making jobs bomb out even though it is logging the lamentable lack of the tools | 22:26 |
| clarkb | and it is working where the tools exist | 22:26 |
| fungi | so i think we're good there once image uploads complete | 22:26 |
| clarkb | fungi: see link above for osic example | 22:26 |
| fungi | once ord has a fresh image, we should probably crank up the max-servers there to something small like 10 and then see what we get | 22:27 |
| *** tpsilva has quit IRC | 22:27 | |
| fungi | though as smcginnis points out, we're seeing it happen within dfw too | 22:27 |
| *** cloudtrainme has quit IRC | 22:27 | |
| clarkb | internap just finished | 22:27 |
| *** annegentle has quit IRC | 22:27 | |
| clarkb | leaving rax and ovh | 22:27 |
| fungi | just not as often as it did in ord (maybe because we booted more instances in ord? but it seems like the error rate there was disproportionately high still even then) | 22:27 |
| clarkb | have we seen any of these failures in not rax regions? if not maybe its a wan problem for rax? | 22:28 |
| *** gordc has quit IRC | 22:29 | |
| fungi | clarkb: we have not, afaik. and since we should in theory be doing v6 from vexxhost to git.o.o that probably means it's not a v6-specific issue | 22:30 |
| *** knikolla has quit IRC | 22:30 | |
| *** andymaier has joined #openstack-infra | 22:30 | |
| fungi | clarkb: e-r entry is http://status.openstack.org//elastic-recheck/#1298006 | 22:31 |
| fungi | hrm, in the last 10 days it's been 18 hits in rax-dfw, 4 in bluebox-sjc1 and 1 in ovh-gra1 | 22:32 |
| *** tmorin has joined #openstack-infra | 22:32 | |
| *** andymaier has quit IRC | 22:32 | |
| fungi | clarkb: given how many more nodes we boot in ovh-gra1 than rax-dfw, i have a feeling this is primarily an intra-rackspace connectivity issue, yes | 22:33 |
| fungi | especially silly given that rax-dfw _is_ where we also host the git farm | 22:33 |
| tmorin | I'm having a hard time understanding why a devstack vm job fails, some help would be welcome | 22:33 |
| *** mixos has joined #openstack-infra | 22:33 | |
| dougwig | tmorin: link? | 22:34 |
| tmorin | console.html : http://logs.openstack.org/60/297260/16/experimental/gate-install-dsvm-networking-bgpvpn-bagpipe/40e4c1a/console.html#_2016-04-04_22_30_44_214 | 22:34 |
| tmorin | dougwig: ^^ | 22:34 |
| *** dimtruck is now known as zz_dimtruck | 22:35 | |
| tmorin | but devstacklog.txt seems to happily end: | 22:35 |
| *** EricGonczer_ has quit IRC | 22:35 | |
| *** ddieterly is now known as ddieterly[away] | 22:35 | |
| *** sdake has quit IRC | 22:36 | |
| clarkb | http://logs.openstack.org/60/297260/16/experimental/gate-install-dsvm-networking-bgpvpn-bagpipe/40e4c1a/logs/devstacklog.txt.gz#_2016-04-04_22_30_41_799 I think that returned 2 | 22:36 |
| *** asettle has joined #openstack-infra | 22:36 | |
| dougwig | tmorin: 2016-04-04 22:30:41.799 | Waiting for bagpipe-bgp to start... | 22:36 |
| dougwig | tmorin: that's the last thing in devstacklog.txt, which is the script that likely failed. did that return non-zero? | 22:37 |
| *** markvoelker has quit IRC | 22:37 | |
| tmorin | clarkb: I wouldn't expect it to fails without talking more / saying why | 22:37 |
| tmorin | clarkb: but that sounds plausible, yes | 22:37 |
| clarkb | tmorin: I am guessing the wait just timed out and it returned 2 check your init scrpt or whatever is making the service start | 22:37 |
| tmorin | clarkb: would stack.sh output the summary lines at then end even if something fails ? | 22:38 |
| clarkb | I have no iea | 22:38 |
| clarkb | I would have to read the code | 22:38 |
| clarkb | since I have to sign like 20 something pages of shit | 22:39 |
| clarkb | gah ww | 22:39 |
| tmorin | https://github.com/Orange-OpenSource/bagpipe-bgp/blob/master/devstack/plugin.sh#L120 | 22:39 |
| dougwig | 5 minute runtime, i doubt it's timeout. but somethign is returning 2. | 22:39 |
| clarkb | house buying for you :) | 22:39 |
| tmorin | it is supposed to "die xxx" if it fails | 22:39 |
| tmorin | can't see why it would silently fail | 22:39 |
| clarkb | tmorin: huh | 22:39 |
| clarkb | tmorin: is it supposed to do anything after that though? | 22:39 |
| clarkb | seems odd it wouldn't continue if it failed there | 22:40 |
| tmorin | clarkb, yes, it's supposed to run something else | 22:40 |
| *** Guest6437 has quit IRC | 22:40 | |
| tmorin | https://github.com/Orange-OpenSource/bagpipe-bgp/blob/master/devstack/plugin.sh#L126 | 22:40 |
| clarkb | seems like it is failing at that start service portion | 22:40 |
| clarkb | if it isn't continuing to do the additional steps | 22:40 |
| dougwig | tmorin: i'd put an exit 0 at the end of that script. | 22:41 |
| tmorin | dougwig: hmm, I wouldn't think that necessary, but I'll try | 22:41 |
| tmorin | clarkb: the other service is not enabled in this gate config | 22:42 |
| clarkb | tmorin: bash will exit with the exit status of the last command run | 22:42 |
| fungi | okay, jjb updates completed successfully on all jenkins masters now | 22:42 |
| clarkb | tmorin: oh so it may actually be done there | 22:42 |
| dougwig | tmorin: i'd have to read deeper to make sure $xtrace was always 0, otherwise. | 22:42 |
| *** ybathia has joined #openstack-infra | 22:43 | |
| *** asettle has quit IRC | 22:43 | |
| tmorin | dougwig, clarkb: ok, you've helped enough to let me investigate more, that helps; I'll investigate -- many thanks | 22:44 |
| *** rebase has joined #openstack-infra | 22:44 | |
| *** david-lyle has quit IRC | 22:44 | |
| zaro | crinkle: would you be able to help with puppet change? | 22:45 |
| clarkb | tmorin: dougwig wait_for_service may not actually exit with a return code that is useable in that way | 22:45 |
| jhesketh | Morning | 22:46 |
| clarkb | the last command run in wait_for_service is time_stop which errors if the timer wasn't started but doesn't reflect the service waiting timeout return code | 22:46 |
| zaro | crinkle: why is this failing? https://review.openstack.org/#/c/300596 | 22:46 |
| fungi | jeblair: not sure if you saw the "[neutron] -2'ing all patches on every gate breakage" thread starting at http://lists.openstack.org/pipermail/openstack-dev/2016-April/091307.html but it starts to delve into discussion of "if only we had some sort of procedural blocking mechanism that wasn't code-review -2..." | 22:46 |
| *** mixos has quit IRC | 22:47 | |
| clarkb | fungi: rax-iad is done but I guess it hasn't exhibited this issue? | 22:47 |
| *** david-lyle has joined #openstack-infra | 22:47 | |
| *** |-paul-| has quit IRC | 22:47 | |
| zaro | crinkle: download object is here: https://review.openstack.org/#/c/300603/2/modules/openstack_project/manifests/review.pp | 22:47 |
| clarkb | ovh-bhs1 failed and has been restarted, ovh-gra1 is in progress | 22:47 |
| fungi | clarkb: 18 times for the past 10 days in the gate. it wasn't as rampant as rax-ord | 22:47 |
| crinkle | zaro: you want .empty? not .empty | 22:47 |
| zaro | crinkle: i want to check that it's not empty | 22:48 |
| tmorin | clarkb, dougwig: ok, I'll investigate, I'll have a look at how it's done elsewhere -- there must be a subtle difference in what we do | 22:48 |
| crinkle | zaro: right, the method to check that is called ".empty?" with a question mark http://ruby-doc.org/core-2.2.0/Hash.html#method-i-empty-3F | 22:48 |
| dougwig | fungi: it just hasn't been a big deal in practice, though. | 22:49 |
| * tmorin gtg eat somewhere... | 22:49 | |
| clarkb | tmorin: to debug you might want to just put a manual curl in there before the die to see the state of things before it bails out | 22:49 |
| fungi | dougwig: sure, it's just a topic which has come up before | 22:49 |
| dougwig | fungi: gotcha. | 22:49 |
| tmorin | clarkb: good idea, I think I'll add a lot of echo foo, echo bar, here and there | 22:49 |
| fungi | dougwig: so provides a new datapoint for another way people might want to use procedural blocking | 22:49 |
| tmorin | clarkb: :) | 22:50 |
| tmorin | bye guys ! | 22:50 |
| *** tmorin has left #openstack-infra | 22:50 | |
| *** nwkarsten has joined #openstack-infra | 22:50 | |
| zaro | crinkle: ohh wat? thanks! | 22:51 |
| crinkle | zaro: yep np :) | 22:51 |
| fungi | i will admit i too had trouble parsing the answer as originally worded | 22:51 |
| clarkb | fungi: animaniacs are on netflix now | 22:52 |
| fungi | clarkb: ooh, if i netflixed... then again i should probably just buy the series on disc anyway | 22:53 |
| fungi | it really was that good | 22:53 |
| clarkb | fungi: and I think amazon prime has pinky and the brain | 22:53 |
| fungi | also excellent, yes | 22:54 |
| *** nwkarsten has quit IRC | 22:55 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's download protocols https://review.openstack.org/300596 | 22:56 |
| pabelanger | clarkb: fungi: indeed, watch s01e01 this morning | 22:56 |
| *** mylu has quit IRC | 22:58 | |
| *** ddieterly[away] is now known as ddieterly | 22:59 | |
| *** ddieterly has quit IRC | 23:00 | |
| *** jamesmcarthur has joined #openstack-infra | 23:03 | |
| *** ams_ has quit IRC | 23:03 | |
| *** rbrndt_ has quit IRC | 23:04 | |
| *** amrith is now known as _amrith_ | 23:04 | |
| clarkb | fungi: rax-dfw new image is ready | 23:04 |
| clarkb | ord failed so retrying that one | 23:05 |
| clarkb | hrm that didn't seem to remove the record from the database | 23:06 |
| *** apoorvad has quit IRC | 23:06 | |
| clarkb | this might be a bug in the manual upload command not catching the exception and removing the db entry | 23:07 |
| *** apoorvad has joined #openstack-infra | 23:07 | |
| clarkb | I will look into that | 23:07 |
| *** apoorvad_ has joined #openstack-infra | 23:08 | |
| *** apoorvad has quit IRC | 23:08 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Make git protocol point to git.o.o https://review.openstack.org/300576 | 23:08 |
| *** alex_xu has quit IRC | 23:09 | |
| clarkb | hrm I want to say there was a fix for this already | 23:10 |
| *** alex_xu has joined #openstack-infra | 23:11 | |
| *** ihrachys has quit IRC | 23:11 | |
| openstackgerrit | Merged openstack/requirements: Blacklist cryptography 1.3.0 https://review.openstack.org/295344 | 23:12 |
| fungi | okay, this is a weird openstackci-beaker job output http://logs.openstack.org/30/301330/1/check/gate-openstackci-beaker-ubuntu-trusty/9238513/console.html | 23:14 |
| *** asettle has joined #openstack-infra | 23:14 | |
| fungi | the change that reported on went nowhere near openstackci or any modules it relies on | 23:14 |
| jeblair | 2016-04-04 20:44:53.579 | Error: /Stage[main]/Openstackci::Logserver/Package[keyring]/ensure: change from 8.7 to latest failed: Could not get latest version: Timeout while contacting pypi.python.org: Timeout::Error | 23:15 |
| jeblair | fungi: ^ | 23:15 |
| jeblair | i wish i understood what was going on there enough to figure out why it's not using our local mirror | 23:16 |
| jeblair | or rather, how to fix it so that it will | 23:16 |
| clarkb | ovh-bhs1 is done | 23:16 |
| *** akshai has quit IRC | 23:17 | |
| *** chlong has joined #openstack-infra | 23:18 | |
| *** asettle has quit IRC | 23:19 | |
| fungi | jeblair: ohhh | 23:19 |
| fungi | i missed that error message sandwiched in there, thanks | 23:19 |
| *** mylu has joined #openstack-infra | 23:19 | |
| fungi | strange that it was ultimately complaining about non-working default values for parameters | 23:20 |
| crinkle | jeblair: i tried to look at that a while ago, found that puppet isn't doing anything that changes /etc/pip.conf (https://review.openstack.org/#/c/288186/) but I don't know what else might be getting in the way | 23:21 |
| *** [1]Thelo has joined #openstack-infra | 23:21 | |
| *** salv-orlando has quit IRC | 23:21 | |
| *** ybathia has quit IRC | 23:22 | |
| *** Thelo has quit IRC | 23:23 | |
| *** [1]Thelo is now known as Thelo | 23:23 | |
| *** jamesmcarthur has quit IRC | 23:24 | |
| *** rebase_ has joined #openstack-infra | 23:25 | |
| jeblair | crinkle: i wonder if we're overwriting it with our own puppet manifests? | 23:26 |
| *** mixos has joined #openstack-infra | 23:26 | |
| crinkle | jeblair: i don't think so, you can see at the end of this puppet run it's the same http://logs.openstack.org/86/288186/2/check/gate-openstackci-beaker-trusty-dsvm/4cb7fa1/console.html#_2016-03-04_00_45_10_111 | 23:27 |
| *** rebase has quit IRC | 23:27 | |
| jeblair | crinkle: oh, did we stick a cat on there for this? | 23:28 |
| crinkle | jeblair: I did just to test it out https://review.openstack.org/#/c/288186/2/spec/acceptance/basic_spec.rb | 23:29 |
| jeblair | crinkle: this is getting interesting now :) | 23:29 |
| *** rebase_ has quit IRC | 23:29 | |
| crinkle | jeblair: i think maybe someone should hold a node and tcpdump while running puppet | 23:30 |
| clarkb | nodepools upload image command is calling the onFailed() handler I see that due to the log | 23:30 |
| jeblair | crinkle: are there any other user accounts involved in this job? (a per-user pip.conf is the only other thing i can think of right now) | 23:30 |
| crinkle | jeblair: because when i tried it locally it seemed to be talking to the mirror | 23:30 |
| clarkb | but for whatever reason it isn't deleting the snap image properly. more digging needed | 23:30 |
| crinkle | jeblair: hmm i don't know | 23:30 |
| *** mylu has quit IRC | 23:31 | |
| crinkle | jeblair: i think beaker runs as root | 23:31 |
| *** mixos has quit IRC | 23:31 | |
| *** mylu has joined #openstack-infra | 23:31 | |
| *** abregman has quit IRC | 23:31 | |
| clarkb | and it seems to only break in the command line, we have tests for the nodepool internals and they work | 23:32 |
| *** mixos has joined #openstack-infra | 23:32 | |
| jeblair | crinkle, fungi: i'm guessing we renamed gate-openstackci-beaker-trusty-dsvm to gate-openstackci-beaker-ubuntu-trusty ? | 23:33 |
| *** mixos has quit IRC | 23:33 | |
| *** ddieterly has joined #openstack-infra | 23:33 | |
| fungi | jeblair: yes, i believe so | 23:34 |
| jeblair | crinkle: have an ssh key handy? | 23:34 |
| fungi | or rather, since the jobs were named for the node type, we renamed and also moved the job | 23:34 |
| *** rfolco has quit IRC | 23:34 | |
| crinkle | jeblair: https://launchpad.net/~krinkle/+sshkeys | 23:34 |
| clarkb | I see the issue, it is a threading problem. | 23:34 |
| fungi | it was previously running on devstack-trusty nodes, now it runs on ubuntu-trusty nodes | 23:34 |
| *** mixos has joined #openstack-infra | 23:35 | |
| jeblair | crinkle: jenkins@23.253.230.161 | 23:35 |
| * crinkle goes adventuring | 23:36 | |
| jeblair | crinkle: can you sudo? | 23:36 |
| crinkle | jeblair: nope it asks for a password | 23:37 |
| *** markvoelker has joined #openstack-infra | 23:37 | |
| jeblair | crinkle: that should be fixed (you may need to login again) | 23:38 |
| *** xarses has quit IRC | 23:38 | |
| jeblair | (i wonder why a job that does things as root removes sudo permissions from jenkins) | 23:38 |
| *** dalgaaf has quit IRC | 23:38 | |
| crinkle | jeblair: looks like my ssh key was removed | 23:39 |
| jeblair | oh, it ssh's as root | 23:39 |
| *** mixos has quit IRC | 23:40 | |
| jeblair | crinkle: that's interesting... | 23:40 |
| jeblair | crinkle: re-added | 23:40 |
| crinkle | jeblair: okay it works now | 23:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Synchronize image delete with handle completion https://review.openstack.org/301401 | 23:41 |
| clarkb | I think that fixes the issue | 23:41 |
| jeblair | maybe that means that puppet ran on the host and puppeted the jenkins homedir again | 23:42 |
| clarkb | fungi: is there a change proposed to reenable rax-ord yet? The image isn't uploaded but we could WIP that change until it is | 23:43 |
| *** Qiming has joined #openstack-infra | 23:44 | |
| anteaya | I can offer that | 23:44 |
| *** _amrith_ is now known as amrith | 23:44 | |
| anteaya | it would just be a revert of my patch to turn it off | 23:45 |
| *** keedya has quit IRC | 23:45 | |
| *** mylu has quit IRC | 23:45 | |
| clarkb | anteaya: fungi suggested we start with a smaller max server count like 10 | 23:45 |
| clarkb | and kick the tires on it first | 23:45 |
| *** mylu has joined #openstack-infra | 23:45 | |
| fungi | yep, go for it anteaya--thanks! | 23:46 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Revert "Turn off rax-ord" https://review.openstack.org/301402 | 23:46 |
| fungi | and yes, we should start small because the failure rate there was high | 23:46 |
| anteaya | ah sorry well I can offer a max-server of 10 | 23:46 |
| anteaya | makes sense, sorry wasn't following along | 23:46 |
| fungi | no problem, i know you were out and about for a while | 23:47 |
| *** mylu has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack/requirements: Add microversion-parse to requirements https://review.openstack.org/300113 | 23:47 |
| clarkb | pabelanger: were did we ned up on the xenial mirror? | 23:48 |
| *** mylu has joined #openstack-infra | 23:49 | |
| clarkb | pabelanger: I see that it apperas to exist now, what was the fix? | 23:49 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Turn rax-ord on slowly, max to 10 https://review.openstack.org/301403 | 23:49 |
| *** ddieterly is now known as ddieterly[away] | 23:49 | |
| anteaya | I abandoned the other one | 23:50 |
| *** nwkarsten has joined #openstack-infra | 23:51 | |
| *** akshai has joined #openstack-infra | 23:52 | |
| *** bdemers has quit IRC | 23:54 | |
| *** asettle has joined #openstack-infra | 23:54 | |
| *** Qiming has quit IRC | 23:55 | |
| *** mylu has quit IRC | 23:55 | |
| *** bdemers has joined #openstack-infra | 23:56 | |
| *** nwkarsten has quit IRC | 23:56 | |
| *** thorst has joined #openstack-infra | 23:56 | |
| openstackgerrit | Paul Van Eck proposed openstack-infra/puppet-refstack: Add db backup cron job https://review.openstack.org/301405 | 23:56 |
| clarkb | in theory this second ord upload will go faster because the swift portion isnt needed due to shades hashing | 23:56 |
| clarkb | but it is still going | 23:56 |
| *** akshai_ has joined #openstack-infra | 23:58 | |
| *** mylu has joined #openstack-infra | 23:58 | |
| *** rhochmuth has quit IRC | 23:58 | |
| *** angdraug has quit IRC | 23:58 | |
| *** ybathia has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!