| *** signed8bit_Zzz is now known as signed8bit | 00:00 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Sign artifacts before we release them https://review.openstack.org/496428 | 00:00 |
|---|---|---|
| pabelanger | mordred: jeblair: clarkb: I've updated 496428^ for gpg keysiging of our artifacts for testpypi | 00:01 |
| clarkb | also there is unmerged edit to private common.yaml | 00:03 |
| clarkb | fungi: ^ looks like it may hvae been your edit | 00:03 |
| clarkb | I wonder if we'll have to restart the entire cloud to get python daemons to pick that change up | 00:03 |
| clarkb | I guess we'll find out when we get there :) | 00:03 |
| *** bnemec has joined #openstack-infra | 00:04 | |
| pabelanger | good ops experience :) | 00:04 |
| clarkb | I'm inclined to defer this until tomorrow (in part because I need to go sort out dinner :) ) | 00:04 |
| clarkb | jeblair: is it urgent to get puppet updates working now? | 00:04 |
| jeblair | clarkb: not for me; i reordered the change that brought this to my attention | 00:05 |
| jeblair | long as we get it sorted out in the next day or so i think | 00:05 |
| clarkb | ok, worst case we can pull that cloud out of the clouds.yaml file | 00:06 |
| clarkb | but I can pick this up first thing in the morning | 00:06 |
| jeblair | i think it's also affecting nodepool, so probably best to press ahead, but not urgently | 00:06 |
| *** ijw has quit IRC | 00:07 | |
| *** funzo has joined #openstack-infra | 00:07 | |
| *** ijw has joined #openstack-infra | 00:07 | |
| *** ijw has quit IRC | 00:11 | |
| *** funzo has quit IRC | 00:11 | |
| *** rcernin has quit IRC | 00:12 | |
| *** dingyichen has joined #openstack-infra | 00:14 | |
| *** Apoorva has quit IRC | 00:27 | |
| lathiat | tonyb; very delayed howdy reply :) | 00:30 |
| tonyb | lathiat: :) | 00:30 |
| tonyb | lathiat: what are you workignon these days? ... clearly OpenStack but a man needs specifics! | 00:30 |
| lathiat | tonyb; Support Team @ Canonical | 00:31 |
| lathiat | tonyb; dabbling in a little code but mostly operational support, but you know how these things go you end up reading code to figure out what is going on half the time | 00:31 |
| lathiat | tonyb; also got a presentation accepted at Sydney, so that'l be interesting! | 00:32 |
| tonyb | lathiat: Oh cool. #link? | 00:32 |
| lathiat | tonyb; you'll laugh slightly when you see it | 00:32 |
| tonyb | lathiat: Yeah sadly we often need to read the code to work out what $log_message actually means and how we got theer ;P | 00:33 |
| tonyb | lathiat: never .... perhaps a chuckle | 00:33 |
| lathiat | tonyb; https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20057/ipv6-primer-for-deployments | 00:33 |
| tonyb | lathiat: okay you were right I did LOL ;P | 00:33 |
| tonyb | lathiat: we'll have to catch-up while we're there | 00:34 |
| lathiat | tonyb; sounds good! | 00:34 |
| tonyb | lathiat: /me still doesn't have IPv6 for anything ;P so nothing changes | 00:35 |
| *** masber has joined #openstack-infra | 00:37 | |
| *** thorst_afk has joined #openstack-infra | 00:45 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Fix testenv creation without network isolation https://review.openstack.org/496967 | 00:46 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Move fix_disk_layout to an Ansible role in setup_host https://review.openstack.org/496935 | 00:46 |
| *** thorst_afk has quit IRC | 00:49 | |
| *** signed8bit is now known as signed8bit_Zzz | 00:54 | |
| *** cshastri has joined #openstack-infra | 00:55 | |
| *** yuanying_ has joined #openstack-infra | 00:58 | |
| *** gmann has quit IRC | 01:01 | |
| *** gmann has joined #openstack-infra | 01:01 | |
| *** yuanying has quit IRC | 01:02 | |
| *** yuanying has joined #openstack-infra | 01:02 | |
| *** bobh has joined #openstack-infra | 01:02 | |
| *** yuanying_ has quit IRC | 01:02 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Add a job to test emit-ara-html https://review.openstack.org/496952 | 01:03 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Add a job to test emit-ara-html https://review.openstack.org/496952 | 01:04 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Add a job to test emit-ara-html https://review.openstack.org/496952 | 01:05 |
| *** LindaWang has joined #openstack-infra | 01:05 | |
| fungi | clarkb: oh, thanks! yes i was cleaning up the sysadmins list and forgot to commit that :/ | 01:05 |
| *** bobh has quit IRC | 01:08 | |
| *** liujiong has joined #openstack-infra | 01:09 | |
| clarkb | fungi: any objection to using self signed snake oil cert in infracloud? | 01:10 |
| clarkb | old cert has expired | 01:10 |
| *** cshastri has quit IRC | 01:12 | |
| *** cshastri has joined #openstack-infra | 01:13 | |
| fungi | clarkb: no objection | 01:14 |
| *** ijw has joined #openstack-infra | 01:14 | |
| *** ijw has quit IRC | 01:19 | |
| *** ramishra has quit IRC | 01:19 | |
| *** masber has quit IRC | 01:20 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 01:21 | |
| *** thorst_afk has joined #openstack-infra | 01:23 | |
| *** xarses_ has joined #openstack-infra | 01:23 | |
| *** xarses_ has quit IRC | 01:24 | |
| *** xarses_ has joined #openstack-infra | 01:24 | |
| *** andreww has quit IRC | 01:27 | |
| *** dave-mccowan has joined #openstack-infra | 01:30 | |
| *** gcb has quit IRC | 01:33 | |
| *** sekelso has joined #openstack-infra | 01:34 | |
| *** gcb has joined #openstack-infra | 01:34 | |
| *** rlandy|bbl is now known as rlandy | 01:35 | |
| *** rlandy has quit IRC | 01:37 | |
| *** thorst_afk has quit IRC | 01:46 | |
| *** yamahata has quit IRC | 01:48 | |
| *** yuanying has quit IRC | 01:50 | |
| *** masber has joined #openstack-infra | 01:51 | |
| *** zhurong has joined #openstack-infra | 01:54 | |
| *** iyamahat has quit IRC | 01:54 | |
| *** cuongnv has joined #openstack-infra | 01:54 | |
| *** dougwig has quit IRC | 02:00 | |
| *** yuanying has joined #openstack-infra | 02:01 | |
| *** gildub has quit IRC | 02:05 | |
| *** funzo has joined #openstack-infra | 02:07 | |
| *** dave-mccowan has quit IRC | 02:07 | |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-infra | 02:11 | |
| *** funzo has quit IRC | 02:12 | |
| *** thorst_afk has joined #openstack-infra | 02:19 | |
| *** thorst_afk has quit IRC | 02:19 | |
| *** dave-mccowan has joined #openstack-infra | 02:23 | |
| *** gildub has joined #openstack-infra | 02:24 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Fix user role grants https://review.openstack.org/496980 | 02:28 |
| *** ramishra has joined #openstack-infra | 02:30 | |
| *** baoli has joined #openstack-infra | 02:34 | |
| *** chenying_ has joined #openstack-infra | 02:36 | |
| chenying_ | hi I want to delete the unused page[1] in openstack wiki. But I don't have permissions. Does anyone have the administrator permissions to the wiki? [1]https://wiki.openstack.org/wiki/smaug | 02:36 |
| *** baoli has quit IRC | 02:38 | |
| *** markvoelker_ has quit IRC | 02:41 | |
| *** hongbin has joined #openstack-infra | 02:46 | |
| *** signed8bit_Zzz is now known as signed8bit | 02:50 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Allow domain_id for roles https://review.openstack.org/496992 | 02:53 |
| *** chenying_ has quit IRC | 02:55 | |
| *** chenying_ has joined #openstack-infra | 02:55 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Allow domain_id for roles https://review.openstack.org/496992 | 03:00 |
| *** signed8bit is now known as signed8bit_Zzz | 03:07 | |
| *** dhajare has joined #openstack-infra | 03:12 | |
| *** signed8bit_Zzz is now known as signed8bit | 03:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Add test-role-ensure-twine job https://review.openstack.org/496994 | 03:15 |
| *** links has joined #openstack-infra | 03:18 | |
| *** thorst_afk has joined #openstack-infra | 03:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Add test-role-ensure-twine job https://review.openstack.org/496994 | 03:23 |
| *** thorst_afk has quit IRC | 03:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Add test-role-ensure-twine job https://review.openstack.org/496994 | 03:25 |
| *** signed8bit has quit IRC | 03:27 | |
| *** dave-mccowan has quit IRC | 03:27 | |
| *** nicolasbock has quit IRC | 03:35 | |
| *** sekelso has quit IRC | 03:51 | |
| *** bobh has joined #openstack-infra | 03:52 | |
| *** rmcall has quit IRC | 03:55 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Release extra-args from multinode environment settings https://review.openstack.org/496676 | 03:55 |
| clarkb | chenying_: can you send email to the infra list with that request and details? I'm not sure what the process is for that but we can get it sorted out | 03:57 |
| *** rmcall has joined #openstack-infra | 03:57 | |
| *** hongbin has quit IRC | 03:57 | |
| *** bobh has quit IRC | 04:01 | |
| *** ykarel has joined #openstack-infra | 04:04 | |
| *** funzo has joined #openstack-infra | 04:05 | |
| *** funzo has quit IRC | 04:09 | |
| *** gildub has quit IRC | 04:10 | |
| *** gildub has joined #openstack-infra | 04:12 | |
| *** s-shiono has joined #openstack-infra | 04:13 | |
| *** gildub has quit IRC | 04:17 | |
| *** gildub has joined #openstack-infra | 04:19 | |
| *** thorst_afk has joined #openstack-infra | 04:21 | |
| *** claudiub has joined #openstack-infra | 04:23 | |
| *** thorst_afk has quit IRC | 04:26 | |
| *** markvoelker has joined #openstack-infra | 04:42 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: Update the refstack meeting with new chair/ptl https://review.openstack.org/496296 | 04:46 |
| tonyb | So meetings in project channels is that a thing we do now? Is meetbot setup to deal with that? | 04:50 |
| tonyb | wondering what I do with: https://review.openstack.org/#/c/496314/1 | 04:50 |
| tonyb | fungi, jeblair, clarkb, ttx, flaper87 ^^^ | 04:50 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update the refstack meeting with new chair/ptl https://review.openstack.org/496296 | 04:54 |
| *** udesale has joined #openstack-infra | 04:57 | |
| *** zhurong has quit IRC | 04:58 | |
| *** pbourke has quit IRC | 05:00 | |
| *** pbourke has joined #openstack-infra | 05:02 | |
| *** bhavik1 has joined #openstack-infra | 05:03 | |
| *** zhurong has joined #openstack-infra | 05:06 | |
| *** iyamahat has joined #openstack-infra | 05:10 | |
| *** iyamahat has quit IRC | 05:11 | |
| *** rcernin has joined #openstack-infra | 05:11 | |
| *** iyamahat has joined #openstack-infra | 05:11 | |
| *** gouthamr has quit IRC | 05:13 | |
| *** markvoelker has quit IRC | 05:17 | |
| *** eranrom has quit IRC | 05:18 | |
| *** thorst_afk has joined #openstack-infra | 05:22 | |
| *** thorst_afk has quit IRC | 05:26 | |
| *** rcernin has quit IRC | 05:27 | |
| *** armax has quit IRC | 05:28 | |
| *** jascott1 has joined #openstack-infra | 05:42 | |
| *** kjackal has joined #openstack-infra | 05:43 | |
| openstackgerrit | Divyansh Acharya proposed openstack/os-testr master: Just changed the opening of the file https://review.openstack.org/494697 | 05:45 |
| *** bhavik1 has quit IRC | 05:51 | |
| *** _jbadiapa has quit IRC | 06:03 | |
| chenying_ | clarkb Thanks. | 06:09 |
| chenying_ | clarkb I will send a email ASAP. | 06:10 |
| *** rcernin has joined #openstack-infra | 06:12 | |
| *** markvoelker has joined #openstack-infra | 06:14 | |
| *** pcaruana has joined #openstack-infra | 06:21 | |
| *** thorst_afk has joined #openstack-infra | 06:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: LVM support for dib-block-device https://review.openstack.org/472065 | 06:23 |
| *** martinkopec has joined #openstack-infra | 06:24 | |
| *** aviau has quit IRC | 06:26 | |
| *** makowals has joined #openstack-infra | 06:27 | |
| *** thorst_afk has quit IRC | 06:27 | |
| *** slaweq has joined #openstack-infra | 06:32 | |
| *** jtomasek has joined #openstack-infra | 06:33 | |
| *** kjackal has quit IRC | 06:36 | |
| *** hichihara has joined #openstack-infra | 06:40 | |
| *** aeng has quit IRC | 06:45 | |
| *** markvoelker has quit IRC | 06:47 | |
| *** eroux has joined #openstack-infra | 06:48 | |
| *** lathiat has quit IRC | 06:51 | |
| *** lathiat has joined #openstack-infra | 06:51 | |
| *** jpich has joined #openstack-infra | 06:59 | |
| *** aviau has joined #openstack-infra | 07:03 | |
| *** florianf_ has joined #openstack-infra | 07:04 | |
| *** gildub has quit IRC | 07:05 | |
| *** florianf has quit IRC | 07:05 | |
| *** florianf_ has quit IRC | 07:05 | |
| *** florianf has joined #openstack-infra | 07:06 | |
| *** kjackal has joined #openstack-infra | 07:10 | |
| *** coolsvap has joined #openstack-infra | 07:13 | |
| *** tesseract has joined #openstack-infra | 07:20 | |
| *** liujiong_lj has joined #openstack-infra | 07:20 | |
| *** liujiong has quit IRC | 07:22 | |
| *** thorst_afk has joined #openstack-infra | 07:24 | |
| *** hichihara has quit IRC | 07:24 | |
| *** liujiong_lj has quit IRC | 07:28 | |
| *** thorst_afk has quit IRC | 07:28 | |
| *** alexchadin has joined #openstack-infra | 07:31 | |
| *** jpena|off is now known as jpena | 07:36 | |
| AJaeger | jeblair, clarkb, part of puppet not running means that project-config changes are not rolled out - zuul has "Last reconfigured: Wed Aug 23 2017 09:41:41 GMT+0200 (CEST)" | 07:36 |
| *** markvoelker has joined #openstack-infra | 07:44 | |
| *** gildub has joined #openstack-infra | 07:45 | |
| *** dtantsur|afk is now known as dtantsur | 07:46 | |
| *** _jbadiapa has joined #openstack-infra | 07:47 | |
| *** tesseract-RH has joined #openstack-infra | 07:49 | |
| *** markus_z has joined #openstack-infra | 07:49 | |
| *** tesseract-RH has quit IRC | 07:50 | |
| *** tesseract-RH has joined #openstack-infra | 07:50 | |
| *** gildub has quit IRC | 07:51 | |
| *** tesseract has quit IRC | 07:52 | |
| *** egonzalez has joined #openstack-infra | 07:56 | |
| *** tesseract-RH has quit IRC | 08:01 | |
| *** ykarel is now known as ykarel|lunch | 08:01 | |
| *** tesseract-RH has joined #openstack-infra | 08:03 | |
| *** gixxer1k has joined #openstack-infra | 08:05 | |
| *** funzo has joined #openstack-infra | 08:07 | |
| *** gildub has joined #openstack-infra | 08:09 | |
| *** yamamoto has joined #openstack-infra | 08:10 | |
| *** funzo has quit IRC | 08:11 | |
| *** sflanigan has quit IRC | 08:12 | |
| *** bogdando has joined #openstack-infra | 08:12 | |
| bogdando | o/, elastic-recheck folks, PTAL e-r queries https://review.openstack.org/#/c/493535/ https://review.openstack.org/#/c/493525/ https://review.openstack.org/#/c/493520/ to reduce unknown/uncategorised numbers | 08:12 |
| bogdando | any chances to have those merged? | 08:12 |
| bogdando | w/o those there is too much 'unknown' noise pops up in the http://status.openstack.org/elastic-recheck/data/others.html | 08:14 |
| *** lin_yang has quit IRC | 08:14 | |
| *** markvoelker has quit IRC | 08:16 | |
| *** thorst_afk has joined #openstack-infra | 08:25 | |
| *** thorst_afk has quit IRC | 08:29 | |
| *** iyamahat has quit IRC | 08:29 | |
| *** rmcall has quit IRC | 08:34 | |
| *** psachin has joined #openstack-infra | 08:36 | |
| *** lucas-afk is now known as lucasagomes | 08:41 | |
| *** pblaho has joined #openstack-infra | 08:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: ensure tripleo scenario003 checks trigger on changes https://review.openstack.org/496862 | 08:44 |
| openstackgerrit | Merged openstack-infra/project-config master: import whereto program https://review.openstack.org/496856 | 08:46 |
| *** pblaho has quit IRC | 08:49 | |
| *** alexchadin has quit IRC | 08:53 | |
| *** alexchadin has joined #openstack-infra | 08:54 | |
| *** pblaho has joined #openstack-infra | 08:56 | |
| frickler | infra-root: ask.o.o is rejecting connections | 08:58 |
| *** dingyichen has quit IRC | 08:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Change job type for 3nodes job to move to oooq runner"" https://review.openstack.org/495853 | 09:00 |
| *** ykarel|lunch is now known as ykarel | 09:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add nodejs4-jobs for designate-dashboard https://review.openstack.org/496724 | 09:03 |
| *** cshastri has quit IRC | 09:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add os-traits to gerritbot https://review.openstack.org/496546 | 09:07 |
| *** sambetts|afk is now known as sambetts | 09:11 | |
| *** slaweq has quit IRC | 09:11 | |
| *** slaweq has joined #openstack-infra | 09:12 | |
| *** markvoelker has joined #openstack-infra | 09:13 | |
| *** tosky has joined #openstack-infra | 09:15 | |
| *** yamamoto has quit IRC | 09:16 | |
| *** slaweq has quit IRC | 09:16 | |
| *** yamamoto has joined #openstack-infra | 09:17 | |
| *** claudiub|2 has joined #openstack-infra | 09:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/497165 | 09:18 |
| openstackgerrit | Merged openstack/diskimage-builder master: LVM support for dib-block-device https://review.openstack.org/472065 | 09:18 |
| *** _jbadiapa has quit IRC | 09:18 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Service Update https://review.openstack.org/482196 | 09:18 |
| *** tesseract-RH has quit IRC | 09:18 | |
| egonzalez | Hi, ask.openstack.org is down | 09:18 |
| *** claudiub has quit IRC | 09:20 | |
| *** ociuhandu has quit IRC | 09:21 | |
| *** tesseract has joined #openstack-infra | 09:23 | |
| *** thorst_afk has joined #openstack-infra | 09:25 | |
| *** yamamoto has quit IRC | 09:27 | |
| *** yamamoto has joined #openstack-infra | 09:29 | |
| *** thorst_afk has quit IRC | 09:30 | |
| *** yamamoto has quit IRC | 09:30 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Service Delete https://review.openstack.org/482197 | 09:30 |
| *** psachin has quit IRC | 09:31 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Switch to _is_client_version in list_services https://review.openstack.org/497171 | 09:34 |
| *** kjackal has quit IRC | 09:39 | |
| *** eumel8 has joined #openstack-infra | 09:40 | |
| *** claudiub has joined #openstack-infra | 09:42 | |
| *** jbadiapa has joined #openstack-infra | 09:42 | |
| *** claudiub|2 has quit IRC | 09:44 | |
| *** markvoelker has quit IRC | 09:46 | |
| *** alexchadin has quit IRC | 09:49 | |
| *** alexchadin has joined #openstack-infra | 09:54 | |
| *** s-shiono has quit IRC | 09:56 | |
| *** slaweq has joined #openstack-infra | 09:59 | |
| *** makowals has quit IRC | 10:03 | |
| *** zoli is now known as zoli|lunch | 10:05 | |
| *** zoli|lunch is now known as zoli | 10:05 | |
| *** funzo has joined #openstack-infra | 10:07 | |
| *** funzo has quit IRC | 10:12 | |
| *** kjackal has joined #openstack-infra | 10:12 | |
| *** dtantsur is now known as dtantsur|brb | 10:14 | |
| *** jamesmcarthur has joined #openstack-infra | 10:16 | |
| *** alexchadin has quit IRC | 10:16 | |
| *** jamesmcarthur has quit IRC | 10:20 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Fix the DLRN API URL for non-master branches https://review.openstack.org/497184 | 10:25 |
| *** gildub has quit IRC | 10:25 | |
| *** thorst_afk has joined #openstack-infra | 10:26 | |
| *** ociuhandu has joined #openstack-infra | 10:26 | |
| *** egonzalez has quit IRC | 10:30 | |
| *** stakeda has quit IRC | 10:30 | |
| *** thorst_afk has quit IRC | 10:31 | |
| *** yamamoto has joined #openstack-infra | 10:31 | |
| *** nicolasbock has joined #openstack-infra | 10:33 | |
| *** yamamoto has quit IRC | 10:36 | |
| *** jkilpatr has quit IRC | 10:39 | |
| *** ykarel is now known as ykarel|away | 10:39 | |
| *** xinliang has quit IRC | 10:44 | |
| *** egonzalez has joined #openstack-infra | 10:51 | |
| *** dpawlik_ is now known as dpawlik | 10:51 | |
| *** nicolasbock has quit IRC | 10:54 | |
| *** xinliang has joined #openstack-infra | 10:57 | |
| *** xinliang has quit IRC | 10:57 | |
| *** xinliang has joined #openstack-infra | 10:57 | |
| *** sean-k-mooney has joined #openstack-infra | 10:57 | |
| sean-k-mooney | o/ | 10:57 |
| *** namnh has quit IRC | 11:01 | |
| *** markus_z has quit IRC | 11:01 | |
| *** udesale has quit IRC | 11:04 | |
| *** jpena is now known as jpena|lunch | 11:05 | |
| *** egonzalez has quit IRC | 11:07 | |
| *** nicolasbock has joined #openstack-infra | 11:08 | |
| *** jkilpatr has joined #openstack-infra | 11:13 | |
| *** markus_z has joined #openstack-infra | 11:17 | |
| *** ociuhandu has quit IRC | 11:17 | |
| flaper87 | tonyb: thanks for the heads up | 11:18 |
| *** ociuhandu has joined #openstack-infra | 11:18 | |
| *** egonzalez has joined #openstack-infra | 11:18 | |
| *** gixxer1k has quit IRC | 11:24 | |
| *** ociuhandu has quit IRC | 11:25 | |
| *** gixxer1k has joined #openstack-infra | 11:25 | |
| *** cuongnv has quit IRC | 11:25 | |
| *** thorst_afk has joined #openstack-infra | 11:27 | |
| *** alexchadin has joined #openstack-infra | 11:31 | |
| sean-k-mooney | hi, i think i have asked this question in the past. does anyone know if there is a way to create an upstream ci job that will be scheduled to a node with nested virt enabled on kvm? | 11:32 |
| *** thorst_afk has quit IRC | 11:32 | |
| sean-k-mooney | is that a facility that could be added in the future? | 11:32 |
| *** yamamoto has joined #openstack-infra | 11:33 | |
| pabelanger | sean-k-mooney: we currently disable nested virt, mostly because not all our clouds support it | 11:34 |
| sean-k-mooney | pabelanger: yes i have seen that. i was wondering if we had the abllity to create an aggregate of providers that did support it and associate specific jobs that required it to run on that aggregate | 11:35 |
| *** numans has quit IRC | 11:35 | |
| *** alexchadin has quit IRC | 11:36 | |
| sean-k-mooney | im also aware that alot of infras current capasity is provided by rackspace which use zen in the wake of osic disolving so kvm node are at a premium and kvm + nested vert even more so | 11:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/494800 | 11:37 |
| pabelanger | sean-k-mooney: usually not, that would mean homing jobs to specific providers. Which makes it more difficult to keep things realiable. | 11:37 |
| *** alexchadin has joined #openstack-infra | 11:37 | |
| pabelanger | sean-k-mooney: something like a 3rd party CI would be good fit for this | 11:38 |
| *** yamamoto has quit IRC | 11:38 | |
| *** mat128 has joined #openstack-infra | 11:39 | |
| *** numans has joined #openstack-infra | 11:39 | |
| *** wolverineav has joined #openstack-infra | 11:39 | |
| sean-k-mooney | pabelanger: yes currently we have a 3rd party ci for ovs-dpdk testing and some other fetures that require nested virt. i was trying to see if we could supprot that testing in the upstream ci going forward | 11:39 |
| *** gixxer1k has quit IRC | 11:39 | |
| *** wolverin_ has joined #openstack-infra | 11:40 | |
| sean-k-mooney | i was investigating removing the nested virt requirements for ovs-dpdk but i hit bugs in qemus vhost-user support in the tci backend which ment that i could not proceed with that path | 11:40 |
| pabelanger | sean-k-mooney: AFAIK, no because of the issue mention before. But we can see what clarkb says | 11:41 |
| *** numans has quit IRC | 11:41 | |
| *** ykarel|away is now known as ykarel | 11:41 | |
| AJaeger | pabelanger, sean-k-mooney, we had that discussion in the past - and cannot go there currently. We choose to document what is possible at https://docs.openstack.org/infra/manual/testing.html | 11:41 |
| *** srobert has joined #openstack-infra | 11:42 | |
| *** numans has joined #openstack-infra | 11:42 | |
| *** makowals has joined #openstack-infra | 11:42 | |
| AJaeger | that includes a paragraph on nested virt | 11:43 |
| *** wolverineav has quit IRC | 11:43 | |
| *** markvoelker has joined #openstack-infra | 11:44 | |
| *** srobert has quit IRC | 11:44 | |
| sean-k-mooney | AJaeger: thanks just read over it now | 11:45 |
| sean-k-mooney | so on a slightly different vain can i ask. if we wehere to make hardware available as a new provider to infra could we then achive this goal? e.g. use infras zull to kick of jobs on that dedicated cloud which has nested virt et enabled. | 11:47 |
| sean-k-mooney | essincially why i am asking is our funding or our thirdparty ci has been reduced significantly over the last 12 months. for the pike cycle we were reduced from 1 full time employee + 2 full time contractor dedicated to maintaining our on cis to 1 contractor who is no part of a different team. | 11:48 |
| sean-k-mooney | as a result while we have the capsity to keep our third party ci alive and functioning it has effectly resulted in canceling all furter extention fo the ci to cover new test cases and canceled or plans to migrate to zuul v3 | 11:50 |
| *** arturb has quit IRC | 11:54 | |
| *** eumel8 has quit IRC | 11:57 | |
| berendt | is ask.openstack.org down? | 12:00 |
| *** trown|outtypewww is now known as trown | 12:01 | |
| *** jpena|lunch is now known as jpena | 12:02 | |
| pabelanger | technically possible, but again we'd be homing jobs to a specific jobs, this is how tripleo-test-cloud-rh1 works today. They have OVB jobs that run on that dedicated cloud from zuul.o.o, however were not fans of the setup. If that cloud goes down, or there is issues, jobs are efficatively blocked. In fact, tripleo-test-cloud-rh1 is in the process of moving toward 3rd party CI. It is also less | 12:02 |
| pabelanger | complicated if all jobs run across all cloud regions from an management POV too | 12:02 |
| pabelanger | berendt: ya, looks like it | 12:02 |
| pabelanger | looking to see why | 12:02 |
| pabelanger | [Thu Aug 24 06:27:11.130008 2017] [core:notice] [pid 15941] AH00060: seg fault or similar nasty error detected in the parent process | 12:03 |
| pabelanger | apache2 died | 12:03 |
| berendt | pabelanger: thanks | 12:03 |
| pabelanger | back online now | 12:04 |
| pabelanger | so, we need to see why apache2 wasn't restarted. I know puppet isn't running, ATM, so that doesn't help | 12:04 |
| *** funzo has joined #openstack-infra | 12:05 | |
| fungi | tonyb: meetings in project channels are a thing after https://review.openstack.org/485117 reaches consensus and gets approved | 12:06 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix unbound variable DELOREAN_LINK https://review.openstack.org/497221 | 12:07 |
| *** quite has joined #openstack-infra | 12:08 | |
| *** ldnunes has joined #openstack-infra | 12:09 | |
| *** funzo has quit IRC | 12:10 | |
| fungi | sean-k-mooney: to expand on the answer about "making dedicated hardware available to the upstream ci" we also don't really have any good solutions for that today. the tripleo cloud isn't really a good example because it's a running cloud operated by the donor, not just a pile of hardware we take care of | 12:10 |
| fungi | the closest example we have is our infra-cloud experiment, but the infra team doesn't really have a lot of available time to be cloud operators on top of running the ci system | 12:11 |
| pabelanger | looking forward to the upcoming talks at PTG too about this. It is an interesting topic | 12:12 |
| fungi | also, as pabelanger and AJaeger pointed out 1. we're not really set up to cater to jobs which can't run consistently everywhere, and 2. nested kvm has been really unstable when we've tried it in the past (even the upstream kernel still has it disabled by default, as i understand) | 12:14 |
| *** eranrom has joined #openstack-infra | 12:15 | |
| fungi | the nested kvm acceleration support i mean (nesting kvm in kvm without acceleration works fine, it's just slow) | 12:15 |
| *** thorst_afk has joined #openstack-infra | 12:16 | |
| *** dprince has joined #openstack-infra | 12:16 | |
| *** markvoelker has quit IRC | 12:16 | |
| fungi | there's also 3. any jobs with super special needs like that are even harder for developers to run locally so they can recreate and debug failures, so they end up treating them as a black box or asking us to debug for them | 12:16 |
| *** gildub has joined #openstack-infra | 12:21 | |
| *** mriedem has quit IRC | 12:21 | |
| pabelanger | fungi: it looks like puppet is still down. I know clark was working to replace infracloud SSL certs last night. I have an errand to run this morning, so won't be able to help for the next 2 hrs | 12:22 |
| fungi | i'm about to disappear as well (still travelling through tomorrow) so i probably don't have time to come up to speed on the issue | 12:23 |
| fungi | ahh, ansible segfaulting because it can't generate the inventory because it can't log into the cloud api there | 12:25 |
| *** gildub has quit IRC | 12:28 | |
| *** rlandy has joined #openstack-infra | 12:28 | |
| *** sflanigan has joined #openstack-infra | 12:30 | |
| *** sflanigan has joined #openstack-infra | 12:30 | |
| mordred | fungi: segfaulting? | 12:30 |
| fungi | mordred: raising a python exception | 12:31 |
| fungi | i'm still short caffeine | 12:31 |
| fungi | pabelanger: clarkb: looks like in clouds.yaml we specify a custom cacert parameter for infra-cloud regions so it will consult the /etc/ssl/certs/ca-certificates.crt file, but that hasn't been updated to include the new snakeoil cert(s)? | 12:31 |
| mordred | ah. ok. I mean - we have that python segfault fix | 12:31 |
| fungi | yeah, i should not have used that word | 12:32 |
| mordred | fungi: that seems possible | 12:32 |
| *** markvoelker has joined #openstack-infra | 12:32 | |
| mordred | (the cacert thing) | 12:32 |
| fungi | the file was last modified a year ago, so suspecting that's (at least part of) the problem | 12:33 |
| *** signed8bit has joined #openstack-infra | 12:34 | |
| *** yamamoto has joined #openstack-infra | 12:34 | |
| *** yamamoto has quit IRC | 12:41 | |
| *** jcoufal has joined #openstack-infra | 12:46 | |
| *** eharney has joined #openstack-infra | 12:46 | |
| *** markus_z has quit IRC | 12:47 | |
| *** kjackal has quit IRC | 12:47 | |
| *** kgiusti has joined #openstack-infra | 12:51 | |
| *** jascott1 has quit IRC | 12:52 | |
| *** alexchadin has quit IRC | 12:52 | |
| *** jascott1 has joined #openstack-infra | 12:53 | |
| *** alexchadin has joined #openstack-infra | 12:53 | |
| *** kornicameister has quit IRC | 12:54 | |
| *** mat128 has quit IRC | 12:55 | |
| *** udesale has joined #openstack-infra | 12:55 | |
| *** jascott1 has quit IRC | 12:57 | |
| *** sflanigan has quit IRC | 12:57 | |
| *** mriedem has joined #openstack-infra | 12:58 | |
| *** signed8bit is now known as signed8bit_Zzz | 12:59 | |
| *** Goneri has joined #openstack-infra | 13:00 | |
| *** LindaWang has quit IRC | 13:00 | |
| *** dtantsur|brb is now known as dtantsur | 13:00 | |
| *** gildub has joined #openstack-infra | 13:00 | |
| fungi | sorry, popping in and out of access... i'll try to see if a puppet apply on the puppetmaster takes care of that | 13:01 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Remove OpenStack/Ceph/Virt repo from CentOS https://review.openstack.org/493003 | 13:03 |
| *** sekelso has joined #openstack-infra | 13:03 | |
| *** mat128 has joined #openstack-infra | 13:04 | |
| *** mat128 has quit IRC | 13:04 | |
| *** amoralej is now known as amoralej|lunch | 13:05 | |
| *** kornicameister has joined #openstack-infra | 13:06 | |
| *** gildub has quit IRC | 13:06 | |
| fungi | i fell down the rabbit hole of attempting to figure out how to run puppet apply consistently with how ansible runs it, and so have run out of time to dig deeper into figuring this out | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Document list of configuration items for include/exclude https://review.openstack.org/496928 | 13:08 |
| *** pblaho has quit IRC | 13:08 | |
| *** sree has joined #openstack-infra | 13:12 | |
| *** markus_z has joined #openstack-infra | 13:13 | |
| *** signed8bit_Zzz is now known as signed8bit | 13:14 | |
| *** kjackal has joined #openstack-infra | 13:15 | |
| *** sree has quit IRC | 13:16 | |
| *** pblaho has joined #openstack-infra | 13:21 | |
| egonzalez | Hi, does post merge jobs having a different instance size? kolla build gate jobs succeed with oraclelinux but failing always in post merge (creates a docker registry) http://logs.openstack.org/79/793949e12af683ef2dbba5640eaf4ff2458d0c57/post/kolla-dsvm-build-oraclelinux-binary-centos-7/db2df54/console.html | 13:21 |
| *** pblaho has quit IRC | 13:22 | |
| egonzalez | *failing because not space on device | 13:22 |
| *** pblaho has joined #openstack-infra | 13:22 | |
| *** gildub has joined #openstack-infra | 13:23 | |
| *** udesale has quit IRC | 13:26 | |
| *** udesale has joined #openstack-infra | 13:26 | |
| *** udesale has quit IRC | 13:26 | |
| *** gouthamr has joined #openstack-infra | 13:27 | |
| *** dave-mccowan has joined #openstack-infra | 13:34 | |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Add irc notifications for pike https://review.openstack.org/497242 | 13:38 |
| *** LindaWang has joined #openstack-infra | 13:39 | |
| *** Julien-zte has quit IRC | 13:40 | |
| *** yamamoto has quit IRC | 13:42 | |
| *** amoralej|lunch is now known as amoralej | 13:43 | |
| *** Julien-zte has joined #openstack-infra | 13:43 | |
| *** links has quit IRC | 13:46 | |
| AJaeger | egonzalez: are you really doing the same stuff in both? WE use the same nodes for all pipelines if you use the same node specification as the jobs do | 13:47 |
| *** xarses has joined #openstack-infra | 13:47 | |
| *** xarses_ has quit IRC | 13:48 | |
| fungi | egonzalez: gate and post jobs get nodes assigned from the same pool, so either your jobs generally use too much space to run in some of our providers and you've just been getting lucky in the gate, or else your post jobs use more space than your gate jobs | 13:48 |
| fungi | that job looks like it ran out of space copying docker logs | 13:49 |
| *** bobh has joined #openstack-infra | 13:50 | |
| *** eharney has quit IRC | 13:50 | |
| egonzalez | fungi, AJaeger difference between regular jobs and post merge is that post-merge creates a tarball with all docker registry volume. That may be filling the disk as is failing always in that step | 13:51 |
| *** camunoz has joined #openstack-infra | 13:51 | |
| egonzalez | so, yeah, different jobs :( | 13:51 |
| fungi | in fact, it may have broken earlier... that console log indicates lots of images failed to build | 13:52 |
| *** bobh has quit IRC | 13:53 | |
| *** bobh has joined #openstack-infra | 13:53 | |
| egonzalez | fungi, those images are expected to fail (have an exception in the code), already checked that | 13:53 |
| fungi | egonzalez: for one suggestion, make sure you set up any available additional block device for scratch space (like devstack-gate does creating /opt) | 13:53 |
| *** eharney has joined #openstack-infra | 13:54 | |
| fungi | we make at least 80gb of space available but it may not all be formatted and mounted (that's one of the things devstack-gate would set up for you) | 13:54 |
| fungi | egonzalez: see the bullet item about available disk at https://docs.openstack.org/infra/manual/testing.html#unprivileged-single-use-vms | 13:59 |
| *** ihrachys has joined #openstack-infra | 14:00 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/infra-manual master: Add Zuul v3 quickstart section https://review.openstack.org/495971 | 14:03 |
| *** mat128 has joined #openstack-infra | 14:03 | |
| *** ldnunes has quit IRC | 14:03 | |
| egonzalez | fungi, yeah, you are right, a disk is not mounted http://logs.openstack.org/06/496706/1/check/gate-kolla-dsvm-build-oraclelinux-binary-centos-7-nv/9f55c3f/logs/node_info_1503569032 | 14:04 |
| *** derekh has joined #openstack-infra | 14:04 | |
| *** mat128 has quit IRC | 14:05 | |
| *** gildub has quit IRC | 14:05 | |
| *** hongbin has joined #openstack-infra | 14:08 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: testing ssl https://review.openstack.org/497405 | 14:10 |
| *** ldnunes has joined #openstack-infra | 14:15 | |
| *** makowals has quit IRC | 14:20 | |
| *** mat128 has joined #openstack-infra | 14:21 | |
| *** marst has joined #openstack-infra | 14:22 | |
| mwhahaha | hey where is the config for the openstackgerrit bot for stable branches? i think i need to update it to pick up on stable/pike stuff | 14:23 |
| jeblair | mwhahaha: project-config/gerritbot | 14:24 |
| mwhahaha | thanks | 14:24 |
| *** jcoufal has quit IRC | 14:25 | |
| *** jcoufal has joined #openstack-infra | 14:26 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Update bot for puppet-openstack to include pike https://review.openstack.org/497411 | 14:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: create stack and tempest users https://review.openstack.org/496959 | 14:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: put repos into $BASE/new https://review.openstack.org/496958 | 14:27 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Update puppet-ceph to use stable/pike for jewel https://review.openstack.org/497413 | 14:28 |
| *** slaweq has quit IRC | 14:29 | |
| sean-k-mooney | fungi: hi sorry was debugin other things. we have not had any issues with nested virt. we do most of our development in vms with it enabled and its always been enable in the intel nfv ci for the last 3 years. but ya we have kept it as a thrid party ci for a reason im jsut trying to see if there was a way we could expand the scope of what we were testing without having to go back and request more | 14:30 |
| sean-k-mooney | headcount which upper management has been trying to reduce | 14:30 |
| sean-k-mooney | anyway for now it sould like what we have today with the thrid party ci is still the best approch. | 14:31 |
| *** eroux has quit IRC | 14:31 | |
| *** rbrndt has joined #openstack-infra | 14:31 | |
| *** jcoufal has quit IRC | 14:33 | |
| *** jcoufal has joined #openstack-infra | 14:36 | |
| *** mat128 has quit IRC | 14:36 | |
| *** yamamoto has joined #openstack-infra | 14:38 | |
| clarkb | fungi: The cert value in hiera gets written out and used by clouds.yaml/shade/nodepool iirc | 14:42 |
| *** armax has joined #openstack-infra | 14:42 | |
| clarkb | fungi: that infracloud cert variable is used on client hosts in site.pp at least | 14:42 |
| *** yamamoto has quit IRC | 14:44 | |
| clarkb | I think once we update the certs on controller by hand (because ansible/puppet wont run until then) everythi g else should fall into place? | 14:45 |
| *** alexchadin has quit IRC | 14:46 | |
| *** ralonsoh has joined #openstack-infra | 14:46 | |
| *** ralonsoh has quit IRC | 14:47 | |
| *** patriciadomin has joined #openstack-infra | 14:48 | |
| *** bobh has quit IRC | 14:53 | |
| pabelanger | and back | 14:58 |
| *** xarses_ has joined #openstack-infra | 14:58 | |
| clarkb | I'm a few minutes away from replacing the cert by hand on controller00 | 14:58 |
| clarkb | I want to get in the hirra changes first so that once I do that the cert updates everywhere | 14:58 |
| pabelanger | ok | 14:59 |
| clarkb | but fungi brings up a good point of double checking the vlient side trust | 15:00 |
| *** ykarel has quit IRC | 15:01 | |
| *** Swami has joined #openstack-infra | 15:01 | |
| *** markus_z has quit IRC | 15:02 | |
| *** jascott1 has joined #openstack-infra | 15:06 | |
| *** jistr|trng is now known as jistr | 15:09 | |
| clarkb | fungi: pabelanger on the nodepool launchers and builders we write out that hiera value but we do not do this on puppetmaster | 15:10 |
| clarkb | any idea what is updating ca-certificates.crt on puppetmaster? I wonder if that was just a manual trust the cert | 15:11 |
| clarkb | ya looks like there is a pem file for each of vanilla and chocolate in /etc/ssl/certs wondering if that was just dropped in manually. In any case I think I can go ahead and update private hiera now and merge the public hiera change | 15:14 |
| clarkb | pabelanger: ^ ready? | 15:14 |
| *** sekelso has quit IRC | 15:14 | |
| *** camunoz has quit IRC | 15:14 | |
| *** camunoz has joined #openstack-infra | 15:15 | |
| *** jamesmcarthur has joined #openstack-infra | 15:15 | |
| *** bhavik1 has joined #openstack-infra | 15:18 | |
| clarkb | ok private hiera updated, approving public change now | 15:19 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Allow domain_id for roles https://review.openstack.org/496992 | 15:26 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Fix user role grants https://review.openstack.org/496980 | 15:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: create stack and tempest users https://review.openstack.org/496959 | 15:27 |
| clarkb | I have updated the certs on controller00 by hand | 15:28 |
| clarkb | and restarted apache. openssl s_client seems happy enough with that ssl setup | 15:28 |
| clarkb | once the public key change merges I will then update (by hand I guess because I can't see where we ever puppeted this) /usr/local/share/ca-certificates on puppetmaster and run update-ca-certificates so that ansible/shade/clouds.yaml is happy there | 15:29 |
| clarkb | then we'll have to make sure that all the compute hosts and all that aren't out to lunch | 15:29 |
| openstackgerrit | Merged openstack-infra/system-config master: Update infracloud-vanilla public ssl cert https://review.openstack.org/496964 | 15:30 |
| clarkb | infra-root ^ considering its puppetmaster, any concern with that plan? | 15:30 |
| *** cody-somerville has joined #openstack-infra | 15:31 | |
| jeblair | clarkb: ++ | 15:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: create stack and tempest users https://review.openstack.org/496959 | 15:33 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Allow domain_id for roles https://review.openstack.org/496992 | 15:33 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Fix user role grants https://review.openstack.org/496980 | 15:33 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Rework the getthelogs helper script for wget recursive https://review.openstack.org/492178 | 15:33 |
| kklimonda | is there a reason why, in puppet-zuul module, apache vhost (and all dependencies) are managed in ::zuul class, and not ::zuul::server? Is there some unexpected dependency I should be aware of? | 15:37 |
| *** ijw has joined #openstack-infra | 15:37 | |
| hwoarang | Hello. would it be easy to hold a node for me to debug http://logs.openstack.org/90/497190/1/check/gate-openstack-ansible-tests-ansible-func-opensuse-423-nv/3ad51fe/console.html#_2017-08-24_11_06_47_346515 ? i can't reproduce it on a vagrant image | 15:37 |
| *** jascott1 has quit IRC | 15:40 | |
| clarkb | infra-root I can successfully catalog list vanilla from puppetmaster now | 15:40 |
| jeblair | clarkb: \o/ | 15:40 |
| *** yamamoto has joined #openstack-infra | 15:40 | |
| pabelanger | clarkb: great | 15:40 |
| clarkb | infra-root I will swing back aroudn once puppet runs to make sure the rest of the cloud is functioning post cert update | 15:40 |
| openstackgerrit | Krzysztof Klimonda proposed openstack-infra/puppet-zuul master: Reload zuul-launcher when launcher-logging.conf changes. https://review.openstack.org/497462 | 15:40 |
| clarkb | then I guess next step is do the same for chocolate | 15:40 |
| clarkb | hwoarang: are you using a dib built image? | 15:41 |
| clarkb | infra-root I'd also like to update puppetmaster to not trust infracloud certs via /etc/ssl/certs/ca-certificates but instead via direct pointing to the public cert file like we do with nodepool | 15:42 |
| jeblair | clarkb: ++ | 15:42 |
| clarkb | I think that ends up being a much clearer deployment method and makes untrusting these certs trivial | 15:43 |
| hwoarang | clarkb: nope but hold on a moment | 15:43 |
| *** mat128 has joined #openstack-infra | 15:43 | |
| pabelanger | clarkb: ++ | 15:44 |
| *** mat128 has quit IRC | 15:45 | |
| *** yamamoto has quit IRC | 15:45 | |
| jeblair | clarkb: do you think you can review the updated d-g networking patch today (435933)? | 15:45 |
| clarkb | jeblair: ya probably. Also puppet still failing to run because we can't list servers because some other thing is failing ssl now. Looking at that | 15:46 |
| *** sambetts is now known as sambetts|afk | 15:47 | |
| clarkb | this is a fun chicken and egg where you can't puppet the cloud because the cloud dne | 15:47 |
| *** amotoki is now known as amotoki__away | 15:47 | |
| clarkb | I'm thinking I may just comment it out of clouds.yaml which will let puppet run on the "under" cloud hosts and fix ssl (in theory) | 15:48 |
| jeblair | wfm | 15:48 |
| *** mat128 has joined #openstack-infra | 15:48 | |
| *** srobert has joined #openstack-infra | 15:49 | |
| *** rcrit has joined #openstack-infra | 15:49 | |
| clarkb | ok thats done, next run in 10 minutes | 15:49 |
| rcrit | the openstack project, novajoin, is currently branchless. If we stick with this for now but find out later we need, say, pike-specific changes is it possible to branch later? | 15:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Move fix_disk_layout to an Ansible role in setup_host https://review.openstack.org/496935 | 15:50 |
| clarkb | rcrit: yes you will be able to create a stable/pike later if necessary. You just need a base ref/sha1/branch to branch off of. You might double check with the release team to make sure this doesn't break any of their tooling though | 15:52 |
| rcrit | thanks. How can I contact them? | 15:52 |
| dmsimard | infra-root: is it possible that codesearch.o.o needs to run puppet to update it's projects file ? Some repos such as zuul-jobs are missing from it. | 15:53 |
| * rcrit assumes you mean openstack release team | 15:53 | |
| *** jascott1 has joined #openstack-infra | 15:53 | |
| jeblair | rcrit: #openstack-release | 15:53 |
| jeblair | clarkb: do the words "ironic_key" mean anything to you in the devstack-gate context? | 15:53 |
| rcrit | thanks | 15:54 |
| pabelanger | dmsimard: ya, that needs to be manually restarted to pick up new projects | 15:54 |
| clarkb | jeblair: sort of. Ironic's fake baremetal driver uses ssh to ssh into hypervisors and manages VMs with virsh. So that is the key that driver uses to ssh | 15:54 |
| *** rcernin has quit IRC | 15:54 | |
| clarkb | jeblair: I've asked them to use the existing test infrastructure key in the past iirc but don't recall why it wasn't used (if it wasn't, maybe it is) | 15:55 |
| dmsimard | pabelanger: it's probably long overdue, zuul-jobs has existed for a while now :) | 15:55 |
| jeblair | clarkb: okay. there's something about the way it's setup that isn't making sense to me. i'm going to put up a patch. | 15:55 |
| clarkb | infra-root reading exceptions more carefully I see that the python daemons on controller00 do indeed appear to need restarting to pick up the new key data. So I am doing that now | 15:55 |
| clarkb | infra-root the way we deploy we are terminating ssl in the python daemons rather than in a proxy (except for keystone which is wsgi'd in apache) | 15:55 |
| pabelanger | dmsimard: yes, it results in an outage of codesearch.o.o. But I can kick here in a moment | 15:56 |
| *** rlandy is now known as rlandy|brb | 15:56 | |
| openstackgerrit | Steve McLellan proposed openstack-infra/irc-meetings master: Move searchlight meeting to biweekly (odd) https://review.openstack.org/497469 | 15:56 |
| *** pcaruana has quit IRC | 15:57 | |
| *** wolverin_ has quit IRC | 15:57 | |
| *** wolverineav has joined #openstack-infra | 15:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Remove IRONIC_KEY_FILE https://review.openstack.org/497470 | 15:58 |
| *** ianychoi has quit IRC | 15:59 | |
| *** ianychoi has joined #openstack-infra | 16:00 | |
| pabelanger | #status log hound service on codesearch.o.o stopped / started to pick up new projects for indexing | 16:00 |
| openstackstatus | pabelanger: finished logging | 16:00 |
| pabelanger | dmsimard: ^ | 16:00 |
| *** egonzalez has quit IRC | 16:01 | |
| dmsimard | pabelanger: thanks | 16:01 |
| *** bhavik1 has quit IRC | 16:02 | |
| *** wolverineav has quit IRC | 16:02 | |
| *** jbadiapa has quit IRC | 16:02 | |
| *** tesseract has quit IRC | 16:03 | |
| *** openstackgerrit has quit IRC | 16:04 | |
| *** vhosakot has joined #openstack-infra | 16:05 | |
| clarkb | \o/ server list works now. I have learned more things. We need to update-ca-certificates on the controller as well because all the things talk to each other and we don't explicitly trust there either | 16:06 |
| *** links has joined #openstack-infra | 16:06 | |
| *** funzo has joined #openstack-infra | 16:07 | |
| dimak | Hey, there are lots of queued jobs in zuul | 16:07 |
| dimak | Anything wrong with the infrastructure? | 16:07 |
| pabelanger | looking | 16:07 |
| *** rlandy|brb is now known as rlandy | 16:08 | |
| pabelanger | dimak: will be better once infracloud is backonlie | 16:09 |
| pabelanger | online* | 16:09 |
| pabelanger | shouldn't be much longer | 16:09 |
| dimak | pabelanger, thanks, looking forward :) | 16:09 |
| clarkb | also puppet may have done some magic to update things I missed | 16:09 |
| clarkb | basically just had to get it far enough along that it could bootstrap itself | 16:10 |
| pabelanger | dimak: but ya, that is the new normal. We only have ~950 nodes now that osic-cloud1 is offline, so please be patient | 16:10 |
| *** openstackgerrit has joined #openstack-infra | 16:10 | |
| openstackgerrit | Merged openstack-infra/project-config master: Sign artifacts before we release them https://review.openstack.org/496428 | 16:10 |
| *** funzo has quit IRC | 16:11 | |
| pabelanger | clarkb: confirm puppet looks to be running | 16:12 |
| *** coolsvap has quit IRC | 16:13 | |
| clarkb | we should know shortly if nodepool is able to boot instances there I think | 16:13 |
| jeblair | the test node graph is very spikey | 16:15 |
| *** rockstar has left #openstack-infra | 16:15 | |
| pabelanger | ya :( | 16:15 |
| clarkb | puppet just upadted the cert on nodepool.o.o | 16:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add infracloud controllers to ssl_cert_check https://review.openstack.org/497479 | 16:18 |
| *** jbadiapa has joined #openstack-infra | 16:18 | |
| pabelanger | clarkb: ^is that something that will work? | 16:18 |
| clarkb | pabelanger: maybe? depends on if it will always fail because self signed | 16:19 |
| pabelanger | clarkb: ya, that's my concern too | 16:19 |
| pabelanger | maybe fungi knows | 16:19 |
| pabelanger | look to be building nodes again | 16:19 |
| clarkb | ya, now to see if they succeed | 16:20 |
| clarkb | (I'm slightly worried we have to go through and kick all the computes to see the new key too) | 16:20 |
| pabelanger | OpenStackCloudException: Error in creating the server. | 16:21 |
| pabelanger | that's all it says | 16:22 |
| *** LindaWang has quit IRC | 16:22 | |
| pabelanger | going to check controller | 16:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Revert "Allow launcher to stop quicker when asked" https://review.openstack.org/497480 | 16:22 |
| jeblair | pabelanger: that happens when the server is in error state | 16:23 |
| jeblair | so we can also ask nova | 16:23 |
| pabelanger | jeblair: clarkb: Ya, I am seeing SSL cert verify failed errors on a compute node | 16:24 |
| pabelanger | SSLError: SSL exception connecting to https://controller00.vanilla.ic.openstack.org:35357/v3/auth/tokens: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed | 16:24 |
| *** deduped has joined #openstack-infra | 16:24 | |
| clarkb | pabelanger: ya I think the python daemons may load the cert data on start and not see the updated stuff until a restart. However it may also be relying on /etc/ssl/certs/ca-certificates and that hasn'ed updated yet for some reason | 16:25 |
| clarkb | so I think we want to double check with s_client first, then track down ca-certificates then possibly restart daemons | 16:26 |
| *** sekelso has joined #openstack-infra | 16:26 | |
| pabelanger | ya, that cert haven't updated | 16:26 |
| pabelanger | -r--r--r-- 1 root root 1261 Aug 26 2016 openstack_infra_ca.crt | 16:26 |
| *** bogdando has quit IRC | 16:26 | |
| pabelanger | let me see where we set that up | 16:26 |
| clarkb | s_client to controller looks good | 16:26 |
| *** sekelso has quit IRC | 16:26 | |
| clarkb | ok | 16:27 |
| *** sekelso has joined #openstack-infra | 16:27 | |
| clarkb | pabelanger: I think the puppet-infracloud::compute manifest drops it in place | 16:29 |
| *** jpich has quit IRC | 16:29 | |
| pabelanger | clarkb: ya | 16:29 |
| pabelanger | ssl_key_file_contents did we update that? | 16:29 |
| pabelanger | in hiera | 16:29 |
| clarkb | yes | 16:29 |
| pabelanger | k | 16:29 |
| clarkb | that is the private key though which compute hosts don't need (though I think get) | 16:29 |
| pabelanger | maybe I am just on node puppet didn't connect too | 16:29 |
| pabelanger | clarkb: that is the cacert_content in compute nodes | 16:30 |
| clarkb | 009 hasn't had puppet-user logs in syslog for a while | 16:30 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/puppet-infracloud/tree/manifests/compute.pp | 16:30 |
| clarkb | no its the key content | 16:30 |
| clarkb | cert content is the public data | 16:30 |
| pabelanger | same with compute-004 | 16:30 |
| clarkb | (I think the puppet manifest si fine, its more a matter of why are we not running puppet) | 16:31 |
| pabelanger | okay, puppet looks to be running now on infracloud controllers | 16:32 |
| pabelanger | I don't think it has made it to compute nodes yet | 16:32 |
| portdirect | hey - does anyone know who maintains the cirros site? | 16:32 |
| clarkb | portdirect: scott moser iirc | 16:32 |
| pabelanger | Aug 24 16:31:18 controller00 puppet-user[31283]: Could not find command '/usr/bin/curl' | 16:34 |
| pabelanger | that is on controllers | 16:34 |
| portdirect | clarkb: thanks, looks like its taking a break for a bit :) https://who.is/whois/cirros-cloud.net | 16:34 |
| pabelanger | which is breaking puppet | 16:34 |
| pabelanger | clarkb: want me to manually install curl, then fix in puppet? | 16:34 |
| clarkb | pabelanger: and ansible is set up to not run on computes if controller fails right? | 16:34 |
| clarkb | pabelanger: ++ please | 16:34 |
| pabelanger | clarkb: ya, that looks like it | 16:34 |
| clarkb | portdirect: you can file a bug on their launchpad or track them down on irc (or both) | 16:35 |
| portdirect | clarkb: roger | 16:35 |
| clarkb | portdirect: smoser is the freenode nick iirc | 16:35 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Support domain_id for user operations https://review.openstack.org/497483 | 16:36 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Support domain_id for user operations https://review.openstack.org/497483 | 16:38 |
| *** links has quit IRC | 16:38 | |
| clarkb | pabelanger: I'm gonna grab some caffeine really quick | 16:38 |
| clarkb | pabelanger: but my hunch is this is the last bit of stuff that needs to be done | 16:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-pip master: Ensure curl is installed https://review.openstack.org/497484 | 16:40 |
| pabelanger | clarkb: curl fix^ | 16:40 |
| *** yamamoto has joined #openstack-infra | 16:41 | |
| *** efoley has joined #openstack-infra | 16:44 | |
| *** ykarel has joined #openstack-infra | 16:45 | |
| pabelanger | puppet wheel started on infracloud again | 16:45 |
| *** iyamahat has joined #openstack-infra | 16:47 | |
| *** yamamoto has quit IRC | 16:47 | |
| *** yamahata has joined #openstack-infra | 16:47 | |
| pabelanger | and passed | 16:47 |
| *** dtantsur is now known as dtantsur|afk | 16:47 | |
| pabelanger | on compute nodes now | 16:47 |
| clarkb | reviewing curl fix now | 16:48 |
| *** Swami has quit IRC | 16:49 | |
| clarkb | I see syslog stuff on 009 that looks like ansible | 16:49 |
| *** psachin has joined #openstack-infra | 16:51 | |
| *** lucasagomes is now known as lucas-afk | 16:51 | |
| *** Apoorva has joined #openstack-infra | 16:51 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Support domain_id for user operations https://review.openstack.org/497483 | 16:51 |
| clarkb | can we get a second reviewing on the curl fix above? It will touch a lot of things so not approving with single reviewer | 16:52 |
| *** efoley has quit IRC | 16:54 | |
| *** derekh has quit IRC | 16:55 | |
| clarkb | pabelanger: syncing puppet modules is slow? I guess that isn't surprising considering the network bw issues we have had there | 16:58 |
| pabelanger | clarkb: ya, think that is the reason | 16:58 |
| pabelanger | clarkb: did HPE ever comeback with reason for it? | 16:59 |
| clarkb | pabelanger: I haven't seen one | 16:59 |
| *** martinkopec has quit IRC | 17:00 | |
| clarkb | pabelanger: looking at ansible on puppetmaster the infracloud run has a bunch of defunct processes so wondering if it will actually progress | 17:05 |
| *** slaweq has joined #openstack-infra | 17:06 | |
| *** ykarel has quit IRC | 17:06 | |
| pabelanger | clarkb: possible they timed out on networking | 17:06 |
| pabelanger | clarkb: we might have to disable infracloud crontab and kick.sh one to see what happens | 17:07 |
| clarkb | ya thats what I am thinking | 17:07 |
| pabelanger | ya, lets try that | 17:07 |
| clarkb | what is the best way to stop the existing run? | 17:07 |
| jlvillal | $ host download.cirros-cloud.net | 17:07 |
| jlvillal | Host download.cirros-cloud.net not found: 2(SERVFAIL) | 17:07 |
| jlvillal | portdirect: ^^ :( | 17:08 |
| pabelanger | clarkb: I've used kill in the past for those processes | 17:08 |
| portdirect | jlvillal: looks like dreamhost DNS is down | 17:08 |
| portdirect | so expect it back up soon (I hope...) | 17:08 |
| clarkb | portdirect: the defunct ones? | 17:08 |
| jlvillal | :( That isn't good... | 17:08 |
| clarkb | er pabelanger ^ | 17:08 |
| pabelanger | clarkb: ya | 17:09 |
| clarkb | pabelanger: ok trying that now | 17:09 |
| pabelanger | I'll get crontab | 17:09 |
| jlvillal | portdirect: Thanks for the info | 17:10 |
| pabelanger | clarkb: k, also added puppetmaster.o.o to emergency host | 17:10 |
| clarkb | pabelanger: I think I actually need to kill the parent process | 17:10 |
| clarkb | since its not cleaning up its children | 17:10 |
| pabelanger | clarkb: k | 17:10 |
| clarkb | pabelanger: ok thats done | 17:11 |
| clarkb | pabelanger: let me know when cron is done and then its kick.sh time | 17:12 |
| pabelanger | clarkb: k, did you want to kick.sh one? | 17:12 |
| pabelanger | cron is done | 17:12 |
| clarkb | ya I'm gonna do it on compute009.vanilla.ic.openstack.org since that is the host I am already one and can watch from both sides | 17:12 |
| pabelanger | k | 17:12 |
| clarkb | does kick.sh require root? | 17:14 |
| * clarkb runs screen as root anyways | 17:14 | |
| pabelanger | clarkb: ya, I usually use root | 17:15 |
| *** trown is now known as trown|lunch | 17:15 | |
| clarkb | pabelanger: I've got screen running if you want to attach | 17:15 |
| pabelanger | clarkb: ready | 17:15 |
| jlvillal | portdirect: https://twitter.com/dhstatus They say a DDoS attack on their DNS | 17:16 |
| *** ldnunes has quit IRC | 17:16 | |
| clarkb | pabelanger: seems to have quickly gotten past the module copy this time... | 17:16 |
| pabelanger | ya | 17:16 |
| clarkb | pabelanger: might just need to throttle it a bit more and do one at a time | 17:16 |
| pabelanger | Ha, hit curl issue too | 17:16 |
| *** ldnunes has joined #openstack-infra | 17:17 | |
| pabelanger | clarkb: ya, let me see what we limit it too | 17:17 |
| clarkb | pabelanger: puppet failed because no curl there either | 17:17 |
| *** lin_yang has joined #openstack-infra | 17:17 | |
| clarkb | but cert seems to have updated | 17:17 |
| pabelanger | Ya, lets see if a node launches | 17:17 |
| clarkb | pabelanger: I'm gonna do a listing of computes that are working and start running kick.sh against them one by one | 17:18 |
| pabelanger | ++ | 17:18 |
| clarkb | pabelanger: let me know if 009 can boot an instance | 17:20 |
| pabelanger | clarkb: we could also add serial: x for compute hosts too. maybe 5 at a time | 17:20 |
| pabelanger | ya, watching | 17:20 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Support domain_id for user operations https://review.openstack.org/497483 | 17:20 |
| clarkb | or even one at a time. Not sure what the bw will support | 17:20 |
| clarkb | but ya | 17:20 |
| pabelanger | no attempts on 009 yet | 17:22 |
| pabelanger | looks like it cleaned up a leaked instance | 17:24 |
| pabelanger | so that is good | 17:24 |
| clarkb | ya I think if it was able to delete then that will exercise everything but glance interaction | 17:24 |
| pabelanger | ya | 17:24 |
| *** rbrndt has quit IRC | 17:26 | |
| pabelanger | actually, I see nodepool uploading something to infracloud | 17:26 |
| pabelanger | I'll check glance logs | 17:26 |
| *** rcernin has joined #openstack-infra | 17:27 | |
| *** jtomasek has quit IRC | 17:27 | |
| *** dmsimard is now known as dmsimard|afk | 17:30 | |
| pabelanger | k, see some movement on compute009 | 17:30 |
| pabelanger | ya, imagecache update. | 17:31 |
| pabelanger | So, think SSL cert is working | 17:31 |
| pabelanger | I didn't see any errors in glance log too | 17:31 |
| clarkb | woo | 17:32 |
| *** Swami has joined #openstack-infra | 17:33 | |
| *** tosky has quit IRC | 17:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Limit ansible-playbooks to 1 host for infracloud https://review.openstack.org/497530 | 17:36 |
| pabelanger | clarkb: ^ | 17:36 |
| *** ykarel has joined #openstack-infra | 17:38 | |
| *** gixxer1k has joined #openstack-infra | 17:39 | |
| clarkb | +2 | 17:39 |
| clarkb | do we have any used nodes in vanilla yet? | 17:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/497165 | 17:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Fix some markup in v3 docu https://review.openstack.org/497532 | 17:41 |
| AJaeger | Shrews: followup to your change with fixing remaining literals ^ | 17:42 |
| *** yamamoto has joined #openstack-infra | 17:43 | |
| clarkb | I still don't see any in use on grafana | 17:44 |
| clarkb | that could just be our low quota against the number of hypervisors | 17:46 |
| pabelanger | Ya | 17:46 |
| AJaeger | clarkb: in-use is 29 on http://grafana.openstack.org/dashboard/db/nodepool-infra-cloud - is that correct? | 17:46 |
| clarkb | AJaeger: ya its all in chocolate though | 17:47 |
| clarkb | AJaeger: vanilla is the one that is currently sad | 17:47 |
| clarkb | speaking of chocolate we should get the pip curl fix in ebfore we update its cert | 17:47 |
| clarkb | and make sure puppet is puppeting both clouds properly | 17:47 |
| AJaeger | clarkb: Ah, I see - didn't read all scrollback. I note that vanilla is deleting 15 nodes, is that correct? | 17:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Update infracloud API queries https://review.openstack.org/497534 | 17:48 |
| pabelanger | AJaeger: clarkb: ^ should help render API requests into grafana | 17:48 |
| clarkb | AJaeger: ya I think those nodes are on compute hosts that we haven't fixed yet | 17:48 |
| clarkb | AJaeger: they should delete properly once I get the cert updated everywhere | 17:48 |
| *** yamamoto has quit IRC | 17:48 | |
| AJaeger | ok, understood. Btw. Zuul is updated again, I saw project-config changes applied and repos imported... | 17:49 |
| clarkb | AJaeger: thank you for confirming that is working | 17:49 |
| pabelanger | clarkb: Oh, I think it might be glance | 17:51 |
| pabelanger | Build of instance d071a56e-c9a7-4b41-8431-1f241b59208a aborted: Connection to glance host https://controller00.vanilla.ic.openstack.org:9292 failed: Error finding address for https://controller00.vanilla.ic.openstack.org:9292/v1/images/d6d7c8aa-47eb-4362- | 17:51 |
| pabelanger | checking that now | 17:51 |
| *** ykarel has quit IRC | 17:52 | |
| pabelanger | ya, glance-api is the issue | 17:52 |
| clarkb | s_client against port 9292 looks ok | 17:52 |
| pabelanger | http://paste.openstack.org/show/619346/ | 17:53 |
| clarkb | pabelanger: those are just warnings and should be ok | 17:53 |
| pabelanger | k | 17:54 |
| pabelanger | yay | 17:54 |
| pabelanger | we have a node online | 17:54 |
| pabelanger | just took a while | 17:54 |
| clarkb | tailing compute009 I haven't seen anything | 17:54 |
| clarkb | ok cool | 17:54 |
| pabelanger | | 10614414 | infracloud-vanilla | None | ubuntu-xenial | zuul | zl01 | ubuntu-xenial-infracloud-vanilla-10614414 | ubuntu-xenial-infracloud-vanilla-10614414 | e2884a31-3d74-4b8d-86d7-dd92d836c466 | 15.184.65.35 | used | 00:00:01:27 | None | | 17:55 |
| *** tosky has joined #openstack-infra | 17:55 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Fix unbound variable DELOREAN_LINK https://review.openstack.org/497221 | 17:55 |
| clarkb | ya I think its jsut a matter of getting all the computes done then checking again if there are errors | 17:55 |
| clarkb | if not then merging fixes and planning for chocolate | 17:56 |
| pabelanger | ++ | 17:56 |
| *** SumitNaiksatam has joined #openstack-infra | 17:56 | |
| pabelanger | clarkb: do we want to land https://review.openstack.org/497530/ and have crontab try to do the rest? | 17:57 |
| AJaeger | pabelanger: you change one grafana file in https://review.openstack.org/#/c/497534 - shouldn't you update *all* nodepool-CLOUD.yaml files? | 17:57 |
| clarkb | pabelanger: I think lets wait. We can double check crontab works after we get it fixed | 17:57 |
| clarkb | (right now I just want a workign cloud0 | 17:57 |
| pabelanger | AJaeger: ya, I was going to propose a follow up once I knew grafana was good. I can do that now too | 17:58 |
| pabelanger | clarkb: wfm | 17:58 |
| pabelanger | grabbing a coffee | 17:58 |
| AJaeger | pabelanger: if you do a followup, I can +2.. | 17:58 |
| *** jbadiapa has quit IRC | 18:00 | |
| clarkb | sean-k-mooney: now that things on infracloud are settlign down... | 18:03 |
| clarkb | sean-k-mooney: re nested virt. We avoid it for several reasons. The biggest is it isn't available everywhere so don't want to rely on it. But even in places where it is in theory available we have found it to be extremely flaky | 18:04 |
| clarkb | sean-k-mooney: which isn't surprising considering the linux kernel default is to turn it off for this reason on intel cpus. (it is enabled on amd thoguh because apparently that crashes less) | 18:04 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Add domain_id to groups https://review.openstack.org/497537 | 18:04 |
| clarkb | sean-k-mooney: this would be less of an issue if we had mroe ability to debug crashes but unfortunately we have little to no insight into the cpu modules, kernel versions, libvirt/qemu versions etc in use on our clouds. We also have no access to hypervisor logs so when things break our only real option is to just not use the feature | 18:05 |
| pabelanger | AJaeger: I'll push up an update to 497534 shortly, we can get them all and see what fails | 18:05 |
| clarkb | *cpu models | 18:06 |
| clarkb | sean-k-mooney: and it isn't just fear of failures we have seen crashes of both the innermost VMs and the VMs the test jobs run on in different clouds and regions when the feature is enabled | 18:07 |
| *** funzo has joined #openstack-infra | 18:07 | |
| mnaser | any issues on the iweb mirrors? | 18:08 |
| mnaser | http://logs.openstack.org/19/497419/1/check/gate-puppet-horizon-puppet-unit-4.8-centos-7/b8d0cd7/console.html | 18:08 |
| pabelanger | mnaser: possible, we've been debugging some 502 bad gateway issues | 18:08 |
| mnaser | okay cool just wanted to know :) | 18:09 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add support for updating initramfs images in CI https://review.openstack.org/479900 | 18:09 |
| pabelanger | mnaser: that's the first time I have seen that. | 18:09 |
| pabelanger | mnaser: I can check apache logs shortly | 18:09 |
| mnaser | pabelanger no worries, another failure - http://logs.openstack.org/19/497419/1/check/gate-puppet-horizon-puppet-beaker-rspec-centos-7/32cc4fb/console.html -- if that helps | 18:10 |
| *** trown|lunch is now known as trown | 18:10 | |
| *** funzo has quit IRC | 18:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Run sign-artifacts role on executor https://review.openstack.org/497539 | 18:15 |
| pabelanger | mnaser: Hmm, another inap failure. Okay, I wonder if this is networking issue | 18:16 |
| *** vhosakot has quit IRC | 18:16 | |
| *** vhosakot has joined #openstack-infra | 18:16 | |
| pabelanger | mnaser: logstash seems to indicate its limited to inap-mtl01 | 18:17 |
| mnaser | pabelanger yeah i've seen 2 jobs fail but i havent seen a lot of jobs over the day today so | 18:17 |
| clarkb | pabelanger: any apache errors? | 18:17 |
| pabelanger | clarkb: nothing new that I see | 18:18 |
| pabelanger | mnaser: clarkb: I am thinking it is Bug 1617309 | 18:19 |
| openstack | bug 1617309 in OpenStack-Gate "internap-mtl01 network connectivity issues" [Undecided,New] https://launchpad.net/bugs/1617309 | 18:19 |
| pabelanger | we have an existing elastic-recheck query | 18:19 |
| pabelanger | I'll see about updating it for gems in a moment | 18:19 |
| mnaser | on a super unrelated note. does dib have the ability of caching packages locally to avoid hitting the internet when installing them when the image is booted? | 18:25 |
| pabelanger | mnaser: ya, we usused to do that with devstack-cache element | 18:25 |
| clarkb | check out the vanilla in use number :) | 18:25 |
| clarkb | last compute is puppeting now | 18:26 |
| mnaser | pabelanger oh awesome, i'll check that out | 18:26 |
| pabelanger | mnaser: we stopped once we started moving more things to the mirrors | 18:26 |
| pabelanger | clarkb: yay | 18:26 |
| clarkb | pabelanger: I'm wondering if maybe those failed ansible processes were trying to hit computes that no longer exist? | 18:26 |
| *** rbrndt has joined #openstack-infra | 18:26 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Document list of configuration items for include/exclude https://review.openstack.org/496928 | 18:26 |
| mnaser | pabelanger the use case is sahara uses dib and it pulls so much data on every install/deployment | 18:26 |
| pabelanger | clarkb: ya, possible | 18:26 |
| pabelanger | mnaser: ya, so look at bindep-cache and devstack-cache in project-config. You'll likely need to use git history to see orginal versions, but we did the same | 18:27 |
| clarkb | ok all up and enabled computes have been puppeted now | 18:28 |
| pabelanger | mnaser: basically, apt-get --download-only foo at build time, then on boot, we did apt-get install foo | 18:28 |
| mnaser | pabelanger exactly what i'm looking for, i figured it was done before. i'll do some more reading :) | 18:28 |
| pabelanger | mnaser: cache-bindep then :) | 18:29 |
| clarkb | infra-root if we can get https://review.openstack.org/#/c/497530/ in then we can let ansible/puppet loose on the infracloud nodes again and hopefully everything will be happy | 18:29 |
| pabelanger | or bindep-cache | 18:29 |
| *** jpena is now known as jpena|off | 18:32 | |
| *** dasanind has joined #openstack-infra | 18:32 | |
| *** olaph has joined #openstack-infra | 18:38 | |
| *** dprince has quit IRC | 18:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Run sign-artifacts role on executor https://review.openstack.org/497539 | 18:40 |
| *** olaph1 has quit IRC | 18:40 | |
| *** ijw has quit IRC | 18:40 | |
| *** eharney has quit IRC | 18:40 | |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| *** Vinod has joined #openstack-infra | 18:45 | |
| Vinod | hello | 18:46 |
| Vinod | can anybody help on why the review is not triggering jenkins tests | 18:47 |
| Vinod | https://review.openstack.org/#/c/497547/ | 18:47 |
| jeblair | clarkb: i +3d that about 30m ago; gate backup i think | 18:47 |
| jeblair | yeah, it's only about halfway through check | 18:47 |
| clarkb | jeblair: ah thanks | 18:47 |
| clarkb | Vinod: it is queued in zuul, you can see that at http://status.openstack.org/zuul | 18:48 |
| Vinod | clarkb: thank you | 18:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix matched condition for gpg signing https://review.openstack.org/497551 | 18:49 |
| clarkb | grafana says we no longer have error'd launch attempts in vanilla | 18:49 |
| *** ramishra has quit IRC | 18:50 | |
| *** yamamoto has quit IRC | 18:50 | |
| * clarkb attempts to catch up on email | 18:52 | |
| *** rmcall has joined #openstack-infra | 18:58 | |
| *** vinod_ has joined #openstack-infra | 18:59 | |
| *** wolverineav has joined #openstack-infra | 18:59 | |
| *** florianf has quit IRC | 19:00 | |
| *** masber has quit IRC | 19:00 | |
| openstackgerrit | Divyansh Acharya proposed openstack/os-testr master: Just changed the opening of the file https://review.openstack.org/494697 | 19:01 |
| clarkb | pabelanger: do you want to update https://review.openstack.org/#/c/497484/1 really quick to fix the curl thing? bug was caught by testing and I commented what the fix is | 19:11 |
| clarkb | pabelanger: I can push the fix too if it helps | 19:11 |
| openstackgerrit | Merged openstack-infra/system-config master: Limit ansible-playbooks to 1 host for infracloud https://review.openstack.org/497530 | 19:11 |
| pabelanger | clarkb: Ah | 19:12 |
| pabelanger | ya, 1 sec | 19:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-pip master: Ensure curl is installed https://review.openstack.org/497484 | 19:13 |
| pabelanger | forgetting puppet syntax now | 19:13 |
| *** SumitNaiksatam has quit IRC | 19:15 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix matched condition for gpg signing https://review.openstack.org/497551 | 19:16 |
| *** jamesmcarthur has quit IRC | 19:17 | |
| *** jamesmcarthur has joined #openstack-infra | 19:18 | |
| *** eharney has joined #openstack-infra | 19:22 | |
| *** camunoz has quit IRC | 19:23 | |
| clarkb | pabelanger: also needs more commas :) | 19:27 |
| pabelanger | blarg | 19:27 |
| pabelanger | clarkb: Oh, is there an issue? because I read that as there is an issue | 19:28 |
| clarkb | pabelanger: yes there is an issue the lines aboev the ones your added need trailing commas | 19:28 |
| clarkb | http://logs.openstack.org/84/497484/2/check/gate-infra-puppet-apply-3-ubuntu-trusty/f62ec25/puppetapplytest01.final.out.FAILED errors there | 19:28 |
| clarkb | (sorry multitasking and failing at communication) | 19:28 |
| *** sshnaidm is now known as sshnaidm|off | 19:29 | |
| pabelanger | ya | 19:29 |
| pabelanger | I see now | 19:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-pip master: Ensure curl is installed https://review.openstack.org/497484 | 19:29 |
| clarkb | unrelated, but I am going to try tackling our ptg etherpad to organize it a bit better. I'm hoping it won't destroy authorship colors but might and if it does, sorry | 19:29 |
| clarkb | but first I am going to review the ansiblified network overlay change | 19:29 |
| pabelanger | k | 19:29 |
| pabelanger | I am trying to debug why gpg signing failed | 19:30 |
| pabelanger | invalid packet from gpg | 19:30 |
| pabelanger | infra-root: has anybody seen invalid packet from gpg before? http://logs.openstack.org/06/f28eceb5651affee7601d66e91ea1d706b56b506/release/release-openstack-python/388de7c/job-output.txt.gz#_2017-08-24_19_19_51_797528 | 19:31 |
| pabelanger | I am wondering if our gpg secret is not correct | 19:31 |
| clarkb | dmsimard|afk: jeblair I am going to recheck the network overlay change again as it ran with the broken dvr ha job and not the dvr grenade one we swapped in to better test the overlay. Also going to check experimental as ithink the non ha dvr job is in there | 19:31 |
| *** kgiusti has quit IRC | 19:32 | |
| clarkb | pabelanger: arch linux bug tracker reports that keys changed format at some point and had to be imported from old into new and that could cause that behavior. So ya I agree that something is probably wrong with the key (maybe just wrong data or wrong format?) | 19:34 |
| pabelanger | clarkb: this is likely a question for fungi, the googles says gpg --export --armor KEY / gpg --export-secret-key --armor KEY are ways of exporting gpgkeys to a new machine. Then, using --import on the new system. Any reason not to do that? Today, I think we are just copying the binary data | 19:42 |
| pabelanger | and unsure if ansible vars would like that | 19:42 |
| clarkb | thats like how you want to store them to make them resilient to different versions of gnupg | 19:45 |
| clarkb | jeblair: dmsimard|afk I left a comment on the ansible change for network overlays. Its mostly a "I think this may not do the right thing unless ansible magic" and hoping you all know more. Other than that it looks good and I think its now just a matter of checking test results | 19:45 |
| *** yamamoto has joined #openstack-infra | 19:46 | |
| clarkb | pabelanger: --armor makes it ascii armored so should be safe in yaml | 19:47 |
| clarkb | but its being encrypted on top of that, all that to say I don't expect thats a problem | 19:47 |
| clarkb | dmsimard|afk: jeblair I am a little torn on the way things have been split out (specifically the peers role delegating to primary seems like a weird way to do it) but thats not a bikeshed we need to paint now | 19:49 |
| pabelanger | clarkb: Right, that's what I've been reading too. let me check out docs on signing01 | 19:49 |
| pabelanger | clarkb: also, we missed spacex launch | 19:50 |
| pabelanger | totally forgot about it | 19:50 |
| clarkb | oh fail | 19:50 |
| *** yamamoto has quit IRC | 19:51 | |
| *** yamamoto has joined #openstack-infra | 19:51 | |
| *** vhosakot_ has joined #openstack-infra | 19:54 | |
| *** vhosakot has quit IRC | 19:54 | |
| *** psachin has quit IRC | 19:57 | |
| *** psachin has joined #openstack-infra | 19:58 | |
| *** gouthamr has quit IRC | 20:01 | |
| *** sshnaidm|off has quit IRC | 20:03 | |
| *** sshnaidm has joined #openstack-infra | 20:04 | |
| pabelanger | jeblair: mordred: fungi: does that make sense? Should we switch to using --export / --import gpg commands for adding out gpg_key secret to executor? I'm not to familiar with this process as you can see. | 20:04 |
| *** sshnaidm is now known as sshnaidm|off | 20:04 | |
| pabelanger | https://www.phildev.net/pgp/gpg_moving_keys.html is what I am currently looking at | 20:04 |
| *** gouthamr has joined #openstack-infra | 20:05 | |
| jeblair | pabelanger: what are you doing currently? | 20:05 |
| *** ijw has joined #openstack-infra | 20:05 | |
| *** caphrim007 has joined #openstack-infra | 20:06 | |
| clarkb | infraers, when you get a chance please look over https://etherpad.openstack.org/p/infra-ptg-queens I've reorganized it a bit in rough chronological order | 20:06 |
| *** blznblzn2 has quit IRC | 20:06 | |
| clarkb | dhellmann: asettle for your docs+infra helproom topic do you know if you have some existing time constraints we will want to accomodate? | 20:06 |
| dhellmann | clarkb : is that on the infra etherpad or the docs etherpad? I'm not sure what we had in mind there. | 20:07 |
| clarkb | dhellmann: its on the infra one linked just above | 20:07 |
| clarkb | you'll see it if you grep your nickl | 20:08 |
| dhellmann | got it | 20:08 |
| dhellmann | oh, retention policy | 20:08 |
| dhellmann | we should make sure pkovar is there, too | 20:08 |
| dhellmann | he's the new ptl | 20:08 |
| clarkb | ++ | 20:08 |
| *** esberglu has quit IRC | 20:08 | |
| dhellmann | well, I think I'm only triple booked on both those days | 20:09 |
| *** esberglu has joined #openstack-infra | 20:09 | |
| dhellmann | I have down that I promised to do docs things on monday, so if we could talk about it at some point that day it would be good | 20:09 |
| *** esberglu has quit IRC | 20:10 | |
| dhellmann | maybe not first thing in the morning, since the docs team is doing some of its own planning then | 20:10 |
| dhellmann | after lunch? | 20:10 |
| *** esberglu has joined #openstack-infra | 20:10 | |
| dhellmann | clarkb: the docs and i18n teams are sharing a room, so that would be a good time for the i18n team to talk about things the docs team isn't needed for, and we could come drop in on the infra room | 20:11 |
| clarkb | dhellmann: ok I'll mark it down for monday afternoon | 20:11 |
| dhellmann | clarkb : thanks | 20:11 |
| *** mat128 has quit IRC | 20:11 | |
| clarkb | pabelanger: ugh my favorite puppet behavior http://logs.openstack.org/84/497484/3/check/gate-infra-puppet-apply-3-ubuntu-trusty/6b6a5dd/puppetapplytest19.final.out.FAILED | 20:13 |
| pabelanger | jeblair: I am not sure, mordred setup the secret, but I think it is just binary format that we are writing directly into pubring.gpg and secring.gpg | 20:14 |
| clarkb | pabelanger: there are a few of those. I'm going to go grab lunch then when I get back I can throw up a bunch of patches to the various things to wrap curl in the conditioanl install | 20:14 |
| pabelanger | clarkb: k, thanks | 20:15 |
| jeblair | pabelanger: okay, i don't think we should explore other options until we understand what we're currently doing. | 20:15 |
| *** thorst_afk has quit IRC | 20:16 | |
| jeblair | pabelanger: why don't you start by checking the sha1sum of the resulting files and make sure they are what you expect? | 20:16 |
| clarkb | dmellado: can you expand on what you are thinking for more flexible way of gating in https://etherpad.openstack.org/p/infra-ptg-queens ? I think that will help start conversation on the right track | 20:16 |
| clarkb | ianw: when you wake https://etherpad.openstack.org/p/infra-ptg-queens has topic for talking about python3.6 can you write down your rough thoughts/plans there? I don't think you will be joining us in Denver right? so would be good to have your input there since you are working on that now | 20:17 |
| pabelanger | jeblair: sure, we can add a stat into the role to get the data, otherwise I need some help figuring out how to do that inside zuul-bwrap | 20:18 |
| openstackgerrit | James E. Blair proposed openstack/gertty master: Speed up loading change screen https://review.openstack.org/497600 | 20:18 |
| *** srobert has quit IRC | 20:18 | |
| jeblair | pabelanger: "command: sha1sum {{ file }}" ? | 20:19 |
| pabelanger | jeblair: ya, stat will do that too | 20:19 |
| pabelanger | 1 sec | 20:19 |
| *** thorst_afk has joined #openstack-infra | 20:19 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade master: Make endpoint id an optional paramater https://review.openstack.org/497601 | 20:20 |
| *** gouthamr has quit IRC | 20:22 | |
| *** iyamahat has quit IRC | 20:23 | |
| *** rcrit has left #openstack-infra | 20:23 | |
| *** thorst_afk has quit IRC | 20:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Stat pubring.gpg / secring.gpg for debugging https://review.openstack.org/497603 | 20:27 |
| openstackgerrit | Merged openstack-infra/shade master: Fix cleaning of Cinder volumes in functional tests https://review.openstack.org/496939 | 20:27 |
| *** ijw has quit IRC | 20:28 | |
| *** eharney has quit IRC | 20:28 | |
| *** iyamahat has joined #openstack-infra | 20:28 | |
| *** ijw has joined #openstack-infra | 20:31 | |
| *** jkilpatr has quit IRC | 20:31 | |
| *** mriedem1 has joined #openstack-infra | 20:32 | |
| *** mriedem has quit IRC | 20:33 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: sha1sum pubring.gpg / secring.gpg for debugging https://review.openstack.org/497603 | 20:33 |
| *** derekh has joined #openstack-infra | 20:35 | |
| *** mriedem1 is now known as mriedem | 20:35 | |
| *** psachin has quit IRC | 20:36 | |
| *** thorst_afk has joined #openstack-infra | 20:36 | |
| *** hamzy has quit IRC | 20:42 | |
| *** srobert has joined #openstack-infra | 20:43 | |
| *** yamamoto has quit IRC | 20:45 | |
| *** derekh has quit IRC | 20:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-ethercalc master: Allow multiple places to install curl https://review.openstack.org/497611 | 20:47 |
| *** slaweq has quit IRC | 20:47 | |
| *** srobert has quit IRC | 20:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstackid master: Allow multiple places to install curl https://review.openstack.org/497613 | 20:49 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-etherpad_lite master: Allow multiple places to install curl https://review.openstack.org/497614 | 20:49 |
| *** trown is now known as trown|outtypewww | 20:49 | |
| clarkb | pabelanger: topic:curl ^ should allow puppet-pip to install curl | 20:50 |
| clarkb | jeblair: ^ | 20:50 |
| * clarkb updates puppet-pip with some depends on | 20:50 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-pip master: Ensure curl is installed https://review.openstack.org/497484 | 20:51 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: sha1sum pubring.gpg / secring.gpg for debugging https://review.openstack.org/497603 | 20:52 |
| pabelanger | +2's | 20:52 |
| *** ldnunes has quit IRC | 20:54 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: OpenStackID Release 1.0.23 https://review.openstack.org/497616 | 20:55 |
| *** jamesmcarthur has quit IRC | 20:56 | |
| *** caphrim007 has quit IRC | 20:56 | |
| *** wolverineav has quit IRC | 20:57 | |
| jeblair | +3s | 20:57 |
| *** wolverineav has joined #openstack-infra | 20:58 | |
| jeblair | well i +2d the last one | 20:58 |
| *** jamesmcarthur has joined #openstack-infra | 20:58 | |
| clarkb | I'll approve puppet-pip one if it comes back gree | 20:58 |
| clarkb | *green | 20:58 |
| *** slaweq has joined #openstack-infra | 20:59 | |
| *** jkilpatr has joined #openstack-infra | 21:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Support Zuul v3 repo layout https://review.openstack.org/497618 | 21:00 |
| *** wolverineav has quit IRC | 21:02 | |
| *** gildub has joined #openstack-infra | 21:06 | |
| *** gouthamr has joined #openstack-infra | 21:06 | |
| *** jkilpatr has quit IRC | 21:07 | |
| *** rmcall has quit IRC | 21:07 | |
| *** ldnunes has joined #openstack-infra | 21:07 | |
| *** jtomasek has joined #openstack-infra | 21:10 | |
| *** jascott1 has quit IRC | 21:10 | |
| *** jtomasek has quit IRC | 21:11 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid master: Allow multiple places to install curl https://review.openstack.org/497613 | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack-infra/puppet-ethercalc master: Allow multiple places to install curl https://review.openstack.org/497611 | 21:17 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Allow multiple places to install curl https://review.openstack.org/497614 | 21:19 |
| *** rcernin has quit IRC | 21:19 | |
| *** jkilpatr has joined #openstack-infra | 21:19 | |
| *** dave-mccowan has quit IRC | 21:20 | |
| *** Goneri has quit IRC | 21:22 | |
| clarkb | jeblair: when you ahve a moment https://review.openstack.org/#/c/494282/ has a question that ianw asked that I think you may be best able to answer? problem was on the zuulv3 node right? I don't recall if syslog was checked | 21:23 |
| *** jascott1 has joined #openstack-infra | 21:23 | |
| pabelanger | okay, sha1sum for pubring / secring do not match signing01 | 21:25 |
| pabelanger | http://logs.openstack.org/a8/31f1a36ad99a25a824e06a86d25765609dc428a8/release/release-openstack-python/aaf2b73/job-output.txt.gz#_2017-08-24_21_20_55_587206 | 21:25 |
| pabelanger | however, I am not sure if that was the original value for the secret | 21:26 |
| jeblair | clarkb, ianw: i'm sorry, i don't know. all i know is that systemd said it couldn't tell me why something wasn't running because it only had a few days of logs. are you saying i should have been able to find the reason in /var/log/syslog? i don't know whether i looked or not, it was a long time ago. | 21:26 |
| jeblair | pabelanger: then i think the next thing is to verify with mordred whether that's what we expect. | 21:27 |
| clarkb | jeblair: ya I guess that is the intent on ubuntu. That said I think double logging in this case may be advantageous simply because search and manipulation is easier with journald | 21:27 |
| *** jascott1 has quit IRC | 21:27 | |
| clarkb | as long as we don't end up filling up our disks I think I would prefer to double log | 21:28 |
| *** ijw has quit IRC | 21:28 | |
| clarkb | (but I don't feel super strongly about that) | 21:28 |
| jeblair | clarkb: i don't agree that search and manipulation are easier. i understand you do, but so far, my only experience with systemd is it telling me that logs are missing (whether here, or in the case of the git servers, completly empty). | 21:29 |
| clarkb | yup, the xenial case would be fixed by my change (but possibly result in double logging). The centos one would not be fixed by my change as it was related to selinux | 21:30 |
| *** jascott1 has joined #openstack-infra | 21:30 | |
| clarkb | I think part of the disconnect here is that if you use systemctl it tells you "go askthis journalctl thing" but in ubuntu's case it should tell you to look in syslog instead | 21:32 |
| *** jamesmcarthur has quit IRC | 21:39 | |
| *** jamesmcarthur has joined #openstack-infra | 21:40 | |
| *** yamamoto has joined #openstack-infra | 21:46 | |
| *** ijw has joined #openstack-infra | 21:49 | |
| *** sekelso has quit IRC | 21:50 | |
| *** yamamoto has quit IRC | 21:51 | |
| *** slaweq has quit IRC | 21:52 | |
| *** jamesmcarthur has quit IRC | 21:53 | |
| *** jamesmcarthur has joined #openstack-infra | 21:54 | |
| *** jamesmcarthur has quit IRC | 21:55 | |
| *** ijw has quit IRC | 21:57 | |
| pabelanger | jeblair: agree | 21:57 |
| *** caphrim007 has joined #openstack-infra | 22:00 | |
| *** ijw has joined #openstack-infra | 22:02 | |
| *** ihrachys_ has joined #openstack-infra | 22:04 | |
| *** ihrachys has quit IRC | 22:06 | |
| *** ihrachys_ is now known as ihrachys | 22:07 | |
| clarkb | something up with the puppet-pip centos rspec job http://logs.openstack.org/84/497484/4/check/gate-puppet-pip-puppet-beaker-rspec-centos-7/16d6b4f/console.html#_2017-08-24_21_21_39_651464 was last run but looks like it is going to do it again | 22:09 |
| *** esberglu has quit IRC | 22:10 | |
| openstackgerrit | Merged openstack/os-testr master: Just changed the opening of the file https://review.openstack.org/494697 | 22:11 |
| clarkb | thats interesting we force pip to talk to the dfw mirror | 22:12 |
| clarkb | then we install a list of packages twice, then point at pypi and install list again | 22:13 |
| clarkb | oh we only point at the mirror in dfw if we don't already have an /etc/pip.conf | 22:14 |
| clarkb | I wonder if pip talks to pypi no matter what and the iptables rule is making that slwo to fail over? | 22:14 |
| *** signed8bit is now known as signed8bit_Zzz | 22:15 | |
| clarkb | currently there are ~24 packages we install. I wonder if we can trim that list down to one and make it a valid test | 22:18 |
| *** jcoufal has quit IRC | 22:18 | |
| pabelanger | ya | 22:18 |
| pabelanger | we've done that before | 22:18 |
| pabelanger | removed broken things | 22:18 |
| pabelanger | python error might be a build dependency issue too | 22:18 |
| clarkb | I think if we trim it down to just shade that will be a real world exercise that is non trivial | 22:19 |
| pabelanger | wfm | 22:19 |
| clarkb | it looks like gerritbot specifically may be failing | 22:21 |
| clarkb | but I'm not super concerned about gerritbot on centos right now. I'll put a patch up to trim the list | 22:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-pip master: Trim the test package list to just shade https://review.openstack.org/497683 | 22:23 |
| clarkb | lets see if things pass there | 22:23 |
| *** rlandy is now known as rlandy|bbl | 22:31 | |
| *** ijw has quit IRC | 22:33 | |
| *** ihrachys has quit IRC | 22:34 | |
| clarkb | pabelanger: ya ^ passed testing | 22:36 |
| clarkb | so maybe get that in then recheck the curl fix in puppet-pip | 22:37 |
| *** caphrim0_ has joined #openstack-infra | 22:41 | |
| *** dhajare has quit IRC | 22:41 | |
| *** gixxer1k has quit IRC | 22:41 | |
| ianw | jeblair: i believe messages go in "text" to /var/log/syslog? so just the message, but looses all the other meta-data each log gets with journal? but if we want the whole journal, i don't care, it's just different to upstream chosen defaults | 22:42 |
| *** caphrim007 has quit IRC | 22:44 | |
| dmsimard|afk | clarkb: Yeah we can definitely work on the overlay patch -- I mostly just rebased it and fixed some of the problems we had commented on before | 22:46 |
| *** dmsimard|afk is now known as dmsimard | 22:46 | |
| clarkb | dmsimard: ya thanks for that, jlk actually answered my question and updated votes | 22:46 |
| dmsimard | need to catch up, was afk most of the day, I'll read up on reviews later | 22:47 |
| clarkb | ianw: pabelanger jeblair if we can get https://review.openstack.org/497683 in (assuming you are ok with the assertion there) that will help us finally get the curl patch in to fix puppet for infracloud | 22:47 |
| *** caphrim0_ has quit IRC | 22:47 | |
| *** yamamoto has joined #openstack-infra | 22:47 | |
| *** rbrndt has quit IRC | 22:53 | |
| *** rbrndt has joined #openstack-infra | 22:53 | |
| *** rbrndt has quit IRC | 22:53 | |
| *** yamamoto has quit IRC | 22:53 | |
| *** tosky has quit IRC | 23:02 | |
| *** marst has quit IRC | 23:05 | |
| *** xarses_ has quit IRC | 23:07 | |
| *** sekelso has joined #openstack-infra | 23:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove unused zuul sites https://review.openstack.org/497694 | 23:16 |
| pabelanger | clarkb: +2 | 23:16 |
| *** jcoufal has joined #openstack-infra | 23:17 | |
| mordred | clarkb: +A | 23:20 |
| mordred | clarkb, pabelanger: ^^ found that while researching somethign else | 23:20 |
| *** jcoufal has quit IRC | 23:21 | |
| clarkb | mordred: api.o.o is 98.129.229.176 according to dns and not files.o.o | 23:22 |
| *** Apoorva has quit IRC | 23:22 | |
| clarkb | I guess we redirect to developer.o.o | 23:22 |
| mordred | yah - we don't publish anyhting to those in jobs any more | 23:22 |
| clarkb | (convincing myself its not used anymore) | 23:22 |
| *** Apoorva has joined #openstack-infra | 23:23 | |
| *** hongbin has quit IRC | 23:23 | |
| mordred | clarkb: yes - please double-check me | 23:24 |
| mordred | clarkb: I used 'git grep "site:"' in project-config | 23:24 |
| *** claudiub has quit IRC | 23:26 | |
| clarkb | ya if you hit that name its an http redirect so no content published there | 23:29 |
| openstackgerrit | Merged openstack-infra/puppet-pip master: Trim the test package list to just shade https://review.openstack.org/497683 | 23:29 |
| *** Swami has quit IRC | 23:32 | |
| *** Apoorva_ has joined #openstack-infra | 23:35 | |
| pabelanger | mordred: clarkb: +3 | 23:36 |
| *** Apoorva has quit IRC | 23:38 | |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack-infra/project-config master: Remove V1 driver gate jobs https://review.openstack.org/497696 | 23:39 |
| *** jkilpatr has quit IRC | 23:40 | |
| *** jkilpatr has joined #openstack-infra | 23:40 | |
| *** rlandy|bbl is now known as rlandy | 23:42 | |
| *** prometheanfire has joined #openstack-infra | 23:42 | |
| prometheanfire | what's up with http://logs.openstack.org/09/496209/1/check/gate-requirements-integration-dsvm-ubuntu-trusty/170140f/logs/devstack-gate-setup-workspace-new.txt ? | 23:42 |
| prometheanfire | rax hosted tests having issues? | 23:44 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove unused zuul sites https://review.openstack.org/497694 | 23:45 |
| ianw | prometheanfire: that's interesting ... is it repeatable? | 23:49 |
| *** yamamoto has joined #openstack-infra | 23:49 | |
| prometheanfire | three times so far | 23:49 |
| prometheanfire | https://review.openstack.org/496209 | 23:49 |
| ianw | /dev/xvdd 506K 506K 0 100% /mnt/config | 23:50 |
| ianw | so it's nothing weird, at least initially | 23:53 |
| *** dingyichen has joined #openstack-infra | 23:54 | |
| *** yamamoto has quit IRC | 23:55 | |
| ianw | it's quite popular in logstash too | 23:57 |
| *** yuanying_ has joined #openstack-infra | 23:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add proc to bubblewrap https://review.openstack.org/497698 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!