| boris-42____ | @ianw so not sure how to fix the things here | 00:00 |
|---|---|---|
| boris-42____ | @ianw I thought enable plugin | 00:00 |
| boris-42____ | @ianw is going to trigger all required things? | 00:01 |
| ianw | boris-42____: i think it probably does. the gate hook looks like it's old code, but i've only looked at this for 5 seconds :) | 00:01 |
| boris-42____ | @ianw I mean we can merge and repeat until get it working | 00:01 |
| *** oomichi has joined #openstack-infra | 00:02 | |
| *** yamamoto has joined #openstack-infra | 00:03 | |
| *** david-lyle has joined #openstack-infra | 00:03 | |
| *** Apoorva_ has quit IRC | 00:04 | |
| *** Apoorva has joined #openstack-infra | 00:05 | |
| *** cody-somerville has joined #openstack-infra | 00:06 | |
| *** yamamoto has quit IRC | 00:08 | |
| *** kambiz has quit IRC | 00:10 | |
| *** xyang1 has joined #openstack-infra | 00:10 | |
| *** kambiz has joined #openstack-infra | 00:13 | |
| *** gmann has quit IRC | 00:13 | |
| *** gmann has joined #openstack-infra | 00:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add missing word to docs https://review.openstack.org/485874 | 00:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Remove refspec https://review.openstack.org/485875 | 00:15 |
| *** rlandy has quit IRC | 00:22 | |
| *** mattmceuen has quit IRC | 00:27 | |
| *** jascott1 has quit IRC | 00:27 | |
| *** zhurong has joined #openstack-infra | 00:31 | |
| *** masber has quit IRC | 00:33 | |
| *** masber has joined #openstack-infra | 00:33 | |
| *** marst has joined #openstack-infra | 00:35 | |
| *** mtanino has quit IRC | 00:35 | |
| *** gildub has joined #openstack-infra | 00:38 | |
| *** thorst has joined #openstack-infra | 00:39 | |
| *** krtaylor has joined #openstack-infra | 00:39 | |
| openstackgerrit | daixianmeng proposed openstack/diskimage-builder master: update element growroot Add support for XFS format https://review.openstack.org/485895 | 00:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log skipped tasks to the job output https://review.openstack.org/485896 | 00:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacer line after loops https://review.openstack.org/485897 | 00:40 |
| *** LindaWang has joined #openstack-infra | 00:42 | |
| *** gildub has quit IRC | 00:43 | |
| *** thorst has quit IRC | 00:44 | |
| *** gildub has joined #openstack-infra | 00:44 | |
| *** jascott1 has joined #openstack-infra | 00:45 | |
| *** sree has joined #openstack-infra | 00:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Display log url when a job completes https://review.openstack.org/485898 | 00:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Bump fedora-minimal DIB_RELEASE 26 https://review.openstack.org/482570 | 00:50 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Move setfiles to outside chroot with runcon https://review.openstack.org/485899 | 00:50 |
| ianw | pabelanger: ^ some progress on selinux setting issues. basically we are doing it wrong, and it's just luck it's been working | 00:51 |
| *** sdague has quit IRC | 00:53 | |
| *** cuongnv has joined #openstack-infra | 00:57 | |
| *** dave-mccowan has joined #openstack-infra | 00:58 | |
| *** jkilpatr has quit IRC | 00:58 | |
| *** dave-mcc_ has joined #openstack-infra | 01:00 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 01:02 |
| *** dave-mccowan has quit IRC | 01:02 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Move setfiles to outside chroot with runcon https://review.openstack.org/485899 | 01:05 |
| openstackgerrit | Merged openstack-infra/release-tools master: Fix a few small things with eol_branch https://review.openstack.org/484738 | 01:11 |
| *** csomerville has joined #openstack-infra | 01:11 | |
| *** thorst has joined #openstack-infra | 01:11 | |
| *** thorst has quit IRC | 01:11 | |
| *** cuongnv has quit IRC | 01:14 | |
| *** cody-somerville has quit IRC | 01:14 | |
| openstackgerrit | Merged openstack-infra/release-tools master: Handle unknown projects in eol_branch https://review.openstack.org/484739 | 01:16 |
| *** rloo has left #openstack-infra | 01:16 | |
| openstackgerrit | Merged openstack-infra/release-tools master: Add feature to exit on warning in eol_branch https://review.openstack.org/484740 | 01:16 |
| *** mattmceuen has joined #openstack-infra | 01:18 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Display log url when a job completes https://review.openstack.org/485898 | 01:18 |
| *** cuongnv has joined #openstack-infra | 01:19 | |
| *** isaacb_ has quit IRC | 01:22 | |
| *** Daisy_ has joined #openstack-infra | 01:25 | |
| *** armax has joined #openstack-infra | 01:26 | |
| openstackgerrit | Merged openstack-infra/shade master: Replace six.itervalues with dict.values() https://review.openstack.org/485633 | 01:30 |
| *** sree has quit IRC | 01:33 | |
| *** sree has joined #openstack-infra | 01:34 | |
| *** armax has quit IRC | 01:40 | |
| *** sree has quit IRC | 01:41 | |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify Group Create https://review.openstack.org/485102 | 01:41 |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify Group List https://review.openstack.org/485108 | 01:42 |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify Group Update https://review.openstack.org/485118 | 01:42 |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify Group Delete https://review.openstack.org/485122 | 01:42 |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify Role Create https://review.openstack.org/482245 | 01:42 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Move setfiles to outside chroot with runcon https://review.openstack.org/485899 | 01:46 |
| *** sree has joined #openstack-infra | 01:48 | |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify Role List https://review.openstack.org/482246 | 01:50 |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify Role Delete https://review.openstack.org/485145 | 01:50 |
| openstackgerrit | Merged openstack-infra/shade master: Consolidate the use of self._get_and_munchify https://review.openstack.org/485523 | 01:50 |
| *** sree has quit IRC | 01:53 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Move setfiles to outside chroot with runcon https://review.openstack.org/485899 | 01:54 |
| *** iyamahat_ has quit IRC | 01:55 | |
| *** yamahata has quit IRC | 01:56 | |
| *** jerryz has quit IRC | 02:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Simplify bindep logic removing fallback support https://review.openstack.org/482650 | 02:04 |
| *** baoli has joined #openstack-infra | 02:06 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Move setfiles to outside chroot with runcon https://review.openstack.org/485899 | 02:08 |
| *** markvoelker has quit IRC | 02:09 | |
| *** markvoelker has joined #openstack-infra | 02:09 | |
| *** markvoelker has quit IRC | 02:14 | |
| *** DFFlanders has quit IRC | 02:24 | |
| *** yamamoto has joined #openstack-infra | 02:36 | |
| *** mwarad has joined #openstack-infra | 02:38 | |
| *** gouthamr has quit IRC | 02:40 | |
| *** dave-mcc_ has quit IRC | 02:42 | |
| *** yamahata has joined #openstack-infra | 02:43 | |
| *** Rockyg has joined #openstack-infra | 02:47 | |
| *** felipemonteiro has joined #openstack-infra | 02:48 | |
| *** sree has joined #openstack-infra | 02:50 | |
| *** liujiong has joined #openstack-infra | 03:04 | |
| *** r-daneel has joined #openstack-infra | 03:05 | |
| *** yamamoto has quit IRC | 03:08 | |
| *** baoli has quit IRC | 03:09 | |
| *** thorst has joined #openstack-infra | 03:12 | |
| *** xyang1 has quit IRC | 03:12 | |
| *** Sukhdev has joined #openstack-infra | 03:14 | |
| *** thorst has quit IRC | 03:17 | |
| *** armax has joined #openstack-infra | 03:18 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 03:18 |
| *** mriedem has quit IRC | 03:35 | |
| *** sree has quit IRC | 03:36 | |
| *** sree has joined #openstack-infra | 03:36 | |
| *** sree has quit IRC | 03:41 | |
| *** gcb has quit IRC | 03:49 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Check out implicit branch in timer jobs https://review.openstack.org/485329 | 03:49 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename tags job variable jobtags https://review.openstack.org/485845 | 03:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename uuid to build https://review.openstack.org/485851 | 03:51 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul.items to job vars https://review.openstack.org/485853 | 03:51 |
| *** gcb has joined #openstack-infra | 03:52 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add missing word to docs https://review.openstack.org/485874 | 03:53 |
| *** Daisy_ has quit IRC | 03:54 | |
| *** Daisy_ has joined #openstack-infra | 03:54 | |
| *** flepied has quit IRC | 04:00 | |
| *** Daisy_ has quit IRC | 04:02 | |
| *** Daisy_ has joined #openstack-infra | 04:03 | |
| *** gouthamr has joined #openstack-infra | 04:05 | |
| *** yamahata has quit IRC | 04:06 | |
| *** yamahata has joined #openstack-infra | 04:07 | |
| *** Daisy_ has quit IRC | 04:08 | |
| *** markvoelker has joined #openstack-infra | 04:10 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Bump fedora-minimal DIB_RELEASE 26 https://review.openstack.org/482570 | 04:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move setfiles to outside chroot with runcon https://review.openstack.org/485899 | 04:13 |
| *** adisky__ has joined #openstack-infra | 04:18 | |
| *** sree has joined #openstack-infra | 04:30 | |
| *** Daisy_ has joined #openstack-infra | 04:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/485943 | 04:34 |
| *** Daisy_ has quit IRC | 04:38 | |
| *** yamamoto has joined #openstack-infra | 04:39 | |
| *** felipemonteiro has quit IRC | 04:40 | |
| *** liujiong has quit IRC | 04:40 | |
| *** dhajare has joined #openstack-infra | 04:42 | |
| *** markvoelker has quit IRC | 04:45 | |
| *** Daisy_ has joined #openstack-infra | 04:46 | |
| *** yamamoto has quit IRC | 04:48 | |
| *** psachin has joined #openstack-infra | 04:49 | |
| *** yamamoto has joined #openstack-infra | 04:51 | |
| *** nicolasbock has joined #openstack-infra | 04:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Avoid dynamic modification of resolv.conf on openSUSE https://review.openstack.org/485809 | 04:55 |
| *** sree has quit IRC | 04:56 | |
| *** sree has joined #openstack-infra | 04:57 | |
| *** sree has quit IRC | 05:01 | |
| *** r-daneel has quit IRC | 05:01 | |
| *** r-daneel has joined #openstack-infra | 05:02 | |
| *** cuongnv has quit IRC | 05:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix gate-rally-dsvm-designate-* jobs https://review.openstack.org/485852 | 05:04 |
| *** shu-mutou-AWAY is now known as shu-mutou | 05:08 | |
| *** thorst has joined #openstack-infra | 05:13 | |
| *** yamamoto has quit IRC | 05:15 | |
| *** thorst has quit IRC | 05:18 | |
| *** armax has quit IRC | 05:22 | |
| *** rcernin has quit IRC | 05:22 | |
| *** Sukhdev has quit IRC | 05:26 | |
| *** Daisy_ has quit IRC | 05:29 | |
| *** rcernin has joined #openstack-infra | 05:36 | |
| *** flepied has joined #openstack-infra | 05:38 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Log items in loops better https://review.openstack.org/485864 | 05:39 |
| *** cshastri has joined #openstack-infra | 05:41 | |
| *** markvoelker has joined #openstack-infra | 05:42 | |
| *** coolsvap has joined #openstack-infra | 05:43 | |
| *** umbarkar has quit IRC | 05:48 | |
| *** alex_xu has quit IRC | 05:48 | |
| *** alex_xu has joined #openstack-infra | 05:49 | |
| *** Daisy_ has joined #openstack-infra | 05:50 | |
| *** esberglu has joined #openstack-infra | 05:54 | |
| *** esberglu has quit IRC | 05:54 | |
| *** esberglu has joined #openstack-infra | 05:54 | |
| *** alex_xu has quit IRC | 05:56 | |
| *** alex_xu has joined #openstack-infra | 05:56 | |
| *** liujiong has joined #openstack-infra | 05:58 | |
| *** nicolasbock has quit IRC | 05:58 | |
| *** esberglu has quit IRC | 05:59 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: prevent overwrite of gentoo init script https://review.openstack.org/484631 | 06:00 |
| *** nicolasbock has joined #openstack-infra | 06:00 | |
| *** ccamacho has quit IRC | 06:02 | |
| *** ccamacho has joined #openstack-infra | 06:02 | |
| *** jbadiapa has joined #openstack-infra | 06:03 | |
| *** larainema has quit IRC | 06:07 | |
| *** pgadiya has joined #openstack-infra | 06:11 | |
| *** wolverineav has joined #openstack-infra | 06:11 | |
| *** markvoelker has quit IRC | 06:14 | |
| dirk | hwoarang: thanks for the info.. | 06:14 |
| *** flepied has quit IRC | 06:15 | |
| *** wolverineav has quit IRC | 06:16 | |
| *** psachin has quit IRC | 06:16 | |
| *** andreas_s has joined #openstack-infra | 06:17 | |
| *** ccamacho is now known as ccamacho|brb | 06:21 | |
| *** sree has joined #openstack-infra | 06:24 | |
| *** thorst has joined #openstack-infra | 06:27 | |
| *** sree has quit IRC | 06:28 | |
| *** cuongnv has joined #openstack-infra | 06:29 | |
| *** pcrews has quit IRC | 06:30 | |
| *** thorst has quit IRC | 06:31 | |
| *** ralonsoh has joined #openstack-infra | 06:32 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: prevent overwrite of gentoo init script https://review.openstack.org/484631 | 06:33 |
| *** karthikp_ has quit IRC | 06:35 | |
| *** jhesketh has quit IRC | 06:35 | |
| *** jhesketh has joined #openstack-infra | 06:35 | |
| *** gouthamr has quit IRC | 06:36 | |
| *** links has joined #openstack-infra | 06:37 | |
| *** sree has joined #openstack-infra | 06:37 | |
| *** Rockyg has quit IRC | 06:40 | |
| *** sree has quit IRC | 06:41 | |
| *** aeng has quit IRC | 06:44 | |
| *** ralonsoh has quit IRC | 06:45 | |
| *** ralonsoh has joined #openstack-infra | 06:46 | |
| *** ccamacho|brb is now known as ccamacho | 06:48 | |
| *** flepied has joined #openstack-infra | 06:48 | |
| *** _milan_ has joined #openstack-infra | 06:49 | |
| *** yamahata has quit IRC | 06:50 | |
| *** csomerville has quit IRC | 06:50 | |
| openstackgerrit | Rajat Sharma proposed openstack-infra/shade master: Update the documentation link for doc migration https://review.openstack.org/486002 | 06:57 |
| *** ralonsoh has quit IRC | 06:57 | |
| *** rcernin has quit IRC | 06:58 | |
| *** ralonsoh has joined #openstack-infra | 06:58 | |
| *** rcernin has joined #openstack-infra | 06:58 | |
| *** dtantsur|afk is now known as dtantsur | 07:00 | |
| *** florianf has joined #openstack-infra | 07:02 | |
| *** _milan_ is now known as milan | 07:07 | |
| *** vgadiraj has quit IRC | 07:08 | |
| *** markvoelker has joined #openstack-infra | 07:11 | |
| *** psachin has joined #openstack-infra | 07:13 | |
| *** tesseract has joined #openstack-infra | 07:16 | |
| *** zhurong has quit IRC | 07:18 | |
| *** gildub has quit IRC | 07:18 | |
| *** yamamoto has joined #openstack-infra | 07:20 | |
| *** hashar has joined #openstack-infra | 07:21 | |
| *** alexchadin has joined #openstack-infra | 07:26 | |
| *** tnarg has quit IRC | 07:36 | |
| *** yamamoto has quit IRC | 07:42 | |
| *** ccamacho is now known as ccamacho|brb | 07:43 | |
| *** cuongnv has quit IRC | 07:45 | |
| *** markvoelker has quit IRC | 07:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Rename uuid to build https://review.openstack.org/486030 | 07:46 |
| *** dizquierdo has joined #openstack-infra | 07:46 | |
| openstackgerrit | Merged openstack-infra/glean master: prevent overwrite of gentoo init script https://review.openstack.org/484631 | 07:56 |
| *** zhurong has joined #openstack-infra | 07:57 | |
| *** neiljerram has joined #openstack-infra | 08:02 | |
| *** lucas-afk is now known as lucasagomes | 08:03 | |
| *** raissa has joined #openstack-infra | 08:05 | |
| *** neiljerram has quit IRC | 08:07 | |
| *** dizquierdo has quit IRC | 08:08 | |
| *** dims has quit IRC | 08:09 | |
| *** EricGonc_ has quit IRC | 08:09 | |
| *** dims has joined #openstack-infra | 08:11 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rename uuid to build https://review.openstack.org/486030 | 08:18 |
| *** egonzalez has joined #openstack-infra | 08:21 | |
| *** dims has quit IRC | 08:21 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test change https://review.openstack.org/486039 | 08:22 |
| *** dims has joined #openstack-infra | 08:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Revert "Rename uuid to build" https://review.openstack.org/486042 | 08:24 |
| *** thorst has joined #openstack-infra | 08:28 | |
| *** jascott1 has quit IRC | 08:30 | |
| *** jascott1 has joined #openstack-infra | 08:30 | |
| *** Apoorva_ has joined #openstack-infra | 08:31 | |
| *** thorst has quit IRC | 08:33 | |
| *** Apoorva has quit IRC | 08:34 | |
| *** jascott1 has quit IRC | 08:35 | |
| *** Daisy_ has quit IRC | 08:39 | |
| *** Daisy_ has joined #openstack-infra | 08:40 | |
| *** liujiong has quit IRC | 08:40 | |
| *** derekh has joined #openstack-infra | 08:42 | |
| *** markvoelker has joined #openstack-infra | 08:42 | |
| *** wolverineav has joined #openstack-infra | 08:44 | |
| *** Daisy_ has quit IRC | 08:44 | |
| *** dizquierdo has joined #openstack-infra | 08:45 | |
| *** Daisy_ has joined #openstack-infra | 08:46 | |
| *** MatthewLi has quit IRC | 08:48 | |
| *** MatthewLi has joined #openstack-infra | 08:49 | |
| *** MatthewLi has quit IRC | 08:49 | |
| *** Daisy_ has quit IRC | 08:50 | |
| *** ccamacho|brb is now known as ccamacho | 08:51 | |
| *** gaurangt has quit IRC | 08:51 | |
| *** MatthewLi has joined #openstack-infra | 08:51 | |
| *** Kevin_Zheng has quit IRC | 08:51 | |
| *** Kevin_Zheng has joined #openstack-infra | 08:52 | |
| *** Daisy_ has joined #openstack-infra | 08:52 | |
| *** gaurangt has joined #openstack-infra | 08:53 | |
| *** shardy has joined #openstack-infra | 08:54 | |
| *** esberglu has joined #openstack-infra | 08:54 | |
| *** mwarad has quit IRC | 08:57 | |
| *** wolverineav has quit IRC | 08:57 | |
| *** Daisy_ has quit IRC | 08:57 | |
| *** Daisy_ has joined #openstack-infra | 08:58 | |
| *** electrofelix has joined #openstack-infra | 08:59 | |
| *** esberglu has quit IRC | 08:59 | |
| *** Daisy_ has quit IRC | 08:59 | |
| *** Daisy_ has joined #openstack-infra | 08:59 | |
| *** ramishra has quit IRC | 09:00 | |
| *** ramishra has joined #openstack-infra | 09:02 | |
| *** neiljerram has joined #openstack-infra | 09:05 | |
| *** markvoelker has quit IRC | 09:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add a py3 designate job https://review.openstack.org/484275 | 09:18 |
| *** liujiong has joined #openstack-infra | 09:19 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: charms: Add deployment guide https://review.openstack.org/483429 | 09:23 |
| openstackgerrit | Merged openstack-infra/project-config master: zuul: layout: Prevent ansible-hardening openSUSE job to run on pre-Pike https://review.openstack.org/485537 | 09:26 |
| openstackgerrit | Merged openstack-infra/project-config master: zuul: layout: ansible-hardening: Promote the openSUSE job to gate https://review.openstack.org/485538 | 09:26 |
| openstackgerrit | Merged openstack-infra/project-config master: zuul: layout: Add non-voting openSUSE OpenStack Ansible jobs https://review.openstack.org/485539 | 09:26 |
| *** Daisy_ has quit IRC | 09:26 | |
| *** Daisy_ has joined #openstack-infra | 09:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Avoid dynamic modification of resolv.conf on openSUSE https://review.openstack.org/485809 | 09:30 |
| *** iyamahat has joined #openstack-infra | 09:30 | |
| *** Daisy_ has quit IRC | 09:31 | |
| *** esberglu has joined #openstack-infra | 09:32 | |
| *** esberglu has quit IRC | 09:32 | |
| *** esberglu has joined #openstack-infra | 09:32 | |
| *** sambetts|afk is now known as sambetts | 09:33 | |
| *** MatthewLi has quit IRC | 09:34 | |
| *** esberglu has quit IRC | 09:37 | |
| *** iyamahat has quit IRC | 09:38 | |
| hwoarang | dirk: do you mind if I rework https://review.openstack.org/#/c/484499/ ? | 09:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/486080 | 09:47 |
| *** katkapilatova has joined #openstack-infra | 09:48 | |
| *** katkapilatova has left #openstack-infra | 09:48 | |
| *** gildub has joined #openstack-infra | 09:54 | |
| *** larainema has joined #openstack-infra | 10:01 | |
| *** stevebaker has quit IRC | 10:03 | |
| *** slagle has quit IRC | 10:04 | |
| *** sdague has joined #openstack-infra | 10:08 | |
| *** alexchadin has quit IRC | 10:11 | |
| *** markvoelker has joined #openstack-infra | 10:12 | |
| *** jaypipes has joined #openstack-infra | 10:13 | |
| *** liujiong has quit IRC | 10:17 | |
| *** HeOS has joined #openstack-infra | 10:22 | |
| *** ramishra has quit IRC | 10:24 | |
| *** ramishra has joined #openstack-infra | 10:26 | |
| *** jkilpatr has joined #openstack-infra | 10:27 | |
| *** Apoorva_ has quit IRC | 10:28 | |
| *** thorst has joined #openstack-infra | 10:29 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: Adds oswin-tempest-plugin repository https://review.openstack.org/486095 | 10:32 |
| *** jkilpatr has quit IRC | 10:33 | |
| *** thorst has quit IRC | 10:33 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: nodepool: elements: infra-package-needs: Install tar package https://review.openstack.org/486097 | 10:34 |
| smcginnis | Did our automatic linking with "Closes-bug" tags and Launchpad break? | 10:35 |
| smcginnis | Doesn't seem to be working anymore from what I've seen. | 10:36 |
| AJaeger | smcginnis: which bug report and which review? | 10:44 |
| AJaeger | smcginnis: which review should be enough... | 10:44 |
| kklimonda | hi, could someone point me in the right direction to learn about your release pipeline? i.e. how archives are made and uploaded | 10:44 |
| *** markvoelker has quit IRC | 10:45 | |
| openstackgerrit | Renat Akhmerov proposed openstack-infra/project-config master: Add a simple version of Mistral HA gate https://review.openstack.org/484691 | 10:46 |
| *** yamamoto_ has joined #openstack-infra | 10:48 | |
| smcginnis | AJaeger: This is the latest, but I've had a few: https://review.openstack.org/#/c/486096/ | 10:48 |
| AJaeger | smcginnis: that indeed looks fine. We only report with the "first" patchset and that one has already the Closes-Bug. so, needs an infra-root to check log files... i don't see anything obvious wrong here | 10:49 |
| AJaeger | "fine" -> I mean: your change is fine, it should have tagged the bug, so this is broken | 10:50 |
| *** zhurong has quit IRC | 10:50 | |
| smcginnis | AJaeger: OK, thanks for the confirmation that I'm not just doing something stupid. | 10:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/486080 | 10:57 |
| *** rook has quit IRC | 10:57 | |
| *** tushar has joined #openstack-infra | 10:57 | |
| *** dtantsur is now known as dtantsur|bbl | 10:57 | |
| *** wfoster has quit IRC | 10:57 | |
| *** rook has joined #openstack-infra | 10:58 | |
| *** rook is now known as Guest11562 | 10:59 | |
| *** wfoster has joined #openstack-infra | 10:59 | |
| *** murphy has joined #openstack-infra | 10:59 | |
| *** flepied has quit IRC | 11:02 | |
| *** flepied has joined #openstack-infra | 11:03 | |
| *** jkilpatr has joined #openstack-infra | 11:09 | |
| *** shardy is now known as shardy_lunch | 11:10 | |
| *** hashar is now known as hasharAway | 11:11 | |
| *** gcb has quit IRC | 11:14 | |
| *** rtjure has quit IRC | 11:16 | |
| *** ccamacho is now known as ccamacho|lunch | 11:17 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: Adds oswin-tempest-plugin repository https://review.openstack.org/486095 | 11:21 |
| *** pgadiya has quit IRC | 11:24 | |
| *** murphy has quit IRC | 11:25 | |
| *** EricGonczer_ has joined #openstack-infra | 11:26 | |
| *** coolsvap has quit IRC | 11:27 | |
| *** yamamoto_ has quit IRC | 11:32 | |
| *** murphy has joined #openstack-infra | 11:37 | |
| *** shardy_lunch is now known as shardy | 11:38 | |
| *** liujiong has joined #openstack-infra | 11:40 | |
| *** hasharAway is now known as hashar | 11:43 | |
| *** markvoelker has joined #openstack-infra | 11:43 | |
| *** lucasagomes is now known as lucas-hungry | 11:47 | |
| *** rossella__ has quit IRC | 11:50 | |
| *** yamamoto has joined #openstack-infra | 11:52 | |
| *** rossella__ has joined #openstack-infra | 11:53 | |
| *** murphy has quit IRC | 11:56 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** rossella__ has quit IRC | 11:59 | |
| *** ldnunes has joined #openstack-infra | 11:59 | |
| *** thorst has joined #openstack-infra | 11:59 | |
| *** thorst has quit IRC | 11:59 | |
| *** arturb has quit IRC | 12:00 | |
| *** rossella__ has joined #openstack-infra | 12:02 | |
| *** thorst has joined #openstack-infra | 12:04 | |
| *** panda|off is now known as panda | 12:15 | |
| *** camunoz has joined #openstack-infra | 12:15 | |
| *** camunoz is now known as camunoz_mtg | 12:15 | |
| *** camunoz_mtg is now known as camunoz | 12:15 | |
| *** slagle has joined #openstack-infra | 12:15 | |
| *** markvoelker has quit IRC | 12:16 | |
| *** baoli has joined #openstack-infra | 12:20 | |
| *** dprince has joined #openstack-infra | 12:24 | |
| *** markvoelker has joined #openstack-infra | 12:26 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config master: Fix typo in designate job definition https://review.openstack.org/486120 | 12:27 |
| therve | AJaeger, yolanda : could you have a quick look at ^^^ | 12:27 |
| therve | I made a stupid typo and broke the designate gate | 12:27 |
| *** esberglu has joined #openstack-infra | 12:33 | |
| *** gordc has joined #openstack-infra | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** esberglu has quit IRC | 12:37 | |
| *** dizquierdo has quit IRC | 12:38 | |
| mnaser | is this a CI issue? if i open http://status.openstack.org/zuul/ and type 483017 in the filter (which is a change was supposed to start getting checked yesterday) -- there are jobs with over 17 hours queued time | 12:39 |
| mnaser | that isdn't right.. is it? | 12:39 |
| *** trown|outtypewww is now known as trown | 12:40 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 12:40 |
| *** dave-mccowan has joined #openstack-infra | 12:41 | |
| *** EricGonczer_ has quit IRC | 12:41 | |
| *** yamamoto has joined #openstack-infra | 12:42 | |
| *** andreas_s has quit IRC | 12:43 | |
| *** lucas-hungry is now known as lucasagomes | 12:45 | |
| *** dave-mcc_ has joined #openstack-infra | 12:45 | |
| *** dave-mccowan has quit IRC | 12:48 | |
| *** xyang1 has joined #openstack-infra | 12:48 | |
| *** jrist has quit IRC | 12:49 | |
| *** ccamacho|lunch is now known as ccamacho | 12:51 | |
| *** mattmceuen has quit IRC | 12:51 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 12:51 |
| *** rhallisey has joined #openstack-infra | 12:51 | |
| *** yamamoto has quit IRC | 12:52 | |
| *** florianf has quit IRC | 12:52 | |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| *** kgiusti has joined #openstack-infra | 12:53 | |
| *** eharney has joined #openstack-infra | 12:54 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Move setfiles to outside chroot with runcon https://review.openstack.org/485899 | 12:55 |
| *** makowals has joined #openstack-infra | 12:56 | |
| *** esberglu has joined #openstack-infra | 12:56 | |
| jlvillal | Ironic review request: https://review.openstack.org/480332 This is to support testing our new boot-from-volume feature. | 12:58 |
| jlvillal | We need an experimental gate job to test it. Thanks! | 12:58 |
| *** florianf has joined #openstack-infra | 12:59 | |
| *** MatthewLi has joined #openstack-infra | 13:01 | |
| *** LindaWang has quit IRC | 13:02 | |
| *** jrist has joined #openstack-infra | 13:03 | |
| *** ramishra has quit IRC | 13:03 | |
| *** jcoufal has joined #openstack-infra | 13:04 | |
| *** ramishra has joined #openstack-infra | 13:05 | |
| *** mriedem has joined #openstack-infra | 13:06 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:06 |
| *** adisky__ has quit IRC | 13:08 | |
| fungi | AJaeger: smcginnis: judging from the error logs, looks like the gerrit server's login to launchpad has expired. i'll have to dig around and figure out how to refresh it in a bit | 13:12 |
| pabelanger | mnaser: tripleo jobs are having large gate times this week: http://status.openstack.org/elastic-recheck/data/others.html | 13:12 |
| pabelanger | mnaser: I haven't looked today, but previous was yum failures for package downloads. We've recommended a few changes job side, and stopped citycloud-sto2 also | 13:13 |
| mnaser | i guess it's just a long queue trying to keep up.. i'll keep an eye out, thanks pabelanger | 13:14 |
| asettle | fungi and AJaeger - so, while fiddling around with the nova docs, I realised that I'm waiting on their full build just for some doc changes. Other projects have been able to change their gates so when changes don't touch the code, it just builds the doc gate | 13:15 |
| pabelanger | mnaser: ya, more triple jobs are running too. Some of them take up to 3hr, so if there is a gate reset, all following patches need to be retested | 13:15 |
| asettle | How easy would that be to roll out now that we have all the docs in the project repos? | 13:16 |
| asettle | That would be quite beneficial | 13:16 |
| asettle | stephenfin: ^ | 13:16 |
| *** askb has quit IRC | 13:16 | |
| mnaser | asettle i believe that is possible if you do it in project-config/zuul/layout.yaml | 13:17 |
| asettle | mnaser: oh neat :) thanks. So it actually shouldn't be that difficult? | 13:17 |
| fungi | asettle: we discourage it for a number of reasons. it's a fiddly sieve of regular expressions which makes it very easy to miss running some jobs on changes that can wedge projects, and also people rarely think through all the things they could change in their project which might break other jobs | 13:17 |
| pabelanger | mnaser: your patch could merge in about 24mins, assuming all jobs before it pass | 13:17 |
| mnaser | but also what fungi said :p | 13:17 |
| mnaser | i know how it can be done, but cant speak about if its good or not | 13:17 |
| asettle | Ahhh I see, so... not recommended, basically, but do-able. | 13:18 |
| mnaser | but i've seen it done for kolla - https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1880-L1887 | 13:18 |
| asettle | mnaser: yeah they have it implemented for kolla, osa, and I think even glance and maybe keystone? | 13:18 |
| mnaser | pabelanger cool! i spend way too much time watching status.openstack.org/zuul/ :p | 13:18 |
| asettle | I just think I might actually pass away in my sleep waiting for hte nova gate :p | 13:18 |
| mnaser | asettle you can checkout the output of the doc jobs in advance though | 13:19 |
| fungi | asettle: however, we try to make sure jobs (especially things like docs jobs) can run locally so you don't have to wait for our ci system when iterating on something | 13:19 |
| pabelanger | mnaser: ya, the large number if pep8 failures in gate means something changed recently in a pep8 dependency. Or broken code was show how merged | 13:19 |
| asettle | fungi: yeah that's true. I did do that... I was just wanting to see the pretty pretty build | 13:20 |
| mnaser | http://status.openstack.org/zuul/ -- paste your review id in the filter, then click on the job name (assuming it's completed) and you can get all the output you need | 13:20 |
| mnaser | thats how i troubleshoot failures if i see them before needing to wait for all jobs to complete | 13:20 |
| * asettle gasps dramatically | 13:20 | |
| asettle | Okay I've definitely used zuul before BUT I DIDNT KNOW THAT | 13:20 |
| asettle | You know those days when you combination feel like an idiot but also learn something cool? | 13:21 |
| mnaser | also i'm not sure if this is a good thing.. or a bad thing... but if you throw a review after you figure out a problem, the existing job terminates and a new one starts.. so you dont have to wait for the jobs to complete to submit another review | 13:21 |
| mnaser | but i dont know if that has any bad impact on the CI infra :X | 13:21 |
| mnaser | i try to avoid churn because donated resources :p | 13:21 |
| fungi | asettle: i usually point my browser at file:///home/fungi/src/openstack/nova/docs/build/html/index.html or whatever after running | 13:22 |
| asettle | fungi: always makes sense. Just not shareable. | 13:22 |
| fungi | asettle: right, i mean when i'm figuring out how to get the content i want, before it's ready for review. once it's ready to push for review, sure it takes a bit before there is feedback from the ci system (especially during the feature freeze rush, less because of job runtime and more because of jobs waiting for available nodes) | 13:23 |
| fungi | i expect things to get better after next week | 13:24 |
| *** annegentle has joined #openstack-infra | 13:25 | |
| *** alexchadin has joined #openstack-infra | 13:25 | |
| asettle | fungi: okay :) thanks. I will scratch out the idea of changing everyone's gate jobs :p | 13:26 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config master: Set OSIC cloud1 Region1 to use 0 servers https://review.openstack.org/485383 | 13:26 |
| *** yamamoto_ has joined #openstack-infra | 13:28 | |
| *** yamamoto_ has quit IRC | 13:28 | |
| *** yamamoto has quit IRC | 13:28 | |
| fungi | AJaeger: smcginnis: hrm, i may be wrong about the gerrit server's launchpad login. the errors i was looking at were a red herring... here's the issue for that change: | 13:29 |
| fungi | [2017-07-21 10:33:07,304] INFO com.google.gerrit.common.ChangeHookRunner : hook[patchset-created] output: assignee_link: No such object "https://api.launchpad.net/devel/~stmcginnis". | 13:29 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config master: Convert the OSIC to only use disk nodes https://review.openstack.org/485378 | 13:30 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:30 |
| *** florianf has quit IRC | 13:32 | |
| *** dtantsur|bbl is now known as dtantsur | 13:32 | |
| fungi | smcginnis: is that your correct lp id? if so, it's not coming up for me either. i wonder whether the lp api returned the wrong lp account for your openid | 13:32 |
| *** florianf has joined #openstack-infra | 13:33 | |
| *** annegentle has quit IRC | 13:37 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: Adds oswin-tempest-plugin repository https://review.openstack.org/486095 | 13:37 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Move py3 jobs for ARA to voting gate https://review.openstack.org/486145 | 13:41 |
| dmsimard | rcarrillocruz: ^^ next on my plate is Ansible 2.4 | 13:41 |
| *** jcoufal_ has joined #openstack-infra | 13:41 | |
| *** links has quit IRC | 13:43 | |
| *** jcoufal has quit IRC | 13:43 | |
| *** alexchadin has quit IRC | 13:44 | |
| *** Qiming has quit IRC | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| rcarrillocruz | Neat! | 13:47 |
| *** nmathew has joined #openstack-infra | 13:48 | |
| *** markmcd has quit IRC | 13:49 | |
| *** baoli has quit IRC | 13:49 | |
| *** Qiming has joined #openstack-infra | 13:50 | |
| *** gouthamr has joined #openstack-infra | 13:51 | |
| *** rossella__ has quit IRC | 13:52 | |
| *** rossella__ has joined #openstack-infra | 13:54 | |
| *** dansmith is now known as superdan | 13:57 | |
| *** markmcd has joined #openstack-infra | 13:59 | |
| *** eharney has quit IRC | 14:02 | |
| *** Goneri has joined #openstack-infra | 14:02 | |
| *** baoli has joined #openstack-infra | 14:02 | |
| *** felipemonteiro has joined #openstack-infra | 14:03 | |
| *** _milan_ has joined #openstack-infra | 14:06 | |
| *** psachin has quit IRC | 14:06 | |
| *** liujiong has quit IRC | 14:07 | |
| *** milan has quit IRC | 14:07 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:07 | |
| *** jaypipes is now known as leakypipes | 14:09 | |
| *** felipemonteiro has quit IRC | 14:11 | |
| *** sm_ has quit IRC | 14:11 | |
| *** EricGonczer_ has joined #openstack-infra | 14:11 | |
| *** jcoufal has joined #openstack-infra | 14:12 | |
| *** baoli has quit IRC | 14:12 | |
| *** baoli has joined #openstack-infra | 14:13 | |
| *** jcoufal_ has quit IRC | 14:13 | |
| *** Daisy_ has joined #openstack-infra | 14:14 | |
| pabelanger | AJaeger: fungi: clarkb: I am ready to support landing: 485383 Brings down osic-cloud1 so we can start using osic-cloud1-disk | 14:14 |
| *** armax has joined #openstack-infra | 14:16 | |
| *** EricGonc_ has joined #openstack-infra | 14:16 | |
| *** eharney has joined #openstack-infra | 14:16 | |
| *** baoli has quit IRC | 14:16 | |
| *** baoli has joined #openstack-infra | 14:17 | |
| stevemar | mtreinish: loved http://blog.kortar.org/?p=380 -- great read | 14:17 |
| *** EricGonczer_ has quit IRC | 14:18 | |
| *** EricGonc_ has quit IRC | 14:19 | |
| *** Guest11562 is now known as rook | 14:19 | |
| fungi | pabelanger: thanks, i've chucked that one in | 14:20 |
| cloudnull | pabelanger: if you need anything from me while this is happening let me know . | 14:21 |
| pabelanger | cloudnull: ack | 14:21 |
| *** EricGonczer_ has joined #openstack-infra | 14:21 | |
| *** mattmceuen has joined #openstack-infra | 14:22 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart https://review.openstack.org/483077 | 14:22 |
| *** Daisy_ has quit IRC | 14:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set OSIC cloud1 Region1 to use 0 servers https://review.openstack.org/485383 | 14:26 |
| *** jrist has quit IRC | 14:26 | |
| *** r-daneel has quit IRC | 14:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Fix stop / restart for zuul-web init script https://review.openstack.org/486159 | 14:29 |
| pabelanger | mordred: jeblair: ^zuul-web init script fixes | 14:30 |
| *** EricGonc_ has joined #openstack-infra | 14:30 | |
| *** EricGonczer_ has quit IRC | 14:32 | |
| *** baoli has quit IRC | 14:33 | |
| *** ihrachys has joined #openstack-infra | 14:33 | |
| *** baoli has joined #openstack-infra | 14:36 | |
| *** jrist has joined #openstack-infra | 14:39 | |
| *** rossella__ has quit IRC | 14:41 | |
| *** marst has quit IRC | 14:42 | |
| *** dhajare has quit IRC | 14:43 | |
| mugsie | is anyone able to look at https://review.openstack.org/486120 ? Designates devstack jobs are all failing without it | 14:43 |
| pabelanger | mugsie: +3 | 14:43 |
| *** cshastri has quit IRC | 14:44 | |
| mugsie | pabelanger: thank you very much | 14:44 |
| *** rossella__ has joined #openstack-infra | 14:44 | |
| pabelanger | np | 14:44 |
| *** baoli has quit IRC | 14:46 | |
| *** rlandy is now known as rlandy|brb | 14:46 | |
| *** eharney has quit IRC | 14:48 | |
| *** rodrigopaiva has joined #openstack-infra | 14:49 | |
| *** marst_ has joined #openstack-infra | 14:50 | |
| mtreinish | stevemar: thanks, glad you enjoyed it | 14:50 |
| mtreinish | stevemar: it wasn't too long? :) | 14:50 |
| *** eharney has joined #openstack-infra | 14:50 | |
| *** dizquierdo has joined #openstack-infra | 14:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo in designate job definition https://review.openstack.org/486120 | 14:54 |
| dmsimard | pabelanger: hey, do you know what's the plan for py3 on the centos-7 image ? | 14:56 |
| pabelanger | dmsimard: no yet. We've talked about using software colllect in the past | 14:58 |
| pabelanger | collections* | 14:58 |
| pabelanger | for now, fedora would be a good option for python3 testing | 14:58 |
| pabelanger | I know ianw plans on doing that for devstack | 14:58 |
| pabelanger | fungi: cloudnull: osic-cloud1 now at max-server: 0 | 14:59 |
| dmsimard | pabelanger: yeah I actually just realized I screwed up by almost adding a py3 job on the centos node, I'll switch to fedora for py3 coverage on an rpm distro | 14:59 |
| *** chandankumar is now known as daccontant | 15:00 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Move py3 jobs for ARA to voting gate https://review.openstack.org/486145 | 15:00 |
| *** shardy is now known as shardy_mtg | 15:00 | |
| pabelanger | dmsimard: yup, I've been using fedora for testing of ansible roles for a while now. Works well | 15:00 |
| pabelanger | dmsimard: trying to get fedora-26 online too | 15:00 |
| dmsimard | pabelanger: ah that was my next question, so I'll use f25 in the meantime | 15:00 |
| pabelanger | dmsimard: we need https://review.openstack.org/#/c/482570/ first | 15:01 |
| pabelanger | hopefully next week | 15:01 |
| *** nmathew has quit IRC | 15:04 | |
| *** _milan_ has quit IRC | 15:04 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Move py3 jobs for ARA to voting gate https://review.openstack.org/486145 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove validate-host from unittests/pre.yaml https://review.openstack.org/485836 | 15:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove gather_facts: true https://review.openstack.org/485834 | 15:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Move subunit processing https://review.openstack.org/485840 | 15:06 |
| *** MatthewLi has quit IRC | 15:06 | |
| *** dprince has quit IRC | 15:07 | |
| *** ekhugen- has quit IRC | 15:10 | |
| *** ekhugen has joined #openstack-infra | 15:10 | |
| *** patriciadomin has quit IRC | 15:11 | |
| *** rcernin has quit IRC | 15:14 | |
| *** rtjure has joined #openstack-infra | 15:16 | |
| *** jrist has quit IRC | 15:18 | |
| *** rlandy|brb is now known as rlandy | 15:19 | |
| *** r-daneel has joined #openstack-infra | 15:20 | |
| *** jesusaurum has joined #openstack-infra | 15:22 | |
| *** aarefiev is now known as aarefiev_afk | 15:23 | |
| *** jesusaur has quit IRC | 15:26 | |
| *** jcoufal_ has joined #openstack-infra | 15:27 | |
| *** jcoufal has quit IRC | 15:28 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Move py3 jobs for ARA to voting gate https://review.openstack.org/486145 | 15:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Stop testing ARA against Ansible 2.1 https://review.openstack.org/486163 | 15:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add fedora to ARA integration gate for python 3 on RPM distro https://review.openstack.org/486164 | 15:32 |
| *** shardy_mtg is now known as shardy | 15:33 | |
| *** mamitchl has joined #openstack-infra | 15:34 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add fedora to ARA integration gate for python 3 on RPM distro https://review.openstack.org/486164 | 15:34 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart https://review.openstack.org/483077 | 15:36 |
| *** yamamoto has quit IRC | 15:44 | |
| *** yamamoto has joined #openstack-infra | 15:44 | |
| *** baoli has joined #openstack-infra | 15:47 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/git-review master: Fix commit-msg hook installation for worktrees https://review.openstack.org/397206 | 15:48 |
| *** annegentle has joined #openstack-infra | 15:48 | |
| *** yamamoto has quit IRC | 15:49 | |
| *** flepied has quit IRC | 15:51 | |
| *** makowals has quit IRC | 15:51 | |
| *** Hal has quit IRC | 15:53 | |
| openstackgerrit | Merged openstack-infra/git-review master: Switch to string format to avoid interpolation issues https://review.openstack.org/389701 | 15:54 |
| *** pcrews has joined #openstack-infra | 15:55 | |
| *** florianf has quit IRC | 15:56 | |
| *** camunoz has quit IRC | 15:57 | |
| *** jrist has joined #openstack-infra | 15:57 | |
| *** jrist has quit IRC | 15:57 | |
| *** jrist has joined #openstack-infra | 15:57 | |
| *** camunoz has joined #openstack-infra | 15:57 | |
| *** mriedem is now known as mriedem_afk | 16:02 | |
| *** HeOS has quit IRC | 16:02 | |
| openstackgerrit | Merged openstack-infra/git-review master: Added topic field to the list output https://review.openstack.org/244083 | 16:04 |
| *** csomerville has joined #openstack-infra | 16:05 | |
| openstackgerrit | Merged openstack-infra/git-review master: Refactor displaying of reviews https://review.openstack.org/398353 | 16:05 |
| *** Rockyg has joined #openstack-infra | 16:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: upload-logs: zuul.newrev can be none now https://review.openstack.org/486172 | 16:08 |
| *** lucasagomes is now known as lucas-afk | 16:09 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Fix stop / restart for zuul-web init script https://review.openstack.org/486159 | 16:10 |
| *** florianf has joined #openstack-infra | 16:11 | |
| *** Rockyg has quit IRC | 16:14 | |
| *** hashar is now known as hasharDinner | 16:15 | |
| *** egonzalez has quit IRC | 16:16 | |
| *** bhavik1 has joined #openstack-infra | 16:16 | |
| *** dizquierdo has quit IRC | 16:19 | |
| *** daccontant is now known as chandankumar | 16:20 | |
| *** rtjure has quit IRC | 16:23 | |
| *** electrofelix has quit IRC | 16:26 | |
| *** jascott1 has joined #openstack-infra | 16:27 | |
| *** yamahata has joined #openstack-infra | 16:27 | |
| *** flepied has joined #openstack-infra | 16:27 | |
| *** vhosakot has joined #openstack-infra | 16:29 | |
| *** bhavik1 has quit IRC | 16:35 | |
| *** dmsimard is now known as dmsimard|afk | 16:37 | |
| *** sdague has quit IRC | 16:37 | |
| *** neiljerram has quit IRC | 16:39 | |
| *** greghaynes has quit IRC | 16:40 | |
| *** pblaho has quit IRC | 16:42 | |
| *** jascott1 has quit IRC | 16:42 | |
| *** r-daneel has quit IRC | 16:47 | |
| *** dave-mccowan has joined #openstack-infra | 16:48 | |
| *** rodrigopaiva_ has joined #openstack-infra | 16:48 | |
| *** dprince has joined #openstack-infra | 16:48 | |
| *** rodrigopaiva has quit IRC | 16:48 | |
| *** dave-mcc_ has quit IRC | 16:50 | |
| *** yamamoto has joined #openstack-infra | 16:50 | |
| *** derekh has quit IRC | 16:54 | |
| *** flepied has quit IRC | 16:54 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review master: Limit reviews by branch when specified https://review.openstack.org/312056 | 16:55 |
| *** trown is now known as trown|lunch | 16:55 | |
| *** ramishra has quit IRC | 16:57 | |
| morgan | i'm seeing a significant uptick in 503 errors on gerrit (and very long "working" popups) | 16:57 |
| morgan | not sure if i'm hitting something wonky on my end as well, having some packet loss | 16:58 |
| morgan | but figured i'd report it here in case it was gerrit involved. | 16:58 |
| *** r-daneel has joined #openstack-infra | 16:58 | |
| morgan | s/503/502 | 16:59 |
| raissa | yeah, and it's somewhat slow | 17:00 |
| *** calebb has quit IRC | 17:00 | |
| pabelanger | Ya, we seem to have a spike in GC | 17:01 |
| pabelanger | give me a sec and I'll restart gerrit | 17:02 |
| *** sambetts is now known as sambetts|afk | 17:02 | |
| pabelanger | #status notice Restarting Gerrit for our weekly memory leak cleanup. | 17:04 |
| openstackstatus | pabelanger: sending notice | 17:04 |
| *** calebb has joined #openstack-infra | 17:04 | |
| *** ralonsoh has quit IRC | 17:04 | |
| morgan | pabelanger: cool | 17:05 |
| morgan | pabelanger: thanks! | 17:05 |
| -openstackstatus- NOTICE: Restarting Gerrit for our weekly memory leak cleanup. | 17:05 | |
| *** yamamoto has quit IRC | 17:05 | |
| pabelanger | should be back online now | 17:06 |
| morgan | tyvm | 17:06 |
| morgan | confirmed it is back. | 17:06 |
| AJaeger | thanks, pabelanger | 17:06 |
| openstackstatus | pabelanger: finished sending notice | 17:07 |
| fungi | if we ever get upgraded to 2.13 that's _supposed_ to stop being a problem ;) | 17:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Undefine Zuul variables in Ansible when appropriate https://review.openstack.org/486183 | 17:09 |
| *** dtantsur is now known as dtantsur|afk | 17:11 | |
| *** iyamahat has joined #openstack-infra | 17:12 | |
| AJaeger | any project-config core to review a cosmetic one liner, please? https://review.openstack.org/485110 | 17:13 |
| *** jerryz has joined #openstack-infra | 17:13 | |
| *** greghaynes has joined #openstack-infra | 17:16 | |
| pabelanger | fungi: cloudnull: osic-cloud1 offline now. I we should be able to move forward on osic-cloud1-disk | 17:17 |
| *** zz_ja has quit IRC | 17:17 | |
| *** thorst has quit IRC | 17:17 | |
| pabelanger | https://review.openstack.org/#/c/485378/ | 17:17 |
| *** thorst has joined #openstack-infra | 17:20 | |
| *** zz_ja has joined #openstack-infra | 17:20 | |
| *** Swami has joined #openstack-infra | 17:21 | |
| *** dave-mccowan has quit IRC | 17:23 | |
| *** thorst has quit IRC | 17:24 | |
| *** dave-mccowan has joined #openstack-infra | 17:26 | |
| clarkb | wmf apparently disables gerrit git gc because "it is broken" another item we should look at with 2.13 | 17:33 |
| *** mriedem_afk is now known as mriedem | 17:33 | |
| *** flepied has joined #openstack-infra | 17:36 | |
| *** jkilpatr has quit IRC | 17:36 | |
| *** shardy has quit IRC | 17:39 | |
| *** jkilpatr has joined #openstack-infra | 17:40 | |
| fungi | eep | 17:41 |
| *** bobh has joined #openstack-infra | 17:41 | |
| fungi | clarkb: also, did you see our recent comment tests at https://review-dev.openstack.org/107958 ? | 17:41 |
| fungi | i think something's screwy with multibyte codepoints | 17:42 |
| clarkb | I saw testing didnt realize something baf was happening. note that gerrits db is only 3 byte utf8 not 4 due to age of mysql server | 17:43 |
| fungi | this arose because waynr got a 500 error on review.o.o when trying to post a comment with a "see-no-evil monkey emoji" (whatever that is) | 17:44 |
| fungi | so we were trying to see if it was repeatable on review-dev/2.13 | 17:44 |
| *** mamitchl has quit IRC | 17:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: Convert the OSIC to only use disk nodes https://review.openstack.org/485378 | 17:44 |
| fungi | clarkb: some of my tests were katakana which should not have been 4-byte, but still got written to the db as "?" | 17:45 |
| *** thorst has joined #openstack-infra | 17:45 | |
| fungi | oh, hrm, maybe they are after all | 17:46 |
| *** Sukhdev has joined #openstack-infra | 17:48 | |
| fungi | hrm, nope. katakana 'ka' is unicode 0x30ab and utf-8 0xe382ab | 17:48 |
| fungi | so not just a 4-byte problem afaict | 17:49 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Undefine Zuul variables in Ansible when appropriate https://review.openstack.org/486183 | 17:50 |
| melwitt | hi, just wanted to let someone know that logstash.openstack.org is showing a blank dashboard, not working | 17:50 |
| mriedem | logstash.o.o down? | 17:51 |
| *** Daisy_ has joined #openstack-infra | 17:52 | |
| *** trown|lunch is now known as trown | 17:52 | |
| fungi | it does indeed look like at least kibana isn't loading completely in my browser there | 17:53 |
| mtreinish | fungi: heh, I think you win the award for leaving the first real comment on my blog | 17:56 |
| clarkb | does a hard refresh work? | 17:56 |
| mtreinish | fungi, melwitt: I'm getting an error saying it can't connect to elasticsearch | 17:57 |
| fungi | clarkb: complete with obligatory grammatical typo! | 17:57 |
| fungi | erm mtreinish: ^ | 17:57 |
| fungi | sorry clarkb | 17:57 |
| fungi | clarkb: force refreshes aren't helping me | 17:57 |
| fungi | i'm checking apis now | 17:57 |
| mtreinish | fungi: I as looking hard for that typo in clarkb's question | 17:58 |
| fungi | mtreinish: yeah, i'm also having trouble keeping multiple streams of irc conversation from crossing today. then again, it's friday | 17:59 |
| *** Daisy_ has quit IRC | 17:59 | |
| fungi | looks like elasticsearch02 fell offline (the whole server) around 21:20 utc yesterday | 18:00 |
| fungi | and since it's the one kibana is configured to query, kibana is hanging | 18:01 |
| fungi | it's responding to ping but not ssh, consistent on ipv4 and ipv6 addresses | 18:02 |
| fungi | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=123 | 18:05 |
| *** nicolasbock has quit IRC | 18:06 | |
| jeblair | fungi: should losing an es node kill logstash like that? | 18:08 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=424&rra_id=all | 18:08 |
| jeblair | shows an anomoly with a different timing | 18:08 |
| *** nicolasbock has joined #openstack-infra | 18:08 | |
| jeblair | fungi: oh, er, that's the same timing isn't it? :) | 18:09 |
| fungi | jeblair: kibana has to be configured to query a single elasticsearch node in the cluster, as i understand it, and it just happens to be configured to query this one | 18:09 |
| jeblair | fungi: yes, i see the smoke now. | 18:09 |
| fungi | out of band console for that nova instance had a login prompt, but hitting enter there gave me a spew of kmesgs about "task xenwatch blocked for more than 120 seconds" over and over | 18:10 |
| fungi | so i have a feeling this is a hypervisor-level issue and will need a (possibly hard but we'll see how it goes) reboot to recover | 18:11 |
| jeblair | fungi: ++ | 18:11 |
| fungi | rebooting elasticsearch02 via nova api now | 18:12 |
| fungi | well, trying the send ctrl-alt-del option from the console emulator first | 18:12 |
| fungi | not that i have high hopes | 18:12 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Adding more openSUSE 42.3 variants to existing 42.2 jobs https://review.openstack.org/484499 | 18:13 |
| *** baoli has quit IRC | 18:13 | |
| fungi | yeah, console is actually unresponsive to further carriage returns as well. seems hung pretty hard | 18:13 |
| pabelanger | fungi: clarkb: osic-cloud1-disk live, uploading DIBs now | 18:15 |
| fungi | thanks pabelanger! | 18:15 |
| fungi | nova api claims es02 was in a "reboot" state for a while, then went back to "active" but i saw no evidence on its console window that it actually rebooted and a couple minutes later i'm still timing out attempting to ssh to it. time for reboot --hard i guess | 18:17 |
| fungi | that definitely cleared the console | 18:17 |
| fungi | console now shows the (disappointingly silent) ubuntu boot screen | 18:19 |
| fungi | and now i have a login prompt | 18:19 |
| fungi | and can ssh into it again | 18:19 |
| *** jascott1 has joined #openstack-infra | 18:19 | |
| fungi | nothing (yet) listening on localhost:9200 though :( | 18:21 |
| fungi | oh, maybe we don't have the service set to start at boot | 18:21 |
| jeblair | nothing in the es error log | 18:21 |
| jeblair | fungi: :( | 18:21 |
| fungi | right | 18:21 |
| fungi | yeah, we have an initscript but no symlink from /etc/rc2.d | 18:22 |
| jeblair | is there a reason for that? | 18:22 |
| fungi | not sure, but i'm attempting to start the elasticsearch service on es02 now | 18:22 |
| jeblair | ++ | 18:23 |
| fungi | definitely wrote some log entries that time | 18:23 |
| jeblair | "number_of_nodes" : 5, | 18:23 |
| jeblair | that used to say 4 | 18:23 |
| fungi | yeah, we have an api again at least | 18:23 |
| *** baoli has joined #openstack-infra | 18:23 | |
| jeblair | why does that say 5 and not 6? | 18:24 |
| fungi | i'm guessing we lost more than es02 | 18:24 |
| jeblair | and the cluster status has changed from green to red | 18:25 |
| fungi | the other 5 servers are reachable at least | 18:25 |
| pabelanger | fungi: cloudnull: and new nodes being used in osic-cloud1-disk | 18:25 |
| fungi | es05 is reachable but not running the service | 18:26 |
| jeblair | did we have instructions for how to use bigdesk somewhere? | 18:26 |
| fungi | es05 has an uptime of a week | 18:27 |
| fungi | reboot system boot Fri Jul 14 01:39 - 18:27 (7+16:48) 0.0.0.0 | 18:28 |
| jeblair | should we let the cluster recover with 02 before adding 05? | 18:28 |
| clarkb | I think we dont have service set to start on boot so that we can do upgrqdes and such in controlled manner | 18:28 |
| clarkb | jeblair: that shouldnt be necessary | 18:28 |
| fungi | and no logins prior to that reboot, so that must have been a spontaneous one | 18:29 |
| fungi | clarkb: thanks, in that case should i go ahead and start the service on es05 as well? | 18:29 |
| *** jrist has quit IRC | 18:29 | |
| jeblair | clarkb: like normal os package upgrades, or 'big event' upgrades? | 18:30 |
| clarkb | jeblair: elasticsearch upgrades | 18:30 |
| jeblair | clarkb: it might be worth considering having the service start on boot normally, and then, when upgrading es, manually disable that? | 18:31 |
| clarkb | we've bundled os/kernel updates with es upgrades in the past so install new es, reboot, do that 6 servers then start vluster | 18:31 |
| clarkb | fungi: yes should be fine to start both | 18:31 |
| fungi | service started on es05 too now | 18:32 |
| fungi | "number_of_nodes" : 6, | 18:32 |
| jeblair | that's a good number | 18:33 |
| fungi | clarkb: will kibana continue to report "Connection Failed" until the cluster stops being in red status, or is there an additional api service which needs to be started as well? | 18:33 |
| jeblair | fungi: i don't get connection failed now | 18:33 |
| jeblair | fungi: i get no results | 18:33 |
| clarkb | fungi: there is just the one service, it ahould be able to manage api queries even when red | 18:33 |
| jeblair | which makes sense | 18:33 |
| fungi | oh, hah! | 18:34 |
| clarkb | red is a data state not api state | 18:34 |
| clarkb | means not all indexes are recovered yet | 18:34 |
| fungi | it had redirected me to http://logstash.openstack.org/#/connectionFailed and i didn't notice that i just kept refreshing that | 18:34 |
| fungi | :/ | 18:34 |
| fungi | beer-thirty might be approaching earlier than usual today | 18:34 |
| fungi | and yeah, much more of the kibana interface is loading for me now, just missing recent data | 18:35 |
| clarkb | yellow is all primary indexes recovered but not secondary indexes | 18:36 |
| clarkb | green is all indexes are happy | 18:36 |
| fungi | querying back 2 days it returns resolys from up to 2017-07-20T21:25:00.372+00:00 now | 18:36 |
| fungi | s/resolys/results/ | 18:36 |
| AJaeger | infra-root, a heads up: docs.o.o is broken, I guess we have a wrong .htaccess. dhellmann and myself are looking at it | 18:37 |
| fungi | AJaeger: thanks for the note | 18:37 |
| *** baoli has quit IRC | 18:37 | |
| AJaeger | #status notice docs.o.o is currently broken, we're investigating | 18:37 |
| openstackstatus | AJaeger: sending notice | 18:37 |
| AJaeger | argh, wanted #status log instead... | 18:38 |
| *** baoli has joined #openstack-infra | 18:38 | |
| fungi | AJaeger: doesn't hurt to make that a little more widely known i guess | 18:38 |
| *** mdrabe has quit IRC | 18:38 | |
| AJaeger | fungi: yep | 18:38 |
| -openstackstatus- NOTICE: docs.o.o is currently broken, we're investigating | 18:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - multinode testing https://review.openstack.org/486195 | 18:39 |
| *** baoli has quit IRC | 18:40 | |
| openstackstatus | AJaeger: finished sending notice | 18:41 |
| fungi | #status log elasticsearch02 has been hard-rebooted via nova after it hung at roughly 21:25 utc yesterday; elasticsearch service on elasticsearch05 also had to be manually started following a spontaneous reboot from 2017-07-14 01:39..18:27 (provider ticket from that date mentions an unresponsive hypervisor host); cluster is recovering now but kibana on logstash.o.o is currently missing entries past | 18:42 |
| openstackstatus | fungi: finished logging | 18:42 |
| fungi | 21:25 utc yesterday | 18:42 |
| fungi | #status log kibana on logstash.o.o is currently missing entries past 21:25 utc yesterday | 18:43 |
| openstackstatus | fungi: finished logging | 18:43 |
| *** mamitchl has joined #openstack-infra | 18:43 | |
| *** mdrabe has joined #openstack-infra | 18:44 | |
| *** bobh has quit IRC | 18:44 | |
| fungi | melwitt: mriedem: ^ | 18:44 |
| mriedem | ok, thanks for digging in anyway | 18:44 |
| melwitt | thanks fungi | 18:44 |
| *** MasterOfBugs has joined #openstack-infra | 18:45 | |
| fungi | mriedem: melwitt: i don't know for sure, but hope that as the cluster recovers and indices catch up we'll catch back up on entries | 18:45 |
| *** nicolasbock has quit IRC | 18:45 | |
| melwitt | hope so too | 18:47 |
| *** jkilpatr has quit IRC | 18:47 | |
| fungi | cluster health numbers at least seem to be improving rapidly as shards continue to get initialized/assigned/relocated and become active | 18:47 |
| *** baoli has joined #openstack-infra | 18:48 | |
| *** sdague has joined #openstack-infra | 18:48 | |
| *** baoli has quit IRC | 18:49 | |
| *** jkilpatr has joined #openstack-infra | 18:50 | |
| *** vponomaryov has quit IRC | 18:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Limit add-build-sshkey to localhost https://review.openstack.org/486199 | 18:52 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 18:53 |
| *** jangutter has joined #openstack-infra | 18:53 | |
| jangutter | Hi, would this be the right channel to ask for tips about setting up a third-party CI? I need to brief a bunch of people to do some work for me. | 18:54 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 18:56 |
| fungi | jangutter: there's also #openstack-third-party-ci which may be better for that purpose | 18:56 |
| *** baoli has joined #openstack-infra | 18:56 | |
| jangutter | Thanks! | 18:56 |
| fungi | it tends to have more third-party ci operators paying attention than we do in here | 18:57 |
| fungi | we're more focused on building/running the first-party ci system | 18:57 |
| jangutter | Heh, of which the third-party guys are just the consumers :-p | 18:57 |
| *** baoli has quit IRC | 18:57 | |
| fungi | nah, they provide useful feedback as well as some bug fixes! | 18:57 |
| fungi | we definitely have some overlap between people helping write the tools and people using them to run other systems | 18:58 |
| *** baoli has joined #openstack-infra | 18:59 | |
| *** annegentle has quit IRC | 19:00 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: invert add-build-sshkey logic https://review.openstack.org/486203 | 19:00 |
| jangutter | OK, one last "real newbie" question. Are two machines sufficient? A logserver and a baremetal server (it needs to run stuff on naked HW). | 19:02 |
| smcginnis | fungi: Sorry for the high latency, I'm on the road. | 19:04 |
| smcginnis | fungi: Launchpad ID should be https://launchpad.net/~sean-mcginnis | 19:04 |
| *** mamitchl has quit IRC | 19:05 | |
| *** mdrabe has quit IRC | 19:06 | |
| *** annegentle has joined #openstack-infra | 19:06 | |
| *** jcoufal_ has quit IRC | 19:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - invert add-build-sshkey logic https://review.openstack.org/486203 | 19:08 |
| jangutter | fungi: thanks, but it seems it's all quiet there now. I'll check later. Would you perhaps know if two nodes (logserver and HW devstack testing server) are sufficient? | 19:10 |
| fungi | jangutter: probably, but our team doesn't really set policies for designing third-party ci systems. you could just as well use a virtual machine to run the entire ci stack and then have that drive jobs remotely on your bare metal system, but it all depends on how you design it and which services you end up using | 19:10 |
| fungi | and whatever project(s) you're going to be testing changes for are the ones who decide what degree of testing they expect | 19:11 |
| *** ldnunes has quit IRC | 19:11 | |
| fungi | we don't run any testing on bare metal here, so i don't have much experience trying to design a ci system to incorporate that element | 19:11 |
| jangutter | fungi: yep, it's about as simple as "How to install OpenStack" :-p. Thanks and have a great weekend! | 19:12 |
| fungi | if on the other hand you want recommendations on ways to run thousands of jobs in parallem on disposable virtual machines across a dozen different and varies public cloud providers, i'm happy to field that ;) | 19:12 |
| *** EricGonc_ has quit IRC | 19:14 | |
| *** EricGonczer_ has joined #openstack-infra | 19:15 | |
| fungi | smcginnis: i've finally freed myself up enough to finish digging into it. i'm willing to bet we've got multiple lp openids for you in gerrit due to an account merge or something and the bug update script is picking thr "wrong" one | 19:15 |
| smcginnis | fungi: That's very likely true. | 19:16 |
| fungi | smcginnis: also it explains why the updates seem to you like they've been broken for a while, yet nobody else has noticed it ;) | 19:16 |
| smcginnis | fungi: I had a "personal" ona and a work one. I've tried to merge them, and I think for the most part they are, but still get weirdness. | 19:17 |
| smcginnis | fungi: Hah, yep. ;) | 19:17 |
| fungi | we did indeed at one point have two accounts for you, it seems. current active account is 11904 and disabled account is 23390 | 19:18 |
| *** iyamahat has quit IRC | 19:19 | |
| fungi | and as suspected, we have two openids listed (in case you tried to log in with the "other" account you would end up in the correct one) | 19:19 |
| fungi | this is probably the culprit. i'll work out which one maps to the correct lp account and delete the other row | 19:19 |
| *** yamahata has quit IRC | 19:20 | |
| *** jangutter has quit IRC | 19:20 | |
| *** mdrabe has joined #openstack-infra | 19:21 | |
| *** mamitchl has joined #openstack-infra | 19:21 | |
| smcginnis | fungi: That logging in with the other account thing got messed up. | 19:21 |
| smcginnis | fungi: I had been able to do that for awhile. | 19:22 |
| smcginnis | fungi: Then severl months back we did some sort of update, and it never worked after that. | 19:22 |
| fungi | hrm, nope those two openids are just to deal with the global transition from login.launchpad.net to login.ubuntu.com | 19:22 |
| smcginnis | fungi: That's when I tried merging the two. | 19:22 |
| smcginnis | fungi: Must have been after that transition | 19:22 |
| fungi | okay, so it _actually_ looks like the bug update hook is finding your incorrect disabled account and using the openid from that | 19:23 |
| smcginnis | Would have preferred to stick with my gmail one, but since I had so much under my work one that seemed the path of least resistance. | 19:23 |
| fungi | this may be a bug in the script | 19:23 |
| *** ldnunes has joined #openstack-infra | 19:24 | |
| smcginnis | fungi: If there's anything I need to do account-wise, just let me know. | 19:24 |
| fungi | smcginnis: i think we should probably be constraining our db query to active accounts | 19:24 |
| smcginnis | fungi: Seems like a good thing to do. :) | 19:25 |
| fungi | smcginnis: yep, that's the bug... http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/update_bug.py#n110 | 19:26 |
| fungi | though this is going to turn into a trickier join since i need to map to the accounts table to check whether inactive is Y or N | 19:27 |
| *** mamitchl has quit IRC | 19:28 | |
| *** mdrabe has quit IRC | 19:28 | |
| smcginnis | fungi: Something like an INNER JOIN account a on a.account_id = original.account_id WHERE a.inactive = FALSE ? | 19:29 |
| smcginnis | fungi: Is that schema documented? | 19:30 |
| fungi | smcginnis: it _might_ be but honestly this is internal implementation details of gerrit which they don't generally consider a stable interface | 19:30 |
| smcginnis | fungi: That's usually the fun stuff to hack around. :) | 19:31 |
| fungi | which is why i've tended to include verbose explanations in code comments there | 19:31 |
| smcginnis | Good call. | 19:31 |
| dirk | hwoarang: no, feel free to rework | 19:32 |
| dirk | hwoarang: I am mostly busy trying to understand why devstack broke again | 19:32 |
| clarkb | fungi: you might be able to get that info via public api | 19:32 |
| fungi | clarkb: not the openid lookup, unfortunately | 19:32 |
| dirk | It looks like keystone is segfaulting somewhere in openssl | 19:32 |
| fungi | clarkb: or at least not the last time i checked | 19:33 |
| clarkb | fungi: can you not query via owner:openid that eay? | 19:33 |
| clarkb | username is valid that way at least | 19:33 |
| fungi | clarkb: we're ultimately querying to _get_ the openid on record in gerrit, so we can pass that to the lp api to ask it which lp account it's for | 19:33 |
| clarkb | ah | 19:34 |
| fungi | and gerrit didn't seem to let you use the public api to find out the openid for an account (though maybe a privileged account can do that with an authenticated api call, we could still probably create a dedicated service account with the right access and do it that way if so) | 19:35 |
| *** dmsimard|afk is now known as dmsimard | 19:35 | |
| fungi | but i think i mostly have the sql syntax figured out for this | 19:35 |
| fungi | especially since i have a reproducer handy! ;) | 19:35 |
| *** iyamahat has joined #openstack-infra | 19:36 | |
| fungi | yeah, the original query returns all openids for accounts associated with that e-mail address | 19:37 |
| fungi | just confirmed by running the query manually through mysqlclient | 19:38 |
| *** florianf has quit IRC | 19:38 | |
| *** patriciadomin has joined #openstack-infra | 19:39 | |
| sdague | clarkb / fungi - http://logs.openstack.org/75/419975/20/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/17f4349/logs/devstack-gate-setup-workspace-new.txt.gz setupworkspace taking 25 minutes on a node? | 19:40 |
| sdague | that caused it to eventually get timed out | 19:40 |
| sdague | but that seems really out of whack | 19:40 |
| sdague | infracloud nodes | 19:41 |
| sdague | taking a full minute to do repo clones some times? | 19:41 |
| *** dougwig has joined #openstack-infra | 19:41 | |
| *** mdrabe has joined #openstack-infra | 19:42 | |
| *** mamitchl has joined #openstack-infra | 19:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/jeepyb master: Update bugs only with active Gerrit accounts https://review.openstack.org/486212 | 19:48 |
| fungi | AJaeger: smcginnis: clarkb: yeesh... ^ | 19:48 |
| pabelanger | infracloud nodes were maxed out around the time too, we had stopped osic-cloud1 | 19:49 |
| pabelanger | wonder if more overcommit issues | 19:49 |
| smcginnis | fungi: Thanks! | 19:49 |
| fungi | pabelanger: maybe we need to dial the max-servers there down even more than we already have? | 19:49 |
| pabelanger | fungi: wonder if I can find out which compute host that was running on | 19:50 |
| pabelanger | and check cacit.o.o | 19:50 |
| pabelanger | btw | 19:51 |
| pabelanger | ubuntu-xenial-infracloud-vanilla-10010347 | 19:51 |
| fungi | pabelanger: should be able to map the server name to a uuid in the nodepool log and then check the nova logs for it? | 19:51 |
| pabelanger | over 10million VMS with nodepool now :D | 19:51 |
| * fungi hangs an "over 10 million (nodes) served" sign on nodepool.o.o | 19:52 | |
| pabelanger | ++ | 19:52 |
| mtreinish | fungi: so I tried a few different images, the fedora released f26 image for openstack to use the config drive with cloud-init: http://paste.openstack.org/show/616207/ | 19:53 |
| mtreinish | but not an f25, cirros, or centos7 from upstream | 19:53 |
| mtreinish | I still need to try creating some images by hand | 19:54 |
| fungi | mtreinish: so f26 cloud-init works out of the box? that's not surprising that other distros/releases don't support it yet | 19:54 |
| *** xyang1 has quit IRC | 19:55 | |
| mtreinish | yep, I was kinda surprised, because that's the first out of the box image that's worked | 19:55 |
| fungi | i want to say the support only made it into a cloud-init release as of 6-9 months ago or something like that | 19:55 |
| mtreinish | well besides ubuntu, but nova was doing file injection there | 19:55 |
| sdague | pabelanger: ok, so the take is that infracloud had an issue which is now resolved? | 19:55 |
| pabelanger | sdague: ya, we've seen preformance issues in infracloud when we max our the compute nodes. Trying to see that that is the case here | 19:55 |
| *** yamahata has joined #openstack-infra | 19:55 | |
| fungi | sdague: theory is that we've still overestimated our effective capacity for the infra-cloud hardware, but since we've only very recently had spikes of the ci system maxing out capacity everywhere we didn't realize it (and so we've been tuning accordingly) | 19:56 |
| *** hasharDinner is now known as hashar | 19:56 | |
| fungi | we already dropped the max-servers there last week (i think) but may not have lowered it quite enough to avoid contention still | 19:57 |
| *** Swami has quit IRC | 19:58 | |
| *** xyang1 has joined #openstack-infra | 19:58 | |
| cloudnull | pabelanger: awesome! | 19:59 |
| cloudnull | thanks everyone for helping getting that done. | 19:59 |
| fungi | cloudnull: thanks for keeping (former) osic online! | 19:59 |
| pabelanger | ++ | 20:00 |
| cloudnull | its my pleasure. I'll keep things limping along so long as they allow me | 20:00 |
| fungi | "old service intel canned" | 20:00 |
| cloudnull | ++ | 20:01 |
| cloudnull | I should make that a landing page for osic.xyz | 20:01 |
| *** dave-mccowan has quit IRC | 20:02 | |
| pabelanger | | 139859be-f0b3-46f1-a61a-bc16e8f1a072 | ubuntu-xenial-infracloud-vanilla-10010347 | DELETED | None | NOSTATE | | ubuntu-xenial-1500584281 | 1cc7329f-a3e8-4d61-a6dd-7258623dc025 | nodepool | a3d1723d-8e05-4736-96de-b523af714db7 | nova | compute032.vanilla.ic.openstack.org | | | 20:03 |
| pabelanger | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=3&leaf_id=436 | 20:03 |
| *** jkilpatr has quit IRC | 20:03 | |
| spotz | He still posts as the boyfriend though they're married | 20:04 |
| pabelanger | fungi: sdague: Oh ya. Load was up to 14 around that time: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=5969&rra_id=all | 20:04 |
| sdague | pabelanger: yeh, that's not ideal | 20:05 |
| spotz | woops wrong channel:) | 20:05 |
| *** bobh has joined #openstack-infra | 20:05 | |
| *** wolverineav has joined #openstack-infra | 20:05 | |
| pabelanger | I'm going to audit infracloud next week and re-caclulate our max-servers | 20:05 |
| pabelanger | we likely need to adjust them | 20:05 |
| fungi | pabelanger: thanks, sounds good. in the meantime with osic back online this shouldn't bite us | 20:06 |
| fungi | c | 20:08 |
| fungi | perhaps i should do that in my mutt window instead of my irc window | 20:08 |
| *** rodrigopaiva_ has quit IRC | 20:09 | |
| *** rodrigopaiva_ has joined #openstack-infra | 20:09 | |
| *** bobh has quit IRC | 20:09 | |
| *** rodrigopaiva has joined #openstack-infra | 20:10 | |
| *** rodrigopaiva_ has quit IRC | 20:10 | |
| *** rodrigopaiva has quit IRC | 20:11 | |
| *** eharney has quit IRC | 20:13 | |
| fungi | pabelanger: clarkb: i cc'd you on my e-mail to daniel at citynet about the intermittent network issues in sto2 since you were on the earlier threads about the "extra" ip addresses instances were getting too | 20:13 |
| fungi | given the time, i don't expect him to see it until monday at this point | 20:14 |
| clarkb | sounds good, thanks | 20:15 |
| *** aviau has quit IRC | 20:16 | |
| *** aviau has joined #openstack-infra | 20:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Only run once our delegate_to locahost commands https://review.openstack.org/486217 | 20:23 |
| *** boris-42____ has quit IRC | 20:28 | |
| *** camunoz has quit IRC | 20:29 | |
| fungi | clarkb: does the elasticsearch cluster status go back to green after there are no shards in relocating, initializing, unassigned or delayed unassigned states? | 20:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix stat check for add-build-sshkey https://review.openstack.org/486219 | 20:29 |
| *** mriedem has quit IRC | 20:30 | |
| *** jrist has joined #openstack-infra | 20:33 | |
| *** kgiusti has left #openstack-infra | 20:33 | |
| *** bobh has joined #openstack-infra | 20:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix stat check for add-build-sshkey https://review.openstack.org/486219 | 20:38 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix stat check for add-build-sshkey https://review.openstack.org/486219 | 20:40 |
| *** chandankumar is now known as dassistant | 20:41 | |
| *** bobh has quit IRC | 20:41 | |
| clarkb | fungi: yes | 20:43 |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config master: Filter out services in designate jobs https://review.openstack.org/486221 | 20:43 |
| clarkb | and yellow after red once all primary shards are good | 20:43 |
| fungi | unassigned shards is still dropping but has slowed down quite a bit | 20:43 |
| fungi | we're at 52 unassigned, 3 initializing nopw | 20:44 |
| fungi | s/nopw/now/ | 20:44 |
| pabelanger | fungi: thanks, just seen email for citycloud | 20:45 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Only run once our delegate_to locahost commands https://review.openstack.org/486217 | 20:46 |
| *** baoli has quit IRC | 20:47 | |
| *** bobh has joined #openstack-infra | 20:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - multinode testing! https://review.openstack.org/486195 | 20:50 |
| *** bnemec has quit IRC | 20:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove refspec https://review.openstack.org/485875 | 20:51 |
| *** jkilpatr has joined #openstack-infra | 20:52 | |
| *** ldnunes has quit IRC | 20:53 | |
| fungi | okay, this is an interesting solution to run across... a python project that recommends using *tox* as its installer: https://www.qutebrowser.org/INSTALL.html#tox | 20:57 |
| fungi | nothing too out of the ordinary in their tox.ini though... mainly just a no-op picking a specific additional reqs file | 21:00 |
| *** baoli has joined #openstack-infra | 21:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use undefined values instead of 40 zeroes https://review.openstack.org/486225 | 21:01 |
| fungi | they could just as well have used extras with their install_requires | 21:01 |
| *** trown is now known as trown|lunch | 21:01 | |
| fungi | to the same effect (though perhaps they're catering to older pip/setuptools) | 21:01 |
| *** trown|lunch is now known as trown|outtypewww | 21:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix broken project config in dynamic config test https://review.openstack.org/486226 | 21:05 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Don't ignore inexistent jobs in dynamic config https://review.openstack.org/486227 | 21:05 |
| *** ldnunes has joined #openstack-infra | 21:05 | |
| jeblair | fungi: wow, erm. considering the number of times i'm like "oops, tox ate my venv; rebuild", i don't favor that as a production deployment strategy. | 21:05 |
| *** eharney has joined #openstack-infra | 21:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Store tox logs by inventory_hostname https://review.openstack.org/486228 | 21:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - multinode testing https://review.openstack.org/486195 | 21:08 |
| *** gordc has quit IRC | 21:09 | |
| *** bnemec has joined #openstack-infra | 21:11 | |
| *** gouthamr has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** otherwiseguy has quit IRC | 21:20 | |
| *** rnoriega has quit IRC | 21:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Support multi node jobs for tox logs https://review.openstack.org/486228 | 21:23 |
| *** annegentle has quit IRC | 21:31 | |
| *** _ari_ is now known as _ari_|gone | 21:33 | |
| *** dprince has quit IRC | 21:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Support multi node jobs for tox logs https://review.openstack.org/486228 | 21:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove refspec https://review.openstack.org/485875 | 21:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_REFNAME https://review.openstack.org/486232 | 21:43 |
| *** baoli has quit IRC | 21:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_OLDREV and ZUUL_NEWREV https://review.openstack.org/486233 | 21:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_COMMIT https://review.openstack.org/486235 | 21:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_REF https://review.openstack.org/486236 | 21:55 |
| *** bobh has quit IRC | 21:59 | |
| *** Goneri has quit IRC | 22:05 | |
| *** wolverineav has quit IRC | 22:05 | |
| *** wolverineav has joined #openstack-infra | 22:05 | |
| *** flepied has quit IRC | 22:07 | |
| *** wolverineav has quit IRC | 22:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_PATCHSET https://review.openstack.org/486240 | 22:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGE https://review.openstack.org/486241 | 22:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix the rendering of item entries https://review.openstack.org/486242 | 22:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGE_IDS https://review.openstack.org/486243 | 22:13 |
| *** rlandy has quit IRC | 22:15 | |
| *** Sukhdev has quit IRC | 22:17 | |
| *** annegentle has joined #openstack-infra | 22:23 | |
| *** tesseract has quit IRC | 22:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGES https://review.openstack.org/486245 | 22:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_BRANCH https://review.openstack.org/486246 | 22:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_VOTING and add zuul.voting https://review.openstack.org/486247 | 22:25 |
| *** hashar has quit IRC | 22:27 | |
| *** marst_ has quit IRC | 22:28 | |
| *** marst has joined #openstack-infra | 22:29 | |
| *** annegentle has quit IRC | 22:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_URL https://review.openstack.org/486249 | 22:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_PIPELINE https://review.openstack.org/486250 | 22:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_PROJECT https://review.openstack.org/486251 | 22:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_UUID https://review.openstack.org/486252 | 22:32 |
| *** sdague has quit IRC | 22:33 | |
| *** marst has quit IRC | 22:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGE_IDS https://review.openstack.org/486243 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGES https://review.openstack.org/486245 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_VOTING and add zuul.voting https://review.openstack.org/486247 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_BRANCH https://review.openstack.org/486246 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_URL https://review.openstack.org/486249 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_PROJECT https://review.openstack.org/486251 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_PIPELINE https://review.openstack.org/486250 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_UUID https://review.openstack.org/486252 | 22:38 |
| *** mattmceuen has quit IRC | 22:49 | |
| *** sdague has joined #openstack-infra | 22:51 | |
| *** EricGonczer_ has quit IRC | 22:52 | |
| *** gouthamr has joined #openstack-infra | 22:59 | |
| *** Goneri has joined #openstack-infra | 23:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_VOTING and add zuul.voting https://review.openstack.org/486247 | 23:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_URL https://review.openstack.org/486249 | 23:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_PROJECT https://review.openstack.org/486251 | 23:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_PIPELINE https://review.openstack.org/486250 | 23:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ZUUL_UUID https://review.openstack.org/486252 | 23:04 |
| *** esberglu has quit IRC | 23:05 | |
| *** felipemonteiro_ has quit IRC | 23:07 | |
| *** xyang1 has quit IRC | 23:11 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove gather_facts: true https://review.openstack.org/485834 | 23:18 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove validate-host from unittests/pre.yaml https://review.openstack.org/485836 | 23:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use undefined values instead of 40 zeroes https://review.openstack.org/486225 | 23:19 |
| *** EricGonczer_ has joined #openstack-infra | 23:20 | |
| *** dougwig has quit IRC | 23:22 | |
| *** EricGonc_ has joined #openstack-infra | 23:23 | |
| *** EricGonc_ has quit IRC | 23:24 | |
| fungi | i fear something has gone sideways with our elasticsearch cluster recovery. it's been sitting at 52 unassigned shards for over an hour now | 23:24 |
| *** thorst has quit IRC | 23:24 | |
| *** EricGonczer_ has quit IRC | 23:24 | |
| jeblair | fungi: io use fell off on es05 around 2120 | 23:25 |
| *** HeOS has joined #openstack-infra | 23:25 | |
| fungi | looks like es02 logged a recovery failed exception around 19:39 | 23:25 |
| fungi | for a specific shard | 23:26 |
| *** EricGonczer_ has joined #openstack-infra | 23:26 | |
| jeblair | fungi: maybe we should drop logstash-2017.07.06 | 23:27 |
| fungi | es05 logged a bunch of dangling index messages at 18:32 | 23:27 |
| fungi | but nothing since | 23:27 |
| fungi | oh, that was at startup for the service | 23:27 |
| fungi | jeblair: yeah, that date's around our expiration timeframe anyway | 23:28 |
| fungi | maybe it was in the process of being expired/culled when the cluster lost coherency | 23:28 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Log skipped tasks to the job output https://review.openstack.org/485896 | 23:29 |
| jeblair | fungi: actually, we're at 10 days? | 23:29 |
| jeblair | so that would be past our expiration | 23:29 |
| fungi | oh, wow | 23:29 |
| jeblair | 0 2 * * * curator --logfile /var/log/curator.log delete indices --older-than 10 --time-unit days --timestring \%Y.\%m.\%d | 23:29 |
| fungi | for some reason i thought it was still around 14/15 | 23:29 |
| jeblair | maybe i should go ahead and run that? | 23:30 |
| fungi | by all means, seems like a fine idea | 23:30 |
| jeblair | The program 'screen' is currently not installed. You can install it by typing: | 23:30 |
| jeblair | apt-get install screen | 23:30 |
| jeblair | i'm going to install that first | 23:30 |
| fungi | right on | 23:30 |
| fungi | i wonder if those were shards under the care of es05 which was offline for a week | 23:31 |
| jeblair | apparently we run that on every es host | 23:31 |
| *** r-daneel has quit IRC | 23:31 | |
| jeblair | 2017-07-21 23:30:50,731 INFO Action delete will be performed on the following indices: ['logstash-2017.07.04', 'logstash-2017.07.05', 'logstash-2017.07.06', 'logstash-2017.07.07', 'logstash-2017.07.08', 'logstash-2017.07.09', 'logstash-2017.07.10', 'logstash-2017.07.11'] | 23:32 |
| jeblair | 2017-07-21 23:31:09,036 INFO Job completed successfully. | 23:32 |
| jeblair | fungi: does it look happier now? | 23:32 |
| fungi | yes! | 23:33 |
| fungi | that cleared out all the unassigned shards | 23:33 |
| fungi | two relocating noww | 23:33 |
| fungi | status GREEN | 23:33 |
| jeblair | still nothing in logstash | 23:33 |
| fungi | i'm checking the workers | 23:34 |
| *** panda is now known as panda|off | 23:34 | |
| jeblair | i don't see any log entries from them since 2017-07-20 21:51:28 | 23:35 |
| *** EricGonczer_ has quit IRC | 23:35 | |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=1245&rra_id=2&view_type=&graph_start=1500075376&graph_end=1500680176 | 23:36 |
| jeblair | fungi: maybe restart all of them? | 23:36 |
| jeblair | (i don't understand why the queue hasn't just increased though) | 23:37 |
| fungi | restarted all 4 workers on l-w01 to see what happens | 23:38 |
| jeblair | it appears to be doing work | 23:39 |
| *** markvoelker has quit IRC | 23:39 | |
| *** csomerville has quit IRC | 23:39 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 23:39 |
| fungi | the worker debug logs claim they're processing entries, yeah | 23:40 |
| jeblair | none of that work involves pushing lines though? | 23:40 |
| fungi | worker logs indicate quite a few 404 errors trying to retrieve logs | 23:41 |
| jeblair | i think that's normal, we're not at all smart about them. but usually we'd hit something.. | 23:41 |
| jeblair | oh hey i think i just saw one push 900 lines | 23:41 |
| jeblair | 2017-07-21 23:41:41,374 Pushing 900 log lines. | 23:42 |
| fungi | ahh, okay | 23:42 |
| *** mattmceuen has joined #openstack-infra | 23:42 | |
| fungi | well, we log very wordy python tracebacks for every 404 | 23:43 |
| fungi | yeah, i see "pushing 900 log lines" there in the log | 23:44 |
| fungi | the vast majority seem to be 0 | 23:44 |
| fungi | now i see more | 23:44 |
| fungi | latest was 2113 lines for a push | 23:45 |
| jeblair | i can't tell if logstash is doing anything :/ | 23:45 |
| fungi | agreed, latest logline kibana reports is still 2017-07-20T21:25:00.372+00:00 | 23:46 |
| jeblair | fungi: maybe we should restart logstash on the worker too? | 23:47 |
| fungi | oh! i forgot that ran there | 23:47 |
| fungi | its log is also quite silent. doing that now | 23:47 |
| fungi | was running since may 3 but its log was silent | 23:47 |
| fungi | "logstash stop failed; still running." | 23:48 |
| fungi | grr | 23:48 |
| jeblair | i consider that promising actually :) | 23:48 |
| jeblair | like, it may actually have been stuck | 23:48 |
| jeblair | i straced it. this is the only interesting thing i saw: | 23:48 |
| jeblair | [pid 14526] futex(0x7f215413e854, FUTEX_WAIT_BITSET_PRIVATE, 1, {36407227, 925966142}, ffffffff) = -1 ETIMEDOUT (Connection timed out) | 23:48 |
| fungi | makes sense | 23:49 |
| jeblair | other than that, it's all ordinary futexes. | 23:49 |
| fungi | i tried to gracefully stop it again, still no dice | 23:49 |
| jeblair | no send/recv | 23:49 |
| fungi | shall i try sigsegv or go straight for sigkill? | 23:49 |
| fungi | actually i'll try a hup first. sometimes that will get it to notice other queued signals | 23:49 |
| jeblair | fungi: i have no preference | 23:50 |
| fungi | though this is java, so... anybody's guess | 23:50 |
| fungi | -7 worked | 23:50 |
| fungi | -1 didn't seem to help | 23:51 |
| fungi | starting it now | 23:51 |
| fungi | lots of log entries now | 23:51 |
| *** mattmceuen has quit IRC | 23:52 | |
| jeblair | fungi: we may need to restart the processors again | 23:52 |
| fungi | in the logstash.log i mean. though all look like startup messages | 23:52 |
| fungi | i'll give that a go now | 23:52 |
| *** baoli has joined #openstack-infra | 23:52 | |
| *** gouthamr has quit IRC | 23:52 | |
| jeblair | (their logs are now idle) | 23:52 |
| jeblair | yay! | 23:53 |
| jeblair | http://logstash.openstack.org/#/dashboard/file/logstash.json has data | 23:53 |
| jeblair | fungi: so i think we should probably stop log processors, stop logstash, start logstash, start processors on all of the hosts | 23:54 |
| fungi | agreed! kibana has new log entries | 23:54 |
| fungi | i'll start looping through those. making sure logstash dies will probably be the fun part | 23:54 |
| jeblair | ya | 23:54 |
| jeblair | fungi: you want to get them all, or want me to take half? | 23:55 |
| fungi | i'll get 'em | 23:55 |
| jeblair | fungi: thanks! | 23:56 |
| fungi | i'll just bludgeon any that don't die gracefully. easy enough to repeat the loop until stuff's down | 23:56 |
| *** baoli has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!