| *** jamesmcarthur has quit IRC | 00:01 | |
| *** pahuang has joined #openstack-infra | 00:04 | |
| *** s-shiono has joined #openstack-infra | 00:04 | |
| *** Goneri has joined #openstack-infra | 00:05 | |
| *** thorst has joined #openstack-infra | 00:08 | |
| *** dingyichen has joined #openstack-infra | 00:11 | |
| *** felipemonteiro_ has joined #openstack-infra | 00:11 | |
| *** thorst has quit IRC | 00:12 | |
| *** jascott1 has joined #openstack-infra | 00:13 | |
| *** sflanigan has joined #openstack-infra | 00:16 | |
| *** sflanigan has joined #openstack-infra | 00:16 | |
| *** jascott1 has quit IRC | 00:17 | |
| *** jamesmcarthur has joined #openstack-infra | 00:17 | |
| *** gmann has quit IRC | 00:18 | |
| *** gmann has joined #openstack-infra | 00:18 | |
| *** slagle has joined #openstack-infra | 00:28 | |
| *** sdake_ has quit IRC | 00:32 | |
| *** sdake_ has joined #openstack-infra | 00:32 | |
| *** jamesmcarthur has quit IRC | 00:32 | |
| *** aeng has quit IRC | 00:33 | |
| *** spzala has joined #openstack-infra | 00:42 | |
| *** zhurong has joined #openstack-infra | 00:43 | |
| *** funzo has joined #openstack-infra | 00:45 | |
| *** bobh has joined #openstack-infra | 00:47 | |
| *** bobh has quit IRC | 00:49 | |
| *** funzo has quit IRC | 00:49 | |
| *** bobh has joined #openstack-infra | 00:50 | |
| *** aeng has joined #openstack-infra | 00:50 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-client-config master: Updated from global requirements https://review.openstack.org/491294 | 00:51 |
|---|---|---|
| *** bobh has quit IRC | 00:52 | |
| *** zhurong has quit IRC | 00:53 | |
| *** aviau has quit IRC | 00:54 | |
| *** aviau has joined #openstack-infra | 00:55 | |
| *** spzala has quit IRC | 00:57 | |
| *** bobh has joined #openstack-infra | 01:01 | |
| *** shu-mutou has joined #openstack-infra | 01:08 | |
| *** jamesmcarthur has joined #openstack-infra | 01:08 | |
| *** slagle has quit IRC | 01:08 | |
| *** lewo has quit IRC | 01:09 | |
| *** lewo has joined #openstack-infra | 01:10 | |
| *** markvoelker has joined #openstack-infra | 01:15 | |
| *** LindaWang has joined #openstack-infra | 01:17 | |
| *** liujiong has joined #openstack-infra | 01:19 | |
| *** hongbin has joined #openstack-infra | 01:20 | |
| *** aeng has quit IRC | 01:20 | |
| *** aeng has joined #openstack-infra | 01:21 | |
| *** gildub has quit IRC | 01:27 | |
| *** bobh has quit IRC | 01:32 | |
| *** jamesmcarthur has quit IRC | 01:32 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update status.o.o footer https://review.openstack.org/491007 | 01:33 |
| *** jamesmcarthur has joined #openstack-infra | 01:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove ansible-role-functional-jobs-centos7 from ansible-role-zuul https://review.openstack.org/491257 | 01:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Run TripleO scenario002 for Telemetry services https://review.openstack.org/491099 | 01:37 |
| *** aeng has quit IRC | 01:39 | |
| *** sekelso has joined #openstack-infra | 01:39 | |
| *** jamesmcarthur has quit IRC | 01:41 | |
| openstackgerrit | Merged openstack-infra/puppet-mailman master: Good Bye Diablo: Update pages with footer https://review.openstack.org/490986 | 01:42 |
| openstackgerrit | Merged openstack-infra/puppet-mailman master: Use https for links https://review.openstack.org/491013 | 01:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Update OpenStack-Helm gates https://review.openstack.org/490647 | 01:43 |
| *** bobh has joined #openstack-infra | 01:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add "check experimental" jobs to mistral https://review.openstack.org/490086 | 01:44 |
| *** cuongnv has joined #openstack-infra | 01:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: Moves the ironic python3 experimental job as non-voting job https://review.openstack.org/489278 | 01:47 |
| *** liusheng has joined #openstack-infra | 01:47 | |
| *** Goneri has quit IRC | 01:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move ironic experimental BFV job to check https://review.openstack.org/490710 | 01:48 |
| openstackgerrit | Merged openstack-infra/project-config master: Add TripleO OVB job to mistral experimental https://review.openstack.org/490871 | 01:48 |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-to-pypi job for monasca-ceilometer https://review.openstack.org/484494 | 01:48 |
| *** sekelso has quit IRC | 01:49 | |
| *** gildub has joined #openstack-infra | 01:59 | |
| *** markvoelker has quit IRC | 02:02 | |
| *** markvoelker has joined #openstack-infra | 02:02 | |
| *** thorst has joined #openstack-infra | 02:13 | |
| *** aeng has joined #openstack-infra | 02:17 | |
| *** thorst has quit IRC | 02:18 | |
| *** makowals has joined #openstack-infra | 02:20 | |
| *** tuanluong has joined #openstack-infra | 02:26 | |
| *** spzala has joined #openstack-infra | 02:32 | |
| *** markvoelker has quit IRC | 02:34 | |
| *** huanxie has joined #openstack-infra | 02:37 | |
| *** spzala has quit IRC | 02:37 | |
| *** spzala has joined #openstack-infra | 02:38 | |
| *** gildub has quit IRC | 02:42 | |
| *** funzo has joined #openstack-infra | 02:43 | |
| *** bobh has quit IRC | 02:45 | |
| *** funzo has quit IRC | 02:46 | |
| *** funzo has joined #openstack-infra | 02:46 | |
| *** sekelso has joined #openstack-infra | 02:49 | |
| *** skelso has joined #openstack-infra | 02:53 | |
| *** sekelso has quit IRC | 02:55 | |
| *** gildub has joined #openstack-infra | 03:06 | |
| *** liusheng has quit IRC | 03:06 | |
| *** liusheng has joined #openstack-infra | 03:07 | |
| *** felipemonteiro__ has joined #openstack-infra | 03:08 | |
| *** dave-mccowan has quit IRC | 03:09 | |
| *** felipemonteiro_ has quit IRC | 03:11 | |
| *** spzala has quit IRC | 03:11 | |
| *** baoli has joined #openstack-infra | 03:13 | |
| *** thorst has joined #openstack-infra | 03:14 | |
| *** felipemonteiro__ has quit IRC | 03:18 | |
| *** thorst has quit IRC | 03:19 | |
| EmilienM | pabelanger: situation with mirror issue is pretty bad | 03:24 |
| *** makowals has quit IRC | 03:24 | |
| *** kfox1111 has quit IRC | 03:29 | |
| *** spzala has joined #openstack-infra | 03:32 | |
| ianw | EmilienM: is this timeouts? | 03:36 |
| *** spzala has quit IRC | 03:37 | |
| EmilienM | ianw: no more mirrors to try | 03:37 |
| EmilienM | ianw: mostly with RAX | 03:37 |
| EmilienM | ianw: let me show you an example | 03:37 |
| EmilienM | ianw: http://logs.openstack.org/74/489874/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/63397e3/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-08-07_00_08_57 | 03:37 |
| EmilienM | ianw: if you have any suggestion for us to do in tripleo ci, I'm all in | 03:38 |
| EmilienM | brb food | 03:38 |
| ianw | EmilienM: is this going via the reverse proxy? | 03:42 |
| *** skelso has quit IRC | 03:45 | |
| EmilienM | ianw: I think so | 03:45 |
| *** ramishra has joined #openstack-infra | 03:46 | |
| *** spzala has joined #openstack-infra | 03:49 | |
| *** links has joined #openstack-infra | 03:49 | |
| ianw | EmilienM: ok ... well ... | 03:50 |
| ianw | the mirror thinks it is "Mon Aug 7 03:49:52 UTC 2017" | 03:50 |
| ianw | oh, hang on, just getting all my stupid date stamps confused | 03:51 |
| *** nicolasbock has joined #openstack-infra | 03:51 | |
| ianw | this happened aug 7 00:08:57,775 | 03:51 |
| ianw | 66.187.229.248 - - [07/Aug/2017:00:08:57 +0000] "GET /buildlogs.centos/centos/7/cloud/x86_64/openstack-ocata/openstack-gnocchi-api-3.1.9-1.el7.noarch.rpm HTTP/1.1" 502 739 "-" "urlgrabber/3.10 yum/3.4.3" | 03:53 |
| ianw | ^ there it is | 03:53 |
| *** spzala has quit IRC | 03:54 | |
| ianw | root@mirror:/var/log/apache2# less mirror.regionone.tripleo-test-cloud-rh1.openstack.org_8080_access.log | grep 'urlgrabber/3.10' | wc -l | 03:56 |
| ianw | 60046 | 03:56 |
| ianw | root@mirror:/var/log/apache2# less mirror.regionone.tripleo-test-cloud-rh1.openstack.org_8080_access.log | grep 'urlgrabber/3.10' | grep '" 502 ' | wc -l | 03:56 |
| ianw | 30 | 03:56 |
| ianw | so 0.05% of requests are hitting a 502 error | 03:57 |
| *** jascott1 has joined #openstack-infra | 03:57 | |
| ianw | [Mon Aug 07 00:08:57.735660 2017] [proxy_http:error] [pid 14909:tid 139785716840192] (70014)End of file found: [client 66.187.229.248:34680] AH01102: error reading status line from remote server buildlogs.centos.org:443 | 03:58 |
| ianw | [Mon Aug 07 00:08:57.735732 2017] [proxy:error] [pid 14909:tid 139785716840192] [client 66.187.229.248:34680] AH00898: Error reading from remote server returned by /buildlogs.centos/centos/7/cloud/x86_64/openstack-ocata/openstack-gnocchi-api-3.1.9-1.el7.noarch.rpm | 03:58 |
| ianw | there's the error from error logs | 03:59 |
| *** veebers has quit IRC | 04:01 | |
| *** hongbin has quit IRC | 04:01 | |
| *** jascott1 has quit IRC | 04:02 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add proxy-initial-not-pooled to proxy https://review.openstack.org/491339 | 04:30 |
| *** spzala has joined #openstack-infra | 04:30 | |
| *** spzala has quit IRC | 04:35 | |
| EmilienM | ianw: thank you for the fix ^ | 04:35 |
| ianw | EmilienM: ^ well, that's my best guess | 04:36 |
| ianw | EmilienM: heh, i don't know about a fix, but given some investigation i think it's worth a try ... | 04:36 |
| EmilienM | ianw: let's see | 04:36 |
| EmilienM | ok | 04:36 |
| EmilienM | I'm out for sleeping now | 04:36 |
| ianw | EmilienM: night night :) | 04:36 |
| EmilienM | enjoy your day :) | 04:37 |
| *** PsionTheory has joined #openstack-infra | 04:37 | |
| openstackgerrit | Artur Basiak proposed openstack-infra/project-config master: Provide unified gate configuration https://review.openstack.org/490790 | 04:38 |
| *** david-lyle has quit IRC | 04:38 | |
| *** slaweq has joined #openstack-infra | 04:42 | |
| *** slaweq has quit IRC | 04:46 | |
| *** spzala has joined #openstack-infra | 05:09 | |
| AJaeger | infra-root, https://review.openstack.org/490986 merged to update lists.openstack.org - but I do not see the changes in the footer of http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev - is there a way to regenerate all these pages? | 05:12 |
| *** spzala has quit IRC | 05:14 | |
| *** thorst has joined #openstack-infra | 05:15 | |
| *** sree has joined #openstack-infra | 05:20 | |
| *** thorst has quit IRC | 05:20 | |
| *** pgadiya has joined #openstack-infra | 05:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Skip tempest job of Mogan if not related to code https://review.openstack.org/490724 | 05:23 |
| *** jaosorior has joined #openstack-infra | 05:24 | |
| *** zhurong has joined #openstack-infra | 05:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add dashboard to display openstack-ansible job runtimes and failures https://review.openstack.org/491247 | 05:28 |
| *** eranrom has quit IRC | 05:31 | |
| *** psachin has joined #openstack-infra | 05:36 | |
| ianw | AJaeger: hmm | 05:36 |
| ianw | the puppet has has "recurse=>true" which you'd think keeps in sync... | 05:38 |
| *** gildub has quit IRC | 05:45 | |
| *** xinliang has joined #openstack-infra | 05:47 | |
| *** xinliang has quit IRC | 05:47 | |
| *** xinliang has joined #openstack-infra | 05:47 | |
| ianw | except lists.o.o is in the disabled list | 05:49 |
| ianw | so, maybe jeblair knows why this is ^ | 05:49 |
| *** spzala has joined #openstack-infra | 05:50 | |
| *** spzala has quit IRC | 05:54 | |
| *** cshastri has joined #openstack-infra | 05:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix specs for https https://review.openstack.org/490996 | 06:03 |
| *** gildub has joined #openstack-infra | 06:03 | |
| *** spzala has joined #openstack-infra | 06:11 | |
| *** spzala has quit IRC | 06:16 | |
| *** gildub has quit IRC | 06:23 | |
| openstackgerrit | Artur Basiak proposed openstack-infra/project-config master: Provide unified gate configuration https://review.openstack.org/490790 | 06:25 |
| *** jtomasek has joined #openstack-infra | 06:26 | |
| *** rwsu has joined #openstack-infra | 06:30 | |
| *** rwsu has quit IRC | 06:30 | |
| *** rwsu has joined #openstack-infra | 06:36 | |
| *** kjackal_ has joined #openstack-infra | 06:44 | |
| *** spzala has joined #openstack-infra | 06:48 | |
| *** spzala has quit IRC | 06:53 | |
| *** gildub has joined #openstack-infra | 06:57 | |
| *** jtomasek has quit IRC | 07:01 | |
| *** kjackal_ has quit IRC | 07:01 | |
| *** stakeda has quit IRC | 07:06 | |
| *** nicolasbock_ has joined #openstack-infra | 07:06 | |
| *** shardy has joined #openstack-infra | 07:07 | |
| *** kjackal_ has joined #openstack-infra | 07:07 | |
| *** nicolasbock_ has quit IRC | 07:07 | |
| *** alexchadin has joined #openstack-infra | 07:07 | |
| *** rcernin has joined #openstack-infra | 07:08 | |
| *** spzala has joined #openstack-infra | 07:09 | |
| *** florianf has joined #openstack-infra | 07:09 | |
| openstackgerrit | Lingyong Xu proposed openstack/os-client-config master: Update the documentation link for doc migration https://review.openstack.org/491374 | 07:10 |
| *** spzala has quit IRC | 07:14 | |
| *** thorst has joined #openstack-infra | 07:16 | |
| *** markus_z has joined #openstack-infra | 07:18 | |
| *** tesseract has joined #openstack-infra | 07:20 | |
| *** yolanda has joined #openstack-infra | 07:20 | |
| *** thorst has quit IRC | 07:21 | |
| *** fabo_ is now known as fabo | 07:23 | |
| *** makowals has joined #openstack-infra | 07:25 | |
| *** mandre is now known as mandre_away | 07:27 | |
| *** priteau has joined #openstack-infra | 07:28 | |
| *** spzala has joined #openstack-infra | 07:34 | |
| *** jascott1 has joined #openstack-infra | 07:34 | |
| *** spzala_ has joined #openstack-infra | 07:35 | |
| *** spzala has quit IRC | 07:38 | |
| *** spzala_ has quit IRC | 07:39 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 07:39 | |
| *** eranrom has joined #openstack-infra | 07:45 | |
| *** gibi_chaosmonkey is now known as gibi | 07:46 | |
| *** rossella_s has joined #openstack-infra | 07:49 | |
| *** eranrom has quit IRC | 07:50 | |
| *** ccamacho has joined #openstack-infra | 07:58 | |
| *** alexchadin has quit IRC | 08:01 | |
| *** sree_ has joined #openstack-infra | 08:04 | |
| *** sree_ is now known as Guest66661 | 08:05 | |
| *** pblaho has joined #openstack-infra | 08:05 | |
| *** sree has quit IRC | 08:07 | |
| *** alexchadin has joined #openstack-infra | 08:07 | |
| *** jtomasek has joined #openstack-infra | 08:12 | |
| *** priteau has quit IRC | 08:14 | |
| *** jtomasek has quit IRC | 08:17 | |
| *** priteau has joined #openstack-infra | 08:17 | |
| *** markvoelker has joined #openstack-infra | 08:17 | |
| *** markvoelker_ has joined #openstack-infra | 08:19 | |
| *** aarefiev_afk is now known as aarefiev | 08:21 | |
| *** markvoelker has quit IRC | 08:23 | |
| *** alexchadin has quit IRC | 08:26 | |
| *** alexchadin has joined #openstack-infra | 08:28 | |
| *** baoli has quit IRC | 08:30 | |
| *** dingyichen has quit IRC | 08:30 | |
| *** s-shiono has quit IRC | 08:35 | |
| *** bhavik1 has joined #openstack-infra | 08:39 | |
| *** spzala has joined #openstack-infra | 08:46 | |
| *** spzala has quit IRC | 08:51 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Install prereqs for undercloud via tripleoclient https://review.openstack.org/473984 | 08:53 |
| *** alexchadin has quit IRC | 08:59 | |
| *** lrossetti_ has quit IRC | 09:06 | |
| *** e0ne has joined #openstack-infra | 09:12 | |
| *** slaweq has joined #openstack-infra | 09:16 | |
| *** thorst has joined #openstack-infra | 09:17 | |
| *** slaweq has quit IRC | 09:21 | |
| *** sambetts_ is now known as sambetts | 09:21 | |
| *** bhavik1 has quit IRC | 09:21 | |
| *** thorst has quit IRC | 09:22 | |
| *** spzala has joined #openstack-infra | 09:22 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Add Kolla Ansible Plugins project https://review.openstack.org/491412 | 09:23 |
| *** lennyb has quit IRC | 09:25 | |
| *** alexchadin has joined #openstack-infra | 09:26 | |
| *** spzala has quit IRC | 09:27 | |
| *** jaosorior has quit IRC | 09:27 | |
| *** arxcruz|sick is now known as arxcruz | 09:32 | |
| *** tuanluong has quit IRC | 09:32 | |
| *** cuongnv has quit IRC | 09:32 | |
| *** tuanluong has joined #openstack-infra | 09:33 | |
| *** cuongnv has joined #openstack-infra | 09:33 | |
| *** Hal has joined #openstack-infra | 09:34 | |
| *** Hal is now known as Guest58906 | 09:34 | |
| *** tosky has joined #openstack-infra | 09:35 | |
| *** cuongnv has quit IRC | 09:36 | |
| *** PsionTheory has quit IRC | 09:36 | |
| *** yamamoto_ has quit IRC | 09:44 | |
| *** jaosorior has joined #openstack-infra | 09:45 | |
| *** sdague has joined #openstack-infra | 09:47 | |
| *** dtantsur|afk is now known as dtantsur | 09:48 | |
| *** spzala has joined #openstack-infra | 09:49 | |
| *** spzala has quit IRC | 09:54 | |
| *** yamamoto has joined #openstack-infra | 10:02 | |
| *** gcb has quit IRC | 10:03 | |
| *** thorst has joined #openstack-infra | 10:05 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Add Kolla Ansible Plugins project https://review.openstack.org/491412 | 10:06 |
| *** thorst has quit IRC | 10:11 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/devstack-gate master: Do not set ephemeral size based on driver https://review.openstack.org/326061 | 10:13 |
| *** yamamoto has quit IRC | 10:16 | |
| *** slaweq has joined #openstack-infra | 10:17 | |
| *** liujiong has quit IRC | 10:18 | |
| *** slaweq has quit IRC | 10:21 | |
| *** raissa has joined #openstack-infra | 10:22 | |
| *** LindaWang has quit IRC | 10:23 | |
| *** spzala has joined #openstack-infra | 10:25 | |
| *** claudiub has joined #openstack-infra | 10:28 | |
| *** spzala has quit IRC | 10:30 | |
| *** tuanluong has quit IRC | 10:35 | |
| *** lennyb has joined #openstack-infra | 10:37 | |
| *** krtaylor has quit IRC | 10:41 | |
| *** dizquierdo has joined #openstack-infra | 10:47 | |
| *** markvoelker has joined #openstack-infra | 10:49 | |
| *** alexchadin has quit IRC | 10:49 | |
| *** spzala has joined #openstack-infra | 10:49 | |
| *** jkilpatr has quit IRC | 10:49 | |
| *** krtaylor has joined #openstack-infra | 10:51 | |
| *** Guest66661 has quit IRC | 10:52 | |
| *** markvoelker_ has quit IRC | 10:53 | |
| *** markvoelker_ has joined #openstack-infra | 10:54 | |
| *** markvoelker has quit IRC | 10:54 | |
| *** spzala has quit IRC | 10:54 | |
| *** markvoelker has joined #openstack-infra | 10:57 | |
| *** markvoelker has quit IRC | 10:59 | |
| *** jascott1 has quit IRC | 10:59 | |
| *** jascott1 has joined #openstack-infra | 11:00 | |
| *** makowals has quit IRC | 11:00 | |
| *** markvoelker_ has quit IRC | 11:00 | |
| *** markvoelker has joined #openstack-infra | 11:01 | |
| *** jascott1 has quit IRC | 11:04 | |
| *** strigazi_OFF is now known as strigazi | 11:05 | |
| *** slaweq has joined #openstack-infra | 11:06 | |
| *** jkilpatr has joined #openstack-infra | 11:07 | |
| *** dizquierdo has quit IRC | 11:08 | |
| *** lrossetti has joined #openstack-infra | 11:09 | |
| *** makowals has joined #openstack-infra | 11:10 | |
| *** slaweq has quit IRC | 11:10 | |
| *** markvoelker_ has joined #openstack-infra | 11:12 | |
| *** yamamoto has joined #openstack-infra | 11:13 | |
| *** pgadiya has quit IRC | 11:14 | |
| *** markvoelker has quit IRC | 11:15 | |
| *** dizquierdo has joined #openstack-infra | 11:22 | |
| *** jaypipes has joined #openstack-infra | 11:25 | |
| *** spzala has joined #openstack-infra | 11:26 | |
| strigazi | ianw AJaeger you there? | 11:26 |
| *** eharney has quit IRC | 11:27 | |
| *** pgadiya has joined #openstack-infra | 11:28 | |
| *** markvoelker_ has quit IRC | 11:28 | |
| *** thorst has joined #openstack-infra | 11:28 | |
| *** markvoelker has joined #openstack-infra | 11:29 | |
| *** spzala has quit IRC | 11:30 | |
| *** slagle has joined #openstack-infra | 11:31 | |
| *** thorst has quit IRC | 11:33 | |
| *** markvoelker has quit IRC | 11:34 | |
| strigazi | ianw AJaeger For magnum jobs, we pull qcow images from our account in fedorapeople, where we store stock fedora or centos images. Does it make sense to have a job to download stock images and upload them to tarballs.o.o ? | 11:36 |
| strigazi | Actually @all ^^ | 11:36 |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| *** markvoelker has joined #openstack-infra | 11:40 | |
| ianw | strigazi: it probably makes more sense for us to add it to the proxy mirror | 11:40 |
| ianw | see https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb | 11:41 |
| *** markvoelker_ has joined #openstack-infra | 11:41 | |
| ianw | recent tripleo changes touching that are a good example | 11:41 |
| ianw | of course, then the jobs need to use the proxy too | 11:41 |
| *** markvoelker_ has quit IRC | 11:42 | |
| *** markvoelker_ has joined #openstack-infra | 11:44 | |
| *** zhurong has quit IRC | 11:44 | |
| *** markvoelker has quit IRC | 11:44 | |
| strigazi | ianw like this one https://review.openstack.org/#/c/489462/ ? | 11:45 |
| *** markvoelker has joined #openstack-infra | 11:47 | |
| *** markvoelker_ has quit IRC | 11:50 | |
| *** markvoelker_ has joined #openstack-infra | 11:51 | |
| *** shardy is now known as shardy_lunch | 11:51 | |
| *** markvoelker has quit IRC | 11:52 | |
| *** markvoelker has joined #openstack-infra | 11:53 | |
| *** markvoelker_ has quit IRC | 11:56 | |
| *** markvoelker_ has joined #openstack-infra | 11:56 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add experimental opensuse-423 job for freezer https://review.openstack.org/491465 | 11:58 |
| *** markvoelker has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-infra | 12:02 | |
| *** markvoelker_ has quit IRC | 12:03 | |
| fungi | AJaeger: ianw: lists.o.o has been in the emergency disable list ever since the spam flood against -owner addresses began, to prevent us having to track the various disabled aliases in puppet | 12:04 |
| *** thorst has joined #openstack-infra | 12:05 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/system-config master: [magnum] Cache fedorapeople.org/groups/magnum https://review.openstack.org/491466 | 12:06 |
| *** slaweq has joined #openstack-infra | 12:07 | |
| strigazi | fungi Can you have a look ? ^^ | 12:09 |
| fungi | probably in an hour or so once i'm up and caffeinated | 12:09 |
| *** thorst has quit IRC | 12:09 | |
| *** thorst has joined #openstack-infra | 12:10 | |
| *** baoli has joined #openstack-infra | 12:11 | |
| *** slaweq has quit IRC | 12:11 | |
| strigazi | fungi sure xD That's my personal preference https://en.wikipedia.org/wiki/Ristretto | 12:13 |
| *** pgadiya has quit IRC | 12:15 | |
| fungi | oh, yeah those are definitely tasty | 12:16 |
| fungi | but i mostly just make a large cold-brew batch once a week, and keep it refrigerated to drink in the mornings. saves me some time | 12:17 |
| *** lewo has quit IRC | 12:18 | |
| *** lewo has joined #openstack-infra | 12:19 | |
| *** pgadiya has joined #openstack-infra | 12:19 | |
| *** jcoufal has joined #openstack-infra | 12:23 | |
| *** trown_pto is now known as trown | 12:31 | |
| *** pgadiya has quit IRC | 12:33 | |
| *** baoli has quit IRC | 12:34 | |
| *** rhallisey has joined #openstack-infra | 12:34 | |
| *** baoli has joined #openstack-infra | 12:34 | |
| *** gcb has joined #openstack-infra | 12:35 | |
| *** coolsvap has quit IRC | 12:36 | |
| openstackgerrit | James Slagle proposed openstack-infra/project-config master: Revert "Change job type for 3nodes job to move to oooq runner" https://review.openstack.org/491477 | 12:38 |
| *** pgadiya has joined #openstack-infra | 12:38 | |
| openstackgerrit | James Slagle proposed openstack-infra/project-config master: Revert "Change job type for 3nodes job to move to oooq runner" https://review.openstack.org/491477 | 12:38 |
| *** baoli has quit IRC | 12:39 | |
| *** pgadiya has quit IRC | 12:43 | |
| *** mikal has quit IRC | 12:44 | |
| *** pgadiya has joined #openstack-infra | 12:44 | |
| openstackgerrit | James Slagle proposed openstack-infra/project-config master: Revert "Change job type for 3nodes job to move to oooq runner" https://review.openstack.org/491477 | 12:47 |
| *** dave-mccowan has joined #openstack-infra | 12:49 | |
| *** shardy_lunch is now known as shardy | 12:52 | |
| *** jamesmcarthur has joined #openstack-infra | 12:56 | |
| *** spzala has joined #openstack-infra | 12:57 | |
| *** mikal has joined #openstack-infra | 13:00 | |
| *** kgiusti has joined #openstack-infra | 13:00 | |
| *** esberglu has joined #openstack-infra | 13:03 | |
| *** sree_ has joined #openstack-infra | 13:04 | |
| *** sree_ is now known as Guest15209 | 13:05 | |
| *** skelso has joined #openstack-infra | 13:07 | |
| *** slaweq has joined #openstack-infra | 13:08 | |
| *** sekelso has joined #openstack-infra | 13:09 | |
| *** skelso has quit IRC | 13:13 | |
| *** slaweq has quit IRC | 13:13 | |
| EmilienM | AJaeger: hi! https://review.openstack.org/#/c/491477/ ready | 13:13 |
| *** baoli has joined #openstack-infra | 13:15 | |
| *** baoli has quit IRC | 13:17 | |
| *** baoli has joined #openstack-infra | 13:19 | |
| *** makowals has quit IRC | 13:26 | |
| *** makowals has joined #openstack-infra | 13:29 | |
| *** pgadiya has quit IRC | 13:32 | |
| *** dizquierdo has quit IRC | 13:32 | |
| *** dizquierdo has joined #openstack-infra | 13:33 | |
| mnaser | is there a way that we can collect logs for timed out jobs somehow? | 13:36 |
| *** lbragstad has joined #openstack-infra | 13:36 | |
| fungi | mnaser: depends on the jobs. for any running under devstack-gate, definitely. it knows to anticipate the timeout, kill off remaining tests and collect logs before the job timeout is reached | 13:38 |
| mnaser | fungi hmm, i think i'll have to study a bit of the code that's used there. puppet-openstack-integration is timing out pretty often and i think there might be network issues causing it | 13:39 |
| fungi | mnaser: in a more general sense, maybe if someone puts together some parser rules so we can ingest zuul ansible logs, then we could find places where it fired a timeout on the job task | 13:39 |
| fungi | though that doesn't get the logs for the job payloads themseles | 13:40 |
| mnaser | it seems that specific part of the jobs take longer but oddly enough they always take longer in multiples of 2 (which feels like some 2 minute timeout is getting hit) | 13:40 |
| mnaser | either 4 minutes, 8 minutes, 10 minutes (almost exactly) | 13:40 |
| *** slagle has quit IRC | 13:40 | |
| fungi | so, yeah, sounds like what you're looking for is more the sort of thing devstack-gate does (one more reason to use devstack-gate instead of reinventing solutions to all the problems it already addresses) | 13:41 |
| mnaser | fungi i dont think puppet openstack integration can use devstack-gate, or can it? it's really meant to install openstack using puppet so devstack would kinda be doing what it has t odo | 13:42 |
| *** slagle has joined #openstack-infra | 13:42 | |
| fungi | devstack-gate is confusingly named. it doesn't have to use devstack at all | 13:42 |
| mnaser | oh | 13:43 |
| fungi | it's just a framework for setting up a job workspace including a list of git repositories you want checked out to the correct branches/refs, overlay network setup for multi-node jobs, soft timeout mechanism, log/artifact collection, and so on | 13:44 |
| fungi | things people using devstack in their jobs might want which devstack itself doesn't provide, so it got named "devstack-gate" (but if you don't use the default gate_hook you can override it with whatever you want instead of devstack) | 13:45 |
| mnaser | fungi i guess we can try adding experimental jobs using it and if it's adding value (giving us logs to check for timeouts) and working well, we can move them to replace the old ones? | 13:47 |
| *** links has quit IRC | 13:50 | |
| *** baoli has quit IRC | 13:55 | |
| *** baoli has joined #openstack-infra | 13:57 | |
| *** baoli has quit IRC | 13:57 | |
| *** asettle_ is now known as asettle | 13:59 | |
| *** felipemonteiro has joined #openstack-infra | 13:59 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:59 | |
| *** sekelso has quit IRC | 14:00 | |
| *** mriedem has joined #openstack-infra | 14:00 | |
| *** sekelso has joined #openstack-infra | 14:01 | |
| fungi | mnaser: that would make sense, you could probably get by with a very minimal d-g configuration which fires your old job as its gate_hook and then work slowly to replace bespoke bits in it with the features d-g is providing | 14:03 |
| *** felipemonteiro has quit IRC | 14:03 | |
| *** ihrachys has joined #openstack-infra | 14:04 | |
| *** jrist has quit IRC | 14:04 | |
| *** baoli has joined #openstack-infra | 14:05 | |
| *** marst has joined #openstack-infra | 14:09 | |
| *** slaweq has joined #openstack-infra | 14:10 | |
| *** slaweq has quit IRC | 14:15 | |
| *** jistr is now known as jistr|mtg | 14:16 | |
| *** skelso has joined #openstack-infra | 14:28 | |
| *** rbrndt has joined #openstack-infra | 14:28 | |
| *** sekelso has quit IRC | 14:30 | |
| *** skelso has quit IRC | 14:32 | |
| *** alexchadin has joined #openstack-infra | 14:35 | |
| *** baoli_ has joined #openstack-infra | 14:43 | |
| *** cshastri has quit IRC | 14:45 | |
| *** baoli has quit IRC | 14:45 | |
| *** gcb_ has joined #openstack-infra | 14:45 | |
| *** sileht has left #openstack-infra | 14:47 | |
| *** gcb has quit IRC | 14:47 | |
| *** psachin has quit IRC | 14:51 | |
| *** rbrndt has quit IRC | 14:52 | |
| *** jistr|mtg is now known as jistr | 14:52 | |
| *** gcb_ has quit IRC | 14:52 | |
| *** hongbin has joined #openstack-infra | 14:54 | |
| *** alexchadin has quit IRC | 14:56 | |
| *** xarses_ has joined #openstack-infra | 14:57 | |
| *** alexchadin has joined #openstack-infra | 14:59 | |
| *** yamahata has joined #openstack-infra | 15:00 | |
| *** Julien-zte has joined #openstack-infra | 15:01 | |
| *** Nil_ has quit IRC | 15:01 | |
| *** jascott1 has joined #openstack-infra | 15:02 | |
| *** markvoelker has quit IRC | 15:03 | |
| *** LindaWang has joined #openstack-infra | 15:04 | |
| *** david-lyle has joined #openstack-infra | 15:04 | |
| *** markvoelker has joined #openstack-infra | 15:06 | |
| *** makowals has quit IRC | 15:06 | |
| *** jascott1 has quit IRC | 15:06 | |
| *** PsionTheory has joined #openstack-infra | 15:08 | |
| *** rbrndt has joined #openstack-infra | 15:08 | |
| pabelanger | fungi: clarkb: ianw has a patch up to help out reverse proxy cache, we are seeing some 502 errors. Do either of you mind reviewing and +3? https://review.openstack.org/#/c/491339/ | 15:09 |
| *** makowals has joined #openstack-infra | 15:09 | |
| pabelanger | infra-root: I'd like to see about landing https://review.openstack.org/490576 this morning. It creates a trusted job in project-config to allow us to upload things to tarballs.o.o for zuulv3 | 15:10 |
| *** slaweq has joined #openstack-infra | 15:11 | |
| *** slaweq has quit IRC | 15:16 | |
| *** LindaWang has quit IRC | 15:18 | |
| *** pas-ha has joined #openstack-infra | 15:21 | |
| *** bswartz has quit IRC | 15:25 | |
| *** annegentle has joined #openstack-infra | 15:25 | |
| *** bswartz has joined #openstack-infra | 15:27 | |
| mordred | pabelanger: +A | 15:29 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Set success-url for tox-docs job https://review.openstack.org/491132 | 15:30 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use zuul.executor.work_root for add-build-sshkey https://review.openstack.org/490969 | 15:30 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove configure-mirrors from unittests/pre https://review.openstack.org/490977 | 15:30 |
| *** camunoz has joined #openstack-infra | 15:34 | |
| *** Julien-zte has quit IRC | 15:35 | |
| *** Julien-zte has joined #openstack-infra | 15:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/elastic-recheck master: Rework confusing job_re / job_regex config logic https://review.openstack.org/491521 | 15:36 |
| robcresswell | The oslo guys left their meeting open in meeting-3 btw; not sure who can fix that | 15:37 |
| *** markus_z has quit IRC | 15:37 | |
| *** beekneemech is now known as bnemec | 15:37 | |
| pabelanger | I can look | 15:38 |
| clarkb | robcresswell: pabelanger after an hour anyone can endmeeting | 15:38 |
| openstackgerrit | Merged openstack-infra/project-config master: Create publish-openstack-tarball job https://review.openstack.org/490576 | 15:39 |
| pabelanger | clarkb: robcresswell: looks like they tried: http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/latest.log.html | 15:39 |
| pabelanger | ah, IRC nick changed | 15:39 |
| robcresswell | Ah cool | 15:40 |
| robcresswell | yeah, awaynick strikes again. | 15:40 |
| * robcresswell wishes people would stop using that :) | 15:40 | |
| clarkb | anyone should be able to endmeeting it now | 15:40 |
| clarkb | because its been over an hour, but also if gcb has renicked and authed it should've worked as well | 15:40 |
| robcresswell | Yep, pabelanger got it. | 15:40 |
| clarkb | s/authed/identified/ | 15:41 |
| robcresswell | I think its because they started as gcb and then didnt go back to gcb, stayed as gcb_ | 15:41 |
| pabelanger | yup | 15:42 |
| pabelanger | jeblair: mordred: mind a review on https://review.openstack.org/490973 too? Move configure-mirror to base job (from base-test) using local site variables | 15:45 |
| *** spzala has quit IRC | 15:47 | |
| mordred | pabelanger: +A - awesome | 15:48 |
| openstackgerrit | Merged openstack-infra/system-config master: Add proxy-initial-not-pooled to proxy https://review.openstack.org/491339 | 15:49 |
| *** alexchadin has quit IRC | 15:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 15:51 |
| *** markvoelker has quit IRC | 15:53 | |
| *** aarefiev is now known as aarefiev_afk | 15:54 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix typo in post-run for publish-openstack-tarball https://review.openstack.org/491530 | 15:55 |
| pabelanger | mordred: jeblair:^ sorry, typo in project-config | 15:56 |
| pabelanger | maybe something we can have zuul check for, if it can access playbook on dynamic-reload? | 15:56 |
| *** ijw has joined #openstack-infra | 15:58 | |
| jeblair | pabelanger: theoretically doable, but that's a bit costly (it either doubles the number of round-trips from the merger, or we need to ask the merger for a list of all files) | 15:58 |
| *** ykarel has joined #openstack-infra | 15:59 | |
| jeblair | the list of all files is probably the least expensive way to implement it | 15:59 |
| jeblair | pabelanger: file a story? :) | 15:59 |
| *** yamamoto has quit IRC | 15:59 | |
| pabelanger | jeblair: sure | 15:59 |
| *** bobh has joined #openstack-infra | 16:00 | |
| *** Apoorva has joined #openstack-infra | 16:00 | |
| *** Apoorva has quit IRC | 16:01 | |
| *** iyamahat has joined #openstack-infra | 16:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Promote configure-mirror role to base job https://review.openstack.org/490973 | 16:04 |
| pabelanger | jeblair: https://storyboard.openstack.org/#!/story/2001143 | 16:06 |
| *** aarefiev_afk has quit IRC | 16:06 | |
| *** rlandy has joined #openstack-infra | 16:07 | |
| jeblair | pabelanger: thanks! | 16:08 |
| jeblair | pabelanger: the linters is a good idea worth exploring too | 16:08 |
| priteau | Is http://docs-draft.openstack.org unavailable? | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-elastic_recheck master: Add dsvm|tripleo to jobs_regex https://review.openstack.org/491535 | 16:10 |
| pabelanger | jeblair: cool, I'll see if I can find some time to work on it | 16:10 |
| *** slaweq has joined #openstack-infra | 16:12 | |
| fungi | priteau: i'm seeing some issues loading certain sites from static.o.o as well right now. trying to figure out why | 16:12 |
| *** Apoorva has joined #openstack-infra | 16:13 | |
| fungi | looks like we had an apache process segfault a few days ago, i wonder if that left some cached urls in a weird state | 16:13 |
| clarkb | the host itself seems happy. Don't see fs errors in dmesg and load is on the low end | 16:14 |
| jeblair | i'm going to install links so i can hit the status page | 16:14 |
| openstackgerrit | Merged openstack-infra/statusbot master: Implement #thanks command in statusbot https://review.openstack.org/473534 | 16:15 |
| fungi | jeblair: sounds good | 16:15 |
| fungi | there's a huge spike in its tcp connections graph | 16:16 |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack-infra/project-config master: Make grenade-linuxbridge-multinode job experimental https://review.openstack.org/490993 | 16:16 |
| jeblair | that request seems to be hanging like everything else | 16:16 |
| fungi | but not its tcp open stats, so probably an effect and not the cause | 16:16 |
| *** slaweq has quit IRC | 16:17 | |
| jeblair | oh, my status request has connected and sent a request; no response yet | 16:17 |
| *** e0ne has quit IRC | 16:17 | |
| *** Qiming has quit IRC | 16:18 | |
| fungi | seems like we had a similar situation a month or two ago which i attributed at the time to a number of very recent apache segfaults, and an apache restart cleared the behavior (though i think that may have been on files01.o.o not static.o.o) | 16:18 |
| jeblair | i ran lsof on some apache processes. they seem to only have /srv/static/tarballs/kolla/images/centos-source-registry-ocata.tar.gz open | 16:18 |
| jeblair | one process has 25 of those | 16:18 |
| fungi | any idea how long those have been going? | 16:19 |
| dtroyer | logs.o.o is also affected by this? | 16:19 |
| fungi | dtroyer: yes, same server | 16:19 |
| dtroyer | k, thanks | 16:19 |
| fungi | any site whose fqdn is a cname in dns for static.openstack.org is potentially impacted | 16:19 |
| *** Qiming has joined #openstack-infra | 16:20 | |
| fungi | i thought kolla was putting its ci use of those registry tarballs through provider-local proxies, so 35 for the same file seems excessive | 16:20 |
| fungi | unless it's outside consumers grabbing them | 16:20 |
| jeblair | fungi: it's 25 for every apache process i have checked (3 so far). i assume it's the whole server. | 16:21 |
| pabelanger | Yup, confirmed. logs.o.o is affected | 16:21 |
| fungi | yikes | 16:21 |
| jeblair | fungi: i'm trying to x-ref ip addrs from lsof | 16:21 |
| jeblair | the first i chose has no reverse dns | 16:21 |
| fungi | all the same filename or different files under that dir? | 16:21 |
| jeblair | all same | 16:21 |
| smcginnis | releases.o.o down? | 16:22 |
| fungi | let's switch that file to read-only and restart apache | 16:22 |
| pabelanger | smcginnis: yes, we are having issues with static.o.o atm | 16:22 |
| fungi | smcginnis: yes, for a minute or two longer probably | 16:22 |
| clarkb | mnaser: (don't want to distract from current issue too much but want to make sure you see this before I forget) the new css for the logo on gitweb seems to have forced gerrit to be extra wide and horizontal scroll. Its not world breaking but would be great if you could have a look when you have time | 16:22 |
| smcginnis | pabelanger, fungi: Thanks, I'll read scrollback and wait. | 16:22 |
| jeblair | fungi: you mean -r ? | 16:22 |
| jeblair | (ie, unreadable?) | 16:23 |
| fungi | jeblair: er, right i mean non-world-readable, sorry | 16:23 |
| clarkb | ya I think we want it to 404 | 16:23 |
| clarkb | ++ | 16:23 |
| jeblair | fungi: gimme just a sec before you do that | 16:23 |
| fungi | so either chmod o-r or mv to a slightly different name | 16:23 |
| fungi | holding | 16:23 |
| fungi | we can get ip addresses from the apache logs afterward i suppose | 16:24 |
| jeblair | fungi: ok go for it | 16:25 |
| pabelanger | ++ | 16:25 |
| jeblair | fungi: i wanted to grab all the lsof because of the way apache logs, we may miss some ip addrs | 16:25 |
| fungi | okay, it should be running again | 16:26 |
| fungi | just really odd that this wasn't reflected in a spike on http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all | 16:26 |
| *** jamesmcarthur has quit IRC | 16:27 | |
| fungi | which makes me suspect those were all old requests which were effectively hung | 16:27 |
| *** jamesmcarthur has joined #openstack-infra | 16:27 | |
| jeblair | fungi: 148 out of 152 connections were from 59.108.63.126 | 16:27 |
| *** vhosakot has joined #openstack-infra | 16:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo in post-run for publish-openstack-tarball https://review.openstack.org/491530 | 16:28 |
| fungi | random broadband address in china | 16:28 |
| fungi | "Beijing Founder Broadband Network Technology Co.,Ltd" | 16:28 |
| *** dizquierdo has quit IRC | 16:30 | |
| tosky | uhm, for no apparent reason 3 gate jobs failed for Sahara, with strange logs (truncated? Maybe timeout?) | 16:31 |
| tosky | is it something related to what you were talking earlier? | 16:31 |
| jeblair | the other 4 were also for kolla files. 2 look like infra-cloud addrs. | 16:31 |
| tosky | I see at least one with: | 16:32 |
| fungi | probably the chocolate and vanilla cache servers (i hope) | 16:32 |
| tosky | Connecting to tarballs.openstack.org (tarballs.openstack.org)|23.253.108.137|:80... failed: Connection refused. | 16:32 |
| fungi | tosky: tarballs.openstack.org was offline for a bit | 16:32 |
| tosky | then tries with IPv6, repest 6 times | 16:32 |
| tosky | ok, I will recheck :/ | 16:32 |
| fungi | jeblair: i can temporarily block 59.108.63.126 in iptables and then readd read access for that file i guess | 16:33 |
| jeblair | fungi: might be worth a try | 16:34 |
| fungi | i have a sinking suspicion someone tried to set up a kolla ci system or something behind that ip address | 16:34 |
| jeblair | server-status isn't working on that host :( | 16:35 |
| jeblair | oh hey | 16:35 |
| jeblair | docs.o.o/server-status works :) | 16:35 |
| jeblair | (the vhost situation there is ... complicated) | 16:35 |
| fungi | strange | 16:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 16:36 |
| jeblair | anyway, when on static.o.o this should work: links http://docs.openstack.org/server-status | 16:36 |
| clarkb | docs.o.o is on a different server, you mean docs-draft right? | 16:36 |
| sdague | docs draft, yes | 16:37 |
| sdague | that seems busted | 16:37 |
| jeblair | turns out, either, and they both allow public access. | 16:37 |
| jeblair | which is not the intent | 16:38 |
| fungi | #status log temporarily blocked 59.108.63.126 in iptables on static.openstack.org due to a denial of service condition involving tarballs.o.o/kolla/images/centos-source-registry-ocata.tar.gz | 16:38 |
| openstackstatus | fungi: finished logging | 16:38 |
| fungi | sdague: still busted? should have been fixed at least 13 minutes ago | 16:38 |
| *** _ari_ is now known as _ari_|lunch | 16:39 | |
| *** slaweq has joined #openstack-infra | 16:39 | |
| *** jamesmcarthur has quit IRC | 16:39 | |
| fungi | i've made that kolla tarball world-readable again | 16:40 |
| fungi | now that the offender is blocked | 16:40 |
| *** ijw has quit IRC | 16:40 | |
| *** rlandy has quit IRC | 16:41 | |
| *** openstackstatus has quit IRC | 16:42 | |
| fungi | jeblair: Location /server-status looks like it's set to Require local so i'm not sure why that's not being enforced :/ | 16:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix tarballs.o.o typo in openstack-tarball.yaml https://review.openstack.org/491545 | 16:43 |
| *** Julien-zte has quit IRC | 16:43 | |
| *** openstackstatus has joined #openstack-infra | 16:43 | |
| *** ChanServ sets mode: +v openstackstatus | 16:43 | |
| *** Julien-zte has joined #openstack-infra | 16:44 | |
| jeblair | fungi: yeah, i don't see what's wrong | 16:44 |
| *** Julien-zte has quit IRC | 16:44 | |
| jeblair | not just docs, governance.o.o allows it too | 16:44 |
| fungi | and we've enabled authz_host | 16:44 |
| fungi | well, docs is on files01 and governance is on static, so it seems to be endemic to our configuration in general yes | 16:45 |
| jeblair | could 'require all granted' be overriding that? | 16:46 |
| fungi | could it be a funky interaction with mod_proxy_http? | 16:46 |
| *** rossella_s has quit IRC | 16:46 | |
| fungi | hrm, maybe | 16:46 |
| mnaser | fungi clarkb i havent noticed that but htat is indeed annoying | 16:46 |
| mnaser | let me work something | 16:46 |
| mnaser | also oops random extra highlight there | 16:47 |
| *** spzala has joined #openstack-infra | 16:47 | |
| *** rossella_s has joined #openstack-infra | 16:48 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove query for bug 1697808 https://review.openstack.org/491118 | 16:49 |
| openstack | bug 1697808 in OpenStack-Gate "wget: unable to resolve host address 'tarballs.openstack.org'" [High,Fix released] https://launchpad.net/bugs/1697808 - Assigned to Clark Boylan (cboylan) | 16:49 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove query for bug 1643911 https://review.openstack.org/491122 | 16:49 |
| openstack | bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,In progress] https://launchpad.net/bugs/1643911 | 16:49 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove query for bug 1701411 https://review.openstack.org/491119 | 16:50 |
| openstack | bug 1701411 in devstack "devstack-version does not handle merge commit messages properly" [Critical,Fix released] https://launchpad.net/bugs/1701411 - Assigned to Manjeet Singh Bhatia (manjeet-s-bhatia) | 16:50 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove query for bug 1531187 https://review.openstack.org/491124 | 16:50 |
| openstack | bug 1531187 in OpenStack-Gate "subnode fails to be accessable over ssh" [High,Fix released] https://launchpad.net/bugs/1531187 - Assigned to Paul Belanger (pabelanger) | 16:50 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove query for bug 1284371 https://review.openstack.org/491126 | 16:50 |
| openstack | bug 1284371 in OpenStack-Gate "gate error: Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/*/slave.log (No such file or directory)" [High,Fix released] https://launchpad.net/bugs/1284371 - Assigned to James E. Blair (corvus) | 16:50 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove query for bug 1638982 https://review.openstack.org/491127 | 16:50 |
| openstack | bug 1638982 in libvirt (Ubuntu) "libvirtd randomly crashes on xenial nodes with "realloc(): invalid next size"" [Undecided,Confirmed] https://launchpad.net/bugs/1638982 | 16:50 |
| fungi | jeblair: i wonder if we need to wrap that in a <RequireAll> block? | 16:51 |
| *** spzala has quit IRC | 16:52 | |
| fungi | or <RequireAny> | 16:54 |
| fungi | i'm trying to wrap my head around the affect of those | 16:54 |
| fungi | i guess the fact that some require directives can return a state which is neither success nor failure makes the interaction between those a little muddy | 16:55 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Make neutron functional job non-voting https://review.openstack.org/491548 | 16:56 |
| fungi | presumably if we put "Require local" as the only directive in the block, than the impact of <RequireAll> vs <RequireAny> will be indistinguishable | 16:57 |
| *** dougwig has joined #openstack-infra | 16:57 | |
| *** pas-ha has left #openstack-infra | 16:57 | |
| *** yamamoto has joined #openstack-infra | 17:00 | |
| fungi | and <RequireAny> seems to be the implied default if not specified, but i can't find any confirmation as to whether other require directives get inherited/merged | 17:01 |
| fungi | aha, controlled by the AuthMerging setting | 17:02 |
| *** beisner has quit IRC | 17:02 | |
| *** beisner has joined #openstack-infra | 17:02 | |
| fungi | "When authorization is enabled, it is normally inherited by each subsequent configuration section, unless a different set of authorization directives is specified. This is the default action, which corresponds to an explicit setting of AuthMerging Off." | 17:02 |
| fungi | so it shouldn't be merging authorizations for Location / with those of Location /server-status based on my reading | 17:03 |
| *** shardy has quit IRC | 17:04 | |
| *** yamamoto has quit IRC | 17:04 | |
| fungi | there's also this security warning: "Exercise caution when setting authorization directives in Location sections that overlap with content served out of the filesystem. By default, these configuration sections overwrite authorization configuration in Directory, and Files sections." | 17:06 |
| fungi | which is what we want | 17:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix tarballs.o.o typo in openstack-tarball.yaml https://review.openstack.org/491545 | 17:07 |
| fungi | so i'm still not clear on why it's being allowed | 17:07 |
| *** ykarel has quit IRC | 17:10 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/elastic-recheck master: Rework confusing job_re / job_regex config logic https://review.openstack.org/491521 | 17:12 |
| *** tesseract has quit IRC | 17:12 | |
| *** dizquierdo has joined #openstack-infra | 17:13 | |
| EmilienM | pabelanger: another mirror issue http://logs.openstack.org/22/491522/1/check/gate-tripleo-quickstart-linters-ubuntu-xenial/04534a7/console.html#_2017-08-07_16_29_28_310535 | 17:13 |
| EmilienM | is this one known already? | 17:13 |
| pabelanger | EmilienM: yes: http://status.openstack.org/elastic-recheck/#1708707 | 17:14 |
| *** pcaruana has quit IRC | 17:15 | |
| EmilienM | ok | 17:15 |
| clarkb | interestingly I think taht error has happened both with the reverse proxy and with straight afs hosting | 17:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Replace all instances of tarball with tarballs https://review.openstack.org/491549 | 17:19 |
| *** Swami has joined #openstack-infra | 17:19 | |
| *** rcernin has quit IRC | 17:22 | |
| *** mriedem has quit IRC | 17:22 | |
| *** e0ne has joined #openstack-infra | 17:24 | |
| *** hongbin has quit IRC | 17:25 | |
| *** hongbin has joined #openstack-infra | 17:26 | |
| *** spzala has joined #openstack-infra | 17:26 | |
| fungi | jeblair: i think i have half of the mystery solved at least... the default vhost on static (ci.openstack.org) is unconditionally redirecting requests for "localhost" or ip addresses to docs.o.o/infra | 17:27 |
| fungi | so the redirect is kicking in before the handoff to location /server-status | 17:30 |
| *** tosky has quit IRC | 17:30 | |
| fungi | if we made the default vhost be one of the non-redirected sites that much would probably start working correctly | 17:31 |
| *** rossella_s has quit IRC | 17:32 | |
| clarkb | worth noting that we appear to be operating at nodepool capacity rightn ow and are mostly keeping up with demand | 17:32 |
| clarkb | (post queue is suffering, but others look happy) | 17:32 |
| *** Guest15209 has quit IRC | 17:33 | |
| pabelanger | Ya, OVH looks to be having issues atm too | 17:33 |
| pabelanger | ovh-bhs1 specifically | 17:34 |
| *** rossella_s has joined #openstack-infra | 17:35 | |
| *** rbrndt has quit IRC | 17:36 | |
| *** sambetts is now known as sambetts|afk | 17:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Replace all instances of tarball with tarballs https://review.openstack.org/491549 | 17:38 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add ER query for bug #1663529 https://review.openstack.org/473851 | 17:39 |
| openstack | bug 1663529 in tempest "test_server_basic_ops: mounting /dev/sr0 on /mnt failed: Device or resource busy " [High,Triaged] https://launchpad.net/bugs/1663529 | 17:39 |
| *** annegentle has quit IRC | 17:41 | |
| *** vhosakot has quit IRC | 17:42 | |
| *** vhosakot has joined #openstack-infra | 17:42 | |
| *** slaweq has quit IRC | 17:43 | |
| *** larsks has joined #openstack-infra | 17:44 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config master: Add support for post-processing stestr runs to run_tox.sh https://review.openstack.org/488212 | 17:45 |
| larsks | mordred: ping re: https://review.openstack.org/#/c/479192/, which is hitting me trying to work with gerrithub. Who else can +2 -infra reviews? | 17:45 |
| mtreinish | pabelanger: ^^^ when you get a sec, can you show me where to update that in the zuul v3 ansible stuff? | 17:45 |
| mtreinish | I think we should land both (I have patches up switching some things to stestr, although they won't likely be mergeable until after pike branches) | 17:46 |
| fungi | jeblair: etherpad.o.o works the way we want, so there's definitely something about the files01 and static server configs which is causing us to be unable to restrict it (i even went to hard as setting "require all denied" instead of "require local" for that location and could still access it from everywhere) | 17:46 |
| pabelanger | mtreinish: openstack-infra/zuul-jobs. We'll need to either update fetch-testr role or create fetch-stestr too | 17:47 |
| pabelanger | mtreinish: likely want to create stestr role | 17:47 |
| fungi | larsks: and that works with earlier gerrit versions just fine? | 17:47 |
| mtreinish | pabelanger: ok, I can do that. Although they're essentially the same thing, just with a 1 character difference :) | 17:48 |
| larsks | fungi: yes (at least, it works also w/ review.openstack.org, as does master) | 17:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use /srv not /src for publish-openstack-tarball root folder https://review.openstack.org/491557 | 17:48 |
| larsks | fungi: it's a super-minimal change that just calls str() on a value returned from gerrit. | 17:48 |
| fungi | larsks: oh, right, i initially read it the other way around | 17:49 |
| *** hongbin has quit IRC | 17:50 | |
| *** iyamahat has quit IRC | 17:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/elastic-recheck master: Switch to jobs_re for config setting https://review.openstack.org/491521 | 17:54 |
| *** markvoelker has joined #openstack-infra | 17:54 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/zuul-jobs master: Add fetch-stestr-output role https://review.openstack.org/491559 | 17:56 |
| mtreinish | pabelanger: ^^^ ok that should do it I think | 17:56 |
| *** _ari_|lunch is now known as _ari_ | 17:56 | |
| pabelanger | mtreinish: I'll look in a bit | 17:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-elastic_recheck master: Add dsvm|tripleo to jobs_regex https://review.openstack.org/491535 | 17:58 |
| *** gouthamr has joined #openstack-infra | 17:58 | |
| *** tosky has joined #openstack-infra | 18:00 | |
| openstackgerrit | Merged openstack-infra/git-review master: Fix listing changes over SSH for 2.14 https://review.openstack.org/479192 | 18:03 |
| *** slaweq has joined #openstack-infra | 18:04 | |
| fungi | clarkb: that foundation member id in invites list change you avoided approving is purposely blocked by a wip parent change anyway, so it won't merge until we're past the start of ptl election runoffs | 18:08 |
| clarkb | fungi: ah ok | 18:08 |
| *** slaweq has quit IRC | 18:09 | |
| *** yamahata has quit IRC | 18:09 | |
| fungi | and agreed on restarting gerrit after the change to the icla, though that should take effect immediately, better safe than sorry when it comes to breaking startup (though it's only removing things so really shouldn't cause problems) | 18:10 |
| *** baoli_ has quit IRC | 18:10 | |
| fungi | er, the parts in the gerrit config file which only get reread on restart are only removals, i should say | 18:10 |
| jeblair | mtreinish: if world domination with stestr is your goal, you may want to think about updating the cti | 18:11 |
| mtreinish | jeblair: that's on the list | 18:11 |
| jeblair | mtreinish: cool | 18:11 |
| mtreinish | I'm trying to just test it out in a few projects and ostestr first | 18:11 |
| pabelanger | ianw: clarkb: reading up on apache2 more, it seems we also might want to try: | 18:12 |
| pabelanger | https://serverfault.com/questions/206738/intermittent-error-when-using-mod-proxy-to-do-reverse-proxy-to-soap-service?answertab=votes#tab-top | 18:12 |
| mtreinish | and then start the inevitable bikeshed discussion on the ML and governance review to start the migration | 18:12 |
| *** dtantsur is now known as dtantsur|afk | 18:12 | |
| clarkb | pabelanger: we probably want to confirm the initial fix doesn't correct it before disabling keepalives | 18:13 |
| pabelanger | clarkb: ya, I am still seeing 502 from buildlogs | 18:13 |
| *** eranrom has joined #openstack-infra | 18:14 | |
| pabelanger | 137.74.31.237 - - [07/Aug/2017:18:09:58 +0000] "GET /buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/common/pysendfile-2.0.0-5.el7.x86_64.rpm HTTP/1.1" 502 729 "-" "urlgrabber/3.10 yum/3.4.3" | 18:14 |
| pabelanger | just happened | 18:14 |
| pabelanger | and apache was restarted an hour ago | 18:14 |
| clarkb | pabelanger: and the no pool change has gone in? | 18:14 |
| pabelanger | clarkb: ya | 18:14 |
| clarkb | pabelanger: and does it only affect buildlogs/centos ? | 18:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-gerrit master: Stop supporting the Contact Store feature https://review.openstack.org/491090 | 18:16 |
| pabelanger | clarkb: seems to be only buildlogs.centos | 18:17 |
| clarkb | pabelanger: any chance we can see their logs to see why its making our proxy unhappy? | 18:17 |
| pabelanger | clarkb: I can ask around | 18:17 |
| clarkb | however, if its only that location we might consider restricting these additional changes (and the original one) to just that backend | 18:17 |
| pabelanger | clarkb: right, was going to suggest that too | 18:18 |
| clarkb | since this set of changes will make a new tcp connection for every request aiui | 18:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Use /srv not /src for publish-openstack-tarball root folder https://review.openstack.org/491557 | 18:20 |
| *** baoli has joined #openstack-infra | 18:21 | |
| *** ijw has joined #openstack-infra | 18:22 | |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr master: Switch to stestr under the covers https://review.openstack.org/488441 | 18:24 |
| *** annegentle has joined #openstack-infra | 18:24 | |
| clarkb | pabelanger: looking at ovh bhs1 doesn't appear that we are successfully booting anything there right now | 18:24 |
| clarkb | OpenStackCloudException: ('Error in creating the server: Timed out waiting for a reply to message ID e998ed2a40fb4039bf584fa734791058' | 18:24 |
| pabelanger | clarkb: Ya, I haven't looked why | 18:25 |
| *** baoli has quit IRC | 18:25 | |
| clarkb | like their rabbitmq is sad | 18:25 |
| clarkb | nova is returning 500 error codes with that message | 18:25 |
| *** slaweq has joined #openstack-infra | 18:25 | |
| tobiash | clarkb: I have a fix for the gerrit topmenu width which is slightly broken since the icon change: https://review.openstack.org/#/c/490981/ | 18:25 |
| *** baoli has joined #openstack-infra | 18:25 | |
| *** iyamahat has joined #openstack-infra | 18:25 | |
| *** baoli has quit IRC | 18:25 | |
| clarkb | we should maybe consider setting max-servers to 0 temporarily (that way we stop trying to schedule nodes there possibly making it worse for them and tricking nodepool into thinking it can boot there) | 18:26 |
| clarkb | infra-root ^ | 18:26 |
| *** slaweq has quit IRC | 18:27 | |
| jeblair | clarkb: yeah, or maybe 1? | 18:27 |
| *** ijw has quit IRC | 18:27 | |
| jeblair | (so we can monitor?) | 18:27 |
| *** slaweq has joined #openstack-infra | 18:27 | |
| fungi | wonder if our account has been disabled for "nonpayment" there again? | 18:27 |
| clarkb | jeblair: ++ to 1 | 18:28 |
| *** baoli has joined #openstack-infra | 18:28 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove no_log from tarballs.o.o synchronize https://review.openstack.org/491569 | 18:28 |
| *** qba73 has joined #openstack-infra | 18:28 | |
| *** markvoelker has quit IRC | 18:28 | |
| clarkb | gra seems to be working (not sure if disabled accounts are per region) | 18:28 |
| fungi | i'm reading through the infra-root mailbox messages from them, but it's all in french and my comprehension of french is très terrible | 18:29 |
| clarkb | mnaser: see tobiash's mesage above. Also thank you tobiash I have +2'd | 18:29 |
| mnaser | clarkb +1 is all i can give but perfect | 18:30 |
| * clarkb makes a change to set max-servers to 1 | 18:30 | |
| mnaser | fungi i speak french fluently | 18:32 |
| mnaser | if you would like me to help | 18:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Set OVH BHS1 max-servers to 1 https://review.openstack.org/491571 | 18:32 |
| clarkb | jeblair: ^ there is that change | 18:33 |
| fungi | mnaser: appreciated, but they mailbomb us with messages for "obligation de paiement" e-mails several times a day which we've ignored for the past couple years, but i'm trying to wade through them to see if i spot one which mentions disconnection | 18:33 |
| *** annegentle has quit IRC | 18:34 | |
| mnaser | fungi gotcha, feel free to reach out | 18:34 |
| clarkb | fungi: let me know if you need jean daniel's email address | 18:34 |
| *** trown is now known as trown|lunch | 18:36 | |
| pabelanger | clarkb: fungi: jeblair: http://travaux.ovh.net/?do=details&id=26480 seems to be known openstack API issues at ovh | 18:36 |
| fungi | hah, okay it looks like this one from today which among (other things) says: | 18:37 |
| fungi | "De ce fait et conformément à nos termes contractuels, nous avons procédé à la suspension de la prestation sus-mentionnée et des services qui y seraient rattachés. Votre dossier est désormais transmis à notre service recouvrement. Toute nouvelle commande réalisée auprès de notre société sera subordonnée à la régularisation de votre situation." | 18:37 |
| fungi | my french is not great, but that's a suspension of service | 18:37 |
| pabelanger | yup | 18:37 |
| *** jamesmcarthur has joined #openstack-infra | 18:40 | |
| mnaser | and it looks like you're being sent to collections too :p | 18:41 |
| fungi | indeed | 18:41 |
| clarkb | yay | 18:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow and document use of the uri module from localhost https://review.openstack.org/490215 | 18:43 |
| *** yamahata has joined #openstack-infra | 18:44 | |
| *** jamesmcarthur has quit IRC | 18:44 | |
| clarkb | fungi: do you want me to update my change to be max-servers: 0 in light of ^ | 18:44 |
| *** mriedem has joined #openstack-infra | 18:45 | |
| fungi | i'm in the process of e-mailing jean-daniel now, but yeah in the interim we should probably put all of ovh to 0 | 18:45 |
| clarkb | well gra1 seems to be working, do you want me to set it to 0 too? | 18:45 |
| fungi | it's likely to stop working once it gets the same cancellation processed, i expect | 18:46 |
| *** slaweq has quit IRC | 18:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Set OVH max-servers to 0 https://review.openstack.org/491571 | 18:48 |
| clarkb | there | 18:48 |
| pabelanger | + | 18:50 |
| pabelanger | 2 | 18:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove no_log from tarballs.o.o synchronize https://review.openstack.org/491569 | 18:52 |
| *** kjackal_ has quit IRC | 18:54 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: enable show_related https://review.openstack.org/491097 | 18:55 |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr master: WIP: Switch to stestr under the covers https://review.openstack.org/488441 | 18:55 |
| *** nicolasbock has quit IRC | 18:57 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: add very simple architecture diagram https://review.openstack.org/491100 | 18:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 18:58 |
| clarkb | pabelanger: jeblair should I go ahead and approve that change? looks like was just race to +2 before seeing the other +2 is why you didn't approve? | 19:01 |
| pabelanger | clarkb: sure | 19:04 |
| clarkb | infra-root other changes that would be nice to have reviews on: https://review.openstack.org/#/c/490981/ to fix gerrit logo horizonal scrolling issue and https://review.openstack.org/#/c/490175/ to list docs 404s | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Write AFS keytab onto zuul v3 executor nodes https://review.openstack.org/488868 | 19:06 |
| mordred | clarkb: on it | 19:06 |
| mordred | clarkb, pabelanger, jeblair: ^^ that's rebased after pabelanger's recent changes landed | 19:06 |
| *** jamesmcarthur has joined #openstack-infra | 19:07 | |
| mordred | clarkb: both lgtm | 19:07 |
| *** slaweq has joined #openstack-infra | 19:08 | |
| *** armax has joined #openstack-infra | 19:08 | |
| *** jamesmcarthur has quit IRC | 19:11 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Allow and document use of the uri module from localhost https://review.openstack.org/490215 | 19:12 |
| *** slaweq has quit IRC | 19:13 | |
| *** jamesmcarthur has joined #openstack-infra | 19:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove verify_host from publish-openstack-tarball https://review.openstack.org/491577 | 19:13 |
| *** jtomasek has joined #openstack-infra | 19:14 | |
| *** baoli has quit IRC | 19:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set OVH max-servers to 0 https://review.openstack.org/491571 | 19:17 |
| *** rbrndt has joined #openstack-infra | 19:17 | |
| *** vhosakot has quit IRC | 19:17 | |
| *** vhosakot has joined #openstack-infra | 19:18 | |
| *** jtomasek has quit IRC | 19:18 | |
| *** slaweq has joined #openstack-infra | 19:18 | |
| openstackgerrit | Merged openstack-infra/system-config master: Generate list of 404s for docs.o.o https://review.openstack.org/490175 | 19:21 |
| openstackgerrit | Merged openstack-infra/system-config master: Adjust gerrit topmenu width to fit on screen https://review.openstack.org/490981 | 19:21 |
| *** jcoufal has quit IRC | 19:32 | |
| *** slaweq has quit IRC | 19:33 | |
| *** xyang1 has joined #openstack-infra | 19:33 | |
| *** camunoz has quit IRC | 19:34 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 19:36 |
| *** vhosakot_ has joined #openstack-infra | 19:36 | |
| *** vhosakot has quit IRC | 19:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove verify_host from publish-openstack-tarball https://review.openstack.org/491577 | 19:39 |
| *** qba73 has quit IRC | 19:40 | |
| *** e0ne has quit IRC | 19:43 | |
| *** priteau has quit IRC | 19:43 | |
| *** priteau has joined #openstack-infra | 19:44 | |
| *** trown|lunch is now known as trown | 19:44 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Log file stats for tox tarball / wheels https://review.openstack.org/490045 | 19:46 |
| *** dizquierdo has quit IRC | 19:47 | |
| clarkb | pabelanger: EmilienM fwiw I'm pretty sure that the main categorization that e-r does is done globally on all voting jobs. This means you don't need to wait for anything to have your queries added to it (also could have added them from the start). The main thing being restriced is the live lookups and gerrit commenting | 19:47 |
| *** priteau has quit IRC | 19:48 | |
| clarkb | I'm going to manually run the docs 404 generation command after lunch to ensure it works before it runs automatically at 0700UTC when most of us are not around | 19:49 |
| *** florianf has quit IRC | 19:49 | |
| *** dizquierdo has joined #openstack-infra | 19:51 | |
| *** annegentle has joined #openstack-infra | 19:52 | |
| *** jamesmcarthur has quit IRC | 19:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 19:53 |
| *** slaweq has joined #openstack-infra | 19:54 | |
| *** camunoz has joined #openstack-infra | 19:55 | |
| clarkb | fungi: I shut down the screen on static.o.o for listing job log fs disk consumption. The results are still on disk and I pasted them to paste.o.o yesterday if you are interested in them | 19:55 |
| fungi | clarkb: saw, thanks! | 19:55 |
| *** slaweq has quit IRC | 19:59 | |
| *** baoli has joined #openstack-infra | 19:59 | |
| *** dizquierdo has quit IRC | 20:00 | |
| *** thorst has quit IRC | 20:00 | |
| *** thorst has joined #openstack-infra | 20:03 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Remove verify_host from publish-openstack-tarball" https://review.openstack.org/491590 | 20:04 |
| *** thorst has quit IRC | 20:07 | |
| *** spzala has quit IRC | 20:07 | |
| *** jamesmcarthur has joined #openstack-infra | 20:11 | |
| *** jamesmcarthur has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #openstack-infra | 20:12 | |
| *** slaweq has joined #openstack-infra | 20:14 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Fix source of 404s.sh script https://review.openstack.org/491593 | 20:15 |
| clarkb | ^^ is a fix for a derp on puppetry | 20:16 |
| *** gouthamr has quit IRC | 20:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 20:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Set basepython to python3 https://review.openstack.org/491594 | 20:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Setup ANSIBLE_ROLES_PATH in tox.ini https://review.openstack.org/491595 | 20:18 |
| *** thorst has joined #openstack-infra | 20:18 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** slaweq has quit IRC | 20:19 | |
| *** vhosakot_ has quit IRC | 20:22 | |
| *** vhosakot has joined #openstack-infra | 20:23 | |
| *** kjackal_ has joined #openstack-infra | 20:23 | |
| weshay_ | pabelanger++ thanks for the recheck | 20:23 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Switch to jobs_re for config setting https://review.openstack.org/491521 | 20:24 |
| *** gouthamr has joined #openstack-infra | 20:24 | |
| pabelanger | clarkb: fungi: mtreinish: mind reviewing puppet change for elastic-recheck too: https://review.openstack.org/491535 | 20:26 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/zuul-jobs master: Add fetch-stestr-output role https://review.openstack.org/491559 | 20:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Remove verify_host from publish-openstack-tarball" https://review.openstack.org/491590 | 20:29 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix source of 404s.sh script https://review.openstack.org/491593 | 20:35 |
| *** slaweq has joined #openstack-infra | 20:35 | |
| *** gouthamr has quit IRC | 20:36 | |
| *** tnovacik has joined #openstack-infra | 20:38 | |
| kevinbenton | hi, can I get some eyes on https://review.openstack.org/#/c/491548/ to unblock the Neutron gate while we figure out what went wrong? | 20:38 |
| *** slaweq has quit IRC | 20:39 | |
| clarkb | kevinbenton: ya looking | 20:40 |
| clarkb | pabelanger: https://review.openstack.org/#/q/topic:thanksbot+status:open is a couple changes to enable the thanks feature in status bot (I think fungi is hoping a non foundation staff member can look at it for reviewer diversity) | 20:40 |
| clarkb | (I've already reviewed them) | 20:40 |
| *** slaweq has joined #openstack-infra | 20:41 | |
| kevinbenton | clarkb: thanks | 20:42 |
| *** jkilpatr has quit IRC | 20:43 | |
| clarkb | kevinbenton: I'm happy to help debug things if infra can help | 20:44 |
| clarkb | kevinbenton: not sure where you are in your investigation but do let us know if we can help | 20:44 |
| kevinbenton | clarkb: thanks. I think the latest was that a test worker was being unexpectedly killed but the source of the SIGKILL was unclear. | 20:46 |
| clarkb | and then the entire job eventually times out ? I think I looked at it briefly last week | 20:47 |
| kevinbenton | i'm going to have a look this evening after the Neutron meeting and then jlibovsa will pick it up again in the morning Europe time | 20:47 |
| kevinbenton | clarkb: i'm not sure. ihrachys just briefly told me about it this morning and I haven't had a chance to dig into it quite yet | 20:47 |
| ihrachys | kevinbenton, actually, when we talked, I was not yet aware of that other issue that currently keeps the gate at 100% | 20:48 |
| ihrachys | I was talking about timeouts that are more like 30% | 20:48 |
| kevinbenton | ihrachys: oooh | 20:49 |
| kevinbenton | so there are two problems then | 20:49 |
| kevinbenton | fun :) | 20:49 |
| ihrachys | yep | 20:49 |
| *** jamesmcarthur has joined #openstack-infra | 20:50 | |
| ihrachys | later I learned about the 2nd and mentioned it in the infra patch alongside the first one | 20:50 |
| *** tnovacik has quit IRC | 20:51 | |
| *** jamesmca_ has joined #openstack-infra | 20:51 | |
| *** jamesmca_ has quit IRC | 20:51 | |
| *** jamesmca_ has joined #openstack-infra | 20:51 | |
| *** jamesmcarthur has quit IRC | 20:52 | |
| *** mriedem has quit IRC | 20:56 | |
| *** hichihara has joined #openstack-infra | 20:59 | |
| *** annegentle has quit IRC | 20:59 | |
| *** jkilpatr has joined #openstack-infra | 21:01 | |
| *** ihrachys has quit IRC | 21:02 | |
| *** trown is now known as trown|outtypewww | 21:05 | |
| clarkb | sdague: dhellmann http://files.openstack.org/docs-404s/2017-08-07.txt is up now. I also responded to the ml thread so everyone following there should see it | 21:05 |
| sdague | clarkb: thank you | 21:07 |
| *** spzala has joined #openstack-infra | 21:08 | |
| sdague | so the stuff that's /trunk is going to need to be put in openstack-manuals for redirects, right? | 21:09 |
| sdague | or somewhere else? | 21:09 |
| sdague | because redirects we put in nova tree only reflect things that hit /nova/latest.... right? | 21:09 |
| clarkb | ya, I think some of those may need to go into the roote level apache config (or possibly an htpasword there) | 21:10 |
| clarkb | I think AJaeger and jeblair likely have the most info paged in for how that all looks top to bottom and would be good to see if they have opinions on how to address those | 21:11 |
| sdague | ok, cool, just making sure I understood what was changable in our tree | 21:11 |
| clarkb | sdague: your htpaasswd file can have rules for whatever urls but you'll only ever hit that htpasswd if rooted under /nova | 21:12 |
| *** spzala has quit IRC | 21:12 | |
| sdague | clarkb: I'm going to ask a more complicated request... and you can say no, and I won't be offended | 21:12 |
| clarkb | I guess /nova/latest or /nova/$branch since its a file per publishing branch | 21:12 |
| sdague | is the user agent string in there so we could bin out crawl bots? | 21:12 |
| clarkb | let me check | 21:13 |
| sdague | which we should still fix, but are lower priority than humans | 21:13 |
| clarkb | also that reminds me did the robots.txt get updated | 21:13 |
| clarkb | ya I see a bunch of mozilla 5.0s | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Revert "Remove verify_host from publish-openstack-tarball"" https://review.openstack.org/491608 | 21:15 |
| clarkb | looks like robots didn't need updating beacuse files.o.o is already disallow / in robots.txt | 21:15 |
| clarkb | yay for having this utility hostname | 21:15 |
| sdague | oh, we should allow robots in at some point | 21:15 |
| sdague | so we get better indexing | 21:15 |
| clarkb | sdague: we do for docs.o.o but not for files.o.o | 21:15 |
| clarkb | files.o.o is like the backdoor utility info for docs.o.o and whatever else we host there | 21:15 |
| sdague | ok, maybe we are taking about different things? | 21:16 |
| clarkb | sdague: ya your think made me think of something else. Sorry for the confusion | 21:16 |
| clarkb | sdague: for your thing would it be suifficient to grep Mozilla lines out and only process those? | 21:16 |
| sdague | my question was, for the 404s do we know which docs.o.o come from bots and which from users? | 21:16 |
| *** jamesmca_ has quit IRC | 21:16 | |
| *** priteau has joined #openstack-infra | 21:16 | |
| sdague | clarkb: yeh, or make 2 lists. We still want to fix them all, but it would be useful to know which ones are breaking real people | 21:17 |
| sdague | anyway, need to drop for the day, if it's easy, it would be cool | 21:17 |
| *** jamesmcarthur has joined #openstack-infra | 21:17 | |
| *** jascott1 has joined #openstack-infra | 21:17 | |
| clarkb | looks like some of the web crawlers use mozilla 5.0 too | 21:17 |
| *** jascott1 has quit IRC | 21:17 | |
| clarkb | sdague: but ya can totally filter that way, just need to come up with a good rule/regex for is a bot vs is not a bot | 21:17 |
| *** jascott1 has joined #openstack-infra | 21:18 | |
| *** slaweq has quit IRC | 21:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Require a base job https://review.openstack.org/491610 | 21:20 |
| *** yamamoto has joined #openstack-infra | 21:20 | |
| clarkb | sdague: doing some quick reading the rules people use are straightforward but large https://www.sumologic.com/apache/identifying-robots-with-apache-logs/ | 21:21 |
| fungi | would be trivial for us to dump the list of user agent strings we see and let people sort those into probably-human/probably-terminator buckets and then we could split the 404 listings filtering on those two sets | 21:23 |
| clarkb | sdague: dhellmann for /developer/\(.*\)/\(.*\) I think we have a redirect to /\1/\2? I guess we have to handle latest/ | 21:23 |
| *** yamamoto has quit IRC | 21:25 | |
| *** markvoelker has joined #openstack-infra | 21:26 | |
| fungi | infra-root: i'm going to make the by-hand configuration adjustment required by https://review.openstack.org/491077 in a moment so i can un-wip that change. would be good to have it merged shortly thereafter and then i'll approve https://review.openstack.org/491130 which depends on it as well so i can announce the behavior change to the -dev ml | 21:27 |
| * clarkb realizes he had english sentence punctuation adjacent to a regex which liekly made that harder to read | 21:29 | |
| *** annegentle has joined #openstack-infra | 21:30 | |
| *** kgiusti has left #openstack-infra | 21:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 21:33 |
| *** hichihara has quit IRC | 21:34 | |
| jeblair | fungi: you have +2s; i suggest you +W it yourself when ready | 21:36 |
| ianw | pabelanger: yeah, i wasn't 100% sure on those other settings with apache 2.4? | 21:36 |
| fungi | thanks jeblair! | 21:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 21:38 |
| *** spzala has joined #openstack-infra | 21:38 | |
| *** slaweq has joined #openstack-infra | 21:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 21:39 |
| pabelanger | ianw: Ya, I think we can try the others. However, buildlogs.centos.org is under a DDOS ATM | 21:40 |
| *** spzala has quit IRC | 21:43 | |
| *** slaweq has quit IRC | 21:44 | |
| *** priteau has quit IRC | 21:44 | |
| *** jamesmcarthur has quit IRC | 21:44 | |
| *** jascott1_ has joined #openstack-infra | 21:47 | |
| *** jascott1 has quit IRC | 21:47 | |
| mnaser | sigh | 21:47 |
| mnaser | why would someone ddos that | 21:47 |
| fungi | might not be intentional | 21:48 |
| fungi | maybe someone simply pointed a ci system at it | 21:48 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Remove verify_host from publish-openstack-tarball"" https://review.openstack.org/491608 | 21:49 |
| fungi | i'm fondly recalling the time we ddos'd git.openstack.org by failing to cache the nova repo on our images due to some logic error, so every devstack-based job performed a full clone of it | 21:50 |
| mnaser | ouch | 21:50 |
| smcginnis | oops | 21:50 |
| mnaser | i think i might be wrong but if i remember correctly buildlogs.centos.org was behind a cdn | 21:50 |
| fungi | as soon as activity ramped up that day, blam | 21:50 |
| fungi | hundreds of machines around the internet doing `git clone .../nova` from the git server farm brought it down hard | 21:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Require a base job https://review.openstack.org/491610 | 21:52 |
| rm_work | for os-loganalyze patch https://review.openstack.org/#/c/490121/ do I need something else or is this acceptable? | 21:54 |
| *** slaweq has joined #openstack-infra | 21:55 | |
| clarkb | rm_work: if you have an example logs dir we can look at really quickly that would be great | 21:55 |
| rm_work | clarkb: like this would do? http://logs.openstack.org/36/491136/2/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/70ef306/logs/ | 21:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add zuul-sphinx as a requirement https://review.openstack.org/491129 | 21:56 |
| rm_work | all of the screen-o-* is there | 21:56 |
| *** dizquierdo has joined #openstack-infra | 21:56 | |
| *** yamamoto has joined #openstack-infra | 21:56 | |
| *** markvoelker has quit IRC | 22:00 | |
| *** xyang1 has quit IRC | 22:00 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Reduce yum-config-manager output https://review.openstack.org/491076 | 22:01 |
| jeblair | zuul meeting time in #openstack-meeting-alt | 22:01 |
| *** ccamacho has quit IRC | 22:01 | |
| *** bobh has quit IRC | 22:01 | |
| *** slaweq has quit IRC | 22:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 22:01 |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr master: WIP: Switch to stestr under the covers https://review.openstack.org/488441 | 22:03 |
| clarkb | rm_work: yup thats it thanks | 22:06 |
| rm_work | no, thank *you* :P | 22:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python-tarball job https://review.openstack.org/491093 | 22:08 |
| *** esberglu has quit IRC | 22:09 | |
| openstackgerrit | Merged openstack-infra/os-loganalyze master: Add Octavia logs to filter regex https://review.openstack.org/490121 | 22:09 |
| clarkb | rm_work: that will get deployed by puppet so you won't see it right away but within an hour it should be in place | 22:10 |
| *** esberglu has joined #openstack-infra | 22:10 | |
| rm_work | ok cool, thanks | 22:10 |
| openstackgerrit | Merged openstack-infra/system-config master: Stop using the Gerrit Contact Store https://review.openstack.org/491077 | 22:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create publish-openstack-python(-branch)-tarball jobs https://review.openstack.org/491093 | 22:13 |
| tonyb | Hmm okay based on the traceback I *think* it was a 404 grabbing the projects_list as we're both passed that I think we'll be okay | 22:14 |
| *** slaweq has joined #openstack-infra | 22:14 | |
| tonyb | *sigh* wrong window :( | 22:14 |
| *** esberglu has quit IRC | 22:14 | |
| *** Apoorva_ has joined #openstack-infra | 22:15 | |
| *** gtrxcb has joined #openstack-infra | 22:15 | |
| *** Apoorva has quit IRC | 22:18 | |
| *** slaweq has quit IRC | 22:19 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul-sphinx as a requirement https://review.openstack.org/491129 | 22:20 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Add firehose schema docs for logstash workers https://review.openstack.org/491615 | 22:20 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Contactectomy the account setup instructions https://review.openstack.org/491130 | 22:20 |
| mtreinish | clarkb, fungi: ^^^ that should be a straightforward review | 22:21 |
| *** kjackal_ has quit IRC | 22:22 | |
| clarkb | mtreinish: +2. Also did you see I had to remove the event per log message as it was making everything slow? | 22:22 |
| clarkb | so we get events per processed log file now | 22:23 |
| mtreinish | yeah, I saw that, I was wondering where all the events went this morning and saw the commit | 22:24 |
| mtreinish | I reflected that in the schema doc, because there is only a retrieve_logs action now | 22:24 |
| *** Apoorva_ has quit IRC | 22:24 | |
| *** ijw has joined #openstack-infra | 22:25 | |
| *** Apoorva has joined #openstack-infra | 22:25 | |
| *** esberglu has joined #openstack-infra | 22:25 | |
| pabelanger | ianw: do you mind reviewing and +3 https://review.openstack.org/491535 changes elastic-recheck to add tripleo jobs too | 22:27 |
| *** gouthamr has joined #openstack-infra | 22:28 | |
| *** yamahata has quit IRC | 22:29 | |
| *** jamesmcarthur has joined #openstack-infra | 22:30 | |
| *** slaweq has joined #openstack-infra | 22:30 | |
| *** dizquierdo has quit IRC | 22:30 | |
| *** thorst has quit IRC | 22:32 | |
| *** markvoelker has joined #openstack-infra | 22:32 | |
| *** yamahata has joined #openstack-infra | 22:34 | |
| *** sree has joined #openstack-infra | 22:34 | |
| *** jamesmcarthur has quit IRC | 22:34 | |
| *** hongbin has joined #openstack-infra | 22:35 | |
| *** marst has quit IRC | 22:36 | |
| *** slaweq has quit IRC | 22:36 | |
| *** jamesmcarthur has joined #openstack-infra | 22:37 | |
| *** sree has quit IRC | 22:38 | |
| openstackgerrit | Joshua Harlow proposed openstack/os-client-config master: Update globals safely https://review.openstack.org/491618 | 22:38 |
| dhellmann | clarkb : I missed most of whatever discussion there was about redirects, but when we render the .htaccess file we know what all of the series names are and which corresponds to latest so we can have conditionals in the template if we need them | 22:41 |
| dhellmann | if we have some specific needs, it would be good to put those on the ML so we can try to find someone to help with patches | 22:41 |
| clarkb | dhellmann: mostly just thinking about all the /developer 404s | 22:43 |
| clarkb | dhellmann: since those should be an easy redirect to /$project/$branch right? | 22:43 |
| openstackgerrit | Joshua Harlow proposed openstack/os-client-config master: Update globals safely https://review.openstack.org/491618 | 22:44 |
| *** ijw has quit IRC | 22:46 | |
| *** Guest58906 has quit IRC | 22:46 | |
| *** Hal has joined #openstack-infra | 22:46 | |
| openstackgerrit | Joshua Harlow proposed openstack/os-client-config master: Update globals safely https://review.openstack.org/491618 | 22:47 |
| *** Hal is now known as Guest62535 | 22:47 | |
| dhellmann | clarkb : I should look, because we should already have basic redirects in place for those | 22:47 |
| dhellmann | where's that list of 404s published again? | 22:47 |
| clarkb | dhellmann: http://files.openstack.org/docs-404s/2017-08-07.txt | 22:47 |
| clarkb | that dir will get daily updates going forward too | 22:47 |
| dhellmann | fuel and gnocchi aren't included in the list of known projects on docs.o.o so that's why they don't have redirects | 22:48 |
| dhellmann | some of those others have series names in the URL, and I don't know if we were trying to handle those with the rules | 22:49 |
| dhellmann | we do, but openstack-ansible-os_barbican isn't listed in the project list | 22:49 |
| *** slaweq has joined #openstack-infra | 22:49 | |
| *** spzala has joined #openstack-infra | 22:50 | |
| dhellmann | a lot of these would be fixed if the projects were added to openstack-manuals/www/project-data/latest.yaml | 22:50 |
| dhellmann | we may want to create a new sub-type for "a real thing that should have a redirect but not be listed on the project list page" though because otherwise all of those ansible and puppet repos are going to make that page useless | 22:51 |
| *** Guest62535 has quit IRC | 22:52 | |
| clarkb | you just need the htpassword file to update right? so you could have it write to that and not your index | 22:52 |
| dhellmann | fwiw, we made the choice to only produce redirects for known things, rather than using a more general pattern, to encourage project teams to list their components | 22:53 |
| *** markvoelker has quit IRC | 22:54 | |
| *** slaweq has quit IRC | 22:54 | |
| *** markvoelker has joined #openstack-infra | 22:54 | |
| dhellmann | clarkb : this list of 404s is fascinating :-) | 22:54 |
| dhellmann | so many random things in here | 22:54 |
| clarkb | looks like we do still host fuel-docs but not gnocci so we probably want to address one of the two in that list | 22:56 |
| *** PsionTheory has quit IRC | 22:58 | |
| *** markvoelker has quit IRC | 22:59 | |
| *** slaweq has joined #openstack-infra | 23:02 | |
| *** rbrndt has quit IRC | 23:03 | |
| *** rbrndt has joined #openstack-infra | 23:03 | |
| *** rbrndt has quit IRC | 23:03 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Make python3 dsvm job as voting https://review.openstack.org/491623 | 23:04 |
| *** xarses_ has quit IRC | 23:06 | |
| *** pbourke has quit IRC | 23:06 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Move multinode job to experimental https://review.openstack.org/491624 | 23:07 |
| *** pbourke has joined #openstack-infra | 23:08 | |
| *** vhosakot has quit IRC | 23:08 | |
| *** slaweq has quit IRC | 23:08 | |
| *** annegentle has quit IRC | 23:09 | |
| *** jamesmcarthur has quit IRC | 23:11 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Kuryr] Turn python3 job to voting https://review.openstack.org/491627 | 23:12 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Fuxi] Turn python3 job to voting https://review.openstack.org/491628 | 23:14 |
| *** jaypipes has quit IRC | 23:19 | |
| *** sdague has quit IRC | 23:21 | |
| *** tosky has quit IRC | 23:23 | |
| *** slaweq has joined #openstack-infra | 23:24 | |
| *** dizquierdo has joined #openstack-infra | 23:24 | |
| *** jamesmcarthur has joined #openstack-infra | 23:24 | |
| *** dizquierdo has quit IRC | 23:29 | |
| *** slaweq has quit IRC | 23:30 | |
| fungi | infra-root: updated gerrit config from 491077 looks correctly applied on review.o.o, so in ~30 minutes i'll proceed with a status notice and gerrit restart to make sure it comes back up properly with the new configuration | 23:31 |
| *** rhallisey has quit IRC | 23:31 | |
| fungi | #status notice The Gerrit service on review.openstack.org will be offline momentarily at 00:00 utc for a quick reconfiguration-related restart | 23:31 |
| openstackstatus | fungi: sending notice | 23:31 |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck master: Add dsvm|tripleo to jobs_regex https://review.openstack.org/491535 | 23:32 |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org will be offline momentarily at 00:00 utc for a quick reconfiguration-related restart | 23:33 | |
| *** Hal has joined #openstack-infra | 23:34 | |
| *** Hal is now known as Guest93343 | 23:34 | |
| openstackstatus | fungi: finished sending notice | 23:34 |
| clarkb | fungi: \o/ | 23:35 |
| pabelanger | clarkb: ianw: looks like centos 7.4 will be landing shortly, maybe this week. So, might be some fun for us coming | 23:35 |
| ianw | fungi: so ... going back 12 hours or so ... what's the end-game for puppet+lists.o.o ? | 23:36 |
| fungi | ianw: good question, i was hoping the spam flood might have subsided by now | 23:36 |
| fungi | however it only seems to be escalating. we rejected nearly 10k messages to -owner addresses yesterday | 23:37 |
| clarkb | pabelanger: newer libvirt will cause problems with stable branches and our libvirt-python constraint | 23:37 |
| dmsimard | ianw: are the f26 jobs available/stable yet ? | 23:38 |
| ianw | dmsimard: yes, nodes are available | 23:38 |
| pabelanger | clarkb: Ya, just thinking of that too | 23:38 |
| dmsimard | ianw: thanks, will upgrade my py35 job | 23:38 |
| ianw | the job has worked in experimental | 23:38 |
| *** stakeda has joined #openstack-infra | 23:38 | |
| pabelanger | ianw: did we land the fedora-26 nodepool jobs in diskimage-builder yet? | 23:38 |
| *** felipemonteiro_ has quit IRC | 23:39 | |
| jeblair | fungi: note db backups happen at 00:00 but i can't think of how that could adversely affect your work. mentioning it in case you think of something i missed. | 23:39 |
| pabelanger | ianw: not yet, I just +2 482570, clarkb do you mind looking too? | 23:40 |
| clarkb | ya dbbackups happen separately so that should be fine | 23:40 |
| clarkb | separate process and all that | 23:40 |
| fungi | jeblair: great point, i'll wait until the mysqldump completes just for sanity | 23:40 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Upgrade the ARA fedora jobs to fedora 26 https://review.openstack.org/491633 | 23:40 |
| dmsimard | ianw: ^ thanks :) | 23:40 |
| ianw | pabelanger: oh yeah, that was hitting our mirror issues last week. it's ready to go | 23:40 |
| ianw | dmsimard: you ok to make them voting straight away? | 23:42 |
| dmsimard | ianw: it's just a py35 unit test job | 23:42 |
| dmsimard | ianw: I've been running those on my f26 laptop for a long time :P | 23:42 |
| dmsimard | oh wait, it's not unit test, it's integration -- but yeah, I've been running those too. | 23:43 |
| ianw | ok, it's your gate :) | 23:43 |
| dmsimard | ianw: I have faith in you ! | 23:43 |
| *** gildub has quit IRC | 23:43 | |
| dmsimard | ianw: if you're not confident I can add them as experimental first | 23:43 |
| ianw | it's likely fine, we can always revert quickly | 23:45 |
| *** sshnaidm is now known as sshnaidm|afk | 23:45 | |
| dmsimard | pabelanger: where did you hear about centos 7.4 this week ? Their turn around time isn't really that fast and they mentioned that this is actually the biggest dot release they've seen | 23:45 |
| ianw | pabelanger: yeah, re previous discussion i've started trying to pull apart some of the centos7 instability -> https://etherpad.openstack.org/p/centos7-dsvm-triage | 23:45 |
| ianw | it's slow work untangling this stuff though :/ | 23:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Import directives/roles from zuul https://review.openstack.org/491635 | 23:46 |
| clarkb | pabelanger: ianw I got sniped on the yaml comment and I'm fairly certain thats valid yaml so +2 (pabelanger has already approved) | 23:47 |
| pabelanger | dmsimard: ah, maybe it is rhel 7.4 | 23:47 |
| dmsimard | yeah rhel 7.4 was shipped last week | 23:47 |
| dmsimard | I think centos 7.4 will not land for a few weeks | 23:47 |
| clarkb | ya rhel happened which is how I know about the libvirt problems | 23:48 |
| dmsimard | pabelanger: I guess they're making good progress https://twitter.com/JohnnyCentOS/status/894537997865103362 | 23:49 |
| clarkb | our old branches can't run on rhel right now due to really new libvirt on rhel 7.4 and old libvirt-python in constraints | 23:49 |
| dmsimard | https://seven.centos.org/2017/08/centos-linux-7-1708-based-on-rhel-7-4-source-code/ | 23:50 |
| dmsimard | mentions before september 12th if all goes well | 23:50 |
| dmsimard | dead in the middle of pike release, yay | 23:50 |
| fungi | i thought rhel avoided updating those sort of things outside major releases? | 23:51 |
| fungi | ianw: i take that back, i was looking at today's exim logs. yesterday we rejected nearly 16k messages to -owner addresses | 23:53 |
| ianw | :( this is why we can't have nice things | 23:54 |
| clarkb | fungi I thought that too :) | 23:54 |
| fungi | and today's well on track to exceed that count | 23:54 |
| *** slaweq has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!