| *** olaph1 has joined #openstack-infra | 00:00 | |
| *** olaph has quit IRC | 00:01 | |
| *** rcernin has joined #openstack-infra | 00:08 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: make ec2-api job as n-v https://review.openstack.org/539740 | 00:09 |
|---|---|---|
| *** felipemonteiro__ has quit IRC | 00:11 | |
| *** hongbin has quit IRC | 00:12 | |
| *** rkukura_ has joined #openstack-infra | 00:13 | |
| *** rkukura has quit IRC | 00:14 | |
| *** rkukura_ is now known as rkukura | 00:14 | |
| *** dingyichen has joined #openstack-infra | 00:14 | |
| *** sree has joined #openstack-infra | 00:15 | |
| johnsom | I am seeing something strange with https://review.openstack.org/539713 in zuul. It looks like all of the jobs start, some go back to queued, others the progress bar goes blank again. | 00:17 |
| *** r-daneel has quit IRC | 00:18 | |
| *** sree has quit IRC | 00:20 | |
| *** jamesmcarthur has quit IRC | 00:22 | |
| corvus | johnsom: that's zuul retrying the job due to an error probably unrelated to the job itself | 00:23 |
| johnsom | Yeah, It's doing it even on the releasenotes job, so pretty low chance I broke that too. | 00:25 |
| corvus | however, it looks like jobs are starting to queue up. | 00:26 |
| johnsom | Yeah, there are a lot of jobs with all blue overview bars like mine | 00:28 |
| corvus | it looks like the executors are not picking up new jobs | 00:31 |
| *** olaph has joined #openstack-infra | 00:33 | |
| *** agopi has quit IRC | 00:34 | |
| *** olaph1 has quit IRC | 00:35 | |
| *** vivsoni__ has quit IRC | 00:35 | |
| *** vivsoni__ has joined #openstack-infra | 00:35 | |
| *** tosky has quit IRC | 00:36 | |
| *** zhurong has joined #openstack-infra | 00:37 | |
| corvus | okay, my theory is that the executor's merger is starving the executor | 00:41 |
| corvus | they share a lock, and a new job can't start executing while the merger is running, and i'm guessing that the executor side just isn't getting the lock | 00:42 |
| *** olaph1 has joined #openstack-infra | 00:44 | |
| *** dhill_ has quit IRC | 00:44 | |
| *** olaph has quit IRC | 00:44 | |
| corvus | it looks like this will self-correct, but it's going to be slow for the next little bit | 00:45 |
| corvus | okay, i think it's unjammed now. things should start speeding up again | 00:50 |
| johnsom | Ha, yeah, big change in the graphs. Hmmm | 00:50 |
| johnsom | Running builds went from well under 50 to almost 150 | 00:51 |
| *** sree has joined #openstack-infra | 00:53 | |
| openstackgerrit | Merged openstack-infra/zuul master: Don't override allowed-projects https://review.openstack.org/538353 | 00:57 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix cross source dependencies for github https://review.openstack.org/537428 | 00:57 |
| *** gyee has quit IRC | 00:58 | |
| *** psachin has joined #openstack-infra | 01:00 | |
| *** sree has quit IRC | 01:01 | |
| *** jistr has quit IRC | 01:01 | |
| *** jistr has joined #openstack-infra | 01:02 | |
| *** cuongnv has joined #openstack-infra | 01:04 | |
| *** armax has joined #openstack-infra | 01:07 | |
| *** gus has quit IRC | 01:07 | |
| *** Swami has quit IRC | 01:08 | |
| *** gus has joined #openstack-infra | 01:08 | |
| *** camunoz has quit IRC | 01:08 | |
| *** lbragstad has quit IRC | 01:10 | |
| *** sree has joined #openstack-infra | 01:13 | |
| *** armax has quit IRC | 01:13 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add memory awareness to system load governor https://review.openstack.org/539426 | 01:14 |
| *** stakeda has joined #openstack-infra | 01:15 | |
| bkero- | a | 01:16 |
| *** sree has quit IRC | 01:18 | |
| *** armax has joined #openstack-infra | 01:20 | |
| *** s-shiono has joined #openstack-infra | 01:22 | |
| *** jamesmca_ has joined #openstack-infra | 01:22 | |
| *** sree has joined #openstack-infra | 01:24 | |
| *** bcafarel has joined #openstack-infra | 01:27 | |
| *** liujiong has joined #openstack-infra | 01:27 | |
| *** namnh has joined #openstack-infra | 01:28 | |
| *** sree has quit IRC | 01:29 | |
| *** lbragstad has joined #openstack-infra | 01:31 | |
| *** liujiong has quit IRC | 01:33 | |
| *** liujiong has joined #openstack-infra | 01:33 | |
| *** apetrich has quit IRC | 01:37 | |
| *** apetrich has joined #openstack-infra | 01:37 | |
| *** gongysh has joined #openstack-infra | 01:40 | |
| *** salv-orlando has joined #openstack-infra | 01:44 | |
| *** salv-orlando has quit IRC | 01:49 | |
| *** sree has joined #openstack-infra | 01:49 | |
| *** dougwig has quit IRC | 01:52 | |
| *** sree has quit IRC | 01:53 | |
| *** jamesmca_ has quit IRC | 01:59 | |
| *** hongbin has joined #openstack-infra | 01:59 | |
| *** felipemonteiro has joined #openstack-infra | 02:00 | |
| *** agopi has joined #openstack-infra | 02:00 | |
| openstackgerrit | Merged openstack-infra/nodepool master: nodeutils: use socket.getaddrinfo instead of ipaddress https://review.openstack.org/539086 | 02:05 |
| openstackgerrit | Merged openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 02:05 |
| *** jamesmcarthur has joined #openstack-infra | 02:15 | |
| *** gcb has joined #openstack-infra | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:20 | |
| *** jamesmcarthur has joined #openstack-infra | 02:21 | |
| *** ramishra has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has quit IRC | 02:25 | |
| *** threestrands has joined #openstack-infra | 02:27 | |
| *** threestrands has quit IRC | 02:27 | |
| *** threestrands has joined #openstack-infra | 02:27 | |
| *** felipemonteiro has quit IRC | 02:29 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Migrate incomplete as todo https://review.openstack.org/529433 | 02:29 |
| *** andreas_s has joined #openstack-infra | 02:34 | |
| *** annp has joined #openstack-infra | 02:34 | |
| *** yamamoto has joined #openstack-infra | 02:37 | |
| *** andreas_s has quit IRC | 02:38 | |
| *** olaph has joined #openstack-infra | 02:40 | |
| *** olaph1 has quit IRC | 02:41 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Zuul: Remove project name https://review.openstack.org/539728 | 02:44 |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/bindep master: Zuul: Remove project name https://review.openstack.org/539777 | 02:52 |
| *** Liuqing has joined #openstack-infra | 02:56 | |
| *** Liuqing has quit IRC | 02:57 | |
| *** priteau has joined #openstack-infra | 02:57 | |
| *** priteau has quit IRC | 02:58 | |
| *** rlandy|bbl is now known as rlandy | 02:58 | |
| *** jamesmcarthur has quit IRC | 03:00 | |
| *** olaph1 has joined #openstack-infra | 03:04 | |
| *** olaph has quit IRC | 03:05 | |
| *** Liuqing has joined #openstack-infra | 03:05 | |
| *** r-daneel has joined #openstack-infra | 03:10 | |
| *** armaan has quit IRC | 03:12 | |
| *** armaan has joined #openstack-infra | 03:13 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove python-keystoneclient from test-requirements https://review.openstack.org/536879 | 03:15 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove pep8 and pyflakes from test-requirements https://review.openstack.org/536880 | 03:15 |
| *** r-daneel has quit IRC | 03:15 | |
| *** rosmaita has quit IRC | 03:17 | |
| *** vivsoni__ has quit IRC | 03:17 | |
| *** ihrachys has quit IRC | 03:19 | |
| *** kiennt26 has quit IRC | 03:19 | |
| *** cuongnv has quit IRC | 03:19 | |
| *** daidv has quit IRC | 03:19 | |
| *** sree has joined #openstack-infra | 03:19 | |
| *** kiennt26 has joined #openstack-infra | 03:19 | |
| *** daidv has joined #openstack-infra | 03:19 | |
| *** cuongnv has joined #openstack-infra | 03:20 | |
| *** sree has quit IRC | 03:20 | |
| *** sree has joined #openstack-infra | 03:20 | |
| *** vivsoni has joined #openstack-infra | 03:22 | |
| *** HeOS has quit IRC | 03:25 | |
| *** dhajare has joined #openstack-infra | 03:26 | |
| *** iyamahat has quit IRC | 03:37 | |
| openstackgerrit | Merged openstack-infra/zuul master: Update docs to use sphinx-build https://review.openstack.org/536881 | 03:37 |
| *** Liuqing has quit IRC | 03:40 | |
| *** olaph has joined #openstack-infra | 03:42 | |
| *** eharney has quit IRC | 03:42 | |
| *** olaph1 has quit IRC | 03:43 | |
| *** cshastri has joined #openstack-infra | 03:43 | |
| *** salv-orlando has joined #openstack-infra | 03:45 | |
| openstackgerrit | lei zhang proposed openstack/diskimage-builder master: Zuul: Remove project name https://review.openstack.org/539794 | 03:45 |
| *** dave-mccowan has quit IRC | 03:48 | |
| *** salv-orlando has quit IRC | 03:50 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Set default label for XFS disks https://review.openstack.org/532279 | 03:59 |
| *** coolsvap has joined #openstack-infra | 04:07 | |
| *** rlandy has quit IRC | 04:10 | |
| *** kiennt26 has quit IRC | 04:10 | |
| *** harlowja has quit IRC | 04:12 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack-infra/puppet-openstackci master: Remove pbr warnerrors in favor of sphinx check https://review.openstack.org/539804 | 04:14 |
| *** olaph1 has joined #openstack-infra | 04:15 | |
| *** gongysh has quit IRC | 04:16 | |
| *** olaph has quit IRC | 04:18 | |
| *** ykarel|away has joined #openstack-infra | 04:27 | |
| *** jamesmcarthur has joined #openstack-infra | 04:30 | |
| *** links has joined #openstack-infra | 04:32 | |
| *** jamesmcarthur has quit IRC | 04:35 | |
| *** harlowja has joined #openstack-infra | 04:41 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack-infra/system-config master: Remove pbr warnerrors in favor of sphinx check https://review.openstack.org/539812 | 04:43 |
| *** hongbin has quit IRC | 04:47 | |
| *** xarses_ has quit IRC | 04:49 | |
| *** zhurong has quit IRC | 04:55 | |
| *** markvoelker has quit IRC | 04:57 | |
| AJaeger | andreaf: please handle networking-odl as well | 05:09 |
| AJaeger | ianw, could you review https://review.openstack.org/538632 and https://review.openstack.org/538631 and https://review.openstack.org/538591 , please? Those remove some jobs from project-config. | 05:12 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Choose appropriate bootloader for block-device https://review.openstack.org/539731 | 05:12 |
| openstackgerrit | Merged openstack-infra/project-config master: remove jobs from ironic-inspector-tempest-plugin https://review.openstack.org/538632 | 05:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove sqlalchemy-migrate legacy tox job https://review.openstack.org/538631 | 05:26 |
| *** HeOS has joined #openstack-infra | 05:28 | |
| *** bfournie has quit IRC | 05:29 | |
| *** xarses_ has joined #openstack-infra | 05:29 | |
| *** andreww has joined #openstack-infra | 05:31 | |
| *** saop has joined #openstack-infra | 05:31 | |
| saop | Hello | 05:31 |
| *** sree has quit IRC | 05:31 | |
| saop | I created one nova vm using openvswitch, as part of nodepool | 05:31 |
| saop | And during installation of devstack there, it failed as openvswitch inside that vm is not able to ping outside | 05:32 |
| saop | Do i need to add anything in nova.conf so it properly use virtual port, ? | 05:32 |
| *** felipemonteiro has joined #openstack-infra | 05:33 | |
| *** xarses_ has quit IRC | 05:34 | |
| *** olaph has joined #openstack-infra | 05:34 | |
| *** olaph1 has quit IRC | 05:34 | |
| *** sree has joined #openstack-infra | 05:40 | |
| *** salv-orlando has joined #openstack-infra | 05:46 | |
| *** olaph1 has joined #openstack-infra | 05:46 | |
| *** olaph has quit IRC | 05:47 | |
| *** threestrands has quit IRC | 05:48 | |
| *** salv-orlando has quit IRC | 05:50 | |
| *** andreww has quit IRC | 05:51 | |
| *** gongysh has joined #openstack-infra | 05:54 | |
| *** gongysh has quit IRC | 05:56 | |
| *** agopi has quit IRC | 05:59 | |
| *** zhurong has joined #openstack-infra | 06:02 | |
| *** gongysh has joined #openstack-infra | 06:04 | |
| *** janki has joined #openstack-infra | 06:10 | |
| *** kjackal has joined #openstack-infra | 06:11 | |
| *** harlowja has quit IRC | 06:12 | |
| *** felipemonteiro has quit IRC | 06:15 | |
| *** bhavik1 has joined #openstack-infra | 06:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Don't gather host keys for non ssh connections https://review.openstack.org/535685 | 06:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 06:17 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/539825 | 06:18 |
| *** bhavik1 has quit IRC | 06:20 | |
| *** sree has quit IRC | 06:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove mirror01.dfw.o.o https://review.openstack.org/539828 | 06:30 |
| ianw | #status log mirror01.dfw.o.o has been retired due to performance issues; it is replaced by mirror02.dfw.o.o | 06:31 |
| *** dbecker has quit IRC | 06:31 | |
| openstackstatus | ianw: finished logging | 06:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Don't gather host keys for non ssh connections https://review.openstack.org/535685 | 06:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 06:35 |
| *** andreww has joined #openstack-infra | 06:38 | |
| *** xarses_ has joined #openstack-infra | 06:39 | |
| *** andreww has quit IRC | 06:43 | |
| *** dbecker has joined #openstack-infra | 06:45 | |
| *** iyamahat has joined #openstack-infra | 06:46 | |
| *** kjackal has quit IRC | 06:46 | |
| *** xarses_ has quit IRC | 06:50 | |
| *** pahuang has quit IRC | 06:52 | |
| *** armaan has quit IRC | 06:55 | |
| *** armaan has joined #openstack-infra | 06:55 | |
| *** lathiat has quit IRC | 06:55 | |
| *** markvoelker has joined #openstack-infra | 06:58 | |
| *** lathiat has joined #openstack-infra | 06:59 | |
| *** e0ne has joined #openstack-infra | 07:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix runtime stats reporting for noop job https://review.openstack.org/537432 | 07:08 |
| *** chenying has quit IRC | 07:11 | |
| *** zhurong has quit IRC | 07:14 | |
| *** andreas_s has joined #openstack-infra | 07:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/539825 | 07:15 |
| *** andreas_s has quit IRC | 07:16 | |
| *** pgadiya has joined #openstack-infra | 07:17 | |
| *** ykarel|away is now known as ykarel | 07:18 | |
| *** sshnaidm|off is now known as sshnaidm | 07:21 | |
| *** iyamahat has quit IRC | 07:22 | |
| *** rcernin has quit IRC | 07:23 | |
| *** zhurong has joined #openstack-infra | 07:24 | |
| *** markvoelker has quit IRC | 07:32 | |
| *** annp has quit IRC | 07:33 | |
| *** namnh has quit IRC | 07:33 | |
| *** annp has joined #openstack-infra | 07:33 | |
| *** namnh has joined #openstack-infra | 07:33 | |
| *** jamesmcarthur has joined #openstack-infra | 07:35 | |
| *** iyamahat has joined #openstack-infra | 07:37 | |
| *** kjackal has joined #openstack-infra | 07:37 | |
| *** jamesmcarthur has quit IRC | 07:39 | |
| *** claudiub has joined #openstack-infra | 07:44 | |
| *** salv-orlando has joined #openstack-infra | 07:46 | |
| *** claudiub|3 has quit IRC | 07:47 | |
| *** sree has joined #openstack-infra | 07:47 | |
| *** pcaruana has joined #openstack-infra | 07:47 | |
| *** slaweq has joined #openstack-infra | 07:50 | |
| *** salv-orlando has quit IRC | 07:51 | |
| *** sree has quit IRC | 07:51 | |
| *** jamesmcarthur has joined #openstack-infra | 07:54 | |
| *** jamesmcarthur has quit IRC | 07:58 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove legacy ec2-api jobs https://review.openstack.org/539109 | 07:59 |
| gmann | AJaeger: ^^ thanks for comment, i am removing job directly to merge ec2-api job migration | 08:00 |
| gmann | AJaeger: once this is merge i can abandon this - https://review.openstack.org/#/c/539740/ | 08:02 |
| *** kiennt26 has joined #openstack-infra | 08:02 | |
| *** dingyichen has quit IRC | 08:06 | |
| *** HeOS has quit IRC | 08:07 | |
| AJaeger | gmann: reviewing... | 08:08 |
| gmann | AJaeger: thanks | 08:08 |
| *** amoralej|off is now known as amoralej | 08:09 | |
| AJaeger | ianw: do you have time for another review, please? gmann asks for https://review.openstack.org/539109 | 08:10 |
| gmann | AJaeger: cool, i will rebase the other patches of that series meanwhile. | 08:12 |
| *** tmorin has joined #openstack-infra | 08:13 | |
| *** kopecmartin has joined #openstack-infra | 08:13 | |
| *** ramishra has quit IRC | 08:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Zuul: Remove project name https://review.openstack.org/539853 | 08:15 |
| mpeterson | AJaeger, andreaf: https://review.openstack.org/539854 networking-odl adaptation | 08:19 |
| *** b_bezak has joined #openstack-infra | 08:24 | |
| *** d0ugal has joined #openstack-infra | 08:27 | |
| *** s-shiono has quit IRC | 08:28 | |
| *** markvoelker has joined #openstack-infra | 08:29 | |
| *** snapiri|AFK has quit IRC | 08:33 | |
| AJaeger | mpeterson: thanks! | 08:36 |
| *** jpich has joined #openstack-infra | 08:39 | |
| *** jpena|off is now known as jpena | 08:40 | |
| *** olaph has joined #openstack-infra | 08:41 | |
| *** saop has quit IRC | 08:42 | |
| *** gongysh has quit IRC | 08:43 | |
| *** olaph1 has quit IRC | 08:43 | |
| *** dsariel has joined #openstack-infra | 08:47 | |
| *** ralonsoh has joined #openstack-infra | 08:47 | |
| *** salv-orlando has joined #openstack-infra | 08:47 | |
| *** pcichy has joined #openstack-infra | 08:49 | |
| *** gongysh has joined #openstack-infra | 08:50 | |
| *** salv-orlando has quit IRC | 08:52 | |
| *** shardy has joined #openstack-infra | 08:54 | |
| *** namnh_ has joined #openstack-infra | 08:58 | |
| *** namnh has quit IRC | 09:02 | |
| *** olaph1 has joined #openstack-infra | 09:03 | |
| andreaf | mpeterson: cool thanks | 09:03 |
| *** markvoelker has quit IRC | 09:03 | |
| *** olaph has quit IRC | 09:04 | |
| *** salv-orlando has joined #openstack-infra | 09:10 | |
| *** oidgar has joined #openstack-infra | 09:15 | |
| *** pcichy has quit IRC | 09:15 | |
| *** pcaruana has quit IRC | 09:19 | |
| *** pcaruana has joined #openstack-infra | 09:19 | |
| *** kiennt26 has quit IRC | 09:20 | |
| *** kopecmartin has quit IRC | 09:20 | |
| *** derekh has joined #openstack-infra | 09:21 | |
| *** zhurong has quit IRC | 09:25 | |
| *** liujiong has quit IRC | 09:27 | |
| *** kjackal has quit IRC | 09:31 | |
| *** zhurong has joined #openstack-infra | 09:33 | |
| *** yamahata has quit IRC | 09:36 | |
| *** iyamahat has quit IRC | 09:36 | |
| *** stakeda has quit IRC | 09:38 | |
| *** eumel8 has joined #openstack-infra | 09:38 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/openstack-zuul-jobs master: Extend project translation template for Queens https://review.openstack.org/539873 | 09:41 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Configure -horizon jobs for monasca-ui https://review.openstack.org/539874 | 09:45 |
| *** oidgar has quit IRC | 09:49 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/539596 | 09:52 |
| *** pcichy has joined #openstack-infra | 09:57 | |
| *** priteau has joined #openstack-infra | 09:57 | |
| *** markvoelker has joined #openstack-infra | 10:00 | |
| *** salv-orlando has quit IRC | 10:11 | |
| *** salv-orlando has joined #openstack-infra | 10:11 | |
| *** dsariel has quit IRC | 10:14 | |
| *** alexchadin has joined #openstack-infra | 10:14 | |
| *** salv-orlando has quit IRC | 10:16 | |
| *** kjackal has joined #openstack-infra | 10:16 | |
| *** cuongnv has quit IRC | 10:19 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/openstack-zuul-jobs master: Add python35-jobs-horizon-nonvoting template https://review.openstack.org/539887 | 10:22 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Configure -horizon jobs for monasca-ui https://review.openstack.org/539874 | 10:23 |
| *** gongysh has quit IRC | 10:25 | |
| *** tosky has joined #openstack-infra | 10:27 | |
| *** jchhatbar has joined #openstack-infra | 10:28 | |
| *** janki has quit IRC | 10:29 | |
| *** gongysh has joined #openstack-infra | 10:30 | |
| *** priteau has quit IRC | 10:30 | |
| *** snapiri has joined #openstack-infra | 10:33 | |
| *** markvoelker has quit IRC | 10:33 | |
| *** zhurong has quit IRC | 10:34 | |
| *** jchhatbar is now known as janki | 10:34 | |
| *** gongysh has quit IRC | 10:35 | |
| *** wolverineav has joined #openstack-infra | 10:36 | |
| *** sambetts|afk is now known as sambetts | 10:41 | |
| *** armaan has quit IRC | 10:42 | |
| *** armaan has joined #openstack-infra | 10:43 | |
| openstackgerrit | Fabio Verboso proposed openstack-infra/project-config master: Iotronic service and Lightning-rod have been moved to python3.5 in order to use the asyncio instead of Twisted https://review.openstack.org/539900 | 10:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy ec2-api jobs https://review.openstack.org/539109 | 10:47 |
| *** pbourke has quit IRC | 10:47 | |
| *** rubbish has joined #openstack-infra | 10:48 | |
| *** pbourke has joined #openstack-infra | 10:49 | |
| andreaf | clarkb, corvus: could you have a look at this devstack change? https://review.openstack.org/#/c/536611/ | 10:52 |
| rubbish | I am trying to execute the tests for jenkins-job-builder. On invocating 'tox -e py34' I finally get the error "'OS_STDOUT_CAPTURE' is not recognized as an internal or external command,". I have found no clue how to tackle this. | 10:55 |
| rubbish | The script ends with "ERROR: InvocationError: '<mypath>\\.tox\\py34\\Scripts\\python.EXE setup.py testr --slowest --testr-args='" | 10:55 |
| rubbish | trying with python 3.4.4 on windows 10 | 10:55 |
| *** lucas-afk is now known as lucasagomes | 10:56 | |
| *** annp has quit IRC | 10:57 | |
| *** priteau has joined #openstack-infra | 11:00 | |
| AJaeger | rubbish: better ask on #openstack-jjb | 11:01 |
| *** vivsoni_ has joined #openstack-infra | 11:02 | |
| *** owalsh_afk is now known as owalsh | 11:03 | |
| *** priteau has quit IRC | 11:04 | |
| *** vivsoni has quit IRC | 11:04 | |
| *** priteau has joined #openstack-infra | 11:04 | |
| *** mtreinish_ has joined #openstack-infra | 11:08 | |
| *** zaro_ has joined #openstack-infra | 11:08 | |
| *** asilenkov_ has joined #openstack-infra | 11:09 | |
| *** priteau has quit IRC | 11:09 | |
| *** armaan has quit IRC | 11:09 | |
| *** Ng__ has joined #openstack-infra | 11:09 | |
| *** armaan has joined #openstack-infra | 11:09 | |
| *** kmARC_ has joined #openstack-infra | 11:11 | |
| *** zaro has quit IRC | 11:11 | |
| *** mtreinish has quit IRC | 11:11 | |
| *** asilenkov has quit IRC | 11:11 | |
| *** Ng has quit IRC | 11:11 | |
| *** bstinson has quit IRC | 11:11 | |
| *** ameade has quit IRC | 11:11 | |
| *** sdake has quit IRC | 11:11 | |
| *** beisner has quit IRC | 11:11 | |
| *** hwoarang has quit IRC | 11:11 | |
| *** DuncanT has quit IRC | 11:11 | |
| *** kmARC has quit IRC | 11:11 | |
| *** edwarnicke has quit IRC | 11:11 | |
| *** nibalizer has quit IRC | 11:11 | |
| *** simonmcc has quit IRC | 11:11 | |
| *** ianw has quit IRC | 11:11 | |
| *** TheJulia has quit IRC | 11:11 | |
| *** ildikov has quit IRC | 11:11 | |
| *** mtreinish_ is now known as mtreinish | 11:11 | |
| *** nibz has joined #openstack-infra | 11:11 | |
| *** kmARC_ is now known as kmARC | 11:11 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:11 | |
| *** sdake has joined #openstack-infra | 11:11 | |
| *** sdake has quit IRC | 11:11 | |
| *** sdake has joined #openstack-infra | 11:11 | |
| *** ameade has joined #openstack-infra | 11:12 | |
| *** salv-orlando has joined #openstack-infra | 11:12 | |
| *** beisner has joined #openstack-infra | 11:12 | |
| *** ildikov has joined #openstack-infra | 11:12 | |
| *** hwoarang has joined #openstack-infra | 11:12 | |
| *** DuncanT has joined #openstack-infra | 11:12 | |
| *** priteau has joined #openstack-infra | 11:12 | |
| *** ianw has joined #openstack-infra | 11:12 | |
| *** simonmcc has joined #openstack-infra | 11:12 | |
| *** TheJulia has joined #openstack-infra | 11:12 | |
| *** edwarnicke has joined #openstack-infra | 11:13 | |
| *** bstinson has joined #openstack-infra | 11:14 | |
| *** salv-orlando has quit IRC | 11:17 | |
| *** adarazs is now known as adarazs_lunch | 11:17 | |
| *** gcb has quit IRC | 11:25 | |
| *** gcb has joined #openstack-infra | 11:25 | |
| *** markvoelker has joined #openstack-infra | 11:30 | |
| *** namnh_ has quit IRC | 11:32 | |
| *** salv-orlando has joined #openstack-infra | 11:38 | |
| *** sree has joined #openstack-infra | 11:45 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Configure -horizon jobs for monasca-ui https://review.openstack.org/539874 | 11:46 |
| *** pblaho has joined #openstack-infra | 11:49 | |
| *** pgadiya has quit IRC | 11:50 | |
| *** alexchadin has quit IRC | 11:50 | |
| *** alexchadin has joined #openstack-infra | 11:51 | |
| *** amoralej is now known as amoralej|off | 11:52 | |
| *** ramishra has joined #openstack-infra | 11:53 | |
| *** rubbish has quit IRC | 11:57 | |
| robcresswell | o/ I need to retire a repo; is there a documented process? I know there's a fairly common "deprecated / deleted" notice I've seen on a few repos | 11:59 |
| *** jpena is now known as jpena|lunch | 12:00 | |
| *** dhajare has quit IRC | 12:02 | |
| *** markvoelker has quit IRC | 12:04 | |
| cmurphy | robcresswell: this might be what you're looking for https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project | 12:04 |
| robcresswell | ahh perfect, thanks cmurphy | 12:05 |
| *** ykarel_ has joined #openstack-infra | 12:09 | |
| *** ykarel has quit IRC | 12:12 | |
| *** dave-mccowan has joined #openstack-infra | 12:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Invalidate flavor and image cache on 400 errors https://review.openstack.org/441215 | 12:15 |
| *** dtantsur|afk is now known as dtantsur | 12:18 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config master: Begin retiring horizon-cisco-ui https://review.openstack.org/539927 | 12:20 |
| *** salv-orlando has quit IRC | 12:25 | |
| *** salv-orlando has joined #openstack-infra | 12:25 | |
| *** links has quit IRC | 12:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Consume Task and TaskManager from shade https://review.openstack.org/414759 | 12:28 |
| *** adarazs_lunch is now known as adarazs | 12:29 | |
| *** salv-orlando has quit IRC | 12:30 | |
| *** ykarel__ has joined #openstack-infra | 12:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove need for findNetwork https://review.openstack.org/398462 | 12:33 |
| *** ykarel_ has quit IRC | 12:33 | |
| *** tpsilva has joined #openstack-infra | 12:33 | |
| *** jlabarre has joined #openstack-infra | 12:36 | |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: Zuul: Remove project name https://review.openstack.org/539930 | 12:37 |
| *** links has joined #openstack-infra | 12:39 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul web: add admin endpoint, enqueue commands https://review.openstack.org/539004 | 12:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Run image object autocleanup after uploading images https://review.openstack.org/539933 | 12:43 |
| *** makowals has quit IRC | 12:43 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/539596 | 12:43 |
| *** makowals has joined #openstack-infra | 12:47 | |
| *** camunoz has joined #openstack-infra | 12:49 | |
| *** makowals has quit IRC | 12:51 | |
| *** makowals has joined #openstack-infra | 12:54 | |
| *** dprince has joined #openstack-infra | 12:54 | |
| *** makowals has quit IRC | 12:55 | |
| *** jpena|lunch is now known as jpena|off | 12:56 | |
| *** panda|off is now known as panda | 12:58 | |
| *** katkapilatova has joined #openstack-infra | 13:00 | |
| *** katkapilatova has left #openstack-infra | 13:00 | |
| *** ykarel__ is now known as ykarel | 13:00 | |
| *** markvoelker has joined #openstack-infra | 13:01 | |
| *** links has quit IRC | 13:03 | |
| *** rosmaita has joined #openstack-infra | 13:05 | |
| *** weshay|ruck is now known as weshay | 13:06 | |
| *** jpena|off is now known as jpena | 13:07 | |
| *** sree has quit IRC | 13:08 | |
| *** sree has joined #openstack-infra | 13:09 | |
| *** oidgar has joined #openstack-infra | 13:10 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:10 | |
| *** jcoufal has joined #openstack-infra | 13:10 | |
| *** shardy is now known as shardy_mtg | 13:12 | |
| *** alexchadin has quit IRC | 13:12 | |
| *** sree has quit IRC | 13:14 | |
| *** trown|outtypewww is now known as trown | 13:14 | |
| *** edmondsw has joined #openstack-infra | 13:15 | |
| *** jamesmcarthur has joined #openstack-infra | 13:15 | |
| *** hemna_ has joined #openstack-infra | 13:15 | |
| mordred | cmurphy: if you have a spare sec, feel like +Aing this https://review.openstack.org/#/c/505339/ ? | 13:19 |
| *** jamesmcarthur has quit IRC | 13:19 | |
| *** armaan has quit IRC | 13:20 | |
| *** alexchadin has joined #openstack-infra | 13:21 | |
| *** jamesmcarthur has joined #openstack-infra | 13:21 | |
| *** coolsvap has quit IRC | 13:22 | |
| *** ethfci has joined #openstack-infra | 13:25 | |
| *** salv-orlando has joined #openstack-infra | 13:26 | |
| *** cshastri has quit IRC | 13:27 | |
| cmurphy | sure | 13:28 |
| *** jamesmcarthur has quit IRC | 13:28 | |
| *** ramishra has quit IRC | 13:28 | |
| *** psachin has quit IRC | 13:29 | |
| *** salv-orlando has quit IRC | 13:30 | |
| *** shardy_mtg is now known as shardy | 13:31 | |
| *** Ng__ is now known as Ng | 13:31 | |
| *** markvoelker has quit IRC | 13:33 | |
| *** markvoelker has joined #openstack-infra | 13:33 | |
| *** dklyle has quit IRC | 13:34 | |
| *** david-lyle has joined #openstack-infra | 13:34 | |
| *** rlandy has joined #openstack-infra | 13:37 | |
| *** olaph1 is now known as olaph | 13:39 | |
| *** janki has quit IRC | 13:39 | |
| *** makowals has joined #openstack-infra | 13:40 | |
| *** david-lyle has quit IRC | 13:40 | |
| *** tmorin1 has joined #openstack-infra | 13:43 | |
| *** tmorin has quit IRC | 13:43 | |
| *** dsariel has joined #openstack-infra | 13:46 | |
| *** hemna_ has quit IRC | 13:47 | |
| *** kgiusti has joined #openstack-infra | 13:47 | |
| *** olaph1 has joined #openstack-infra | 13:49 | |
| *** olaph has quit IRC | 13:51 | |
| *** makowals has quit IRC | 13:53 | |
| smcginnis | I'm looking at some stable branch failures. I see vmware-nsx failed with this: | 13:53 |
| smcginnis | http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/vmware-nsx/stable/ocata/build-openstack-sphinx-docs/cbc7b57/job-output.txt.gz#_2018-02-01_06_31_03_327272 | 13:53 |
| smcginnis | But project-config has the missing repo listed for that job here: https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L17030 | 13:53 |
| smcginnis | Is this handled differently for stable branches? | 13:54 |
| *** yamamoto has quit IRC | 13:55 | |
| *** annp has joined #openstack-infra | 13:55 | |
| *** trown is now known as trown|brb | 13:56 | |
| *** Goneri has joined #openstack-infra | 13:56 | |
| *** ykarel is now known as ykarel|afk | 13:58 | |
| *** trown|brb is now known as trown | 13:58 | |
| pabelanger | morning | 13:59 |
| *** makowals has joined #openstack-infra | 13:59 | |
| *** hemna_ has joined #openstack-infra | 14:00 | |
| Roamer` | hm, I asked earlier, but has anybody noticed review.o.o being flaky in the past 36 hours? Our CI, hosted at Hetzner, gets "fatal: unable to access 'https://review.openstack.org/openstack/nova/': Failed to connect to review.openstack.org port 443: Connection timed out" several times a day | 14:01 |
| Roamer` | shoud I bug Hetzner about it, or is it possible that something is up with Gerrit? | 14:01 |
| *** ihrachys has joined #openstack-infra | 14:01 | |
| *** alexchadin has quit IRC | 14:02 | |
| *** bobh has joined #openstack-infra | 14:02 | |
| *** sshnaidm is now known as sshnaidm|mtg | 14:06 | |
| *** jamesmcarthur has joined #openstack-infra | 14:06 | |
| *** ykarel|afk has quit IRC | 14:07 | |
| mugsie | random question - but should we be able to use journalctl on the devstack.journal.xz file from devstack runs? | 14:10 |
| mugsie | I keep getting this: | 14:10 |
| mugsie | ➜ Downloads journalctl --file devstack.journal | 14:10 |
| mugsie | Failed to open files: Bad message | 14:10 |
| mugsie | designate's log files also seem to be missing the ability to filter the log levels - is that a nova + friends thing only? | 14:12 |
| mugsie | http://logs.openstack.org/30/471530/23/check/designate-devstack-bind9/4712431/logs/screen-keystone.txt.gz vs http://logs.openstack.org/30/471530/23/check/designate-devstack-bind9/4712431/logs/screen-designate-worker.txt.gz | 14:12 |
| *** jhesketh has quit IRC | 14:12 | |
| *** agopi has joined #openstack-infra | 14:13 | |
| Roamer` | mugsie, ICBW, but doesn't journalctl expect the logfiles to be in the systemd internal binary format? | 14:13 |
| *** jhesketh has joined #openstack-infra | 14:13 | |
| mugsie | Roamer`: I have no idea :) | 14:13 |
| mugsie | but that would explain it | 14:13 |
| *** makowals has quit IRC | 14:13 | |
| *** iyamahat has joined #openstack-infra | 14:14 | |
| Roamer` | or maybe it's not that, let me check something much simpler | 14:14 |
| fungi | smcginnis: i'll see if i can piece that together in a moment | 14:15 |
| *** makowals has joined #openstack-infra | 14:16 | |
| Roamer` | mugsie, devstack.journal.xz is generated by journalctl -o export, which produces a slightly different format | 14:16 |
| *** yamahata has joined #openstack-infra | 14:16 | |
| *** olaph1 has quit IRC | 14:17 | |
| *** olaph has joined #openstack-infra | 14:17 | |
| Roamer` | mugsie, and if your next question is "so how in blazes to I read that?", I just learned about its existence, I don't know yet :) | 14:17 |
| fungi | Roamer`: the most common misbehavior we see from the gerrit rest api is 5xx from the apache frontend when jvm garbage collection activity gets too heavy and bogs down the backend. that doesn't seem to be the case right now and connection timeouts suggest the problem may be on the route between your client and the server... maybe a problem at a backbone peering point or something | 14:17 |
| *** yamamoto has joined #openstack-infra | 14:17 | |
| *** agopi has quit IRC | 14:18 | |
| mugsie | Roamer`: yeah - that seems to be it. | 14:18 |
| * mugsie feels some awful grep coming on | 14:18 | |
| fungi | mugsie: Roamer`: i believe there's a systemd utility you need to apply... i'll see if we've noted that in the devstack docs/readme | 14:18 |
| smcginnis | Thanks fungi, low priority. | 14:18 |
| mugsie | fungi: I dont think it is. When I figure it out, I will update the docs | 14:20 |
| Roamer` | mugsie, there's always https://pypi.python.org/pypi/journalparse/0.1.1 ... I think :) Never tried it, of course, but who knows | 14:21 |
| mugsie | Roamer`: thanks :) | 14:21 |
| * mugsie will give it a shot | 14:21 | |
| fungi | man journalctl and look at the -o export paragraph | 14:22 |
| fungi | people do still use manpages, right? ;) | 14:22 |
| fungi | To import the binary stream back into native journald format use systemd-journal-remote | 14:23 |
| Roamer` | fungi, ok, so *now* I'm feeling stupid, because I did read that manual page three times and I completely missed that sentence | 14:23 |
| *** olaph1 has joined #openstack-infra | 14:24 | |
| fungi | the job also mentions it in a code comment, but i'll admit that's probably not super discoverable... http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/export-devstack-journal/tasks/main.yaml#n21 | 14:24 |
| Roamer` | I basically stopped at "see Journal Export Format", opened a browser, and forgot to read the rest of the paragraph | 14:24 |
| *** olaph has quit IRC | 14:24 | |
| fungi | the code comment in the job also explains why we ended up using this format | 14:25 |
| Roamer` | mugsie, so, yeah, what fungi said about systemd-journal-remote :) A separate package must be installed though, not sure if it's installed by default | 14:25 |
| Roamer` | (at least on Debian systems it's in its own systemd-journal-remote package) | 14:25 |
| mugsie | cool - thanks fungi Roamer` | 14:26 |
| fungi | any time | 14:27 |
| *** beagles is now known as beagles_mtg | 14:27 | |
| *** alex_xu has quit IRC | 14:27 | |
| *** slaweq has quit IRC | 14:27 | |
| *** jcoufal_ has joined #openstack-infra | 14:28 | |
| *** slaweq has joined #openstack-infra | 14:28 | |
| Roamer` | or, of course, you could just parse the file itself, it seems to be almost straightforward, but it might require some parsing, grep is not the best tool for handling multiline text records | 14:29 |
| *** jamesmcarthur has quit IRC | 14:29 | |
| Roamer` | especially those also separated by sequences of null characters... gah! | 14:29 |
| *** swest has quit IRC | 14:30 | |
| fungi | well, you could tell grep that your line separator is \0 instead of \n i think, but yeah still hacky | 14:30 |
| *** alex_xu has joined #openstack-infra | 14:30 | |
| fungi | there is a tool designed to do this already, so just use it | 14:30 |
| Roamer` | actually writing grammar parsers in Perl is already almost second nature to me, but 1. this is getting off-topic, and 2. there is a tool already indeed, so I'll shut up now | 14:31 |
| fungi | (as much as systemd/journald still annoy me, i'm learning to avoid working around them at least) | 14:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: Zuul: Remove project name https://review.openstack.org/539930 | 14:31 |
| *** jcoufal has quit IRC | 14:31 | |
| Roamer` | fungi, and thanks for your response about review.o.o; if the problem persists, I'll bug the hosting provider | 14:32 |
| *** lucasagomes is now known as lucas-hungry | 14:32 | |
| *** slaweq has quit IRC | 14:33 | |
| *** hemna_ has quit IRC | 14:33 | |
| *** pblaho has quit IRC | 14:33 | |
| fungi | Roamer`: well, if you can replicate the problem from other locations or we get more reports of similar issues it may still indicate a problem at our side... i'm not ruling that out, just i haven't seen anything to suggest it's likely (yet) | 14:34 |
| *** dsariel has quit IRC | 14:34 | |
| fungi | our service provider is certainly not perpetually free of network problems from time to time | 14:34 |
| fungi | er | 14:34 |
| fungi | you know what i mean. still struggling with low blood-caffeine levels at this time of day | 14:35 |
| *** mriedem is now known as mriedem_afk | 14:35 | |
| *** esberglu has joined #openstack-infra | 14:35 | |
| *** isviridov_away has quit IRC | 14:37 | |
| *** vaidy has quit IRC | 14:37 | |
| *** eharney has joined #openstack-infra | 14:37 | |
| mugsie | fungi: system-journal-remote worked great - just had to find the program, as ubuntu seems to not add it to your path for some reason | 14:37 |
| Roamer` | "I open my mouth / Words come out / That make no sense / To a stranger's ear"? Oh man, do I ever know the feeling | 14:38 |
| mugsie | fungi: is there anything I have to add to get the log filtering toolbar at the top of the service logs to work ? | 14:40 |
| fungi | mugsie: it's dependent on matching rules in the os-loganalyze tool... looking for where now | 14:42 |
| *** efried has quit IRC | 14:43 | |
| *** efried has joined #openstack-infra | 14:43 | |
| *** sree has joined #openstack-infra | 14:44 | |
| *** hongbin has joined #openstack-infra | 14:46 | |
| *** r-daneel has joined #openstack-infra | 14:47 | |
| *** nikhil has joined #openstack-infra | 14:47 | |
| fungi | mugsie: looks like maybe the double timestamps in the lines could be throwing it off? | 14:48 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Include list of commits as Zuul variable https://review.openstack.org/539975 | 14:48 |
| mugsie | ah, maybe | 14:49 |
| fungi | it's depending on field order to determine where in the line to find the severity name | 14:49 |
| mugsie | OK, will experiment with os-loganalyze, thanks for the pointer | 14:50 |
| fungi | so the designate log example you provided has a second date in the field where teh keystone log has its severity | 14:50 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing min_avail_mem in memory governor https://review.openstack.org/539976 | 14:51 |
| fungi | i believe i remember someone saying that our os_loganalyze primarily supports whatever the oslo.log standard is | 14:51 |
| *** slaweq has joined #openstack-infra | 14:53 | |
| mugsie | fungi: found the issue | 14:53 |
| mugsie | https://github.com/openstack/designate/blob/master/devstack/plugin.sh#L26 | 14:53 |
| mugsie | we override the log format for some reason | 14:53 |
| lennyb | Is there a simple cli way to get a list of commits that are currently in zuul v2 gate? | 14:54 |
| fungi | aha | 14:54 |
| fungi | lennyb: i think harlowja wrote something called czuul... looking for it now | 14:54 |
| fungi | though parsing the status.json shouldn't be terribly hard either | 14:54 |
| d0ugal | Why are we missing logs for all these CI results? https://review.openstack.org/#/c/535912/ | 14:54 |
| d0ugal | I remember this was an issue recently, but that just completed ~30 mins ago | 14:55 |
| lennyb | fungi, thanks a lot, you just saved my day | 14:55 |
| *** camunoz has quit IRC | 14:56 | |
| fungi | lennyb: looks like maybe https://pypi.org/project/gerrit-view provides it | 14:56 |
| *** hemna_ has joined #openstack-infra | 14:56 | |
| *** pcichy has quit IRC | 14:57 | |
| *** pcichy has joined #openstack-infra | 14:57 | |
| fungi | d0ugal: yeah, this seems not good. i'm taking a look at static.o.o (which serves the logs site) now | 14:57 |
| d0ugal | fungi: thanks | 14:58 |
| *** mriedem_afk is now known as mriedem | 14:58 | |
| Roamer` | lennyb, curl + jq kind of works for me, [roam@straylight ~]$ curl -s http://ci-openstack.storpool.com/status.json | jq -r '.pipelines[0].change_queues[].heads | map(.[0].id)[] | 14:58 |
| lennyb | thanks Roamer` | 14:59 |
| Roamer` | this gives you just the first... erm, that's not what you want, is it... argh, sorry | 14:59 |
| Roamer` | curl -s http://ci-openstack.storpool.com/status.json | jq -r '.pipelines[0].change_queues[].heads | 15:00 |
| Roamer` | | map(.[-1].id)[]' | 15:00 |
| fungi | d0ugal: the good news is that the files are actually there. bad news is it looks like we may have some screwy permissions on some logs dirs... | 15:00 |
| fungi | drwx------ 3 jenkins jenkins 4096 Feb 1 11:19 /srv/static/logs/12/535912 | 15:00 |
| Roamer` | ...and trust the terminal to copy the newlines in two different ways when I selected them the same | 15:00 |
| fungi | d0ugal: check again, i've fixed that one but will run a find to correct other similarly broken directories | 15:01 |
| Roamer` | lennyb, curl -s http://ci-openstack.storpool.com/status.json | jq -r '.pipelines[0].change_queues[].heads | map(.[-1].id)[]' should give you just the commits that are actually being tested, the ones on top of the queues, although Zuul will show them at the bottom | 15:01 |
| fungi | thanks for pointing that out! | 15:01 |
| Roamer` | sorry about the .[0].id thing, this was completely useless | 15:01 |
| d0ugal | fungi: great, thanks! That works :) | 15:02 |
| smcginnis | Yelp - http://lists.openstack.org/pipermail/release-job-failures/2018-February/000751.html | 15:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: do not allow story id to change via payload https://review.openstack.org/539332 | 15:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: set up tests to run with sqlite https://review.openstack.org/539308 | 15:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: remove unused variable in test-cleanup.sh https://review.openstack.org/539695 | 15:04 |
| *** ijw has joined #openstack-infra | 15:06 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 15:06 |
| fungi | smcginnis: eep, looks like we may not be properly checking for that case still? | 15:08 |
| fungi | i bet it's because we don't run post pipeline jobs in check | 15:08 |
| smcginnis | fungi: dhellmann is going to revert the addition of a semaphore on the publish job for now. | 15:09 |
| smcginnis | I'm still not clear why we hit the rsync contention now but we never had that issue in the past. | 15:10 |
| fungi | looks like addition of the static-publish-releases semaphore attempts to modify the publish-static job, which can't be modified indirectly since it's marked as final | 15:10 |
| *** sree has quit IRC | 15:13 | |
| smcginnis | fungi: A little conversation on this going on in -release | 15:13 |
| * fungi switches context | 15:13 | |
| AJaeger | fungi, pabelanger, could you review andreaf's https://review.openstack.org/#/c/539683/ - and be around in case of problems? | 15:19 |
| AJaeger | and could you also review https://review.openstack.org/#/c/538591/ , please? Easy job removal | 15:19 |
| andreaf | ttx mordred corvus added my blog to openstack-planet https://review.openstack.org/#/c/539979/ | 15:21 |
| ttx | +2 | 15:22 |
| mordred | +A | 15:22 |
| *** sree has joined #openstack-infra | 15:24 | |
| *** dhill_ has joined #openstack-infra | 15:25 | |
| pabelanger | AJaeger: have we tested the changes via depends-on? having issues locating patches | 15:26 |
| andreaf | ttx mordred thanks | 15:26 |
| *** salv-orlando has joined #openstack-infra | 15:26 | |
| andreaf | pabelanger: yeah one sec https://review.openstack.org/#/c/539686/2 | 15:27 |
| *** Spazmotic has left #openstack-infra | 15:28 | |
| andreaf | https://review.openstack.org/#/c/539708/ | 15:28 |
| andreaf | pabelanger ^^^ | 15:28 |
| AJaeger | https://review.openstack.org/539854 as well | 15:28 |
| AJaeger | but then is not finished ^ | 15:29 |
| pabelanger | AJaeger: andreaf: what is the risk of breaking things? Given outages last week, and release schedules... | 15:30 |
| pabelanger | should we hold off until PTG? | 15:30 |
| AJaeger | pabelanger: that risk is why I asked for somebody to be around when we merge it. | 15:31 |
| AJaeger | pabelanger, andreaf, waiting until PTG is an option as well (or coordinate with release team) | 15:31 |
| *** salv-orlando has quit IRC | 15:31 | |
| pabelanger | AJaeger: yah, integrated queue is at 12 hours right now, we should consider that. | 15:31 |
| efried | Does zuul.openstack.org take a querystring for filters? | 15:32 |
| *** ociuhandu has joined #openstack-infra | 15:33 | |
| *** ociuhandu has quit IRC | 15:33 | |
| *** slaweq_ has joined #openstack-infra | 15:34 | |
| AJaeger | pabelanger: let's hear what andreaf thinks - I know that I won't be around later today, so will not +A now | 15:34 |
| *** dsariel has joined #openstack-infra | 15:35 | |
| *** sree has quit IRC | 15:36 | |
| *** sree has joined #openstack-infra | 15:37 | |
| fungi | efried: there are some status queries you can do... i believe by change/commit identifier is one | 15:37 |
| efried | fungi: Neat - is that documented anywhere? | 15:37 |
| fungi | efried: oh, you mean in the filter box for its webui? | 15:37 |
| fungi | or for direct status.json gets? | 15:37 |
| *** lucas-hungry is now known as lucasagomes | 15:38 | |
| efried | fungi: I love the filter box; but e.g. on my mobile the page seems to refresh at random and I lose that box. So I was wondering if I could get the same effect with e.g. ?filter=123456,654321,... | 15:38 |
| efried | ...&expand_by_default=True | 15:38 |
| *** slaweq_ has quit IRC | 15:38 | |
| AJaeger | pabelanger: could you also review https://review.openstack.org/#/c/538591/ and https://review.openstack.org/#/c/538300/ , please? Easy job removals | 15:38 |
| efried | lose that box => the contents of the box disappear (when the page refreshes) | 15:39 |
| fungi | mmm, https://docs.openstack.org/infra/zuul/search.html?q=status seems to indicate we're missing documentation about the status api | 15:40 |
| fungi | or else the search in sphinx is not so great | 15:41 |
| *** ijw has quit IRC | 15:41 | |
| *** sree has quit IRC | 15:41 | |
| *** eumel8 has quit IRC | 15:42 | |
| efried | fungi: I know you guys are busy with actual important stuff; if you want to point me to the appropriate repo and some way I can discover/derive the current functionality, I'd be happy to propose a doc patch. | 15:42 |
| *** david-lyle has joined #openstack-infra | 15:43 | |
| fungi | efried: well, it's all in the master branch of the openstack-infra/zuul repo... i'll see if i can get you a more precise directory/file starting point | 15:45 |
| *** beagles_mtg is now known as beagles | 15:46 | |
| *** salv-orlando has joined #openstack-infra | 15:46 | |
| *** myoung|off is now known as myoung | 15:47 | |
| efried | fungi: For purposes of https://docs.openstack.org/infra/zuul/user/index.html what is a "user" of zuul? Does "user" incorporate someone like me, who just thinks of it as a thing that needs to happen before my nova patches merge? Seems like maybe so; in which case, this could be a good place to spin off a "monitoring your jobs" page. | 15:51 |
| andreaf | pabelanger, AJaeger: the code for the list input is the same as before and the code for the dict input has an "ignore errors" - so I think it's very low risk | 15:51 |
| fungi | efried: i'm really not sure, it's probably a more appropriate topic for the crowd in the #zuul channel actually since that's more of an upstream question rather than downstream (we do have a lot of overlap in people here, granted) | 15:52 |
| andreaf | it's not terribly urgent either but I'd like to merge it at least 1 day before PTG so that if we work on jobs during the PTG we have the right format in place | 15:52 |
| *** eglute_s is now known as eglute | 15:52 | |
| *** StaceyF has joined #openstack-infra | 15:52 | |
| pabelanger | fungi: http://zuul.openstack.org/stream.html?uuid=783764a7cce74a02a90ca60a96c925aa&logfile=console.log looks to be another job hung in inap. Going to see if I can collect info about instance | 15:53 |
| fungi | pabelanger: cool, definitely grab an openstack server show as well as openstack console log show | 15:53 |
| *** agopi has joined #openstack-infra | 15:54 | |
| fungi | and see if the ssh behavior mimics what we saw yesterday | 15:54 |
| pabelanger | fungi: how are you mapping UUID to running jobs, since we no longer log that info into console logs | 15:54 |
| *** jamesmcarthur has joined #openstack-infra | 15:54 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add available RAM to statsd https://review.openstack.org/539707 | 15:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove oslo-cookiecutter legacy job https://review.openstack.org/538591 | 15:54 |
| StaceyF | Hello everyone, I'm back. I have a general question. I went through your gerrit configurations and I was wondering, if say a Core does a workflow +1 with the 2.. +2 code reviews, does zuul then take that patchset, merge it if the parent commit has moved and then zuul actually merges after the gates pass? | 15:54 |
| fungi | pabelanger: the console log mentions the executor, and then i look in the executor log there | 15:55 |
| *** armax has quit IRC | 15:55 | |
| andreaf | pabelanger AJaeger the only risk of waiting is that more people will write jobs the wrong way | 15:55 |
| fungi | pabelanger: though there's likely a better way i just don't know about | 15:55 |
| pabelanger | fungi: yah, that is what I just did: ubuntu-xenial-inap-mtl01-0002321991 | 15:55 |
| andreaf | if we have the right examples in devstack / tempest then copy paste will help us :) | 15:55 |
| mgagne | ceae204f-8224-44ac-8f04-468bc6b430f2 | 15:55 |
| mgagne | hostId: 698ec2c634e13382c007975b83c80e7b5cfdaffdec0d02f2a81724e7 | 15:56 |
| mgagne | new one again | 15:56 |
| fungi | StaceyF: almost. we have merge-if-necessary set for the merge strategy in gerrit, and zuul "pushes the submit button" in gerrit (via the gerrit api) for that change if all tests pass | 15:56 |
| StaceyF | perfect, that is what I was wondering, thank you | 15:56 |
| pabelanger | mgagne: yah ceae204f-8224-44ac-8f04-468bc6b430f2 | 15:56 |
| mgagne | pabelanger: task jbd2/vda1-8:211 blocked for more than 120 seconds. | 15:57 |
| mgagne | similar problem again | 15:57 |
| fungi | StaceyF: zuul does though also locally merge that change to the branch tip for purposes of testing what the repository would be like in that state | 15:57 |
| *** claudiub|2 has joined #openstack-infra | 15:57 | |
| *** annp has quit IRC | 15:57 | |
| fungi | StaceyF: and the dependent pipeline mechanism in zuul (which we use for our "gate" pipeline) ensures that the repository state which is tested matches the same commit order in which zuul will ask gerrit to merge those changes (even across repositories in the case of shared queues) | 15:58 |
| pabelanger | mgagne: ouch | 15:58 |
| pabelanger | mgagne: fungi: maybe we can hold the node an see if we can get into it via console | 15:59 |
| AJaeger | andreaf, pabelanger: gave a +2 now on https://review.openstack.org/#/c/539683 after checking again - but not a +A since I'll be offline in a bit... | 15:59 |
| andreaf | AJaeger ok cool thanks | 15:59 |
| *** olaph1 is now known as olaph | 15:59 | |
| StaceyF | thanks for the info fungi | 15:59 |
| *** claudiub has quit IRC | 16:00 | |
| pabelanger | mgagne: fungi: or maybe we disable inap, and see if we can reproduce issues using xenial images ourself | 16:00 |
| StaceyF | Do you have any plans on creating a helm chart for zuul v3? or has there already been one created that I've missed? | 16:00 |
| mgagne | pabelanger: what I can do is fast forward our maintenance plan for that region and upgrade a bunch of softwares on the compute nodes and see if it fixes anything | 16:01 |
| corvus | StaceyF: i'm not aware of anyone working on that, but it would be welcome! if you're interested in that, we could host it in our gerrit along with zuul itself. and you may want to join the #zuul channel where we discuss 'upstream' zuul -- ie, zuul itself, not just related to openstack. | 16:03 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix missing min_avail_mem in memory governor https://review.openstack.org/539976 | 16:05 |
| StaceyF | cool thanks | 16:05 |
| fungi | AJaeger: not sure we can still hold a node for an already-running job? is that still a thing or do we only have advance autohold now? | 16:05 |
| fungi | StaceyF: yeah, asking in #zuul is likely to get you better answers i suspect. i, for one, have no idea what a "helm chart" is or why zuul might want one ;) | 16:06 |
| fungi | AJaeger: sorry, i mean to tab-complete pabelanger | 16:07 |
| * fungi is juggling discussions in too many channels again | 16:07 | |
| *** rfolco is now known as rfolco|ruck | 16:07 | |
| pabelanger | corvus: Shrews: if we create an autohold on a running jobs, is that too late in the process to have zuul hold it? I'd like to see how we could hold ubuntu-xenial-inap-mtl01-0002321991 but unsure how | 16:07 |
| pabelanger | fungi: just seen, I asked the experts^ | 16:08 |
| *** felipemonteiro has joined #openstack-infra | 16:08 | |
| clarkb | the bigger issue is you can't say hold a node in this cloud right? | 16:08 |
| clarkb | you can only say hold a node for this job? | 16:08 |
| Shrews | pabelanger: i don't think so. when the job completes, that's when it checks to see if it should hold the node | 16:08 |
| *** rossella_s has quit IRC | 16:08 | |
| clarkb | also I am pre caffeine so ignore anything I'm saying probably | 16:08 |
| Shrews | so as long as it's set before completing, should be ok | 16:08 |
| pabelanger | clarkb: yah, when I tried using nodepool hold, unable to because of locked being held by zuul | 16:09 |
| Shrews | pabelanger: 'nodepool hold' should NOT be used | 16:09 |
| * Shrews is going to put up a change to remove that right now | 16:09 | |
| pabelanger | Shrews: yah, last I used it (not today) didn't work. just times out on lock | 16:09 |
| *** felipemonteiro_ has joined #openstack-infra | 16:09 | |
| pabelanger | but, do think we need a way to hold running nodes like nodepoolv2 | 16:10 |
| pabelanger | I'll set an autohold in zuul and see if that works | 16:10 |
| corvus | pabelanger: setting the hold in zuul is that way. | 16:10 |
| *** haleyb has quit IRC | 16:10 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Remove the hold command https://review.openstack.org/540002 | 16:10 |
| *** rossella_s has joined #openstack-infra | 16:11 | |
| *** kgiusti has quit IRC | 16:11 | |
| fungi | looks like there's a zuul autohold which takes a job as a parameter, but no zuul hold which takes a specific node id as a parameter | 16:12 |
| corvus | https://docs.openstack.org/infra/zuul/admin/client.html#autohold | 16:12 |
| Shrews | fungi: nope, you can't specify a particular node b/c you have no idea which node it will run on | 16:12 |
| pabelanger | zuul autohold --tenant openstack --project openstack/tempest --job neutron-grenade --reason "pabelanger: debug inap failures" | 16:12 |
| Shrews | nor can you name a particular provider | 16:12 |
| *** haleyb has joined #openstack-infra | 16:13 | |
| *** slaweq has quit IRC | 16:13 | |
| pabelanger | yah, it would be nice to also support node id | 16:13 |
| corvus | well, there are patches in flight to support holding for specific changes. that may be sufficent. | 16:13 |
| fungi | Shrews: use case is when the build is underway and you see the (rare, perhaps also provider-specific) failure condition you're concerned with unfolding want to avoid it getting deleted once that job completes | 16:14 |
| clarkb | problem is a specific change may not run on a specific provider | 16:14 |
| pabelanger | corvus: great | 16:14 |
| *** felipemonteiro has quit IRC | 16:14 | |
| clarkb | so in this particular case we could recheck a specific change until it did, but may be better ux to be able to specify hold the next job foo that runs on provider bar | 16:14 |
| corvus | clarkb: i mean that if the job is running, you know what criterea to give it. | 16:14 |
| fungi | clarkb: yeah, again this is not about asking in advance to have a node held, but asking to have a currently used node not be deleted (like in the old nodepool hold days) | 16:15 |
| corvus | clarkb: the case we're dealing with is not a consistent provider failure, it's a specific node. | 16:15 |
| *** lucasagomes has quit IRC | 16:15 | |
| fungi | the autohold feature is great, and more useful than the old node-specific holds, but there are still rare circumstances where the latter would still be nice to have | 16:16 |
| *** hemna_ has quit IRC | 16:16 | |
| corvus | i'd like to see if, when we add the additional specificity of autoholding by change, that's sufficient | 16:17 |
| corvus | (you would be able to hold project+job+change) | 16:17 |
| pabelanger | http://zuul.openstack.org/stream.html?uuid=d11508a1716441dba61e4bf47a86a016&logfile=console.log looks to also be a hung inap jobs | 16:17 |
| pabelanger | job* | 16:18 |
| clarkb | re the jbd2 blocked for more than 120 seconds error that is coming from the VM guest right? | 16:18 |
| pabelanger | let me get UUID of node | 16:18 |
| *** lucasagomes has joined #openstack-infra | 16:18 | |
| corvus | clarkb: yes | 16:18 |
| *** kgiusti has joined #openstack-infra | 16:18 | |
| clarkb | mgagne: are the disks for the VM guest local to the hypervisor or remote? | 16:18 |
| pabelanger | fungi: do you know which zuul-executor other failures were on? | 16:18 |
| corvus | clarkb: it is from the kernel running on the openstack instance we get from inap | 16:18 |
| mgagne | clarkb: local | 16:18 |
| pabelanger | fungi: both so far, are ze02.o.o today. Not sure if related | 16:19 |
| pabelanger | ubuntu-xenial-inap-mtl01-0002323954 is 2nd instance | 16:19 |
| mgagne | 65b1308f-f1f1-4077-8a70-7435d903940f | 16:20 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Remove the hold command https://review.openstack.org/540002 | 16:20 |
| pabelanger | mgagne: yes, just confirmed | 16:20 |
| mgagne | we seen that hostId before: d50b9618e3e33c1d0b08ebe16ca501226e1ee405d838e1b69a79c600 | 16:20 |
| pabelanger | [ 1560.543555] INFO: task jbd2/vda1-8:216 blocked for more than 120 seconds. | 16:21 |
| pabelanger | from console log | 16:21 |
| *** e0ne has quit IRC | 16:22 | |
| mgagne | so same error again | 16:22 |
| clarkb | we switched from raw to qcow2 right? I wonder if this is a result of that (though we saw similar errors early in the month before that switch I guess) | 16:24 |
| pabelanger | yah, was thinking that too, but I think issue that fungi reported in history is before that change | 16:25 |
| mgagne | pabelanger: I can check creation date of the instance | 16:25 |
| mgagne | pabelanger: cf197071-c825-4771-b824-0d2eedb9b9ab (2018-01-05 18:04:42) | 16:26 |
| mgagne | pabelanger: so problem was present before | 16:26 |
| *** ykarel|afk has joined #openstack-infra | 16:28 | |
| pabelanger | yah, I'm checking to see if something in diskimage-builder changed. 2.10.1 is latest release, about 3weeks ago. before that 2.9.0 was release 4 months go. | 16:28 |
| mgagne | and also before the kernel update earlier for meltdown (by Ubuntu) | 16:28 |
| pabelanger | yah | 16:29 |
| mgagne | and nothing changed on the compute node since... (except qcow2 base file) | 16:29 |
| *** bhavik1 has joined #openstack-infra | 16:30 | |
| *** felipemonteiro_ has quit IRC | 16:30 | |
| *** pcichy has quit IRC | 16:31 | |
| pabelanger | clarkb: mgagne: in the mean time, do we can to consider setting inap max-servers: 0 and start spending time out manually reproducing the issue? To keep merging patches in gate | 16:31 |
| mgagne | pabelanger: I wouldn't mind and I could also update compute nodes without impacting jobs | 16:32 |
| *** StaceyF has quit IRC | 16:33 | |
| *** bhavik1 has quit IRC | 16:36 | |
| *** iyamahat has quit IRC | 16:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zone-zuul-ci.org master: Add records for web site https://review.openstack.org/540029 | 16:36 |
| corvus | fungi, pabelanger: ^ can you review that please? | 16:37 |
| *** agopi_ has joined #openstack-infra | 16:38 | |
| *** yamahata has quit IRC | 16:38 | |
| openstackgerrit | Merged openstack-infra/zuul master: Zuul: Remove project name https://review.openstack.org/539853 | 16:40 |
| *** agopi has quit IRC | 16:41 | |
| pabelanger | +3 | 16:41 |
| *** pcichy has joined #openstack-infra | 16:42 | |
| pabelanger | mgagne: sure, if you want to propose patch to project-config, we can have clarkb review too. It will be nodepool/nl03.o.o.yaml now. | 16:42 |
| *** agopi_ has quit IRC | 16:44 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config master: Temporary disable inap-mtl01 region for maintenance https://review.openstack.org/540032 | 16:44 |
| *** gyee has joined #openstack-infra | 16:45 | |
| *** pcaruana has quit IRC | 16:45 | |
| pabelanger | dmsimard|afk: I see 2 nodes on hold in nodepool, are those still needed or can we release them? | 16:46 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: do not allow story id to change via payload https://review.openstack.org/539332 | 16:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: set up tests to run with sqlite https://review.openstack.org/539308 | 16:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: remove unused variable in test-cleanup.sh https://review.openstack.org/539695 | 16:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: Stop storing zuul references https://review.openstack.org/540033 | 16:51 |
| *** yee37933 has quit IRC | 16:51 | |
| *** yee379 has joined #openstack-infra | 16:51 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Allow specifying node's TTL when holding https://review.openstack.org/539493 | 16:52 |
| *** shardy has quit IRC | 16:52 | |
| *** sshnaidm|mtg is now known as sshnaidm|rover | 16:53 | |
| clarkb | mgagne: pabelanger I +2'd but didnt approve as I'm currently making breakfast for kids | 16:54 |
| pabelanger | clarkb: mgagne: okay, I can monitor approval | 16:55 |
| *** b_bezak has quit IRC | 16:57 | |
| *** b_bezak has joined #openstack-infra | 16:57 | |
| *** tmorin1 has quit IRC | 16:58 | |
| openstackgerrit | Merged openstack-infra/zone-zuul-ci.org master: Add records for web site https://review.openstack.org/540029 | 16:58 |
| *** janki has joined #openstack-infra | 17:00 | |
| *** armaan has joined #openstack-infra | 17:01 | |
| openstackgerrit | Krzysztof Klimonda proposed openstack-infra/zuul master: Support autoholding nodes for specific changes/refs https://review.openstack.org/540035 | 17:01 |
| *** dsariel has quit IRC | 17:02 | |
| *** b_bezak has quit IRC | 17:02 | |
| *** d0ugal has quit IRC | 17:03 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 17:05 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 17:06 |
| *** agopi_ has joined #openstack-infra | 17:09 | |
| *** agopi_ is now known as agopi | 17:09 | |
| *** harlowja has joined #openstack-infra | 17:09 | |
| *** StaceyF has joined #openstack-infra | 17:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Temporary disable inap-mtl01 region for maintenance https://review.openstack.org/540032 | 17:16 |
| *** olaph1 has joined #openstack-infra | 17:17 | |
| *** olaph has quit IRC | 17:17 | |
| *** iyamahat has joined #openstack-infra | 17:17 | |
| *** d0ugal has joined #openstack-infra | 17:19 | |
| *** ykarel|afk has quit IRC | 17:22 | |
| *** oidgar has quit IRC | 17:23 | |
| pabelanger | corvus: do we want to consider zuul-executor restarts to pick up memory governor changes? Or hold off until later time. | 17:24 |
| pabelanger | ze01.o.o still needs to be restart to bring console stream back online | 17:24 |
| smcginnis | fungi: I don't suppose you had a chance to take a look at that failed vmware-nsx stable job? I know there have been many interupts since this morning. | 17:25 |
| *** yamahata has joined #openstack-infra | 17:27 | |
| *** derekh has quit IRC | 17:27 | |
| *** pcichy has quit IRC | 17:28 | |
| corvus | pabelanger: also hostname changes | 17:30 |
| pabelanger | corvus: Yah, I keep forgetting about that | 17:30 |
| *** jtomasek_ has quit IRC | 17:31 | |
| *** psachin has joined #openstack-infra | 17:31 | |
| openstackgerrit | Merged openstack-infra/zuul master: requirements: remove paramiko <2.0 cap https://review.openstack.org/535540 | 17:32 |
| fungi | smcginnis: thanks for the reminder, i did start but got sidetracked. taking a closer look now | 17:33 |
| *** armaan has quit IRC | 17:35 | |
| fungi | smcginnis: http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/vmware-nsx/stable/ocata/build-openstack-sphinx-docs/cbc7b57/zuul-info/inventory.yaml is where to look to double-check what projects were included, and i agree several from the required-projects list in project-config for that job don't appear to have been included (not just networking-l2gw) | 17:36 |
| smcginnis | Probably just the first that got hit. | 17:37 |
| smcginnis | Are stable jobs handled differently? I didn't think so, but this doesn't appear to be an issue on master. | 17:37 |
| fungi | yeah, looks like it only used the project itself plus horizon, neutron and requirements | 17:37 |
| smcginnis | But I saw stable jobs fail in both pike and ocata. | 17:37 |
| andreaf | the patch on top of the gate queue failed the grenade job but it seems to be stuck there - I guess everyone knows already but just in case :) | 17:38 |
| *** sambetts is now known as sambett|afk | 17:41 | |
| *** jpich has quit IRC | 17:45 | |
| *** trown is now known as trown|lunch | 17:46 | |
| *** Swami has joined #openstack-infra | 17:47 | |
| *** Qiming has quit IRC | 17:49 | |
| *** Qiming has joined #openstack-infra | 17:49 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove the hold command https://review.openstack.org/540002 | 17:49 |
| fungi | smcginnis: i'm trying to see if the _inheritance_path list tells us anything useful in this context | 17:50 |
| corvus | files.o.o is in the emergency file due to an error with the zuul-ci.org intermediate cert. i think that has been corrected, so i'm going to run puppet to verify and then remove | 17:51 |
| fungi | thanks for the heads up, corvus! | 17:51 |
| corvus | clarkb: https://review.openstack.org/538041 | 17:52 |
| *** iyamahat has quit IRC | 17:52 | |
| *** psachin has quit IRC | 17:52 | |
| fungi | smcginnis: '<Job build-openstack-sphinx-docs branches: {MatchAny:{BranchMatcher:stable/ocata},{BranchMatcher:stable/pike}} | 17:52 |
| fungi | source: openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#698>' | 17:52 |
| *** iyamahat has joined #openstack-infra | 17:54 | |
| fungi | smcginnis: so http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n698 | 17:54 |
| smcginnis | fungi: So that's just the setup telling the periodic jobs to run py27 and docs, right? | 17:54 |
| fungi | smcginnis: i wonder if that line has moved since the job ran... digging | 17:54 |
| smcginnis | fungi: Oh! So the required-projects need to be added there? | 17:55 |
| smcginnis | fungi: Yeah, looks like it should actually be the job at line 683 | 17:55 |
| *** rkukura_ has joined #openstack-infra | 17:55 | |
| smcginnis | OK, the job names do match with what is defined in project-config/zuul.d/projects.yaml for that repo where the required-projects are added. | 17:56 |
| corvus | #status log files02.openstack.org removed from emergency file after zuul-ci.org intermediate cert problem resolved | 17:57 |
| openstackstatus | corvus: finished logging | 17:57 |
| *** rkukura has quit IRC | 17:57 | |
| *** rkukura_ is now known as rkukura | 17:57 | |
| *** HeOS has joined #openstack-infra | 17:57 | |
| *** panda is now known as panda|off | 17:58 | |
| corvus | https://zuul-ci.org/ exists | 17:58 |
| *** hemna_ has joined #openstack-infra | 17:58 | |
| fungi | smcginnis: still looking to figure out where it's getting horizon, neutron and requirements from instead of the required-projects list for the variant in projects.yaml | 17:59 |
| smcginnis | corvus: What, not zuul-ci.io. All the cool kids are .io now, didn't you know? | 17:59 |
| *** SumitNaiksatam has joined #openstack-infra | 18:00 | |
| *** slaweq has joined #openstack-infra | 18:00 | |
| fungi | smcginnis: feel free to blame me for pushing for a .org like the crufty old curmudgeons we are ;) | 18:00 |
| corvus | fungi: you must share that blame with me | 18:00 |
| smcginnis | :) | 18:00 |
| fungi | done | 18:00 |
| corvus | there's nothing certain about anything in the DNS, but many cctlds (especially smaller ones) have in general been more volatile than .org | 18:01 |
| fungi | yeah, that was precisely my concern as well | 18:01 |
| pabelanger | corvus: yay, looks great | 18:02 |
| corvus | we're in this for the long-haul. i mean, we didn't even set up a website for 6 years. | 18:03 |
| pabelanger | ha | 18:03 |
| fungi | i mean, for example i do personally have an icelandic domain registered (among many others), and iceland's a very nice country with a perfectly serviceable and modern registry, but... i don't trust i'll be as able to hold onto it as a .org, .com, .net, et cetera | 18:03 |
| *** slaweq has quit IRC | 18:04 | |
| fungi | and for a project like zuul, updating domain names for all the infrastructure because the registrar decided we don't qualify to renew it under some new policy would be a pain | 18:04 |
| fungi | especially for a situation so easily avoided | 18:05 |
| fungi | corvus: the new web site looks great! | 18:06 |
| *** janki has quit IRC | 18:06 | |
| corvus | of course, we should really just do a kickstarter for the $100,000 needed to register .zuul tld | 18:06 |
| corvus | fungi: thanks! hopefully other people will help make it look better soon :) | 18:06 |
| *** jpena is now known as jpena|off | 18:07 | |
| fungi | possible i have terrible taste in web design, but aside from adding more content it seems entirely functional already | 18:08 |
| smcginnis | ++ looks good to me! | 18:08 |
| fungi | and the content it does have covers the most important bits i can think of | 18:08 |
| fungi | ooh, page title metadata is boilerplate | 18:09 |
| fungi | "Arcana by HTML5 UP" | 18:09 |
| corvus | fungi: ooh, your first patch! | 18:09 |
| fungi | i'll push a patch! | 18:09 |
| fungi | indeed ;) | 18:09 |
| fungi | and a second patch... footer link to where the page source code lives ;) | 18:10 |
| fungi | (unless i'm blind, which i probably am) | 18:10 |
| fungi | luckily i know where we keep all our source code | 18:11 |
| pabelanger | indeed | 18:12 |
| *** e0ne has joined #openstack-infra | 18:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add zuul-website to #openstack-infra gerritbot https://review.openstack.org/538041 | 18:14 |
| *** jamesmcarthur has quit IRC | 18:16 | |
| *** david-lyle has quit IRC | 18:16 | |
| *** armax has joined #openstack-infra | 18:16 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add noop job description https://review.openstack.org/535534 | 18:16 |
| openstackgerrit | Merged openstack-infra/zuul master: pep8: ignore E124 rule https://review.openstack.org/535536 | 18:16 |
| *** jamesmcarthur has joined #openstack-infra | 18:16 | |
| *** yamamoto has quit IRC | 18:17 | |
| pabelanger | Shrews: Just noticed we have some 2+ days ready nodes still locked in nodepool. might be something you are interested in. Trying to see how has looks currently | 18:18 |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** openstackgerrit has joined #openstack-infra | 18:19 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config master: Create 'Review-Priority' for designate repos https://review.openstack.org/295253 | 18:19 |
| *** hashar has joined #openstack-infra | 18:20 | |
| *** jamesmcarthur has quit IRC | 18:21 | |
| *** hashar has quit IRC | 18:21 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config master: Create 'Review-Priority' for designate repos https://review.openstack.org/295253 | 18:21 |
| pabelanger | Shrews: okay, I can see on nl02: Unlocked node 0002238765 for request 200-0002419205 | 18:23 |
| pabelanger | checking zuul logs now | 18:23 |
| *** xarses_ has joined #openstack-infra | 18:24 | |
| openstackgerrit | Merged openstack-infra/zuul master: scheduler: better handle format status error https://review.openstack.org/535539 | 18:24 |
| *** xarses_ has quit IRC | 18:25 | |
| *** xarses_ has joined #openstack-infra | 18:25 | |
| pabelanger | Shrews: http://paste.openstack.org/show/658348/ is related zuul debug logs | 18:27 |
| *** harlowja has quit IRC | 18:29 | |
| *** andreww has joined #openstack-infra | 18:30 | |
| *** slaweq has joined #openstack-infra | 18:32 | |
| *** jbadiapa_ has joined #openstack-infra | 18:33 | |
| *** xarses_ has quit IRC | 18:33 | |
| openstackgerrit | Merged openstack-infra/zuul-website master: Update page title https://review.openstack.org/540076 | 18:34 |
| *** jbadiapa has quit IRC | 18:35 | |
| *** slaweq has quit IRC | 18:38 | |
| *** tosky has quit IRC | 18:38 | |
| pabelanger | Shrews: I'll have to wait for your help, I'm confused to what is happening | 18:38 |
| *** salv-orlando has quit IRC | 18:41 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Convert Arcana files from DOS to UNIX line endings https://review.openstack.org/540082 | 18:41 |
| *** salv-orlando has joined #openstack-infra | 18:41 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Link page source from the index footer https://review.openstack.org/540083 | 18:44 |
| *** dtantsur is now known as dtantsur|afk | 18:45 | |
| corvus | fungi: oh nice, of course we should do that! | 18:45 |
| *** salv-orlando has quit IRC | 18:45 | |
| Shrews | pabelanger: last time we saw that it looked related to zuul semaphore jobs. Coming back from lunch now so will look in a bit. | 18:47 |
| *** kjackal has quit IRC | 18:47 | |
| *** _jbadiapa has joined #openstack-infra | 18:48 | |
| pabelanger | Shrews: Oh, that could be it. | 18:48 |
| *** jbadiapa_ has quit IRC | 18:49 | |
| *** e0ne has quit IRC | 18:49 | |
| *** slaweq has joined #openstack-infra | 18:52 | |
| *** camunoz has joined #openstack-infra | 18:56 | |
| *** slaweq has quit IRC | 18:56 | |
| *** bobh has quit IRC | 18:59 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 18:59 | |
| *** trown|lunch is now known as trown | 19:01 | |
| *** kjackal has joined #openstack-infra | 19:01 | |
| *** camunoz has quit IRC | 19:03 | |
| *** ralonsoh has quit IRC | 19:04 | |
| *** tosky has joined #openstack-infra | 19:04 | |
| *** sshnaidm|off has quit IRC | 19:05 | |
| *** harlowja has joined #openstack-infra | 19:09 | |
| Shrews | pabelanger: does nova have jobs that use the semaphore? | 19:11 |
| *** harlowja_ has joined #openstack-infra | 19:11 | |
| *** agopi has quit IRC | 19:11 | |
| Shrews | pabelanger: but yeah, once you see the "unlocked node for request" message from nodepool, it's pretty certain zuul will have the lock after that | 19:11 |
| pabelanger | Shrews: I am not sure. I can look now | 19:12 |
| *** harlowja has quit IRC | 19:14 | |
| *** agopi has joined #openstack-infra | 19:16 | |
| pabelanger | Shrews: AJaeger: I think we only have semaphore configured for translations, propose-requirements and now release publishing. Which I think nova might run translations? | 19:17 |
| *** yamamoto has joined #openstack-infra | 19:18 | |
| Shrews | pabelanger: zookeeper 'dump' command seems to indicate zuul still holds the lock. Might need corvus to explore past this point | 19:18 |
| *** dprince has quit IRC | 19:19 | |
| pabelanger | ack | 19:19 |
| *** armax_ has joined #openstack-infra | 19:23 | |
| *** armax has quit IRC | 19:23 | |
| *** armax_ is now known as armax | 19:23 | |
| *** david-lyle has joined #openstack-infra | 19:24 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to send speaker announcement email https://review.openstack.org/540099 | 19:25 |
| *** nikhil has quit IRC | 19:26 | |
| *** jamesmcarthur has joined #openstack-infra | 19:27 | |
| corvus | pabelanger, Shrews: if that node id doesn't show up any more places in the logs, i think the next place to look would be to follow that job and item... did something happen to those? | 19:27 |
| *** rlandy is now known as rlandy|brb | 19:28 | |
| *** b_bezak has joined #openstack-infra | 19:28 | |
| *** caphrim007 has joined #openstack-infra | 19:28 | |
| *** yamamoto has quit IRC | 19:29 | |
| *** bobh has joined #openstack-infra | 19:29 | |
| pabelanger | I'm about to grab some lunch, but can dive more into that when return | 19:30 |
| *** olaph has joined #openstack-infra | 19:31 | |
| *** olaph1 has quit IRC | 19:32 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to send speaker announcement email https://review.openstack.org/540099 | 19:34 |
| *** bobh has quit IRC | 19:34 | |
| fungi | smcginnis: so back to the stable job required-projects wierdness... looks like http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n202 is where we got them from and so the variant in projects.yaml isn't overriding that | 19:41 |
| fungi | (third entry in the _inheritance_path list) | 19:42 |
| *** slaweq has joined #openstack-infra | 19:42 | |
| *** slaweq_ has joined #openstack-infra | 19:42 | |
| smcginnis | fungi: So because the job defines required-projects, the repo specific ones in projects.yaml gets ignored? | 19:46 |
| *** slaweq has quit IRC | 19:46 | |
| *** slaweq_ has quit IRC | 19:47 | |
| mtreinish | clarkb, fungi, pabelanger, mordred: when you get a sec can you take a look at: https://review.openstack.org/#/c/534953/ | 19:48 |
| mtreinish | it should be a super quick | 19:48 |
| *** b_bezak has quit IRC | 19:51 | |
| fungi | smcginnis: that's what it's looking like to me. wouldn't mind getting a second opinion | 19:52 |
| smcginnis | fungi: Would that be desired/expected behavior? | 19:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't use run_lock in executor's merger https://review.openstack.org/540112 | 19:52 |
| mordred | fungi, smcginnis: what's the tl;dr on stable job issue? | 19:53 |
| fungi | smcginnis: i'm still a little fuzzy on the inheritance model, so i don't actually know whether that's intended behavior or buggy/unexpected behavior | 19:53 |
| smcginnis | mordred: We have a project that has required-projects listed, but they are not getting pulled in. | 19:53 |
| *** bobh has joined #openstack-infra | 19:53 | |
| *** iyamahat_ has joined #openstack-infra | 19:54 | |
| smcginnis | mordred: Let me see if I can find the links to the failures that led to this. | 19:54 |
| mordred | smcginnis: awesome! | 19:54 |
| fungi | mordred: basically openstack/vmware-nsx wants to override the required projects for build-openrtack-sphinx-docs in project config zuul.d/projects.yaml | 19:54 |
| smcginnis | mordred: So, found this in a periodic stable job: http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/vmware-nsx/stable/ocata/build-openstack-sphinx-docs/cbc7b57/job-output.txt.gz#_2018-02-01_06_31_03_327272 | 19:55 |
| *** lucasagomes is now known as lucas-afk | 19:55 | |
| smcginnis | mordred: Even though that's listed as a required-project for them: https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L17022 | 19:55 |
| fungi | mordred: but instead we're getting the required projects list for it from openstack-zuul-jobs zuul.d/jobs.yaml | 19:55 |
| smcginnis | mordred: Then fungi noticed none of those are getting pulled in, but things like horizon and neutron are. | 19:55 |
| fungi | so the job definition includes required-projects and the variant isn't successfully extending/replacing that | 19:56 |
| *** iyamahat has quit IRC | 19:56 | |
| * mordred looking at al th ethings | 19:57 | |
| fungi | or rather, sounds like maybe it _is_ succeeding at doing that when the job is run against master but not for runs against stable'* branches? | 19:57 |
| fungi | smcginnis: ^ if so, then sounds like a bug to me | 19:57 |
| smcginnis | fungi: You're right. Because this isn't failing on master. | 19:57 |
| mordred | I believe this may have something to do with the job coming from a project-template | 19:58 |
| fungi | so could be a regression in the recent changes to implied branch matchers maybe? | 19:58 |
| fungi | ahh | 19:58 |
| mordred | (looking at the inheritance path here: http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/vmware-nsx/stable/ocata/build-openstack-sphinx-docs/cbc7b57/zuul-info/inventory.yaml | 19:58 |
| mordred | BUT - I'm still in 'thinking out loud' mode | 19:59 |
| *** rlandy|brb is now known as rlandy | 19:59 | |
| *** bobh has quit IRC | 19:59 | |
| fungi | sure, i lack the depth of understanding here to do much more than wildly speculate | 19:59 |
| smcginnis | mordred: The periodic-stable-jobs pull them in here: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n683 | 20:00 |
| *** rkukura has quit IRC | 20:00 | |
| *** felipemonteiro has joined #openstack-infra | 20:00 | |
| pabelanger | infra-root: I was reminded of installation of internet tomorrow at house, so I'll be offline tomorrow until that has been completed. | 20:01 |
| fungi | thanks for the heads up, pabelanger! | 20:01 |
| fungi | happy internets! | 20:01 |
| smcginnis | mordred: Oh, I had the line wrong there. They are using the -neutron flavor just below there: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n699 | 20:02 |
| smcginnis | Which also has required-projects defined. | 20:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear master: Make workers admin command py3 safe https://review.openstack.org/540116 | 20:02 |
| mordred | smcginnis, fungi: so - to summarize - the issue is that in the periodic stable jobs, vmware-nsx does not have the correct required-projects for build-openstack-sphinx-docs yes? | 20:02 |
| smcginnis | mordred: Correct | 20:02 |
| fungi | mordred: that's what i'm seeing at least | 20:02 |
| mordred | yes. I agree- that is the correct behavior based on these files | 20:03 |
| mtreinish | mordred: thanks | 20:03 |
| smcginnis | mordred: What is the correct behavior? | 20:03 |
| mordred | smcginnis, fungi: they are using periodic-stable-jobs-neutron which does not define networking-l2gw | 20:03 |
| mordred | they defined networking-l2gw as a required project in the check pipeline in project-config | 20:04 |
| mordred | but it is not defined in periodic-stable | 20:04 |
| *** dprince has joined #openstack-infra | 20:04 | |
| *** bobh has joined #openstack-infra | 20:04 | |
| smcginnis | Ah! | 20:04 |
| fungi | aha, yes that explains the difference... | 20:04 |
| fungi | so just need to add another pipeline override | 20:04 |
| smcginnis | mordred: So they need to define a periodic-stable pipeline in project-config with those requirements? | 20:04 |
| mordred | sometimes a fresh set of eyes is nice | 20:04 |
| smcginnis | ++ | 20:04 |
| fungi | absolutely. i got very tunnel-visiony there | 20:04 |
| fungi | thanks! | 20:05 |
| mordred | they have periodic-stable in project-config already - it just doesn't have an override for the docs job | 20:05 |
| smcginnis | Oh, surenuf, there it is. | 20:05 |
| mordred | also - we should keep track of this particular use-case and the confusion around it and what we'd like to get done .. I think it's a good example for us to keep in mind when we discuss adding the ability to have a *project* define required-projects and not just a job or a variant | 20:06 |
| smcginnis | Which explains why their openstack-tox-py27 job does not fail in the same way. | 20:06 |
| mordred | smcginnis: ++ | 20:06 |
| fungi | yep | 20:06 |
| smcginnis | mordred: That would seem to be cleaner and less error prone to define it at the project level. | 20:06 |
| fungi | i completely spaced on the fact that i was looking at a check pipeline variant for a job which ran strangely in a different pipeline | 20:07 |
| mordred | yah. it's a thought that's comeup a few times - but we haven't had the space yet to consider it from all angles | 20:07 |
| smcginnis | I suppose it could be a problem if there are different required-projects with older stable branches maybe. | 20:07 |
| *** b_bezak has joined #openstack-infra | 20:07 | |
| *** slaweq has joined #openstack-infra | 20:08 | |
| corvus | that is a good angle to consider indeed :) | 20:09 |
| *** bobh has quit IRC | 20:09 | |
| *** slaweq_ has joined #openstack-infra | 20:09 | |
| dmsimard|afk | pabelanger: yes Shrews reminded me about those two instances earlier this week.. it's not a good time for me this week and I won't be able to troubleshoot why that job is failing | 20:10 |
| dmsimard|afk | feel free to release them and I'll re-check when I have the time to look into it | 20:10 |
| dmsimard|afk | it's blocking a stack of patches, the review number is in the hold comment | 20:10 |
| *** camunoz has joined #openstack-infra | 20:11 | |
| pabelanger | dmsimard|afk: okay, will do that | 20:11 |
| *** b_bezak has quit IRC | 20:12 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed CSV export format issues https://review.openstack.org/540118 | 20:13 |
| *** d0ugal has quit IRC | 20:13 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Fix vmware-nsx stable-periodic docs job https://review.openstack.org/540119 | 20:14 |
| smcginnis | fungi, mordred: Thanks for your help figuring that out. | 20:14 |
| *** slaweq_ has quit IRC | 20:14 | |
| fungi | smcginnis: sorry i totally spaced on the actual problem right in front of me :/ | 20:14 |
| *** agopi has quit IRC | 20:14 | |
| *** agopi has joined #openstack-infra | 20:15 | |
| smcginnis | fungi: I missed it too, but you really helped tracking down all the pieces. | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| *** bobh has joined #openstack-infra | 20:16 | |
| fungi | well, the _inheritance_path is where the magic goodies are. i'm just surprised i remembered to look there now that we have it | 20:16 |
| smcginnis | :) | 20:16 |
| smcginnis | Hmm, OK, new one. :) | 20:17 |
| smcginnis | requestsexceptions was released a few hours ago. | 20:17 |
| smcginnis | It's present on https://pypi.python.org/pypi/requestsexceptions/1.4.0 | 20:17 |
| smcginnis | But a u-c patch testing it just failed a few minutes ago: http://logs.openstack.org/05/540105/2/check/cross-cinder-py27/303683a/job-output.txt.gz#_2018-02-01_20_12_49_401828 | 20:17 |
| smcginnis | Pypi mirror out of sync? | 20:17 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed CSV export format issues https://review.openstack.org/540118 | 20:18 |
| *** jamesmcarthur has quit IRC | 20:20 | |
| *** jamesmcarthur has joined #openstack-infra | 20:20 | |
| *** bobh has quit IRC | 20:21 | |
| *** SumitNaiksatam has quit IRC | 20:22 | |
| mgagne | pabelanger: once there are no more nodes in "in use" state in inap-mtl01, do you allow me to reboot compute nodes which still have instances on them? (except mirror which has been live migrated already) | 20:23 |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** jamesmcarthur has joined #openstack-infra | 20:24 | |
| mgagne | pabelanger: there is a "fun" bug in Glance v1. When Nova tries to get details on a deleted image, API returns 500, crashing live migration process and it looks you can't enforce Glance v2 in Nova Mitaka, v1 is default :-/ | 20:24 |
| *** d0ugal has joined #openstack-infra | 20:24 | |
| pabelanger | mgagne: yah, I think once they are 0, you are safe to proceed. | 20:24 |
| openstackgerrit | Merged openstack-infra/zuul master: Merger: Stop storing zuul references https://review.openstack.org/540033 | 20:25 |
| fungi | smcginnis: when in doubt... http://mirror.dfw.rax.openstack.org/pypi/last-modified | 20:25 |
| fungi | looks ~6 hours stale to me | 20:25 |
| *** hemna_ has quit IRC | 20:25 | |
| mgagne | pabelanger: thanks, almost done, waiting for last "in use" nodes to be deleted | 20:25 |
| pabelanger | ack | 20:25 |
| smcginnis | fungi: That's not normal operating parameters for that sync, right? | 20:26 |
| smcginnis | I did verify it is not there: http://mirror.iad.rax.openstack.org/pypi/simple/requestsexceptions/ | 20:26 |
| fungi | smcginnis: right, we usually update it every ~5 minutes. i'm taking a look at the update logs | 20:26 |
| fungi | NameError: global name 'request_url' is not defined | 20:27 |
| fungi | that's not good | 20:27 |
| mordred | fungi: that doesn't seem good at all - did bandersnatch update? | 20:29 |
| fungi | no, i think we have a latent bug in our script | 20:29 |
| mordred | fantastic | 20:29 |
| *** eharney has quit IRC | 20:29 | |
| fungi | mordred: http://git.openstack.org/cgit/openstack-infra/puppet-bandersnatch/tree/files/run_bandersnatch.py#n68 uses an undefined python variable | 20:30 |
| smcginnis | Whoops | 20:30 |
| fungi | checking for any recent regression there | 20:31 |
| *** bobh has joined #openstack-infra | 20:31 | |
| mordred | fungi: heh. how did that ever work? | 20:31 |
| fungi | last time that script was touched was nearly two years ago | 20:31 |
| fungi | bug introduced by https://review.openstack.org/319449 | 20:32 |
| mordred | yah. wow. that's ... I wonder why it worked and why it's just now ... | 20:32 |
| mordred | oh - it'll only get hit when there are stale urls | 20:32 |
| fungi | i'm guessing we just haven't caught this condition previously | 20:32 |
| fungi | right, that | 20:33 |
| fungi | and will probably "fix itself" when fastly expires them | 20:33 |
| mordred | well - it seems like s/request_url/stale_url/ will also fix the script | 20:33 |
| openstackgerrit | Merged openstack-infra/zuul master: Merger: retry network operations https://review.openstack.org/539356 | 20:33 |
| fungi | yeah, i have a patch on the way | 20:33 |
| mordred | ossum. I have a review to go with it | 20:33 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-bandersnatch master: Use the correct name for the stale_url variable https://review.openstack.org/540129 | 20:34 |
| fungi | i'm going to hand-patch that ^ on the server to make sure it's working as intended | 20:35 |
| *** bobh has quit IRC | 20:36 | |
| *** rfolco|ruck is now known as rfolco|ruck|off | 20:37 | |
| fungi | looks like bandersnatch got a large update to chew on though, since its last pulse from 19:20 is still underway | 20:37 |
| corvus | sorry 'bout that | 20:39 |
| fungi | corvus: statute of limitations expired on that one ;) | 20:40 |
| *** bobh has joined #openstack-infra | 20:40 | |
| *** slaweq_ has joined #openstack-infra | 20:41 | |
| *** slaweq has quit IRC | 20:41 | |
| mordred | corvus: yah - I'm pretty impressed it took 2 years for that to surface :) | 20:42 |
| *** dbecker has quit IRC | 20:42 | |
| *** dbecker has joined #openstack-infra | 20:43 | |
| corvus | well, it's no meltdown. i'll try harder next time. | 20:43 |
| openstackgerrit | Merged openstack-infra/gear master: Make workers admin command py3 safe https://review.openstack.org/540116 | 20:44 |
| corvus | we'll want a gear release for that ^ | 20:44 |
| corvus | shall i make that now? | 20:44 |
| mordred | corvus: ++ | 20:44 |
| fungi | yes please | 20:45 |
| corvus | looks like 0.11.1 | 20:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not submit min-ready requests with min-ready 0 https://review.openstack.org/540133 | 20:45 |
| fungi | sounds right | 20:45 |
| *** slaweq_ has quit IRC | 20:45 | |
| Shrews | ^^ super easy review there | 20:45 |
| *** bobh has quit IRC | 20:45 | |
| corvus | gear 0.11.1 pushed | 20:46 |
| AJaeger | smcginnis: I think we need to fix tox_install.sh for vmware-nsx - backport from master to the branch | 20:47 |
| AJaeger | pabelanger: nova runs translations, yes | 20:48 |
| *** slaweq has joined #openstack-infra | 20:48 | |
| *** bobh has joined #openstack-infra | 20:48 | |
| smcginnis | AJaeger: ack | 20:48 |
| AJaeger | fungi, smcginnis, you're on the wrong track. mordred and myself worked on the PTI changs for docs. The periodic job should be the same as the normal build-docs. If either fails, backport from master branch what fixes it ;) | 20:49 |
| fungi | AJaeger: well, in this case the project had required-projects overridden for that job in check but not in periodic-stable | 20:50 |
| fungi | so it actually _wasn't_ being run the same way in the two pipelines | 20:50 |
| *** baoli has joined #openstack-infra | 20:50 | |
| smcginnis | AJaeger: I fixed a lot of stable branches by backporting the PTI change, but this one does look different. | 20:50 |
| AJaeger | fungi: oh, indeed - that shouldn't be needed anymore | 20:50 |
| smcginnis | Or more. | 20:50 |
| AJaeger | smcginnis, fungi: ok, I might be wrong ;( - need to dig deeper... | 20:51 |
| *** daidv has quit IRC | 20:51 | |
| AJaeger | smcginnis: let me push a test change to vmware-nsx to debug something | 20:51 |
| *** baoli has quit IRC | 20:52 | |
| *** daidv has joined #openstack-infra | 20:53 | |
| *** bobh has quit IRC | 20:53 | |
| *** Thaelim has joined #openstack-infra | 20:53 | |
| AJaeger | smcginnis, fungi https://review.openstack.org/540136 - that build vmware-nsx without required projects. Let's see what happens - and then I'll dig into fixing it... | 20:54 |
| smcginnis | AJaeger: Ah, you're saying the required-projects shouldn't be needed in the first place because docs jobs should be able to run without installing everything, right? | 20:56 |
| AJaeger | smcginnis: yes, exactly. | 20:57 |
| fungi | even better! | 21:00 |
| * AJaeger just does not remember what we did for other repos, might need either mordred's help or a fresh mind | 21:01 | |
| *** daidv has quit IRC | 21:01 | |
| smcginnis | AJaeger: I backported several of those changes, so I can probably look at doing that. | 21:01 |
| *** priteau has quit IRC | 21:03 | |
| AJaeger | smcginnis: let's first check master - if you have an idea, feel free to use my change for testing... | 21:03 |
| *** daidv has joined #openstack-infra | 21:03 | |
| *** bobh has joined #openstack-infra | 21:03 | |
| *** yamahata has quit IRC | 21:06 | |
| *** iyamahat_ has quit IRC | 21:06 | |
| weshay | anything going on w/ the various cloud providers today? I'm seeing a lot of timeouts today | 21:06 |
| weshay | elastic-recheck-query 1686542.yaml doesn't show a ton of ooo jobs, but just about every job is timing out | 21:07 |
| *** bobh has quit IRC | 21:08 | |
| AJaeger | so, log files are at http://logs.openstack.org/36/540136/1/check/vmware-nsx-docs/652097f/ - and this did not fail at all. It throws warnings for autodocs but that's it. | 21:08 |
| AJaeger | smcginnis: tools/tox_install.sh is different between ocata and master | 21:09 |
| *** sshnaidm|off has joined #openstack-infra | 21:09 | |
| * AJaeger calls it a day now, if anybody wants to take it over, please do. Will read backscroll tomorrow... | 21:09 | |
| smcginnis | AJaeger: Thanks, I'll try to get to it yet today. | 21:10 |
| *** daidv has quit IRC | 21:14 | |
| *** jamesmcarthur has quit IRC | 21:14 | |
| *** daidv has joined #openstack-infra | 21:16 | |
| *** bobh has joined #openstack-infra | 21:16 | |
| *** eharney has joined #openstack-infra | 21:17 | |
| *** jpena|off is now known as jpena | 21:18 | |
| *** ianychoi_ has joined #openstack-infra | 21:21 | |
| *** bobh has quit IRC | 21:21 | |
| *** dprince has quit IRC | 21:22 | |
| *** ianychoi has quit IRC | 21:24 | |
| *** slaweq has quit IRC | 21:24 | |
| *** daidv has quit IRC | 21:25 | |
| *** kgiusti has left #openstack-infra | 21:26 | |
| *** daidv has joined #openstack-infra | 21:28 | |
| *** bobh has joined #openstack-infra | 21:31 | |
| *** Goneri has quit IRC | 21:32 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: Always save the logs from tag-releases https://review.openstack.org/540147 | 21:33 |
| *** aeng has joined #openstack-infra | 21:35 | |
| *** daidv has quit IRC | 21:36 | |
| *** bobh has quit IRC | 21:37 | |
| *** bobh has joined #openstack-infra | 21:38 | |
| *** daidv has joined #openstack-infra | 21:38 | |
| *** apevec has joined #openstack-infra | 21:43 | |
| *** bobh has quit IRC | 21:43 | |
| apevec | looks like zuulv3.openstack.org points to some old inactive machine, could it be changed to CNAME zuul.openstack.org ? | 21:43 |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch master: Use the correct name for the stale_url variable https://review.openstack.org/540129 | 21:44 |
| apevec | (thanks to FF address bar's long memory...) | 21:44 |
| *** daidv has quit IRC | 21:44 | |
| fungi | apevec: maybe? it was only ever a http 302 _temporary_ redirect from zuul.o.o | 21:45 |
| fungi | zuul.openstack.org has remained the canonical name for the service | 21:45 |
| apevec | ah, dunno, my firefox remembered it | 21:46 |
| apevec | could also just remove that record | 21:46 |
| apevec | zuulv3.openstack.org has address 104.130.246.67 | 21:46 |
| fungi | oh, yeah we should do that if we've deleted the old server, for sure | 21:46 |
| fungi | looks like the server is still in existence though. probably time to delete it anyway | 21:47 |
| apevec | yes early spring cleanup! | 21:47 |
| *** slaweq has joined #openstack-infra | 21:47 | |
| fungi | infra-root: anyone object to removing zuulv3.o.o now that zuul01.o.o has been in production for over 2 weeks? | 21:47 |
| fungi | anyone have any old data on it they want preserved? | 21:48 |
| clarkb | I have none and say go for it | 21:48 |
| fungi | or should we hold onto it a little longer for historical service logs? | 21:48 |
| corvus | fungi: ok to delete from me | 21:49 |
| mgagne | so there is a node stuck in deleting state, anything abnormal? http://grafana.openstack.org/dashboard/db/nodepool-inap | 21:51 |
| *** salv-orlando has joined #openstack-infra | 21:51 | |
| *** slaweq has quit IRC | 21:51 | |
| mgagne | and also still waiting for last 3 nodes to delete, seems to be taking forever | 21:52 |
| *** jpena is now known as jpena|off | 21:52 | |
| *** armaan has joined #openstack-infra | 21:53 | |
| smcginnis | I'm watching a couple jobs that are just sitting out there too. | 21:53 |
| *** bobh has joined #openstack-infra | 21:53 | |
| smcginnis | Hasn't even updated with their jobs yet. | 21:53 |
| mordred | fungi: ok from me | 21:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Report zuul failures on github PRs to MySQL https://review.openstack.org/540156 | 21:58 |
| mordred | corvus, fungi, pabelanger: ^^ | 21:58 |
| *** bobh has quit IRC | 21:59 | |
| *** threestrands has joined #openstack-infra | 21:59 | |
| *** daidv has joined #openstack-infra | 22:00 | |
| *** threestrands has quit IRC | 22:00 | |
| *** threestrands has joined #openstack-infra | 22:00 | |
| clarkb | its 2 oclock and this is the first time I'm managing to get a keyboard since like 8:30 :( | 22:00 |
| *** trown is now known as trown|outtypewww | 22:01 | |
| clarkb | smcginnis: that is normal ish when waiting for the nodes to be assigned (its no longer a fifo like old nodepool) | 22:01 |
| clarkb | smcginnis: eventually they should get nodes and start running though | 22:01 |
| *** slaweq has joined #openstack-infra | 22:01 | |
| smcginnis | clarkb: OK. Just looks a little odd that it's been out there for 30 minutes with only 75 in queue. I'll be patient. | 22:02 |
| clarkb | mgagne: http://paste.openstack.org/show/658360/ is what I see | 22:02 |
| clarkb | mgagne: the ready nodes are all more than 2 days old and I think we may need to delete them? | 22:03 |
| clarkb | mgagne: I wonder if grafana is out of sync on that 1 deleitng now | 22:03 |
| clarkb | *1 deleting node | 22:03 |
| mgagne | clarkb: I don't know, I have no idea about the matching instance UUIDs | 22:04 |
| fungi | #status log deleted zuulv3.openstack.org and corresponding dns records now that zuul01.openstack.org has been in production for two weeks | 22:05 |
| openstackstatus | fungi: finished logging | 22:05 |
| *** slaweq has quit IRC | 22:06 | |
| clarkb | corvus: what does ready and locked imply? that zuul intends on using the instance? | 22:06 |
| clarkb | I wonder if the zuul restart form earlier in the week is what has orphaned these though | 22:07 |
| mgagne | is there a way to see running jobs and their instances? | 22:07 |
| clarkb | mgagne: there is no direct way no. | 22:08 |
| mgagne | why not? =) | 22:08 |
| mgagne | would be a very useful info imo | 22:08 |
| clarkb | its not something we need frequently I don't think? like you get all the node info in the job logs so its only when you want the info upfront | 22:09 |
| clarkb | I'm sure it could be implemented | 22:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Executor: Don't start too many jobs at once https://review.openstack.org/540162 | 22:09 |
| clarkb | zuul knows what the instances are that it got from the node request so in theory can expose that via the zuul web dashboard | 22:09 |
| clarkb | its just not there | 22:09 |
| mgagne | for now, should I assume all jobs are completed and grafana is just out of sync? | 22:09 |
| clarkb | I think it is possible the three in-use instances from my paste are running jobs | 22:10 |
| corvus | clarkb: yes, ready+locked means zuul intends to use it. or it has leaked them. pabelanger thought maybe we had leaked some earlier and was looking into it. a zuul restart can not cause this -- a restart releases all locks. | 22:10 |
| clarkb | corvus: ok I'll leave those alone | 22:11 |
| clarkb | mgagne: let me check really quickly if I can find any of those in use as actually in use | 22:11 |
| *** daidv has quit IRC | 22:12 | |
| clarkb | mgagne: 2018-02-01 15:27:44,227 DEBUG zuul.AnsibleJob: [build: 28a3a26c97164d419aac0b9bb9110022] Ansible output: b' "ansible_nodename": "centos-7-inap-mtl01-0002326341",' is the lsat log for the first one on ze06 | 22:13 |
| fungi | smcginnis: http://mirror.dfw.rax.openstack.org/pypi/last-modified is showing a relatively recent update stamp now, so hopefully the earlier failure is no longer an issue (it's taking quite a while to run bandersnatch lately though, looks like, so we may need to look into that as well) | 22:13 |
| *** daidv has joined #openstack-infra | 22:13 | |
| clarkb | mgagne: that looks like maybe the intsance ran into the problem we are trying to address. Considering that I think it is probably safe to proceed on your end and those jobs will either fial due to timeouts or rereun due to broken ssh connections | 22:13 |
| fungi | i need to disappear in about 5 minutes but may be able to dig deeper on bandersnatch slowness later tonight if nobody else gets curious before then | 22:13 |
| smcginnis | fungi: Thanks, I did verify requestsexceptions is now out there. | 22:14 |
| fungi | awesome | 22:15 |
| *** bobh has joined #openstack-infra | 22:15 | |
| mgagne | clarkb: ok, rebooting all remaining nodes | 22:16 |
| *** bobh has quit IRC | 22:18 | |
| *** bobh has joined #openstack-infra | 22:18 | |
| mgagne | interesting "Service started deleting the instance during the previous run, but did not finish. Restarting the deletion now." | 22:18 |
| mgagne | could be the stuck one | 22:18 |
| openstackgerrit | Merged openstack-infra/nodepool master: Do not submit min-ready requests with min-ready 0 https://review.openstack.org/540133 | 22:19 |
| *** mriedem is now known as mriedem_parent | 22:20 | |
| *** hongbin has quit IRC | 22:23 | |
| *** salv-orl_ has joined #openstack-infra | 22:28 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:28 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config master: Revert "Temporary disable inap-mtl01 region for maintenance" https://review.openstack.org/540167 | 22:28 |
| clarkb | mgagne: ^ you ready for that to go in now? | 22:29 |
| *** tpsilva has quit IRC | 22:29 | |
| mgagne | clarkb: yes, all compute nodes were updated | 22:30 |
| *** dave-mccowan has quit IRC | 22:30 | |
| mordred | clarkb: have a sec to +A a one-liner? https://review.openstack.org/#/c/540156/ | 22:30 |
| *** salv-orlando has quit IRC | 22:31 | |
| *** felipemonteiro has quit IRC | 22:32 | |
| clarkb | mordred: can you review mgagne ^ change above? | 22:33 |
| mordred | clarkb: yup | 22:33 |
| *** wolverin_ has joined #openstack-infra | 22:38 | |
| *** wolverineav has quit IRC | 22:39 | |
| *** iyamahat has joined #openstack-infra | 22:39 | |
| *** wolverin_ has quit IRC | 22:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Report zuul failures on github PRs to MySQL https://review.openstack.org/540156 | 22:45 |
| clarkb | mordred: ^ | 22:46 |
| *** salv-orl_ has quit IRC | 22:49 | |
| *** salv-orlando has joined #openstack-infra | 22:49 | |
| *** salv-orlando has quit IRC | 22:53 | |
| *** edmondsw has quit IRC | 22:56 | |
| *** eharney has quit IRC | 22:56 | |
| *** jcoufal_ has quit IRC | 22:58 | |
| *** yamahata has joined #openstack-infra | 22:58 | |
| *** bhuvan has joined #openstack-infra | 22:58 | |
| *** bhuvan has joined #openstack-infra | 22:59 | |
| *** gyee has quit IRC | 22:59 | |
| *** bhuvan has quit IRC | 22:59 | |
| *** priteau has joined #openstack-infra | 23:03 | |
| *** priteau_ has joined #openstack-infra | 23:07 | |
| *** priteau has quit IRC | 23:08 | |
| *** priteau_ has quit IRC | 23:11 | |
| *** apevec has quit IRC | 23:12 | |
| *** olaph1 has joined #openstack-infra | 23:12 | |
| *** bobh has quit IRC | 23:13 | |
| *** olaph has quit IRC | 23:14 | |
| *** wolverineav has joined #openstack-infra | 23:14 | |
| *** felipemonteiro_ has quit IRC | 23:18 | |
| *** inc0 has quit IRC | 23:22 | |
| *** inc0 has joined #openstack-infra | 23:22 | |
| *** esberglu has quit IRC | 23:24 | |
| *** rlandy is now known as rlandy|biab | 23:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Temporary disable inap-mtl01 region for maintenance" https://review.openstack.org/540167 | 23:29 |
| *** StaceyF has quit IRC | 23:29 | |
| *** r-daneel has quit IRC | 23:35 | |
| *** wolverineav has quit IRC | 23:36 | |
| *** esberglu has joined #openstack-infra | 23:36 | |
| *** stakeda has joined #openstack-infra | 23:36 | |
| *** wolverineav has joined #openstack-infra | 23:39 | |
| *** efried has quit IRC | 23:43 | |
| *** wolverineav has quit IRC | 23:45 | |
| *** esberglu_ has joined #openstack-infra | 23:45 | |
| *** esberglu has quit IRC | 23:47 | |
| *** salv-orlando has joined #openstack-infra | 23:50 | |
| *** esberglu_ has quit IRC | 23:51 | |
| *** agopi_ has joined #openstack-infra | 23:53 | |
| *** salv-orlando has quit IRC | 23:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Executor: Don't start too many jobs at once https://review.openstack.org/540162 | 23:55 |
| *** agopi has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!