| *** aeng has quit IRC | 00:04 | |
| *** germs has joined #openstack-infra | 00:05 | |
| *** germs has quit IRC | 00:05 | |
| *** germs has joined #openstack-infra | 00:05 | |
| *** odyssey4me has quit IRC | 00:08 | |
| *** odyssey4me has joined #openstack-infra | 00:08 | |
| *** germs has quit IRC | 00:10 | |
| *** jbadiapa_ has joined #openstack-infra | 00:10 | |
| *** pahuang has quit IRC | 00:11 | |
| *** jbadiapa has quit IRC | 00:12 | |
| *** inc0 has joined #openstack-infra | 00:14 | |
| *** esberglu has quit IRC | 00:15 | |
| *** esberglu has joined #openstack-infra | 00:16 | |
| *** aeng has joined #openstack-infra | 00:17 | |
| *** esberglu has quit IRC | 00:20 | |
| *** HeOS has quit IRC | 00:22 | |
| *** pahuang has joined #openstack-infra | 00:23 | |
| *** tosky has quit IRC | 00:23 | |
| *** yamamoto has joined #openstack-infra | 00:28 | |
| *** caphrim007_ has joined #openstack-infra | 00:32 | |
| *** lbragstad has quit IRC | 00:33 | |
| *** caphrim007 has quit IRC | 00:33 | |
| *** yamamoto has quit IRC | 00:34 | |
| *** eernst has quit IRC | 00:34 | |
| *** caphrim007_ has quit IRC | 00:37 | |
| *** caphrim007 has joined #openstack-infra | 00:38 | |
| *** inc0 has quit IRC | 00:40 | |
| *** caphrim007 has quit IRC | 00:42 | |
| *** rosmaita has quit IRC | 00:43 | |
| *** caphrim007 has joined #openstack-infra | 00:43 | |
| *** caphrim007 has quit IRC | 00:48 | |
| *** caphrim007 has joined #openstack-infra | 00:49 | |
| *** caphrim007 has quit IRC | 00:54 | |
| *** caphrim007 has joined #openstack-infra | 00:55 | |
| *** hongbin has joined #openstack-infra | 00:57 | |
| *** inc0 has joined #openstack-infra | 00:58 | |
| *** caphrim007 has quit IRC | 00:59 | |
| openstackgerrit | Merged openstack/diskimage-builder master: delete unused module https://review.openstack.org/558167 | 01:04 |
|---|---|---|
| dmsimard | pabelanger: I don't understand https://review.openstack.org/#/c/557948/2/playbooks/base-test/pre.yaml | 01:08 |
| *** armaan has quit IRC | 01:09 | |
| dmsimard | nevermind, needed to look at another patch. | 01:12 |
| *** gyan__ has joined #openstack-infra | 01:13 | |
| *** spzala has quit IRC | 01:14 | |
| *** harlowja has quit IRC | 01:19 | |
| *** cshastri has joined #openstack-infra | 01:20 | |
| *** zhangfei has joined #openstack-infra | 01:25 | |
| *** yamamoto has joined #openstack-infra | 01:30 | |
| *** armaan has joined #openstack-infra | 01:35 | |
| *** yamamoto has quit IRC | 01:35 | |
| *** inc0 has quit IRC | 01:35 | |
| *** nicolasbock has quit IRC | 01:36 | |
| *** wolverineav has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| *** jamesmcarthur has quit IRC | 01:47 | |
| *** VW has joined #openstack-infra | 01:48 | |
| *** eernst has joined #openstack-infra | 01:49 | |
| *** inc0 has joined #openstack-infra | 01:50 | |
| *** jogo has quit IRC | 01:58 | |
| *** VW has quit IRC | 01:58 | |
| *** germs has joined #openstack-infra | 02:06 | |
| *** germs has quit IRC | 02:06 | |
| *** germs has joined #openstack-infra | 02:06 | |
| *** eernst has quit IRC | 02:10 | |
| *** inc0 has quit IRC | 02:10 | |
| *** germs has quit IRC | 02:11 | |
| *** wolverineav has quit IRC | 02:11 | |
| *** wolverineav has joined #openstack-infra | 02:11 | |
| *** ykarel|afk has joined #openstack-infra | 02:16 | |
| *** inc0 has joined #openstack-infra | 02:17 | |
| *** aeng has quit IRC | 02:18 | |
| *** wolverineav has quit IRC | 02:18 | |
| *** aeng has joined #openstack-infra | 02:18 | |
| *** pahuang has quit IRC | 02:18 | |
| *** lbragstad has joined #openstack-infra | 02:20 | |
| *** felipemonteiro has joined #openstack-infra | 02:22 | |
| *** hongbin has quit IRC | 02:22 | |
| *** dave-mccowan has joined #openstack-infra | 02:24 | |
| *** annp has quit IRC | 02:24 | |
| *** annp has joined #openstack-infra | 02:25 | |
| *** ykarel|afk has quit IRC | 02:29 | |
| *** zhurong has joined #openstack-infra | 02:29 | |
| *** pahuang has joined #openstack-infra | 02:30 | |
| *** rfolco|away is now known as rfolco|ruck | 02:31 | |
| *** yamamoto has joined #openstack-infra | 02:32 | |
| *** jogo has joined #openstack-infra | 02:33 | |
| *** jogo has joined #openstack-infra | 02:33 | |
| *** yamamoto has quit IRC | 02:37 | |
| *** armaan has quit IRC | 02:43 | |
| *** dhajare__ has joined #openstack-infra | 02:43 | |
| *** armaan has joined #openstack-infra | 02:43 | |
| *** psachin has joined #openstack-infra | 02:44 | |
| *** threestrands has quit IRC | 02:47 | |
| *** threestrands has joined #openstack-infra | 02:47 | |
| *** eernst has joined #openstack-infra | 02:47 | |
| *** threestrands has quit IRC | 02:48 | |
| *** threestrands has joined #openstack-infra | 02:49 | |
| *** felipemonteiro has quit IRC | 02:54 | |
| *** hemna_ has quit IRC | 02:59 | |
| *** gagehugo has quit IRC | 02:59 | |
| *** dave-mccowan has quit IRC | 03:01 | |
| *** gagehugo has joined #openstack-infra | 03:03 | |
| dmsimard | infra-root: I'd like to test something low-risk on logs.openstack.org to reproduce (and fix) the ARA middleware performance issues. | 03:04 |
| dmsimard | well, not just performance -- the issue about half loaded reports, too. | 03:04 |
| *** shu-mutow has joined #openstack-infra | 03:04 | |
| dmsimard | I'll send a patch and we can discuss it there I guess | 03:06 |
| *** caphrim007 has joined #openstack-infra | 03:11 | |
| *** caphrim007 has quit IRC | 03:15 | |
| *** inc0 has quit IRC | 03:16 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add a backoff for failed builds https://review.openstack.org/558686 | 03:21 |
| *** lbragstad has quit IRC | 03:25 | |
| *** ramishra has joined #openstack-infra | 03:33 | |
| *** yamamoto has joined #openstack-infra | 03:33 | |
| *** caphrim007 has joined #openstack-infra | 03:34 | |
| dmsimard | logan-: pretty sure I found the culprit for the incomplete loaded reports | 03:34 |
| logan- | o rly | 03:35 |
| *** yamamoto has quit IRC | 03:38 | |
| *** rfolco|ruck is now known as rfolco|off | 03:39 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Setup WSGIDaemonProcess, ProcessGroup + ApplicationGroup for logs-dev https://review.openstack.org/558687 | 03:42 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Enable tuned WSGI settings for logs.openstack.org https://review.openstack.org/558688 | 03:42 |
| dmsimard | logan-: ^ orly | 03:42 |
| logan- | ahh makes sense | 03:43 |
| *** inc0 has joined #openstack-infra | 03:43 | |
| dmsimard | see the difference in the process cpu consumption with different configurations for processes https://imgur.com/a/e2YFR | 03:44 |
| *** eernst has quit IRC | 03:44 | |
| *** VW has joined #openstack-infra | 03:45 | |
| dmsimard | hmm for some reason, one of the screenshots was left out | 03:45 |
| *** inc0 has quit IRC | 03:47 | |
| dmsimard | okay, added it | 03:47 |
| *** lpetrut has joined #openstack-infra | 03:49 | |
| *** Sukhdev has joined #openstack-infra | 03:50 | |
| logan- | yup | 03:51 |
| *** hongbin has joined #openstack-infra | 03:56 | |
| *** Sukhdev has quit IRC | 03:56 | |
| *** hongbin has quit IRC | 04:05 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add a backoff for failed builds https://review.openstack.org/558686 | 04:06 |
| *** VW has quit IRC | 04:06 | |
| *** udesale has joined #openstack-infra | 04:07 | |
| *** ykarel|afk has joined #openstack-infra | 04:09 | |
| *** ykarel|afk is now known as ykarel | 04:10 | |
| *** asilenkov has quit IRC | 04:12 | |
| *** asilenkov has joined #openstack-infra | 04:13 | |
| *** caphrim007_ has joined #openstack-infra | 04:21 | |
| *** lpetrut has quit IRC | 04:24 | |
| *** caphrim007 has quit IRC | 04:24 | |
| ianw | mordred: 2018-04-04 04:25:45.078131 | contrFailure from finger client: 'utf-8' codec can't decode byte 0xe2 in position 1023: unexpected end of data | 04:26 |
| ianw | http://zuul.openstack.org/stream.html?uuid=cbfc6ff53e434d1c99777842aed923ff&logfile=console.log | 04:26 |
| ianw | fyi | 04:26 |
| ianw | unicode strikes ... something ... again | 04:26 |
| *** pcaruana has joined #openstack-infra | 04:27 | |
| *** pcaruana has quit IRC | 04:30 | |
| *** olaph1 has joined #openstack-infra | 04:33 | |
| *** yamamoto has joined #openstack-infra | 04:35 | |
| *** olaph has quit IRC | 04:36 | |
| *** pgadiya has joined #openstack-infra | 04:37 | |
| *** olaph has joined #openstack-infra | 04:37 | |
| *** olaph1 has quit IRC | 04:37 | |
| *** threestrands has quit IRC | 04:38 | |
| *** links has joined #openstack-infra | 04:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 04:40 |
| *** links has quit IRC | 04:40 | |
| *** yamamoto has quit IRC | 04:40 | |
| *** lpetrut has joined #openstack-infra | 04:40 | |
| *** tonyb has quit IRC | 04:45 | |
| *** e0ne has joined #openstack-infra | 04:45 | |
| *** links has joined #openstack-infra | 04:46 | |
| *** e0ne has quit IRC | 04:48 | |
| *** yamamoto has joined #openstack-infra | 04:51 | |
| prometheanfire | ianw: we had to make a fix for that too | 04:54 |
| prometheanfire | well, a unicode thing anyway | 04:55 |
| prometheanfire | https://review.openstack.org/558566 | 04:55 |
| *** zhurong has quit IRC | 04:55 | |
| dmsimard | I remember when utf8 chars would crash puppet | 04:57 |
| dmsimard | https://review.openstack.org/#/c/166379/ | 04:58 |
| *** lpetrut has quit IRC | 05:00 | |
| ianw | pabelanger: ok, the main clone of debian arm64 just finished up. it has to do it's thing and release the volume, but hopefully ok | 05:03 |
| *** dtruong2 has joined #openstack-infra | 05:08 | |
| *** dtruong2 has quit IRC | 05:10 | |
| *** harlowja has joined #openstack-infra | 05:16 | |
| *** e0ne has joined #openstack-infra | 05:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Formalise saving of /etc/resolv.conf https://review.openstack.org/558362 | 05:19 |
| *** harlowja has quit IRC | 05:20 | |
| *** eernst has joined #openstack-infra | 05:23 | |
| *** caphrim007_ has quit IRC | 05:24 | |
| *** olaph1 has joined #openstack-infra | 05:26 | |
| *** olaph has quit IRC | 05:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove ansible-role-functional-jobs-fedora-26 for windmill https://review.openstack.org/558578 | 05:27 |
| *** dhajare__ has quit IRC | 05:38 | |
| *** jtomasek has joined #openstack-infra | 05:39 | |
| *** notmyname has quit IRC | 05:40 | |
| *** notmyname has joined #openstack-infra | 05:42 | |
| *** dave-mccowan has joined #openstack-infra | 05:42 | |
| *** jamesmcarthur has joined #openstack-infra | 05:43 | |
| *** jamesmcarthur has quit IRC | 05:48 | |
| *** eernst has quit IRC | 05:50 | |
| *** greghaynes has quit IRC | 05:52 | |
| *** xinni9e has joined #openstack-infra | 05:52 | |
| *** iranzo has joined #openstack-infra | 05:52 | |
| *** kjackal has quit IRC | 05:53 | |
| *** notmyname has quit IRC | 05:54 | |
| *** notmyname has joined #openstack-infra | 05:57 | |
| *** dhajare__ has joined #openstack-infra | 06:01 | |
| *** janki has joined #openstack-infra | 06:06 | |
| *** gmann_ is now known as gmann | 06:06 | |
| *** agopi has quit IRC | 06:07 | |
| *** dave-mccowan has quit IRC | 06:09 | |
| *** hashar has joined #openstack-infra | 06:12 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Restore tracing on exit points of block_device_create_config_file https://review.openstack.org/558358 | 06:13 |
| *** sree_ has joined #openstack-infra | 06:16 | |
| *** sree_ is now known as Guest90995 | 06:16 | |
| *** dsariel has joined #openstack-infra | 06:19 | |
| *** zhipeng has joined #openstack-infra | 06:20 | |
| *** jaosorior has quit IRC | 06:25 | |
| *** aeng has quit IRC | 06:26 | |
| *** e0ne has quit IRC | 06:28 | |
| *** garyk has quit IRC | 06:29 | |
| *** pcichy has joined #openstack-infra | 06:34 | |
| *** shardy has quit IRC | 06:36 | |
| *** shardy has joined #openstack-infra | 06:36 | |
| *** alexchadin has joined #openstack-infra | 06:37 | |
| *** mpjetta has quit IRC | 06:39 | |
| *** dsariel has quit IRC | 06:41 | |
| *** kjackal has joined #openstack-infra | 06:45 | |
| *** mpjetta has joined #openstack-infra | 06:51 | |
| *** aojea has joined #openstack-infra | 06:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 06:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update to Ansible 2.4 https://review.openstack.org/535781 | 06:56 |
| dirk | clarkb: done. there is actually an open review for the bindep/tumbleweed detection problem since september last year waiting for a review.. | 06:58 |
| *** jaosorior has joined #openstack-infra | 06:58 | |
| *** pcaruana has joined #openstack-infra | 07:01 | |
| *** imacdonn has quit IRC | 07:04 | |
| *** imacdonn has joined #openstack-infra | 07:04 | |
| gmann | fungi: seems like https://ask.openstack.org/ is down | 07:11 |
| *** armaan has quit IRC | 07:12 | |
| *** prometheanfire has quit IRC | 07:12 | |
| *** andreas_s has joined #openstack-infra | 07:12 | |
| *** armaan has joined #openstack-infra | 07:12 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tmpdir test https://review.openstack.org/558719 | 07:12 |
| *** prometheanfire has joined #openstack-infra | 07:13 | |
| *** florianf has joined #openstack-infra | 07:14 | |
| *** florianf has quit IRC | 07:14 | |
| *** florianf_ has joined #openstack-infra | 07:14 | |
| ianw | gmann: i think it might just have some periodic issues. working now? | 07:16 |
| gmann | ianw: ohk let me check | 07:16 |
| gmann | ianw: yea its fine now, thanks | 07:16 |
| *** lewo has quit IRC | 07:20 | |
| *** rcernin has quit IRC | 07:23 | |
| *** tesseract has joined #openstack-infra | 07:26 | |
| *** lewo has joined #openstack-infra | 07:29 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tmpdir test https://review.openstack.org/558719 | 07:30 |
| *** olaph has joined #openstack-infra | 07:31 | |
| *** olaph1 has quit IRC | 07:32 | |
| *** amoralej|off is now known as amoralej | 07:34 | |
| *** lpetrut has joined #openstack-infra | 07:36 | |
| *** HeOS has joined #openstack-infra | 07:36 | |
| *** electrofelix has joined #openstack-infra | 07:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update zuul-jobs to use fedora-27 for testing https://review.openstack.org/558579 | 07:42 |
| *** shardy has quit IRC | 07:45 | |
| *** shardy has joined #openstack-infra | 07:45 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make fedora-27 jobs voting https://review.openstack.org/558580 | 07:45 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch openstack-tox-py36 to fedora-27 https://review.openstack.org/558582 | 07:45 |
| *** jpena|off is now known as jpena | 07:46 | |
| *** d0ugal_ has quit IRC | 07:46 | |
| *** d0ugal has joined #openstack-infra | 07:47 | |
| *** d0ugal has quit IRC | 07:47 | |
| *** d0ugal has joined #openstack-infra | 07:47 | |
| *** tosky has joined #openstack-infra | 07:48 | |
| *** prometheanfire has quit IRC | 07:52 | |
| *** prometheanfire has joined #openstack-infra | 07:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 07:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix publishing infra/publications https://review.openstack.org/558727 | 07:57 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update to Ansible 2.4 https://review.openstack.org/535781 | 07:58 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Disable action and lookup plugins from 2.4 https://review.openstack.org/535839 | 07:58 |
| *** alexchadin has quit IRC | 08:02 | |
| *** jpich has joined #openstack-infra | 08:02 | |
| *** garyk has joined #openstack-infra | 08:03 | |
| *** tinwood_ is now known as tinwood | 08:06 | |
| *** germs has joined #openstack-infra | 08:08 | |
| *** germs has quit IRC | 08:08 | |
| *** germs has joined #openstack-infra | 08:08 | |
| *** dsariel has joined #openstack-infra | 08:08 | |
| *** efoley has joined #openstack-infra | 08:08 | |
| *** alexchadin has joined #openstack-infra | 08:12 | |
| *** germs has quit IRC | 08:12 | |
| *** e0ne has joined #openstack-infra | 08:14 | |
| *** garyk has quit IRC | 08:14 | |
| *** garyk has joined #openstack-infra | 08:14 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tmpdir test https://review.openstack.org/558736 | 08:15 |
| *** garyk has quit IRC | 08:15 | |
| *** garyk has joined #openstack-infra | 08:15 | |
| *** garyk has quit IRC | 08:16 | |
| *** lucas-afk is now known as lucasagomes | 08:17 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove ansible-role-functional-fedora-26 jobs https://review.openstack.org/558584 | 08:17 |
| *** garyk has joined #openstack-infra | 08:17 | |
| *** garyk has quit IRC | 08:17 | |
| *** garyk has joined #openstack-infra | 08:17 | |
| *** garyk has quit IRC | 08:18 | |
| *** AJaeger has joined #openstack-infra | 08:18 | |
| *** garyk has joined #openstack-infra | 08:18 | |
| *** garyk has quit IRC | 08:19 | |
| *** garyk has joined #openstack-infra | 08:19 | |
| *** garyk has quit IRC | 08:19 | |
| *** garyk has joined #openstack-infra | 08:20 | |
| AJaeger | infra-root, at least one of our git servers has a broken ara-client and ara-plugins repo. | 08:20 |
| *** garyk has quit IRC | 08:20 | |
| AJaeger | infra-root, running "git clone -vvv https://git.openstack.org/openstack/ara-clients.git" repeatedly, I sometimes get errors like "warning: You appear to have cloned an empty repository." instead of the actual content. | 08:20 |
| *** garyk has joined #openstack-infra | 08:21 | |
| *** garyk has quit IRC | 08:21 | |
| *** garyk has joined #openstack-infra | 08:21 | |
| *** bmace has quit IRC | 08:22 | |
| *** garyk has quit IRC | 08:22 | |
| *** bmace has joined #openstack-infra | 08:22 | |
| *** garyk has joined #openstack-infra | 08:22 | |
| *** garyk has quit IRC | 08:23 | |
| *** garyk has joined #openstack-infra | 08:23 | |
| *** garyk has quit IRC | 08:23 | |
| *** garyk has joined #openstack-infra | 08:24 | |
| *** garyk has quit IRC | 08:24 | |
| *** garyk has joined #openstack-infra | 08:24 | |
| *** garyk has quit IRC | 08:25 | |
| *** shu-mutow has quit IRC | 08:25 | |
| *** garyk has joined #openstack-infra | 08:25 | |
| *** garyk has quit IRC | 08:26 | |
| *** garyk has joined #openstack-infra | 08:26 | |
| frickler | AJaeger: you can talk to the backend servers directly avoiding the load-balancer. doing so shows that http://git08.openstack.org:8080/openstack/ara-clients.git is broken, but I'm not sure how to fix that. | 08:26 |
| *** garyk has quit IRC | 08:27 | |
| *** garyk has joined #openstack-infra | 08:27 | |
| *** garyk has quit IRC | 08:27 | |
| *** garyk has joined #openstack-infra | 08:28 | |
| *** garyk has quit IRC | 08:28 | |
| *** garyk has joined #openstack-infra | 08:28 | |
| lifeless | mtreinish: so I'm trying to do the release. challenges overcome: | 08:29 |
| *** garyk has quit IRC | 08:29 | |
| lifeless | - my hypervisor can't mount my usb key to get at my key. | 08:29 |
| *** garyk has joined #openstack-infra | 08:29 | |
| *** garyk has quit IRC | 08:30 | |
| lifeless | - broken release virtualenvs | 08:30 |
| *** garyk has joined #openstack-infra | 08:30 | |
| lifeless | - broken gpg2 config needed to start that over | 08:30 |
| lifeless | - now working on SSL verification errors on upload.pypi.org :/ | 08:30 |
| *** garyk has quit IRC | 08:31 | |
| lifeless | mtreinish: I sometimes wish the world would stop and we could just hop off | 08:31 |
| *** garyk has joined #openstack-infra | 08:31 | |
| *** garyk has quit IRC | 08:31 | |
| *** garyk has joined #openstack-infra | 08:32 | |
| *** garyk has quit IRC | 08:32 | |
| frickler | #status log git08 is showing broken repos e.g. openstack/ara-clients is empty. placed git.openstack.org into emergency file and removed git08 from the list of backends for haproxy as temporary fix | 08:32 |
| openstackstatus | frickler: finished logging | 08:32 |
| frickler | infra-root: ^^ | 08:32 |
| *** garyk has joined #openstack-infra | 08:32 | |
| *** zhongjun_ has joined #openstack-infra | 08:33 | |
| *** garyk has quit IRC | 08:33 | |
| *** garyk has joined #openstack-infra | 08:33 | |
| *** iyamahat_ has quit IRC | 08:33 | |
| *** garyk has quit IRC | 08:34 | |
| *** garyk has joined #openstack-infra | 08:34 | |
| *** garyk has quit IRC | 08:34 | |
| *** garyk has joined #openstack-infra | 08:35 | |
| *** garyk has quit IRC | 08:35 | |
| *** garyk has joined #openstack-infra | 08:36 | |
| *** garyk has quit IRC | 08:36 | |
| *** garyk has joined #openstack-infra | 08:36 | |
| *** garyk has quit IRC | 08:37 | |
| *** garyk has joined #openstack-infra | 08:37 | |
| *** garyk has quit IRC | 08:38 | |
| *** garyk has joined #openstack-infra | 08:38 | |
| *** garyk has quit IRC | 08:38 | |
| *** garyk has joined #openstack-infra | 08:39 | |
| *** garyk has quit IRC | 08:39 | |
| *** gfidente has joined #openstack-infra | 08:42 | |
| *** gfidente has joined #openstack-infra | 08:42 | |
| AJaeger | thanks, frickler. corvus has been doing some testing on git08 - that might explain it. Still, we need to fix it. | 08:43 |
| AJaeger | frickler: ah, port 8080... | 08:44 |
| ianw | [2018-03-27 11:35:19,942] [9461f597] Rescheduling replication to cgit@git08.openstack.org:/var/lib/git/openstack/ara-clients.git to avoid collision with an in-flight push. | 08:46 |
| ianw | [2018-03-27 11:35:20,942] [9461f597] Replication to cgit@git08.openstack.org:/var/lib/git/openstack/ara-clients.git started... | 08:46 |
| ianw | [2018-03-27 11:35:26,963] [9461f597] Created remote repository: cgit@git08.openstack.org:/var/lib/git/openstack/ara-clients.git | 08:46 |
| ianw | last things in the logs about this | 08:46 |
| ianw | it never completed, afaict | 08:47 |
| ianw | that also might have been around the time it was being fiddled with? | 08:47 |
| frickler | ianw: status log says "2018-03-27 16:21:25 UTC git08 removed from emergency file and added back to git loadbalancer" so that fits the picture | 08:48 |
| *** zhipeng has quit IRC | 08:50 | |
| ianw | frickler: ok, i just uploaded a fake change to get it replicating again ... | 08:51 |
| ianw | [2018-04-04 08:51:07,025] [8b262cab] Replication to cgit@git08.openstack.org:/var/lib/git/openstack/ara-clients.git completed in 1956ms, 1000ms delay, 0 retries | 08:51 |
| ianw | is it ok now? | 08:52 |
| frickler | ianw: AJaeger mentioned two broken repos, there might also be more. also different error now: "warning: remote HEAD refers to nonexistent ref, unable to checkout." | 08:52 |
| lifeless | mtreinish: and a 2 month slow VM clock! | 08:52 |
| *** links has quit IRC | 08:53 | |
| AJaeger | ianw, frickler: please check ara-plugin and ansible-role-k8s-rabbitmq as well | 08:53 |
| AJaeger | ianw: I still cannot clone from it - "warning: remote HEAD refers to nonexistent ref, unable to checkout." | 08:54 |
| AJaeger | ianw: running "git clone -vvv http://git08.openstack.org:8080/openstack/ara-clients.git" | 08:54 |
| ianw | fricker / AJaeger: my inclination is to just delete it, and upload a new change | 08:55 |
| frickler | ianw: my fear is that other repos might also be broken in more subtle ways. so to be on the safe side we'd have to rebuild the node from scratch probably. but for that I'd first like to hear feedback from corvus, maybe he wants to debug further why this broke in the first place | 08:56 |
| *** links has joined #openstack-infra | 08:57 | |
| frickler | ianw: the node should be removed from the lb again, so no immediate impact anymore | 08:57 |
| ianw | frickler: ok, yep happy to leave it alone :) | 09:00 |
| *** sree_ has joined #openstack-infra | 09:05 | |
| *** sree_ is now known as Guest18520 | 09:05 | |
| *** pblaho has joined #openstack-infra | 09:07 | |
| *** Guest90995 has quit IRC | 09:08 | |
| *** edmondsw has joined #openstack-infra | 09:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add gentoo test to nodepool experimental https://review.openstack.org/558761 | 09:13 |
| *** ykarel_ has joined #openstack-infra | 09:14 | |
| AJaeger | frickler: thansk for your investigation last week regarding publications repo - I pushed the change now since I didn't see anything after double checking your findings: https://review.openstack.org/#/c/558727/ | 09:15 |
| *** edmondsw has quit IRC | 09:15 | |
| *** ykarel has quit IRC | 09:17 | |
| *** sree_ has joined #openstack-infra | 09:26 | |
| *** sree_ is now known as Guest55808 | 09:26 | |
| *** panda|bbl is now known as panda | 09:26 | |
| *** Guest18520 has quit IRC | 09:28 | |
| *** Guest55808 has quit IRC | 09:31 | |
| *** alexchadin has quit IRC | 09:35 | |
| *** ykarel__ has joined #openstack-infra | 09:39 | |
| *** ykarel_ has quit IRC | 09:43 | |
| *** yamahata has quit IRC | 09:44 | |
| *** dtantsur|afk is now known as dtantsur | 09:54 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Prefix the project description with RETIRED for retired projects https://review.openstack.org/557166 | 09:54 |
| *** cshastri has quit IRC | 10:04 | |
| *** efoley_ has joined #openstack-infra | 10:05 | |
| *** efoley has quit IRC | 10:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Prefix the project description with RETIRED for retired projects https://review.openstack.org/557166 | 10:05 |
| *** dtruong_ has joined #openstack-infra | 10:05 | |
| *** jtomasek_ has joined #openstack-infra | 10:06 | |
| *** rkukura_ has joined #openstack-infra | 10:07 | |
| *** armaan_ has joined #openstack-infra | 10:07 | |
| *** yamamoto_ has joined #openstack-infra | 10:08 | |
| *** auristor1 has joined #openstack-infra | 10:08 | |
| kashyap | Hi there; strangely sending email to openstack-{dev,operators} is failing for me locally: | 10:09 |
| kashyap | (From my `journalctl`) | 10:09 |
| kashyap | Apr 04 12:07:53 paraplu postfix/smtp[6262]: 59AED3E029C: to=<openstack-dev@lists.openstack.org>, relay=none, delay=4966, delays=4965/0/0.62/0, dsn=4.4.1, status=deferred (connect to lists.openstack.org[2001:4800:780e:510:3bc3:d7f6:ff04:b736]:25: Network is unreachable) | 10:09 |
| * kashyap uses `postfix` locally; and tried to flush it via `postque -f`, still to no avail. | 10:09 | |
| *** annp_ has joined #openstack-infra | 10:09 | |
| *** beekneemech has joined #openstack-infra | 10:10 | |
| kashyap | And this works: | 10:10 |
| kashyap | $> ping -c 2 lists.openstack.org | 10:10 |
| kashyap | PING lists.openstack.org (50.56.173.222) 56(84) bytes of data. | 10:10 |
| kashyap | 64 bytes from lists.openstack.org (50.56.173.222): icmp_seq=1 ttl=52 time=160 ms | 10:10 |
| kashyap | 64 bytes from lists.openstack.org (50.56.173.222): icmp_seq=2 ttl=52 time=157 ms | 10:10 |
| *** markmcd1 has joined #openstack-infra | 10:10 | |
| *** kzaitsev1pi has joined #openstack-infra | 10:11 | |
| *** bradm_ has joined #openstack-infra | 10:11 | |
| *** darvon_ has joined #openstack-infra | 10:11 | |
| *** jgrassle1 has joined #openstack-infra | 10:11 | |
| *** colettecello has joined #openstack-infra | 10:11 | |
| *** lifeless_ has joined #openstack-infra | 10:11 | |
| *** zhangfei_ has joined #openstack-infra | 10:11 | |
| *** aluria` has joined #openstack-infra | 10:12 | |
| *** pcaruana|afk| has joined #openstack-infra | 10:12 | |
| *** krenczewski has joined #openstack-infra | 10:12 | |
| *** pgadiya_ has joined #openstack-infra | 10:13 | |
| *** kjackal has quit IRC | 10:14 | |
| *** gfidente^2nd has joined #openstack-infra | 10:14 | |
| *** rodrigod` has joined #openstack-infra | 10:14 | |
| *** weshay_ has joined #openstack-infra | 10:14 | |
| *** ktsuyuzaki has joined #openstack-infra | 10:14 | |
| *** olivierb- has joined #openstack-infra | 10:14 | |
| *** niedbalski_ has joined #openstack-infra | 10:14 | |
| *** dayou_ has joined #openstack-infra | 10:15 | |
| *** gary_perkins_ has joined #openstack-infra | 10:15 | |
| *** wznoinsk_ has joined #openstack-infra | 10:15 | |
| *** haleyb_ has joined #openstack-infra | 10:15 | |
| *** gfidente has quit IRC | 10:15 | |
| *** jtomasek has quit IRC | 10:15 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** annp has quit IRC | 10:15 | |
| *** rkukura has quit IRC | 10:15 | |
| *** kota_ has quit IRC | 10:15 | |
| *** auristor has quit IRC | 10:15 | |
| *** kzaitsev_pi has quit IRC | 10:15 | |
| *** owalsh has quit IRC | 10:15 | |
| *** bandini has quit IRC | 10:15 | |
| *** gary_perkins has quit IRC | 10:15 | |
| *** bnemec has quit IRC | 10:15 | |
| *** lifeless has quit IRC | 10:15 | |
| *** niedbalski has quit IRC | 10:15 | |
| *** rodrigods has quit IRC | 10:15 | |
| *** gothicmindfood has quit IRC | 10:15 | |
| *** tosky has quit IRC | 10:15 | |
| *** armaan has quit IRC | 10:15 | |
| *** pcaruana has quit IRC | 10:15 | |
| *** pgadiya has quit IRC | 10:15 | |
| *** zhangfei has quit IRC | 10:15 | |
| *** olivierbourdon38 has quit IRC | 10:15 | |
| *** dayou has quit IRC | 10:15 | |
| *** dtruong has quit IRC | 10:15 | |
| *** jrist has quit IRC | 10:15 | |
| *** weshay has quit IRC | 10:15 | |
| *** nhandler has quit IRC | 10:15 | |
| *** haleyb has quit IRC | 10:15 | |
| *** wznoinsk has quit IRC | 10:15 | |
| *** dmsimard has quit IRC | 10:15 | |
| *** lucasagomes has quit IRC | 10:15 | |
| *** bradm has quit IRC | 10:15 | |
| *** aluria has quit IRC | 10:15 | |
| *** darvon has quit IRC | 10:15 | |
| *** jgrassler has quit IRC | 10:15 | |
| *** gary_perkins_ is now known as gary_perkins | 10:15 | |
| *** rkukura_ is now known as rkukura | 10:16 | |
| *** darvon_ is now known as darvon | 10:16 | |
| *** aluria` is now known as aluria | 10:16 | |
| *** aluria has quit IRC | 10:16 | |
| *** aluria has joined #openstack-infra | 10:16 | |
| *** tosky has joined #openstack-infra | 10:16 | |
| *** owalsh has joined #openstack-infra | 10:17 | |
| *** nhandler has joined #openstack-infra | 10:17 | |
| *** dmsimard has joined #openstack-infra | 10:18 | |
| *** lucasagomes has joined #openstack-infra | 10:19 | |
| *** 18VADOPXF has joined #openstack-infra | 10:19 | |
| *** jrist has joined #openstack-infra | 10:20 | |
| *** 18VADOPXF has quit IRC | 10:22 | |
| *** udesale has quit IRC | 10:23 | |
| *** udesale has joined #openstack-infra | 10:24 | |
| *** bandini has joined #openstack-infra | 10:26 | |
| *** zhangfei_ has quit IRC | 10:28 | |
| *** pcaruana|afk| has quit IRC | 10:28 | |
| *** pcaruana|afk| has joined #openstack-infra | 10:28 | |
| *** weshay_ has quit IRC | 10:28 | |
| *** weshay_ has joined #openstack-infra | 10:28 | |
| *** wznoinsk_ has quit IRC | 10:28 | |
| *** wznoinsk_ has joined #openstack-infra | 10:28 | |
| *** haleyb_ has quit IRC | 10:28 | |
| *** haleyb_ has joined #openstack-infra | 10:28 | |
| *** lucasagomes has quit IRC | 10:28 | |
| *** lucasagomes has joined #openstack-infra | 10:28 | |
| *** nicolasbock has joined #openstack-infra | 10:30 | |
| *** zoli is now known as zoli|lunch | 10:31 | |
| AJaeger | kashyap: above is a IPv6 address, below you use IPv4. I can reach lists.o.o via IPv6 as well. Can you? | 10:34 |
| kashyap | AJaeger: Yep; that's the problem | 10:34 |
| kashyap | Just doing this: | 10:34 |
| kashyap | $> grep inet_protocols /etc/postfix/main.cf | 10:34 |
| kashyap | #inet_protocols = all | 10:34 |
| kashyap | inet_protocols = ipv4 | 10:34 |
| kashyap | AJaeger: I can't reach via IPv6 | 10:34 |
| kashyap | $> ping6 -c2 lists.openstack.org | 10:34 |
| kashyap | connect: Network is unreachable | 10:34 |
| AJaeger | kashyap: looks like a local problem since it works for me | 10:35 |
| kashyap | AJaeger: Definitely local | 10:35 |
| AJaeger | kashyap: so, nothing we can help here. | 10:35 |
| kashyap | Yeah; I was just trying to see what I could fix; after a bit of looking around found it | 10:36 |
| *** boden has joined #openstack-infra | 10:37 | |
| *** efoley_ has quit IRC | 10:38 | |
| *** bandini has quit IRC | 10:38 | |
| *** kjackal has joined #openstack-infra | 10:40 | |
| *** alexchadin has joined #openstack-infra | 10:41 | |
| krenczewski | Where I can find example zuul job that test plugin and install this plugin in job and run devstack? | 10:42 |
| *** rcernin has joined #openstack-infra | 10:44 | |
| *** salv-orlando has joined #openstack-infra | 10:44 | |
| AJaeger | krenczewski: I suggest to ask on #openstack-qa since they develop these | 10:45 |
| *** bandini has joined #openstack-infra | 10:45 | |
| krenczewski | AJaeger: OK, will try there, thanks | 10:46 |
| *** annp_ has quit IRC | 10:46 | |
| *** pbourke has quit IRC | 10:47 | |
| *** pbourke has joined #openstack-infra | 10:47 | |
| *** niedbalski_ is now known as niedbalski | 10:52 | |
| *** niedbalski has joined #openstack-infra | 10:52 | |
| *** panda is now known as panda|lunch | 10:58 | |
| *** garyk has joined #openstack-infra | 11:05 | |
| *** olaph1 has joined #openstack-infra | 11:08 | |
| *** olaph has quit IRC | 11:09 | |
| *** gfidente^2nd has quit IRC | 11:11 | |
| *** lucasagomes is now known as lucas-hungry | 11:19 | |
| *** gfidente has joined #openstack-infra | 11:22 | |
| *** gfidente has joined #openstack-infra | 11:22 | |
| *** nicolasbock has quit IRC | 11:26 | |
| *** rosmaita has joined #openstack-infra | 11:26 | |
| *** zoli|lunch is now known as zoli | 11:30 | |
| *** nicolasbock has joined #openstack-infra | 11:31 | |
| *** katkapilatova has joined #openstack-infra | 11:33 | |
| *** olaph has joined #openstack-infra | 11:36 | |
| *** olaph1 has quit IRC | 11:37 | |
| *** gyan__ has quit IRC | 11:38 | |
| *** katkapilatova has quit IRC | 11:40 | |
| *** hu_nokia has joined #openstack-infra | 11:40 | |
| hu_nokia | hello | 11:41 |
| *** ykarel__ has quit IRC | 11:43 | |
| *** rfolco|off is now known as rfolco|ruck | 11:43 | |
| *** anteaya has quit IRC | 11:43 | |
| *** alexchadin has quit IRC | 11:47 | |
| *** cshastri has joined #openstack-infra | 11:47 | |
| *** ldnunes has joined #openstack-infra | 11:48 | |
| *** ykarel has joined #openstack-infra | 11:50 | |
| *** katkapilatova has joined #openstack-infra | 11:51 | |
| *** jpena is now known as jpena|lunch | 11:53 | |
| *** dprince has joined #openstack-infra | 11:55 | |
| *** jlabarre has joined #openstack-infra | 12:00 | |
| *** jcoufal has joined #openstack-infra | 12:03 | |
| dmsimard | frickler, AJaeger: those are new ara repos | 12:05 |
| *** sree_ has joined #openstack-infra | 12:05 | |
| dmsimard | I remember that corvus took git08 off puppet to do some testing with the Zuul cgit. Maybe something related to that ? Or just an odd coincidence | 12:05 |
| *** sree_ is now known as Guest18685 | 12:06 | |
| *** alexchadin has joined #openstack-infra | 12:06 | |
| dmsimard | How did someone find this out btw ? I haven't even submitted anything there yet heh | 12:07 |
| *** katkapilatova has quit IRC | 12:09 | |
| *** Guest18685 has quit IRC | 12:09 | |
| *** germs has joined #openstack-infra | 12:10 | |
| *** germs has quit IRC | 12:10 | |
| *** germs has joined #openstack-infra | 12:10 | |
| *** jbadiapa_ is now known as jbadiapa | 12:10 | |
| *** ldnunes has quit IRC | 12:11 | |
| AJaeger | dmsimard: I have all repos checked out locally - and run into problems with these... | 12:11 |
| *** lucas-hungry is now known as lucasagomes | 12:11 | |
| dmsimard | Oh wow | 12:13 |
| * dmsimard doesn't do that | 12:13 | |
| *** ifat_afek has joined #openstack-infra | 12:13 | |
| dmsimard | Nice find :) | 12:13 |
| AJaeger | dmsimard: helps if you do cross-repo changes... | 12:13 |
| dmsimard | Yeah but I only check out the repos I need/work with | 12:14 |
| *** germs has quit IRC | 12:15 | |
| *** mhayden has left #openstack-infra | 12:15 | |
| *** salv-orlando has quit IRC | 12:18 | |
| *** salv-orlando has joined #openstack-infra | 12:18 | |
| *** salv-orlando has quit IRC | 12:23 | |
| *** odyssey4me has quit IRC | 12:23 | |
| *** odyssey4me has joined #openstack-infra | 12:23 | |
| *** ldnunes has joined #openstack-infra | 12:24 | |
| *** lbragstad has joined #openstack-infra | 12:29 | |
| *** amoralej is now known as amoralej|lunch | 12:29 | |
| *** panda|lunch is now known as panda | 12:31 | |
| *** cshastri has quit IRC | 12:31 | |
| *** jlabarre is now known as rh-jelabarre | 12:33 | |
| *** salv-orlando has joined #openstack-infra | 12:36 | |
| *** adrianreza has quit IRC | 12:37 | |
| *** snapiri- has joined #openstack-infra | 12:38 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add static driver doc to Zuul From Scratch https://review.openstack.org/558802 | 12:38 |
| *** rh-jelabarre is now known as rh-jlabarre | 12:38 | |
| dmsimard | infra-root: please see https://review.openstack.org/#/q/topic:logs-wsgi today to look at performance improvements for loganalyze and ara on logs.o.o | 12:40 |
| *** gcb has quit IRC | 12:40 | |
| *** rh-jlabarre is now known as rh-jelabarre | 12:40 | |
| *** trown|outtypewww is now known as trown | 12:43 | |
| *** hu_nokia has quit IRC | 12:44 | |
| *** jamesmcarthur has joined #openstack-infra | 12:46 | |
| *** edmondsw has joined #openstack-infra | 12:47 | |
| mtreinish | lifeless: heh, it's like things are aligned to block things | 12:48 |
| *** jpena|lunch is now known as jpena | 12:53 | |
| *** shardy is now known as shardy_mtg | 12:58 | |
| *** eharney has joined #openstack-infra | 13:01 | |
| *** psachin has quit IRC | 13:04 | |
| *** Goneri has joined #openstack-infra | 13:05 | |
| *** weshay_ is now known as weshay | 13:06 | |
| *** mriedem has joined #openstack-infra | 13:07 | |
| *** VW has joined #openstack-infra | 13:08 | |
| *** kgiusti has joined #openstack-infra | 13:10 | |
| *** cshastri has joined #openstack-infra | 13:20 | |
| *** udesale has quit IRC | 13:20 | |
| jaosorior | Does anybody know where I can find the openstack-tox-linters job definition? | 13:21 |
| *** ykarel has quit IRC | 13:21 | |
| *** amoralej|lunch is now known as amoralej | 13:21 | |
| *** cshastri has quit IRC | 13:22 | |
| *** yolanda_ has joined #openstack-infra | 13:22 | |
| *** armaan_ has quit IRC | 13:22 | |
| *** zara_the_lemur__ has quit IRC | 13:23 | |
| *** armaan has joined #openstack-infra | 13:23 | |
| cmurphy | jaosorior: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n388 | 13:24 |
| *** yolanda has quit IRC | 13:24 | |
| jaosorior | cmurphy: thanks! | 13:25 |
| *** SotK has quit IRC | 13:25 | |
| *** hashar is now known as hasharAway | 13:25 | |
| *** esberglu has joined #openstack-infra | 13:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add gentoo test to nodepool experimental https://review.openstack.org/558761 | 13:34 |
| *** links has quit IRC | 13:35 | |
| *** mugsie_ is now known as mugsie | 13:35 | |
| *** ifat_afek has quit IRC | 13:35 | |
| *** mugsie has quit IRC | 13:36 | |
| *** mugsie has joined #openstack-infra | 13:36 | |
| *** inc0 has joined #openstack-infra | 13:36 | |
| *** SotK has joined #openstack-infra | 13:38 | |
| fungi | jaosorior: until we have the configuration browser working in the zuul dashboard, searching like http://codesearch.openstack.org/?q=openstack-tox-linters is usually pretty fast way to find a job definition | 13:40 |
| jaosorior | fungi: that's good to know! | 13:41 |
| *** dklyle has quit IRC | 13:42 | |
| openstackgerrit | YumengBao proposed openstack-infra/project-config master: Adds python-cyborgclient to StoryBoard https://review.openstack.org/558821 | 13:43 |
| *** pgadiya_ has quit IRC | 13:44 | |
| *** camunoz has joined #openstack-infra | 13:47 | |
| *** armaan has quit IRC | 13:49 | |
| *** armaan has joined #openstack-infra | 13:49 | |
| *** links has joined #openstack-infra | 13:49 | |
| *** ykarel has joined #openstack-infra | 13:49 | |
| *** anteaya has joined #openstack-infra | 13:50 | |
| *** rpioso|afk is now known as rpioso | 13:51 | |
| *** neiloy has joined #openstack-infra | 13:53 | |
| *** kiennt26 has joined #openstack-infra | 13:54 | |
| andreykurilin_ | hi folks, please look at https://review.openstack.org/#/c/558204/ it disables several jobs == decreases price for rally gates | 13:54 |
| *** caphrim007 has joined #openstack-infra | 13:55 | |
| *** caphrim007 has quit IRC | 13:55 | |
| *** lbragstad has quit IRC | 13:56 | |
| *** hongbin_ has joined #openstack-infra | 14:00 | |
| *** jamesmcarthur has quit IRC | 14:00 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 14:01 |
| *** zhipeng has joined #openstack-infra | 14:02 | |
| *** gouthamr has joined #openstack-infra | 14:02 | |
| *** agopi has joined #openstack-infra | 14:03 | |
| *** inc0 has quit IRC | 14:03 | |
| *** eharney has quit IRC | 14:04 | |
| *** eharney has joined #openstack-infra | 14:04 | |
| frickler | config-core: some project-config patches for you: https://review.openstack.org/558204 https://review.openstack.org/558727 https://review.openstack.org/295253 | 14:04 |
| corvus | frickler, AJaeger: i don't expect anything i did on git08 to interfere with the actual git repositories on disk (and i never interfered with replication) so we should keep looking into it. | 14:05 |
| corvus | frickler, AJaeger: (basically, i was creating symlinks to those repos and virtual host configs) | 14:06 |
| corvus | frickler, AJaeger: oh, maybe repositories created while git08 was in the emergency file are invalid -- because jeepyb didn't run to set them up correctly | 14:10 |
| *** caphrim007 has joined #openstack-infra | 14:11 | |
| *** bobh has joined #openstack-infra | 14:11 | |
| corvus | i had assumed that would cause all of the git-related tasks not to run, but now that i think about it, they probably just ran everywhere else. | 14:11 |
| corvus | so it's not really safe to put one git server in the emergency file | 14:12 |
| *** caphrim007_ has joined #openstack-infra | 14:12 | |
| *** lpetrut_ has joined #openstack-infra | 14:12 | |
| mordred | corvus: good point | 14:12 |
| fungi | if that's the only impacted repo, we can in theory delete it from disk and then rerun manage-projects | 14:12 |
| fungi | deleteit from disk on git08 i mean | 14:13 |
| corvus | ya | 14:13 |
| corvus | i'll xref project-config with status log | 14:13 |
| *** ramishra has quit IRC | 14:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: [rally] Remove several redundant jobs https://review.openstack.org/558204 | 14:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix publishing infra/publications https://review.openstack.org/558727 | 14:14 |
| *** lpetrut has quit IRC | 14:14 | |
| *** inc0 has joined #openstack-infra | 14:15 | |
| *** caphrim007 has quit IRC | 14:16 | |
| *** alexchadin has quit IRC | 14:16 | |
| *** sree_ has joined #openstack-infra | 14:20 | |
| *** sree_ is now known as Guest51139 | 14:20 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/openstack-zuul-jobs master: [rally] Remove unused jobs https://review.openstack.org/558205 | 14:21 |
| corvus | frickler, AJaeger, fungi, mordred: here's the list i've come up with: https://etherpad.openstack.org/p/lKQm5txVwu | 14:21 |
| *** kiennt26 has quit IRC | 14:21 | |
| corvus | i'll delete those from git08, run manage projects, then tell gerrit to replicate those projects | 14:21 |
| fungi | sounds good (i didn't get a chance to check that those others were also corrupt, but can't hurt) | 14:23 |
| corvus | huh, puppet runs /usr/local/bin/manage-projects, but it's installed in /usr/bin on git08 | 14:25 |
| *** armaan has quit IRC | 14:25 | |
| corvus | oh, we don't actually run manage-projects on git08, i think we only run create_cgitrepos | 14:26 |
| *** jcoufal_ has joined #openstack-infra | 14:27 | |
| *** alexchadin has joined #openstack-infra | 14:27 | |
| *** eernst has joined #openstack-infra | 14:27 | |
| *** anteaya has quit IRC | 14:27 | |
| *** Guest51139 has quit IRC | 14:28 | |
| fungi | i meant rerunning manage-projects on review.o.o, but yeah i guess that doesn't directly trigger anything on the git servers | 14:28 |
| *** sree has joined #openstack-infra | 14:28 | |
| corvus | yeah, i think create_cgitrepos + replicate should take care ofi t | 14:28 |
| fungi | agreed | 14:28 |
| *** r-daneel has joined #openstack-infra | 14:29 | |
| *** jcoufal has quit IRC | 14:30 | |
| mordred | corvus: ++ | 14:30 |
| mordred | corvus: fwiw, create_cgitrepos is very cheap - we could also/additionally consider making git* just run it every puppet run rather than only when projects.yaml changes | 14:32 |
| *** rcernin has quit IRC | 14:32 | |
| corvus | mordred: well, it works pretty well until a sysadmin gets involved :) | 14:32 |
| mordred | corvus: :) | 14:32 |
| mordred | corvus: silly sysadmins | 14:32 |
| *** salv-orlando has quit IRC | 14:33 | |
| corvus | frickler, AJaeger, fungi, mordred: okay, that's done. i think we can add git08 back into lb now | 14:34 |
| *** inc0 has quit IRC | 14:34 | |
| *** dhajare__ has quit IRC | 14:34 | |
| *** inc0 has joined #openstack-infra | 14:34 | |
| corvus | how was git08 removed? | 14:35 |
| *** lbragstad has joined #openstack-infra | 14:35 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Setup WSGIDaemonProcess, ProcessGroup + ApplicationGroup for logs-dev https://review.openstack.org/558687 | 14:35 |
| corvus | i assumed it would have just been put into maintenance mode | 14:36 |
| corvus | but it's not showing up in the lb config at all | 14:36 |
| *** jamesmcarthur has joined #openstack-infra | 14:36 | |
| *** VW_ has joined #openstack-infra | 14:37 | |
| frickler | corvus: I edited haproxy.conf on git.o.o manually, will revert now | 14:38 |
| corvus | frickler: okay thanks. does haproxy reload its config when you edit it? | 14:38 |
| frickler | corvus: I don't think so, I restarted the service for that | 14:39 |
| *** VW has quit IRC | 14:39 | |
| *** d0ugal has quit IRC | 14:39 | |
| corvus | frickler: okay. i'll write up a zero-downtime process for removing backends we can use in the future | 14:39 |
| frickler | corvus: o.k., great | 14:39 |
| *** inc0 has quit IRC | 14:40 | |
| *** snapiri- has quit IRC | 14:40 | |
| *** d0ugal has joined #openstack-infra | 14:41 | |
| fungi | yeah, you can take servers in and out of the pool through the haproxy control socket | 14:41 |
| fungi | no need to drop connections | 14:41 |
| frickler | corvus: while we're at it, cacti doesn't seem to have data for git08 since december | 14:41 |
| frickler | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2568&rra_id=all | 14:42 |
| fungi | looks like it was rebooted around when the graphs stop | 14:44 |
| fungi | snmpd is still running | 14:45 |
| *** dsariel has quit IRC | 14:45 | |
| fungi | the process was last started on january 4, which corresponds with the reboot and the end of the graph data | 14:45 |
| *** r-daneel has quit IRC | 14:47 | |
| *** dosaboy_ is now known as dosaboy | 14:47 | |
| fungi | `sudo journalctl _SYSTEMD_UNIT=snmpd.service` shows it logging connections from cacti02.o.o | 14:47 |
| fungi | and logging no errors | 14:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add haproxy maintenance instructions https://review.openstack.org/558840 | 14:48 |
| corvus | frickler, fungi: ^ | 14:48 |
| *** dklyle has joined #openstack-infra | 14:48 | |
| *** dkehn has quit IRC | 14:49 | |
| *** dkehn has joined #openstack-infra | 14:50 | |
| *** dkehn has quit IRC | 14:50 | |
| *** jcoufal_ has quit IRC | 14:50 | |
| *** dkehn has joined #openstack-infra | 14:51 | |
| *** jcoufal has joined #openstack-infra | 14:51 | |
| *** haleyb_ is now known as haleyb | 14:52 | |
| *** felipemonteiro has joined #openstack-infra | 14:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove unused fedora-26 jobs https://review.openstack.org/558841 | 14:54 |
| fungi | frickler: aha... i think this is another case of broken ipv6 routing somewhere in rackspace's dfw region | 14:55 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import ansible-role-grafyaml repository https://review.openstack.org/558843 | 14:56 |
| *** david-lyle has joined #openstack-infra | 14:57 | |
| *** Kevin_Zheng has joined #openstack-infra | 14:57 | |
| *** dklyle has quit IRC | 14:57 | |
| *** dsariel has joined #openstack-infra | 14:58 | |
| *** inc0 has joined #openstack-infra | 14:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add haproxy maintenance instructions https://review.openstack.org/558840 | 14:58 |
| corvus | pabelanger: ^ sorry, i simplified some commands | 14:59 |
| *** sree has quit IRC | 14:59 | |
| *** sree has joined #openstack-infra | 15:00 | |
| *** links has quit IRC | 15:00 | |
| fungi | frickler: bingo... tcpdump on git08 shows queries arriving from cacti02 and corresponding outbound responses, but tcpdump on cacti02 only shows the outgoing queries to git08 and no inbound responses for them | 15:00 |
| fungi | frickler: similarly, i can't ping6 between those two servers | 15:01 |
| fungi | packets make it from cacti02 to git08 and it replies but packets from git08 do not make it to cacti02 | 15:02 |
| *** r-daneel has joined #openstack-infra | 15:04 | |
| *** alexchadin has quit IRC | 15:04 | |
| *** sree has quit IRC | 15:04 | |
| *** dizquierdo has joined #openstack-infra | 15:04 | |
| *** rcarrillocruz has quit IRC | 15:05 | |
| *** r-daneel_ has joined #openstack-infra | 15:08 | |
| AJaeger | corvus: thanks! | 15:08 |
| *** r-daneel has quit IRC | 15:08 | |
| *** r-daneel_ is now known as r-daneel | 15:08 | |
| *** rcarrillocruz has joined #openstack-infra | 15:08 | |
| boden | hi.. with the neutron-lib requirements-check job, I’m hitting an “packaging.markers.InvalidMarker” error http://logs.openstack.org/09/557809/4/check/requirements-check/f7d14a7/job-output.txt.gz#_2018-04-04_12_56_23_194940 could it be the parsing doesn’t handle the requirement string “weakrefmethod>=1.0.2;python_version=='2.7' # PSF” ?? | 15:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Set min-ready for fedora-26 to 0 https://review.openstack.org/558846 | 15:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove fedora-26 nodes from nodepool https://review.openstack.org/558847 | 15:10 |
| clarkb | boden: I wonder if the marker parser can't handle the comment too | 15:10 |
| boden | clarkb yeah I was wondering.. mitigation suggestions? | 15:11 |
| fungi | AJaeger: your comment on https://review.openstack.org/558821 confuses me. what does python-cyborgclient have to do with python-openstackclient? | 15:11 |
| clarkb | boden: I would drop the comment to confirm and if that fixes it probably drop the comment until it can be parsed properly? dhellmann is probably the best person to weigh in on that though | 15:11 |
| *** kashyap has left #openstack-infra | 15:12 | |
| *** dkehn has quit IRC | 15:12 | |
| fungi | clarkb: boden: i thought i saw someone (dhellmann? prometheanfire?) mention there was a problem with inline comments and that someone has a patch up to fix that | 15:12 |
| *** dkehn has joined #openstack-infra | 15:12 | |
| fungi | AJaeger: oh! did you maybe mean to leave that comment on https://review.openstack.org/554663 instead? | 15:13 |
| *** janki has quit IRC | 15:14 | |
| pabelanger | corvus: +2, so frickler has a chance to review | 15:14 |
| *** zhurong has joined #openstack-infra | 15:14 | |
| *** dkehn_ has joined #openstack-infra | 15:19 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import ansible-role-grafyaml repository https://review.openstack.org/558843 | 15:19 |
| *** felipemonteiro_ has joined #openstack-infra | 15:22 | |
| *** dkehn has quit IRC | 15:22 | |
| prometheanfire | fungi: yes, https://review.openstack.org/558517 | 15:23 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Create ansible-role-grafyaml repository https://review.openstack.org/558843 | 15:25 |
| *** felipemonteiro has quit IRC | 15:26 | |
| AJaeger | fungi: checking... | 15:27 |
| AJaeger | fungi: no, wanted to link to https://review.openstack.org/#/c/558327/ | 15:28 |
| AJaeger | fungi: updated 558821 | 15:28 |
| pabelanger | config-core: clarkb: I've proposed the patches to remove fedora-26 from nodepool: https://review.openstack.org/#/q/topic:remove-fedora-26 | 15:31 |
| *** yamahata has joined #openstack-infra | 15:31 | |
| *** iyamahat has joined #openstack-infra | 15:32 | |
| *** felipemonteiro__ has joined #openstack-infra | 15:33 | |
| *** felipemonteiro_ has quit IRC | 15:33 | |
| *** salv-orlando has joined #openstack-infra | 15:33 | |
| *** evrardjp_ has joined #openstack-infra | 15:35 | |
| fungi | thanks AJaeger! | 15:35 |
| *** evrardjp has quit IRC | 15:36 | |
| *** felipemonteiro has joined #openstack-infra | 15:36 | |
| *** salv-orlando has quit IRC | 15:38 | |
| *** germs has joined #openstack-infra | 15:38 | |
| *** germs has quit IRC | 15:38 | |
| *** germs has joined #openstack-infra | 15:38 | |
| *** tesseract has quit IRC | 15:38 | |
| *** dtantsur is now known as dtantsur|brb | 15:38 | |
| *** germs has quit IRC | 15:39 | |
| *** germs has joined #openstack-infra | 15:39 | |
| *** germs has quit IRC | 15:39 | |
| *** germs has joined #openstack-infra | 15:39 | |
| *** anteaya has joined #openstack-infra | 15:39 | |
| *** agopi has quit IRC | 15:40 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoint to create summit https://review.openstack.org/558852 | 15:40 |
| dmsimard | infra-root: May I put static.o.o in the emergency file ? I was testing https://review.openstack.org/#/c/558687/ and found an unrelated issue that I'm not sure about yet | 15:40 |
| *** jcoufal_ has joined #openstack-infra | 15:40 | |
| *** agopi has joined #openstack-infra | 15:40 | |
| dmsimard | I can't easily explain why these URLs yields different results | 15:40 |
| clarkb | dmsimard: fine with me, you are hackign on the logs-dev vhost I assume? | 15:40 |
| dmsimard | logs.o.o: http://logs.openstack.org/21/557921/6/check/openstack-ansible-functional-ubuntu-xenial/6411906/logs/ara-report/ | 15:41 |
| *** andreas_s has quit IRC | 15:41 | |
| dmsimard | logs-dev.o.o: http://logs-dev.openstack.org/21/557921/6/check/openstack-ansible-functional-ubuntu-xenial/6411906/logs/ara-report/ | 15:41 |
| dmsimard | clarkb: yes, logs-dev | 15:41 |
| *** tesseract has joined #openstack-infra | 15:41 | |
| *** dtruong_ has quit IRC | 15:41 | |
| *** dtruong has joined #openstack-infra | 15:42 | |
| *** iranzo has quit IRC | 15:43 | |
| *** links has joined #openstack-infra | 15:43 | |
| *** jcoufal has quit IRC | 15:43 | |
| fungi | sounds fine | 15:44 |
| *** pcaruana|afk| has quit IRC | 15:45 | |
| *** armaan has joined #openstack-infra | 15:46 | |
| *** panda is now known as panda|training | 15:47 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Synchronize vhost configuration between logs.o.o and logs-dev.o.o https://review.openstack.org/558856 | 15:47 |
| anteaya | corvus: when you are about https://review.openstack.org/#/c/557979 is the current status of my patch to impliment the survey spec | 15:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Enable tuned WSGI settings for logs.openstack.org https://review.openstack.org/558688 | 15:48 |
| anteaya | corvus, I was talking with clarkb and he found example code for me to impliment the Apache auth module but it requires the code to go into the puppet module | 15:48 |
| anteaya | corvus, in the example he found we were hosting the puppet module, and in this case, we are not | 15:49 |
| anteaya | I emailed the maintainer of the puppet-limesurvey module yesterday and will give it a few days for that person to get back to me | 15:49 |
| anteaya | I was wondering about plan B, in case they don't get back to me in the foreseeable future and clarkb suggested we might consider moving ahead and including it after the server is up | 15:50 |
| *** ianychoi__ is now known as ianychoi | 15:50 | |
| anteaya | I'm interested to know where you would stand on this idea | 15:50 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoint to create summit https://review.openstack.org/558852 | 15:51 |
| anteaya | also I'm unfortunately net-splitting a lot, so if I disappear I will check the logs for my nick | 15:52 |
| *** jamesmcarthur has quit IRC | 15:52 | |
| *** jamesmcarthur has joined #openstack-infra | 15:52 | |
| *** iyamahat has quit IRC | 15:55 | |
| * fungi needs to go run some lunch errands before the storyboard meeting, but will return shortly | 15:55 | |
| anteaya | happy errands | 15:55 |
| *** dsariel has quit IRC | 15:57 | |
| *** zhipeng has quit IRC | 15:59 | |
| clarkb | I'm going to try and make today my go through review backlog and file expenses day | 16:02 |
| *** jamesmcarthur has quit IRC | 16:03 | |
| *** evrardjp_ is now known as evrardjp | 16:05 | |
| *** links has quit IRC | 16:05 | |
| ssbarnea | mordred: hi! when you have minute please have a look at https://review.openstack.org/#/c/550087/ -- i need some hints for adding cross testing. | 16:08 |
| *** salv-orlando has joined #openstack-infra | 16:09 | |
| *** iyamahat has joined #openstack-infra | 16:10 | |
| *** eharney has quit IRC | 16:13 | |
| *** eharney has joined #openstack-infra | 16:14 | |
| *** trown is now known as trown|lunch | 16:14 | |
| *** mriedem has quit IRC | 16:15 | |
| *** jamesmcarthur has joined #openstack-infra | 16:16 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add Gerrit docs to Zuul From Scratch https://review.openstack.org/558600 | 16:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add static driver doc to Zuul From Scratch https://review.openstack.org/558802 | 16:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 16:20 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Move zuul jobs config to zuul config section https://review.openstack.org/558867 | 16:20 |
| *** links has joined #openstack-infra | 16:20 | |
| mordred | ssbarnea: done - let me know if the comments make sense | 16:21 |
| *** efried is now known as efried_rollin | 16:25 | |
| *** udesale has joined #openstack-infra | 16:26 | |
| *** florianf_ has quit IRC | 16:28 | |
| *** zhurong has quit IRC | 16:29 | |
| ssbarnea | mordred: yep, really clear comments. only question is regarding implementing the 'tips' feature, i was looking at https://github.com/openstack/python-openstackclient/blob/master/tox.ini#L81 | 16:31 |
| ssbarnea | mordred: the only problem with your implementation is that it assumes that developer already cloned these in that location. | 16:32 |
| *** jamesmcarthur has quit IRC | 16:33 | |
| ssbarnea | mordred: i would rather do some bash logic like: if -d path: use pip install git+file, else: pip install git+http: ... -- it would look bit ugly inside tox.ini but should work. | 16:34 |
| ssbarnea | mordred: in fact it your code would work for me as I happen to have the sibling in the same location, but I was considering other devs that may not have that. | 16:35 |
| prometheanfire | anyone other than tony review irc-meetings? | 16:37 |
| odyssey4me | is the data in http://logs.openstack.org/29/558229/1/gate/openstack-ansible-deploy-ceph-ubuntu-xenial/aa549d5/zuul-info/inventory.yaml accessible on the host in some way? ie if I'm running a script or my own ansible version in the job, can I ingest the contents of that file somehow? I need to detect the region and do things based on the result | 16:37 |
| *** mriedem has joined #openstack-infra | 16:37 | |
| *** e0ne has quit IRC | 16:38 | |
| *** beekneemech is now known as bnemec | 16:42 | |
| *** lucasagomes is now known as lucas-afk | 16:45 | |
| *** claudiub|2 has quit IRC | 16:45 | |
| *** dizquierdo has quit IRC | 16:45 | |
| *** claudiub|2 has joined #openstack-infra | 16:46 | |
| dansmith | are the TIMED_OUT errors known? | 16:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Check out more appropriate branches of role and playbook repos https://review.openstack.org/558643 | 16:46 |
| *** udesale has quit IRC | 16:46 | |
| pabelanger | dansmith: which provider? | 16:46 |
| pabelanger | dansmith: we've seen some slowness with limestone recently | 16:47 |
| dansmith | pabelanger: I dunno, I can look. I've seen a bunch in the last 12-18 hours | 16:47 |
| pabelanger | I want to say we have an elastic-recheck query setup for them too | 16:47 |
| pabelanger | yah | 16:47 |
| pabelanger | http://status.openstack.org/elastic-recheck/#1686542 | 16:47 |
| pabelanger | seems to be trending up | 16:47 |
| dansmith | first one I checked is limestone | 16:48 |
| dansmith | and the second | 16:48 |
| dansmith | so that must be it | 16:48 |
| clarkb | we might need to reduce max servers then? I think its cpu contention according to logan | 16:48 |
| dansmith | it seems to hit often enough to take out at least one job on most of the runs I've seen lately | 16:48 |
| clarkb | (I also thought ut was expected ti be resolved yesterday but maybe I misunderstoof) | 16:50 |
| logan- | yea everything is full capacity since yesterday evening | 16:52 |
| *** wolverineav has joined #openstack-infra | 16:52 | |
| clarkb | dial back to ~30 servers then? | 16:52 |
| logan- | sounds good | 16:52 |
| logan- | ill dig into it more and see if theres a node misbehaving or something... | 16:53 |
| snapiri | infra: I have some issue with zuul that I need help understanding. does anyone have free resources to assist? | 16:53 |
| pabelanger | clarkb: logan-: 30 sounds good | 16:54 |
| logan- | dansmith: im trolling thru logstash, but if you run across troublesome limestone jobs feel free to drop me a url and ill correlate with nodes on our end to see whats up | 16:54 |
| pabelanger | snapiri: usually just best to ask the question and somebody will likely respond :) | 16:54 |
| snapiri | pabelanger: ok... | 16:54 |
| snapiri | On https://review.openstack.org/#/c/558420/ I have added some library as required project to my job, but still the job fails and says this library is missing (http://logs.openstack.org/20/558420/2/check/dragonflow-tempest/321c5ea/job-output.txt.gz#_2018-04-03_17_42_56_236032) any ideas? | 16:55 |
| dansmith | logan-: okay, all the ones I've checked have been limestone.. you can probably get the authoritative list from logstash | 16:55 |
| logan- | ++ will do thanks | 16:55 |
| *** greghaynes has joined #openstack-infra | 16:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Reduce limestone max-servers to 30 https://review.openstack.org/558881 | 16:57 |
| clarkb | pabelanger: logan- ^ | 16:57 |
| *** HeOS has quit IRC | 16:58 | |
| pabelanger | snapiri: I _think_ https://review.openstack.org/549252/ will help with that. But we might want to confirm with corvus and andreaf | 16:59 |
| *** dprince has quit IRC | 16:59 | |
| pabelanger | clarkb: +2 | 16:59 |
| *** auristor1 has quit IRC | 17:01 | |
| *** anteaya has quit IRC | 17:01 | |
| *** auristor has joined #openstack-infra | 17:02 | |
| *** germs has quit IRC | 17:03 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:05 | |
| *** germs has joined #openstack-infra | 17:06 | |
| *** germs has quit IRC | 17:06 | |
| *** germs has joined #openstack-infra | 17:06 | |
| *** Shrews has quit IRC | 17:06 | |
| *** Shrews has joined #openstack-infra | 17:06 | |
| *** blessin has joined #openstack-infra | 17:06 | |
| *** Kevin_Zheng has quit IRC | 17:07 | |
| blessin | HELLO | 17:07 |
| blessin | anyone to help me out | 17:07 |
| *** links has quit IRC | 17:08 | |
| *** felipemonteiro__ has quit IRC | 17:08 | |
| blessin | anyone | 17:09 |
| *** Shrews has quit IRC | 17:11 | |
| *** jpich has quit IRC | 17:11 | |
| *** Shrews has joined #openstack-infra | 17:11 | |
| clarkb | I single core approved the reduction in max-servers since it is affecting jobs | 17:12 |
| clarkb | blessin: how can we help you | 17:12 |
| *** felipemonteiro_ has quit IRC | 17:13 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:14 | |
| *** david-lyle has quit IRC | 17:16 | |
| *** felipemonteiro_ has quit IRC | 17:18 | |
| *** jpena is now known as jpena|off | 17:19 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:20 | |
| snapiri | pabelanger: Thanks, so as a workaround to check it, can I add the neutron-tempest-plugin to LIBS_FROM_GIT? | 17:21 |
| *** olaph has quit IRC | 17:22 | |
| *** olaph has joined #openstack-infra | 17:22 | |
| pabelanger | snapiri: yah, i think so. However, it would be good to confirm with openstack-qa if that is the right path forward. | 17:22 |
| *** lpetrut_ has quit IRC | 17:22 | |
| *** pcichy has quit IRC | 17:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Reduce limestone max-servers to 30 https://review.openstack.org/558881 | 17:23 |
| *** gouthamr has quit IRC | 17:24 | |
| snapiri | pabelanger: will ask in openstack-qa. Thanks! | 17:25 |
| *** trown|lunch is now known as trown | 17:26 | |
| *** dprince has joined #openstack-infra | 17:26 | |
| *** e0ne has joined #openstack-infra | 17:27 | |
| *** olaph1 has joined #openstack-infra | 17:32 | |
| *** olaph has quit IRC | 17:32 | |
| mordred | ssbarnea: yah - I agree, that would be abetter logic ... | 17:36 |
| *** adriano has joined #openstack-infra | 17:36 | |
| mordred | ssbarnea: I've actualy been working on extracting the siblings logic into a tool that could also be used locally | 17:36 |
| dmsimard | did a non-scientific benchmark and found an easy fix to shave between 300ms and 400ms for every ara middleware sqlite request, yikes | 17:36 |
| mordred | ssbarnea: maybe I should finish doing that | 17:36 |
| ssbarnea | mordred: make it a tox plugin for example. | 17:37 |
| *** olaph1 is now known as olaph | 17:37 | |
| *** Diabelko has quit IRC | 17:38 | |
| mordred | ssbarnea: well ... I would, but unfortunately tox plugins don't let you express that you need a tox plugin for tox to work | 17:38 |
| mordred | ssbarnea: but yes - something like that | 17:38 |
| mordred | ssbarnea: (I took a stab at implementing this as a tox plugin already) | 17:39 |
| pabelanger | config-core: https://review.openstack.org/558846/ drops fedora-26 min-ready to 0, first step in remove images from nodepool | 17:39 |
| ssbarnea | mordred: you may find this useful too https://github.com/pycontribs/rtox | 17:39 |
| mordred | ssbarnea: nice | 17:41 |
| ssbarnea | mordred: got me busy over the weekend, fixing few issues but so far it seems more than usable. also added some incipient bindeps.txt support. | 17:42 |
| *** gfidente is now known as gfidente|afk | 17:42 | |
| *** d0ugal has quit IRC | 17:42 | |
| ssbarnea | mordred: my plan is to make it able to install bindep deps, maybe in the future adding this directly into bindep. | 17:43 |
| blessin | could someone tell me the application examples that are easy to deploy in openstack magnum project | 17:43 |
| *** alexchadin has joined #openstack-infra | 17:44 | |
| ssbarnea | mordred: timet to recheck https://review.openstack.org/#/c/550087/ -- now with smart "tips". | 17:44 |
| *** mikal_ has joined #openstack-infra | 17:45 | |
| *** hongbin_ has quit IRC | 17:45 | |
| *** gouthamr has joined #openstack-infra | 17:45 | |
| *** electrofelix has quit IRC | 17:47 | |
| ssbarnea | mordred: now that I think a little bit, this feature of installing tips is more likely to become a feature of pip. Is bit similar with --pre (PIP_PRE=1) option. We could have it PIP_PRE=2 where instead of pre-release, it will attempt to clone the master (if it can detect where the codebase is hosted) | 17:48 |
| *** wolverineav has quit IRC | 17:48 | |
| *** mikal has quit IRC | 17:48 | |
| *** hongbin_ has joined #openstack-infra | 17:52 | |
| *** yamamoto_ has quit IRC | 17:58 | |
| *** yamamoto has joined #openstack-infra | 17:59 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:01 | |
| *** eernst has quit IRC | 18:01 | |
| *** eernst has joined #openstack-infra | 18:02 | |
| *** felipemonteiro_ has quit IRC | 18:02 | |
| *** adriano has quit IRC | 18:02 | |
| cmurphy | so on patchset 15 here https://review.openstack.org/#/c/557011/15/playbooks/infra-puppet-apply/post.yaml i managed to get a passing puppet-apply run https://review.openstack.org/#/c/557515/ (two rechecks ago) but the test output didn't actually end up in the logs | 18:04 |
| cmurphy | looking at how devstack does it, it uses the stage-output role but then has another role that does a synchronize to the log server http://git.openstack.org/cgit/openstack-dev/devstack/tree/playbooks/post.yaml | 18:04 |
| cmurphy | which if i copied that would bring me back to what i had on patchset 1 | 18:04 |
| cmurphy | advice? | 18:04 |
| clarkb | cmurphy: I think you copy the logs onto the executor into the logs/ dir then the base job post run copies that to the log server proper? | 18:05 |
| *** wolverineav has joined #openstack-infra | 18:06 | |
| *** shardy_mtg has quit IRC | 18:06 | |
| *** yamamoto_ has joined #openstack-infra | 18:06 | |
| cmurphy | clarkb: isn't that what https://review.openstack.org/#/c/557011/17/zuul.d/jobs.yaml@1116 should be doing? | 18:08 |
| clarkb | maybe? I'm not sure what consumes zuul_copy_output but its probably that | 18:09 |
| cmurphy | it's consumed by the stage-output role | 18:09 |
| cmurphy | which i thought was going to put the logs into a magic place | 18:09 |
| clarkb | ah then ya that should be it | 18:09 |
| cmurphy | but that didn't seem to happen | 18:09 |
| clarkb | mordred: AJaeger ^ I think did a lot of work on that | 18:10 |
| clarkb | and may know more | 18:10 |
| *** yamamoto has quit IRC | 18:10 | |
| *** felipemonteiro has quit IRC | 18:10 | |
| *** d0ugal has joined #openstack-infra | 18:11 | |
| *** lpetrut_ has joined #openstack-infra | 18:12 | |
| *** Diabelko has joined #openstack-infra | 18:13 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed CORS post without preflight https://review.openstack.org/558886 | 18:14 |
| *** tesseract has quit IRC | 18:14 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed CORS post without preflight https://review.openstack.org/558886 | 18:14 |
| mordred | clarkb: reading | 18:15 |
| mordred | cmurphy, clarkb: yah - sorry, I'm behind on finishing things up there | 18:16 |
| *** armaan has quit IRC | 18:16 | |
| mordred | the *theory*/*plan* is to do what cmurphy thought - put the logs into the correct location on the remote note, then the base job will copy them back to the executor and then up to the log server | 18:17 |
| *** wolverineav has quit IRC | 18:17 | |
| mordred | however, that work is not finished yet, so for now jobs have to copy the logs back to the executor themselves | 18:17 |
| clarkb | gotcha | 18:17 |
| *** wolverineav has joined #openstack-infra | 18:17 | |
| cmurphy | okay so i'm just gonna go back to basically https://review.openstack.org/#/c/557011/1/playbooks/infra-puppet-apply/post.yaml then | 18:18 |
| *** alexchadin has quit IRC | 18:18 | |
| mordred | clarkb: https://review.openstack.org/#/q/topic:zuulv3-output | 18:19 |
| *** blessin has quit IRC | 18:20 | |
| clarkb | looks like it needs rebasing? is that complicated and touching enough things that we may just want to get them updated then ask people to update them before other zuul-jobs changes? | 18:20 |
| *** Swami has joined #openstack-infra | 18:21 | |
| *** wolverineav has quit IRC | 18:22 | |
| *** ihrachys has joined #openstack-infra | 18:23 | |
| clarkb | I've confirmed the max-servers: 30 change is now in effect, we should be able to use e-r to see if there is a drop off in timeouts | 18:24 |
| *** salv-orlando has quit IRC | 18:24 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 18:27 |
| *** wolverineav has joined #openstack-infra | 18:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Check out more appropriate branches of role and playbook repos https://review.openstack.org/558643 | 18:28 |
| *** dtantsur|brb is now known as dtantsur | 18:29 | |
| *** armaan has joined #openstack-infra | 18:30 | |
| *** armaan has quit IRC | 18:35 | |
| *** Sukhdev has joined #openstack-infra | 18:36 | |
| *** armaan has joined #openstack-infra | 18:36 | |
| *** rosmaita has quit IRC | 18:37 | |
| *** felipemonteiro has joined #openstack-infra | 18:38 | |
| *** felipemonteiro__ has quit IRC | 18:38 | |
| *** Sukhdev has quit IRC | 18:41 | |
| *** VW has joined #openstack-infra | 18:50 | |
| pabelanger | clarkb: mind reviewing: https://review.openstack.org/558846/ and https://review.openstack.org/558847/ for fedora-26 removals | 18:50 |
| *** amoralej is now known as amoralej|off | 18:51 | |
| clarkb | pabelanger: are you wanting to remove them before the distro EOLs? | 18:51 |
| *** sshnaidm is now known as sshnaidm|afk | 18:51 | |
| clarkb | fwiw this is why we don't do intermediate ubuntu releases | 18:51 |
| *** dtantsur is now known as dtantsur|afk | 18:52 | |
| *** VW_ has quit IRC | 18:52 | |
| clarkb | just thinking this is a different message than we've sort of said around extended maintence of we can probably keep images up until they eol | 18:52 |
| clarkb | but fedora is different in that it only lives for ~13 months | 18:53 |
| pabelanger | clarkb: yah, until now, we've always only have 1 version of fedora online. Due to zuul roll out, we never got around to removing fedora-26 sooner | 18:53 |
| pabelanger | once we bring latest version of fedora online, we've ways migrated jobs and remove the older version, regardless of EOL status | 18:53 |
| clarkb | (fedora will eol before openstack support for $release eols which makes fedora different) | 18:53 |
| clarkb | gotcha | 18:54 |
| fungi | i have to say, i'm amused by that one job mentioned on the ml which uses 10 nodes (5x centos and 5x fedora?) | 18:55 |
| clarkb | fungi: I think its two jobs? | 18:55 |
| clarkb | but total is 10 nodes | 18:55 |
| mtreinish | clarkb: even 5 nodes per job is a lot :) | 18:55 |
| fungi | oh, got it. that wasn't clear to me from the ml. though it was doing mixed-distro testnig | 18:55 |
| fungi | testing | 18:55 |
| clarkb | mtreinish: it is especially if the job doesn't work (I've been working with OSH in their irc channel on that the last few days) | 18:55 |
| mtreinish | lol, ok that's even better | 18:56 |
| pabelanger | fungi: yah, there is also 5 node ubuntu too, which seems to be passing but non-voting | 18:56 |
| *** gfidente|afk has quit IRC | 18:56 | |
| pabelanger | I didn't really look to see why it needed 5 nodes vs 2 | 18:57 |
| *** olaph1 has joined #openstack-infra | 19:01 | |
| *** olaph has quit IRC | 19:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set min-ready for fedora-26 to 0 https://review.openstack.org/558846 | 19:04 |
| clarkb | pabelanger: thinking about this more we should sync up with stable team and make it clear that ubuntu lts and centos are likely the only two images to live for any long period of time. Simply because they are supported for a long peiod of time and we have users using them. Debian maybe too? I don't really want to commit to debian that way but its theoretically doable | 19:07 |
| *** jamesmcarthur has joined #openstack-infra | 19:07 | |
| *** hasharAway is now known as hashar | 19:07 | |
| pabelanger | clarkb: Yah, I think that is good. I've always seen fedora as a way for people to get newer packages for testing but not actually doing stable branches on | 19:08 |
| clarkb | pabelanger: ya especially since our stable branches live logner than fedora I don't think its an issue with fedora | 19:08 |
| pabelanger | clarkb: +1 | 19:09 |
| *** eernst has quit IRC | 19:10 | |
| *** harlowja has joined #openstack-infra | 19:10 | |
| pabelanger | clarkb: in fact, I think we should look to remove debian-jessie once we get debian-stretch online, to reduce the burden of images we maintain | 19:10 |
| *** jamesmcarthur has quit IRC | 19:11 | |
| *** rosmaita has joined #openstack-infra | 19:11 | |
| *** kgiusti has quit IRC | 19:13 | |
| *** diablo_rojo has quit IRC | 19:14 | |
| *** jamesmcarthur has joined #openstack-infra | 19:16 | |
| clarkb | pabelanger: fwiw most of the burden tends to be on adding new releases or distros | 19:17 |
| clarkb | Once we've done that that particular distro release tends to just keep working thanks to dib | 19:17 |
| fungi | rolling release dsitros (debian/sid, gentoo, opensuse/tumbleweed) may be viable exceptions there | 19:18 |
| fungi | though maybe also terrible choices for stable branch testing ;) | 19:18 |
| *** salv-orlando has joined #openstack-infra | 19:18 | |
| clarkb | indeed :) | 19:18 |
| *** felipemonteiro_ has joined #openstack-infra | 19:18 | |
| fungi | pabelanger: yes, debian-jessie (currend oldstable) will be transitioning out of normal support and into lts any day now | 19:19 |
| fungi | yeah, https://wiki.debian.org/LTS says the lts team takes over jessie in june | 19:20 |
| fungi | and drops wheezy entirely next month | 19:20 |
| pabelanger | okay, so we'd be okay keeping debian-jessie online? | 19:20 |
| clarkb | fungi: this discussion led me to https://review.openstack.org/#/c/505657/3 fwiw I'm fine with removing old support if that is what tumbleweed has done. New installs will get new stuff and if you aren't updating then you should be | 19:21 |
| fungi | if we don't have anything important testing jessie, i'd drop it | 19:21 |
| clarkb | fungi: would you be willing to merge 505657 and make a release in that case? | 19:21 |
| *** felipemonteiro has quit IRC | 19:21 | |
| clarkb | fungi: or do you think bindep should be conservative? | 19:21 |
| *** eernst has joined #openstack-infra | 19:21 | |
| fungi | clarkb: i'm fine with it if the opinion of users of that version is that rolling releases should continue rolling | 19:22 |
| pabelanger | fungi: I think OSA is using it but would need to confirm | 19:22 |
| clarkb | fungi: yes I think that is the case | 19:22 |
| pabelanger | http://git.openstack.org/cgit/openstack/ansible-hardening/tree/zuul.d/project.yaml | 19:22 |
| clarkb | fungi: I've +2'd the change that isn't mine and will abandon my chagne if you want to also leave a review with ^ | 19:22 |
| pabelanger | clarkb: fungi: helm patch moving centos / fedora to experimental was approved, FYI | 19:24 |
| fungi | clarkb: thanks | 19:25 |
| *** anteaya has joined #openstack-infra | 19:26 | |
| odyssey4me | pabelanger I think ansible-hardening may be using it, but none of the other OSA repositories to my knowledge - assuming we're talking about debian jessie? | 19:26 |
| *** kgiusti has joined #openstack-infra | 19:26 | |
| fungi | odyssey4me: yeah, asking whether we need to keep debian-jessie around once we're sure our debian-stretch images are working okay | 19:28 |
| pabelanger | odyssey4me: yah, that is what codesearch.o.o shows too | 19:28 |
| fungi | worth noting, we also have work in progress to add debian-sid images too, in case people want to test something much newer | 19:29 |
| *** efried_rollin is now known as efried | 19:29 | |
| * fungi will be curious to find out how often image building breaks for that | 19:29 | |
| fungi | i run sid on my workstations and portable systems (not my servers though) so it's not all _that_ unstable | 19:30 |
| *** eernst has quit IRC | 19:30 | |
| *** eernst has joined #openstack-infra | 19:30 | |
| *** armaan has quit IRC | 19:31 | |
| *** adriano_ has joined #openstack-infra | 19:33 | |
| *** ykarel is now known as ykarel|away | 19:35 | |
| *** eernst has quit IRC | 19:36 | |
| *** tosky has quit IRC | 19:36 | |
| openstackgerrit | Merged openstack-infra/bindep master: SUSE: Update openSUSE Tumbleweed references for lsb-release-3.X.X https://review.openstack.org/505657 | 19:36 |
| *** tosky has joined #openstack-infra | 19:37 | |
| *** armaan has joined #openstack-infra | 19:38 | |
| *** david-lyle has joined #openstack-infra | 19:42 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:48 | |
| *** felipemonteiro__ has quit IRC | 19:49 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:49 | |
| *** felipemonteiro_ has quit IRC | 19:52 | |
| *** gouthamr has quit IRC | 19:52 | |
| *** felipemonteiro_ has joined #openstack-infra | 19:54 | |
| *** claudiub|2 has quit IRC | 19:54 | |
| *** armaan has quit IRC | 19:54 | |
| *** eharney has quit IRC | 19:56 | |
| *** felipemonteiro__ has quit IRC | 19:57 | |
| fungi | clarkb: ^ | 20:01 |
| clarkb | fungi: cool that probably deserves a release too? I can do that after lunch if you dont want to do it | 20:02 |
| *** esberglu has quit IRC | 20:03 | |
| *** ihrachys has quit IRC | 20:04 | |
| *** ihrachys has joined #openstack-infra | 20:05 | |
| fungi | clarkb: i can push a tag. probably should be a minor bump? i'd feel silly about doing a major bump for dropping compat with old unupdated deployments of rolling release distros | 20:05 |
| clarkb | ya I think so its a bugfix unless any other changes got in too | 20:06 |
| *** rosmaita has quit IRC | 20:07 | |
| *** esberglu_ has joined #openstack-infra | 20:08 | |
| *** VW has quit IRC | 20:08 | |
| *** eernst has joined #openstack-infra | 20:09 | |
| *** VW has joined #openstack-infra | 20:09 | |
| *** myoung is now known as myoung|mtg | 20:10 | |
| *** olaph has joined #openstack-infra | 20:13 | |
| *** VW has quit IRC | 20:14 | |
| *** VW has joined #openstack-infra | 20:15 | |
| *** olaph1 has quit IRC | 20:16 | |
| pabelanger | clarkb: fungi: I think we are ready to move forward on https://review.openstack.org/558847/ no fedora-26 nodes online and emails sent to ML about potential NODE_FAILURE | 20:17 |
| fungi | clarkb: looks like we added support for apple darwin and rhel workstation, so bumping from 2.6.0 to 2.7.0 seems appropriate. do you agree? | 20:23 |
| fungi | per `git log --no-merges --oneline 2.6.0..origin/master` | 20:24 |
| *** ykarel|away has quit IRC | 20:25 | |
| *** anteaya has quit IRC | 20:25 | |
| *** lpetrut_ has quit IRC | 20:27 | |
| clarkb | fungi: ya | 20:27 |
| clarkb | though i can double chck in a few almost back to desktop | 20:27 |
| fungi | cool, pushed | 20:27 |
| fungi | no need | 20:27 |
| clarkb | yay | 20:27 |
| fungi | #status log released bindep 2.7.0 | 20:27 |
| openstackstatus | fungi: finished logging | 20:27 |
| *** snapiri has quit IRC | 20:28 | |
| *** Sukhdev has joined #openstack-infra | 20:29 | |
| *** snapiri has joined #openstack-infra | 20:30 | |
| *** yamamoto_ has quit IRC | 20:31 | |
| *** snapiri has quit IRC | 20:33 | |
| *** snapiri has joined #openstack-infra | 20:33 | |
| clarkb | fungi: thanks | 20:34 |
| *** anteaya has joined #openstack-infra | 20:36 | |
| openstackgerrit | Merged openstack/python-jenkins master: Pass form data as body instead of in the url https://review.openstack.org/474952 | 20:37 |
| openstackgerrit | Merged openstack/python-jenkins master: Migration to using requests https://review.openstack.org/205637 | 20:40 |
| *** panda|training is now known as panda|off | 20:42 | |
| *** Nil_ has quit IRC | 20:42 | |
| *** jtomasek_ has quit IRC | 20:44 | |
| *** e0ne has quit IRC | 20:46 | |
| *** hashar has quit IRC | 20:48 | |
| *** edmondsw has quit IRC | 20:50 | |
| *** Nil_ has joined #openstack-infra | 20:52 | |
| *** rfolco|ruck is now known as rfolco|off | 20:52 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/python-jenkins master: Replace testr with stestr https://review.openstack.org/558924 | 20:53 |
| *** florianf has joined #openstack-infra | 20:56 | |
| *** e0ne has joined #openstack-infra | 20:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add debian stretch nodes to arm64/amd64 clouds https://review.openstack.org/557902 | 20:58 |
| ianw | pabelanger / fungi : that should be ready to go. there's something in reprepro about "flooding" all-arch packages when you add a new architecture, but i'm not sure if it's required when mirroring or not | 20:59 |
| *** ktsuyuzaki is now known as kota_ | 20:59 | |
| *** rcernin has joined #openstack-infra | 20:59 | |
| ianw | ^ i guess we try it, and if arm64 starts complaining about any missing _all packages, we'll know | 20:59 |
| *** olaph1 has joined #openstack-infra | 21:01 | |
| pabelanger | ianw: -1, i don't think universe is a thing in debian | 21:01 |
| pabelanger | just main | 21:01 |
| clarkb | fungi: do you mind if I put 513825 back to a failing state if upstream pip isn't going to merge your fixes? | 21:01 |
| fungi | clarkb: yeah, please do | 21:01 |
| clarkb | fungi: pabelanger I've got a super hacky installeverything in a virtualenv patch for devstack that I'd like to check for viability | 21:02 |
| ianw | oh ha, good point, copy-paste issues :) | 21:02 |
| pabelanger | clarkb: cool, I ran into a fair bit of issues with the USE_VENV logic | 21:02 |
| fungi | i half expect they'll revisit my patch soon after 10 releases and lots of people start complaining and they remember someone already pushed up a get-out-of-jail-free card | 21:02 |
| ianw | we probably want security in there? | 21:02 |
| *** olaph has quit IRC | 21:03 | |
| persia | ianw: You should only need to flood arch_all packages when doing something very weird, unless I misread the documentation. I don't think it necessary for this use case. | 21:03 |
| pabelanger | ianw: that should be there by default I think | 21:03 |
| pabelanger | ianw: just the component is main | 21:03 |
| pabelanger | clarkb: https://review.openstack.org/558090/ might apply to use also | 21:03 |
| *** myoung|mtg is now known as myoung | 21:03 | |
| *** Sukhdev has quit IRC | 21:04 | |
| ianw | persia: yeah i hope so. adding an architecture to an existing mirror might actually be considered weird :) | 21:04 |
| persia | ianw / pabelanger: You need component "main" and three different pockets (regular, updates, security) | 21:04 |
| *** Goneri has quit IRC | 21:05 | |
| clarkb | persia: ya I think that is the wrong appraoch since we don't actually care about doing more than one virtualenv per python | 21:05 |
| clarkb | er pabelanger ^ | 21:05 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add debian stretch nodes to arm64/amd64 clouds https://review.openstack.org/557902 | 21:05 |
| persia | ianw: It's for the case where you are running your mirror out-of-sync for some architectures and not others, and want to update some arch:all packages from one place to another. | 21:05 |
| clarkb | sorry persia | 21:05 |
| clarkb | pabelanger: https://review.openstack.org/558930 something more like that more closely mimics the current situation | 21:05 |
| *** r-daneel has quit IRC | 21:06 | |
| clarkb | pabelanger: basically just make a virtualenv, stick it at the front of the path and continue on like before | 21:06 |
| clarkb | I'm sure stuff will still break though | 21:06 |
| ianw | clarkb: yeah i think the issues has been people basically assuming the location of binaries; but probably that has to break | 21:08 |
| clarkb | ianw: ya I'm noticing that which is why I put the venv/bin at front of path :) | 21:09 |
| clarkb | I expect it may still be an issue with systemd unit files | 21:10 |
| *** rosmaita has joined #openstack-infra | 21:10 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoint update summit https://review.openstack.org/558933 | 21:10 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add debian stretch nodes to arm64/amd64 clouds https://review.openstack.org/557902 | 21:11 |
| clarkb | ianw: pabelanger I don't expect I'll have time to dig much more I just want to get a feel for how broken it is if we try the hacky thing | 21:12 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoint update summit https://review.openstack.org/558933 | 21:12 |
| clarkb | if it gets close to working then maybe we run with it. If it isn't close to working then back to drawing board potentially | 21:12 |
| openstackgerrit | Sorin Sbarnea proposed openstack/python-jenkins master: Replace testr with stestr https://review.openstack.org/558924 | 21:12 |
| ianw | clarkb: we have dtroyer's stuff which was close to working at one point | 21:13 |
| clarkb | ianw: for single virtualenv? | 21:13 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Track tripleo-validations tasks on StoryBoard https://review.openstack.org/558934 | 21:13 |
| clarkb | (unsure of what specific stuff it is) | 21:13 |
| ianw | clarkb: basically USE_VENV inside devstack | 21:14 |
| clarkb | that doesn't help us though because it doesn't check coinstallability | 21:15 |
| *** iyamahat has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| fungi | USE_VENV is separate per-service venvs, right? | 21:15 |
| clarkb | fungi: ya | 21:15 |
| ianw | no, iirc it was actually to avoid co-installing | 21:15 |
| clarkb | which is what pabelanger is working on fixing but I don't think its quite what we want | 21:15 |
| clarkb | it may address the pip10 problem but then we still have to figure out coinstallability | 21:15 |
| *** shoogz has joined #openstack-infra | 21:15 | |
| ianw | right, well i thought that the requirements project was sort of the keeper of that? | 21:16 |
| clarkb | ianw: it sort of was but then they just changed it to stop doing that | 21:16 |
| ianw | as in the lower-constraints stuff | 21:16 |
| clarkb | maybe this was a bad time to make that change :/ | 21:16 |
| ianw | ? | 21:16 |
| clarkb | ianw: each project can have its own constraints that differ from the main constraints as long as it lies within upper and lower bounds | 21:17 |
| clarkb | so we aren't actively testing coinstallability there aiui | 21:17 |
| clarkb | the integration test is being relied on to do that | 21:17 |
| ianw | ok; we still have that test that installs all requirements together? | 21:18 |
| ianw | (sorry, I have read no scrollback, tell me to just catch up if i'm rehashing) | 21:18 |
| clarkb | ianw: I'm not fully up to date on it myself. But have fielded questions from proejcts wondering why requirements checks are failing their now special repo local constraints | 21:18 |
| clarkb | (and part of it is no longer having global requirements do that for them and getting it wrong locally) | 21:19 |
| *** yamahata has quit IRC | 21:19 | |
| clarkb | dhellmann: would probably know off the top of his head. If devstack stops installing everything in one python environment and instead uses per service virtualenvs would that be a risk for the new requirements system? | 21:20 |
| clarkb | dhellmann: or do we have something else somewhere checking coinstallability | 21:20 |
| *** d0ugal has quit IRC | 21:21 | |
| ianw | clarkb: ok; a priori i feel like devstack is probably the wrong place to be checking coinstallability; it seems there's probably too much else going on in there to confuse things | 21:21 |
| ianw | but also things get confused if we're doing a virtualenv for pip reasons | 21:22 |
| ianw | tl;dr : i'm confused :) | 21:22 |
| clarkb | it is all a mess :P fwiw my hack change is still running much longer than I expected it to | 21:22 |
| clarkb | ianw: I think there is value in having your integration tests check if you can actually install the software together because that way you are exercising it in addition to just installing it | 21:23 |
| ianw | i probably wouldn't disagree with that ... but also things come and go in the devstack environment so i'm not sure it could be called canonical. doing something like grabbing all of requirements and mass installing it would be the gold standard? | 21:24 |
| clarkb | ianw: ya that was the old method of constraints generation. Unsure if that is still done | 21:25 |
| clarkb | hrm zuul console log follower autoscrolling option isn't working as I expect | 21:26 |
| clarkb | can't scroll to the bottom if auto scroll off and with autoscroll on scrolling up doesn't stop the scrolling | 21:26 |
| ianw | apropos nothing; could i get some sanity reviews on https://review.openstack.org/#/c/557588/ which fixes up when we mirror two repos to the same volume. that will finally fix mariadb | 21:26 |
| clarkb | my hack change did finally fail but I can't see why yet due to that autoscrolling weirdness :( | 21:27 |
| ianw | also slightly related https://review.openstack.org/553894 uses a better method to offset the mirror cron jobs i think | 21:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/gear master: Statsd is a requirement, not test-requirement https://review.openstack.org/558938 | 21:28 |
| *** yamamoto has joined #openstack-infra | 21:28 | |
| pabelanger | anybody interested in reviewing a few gear patches? ^ | 21:28 |
| pabelanger | parent adds zuul.yaml file for testing too | 21:28 |
| *** kgiusti has left #openstack-infra | 21:29 | |
| pabelanger | oh, let me rebase | 21:29 |
| pabelanger | parent is abandoned | 21:29 |
| mordred | pabelanger: I believe statsd is optional on purpose | 21:29 |
| *** rcernin has quit IRC | 21:29 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/gear master: Statsd is a requirement, not test-requirement https://review.openstack.org/558938 | 21:29 |
| *** iyamahat has joined #openstack-infra | 21:29 | |
| pabelanger | mordred: oh, do you know why? | 21:29 |
| *** eernst has quit IRC | 21:30 | |
| mordred | pabelanger: the intent being that if you install statsd, you'll get stats, but if you aren't interested in stats, you don't have to install it | 21:30 |
| *** eernst has joined #openstack-infra | 21:30 | |
| pabelanger | mordred: okay, that is different on how we do nodepool / zuul today | 21:30 |
| mordred | pabelanger: you'll see in gear/__init__.py a try: import statsd except ImportError: statsd = None | 21:31 |
| pabelanger | which is fine, just wanted to see if they would work the same :) | 21:31 |
| mordred | pabelanger: :)) | 21:31 |
| mordred | pabelanger: I think because gear is otherwise a very lightweight library ... | 21:31 |
| mordred | pabelanger: but ... | 21:32 |
| mordred | corvus: ^^ any thoughts on the above? | 21:32 |
| *** shoogz has quit IRC | 21:33 | |
| mordred | pabelanger: I think also statsd initialization used to be a bit different than it is now in 3.x | 21:33 |
| mordred | pabelanger: so I do htink it's worth checking in on the approach | 21:33 |
| pabelanger | mordred: yah, it is still pinned <3.0 with gear. Was going to update like we did in zuul / nodepool to uncap it | 21:33 |
| *** yamamoto has quit IRC | 21:33 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoint to delete summit by id https://review.openstack.org/558939 | 21:34 |
| mordred | pabelanger: ah - nod | 21:35 |
| clarkb | fwiw I think not needing statds is a nice thing | 21:35 |
| mordred | pabelanger: well, I left a comment on the change too | 21:35 |
| clarkb | I run geard all the time locally for random testing and such and don't want to have to set up statsd server or otherwise deal with packets being sent places | 21:35 |
| mordred | clarkb: yah - not requiring a statsd setup I think is important - but I think the latest version of the library properly no-ops if you don't have one | 21:35 |
| clarkb | ah | 21:35 |
| pabelanger | yah, it isn't a deal breaker adding it to requirements.txt, just noticed it wasn't installed when I tried to testing with geard | 21:35 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoint to delete summit by id https://review.openstack.org/558939 | 21:36 |
| clarkb | ianw: on the rand one I don't know that it will help https://github.com/puppetlabs/puppet/blob/3.8.7/lib/puppet/parser/functions/fqdn_rand.rb#L10 all those resources are on the same host so the value should be the same for each fqdn_rand(45) right? | 21:38 |
| clarkb | oh wait thats why you also have a seed derp | 21:38 |
| *** kjackal has quit IRC | 21:38 | |
| *** kjackal has joined #openstack-infra | 21:39 | |
| clarkb | ianw: the other drawback with per service venvs is every plugin will need to be updated too I Think | 21:40 |
| clarkb | Ideally we'd be able to fix this in a like for like of current behavior then change functionality on our own schedule (rather than pips) | 21:40 |
| corvus | pabelanger, mordred, clarkb: correct, statsd is optional on purpose. i think we support passing in the arguments though, so we should be set to upgrade our support to post 3.0 | 21:42 |
| corvus | (ie, we don't depend on the old env-var based initialization) | 21:42 |
| *** e0ne has quit IRC | 21:43 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: python-jenkins project updates https://review.openstack.org/558941 | 21:43 |
| ianw | clarkb: yeah, i think the per-service thing was supposed to be a bit of an api, but what plugins do, who knows | 21:44 |
| clarkb | ianw: I think the nodepool plugin is all venv already because we wanted to isolate nodepool the application from the cloud as far as python deps go | 21:45 |
| clarkb | but I'd have to double check and my hacky change may even break that | 21:45 |
| corvus | ianw, clarkb: i believe a single venv in devstack is the most expedient and correct thing to do to resolve the pip situation without changing too much else. i think anything more significant than that warrants a forum/ptg session to hash out all the implications. | 21:45 |
| corvus | clarkb: and yeah, nodepool is acting as a cloud consumer there. the fact that it happens to abuse the devstack plugin interface to set itself up shouldn't really mean it's part of the cloud. | 21:46 |
| corvus | if it's a problem, the right thing to do is not be a devstack plugin and just re-arrange the job to install nodepool after devstack | 21:47 |
| corvus | clarkb: i suspect it's set up that way because that was the easiest thing to do before zuulv3, but "gimme a devstack" is a bit easier now. | 21:47 |
| clarkb | it actually does the right thing | 21:48 |
| clarkb | and the reason we did it that way was to make it easier for local devs more so than testing | 21:48 |
| clarkb | if you want to dev on nodepool and test it functionally its like a two line devstack plugin config change and you are off | 21:48 |
| clarkb | nodepool devstack plugin isolates itself properly | 21:48 |
| clarkb | because one of the complaints we heard a lot of at the time was I don't have a cloud how do I test this | 21:49 |
| *** d0ugal has joined #openstack-infra | 21:49 | |
| *** d0ugal has quit IRC | 21:49 | |
| *** yamahata has joined #openstack-infra | 21:50 | |
| *** d0ugal has joined #openstack-infra | 21:50 | |
| corvus | ack | 21:52 |
| ianw | corvus: we do already have USE_VENV in devstack (for the separate venv's) and it looks like there's a few people who have used it http://codesearch.openstack.org/?q=USE_VENV&i=nope&files=&repos= | 21:52 |
| anteaya | thank you for the lovely list of comments on https://review.openstack.org/#/c/557979/ | 21:52 |
| anteaya | fungi, have you any initial thoughts on how I should address the fact that the puppet module requires puppet 4.5 or better with us being on 3 | 21:53 |
| anteaya | I hadn't even checked that myself so thank you | 21:53 |
| anteaya | I'm happy to make the other changes you suggest but if the module requires 4.5 I wonder about the use of doing so | 21:54 |
| fungi | anteaya: well, there's the "what happens if we just try it with puppet 3 anyway?" route, the "can we find an older revision in their git history compatible with puppet 3?" option and then the nuclear "let's update our systems to puppet 4!" | 21:54 |
| ianw | corvus: but even then, a codesearch on virtualenv in .*devstack.* files brings up things like http://git.openstack.org/cgit/openstack/valet/tree/devstack/plugin.sh#n36 which seems to source it's own virtualenv | 21:54 |
| anteaya | fungi, I'm not banking on the third option | 21:55 |
| anteaya | let me look at the second option | 21:55 |
| anteaya | I'm fine with the first option as well | 21:56 |
| anteaya | but would be nice to have some direction as a backup if that fails | 21:56 |
| fungi | looks like the earliest tagged state still claims to need newer puppet: https://github.com/martialblog/puppet-limesurvey/blob/v0.1.0/README.md | 21:56 |
| *** neiloy has quit IRC | 21:56 | |
| *** esberglu_ has quit IRC | 21:57 | |
| fungi | note it technically says it's "tested on...Puppet >=4.5" not that it "requires Puppet >=4.5" | 21:57 |
| anteaya | booo | 21:57 |
| anteaya | well okay so I'll go ahead and make your suggested changes and maybe it will work on 3? | 21:58 |
| *** boden has quit IRC | 21:59 | |
| *** florianf has quit IRC | 21:59 | |
| fungi | but yeah, the third commit to that repository added the mention of a puppet version to the readme, so looks like it's basically expected (or at least never been tested on anything older than) 4 | 22:00 |
| anteaya | doesn't hurt to try | 22:01 |
| anteaya | thanks | 22:01 |
| anteaya | I'm going to do my artistic interpretation of your recommendations and what I get wrong we can clarify from there? | 22:02 |
| fungi | take it away | 22:02 |
| anteaya | thanks | 22:03 |
| openstackgerrit | Merged openstack/python-jenkins master: repair .gitgnore https://review.openstack.org/548547 | 22:03 |
| pabelanger | I think the issue might be the data types syntax the puppet module is using, I don't believe puppet 3 supports that. However, our puppet-apply testing should try to use it | 22:04 |
| *** jamesmcarthur has quit IRC | 22:05 | |
| *** diablo_rojo has joined #openstack-infra | 22:06 | |
| mordred | pabelanger: oh look at that- type information! | 22:06 |
| corvus | ianw: i'm aware of USE_VENV. i'm trying to emphasize that i think the best solution to the looming pip10 crisis is not to use that, but instead, to add a new feature to install all of devstack in one venv, just as we do today at the system level. | 22:08 |
| clarkb | oh wow latest patchset of my hacky thing is working on python2 and is now going to run tempest | 22:08 |
| clarkb | there is at least one more bug there though that a new patchset will address | 22:08 |
| pabelanger | ianw: https://review.openstack.org/552939/ was my attempt to get USE_VENV working again, but i cannot figure out why cinder-rootwrap is failing | 22:09 |
| *** jamesmcarthur has joined #openstack-infra | 22:09 | |
| corvus | ianw: ie, clarkb's hacky thing :) | 22:10 |
| pabelanger | ianw: also seems neutron doesn't have logic for USE_VENV too | 22:10 |
| openstackgerrit | Merged openstack/python-jenkins master: get_build_console_output: fix docs for "number" parameter https://review.openstack.org/511954 | 22:10 |
| anteaya | fungi, am I removing the survey:apache class then in https://review.openstack.org/#/c/557979/1/modules/openstack_project/manifests/survey.pp ? | 22:13 |
| anteaya | sounds like I am | 22:13 |
| clarkb | turns out waiting for devstack/tempest to run is an excellent time to file expenses | 22:14 |
| *** rpioso is now known as rpioso|afk | 22:14 | |
| ianw | pabelanger: that is probably correctly, with the long history of q-* services, neutron-legacy and neutron | 22:14 |
| anteaya | clarkb, good to know | 22:15 |
| fungi | anteaya: yeah, we likely need to use ::httpd instead there, i'll get you an example | 22:15 |
| *** xgerman_ has quit IRC | 22:15 | |
| *** xgerman_ has joined #openstack-infra | 22:15 | |
| anteaya | ah okay I think I understand now, but I love examples | 22:15 |
| anteaya | fungi, and the class would be limesurvey::httpd yes? | 22:16 |
| *** lbragstad has quit IRC | 22:18 | |
| fungi | anteaya: nope, the limesurvey module doesn't have an httpd class | 22:18 |
| anteaya | okay | 22:18 |
| fungi | i'm suggesting putting something like this inside the openstack_project::survey class: http://git.openstack.org/cgit/openstack-infra/puppet-etherpad_lite/tree/manifests/apache.pp | 22:19 |
| * anteaya looks | 22:19 | |
| fungi | basically telling it to use the ::httpd base class (which is provided by our openstack-infra/puppet-httpd module) | 22:20 |
| fungi | so including ::httpd and instantiating the httpd::vhost class | 22:21 |
| *** felipemonteiro_ has quit IRC | 22:21 | |
| ianw | clarkb: i'm really surprised it's working without having to modify the $command in write_user_unit_file. how is systemd finding the virtualenv binary? | 22:21 |
| anteaya | ah thank you, I'll do my interpretive dance of what I'm seeing and hopefully it provides some entertainment | 22:21 |
| ianw | (virtualenv-ed binary, i.e. the command to start inside the virtualenv, in case not clear) | 22:21 |
| mordred | ianw: how does systemd do anything really? | 22:21 |
| clarkb | ianw: https://review.openstack.org/#/c/558930/3/inc/python ugly hack starting on line 49 | 22:21 |
| clarkb | ianw: apparently all the things use that function to determine their bin path | 22:22 |
| ianw | clarkb: oooohhhh, ok ... for python3. that's an interesting hack for that :) | 22:23 |
| clarkb | ianw: I don't think that hack is viable long term because people want to be able to run all of openstack but swift under python3 then do swift under python2 | 22:24 |
| clarkb | but for a quick can we make this work it seems to be doing well enough | 22:24 |
| openstackgerrit | Sorin Sbarnea proposed openstack/python-jenkins master: Replace testr with stestr https://review.openstack.org/558924 | 22:25 |
| anteaya | fungi, that example has code for bringing in auth_openid, I'm thinking I just copy everything there that looks useful | 22:26 |
| *** wolverineav has quit IRC | 22:26 | |
| fungi | anteaya: indeed, that's in fact why i chose it as an example ;) | 22:26 |
| anteaya | thank you :) | 22:26 |
| *** wolverineav has joined #openstack-infra | 22:26 | |
| clarkb | pabelanger: I think I may grok the rootwrap problem now too. I think the problem there is that rootwrap is installed in the venv but the path env for the running daemons is not set to include the venv? | 22:26 |
| clarkb | pabelanger: I think that needs fixing in my hacks and using USE_VENV | 22:27 |
| clarkb | pabelanger: we need to set the PATH var in the unit files I expect | 22:27 |
| *** inc01 has joined #openstack-infra | 22:27 | |
| *** jamesmcarthur has quit IRC | 22:29 | |
| *** yamamoto has joined #openstack-infra | 22:30 | |
| *** inc0 has quit IRC | 22:31 | |
| *** wolverineav has quit IRC | 22:31 | |
| pabelanger | clarkb: maybe, but I think it is only cinder that is failing. I was able to fix nova by doing: https://review.openstack.org/#/c/552939/19/inc/rootwrap | 22:32 |
| pabelanger | but the unit file for rootwrap seems right | 22:33 |
| pabelanger | I'll have to find time to run it locally and inspect | 22:33 |
| clarkb | oh is rootwrap running as a daemon everywhere now? if so then ya likely issue I thought we had is non issue and just have to configure the daemon like you did | 22:34 |
| clarkb | pabelanger: the sudo stuff seems to do all the right things to allow rootwrap in a venv to be executed by stack users with sudo | 22:35 |
| pabelanger | not sure, I'm not really up to speed on how rootwrap works. But looking at OSA, that seemed to be how they supported it | 22:35 |
| clarkb | pabelanger: ya there are two modes, one as daemon and one openstack just forks it each time. I'm not sure which setup devstack uses | 22:36 |
| *** yamamoto has quit IRC | 22:36 | |
| pabelanger | k | 22:36 |
| *** olaph has joined #openstack-infra | 22:38 | |
| *** olaph1 has quit IRC | 22:39 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 22:41 |
| *** aeng has joined #openstack-infra | 22:42 | |
| clarkb | pabelanger: http://logs.openstack.org/30/558930/3/check/devstack-platform-centos-7/a4e66e2/job-output.txt.gz#_2018-04-04_22_35_49_120835 that looks suspiciously like what I'm worried about | 22:42 |
| pabelanger | clarkb: yah, that looks like the issue I have | 22:42 |
| clarkb | pabelanger: http://logs.openstack.org/30/558930/3/check/devstack-platform-centos-7/a4e66e2/controller/logs/screen-c-vol.txt.gz#_Apr_04_22_35_29_327662 except it isn't rootwrap its some other thing | 22:43 |
| clarkb | pabelanger: so ya I think we need to put the venvs in the unit paths | 22:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Check out more appropriate branches of role and playbook repos https://review.openstack.org/558643 | 22:43 |
| *** bradm_ is now known as bradm | 22:44 | |
| pabelanger | clarkb: http://logs.openstack.org/39/552939/19/check/tempest-full/8436526/controller/logs/screen-c-vol.txt.gz#_Apr_02_05_43_41_231994 was my error, but you can see that is inside the virtualenv | 22:44 |
| clarkb | pabelanger: ya the problem is it can't find the cinder-rootwrap command | 22:45 |
| clarkb | so same underlying issue we need to put venvs in the path | 22:45 |
| pabelanger | http://logs.openstack.org/39/552939/19/check/tempest-full/8436526/job-output.txt.gz#_2018-04-02_05_31_51_256186 | 22:45 |
| pabelanger | is where it should be | 22:45 |
| clarkb | I'm adding this to my patch now | 22:45 |
| *** rcernin has joined #openstack-infra | 22:45 | |
| pabelanger | k | 22:46 |
| clarkb | pabelanger: https://review.openstack.org/558930 the functions-common file diff | 22:47 |
| clarkb | pabelanger: note that since my hack is adding the virtualenvs to the path I just pass path through directly | 22:47 |
| clarkb | in your case you'll likely want venvpath/bin:$PATH | 22:47 |
| *** rh-jelabarre has quit IRC | 22:49 | |
| clarkb | oddly that doesn't seem to have affected the ubuntu jobs as much | 22:49 |
| clarkb | I suspect I'm getting close to needing to let the main python2 and python3 jobs finish then dig through logs to see why tempest is failing | 22:50 |
| *** mriedem is now known as mriedem_away | 22:51 | |
| pabelanger | yah, lets see if that works | 22:51 |
| dmsimard | oh dear.. I installed cowsay to get presentty to print cows and now there's cows all over my ansible things | 22:51 |
| *** eernst has quit IRC | 22:51 | |
| dmsimard | the cowsay implementation defaults to true if cowsay is installed | 22:52 |
| dmsimard | T_T | 22:52 |
| *** eernst has joined #openstack-infra | 22:52 | |
| clarkb | dmsimard: yup | 22:52 |
| corvus | dmsimard: not that you would *ever* want to change it.... but: export ANSIBLE_NOCOWS=1 | 22:52 |
| dmsimard | corvus: :D | 22:53 |
| corvus | (that may or may not be in my .bash_profile) | 22:53 |
| dmsimard | I thought that was new from Ansible 2.5 for a moment | 22:53 |
| dmsimard | I was like.. did they really add force default cows in ? | 22:53 |
| *** eernst_ has joined #openstack-infra | 22:53 | |
| *** eernst_ has quit IRC | 22:56 | |
| *** eernst has quit IRC | 22:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add haproxy maintenance instructions https://review.openstack.org/558840 | 22:57 |
| openstackgerrit | Merged openstack-infra/system-config master: Update connection construction in launch_node https://review.openstack.org/558583 | 22:57 |
| *** bobh has quit IRC | 22:57 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add error messages to dns.py about old sdk https://review.openstack.org/558256 | 22:57 |
| *** felipemonteiro has joined #openstack-infra | 23:00 | |
| pabelanger | ianw: clarkb: do we want to land fedora-26 remove today or leave until morning: https://review.openstack.org/558847/ | 23:02 |
| clarkb | pabelanger: maybe leave til morning so that we can field questions? I guess ianw can field them solo if he wants | 23:02 |
| pabelanger | sure | 23:02 |
| *** d0ugal has quit IRC | 23:03 | |
| pabelanger | https://review.openstack.org/558841/ also removes some unused fedora-26 jobs too | 23:03 |
| *** wolverineav has joined #openstack-infra | 23:06 | |
| *** hongbin_ has quit IRC | 23:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add git timeout setting for nodepool-builder https://review.openstack.org/557845 | 23:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add git timeout setting for nodepool-builder https://review.openstack.org/557845 | 23:09 |
| pabelanger | ianw: we should maybe rebase the debian-stretch a top of ^ | 23:10 |
| *** dhinesh has joined #openstack-infra | 23:10 | |
| ianw | pabelanger: ok | 23:11 |
| *** dhinesh has quit IRC | 23:12 | |
| anteaya | zuul is unhappy with https://review.openstack.org/557979 the puppet apply tests, I'm rusty reading logs anyway and the logs have changed significantly since the last time I looked | 23:14 |
| anteaya | anyone feel like helping me find the dir/file with the actual error in it? | 23:14 |
| anteaya | I found http://logs.openstack.org/79/557979/2/check/legacy-infra-puppet-apply-3/6583728/job-output.txt.gz#_2018-04-04_22_54_22_727909 | 23:14 |
| anteaya | but I don't know if that is THE error | 23:14 |
| pabelanger | anteaya: if you look in the applytest folder, you should see out.FAILED, that will give more details: http://logs.openstack.org/79/557979/2/check/legacy-infra-puppet-apply-3/6583728/openstack-infra/system-config/applytest/ | 23:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add debian stretch nodes to arm64/amd64 clouds https://review.openstack.org/557902 | 23:17 |
| ianw | pabelanger: ^ rebased. happy to put the f26 removal in today | 23:17 |
| *** aeng has quit IRC | 23:18 | |
| anteaya | pabelanger, thanks Paul | 23:19 |
| *** aeng has joined #openstack-infra | 23:19 | |
| pabelanger | ianw: -1, i think you removed debian-stretch diskimage in nodepool.yaml | 23:20 |
| pabelanger | anteaya: np | 23:20 |
| ianw | pabelanger: urgh, sorry must have pushed the wrong button in emerge | 23:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add debian stretch nodes to arm64/amd64 clouds https://review.openstack.org/557902 | 23:22 |
| pabelanger | +2 | 23:23 |
| pabelanger | clarkb: mind a +3 again on https://review.openstack.org/557845/ your vote was dropped on a rebase conflict | 23:24 |
| anteaya | I see the error: http://logs.openstack.org/79/557979/2/check/legacy-infra-puppet-apply-3/6583728/openstack-infra/system-config/applytest/puppetapplytest38.final.out.FAILED I don't understand what the problem is though | 23:24 |
| anteaya | it appears to be telling me I didn't give it enough )'s | 23:24 |
| *** wolverineav has quit IRC | 23:24 | |
| anteaya | but they all look closed to me | 23:24 |
| *** wolverineav has joined #openstack-infra | 23:25 | |
| *** tosky has quit IRC | 23:25 | |
| ianw | anteaya: is https://github.com/martialblog/puppet-limesurvey/blob/master/manifests/init.pp#L65 the line it's talking about | 23:26 |
| ianw | ? | 23:26 |
| ianw | i feel like "String $foo" is the data type stuff, and may not be puppet3 ? | 23:27 |
| pabelanger | yah, I think this might be puppet 3 raising a syntax error for puppet 4 code | 23:27 |
| anteaya | perhaps that is the line 65 it is talking about | 23:27 |
| ianw | i'm fairly sure that's it, i have great deja vu seeing that before | 23:28 |
| anteaya | thanks ianw | 23:28 |
| anteaya | I'm stuck on what to do about it, but yeah that sounds like that is the problem | 23:29 |
| *** wolverineav has quit IRC | 23:29 | |
| ianw | yeah, i think in my case i was lucky that i went back to a tag that worked with puppet3 ... i'm not sure there's good solutions other than maintaining a fork, or yak shaving puppet >=4 | 23:30 |
| *** Guest98296 has joined #openstack-infra | 23:30 | |
| anteaya | well we aren't even sure that this module with stand up a server since we use puppet 3 | 23:31 |
| anteaya | so I will look for other opinions tomorrow and see if we have additional thoughts | 23:31 |
| anteaya | since I'm copy pasting here any yak shaving I do will take a lot of time of others | 23:32 |
| *** yamamoto has joined #openstack-infra | 23:32 | |
| anteaya | so that isn't my first choice | 23:32 |
| anteaya | but thanks for the help locating the error | 23:32 |
| *** yamamoto has quit IRC | 23:38 | |
| *** olaph1 has joined #openstack-infra | 23:39 | |
| *** olaph has quit IRC | 23:40 | |
| *** yamahata has quit IRC | 23:42 | |
| *** iyamahat has quit IRC | 23:44 | |
| clarkb | pabelanger: https://review.openstack.org/558930 now has a slightly different appraoch than yours for the rootwrap config. In this case we have to set path in those envs anyways so just relying on that for rootwrap | 23:47 |
| clarkb | pabelanger: it is potentially less secure but its devstack | 23:47 |
| clarkb | dansmith: just to follow up before end of the day the last limestone timeout was about two housr ago and the one before that was 3 hours before that and the mitigation went in about an hour before the one ~5 hours ago | 23:50 |
| clarkb | dansmith: all that to say I think we've largely addressed it at this poit but I'll keep an eye on that logstash query | 23:50 |
| dansmith | clarkb: sweet thanks | 23:52 |
| johnsom | Wondering if there are known issues with the new limestone hosts? Have a gate run that timed out and it doesn't appear devstack ever started: http://logs.openstack.org/92/558892/1/gate/octavia-v1-dsvm-py3x-scenario-multinode/ca6c90a/ | 23:57 |
| johnsom | http://logs.openstack.org/92/558892/1/gate/octavia-v1-dsvm-py3x-scenario-multinode/ca6c90a/job-output.txt.gz#_2018-04-04_23_43_26_148350 | 23:57 |
| *** wolverineav has joined #openstack-infra | 23:57 | |
| clarkb | johnsom: thats what I was just talking about with dansmith above. tldr is we were oversubscribed on cpu we have since moved to 60% of the old instance quota which has drastically reduced the noise | 23:58 |
| johnsom | Ah, ok. I scrolled back but didn't put it together. | 23:58 |
| ianw | fungi / corvus : just reading the scrollback on the git08 stuff. to get networking back, i had to detach and re-attach the interface on the host, and then update dns. lmn, i can do that for git08 if you want | 23:58 |
| clarkb | johnsom: I think its largely corrected at this point but we'll have to continue to monitor it | 23:58 |
| ianw | fungi / corvus: it probably would have been easier to just recreate the ze instance i did it on; not sure with a git host | 23:59 |
| johnsom | Ok, cool! Thanks - recheck in flight | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!