| *** dingyichen has joined #openstack-infra | 00:00 | |
| *** SumitNaiksatam has quit IRC | 00:01 | |
| *** SumitNaiksatam has joined #openstack-infra | 00:02 | |
| *** bobh has joined #openstack-infra | 00:04 | |
| *** hamerins has quit IRC | 00:05 | |
| *** jamesmcarthur has joined #openstack-infra | 00:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 00:06 |
|---|---|---|
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 00:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 00:06 |
| mordred | fungi: ^^ | 00:06 |
| *** SumitNaiksatam has quit IRC | 00:07 | |
| mordred | I think that's still missing one thing - which is updating the update_puppet_version.yaml playbook to handle installing puppet when needed | 00:07 |
| mordred | but it's EOD now | 00:07 |
| *** bobh has quit IRC | 00:09 | |
| fungi | taking a look now | 00:13 |
| *** bobh has joined #openstack-infra | 00:15 | |
| *** bobh has quit IRC | 00:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Enable Python 2.7 unit tests https://review.openstack.org/592767 | 00:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add tox-py27 jobs to zuul-jobs https://review.openstack.org/592768 | 00:23 |
| *** bobh has joined #openstack-infra | 00:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Enable Python 2.7 unit tests https://review.openstack.org/592767 | 00:23 |
| *** bobh has quit IRC | 00:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: verify tox-py27 works https://review.openstack.org/592771 | 00:34 |
| *** nicolasbock has quit IRC | 00:36 | |
| *** longkb has joined #openstack-infra | 00:36 | |
| *** bobh has joined #openstack-infra | 00:47 | |
| *** bobh has quit IRC | 00:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: small python2/3 update https://review.openstack.org/592774 | 00:56 |
| *** dave-mccowan has joined #openstack-infra | 00:58 | |
| *** hongbin has joined #openstack-infra | 01:04 | |
| *** slaweq has joined #openstack-infra | 01:06 | |
| *** bobh has joined #openstack-infra | 01:10 | |
| *** sambetts has quit IRC | 01:14 | |
| *** bobh has quit IRC | 01:14 | |
| *** bobh has joined #openstack-infra | 01:15 | |
| *** yamahata has quit IRC | 01:16 | |
| *** slaweq has quit IRC | 01:16 | |
| *** bobh has quit IRC | 01:19 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:20 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Bump ansible test version to 2.5 https://review.openstack.org/592776 | 01:20 |
| *** jamesmcarthur has quit IRC | 01:24 | |
| *** bobh has joined #openstack-infra | 01:26 | |
| *** mrsoul has joined #openstack-infra | 01:26 | |
| *** bobh has quit IRC | 01:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Use collections Sequence for FileList https://review.openstack.org/592778 | 01:34 |
| *** r-daneel has quit IRC | 01:35 | |
| *** bobh has joined #openstack-infra | 01:36 | |
| *** ianychoi has joined #openstack-infra | 01:47 | |
| *** jamesmcarthur has joined #openstack-infra | 01:51 | |
| *** jamesmcarthur has quit IRC | 01:56 | |
| *** bobh has quit IRC | 02:14 | |
| *** rosmaita has quit IRC | 02:15 | |
| *** gcb_ has joined #openstack-infra | 02:19 | |
| *** agopi|out has joined #openstack-infra | 02:22 | |
| *** agopi|out is now known as agopi | 02:24 | |
| *** bobh has joined #openstack-infra | 02:35 | |
| *** rlandy|bbl is now known as rlandy | 02:41 | |
| *** rlandy has quit IRC | 02:42 | |
| *** armax has quit IRC | 02:45 | |
| *** jamesmcarthur has joined #openstack-infra | 02:49 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| *** bobh has quit IRC | 02:58 | |
| *** roman_g has quit IRC | 03:06 | |
| *** jamesmcarthur has joined #openstack-infra | 03:09 | |
| *** ssbarnea has joined #openstack-infra | 03:28 | |
| *** jamesmcarthur has quit IRC | 03:35 | |
| *** jamesmcarthur has joined #openstack-infra | 03:39 | |
| *** dave-mccowan has quit IRC | 03:45 | |
| *** udesale has joined #openstack-infra | 03:47 | |
| *** SumitNaiksatam has quit IRC | 03:50 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:51 | |
| *** hongbin has quit IRC | 03:57 | |
| *** vivsoni_ has quit IRC | 03:59 | |
| *** jamesmcarthur has quit IRC | 04:01 | |
| *** jamesmcarthur has joined #openstack-infra | 04:05 | |
| *** Bhujay has joined #openstack-infra | 04:06 | |
| *** jamesmcarthur has quit IRC | 04:11 | |
| *** ssbarnea has quit IRC | 04:12 | |
| *** ssbarnea has joined #openstack-infra | 04:21 | |
| *** vivsoni has joined #openstack-infra | 04:25 | |
| *** ramishra has joined #openstack-infra | 04:32 | |
| *** yamahata has joined #openstack-infra | 04:38 | |
| *** janki has joined #openstack-infra | 04:39 | |
| *** SumitNaiksatam has quit IRC | 04:48 | |
| *** jamesdenton has quit IRC | 04:51 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:54 | |
| *** apetrich has joined #openstack-infra | 05:05 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Enable Python 2.7 unit tests https://review.openstack.org/592767 | 05:07 |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for vitrage repositories https://review.openstack.org/592828 | 05:16 |
| *** e0ne has joined #openstack-infra | 05:20 | |
| *** Douhet has quit IRC | 05:25 | |
| *** Douhet has joined #openstack-infra | 05:26 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: [WIP] remove job settings for vitrage repositories https://review.openstack.org/592828 | 05:35 |
| AJaeger | Ng: are you nguyen Hai? "[WIP]" is not needed at all, there's a Workflow -1 button which marks a change as work-in-progress. | 05:41 |
| AJaeger | Ng, that single click is all that needs - now you have to update your change several times, rerun tests etc. | 05:41 |
| * AJaeger should have run whois directly - sorry, Ng. Wrong person... | 05:42 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for vitrage repositories https://review.openstack.org/592828 | 05:50 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 05:51 |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for cyborg repositories https://review.openstack.org/592849 | 05:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Small python2/3 update https://review.openstack.org/592774 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Bump ansible test version to 2.5 https://review.openstack.org/592776 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Use collections Sequence for FileList https://review.openstack.org/592778 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Cleanup temporary directories https://review.openstack.org/592340 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Create a download script https://review.openstack.org/592341 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Turn FileList into a context manager https://review.openstack.org/592850 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Keep the FileList in the indexer class https://review.openstack.org/592851 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Make indexer more generic https://review.openstack.org/592852 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.openstack.org/592853 | 05:57 |
| *** mrsoul is now known as mschuppert | 05:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Create a download script https://review.openstack.org/592341 | 06:05 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.openstack.org/592853 | 06:05 |
| *** Tahvok has quit IRC | 06:06 | |
| *** Bhujay has quit IRC | 06:07 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for designate repositories https://review.openstack.org/592865 | 06:08 |
| *** Tahvok has joined #openstack-infra | 06:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 06:14 |
| *** graphene has quit IRC | 06:19 | |
| *** graphene has joined #openstack-infra | 06:21 | |
| *** jamesmcarthur has joined #openstack-infra | 06:22 | |
| *** jamesmcarthur has quit IRC | 06:27 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 06:40 |
| *** e0ne has quit IRC | 06:44 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 06:47 |
| *** pcaruana has joined #openstack-infra | 06:54 | |
| *** alexchadin has joined #openstack-infra | 06:54 | |
| *** rcernin has quit IRC | 06:55 | |
| *** slaweq has joined #openstack-infra | 06:57 | |
| *** annp has quit IRC | 07:02 | |
| *** longkb has quit IRC | 07:03 | |
| *** longkb has joined #openstack-infra | 07:03 | |
| *** dpawlik has joined #openstack-infra | 07:08 | |
| *** dpawlik has quit IRC | 07:13 | |
| *** dpawlik has joined #openstack-infra | 07:14 | |
| *** Bhujay has joined #openstack-infra | 07:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 07:22 |
| *** SteffanW has joined #openstack-infra | 07:27 | |
| *** ykarel has joined #openstack-infra | 07:31 | |
| *** shardy has joined #openstack-infra | 07:39 | |
| *** shardy has quit IRC | 07:40 | |
| *** shardy has joined #openstack-infra | 07:41 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for etherpad.o.o https://review.openstack.org/590029 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for ethercalc.o.o https://review.openstack.org/590030 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for eavesdrop.o.o https://review.openstack.org/590048 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for paste.o.o https://review.openstack.org/590492 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for planet.o.o https://review.openstack.org/590493 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for logstash-worker01 https://review.openstack.org/590499 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for all logstash workers https://review.openstack.org/590500 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for firehose.o.o https://review.openstack.org/591500 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for review and review01 https://review.openstack.org/591516 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for grafana.o.o https://review.openstack.org/592640 | 07:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn the future parser on for health.o.o https://review.openstack.org/592647 | 07:44 |
| *** graphene has quit IRC | 07:44 | |
| *** graphene has joined #openstack-infra | 07:46 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:46 | |
| *** jaosorior has quit IRC | 07:47 | |
| *** zhenguo has joined #openstack-infra | 07:50 | |
| *** electrofelix has joined #openstack-infra | 07:59 | |
| *** e0ne has joined #openstack-infra | 07:59 | |
| *** longkb has quit IRC | 08:02 | |
| *** longkb has joined #openstack-infra | 08:02 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for solum repositories https://review.openstack.org/592923 | 08:08 |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 08:18 |
| *** gcb_ has quit IRC | 08:20 | |
| *** jpich has joined #openstack-infra | 08:21 | |
| *** annp has joined #openstack-infra | 08:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Make indexer more generic https://review.openstack.org/592852 | 08:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Create a download script https://review.openstack.org/592341 | 08:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.openstack.org/592853 | 08:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Stub out dry run in the uploader https://review.openstack.org/592929 | 08:33 |
| *** derekh has joined #openstack-infra | 08:35 | |
| *** e0ne has quit IRC | 08:37 | |
| *** ykarel|lunch is now known as ykarel | 08:56 | |
| *** mandre is now known as mandre_off | 09:04 | |
| *** alexchadin has quit IRC | 09:07 | |
| *** panda|off is now known as panda | 09:08 | |
| *** alexchadin has joined #openstack-infra | 09:09 | |
| *** roman_g has joined #openstack-infra | 09:10 | |
| *** Bhujay has quit IRC | 09:13 | |
| *** Bhujay has joined #openstack-infra | 09:15 | |
| *** udesale has quit IRC | 09:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-role-puppet master: Add ability to override mgmt manifest path https://review.openstack.org/592946 | 09:52 |
| *** jiapei has joined #openstack-infra | 09:56 | |
| *** dayou has joined #openstack-infra | 09:56 | |
| *** zhenguo has quit IRC | 10:00 | |
| *** e0ne has joined #openstack-infra | 10:09 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for congress repositories https://review.openstack.org/592970 | 10:12 |
| *** alexchadin has quit IRC | 10:15 | |
| *** EmilienM is now known as EvilienM | 10:15 | |
| *** ansmith has quit IRC | 10:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-role-puppet master: Add ability to override mgmt manifest path https://review.openstack.org/592946 | 10:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add base.yaml to kick.sh https://review.openstack.org/592760 | 10:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 10:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 10:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 10:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Turn on fact caching https://review.openstack.org/592971 | 10:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppet variables and playbooks https://review.openstack.org/592972 | 10:22 |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for masakari repositories https://review.openstack.org/592992 | 10:30 |
| *** ansmith has joined #openstack-infra | 10:31 | |
| *** udesale has joined #openstack-infra | 10:31 | |
| *** ociuhandu has joined #openstack-infra | 10:32 | |
| *** ociuhandu has quit IRC | 10:34 | |
| *** graphene has quit IRC | 10:35 | |
| *** nicolasbock has joined #openstack-infra | 10:35 | |
| *** graphene has joined #openstack-infra | 10:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppet variables and playbooks https://review.openstack.org/592972 | 10:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 10:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 10:41 |
| *** graphene has quit IRC | 10:43 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 10:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Turn on fact caching https://review.openstack.org/592971 | 10:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add base.yaml to kick.sh https://review.openstack.org/592760 | 10:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 10:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppet variables and playbooks https://review.openstack.org/592972 | 10:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 10:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 10:44 |
| *** graphene has joined #openstack-infra | 10:45 | |
| *** alexchadin has joined #openstack-infra | 10:45 | |
| *** alexchadin has quit IRC | 10:51 | |
| *** dklyle has quit IRC | 10:55 | |
| *** david-lyle has joined #openstack-infra | 10:55 | |
| *** ramishra has quit IRC | 11:04 | |
| *** jpena is now known as jpena|lunch | 11:04 | |
| *** graphene has quit IRC | 11:04 | |
| *** graphene has joined #openstack-infra | 11:06 | |
| *** longkb has quit IRC | 11:07 | |
| *** annp has quit IRC | 11:09 | |
| *** rh-jelabarre has joined #openstack-infra | 11:15 | |
| *** dave-mccowan has joined #openstack-infra | 11:19 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for tacker repositories https://review.openstack.org/593027 | 11:23 |
| *** ramishra has joined #openstack-infra | 11:23 | |
| *** alexchadin has joined #openstack-infra | 11:24 | |
| *** shardy has quit IRC | 11:24 | |
| *** shardy has joined #openstack-infra | 11:24 | |
| *** graphene has quit IRC | 11:26 | |
| *** udesale has quit IRC | 11:26 | |
| *** graphene has joined #openstack-infra | 11:27 | |
| *** alexchadin has quit IRC | 11:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppetmaster puppet config handled by ansible https://review.openstack.org/592215 | 11:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add playbook to install openstack credentials https://review.openstack.org/593029 | 11:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove purge_apt_sources https://review.openstack.org/592214 | 11:37 |
| *** graphene has quit IRC | 11:37 | |
| *** alexchadin has joined #openstack-infra | 11:38 | |
| *** graphene has joined #openstack-infra | 11:39 | |
| *** notmyname has quit IRC | 11:42 | |
| *** notmyname has joined #openstack-infra | 11:43 | |
| *** tpsilva has joined #openstack-infra | 11:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 11:47 |
| *** jamesdenton has joined #openstack-infra | 11:47 | |
| *** jamesmcarthur has joined #openstack-infra | 11:52 | |
| *** graphene has quit IRC | 11:55 | |
| *** graphene has joined #openstack-infra | 11:56 | |
| *** jamesmcarthur has quit IRC | 11:57 | |
| *** boden has joined #openstack-infra | 11:59 | |
| *** jpena|lunch is now known as jpena | 12:00 | |
| *** graphene has quit IRC | 12:07 | |
| *** graphene has joined #openstack-infra | 12:08 | |
| *** rcarrillocruz has joined #openstack-infra | 12:12 | |
| *** dpawlik has quit IRC | 12:18 | |
| mordred | cmurphy: if you have a sec - I feel like I'm mising something dumb on the failures here: https://review.openstack.org/#/c/592763 | 12:18 |
| * cmurphy looks | 12:20 | |
| *** eharney has joined #openstack-infra | 12:22 | |
| cmurphy | mordred: hiera.yaml relies on the puppet environment http://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/templates/hiera.yaml.j2#n16 did you change that somewhere? | 12:24 |
| mordred | cmurphy: ah. I bet that's it | 12:25 |
| *** dpawlik has joined #openstack-infra | 12:25 | |
| *** efried has quit IRC | 12:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 12:28 |
| mordred | cmurphy: trying a hacky workaround first | 12:29 |
| *** efried has joined #openstack-infra | 12:29 | |
| cmurphy | the best kind of workaround | 12:29 |
| mordred | you know it | 12:29 |
| *** dpawlik has quit IRC | 12:29 | |
| *** dbecker has joined #openstack-infra | 12:31 | |
| cmurphy | wow the way that gets rendered in the gerrit view and the git diff does not make it obvious what it is | 12:32 |
| cmurphy | but i think it's not going to work | 12:33 |
| cmurphy | $ ls production | 12:33 |
| cmurphy | ls: cannot access 'production': Too many levels of symbolic link | 12:33 |
| *** bobh has joined #openstack-infra | 12:35 | |
| *** trown|outtypewww is now known as trown | 12:35 | |
| *** bobh has quit IRC | 12:36 | |
| *** bobh has joined #openstack-infra | 12:36 | |
| mordred | oh - poop | 12:37 |
| mordred | cmurphy: I just can't make symlinks | 12:38 |
| mordred | it's too hard | 12:38 |
| cmurphy | it is pretty much the hardest part of *nix | 12:38 |
| *** rlandy has joined #openstack-infra | 12:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 12:40 |
| *** bobh has quit IRC | 12:41 | |
| mordred | cmurphy: fwiw, "ln -s production ." is bad. "ln -s . production" is good | 12:41 |
| cmurphy | yeah that looks better | 12:41 |
| slaweq | hi, question about grafana | 12:46 |
| slaweq | is there any way to change graphs "online" - I would like to check some options, how graph would looks like and then propose a patch to project_config repo | 12:47 |
| slaweq | is it possible somehow? | 12:47 |
| slaweq | (I'm asking about http://grafana.openstack.org/) | 12:47 |
| *** jamesdenton has quit IRC | 12:48 | |
| *** jamesdenton has joined #openstack-infra | 12:49 | |
| *** jcoufal has joined #openstack-infra | 12:49 | |
| mordred | slaweq: I don't _think_ so - other than being able to make individual graphs at graphite.openstack.org | 12:50 |
| mordred | pabelanger: ^^ you're smarter about that than I am | 12:50 |
| *** jamesdenton has quit IRC | 12:52 | |
| *** alexchadin has quit IRC | 12:52 | |
| *** rosmaita has joined #openstack-infra | 12:53 | |
| *** TheJulia is now known as needssleep | 12:55 | |
| *** jiapei has quit IRC | 12:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 12:56 |
| mordred | cmurphy: ok - I think maybe that ^^ will do a better job. I think the issue is that on the _remote_ nodes (where puppet is running) we still need production in the mix, so update prep-apply to remove the production from the path is wrong | 12:57 |
| *** jamesdenton has joined #openstack-infra | 12:57 | |
| *** dpawlik has joined #openstack-infra | 12:58 | |
| cmurphy | mordred: ++ that sounds probably true | 12:58 |
| *** dbecker has quit IRC | 13:04 | |
| *** janki has quit IRC | 13:08 | |
| *** janki has joined #openstack-infra | 13:12 | |
| *** efried is now known as fried_rice | 13:12 | |
| mordred | cmurphy: woot! that time it's passing tests. thanks for the help! | 13:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppet variables and playbooks https://review.openstack.org/592972 | 13:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 13:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 13:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add playbook to install openstack credentials https://review.openstack.org/593029 | 13:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppetmaster puppet config handled by ansible https://review.openstack.org/592215 | 13:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove purge_apt_sources https://review.openstack.org/592214 | 13:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove restricted and multiverse from ubuntu servers https://review.openstack.org/593047 | 13:13 |
| mordred | fungi: ^^ I made you a bunch of presents this morning | 13:13 |
| cmurphy | \o/ | 13:15 |
| *** jcoufal has quit IRC | 13:18 | |
| *** jcoufal has joined #openstack-infra | 13:19 | |
| *** kevinshort has joined #openstack-infra | 13:21 | |
| *** ramishra has quit IRC | 13:33 | |
| *** dbecker has joined #openstack-infra | 13:33 | |
| *** bobh has joined #openstack-infra | 13:35 | |
| *** electrofelix has quit IRC | 13:36 | |
| *** derekh has quit IRC | 13:38 | |
| *** kevinshort has quit IRC | 13:43 | |
| *** kevinshort has joined #openstack-infra | 13:45 | |
| *** felipemonteiro has joined #openstack-infra | 13:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Install ssh key on bridge.openstack.org https://review.openstack.org/593053 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set mgmt_puppet_module_dir publically https://review.openstack.org/593054 | 13:51 |
| *** derekh has joined #openstack-infra | 13:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set mgmt_puppet_module_dir publically https://review.openstack.org/593054 | 13:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppetmaster puppet config handled by ansible https://review.openstack.org/592215 | 13:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove purge_apt_sources https://review.openstack.org/592214 | 13:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove restricted and multiverse from ubuntu servers https://review.openstack.org/593047 | 13:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Install ssh key on bridge.openstack.org https://review.openstack.org/593053 | 13:54 |
| *** dansmith is now known as steelydan | 13:58 | |
| *** steelydan is now known as SteelyDan | 13:58 | |
| mnaser | clarkb: good morning, do you know if there is any archives of the discussions that happened regarding nested virt with ovh and limestone | 13:58 |
| *** pbourke has quit IRC | 13:58 | |
| mnaser | question is we're rolling it out soon across all HVs, i think it has been relatively stable with kata, but i don't know how stable it will be for openstack | 13:59 |
| *** Bhujay has quit IRC | 13:59 | |
| *** ramishra has joined #openstack-infra | 14:00 | |
| *** pbourke has joined #openstack-infra | 14:00 | |
| *** rosmaita has quit IRC | 14:00 | |
| mordred | mnaser: well, we disable it by default even if it's there - so you rolling it out shouldn't impact anything openstack-wise | 14:00 |
| mnaser | mordred: really? i could swear projects had workarounds to disable it.. unless they have to do without devstack-gate? | 14:01 |
| *** rpioso|afk is now known as rpioso | 14:03 | |
| mordred | mnaser: the general policy is to not use nested-virt because it's too unstable and the errors are too weird. there are a few projects who have done some work to enable it anyway I think? that said - for people who don't make use of the base node setup in the multinode base job (or devstack-gate for legacy jobs) - it's possible people will have had to duplicate existing work | 14:03 |
| mordred | mnaser: but - I think you were right to ping clarkb - he knows much more the state than I do | 14:05 |
| *** armax has joined #openstack-infra | 14:08 | |
| mnaser | mordred: thanks for all that information | 14:08 |
| mnaser | good to know i wont break everyone in rc time :) | 14:09 |
| * mordred just says words | 14:09 | |
| *** derekh has quit IRC | 14:11 | |
| fungi | AJaeger: yeah, Ng is an old-timer in here (he's awesome, btw!). no clue if the author of 592828 is in here, but plugging their committer e-mail into the openstackid-resources lookup api, the irc nick field on their foundation profile isn't filled in | 14:16 |
| mordred | corvus: I see you stirring ... hopefully you're enjoy the giant pile of new update-cfg-mgmt patches I've prepared for your enjoyment | 14:17 |
| *** dbecker has quit IRC | 14:17 | |
| mordred | s/you're/you'll/ | 14:18 |
| fungi | it is rather a pile (in a good way) | 14:18 |
| *** rosmaita has joined #openstack-infra | 14:18 | |
| smcginnis | Would it be possible to check if the image cache has been refreshed since yesterday? We have a branch recreation to do yet and I'd like to make sure it won't still see the old deleted one. | 14:19 |
| fungi | mmm, rax fantastic support says they rebooted the host under ze02, checking on it real quick | 14:20 |
| mnaser | infra-root: can we check on the rsync job for opensuse mirrors | 14:25 |
| mnaser | http://logs.openstack.org/33/593033/3/check/openstack-ansible-functional-opensuse-423/539092a/logs/host/lxc-cache-prep-commands.log.txt.gz shows it giving a 404 | 14:25 |
| mnaser | i cant find that file in here either http://mirror.iad.rax.openstack.org/opensuse/update/leap/42.3/oss/x86_64/ | 14:25 |
| mnaser | i dont know much how suse's packager works but it looks like it syncd the metadata without the packages (or failed midway) | 14:26 |
| *** dbecker has joined #openstack-infra | 14:27 | |
| mnaser | well looks like the source is missing that file too http://mirrors.rit.edu/opensuse/update/leap/42.3/oss/x86_64/ | 14:28 |
| mnaser | http://mirror.rackspace.com/openSUSE/update/leap/42.3/oss/x86_64/ i cant find it anywhere | 14:29 |
| *** shardy has quit IRC | 14:29 | |
| cmurphy | it's in here http://download.opensuse.org/update/leap/42.3/oss/x86_64/ | 14:30 |
| fungi | #status log the hypervisor host for ze02 was restarted, server up since 22:53z, seems to be running jobs normally | 14:30 |
| openstackstatus | fungi: finished logging | 14:30 |
| mnaser | cmurphy: interesting that im not finding it on any other mirror except that | 14:32 |
| cmurphy | that's pretty weird | 14:32 |
| corvus | mordred, slaweq: what i do is run grafana locally in a container, and add graphite.openstack.org as a data source. you can copy the "panel json" from grafana.openstack.org into the local instance and modify it. | 14:32 |
| fungi | mnaser: cmurphy: i believe i heard that opensuse rsync source is itself a separate mirror, so the file may be broken there and not on the official download site | 14:32 |
| fungi | which could explain why other mirrors show it broken | 14:32 |
| evrardjp | mnaser: I checked other mirrors | 14:32 |
| slaweq | corvus: thx for suggestion, I will try that | 14:32 |
| mordred | cmurphy: is it possible that the puppet-apply tests could be occasionally racey/flaky? | 14:32 |
| evrardjp | some are fine | 14:32 |
| cmurphy | mordred: lol | 14:32 |
| mordred | cmurphy: https://review.openstack.org/#/c/592215/4 failed one of them - but the tests before it and after it passed | 14:32 |
| evrardjp | mnaser: you can see the state: https://mirrors.opensuse.org/ | 14:32 |
| mordred | cmurphy: and nothing in that patch really should have had the sort of impact for what seems to have failed there | 14:32 |
| mnaser | im clicking arbitrary ones trying to find an up to date | 14:32 |
| fungi | evrardjp: yeah, i wonder if the ones updating via rsync are broken and the ones using other mirroring solutions are okay, or whether there was a particular period of time where it was broken at the source and some which have updated since are fixed | 14:33 |
| cmurphy | mordred: hmm i was expecting to see the ssh key problem | 14:33 |
| cmurphy | but http://logs.openstack.org/15/592215/4/check/infra-puppet-apply-4-centos-7/e7e57c2/applytest/puppetapplytest26.final.out.FAILED is new to me | 14:33 |
| mordred | cmurphy: yeah. it's weird | 14:34 |
| mnaser | fungi: well our mirror source is broken, so there's that | 14:34 |
| mnaser | http://opensuse-mirror-gce-us.opensu.se/ seems down too | 14:34 |
| mordred | cmurphy: maybe in this case a recheck is warranted, and keep our eyes out for recurrance? | 14:34 |
| cmurphy | mordred: "msg": "rsync: symlink \"/opt/system-config/system-config/system-config/production\" -> \"/home/zuul/src/git.openstack.org/openstack-infra/system-config\" failed: File exists (17)\nrsync error: some files/attrs were not transferred (see previous errors) | 14:35 |
| cmurphy | i think it's you | 14:35 |
| mordred | oh - extra weird | 14:35 |
| cmurphy | yeah | 14:35 |
| mordred | cmurphy: the symlink version of the patch was removed a while ago | 14:35 |
| fungi | smcginnis: i would wait a bit longer to retry that branch creation. looking at the recent images most were uploaded between 23-25 hours ago so we're likely building new ones right now | 14:36 |
| mordred | and why the 2 extra system-configs | 14:36 |
| cmurphy | oh | 14:36 |
| cmurphy | very weird | 14:36 |
| mordred | and why only on puppet4 | 14:36 |
| mordred | I think I want to blame sunspots | 14:36 |
| fungi | stray cosmic rays | 14:36 |
| corvus | mordred: 592763 commit message says it symlinks, it's wrong? | 14:36 |
| mordred | it is! it's a comment for an older version of the patch that I didn't fix | 14:37 |
| mordred | corvus: you want me to fix and re-push? | 14:37 |
| slaweq | mordred: I think that graphite will be enough for me at least for now, thx also :) | 14:38 |
| corvus | mordred: yeah, i think we need to be nice to our future forensic archeologists. | 14:38 |
| mordred | kk. | 14:39 |
| cmurphy | ++ | 14:39 |
| mordred | corvus: I'm going to wait a few to see if there are any other feedbacks from the stack - just to reduce churn | 14:39 |
| corvus | mordred: ++ | 14:39 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/system-config master: Use Kernel.org mirror for OpenSUSE https://review.openstack.org/593065 | 14:40 |
| mnaser | evrardjp, cmurphy, fungi ^ | 14:40 |
| mnaser | that mirror has the package, and i guess it is somewhat more reliably maintained | 14:41 |
| mnaser | be nice if we can push that through considering every single role job will result in a failure now for opensuse in OSA :( | 14:41 |
| mordred | wfm | 14:43 |
| evrardjp | mnaser: kernel.org is not in sync | 14:43 |
| mnaser | evrardjp: how so? i see the missing opensuse-release file | 14:43 |
| mnaser | https://mirrors.edge.kernel.org/opensuse/update/leap/42.3/oss/x86_64/ => openSUSE-release-42.3-2.1.x86_64.rpm | 14:43 |
| evrardjp | other files are missing, else you'd get a nice tickbox on https://mirrors.opensuse.org/list/42.3.html | 14:43 |
| *** r-daneel has joined #openstack-infra | 14:44 | |
| mnaser | evrardjp: well rackspace has a tick and it doenst have openSUSE-release-42.3-2.1.x86_64.rpm (sorry, not picking on anyone here, it was just one i tested!!) | 14:44 |
| evrardjp | that's fair. | 14:44 |
| smcginnis | fungi: ack, thanks | 14:45 |
| *** janki has quit IRC | 14:46 | |
| *** jamesdenton has quit IRC | 14:46 | |
| fungi | does kernel.org support rsync, i guess? | 14:46 |
| *** felipemonteiro has quit IRC | 14:47 | |
| mnaser | fungi: yep, i tried this command and it worked | 14:47 |
| mnaser | rsync -rlptDvz --delete --stats --delete-excluded --exclude="iso" --dry-run rsync://mirrors.kernel.org/mirrors/opensuse/distribution/leap/42.3 test | 14:47 |
| corvus | mordred: a kind of weak -1 on https://review.openstack.org/592972 with some comments to prompt a discussion which i think you will enjoy! | 14:48 |
| mnaser | note you can run this locally with because of the --dry-run | 14:48 |
| *** Dobroslaw has quit IRC | 14:49 | |
| *** felipemonteiro has joined #openstack-infra | 14:50 | |
| mordred | corvus: aha! I will respond, but tl;dr I don't think it's as complex - the puppet _role_ isn't upstream, only the puppet module - so I think we can definitely update the role to have better parameter names | 14:50 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: [Neutron] Fix graphs with number of jobs running recently https://review.openstack.org/593070 | 14:51 |
| *** SumitNaiksatam has quit IRC | 14:51 | |
| corvus | mordred: oh w00t! | 14:53 |
| mordred | corvus: that said- maybe let's do that dance as a followup patch, otherwise the rebasing game might get gnarly | 14:54 |
| *** rpittau has quit IRC | 14:54 | |
| corvus | mordred: that works -- we just need to do it before, you know, the next ansible thing that wants it's "config" "managed" :) | 14:55 |
| mordred | ++ | 14:55 |
| mordred | corvus: I gotta afk for about 1.5 hours - want me to push up the fix to your existing thing before I go or just wait til I get back? | 14:55 |
| corvus | mordred: go for it, i'll continue through the stack and maybe we'll have a thing by the time you get back | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set mgmt_puppet_module_dir publically https://review.openstack.org/593054 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppet variables and playbooks https://review.openstack.org/592972 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add playbook to install openstack credentials https://review.openstack.org/593029 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppetmaster puppet config handled by ansible https://review.openstack.org/592215 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove purge_apt_sources https://review.openstack.org/592214 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove restricted and multiverse from ubuntu servers https://review.openstack.org/593047 | 14:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Install ssh key on bridge.openstack.org https://review.openstack.org/593053 | 14:57 |
| mordred | corvus: bomb's away | 14:57 |
| *** ijw has joined #openstack-infra | 14:57 | |
| *** ijw has quit IRC | 14:58 | |
| *** graphene has quit IRC | 14:59 | |
| *** r-daneel has quit IRC | 15:00 | |
| *** graphene has joined #openstack-infra | 15:01 | |
| *** felipemonteiro has quit IRC | 15:01 | |
| Ng | fungi: you are too kind :) | 15:02 |
| *** cburgess has quit IRC | 15:02 | |
| *** dpawlik has quit IRC | 15:02 | |
| *** cburgess has joined #openstack-infra | 15:03 | |
| *** ykarel is now known as ykarel|away | 15:04 | |
| *** rpioso is now known as rpioso|car | 15:05 | |
| fungi | Ng: not at all. we miss you! | 15:06 |
| *** dbecker has quit IRC | 15:07 | |
| Ng | I kinda miss the hustle and bustle of being in gerrit all the time, but I like to hope that I'm now a force multiplier and many people are gerriting on my behalf ;) | 15:08 |
| *** cburgess has quit IRC | 15:08 | |
| *** cburgess has joined #openstack-infra | 15:09 | |
| *** ykarel|away has quit IRC | 15:12 | |
| fungi | i'm certain that is the case | 15:13 |
| fungi | i feel the same way... all those developers fixing the bugs i introduced ;) | 15:14 |
| *** jamesmcarthur has joined #openstack-infra | 15:21 | |
| clarkb | mnaser: mordred right devstack-gate (and I think new devstack base jobs) disable nested virt by default beacuse even if the cloud says its a feature doesn't mean it will work | 15:23 |
| *** boden has quit IRC | 15:23 | |
| mnaser | clarkb: ok, thats reasonable | 15:23 |
| clarkb | mnaser: mordred more recently (like the last month?) ovh and limestone did some hypervisor kernel updates that improved reliability for xenial guest kernels | 15:23 |
| mnaser | we deploy on centos/rdo so that might put us in a different position | 15:23 |
| *** kevinshort has quit IRC | 15:23 | |
| clarkb | mnaser: mordred with kata we've foudn that newer kernels (like fedora 28 kernel) seems to crash at a noticeable rate on vexxhost but I haven't looked recently to see if that has changed | 15:24 |
| clarkb | mnaser: ya they were both running ubuntu kernels iirc | 15:24 |
| mnaser | xenial seems to be super super stable | 15:24 |
| clarkb | mnaser: I've asked johnsom to add a canary job on bionic to see if that changes behavior since octavia is one of the projects overriding the our forced use of emu | 15:25 |
| clarkb | *qemu | 15:25 |
| clarkb | mnaser: but I'm not sure that has been done yet | 15:25 |
| johnsom | clarkb Sadly it has not..... | 15:25 |
| mnaser | once we start flipping things over, we won't need that kata specific flavor | 15:25 |
| clarkb | mnaser: cool | 15:25 |
| mnaser | also, any idea why `build-sphinx-docs` and `openstackci-beaker-ubuntu-trusty` seems to be stuck forever in gate on 593065,1 | 15:25 |
| *** kevinshort has joined #openstack-infra | 15:25 | |
| clarkb | mnaser: mostly its a try it and see if it works and when it breaks work with cloud and kernely people like kashyap to debug and fix it | 15:26 |
| clarkb | kernel updates on the guests and hypervisors are what I expect to be the biggest impacts (both positive and negative) but possible also different hardware could affect it | 15:26 |
| clarkb | mnaser: the kernel has nested virt enabled by default on amd cpus fwiw but not intel | 15:26 |
| clarkb | mordred: anything need eyeballs for bridge right now? | 15:27 |
| *** slaweq has quit IRC | 15:27 | |
| clarkb | I still need caffeine but can take a look at anything urgent | 15:27 |
| clarkb | mnaser: johnsom fwiw I think its great we've got clouds thinking about this now :) I feel like we are finally in a position to make use of this featuer and have the ability to work with people to fix it if/when it breaks \o/ so thank you | 15:29 |
| mnaser | clarkb: yeah i'm excited as that should bring new interesting use cases | 15:29 |
| johnsom | +1 | 15:30 |
| fungi | mnaser: possible those stuck jobs were running on ze02 when it spontaneously rebooted ~16 hours ago? how long have they been stuck? | 15:32 |
| *** graphene has quit IRC | 15:32 | |
| fungi | clarkb: there is a fresh pile of changes to continue the config management redress, yes | 15:32 |
| mnaser | fungi: not wildly long, but they just queued up | 15:32 |
| fungi | okay, so probably not related to the ze02 reboot in that case | 15:33 |
| clarkb | fungi: ya I see the list , are they things we want to get in before turning on the cron? Looks like they can't hurt, just not sure where they fit into the lets get deployments happening automagically again | 15:33 |
| mnaser | so once things merge in system-config, we need to ask nicely because cronjobs dont run anymore? | 15:33 |
| *** graphene has joined #openstack-infra | 15:34 | |
| fungi | clarkb: yeah, not sure. i was planning to look closer once the release meeting wrapped up, which it just did | 15:34 |
| clarkb | ok, I'm gonna grab tea then and will dig into the list after that | 15:34 |
| fungi | mnaser: basically yes, until sometime later today probably | 15:34 |
| mnaser | ok, ill do so once the opensuse merge changes (and maybe ask extra nicely to run the rsync instead of waiting because all of our jobs are broken :X) | 15:34 |
| fungi | mnaser: except job configuration, which zuul should notice on its own | 15:34 |
| fungi | but yeah the mirror script change will need help getting deployed | 15:35 |
| *** boden has joined #openstack-infra | 15:39 | |
| openstackgerrit | Merged openstack-infra/system-config master: Use Kernel.org mirror for OpenSUSE https://review.openstack.org/593065 | 15:44 |
| *** dbecker has joined #openstack-infra | 15:46 | |
| mnaser | infra-root: can someone manually kick off a puppet run and run the mirroring script ? | 15:49 |
| mnaser | ^ | 15:49 |
| fungi | corvus: mordred: is the best way to do that to just run `kick.sh mirror-update01.openstack.org` ? | 15:51 |
| fungi | on bridge | 15:51 |
| fungi | smcginnis: we have a lot of 30-40 minute old images showing ready in providers now. if you want to maybe wait another 30 minutes to be sure, then it'll probably be safe to retry that branch creation run | 15:53 |
| smcginnis | fungi: Great, thanks for checking on that. | 15:53 |
| *** graphene has quit IRC | 15:54 | |
| corvus | fungi: yes, that's my understanding | 15:55 |
| fungi | thanks, i'll give it a shot | 15:55 |
| *** graphene has joined #openstack-infra | 15:56 | |
| fungi | should i open a new window in that existing root screen session and run it there, for faster triage in case of errors? | 15:56 |
| corvus | fungi: sounds good | 15:56 |
| fungi | taking the conn for a moment to do so, in that case ;) | 15:57 |
| corvus | clarkb: our action item right now is to review mordred's stack while he's afk | 15:57 |
| pabelanger | mnaser: left a comment on 593065 as FYI | 15:57 |
| mnaser | pabelanger: aw darn | 15:58 |
| fungi | it's running now | 15:58 |
| mnaser | i can email them to ask if they can extend something to us | 15:58 |
| mnaser | is there a specific host ip/hostname that does the rsyncing? | 15:58 |
| pabelanger | mnaser: I'd give it a try, maybe it has improved, but may need to try another host too, if too many connections keep happening | 15:58 |
| *** e0ne has quit IRC | 15:59 | |
| corvus | fungi: not very chatty so far, eh? | 15:59 |
| fungi | presumably another inventory cache repopulation | 15:59 |
| *** dbecker has quit IRC | 15:59 | |
| corvus | oh, i guess that was the fact gathering (there's a patch in reviewto enable caching) | 15:59 |
| corvus | or that | 15:59 |
| fungi | pabelanger: mnaser: ahh, yeah we'd end up going days without mirror updates because kernel.org only allows 50 rsync sessions total at any point in time and whenever a popular distro has a major update they're basically unusable | 16:00 |
| mnaser | i can try reaching out to them if we have an ip that they can perhaps whitelist | 16:00 |
| fungi | it won't be the same ip if we need to rebuild the server, and also if we convert it to a zuul job it will be a different ip address on every run | 16:01 |
| fungi | well, probably at least (unless we decide to use a static job node for it) | 16:01 |
| pabelanger | yah, if we can get away from static IP, that would be better I think. Maybe we can ask in opensuse for a good 2nd mirror | 16:02 |
| fungi | looks like the kick completed. checking state on the server now | 16:02 |
| fungi | kick.sh doesn't seem to have altered /usr/local/bin/opensuse-mirror-update on the server :( | 16:03 |
| *** r-daneel has joined #openstack-infra | 16:04 | |
| fungi | i wonder if it ran with an old system-config? | 16:04 |
| fungi | do i need to wait for the copy of system-config on bridge to update? or do i need to git pull in there? | 16:04 |
| *** strigazi has quit IRC | 16:05 | |
| mnaser | i remember seeing mordred saying that something had to be ran in order for system-config to update | 16:06 |
| *** strigazi has joined #openstack-infra | 16:06 | |
| *** strigazi has quit IRC | 16:06 | |
| *** strigazi has joined #openstack-infra | 16:07 | |
| fungi | if it's that i need to pull into /opt/system-config/production there's a (minor) uncommitted local change i'll need to make sure is preserved | 16:08 |
| clarkb | fungi: ya I don't think system config is updating automatically beause the cron does that too | 16:10 |
| *** rpioso|car is now known as rpioso | 16:10 | |
| mnaser | is there a way to get build stats beyond the 50 records shown in zuul.o.o ? | 16:11 |
| *** strigazi has quit IRC | 16:13 | |
| *** strigazi has joined #openstack-infra | 16:14 | |
| corvus | mnaser: ?skip=... should do it i think | 16:14 |
| corvus | mnaser: or also ?limit=... | 16:15 |
| corvus | mnaser: i don't think we have bounds checks for those yet, so... um, be nice :) | 16:15 |
| fungi | clarkb: okay, i'll pull --ff-only in /opt/system-config/production and make sure the uncommitted diff is preserved. for the record (so nobody has to futz with reflog), its head prior to pulling was 3b0dc9df7c0a256ece1ce99d3bb4f0fa3b3ec687 | 16:16 |
| mnaser | corvus: i promise i wont do anything over a million | 16:16 |
| mnaser | :p | 16:16 |
| mnaser | i want to run something simple just to see failure % for certain jobs in OSA to see if they can stay/go nonvoting | 16:16 |
| corvus | mnaser: there's also graphite | 16:18 |
| fungi | seems the uncommitted diff on that detached head was no longer relevant on latest master branch tip | 16:19 |
| *** ramishra has quit IRC | 16:19 | |
| fungi | i think it must have been a divergence in an updated patchset from a locally-applied change | 16:19 |
| corvus | sounds wholly plausible | 16:20 |
| fungi | because git log for the file it was changing shows the task it removed never existed in master branch history | 16:20 |
| fungi | rerunning kick.sh for the mirror update server now | 16:20 |
| *** jamesdenton has joined #openstack-infra | 16:20 | |
| *** ykarel has joined #openstack-infra | 16:21 | |
| fungi | MIRROR="rsync://mirrors.kernel.org/mirrors/opensuse" | 16:22 |
| fungi | that seems to have done the trick | 16:22 |
| fungi | mnaser: ^ | 16:22 |
| fungi | i'll leave the session in that new screen window intact and toggle back to the previous window where we were working yesterday | 16:23 |
| fungi | but it's there in case anyone needs to check its history | 16:23 |
| *** rcarrillocruz has quit IRC | 16:24 | |
| fungi | mnaser: i've manually started a mirror update for that now so it should hopefully have good content shortly | 16:25 |
| fungi | i'll let you know once it completes | 16:26 |
| *** fried_rice is now known as efried_pto | 16:29 | |
| fungi | mnaser: it should be done updating now if you want to double-check what was broken earlier | 16:29 |
| * mordred waves to people | 16:30 | |
| mnaser | fungi: i see it, thank you so much | 16:31 |
| mordred | fungi, corvus, clarkb: it seems we were able to do the mirror kick successfully? | 16:31 |
| *** ykarel has quit IRC | 16:32 | |
| fungi | mordred: yes, after changing to the master branch and pulling up to latest origin/master state | 16:32 |
| fungi | in /opt/system-config/production on bridge.o.o | 16:33 |
| clarkb | mordred: corvus and I have been leaving comments on some of those reviews (not all are -1s) | 16:34 |
| *** yamahata has quit IRC | 16:36 | |
| clarkb | I've not approved anything because still catching up on where we are at this morning | 16:39 |
| mordred | fungi: \o/ | 16:39 |
| mordred | clarkb: yah. I'm going through those right now | 16:39 |
| *** eernst has joined #openstack-infra | 16:39 | |
| *** jpena is now known as jpena|off | 16:40 | |
| *** pas-ha has quit IRC | 16:41 | |
| *** pas-ha has joined #openstack-infra | 16:42 | |
| *** jpich has quit IRC | 16:46 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:47 | |
| *** roman_g has quit IRC | 16:48 | |
| clarkb | fungi: openstackid-resources.openstack.org showed up in the cert renewal list recently but then fell off. Looking at its current cert it has until november 2018, implying that something like letsencrypt is set up for it. I think you said that server was no longer needed though? maybe we should delete it? | 16:52 |
| clarkb | (smarcet isn't here but I bet smarcet would know0 | 16:52 |
| *** gyee has joined #openstack-infra | 16:52 | |
| fungi | clarkb: i think it's either that or cloudflare managing the cert for it | 16:52 |
| fungi | but yeah, probably no point in us tracking/monitoring that | 16:53 |
| fungi | we don't manage that server | 16:53 |
| clarkb | got it | 16:53 |
| corvus | i've gone through clarkb's comments so i'm up to date | 16:54 |
| mordred | clarkb, corvus: ok. I got your comments taken care of - and also have reorganized the stack a bit so that things we should do while we're doing manual runs/tests are up front | 16:55 |
| mordred | patch-bomb incoming | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove purge_apt_sources https://review.openstack.org/592214 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove restricted and multiverse from ubuntu servers https://review.openstack.org/593047 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Clean up puppetmaster puppet config handled by ansible https://review.openstack.org/592215 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add playbook to install openstack credentials https://review.openstack.org/593029 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Install ssh key on bridge.openstack.org https://review.openstack.org/593053 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Fix AFS and CA docs references to puppetmaster https://review.openstack.org/593107 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove infracloud references https://review.openstack.org/593108 | 16:55 |
| openstackgerrit | Saul Wold proposed openstack-infra/project-config master: stx-specs: Add new repo for specifications https://review.openstack.org/592676 | 16:56 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Remove unneeded ssl cert expiry check https://review.openstack.org/593109 | 16:57 |
| clarkb | fungi: ^ that is the cleanup | 16:57 |
| fungi | thanks! | 16:57 |
| mordred | clarkb, corvus, fungi: if we're happy with the production-path-ectomy, I think we can land the patches up through that at least while the others are reviewed and I can manually move the existing dir on bridge in preparation to run things again | 16:59 |
| mordred | https://review.openstack.org/#/c/592763/ - ftr | 16:59 |
| mordred | I think it would be nice to land through https://review.openstack.org/#/c/592215 while we're in manual check mode | 17:00 |
| *** graphene has quit IRC | 17:00 | |
| clarkb | mordred: that seems like a reasonable plan | 17:01 |
| clarkb | I'm reworking through the stack now | 17:01 |
| mordred | I'm also going to go ahead and land https://review.openstack.org/#/c/592946 | 17:01 |
| *** graphene has joined #openstack-infra | 17:02 | |
| corvus | mnaser: ok, approved through 592763 | 17:02 |
| corvus | mordred: ^ (sorry mnaser) | 17:02 |
| mnaser | np, i was curious given i didnt have anything proposed :p | 17:02 |
| mordred | woot | 17:03 |
| corvus | mnaser: if you did i'd totally land it | 17:03 |
| mordred | I have moved /opt/system-config/production to /opt/system-config | 17:03 |
| mordred | I've also removed this: | 17:03 |
| mordred | mgmt_puppet_module_dir: /etc/puppet/modules | 17:03 |
| mordred | mgmt_hieradata: /etc/ansible/hosts | 17:03 |
| mordred | from private group_vars/all.yaml, since that's covered in the patches in public group_vars now | 17:04 |
| clarkb | mordred: for launch node you mean deploy server with ansible then if you are puppeting it run ansible to puppet it after launch node is done? (this might happen autoamtically or manually dependning) | 17:04 |
| *** SteffanW has quit IRC | 17:04 | |
| mordred | yah | 17:06 |
| mordred | don't need to special-case the installation of puppet any more | 17:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Add tox-py27 jobs to zuul-jobs https://review.openstack.org/592768 | 17:07 |
| *** yamahata has joined #openstack-infra | 17:07 | |
| *** trown is now known as trown|lunch | 17:24 | |
| clarkb | mordred: corvus are we just waiting for things to merge then retest again? | 17:26 |
| mordred | yah - I was thinking of pulling from gerrit to move us along | 17:27 |
| mordred | corvus: how do you feel about https://review.openstack.org/#/c/592764 ? | 17:32 |
| clarkb | I'm going to step away for a few, but think I've got through the stack up to the point where you want to test | 17:33 |
| mordred | awesome - and yes, I thnk you're good | 17:34 |
| corvus | mordred: i feel fondly about it | 17:34 |
| mordred | I'm also not expecting these to be earthshattering or scary like the exim patches yesterday | 17:34 |
| corvus | i see i'm lacking +2 on some others, going through again now | 17:34 |
| mordred | I blame rebasing | 17:34 |
| mordred | I'm going to pull through 593047 onto bridge and then start in through the testing game again | 17:35 |
| *** dpawlik has joined #openstack-infra | 17:36 | |
| corvus | mordred: why the free strategy in 593029? | 17:36 |
| mordred | corvus: because the hosts in the group don't depend on each other in any way | 17:36 |
| mordred | corvus: no need for one host to wait on the other finishing a step or anything | 17:37 |
| mordred | cmurphy: the puppet apply tests definitely seem to have grown a flakiness | 17:38 |
| mordred | corvus, cmurphy, clarkb: the test failure in https://review.openstack.org/#/c/592765/7 is similar to the other transient failures from this morning, and I still don't understand it | 17:39 |
| *** dpawlik has quit IRC | 17:40 | |
| mordred | ok - I'm going to run bridge.yaml and base.yaml on bridge with the new patches and see what I got wrong | 17:43 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Add ability to override mgmt manifest path https://review.openstack.org/592946 | 17:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove infracloud references https://review.openstack.org/593108 | 17:50 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on fact caching https://review.openstack.org/592971 | 17:50 |
| openstackgerrit | Merged openstack-infra/system-config master: Add base.yaml to kick.sh https://review.openstack.org/592760 | 17:50 |
| mordred | corvus, clarkb: ok - so it seems state: touch on a file will always result in the file being touched and thus always report changed | 17:53 |
| mordred | we're doing it in bridge.yaml right now to make sure /var/log/ansible.log is writable - but we run ansible as root, so maybe that's not important? or maybe we should make a /var/log/ansible directory instead? | 17:54 |
| mordred | (this is very nitpicky - but we're looking at it now) | 17:54 |
| openstackgerrit | Merged openstack-infra/system-config master: Set mgmt_puppet_module_dir publically https://review.openstack.org/593054 | 17:54 |
| clarkb | ya maybe make /var/log/ansible group owned by admin | 17:55 |
| clarkb | then anyone running ansible that is an admin can write to files there | 17:55 |
| mordred | yah | 17:55 |
| *** e0ne has joined #openstack-infra | 17:56 | |
| clarkb | mordred: the failure there is a failure to parse a valid node {} block out of site.pp | 17:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move ansible log into directory https://review.openstack.org/593118 | 17:57 |
| fungi | i'm in favor of anything that reduces gratuitous sudo invocation on an already admin-only system | 17:57 |
| mordred | clarkb: like that ^^ ? | 17:57 |
| mordred | clarkb: oh yeah? weird | 17:57 |
| clarkb | http://logs.openstack.org/65/592765/7/check/infra-puppet-apply-3-ubuntu-xenial/f0ec40c/applytest/puppetapplytest28.final.out.FAILED its missing the closing } | 17:57 |
| clarkb | (I don't know why yet) | 17:57 |
| clarkb | oh wait we comment out the node | 17:58 |
| clarkb | nevermind | 17:58 |
| *** bobh has quit IRC | 17:59 | |
| clarkb | mordred: http://logs.openstack.org/65/592765/7/check/infra-puppet-apply-3-ubuntu-xenial/f0ec40c/job-output.txt.gz#_2018-08-17_17_08_27_240324 a path issue? | 17:59 |
| clarkb | note that it is trying to link /opt/system-config/system-config/system-config/system-config/production but with more system-configs | 18:00 |
| mordred | clarkb: yah. that happened in the earlier failure too | 18:01 |
| mordred | clarkb: but it works most of the time, so I'm super confused | 18:02 |
| mordred | clarkb: maybe something is fighting and trying to make the symlink more than once? | 18:02 |
| mordred | which is why we only see it occasionally? | 18:02 |
| clarkb | mordred: ya, we run that in parallel using xargs iirc | 18:03 |
| clarkb | so depending on how those line up they could be fighting over that symlink | 18:03 |
| clarkb | but why the extra system-configs too | 18:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Swap order of roles path https://review.openstack.org/593120 | 18:05 |
| mordred | clarkb, corvus: ^^ that just surfaced in testing on bridge | 18:05 |
| clarkb | its trying to copy mgmt_manifestpath which is a symlink to the zuul installed dir | 18:05 |
| clarkb | mgmt_manifestpath seems all borked though | 18:05 |
| clarkb | and symlinks don't rsync cleanly on top of each toher I guess? | 18:05 |
| clarkb | oh its copying /opt/system-config to /opt/system-config | 18:06 |
| mordred | we don't really need it to do that | 18:06 |
| clarkb | but /opt/system-config is lready a link back to /home/zuul/src/etc | 18:06 |
| corvus | huh, that's a really unfortunate overloading of the roles_path | 18:06 |
| clarkb | mordred: I think if we copied the repo instead of symlinking it this would work because rsync would noop | 18:06 |
| corvus | like, the order in which roles are loaded should not be determined by where galaxy installs things :/ | 18:06 |
| AJaeger | fungi: thanks for the reverse lookup. | 18:06 |
| mordred | corvus: oh! we can control it | 18:06 |
| clarkb | mordred: in system-config/tools/prep-apply.sh | 18:07 |
| *** r-daneel_ has joined #openstack-infra | 18:07 | |
| *** r-daneel has quit IRC | 18:08 | |
| *** r-daneel_ is now known as r-daneel | 18:08 | |
| corvus | mordred: so when we invoke ansible-galaxy use "-p /etc/ansible/roles", then leave roles_path in install-ansible as-is? | 18:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Be explicit about where to install roles https://review.openstack.org/593120 | 18:09 |
| mordred | corvus: you'll like that better I hope ^^ | 18:09 |
| mordred | corvus: yah | 18:09 |
| clarkb | can I just say thats a stupid behavior | 18:09 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove unneeded ssl cert expiry check https://review.openstack.org/593109 | 18:09 |
| clarkb | typically you want overrides earlier in the path then let "global" installed things be at the end | 18:09 |
| corvus | mordred: you changed an opt to etc | 18:09 |
| AJaeger | dhellmann: do you really want to change that many repos already for python3-first? https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:python3-first | 18:09 |
| mordred | corvus: well, I suck | 18:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Be explicit about where to install roles https://review.openstack.org/593120 | 18:10 |
| mordred | corvus: how about now ? | 18:10 |
| corvus | clarkb: agreed, but at least we can override it, so i think we're ok | 18:10 |
| clarkb | ya the fix is fine | 18:10 |
| clarkb | just extremely surprising behavior from the tooling | 18:10 |
| mordred | ok. I'm starting down the list again with review-dev | 18:12 |
| clarkb | mordred: ok I think we can fix that if we set mgmt_manifestpath to /home/zuul/src/git.openstack.org/openstack-infra/system-config and have it copy to /opt/system-config and remove the ln -s tools/prep-apply.sh | 18:13 |
| mordred | clarkb: kk. and we can set mgmt_manifestpath with a -e I believe | 18:13 |
| mordred | well poo. that didn't work | 18:14 |
| clarkb | mordred: it thinks it can find its puppet locally | 18:15 |
| mordred | yeah- I moved mgmt_hieradata to the wrong file | 18:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: test container push job https://review.openstack.org/593125 | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set mgmt_hieradata in puppet group_vars https://review.openstack.org/593126 | 18:17 |
| mordred | clarkb, corvus: ^^ | 18:17 |
| * mordred made a conceptual mistake before - but it's a mistake that's probably good for all of us to grok moving forward | 18:18 | |
| clarkb | right the ansible role puppet vars apply to the hosts that are being puppeted (even if they are data about bridge) | 18:19 |
| mordred | yup | 18:20 |
| corvus | makes sense | 18:21 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: upload-logs-swift: Small python2/3 update https://review.openstack.org/592774 | 18:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: test container push job https://review.openstack.org/593125 | 18:21 |
| mordred | and another hiccup - but this is good, it's definitely why I wanted to get these in while we were still watching | 18:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: test container push job https://review.openstack.org/593125 | 18:23 |
| clarkb | mordred: it copied to /opt/system-config/system-config | 18:24 |
| clarkb | on review-dev01 | 18:24 |
| mordred | of course it did | 18:24 |
| clarkb | which may also help explain all the system-configs in your failing test | 18:24 |
| mordred | yah | 18:24 |
| clarkb | hrm | 18:25 |
| clarkb | beacuse /opt/system-config/product is a git dir and we want to put a git dir at /opt/sytem-config too that may cause some confusion | 18:25 |
| clarkb | though we don't really run git on that stuff so maybe its fine | 18:25 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: test container push job https://review.openstack.org/593125 | 18:25 |
| mordred | well - we want stuff to go to /opt/system-config/production on the remote hosts still | 18:26 |
| mordred | their puppet hiera config includes an environment dir | 18:26 |
| clarkb | oh | 18:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: test container push job https://review.openstack.org/593125 | 18:28 |
| clarkb | mordred: ansible-role-puppet copies to manifest_base which doesn't specify the environment because it was getting that from the source | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Be more explicit about puppet paths https://review.openstack.org/593130 | 18:28 |
| mordred | clarkb: how about if we just do that ^^ | 18:28 |
| clarkb | mordred: but now we've replaced the source with /opt/system-config which copies system-config to /opt/system-config | 18:28 |
| mordred | clarkb: oh- so we also want the source to have a trailing / ? | 18:29 |
| clarkb | mordred: yes I think so | 18:29 |
| clarkb | current patch will produce something like /opt/system-config/production/system-config on the remote node | 18:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Be more explicit about puppet paths https://review.openstack.org/593130 | 18:31 |
| clarkb | mordred: also puppet_module_dir is supposed to be /etc/puppet/modules still I think | 18:31 |
| clarkb | (that doesn't need to change?) | 18:31 |
| mordred | yeah. you're right | 18:32 |
| fungi | dhellmann: are these zuul in-repo job migration efforts Nguyễn is notifying us about (vitrage, cyborg, solum, tacker, masakari, congress, designate) being coordinated? http://lists.openstack.org/pipermail/openstack-infra/2018-August/ | 18:32 |
| mordred | this role needs a good reworking with a bunch of better variable names | 18:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Be more explicit about puppet paths https://review.openstack.org/593130 | 18:32 |
| dhellmann | fungi : he is one of the goal champions; I do not know if he has talked to those PTLs yet | 18:32 |
| fungi | dhellmann: they look similar to the one you sent about oslo, so just wanted to make sure | 18:32 |
| fungi | thanks! | 18:32 |
| dhellmann | fungi : yeah, part of the arrangement with clarkb, corvus, and AJaeger was that we would send emails like that | 18:33 |
| fungi | he wasn't subscribed to the infra ml so i just approved all those for delivery a few minutes ago | 18:33 |
| fungi | if he's going to send more he may benefit from subscribing | 18:33 |
| dhellmann | I will suggest that he sign up for the list, if you haven't already | 18:33 |
| fungi | i haven't | 18:33 |
| *** eharney has quit IRC | 18:33 | |
| dhellmann | I'll handle it | 18:33 |
| AJaeger | dhellmann: and it's good to have those - I reviewed them and found some problems, expect the problems to be in the in-repo imports as well... | 18:33 |
| clarkb | mordred: that is still not quite right because of hieradata :/ | 18:34 |
| AJaeger | dhellmann: I just thought you wanted to start slowly, I was surprised to see so many in flight at this point of time. | 18:34 |
| fungi | happen to know his irc nick? i think AJaeger was trying to find him in here earlier | 18:34 |
| clarkb | mordred: /opt/system-config/hieradata is linked to /etc/puppet/hieradata on the remote (I do not know why) | 18:35 |
| AJaeger | fungi: he followed up to my review commend - he now knows how to WIP a change ;) | 18:35 |
| clarkb | mordred: but changing manifest_base makes it /opt/system-config/production/hierdata | 18:35 |
| dhellmann | he's in #openstack-python3 as nguyenhai_ | 18:35 |
| fungi | cool--thanks again! | 18:35 |
| *** jamesmcarthur has quit IRC | 18:35 | |
| fungi | also glad to see the #openstack-python3 channel has been resurrected (it was dead for the longest time) | 18:36 |
| clarkb | mordred: we can either continue to keep /production on bridge or modify ansible-role-puppet to not assume the dirs under "base" line up on both sides | 18:36 |
| mordred | clarkb: hrm. why do we link that at all | 18:36 |
| clarkb | mordred: ya that I don't know | 18:36 |
| AJaeger | dhellmann: ah, wasn't aware of that channel - thanks | 18:36 |
| clarkb | mordred: /etc/hiera.yaml datadir is set to /opt/system-config and the paths are relative to that | 18:37 |
| clarkb | mordred: so that link gets us back to /etc/puppet hierdata | 18:37 |
| mordred | clarkb: oy. | 18:38 |
| *** bobh has joined #openstack-infra | 18:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: haproxy: don't manage rsyslog service https://review.openstack.org/592637 | 18:38 |
| mordred | clarkb: so why are we putting anyting in /etc/puppet/hieradata on the remote host at all? | 18:38 |
| clarkb | mordred: I think because that is the puppety place for it | 18:38 |
| mordred | but then we configure it to not use it ... | 18:38 |
| mordred | anyway | 18:38 |
| clarkb | right I think this was a hack to get public hiera in repo working with normal private puppet hiera | 18:38 |
| mordred | oh right | 18:39 |
| clarkb | we made all the hieradata paths relative to system-config so it could find both | 18:39 |
| mordred | kk. lemme see if I can make a patch that makes sense - if I can't, we can give up on this particular thing for now | 18:40 |
| *** gyee has quit IRC | 18:43 | |
| *** r-daneel_ has joined #openstack-infra | 18:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-role-puppet master: Allow overiding the remote hieradata link dest https://review.openstack.org/593134 | 18:44 |
| *** r-daneel has quit IRC | 18:44 | |
| *** r-daneel_ is now known as r-daneel | 18:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Be more explicit about puppet paths https://review.openstack.org/593130 | 18:46 |
| mordred | clarkb: how do those two look? | 18:46 |
| *** r-daneel has quit IRC | 18:46 | |
| *** rlandy is now known as rlandy|afk | 18:48 | |
| clarkb | mordred: ya I think that is right | 18:49 |
| mordred | clarkb: lemme pull and try it | 18:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Clarify project creation now that jobs are in-repo https://review.openstack.org/593138 | 18:51 |
| AJaeger | infra-root, another update for infra-manual, we missed one place with the in-repo move ^ | 18:51 |
| mordred | clarkb: ok. I think that time it did the right thing | 18:53 |
| mordred | clarkb: but it would not be a bad idea to double-check me on that | 18:54 |
| clarkb | ya, you may want to double check the paths in the ansible output | 18:54 |
| clarkb | looking at syslog the change in puppet was just updating vcsrepo | 18:54 |
| *** trown|lunch is now known as trown | 18:55 | |
| mordred | clarkb: I could delete everythign in /opt/system-config on review01 and run it again | 18:55 |
| clarkb | the system-config on review-dev01 is at fa5ced4c7c5ed09e79d379a7a107c81b0b2f3c05 which is your commit just above so I think it copied it properly | 18:55 |
| mordred | clarkb: and everything in /etc/puppet too | 18:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: DNM: Add container push job https://review.openstack.org/593125 | 18:55 |
| clarkb | mordred: that might be a reasonable check for this first pass host | 18:55 |
| mordred | lemme do that | 18:55 |
| clarkb | that said I don't see anything wrong with this latest pass | 18:55 |
| mordred | me either | 18:56 |
| mordred | but I'm paranoid now | 18:56 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: DNM: Add container push job https://review.openstack.org/593125 | 18:56 |
| mordred | clarkb: I thik we should add a file: state: directoryin to the mix - but we can do that as a followup | 18:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-role-puppet master: Make sure target manifest directory exists https://review.openstack.org/593140 | 18:59 |
| mordred | clarkb, corvus: ^^ that is needed for bare hosts | 18:59 |
| corvus | clarkb, mordred: lunch now, biab. | 18:59 |
| mordred | corvus: kk | 19:00 |
| corvus | mordred: er i can look at that real quick :) | 19:00 |
| mordred | corvus: the real fun is the one below it | 19:00 |
| mordred | but I've tested thatand it works | 19:00 |
| mordred | corvus: also -your pointyesterday about the variables in that role ... _definitley_ valid | 19:00 |
| corvus | haha | 19:01 |
| *** jamesmcarthur has joined #openstack-infra | 19:01 | |
| corvus | +2d, lunching now | 19:01 |
| mordred | woot. thanks | 19:01 |
| mordred | clarkb: I'm going to run base and puppet again on review, then move on to etherpad-dev | 19:02 |
| clarkb | mordred: are you going to rerun with puppet dirs moved aside/deleted? | 19:02 |
| clarkb | review-dev you mean? | 19:02 |
| mordred | clarkb: I did already | 19:02 |
| clarkb | ah cool | 19:02 |
| mordred | although- lemme do it again with that file: directory patch in place | 19:02 |
| mordred | just to make sure | 19:02 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Bump ansible test version to 2.5 https://review.openstack.org/592776 | 19:02 |
| mordred | clarkb: I've deleted /opt/system-confi and /etc/puppet/hieradata/production on review-dev- running again | 19:03 |
| clarkb | ok I checked and they are there again but looks like ansible puppet is rerunning | 19:04 |
| clarkb | and timestamp from a minute ago | 19:04 |
| mordred | clarkb: yah. the screen from yesterday is still around fwiw | 19:04 |
| mordred | clarkb: and yes - I thnk this will also solve that test failure- but that we can go back and trim that prep-apply script to let the ansible puppet machinery do more of the work | 19:04 |
| mordred | and just pass a few variables to it | 19:05 |
| *** jamesmcarthur has quit IRC | 19:05 | |
| mordred | clarkb: ok. I think I'm happy with review-dev ... gonna do etherpad-dev | 19:12 |
| mordred | clarkb: then if it's good, firehose and storyboard again | 19:12 |
| mordred | then lists - then I think maybe we just do a full run_all ? | 19:13 |
| clarkb | may want to do review and zuul in foreground before a full run all just to make sure they are happy in a controlled manner first? | 19:14 |
| mordred | ++ | 19:14 |
| *** r-daneel has joined #openstack-infra | 19:15 | |
| *** owalsh_ has joined #openstack-infra | 19:16 | |
| mordred | clarkb: crud. unittests failed. because puppet3/puppet4 | 19:18 |
| *** owalsh has quit IRC | 19:20 | |
| clarkb | oh we haev to set different paths for each huh | 19:20 |
| clarkb | that probably explains why its fancy in the role | 19:20 |
| *** jamesmcarthur has joined #openstack-infra | 19:20 | |
| *** owalsh_ has quit IRC | 19:20 | |
| mordred | actually - the error message seems earlier | 19:21 |
| *** owalsh has joined #openstack-infra | 19:21 | |
| *** jamesmcarthur has quit IRC | 19:25 | |
| *** boden has quit IRC | 19:25 | |
| *** eernst has quit IRC | 19:26 | |
| *** boden has joined #openstack-infra | 19:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set mgmt_hieradata in puppet group_vars https://review.openstack.org/593126 | 19:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Be more explicit about puppet paths https://review.openstack.org/593130 | 19:26 |
| *** apetrich has quit IRC | 19:26 | |
| mordred | clarkb: I think that ^^ should fix the test issue | 19:26 |
| clarkb | ya that looks about right | 19:28 |
| clarkb | zuul will let us know :) | 19:28 |
| mordred | etherpad-dev looks fine - I'mma do firehose | 19:29 |
| *** bobh has quit IRC | 19:30 | |
| *** bobh_ has joined #openstack-infra | 19:30 | |
| *** owalsh_ has joined #openstack-infra | 19:32 | |
| mordred | clarkb: can haz 2nd +2 on https://review.openstack.org/#/c/593118/1 | 19:33 |
| mordred | k. firehose happy | 19:33 |
| mordred | I think we're hitting our stride now | 19:33 |
| mordred | doing storyboard | 19:34 |
| *** dbecker has joined #openstack-infra | 19:34 | |
| clarkb | mordred: done | 19:34 |
| mordred | tyty | 19:35 |
| *** owalsh has quit IRC | 19:35 | |
| *** owalsh has joined #openstack-infra | 19:36 | |
| *** owalsh_ has quit IRC | 19:37 | |
| mordred | corvus: whence you are back, https://review.openstack.org/593126 https://review.openstack.org/593130 | 19:42 |
| mordred | corvus: and then let's do lists | 19:42 |
| clarkb | mordred: I need to afk for a bit myself. I haven't seen anything cranky myself fwiw | 19:43 |
| * mordred is going to afk for a sandwich | 19:43 | |
| mordred | clarkb: yah all seems to be swimming along as we want it to | 19:43 |
| clarkb | mordred: I think if we get through lists* and then do review and zuul in foreground then enxt step is running everything? | 19:43 |
| mordred | yup | 19:43 |
| mordred | as as long as that works, we can enable the cron job and have a cocktail | 19:43 |
| *** jamesdenton has quit IRC | 19:46 | |
| corvus | back, catching up | 19:47 |
| *** nicolasbock has quit IRC | 19:50 | |
| *** eharney has joined #openstack-infra | 19:51 | |
| *** bobh_ has quit IRC | 19:52 | |
| *** graphene has quit IRC | 19:54 | |
| *** nicolasbock has joined #openstack-infra | 19:55 | |
| openstackgerrit | Merged openstack-infra/system-config master: Move /opt/system-config/production to /opt/system-config https://review.openstack.org/592763 | 19:55 |
| *** graphene has joined #openstack-infra | 19:55 | |
| openstackgerrit | Merged openstack-infra/system-config master: Clean up puppet variables and playbooks https://review.openstack.org/592972 | 19:57 |
| openstackgerrit | Merged openstack-infra/system-config master: Use openstack and groups inventory from system-config https://review.openstack.org/592764 | 19:57 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove purge_apt_sources https://review.openstack.org/592214 | 19:58 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove restricted and multiverse from ubuntu servers https://review.openstack.org/593047 | 19:58 |
| openstackgerrit | Merged openstack-infra/system-config master: Move ansible log into directory https://review.openstack.org/593118 | 19:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add tox-with-sudo job https://review.openstack.org/593150 | 20:00 |
| mordred | corvus: ok. now I'm back too | 20:10 |
| mordred | corvus: you ready for the magic that is lists.o.o ? | 20:10 |
| *** jcoufal has quit IRC | 20:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: htmlify: fix py3 gzip issue https://review.openstack.org/593165 | 20:13 |
| corvus | mordred: 1 sec | 20:13 |
| *** haleyb|away has quit IRC | 20:17 | |
| fungi | is it safe to merge 592215 yet? | 20:17 |
| fungi | saw it had 2x+2 but no approval | 20:17 |
| fungi | seems like everything before it is in now? | 20:17 |
| *** jamesmcarthur has joined #openstack-infra | 20:18 | |
| *** jamesmcarthur has quit IRC | 20:18 | |
| fungi | also was there a determination on why infra-puppet-apply-3-ubuntu-xenial is failing on 592765? | 20:18 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: upload-logs-swift: Use collections Sequence for FileList https://review.openstack.org/592778 | 20:18 |
| *** jamesmcarthur has joined #openstack-infra | 20:18 | |
| clarkb | fungi: I think we are testing the whole stack as we go first | 20:19 |
| *** mattmceuen has left #openstack-infra | 20:19 | |
| clarkb | fungi: I believe that fialure is due to the symlink and path derpage. I think the fix is to stop symlinking and have ansible role puppet copy from zuul repo to /opt/system-config | 20:20 |
| mordred | clarkb, fungi: oh - good- zuul is happy now | 20:20 |
| corvus | mordred: ready! | 20:20 |
| mordred | I've been testing with 593130 and 593134 applied | 20:21 |
| fungi | okay, got it | 20:21 |
| mordred | corvus: awesome! I shall now run base on lists, then we'll do puppet, then base again | 20:21 |
| mordred | corvus: ok. base ran successfully - does exim seem happy to you? (itdidn't show any changes, so I'm guessing "yes") | 20:24 |
| corvus | mordred: lgtm | 20:25 |
| mordred | k. doing puppet | 20:25 |
| corvus | the puppet module rsync output is quite valuable | 20:28 |
| corvus | oh wait i meant voluminous | 20:28 |
| corvus | not valuable | 20:28 |
| mordred | yes. definitely not valuable | 20:29 |
| mordred | ok. giant rsync and then it seems all good | 20:29 |
| *** kevinsho_ has joined #openstack-infra | 20:29 | |
| *** bobh has joined #openstack-infra | 20:29 | |
| mordred | corvus: I feel like we're good to run base again if you do | 20:30 |
| *** kevinshort has quit IRC | 20:30 | |
| corvus | mordred: yep | 20:30 |
| *** kevinsho_ has quit IRC | 20:31 | |
| *** dbecker has quit IRC | 20:31 | |
| mordred | k. that looks good | 20:32 |
| mordred | corvus, clarkb: so - I think we were tinking review and/or zuul now? | 20:33 |
| corvus | yeah that was nice and boring | 20:33 |
| *** apetrich has joined #openstack-infra | 20:33 | |
| clarkb | mordred: what was the changed task? | 20:33 |
| mordred | we should maybe do one of the git farm, since they're centos | 20:33 |
| clarkb | oh good idea | 20:33 |
| mordred | clarkb: it'sthe one that always changes - set ssh key for management | 20:33 |
| clarkb | ah | 20:33 |
| mordred | best I can tell it's because we have the key_options | 20:33 |
| mordred | next time we're bored it might be a nice upstream bug to file/fix | 20:34 |
| mordred | but for now, 1 changed on base is what we should see | 20:34 |
| clarkb | ok | 20:34 |
| clarkb | 1 is the new 0 | 20:34 |
| clarkb | mordred: I would do a single git0X host to start | 20:34 |
| clarkb | mordred: since we have many of them gives us a sane fallback :) | 20:35 |
| mordred | yah | 20:35 |
| clarkb | but I think that is a great idea, then we do zuul then review? | 20:35 |
| mordred | let's do git01 | 20:35 |
| mordred | ++ | 20:35 |
| *** e0ne has quit IRC | 20:35 | |
| *** rlandy|afk is now known as rlandy | 20:36 | |
| mordred | ok. mostly ok - let's fix that error real quick | 20:36 |
| clarkb | mordred: oh also is remote ansible running python2.7? | 20:37 |
| clarkb | beacuse on centos there is no python3 | 20:37 |
| mordred | clarkb: yes - it seems to be working - so woot | 20:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add skip: true to disable-puppet-agent https://review.openstack.org/593171 | 20:39 |
| mordred | that should fix the derp there | 20:39 |
| corvus | mordred: query, not lookup? | 20:40 |
| mordred | corvus: yah - query returns a list - so it works with loop | 20:41 |
| mordred | which you have to do because of skip | 20:41 |
| mordred | skip causes it to return an empty list if nothing is found | 20:41 |
| corvus | ooh, that's the workaround for that. hrm. | 20:41 |
| clarkb | ah empty list vs error | 20:42 |
| mordred | yah | 20:42 |
| corvus | this is still far, far inferior to with_first_found. grumble. | 20:42 |
| mordred | totally agree | 20:42 |
| mordred | although I now undestand many more internals than before | 20:42 |
| corvus | mordred: yep. maybe someday i'll understand enough internals to write that code from scratch. but i don't today, which means i need to remember where it is and copy-pasta. | 20:43 |
| mordred | ok. git01 seemed to work that time - let's try some puppet | 20:43 |
| corvus | mordred: anyway, assuming all the things you've said are correct, that patch also looks correct. :) | 20:43 |
| mordred | \o/ | 20:44 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Allow overiding the remote hieradata link dest https://review.openstack.org/593134 | 20:44 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Make sure target manifest directory exists https://review.openstack.org/593140 | 20:44 |
| mordred | that looks mostly right- gonna do base one more time | 20:46 |
| *** jamesmcarthur has quit IRC | 20:46 | |
| *** jamesmcarthur has joined #openstack-infra | 20:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add loop_control and loop_var to silence a warning https://review.openstack.org/593173 | 20:48 |
| mordred | ok. I think git is good | 20:48 |
| mordred | that ^^ is some window dressing from a warning that went by | 20:48 |
| corvus | yeah, saw that too. ++ | 20:49 |
| mordred | corvus, clarkb: so - zuul - and review | 20:50 |
| mordred | we could run playbooks/remote_puppet_git.yaml | 20:50 |
| mordred | that'll only do the zuul scheduler | 20:50 |
| mordred | but it'll do git*, review and zuul | 20:51 |
| mordred | or - we could pick some zuul to do indivually? | 20:51 |
| clarkb | that makes me realize if we've merged any project adds and we don't add them to review first zuul will have a sad | 20:51 |
| clarkb | I think | 20:51 |
| corvus | mordred: the scheduler's the only scary one, so we might as well do that | 20:51 |
| *** jamesmcarthur has quit IRC | 20:51 | |
| clarkb | so we may want to do review first | 20:51 |
| mordred | kk. I'll do review by itself first | 20:51 |
| mordred | then scheduler by itself | 20:52 |
| corvus | by that i meant remote_puppet_git | 20:52 |
| mordred | oh - y'all are saying different things | 20:52 |
| corvus | if we do hosts alone, don't we risk messing up project creation? | 20:52 |
| corvus | how about this -- if we want to do one host at a time, let's do it in order. so let's run all the git* hosts now | 20:53 |
| corvus | then we can do review | 20:53 |
| corvus | then zuul | 20:53 |
| clarkb | corvus: wfm | 20:53 |
| mordred | kk | 20:54 |
| mordred | doing all git | 20:54 |
| mordred | oh - wait | 20:54 |
| mordred | we pass a project-config ref in | 20:54 |
| mordred | in the playbook | 20:54 |
| clarkb | ya so they all use the same reference (that way if project addition merges and races them they don't end up creating different sets of projcets) | 20:55 |
| mordred | facts: | 20:55 |
| fungi | in our favor, there are no pending project creations that i can see merged | 20:55 |
| mordred | project_config_ref: "{{ hostvars.localhost.gitinfo.after }}" | 20:55 |
| mordred | \o/ | 20:55 |
| mordred | so we can do them individually without worry. yay. git-server here we come | 20:55 |
| clarkb | you can probably just set that to whatever the ref is for project-config master and it will be happy | 20:55 |
| fungi | we've merged very little to project-config at all in the past 24 hours or so | 20:55 |
| clarkb | but ya individually should be fine given fungi's statement | 20:55 |
| mordred | SO MANY MORE HOSTS RUNNING AT ONCE | 20:56 |
| fungi | oh yes, that certainly looks impressive | 20:56 |
| clarkb | zoom zoom zoom | 20:57 |
| fungi | the sort of rapidly streaming output which would be right at home in any hollywood blockbuster with minimal investment in realism | 20:57 |
| fungi | he's just hacked the pentagon! let's worm their internal firewalls | 20:58 |
| mordred | ok. doing the same but with the puppets | 20:58 |
| mordred | 8x the rsync output | 20:59 |
| * fungi hopes someone just read all of that which flashed by | 21:00 | |
| mordred | I'm narrating it out loud | 21:00 |
| mordred | sandy isn't super thrilled | 21:00 |
| *** trown is now known as trown|outtypewww | 21:00 | |
| corvus | it's my favorite part again! | 21:00 |
| clarkb | just be glad cowsay isn't installed | 21:01 |
| clarkb | the poor cows | 21:01 |
| fungi | this wasn't here. none of this was here. and i heard a voice say "zuul" | 21:01 |
| * fungi saw a glimpse of "zuul" in the rsync output | 21:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/ansible-role-puppet master: Don't log puppot module sync https://review.openstack.org/593186 | 21:03 |
| fungi | heh, thanks! | 21:03 |
| corvus | fungi: yeah, there's a puppet zuul module | 21:03 |
| mordred | corvus: can we please write something and call it puppot | 21:03 |
| fungi | i agree that's not the best use of phosphors on my crt (if i actually had a crt) | 21:03 |
| mordred | k. doing base again | 21:04 |
| corvus | ftr, that's a legit dvorak typo. | 21:04 |
| clarkb | I thought dvorak made you type without typos because it is better for your fingers :P | 21:05 |
| corvus | clarkb: ain't nothin gonna make me type without typos | 21:05 |
| fungi | dvorak massages your fingers while you type | 21:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use is changed instead of | changed https://review.openstack.org/593189 | 21:05 |
| corvus | i've also worn the 'e' label off of my keyboard... that's... not something i've seen very often on a model m | 21:06 |
| mordred | ok. git servers seem good | 21:06 |
| mordred | I will now do review.o.o ? | 21:06 |
| corvus | wfm | 21:06 |
| clarkb | ++ | 21:06 |
| mordred | the extra time is nice for squashing all these little nigglies :) | 21:07 |
| mordred | ok. I can still log in and gerrit is still running | 21:08 |
| mordred | puppet | 21:08 |
| *** boden has quit IRC | 21:09 | |
| mordred | corvus: I'm totally pulling your nolog patch in for the next run | 21:09 |
| corvus | ++ | 21:09 |
| mordred | fwiw - I cloned a copy of ansible-role-puppet into system-config/roles so that we could pull patches without needing to land them | 21:10 |
| mordred | so that roles path turns out to have some nice benefits :) | 21:10 |
| *** apetrich has quit IRC | 21:10 | |
| mordred | re-running base - puppet seemed fine | 21:10 |
| *** masayukig has quit IRC | 21:10 | |
| mordred | after this we're doing zuul-scheduler, yes? | 21:11 |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| clarkb | hax | 21:11 |
| clarkb | yes zuul scheduler next | 21:11 |
| mordred | kk. here goes nothing | 21:12 |
| clarkb | assuming that goes well too I think we should consider getting things merged such that we aren't in a one off unmanaged state, then run against everything then turn it on? | 21:12 |
| mordred | ++ | 21:12 |
| clarkb | I wish the diff -'s weren't red | 21:12 |
| mordred | yes | 21:12 |
| mordred | land all the things next, then run run_all | 21:12 |
| clarkb | maybe rerun against review-dev again just to be sure we landed it all properly :) | 21:13 |
| mordred | yah | 21:13 |
| mordred | yay no_log! | 21:14 |
| *** rh-jelabarre has quit IRC | 21:15 | |
| mordred | kk. that seems all good | 21:15 |
| mordred | let's get these bad boys landed | 21:15 |
| mordred | I hit +A again on the one with a zuul failure - that's the flaky fail that's fixed later in the stack | 21:16 |
| *** jamesmcarthur has quit IRC | 21:16 | |
| corvus | i hardly noticed it running with the no_log | 21:16 |
| mordred | clarkb: you wanna +2/+A https://review.openstack.org/#/c/593126/2 ? | 21:16 |
| clarkb | which change addresses the flaky fail? | 21:16 |
| mordred | clarkb: https://review.openstack.org/#/c/593130/5 | 21:16 |
| mordred | it's two up the stack | 21:17 |
| mordred | we could squash those three | 21:17 |
| *** jamesmcarthur has joined #openstack-infra | 21:17 | |
| clarkb | oh right that fixed where and how we copy system-config which should make the tests happier | 21:18 |
| *** bobh has quit IRC | 21:18 | |
| clarkb | I think if we continue to not be able to merge things then ya either rebase and reorder or squash if necessary | 21:18 |
| mordred | cool. fornow we can recheck hammer | 21:18 |
| *** jianghuaw_ has quit IRC | 21:20 | |
| *** jamesmcarthur has quit IRC | 21:21 | |
| *** jamesmcarthur has joined #openstack-infra | 21:23 | |
| clarkb | just a few more minutes before everything goes into the gate hopefully | 21:33 |
| corvus | clarkb: https://review.openstack.org/593165 if you have a sec | 21:36 |
| clarkb | corvus: isn't text mode the default in python? | 21:40 |
| clarkb | eg you have to do wb to get binary | 21:40 |
| corvus | clarkb: not with gzip apparently | 21:40 |
| clarkb | huh | 21:40 |
| corvus | clarkb: https://docs.python.org/3/library/gzip.html#gzip.open | 21:40 |
| clarkb | I guess that kind of makes sense since gzip operates on bytes | 21:40 |
| clarkb | but weird to go against otherwise default behavior in the stdlib | 21:40 |
| mordred | clarkb: watching this patch try to land is like watching very boring paint dry slowly | 21:42 |
| mnaser | hmm | 21:42 |
| mnaser | clarkb: have you seen this in relation to nested virt "KVM: entry failed, hardware error 0x0" | 21:42 |
| mordred | clarkb: also - it seems like it's stuck | 21:42 |
| clarkb | mordred: ya thats curious no stream data? could just be a bug in zuul? | 21:42 |
| mnaser | this is a vm that landed on ovh and looks like is failing to run nested virt | 21:42 |
| mordred | clarkb: nevermind. I'm just too impatient | 21:42 |
| clarkb | mnaser: I can't recall the exact string but ya that sounds about right | 21:43 |
| mnaser | i guess we should disable kvm afterall | 21:43 |
| mnaser | (we don't do that in openstack ansible world) | 21:43 |
| mordred | clarkb: do we only disable kvm in devstack base job? | 21:44 |
| clarkb | mordred: ya devstack/devstack-gate is our control for that | 21:44 |
| clarkb | some groups then override it back to on (octavia) and others (osa I guess) likely don't control for it | 21:44 |
| mnaser | https://bugs.launchpad.net/cloud-archive/+bug/1739585/comments/21 | 21:45 |
| openstack | Launchpad bug 1739585 in OpenStack Charm Test Infra "L2 guest failed to boot under nested KVM: entry failed, hardware error 0x0" [Critical,Fix released] - Assigned to David Ames (thedac) | 21:45 |
| mnaser | ooou interesting | 21:45 |
| mnaser | i mean it's only in the infra side | 21:45 |
| clarkb | mordred: there is a nice stack in the gate now | 21:45 |
| johnsom | Yeah, I noticed one or two ovh bombing again. Not sure what caused that to regress | 21:45 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Don't log puppot module sync https://review.openstack.org/593186 | 21:46 |
| johnsom | The last round a new kernel version for the nodepool hosts resolved the issue, but I'm not sure why it would have gone away. | 21:47 |
| clarkb | johnsom: the cloud hypervisors that nodepool gets VMs from you mean | 21:47 |
| mnaser | yeah i just managed to make the CI for OSA stable and this is throwing it off :( | 21:47 |
| johnsom | clarkb No, it was the openstack images that fixed it | 21:47 |
| johnsom | The OS on the hosts didn't change | 21:48 |
| clarkb | ah, well there are new kernels as of like two days ago | 21:48 |
| clarkb | its possible they regressed | 21:48 |
| johnsom | Yeah, that is my guess | 21:48 |
| johnsom | Haven't had time to go dig into the ubuntu changelog to see the diff | 21:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 21:51 |
| mordred | clarkb, corvus, fungi: ^^ updated/rebaed version of 'turn cron on' | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 21:53 |
| mordred | sorry- fixed a comment | 21:53 |
| corvus | mordred: what'd you change? | 21:53 |
| clarkb | made comment more accurate about what update system-config playbook does | 21:54 |
| mordred | yeah. that | 21:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: htmlify: fix py3 gzip issue https://review.openstack.org/593165 | 21:58 |
| *** nicolasbock has quit IRC | 21:59 | |
| mordred | I'm going to clean up after myself on bridge a little bit - reset git repos and remove the extra roles thing | 22:06 |
| clarkb | ok | 22:06 |
| fungi | is 592215 safe to approve yet? | 22:06 |
| mordred | yah, I think so | 22:07 |
| clarkb | fungi: I think so, that stack is sort of independent of the transition itself | 22:07 |
| clarkb | assuming we don't want to rollback and revert | 22:07 |
| fungi | seemed like it should be, just thought i might be missing something | 22:07 |
| *** graphene has quit IRC | 22:08 | |
| mordred | I think we might want to manually test 593053 since it touches keys | 22:08 |
| mordred | but up til there in that stack I think it's safe | 22:08 |
| clarkb | ++ | 22:09 |
| mordred | and we can test that one quickly, since it only touches bridge | 22:09 |
| mordred | once the other things land | 22:09 |
| *** graphene has joined #openstack-infra | 22:09 | |
| mordred | also - https://review.openstack.org/#/c/593107 and https://review.openstack.org/#/c/593108 are docs updates | 22:10 |
| mordred | oh - and a big removal :) | 22:10 |
| openstackgerrit | Merged openstack-infra/system-config master: Be explicit about where to install roles https://review.openstack.org/593120 | 22:10 |
| openstackgerrit | Merged openstack-infra/system-config master: Set mgmt_hieradata in puppet group_vars https://review.openstack.org/593126 | 22:10 |
| openstackgerrit | Merged openstack-infra/system-config master: Be more explicit about puppet paths https://review.openstack.org/593130 | 22:10 |
| openstackgerrit | Merged openstack-infra/system-config master: Add skip: true to disable-puppet-agent https://review.openstack.org/593171 | 22:10 |
| fungi | so is 592765 likely to pass infra-puppet-apply-3-ubuntu-xenial now if we recheck it? | 22:10 |
| mordred | \o/ | 22:10 |
| mordred | fungi: yes | 22:10 |
| mordred | those one-off puppet-apply issues should be fixed now | 22:10 |
| mordred | they were ... fun | 22:11 |
| fungi | i thought one of the changes i reviewed in the other stack should have addressed it but wasn't clear | 22:11 |
| *** graphene has quit IRC | 22:11 | |
| mordred | yeah. it should have | 22:12 |
| clarkb | ya we had a path copying issue that intersected wtih a symlink in such a way to make rsync unhappy | 22:14 |
| *** graphene has joined #openstack-infra | 22:14 | |
| mordred | puppet-beaker-rspec-infra-system-config is boring | 22:15 |
| * mordred jumps up and down and pouts | 22:15 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add loop_control and loop_var to silence a warning https://review.openstack.org/593173 | 22:15 |
| clarkb | looks like two more changes before we run things off of master against our review-dev canary and then try run all? | 22:15 |
| openstackgerrit | Merged openstack-infra/system-config master: Use is changed instead of | changed https://review.openstack.org/593189 | 22:15 |
| mordred | woot | 22:15 |
| clarkb | and now no more changes | 22:15 |
| mordred | yup | 22:15 |
| mordred | ok. so now it's running run_all.sh | 22:16 |
| mordred | yeah? | 22:16 |
| clarkb | do we want to amke sure that what we merged works against review-dev first? | 22:16 |
| clarkb | (I kinda think its a good idea just because there were so many moving parts) | 22:16 |
| mordred | oh - right. one sec, doing | 22:16 |
| mordred | running update-system-config | 22:17 |
| mordred | k. now runing review-dev | 22:19 |
| clarkb | thats a lot of fetching origins | 22:19 |
| mordred | yeah | 22:20 |
| clarkb | that must be the puppet module installation | 22:20 |
| mordred | yah | 22:20 |
| mordred | clarkb: it's looking good ... | 22:20 |
| clarkb | ya so far seems to be pretty noopy | 22:21 |
| mordred | yup | 22:21 |
| clarkb | even the puppet run itself was a noop | 22:21 |
| mordred | ok. I think it's time for the big boy | 22:21 |
| * fungi makes thumbs-up sign | 22:22 | |
| clarkb | seems like it | 22:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use ansible.cfg for ansible logging config https://review.openstack.org/593208 | 22:23 |
| clarkb | mordred: I didn't catch the start of the current screen proecess running, this is the run all script ya? | 22:26 |
| mordred | clarkb: hah. well - this run of run_all is going to be pretty boring | 22:26 |
| mordred | yup | 22:26 |
| clarkb | mordred: boring is what we want :) | 22:26 |
| mordred | but we haven't added the line for base.yaml yes | 22:26 |
| clarkb | (or you mean no --verbose) | 22:26 |
| mordred | yet | 22:26 |
| clarkb | ah | 22:26 |
| mordred | so - it's good - but there's one more bit of 'excitement' left | 22:26 |
| fungi | more like a dry-run i guess | 22:27 |
| openstackgerrit | Merged openstack-infra/system-config master: Clean up puppetmaster puppet config handled by ansible https://review.openstack.org/592215 | 22:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove infracloud references from remote_puppet_else https://review.openstack.org/593210 | 22:31 |
| mordred | I am bored waiting for whatever I am waiting on right now | 22:32 |
| clarkb | generating a list of nodes to run against? | 22:33 |
| mordred | I'm guessing that's what's happening | 22:33 |
| mordred | but it should be operating on cache | 22:33 |
| mordred | so there might be a performance bug in that cache codepath somewhere | 22:33 |
| mordred | such that that expression is causing it to really churn | 22:34 |
| clarkb | mordred: there are 10 ansibles and control persist sshes running | 22:35 |
| mordred | yah | 22:36 |
| mordred | I agree with you | 22:36 |
| clarkb | /usr/bin/python && sleep 0 | 22:36 |
| mordred | this is putting a damper on my excitement | 22:36 |
| clarkb | fact gathering? | 22:36 |
| clarkb | hrm the IPs aren't changing though | 22:37 |
| corvus | --- stopped by SIGTTOU --- | 22:37 |
| corvus | i straced one | 22:37 |
| mordred | hrm | 22:37 |
| corvus | (that means the parent process isn't reading the child processes output) | 22:38 |
| mordred | should we kill the script and run the ansible-playbook command directly? | 22:38 |
| clarkb | its been about 10 minutes | 22:39 |
| corvus | 1 sec | 22:39 |
| clarkb | last doesn't seem to show the sshes ever connecting | 22:40 |
| *** jamesmcarthur has quit IRC | 22:42 | |
| corvus | the main ansible process is also stopped by sigttou | 22:43 |
| mordred | corvus: we are running these inside of the timeout command | 22:44 |
| mordred | maybe it has gone sideways? | 22:44 |
| fungi | could there be a read buffer problem in the parent? | 22:44 |
| corvus | maybe? i can't tell. it's sitting in rt_sigsuspend, i don't know if there's anything else it should be doing | 22:44 |
| clarkb | its also running under screen right? | 22:44 |
| mordred | yah | 22:44 |
| fungi | like, child process filled a buffer the parent wasn't reading fast enough (or at all) | 22:44 |
| corvus | i remove my objection to killing/restarting (i'm out of ideas atm) | 22:45 |
| mordred | me too | 22:45 |
| clarkb | if it happens againt we might run it without timeout and screen and just refer to the log fiel? | 22:46 |
| mordred | let's try restarting it outside of the run_all script just to see if we get any output | 22:46 |
| fungi | at least see if it recurs, yeah | 22:46 |
| mordred | oh good. ctrl-c doesn't work | 22:46 |
| *** SumitNaiksatam has quit IRC | 22:46 | |
| fungi | yay? | 22:46 |
| mordred | shall we kill the timeout process? | 22:47 |
| corvus | ya | 22:47 |
| fungi | we probably have to | 22:47 |
| *** SumitNaiksatam has joined #openstack-infra | 22:47 | |
| corvus | can also try c-\ | 22:47 |
| mordred | ok. kill worked | 22:47 |
| fungi | it might wake up from a sighup nudge, but may as well just sigterm it | 22:47 |
| mordred | that commandline look good ? | 22:48 |
| clarkb | I think so | 22:48 |
| * clarkb reads run_all.sh | 22:48 | |
| mordred | I catted it right before | 22:48 |
| fungi | yup | 22:48 |
| mordred | k. running | 22:49 |
| mordred | well. there we go | 22:49 |
| fungi | hah | 22:49 |
| clarkb | oh we are going to have to accept all the ssh keys aren't we | 22:49 |
| *** tpsilva has quit IRC | 22:50 | |
| mordred | ethercalc02 | 22:50 |
| mordred | no - just the once since we copied /root | 22:50 |
| clarkb | ah right I did that one after bridge was turned on | 22:50 |
| corvus | that is a plausible explanation :) | 22:50 |
| clarkb | review.o.o and citycloud mirrors not being accessible is expected | 22:50 |
| clarkb | I would expect packethost to be reachable though | 22:50 |
| fungi | that may be a second offline mirror instance? | 22:51 |
| mordred | clarkb: review.o.o ? | 22:51 |
| clarkb | mordred: I think we may have not deleted old gerrit yet | 22:51 |
| clarkb | pabelanger: ^ fyi | 22:51 |
| mordred | ah! review.o.o - not review01 | 22:51 |
| mordred | PHEW | 22:51 |
| clarkb | ribght | 22:51 |
| clarkb | but we should totally do that at this point | 22:52 |
| mordred | if some of these have been offline for a sec, we migh tneed to shell in as us or as puppetmaster to add bridge's ip to the authorized_keys info | 22:52 |
| mordred | like, if they weren't live when we added it via puppet from puppetmaster | 22:53 |
| mordred | corvus: it is SO GOOD that we landed the no_log change | 22:53 |
| corvus | mordred: yay i feel like i helped! | 22:54 |
| clarkb | well I don't think we want review.o.o to be ansible puppeted | 22:54 |
| clarkb | it should just go away (but I'm not doing that on a friday evening :) ) | 22:54 |
| mordred | totes | 22:54 |
| mordred | just saying - if any of the others are valid things we want - we may need to manual them back into the fold | 22:55 |
| clarkb | ya | 22:55 |
| fungi | same goes for systems in emergency disable or normal disable groups? | 22:56 |
| mordred | yah | 22:56 |
| *** graphene has quit IRC | 22:58 | |
| *** graphene has joined #openstack-infra | 22:59 | |
| corvus | something's running on ethercalc02 still | 23:01 |
| *** r-daneel has quit IRC | 23:01 | |
| corvus | nevermind | 23:01 |
| openstackgerrit | Merged openstack-infra/system-config master: Update launch-node for the new Ansible world https://review.openstack.org/592765 | 23:02 |
| mordred | so - I'm starting to run in to an EOD-related logistics issue- mostly that I need to go pick up some gear before the shop closes. do any of you feel comfortable driving final steps? (sorry, I wasn't quite expecting it to linger this long) | 23:04 |
| mordred | or I can also continue to followup once I'm back bearing gear | 23:04 |
| *** bobh has joined #openstack-infra | 23:04 | |
| corvus | what are final steps? | 23:05 |
| mordred | I think we need to run the base playbook without any --limit exclusions | 23:05 |
| mordred | then it's about landing the enable-cron patch - and running bridge.yaml to get the cronjob added | 23:06 |
| corvus | hrm. i'm leaning toward saying i'm not super comfortable driving. i may be more of a supporting player at this point. | 23:07 |
| mordred | totally fair. I'm happy to either drive again in a bit - or we can hit the pause button on the process again -what do y'all think? | 23:07 |
| corvus | clarkb: are you comfortable continuing with me around to help with issues, or you want to pause for now? | 23:08 |
| clarkb | pause makes me most comfortable :) also I'm trying to be on vacation at EOD today so don't want to end up going too late | 23:08 |
| fungi | i can mash some buttons for sure, but worried if something goes sideways i may not be well suited for troubleshooting it | 23:08 |
| clarkb | I can be around in the morning if that works? | 23:08 |
| mordred | I can't do morning - we've got scuba training in the morning (this is why I have to go get gear before the shop closes) | 23:09 |
| corvus | mordred: yeah, let's pause for now. like yesterday -- nothing is broken now which is good :) | 23:09 |
| mordred | ++ | 23:09 |
| mordred | and it's unlikely changes are going to happen over the weekend that are going to need to get applied immediately | 23:09 |
| fungi | and we're caught up on applying configuration anyway i think, based on what i saw recent in project-config | 23:09 |
| mordred | yah | 23:10 |
| mordred | so resume monday morning? | 23:10 |
| fungi | wfm | 23:10 |
| mordred | ossum. I'll send out a followup email to the infra list when I get back from the store | 23:10 |
| mordred | and with that ... I'll chat with everybody later - thanks for the help!!! | 23:11 |
| clarkb | mordred: thanks! | 23:11 |
| clarkb | I think this is real close now | 23:11 |
| corvus | mordred: yay! thanks | 23:11 |
| fungi | really quite great. i didn't expect we'd be this far along with the effort so soon | 23:16 |
| clarkb | I think ethercalc02 is still unhappy | 23:16 |
| clarkb | it has the ip in /root/.ssh/authorized_keys | 23:17 |
| * clarkb going to test ssh from bridge | 23:17 | |
| clarkb | I had to accept the ssh host key | 23:18 |
| clarkb | I don't think that is going to make the ssh process happy though | 23:18 |
| clarkb | corvus: fungi should we consider killing just that process to allow the rest of ansible to complete? | 23:18 |
| clarkb | I imagine that it will eventually time out? | 23:19 |
| *** SumitNaiksatam has quit IRC | 23:21 | |
| clarkb | I'm going to take a break and then check back in a bit to see if it has timed out or if anyone else decided we should kill it | 23:21 |
| fungi | looking at it now | 23:21 |
| *** SumitNaiksatam has joined #openstack-infra | 23:22 | |
| fungi | yeah, it will need to retry the authentication... possible killing the ssh child process will trigger that or just cause it to get skipped | 23:23 |
| fungi | not sure what ansible's behavior would be in that situation | 23:24 |
| *** rlandy has quit IRC | 23:27 | |
| *** SumitNaiksatam has quit IRC | 23:30 | |
| *** bobh has quit IRC | 23:34 | |
| clarkb | I think it will treat it as a failure to connect | 23:48 |
| clarkb | it is still there though, coming up on an hour which seems like a sane timeout (so maybe wait for that to go past then consider killing it) | 23:48 |
| clarkb | fungi: corvus ^ what do you think | 23:48 |
| fungi | sounds good | 23:49 |
| clarkb | ok past an hour now | 23:50 |
| clarkb | fungi: I'm going to kill 26120 that look right ot you? | 23:50 |
| fungi | yep, that looks like the correct pid to me | 23:52 |
| clarkb | done and that caused the process ins screen to finish up and report | 23:53 |
| clarkb | I did accept the key as root on bridge so I think it is good for the next attempt on monday | 23:53 |
| fungi | sounds good | 23:54 |
| fungi | some of those failures are due to being listed in the disable groups, right? | 23:54 |
| clarkb | I don't think so | 23:55 |
| fungi | actually logstash-worker08 and openstackid-dev are some other failure, not unreachable | 23:55 |
| clarkb | review.openstack.org is unreachable because its effectively off iirc but we haven't deleted it | 23:56 |
| clarkb | us-west-1.packethost mirror likely need to allow this new ip address? | 23:56 |
| clarkb | openstackid-dev failed to npm deploy a thing | 23:56 |
| clarkb | possibly due to too old of a nodejs? its complaining about methods not existing | 23:57 |
| clarkb | I wish https://docs.ansible.com/ansible/2.4/intro_configuration.html#timeout had a unit | 23:57 |
| clarkb | I'm going to assume seconds? | 23:57 |
| clarkb | but if 10 seconds is the default then why doesn't it timeout after an hour | 23:58 |
| clarkb | maybe someone that understands ansible can explain that behavior | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!