| clarkb | dmsimard: is that because it forks anoyher ansible to run the actual base playbook? | 00:01 |
|---|---|---|
| clarkb | I think you want to edit the bridge.o.o hostvar to set that install flag instead | 00:01 |
| clarkb | then when we ansible for bridge.o.o it will see the var is set | 00:01 |
| dmsimard | but the job doesn't seem to use hostvars | 00:01 |
| clarkb | it does | 00:01 |
| dmsimard | it's on a nodepool vm right ? | 00:01 |
| dmsimard | not the real thing ? | 00:01 |
| clarkb | yes | 00:02 |
| dmsimard | so it doesn't have all the private hostvars ? | 00:02 |
| dmsimard | or does it | 00:02 |
| clarkb | correct we have fake private hostvars but this is a public hostvar so you can just edit that | 00:02 |
| dmsimard | where are they ? | 00:02 |
| clarkb | playbooks/host_vars/ iirc | 00:02 |
| dmsimard | that's probably what I'm missing | 00:02 |
| dmsimard | ohhhhh | 00:03 |
| *** betherly has joined #openstack-infra | 00:03 | |
| dmsimard | ok | 00:03 |
| dmsimard | funny thing is ansible has hiera lookups | 00:04 |
| dmsimard | never used it though | 00:04 |
| clarkb | we converted our hiera to ansible grouo and host vars | 00:04 |
| clarkb | so hiera does lookups in /etc/ansible/*_vars | 00:04 |
| dmsimard | ohhhh | 00:05 |
| dmsimard | clever | 00:05 |
| dmsimard | I've been missing out so much :( | 00:05 |
| *** betherly has quit IRC | 00:08 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add support for enabling the ARA callback plugin in install-ansible https://review.openstack.org/611228 | 00:14 |
| *** sthussey has quit IRC | 00:16 | |
| *** betherly has joined #openstack-infra | 00:23 | |
| *** betherly has quit IRC | 00:28 | |
| *** jamesmcarthur has quit IRC | 00:42 | |
| *** fuentess_ has quit IRC | 00:46 | |
| *** markvoelker has joined #openstack-infra | 00:58 | |
| *** jamesmcarthur has joined #openstack-infra | 01:03 | |
| *** jamesmcarthur has quit IRC | 01:12 | |
| *** jamesmcarthur has joined #openstack-infra | 01:13 | |
| *** jamesmcarthur has quit IRC | 01:13 | |
| *** beekneemech has joined #openstack-infra | 01:17 | |
| *** bnemec has quit IRC | 01:19 | |
| *** imacdonn has quit IRC | 01:22 | |
| *** imacdonn has joined #openstack-infra | 01:22 | |
| *** mrsoul has quit IRC | 01:25 | |
| *** markvoelker has quit IRC | 01:32 | |
| *** kjackal has quit IRC | 01:42 | |
| *** zhangfei has joined #openstack-infra | 01:42 | |
| *** kjackal has joined #openstack-infra | 01:42 | |
| *** annp has joined #openstack-infra | 01:47 | |
| *** dklyle has joined #openstack-infra | 01:50 | |
| *** david-lyle has joined #openstack-infra | 01:53 | |
| *** dklyle has quit IRC | 01:56 | |
| *** dklyle has joined #openstack-infra | 01:56 | |
| *** david-lyle has quit IRC | 01:59 | |
| imacdonn | zuul dashboard still having issues? I'm trying to monitor 598426 (only), but when I try to filter on that, I see a bunch of unrelated jobs in the "integrated" queue | 02:01 |
| *** jmccrory has joined #openstack-infra | 02:02 | |
| ianw | imacdonn: i think that might be a feature showing you the queue | 02:03 |
| *** hongbin has joined #openstack-infra | 02:03 | |
| imacdonn | seems to be a big chain of connected jobs .... some in progress, others not started | 02:03 |
| imacdonn | some been running for over 10 hours ..... that's probably not good | 02:04 |
| imacdonn | my job is way at the bottom of the chain, so I guess it is the queue .... I think it's stuck, though | 02:08 |
| *** bnemec has joined #openstack-infra | 02:13 | |
| *** beekneemech has quit IRC | 02:15 | |
| *** beekneemech_ has joined #openstack-infra | 02:15 | |
| fungi | not stuck, the two times listed for each change in that queue are estimated remaining time for jobs to finish (top) and time that change has been in the queue (bottom) | 02:16 |
| fungi | looks like the openstack integrated gate queue is running a ~9.5-hour backlog | 02:17 |
| *** bnemec has quit IRC | 02:18 | |
| fungi | if the top time is "unknown" that's because one or more of its jobs hasn't started running yet | 02:19 |
| *** dklyle has quit IRC | 02:20 | |
| imacdonn | OK, that helps ... thanks, fungi | 02:21 |
| *** betherly has joined #openstack-infra | 02:26 | |
| *** bnemec has joined #openstack-infra | 02:26 | |
| *** beekneemech_ has quit IRC | 02:29 | |
| *** kazsh has quit IRC | 02:30 | |
| *** betherly has quit IRC | 02:30 | |
| *** beekneemech has joined #openstack-infra | 02:31 | |
| openstackgerrit | Andriy Shevchenko proposed openstack/ansible-role-cloud-launcher master: Update min tox version to 2.0 https://review.openstack.org/611745 | 02:32 |
| *** bnemec has quit IRC | 02:34 | |
| *** kazsh has joined #openstack-infra | 02:35 | |
| *** yamamoto has quit IRC | 02:38 | |
| *** jamesmcarthur has joined #openstack-infra | 02:38 | |
| *** yamamoto has joined #openstack-infra | 02:38 | |
| *** jamesmcarthur has quit IRC | 02:43 | |
| *** auristor has quit IRC | 02:44 | |
| *** bobh has joined #openstack-infra | 02:45 | |
| *** bobh has quit IRC | 02:46 | |
| *** bobh has joined #openstack-infra | 02:46 | |
| *** bnemec has joined #openstack-infra | 02:48 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move common ubuntu environment setting to ubuntu-common element https://review.openstack.org/563731 | 02:50 |
| *** bobh has quit IRC | 02:51 | |
| *** beekneemech has quit IRC | 02:51 | |
| *** auristor has joined #openstack-infra | 02:54 | |
| *** betherly has joined #openstack-infra | 02:56 | |
| *** betherly has quit IRC | 03:00 | |
| *** jamesmcarthur has joined #openstack-infra | 03:02 | |
| openstackgerrit | Merged openstack/diskimage-builder master: ubuntu-common: Update default DIB_RELEASE to bionic https://review.openstack.org/603940 | 03:23 |
| openstackgerrit | Merged openstack/diskimage-builder master: Allow debootstrap to cleanup without a kernel https://review.openstack.org/604692 | 03:23 |
| *** jamesmcarthur has quit IRC | 03:24 | |
| *** zhangfei has quit IRC | 03:27 | |
| *** zhangfei has joined #openstack-infra | 03:27 | |
| *** hongbin has quit IRC | 03:46 | |
| *** betherly has joined #openstack-infra | 03:47 | |
| *** jamesmcarthur has joined #openstack-infra | 03:49 | |
| *** betherly has quit IRC | 03:52 | |
| *** dave-mccowan has quit IRC | 03:53 | |
| *** zhangfei has quit IRC | 04:01 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE regex typo https://review.openstack.org/611559 | 04:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add a workflow overview for adding a cloud https://review.openstack.org/611200 | 04:03 |
| *** longkb has joined #openstack-infra | 04:07 | |
| *** jamesmcarthur has quit IRC | 04:07 | |
| *** beekneemech has joined #openstack-infra | 04:10 | |
| *** bnemec has quit IRC | 04:13 | |
| *** jaosorior has joined #openstack-infra | 04:24 | |
| *** bnemec has joined #openstack-infra | 04:45 | |
| *** beekneemech has quit IRC | 04:48 | |
| *** bnemec has quit IRC | 04:51 | |
| *** bnemec has joined #openstack-infra | 04:51 | |
| *** beekneemech has joined #openstack-infra | 05:03 | |
| *** bnemec has quit IRC | 05:06 | |
| *** andreaf has quit IRC | 05:10 | |
| tobiash | ianw: thanks for review on https://review.openstack.org/611644, responded with my use case and a question about an alternative solution | 05:12 |
| *** andreaf has joined #openstack-infra | 05:15 | |
| *** rlandy has quit IRC | 05:17 | |
| *** quiquell|off is now known as quiquell | 05:36 | |
| *** felipemonteiro has quit IRC | 05:41 | |
| *** apetrich has joined #openstack-infra | 05:47 | |
| *** annp has quit IRC | 05:56 | |
| *** jtomasek has quit IRC | 05:58 | |
| openstackgerrit | Tobias Henkel proposed openstack/diskimage-builder master: Also get DIB_IMAGE_SIZE variable from block-device phase https://review.openstack.org/611780 | 05:58 |
| *** jtomasek has joined #openstack-infra | 06:04 | |
| ianw | tobiash: oh ... interesting :) | 06:07 |
| tobiash | ianw: so what do you think about the DIB_IMAGE_SIZE approach? | 06:08 |
| tobiash | I think that would be better than my first proposal | 06:08 |
| ianw | well ... i'm not really following how that helps, because the block-device phase will actually create the loopback image of that size ... so by the time it has exported something it's too late? | 06:09 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/611783 | 06:10 |
| tobiash | ianw: reading the code the block device is created after the block-device phase (if nothing in the block-device phase really creates it) | 06:10 |
| ianw | tobiash: oh ... ok maybe i see. so you *don't* export IMAGE_BLOCK_DEVICE at all | 06:10 |
| tobiash | yes | 06:10 |
| ianw | so we go into the standard creation path | 06:10 |
| tobiash | exactly | 06:10 |
| ianw | ahh, that makes some sense | 06:11 |
| ianw | well a lot of sense :) just surprising as it was not the intended way things were supposed to be going with the block device :) | 06:11 |
| tobiash | ianw: yea, I just needed any phase between install and block device creation that doesn't run within the chroot | 06:12 |
| tobiash | and that's the only candidate | 06:12 |
| *** pcaruana has joined #openstack-infra | 06:14 | |
| ianw | tobiash: we just added a pre-finalise.d phase that runs outside the chroot -> https://review.openstack.org/#/c/609863/ | 06:17 |
| ianw | i know that doesn't help, but maybe we should consider another phase .... | 06:18 |
| ianw | we already have post-install.d ... a post-post-install.d :) | 06:18 |
| tobiash | that could maybe solve that problem | 06:19 |
| tobiash | ianw: hrm, the block device is still too small, I think because cp -ra might not preserve hard links: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/lib/disk-image-create#n485 | 06:22 |
| AJaeger | config-core, couple of cleanups and small changes for review, please: https://review.openstack.org/609339 https://review.openstack.org/609270 https://review.openstack.org/610428 https://review.openstack.org/610175 https://review.openstack.org/608867 https://review.openstack.org/#/c/611431/ | 06:22 |
| ianw | tobiash: oh, that could be right | 06:25 |
| ianw | tobiash: at this point, i'm thinking an intermediate phase, outside the chroot before block-device.d works best | 06:25 |
| ianw | otherwise we're documenting all these tricks about putting block-device.d scripts in, exporting magic variables ... it just seems simpler | 06:26 |
| ianw | i don't think there's any reason why it's *not* there ... just nobody need it (similar to the other phase we added) | 06:26 |
| tobiash | ianw: yes, that would be the best (under the assumption we can solve that cp -ar issue) | 06:27 |
| tobiash | ianw: in an acient version of diskimage builder I have docker pull working (which is broken by some refactorings in current diskimage builder) and there I'm using 'tar -c ${SRC_DIR} | tar -C ${TARGET_DIR} -xp' for copying the docker graph into the final image | 06:28 |
| tobiash | I'll try another run with tar instead of cp | 06:31 |
| *** jaosorior has quit IRC | 06:32 | |
| tobiash | ianw: how would you name the the post-post-install.d? | 06:33 |
| tobiash | pre-block-devide.d? | 06:33 |
| *** jaosorior has joined #openstack-infra | 06:34 | |
| ianw | tobiash: yeah, i'm not feeling pre-block-device.d because it's not *really* to do with block devices. hrm ... | 06:36 |
| ianw | tobiash: i'm thinking, maybe post-root.d ? that feels symmetric with root.d? | 06:37 |
| tobiash | ianw: sounds good | 06:37 |
| *** e0ne has joined #openstack-infra | 06:37 | |
| *** kjackal has quit IRC | 06:42 | |
| *** gfidente has joined #openstack-infra | 06:45 | |
| *** ccamacho has joined #openstack-infra | 06:45 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add nodepool to project-config-nodepool required-projects https://review.openstack.org/611629 | 06:51 |
| openstackgerrit | wuwei proposed openstack-infra/project-config master: okey https://review.openstack.org/611787 | 06:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/611783 | 07:04 |
| *** pcaruana has quit IRC | 07:09 | |
| *** shardy has joined #openstack-infra | 07:10 | |
| *** stakeda has joined #openstack-infra | 07:17 | |
| *** rcernin has quit IRC | 07:17 | |
| *** kjackal has joined #openstack-infra | 07:22 | |
| *** pcaruana has joined #openstack-infra | 07:28 | |
| *** pcaruana is now known as pcaruana|elisa| | 07:30 | |
| *** slaweq has quit IRC | 07:32 | |
| *** slaweq has joined #openstack-infra | 07:33 | |
| *** tosky has joined #openstack-infra | 07:36 | |
| *** eernst has joined #openstack-infra | 07:38 | |
| *** jbadiapa has quit IRC | 07:53 | |
| *** electrofelix has joined #openstack-infra | 07:56 | |
| *** e0ne has quit IRC | 08:03 | |
| *** eernst has quit IRC | 08:08 | |
| *** jbadiapa has joined #openstack-infra | 08:08 | |
| *** jbadiapa has quit IRC | 08:18 | |
| *** quite has quit IRC | 08:21 | |
| amorin | ianw fungi AJaeger hi | 08:22 |
| amorin | can you re-enable your spawn on GRA1 ? | 08:23 |
| amorin | on OVH | 08:23 |
| amorin | I would like to monitor neutron behavior on your tenant | 08:23 |
| dpawlik | revert the patch :D | 08:23 |
| dpawlik | https://review.openstack.org/#/q/project:openstack-infra/project-config | 08:24 |
| *** quite has joined #openstack-infra | 08:24 | |
| amorin | hum,seems that it already re-eanbled | 08:24 |
| amorin | for 20 instances | 08:24 |
| dpawlik | yup | 08:24 |
| *** derekh has joined #openstack-infra | 08:26 | |
| *** witek has quit IRC | 08:27 | |
| AJaeger | amorin: I don#t have that access. I think ianw enabled to 20 nodes. Is that ok? | 08:27 |
| *** witek has joined #openstack-infra | 08:27 | |
| AJaeger | dpawlik, amorin should we go to 79 again? | 08:28 |
| amorin | nop that's correct for now | 08:28 |
| amorin | thanks | 08:28 |
| dpawlik | AJaeger, ianw, amorin: problem that we have is in WIP state: https://review.openstack.org/#/c/611341/ | 08:30 |
| AJaeger | ianw, frickler: are you around to help? | 08:30 |
| amorin | dont worry, 20 is enough for now | 08:30 |
| AJaeger | amorin: Good! Then we stay as is. | 08:31 |
| AJaeger | We have right now the nodepool launcher in our emergency file, so even if 611341 merges, it won't make a difference. An admin needs to take it out - or manually edit... | 08:31 |
| openstackgerrit | Tobias Henkel proposed openstack/diskimage-builder master: Add a post-root.d phase https://review.openstack.org/611806 | 08:35 |
| tobiash | ianw: confirmed that this works for my use case ^ | 08:36 |
| *** jbadiapa has joined #openstack-infra | 08:36 | |
| *** vabada has quit IRC | 08:42 | |
| *** vabada has joined #openstack-infra | 08:43 | |
| ianw | amorin AJaeger: i took nb04 out this morning, the difference atm is that i have a script running to clear leaked ports | 08:43 |
| ianw | (this morning my time :) | 08:43 |
| ianw | https://review.openstack.org/609133 script in particular. it removes quite a few ports each time, but it seems to keep the region going | 08:44 |
| ianw | it runs once every 20 minutes | 08:44 |
| ianw | tobiash: thanks, if ok i'll take a look later | 08:45 |
| tobiash | thanks | 08:45 |
| tobiash | sure, ok | 08:45 |
| ianw | ^ by "out" I mean "out of emergency". so we can merge changes to modify quotas | 08:48 |
| AJaeger | ianw: thanks | 08:50 |
| *** kjackal has quit IRC | 08:52 | |
| *** kjackal has joined #openstack-infra | 08:52 | |
| *** roman_g has joined #openstack-infra | 08:59 | |
| *** kjackal has quit IRC | 09:02 | |
| *** kjackal has joined #openstack-infra | 09:03 | |
| *** roman_g has quit IRC | 09:06 | |
| *** gfidente is now known as gfidente|giulivo | 09:07 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/openstack-zuul-jobs master: Remove ovsdbapp job https://review.openstack.org/604114 | 09:08 |
| *** e0ne has joined #openstack-infra | 09:09 | |
| *** zigo has joined #openstack-infra | 09:13 | |
| *** maciejjozefczyk has quit IRC | 09:30 | |
| *** quiquell is now known as quiquell|bbl | 09:30 | |
| *** quiquell|bbl has quit IRC | 09:30 | |
| *** maciejjozefczyk has joined #openstack-infra | 09:31 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Remove sqlalchemy from requirements https://review.openstack.org/611821 | 09:36 |
| *** rossella_s has quit IRC | 09:41 | |
| *** rossella_s has joined #openstack-infra | 09:41 | |
| *** roman_g has joined #openstack-infra | 09:45 | |
| *** roman_g has quit IRC | 09:53 | |
| *** yamamoto has quit IRC | 10:13 | |
| *** yamamoto has joined #openstack-infra | 10:14 | |
| *** sambetts|afk is now known as sambetts | 10:14 | |
| *** stakeda has quit IRC | 10:16 | |
| *** yamamoto has quit IRC | 10:19 | |
| *** dtantsur|afk is now known as dtantsur | 10:43 | |
| *** dave-mccowan has joined #openstack-infra | 10:53 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** e0ne has quit IRC | 11:01 | |
| fungi | reminder: i won't be around today, but should be back by sunday | 11:21 |
| *** carl_cai has joined #openstack-infra | 11:23 | |
| *** longkb has quit IRC | 11:23 | |
| *** quiquell has joined #openstack-infra | 11:23 | |
| *** xek has joined #openstack-infra | 11:24 | |
| jaosorior | fungi: hey, if we want to add functional tests... like a devstack-based or something of the sort; to a project, what would be the relevant docs or steps to follow? | 11:24 |
| *** roman_g has joined #openstack-infra | 11:32 | |
| AJaeger | jaosorior: I doubt we have much docs here | 11:33 |
| AJaeger | jaosorior: for devstack-based, ask on #openstack-qa. There should be examples on how to do it | 11:33 |
| AJaeger | jaosorior: So, it's defining a job in one of your .zuul.yaml files - and then using it. | 11:34 |
| AJaeger | jaosorior: I think Qa has docs, let me check quickly... | 11:34 |
| AJaeger | jaosorior: https://docs.openstack.org/devstack/latest/zuul_ci_jobs_migration.html | 11:34 |
| jaosorior | Thanks AJaeger! I'll check that out | 11:36 |
| *** yamamoto has quit IRC | 11:39 | |
| *** yamamoto has joined #openstack-infra | 11:40 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder master: Add an element to configure iBFT network interfaces https://review.openstack.org/391787 | 11:43 |
| *** yamamoto has quit IRC | 11:44 | |
| *** e0ne has joined #openstack-infra | 11:52 | |
| *** tpsilva has joined #openstack-infra | 11:53 | |
| *** panda is now known as panda|lunch | 11:53 | |
| *** EmilienM is now known as EvilienM | 11:58 | |
| *** yamamoto has joined #openstack-infra | 12:01 | |
| *** sshnaidm is now known as sshnaidm|off | 12:06 | |
| *** yamamoto has quit IRC | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:08 | |
| *** yamamoto has quit IRC | 12:08 | |
| *** gfidente|giulivo has quit IRC | 12:11 | |
| *** kjackal has quit IRC | 12:11 | |
| *** yamamoto has joined #openstack-infra | 12:14 | |
| *** apetrich has quit IRC | 12:14 | |
| *** mriedem has joined #openstack-infra | 12:19 | |
| *** yamamoto has quit IRC | 12:20 | |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| mordred | jaosorior, AJaeger: there is a devstack-tox-functional base job that can be used for running tox-based functional tests on a devstack node | 12:23 |
| mordred | jaosorior, AJaeger: openstack/openstacksdk repo has an example of using it, as does openstack/python-openstackclient | 12:24 |
| *** jangutter has quit IRC | 12:30 | |
| *** dpawlik has quit IRC | 12:30 | |
| *** yamamoto has quit IRC | 12:30 | |
| *** dpawlik has joined #openstack-infra | 12:30 | |
| *** eharney has joined #openstack-infra | 12:33 | |
| *** priteau has joined #openstack-infra | 12:34 | |
| *** dpawlik has quit IRC | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:38 | |
| *** trown|outtypewww is now known as trown | 12:41 | |
| *** yamamoto has joined #openstack-infra | 12:41 | |
| *** dansmith is now known as SteelyDan | 12:45 | |
| *** yamamoto has quit IRC | 12:46 | |
| *** jamesmcarthur has joined #openstack-infra | 12:48 | |
| *** kgiusti has joined #openstack-infra | 12:48 | |
| *** kjackal has joined #openstack-infra | 12:53 | |
| *** yamamoto has joined #openstack-infra | 12:56 | |
| *** stephenfin is now known as finucannot | 12:57 | |
| *** slaweq has quit IRC | 12:57 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder master: Add an element to configure iBFT network interfaces https://review.openstack.org/391787 | 13:03 |
| *** slaweq has joined #openstack-infra | 13:07 | |
| *** agopi|brb is now known as agopi | 13:08 | |
| *** roman_g has quit IRC | 13:10 | |
| *** derekh has quit IRC | 13:11 | |
| *** derekh has joined #openstack-infra | 13:11 | |
| *** sthussey has joined #openstack-infra | 13:14 | |
| *** jamesmcarthur has quit IRC | 13:14 | |
| *** jamesmcarthur has joined #openstack-infra | 13:14 | |
| *** jamesmcarthur has quit IRC | 13:17 | |
| *** lbragstad is now known as elbragstad | 13:23 | |
| *** slaweq has quit IRC | 13:27 | |
| *** dpawlik has joined #openstack-infra | 13:37 | |
| *** dpawlik has quit IRC | 13:38 | |
| *** dpawlik has joined #openstack-infra | 13:38 | |
| *** jamesmcarthur has joined #openstack-infra | 13:39 | |
| *** jamesmcarthur has quit IRC | 13:40 | |
| *** jamesmcarthur has joined #openstack-infra | 13:40 | |
| *** strigazi has quit IRC | 13:44 | |
| *** strigazi has joined #openstack-infra | 13:44 | |
| *** verdurin has quit IRC | 13:49 | |
| *** slaweq has joined #openstack-infra | 13:49 | |
| *** verdurin has joined #openstack-infra | 13:52 | |
| *** munimeha1 has joined #openstack-infra | 13:53 | |
| amorin | hi ianw AJaeger fungi and all | 13:55 |
| amorin | GRA1 on OVH seems in better shape | 13:55 |
| amorin | maybe we can go to 79 instances? | 13:55 |
| *** rpittau has quit IRC | 13:56 | |
| ttx | ping infra-core ^ | 13:56 |
| amorin | ttx: how? | 13:56 |
| ttx | oh, my mention should do it :) | 13:56 |
| ttx | (mentioning infra-core to make sure the others also see it) | 13:57 |
| amorin | ah ok | 13:57 |
| ttx | I know fungi is mostly off today | 13:57 |
| amorin | ok | 13:57 |
| mordred | amorin, ttx: lemme go poke at the quota | 13:58 |
| amorin | I was wondering why 79 btw? | 13:58 |
| amorin | why not more? | 13:58 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/openstack-zuul-jobs master: Avoid ignore_errors on Set ephemeral device by label https://review.openstack.org/611876 | 13:58 |
| mordred | amorin: I think that's just our quota there? (we also have a long-lived server in that project to handle the mirror) | 13:59 |
| mordred | amorin, ttx: I have set the quota on GRA1 to 79 | 13:59 |
| amorin | mordred: let me check the quota on our side | 13:59 |
| mordred | let's see how it does | 13:59 |
| AJaeger | amorin: sure, let's merge the change... | 14:00 |
| amorin | thanks | 14:00 |
| mordred | AJaeger: that launcher is in the emergency file, yes? | 14:00 |
| AJaeger | mordred: not anymore | 14:00 |
| AJaeger | mordred: just rebasing... | 14:00 |
| mordred | oh - ok. sweet - I read scrollback poorly | 14:00 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Restore full OVH-GRA1 quota https://review.openstack.org/611553 | 14:01 |
| AJaeger | mordred: ^ let's merge ... | 14:01 |
| amorin | | instances | 200 | | 14:01 |
| amorin | this is your quota | 14:01 |
| amorin | I think you can spawn more that 79 | 14:01 |
| amorin | hum | 14:02 |
| amorin | maybe not, the aggregate seems to be able to handle on 80 | 14:02 |
| mordred | AJaeger: ++ | 14:02 |
| amorin | ok, nevermind, stay at 79 for now, we'll see | 14:02 |
| mordred | :) | 14:02 |
| mordred | we'd love more quota of course :) | 14:03 |
| amorin | I will try to find more hosts if possible :p | 14:03 |
| ttx | I'll take taht 1,000,000th server that you plan to put in the museum | 14:04 |
| AJaeger | amorin: I think we never used more than 79 so far... | 14:04 |
| *** mriedem has quit IRC | 14:06 | |
| *** felipemonteiro has joined #openstack-infra | 14:08 | |
| *** mriedem has joined #openstack-infra | 14:09 | |
| amorin | 62 active instances so far | 14:12 |
| amorin | some are in error, checking why | 14:13 |
| amorin | nova scheduler was not able to find a host | 14:13 |
| maciejjozefczyk | amorin: Slawek Kaplonski noticed some neutron tempest jobs failures due timeout on db migrations | 14:25 |
| maciejjozefczyk | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:\"in%20test_walk_versions\"%20AND%20build_name:\"neutron-functional\" | 14:25 |
| maciejjozefczyk | amorin: I verified that the openstack has dedicated resources. But in fact while one host runs 10 of tempest tests its kind overloaded | 14:26 |
| *** efried is now known as efried_pto | 14:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove fetch-tox-output from release-openstack-python https://review.openstack.org/611886 | 14:26 |
| maciejjozefczyk | maybe it is possible to mix tempest runs with unittest runs? and be ensured that no more than X tempests runs on the same hypervisor? | 14:26 |
| amorin | euh | 14:27 |
| *** boden has joined #openstack-infra | 14:28 | |
| maciejjozefczyk | for now some of hosts are overloaded due this, some are not, that impacts tempest; basically it runs vms inside devstack, so that is the issue | 14:29 |
| *** jistr is now known as jistr|call | 14:29 | |
| *** apetrich has joined #openstack-infra | 14:36 | |
| *** bobh has joined #openstack-infra | 14:38 | |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New Repo - OpenStack-Helm Images https://review.openstack.org/611892 | 14:38 |
| *** slaweq has quit IRC | 14:40 | |
| *** felipemonteiro has quit IRC | 14:43 | |
| *** apetrich has quit IRC | 14:43 | |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New Repo: OpenStack-Helm Docs https://review.openstack.org/611893 | 14:43 |
| *** apetrich has joined #openstack-infra | 14:44 | |
| *** maciejjozefczyk has quit IRC | 14:45 | |
| *** jistr|call is now known as jistr | 14:47 | |
| *** panda|lunch is now known as panda | 14:49 | |
| *** e0ne has quit IRC | 14:52 | |
| *** armstrong has joined #openstack-infra | 14:53 | |
| *** adriancz has joined #openstack-infra | 14:53 | |
| *** roman_g has joined #openstack-infra | 14:55 | |
| *** yamamoto has quit IRC | 14:56 | |
| *** pcaruana|elisa| has quit IRC | 14:56 | |
| *** yamamoto has joined #openstack-infra | 14:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Restore full OVH-GRA1 quota https://review.openstack.org/611553 | 14:56 |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New Repo - OpenStack-Helm Images https://review.openstack.org/611892 | 14:57 |
| *** pcaruana|elisa| has joined #openstack-infra | 14:57 | |
| aspiers | just found this pretty interesting comparison of communication protocols https://docs.google.com/spreadsheets/d/1-UlA4-tslROBDS9IqHalWVztqZo7uxlCeKPQ-8uoFOU/edit#gid=0 | 14:58 |
| aspiers | as I've said before, I really think we should look more closely into Matrix | 14:58 |
| *** quiquell is now known as quiquell|off | 14:58 | |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New Repo: OpenStack-Helm Docs https://review.openstack.org/611893 | 14:59 |
| mordred | aspiers: yah- it's on my todo list :) | 14:59 |
| aspiers | I've been using it for a while now | 14:59 |
| *** felipemonteiro has joined #openstack-infra | 15:00 | |
| cmurphy | curious about the "partial" support for TLS with IRC, is that because it's optional? | 15:01 |
| *** gfidente has joined #openstack-infra | 15:01 | |
| mordred | cmurphy: I think so | 15:02 |
| aspiers | They're putting in a lot of effort into making Riot more user-friendly too https://medium.com/@dharmaone/redesigning-matrix-riot-chat-c7bffd9eb841 | 15:02 |
| aspiers | https://github.com/vector-im/riot-web/projects/12 | 15:02 |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New Repo: OpenStack-Helm Docs https://review.openstack.org/611893 | 15:03 |
| mordred | aspiers: yah - the next step in the exploration is to stand up a home server and see what integration with our other stuff and scalability looks like | 15:04 |
| mordred | and things like "how do upgrades work - can we do them without user downtime" | 15:06 |
| aspiers | well since it's federated, maybe it's possible for one server to take over the load of another during maintenance - not sure | 15:07 |
| *** pcaruana|elisa| has quit IRC | 15:08 | |
| mordred | aspiers: yeah - from what I can tell on initial reading the protocol is federated across home servers, but I'm not sure about support for a home server to be a distributed cluster of home servers | 15:08 |
| *** pcaruana has joined #openstack-infra | 15:08 | |
| aspiers | yeah, I dunno | 15:08 |
| *** markmcd has quit IRC | 15:08 | |
| mordred | me either :) | 15:08 |
| aspiers | I think it's time for IRC to finally die though ;-) | 15:08 |
| clarkb | As the weirdo that uses an irc client to talk slack I cant help but feel IRC got a lot of stuff right :P | 15:10 |
| clarkb | re matrix, would we even need to run servers? there is the central set of servers similar to how freenode is hosted right? | 15:10 |
| mordred | clarkb: there are generic servers that we could use - but there isn't a 'main' central set of servers | 15:18 |
| mordred | clarkb: by running our own homeserver, we get namespacing - so channels can be #infra:opendev.org - instead of needing to be #openstack-infra:matrix.org | 15:19 |
| mordred | clarkb: you are right though - we can totally choose to not run a homeserver and instead just choose one of the existing homeservers as the place to create our channels | 15:19 |
| mordred | clarkb: oh - also, if we run our own homeserver, we should be able to integrate it with our SSO | 15:20 |
| mordred | but this is all stuff that an investigation should poke at and have real answers for | 15:21 |
| clarkb | mostly bring it up because after doing maintenancy type work for the servers we run this week I can't help but feel adding more services right now would be a mistake. There is a huge backlog of work and not enough hours to do it | 15:21 |
| *** mtreinish has quit IRC | 15:22 | |
| clarkb | one of the benefits of IRC is that there are existing hosts for the service that cater to the needs of open source projects and do the work for us which is cool | 15:22 |
| mordred | yup to both - which is also why investigation is still backburnered | 15:22 |
| clarkb | does matrix allow anonymous connections at all? or is it always authenticated? (I mean right now with freenode and our channels set to +r we are always auth'd too) | 15:23 |
| mordred | yeah - there is a guest mode | 15:25 |
| *** electrofelix has quit IRC | 15:25 | |
| mordred | and also federation, so someone can log in to matrix.org and join #infra:opendev.org | 15:25 |
| *** mriedem is now known as mriedem_afk | 15:27 | |
| clarkb | that table is pretty comprehensive. I like that there is a phoneless column | 15:27 |
| *** roman_g has quit IRC | 15:31 | |
| *** pcaruana has quit IRC | 15:31 | |
| AJaeger | config-core, the openstack-release jobs fails, and I think https://review.openstack.org/611886 fixes it - could you carefully review, please? | 15:32 |
| clarkb | AJaeger: yes I'll take a look shortly | 15:32 |
| AJaeger | thanks | 15:33 |
| *** dpawlik has quit IRC | 15:34 | |
| *** dpawlik has joined #openstack-infra | 15:34 | |
| *** dpawlik has quit IRC | 15:34 | |
| *** dpawlik has joined #openstack-infra | 15:35 | |
| clarkb | AJaeger: http://logs.openstack.org/33/33d839da8acb93a36a45186d158262f269e9bbd6/release/release-openstack-python/3a9339d/ara-report/file/1e91e730-c7e5-4ad8-a4b0-0eeed401758d/#line-1 confirms we don't run tox to generate the release. +2 lgtm | 15:35 |
| *** dpawlik has quit IRC | 15:36 | |
| *** dpawlik has joined #openstack-infra | 15:36 | |
| *** rpioso|afk is now known as rpioso | 15:36 | |
| clarkb | infra-root I'd like to finish up the etherpad.openstack.org and logstash.openstack.org upgrades today by deleting the old trusty servers. For logstash01.openstack.org four of our logstash workers cannot hit the geard server over ipv6 but do eventually fall back to ipv4 so are functional. Any objection to me deleting the old logstash.openstack.org now? | 15:38 |
| *** mtreinish has joined #openstack-infra | 15:38 | |
| AJaeger | clarkb, frickler, thanks | 15:38 |
| clarkb | infra-root also https://review.openstack.org/#/c/611727/ fixes an issue with geard not listneing on ipv6. Merging that will allow my to take the new server out of the emergency list | 15:39 |
| AJaeger | config-core, a couple of older reviews for your attention as you have time: https://review.openstack.org/608867 https://review.openstack.org/609339 https://review.openstack.org/609270 https://review.openstack.org/610428 https://review.openstack.org/611183 | 15:39 |
| *** mtaylor[m] has joined #openstack-infra | 15:40 | |
| *** dpawlik has quit IRC | 15:40 | |
| clarkb | AJaeger: I'll take a look. Also thank you for helping to drive these reviews! | 15:41 |
| AJaeger | clarkb: finish your updates first - nothing is critical - just a longer queue. | 15:41 |
| AJaeger | clarkb: and thanks for both! | 15:41 |
| clarkb | AJaeger: oh I'm mostly just waiting on reviews on that puppet change then I'll delete stuff in a bit | 15:42 |
| hogepodge | I was wondering if someone could help us understand why the build logs aren't being copied over in these failed jobs. https://review.openstack.org/#/c/611303/ | 15:42 |
| clarkb | hogepodge: http://logs.openstack.org/03/611303/18/check/loci-cinder/8cf81a0/ara-report/result/f27ffd8d-836a-4e2a-952a-6bd46c673e9e/ that says permissions issue | 15:43 |
| clarkb | hogepodge: ansible runs as the zuul user there so if not readable by zuul that may explain it? | 15:43 |
| hogepodge | Ok, so it's local permissions. That seems simple enough to fix | 15:44 |
| clarkb | hogepodge: ya I think the zuul user is trying to read the remote files to copy them and can't due to perms | 15:45 |
| clarkb | AJaeger: I won't review https://review.openstack.org/#/c/611183/ since I pushed it, but I do think we can ervert that. Maybe I'll give it a +1 to make that clear | 15:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove fetch-tox-output from release-openstack-python https://review.openstack.org/611886 | 15:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Add kayobe-config[-dev] to gerritbot for #openstack-kayobe https://review.openstack.org/609339 | 15:50 |
| *** dpawlik has joined #openstack-infra | 15:52 | |
| AJaeger | clarkb: yeah, hope another config-core can help reviewing 611183 | 15:53 |
| *** ccamacho has quit IRC | 15:56 | |
| *** dpawlik has quit IRC | 15:56 | |
| *** ginopc has quit IRC | 16:02 | |
| AJaeger | clarkb: want to empty the queue? I can feed you some more reviews if you want... | 16:02 |
| *** bobh has quit IRC | 16:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-midonet: Update the dashboard for recent changes https://review.openstack.org/610428 | 16:03 |
| AJaeger | mordred: want to review https://review.openstack.org/#/c/611183/ , please? | 16:05 |
| mordred | AJaeger: done! | 16:08 |
| *** beekneemech has quit IRC | 16:08 | |
| *** bnemec has joined #openstack-infra | 16:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add cells charm and interfaces https://review.openstack.org/608867 | 16:09 |
| AJaeger | thanks, mordred | 16:09 |
| AJaeger | mnaser: do you want to +2 https://review.openstack.org/#/c/608345/ ? | 16:09 |
| mnaser | clarkb: done thanks | 16:10 |
| AJaeger | mordred: do you want to abandon or continue on https://review.openstack.org/604820 - see my comments in there | 16:10 |
| AJaeger | mordred: what do you make out of this one - https://review.openstack.org/599028 - looks sane to me but I wanted somebody else to check that is more familiar first | 16:12 |
| *** njohnston has joined #openstack-infra | 16:15 | |
| *** njohnston has left #openstack-infra | 16:16 | |
| mordred | AJaeger: wow. I'm so glad to learn that that's weird | 16:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Octavia charm https://review.openstack.org/609270 | 16:17 |
| AJaeger | mordred: ;) | 16:18 |
| *** gfidente is now known as gfidente|afk | 16:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Reduce gate window floor to 10" https://review.openstack.org/611183 | 16:21 |
| *** openstackgerrit has quit IRC | 16:24 | |
| *** openstackgerrit has joined #openstack-infra | 16:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add os_manila role to openstack-ansible https://review.openstack.org/608345 | 16:29 |
| *** shardy has quit IRC | 16:30 | |
| *** sambetts is now known as sambetts|afk | 16:32 | |
| clarkb | AJaeger: can you review https://review.openstack.org/#/c/611200/3 with your docs hat on? | 16:33 |
| AJaeger | clarkb: sure | 16:34 |
| clarkb | mordred: can you review https://review.openstack.org/#/c/611727/ to fix the geard logstash thing? | 16:35 |
| *** bobh has joined #openstack-infra | 16:35 | |
| *** e0ne has joined #openstack-infra | 16:35 | |
| mordred | clarkb: should we updated geard to listen on :: instead of 127.0.0.1 by default? | 16:36 |
| clarkb | mordred: it listens on 0.0.0.0, but ya I think we should also update geard | 16:36 |
| mordred | clarkb: +2 on the patch - just seems like we should fix it | 16:36 |
| clarkb | I can take a look at writing that patch | 16:36 |
| *** e0ne has quit IRC | 16:37 | |
| clarkb | AJaeger: mordred mnaser I'm about ready to start looking at deleting old etherpad.o.o. Current one has two mysql backup files on it that match the size of the mysql backups on the old server. I also manually set up the off host backups when I booted the server | 16:37 |
| *** kopecmartin is now known as kopecmartin|off | 16:37 | |
| clarkb | as far as I can tell the srvice is working too. Anything else we should check on before deleting the old server? | 16:37 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config master: Add a workflow overview for adding a cloud https://review.openstack.org/611200 | 16:38 |
| mordred | clarkb: I can't think of anything | 16:38 |
| AJaeger | clarkb: fixed a nit ^ | 16:38 |
| AJaeger | clarkb: regarding old etherpad.o.o - no but I'm not an expert | 16:39 |
| AJaeger | mordred: could you review - https://review.openstack.org/599028 - looks sane to me but I wanted somebody else to check that is more familiar first, please | 16:39 |
| clarkb | ok I'll work on geard lsiten address and once that is done delete old etherpad if no one comes up with reason not to | 16:40 |
| mordred | AJaeger: lgtm | 16:40 |
| AJaeger | mordred: wnat to +2A this docs change for system-config, please? https://review.openstack.org/#/c/611200 | 16:40 |
| AJaeger | thanks, mordred | 16:40 |
| mordred | done! | 16:41 |
| AJaeger | thanks | 16:41 |
| mordred | infra-root: heads up - a new openstacksdk release was cut. it shouldn't impact nodepool - but if anybody sees any weird errors in nodepool land, let me know | 16:42 |
| AJaeger | pabelanger: did you see the comments on https://review.openstack.org/#/c/591822/ ? Could you address them, please? | 16:42 |
| AJaeger | now I'm scared - "it shouldn't impact..." ;) | 16:43 |
| *** bnemec is now known as beekneemech | 16:43 | |
| *** dtantsur is now known as dtantsur|afk | 16:44 | |
| clarkb | mordred: pabelanger it turns out we reviewd the change that added listen-address and thought that maintaining None as the value would make ipv6 work https://review.openstack.org/#/c/561705/1..2/gear/cmd/geard.py problem is the first thing returned in the list is AF_INET not AF_INET4 | 16:45 |
| clarkb | I think we can set the default to :: instead of 0.0.0.0 and that will work for most users. Gear already requires linux due to its event polling mechanism | 16:45 |
| *** felipemonteiro has quit IRC | 16:53 | |
| *** derekh has quit IRC | 17:01 | |
| *** carl_cai has quit IRC | 17:02 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/gear master: Prefer ipv6 for listen addrs if available https://review.openstack.org/611920 | 17:04 |
| clarkb | mordred: ^ I think that will do it in a safe way | 17:04 |
| clarkb | pabelanger: ^ that probably interests you too | 17:05 |
| clarkb | ok I'm ready to delete old etherpad server. I think I have decided new one is working. mordred want to double check me that 8e3ab3b5-b264-494a-abfc-026ad29744da is the right uuid to delete? | 17:07 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: ensure-sphinx: do not attempt a gettext install if exists https://review.openstack.org/599028 | 17:11 |
| *** gyee has joined #openstack-infra | 17:12 | |
| *** trown is now known as trown|lunch | 17:14 | |
| *** mtaylor[m] is now known as mordred[m] | 17:16 | |
| *** tosky has quit IRC | 17:18 | |
| clarkb | #status log Old etherpad.openstack.org server (8e3ab3b5-b264-494a-abfc-026ad29744da) deleted as it has been replaced by a new etherpad01.openstack.org server running Xenial. | 17:18 |
| openstackstatus | clarkb: finished logging | 17:18 |
| *** jamesmcarthur has quit IRC | 17:19 | |
| hogepodge | thanks clarkb | 17:20 |
| *** yamamoto has quit IRC | 17:20 | |
| *** yamamoto has joined #openstack-infra | 17:21 | |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| *** florianf has quit IRC | 17:26 | |
| *** yamamoto has quit IRC | 17:27 | |
| *** yamamoto has joined #openstack-infra | 17:32 | |
| *** bobh has quit IRC | 17:34 | |
| *** yamamoto has quit IRC | 17:37 | |
| clarkb | hogepodge: ara is often useful when ansible doesn't do what you expect | 17:39 |
| clarkb | corvus: if I want to restart zuul scheduler later today, can I do that safely without needign to restart the other services? | 17:41 |
| *** adriancz has quit IRC | 17:42 | |
| *** apetrich has quit IRC | 17:51 | |
| *** e0ne has joined #openstack-infra | 17:53 | |
| *** e0ne has quit IRC | 18:00 | |
| *** apetrich has joined #openstack-infra | 18:04 | |
| *** e0ne has joined #openstack-infra | 18:08 | |
| *** AJaeger_ has joined #openstack-infra | 18:11 | |
| AJaeger_ | config-core, a couple more reviews if anybody has time: https://review.openstack.org/611013 https://review.openstack.org/611431 https://review.openstack.org/611622 https://review.openstack.org/611631 https://review.openstack.org/611630 https://review.openstack.org/610175 https://review.openstack.org/611876 - that empties the queue... | 18:13 |
| clarkb | AJaeger_: the first one's depends on isn't merged yet fwiw | 18:14 |
| *** AJaeger has quit IRC | 18:14 | |
| *** yamamoto has joined #openstack-infra | 18:17 | |
| clarkb | mordred: AJaeger_ I think http://logs.openstack.org/31/611631/2/check/project-config-nodepool/cd08579/job-output.txt.gz#_2018-10-19_07_01_20_503512 shows that nodepool is not installed as a sibling | 18:18 |
| mordred | clarkb: oh how funny | 18:19 |
| AJaeger_ | clarkb: ah - thanks | 18:21 |
| clarkb | similar situation wiht https://review.openstack.org/#/c/611630/1 | 18:21 |
| AJaeger_ | yeah - wonder how it worked? | 18:22 |
| mordred | yeah - that stack is operating under a less than ideal assumption - I'll need to rework it | 18:22 |
| AJaeger_ | ah, installed from pypi ;( | 18:22 |
| clarkb | AJaeger_: it works because nodepool and zuul are on pypi | 18:22 |
| clarkb | yup | 18:23 |
| clarkb | which might be good enough? but the assertion in the commit message isn't true so I Figured worth calling out | 18:23 |
| AJaeger_ | clarkb: thanks for catching that | 18:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update ansible library location to envsitepackagesdir https://review.openstack.org/611630 | 18:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop installing nodepool in all the test envs https://review.openstack.org/611631 | 18:24 |
| mordred | clarkb: that should fix the siblings part | 18:24 |
| mordred | clarkb: otoh - maybe we don't want this to work with siblings? | 18:25 |
| clarkb | mordred: ya I do wonder about that, infra runs both zuul and nodepool from source and we make use of functionality that exists before they release | 18:25 |
| clarkb | I think for this reason we do want them installed from source | 18:25 |
| *** AJaeger has joined #openstack-infra | 18:25 | |
| clarkb | and there is value in that for zuul and nodepool beacuse you get a massive beta test isntallation essentially | 18:26 |
| AJaeger | why is https://review.openstack.org/#/c/611200/ failing now? New library release? http://logs.openstack.org/00/611200/4/check/tox-linters/8d93700/job-output.txt.gz#_2018-10-19_18_14_28_960982 | 18:26 |
| *** bobh has joined #openstack-infra | 18:26 | |
| clarkb | wow that failed in setuptools, | 18:27 |
| AJaeger | mordred: do we really need build_sphinx in https://review.openstack.org/#/c/611630/2/setup.cfg ? | 18:27 |
| *** AJaeger_ has quit IRC | 18:28 | |
| clarkb | AJaeger: I do not know why that would have happened | 18:29 |
| AJaeger | clarkb: yeah, strange - I rechecked, let's see whether it's reproduceable | 18:30 |
| clarkb | if it reproduces next step is probably trying to reproduce locally | 18:31 |
| AJaeger | yep | 18:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove airship-deckhand jobs, as they are all in project https://review.openstack.org/611431 | 18:32 |
| openstackgerrit | Merged openstack-infra/project-config master: New Airship project - Tempest Plugin https://review.openstack.org/610175 | 18:32 |
| mordred | AJaeger: probalby not - I totally cargo-culted it in from zuul-jobs :) | 18:33 |
| AJaeger | let's remove it then, please | 18:33 |
| *** jamesmcarthur has quit IRC | 18:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update ansible library location to envsitepackagesdir https://review.openstack.org/611630 | 18:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop installing nodepool in all the test envs https://review.openstack.org/611631 | 18:34 |
| mordred | AJaeger: done! | 18:34 |
| AJaeger | thanks - so let's wait for test results... | 18:35 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Avoid ignore_errors on Set ephemeral device by label https://review.openstack.org/611876 | 18:36 |
| *** mriedem_afk is now known as mriedem | 18:41 | |
| *** roman_g has joined #openstack-infra | 18:45 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update ANSIBLE_LIBRARY to use envsitepackagesdir https://review.openstack.org/611622 | 18:45 |
| clarkb | mordred: thoughts on me going ahead and approving https://review.openstack.org/#/c/611727/1 ? I am getting the sense that its friday and I shouldn't expect further motion on that otherwise | 18:46 |
| clarkb | mordred: but would like to tick off logstash rebuild as done today | 18:46 |
| *** trown|lunch is now known as trown | 18:46 | |
| *** AJaeger_ has joined #openstack-infra | 18:46 | |
| mordred | clarkb: I say go for it | 18:47 |
| mordred | clarkb: it's a thing you're actively working on - I think the only risk would be it landing and breaking silently | 18:47 |
| clarkb | ok I'll approve it and watch it, thanks | 18:48 |
| *** AJaeger has quit IRC | 18:49 | |
| *** roman_g has quit IRC | 18:49 | |
| *** AJaeger_ is now known as AJaeger | 18:53 | |
| *** gfidente|afk has quit IRC | 18:55 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: WIP https://review.openstack.org/611936 | 19:01 |
| AJaeger | config-core, one more review https://review.openstack.org/609010 | 19:04 |
| clarkb | AJaeger: done | 19:05 |
| AJaeger | thanks! | 19:07 |
| *** armstrong has quit IRC | 19:12 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Cleanup down ports https://review.openstack.org/609829 | 19:14 |
| *** bobh has quit IRC | 19:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Cleanup down ports https://review.openstack.org/609829 | 19:19 |
| *** bobh has joined #openstack-infra | 19:19 | |
| *** bobh has quit IRC | 19:23 | |
| *** bobh has joined #openstack-infra | 19:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack/flare project. https://review.openstack.org/609010 | 19:24 |
| openstackgerrit | Merged openstack-infra/puppet-log_processor master: Force geard to listen on :: https://review.openstack.org/611727 | 19:32 |
| clarkb | as soon as ^ is on bridge.o.o I'll remove logstash01 from the emergency list, double check all is happy then delete the old logstash.openstack.org server | 19:32 |
| *** e0ne has quit IRC | 19:34 | |
| *** e0ne has joined #openstack-infra | 19:34 | |
| *** sshnaidm|off has quit IRC | 19:34 | |
| *** e0ne has quit IRC | 19:36 | |
| AJaeger | mordred, clarkb, seems siblings is now working with nodepool, see http://logs.openstack.org/31/611631/4/check/project-config-nodepool/eeca43b/job-output.txt.gz#_2018-10-19_19_43_09_826704 | 19:45 |
| clarkb | moving zuul dns conversation from #zuul to here | 19:46 |
| AJaeger | clarkb, want to remove your -1 on https://review.openstack.org/611631 - or want to wait for all tests to finish and then change to +2 (or leave as is)? | 19:46 |
| clarkb | AJaeger: I'll take a look once I figure out this dns issue | 19:46 |
| clarkb | it appears that adns1.openstack.org does not ping | 19:46 |
| AJaeger | clarkb: sure | 19:46 |
| AJaeger | argh ;( | 19:46 |
| clarkb | ns1 and ns2 are both up and reachable but resolving zuul-ci.org against them fails | 19:47 |
| *** sshnaidm|off has joined #openstack-infra | 19:50 | |
| *** dave-mccowan has quit IRC | 19:57 | |
| clarkb | adns1.openstack.org is up and running according to the console | 19:59 |
| clarkb | Oct 19 16:01:07 ns2 nsd[22004]: [2018-10-19 16:01:07.925] nsd[22004]: error: xfrd: zone zuulci.org has expired | 19:59 |
| clarkb | is what the nsd on ns2 says | 19:59 |
| clarkb | I cannot reach adns1 via ping or ssh | 19:59 |
| clarkb | I think my next step is to attempt a reboot | 20:00 |
| pabelanger | clarkb: I too cannot ssh or ping | 20:00 |
| pabelanger | +1 for reboot | 20:00 |
| clarkb | corvus: ^ I'll give you a couple minutes before doing that to point out you are doing somethign with it or that this is otherwise expected | 20:00 |
| openstackgerrit | Zane Bitter proposed openstack-infra/openstack-zuul-jobs master: Example per-release Python3 unit test templates https://review.openstack.org/611947 | 20:01 |
| pabelanger | clarkb: in this instances, if we had a adns2.o.o setup, would that imply some sort of HA? | 20:01 |
| pabelanger | not up to speed on the new DNS settings | 20:01 |
| clarkb | pabelanger: aiui the ha is ns1 and ns2, but they seem to not serve the transferred zones forever | 20:02 |
| mordred | clarkb: I believe corvus is out todayu | 20:02 |
| pabelanger | clarkb: right | 20:02 |
| clarkb | adns1 feeds the zone info into ns1 and ns2 | 20:02 |
| pabelanger | okay, that is how I also understand it | 20:02 |
| clarkb | nothing else talks to adns1 it should be able to go away for an extended period of time | 20:02 |
| clarkb | mordred: you good with reboot? | 20:02 |
| mordred | clarkb: yup | 20:03 |
| mordred | clarkb: I can't come up with anything else better | 20:03 |
| clarkb | ok doing that now | 20:03 |
| *** sshnaidm|off has quit IRC | 20:04 | |
| clarkb | it sshes now and I'm in. syslog doesn't show anyhting catastrophic from its perspecitive | 20:05 |
| *** diablo_rojo has joined #openstack-infra | 20:06 | |
| clarkb | ns2 shows that it got updated from adns1 | 20:06 |
| clarkb | ct 19 20:04:24 ns2 nsd[22004]: [2018-10-19 20:04:24.490] nsd[22004]: info: zone zuul-ci.org serial 1522167811 is updated to 1522167816. | 20:06 |
| clarkb | that did it | 20:06 |
| pabelanger | yay | 20:07 |
| clarkb | reading the adns1 logs we appear to do hourly updates from adns1 to ns* | 20:07 |
| clarkb | the zone expiry is 10 days | 20:11 |
| clarkb | so adns1 has been afk for 10 days? | 20:11 |
| clarkb | considering we should be good for another 10 days from now I am not going to try an engineer a solution this second | 20:12 |
| clarkb | but we should probably consider monitoring of some sort for adns1 | 20:12 |
| clarkb | ideally we would check that we can resolve against adns1 but I don't think the firewall rules allow that currently | 20:14 |
| clarkb | maybe we can open it up a bit for some well known sources? | 20:14 |
| clarkb | AJaeger: haven't forgotten about you, will look at the testing stuff shortly | 20:15 |
| *** kgiusti has left #openstack-infra | 20:21 | |
| *** sshnaidm|off has joined #openstack-infra | 20:22 | |
| clarkb | AJaeger: I +2'd both changes | 20:25 |
| clarkb | double checked the test logs too | 20:25 |
| *** e0ne has joined #openstack-infra | 20:28 | |
| clarkb | logstash01 is puppeting again with the :: listen address fix for geard. I am going to delete the old logstash.o.o now | 20:28 |
| clarkb | #status log Old logstash.openstack.org server (08c356e5-d225-4163-9dce-c57b4d68eb55) running trusty has been deleted in favor of new logstash01.openstack.org server running xenial | 20:30 |
| openstackstatus | clarkb: finished logging | 20:30 |
| AJaeger | clarkb, indeed, the logs look fine. +2A for both of mordred's changes | 20:31 |
| * AJaeger waves good night and wishes everybody a great weekend | 20:31 | |
| clarkb | AJaeger: good night! and thanks for all the help today | 20:32 |
| *** e0ne has quit IRC | 20:33 | |
| clarkb | infra-root: I think I may investigate what would be required to finally switch over zk to the zk cluster next week | 20:34 |
| clarkb | then we can delete the old nodepool.o.o server | 20:34 |
| clarkb | this will liekly require a full zuul and nodepool downtime | 20:34 |
| clarkb | but its early in openstack release cycle so now is a good time to do that | 20:34 |
| pabelanger | that would be cool to make the change | 20:34 |
| *** owalsh has quit IRC | 20:36 | |
| *** dklyle has joined #openstack-infra | 20:37 | |
| clarkb | pabelanger: I'm thinking that maybe if we do it friday afternoon we just cut over to new zk and let alien cleanup happen in the clouds and not try to preserve the database | 20:39 |
| AJaeger | oh, https://review.openstack.org/611200 fails again in setup tools ;/ I won't investigate this... | 20:39 |
| AJaeger | thanks, clarkb | 20:39 |
| clarkb | AJaeger: interesting. I'll look at reproducing that locally now | 20:39 |
| pabelanger | clarkb: we should be able to do nodepool first, then zuul | 20:41 |
| clarkb | pabelanger: they need to talk to the same db for it all to work. I think we turn them all off, update their config to point at new zk cluster, then turn then on | 20:41 |
| clarkb | let nodepool clean up the aliens and it should be fine | 20:42 |
| clarkb | AJaeger: it does reproduce fwiw | 20:42 |
| clarkb | testinfra 1.17.0 seems to be the problem | 20:43 |
| *** e0ne has joined #openstack-infra | 20:44 | |
| clarkb | working to confirm that and if that is the issue I'll see if I can fix testinfra or we can pin it | 20:45 |
| *** e0ne has quit IRC | 20:48 | |
| clarkb | installing from source works fine in testinfra and with pip install --no-binary :all: I think this means their wheel is broken | 20:50 |
| *** dklyle has quit IRC | 20:50 | |
| *** munimeha1 has quit IRC | 20:51 | |
| *** beekneemech has quit IRC | 20:53 | |
| clarkb | https://github.com/philpep/testinfra/issues/380 issue submitted. I'm going to push a pin for us in a sec | 20:57 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Migration to PHP 7.x https://review.openstack.org/611936 | 21:00 |
| *** trown is now known as trown|outtypewww | 21:00 | |
| *** bobh has quit IRC | 21:01 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Don't install testinfra 1.17.0 https://review.openstack.org/611960 | 21:01 |
| *** smarcet has joined #openstack-infra | 21:01 | |
| *** bobh has joined #openstack-infra | 21:01 | |
| *** dave-mccowan has joined #openstack-infra | 21:05 | |
| smarcet | fungi: afternoon, hope that everything is ok on your side, i am currently working on a patch for openstack id so it could be php 7.x compatible, this is due bc php 5.6 will be at EOL 31 Dec 2018 and after that will not receive any support neither security fixes | 21:05 |
| smarcet | fungi: wonder if infra its supporting ubuntu 18.04 LTS flavor | 21:05 |
| *** bobh has quit IRC | 21:05 | |
| clarkb | smarcet: we are but not with puppet, you'll have to containerize or use ansible | 21:06 |
| smarcet | clarkb: actually container would be better i think | 21:07 |
| clarkb | smarcet: https://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html is the spec detailing this | 21:07 |
| clarkb | smarcet: we haven't gotten to the point where we are deploying anything with containers yet so that aspect of it is still very new, but the ansible stuff is up and running | 21:07 |
| smarcet | i could write a docker, but there is any orchestration engine like kubernetes ? | 21:07 |
| smarcet | oh ok | 21:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Update ansible library location to envsitepackagesdir https://review.openstack.org/611630 | 21:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop installing nodepool in all the test envs https://review.openstack.org/611631 | 21:08 |
| clarkb | smarcet: currently no, see the spec for details. We wanted to be able to take baby steps and replace what we have with something similar ish | 21:08 |
| clarkb | smarcet: once we get that going the next step would be to consider something like k8s | 21:08 |
| clarkb | smarcet: xenial also supports php 7 so we could do an intermediate step to xenial with puppet if that helps | 21:09 |
| smarcet | clarkb: ok cool i will read the spec, and we could discuss on berlin if u want 2 | 21:09 |
| clarkb | basically get to xenail + php7 to get off trusty and old php. Then migrate to containers as a followup. Will depend on how much work the full jump is compared to the smaller steps | 21:09 |
| clarkb | smarcet: probably a good idea to talk in berlin if we haven't started moving things already | 21:09 |
| smarcet | clarkb: sure, i will try then xenial | 21:10 |
| smarcet | :) | 21:10 |
| *** owalsh has joined #openstack-infra | 21:10 | |
| smarcet | clarkb: i am also trying to move www site to php 7.x and for that already wrote a docker, idea its to move also that to kubernetes :) | 21:11 |
| *** dave-mccowan has quit IRC | 21:13 | |
| *** kjackal has quit IRC | 21:16 | |
| *** kjackal has joined #openstack-infra | 21:16 | |
| *** smarcet has left #openstack-infra | 21:17 | |
| *** ssbarnea has quit IRC | 21:21 | |
| *** ssbarnea|bkp2 has joined #openstack-infra | 21:22 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/gear master: Prefer ipv6 for listen addrs if available https://review.openstack.org/611920 | 21:33 |
| *** rlandy has quit IRC | 21:40 | |
| *** mriedem has quit IRC | 21:40 | |
| *** yamamoto has quit IRC | 21:40 | |
| *** rh-jelabarre has quit IRC | 21:41 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Migration to PHP 7.x https://review.openstack.org/611936 | 21:48 |
| *** boden has quit IRC | 21:52 | |
| *** diablo_rojo has quit IRC | 21:58 | |
| *** dklyle has joined #openstack-infra | 22:03 | |
| clarkb | https://review.openstack.org/#/c/611960/ has passed testing and should fix the docs builds for infra | 22:04 |
| clarkb | er linters builds | 22:04 |
| *** yamamoto has joined #openstack-infra | 22:10 | |
| *** xek has quit IRC | 22:11 | |
| *** priteau has quit IRC | 22:12 | |
| *** tpsilva has quit IRC | 22:13 | |
| *** diablo_rojo has joined #openstack-infra | 22:20 | |
| *** ssbarnea has joined #openstack-infra | 22:21 | |
| *** pbourke has quit IRC | 22:36 | |
| *** pbourke has joined #openstack-infra | 22:37 | |
| *** harlowja has quit IRC | 22:46 | |
| *** jamesmcarthur has joined #openstack-infra | 23:10 | |
| *** sthussey has quit IRC | 23:11 | |
| *** jamesmcarthur has quit IRC | 23:15 | |
| *** rpioso is now known as rpioso|afk | 23:15 | |
| *** diablo_rojo has quit IRC | 23:21 | |
| *** bnemec has joined #openstack-infra | 23:25 | |
| *** harlowja has joined #openstack-infra | 23:25 | |
| *** ssbarnea has quit IRC | 23:35 | |
| *** bobh has joined #openstack-infra | 23:37 | |
| *** yamamoto has quit IRC | 23:42 | |
| *** felipemonteiro has joined #openstack-infra | 23:44 | |
| *** gyee has quit IRC | 23:50 | |
| *** bobh has quit IRC | 23:50 | |
| *** dklyle has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!