| *** spatel has joined #openstack-ansible | 00:09 | |
| *** macz_ has quit IRC | 00:09 | |
| *** spatel has quit IRC | 00:13 | |
| *** macz_ has joined #openstack-ansible | 00:20 | |
| djhankb | jrosser: thanks for the heads-up. I'll dig through the role to see if there's something I can add there, but it sounds like OVS on the controller is more trouble than its worth. | 00:28 |
|---|---|---|
| *** macz_ has quit IRC | 00:33 | |
| *** errr has joined #openstack-ansible | 00:38 | |
| *** errr has quit IRC | 00:38 | |
| *** stingrayza has joined #openstack-ansible | 00:39 | |
| *** errr has joined #openstack-ansible | 00:39 | |
| *** errr has quit IRC | 00:40 | |
| *** errr has joined #openstack-ansible | 00:41 | |
| *** errr has quit IRC | 00:42 | |
| *** errr has joined #openstack-ansible | 00:43 | |
| *** bgmccollum has quit IRC | 00:44 | |
| *** errr has quit IRC | 00:44 | |
| *** errr has joined #openstack-ansible | 00:46 | |
| *** errr has quit IRC | 00:47 | |
| *** DanyC has quit IRC | 00:57 | |
| *** DanyC has joined #openstack-ansible | 00:58 | |
| *** errr has joined #openstack-ansible | 01:01 | |
| *** DanyC has quit IRC | 01:02 | |
| *** errr has quit IRC | 01:02 | |
| *** errr has joined #openstack-ansible | 01:04 | |
| *** errr has joined #openstack-ansible | 01:06 | |
| *** errr has quit IRC | 01:07 | |
| *** nicolasbock has quit IRC | 01:15 | |
| *** errr has joined #openstack-ansible | 01:19 | |
| *** errr has joined #openstack-ansible | 01:21 | |
| *** errr has joined #openstack-ansible | 01:22 | |
| *** errr has quit IRC | 01:23 | |
| *** errr has joined #openstack-ansible | 01:24 | |
| *** errr has quit IRC | 01:25 | |
| *** errr has joined #openstack-ansible | 01:27 | |
| *** errr has quit IRC | 01:28 | |
| *** errr has joined #openstack-ansible | 01:29 | |
| *** errr has quit IRC | 01:30 | |
| *** errr has joined #openstack-ansible | 01:31 | |
| *** errr has quit IRC | 01:33 | |
| *** errr has joined #openstack-ansible | 01:34 | |
| *** errr has quit IRC | 01:35 | |
| *** errr has joined #openstack-ansible | 01:39 | |
| *** errr has quit IRC | 01:40 | |
| *** errr has joined #openstack-ansible | 01:41 | |
| *** errr has joined #openstack-ansible | 01:42 | |
| *** errr has quit IRC | 01:43 | |
| *** errr has joined #openstack-ansible | 01:45 | |
| *** errr has quit IRC | 01:45 | |
| *** errr has joined #openstack-ansible | 01:47 | |
| *** errr has joined #openstack-ansible | 01:48 | |
| *** errr has quit IRC | 01:49 | |
| *** errr has joined #openstack-ansible | 01:50 | |
| *** errr has quit IRC | 01:51 | |
| *** errr has joined #openstack-ansible | 01:53 | |
| *** errr has quit IRC | 01:53 | |
| *** errr has joined #openstack-ansible | 01:55 | |
| *** errr has quit IRC | 01:56 | |
| *** errr has joined #openstack-ansible | 01:57 | |
| *** errr has quit IRC | 01:58 | |
| *** errr has joined #openstack-ansible | 02:00 | |
| *** errr has quit IRC | 02:00 | |
| *** errr has joined #openstack-ansible | 02:02 | |
| *** errr has quit IRC | 02:03 | |
| *** errr has joined #openstack-ansible | 02:04 | |
| *** henryb has joined #openstack-ansible | 02:05 | |
| *** errr has quit IRC | 02:05 | |
| *** errr has joined #openstack-ansible | 02:07 | |
| *** errr has joined #openstack-ansible | 02:08 | |
| *** errr has joined #openstack-ansible | 02:10 | |
| *** errr has joined #openstack-ansible | 02:12 | |
| *** errr has quit IRC | 02:13 | |
| *** errr has joined #openstack-ansible | 02:14 | |
| *** errr has quit IRC | 02:15 | |
| *** errr has joined #openstack-ansible | 02:16 | |
| *** errr has quit IRC | 02:18 | |
| *** errr has joined #openstack-ansible | 02:30 | |
| *** errr has quit IRC | 02:30 | |
| *** mmercer has quit IRC | 02:34 | |
| *** errr has joined #openstack-ansible | 02:36 | |
| *** errr has joined #openstack-ansible | 02:38 | |
| *** errr has quit IRC | 02:39 | |
| *** errr has joined #openstack-ansible | 02:40 | |
| *** errr has quit IRC | 02:41 | |
| *** errr has joined #openstack-ansible | 02:45 | |
| *** errr has joined #openstack-ansible | 02:47 | |
| *** errr has joined #openstack-ansible | 02:48 | |
| *** errr has quit IRC | 02:49 | |
| *** errr has joined #openstack-ansible | 02:51 | |
| *** errr has quit IRC | 02:52 | |
| *** errr has joined #openstack-ansible | 02:53 | |
| *** errr has quit IRC | 02:54 | |
| *** errr has joined #openstack-ansible | 02:55 | |
| *** errr has joined #openstack-ansible | 02:58 | |
| *** errr has quit IRC | 02:59 | |
| *** errr has joined #openstack-ansible | 03:00 | |
| *** errr has quit IRC | 03:01 | |
| *** henryb has quit IRC | 03:02 | |
| *** henryb has joined #openstack-ansible | 03:02 | |
| *** errr has joined #openstack-ansible | 03:02 | |
| *** errr has quit IRC | 03:03 | |
| *** errr has joined #openstack-ansible | 03:05 | |
| *** errr has joined #openstack-ansible | 03:06 | |
| *** errr has quit IRC | 03:06 | |
| *** errr has joined #openstack-ansible | 03:08 | |
| *** errr has joined #openstack-ansible | 03:10 | |
| *** errr has joined #openstack-ansible | 03:15 | |
| *** errr has quit IRC | 03:16 | |
| *** errr has joined #openstack-ansible | 03:16 | |
| *** errr has quit IRC | 03:17 | |
| *** errr has joined #openstack-ansible | 03:19 | |
| *** errr has quit IRC | 03:19 | |
| *** gyee has quit IRC | 03:25 | |
| *** gyee has joined #openstack-ansible | 03:27 | |
| *** djhankb has quit IRC | 03:29 | |
| *** henryb is now known as djhankb | 03:30 | |
| *** errr has joined #openstack-ansible | 03:48 | |
| *** timburke has quit IRC | 03:50 | |
| *** errr has joined #openstack-ansible | 03:50 | |
| *** errr has quit IRC | 03:51 | |
| *** johanssone has quit IRC | 03:52 | |
| *** ebbex has quit IRC | 03:52 | |
| *** redkrieg has quit IRC | 03:52 | |
| *** johanssone has joined #openstack-ansible | 03:53 | |
| *** timburke has joined #openstack-ansible | 03:53 | |
| *** redkrieg has joined #openstack-ansible | 03:54 | |
| *** ebbex has joined #openstack-ansible | 03:54 | |
| *** johanssone has quit IRC | 04:00 | |
| *** johanssone has joined #openstack-ansible | 04:01 | |
| *** errr has joined #openstack-ansible | 04:03 | |
| *** errr has quit IRC | 04:03 | |
| *** errr has joined #openstack-ansible | 04:05 | |
| *** errr has quit IRC | 04:06 | |
| *** errr has joined #openstack-ansible | 04:09 | |
| *** errr has joined #openstack-ansible | 04:10 | |
| *** errr has joined #openstack-ansible | 04:12 | |
| *** errr has joined #openstack-ansible | 04:13 | |
| *** errr has quit IRC | 04:14 | |
| *** errr has joined #openstack-ansible | 04:15 | |
| *** errr has quit IRC | 04:16 | |
| *** errr has joined #openstack-ansible | 04:19 | |
| *** errr has quit IRC | 04:20 | |
| *** errr has joined #openstack-ansible | 04:21 | |
| *** errr has quit IRC | 04:22 | |
| *** udesale has joined #openstack-ansible | 04:23 | |
| *** errr has joined #openstack-ansible | 04:23 | |
| *** errr has joined #openstack-ansible | 04:25 | |
| *** errr has quit IRC | 04:26 | |
| *** errr has joined #openstack-ansible | 04:27 | |
| *** errr has quit IRC | 04:28 | |
| *** errr has joined #openstack-ansible | 04:29 | |
| *** errr has quit IRC | 04:29 | |
| *** errr has joined #openstack-ansible | 04:30 | |
| *** errr has joined #openstack-ansible | 04:48 | |
| *** errr has joined #openstack-ansible | 04:49 | |
| *** errr has joined #openstack-ansible | 04:50 | |
| *** errr has quit IRC | 04:51 | |
| *** errr has joined #openstack-ansible | 04:52 | |
| *** errr has quit IRC | 04:53 | |
| *** errr has joined #openstack-ansible | 04:54 | |
| *** errr has quit IRC | 04:55 | |
| *** errr has joined #openstack-ansible | 04:57 | |
| *** errr has quit IRC | 04:58 | |
| *** errr has joined #openstack-ansible | 05:01 | |
| *** errr has quit IRC | 05:02 | |
| *** errr has joined #openstack-ansible | 05:03 | |
| *** errr has quit IRC | 05:04 | |
| *** errr has joined #openstack-ansible | 05:06 | |
| *** errr has joined #openstack-ansible | 05:08 | |
| *** errr has joined #openstack-ansible | 05:10 | |
| *** errr has quit IRC | 05:12 | |
| *** errr has joined #openstack-ansible | 05:13 | |
| *** errr has quit IRC | 05:14 | |
| *** errr has joined #openstack-ansible | 05:15 | |
| *** errr has quit IRC | 05:16 | |
| *** errr has joined #openstack-ansible | 05:17 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-ansible | 05:35 | |
| *** errr has joined #openstack-ansible | 05:36 | |
| *** errr has quit IRC | 05:37 | |
| *** errr has joined #openstack-ansible | 05:38 | |
| *** errr has quit IRC | 05:39 | |
| *** errr has joined #openstack-ansible | 05:41 | |
| *** errr has joined #openstack-ansible | 05:42 | |
| *** errr has joined #openstack-ansible | 05:43 | |
| *** errr has joined #openstack-ansible | 05:44 | |
| *** errr has quit IRC | 05:44 | |
| *** errr has joined #openstack-ansible | 05:45 | |
| *** errr has joined #openstack-ansible | 05:47 | |
| *** errr has quit IRC | 05:48 | |
| *** gyee has quit IRC | 05:49 | |
| *** errr has joined #openstack-ansible | 05:49 | |
| *** errr has quit IRC | 05:50 | |
| *** errr has joined #openstack-ansible | 05:52 | |
| *** errr has quit IRC | 05:52 | |
| *** errr has joined #openstack-ansible | 05:54 | |
| *** errr has joined #openstack-ansible | 05:59 | |
| *** errr has quit IRC | 06:00 | |
| *** errr has joined #openstack-ansible | 06:02 | |
| *** errr has quit IRC | 06:02 | |
| *** raukadah is now known as chandankumar | 06:03 | |
| *** elenalindq has joined #openstack-ansible | 06:06 | |
| *** errr has joined #openstack-ansible | 06:07 | |
| *** errr has quit IRC | 06:07 | |
| *** errr has joined #openstack-ansible | 06:09 | |
| *** errr has quit IRC | 06:10 | |
| *** errr has joined #openstack-ansible | 06:24 | |
| *** errr has quit IRC | 06:25 | |
| *** errr has joined #openstack-ansible | 06:26 | |
| *** errr has quit IRC | 06:27 | |
| *** errr has joined #openstack-ansible | 06:29 | |
| *** errr has joined #openstack-ansible | 06:30 | |
| *** errr has quit IRC | 06:30 | |
| *** errr has joined #openstack-ansible | 06:32 | |
| *** errr has quit IRC | 06:33 | |
| *** errr has joined #openstack-ansible | 06:34 | |
| *** errr has quit IRC | 06:35 | |
| *** errr has joined #openstack-ansible | 06:37 | |
| *** errr has joined #openstack-ansible | 06:39 | |
| *** errr has quit IRC | 06:40 | |
| *** errr has joined #openstack-ansible | 06:42 | |
| *** errr has quit IRC | 06:42 | |
| *** errr has joined #openstack-ansible | 06:43 | |
| *** errr has quit IRC | 06:43 | |
| *** errr has joined #openstack-ansible | 06:45 | |
| *** errr has joined #openstack-ansible | 06:47 | |
| *** errr has quit IRC | 06:48 | |
| *** errr has joined #openstack-ansible | 06:50 | |
| *** errr has quit IRC | 06:50 | |
| *** errr has joined #openstack-ansible | 06:52 | |
| *** errr has quit IRC | 06:53 | |
| *** errr has joined #openstack-ansible | 06:54 | |
| *** errr has joined #openstack-ansible | 06:57 | |
| *** errr has joined #openstack-ansible | 06:59 | |
| *** pcaruana has joined #openstack-ansible | 07:11 | |
| jrosser | mnaser: gshippey guilhermesp we need this stack of patches to remove testing on debian stretch on master https://review.opendev.org/#/q/topic:buster_functional+(status:open+OR+status:merged) | 07:14 |
| *** DanyC has joined #openstack-ansible | 07:18 | |
| *** cshen has joined #openstack-ansible | 07:22 | |
| *** errr has joined #openstack-ansible | 07:24 | |
| *** errr has quit IRC | 07:25 | |
| *** errr has joined #openstack-ansible | 07:27 | |
| *** DanyC has quit IRC | 07:27 | |
| *** errr has quit IRC | 07:27 | |
| *** errr has joined #openstack-ansible | 07:31 | |
| *** errr has quit IRC | 07:32 | |
| *** errr has joined #openstack-ansible | 07:34 | |
| *** errr has quit IRC | 07:34 | |
| *** errr has joined #openstack-ansible | 07:36 | |
| *** errr has quit IRC | 07:37 | |
| *** errr has joined #openstack-ansible | 07:38 | |
| *** errr has quit IRC | 07:39 | |
| *** errr has joined #openstack-ansible | 07:41 | |
| *** errr has quit IRC | 07:41 | |
| *** errr has joined #openstack-ansible | 07:43 | |
| *** errr has quit IRC | 07:43 | |
| *** andrea15 has quit IRC | 07:46 | |
| *** fghaas has joined #openstack-ansible | 07:48 | |
| *** errr has joined #openstack-ansible | 07:51 | |
| *** errr has joined #openstack-ansible | 07:52 | |
| *** errr has quit IRC | 07:53 | |
| *** errr has joined #openstack-ansible | 07:55 | |
| *** errr has quit IRC | 07:56 | |
| *** errr has joined #openstack-ansible | 07:57 | |
| *** errr has joined #openstack-ansible | 07:58 | |
| *** errr has quit IRC | 07:59 | |
| *** errr has joined #openstack-ansible | 08:00 | |
| *** errr has quit IRC | 08:01 | |
| *** errr has joined #openstack-ansible | 08:02 | |
| *** errr has quit IRC | 08:03 | |
| *** tosky has joined #openstack-ansible | 08:05 | |
| *** errr has joined #openstack-ansible | 08:05 | |
| *** errr has quit IRC | 08:05 | |
| *** errr has joined #openstack-ansible | 08:06 | |
| *** errr has quit IRC | 08:07 | |
| *** errr has joined #openstack-ansible | 08:08 | |
| *** errr has quit IRC | 08:09 | |
| *** errr has joined #openstack-ansible | 08:11 | |
| *** errr has quit IRC | 08:11 | |
| *** errr has joined #openstack-ansible | 08:13 | |
| *** errr has quit IRC | 08:14 | |
| *** xakaitetoia1 has joined #openstack-ansible | 08:16 | |
| *** errr has joined #openstack-ansible | 08:22 | |
| *** errr has quit IRC | 08:23 | |
| *** errr has joined #openstack-ansible | 08:24 | |
| *** errr has quit IRC | 08:24 | |
| *** errr has joined #openstack-ansible | 08:26 | |
| *** errr has quit IRC | 08:26 | |
| *** errr has joined #openstack-ansible | 08:27 | |
| *** errr has quit IRC | 08:28 | |
| *** errr has joined #openstack-ansible | 08:29 | |
| *** errr has quit IRC | 08:31 | |
| *** ysastri has joined #openstack-ansible | 08:36 | |
| *** fghaas has left #openstack-ansible | 08:47 | |
| *** errr has joined #openstack-ansible | 09:00 | |
| *** errr has quit IRC | 09:00 | |
| *** yolanda__ is now known as yolanda | 09:00 | |
| *** udesale has quit IRC | 09:07 | |
| *** errr has joined #openstack-ansible | 09:12 | |
| *** errr has quit IRC | 09:13 | |
| *** errr has joined #openstack-ansible | 09:14 | |
| *** errr has quit IRC | 09:15 | |
| *** errr has joined #openstack-ansible | 09:17 | |
| *** errr has quit IRC | 09:18 | |
| *** errr has joined #openstack-ansible | 09:20 | |
| *** errr has quit IRC | 09:20 | |
| *** errr has joined #openstack-ansible | 09:22 | |
| jrosser | mnaser: i'm going to start fresh again today on my centos8 vm - want to get to the point where i'm stuck with galera and have patches up that do the same | 09:23 |
| *** errr has quit IRC | 09:23 | |
| *** errr has joined #openstack-ansible | 09:24 | |
| *** DanyC has joined #openstack-ansible | 09:25 | |
| *** errr has quit IRC | 09:25 | |
| *** errr has joined #openstack-ansible | 09:27 | |
| *** errr has quit IRC | 09:27 | |
| *** errr has joined #openstack-ansible | 09:29 | |
| *** errr has quit IRC | 09:30 | |
| *** errr has joined #openstack-ansible | 09:31 | |
| *** errr has quit IRC | 09:32 | |
| *** errr has joined #openstack-ansible | 09:34 | |
| *** errr has quit IRC | 09:34 | |
| *** errr has joined #openstack-ansible | 09:35 | |
| *** errr has quit IRC | 09:35 | |
| openstackgerrit | kourosh vivan proposed openstack/openstack-ansible-os_manila stable/stein: Add variable files for RedHat and Suse https://review.opendev.org/709673 | 09:36 |
| *** errr has joined #openstack-ansible | 09:38 | |
| *** errr has quit IRC | 09:38 | |
| *** errr has joined #openstack-ansible | 09:38 | |
| kvivan | Hi, I am working on using osa stein (centos) manila with cephFS backend, just notice manila stein don't support centos and suse (i proposed a cherry pick from train) | 09:39 |
| *** errr has quit IRC | 09:39 | |
| *** errr has joined #openstack-ansible | 09:41 | |
| *** errr has quit IRC | 09:42 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: virtualenv: use native python module https://review.opendev.org/709461 | 09:43 |
| *** errr has joined #openstack-ansible | 09:43 | |
| *** errr has quit IRC | 09:44 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Do not include docs/requirements.txt for functional tests https://review.opendev.org/709320 | 09:44 |
| *** errr has joined #openstack-ansible | 09:45 | |
| *** errr has quit IRC | 09:47 | |
| *** errr has joined #openstack-ansible | 09:48 | |
| *** errr has quit IRC | 09:49 | |
| *** errr has joined #openstack-ansible | 09:50 | |
| mnaser | jrosser: nice | 09:51 |
| *** errr has quit IRC | 09:51 | |
| mnaser | jrosser: any chances you or gshippey might be in yvr? :> | 09:51 |
| *** errr has joined #openstack-ansible | 09:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-ansible-rabbitmq_server master: DNM: Testing jobs https://review.opendev.org/709678 | 09:53 |
| *** errr has quit IRC | 09:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Use /tmp for ansible temporary directory https://review.opendev.org/709327 | 09:53 |
| *** errr has joined #openstack-ansible | 09:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-ansible-rabbitmq_server master: DNM: Testing jobs https://review.opendev.org/709678 | 09:55 |
| *** errr has quit IRC | 09:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-ansible-rabbitmq_server master: DNM: Testing jobs https://review.opendev.org/709678 | 09:56 |
| *** errr has joined #openstack-ansible | 09:57 | |
| *** errr has joined #openstack-ansible | 09:58 | |
| *** errr has quit IRC | 09:59 | |
| *** errr has joined #openstack-ansible | 10:02 | |
| *** errr has quit IRC | 10:03 | |
| jrosser | mnaser: so i can't reproduce this locally https://zuul.opendev.org/t/openstack/build/c51c936c5bf54e12a0a6135f88fffcaa/log/job-output.txt#9733 | 10:09 |
| jrosser | oh interesting https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c51/689629/23/check/openstack-ansible-deploy-aio_metal-centos-8/c51c936/logs/ara-report/result/6f2ef2b6-af1a-4e4a-a055-e579e8351761/ | 10:10 |
| *** errr has joined #openstack-ansible | 10:16 | |
| openstackgerrit | kourosh vivan proposed openstack/openstack-ansible-os_manila master: Fix python path when deploy host is not localhost https://review.opendev.org/709684 | 10:17 |
| *** errr has joined #openstack-ansible | 10:30 | |
| *** errr has quit IRC | 10:30 | |
| *** errr has joined #openstack-ansible | 10:32 | |
| *** errr has joined #openstack-ansible | 10:33 | |
| *** errr has joined #openstack-ansible | 10:34 | |
| *** errr has quit IRC | 10:35 | |
| *** errr has joined #openstack-ansible | 10:36 | |
| *** errr has quit IRC | 10:37 | |
| *** errr has joined #openstack-ansible | 10:38 | |
| *** errr has quit IRC | 10:39 | |
| ChosSimbaOne_ | Hi. I am in need of some help. All my nova-compute is in state down, and i cannot figure out how to get them online again. | 10:44 |
| ChosSimbaOne_ | There is connection to the rabbitmq cluster and the nodes is up. | 10:44 |
| ChosSimbaOne_ | I cannot find any clue in the nova-compute.log on what is wrong. Reboot of the nova-compute did not fix it. | 10:44 |
| ChosSimbaOne_ | It all started with our tempest job not terminating correctly. It started an avalanch where failed tempest job ontop of failed tempest jobs, clutted the machines, and now nothing seems to be working. | 10:44 |
| ChosSimbaOne_ | Any good input on how to go about it? | 10:44 |
| ChosSimbaOne_ | I tired forcing it up with the cli, but that just failed. | 10:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Ironic fixes from inspector addition https://review.opendev.org/708693 | 10:48 |
| *** errr has joined #openstack-ansible | 10:52 | |
| *** errr has quit IRC | 10:53 | |
| *** errr has joined #openstack-ansible | 10:54 | |
| *** errr has quit IRC | 10:55 | |
| jrosser | ChosSimbaOne_: i would probably be starting with the rabbitmq cluster | 10:55 |
| jrosser | thats not very specific but "everything is broken" problems quite often lead there | 10:56 |
| *** errr has joined #openstack-ansible | 10:56 | |
| *** errr has quit IRC | 10:57 | |
| ChosSimbaOne_ | jrosser: Okay, I am just a bit lost with the rabbitmq cluster. i looked at the steps here <https://docs.openstack.org/openstack-ansible/queens/admin/maintenance-tasks.html#rabbitmq-cluster-maintenance> for recovery and it does not seem broken. I can telnet from the nova hosts to the rabbitmq cluster. | 10:58 |
| ChosSimbaOne_ | Where is a sane place to start looking? | 10:58 |
| *** errr has joined #openstack-ansible | 10:59 | |
| *** errr has quit IRC | 11:00 | |
| *** errr has joined #openstack-ansible | 11:01 | |
| *** errr has quit IRC | 11:02 | |
| *** leo8a has joined #openstack-ansible | 11:02 | |
| *** elenalindq has quit IRC | 11:02 | |
| *** errr has joined #openstack-ansible | 11:03 | |
| *** errr has quit IRC | 11:04 | |
| *** errr has joined #openstack-ansible | 11:06 | |
| *** errr has quit IRC | 11:07 | |
| ChosSimbaOne_ | so from: | 11:08 |
| ChosSimbaOne_ | root@infra1-rabbit-mq-container-59f5e474:~# rabbitmqctl cluster_status | 11:08 |
| ChosSimbaOne_ | Cluster status of node 'rabbit@infra1-rabbit-mq-container-59f5e474' | 11:08 |
| ChosSimbaOne_ | [{nodes,[{disc,['rabbit@infra1-rabbit-mq-container-59f5e474', | 11:08 |
| ChosSimbaOne_ | 'rabbit@infra2-rabbit-mq-container-54e5d1fb', | 11:08 |
| ChosSimbaOne_ | 'rabbit@infra3-rabbit-mq-container-5bccc14c']}]}, | 11:08 |
| ChosSimbaOne_ | {running_nodes,['rabbit@infra2-rabbit-mq-container-54e5d1fb', | 11:08 |
| ChosSimbaOne_ | 'rabbit@infra3-rabbit-mq-container-5bccc14c', | 11:08 |
| ChosSimbaOne_ | 'rabbit@infra1-rabbit-mq-container-59f5e474']}, | 11:08 |
| ChosSimbaOne_ | {cluster_name,<<"openstack">>}, | 11:08 |
| ChosSimbaOne_ | {partitions,[]}, | 11:08 |
| ChosSimbaOne_ | {alarms,[{'rabbit@infra2-rabbit-mq-container-54e5d1fb',[]}, | 11:08 |
| ChosSimbaOne_ | {'rabbit@infra3-rabbit-mq-container-5bccc14c',[]}, | 11:08 |
| ChosSimbaOne_ | {'rabbit@infra1-rabbit-mq-container-59f5e474',[]}]}] | 11:08 |
| ChosSimbaOne_ | root@infra1-rabbit-mq-container-59f5e474:~# | 11:08 |
| ChosSimbaOne_ | this is outbut from the rabbitmq container. | 11:08 |
| ChosSimbaOne_ | well this might be something: | 11:09 |
| ChosSimbaOne_ | =WARNING REPORT==== 25-Feb-2020::11:23:54 === | 11:09 |
| ChosSimbaOne_ | closing AMQP connection <0.3622.0> (172.21.234.88:53944 -> 172.21.232.199:5671 - uwsgi:14964:22057060-7642-4b60-9d7a-c3c4c4962a62, vhost: '/nova', user: 'nova'): | 11:09 |
| ChosSimbaOne_ | client unexpectedly closed TCP connectionvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv | 11:09 |
| ChosSimbaOne_ | disregard all the v's my keyboard locked up. | 11:10 |
| *** cshen has quit IRC | 11:14 | |
| jrosser | ChosSimbaOne_: we have the rabbitmq web interface configured and it's easy to use to look at the queue lengths and cluster status | 11:15 |
| jrosser | you can do the same with rabbtimqctl | 11:15 |
| ChosSimbaOne_ | cool, it is not setup as part of openstack ansible? | 11:16 |
| *** udesale has joined #openstack-ansible | 11:17 | |
| jrosser | i think the basic config is done but you need to give the monitoring user more access rights than the OSA playbooks do to make the web interface useful | 11:17 |
| ChosSimbaOne_ | okay, will give it a go | 11:17 |
| *** errr has joined #openstack-ansible | 11:19 | |
| *** errr has quit IRC | 11:20 | |
| *** fghaas has joined #openstack-ansible | 11:23 | |
| *** errr has joined #openstack-ansible | 11:23 | |
| *** leo8a has quit IRC | 11:24 | |
| *** errr has quit IRC | 11:24 | |
| *** leo8a has joined #openstack-ansible | 11:24 | |
| ebbex | Does the ceph-install.yml upgrade from mimic to nautilus work on train? I'm stuck waiting for monitors to form the quorum. | 11:25 |
| *** ansmith has joined #openstack-ansible | 11:26 | |
| ebbex | From the logs on a mon_container it seems pretty bad. | 11:26 |
| noonedeadpunk | gshippey: mnaser can you kindluy review https://review.opendev.org/#/c/709308/ ? | 11:29 |
| *** ivve has joined #openstack-ansible | 11:30 | |
| *** leo8a has quit IRC | 11:31 | |
| mathlin | ChosSimbaOne_: I am facing what sounds like a similar problem, i think mine is in this bugreport https://bugs.launchpad.net/nova/+bug/1835637 | 11:31 |
| openstack | Launchpad bug 1835637 in OpenStack Compute (nova) "(404) NOT_FOUND - failed to perform operation on queue 'notifications.info' in vhost '/nova' due to timeout" [Undecided,Incomplete] | 11:31 |
| noonedeadpunk | mnaser: can you help me figuring out what's wrong with https://review.opendev.org/#/c/704935/ ? | 11:33 |
| noonedeadpunk | As I'm pretty confused why uwsgi systemd_service tasks are just skipped https://zuul.opendev.org/t/openstack/build/721a43912e764f448b2ae1875a6c4077/log/job-output.txt#17881 | 11:35 |
| noonedeadpunk | as there are kinda no conditionas why config_template runs, but systemd_services not https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/tasks/uwsgi_post_install.yml | 11:36 |
| noonedeadpunk | I have a sandbox where issue is reproduced just in case | 11:36 |
| noonedeadpunk | I may miss somth pretty obvious... | 11:36 |
| ChosSimbaOne_ | mathlin: Thank you for the suggestion. will have a look, but running queens, and it seems like that should be fine according to the bugrebprot | 11:37 |
| *** rgogunskiy has joined #openstack-ansible | 11:41 | |
| noonedeadpunk | ok, with debug I see the following: | 11:41 |
| noonedeadpunk | <aio1> Task "Set systemd version" has been omitted from the job because the conditional "["inventory_hostname in groups['manila_api']", "not inventory_hostname in groups['manila_api']", "inventory_hostname in groups['manila_api']", "manila_services['manila-api']['group'] in group_names", 'uwsgi_services']" was evaluated as "False" | 11:41 |
| *** cshen has joined #openstack-ansible | 11:42 | |
| jrosser | ["inventory_hostname in groups['manila_api']", "not inventory_hostname in groups['manila_api']" <- Uh Oh | 11:42 |
| *** ansmith has quit IRC | 11:42 | |
| *** errr has joined #openstack-ansible | 11:42 | |
| noonedeadpunk | yeah, but where does it comes from.... | 11:43 |
| jrosser | theres not some overloading of 'item' going on here is there? | 11:43 |
| *** errr has quit IRC | 11:43 | |
| jrosser | outer with_items messing up an inner one by both using 'item' | 11:43 |
| noonedeadpunk | oh, I see what you mean | 11:44 |
| *** errr has joined #openstack-ansible | 11:45 | |
| *** jamesdenton has quit IRC | 11:45 | |
| ChosSimbaOne_ | jrosser: | 11:45 |
| *** errr has quit IRC | 11:46 | |
| ChosSimbaOne_ | jrosser: it does not seems like anything is wrong to me: https://imgur.com/a/RIKSC0h | 11:46 |
| *** jamesdenton has joined #openstack-ansible | 11:46 | |
| *** errr has joined #openstack-ansible | 11:47 | |
| *** errr has quit IRC | 11:48 | |
| noonedeadpunk | jrosser: but I don't see loop.... and while including systemd role loop_control is used... | 11:51 |
| *** errr has joined #openstack-ansible | 11:51 | |
| *** errr has quit IRC | 11:52 | |
| *** errr has joined #openstack-ansible | 11:52 | |
| jrosser | theres lots of use of item here https://github.com/openstack/ansible-role-systemd_service/blob/master/tasks/main.yml#L66 | 11:52 |
| *** nicolasbock has joined #openstack-ansible | 11:53 | |
| *** errr has quit IRC | 11:53 | |
| noonedeadpunk | jrosser: the thing is, that I've added debig: var=systemd_services here https://github.com/openstack/ansible-role-systemd_service/blob/master/tasks/main.yml#L15 and it's also skipped with the same message | 11:54 |
| *** jawad_axd has joined #openstack-ansible | 11:54 | |
| jrosser | wheres your patch again? | 11:54 |
| noonedeadpunk | https://review.opendev.org/#/c/704935/ | 11:54 |
| *** errr has joined #openstack-ansible | 11:55 | |
| noonedeadpunk | dropping when on L131 https://review.opendev.org/#/c/704935/16/tasks/main.yml don't help | 11:55 |
| *** errr has quit IRC | 11:55 | |
| *** errr has joined #openstack-ansible | 11:57 | |
| noonedeadpunk | I can kinda thing that this might be related to this stuff... https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-manila-install.yml#L44-L49 | 11:57 |
| noonedeadpunk | But like not sure how | 11:57 |
| *** errr has quit IRC | 11:58 | |
| *** errr has joined #openstack-ansible | 12:00 | |
| *** errr has quit IRC | 12:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-plugins master: Expand paths to user home directories in connection plugin https://review.opendev.org/709308 | 12:01 |
| *** errr has joined #openstack-ansible | 12:02 | |
| *** errr has quit IRC | 12:03 | |
| noonedeadpunk | jrosser: lol - commenting out https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-manila-install.yml#L47-L49 helped.... | 12:04 |
| noonedeadpunk | but it's pretty weird.... | 12:04 |
| *** errr has joined #openstack-ansible | 12:05 | |
| noonedeadpunk | why both conditions are applied? Ansible bug? | 12:05 |
| jrosser | noonedeadpunk: the recent documentation is not very clear on this | 12:05 |
| *** errr has quit IRC | 12:05 | |
| jrosser | but look https://docs.ansible.com/ansible/2.3/playbooks_roles.html | 12:05 |
| jrosser | "This works by applying the conditional to every task in the role" | 12:05 |
| *** errr has joined #openstack-ansible | 12:05 | |
| jrosser | so it probably depends how the playbook is rendered, if the tasks have their conditions applied all up front | 12:06 |
| jrosser | and i don't think it's like a programming languange where you have two instances of the role | 12:06 |
| *** errr has quit IRC | 12:06 | |
| *** dave-mccowan has quit IRC | 12:06 | |
| * noonedeadpunk tries to recall why do we run role twice | 12:07 | |
| jrosser | does the api need to be up first? | 12:08 |
| jrosser | then the other components somehow need it when they are deployed | 12:08 |
| *** ysastri has quit IRC | 12:09 | |
| noonedeadpunk | But this can probably be handled with flush_handlers... | 12:09 |
| jrosser | why not just split the whole thing into two plays | 12:09 |
| noonedeadpunk | or this< yes | 12:10 |
| jrosser | manila_api followed by manila_all:!manila_api | 12:10 |
| jrosser | its just too clever at the moment and you have to study it hard to understand whats happening | 12:10 |
| jrosser | it feels like the intent is to do the API first, followed by the rest | 12:10 |
| noonedeadpunk | jrosser: oh, wait https://review.opendev.org/#/c/704942/ | 12:11 |
| noonedeadpunk | I just thought it's already merged lol | 12:11 |
| jrosser | lol hah | 12:11 |
| * jrosser rechecks | 12:12 | |
| jrosser | that should merge now that the upgrade jobs are nv | 12:12 |
| noonedeadpunk | yeah, sorry for bothering you | 12:14 |
| jrosser | no worries - while you are here ...... | 12:14 |
| jrosser | this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c51/689629/23/check/openstack-ansible-deploy-aio_metal-centos-8/c51c936/logs/ara-report/result/6f2ef2b6-af1a-4e4a-a055-e579e8351761/ | 12:14 |
| jrosser | that looks like left over symlinks in /usr/local/bin after the gate cleanup playbook has run? | 12:15 |
| *** dave-mccowan has joined #openstack-ansible | 12:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/709517 | 12:16 |
| noonedeadpunk | jrosser: hm, yes, I could possibly leave symlinks if they were there... I was kinda only removing pip and system packages | 12:18 |
| noonedeadpunk | jrosser: I think for centos 8 it should be python3-virtualenv? https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/playbooks/vars/redhat.yml#L17 | 12:19 |
| jrosser | i think the issue is that there is a leftover binary in /usr/local/bin which gets used before anything from python3-virtualenv package | 12:20 |
| jrosser | binary/symlink | 12:20 |
| noonedeadpunk | so we just don't touch python3-virtualenv. I think ideally if packages are removed, symlink shouldn't cause issues as it's going to be broken | 12:21 |
| noonedeadpunk | except it's hardlink ofc.... | 12:21 |
| jrosser | do you have a held node right now? | 12:22 |
| noonedeadpunk | with centos8? no.... | 12:22 |
| jrosser | anything will do, just to see what actually is in /usr/local/bin | 12:22 |
| jrosser | hmm perhaps this is different by OS of course | 12:22 |
| noonedeadpunk | or you mean any? I think I have one with s->t for ubuntu | 12:23 |
| noonedeadpunk | let,s try just adding python3-virtualenv to https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/playbooks/vars/redhat.yml#L17 list | 12:24 |
| jrosser | it's already installed via openstack_hosts | 12:24 |
| noonedeadpunk | yes, it's installed, but it's not removed before aio | 12:24 |
| noonedeadpunk | so infra installs latest version of packages which are build with their own I guess | 12:25 |
| noonedeadpunk | so this is a list of packages to remove from gates only | 12:25 |
| *** errr has joined #openstack-ansible | 12:27 | |
| *** errr has quit IRC | 12:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_panko master: Updated from OpenStack Ansible Tests https://review.opendev.org/709503 | 12:30 |
| *** errr has joined #openstack-ansible | 12:30 | |
| *** errr has quit IRC | 12:31 | |
| *** errr has joined #openstack-ansible | 12:32 | |
| *** errr has quit IRC | 12:32 | |
| *** errr has joined #openstack-ansible | 12:33 | |
| *** errr has quit IRC | 12:34 | |
| *** errr has joined #openstack-ansible | 12:35 | |
| mnaser | https://review.opendev.org/#/c/709435/4 should help us get integrated | 12:36 |
| mnaser | to use py3 venv | 12:36 |
| *** errr has quit IRC | 12:36 | |
| noonedeadpunk | nice! | 12:37 |
| *** errr has joined #openstack-ansible | 12:38 | |
| *** errr has quit IRC | 12:39 | |
| *** errr has joined #openstack-ansible | 12:40 | |
| *** jbadiapa has joined #openstack-ansible | 12:40 | |
| *** errr has quit IRC | 12:41 | |
| openstackgerrit | Merged openstack/ansible-role-systemd_networkd master: Updated from OpenStack Ansible Tests https://review.opendev.org/709523 | 12:43 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Updated from OpenStack Ansible Tests https://review.opendev.org/709521 | 12:49 |
| *** sshnaidm is now known as sshnaidm|bbl | 12:59 | |
| *** errr has joined #openstack-ansible | 13:00 | |
| *** errr has quit IRC | 13:00 | |
| *** errr has joined #openstack-ansible | 13:02 | |
| *** elenalindq has joined #openstack-ansible | 13:03 | |
| *** errr has quit IRC | 13:03 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Clean CI virtualenv installation https://review.opendev.org/709720 | 13:03 |
| *** errr has joined #openstack-ansible | 13:04 | |
| *** errr has quit IRC | 13:05 | |
| *** errr has joined #openstack-ansible | 13:07 | |
| *** errr has quit IRC | 13:07 | |
| *** elenalindq has left #openstack-ansible | 13:07 | |
| *** errr has joined #openstack-ansible | 13:09 | |
| *** errr has joined #openstack-ansible | 13:10 | |
| *** errr has quit IRC | 13:11 | |
| *** errr has joined #openstack-ansible | 13:12 | |
| *** errr has quit IRC | 13:13 | |
| *** leo8a has joined #openstack-ansible | 13:13 | |
| *** errr has joined #openstack-ansible | 13:15 | |
| *** errr has quit IRC | 13:15 | |
| *** errr has joined #openstack-ansible | 13:17 | |
| *** errr has quit IRC | 13:19 | |
| *** errr has joined #openstack-ansible | 13:20 | |
| *** oligau has quit IRC | 13:21 | |
| *** errr has quit IRC | 13:21 | |
| *** oligau has joined #openstack-ansible | 13:21 | |
| *** errr has joined #openstack-ansible | 13:23 | |
| *** errr has quit IRC | 13:24 | |
| *** errr has joined #openstack-ansible | 13:25 | |
| *** errr has quit IRC | 13:26 | |
| *** errr has joined #openstack-ansible | 13:27 | |
| *** errr has quit IRC | 13:28 | |
| *** errr has joined #openstack-ansible | 13:30 | |
| *** errr has quit IRC | 13:30 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.opendev.org/709480 | 13:32 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/709472 | 13:32 |
| *** errr has joined #openstack-ansible | 13:32 | |
| *** CeeMac has joined #openstack-ansible | 13:32 | |
| CeeMac | afternoon | 13:33 |
| *** errr has quit IRC | 13:33 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:33 | |
| *** nicolasbock has quit IRC | 13:33 | |
| *** nicolasbock has joined #openstack-ansible | 13:34 | |
| *** errr has joined #openstack-ansible | 13:35 | |
| *** errr has quit IRC | 13:35 | |
| *** errr has joined #openstack-ansible | 13:36 | |
| *** nicolasbock has quit IRC | 13:36 | |
| *** errr has quit IRC | 13:37 | |
| *** errr has joined #openstack-ansible | 13:38 | |
| *** errr has quit IRC | 13:39 | |
| *** ansmith has joined #openstack-ansible | 13:39 | |
| *** errr has joined #openstack-ansible | 13:40 | |
| *** errr has quit IRC | 13:41 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_openrc master: Updated from OpenStack Ansible Tests https://review.opendev.org/709476 | 13:41 |
| *** errr has joined #openstack-ansible | 13:43 | |
| *** errr has quit IRC | 13:43 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/709468 | 13:43 |
| *** errr has joined #openstack-ansible | 13:44 | |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount master: Updated from OpenStack Ansible Tests https://review.opendev.org/709522 | 13:44 |
| *** errr has quit IRC | 13:44 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Updated from OpenStack Ansible Tests https://review.opendev.org/709510 | 13:45 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests https://review.opendev.org/709483 | 13:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests https://review.opendev.org/709493 | 13:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Updated from OpenStack Ansible Tests https://review.opendev.org/709501 | 13:47 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: Updated from OpenStack Ansible Tests https://review.opendev.org/709524 | 13:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests https://review.opendev.org/709489 | 13:49 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests https://review.opendev.org/709486 | 13:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests https://review.opendev.org/709509 | 13:54 |
| *** fghaas has left #openstack-ansible | 13:56 | |
| *** mhayden has quit IRC | 13:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.opendev.org/709491 | 13:59 |
| *** mhayden has joined #openstack-ansible | 14:01 | |
| *** errr has joined #openstack-ansible | 14:03 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests https://review.opendev.org/709487 | 14:04 |
| *** errr has quit IRC | 14:04 | |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client master: Updated from OpenStack Ansible Tests https://review.opendev.org/709518 | 14:05 |
| *** errr has joined #openstack-ansible | 14:05 | |
| *** errr has quit IRC | 14:06 | |
| *** errr has joined #openstack-ansible | 14:08 | |
| *** errr has quit IRC | 14:09 | |
| *** fghaas has joined #openstack-ansible | 14:10 | |
| *** fghaas has quit IRC | 14:10 | |
| *** fghaas has joined #openstack-ansible | 14:11 | |
| *** errr has joined #openstack-ansible | 14:11 | |
| *** errr has quit IRC | 14:11 | |
| *** errr has joined #openstack-ansible | 14:13 | |
| *** errr has quit IRC | 14:13 | |
| *** errr has joined #openstack-ansible | 14:14 | |
| *** errr has quit IRC | 14:15 | |
| *** errr has joined #openstack-ansible | 14:16 | |
| *** errr has quit IRC | 14:18 | |
| *** errr has joined #openstack-ansible | 14:20 | |
| *** errr has quit IRC | 14:21 | |
| *** errr has joined #openstack-ansible | 14:22 | |
| *** errr has quit IRC | 14:23 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Allow non-url Elastic gpg key specification https://review.opendev.org/709732 | 14:24 |
| *** errr has joined #openstack-ansible | 14:25 | |
| *** errr has quit IRC | 14:26 | |
| *** fghaas has left #openstack-ansible | 14:27 | |
| *** errr has joined #openstack-ansible | 14:27 | |
| *** errr has quit IRC | 14:28 | |
| *** errr has joined #openstack-ansible | 14:29 | |
| *** errr has quit IRC | 14:30 | |
| *** errr has joined #openstack-ansible | 14:32 | |
| *** errr has quit IRC | 14:33 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Allow non-url Elastic gpg key specification https://review.opendev.org/709732 | 14:34 |
| *** errr has joined #openstack-ansible | 14:34 | |
| *** errr has quit IRC | 14:35 | |
| *** errr has joined #openstack-ansible | 14:37 | |
| *** errr has quit IRC | 14:37 | |
| *** errr has joined #openstack-ansible | 14:37 | |
| *** errr has quit IRC | 14:38 | |
| *** errr has joined #openstack-ansible | 14:40 | |
| *** errr has quit IRC | 14:40 | |
| *** errr has joined #openstack-ansible | 14:41 | |
| *** errr has quit IRC | 14:41 | |
| *** jawad_axd has quit IRC | 14:42 | |
| *** jawad_axd has joined #openstack-ansible | 14:43 | |
| *** jawad_axd has quit IRC | 14:43 | |
| *** errr has joined #openstack-ansible | 14:43 | |
| *** errr has quit IRC | 14:44 | |
| *** jawad_axd has joined #openstack-ansible | 14:45 | |
| *** errr has joined #openstack-ansible | 14:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Updated from OpenStack Ansible Tests https://review.opendev.org/709490 | 14:46 |
| *** errr has quit IRC | 14:46 | |
| *** dave-mccowan has quit IRC | 14:47 | |
| *** jawad_ax_ has joined #openstack-ansible | 14:48 | |
| *** errr has joined #openstack-ansible | 14:48 | |
| *** errr has quit IRC | 14:48 | |
| *** errr has joined #openstack-ansible | 14:49 | |
| *** jawad_axd has quit IRC | 14:49 | |
| *** errr has quit IRC | 14:50 | |
| *** errr has joined #openstack-ansible | 14:51 | |
| openstackgerrit | Cam J. Loader (cjloader) proposed openstack/openstack-ansible-os_ironic stable/train: Ironic fixes from inspector addition https://review.opendev.org/709747 | 14:51 |
| *** dave-mccowan has joined #openstack-ansible | 14:52 | |
| cjloader | jrosser: noonedeadpunk ^ backport | 14:52 |
| *** errr has quit IRC | 14:52 | |
| *** jawad_ax_ has quit IRC | 14:52 | |
| *** errr has joined #openstack-ansible | 14:53 | |
| *** errr has quit IRC | 14:54 | |
| *** errr has joined #openstack-ansible | 14:56 | |
| *** errr has quit IRC | 14:57 | |
| *** errr has joined #openstack-ansible | 14:58 | |
| *** errr has quit IRC | 14:59 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Clean CI virtualenv installation https://review.opendev.org/709720 | 14:59 |
| *** errr has joined #openstack-ansible | 15:00 | |
| *** errr has quit IRC | 15:01 | |
| *** errr has joined #openstack-ansible | 15:03 | |
| *** errr has quit IRC | 15:03 | |
| *** errr has joined #openstack-ansible | 15:05 | |
| *** errr has quit IRC | 15:05 | |
| *** errr has joined #openstack-ansible | 15:06 | |
| *** errr has quit IRC | 15:07 | |
| *** errr has joined #openstack-ansible | 15:08 | |
| *** errr has quit IRC | 15:09 | |
| *** errr has joined #openstack-ansible | 15:11 | |
| *** errr has quit IRC | 15:12 | |
| *** errr has joined #openstack-ansible | 15:13 | |
| *** errr has quit IRC | 15:14 | |
| *** errr has joined #openstack-ansible | 15:15 | |
| *** errr has quit IRC | 15:16 | |
| *** errr has joined #openstack-ansible | 15:18 | |
| *** errr has quit IRC | 15:18 | |
| *** errr has joined #openstack-ansible | 15:20 | |
| *** errr has quit IRC | 15:22 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Remove global python interpreter setting. https://review.opendev.org/709756 | 15:22 |
| jrosser | mnaser: ^ was that the sort of thing for stripping back the py2/3 switch variables? Is there value in also removing the default? | 15:23 |
| *** errr has joined #openstack-ansible | 15:23 | |
| *** errr has quit IRC | 15:24 | |
| jrosser | once this merges we can take out even more and make the roles be completely py3 only? https://review.opendev.org/#/c/709441/ | 15:25 |
| *** rgogunskiy has quit IRC | 15:27 | |
| *** irclogbot_0 has quit IRC | 15:27 | |
| *** johanssone has quit IRC | 15:27 | |
| *** irclogbot_3 has joined #openstack-ansible | 15:27 | |
| *** johanssone has joined #openstack-ansible | 15:29 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose apt repo configuration https://review.opendev.org/709543 | 15:45 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Allow non-url Elastic gpg key specification https://review.opendev.org/709732 | 15:45 |
| *** errr has joined #openstack-ansible | 15:49 | |
| *** errr has quit IRC | 15:49 | |
| *** errr has joined #openstack-ansible | 15:50 | |
| *** errr has quit IRC | 15:51 | |
| *** errr has joined #openstack-ansible | 15:52 | |
| *** errr has quit IRC | 15:53 | |
| *** nicolasbock has joined #openstack-ansible | 15:54 | |
| *** cshen has quit IRC | 15:54 | |
| *** errr has joined #openstack-ansible | 16:01 | |
| *** errr has quit IRC | 16:02 | |
| *** jawad_axd has joined #openstack-ansible | 16:02 | |
| *** kvivan has quit IRC | 16:02 | |
| *** errr has joined #openstack-ansible | 16:03 | |
| *** errr has quit IRC | 16:04 | |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 16:06 |
| openstack | Meeting started Tue Feb 25 16:06:26 2020 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:06 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:06 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:06 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:06 |
| noonedeadpunk | #topic office hours | 16:06 |
| *** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)" | 16:06 | |
| *** jawad_axd has quit IRC | 16:06 | |
| cyberpear | Is this the same group of folks who maintain the `os_*` ansible modules, or is this just the roles/playbooks for installing/managing OpenStack? | 16:08 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: [WIP] Centos-8 support https://review.opendev.org/689629 | 16:08 |
| noonedeadpunk | cyberpear: second option:) you can join #openstack-ansible-sig - there you can find ansible modules maintainers as well | 16:09 |
| cjloader | o/ | 16:09 |
| noonedeadpunk | o/ | 16:09 |
| cyberpear | thx! | 16:09 |
| jrosser | o/ | 16:12 |
| noonedeadpunk | ok, so what do we have, except everything being broken?:) | 16:13 |
| *** errr has joined #openstack-ansible | 16:22 | |
| *** errr has quit IRC | 16:23 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Allow non-url Elastic gpg key specification https://review.opendev.org/709732 | 16:24 |
| *** errr has joined #openstack-ansible | 16:25 | |
| *** errr has quit IRC | 16:26 | |
| *** gyee has joined #openstack-ansible | 16:27 | |
| jrosser | so we have a bunch of things going on - centos8 WIP support is here https://review.opendev.org/#/q/topic:osa/el8 | 16:27 |
| *** errr has joined #openstack-ansible | 16:27 | |
| *** errr has quit IRC | 16:28 | |
| jrosser | then about to merge switching debian stretch to buster on master https://review.opendev.org/#/q/topic:buster_functional+(status:open+OR+status:merged) | 16:29 |
| jrosser | that will likley break a few things | 16:29 |
| *** errr has joined #openstack-ansible | 16:29 | |
| jrosser | and a reasonable view of the things that need working on is here https://review.opendev.org/#/q/topic:openstack/openstack-ansible-tests/sync-tests+status:open | 16:30 |
| *** errr has quit IRC | 16:30 | |
| noonedeadpunk | yeah this is lot of work ^ | 16:30 |
| jrosser | there will be some common things - need to fix the pre-run cleanup stuff into the tests repo | 16:31 |
| noonedeadpunk | btw, what do we think about https://review.opendev.org/#/c/709457/ ? | 16:31 |
| jrosser | thats breaking a bunch of those roles | 16:31 |
| *** errr has joined #openstack-ansible | 16:32 | |
| *** errr has quit IRC | 16:32 | |
| noonedeadpunk | regarding suse personally I don't feel right now that it takes a lot of time for it's maintennce | 16:33 |
| *** errr has joined #openstack-ansible | 16:33 | |
| *** errr has quit IRC | 16:33 | |
| jrosser | it would be nice for someone to be getting the things currently broken to be working, but that needs a contributor to come forward | 16:35 |
| noonedeadpunk | yes, agreed. But in case we remove like CI, it means we stop supporting it. And currently CI is working, at least for source installs | 16:36 |
| noonedeadpunk | we can kinda drop distro ones | 16:36 |
| noonedeadpunk | maybe it's worth calling for interested parties in ML | 16:36 |
| noonedeadpunk | as we currently have no idea if suse is used and how widely | 16:37 |
| jrosser | i think https://review.opendev.org/689629 fails on virtualenv again | 16:38 |
| CeeMac | hi jrosser | 16:38 |
| jrosser | Hi CeeMac | 16:38 |
| noonedeadpunk | yeah. that's true..... | 16:38 |
| CeeMac | jrosser: if you get a few spare minutes I don't suppose you'd mind peeking into the neutron channel at my epic question? I've hit a brick wall. no rush, i'm about to head out for the day but wouldn't mind a chat tomorrow if you've any ideas | 16:40 |
| jrosser | CeeMac: this is OVS? | 16:42 |
| CeeMac | it is | 16:42 |
| CeeMac | oh, you dont' ovs yet do you? | 16:42 |
| jrosser | no i dont | 16:42 |
| CeeMac | ah well, never mind then :D | 16:43 |
| jrosser | though the good mr jamesdenton is always a good source of tips | 16:43 |
| CeeMac | unless you can see any issues generally | 16:43 |
| CeeMac | yeah, i'll hit him up later :) | 16:43 |
| CeeMac | not sure if its a port security / matching pairs issue or something more fundamental | 16:44 |
| spotz | Yeah if it's a networking question it's almost always worth waiting to catch jamesdenton | 16:46 |
| *** errr has joined #openstack-ansible | 17:02 | |
| *** errr has quit IRC | 17:02 | |
| *** errr has joined #openstack-ansible | 17:04 | |
| *** udesale has quit IRC | 17:04 | |
| *** errr has quit IRC | 17:05 | |
| noonedeadpunk | #endmeeting | 17:06 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://bit.ly/2SAcGAn" | 17:06 | |
| openstack | Meeting ended Tue Feb 25 17:06:14 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:06 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-02-25-16.06.html | 17:06 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-02-25-16.06.txt | 17:06 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-02-25-16.06.log.html | 17:06 |
| cjloader | could use another review https://review.opendev.org/#/c/709747/ | 17:06 |
| *** errr has joined #openstack-ansible | 17:06 | |
| *** errr has quit IRC | 17:06 | |
| cjloader | passed gate, backport | 17:06 |
| *** errr has joined #openstack-ansible | 17:07 | |
| *** errr has quit IRC | 17:08 | |
| *** errr has joined #openstack-ansible | 17:09 | |
| *** errr has quit IRC | 17:10 | |
| *** errr has joined #openstack-ansible | 17:12 | |
| *** errr has quit IRC | 17:12 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Update ansible to 2.8.8 https://review.opendev.org/659658 | 17:13 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Use buster jobs instead of stretch https://review.opendev.org/707943 | 17:13 |
| *** errr has joined #openstack-ansible | 17:13 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/train: Do not include docs/requirements.txt for functional tests https://review.opendev.org/709795 | 17:13 |
| *** errr has quit IRC | 17:13 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/train: Use /tmp for ansible temporary directory https://review.opendev.org/709796 | 17:14 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/train: Update ansible to 2.8.8 https://review.opendev.org/709797 | 17:14 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/train: Use /tmp for ansible temporary directory https://review.opendev.org/709796 | 17:14 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: [WIP] Centos-8 support https://review.opendev.org/689629 | 17:15 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/train: Update ansible to 2.8.8 https://review.opendev.org/709797 | 17:15 |
| *** errr has joined #openstack-ansible | 17:15 | |
| *** errr has quit IRC | 17:16 | |
| spotz | jrosser: All those ready? | 17:17 |
| jrosser | which ones :) ? | 17:17 |
| *** errr has joined #openstack-ansible | 17:17 | |
| *** errr has quit IRC | 17:18 | |
| *** errr has joined #openstack-ansible | 17:20 | |
| *** errr has quit IRC | 17:21 | |
| *** errr has joined #openstack-ansible | 17:22 | |
| *** errr has quit IRC | 17:23 | |
| *** errr has joined #openstack-ansible | 17:24 | |
| *** errr has quit IRC | 17:25 | |
| cjloader | thx jrosser | 17:26 |
| *** errr has joined #openstack-ansible | 17:27 | |
| *** errr has quit IRC | 17:28 | |
| *** errr has joined #openstack-ansible | 17:29 | |
| *** errr has quit IRC | 17:30 | |
| *** errr has joined #openstack-ansible | 17:31 | |
| *** errr has quit IRC | 17:32 | |
| spotz | jrosser: The ones you just submitted?:) | 17:32 |
| spotz | Or if there's other ones you want reviewed let me know | 17:33 |
| jrosser | theres a stack of backports here https://review.opendev.org/#/q/topic:buster_functional-stable/train+(status:open+OR+status:merged) | 17:33 |
| jrosser | which was those | 17:33 |
| *** errr has joined #openstack-ansible | 17:34 | |
| spotz | ok will go through them | 17:34 |
| *** chandankumar is now known as raukadah | 17:35 | |
| *** errr has quit IRC | 17:35 | |
| *** errr has joined #openstack-ansible | 17:36 | |
| *** errr has quit IRC | 17:37 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: Update octavia configuration documentation https://review.opendev.org/635402 | 17:38 |
| *** errr has joined #openstack-ansible | 17:39 | |
| *** evrardjp has quit IRC | 17:39 | |
| *** evrardjp has joined #openstack-ansible | 17:39 | |
| jamesdenton | hi | 17:39 |
| *** errr has quit IRC | 17:40 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: [WIP] Centos-8 support https://review.opendev.org/689629 | 17:41 |
| *** errr has joined #openstack-ansible | 17:41 | |
| *** errr has quit IRC | 17:42 | |
| *** errr has joined #openstack-ansible | 17:44 | |
| *** errr has quit IRC | 17:44 | |
| *** errr has joined #openstack-ansible | 17:46 | |
| *** errr has quit IRC | 17:47 | |
| *** ivve has quit IRC | 17:47 | |
| *** leo8a has quit IRC | 17:47 | |
| *** errr has joined #openstack-ansible | 17:48 | |
| *** pcaruana has quit IRC | 17:50 | |
| *** errr has joined #openstack-ansible | 17:51 | |
| *** errr has quit IRC | 17:52 | |
| *** errr has joined #openstack-ansible | 17:53 | |
| jamesdenton | CeeMac allowed-address-pairs w/ ovs vxlan may not work | 17:54 |
| *** errr has quit IRC | 17:54 | |
| jamesdenton | or rather, your use case may be broken currently | 17:54 |
| *** errr has joined #openstack-ansible | 17:54 | |
| *** errr has quit IRC | 17:55 | |
| *** errr has joined #openstack-ansible | 17:57 | |
| *** errr has quit IRC | 17:57 | |
| CeeMac | Hi jamesdenton oh, that doesn't sound good. | 17:58 |
| jamesdenton | lemme find the bug we're working with | 17:58 |
| jamesdenton | when you disabled port security, that was on the 'WAN' side of your firewall device, right? | 17:59 |
| *** errr has joined #openstack-ansible | 17:59 | |
| CeeMac | Yes | 17:59 |
| jamesdenton | k | 17:59 |
| *** errr has quit IRC | 18:00 | |
| CeeMac | Well, all interfaces but it's the return traffic from wan that is getting lost | 18:00 |
| CeeMac | I may have turned it back on again I'll need to double check | 18:00 |
| *** pcaruana has joined #openstack-ansible | 18:00 | |
| *** errr has joined #openstack-ansible | 18:01 | |
| *** errr has quit IRC | 18:02 | |
| *** sshnaidm|bbl is now known as sshnaidm | 18:04 | |
| *** errr has joined #openstack-ansible | 18:04 | |
| jamesdenton | right. your issue may be a variation of this: https://review.opendev.org/#/c/601336/ | 18:04 |
| *** DanyC has quit IRC | 18:04 | |
| *** errr has quit IRC | 18:04 | |
| CeeMac | OK, I'll take a look at that later, just in transit currently | 18:05 |
| jamesdenton | for use, we were not able to use a keepalived instance and have a VIP float between instances | 18:05 |
| CeeMac | Ok | 18:05 |
| *** pcaruana has quit IRC | 18:06 | |
| *** errr has joined #openstack-ansible | 18:06 | |
| CeeMac | This is a single device acting as a router so hopefully slightly different | 18:06 |
| *** errr has quit IRC | 18:07 | |
| CeeMac | Is the nat for floating ip performed in the netns or the ovs? | 18:07 |
| jamesdenton | should be the namespace via iptables | 18:08 |
| CeeMac | Ok | 18:08 |
| *** errr has joined #openstack-ansible | 18:08 | |
| CeeMac | I'll have a look at that too | 18:09 |
| jamesdenton | is this floating IP assigned to the 'primary' ip of this firewall device? or some other IP that the firewall has configured, like a secondary address? | 18:09 |
| CeeMac | Primary | 18:09 |
| CeeMac | Not using virtual addresses yet | 18:09 |
| *** errr has quit IRC | 18:09 | |
| CeeMac | Although if the traffic isn't leaving the compute node it probably isn't nat related... | 18:10 |
| *** errr has joined #openstack-ansible | 18:11 | |
| jamesdenton | but you can ping out from the firewall OK? Is it using the float on the outbound? | 18:11 |
| *** strattao has quit IRC | 18:11 | |
| *** errr has quit IRC | 18:11 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: [WIP] Centos-8 support https://review.opendev.org/689629 | 18:11 |
| *** strattao has joined #openstack-ansible | 18:12 | |
| *** strattao has quit IRC | 18:13 | |
| *** strattao has joined #openstack-ansible | 18:13 | |
| *** errr has joined #openstack-ansible | 18:13 | |
| *** errr has joined #openstack-ansible | 18:14 | |
| *** errr has quit IRC | 18:15 | |
| CeeMac | Yes, and I need to double check | 18:16 |
| *** errr has joined #openstack-ansible | 18:17 | |
| *** errr has quit IRC | 18:18 | |
| *** errr has joined #openstack-ansible | 18:19 | |
| *** errr has quit IRC | 18:20 | |
| *** errr has joined #openstack-ansible | 18:21 | |
| *** errr has quit IRC | 18:22 | |
| *** errr has joined #openstack-ansible | 18:22 | |
| *** errr has quit IRC | 18:23 | |
| *** errr has joined #openstack-ansible | 18:25 | |
| *** errr has quit IRC | 18:25 | |
| *** errr has joined #openstack-ansible | 18:27 | |
| *** errr has quit IRC | 18:28 | |
| *** ansmith_ has joined #openstack-ansible | 18:29 | |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Use python3 version by default https://review.opendev.org/709441 | 18:32 |
| *** ansmith has quit IRC | 18:32 | |
| *** fghaas has joined #openstack-ansible | 18:33 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: manila: refactor playbooks https://review.opendev.org/704942 | 18:41 |
| CeeMac | jamesdenton: fyi traffic outbound from instances behind the fw works fine, but I will need to confirm what IP that is translating too. | 18:43 |
| CeeMac | It's return traffic inbound to instances NATd behind the firewall where the session is not completing after the reply leaves the fw wan | 18:44 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Use blockinfile for hosts file generation https://review.opendev.org/709153 | 18:44 |
| *** macz_ has joined #openstack-ansible | 18:51 | |
| *** trident has quit IRC | 18:52 | |
| *** errr has joined #openstack-ansible | 18:53 | |
| jrosser | mnaser: https://zuul.opendev.org/t/openstack/build/fbc083c518f9484e8d3a99afc2fb70f1/log/job-output.txt#10466 | 18:53 |
| jrosser | we are now at the point where the galera repo priorities go wrong | 18:53 |
| *** errr has quit IRC | 18:54 | |
| jrosser | could do with some help with that | 18:54 |
| jrosser | atm the galera role is probably still using centos7 url..... | 18:54 |
| *** errr has joined #openstack-ansible | 18:55 | |
| *** errr has quit IRC | 18:56 | |
| *** errr has joined #openstack-ansible | 18:58 | |
| *** trident has joined #openstack-ansible | 18:58 | |
| *** errr has quit IRC | 18:59 | |
| *** errr has joined #openstack-ansible | 19:00 | |
| *** errr has quit IRC | 19:01 | |
| *** errr has joined #openstack-ansible | 19:03 | |
| *** errr has quit IRC | 19:03 | |
| *** errr has joined #openstack-ansible | 19:05 | |
| *** errr has quit IRC | 19:06 | |
| *** errr has joined #openstack-ansible | 19:07 | |
| *** errr has quit IRC | 19:08 | |
| *** errr has joined #openstack-ansible | 19:10 | |
| *** errr has quit IRC | 19:10 | |
| *** errr has joined #openstack-ansible | 19:11 | |
| *** errr has quit IRC | 19:11 | |
| *** errr has joined #openstack-ansible | 19:13 | |
| *** errr has quit IRC | 19:14 | |
| mnaser | okay I can try and dig jrosser | 19:14 |
| jrosser | I think probably the same has happened in openstack_hosts and other places we define repos | 19:15 |
| jrosser | just they don’t break it | 19:15 |
| *** errr has joined #openstack-ansible | 19:15 | |
| *** errr has quit IRC | 19:16 | |
| *** errr has joined #openstack-ansible | 19:17 | |
| *** errr has quit IRC | 19:18 | |
| *** errr has joined #openstack-ansible | 19:20 | |
| *** errr has quit IRC | 19:20 | |
| *** errr has joined #openstack-ansible | 19:21 | |
| *** errr has quit IRC | 19:21 | |
| *** errr has joined #openstack-ansible | 19:23 | |
| *** errr has joined #openstack-ansible | 19:24 | |
| *** errr has quit IRC | 19:25 | |
| *** errr has joined #openstack-ansible | 19:26 | |
| *** errr has quit IRC | 19:27 | |
| *** sshnaidm has quit IRC | 19:28 | |
| *** sshnaidm has joined #openstack-ansible | 19:28 | |
| *** errr has joined #openstack-ansible | 19:29 | |
| *** errr has quit IRC | 19:29 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic stable/train: Ironic fixes from inspector addition https://review.opendev.org/709747 | 19:30 |
| *** errr has joined #openstack-ansible | 19:31 | |
| *** errr has quit IRC | 19:32 | |
| *** errr has joined #openstack-ansible | 19:34 | |
| *** errr has quit IRC | 19:34 | |
| *** errr has joined #openstack-ansible | 19:35 | |
| *** errr has quit IRC | 19:35 | |
| *** ansmith has joined #openstack-ansible | 19:36 | |
| *** ansmith_ has quit IRC | 19:36 | |
| *** errr has joined #openstack-ansible | 19:37 | |
| *** errr has quit IRC | 19:38 | |
| *** errr has joined #openstack-ansible | 19:39 | |
| *** jtanner has left #openstack-ansible | 19:40 | |
| *** errr has quit IRC | 19:40 | |
| *** errr has joined #openstack-ansible | 19:42 | |
| *** errr has quit IRC | 19:42 | |
| *** errr has joined #openstack-ansible | 19:45 | |
| *** errr has quit IRC | 19:45 | |
| *** errr has joined #openstack-ansible | 19:47 | |
| *** errr has quit IRC | 19:48 | |
| *** errr has joined #openstack-ansible | 19:49 | |
| *** errr has quit IRC | 19:49 | |
| *** mmercer has joined #openstack-ansible | 19:55 | |
| *** errr has joined #openstack-ansible | 20:07 | |
| *** errr has quit IRC | 20:08 | |
| *** errr has joined #openstack-ansible | 20:10 | |
| *** errr has quit IRC | 20:10 | |
| *** errr has joined #openstack-ansible | 20:12 | |
| *** errr has quit IRC | 20:13 | |
| strattao | grrrr, so I've been off trying to replicate our networking issues in our lab environment. Come back to our problem child production region and see that the neutron-dhcp-agent, neutron-l3-agent, and neutron-openvswitch agent shows as "XXX | UP" in the network agent list for one host. However, that network node has all of it's dnsmasq processes still running, but ALL of the dnsmasq processes on the other two | 20:13 |
| strattao | network nodes have stopped. I would have assumed that it would be the other way around. In the logs it looks like there are plenty of rabbit timeouts and ovsdb timeouts, but I'm at my wits end and could use some bigger brains to help me think about what the root cause could be. jrosser, jamesdenton, got any thoughts off the top of your head that would be good to check on? | 20:13 |
| *** errr has joined #openstack-ansible | 20:14 | |
| *** errr has quit IRC | 20:15 | |
| strattao | In the past we have a process of limping along by stopping all neutron services, neutron-ovs-cleanup, then restarting the neutron services. | 20:16 |
| strattao | on all network nodes | 20:16 |
| *** errr has joined #openstack-ansible | 20:17 | |
| strattao | Is there something different/better that I can try? | 20:17 |
| *** errr has quit IRC | 20:17 | |
| *** errr has joined #openstack-ansible | 20:18 | |
| *** errr has quit IRC | 20:19 | |
| *** errr has joined #openstack-ansible | 20:21 | |
| *** errr has joined #openstack-ansible | 20:21 | |
| *** errr has quit IRC | 20:22 | |
| *** errr has joined #openstack-ansible | 20:24 | |
| *** errr has quit IRC | 20:24 | |
| jamesdenton | NTP is OK? | 20:24 |
| jamesdenton | like, are they really dead? Or just, slow to check in | 20:25 |
| strattao | NTP looks good, dates are in sync, chronyc looks happy. I would say those services on the one host aren't really dead, at least a check on that host's neutron.slice says it's active. | 20:31 |
| *** errr has joined #openstack-ansible | 20:41 | |
| *** errr has quit IRC | 20:42 | |
| *** errr has joined #openstack-ansible | 20:47 | |
| *** errr has quit IRC | 20:48 | |
| *** errr has joined #openstack-ansible | 20:49 | |
| *** fghaas has left #openstack-ansible | 20:50 | |
| *** errr has quit IRC | 20:50 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Clean CI virtualenv installation https://review.opendev.org/709720 | 20:51 |
| *** errr has joined #openstack-ansible | 20:51 | |
| *** errr has quit IRC | 20:52 | |
| *** errr has joined #openstack-ansible | 20:54 | |
| *** errr has quit IRC | 20:55 | |
| *** errr has joined #openstack-ansible | 20:56 | |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/709515 | 20:57 |
| *** errr has quit IRC | 20:57 | |
| CeeMac | jamesdenton: I can confirm that pings coming directly from the FW leave the qg interface with the floating ip | 21:00 |
| *** mrda has quit IRC | 21:04 | |
| *** mrda has joined #openstack-ansible | 21:07 | |
| CeeMac | so it seems to be an issue just with responses to inbound requests that pass through the fw WAN -> LAN direction. Curious. | 21:07 |
| CeeMac | I tried MSS wrangling on the WAN and LAN interfaces incase that was causing an issue, I could only get responses to ping against the WAN address for up to 1422 size packets | 21:09 |
| CeeMac | but no dice | 21:09 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests https://review.opendev.org/709500 | 21:11 |
| *** errr has joined #openstack-ansible | 21:12 | |
| *** errr has quit IRC | 21:13 | |
| *** errr has joined #openstack-ansible | 21:14 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.opendev.org/709512 | 21:14 |
| *** errr has quit IRC | 21:15 | |
| *** errr has joined #openstack-ansible | 21:17 | |
| *** errr has quit IRC | 21:17 | |
| *** errr has joined #openstack-ansible | 21:19 | |
| *** errr has quit IRC | 21:20 | |
| *** errr has joined #openstack-ansible | 21:22 | |
| *** errr has quit IRC | 21:23 | |
| *** errr has joined #openstack-ansible | 21:24 | |
| *** errr has joined #openstack-ansible | 21:26 | |
| *** errr has quit IRC | 21:27 | |
| *** errr has joined #openstack-ansible | 21:29 | |
| *** errr has quit IRC | 21:29 | |
| *** errr has joined #openstack-ansible | 21:31 | |
| *** errr has quit IRC | 21:32 | |
| *** errr has joined #openstack-ansible | 21:33 | |
| *** errr has quit IRC | 21:34 | |
| *** errr has joined #openstack-ansible | 21:36 | |
| *** errr has quit IRC | 21:37 | |
| *** ansmith has quit IRC | 21:37 | |
| *** errr has joined #openstack-ansible | 21:38 | |
| *** errr has quit IRC | 21:39 | |
| *** errr has joined #openstack-ansible | 21:40 | |
| *** errr has quit IRC | 21:41 | |
| *** errr has joined #openstack-ansible | 21:43 | |
| *** errr has quit IRC | 21:43 | |
| *** errr has joined #openstack-ansible | 21:44 | |
| *** errr has quit IRC | 21:44 | |
| *** errr has joined #openstack-ansible | 21:46 | |
| *** errr has quit IRC | 21:47 | |
| *** errr has joined #openstack-ansible | 21:48 | |
| *** errr has quit IRC | 21:49 | |
| *** errr has joined #openstack-ansible | 21:51 | |
| *** errr has quit IRC | 21:51 | |
| djhankb | Could someone please help me understand what I did wrong - I re-ran playbooks yesterday and my venvs incremented from '20.0.1' to '20.0.2.dev24' and I'm having issues with Swift/Ceilometer. I was under the impression that if I pulled a tagged release, i.e. 20.0.1, then these would stay static if I re-ran the playbooks again. What | 22:00 |
| djhankb | process/playbook defines which venv versions to pull? is it 'ansible-role-requirements.yml' ? | 22:00 |
| *** errr has joined #openstack-ansible | 22:03 | |
| *** errr has quit IRC | 22:04 | |
| *** errr has joined #openstack-ansible | 22:06 | |
| *** errr has quit IRC | 22:06 | |
| *** errr has joined #openstack-ansible | 22:08 | |
| *** errr has quit IRC | 22:09 | |
| *** DanyC has joined #openstack-ansible | 22:09 | |
| *** errr has joined #openstack-ansible | 22:10 | |
| *** errr has quit IRC | 22:11 | |
| *** ansmith has joined #openstack-ansible | 22:13 | |
| *** errr has joined #openstack-ansible | 22:13 | |
| *** errr has quit IRC | 22:13 | |
| *** errr has joined #openstack-ansible | 22:15 | |
| djhankb | hmm - as I dig further I can totally see that the '20.0.2.dev24' was added to the repo server, although it's not clear what is actually controlling it. | 22:15 |
| *** errr has quit IRC | 22:16 | |
| *** errr has joined #openstack-ansible | 22:18 | |
| *** errr has quit IRC | 22:19 | |
| *** errr has joined #openstack-ansible | 22:20 | |
| *** dave-mccowan has quit IRC | 22:21 | |
| *** errr has quit IRC | 22:21 | |
| *** errr has joined #openstack-ansible | 22:23 | |
| *** errr has quit IRC | 22:23 | |
| *** leo8a has joined #openstack-ansible | 22:25 | |
| *** errr has joined #openstack-ansible | 22:25 | |
| *** errr has quit IRC | 22:26 | |
| *** errr has joined #openstack-ansible | 22:27 | |
| *** errr has quit IRC | 22:28 | |
| *** errr has joined #openstack-ansible | 22:30 | |
| strattao | jamesdenton can you help me understand why if I have 3 network nodes, and the dhcp-agent is in a XXX state, why do all the dnsmasq processes on the other two nodes get killed off? Isn't the point of an HA setup that if one node is having an issue, that one of the other ones will pick up the slack? | 22:30 |
| *** errr has quit IRC | 22:30 | |
| *** errr has joined #openstack-ansible | 22:31 | |
| *** errr has quit IRC | 22:31 | |
| *** errr has joined #openstack-ansible | 22:33 | |
| *** errr has quit IRC | 22:34 | |
| jamesdenton | is the network scheduled to all three nodes? | 22:35 |
| jamesdenton | i would not expect the dnsmasq processes to be killed off on unrelated nodes | 22:35 |
| *** errr has joined #openstack-ansible | 22:35 | |
| *** errr has quit IRC | 22:36 | |
| *** errr has joined #openstack-ansible | 22:38 | |
| strattao | we typically see the exact same number of dnsmasq processes running on all network nodes - I assume that means that the network would be scheduled to all three... | 22:38 |
| *** errr has quit IRC | 22:38 | |
| *** errr has joined #openstack-ansible | 22:40 | |
| strattao | would that not be typical behavior in an HA setup? | 22:40 |
| *** errr has quit IRC | 22:41 | |
| *** errr has joined #openstack-ansible | 22:42 | |
| *** errr has quit IRC | 22:43 | |
| jamesdenton | there is a setting somewhere that controls that, it might be 3, i dunno | 22:44 |
| jamesdenton | but if the agent is down on one, i would not expect to see dnsmasq killed off on the others | 22:44 |
| *** errr has joined #openstack-ansible | 22:44 | |
| *** errr has quit IRC | 22:45 | |
| *** errr has joined #openstack-ansible | 22:47 | |
| *** errr has quit IRC | 22:48 | |
| *** errr has joined #openstack-ansible | 22:49 | |
| *** xakaitetoia1 has quit IRC | 22:49 | |
| *** errr has quit IRC | 22:50 | |
| *** errr has joined #openstack-ansible | 22:51 | |
| *** errr has quit IRC | 22:52 | |
| CeeMac | jamesdenton: just done some additional tcpdumps and looks like the replies are getting lost in between the tap interface and br-tun | 22:53 |
| *** errr has joined #openstack-ansible | 22:54 | |
| *** errr has quit IRC | 22:55 | |
| *** errr has joined #openstack-ansible | 22:56 | |
| *** errr has quit IRC | 22:57 | |
| *** errr has joined #openstack-ansible | 22:57 | |
| *** errr has quit IRC | 22:58 | |
| *** errr has joined #openstack-ansible | 23:00 | |
| *** errr has quit IRC | 23:00 | |
| *** errr has joined #openstack-ansible | 23:02 | |
| *** errr has quit IRC | 23:03 | |
| *** errr has joined #openstack-ansible | 23:04 | |
| *** errr has quit IRC | 23:05 | |
| CeeMac | s/br-tun/patch-tun | 23:05 |
| *** errr has joined #openstack-ansible | 23:06 | |
| *** errr has quit IRC | 23:07 | |
| *** errr has joined #openstack-ansible | 23:12 | |
| *** errr has quit IRC | 23:13 | |
| *** errr has joined #openstack-ansible | 23:14 | |
| *** errr has quit IRC | 23:14 | |
| *** errr has joined #openstack-ansible | 23:34 | |
| *** errr has quit IRC | 23:34 | |
| *** leo8a has quit IRC | 23:35 | |
| *** errr has joined #openstack-ansible | 23:35 | |
| *** errr has joined #openstack-ansible | 23:36 | |
| *** errr has quit IRC | 23:37 | |
| *** errr has joined #openstack-ansible | 23:38 | |
| *** errr has quit IRC | 23:39 | |
| *** errr has joined #openstack-ansible | 23:41 | |
| *** errr has quit IRC | 23:41 | |
| *** errr has joined #openstack-ansible | 23:43 | |
| *** errr has quit IRC | 23:44 | |
| *** errr has joined #openstack-ansible | 23:45 | |
| *** errr has quit IRC | 23:46 | |
| *** errr has joined #openstack-ansible | 23:48 | |
| *** errr has quit IRC | 23:49 | |
| *** errr has joined #openstack-ansible | 23:51 | |
| *** errr has quit IRC | 23:51 | |
| *** tosky has quit IRC | 23:52 | |
| *** errr has joined #openstack-ansible | 23:53 | |
| *** DanyC has quit IRC | 23:54 | |
| *** errr has joined #openstack-ansible | 23:54 | |
| *** errr has joined #openstack-ansible | 23:55 | |
| *** errr has quit IRC | 23:56 | |
| *** errr has joined #openstack-ansible | 23:57 | |
| *** errr has quit IRC | 23:58 | |
| *** errr has joined #openstack-ansible | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!