| *** macza_ has quit IRC | 00:22 | |
| *** ansmith has joined #openstack-ansible | 00:30 | |
| *** ansmith has quit IRC | 00:36 | |
| *** tosky has quit IRC | 00:43 | |
| *** macza has joined #openstack-ansible | 01:20 | |
| *** macza has joined #openstack-ansible | 01:21 | |
| *** trident has quit IRC | 01:36 | |
| *** markvoelker has quit IRC | 01:50 | |
| *** hwoarang has quit IRC | 01:54 | |
| *** hwoarang has joined #openstack-ansible | 01:56 | |
| *** gyee has quit IRC | 02:00 | |
| *** DanyC has joined #openstack-ansible | 02:12 | |
| *** DanyC has quit IRC | 02:16 | |
| *** Bhujay has joined #openstack-ansible | 02:40 | |
| *** Bhujay has quit IRC | 02:47 | |
| *** Bhujay has joined #openstack-ansible | 02:47 | |
| *** Bhujay has quit IRC | 02:48 | |
| *** Bhujay has joined #openstack-ansible | 02:49 | |
| *** Bhujay has quit IRC | 02:50 | |
| *** Bhujay has joined #openstack-ansible | 02:50 | |
| *** Bhujay has quit IRC | 02:51 | |
| *** Bhujay has joined #openstack-ansible | 02:52 | |
| *** Bhujay has quit IRC | 02:53 | |
| *** Bhujay has joined #openstack-ansible | 02:53 | |
| *** Bhujay has joined #openstack-ansible | 02:55 | |
| *** Bhujay has quit IRC | 02:56 | |
| *** Bhujay has joined #openstack-ansible | 02:56 | |
| *** Bhujay has quit IRC | 02:57 | |
| *** Bhujay has joined #openstack-ansible | 02:58 | |
| *** trident has joined #openstack-ansible | 02:58 | |
| *** Bhujay has quit IRC | 02:59 | |
| *** Bhujay has joined #openstack-ansible | 02:59 | |
| *** Bhujay has quit IRC | 03:00 | |
| *** Bhujay has joined #openstack-ansible | 03:01 | |
| *** Bhujay has quit IRC | 03:02 | |
| *** Bhujay has joined #openstack-ansible | 03:02 | |
| *** Bhujay has quit IRC | 03:03 | |
| *** Bhujay has joined #openstack-ansible | 03:04 | |
| *** Bhujay has quit IRC | 03:05 | |
| *** Bhujay has joined #openstack-ansible | 03:05 | |
| *** Bhujay has quit IRC | 03:06 | |
| *** Bhujay has joined #openstack-ansible | 03:07 | |
| *** Bhujay has quit IRC | 03:08 | |
| *** Bhujay has joined #openstack-ansible | 03:08 | |
| *** Bhujay has quit IRC | 03:09 | |
| *** Bhujay has joined #openstack-ansible | 03:10 | |
| *** nurdie has joined #openstack-ansible | 03:12 | |
| *** Bhujay has quit IRC | 03:15 | |
| *** hwoarang has quit IRC | 03:18 | |
| *** hwoarang has joined #openstack-ansible | 03:20 | |
| *** Bhujay has joined #openstack-ansible | 03:41 | |
| *** nurdie has quit IRC | 03:43 | |
| *** nurdie has joined #openstack-ansible | 03:43 | |
| *** Bhujay has quit IRC | 03:46 | |
| *** nurdie has quit IRC | 03:47 | |
| *** Bhujay has joined #openstack-ansible | 05:02 | |
| *** macza_ has joined #openstack-ansible | 05:03 | |
| *** macza has quit IRC | 05:06 | |
| *** shyamb has joined #openstack-ansible | 05:34 | |
| *** chkumar|vanished is now known as chandankumare | 05:42 | |
| *** chandankumare is now known as chandankumar | 05:42 | |
| *** nurdie has joined #openstack-ansible | 05:54 | |
| *** DanyC has joined #openstack-ansible | 06:12 | |
| *** DanyC has quit IRC | 06:16 | |
| *** mgagne has quit IRC | 06:33 | |
| *** mgagne has joined #openstack-ansible | 06:40 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add local facts for template creations https://review.openstack.org/626411 | 06:46 |
|---|---|---|
| *** hwoarang has quit IRC | 06:47 | |
| *** hwoarang has joined #openstack-ansible | 06:51 | |
| *** nurdie has quit IRC | 06:56 | |
| *** nurdie has joined #openstack-ansible | 06:56 | |
| *** nurdie has quit IRC | 07:01 | |
| *** shyamb has quit IRC | 07:12 | |
| *** mathlin has joined #openstack-ansible | 07:12 | |
| *** Bhujay has quit IRC | 07:13 | |
| *** DanyC has joined #openstack-ansible | 07:20 | |
| *** Bhujay has joined #openstack-ansible | 07:23 | |
| *** Bhujay has quit IRC | 07:24 | |
| *** shyamb has joined #openstack-ansible | 07:24 | |
| *** Bhujay has joined #openstack-ansible | 07:24 | |
| *** pcaruana has joined #openstack-ansible | 07:25 | |
| *** Bhujay has quit IRC | 07:26 | |
| *** Bhujay has joined #openstack-ansible | 07:26 | |
| *** Bhujay has quit IRC | 07:27 | |
| *** Bhujay has joined #openstack-ansible | 07:27 | |
| *** Bhujay has quit IRC | 07:29 | |
| *** Bhujay has joined #openstack-ansible | 07:29 | |
| *** Bhujay has quit IRC | 07:30 | |
| *** shyamb has quit IRC | 07:30 | |
| *** Bhujay has joined #openstack-ansible | 07:30 | |
| *** Bhujay has quit IRC | 07:32 | |
| *** Bhujay has joined #openstack-ansible | 07:32 | |
| *** Bhujay has quit IRC | 07:33 | |
| *** Bhujay has joined #openstack-ansible | 07:33 | |
| *** shyamb has joined #openstack-ansible | 07:34 | |
| *** Bhujay has quit IRC | 07:35 | |
| *** Bhujay has joined #openstack-ansible | 07:35 | |
| *** Bhujay has quit IRC | 07:36 | |
| *** Bhujay has joined #openstack-ansible | 07:36 | |
| *** Bhujay has quit IRC | 07:38 | |
| *** Bhujay has joined #openstack-ansible | 07:38 | |
| *** Bhujay has quit IRC | 07:39 | |
| *** Bhujay has joined #openstack-ansible | 07:39 | |
| *** Bhujay has quit IRC | 07:41 | |
| *** Bhujay has joined #openstack-ansible | 07:41 | |
| *** Bhujay has quit IRC | 07:42 | |
| *** Bhujay has joined #openstack-ansible | 07:42 | |
| *** Bhujay has quit IRC | 07:44 | |
| *** Bhujay has joined #openstack-ansible | 07:44 | |
| *** Bhujay has quit IRC | 07:45 | |
| *** Bhujay has joined #openstack-ansible | 07:45 | |
| *** Bhujay has quit IRC | 07:47 | |
| *** Bhujay has joined #openstack-ansible | 07:47 | |
| *** Bhujay has quit IRC | 07:48 | |
| *** Bhujay has joined #openstack-ansible | 07:48 | |
| *** gkadam has joined #openstack-ansible | 07:49 | |
| *** Bhujay has quit IRC | 07:50 | |
| *** Bhujay has joined #openstack-ansible | 07:50 | |
| *** Bhujay has quit IRC | 07:51 | |
| *** Bhujay has joined #openstack-ansible | 07:51 | |
| *** Bhujay has quit IRC | 07:53 | |
| *** Bhujay has joined #openstack-ansible | 07:53 | |
| *** Bhujay has quit IRC | 07:54 | |
| *** Bhujay has joined #openstack-ansible | 07:54 | |
| *** Bhujay has quit IRC | 07:56 | |
| *** Bhujay has joined #openstack-ansible | 07:56 | |
| *** Bhujay has quit IRC | 07:57 | |
| *** Bhujay has joined #openstack-ansible | 07:57 | |
| *** Bhujay has quit IRC | 07:59 | |
| *** Bhujay has joined #openstack-ansible | 07:59 | |
| *** macza has joined #openstack-ansible | 07:59 | |
| *** Bhujay has quit IRC | 08:00 | |
| *** Bhujay has joined #openstack-ansible | 08:00 | |
| *** Bhujay has joined #openstack-ansible | 08:02 | |
| *** Bhujay has quit IRC | 08:03 | |
| *** Bhujay has joined #openstack-ansible | 08:03 | |
| *** DanyC has quit IRC | 08:04 | |
| *** macza has quit IRC | 08:04 | |
| *** Bhujay has quit IRC | 08:05 | |
| *** Bhujay has joined #openstack-ansible | 08:05 | |
| fnpanic | good morning | 08:05 |
| *** shyamb has quit IRC | 08:06 | |
| *** Bhujay has quit IRC | 08:06 | |
| *** kopecmartin|off is now known as kopecmartin | 08:06 | |
| *** Bhujay has joined #openstack-ansible | 08:07 | |
| *** hamzaachi has joined #openstack-ansible | 08:19 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Update the nspawn unit services https://review.openstack.org/626513 | 08:20 |
| *** tosky has joined #openstack-ansible | 08:24 | |
| *** thuydang has joined #openstack-ansible | 08:39 | |
| *** thuydang has quit IRC | 08:43 | |
| *** Bhujay has quit IRC | 08:46 | |
| *** nurdie has joined #openstack-ansible | 08:55 | |
| *** markvoelker has joined #openstack-ansible | 08:56 | |
| *** nurdie has quit IRC | 09:00 | |
| admin0 | \o | 09:05 |
| *** shyamb has joined #openstack-ansible | 09:09 | |
| *** DanyC has joined #openstack-ansible | 09:13 | |
| *** DanyC has quit IRC | 09:14 | |
| *** DanyC has joined #openstack-ansible | 09:15 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Update the systemd-nspawn template for legacy systemd environments https://review.openstack.org/626525 | 09:17 |
| *** shardy has joined #openstack-ansible | 09:24 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: WIP: Enable passing nspawn jobs https://review.openstack.org/608433 | 09:30 |
| chandankumar | cloudnull: jrosser Hello | 09:31 |
| *** thuydang has joined #openstack-ansible | 09:31 | |
| chandankumar | cloudnull: jrosser does libxml2-devel suse issue got fixed? http://logs.openstack.org/92/626092/1/check/openstack-ansible-functional-opensuse-423/f9d5c24/job-output.txt.gz#_2018-12-20_06_11_49_539790 | 09:31 |
| chandankumar | cloudnull: I am seeing one more issue http://logs.openstack.org/28/625828/3/check/openstack-ansible-functional-centos-7/aa8d5c4/job-output.txt.gz#_2018-12-20_06_29_15_747502 | 09:32 |
| chandankumar | cloudnull: faailing at os_neutron check fo available offline migration | 09:32 |
| chandankumar | please have a look, thanks :-) | 09:32 |
| *** shyamb has quit IRC | 09:42 | |
| *** thuydang has quit IRC | 09:45 | |
| cloudnull | o/ chandankumar idk if its fixed yet , there was work with the upstream suse community to get it resolved, however idk where that is at the moment. -cc evrardjp | 09:48 |
| *** shyamb has joined #openstack-ansible | 09:52 | |
| *** mkuf_ has joined #openstack-ansible | 09:56 | |
| *** mkuf has quit IRC | 09:59 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Update the nspawn unit services https://review.openstack.org/626513 | 09:59 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Update the nspawn unit services https://review.openstack.org/626513 | 10:12 |
| *** Bhujay has joined #openstack-ansible | 10:23 | |
| chandankumar | cloudnull: ack thanks :-) | 10:24 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Update the systemd-nspawn template for legacy systemd environments https://review.openstack.org/626525 | 10:25 |
| *** thuydang has joined #openstack-ansible | 10:45 | |
| *** mkuf has joined #openstack-ansible | 10:46 | |
| *** mkuf_ has quit IRC | 10:49 | |
| *** shyamb has quit IRC | 11:02 | |
| *** kopecmartin is now known as kopecmartin|off | 11:02 | |
| odyssey4me | o/ folks, how go things? | 11:03 |
| *** shyamb has joined #openstack-ansible | 11:04 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Update the systemd-nspawn template for legacy systemd environments https://review.openstack.org/626525 | 11:10 |
| jamesdenton | looks like cloudnull is still adjusting to timezones | 11:12 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Enable passing nspawn jobs https://review.openstack.org/608433 | 11:13 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Update the nspawn unit services https://review.openstack.org/626513 | 11:22 |
| odyssey4me | chandankumar that neutron failure looks like something may have merged upstream which has broken things | 11:31 |
| *** ansmith has joined #openstack-ansible | 11:32 | |
| odyssey4me | chandankumar hmm, maybe not - that task may not have something, but it's odd that it's only showing up now | 11:34 |
| odyssey4me | nope, it seems that something in the interface has changed - the return code was 1, but the message that used to be there wasn't | 11:39 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Update the systemd-nspawn template for legacy systemd environments https://review.openstack.org/626525 | 11:39 |
| odyssey4me | https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/neutron_db_setup.yml#L59-L69 | 11:39 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Update the systemd-nspawn template for legacy systemd environments https://review.openstack.org/626525 | 11:40 |
| *** dave-mccowan has quit IRC | 11:42 | |
| *** shyamb has quit IRC | 11:46 | |
| *** udesale has joined #openstack-ansible | 11:56 | |
| odyssey4me | evrardjp any news on resolving that package issue for suse? otherwise it looks like we'll have to making it non-voting I guess, given it's been two days | 11:59 |
| odyssey4me | cloudnull or jrosser you around? I could do with a review for https://review.openstack.org/#/c/625929/ | 12:02 |
| *** ansmith has quit IRC | 12:04 | |
| *** udesale has quit IRC | 12:11 | |
| *** udesale has joined #openstack-ansible | 12:12 | |
| *** udesale has quit IRC | 12:15 | |
| *** udesale has joined #openstack-ansible | 12:16 | |
| *** udesale has quit IRC | 12:21 | |
| *** udesale has joined #openstack-ansible | 12:21 | |
| jamesdenton | odyssey4me what is the process for adding upgrade notes and deprecating vars or processes in upcoming releases? | 12:25 |
| jamesdenton | odyssey4me Also, i will work on testing your neutron migration patches today | 12:27 |
| odyssey4me | jamesdenton great, thanks :) | 12:29 |
| odyssey4me | jamesdenton if OSA vars are deprecated/removed/changed - add a release note with reno | 12:30 |
| jamesdenton | is there an example of that? | 12:30 |
| odyssey4me | jamesdenton https://docs.openstack.org/openstack-ansible/rocky/contributor/code-rules.html#release-notes | 12:31 |
| jamesdenton | perfect, i'll take it from there. thank you | 12:31 |
| odyssey4me | here's an example from one recently merged: https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/83affc627fd2132bb5a65c4b1f5a07a9f95a7998/releasenotes/notes/rabbitmq-gpg-keys-042a47164265ea40.yaml | 12:31 |
| jamesdenton | ahh great, that's helpful | 12:34 |
| jamesdenton | and just push the note with the rest of the commit? | 12:34 |
| odyssey4me | jamesdenton yep - reno will generate the hash for you, those have to be unique so it's best to let it do it for you | 12:35 |
| odyssey4me | then just wipe out the sections from the releasenote which you aren't using and poopulate the content - then just add it to the patch making the changes | 12:36 |
| jamesdenton | k | 12:37 |
| *** mkuf_ has joined #openstack-ansible | 12:42 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: [DOC] fix some typos and fix some stuff https://review.openstack.org/625897 | 12:43 |
| *** mkuf has quit IRC | 12:45 | |
| odyssey4me | chandankumar FYI I've proposed https://review.openstack.org/626565 to resolve the neutron issue. | 12:47 |
| *** cshen has joined #openstack-ansible | 12:48 | |
| odyssey4me | for now, though, while we wait for that - I'll submit patches to pin the SHA we use to something that works | 12:48 |
| *** udesale has quit IRC | 12:49 | |
| *** udesale has joined #openstack-ansible | 12:50 | |
| *** udesale has quit IRC | 12:52 | |
| *** udesale has joined #openstack-ansible | 12:52 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Pin neutron SHA until migrations output is fixed https://review.openstack.org/626567 | 12:53 |
| *** udesale has quit IRC | 12:53 | |
| *** udesale has joined #openstack-ansible | 12:53 | |
| *** udesale has quit IRC | 12:54 | |
| *** udesale has joined #openstack-ansible | 12:55 | |
| chandankumar | odyssey4me: thanks :-) | 12:56 |
| chandankumar | odyssey4me: can we propose one of the openstack-ansible jobs against neutron projects to find issue early? | 12:56 |
| odyssey4me | chandankumar not just yet - I'm busy reworking our jobs to make that possible | 12:56 |
| *** markvoelker has quit IRC | 13:12 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: When upgrading, only stop the service if it exists https://review.openstack.org/625929 | 13:13 |
| *** fnpanic has quit IRC | 13:13 | |
| *** rodolof has quit IRC | 13:20 | |
| *** pmannidi has joined #openstack-ansible | 13:20 | |
| *** rodolof has joined #openstack-ansible | 13:22 | |
| *** mkuf has joined #openstack-ansible | 13:24 | |
| *** gkadam has quit IRC | 13:25 | |
| *** ansmith has joined #openstack-ansible | 13:26 | |
| *** mkuf_ has quit IRC | 13:27 | |
| *** Bhujay has quit IRC | 13:31 | |
| *** cshen has quit IRC | 13:34 | |
| *** fnpanic has joined #openstack-ansible | 13:37 | |
| *** cshen has joined #openstack-ansible | 13:55 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Clean-up and simplify the major upgrade https://review.openstack.org/624773 | 13:58 |
| *** shyamb has joined #openstack-ansible | 13:58 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Clean-up and simplify the major upgrade https://review.openstack.org/624773 | 13:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Clean-up and simplify the major upgrade https://review.openstack.org/624773 | 14:01 |
| *** pmannidi is now known as pmannidi_1 | 14:02 | |
| *** udesale has quit IRC | 14:05 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/queens: Correct failure when stopping cluster state is ignored https://review.openstack.org/626586 | 14:05 |
| *** udesale has joined #openstack-ansible | 14:06 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/rocky: Correct failure when stopping cluster state is ignored https://review.openstack.org/626587 | 14:06 |
| *** mathlin has quit IRC | 14:16 | |
| *** pmannidi_1 is now known as pmannidi | 14:17 | |
| *** pmannidi is now known as pmannidi_1 | 14:17 | |
| *** pmannidi_1 is now known as pmannidi | 14:21 | |
| *** pmannidi is now known as pmannidi_1 | 14:21 | |
| openstackgerrit | Frank Kloeker proposed openstack/openstack-ansible-haproxy_server master: Add feature Letsencrypt SSL certification https://review.openstack.org/586774 | 14:28 |
| *** shyamb has quit IRC | 14:32 | |
| *** jamesdenton has quit IRC | 14:45 | |
| *** jamesden_ has joined #openstack-ansible | 14:45 | |
| *** shyamb has joined #openstack-ansible | 14:49 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Add automated migration of neutron agents to bare metal https://review.openstack.org/625331 | 14:52 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Add automated migration of neutron agents to bare metal https://review.openstack.org/625331 | 14:53 |
| openstackgerrit | Paul Martin proposed openstack/openstack-ansible-os_tempest master: Configuration drives don't appear to work on aarch64+kvm. https://review.openstack.org/626592 | 14:55 |
| openstackgerrit | Paul Martin proposed openstack/openstack-ansible master: Spice console doesn't work on aarch64+kvm. https://review.openstack.org/626593 | 14:57 |
| *** cshen has quit IRC | 14:57 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Add automated migration of neutron agents to bare metal https://review.openstack.org/625331 | 14:57 |
| *** mkuf_ has joined #openstack-ansible | 14:58 | |
| *** jackivanov has joined #openstack-ansible | 14:58 | |
| odyssey4me | cloudnull sorry - did a minor tweak there, no more changes incoming - jamesden_ it worked for me, although I didn't test whether connectivity was maintained - only that the new agents were implemented, then the old ones removed | 14:58 |
| cloudnull | LGTM - very nicely done btw. great seeing that all being cleaned up ! | 14:59 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-plugins master: Update provider networks plugin to support multiple network mappings https://review.openstack.org/626594 | 15:00 |
| *** jamesden_ is now known as jamesdenton | 15:00 | |
| jamesdenton | odyssey4me I'm deploying MNAIO now, but it's slow going | 15:00 |
| *** mkuf has quit IRC | 15:01 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Enable passing nspawn jobs https://review.openstack.org/608433 | 15:09 |
| *** shyamb has quit IRC | 15:10 | |
| *** cshen has joined #openstack-ansible | 15:12 | |
| *** kaiokmo has quit IRC | 15:16 | |
| cloudnull | mnaser ^ with those couple of updates, nspwan with cent7 seems to be humming right along. | 15:25 |
| cloudnull | guilhermesp, mnaser if you all have a moment to give that spin it'd be appreciated. | 15:25 |
| mnaser | cloudnull: woohoooo | 15:27 |
| mnaser | i will have a look | 15:27 |
| mnaser | i can take on splitting that into a few digestable bits (or we can keep it as one bigger patch too) | 15:27 |
| cloudnull | whatever's easiest :) | 15:28 |
| cloudnull | I have it all running an AIO, on my local test cluster . i started it all with `./scripts/gate-check-commit.sh aio_nspawn` | 15:30 |
| cloudnull | https://i.snag.gy/ANYcIT.jpg | 15:30 |
| cloudnull | might still run into "some other failure" however there doesn't seem to be any connectivity issues and the playbooks have been running fine on this latest iteration. | 15:31 |
| *** evrardjp_ has joined #openstack-ansible | 15:31 | |
| *** nurdie has joined #openstack-ansible | 15:32 | |
| *** nurdie_ has joined #openstack-ansible | 15:33 | |
| odyssey4me | cloudnull is that all including the tempest tests? | 15:36 |
| cloudnull | its not run tempest yet. | 15:36 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Provide support for network interface mappings without override https://review.openstack.org/623620 | 15:37 |
| *** evrardjp has quit IRC | 15:37 | |
| *** nurdie has quit IRC | 15:37 | |
| jamesdenton | ^^ that may be ambitious | 15:37 |
| cloudnull | ooooo. nice! | 15:38 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-plugins master: Update provider networks plugin to support multiple network mappings https://review.openstack.org/626594 | 15:41 |
| *** evrardjp has joined #openstack-ansible | 15:41 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Set OpenSUSE jobs to non-voting https://review.openstack.org/626601 | 15:54 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Revert "Set OpenSUSE jobs to non-voting" https://review.openstack.org/626602 | 15:55 |
| odyssey4me | mnaser cloudnull evrardjp to get us unblocked https://review.openstack.org/626601 | 15:56 |
| cloudnull | done | 15:56 |
| *** nurdie_ has quit IRC | 16:00 | |
| *** nurdie has joined #openstack-ansible | 16:01 | |
| *** thuydang has quit IRC | 16:03 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server stable/rocky: Correct failure when stopping cluster state is ignored https://review.openstack.org/626587 | 16:04 |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_container_create master: Update the systemd-nspawn template for legacy systemd environments https://review.openstack.org/626525 | 16:05 |
| jrosser | odyssey4me: I’m around-ish ping me when that unblocking patch passes for more votes | 16:06 |
| odyssey4me | jrosser yeah, I think it'll need the patch to neutron to merge first - that's in the gate, but will take some time still | 16:07 |
| openstackgerrit | Merged openstack/openstack-ansible master: Set Keepalived script_user to root https://review.openstack.org/621125 | 16:08 |
| *** udesale has quit IRC | 16:11 | |
| CeeMac | afternoon channel | 16:17 |
| CeeMac | i dont suppose there happens to be any linux/lxc network experts in the house? | 16:18 |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_container_create master: Add bugs url link to README https://review.openstack.org/622873 | 16:25 |
| *** mkuf_ has quit IRC | 16:27 | |
| *** mkuf_ has joined #openstack-ansible | 16:27 | |
| *** mkuf has joined #openstack-ansible | 16:28 | |
| *** mkuf_ has quit IRC | 16:32 | |
| *** DanyC has quit IRC | 16:34 | |
| *** macza has joined #openstack-ansible | 16:34 | |
| *** DanyC has joined #openstack-ansible | 16:35 | |
| cloudnull | CeeMac - im not an "expert" but happy to help. whats up ? | 16:35 |
| *** DanyC has quit IRC | 16:35 | |
| *** gyee has joined #openstack-ansible | 16:36 | |
| CeeMac | hi cloudnull | 16:37 |
| guilhermesp | nice cloudnull I will take a look soon :D | 16:38 |
| CeeMac | i'm having some issues with arp between lxc containers and across hosts | 16:38 |
| CeeMac | as far as I can see, everything should be fine. Its obviously not though | 16:38 |
| CeeMac | so. there are 2 hosts running lxc, one for the non-os infrastructure roles, the other for the os-controller roles | 16:39 |
| CeeMac | setup-hosts ran fine | 16:39 |
| CeeMac | setup-infrastructure fails on utility container with "no route to host" | 16:40 |
| CeeMac | I can ping the utility container from both hosts | 16:40 |
| CeeMac | I can ping each host from the utility container | 16:40 |
| CeeMac | if i try and ping the vip of the haproxy, or the default gateway, i get nothing | 16:40 |
| CeeMac | i've installed net-tools in the container and the arp table is not being populated for these addresses for some reason | 16:41 |
| CeeMac | I've double/triple checked the interfaces configuration for bond/bridge/vlan config on the hosts | 16:41 |
| CeeMac | the container veth is attached to the bridge | 16:41 |
| CeeMac | if i run tcpdump against the bond i can see the vlan tags | 16:42 |
| CeeMac | face->wall | 16:42 |
| CeeMac | help? | 16:42 |
| CeeMac | oh, and as far as I can tell, IPtables is forwarding with accept by default on the hosts | 16:43 |
| CeeMac | the only things i can think ive done different on these hosts than my last build is upgrade the kernel, not sure if that would make a difference | 16:44 |
| jrosser | This feels like it might be a mismatch between a netmask and the actual addresses in the subnet | 16:46 |
| jrosser | I would look at the routing tables to try to understand the “no route to host” error first | 16:47 |
| CeeMac | "no route to host" is a result of destination host unreachable i thinkg | 16:47 |
| CeeMac | they're on the same lan, so wouldn't use route lookup | 16:48 |
| jrosser | The routing table while say which subnets live down each interface | 16:49 |
| CeeMac | default gateway is configured with /24, br-mgmt is configured with /24 on host, container has /24 showing in ifconfig | 16:49 |
| jrosser | And this is also matching the network definitions in your openstack_user_config? They are /22 in the examples? | 16:50 |
| CeeMac | same in routing tables, just double checked | 16:50 |
| CeeMac | yes, just double-checked | 16:50 |
| *** cshen has quit IRC | 16:51 | |
| CeeMac | /24 | 16:51 |
| CeeMac | for both container network and haproxy_keepalived_internal_vip_cidr | 16:51 |
| CeeMac | interestingly, the arp table on the router that hosts the DG and on the hosts themselves have the arp table populated for the container address | 16:52 |
| CeeMac | which makes sense, given they can both ping it. So, its responding to "who has" arp requests | 16:53 |
| CeeMac | so either its own "who has" requests are getting lost, or the "i have" replies are getting lost | 16:54 |
| *** rodolof has quit IRC | 16:55 | |
| *** DanyC has joined #openstack-ansible | 16:58 | |
| CeeMac | ok, now i know i've gone insane | 16:58 |
| CeeMac | installed tcpdump in the utility container, set it away, went to ping to/from the container in another session and its working>! | 16:59 |
| CeeMac | and I can now ping from the router too | 16:59 |
| CeeMac | fml | 16:59 |
| CeeMac | oh well, thanks jrosser, maybe you have the technicial effect :) | 16:59 |
| CeeMac | *technician | 16:59 |
| *** rodolof has joined #openstack-ansible | 17:01 | |
| CeeMac | hmm, actually, maybe it was an arp timeout | 17:01 |
| jrosser | Is this all bare metal? | 17:02 |
| CeeMac | vmware | 17:02 |
| jrosser | Aaaahhhhhh | 17:02 |
| CeeMac | i think actually, it was me | 17:02 |
| jrosser | jamesdenton: ^ can you advise here? | 17:02 |
| CeeMac | i had fudged the interface config on the controller node, fixed it and rolled back to snapshot, then rebooted that server | 17:02 |
| *** DanyC has quit IRC | 17:02 | |
| CeeMac | but didn't reboot the inf node | 17:03 |
| CeeMac | so it probably had a cached incomplete arp entry | 17:03 |
| CeeMac | maybe | 17:03 |
| jrosser | So up and running now? | 17:03 |
| CeeMac | i'm rolling back again and going to reboot both and see if it still works | 17:03 |
| CeeMac | yeah | 17:03 |
| CeeMac | mental | 17:03 |
| jrosser | Cool :) | 17:03 |
| CeeMac | i had issues with the vmware config on the portgroup for vlan trunk initially which drove me mad for days | 17:03 |
| CeeMac | but i resolved that | 17:04 |
| jrosser | There are folks here running labs on esxi so hopefully you can get some specific pointers if needed | 17:04 |
| CeeMac | cool | 17:05 |
| CeeMac | in an semirelated note | 17:05 |
| CeeMac | how does one get around the haproxy not starting on boot issue, i'm sure i've seen a forum post about it but can't find it again | 17:06 |
| *** priteau has joined #openstack-ansible | 17:10 | |
| CeeMac | so, the problem comes back after a reboot | 17:13 |
| *** cshen has joined #openstack-ansible | 17:13 | |
| jamesdenton | jrosser CeeMac everything good now? | 17:13 |
| cloudnull | CeeMac sorry, was getting a bite. | 17:13 |
| cloudnull | looks like jrosser got you going? | 17:13 |
| *** electrofelix has quit IRC | 17:14 | |
| CeeMac | jamesdenton, i thought so, but its not again | 17:14 |
| CeeMac | cloudnull, no problem, feel free to jump in! | 17:14 |
| jrosser | jamesdenton: any tips for VMware? | 17:14 |
| cloudnull | is this a vmware issue ? | 17:16 |
| cloudnull | spatel did a bunch of vmware things | 17:16 |
| jamesdenton | On ESXi, you'll want to enable 'Promiscuous Mode and MAC Address Changes' on the virtual switch. I think i enabled forged transmits, too. | 17:17 |
| cloudnull | I remember he had to do some config to allow spoofing ? | 17:17 |
| cloudnull | what jamesdenton said | 17:17 |
| cloudnull | :) | 17:17 |
| *** thuydang has joined #openstack-ansible | 17:17 | |
| jamesdenton | and when you create the port groups, those values should be inherited. | 17:17 |
| CeeMac | jamesdenton, yep, all 3 options set to accept (i'd forgotten to do promisc initially a few days ago) | 17:18 |
| *** rodolof has quit IRC | 17:18 | |
| *** cshen has quit IRC | 17:18 | |
| CeeMac | the vlan is passing up ok and I can ping the container from the other host br-mgmt address | 17:18 |
| CeeMac | which i couldn't when the dvport config was incorrect | 17:19 |
| jamesdenton | and i also have a port group named TRUNK associated with the vSwitch, with VLAN ID set to 4095. Allows me to tag in the VM and interfaces with VLANs on physnet | 17:19 |
| jamesdenton | what is the issue you're seeing now? | 17:19 |
| CeeMac | i can ping the container from the 2 hosts | 17:21 |
| CeeMac | i can ping the 2 hosts from the container | 17:21 |
| CeeMac | i cant ping the container from the router, but i get an arp entry with the correct mac | 17:21 |
| CeeMac | i cant ping the router from the container and I don't get an arp entry | 17:22 |
| CeeMac | so arps aren't working 100% from the container | 17:22 |
| CeeMac | except eventually they did while i was troubleshooting earlier | 17:22 |
| CeeMac | after i'd installed net-tools and tcpdump in the container to try and capture arp traffic | 17:22 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Set OpenSUSE jobs to non-voting https://review.openstack.org/626601 | 17:23 |
| jamesdenton | and the router is in the same network as the containers? i assume this is a typical deploy, right? eth0 on a 10dot? | 17:23 |
| *** cshen has joined #openstack-ansible | 17:24 | |
| CeeMac | yeah, all on the same vlan | 17:24 |
| jamesdenton | And you're sure there's no kind of arp conflict? Like, one of the containers didn't grab the IP of the router? | 17:25 |
| *** rodolof has joined #openstack-ansible | 17:26 | |
| jamesdenton | is this a single container behaving this way? Or all containers? | 17:27 |
| CeeMac | let me double check again | 17:27 |
| CeeMac | ok, and now its working again | 17:28 |
| CeeMac | in that container, but not in a different one | 17:28 |
| jamesdenton | grab the MAC that's showing up in the arp cache | 17:28 |
| CeeMac | on? | 17:28 |
| jamesdenton | in the container. | 17:28 |
| jamesdenton | it should be the router's mac, right? | 17:29 |
| CeeMac | should be | 17:29 |
| CeeMac | its the same mac as is showing in the arp table of the hosts | 17:30 |
| CeeMac | its an irb on the switch, just trying to remember how to track the mac of an irb | 17:30 |
| CeeMac | got it, yep its the mac of the irb that is showing in the container now | 17:32 |
| CeeMac | thats a weird delay though | 17:32 |
| jamesdenton | it wasn't resolving for a bit, and now it is? | 17:32 |
| CeeMac | now its working in the second container | 17:32 |
| CeeMac | let me try another onw | 17:32 |
| CeeMac | correct | 17:32 |
| jamesdenton | k | 17:32 |
| CeeMac | could it be a kernel version issue? | 17:32 |
| CeeMac | i updated to 4.15.0-42-generic for this run through | 17:33 |
| jamesdenton | i doubt it. what version? | 17:33 |
| jamesdenton | i'm on that, too. no issues here | 17:33 |
| CeeMac | ah, good to know | 17:33 |
| *** hamzaachi has quit IRC | 17:33 | |
| CeeMac | 3rd container on same host can't ping (destination host unreachable) | 17:34 |
| jamesdenton | You might try performing a pcap on that container net vlan on the infra node and see if those arp requests are making their way out | 17:34 |
| jrosser | With a rebooted vm do the container veth rewire themselves by magic? | 17:36 |
| jamesdenton | can you share your 'brctl show' from the infra node and 'ip addr show' from the container? | 17:36 |
| CeeMac | 3rd container working now, i'm running tcpdump on another session, let me try another container real quick | 17:36 |
| CeeMac | jrosser, yes after reboot they all reattach to the bridge | 17:36 |
| CeeMac | the arp request is coming out of the container | 17:37 |
| CeeMac | root@e6-00-cistg01-memcached-container-96e6028b:~# ip addr show eth1 | 17:38 |
| CeeMac | 30: eth1@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | 17:38 |
| CeeMac | link/ether 00:16:3e:d6:b4:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0 | 17:38 |
| CeeMac | inet 10.1.211.20/24 brd 10.1.211.255 scope global eth1 | 17:38 |
| CeeMac | valid_lft forever preferred_lft forever | 17:38 |
| CeeMac | inet6 fe80::216:3eff:fed6:b40b/64 scope link | 17:38 |
| CeeMac | valid_lft forever preferred_lft forever | 17:38 |
| CeeMac | bridge name bridge id STP enabled interfaces | 17:38 |
| CeeMac | br-mgmt 8000.005056b32e1a no 20385a94_eth1 | 17:38 |
| CeeMac | 2ef06fa6_eth1 | 17:38 |
| CeeMac | 31f0b760_eth1 | 17:38 |
| CeeMac | 37d62ba3_eth1 | 17:38 |
| CeeMac | 4507a3e5_eth1 | 17:38 |
| CeeMac | 6ab83fa5_eth1 | 17:38 |
| jamesdenton | careful not to paste here. try paste.openstack.org or pasted.tech | 17:38 |
| CeeMac | 96e6028b_eth1 | 17:38 |
| CeeMac | ab9f07ff_eth1 | 17:38 |
| CeeMac | bcffa313_eth1 | 17:38 |
| CeeMac | bond0.211 | 17:38 |
| CeeMac | ebac552e_eth1 | 17:38 |
| CeeMac | yeah | 17:38 |
| CeeMac | my bad | 17:39 |
| CeeMac | brain dead now, been a long day | 17:39 |
| jamesdenton | it's all good | 17:39 |
| CeeMac | http://paste.openstack.org/show/737791/ | 17:39 |
| CeeMac | now that container is working | 17:40 |
| CeeMac | so the delay isnt as long, but still there | 17:40 |
| jamesdenton | i recently saw something related to that container nat network. jrosser is eth0 going away, or is it a deployer decision? | 17:40 |
| jrosser | My http proxy test scenario removes it | 17:41 |
| jrosser | It’s kinda up to the deployer how the default route works I guess | 17:41 |
| jamesdenton | it sets up an http proxy config inside the container | 17:41 |
| jamesdenton | ? | 17:41 |
| CeeMac | from the outside it looks like arp is running fine http://paste.openstack.org/show/737792/ | 17:42 |
| jrosser | jamesdenton: accessible via br-mgmt | 17:42 |
| *** thuydang has quit IRC | 17:43 | |
| jamesdenton | CeeMac Can you start that cap again and try to ping from another container and see what the delay is? | 17:43 |
| jrosser | jamesdenton: see https://review.openstack.org/#/c/625523/ that’s an example with proxy and no container eth0 | 17:43 |
| CeeMac | sure | 17:43 |
| jamesdenton | jrosser right on, dude. thanks | 17:44 |
| CeeMac | the arp request/reply is showing straight away on the cap | 17:45 |
| CeeMac | but the reply is still not happening from the container | 17:45 |
| jamesdenton | where are you performing the cap? which interface? | 17:45 |
| CeeMac | oh,wait 27 pings before reply | 17:45 |
| CeeMac | bond0 on the controller | 17:46 |
| CeeMac | the containers are on the inf node that i'm testing currently | 17:46 |
| jamesdenton | CeeMac Similar issue to you: https://lists.linuxfoundation.org/pipermail/bridge/2017-December/010830.html | 17:53 |
| jamesdenton | CeeMac the infra nodes... those are hosted on ESX? If so, what does your bond configuration look like? | 17:58 |
| CeeMac | similar, but slightly different, although I'm still reading through it. If I connect to a container and check eth1 mac, it is not showing in brctl showmacs br-mgmt | 17:59 |
| *** cshen has quit IRC | 17:59 | |
| CeeMac | if i then ping the host IP and check showmacs again, it is there | 17:59 |
| CeeMac | if i then try ping DG, it doesn't work | 17:59 |
| jamesdenton | ok, the former behavior is what i would expect. those entries in mac table would timeout and show up upon traffic. Do you happen to see DG mac in that table? | 18:01 |
| CeeMac | jamesdenton http://paste.openstack.org/show/737794/ | 18:01 |
| CeeMac | DG mac is in br-mgmt yep | 18:01 |
| CeeMac | actually, reading through that mailtrail some more, when i first saw the eth1 mac it was on port1, now that I can ping it is on port 9 | 18:04 |
| CeeMac | so could be similar issue, i'll keep reading | 18:04 |
| *** cshen has joined #openstack-ansible | 18:07 | |
| jamesdenton | "brctl showstp br-mgmt" should show interfaces and respective port numbers. will help figure out 1 vs 9 | 18:08 |
| CeeMac | i have another weird going on | 18:11 |
| CeeMac | when i was looking at the cap for the arp | 18:11 |
| CeeMac | i saw a request for the DG IP coming from an IP that I don't have configured, and the reply went back to that iP | 18:12 |
| *** cshen has quit IRC | 18:12 | |
| CeeMac | I can ping the IP from the switch, tracing the MAC back takes me to br-mgmt on the controller | 18:12 |
| CeeMac | i dont see that ip in ifconfig or attached to a container | 18:13 |
| jamesdenton | And the IP isn't a VIP or something? | 18:14 |
| CeeMac | nope | 18:15 |
| CeeMac | didn't run setup-infrastructure again yet | 18:16 |
| CeeMac | its showing in the arp table against br-mgmt | 18:16 |
| CeeMac | showmacs on br-mgmt shows it against port 1, which is bond0.211 | 18:16 |
| CeeMac | which doesn't have an IP assigned | 18:16 |
| CeeMac | and its not a mac of any of the physical interfaces on that node | 18:19 |
| cjloader | is it possible in an aio, to create duplicate service containers (for example, two barbican containers to test haproxy)? | 18:19 |
| jamesdenton | it may be a byproduct of promisc mode, not sure | 18:19 |
| jamesdenton | cjloader i think you can define affinity? | 18:20 |
| cjloader | okay | 18:20 |
| CeeMac | my brain is dead and i'm way overdue going home. thanks for the help, think i'll kick it in for tonight and try again in the morning | 18:20 |
| jamesdenton | CeeMac Can you verify Forged Transmits is also enabled? | 18:20 |
| jamesdenton | wish i had a better answer for you. i'm sure it's something dumb. | 18:20 |
| CeeMac | this is something perculiar to this build. I got past this stage on my previous builds fine | 18:21 |
| CeeMac | yeah, it always is | 18:21 |
| CeeMac | yep, promisc, mac changes and forged transmits all allowed | 18:21 |
| *** ianychoi has quit IRC | 18:22 | |
| CeeMac | "vlan trunking" type with specific vlan list in the trunk range | 18:22 |
| jamesdenton | k, thanks. the only other thing i would consider would be breaking the bond and working with only one interface - ens160 or ens192. Just to eliminate any funkiness there. You can also try to set max age to 0 and see if that improves things, but it's not ideal | 18:23 |
| CeeMac | might try vmotioning them to different hosts | 18:23 |
| CeeMac | tomorrow see if the issue persists | 18:23 |
| jamesdenton | wish i could replicate, it would help | 18:23 |
| CeeMac | yeah | 18:23 |
| jamesdenton | cjloader https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/app-advanced-config-affinity.html -- I think you can define 1 host and set affinity to 2 or 3 | 18:23 |
| CeeMac | i'm using pretty much the same interfaces config as my previous build, thats the weird thing. Built the VMs from same ISO and using same dvport | 18:24 |
| cjloader | jamesdenton: thx! | 18:24 |
| CeeMac | only difference is kernel upgrade and git branch, but cant see git branch affecting container networking? | 18:25 |
| *** cshen has joined #openstack-ansible | 18:25 | |
| CeeMac | that mail trail doesn't get anywhere either sadly :( | 18:27 |
| jamesdenton | yeah, i know. Found another one, but the fix involved patches or a NIC swap. Neither are concrete IMO | 18:28 |
| CeeMac | yeah | 18:28 |
| jamesdenton | ARP gremlins | 18:28 |
| CeeMac | little buggers | 18:28 |
| CeeMac | hmm | 18:28 |
| jamesdenton | There might be something to the way the MAC moves from one port to another. | 18:28 |
| CeeMac | the VMs are on different hosts than the last build, wonder if that is just coincidence | 18:29 |
| CeeMac | not sure if thats just the ens port proxy-arping for the veth though? | 18:29 |
| jamesdenton | what did port 1 correspond to? and port 9? | 18:29 |
| CeeMac | port 1 was bond0.211, port 9 was the veth for the container | 18:29 |
| jamesdenton | ok, so i would expect to see it always on port 9 | 18:29 |
| CeeMac | *eth1 | 18:29 |
| CeeMac | presuming there is no proxy-arping going on, then yes | 18:30 |
| *** cshen has quit IRC | 18:30 | |
| jamesdenton | And when it wasn't working, the MAC was on eth1. Then aged out (maybe), then showed up on port 9 and worked. | 18:30 |
| CeeMac | thats the presumption i'm working on | 18:30 |
| CeeMac | let me test again from a different container | 18:30 |
| jamesdenton | CeeMac Can you also just check this, for grins? [grep -rnw /proc/sys/net/ipv4/conf/*/proxy_arp -e '1'] | 18:35 |
| CeeMac | sure | 18:35 |
| *** nurdie has quit IRC | 18:36 | |
| CeeMac | from the container? | 18:36 |
| jamesdenton | on the infra node | 18:36 |
| CeeMac | not showing anything yet | 18:36 |
| *** nurdie has joined #openstack-ansible | 18:37 | |
| jamesdenton | k. i would expect no results | 18:37 |
| CeeMac | ah, thats good then? | 18:37 |
| jamesdenton | it's something. heh | 18:37 |
| CeeMac | hah | 18:37 |
| CeeMac | just waiting for the arp reply to come back | 18:37 |
| jamesdenton | just wanted to verify if proxy arp was enabled on anything | 18:37 |
| *** shardy has quit IRC | 18:37 | |
| jamesdenton | while you do that, run the 'brctl showmacs br-mgmt' and see if you see the eth1 mac associated with a port | 18:38 |
| jamesdenton | and which port it is | 18:38 |
| CeeMac | so, eth1 is associated with port1 until it gets a reply, then switched to the port for its veth | 18:39 |
| CeeMac | 7 in this instance | 18:39 |
| CeeMac | once the reply starts coming back. I was a little slow running the tcpdump so cant see which device got the reply this time | 18:40 |
| jamesdenton | i suspect it's the other way around... becomes associated with port 7 then gets the response | 18:40 |
| *** nurdie has quit IRC | 18:41 | |
| CeeMac | yeah | 18:41 |
| CeeMac | ok, so i've jumped on a new container | 18:41 |
| CeeMac | no entry in show macs | 18:41 |
| jamesdenton | out of curiousity... what NICs are you using? | 18:41 |
| jamesdenton | physical nics | 18:41 |
| CeeMac | ping remote host, its in showmacs for port 1 | 18:42 |
| CeeMac | ping local host it goes to port 3 | 18:42 |
| CeeMac | ping dg it goes back to 1 | 18:43 |
| jamesdenton | at that point the remote starts working, too? | 18:43 |
| jamesdenton | oh interesting | 18:43 |
| *** cshen has joined #openstack-ansible | 18:43 | |
| CeeMac | nope | 18:43 |
| CeeMac | still timeout on the dg and remote | 18:43 |
| CeeMac | ens160 and ens224 in bond0 | 18:44 |
| CeeMac | trying to emulate how it would be in physical set up (nic1 and nic3) | 18:44 |
| jamesdenton | sure. I assume these correspond to two port groups and two vSwitches with different physical nics? | 18:45 |
| CeeMac | its a dvswitch | 18:45 |
| jamesdenton | ahh ok. | 18:46 |
| CeeMac | both vmnics connect to the same dvport | 18:46 |
| CeeMac | interestingly, i see the same 3 arp requests just before a reply as that thread, first request is 28 length then next 2 46 | 18:46 |
| CeeMac | no idea if that is significant | 18:46 |
| CeeMac | maybe i'll bounce that VM to a different host before I go | 18:47 |
| CeeMac | looking back through the pcap, it looks like the reply was going back a good few times before the ping kicked it | 18:51 |
| CeeMac | *in | 18:51 |
| *** thuydang has joined #openstack-ansible | 18:51 | |
| CeeMac | hmm, wonder if having a few snapshots on the vm would cause issues | 18:51 |
| CeeMac | i've seen it do weird rtt's when there are snapshots around for a while | 18:52 |
| jamesdenton | hmm, i wouldn't think so? Unless there's another VM out there with the same MACs? | 18:52 |
| CeeMac | nope | 18:52 |
| CeeMac | all unique | 18:52 |
| *** aludwar has quit IRC | 18:53 | |
| CeeMac | ok that was weird, i lost networking to the host for a while after the vmotion there | 18:53 |
| CeeMac | ok, issue persists after vmotion | 18:54 |
| CeeMac | i'm done. | 18:54 |
| CeeMac | will try looking at it with fresh eyes in the morning | 18:54 |
| jamesdenton | understood. get some rest and maybe tomorrow it will come to you | 18:55 |
| CeeMac | thanks again for sticking with me | 18:55 |
| jamesdenton | sure. I've seen this type of behavior before, but not under the same circumstances | 18:55 |
| CeeMac | if you have any epiphanies drop me a dm | 18:55 |
| jamesdenton | Last time i encountered something like this was with an Intel X710 NIC and it could only be corrected with latest drivers | 18:55 |
| CeeMac | on the ESXi or on the VM? | 18:56 |
| jamesdenton | No ESXi in this case. | 18:56 |
| CeeMac | right | 18:56 |
| CeeMac | vmxnet3 adapters on these vms | 18:56 |
| jamesdenton | but the MAC reflection issue was causing drops like we're seeing here | 18:56 |
| jamesdenton | yeah | 18:56 |
| CeeMac | using open vmware tools | 18:57 |
| CeeMac | actually | 18:57 |
| jamesdenton | get some rest | 18:57 |
| CeeMac | theres a thing | 18:57 |
| CeeMac | on my last build i used a template where i'd removed open-vmware tools and installed the official vmware tools | 18:57 |
| CeeMac | but then because i ran into neutron issues i thought id go all stock for this next build | 18:57 |
| CeeMac | i'll take a snapshot and try that in the morning | 18:58 |
| jamesdenton | i don't ever install the vmware toold | 18:58 |
| jamesdenton | sure. keep me posted | 18:58 |
| CeeMac | you mean the offical ones, or at all? | 18:58 |
| jamesdenton | at all. | 18:58 |
| CeeMac | ubuntu installs open-vmware tools automagically | 18:58 |
| jamesdenton | Oh, well in that case, maybe i should retract that statement | 18:59 |
| nsmeds | this isn't exactly OSA specific - however defaults (and capp) is set by OSA so figured I'd ask in here. Our MySQL `max_connections` set at 1600, but appears that we sit ~1k threads connected at most times | 18:59 |
| CeeMac | i certainly got prompted to upgrade them when i ran apt update && apt dist-upgrade | 18:59 |
| nsmeds | anyone else had to increase max_connections beyond the 1600? | 18:59 |
| jamesdenton | yeah ok, open-vm-tools is installed. | 18:59 |
| CeeMac | anyway, thats a chat for the morrow | 19:00 |
| jamesdenton | yep, see ya | 19:00 |
| CeeMac | good to know | 19:00 |
| CeeMac | later | 19:00 |
| *** aludwar has joined #openstack-ansible | 19:04 | |
| redkrieg | Hi all, is it possible to target nova_virt_type based on the individual host instead of making it global for the entire cloud? I'd like a mix of kvm and lxd nodes. | 19:14 |
| jamesdenton | redkrieg you might check this out: https://docs.openstack.org/openstack-ansible/rocky/reference/configuration/using-overrides.html#overriding-openstack-configuration-defaults | 19:17 |
| redkrieg | jamesdenton: perfect, thank you. | 19:17 |
| jamesdenton | CeeMac This may be relevant - not sure in your dvSwitch scenario: https://communities.vmware.com/message/1507261. Since you're running a dvSwitch, can't you implement failover at the ESX level, rather than the VM? I think if you break the bond on the VM and stick to one interface you may have better luck | 19:21 |
| jamesdenton | Is the gate still busted on [os_neutron : Check for available offline migrations]? | 19:25 |
| guilhermesp | cloudnull: thoughts? http://paste.openstack.org/show/737800/ | 19:26 |
| *** thuydang has quit IRC | 19:46 | |
| *** thuydang has joined #openstack-ansible | 19:47 | |
| *** hamzaachi has joined #openstack-ansible | 19:49 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Spice console doesn't work on aarch64+kvm. https://review.openstack.org/626593 | 19:50 |
| *** thuydang has quit IRC | 19:51 | |
| *** thuydang has joined #openstack-ansible | 19:51 | |
| *** jackivanov has quit IRC | 19:54 | |
| *** jackivanov has joined #openstack-ansible | 19:54 | |
| *** pcaruana has quit IRC | 19:57 | |
| *** jackivanov has quit IRC | 19:59 | |
| *** DanyC has joined #openstack-ansible | 20:01 | |
| *** CeeMac has quit IRC | 20:05 | |
| *** jackivanov has joined #openstack-ansible | 20:06 | |
| *** jackivanov has quit IRC | 20:09 | |
| *** jackivanov has joined #openstack-ansible | 20:09 | |
| *** DanyC has quit IRC | 20:10 | |
| *** jackivanov has quit IRC | 20:14 | |
| *** DanyC has joined #openstack-ansible | 20:20 | |
| *** jackivanov has joined #openstack-ansible | 20:22 | |
| *** jackivanov has quit IRC | 20:23 | |
| *** jackivanov has joined #openstack-ansible | 20:24 | |
| *** jackivanov has quit IRC | 20:26 | |
| *** jackivanov has joined #openstack-ansible | 20:27 | |
| openstackgerrit | Justin Alford proposed openstack/openstack-ansible-lxc_hosts master: Increase LXC container shutdown delay https://review.openstack.org/626716 | 20:30 |
| *** jackivanov has quit IRC | 20:31 | |
| *** DanyC has quit IRC | 20:36 | |
| *** jlalford has joined #openstack-ansible | 20:50 | |
| *** radeks has joined #openstack-ansible | 20:56 | |
| openstackgerrit | Justin Alford proposed openstack/openstack-ansible-lxc_hosts master: Increasing LXC container default shutdown delay https://review.openstack.org/625998 | 21:11 |
| *** jlalford_ has joined #openstack-ansible | 21:13 | |
| *** jlalford has quit IRC | 21:13 | |
| *** thuydang has quit IRC | 21:18 | |
| *** thuydang has joined #openstack-ansible | 21:21 | |
| *** ansmith has quit IRC | 21:26 | |
| *** sawblade6 has joined #openstack-ansible | 21:36 | |
| *** hamzaachi has quit IRC | 21:44 | |
| *** hamzaachi has joined #openstack-ansible | 21:45 | |
| *** priteau has quit IRC | 21:56 | |
| *** DanyC has joined #openstack-ansible | 22:01 | |
| *** thuydang has quit IRC | 22:03 | |
| *** ivve has quit IRC | 22:08 | |
| *** ivve has joined #openstack-ansible | 22:20 | |
| *** hamzaachi has quit IRC | 22:28 | |
| *** pmannidi_1 has quit IRC | 22:29 | |
| *** nurdie has joined #openstack-ansible | 22:35 | |
| *** nurdie has quit IRC | 22:40 | |
| *** radeks has quit IRC | 22:40 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Fence options before casting to json https://review.openstack.org/626763 | 23:00 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Fence options before casting to json https://review.openstack.org/626763 | 23:07 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Fence options before casting to json https://review.openstack.org/626763 | 23:35 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Fence options before casting to json https://review.openstack.org/626763 | 23:38 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
