| *** spatel has quit IRC | 00:04 | |
| *** ChiTo has quit IRC | 00:05 | |
| *** cshen has joined #openstack-ansible | 00:31 | |
| *** jawad_axd has joined #openstack-ansible | 00:34 | |
| *** cshen has quit IRC | 00:36 | |
| *** jawad_axd has quit IRC | 00:38 | |
| *** jawad_axd has joined #openstack-ansible | 01:15 | |
| *** jawad_axd has quit IRC | 01:19 | |
| *** gyee has quit IRC | 01:22 | |
| *** jawad_axd has joined #openstack-ansible | 01:36 | |
| *** jawad_axd has quit IRC | 01:41 | |
| *** watersj has quit IRC | 01:42 | |
| *** jawad_axd has joined #openstack-ansible | 01:57 | |
| *** jawad_axd has quit IRC | 02:01 | |
| *** cshen has joined #openstack-ansible | 02:04 | |
| *** watersj has joined #openstack-ansible | 02:06 | |
| watersj | re-ip br-storage network. Other than updating used_ips, manually edit openstack_inventory.json and update storage_address? | 02:08 |
|---|---|---|
| *** cshen has quit IRC | 02:08 | |
| *** spatel has joined #openstack-ansible | 02:50 | |
| *** spatel has quit IRC | 02:55 | |
| *** shyamb has joined #openstack-ansible | 02:57 | |
| *** spatel has joined #openstack-ansible | 03:01 | |
| *** MickyMan77 has joined #openstack-ansible | 03:04 | |
| *** spatel has quit IRC | 03:06 | |
| *** rh-jelabarre has quit IRC | 03:12 | |
| *** MickyMan77 has quit IRC | 03:13 | |
| *** djhankb has quit IRC | 03:21 | |
| *** djhankb has joined #openstack-ansible | 03:22 | |
| *** MickyMan77 has joined #openstack-ansible | 03:48 | |
| *** MickyMan77 has quit IRC | 03:56 | |
| *** cshen has joined #openstack-ansible | 04:04 | |
| *** cshen has quit IRC | 04:09 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-ansible | 04:33 | |
| *** MickyMan77 has joined #openstack-ansible | 04:33 | |
| *** MickyMan77 has quit IRC | 04:41 | |
| *** shyamb has quit IRC | 04:43 | |
| *** MickyMan77 has joined #openstack-ansible | 05:33 | |
| *** MickyMan77 has quit IRC | 05:41 | |
| *** shyamb has joined #openstack-ansible | 05:42 | |
| *** cshen has joined #openstack-ansible | 06:00 | |
| *** cshen has quit IRC | 06:04 | |
| *** itandops has quit IRC | 06:05 | |
| *** cshen has joined #openstack-ansible | 06:21 | |
| *** cshen has quit IRC | 06:26 | |
| *** jbadiapa has joined #openstack-ansible | 06:26 | |
| *** cshen has joined #openstack-ansible | 06:31 | |
| *** miloa has joined #openstack-ansible | 06:34 | |
| *** cshen has quit IRC | 06:36 | |
| *** BlackFX has joined #openstack-ansible | 06:46 | |
| BlackFX | Hi all, I am writing a custom playbook to add some stuff to my containers - however it errors with ""ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "msg": "apt cache update failed"}" | 06:46 |
| BlackFX | any thoughts | 06:47 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance master: Revert use of _cinder_first_play_host for post install tasks https://review.opendev.org/753227 | 06:48 |
| *** andrewbonney has joined #openstack-ansible | 06:53 | |
| *** BlackFX has quit IRC | 06:53 | |
| *** shyamb has quit IRC | 07:12 | |
| *** cshen has joined #openstack-ansible | 07:18 | |
| *** jawad_axd has joined #openstack-ansible | 07:22 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Revert use of _ceilometer_is_first_play_host for post install tasks https://review.opendev.org/753238 | 07:23 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_glance master: Revert use of _glance_first_play_host for post install tasks https://review.opendev.org/753227 | 07:24 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Revert use of _cinder_first_play_host for post install tasks https://review.opendev.org/753133 | 07:25 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Revert use of _neutron_first_play_host for post install tasks https://review.opendev.org/753133 | 07:27 |
| *** spatel has joined #openstack-ansible | 07:29 | |
| *** spatel has quit IRC | 07:34 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_heat master: Revert use of _heat_first_play_host for post install tasks https://review.opendev.org/753241 | 07:38 |
| *** tosky has joined #openstack-ansible | 07:40 | |
| *** pcaruana has joined #openstack-ansible | 07:43 | |
| *** BlackFX has joined #openstack-ansible | 07:46 | |
| *** pcaruana has quit IRC | 07:47 | |
| *** pcaruana has joined #openstack-ansible | 07:47 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Install current ARA version into the ansible-runtime venv https://review.opendev.org/682587 | 08:03 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Test ansible-base 2.10.1 https://review.opendev.org/752963 | 08:04 |
| admin0 | guilhermesp, "openstack role assignment list --domain heat" returns empty .. OSA version is 20.0.2 | 08:26 |
| *** shyamb has joined #openstack-ansible | 08:39 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Add neutron_keepalived_no_track variable https://review.opendev.org/753252 | 08:43 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Revert use of _cinder_first_play_host for post install tasks https://review.opendev.org/753131 | 08:46 |
| *** itandops has joined #openstack-ansible | 09:00 | |
| *** shyam89 has joined #openstack-ansible | 09:04 | |
| *** shyamb has quit IRC | 09:08 | |
| *** shyam89 has quit IRC | 09:14 | |
| *** itandops has quit IRC | 09:14 | |
| snadge | can i disable q-reports-plugin or what setting will logically give neutron better ability to process rabbit messages without blocking? | 09:21 |
| snadge | its difficult that I have to wait until my evening to ask questions.. and that I start drinking in the evenings :P | 09:22 |
| *** shyamb has joined #openstack-ansible | 09:24 | |
| *** shyam89 has joined #openstack-ansible | 09:27 | |
| *** shyamb has quit IRC | 09:30 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Test ansible-base 2.10.1 https://review.opendev.org/752963 | 09:39 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Handle deprecation of ANSIBLE_COLLECTIONS_PATHS https://review.opendev.org/753272 | 09:39 |
| *** mindthecap has joined #openstack-ansible | 10:03 | |
| *** pcaruana has quit IRC | 10:05 | |
| *** miloa has quit IRC | 10:08 | |
| *** miloa has joined #openstack-ansible | 10:10 | |
| *** shyam89 has quit IRC | 10:11 | |
| snadge | ok so i increased api_workers and rpc_workers from 2 to 8 | 10:12 |
| snadge | restarted neutron-server in the neutron slice | 10:12 |
| snadge | and now only q-reports-plugin is backing up.. it gets to about 13 unacked | 10:13 |
| *** owalsh has quit IRC | 10:15 | |
| *** shyamb has joined #openstack-ansible | 10:15 | |
| *** miloa has quit IRC | 10:16 | |
| *** pcaruana has joined #openstack-ansible | 10:17 | |
| snadge | i wonder if its related to this? https://bugs.launchpad.net/neutron/+bug/1853071 | 10:19 |
| openstack | Launchpad bug 1853071 in neutron "AMQP disconnects, q-reports-plugin queue grows, leading to DBDeadlocks while trying to update agent heartbeats" [High,In progress] - Assigned to Arjun Baindur (abaindur) | 10:19 |
| snadge | im running stable/train | 10:19 |
| snadge | except i dont really understand much about that fix and what it means etc | 10:20 |
| *** spatel has joined #openstack-ansible | 10:21 | |
| *** spatel has quit IRC | 10:25 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Test ansible-base 2.10.1 https://review.opendev.org/752963 | 10:26 |
| *** suryasingh has joined #openstack-ansible | 10:32 | |
| *** ygk_12345 has joined #openstack-ansible | 10:38 | |
| *** shyam89 has joined #openstack-ansible | 10:45 | |
| *** shyamb has quit IRC | 10:48 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/ussuri: Use cinder_service_setup_host for qos and types creation https://review.opendev.org/753016 | 10:49 |
| *** shyam89 has quit IRC | 11:00 | |
| *** mgariepy has quit IRC | 11:22 | |
| *** shyamb has joined #openstack-ansible | 11:25 | |
| *** ygk_12345 has quit IRC | 11:29 | |
| *** ygk_12345 has joined #openstack-ansible | 11:35 | |
| *** owalsh has joined #openstack-ansible | 11:42 | |
| *** shyam89 has joined #openstack-ansible | 11:42 | |
| *** shyamb has quit IRC | 11:45 | |
| *** owalsh has quit IRC | 11:49 | |
| *** shyam89 has quit IRC | 11:54 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Test ansible-base 2.10.1 https://review.opendev.org/752963 | 11:57 |
| *** rh-jelabarre has joined #openstack-ansible | 11:57 | |
| *** rh-jelabarre has quit IRC | 11:58 | |
| *** rh-jelabarre has joined #openstack-ansible | 11:58 | |
| *** owalsh has joined #openstack-ansible | 12:07 | |
| jrosser | noonedeadpunk: can you take a look at this https://review.opendev.org/#/c/752684/ - it breaks T->U upgrades | 12:09 |
| jrosser | not sure why we don't see that in CI tbh | 12:09 |
| noonedeadpunk | in CI it's U->master | 12:10 |
| noonedeadpunk | or you mean why for train it not fails... | 12:11 |
| jrosser | yes, i think so | 12:11 |
| jrosser | i'm wondering if it's because our deploy is S->T->U | 12:12 |
| noonedeadpunk | we kind of not running tempest during first run | 12:12 |
| jrosser | rather than freshly installed T | 12:12 |
| jrosser | ahhhh | 12:12 |
| noonedeadpunk | so when tempest_run: false we jsut don't deploy it I guess | 12:12 |
| jrosser | ah sure, yes | 12:12 |
| jrosser | so this was pretty much the only thing in OSA which we had upgrade trouble with this week | 12:13 |
| noonedeadpunk | oh, cool. upgrade jobs seems doing their stuff, despite we having so many issues with them | 12:14 |
| *** ygk_12345 has quit IRC | 12:14 | |
| jrosser | in cinder we hit this https://review.opendev.org/#/c/748482/ | 12:14 |
| *** ygk_12345 has joined #openstack-ansible | 12:15 | |
| noonedeadpunk | btw have you noticed that centos 8 is the way faster than 7? I mean even upgrade jobs have almost same timig | 12:15 |
| jrosser | but i think next OSA bump will inc;ide that | 12:15 |
| noonedeadpunk | ah, yeah | 12:15 |
| jrosser | need to make sure that train online data migrations are done with that patch applied before upgrade | 12:15 |
| noonedeadpunk | wondering if it's related to https://bugs.launchpad.net/cinder/+bug/1806156 | 12:16 |
| openstack | Launchpad bug 1806156 in Cinder "shared_targets_online_data_migration fails when cinder-volume service not running" [Undecided,Confirmed] | 12:16 |
| *** miloa has joined #openstack-ansible | 12:18 | |
| *** mgariepy has joined #openstack-ansible | 12:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Revert use of _glance_first_play_host for post install tasks https://review.opendev.org/753227 | 12:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Revert use of _heat_first_play_host for post install tasks https://review.opendev.org/753241 | 12:22 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove Centos-7 support https://review.opendev.org/742103 | 12:27 |
| *** mmethot has quit IRC | 12:27 | |
| *** jhesketh has quit IRC | 12:32 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Revert use of _ceilometer_is_first_play_host for post install tasks https://review.opendev.org/753238 | 12:33 |
| *** ygk_12345 has quit IRC | 12:37 | |
| *** ygk_12345 has joined #openstack-ansible | 12:38 | |
| admin0 | guilhermesp, please ping when you are online | 12:51 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-lxc_hosts master: Remove support for Centos-7 https://review.opendev.org/742124 | 12:53 |
| *** mmethot has joined #openstack-ansible | 12:58 | |
| admin0 | anyone seen this before ? only comes when i want to run os-keystone playbook . .. it ssomething to do with set haproxy service state .. https://gist.github.com/a1git/8e7dbc0ce770303081badd91406c4eda | 13:06 |
| admin0 | ansible keystone -m shell -a "uptime" -- works fine | 13:07 |
| arxcruz | guilhermesp: noonedeadpunk can you guys take a look at https://review.opendev.org/#/c/752571/ | 13:14 |
| noonedeadpunk | arxcruz: may I ask why do we rename them at all? | 13:16 |
| noonedeadpunk | I mean we probably should use ither existing one or just drop old one and re-create when needed? | 13:17 |
| jrosser | admin0: looks to me like it tries to connect to the haproxy control socket but fails, i guess start with looking at if haproxy is running or broken? | 13:18 |
| *** jhesketh has joined #openstack-ansible | 13:19 | |
| arxcruz | noonedeadpunk: we are having problems in the update jobs | 13:20 |
| arxcruz | it was hardcoded workspace, and it could be another variable, like "tempest" | 13:20 |
| arxcruz | noonedeadpunk: plus I don't see a problem to list all and rename, the code is more simple than check, return 3 check if return was 3 etc | 13:21 |
| noonedeadpunk | I mean maybe we can just clean workspace at all with removing this dir or smth... | 13:22 |
| jrosser | arxcruz: that change means it's not idempotent any more | 13:22 |
| jrosser | thats why all the 'complication' with return codes previously | 13:22 |
| arxcruz | noonedeadpunk: hmmm, not really, in our case, we would like to collect the logs from both | 13:22 |
| noonedeadpunk | and +1 to jrosser comment | 13:23 |
| *** rfolco|ruck has joined #openstack-ansible | 13:23 | |
| arxcruz | jrosser: noonedeadpunk okay, so what about just get rid of it? | 13:24 |
| arxcruz | or idempotency in this case is important ? | 13:24 |
| noonedeadpunk | I think we want to re-init worspace only in specific cases? | 13:24 |
| arxcruz | we should not re-init workspaces, we should create new ones for each tempest run | 13:25 |
| arxcruz | of course, keep the workspace in case you want to reproduce something | 13:25 |
| noonedeadpunk | I can hardly understand why we need new worspace on each playbook run.... | 13:26 |
| noonedeadpunk | or you need it that way as well | 13:27 |
| noonedeadpunk | and I think it's possible to run tempest in the same workspace several times? | 13:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Add neutron_keepalived_no_track variable https://review.opendev.org/753252 | 13:29 |
| arxcruz | I don't need actually, but in the upgrade/update job we call os_tempest twice | 13:29 |
| arxcruz | noonedeadpunk: yes, you can, but in os_tempest case, we are calling twice, we either need to fix the workspace cleanup script, or detect there's a workspace and not create a new one | 13:30 |
| noonedeadpunk | I'd vote for detecting existing workspaces | 13:30 |
| admin0 | jrosser, haproxy is running fine | 13:31 |
| noonedeadpunk | and using them if possible | 13:31 |
| admin0 | jrosser, also haproxy-install playbook runs fine without any errors | 13:31 |
| arxcruz | noonedeadpunk: this is also wrong 100 if ! tempest workspace list | grep ' workspace ' | grep -w {{ tempest_workspace }}; then | 13:31 |
| arxcruz | the logic | 13:31 |
| arxcruz | if the workspace doesn't have 'workspace' it never enter in this if | 13:31 |
| guilhermesp | admin0: im around nbow | 13:32 |
| guilhermesp | now* | 13:32 |
| noonedeadpunk | I don't say current state is ok:) I just think we can improve it even more and don't recreate on each play run, but only when it's really needed (ie no workspace or we need to recreate it by providing some variable) | 13:32 |
| admin0 | hi guilhermesp .. so i found that there is no role assignment | 13:33 |
| admin0 | i was expecting to run os-mangum install and hoping it will do everythign necessary :) | 13:33 |
| guilhermesp | yeah so, i believe that in heat domain, the user stack_domain_admin should have admin role | 13:33 |
| guilhermesp | yeah im not sure how os_magnum is nowadays, it probably needs a fix :P let try that first then we look for a fix | 13:34 |
| admin0 | i am on 20.0.2 .. so not a recent branch ;) | 13:34 |
| guilhermesp | i do remember a similar issue on older releases... im impressed that this is still happening | 13:37 |
| * guilhermesp looks for an explanation | 13:37 | |
| guilhermesp | so admin0 maybe you can do ` openstack role add --user stack_domain_admin --domain heat admin` and show me the output of `openstack role assignment list --domain heat` | 13:39 |
| admin0 | No user with a name or ID of 'stack_domain_admin' exists. | 13:41 |
| admin0 | i copy/paste it literally | 13:41 |
| guilhermesp | https://github.com/openstack/openstack-ansible-os_heat/commit/2c0323c9efa2d3b95a4d264ea9dc0e5155250ca9 that was my commit when i had a similar issue. Does this task runs successfully on your side admin0 ? | 13:47 |
| guilhermesp | btw are you able to create stacks isolated? does magnum fails without creating a stack? | 13:48 |
| admin0 | stacks are fine | 13:49 |
| admin0 | i can create coe template .. | 13:49 |
| admin0 | but when launching instances from the template, it fails | 13:49 |
| guilhermesp | e coe template doesnt involve creating stacks. | 13:51 |
| guilhermesp | Q: when you ran os_heat, did it ran heat_service_setup fine? | 13:51 |
| guilhermesp | it seems the stack_domain_admin is not added, which tells me the tasks didnt run successfully... | 13:53 |
| jrosser | even the most recent patch to os_heat is related to all this https://github.com/openstack/openstack-ansible-os_heat/commit/9f752d4ffa5e61cbc966c51f7120b96e84374fab | 13:55 |
| guilhermesp | i was looking at this right now jrosser | 13:56 |
| jrosser | so i think running an old version of os_heat and not reviewing later changes may be an issue | 13:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: defaults: use a versioned path for stackviz/tempestconf virtualenv https://review.opendev.org/752684 | 13:57 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_tempest stable/ussuri: defaults: use a versioned path for stackviz/tempestconf virtualenv https://review.opendev.org/753318 | 13:58 |
| *** mindthecap has quit IRC | 13:59 | |
| guilhermesp | admin0: i'd suggest cherry-picking this https://review.opendev.org/#/c/745491/1 and try re-running heat playbooks. then see if the role assignments are in place | 14:01 |
| admin0 | guilhermesp, this is how mine looks -- the 1st line has the git checksum: https://gist.github.com/a1git/6bc8efee02a746c0c174a03a61239cc0 | 14:02 |
| admin0 | guilhermesp, which tag has this .. then i can maybe run an upgrade playbook | 14:03 |
| guilhermesp | admin0: what i'd do is just `git fetch https://review.opendev.org/openstack/openstack-ansible-os_heat refs/changes/91/745491/1 && git cherry-pick FETCH_HEAD` and re-run os-heat-install | 14:07 |
| admin0 | guilhermesp, i do this from inside the /etc/ansible/roles/os_heat folder right ? | 14:07 |
| guilhermesp | yep | 14:07 |
| guilhermesp | if it works i guess you can try running a minor upgrade to the latest train afterwards | 14:08 |
| admin0 | playbook running | 14:09 |
| admin0 | guilhermesp, i now see an output in openstack role assignment list --domain heat line | 14:13 |
| admin0 | will try to relaunch and test | 14:13 |
| guilhermesp | good | 14:14 |
| admin0 | guilhermesp, at least that permission denied error is gone | 14:18 |
| admin0 | its stuck at spawning now .. which i think is nova issue that i will look into | 14:18 |
| guilhermesp | good stuff :) | 14:19 |
| guilhermesp | let me know if i can help and sorry for the delays, pretty busy lately | 14:19 |
| admin0 | guilhermesp, thank you .. one last .. from which tag is this solution available so that i do not have to cherry pick ? | 14:21 |
| guilhermesp | what it seems to me that this fix was the last commit to train branch https://github.com/openstack/openstack-ansible-os_heat/commits/stable/train not sure if that was tagged | 14:22 |
| guilhermesp | cc: noonedeadpunk | 14:22 |
| noonedeadpunk | pretty sure it was | 14:23 |
| noonedeadpunk | 20.1.5 and later | 14:23 |
| guilhermesp | yeah indeed | 14:24 |
| guilhermesp | admin0: https://github.com/openstack/openstack-ansible/blob/23b0ec49da1736bdae0c6f6e35847b0c0ebd5122/ansible-role-requirements.yml#L118 | 14:24 |
| *** ygk_12345 has quit IRC | 14:26 | |
| *** ygk_12345 has joined #openstack-ansible | 14:31 | |
| *** ygk_12345 has quit IRC | 14:31 | |
| *** shyamb has joined #openstack-ansible | 14:35 | |
| admin0 | is 20.0.2 -> 20.1.6 still considered minor ? | 14:37 |
| admin0 | doing it anyway | 14:38 |
| *** miloa has quit IRC | 14:45 | |
| noonedeadpunk | it is | 14:46 |
| noonedeadpunk | but you could also just bump heat specific version | 14:47 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Revert use of _neutron_first_play_host for post install tasks https://review.opendev.org/753133 | 14:47 |
| jrosser | ^ i think we can recheck things now that neuton patch is passed | 14:48 |
| *** cshen has quit IRC | 14:54 | |
| *** jawad_axd has quit IRC | 14:56 | |
| *** shyam89 has joined #openstack-ansible | 14:58 | |
| *** shyamb has quit IRC | 15:00 | |
| *** mgariepy has quit IRC | 15:01 | |
| *** miloa has joined #openstack-ansible | 15:04 | |
| *** spatel has joined #openstack-ansible | 15:07 | |
| admin0 | jrosser, haproxy looks fine .. is there any other way i can check it .. or maybe remove the haproxy files and re-run the haprox role again to fix it ? | 15:12 |
| admin0 | always fails at Set haproxy service state | 15:12 |
| jrosser | you should run the command that the keystone role is trying to use manually | 15:12 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix focal ceph jobs https://review.opendev.org/753376 | 15:13 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix focal ceph jobs https://review.opendev.org/753376 | 15:16 |
| *** partlycloudy has quit IRC | 15:32 | |
| watersj | could someone help me with this error @TASK [Set haproxy service state] http://paste.openstack.org/show/798223/ | 15:32 |
| *** partlycloudy has joined #openstack-ansible | 15:32 | |
| admin0 | watersj, i am facing the same issue while doing a minor upgrade .. and now stuck | 15:43 |
| *** mgariepy has joined #openstack-ansible | 15:43 | |
| admin0 | looks like every playbook from galera to rabbit to keystone wants to execute that step | 15:43 |
| watersj | admin, i'm trying re-ip my test env for storage vlan/bridge device | 15:44 |
| admin0 | well, something is causing this to happen | 15:51 |
| jrosser | you just have to be methodical | 15:52 |
| jrosser | try running it with -vv (v?) to see all of the module parameters | 15:52 |
| jrosser | this is the failing task https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-tasks/haproxy-endpoint-manage.yml | 15:53 |
| jrosser | so then check that /var/run/haproxy.stat seems the same (exists / permissions etc) as the first node which worked | 15:53 |
| *** shyam89 has quit IRC | 15:58 | |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 16:00 |
| openstack | Meeting started Tue Sep 22 16:00:27 2020 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:00 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:00 |
| noonedeadpunk | #topic office hours | 16:00 |
| *** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)" | 16:00 | |
| noonedeadpunk | \o/ | 16:00 |
| jrosser | o/ | 16:00 |
| noonedeadpunk | so, regarding what we want to end up with for V. I think usage of ansible 2.10, mariadb 10.5 and cleaning up centos and suse might be really enough | 16:02 |
| noonedeadpunk | (maybe add new ceph release as well) | 16:02 |
| noonedeadpunk | galera is painfull and broken for centos 8 right now... | 16:03 |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: Fix tempest init logic https://review.opendev.org/753393 | 16:03 |
| noonedeadpunk | (eventually it's broken even on 10.4 so) | 16:03 |
| noonedeadpunk | I hope I will have time for fixing this up during this week | 16:04 |
| noonedeadpunk | and jrosserdid really awesome job for ansible 2.10 | 16:04 |
| noonedeadpunk | another thing to raise is PTG | 16:05 |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: Fix tempest init logic https://review.opendev.org/753393 | 16:05 |
| arxcruz | noonedeadpunk: ^ | 16:05 |
| arxcruz | fully idempotent | 16:05 |
| arxcruz | and working properly | 16:05 |
| arxcruz | jrosser: ^ | 16:06 |
| arxcruz | oh, sorry, didn't new it was meeting time | 16:06 |
| jrosser | noonedeadpunk: i nearly got /etc/openstack_deploy/user-collection-requirements.yml done now | 16:06 |
| jrosser | just made small mistake where ....... json_query ....... is in a ....... collection :( DOH | 16:07 |
| noonedeadpunk | That is so awesome! I actually also start looking into that, so thanks for letting me know) | 16:07 |
| jrosser | need to find an alterntive with ansible built ins, should be OK | 16:07 |
| noonedeadpunk | yeah, I think we will be missing some collections for a while tbh | 16:07 |
| noonedeadpunk | as CI doesn;t cover everything we have | 16:08 |
| *** irclogbot_0 has quit IRC | 16:08 | |
| noonedeadpunk | and the most interesting things are not default actually | 16:08 |
| jrosser | here is my WIP http://paste.openstack.org/show/798226/ | 16:08 |
| noonedeadpunk | (like lets encrypt) | 16:08 |
| *** irclogbot_0 has joined #openstack-ansible | 16:08 | |
| noonedeadpunk | jrosser: doesn't from_yaml load to native list of dict? | 16:10 |
| jrosser | then i would also like to look at pre/post playbok hooks | 16:10 |
| noonedeadpunk | an we can use map? | 16:10 |
| jrosser | yes, i think that json_query is just more readable (came from the user roles stuff) but map will do the same in this case | 16:11 |
| noonedeadpunk | user_collections | map(attribute=name) | list ? | 16:12 |
| noonedeadpunk | can't find json_query collection... | 16:13 |
| noonedeadpunk | but yeah it's neat and in some cases we can hardly find an alternative | 16:14 |
| jrosser | it's in community.general | 16:14 |
| noonedeadpunk | ah, lol, I got the issue | 16:14 |
| noonedeadpunk | haha | 16:14 |
| noonedeadpunk | yeah, that's nice one | 16:15 |
| noonedeadpunk | yeah, cool. | 16:15 |
| jrosser | fg | 16:15 |
| jrosser | argh | 16:15 |
| jrosser | did you have any thoughts on pre/post playbook hooks? | 16:16 |
| admin0 | jrosser, .. even with -vvvv i can't seem to figure out what its exactly trying to do .. https://gist.github.com/a1git/6edc27b686550bf2701f381746a2c944 .. | 16:17 |
| noonedeadpunk | I'm not eactly sure how it would behave if there would be no files - I think we can workaround that | 16:17 |
| noonedeadpunk | but we can do that if needed | 16:17 |
| admin0 | haproxy.stat is present in all 3 controllers with root:root and srw permission | 16:17 |
| noonedeadpunk | I would also probably add some mine customizations there then | 16:18 |
| jrosser | i've not tried anything yet, but was going to look at adding a common_tasks file | 16:19 |
| jrosser | so we can include_tasks that in pre_tasks / post_tasks of the existing playbooks, and pass some var for /etc/openstack_deploy/pre_hooks/<blah>/ | 16:20 |
| noonedeadpunk | I don't have much time this week unfortunatelly, so won't be able to try out | 16:20 |
| jrosser | then use with_fileglob | sort to include playbooks in alphanumeric order | 16:20 |
| jrosser | or something | 16:20 |
| jrosser | :) | 16:20 |
| admin0 | i am going to # all lines there and see what it does | 16:21 |
| noonedeadpunk | I mean would be great to separate them per specific role | 16:21 |
| admin0 | since the services are already setup, its already enabled | 16:21 |
| noonedeadpunk | and eventually so that they were placed on custom location as well.... | 16:21 |
| noonedeadpunk | As for instance I have repo with roles and playbooks, which are separated with some patches per role | 16:22 |
| noonedeadpunk | I ran them after all roles, but if they were included by plays, that would be awesome | 16:22 |
| jrosser | oh yes, custom location, good idea | 16:23 |
| noonedeadpunk | I mean maybe we can define path for pre/post hook with some variable and run them only if it's defined? | 16:23 |
| noonedeadpunk | not sure, needs thinking through | 16:24 |
| jrosser | sure, i'll try to spend some time on this tomorrow for POC | 16:25 |
| watersj | jrosser, thanks was push i needed. haproxy failing start because cinder-api not starting, because, i changed ips. in short my fault. I need to re-ip my storage network, other than changing openstack_inventory.json file, what else should I touch 1st. looks like a lot | 16:25 |
| *** redrobot has quit IRC | 16:25 | |
| jrosser | watersj: that is strange - haproxy should generally always start even if the services are down | 16:25 |
| watersj | cinder, glance atleast | 16:26 |
| noonedeadpunk | jrosser: if you're busy, I can try looking into it next week as well | 16:26 |
| *** johnsom has quit IRC | 16:26 | |
| jrosser | noonedeadpunk: this week is OK - i have reasonable time for OSA stuff | 16:26 |
| jrosser | but like you very concerned with galera :( | 16:26 |
| noonedeadpunk | yeah | 16:26 |
| noonedeadpunk | I just feel it may take _some_ time | 16:26 |
| noonedeadpunk | ok, great | 16:27 |
| *** johnsom has joined #openstack-ansible | 16:27 | |
| jrosser | i can maybe ask andrewbonney to also look at that, it's a good match problem | 16:27 |
| noonedeadpunk | so returning back to ptg, I booked slots for Monday, October 26 from 13 UTC till 17 UTC and October 27 from 21-00 UTC till 23-00 UTC | 16:28 |
| noonedeadpunk | so everybody is warmly welcome | 16:28 |
| noonedeadpunk | as well as input and ideas for W cycle | 16:28 |
| *** SecOpsNinja has joined #openstack-ansible | 16:29 | |
| jrosser | we should start an etherpad | 16:29 |
| noonedeadpunk | Was just writing that | 16:29 |
| jrosser | need to do a bit of planning for finishing V too, decide what we try to tackle | 16:29 |
| noonedeadpunk | https://etherpad.opendev.org/p/osa-wallaby-ptg ? | 16:29 |
| jrosser | bind-to-mgmt still so close :/ | 16:29 |
| *** miloa has quit IRC | 16:29 | |
| noonedeadpunk | we should finish it | 16:30 |
| noonedeadpunk | it's really too close | 16:30 |
| noonedeadpunk | I will fill in with current status of things, so we could decide what should be moved to another cycle | 16:30 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/osa-wallaby-ptg | 16:31 |
| noonedeadpunk | btw, PTG schedule is available here | 16:31 |
| noonedeadpunk | #link https://ethercalc.openstack.org/7xp2pcbh1ncb | 16:31 |
| admin0 | oh well .. now my vip is not coming up :D | 16:34 |
| noonedeadpunk | I guess that's it for me:) | 16:38 |
| noonedeadpunk | There's also a bug that I'd probably want to discuss, but have pretty low energy for that... | 16:38 |
| noonedeadpunk | https://bugs.launchpad.net/bugs/1896589 | 16:39 |
| openstack | Launchpad bug 1896589 in openstack-ansible "main interface renamed from ens160 to eth0 after AIO VM restart" [Undecided,New] | 16:39 |
| jrosser | hmm yes it does rename it in the AIO host prep i think? | 16:41 |
| noonedeadpunk | yep it did | 16:41 |
| noonedeadpunk | and yeah, it's not working after reboots which is annoying | 16:41 |
| noonedeadpunk | so I guess it's confirmed, but don't have aio on hands I can reboot | 16:42 |
| jrosser | i think that mine might boot as eth0 in the first place? | 16:43 |
| noonedeadpunk | um, so mine booted with ens3 and was not changed instantly | 16:43 |
| noonedeadpunk | but yeah, I suggest that after reboot it might change naming | 16:44 |
| jrosser | actually you are right, mine here is ens3 too | 16:44 |
| jrosser | but then again i kind of never reboot them | 16:44 |
| noonedeadpunk | yeah mee too. | 16:44 |
| noonedeadpunk | I got used to that we can't rely on aio, but that's probably not really right | 16:45 |
| noonedeadpunk | and yeah http://paste.openstack.org/show/798227/ | 16:45 |
| noonedeadpunk | tons of configuration errors | 16:45 |
| noonedeadpunk | so ansible-role-systemd_networkd worth having a look | 16:46 |
| jrosser | oh my, yes | 16:46 |
| noonedeadpunk | I guess that's it https://opendev.org/openstack/ansible-role-systemd_networkd/src/branch/master/templates/systemd-link.j2 | 16:46 |
| noonedeadpunk | but again that might depend on systmed version... | 16:47 |
| noonedeadpunk | will also try to pickup that then this week | 16:48 |
| noonedeadpunk | but actually reboot will still be impossible I guess due to the way we mount loop devices... | 16:48 |
| jrosser | i have a feeling we do that with systemd units, and this has come up before | 16:49 |
| jrosser | there was some patches to change the dependancy of those to try to get the ordering correct at a reboot | 16:49 |
| noonedeadpunk | we're doing it here https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml#L37 | 16:50 |
| *** cshen has joined #openstack-ansible | 16:50 | |
| *** cshen has quit IRC | 16:55 | |
| noonedeadpunk | #endmeeting | 16:57 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://bit.ly/2SAcGAn" | 16:57 | |
| openstack | Meeting ended Tue Sep 22 16:57:22 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-09-22-16.00.html | 16:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-09-22-16.00.txt | 16:57 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-09-22-16.00.log.html | 16:57 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Add user defined collections https://review.opendev.org/753411 | 17:10 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Add user defined collections https://review.opendev.org/753411 | 17:12 |
| jrosser | noonedeadpunk: ^ first stab at user collections, I still need to do some docs | 17:12 |
| *** andrewbonney has quit IRC | 17:13 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Add user defined collections https://review.opendev.org/753411 | 17:13 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Add user defined collections https://review.opendev.org/753411 | 17:14 |
| admin0 | hi ... how do I force rebuild an infra ? i am trying to upgrade from 20.0.2 to 20.1.6 .. the playbooks for setup hosts and infra ran without issues .. on keystone, it fails at this step .. /openstack/venvs/keystone-20.1.6/bin/keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone .. and when I check, there is no keystone_manage in that 20.1.6 venv .. but it is in the old 20.0.2 venv | 17:26 |
| admin0 | is this a result of a bad venv build ? | 17:26 |
| *** itandops has joined #openstack-ansible | 17:32 | |
| *** SecOpsNinja has left #openstack-ansible | 18:20 | |
| *** gyee has joined #openstack-ansible | 18:34 | |
| *** suryasingh has quit IRC | 18:49 | |
| *** cshen has joined #openstack-ansible | 18:50 | |
| *** cshen has quit IRC | 18:55 | |
| *** BlackFX has quit IRC | 18:58 | |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: Fix tempest init logic https://review.opendev.org/753393 | 19:24 |
| *** MickyMan77 has joined #openstack-ansible | 20:07 | |
| *** MickyMan77 has quit IRC | 20:15 | |
| *** cshen has joined #openstack-ansible | 20:18 | |
| *** cshen has quit IRC | 20:23 | |
| *** ianychoi_ has joined #openstack-ansible | 20:23 | |
| *** ianychoi has quit IRC | 20:27 | |
| *** rfolco|ruck has quit IRC | 20:59 | |
| *** tosky_ has joined #openstack-ansible | 21:09 | |
| *** tosky is now known as Guest33200 | 21:10 | |
| *** tosky_ is now known as tosky | 21:10 | |
| *** MickyMan77 has joined #openstack-ansible | 21:12 | |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: Fix tempest init logic https://review.opendev.org/753393 | 21:15 |
| *** MickyMan77 has quit IRC | 21:21 | |
| *** spatel has quit IRC | 21:59 | |
| *** spatel has joined #openstack-ansible | 21:59 | |
| *** spatel has quit IRC | 21:59 | |
| *** strattao has quit IRC | 22:04 | |
| *** mwhahaha has quit IRC | 22:04 | |
| *** mubix has quit IRC | 22:04 | |
| *** mwhahaha has joined #openstack-ansible | 22:04 | |
| *** gregwork has quit IRC | 22:04 | |
| *** gundalow has quit IRC | 22:04 | |
| *** nicolasbock has quit IRC | 22:04 | |
| *** strattao has joined #openstack-ansible | 22:05 | |
| *** mubix has joined #openstack-ansible | 22:05 | |
| *** gundalow has joined #openstack-ansible | 22:05 | |
| *** nicolasbock has joined #openstack-ansible | 22:05 | |
| *** gouthamr has quit IRC | 22:06 | |
| *** mnaser has quit IRC | 22:06 | |
| *** gregwork has joined #openstack-ansible | 22:09 | |
| *** gouthamr has joined #openstack-ansible | 22:09 | |
| *** gouthamr has quit IRC | 22:09 | |
| *** mnaser has joined #openstack-ansible | 22:10 | |
| *** gouthamr has joined #openstack-ansible | 22:11 | |
| *** cshen has joined #openstack-ansible | 22:18 | |
| *** cshen has quit IRC | 22:23 | |
| *** tosky has quit IRC | 22:48 | |
| *** jbadiapa has quit IRC | 22:57 | |
| *** gokhani has quit IRC | 23:09 | |
| *** yasemind has quit IRC | 23:09 | |
| *** spatel has joined #openstack-ansible | 23:44 | |
| *** spatel has quit IRC | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!