| *** spatel has joined #openstack-ansible | 02:05 | |
| *** spatel has quit IRC | 02:51 | |
| *** spatel has joined #openstack-ansible | 02:54 | |
| *** spatel has quit IRC | 03:22 | |
| *** opendevreview has quit IRC | 05:26 | |
| *** rpittau|afk is now known as rpittau | 06:54 | |
| *** tosky has joined #openstack-ansible | 07:03 | |
| *** frenzy_friday is now known as anbanerj|rover | 07:04 | |
| *** andrewbonney has joined #openstack-ansible | 07:20 | |
| *** luksky has joined #openstack-ansible | 07:29 | |
| *** opendevreview has joined #openstack-ansible | 08:58 | |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Use PKI role to install CA certificates https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/790431 | 08:58 |
|---|---|---|
| *** jnamdar has joined #openstack-ansible | 09:16 | |
| jnamdar | hi all | 09:16 |
| jnamdar | I'm having some trouble with the murano dashboard in horizon https://pastebin.com/WAvqmPPL | 09:17 |
| jnamdar | it's happening at the TASK [os_horizon : Ensure static files are collected and compressed] | 09:17 |
| jnamdar | Is there any way I can disable murano dashboard install maybe? | 09:17 |
| jnamdar | ok I commented out the murano line in openstack-ansible/inventory/group_vars/horizon_all.yml, that should do the trick | 09:22 |
| *** frenzy_friday has joined #openstack-ansible | 09:51 | |
| *** frenzy_friday has left #openstack-ansible | 09:51 | |
| *** er-bot has joined #openstack-ansible | 09:52 | |
| *** er-bot has left #openstack-ansible | 09:53 | |
| noonedeadpunk | jnamdar: you could also just set a variable `horizon_enable_murano_ui: false` in your user_variables | 09:53 |
| noonedeadpunk | jrosser: andrewbonney: would be great if you could take a look at https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/795670 | 10:01 |
| andrewbonney | Will do | 10:02 |
| noonedeadpunk | will push roles freeze for RC later today (once https://review.opendev.org/q/topic:%22osa%252Fpki%22+(status:open) will be rechecked) | 10:09 |
| *** ioni has joined #openstack-ansible | 10:37 | |
| noonedeadpunk | btw I wonder why centos8 upgrade is failing for infra (cluster) https://zuul.opendev.org/t/openstack/build/d0e1c62eb3904c6b894527197581d37a | 11:37 |
| jrosser | logs are very slow to load for me :/ | 11:42 |
| noonedeadpunk | so actually feels like mariadb fails to start after container start | 11:51 |
| noonedeadpunk | or it's not auto-started | 11:51 |
| *** spatel has joined #openstack-ansible | 12:19 | |
| noonedeadpunk | Seems like github may become a better thing with https://github.com/github/roadmap/issues/211 :) | 12:28 |
| *** spotz has quit IRC | 13:26 | |
| *** spotz has joined #openstack-ansible | 13:27 | |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Use ansible-role-pki to generate SSL certificates https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/788032 | 13:30 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Use external PKI role to manage haproxy self-signed certificates https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/790078 | 13:38 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty master: Cloudkitty role cleanup and config updates https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/794199 | 14:11 |
| spatel | jamesdenton around? | 14:18 |
| jamesdenton | sorrta? | 14:26 |
| spatel | something is wrong with aio/metal deployment | 14:28 |
| spatel | i am seeing two 6640 manager in ovs | 14:29 |
| jamesdenton | hmm | 14:29 |
| jamesdenton | what error are you seeing? | 14:29 |
| spatel | root@aio1:~# ovs-vsctl get-manager | 14:29 |
| spatel | ptcp:6640 | 14:29 |
| spatel | ptcp:6640:127.0.0.1 | 14:29 |
| jamesdenton | what OS? | 14:29 |
| spatel | ubuntu focal | 14:29 |
| noonedeadpunk | doh, parallel clone still fails from time to time :( http://paste.openstack.org/show/806534/ | 14:30 |
| jamesdenton | is it causing issues? | 14:30 |
| spatel | vm failed to build | 14:30 |
| spatel | trying to debug | 14:30 |
| jamesdenton | is this OVN? | 14:30 |
| spatel | Yes OVN | 14:30 |
| jamesdenton | has the AIO been restarted? | 14:30 |
| spatel | you mean whole VM? | 14:31 |
| jamesdenton | sure | 14:31 |
| spatel | no i didn't but i can do that if its requirement | 14:31 |
| jamesdenton | no no | 14:31 |
| jamesdenton | curious, can you post 'ovs-vsctl list open_vswitch' please | 14:31 |
| spatel | http://paste.openstack.org/show/806535/ | 14:32 |
| spatel | i have also match hostname is correct without FQDN | 14:32 |
| jamesdenton | kk. i think the fix for that will be to set hostname in nova.conf, btw | 14:33 |
| jamesdenton | have you tried deleting one of the managers? | 14:33 |
| spatel | before my ovsdb was complaining that port already in use so i have delete one of manager and that error is gone | 14:34 |
| spatel | where should i set hostname in nova.conf (i remember in past i didn't do anything and it just worked) not sure what change this time.. | 14:35 |
| jamesdenton | [default] | 14:35 |
| spatel | only change is i have applied my cluster patch | 14:35 |
| jamesdenton | sure | 14:35 |
| jamesdenton | i don't think that's it in this case | 14:35 |
| jamesdenton | but can you show me 'openstack hypervisor list'? | 14:35 |
| spatel | http://paste.openstack.org/show/806536/ | 14:37 |
| jamesdenton | ok, so it may actually be that mismatch causing the issue | 14:37 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/train: config: add region_name to neutron https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/795467 | 14:37 |
| spatel | you want to set host = aio in nova.conf? | 14:38 |
| jamesdenton | not yet | 14:38 |
| mgariepy | spatel, did you tried with : https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/795528 | 14:38 |
| jamesdenton | try: ovs-vsctl set open_vswitch . external-ids:hostname='aio1.openstack.local' | 14:38 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/train: config: add region_name to nova https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/795468 | 14:38 |
| spatel | mgariepy i removed manager by hand | 14:39 |
| jamesdenton | mgariepy good find on that, too. i guess OVS is setting up it's own manager now? | 14:39 |
| jamesdenton | spatel OVS seems to set hostname to long name, which is why i did this: https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/793009 | 14:39 |
| jamesdenton | but nova defaults to shortname, in my experience, thus the recommendation to set long in nova.conf | 14:40 |
| spatel | jamesdenton no luck | 14:41 |
| spatel | same error "message": "Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures | 14:41 |
| spatel | i think something else going on | 14:43 |
| spatel | let me reboot host and see | 14:44 |
| jamesdenton | also check nova-compute logs | 14:48 |
| spatel | seeing error in neutron-log | 14:51 |
| spatel | http://paste.openstack.org/show/806537/ | 14:51 |
| spatel | feel like still host name related issue somewhere | 14:52 |
| jamesdenton | yes, can you send me 'openstack host list'? | 14:55 |
| spatel | hold on, i have rebooted my host vm after setting up hostname | 14:55 |
| spatel | jamesdenton here you go http://paste.openstack.org/show/806538/ | 14:57 |
| jamesdenton | that vm it's complaining about - old or new? | 14:58 |
| spatel | old or new?? | 14:58 |
| jamesdenton | can you spin up a new vm? and see if you get the same error in neutron log | 14:59 |
| spatel | ok | 14:59 |
| spatel | now neutron not showing any error and i can see error code is change to {"code": 500, "created": "2021-06-10T15:00:42Z", "message": "No valid host was found. There are not enough hosts available.", "details": "Traceback (most recent call last) | 15:02 |
| spatel | after reboot i am seeing two Hypervisor | 15:03 |
| spatel | http://paste.openstack.org/show/806539/ | 15:03 |
| jamesdenton | yeah, nova service-list probably shows aio and aio.openstack.local | 15:03 |
| jamesdenton | which is fine | 15:03 |
| jamesdenton | (for now) | 15:04 |
| spatel | nova logs throwing error - "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: aio1.openstack.local already exists. ", "request_id": "req-49b5a70e-c4e5-4a73-989a-71fdc4235eab"} | 15:05 |
| spatel | nova is confused | 15:05 |
| spatel | do you want me to delete one? | 15:06 |
| jamesdenton | yes | 15:06 |
| spatel | try this to remove but it has same name so how do i remove using ID https://docs.openstack.org/openstack-ansible/newton/developer-docs/ops-remove-computehost.html | 15:08 |
| spatel | let me try openstack compute service delete | 15:09 |
| spatel | jamesdenton got this error http://paste.openstack.org/show/806540/ | 15:12 |
| spatel | ovsdb was dead, just started back | 15:15 |
| spatel | now getting error - Failed to build and run instance: libvirt.libvirtError: Cannot get interface MTU on 'br-int': No such device | 15:15 |
| spatel | but i can see it in ovs-vsctl show | 15:15 |
| spatel | it works now | 15:20 |
| spatel | after restarting openvswitch-switch.service | 15:21 |
| jamesdenton | weird, but ok | 15:22 |
| spatel | now i lost that where was the issue | 15:22 |
| spatel | duplicate manager or hostname chaos | 15:23 |
| spatel | i think both | 15:23 |
| spatel | mgariepy patch is important to fix manager issue. (which i did by hand) | 15:23 |
| spatel | curious why its failing here - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/795528 | 15:24 |
| mgariepy | seems like ovsdb is stopped for ${INSERT_FRUSTRATING_REASON_HERE} | 15:25 |
| opendevreview | Merged openstack/openstack-ansible stable/ussuri: Disable magnum zero nodes test. https://review.opendev.org/c/openstack/openstack-ansible/+/795137 | 15:25 |
| spatel | hmm | 15:26 |
| spatel | you patch failed at TASK [os_tempest : Ping router ip address] | 15:26 |
| noonedeadpunk | this basically means that ovn is not working I guess | 15:31 |
| *** anbanerj|rover has quit IRC | 15:32 | |
| *** frenzy_friday has joined #openstack-ansible | 15:33 | |
| mgariepy | ovsdb-server was off. and openvswitch-switch.service | 15:33 |
| *** frenzy_friday is now known as anbanerj|rover | 15:33 | |
| mgariepy | was also off. | 15:33 |
| spatel | noonedeadpunk can we get access of that build VM so see what is the state of build? | 15:33 |
| mgariepy | after starting both service the router goes up and can ping | 15:33 |
| spatel | mgariepy what do you mean by off? | 15:34 |
| mgariepy | they were stopped. | 15:34 |
| spatel | may be ansible didn't started them or possible issue like i had (multiple manager) | 15:35 |
| mgariepy | the manager stuff was an issue. | 15:35 |
| spatel | but you patch should fix that part | 15:35 |
| spatel | i am going to re-build my aio again and see if in first short i spin up vm | 15:36 |
| noonedeadpunk | spatel: we can ask for a hold, then do recheck, and once that fails, we can ask for an access to the VM. Otherwise VM is deleted right after the CI is done | 15:36 |
| mgariepy | yes but still the services were not started. | 15:36 |
| spatel | noonedeadpunk let me do couple of more test in my lab and if they works then we should ask for hold to see what is going on inside build VM | 15:37 |
| noonedeadpunk | feels like mgariepy reproduced it?:) | 15:37 |
| mgariepy | aio with ovn scenario | 15:38 |
| mgariepy | + the 2 patches for neutron | 15:38 |
| mgariepy | and it's reproduced. | 15:38 |
| spatel | he did but i want to check if my ovsdb-server come up clean | 15:38 |
| noonedeadpunk | jrosser: yeah, mariadb is not enabled nowadays in centos | 15:40 |
| mgariepy | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266/9/tasks/providers/ovn_cluster_setup.yml#1 | 15:42 |
| mgariepy | oops line 35.. ! | 15:43 |
| mgariepy | ovsdb is stopped and never started i guess. | 15:43 |
| spatel | hmm i thought ovn-central start all the services including ovsdb-server | 15:53 |
| spatel | may be i am wrong | 15:53 |
| mgariepy | http://paste.openstack.org/show/806542/ | 15:54 |
| spatel | mgariepy you got it | 15:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Ensure mariadb service is enabled https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/795839 | 15:54 |
| spatel | let me fix that patch and add ovsdb-server exclusively | 15:54 |
| mgariepy | can you fix it ? inregrate my fix into your patch | 15:55 |
| spatel | yes fixing it now | 15:55 |
| mgariepy | thanks | 15:55 |
| spatel | do you want me to make your patch changes in my patch ? | 15:55 |
| spatel | remove manager part | 15:55 |
| spatel | or add dependency | 15:56 |
| mgariepy | yes | 15:57 |
| mgariepy | merge it into yours | 15:57 |
| mgariepy | i'll abandon mine. | 15:57 |
| spatel | sounds good | 15:57 |
| spatel | mgariepy do you think we need to restart ovs-vswitchd.service also? | 16:00 |
| opendevreview | Merged openstack/openstack-ansible-os_panko stable/ussuri: Fix wsgi_venv path for panko-api https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/795040 | 16:00 |
| opendevreview | Merged openstack/openstack-ansible-os_panko stable/victoria: Fix wsgi_venv path for panko-api https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/795039 | 16:00 |
| mgariepy | not sure why it was stopped. | 16:00 |
| mgariepy | :/ | 16:00 |
| spatel | lets restart no harm in that | 16:01 |
| spatel | aio is all-in-one so hard to isolate services ( in multi-node i don't think we will see that issue) | 16:02 |
| spatel | lets just do ovsdb at present and will see later | 16:02 |
| jnamdar | do I need to use the metal scenario if I want to deploy ironic in AIO? is that even possible? i'm kind of struggling to understand the role's doc https://docs.openstack.org/openstack-ansible-os_ironic/ussuri/configure-ironic.html, is the guide still relevan? | 16:04 |
| jnamdar | I tried to deploy with lxc, but nova seems messed up, I have no hypervisors | 16:04 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 16:05 |
| mgariepy | dependency, ovsdb stops both ovsdb and opensvswitch stops, same with openvswitch,. but when i start openvswitch they all come back. | 16:05 |
| mgariepy | not the case with ovsdb-server. | 16:05 |
| mgariepy | so i guess we should use openvswitch-switch.service | 16:06 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 16:08 |
| spatel | i can replace ovsdb-server with openvswitch-switch if that is correct way | 16:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Ensure mariadb service is enabled https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/795839 | 16:09 |
| mgariepy | i think it is | 16:09 |
| mgariepy | why does ovsdb needs to be stopped? | 16:09 |
| spatel | in my patch? | 16:10 |
| mgariepy | yes | 16:10 |
| spatel | i just did for clean up other stuff before create cluster db | 16:11 |
| mgariepy | isn't the ovsdb for the ""local"" openvswith stuff and only the ovn nb and sb needs to be replicated? | 16:11 |
| mgariepy | replicated/clusterd. | 16:11 |
| mgariepy | replicated/clustered | 16:11 |
| spatel | nb/sb use ovsdb for database right? | 16:11 |
| mgariepy | yes but in their own files ? /var/lib/ovn/ovnnb_db.db ? | 16:13 |
| mgariepy | ovsdb_server only serve for the local openvswitch-switch if I understand correctly | 16:14 |
| mgariepy | i might be mistaken tho. | 16:14 |
| spatel | frankly speaking i stoped ovsdb-server just for safely | 16:14 |
| spatel | if you check ovn-northd you will see ovsdb-server is running there | 16:15 |
| spatel | we can cut other patch where remove ovsdb-server stop line and see if that work or fail | 16:15 |
| mgariepy | give me 10 minutes i should have an answer, running setup-openstack without hthe ovsdb service management | 16:17 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 16:17 |
| spatel | ok | 16:17 |
| mgariepy | i hope it works ! :D | 16:19 |
| *** rpittau is now known as rpittau|afk | 16:24 | |
| mgariepy | arf my server is not fast enough ! | 16:26 |
| spatel | rebuilding aio_metal again to verify my patch | 16:34 |
| mgariepy | i'm at os-nova role :/ | 16:34 |
| mgariepy | i got the manager in ovs-vsctl show ! | 16:37 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Update distro mirrors to match release https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/795670 | 16:38 |
| mgariepy | moment of truth ! tempest is running ! | 16:41 |
| mgariepy | great success ! | 16:43 |
| mgariepy | http://paste.openstack.org/show/806544/ | 16:44 |
| mgariepy | we could drop the handler for ovsdb-server also. | 16:45 |
| spatel | mgariepy you are saying we don't need to touch ovsdb and just remove it from everywhere ? | 16:46 |
| mgariepy | yep | 16:48 |
| spatel | let me patch it | 16:49 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 16:50 |
| spatel | done | 16:50 |
| spatel | noonedeadpunk just noticed my new aio failing here - http://paste.openstack.org/show/806545/ | 16:51 |
| spatel | look like bad patch or something | 16:51 |
| noonedeadpunk | looks like you haven't bootstrtap ansible properly | 16:51 |
| spatel | hmm i think i did | 16:52 |
| spatel | let me try again | 16:52 |
| noonedeadpunk | you should have https://review.opendev.org/c/openstack/openstack-ansible/+/788031/12/ansible-role-requirements.yml | 16:52 |
| *** jnamdar has quit IRC | 16:56 | |
| spatel | i did run scripts/bootstrap-aio.sh but still same error let me check that file | 16:58 |
| spatel | noonedeadpunk pki section is missing | 16:59 |
| spatel | is this new change which i encountered suddenly | 17:00 |
| noonedeadpunk | then you need to pull changes :) | 17:00 |
| spatel | :) | 17:00 |
| spatel | thats it | 17:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Freeze roles for Wallaby RC1 https://review.opendev.org/c/openstack/openstack-ansible/+/795849 | 17:05 |
| spatel | noonedeadpunk damn it now getting error here - TASK [Create CA certificates] | 17:10 |
| spatel | let me start over again | 17:12 |
| noonedeadpunk | yeah, might be best option... | 17:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Unfreeze roles after RC1 https://review.opendev.org/c/openstack/openstack-ansible/+/795851 | 17:14 |
| *** raildo has joined #openstack-ansible | 17:18 | |
| noonedeadpunk | would be awesome to get this merged https://review.opendev.org/c/openstack/openstack-ansible/+/795849 asap | 17:24 |
| mgariepy | done my part :D | 17:40 |
| spatel | what is the difference between +2 vs +1 code-review ? | 17:59 |
| spatel | just curious | 17:59 |
| *** ioni has quit IRC | 18:03 | |
| mgariepy | +2 is approoved by core-reviewer | 18:04 |
| spatel | Thanks!! may be one day :) | 18:05 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum stable/ussuri: Define region for Magnum trust https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/795044 | 18:09 |
| mgariepy | spatel, added a few comments to fix ansible-lint stuff. | 18:20 |
| spatel | mgariepy thanks | 18:24 |
| spatel | use the service module and on L115 please - (why?) | 18:24 |
| spatel | what is the advantage ? | 18:24 |
| mgariepy | it's more generic. | 18:30 |
| mgariepy | and it's what is used everywhere in osa. | 18:30 |
| spatel | ok cool | 18:31 |
| mgariepy | for now most distro uses systemd but if one would use sysvinit or *cought* upstart *cought* it would work with the service module | 18:32 |
| mgariepy | just like using the package module instead of the apt one. | 18:33 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 19:22 |
| spatel | mgariepy look like after this patch all check will be green | 19:26 |
| spatel | did you verify ovn clustering on your PC ? | 19:28 |
| noonedeadpunk | spatel: jsut please, drop trailing whitespaces from license | 19:31 |
| spatel | let me see | 19:33 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 19:36 |
| mgariepy | spatel, testing it now into a vm. | 19:38 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 19:40 |
| spatel | mgariepy thanks | 19:40 |
| mgariepy | i'll most likely check it out next week tho. | 19:40 |
| spatel | sure no worry | 19:40 |
| spatel | enjoy your weekend | 19:41 |
| *** ioni has joined #openstack-ansible | 19:59 | |
| *** andrewbonney has quit IRC | 20:00 | |
| *** spatel has quit IRC | 20:06 | |
| *** spatel has joined #openstack-ansible | 20:35 | |
| spatel | noonedeadpunk hey | 20:37 |
| spatel | "{{ groups[neutron_services['neutron-ovn-northd']['group']] | first }}" | 20:37 |
| spatel | you want to do like this neutron_ovn_primary_cluster_node: "{{ groups[neutron_services['neutron-ovn-northd']['group']] | first }}" | 20:37 |
| spatel | what is the benefit here? or this is best practice thing? | 20:38 |
| *** spatel has quit IRC | 20:52 | |
| *** spatel has joined #openstack-ansible | 21:47 | |
| *** spatel has quit IRC | 22:06 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!