| opendevreview | Merged openstack/openstack-ansible stable/yoga: Switch spice-html5 source to freedesktop gitlab https://review.opendev.org/c/openstack/openstack-ansible/+/881317 | 01:09 |
|---|---|---|
| psymin | making more progress it seems, now when I log into horizon I get this output (and more) https://paste.opendev.org/show/beJ7D1e0NnOe9ObCD9Or/ | 03:02 |
| noonedeadpunk | psymin[m]: I don't actually see an end of the provided stack trace with result error message | 05:13 |
| jrosser | i would guess that neutron api is down maybe? | 05:38 |
| noonedeadpunk | yeah. might be that. at least it leans toward this | 05:54 |
| jrosser | this fails across the board on jammy which is odd https://review.opendev.org/c/openstack/openstack-ansible/+/879085?tab=change-view-tab-header-zuul-results-summary | 08:41 |
| noonedeadpunk | I kinda wonder if now ubuntu has packed faulty OVN version | 09:09 |
| noonedeadpunk | that is simmilar to what we had with CentOS | 09:09 |
| noonedeadpunk | ` nova.exception.PortBindingFailed: Binding failed for port 425f7fdf-455b-4783-84df-e63353ce5b37, please check neutron logs for more information.` | 09:10 |
| noonedeadpunk | yup, pretty much https://zuul.opendev.org/t/openstack/build/e66c75a713364ae69fca88ce3cb87e39/log/logs/host/ovn/ovn-northd.log.txt#33 | 09:10 |
| noonedeadpunk | openvswitch-common=3.1.0-1~cloud0 | 09:12 |
| noonedeadpunk | pfffff | 09:12 |
| noonedeadpunk | I think we've merged switch to antelope packages | 09:13 |
| jrosser | ahha that would explain it | 09:15 |
| noonedeadpunk | https://bugs.launchpad.net/cloud-archive/+bug/2017757 | 09:45 |
| noonedeadpunk | In the meanwhile I guess we need to update https://opendev.org/openstack/openstack-ansible-os_neutron/commit/f1a8c358531bdf86d8aeda725bc0b1c347d325c1 | 09:45 |
| jrosser | noonedeadpunk: so something about this comment is now changed? https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/878911/6/tasks/providers/ovn_config.yml#55 | 11:41 |
| noonedeadpunk | nope | 11:42 |
| noonedeadpunk | jsut affect jammy as well | 11:42 |
| noonedeadpunk | and well. RDO is fixed now :D | 11:42 |
| noonedeadpunk | So yes, smth has changed :D | 11:42 |
| jrosser | i thought that `--no-record-hostname` would be helping | 11:44 |
| noonedeadpunk | it's `when: - ansible_facts['pkg_mgr'] == 'dnf'` | 12:33 |
| noonedeadpunk | the reason why it's not | 12:34 |
| noonedeadpunk | also path should be diffferent for Ubuntu | 12:34 |
| noonedeadpunk | I will patch it now | 12:34 |
| NeilHanlon | psymin[m]: if you're using my openstack_user_config as a base, it's missing some config to use linux bridges with Zed.. i am updating my stuff now to use straight OVS instead of LXB.. | 13:19 |
| NeilHanlon | because I can't be bothered to figure out OVN for my lab lol | 13:19 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Fix protocol in magnum_url https://review.opendev.org/c/openstack/openstack-ansible/+/881565 | 14:02 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: [DNM] Do not define cluster template labels https://review.opendev.org/c/openstack/openstack-ansible/+/881566 | 14:02 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_magnum master: Rename floating IP option for coe_cluster_template https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/880484 | 14:02 |
| damiandabrowski | noonedeadpunk: I spent some time on magnum issue. It's quite messed up :/ | 14:05 |
| damiandabrowski | commit msg in https://review.opendev.org/c/openstack/openstack-ansible/+/881566 describes the problem | 14:05 |
| noonedeadpunk | I still think it's module issue | 14:05 |
| noonedeadpunk | damiandabrowski: do you have some sandbox/test playbook? | 14:06 |
| noonedeadpunk | or at least sandbox with magnum?:) | 14:07 |
| damiandabrowski | technically yes, but magnum tends to go down during tempest tests for some reason | 14:08 |
| damiandabrowski | but magnum playbook works fine there, so you can jump in if you want :D | 14:08 |
| psymin[m] | <NeilHanlon> "psymin: if you're using my..." <- I am using that config. I added the variable neutron_plugin_type: "ml2.lxb" | 14:24 |
| psymin[m] | Not sure if that was the right thing to do or not, but it made it through the playbook. | 14:24 |
| NeilHanlon | that works, too :) | 14:26 |
| psymin[m] | I'm open to using OVS, it might be a benefit, not sure. I'm mostly eager to have a functional system before digging too deeply into which parts to tinker with | 14:29 |
| NeilHanlon | OVS is a bear, but with the LXB plugin going experimental, it might be worth looking at. Thankfully they can re-use the same bridges you're using, if you choose to | 14:29 |
| NeilHanlon | and that's definitely fair :) | 14:30 |
| psymin[m] | I have three bridges set up on each server, br-mgmt, br-storage and br-vxlan. Each tied to a separate physical nic. I didn't set up any vlan tagging manually, I wonder if I should? I also sort of wonder if I should use the private networks in the example rather than the private networks I'm using, just to be more similar to normal. | 14:31 |
| psymin[m] | I would prefer to do it the "normal" way, whichever way that is. So if you do get a config working with OVS I'd love to see it :) | 14:35 |
| psymin[m] | <noonedeadpunk> "psymin: I don't actually see..." <- I'm not sure which portion of the webpage debug output to share. | 14:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Deprecate certbot-auto https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/881578 | 14:46 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Deprecate certbot-auto https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/881578 | 14:48 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Deprecate certbot-auto https://review.opendev.org/c/openstack/openstack-ansible/+/881579 | 14:48 |
| psymin[m] | I did get this warning during the playbook. Digging into apache stuff in that container now. https://paste.opendev.org/show/bxzrIdXzJMD9iXu8tj1u/ | 14:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Workaround ovs bug that resets hostname with add command https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/881582 | 15:33 |
| noonedeadpunk | I hope that will fix ubuntu ^ | 15:33 |
| jamesdenton | doh | 15:33 |
| psymin | I think every healthcheck is failing https://paste.opendev.org/show/byUddAo0IZee2EoZBfQ6/ | 15:55 |
| noonedeadpunk | psymin: but has playbook failed? | 16:01 |
| psymin | playbooks had success | 16:02 |
| psymin | my manually created linux bridges might be wrong? or setting the variable neutron_plugin_type: "ml2.lxb" might have been the wrong option? | 16:03 |
| psymin | I guess the 404 for the healthcheck isn't occuring at the moment. | 16:08 |
| NeilHanlon | psymin: run this and send a paste w/ the output `echo "show stat" | nc -U /run/haproxy.stat | cut -d "," -f 1,2,5-11,18,24,27,30,36,50,37,56,57,62 | column -s, -t ` | 16:15 |
| psymin | https://paste.opendev.org/show/b3olFAdNqPiHLW09aqo8/ | 16:16 |
| NeilHanlon | `lxc-attach infra1_neutron_server_container-a831628d` and then run `systemctl status neutron.slice` | 16:17 |
| psymin | nice, thank you. I'm also seeing some errors in syslog on the host regarding neutron | 16:18 |
| psymin | https://paste.opendev.org/show/bVA3720OrSz5Z7ksE6Xb/ | 16:18 |
| noonedeadpunk | psymin: are you deploying OVN? | 16:21 |
| noonedeadpunk | As then vxlan is not valid thing indeed | 16:21 |
| noonedeadpunk | We haven't updated docs though... | 16:21 |
| noonedeadpunk | (everywhere) | 16:22 |
| noonedeadpunk | vxlan is valid for ovs/lxb networking | 16:22 |
| noonedeadpunk | and according to the paste it should be ovn | 16:22 |
| psymin | noonedeadpunk, Okay. I probably need to rethink my network and network config. I wasn't intending to deploy ovn but if that is the path of least resistance I'm all for it. | 16:23 |
| noonedeadpunk | well, it's just default which you can easily override | 16:24 |
| noonedeadpunk | but then it should be geneve instead of vxlan | 16:24 |
| admin1 | ovn works nicely :) | 16:44 |
| NeilHanlon | but change is scary! | 16:45 |
| admin1 | i deployed in new | 16:45 |
| psymin | Is there a URL I should be reading to help me configure the target servers, the openstack_user_config.yml and the user_variables.yml for ovn? | 16:46 |
| psymin | I manually configured br-mgmt, br-storage and br-vxlan on the targets, should I not do that for ovn? | 16:47 |
| psymin | https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html | 16:49 |
| psymin | admin1, when you deployed in new, did you deploy antelope? zed? Did you deploy to Ubuntu 22.04.2 server? | 17:19 |
| jrosser | psymin: it's always worth remembering that building an all-in-one will get you a reference deployment to work off | 17:26 |
| jrosser | the default will be OVN | 17:26 |
| jrosser | and tbh it is not the right question to ask if some branch or other was a successful deployment | 17:27 |
| jrosser | becasue it pretty much always comes down to getting the config right | 17:27 |
| psymin | okay, I'll deploy aio to baremetal after this | 17:33 |
| jrosser | the aio contains a bunch of config to make it fit in a small system too | 17:38 |
| jrosser | but the openstack_user_config should be pretty close to what you need | 17:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Workaround ovs bug that resets hostname with add command https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/881582 | 17:42 |
| noonedeadpunk | psymin: you can also cheat by checking configs in CI ;) | 17:43 |
| noonedeadpunk | but this is actually the doc https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html | 17:44 |
| noonedeadpunk | I think in your current state you need to ensure that neutron_plugin_base and neutron_ml2_drivers_type are correct | 17:44 |
| noonedeadpunk | or well, default :) | 17:45 |
| noonedeadpunk | so `neutron_plugin_base: ['ovn-router']` and neutron_ml2_drivers_type: "geneve,vlan,flat" | 17:46 |
| psymin | thank you | 17:57 |
| admin1 | psymin, ubuntu 22.04 , tag 26.0.1 | 18:09 |
| admin1 | got magnum on it deploying k8s .. octavia works good a well .. | 18:09 |
| admin1 | so far happy with it | 18:09 |
| psymin | I haven't been setting up vlan tagging, since I have physical interfaces for each network. I assume I should probably set up those tags. | 18:13 |
| psymin | https://docs.openstack.org/openstack-ansible/zed/_images/network-arch-multiple-interfaces.png | 18:14 |
| jrosser | it doesn’t matter so much, OSA cares about the bridges for the most part | 18:14 |
| jrosser | and neutron the interfaces you specify in provider_networks | 18:15 |
| jrosser | where I recommend using a vlan beyond just a lab toy is your external network because using type “flat” for that just stores up trouble for later | 18:16 |
| jrosser | but even then, you would be using a trunk port for that and dealing with the vlan stuff 1) in your switch and 2) in the neutron setup via its api | 18:16 |
| jrosser | not as part of the deploy environment | 18:17 |
| noonedeadpunk | I'm usually setting a flat network to a vlan :D and then add vlan network as a parent interface :D | 18:29 |
| noonedeadpunk | Probably I could use jsut vlan.... | 18:29 |
| spatel | jamesdenton around? | 19:10 |
| spatel | I have quick question related OVN bridge mapping, I have ovn-bridge-mappings="physnet1:br-ex" | 19:28 |
| spatel | how does physnet map with our external interface? | 19:28 |
| psymin | heh, in the aio page for zed, it says "The Zed release is only compatible with Debian 11 (bullseye), Ubuntu 20.04 (Focal Fossa), CentOS 9 Stream, and derivitives of CentOS Stream/RHEL such as Rocky Linux." I'm booted to Ubuntu 22.04.2 :P | 19:37 |
| psymin | spatel, I'm way over my head on all this stuff, but here is a ChatGPTv3.5 answer to your query https://paste.opendev.org/show/bZSL0vdLh6Bp6GtkTpgL/ | 19:39 |
| spatel | psymin that is cool but still they didn't answer that question how physnet1 is map with my external interface (in my case br-vlan) | 19:42 |
| spatel | I can understand physnet1 is logical name. It doesn't existing on OS | 19:43 |
| admin1 | psymin, it works on 22.04 | 20:06 |
| psymin | good to hear, ran into an issue with task "TASK [openstack.osa.db_setup : Create database for service]" Setting no_log: false and redoing | 20:09 |
| psymin | https://paste.opendev.org/show/bJy5VvHbY0tokpsBw67a/ | 20:11 |
| jrosser | psymin: here is the compatibility matrix https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/compatibility-matrix.html | 20:27 |
| psymin | that container has /usr/bin/python3.10 and a symlink for /usr/bin/python3 so I assume _oslodb_ansible_python_interpreter should resolve to one of those | 20:27 |
| jrosser | ignore that :) | 20:28 |
| jrosser | look that is says "See stdout/stderr for the exact error" | 20:28 |
| jrosser | the interpreter message is a generic error message from the ansible module | 20:28 |
| jrosser | nothing at all specific about what actually happened | 20:28 |
| psymin | hmm I wonder how to find the stdout for that | 20:28 |
| jrosser | unset no_log | 20:28 |
| jrosser | https://github.com/openstack/openstack-ansible-plugins/blob/stable/zed/roles/db_setup/tasks/main.yml | 20:29 |
| jrosser | is this your all-in-one build? | 20:30 |
| psymin | this is aio, yes | 20:30 |
| psymin | the paste that mentions stdout is when no_log: false | 20:30 |
| psymin | in keystone container path /openstack/venvs/ doesn't exist currently | 20:31 |
| jrosser | but it's /openstack/venvs/utility-26.1.0/bin/python | 20:32 |
| jrosser | ^ utility container | 20:32 |
| jrosser | thats where all the db operations are done | 20:32 |
| psymin | that path also doesn't exist on utility container | 20:33 |
| jrosser | then something has failed with deploying the utility container | 20:34 |
| psymin | I'll wipe out containers and rebuil | 20:34 |
| psymin | this time with _venv_wheel_build_enable: "true" | 20:34 |
| jrosser | did you start with a completely clean system for this AIO? | 20:35 |
| psymin | yes, fresh ubuntu 22.04.2 install | 20:35 |
| jrosser | then it should work really without errors | 20:35 |
| jrosser | as thats exactly what we run in CI many many times a day | 20:36 |
| jrosser | +/- external factors of course | 20:36 |
| jrosser | this is less than 24h ago https://zuul.opendev.org/t/openstack/build/4ba9aadda754468c8338fa0c5892f786 | 20:37 |
| jrosser | you can browse the logs there and compare | 20:37 |
| jrosser | here is the start of setup-hosts.yml https://zuul.opendev.org/t/openstack/build/4ba9aadda754468c8338fa0c5892f786/log/job-output.txt#7234 | 20:39 |
| psymin | reinstalling and keeping a record of my steps | 21:02 |
| psymin | (reinstalling os) | 21:02 |
| psymin | I'm hopeful for this aio deploy to succeed. Two changes from default BOOTSTRAP_OPTS="bootstrap_host_public_interface=eno4" and lxc_container_default_mtu: 1464 | 22:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
