| *** ysandeep|out is now known as ysandeep | 05:04 | |
| *** ysandeep is now known as ysandeep|afk | 07:34 | |
| Mouaa | Hi guys, We are trying to use octavia on OSA train in distro install method. We notice a malfunction during the openstack-ansible playbooks/os-octavia-install.yml phase, we systematically come across an error indicating "Cloud default was not found." regarding the creation of the octavia user. | 07:55 |
|---|---|---|
| Mouaa | Has anyone come across this problem before us? | 07:55 |
| Mouaa | See: https://paste.opendev.org/show/bIFYLaggZCDkmttIu8ep/ | 07:55 |
| jrosser_ | Mouaa: is this a new deployment you are doing? | 08:24 |
| Mouaa | @jrosser_ Attempt to add Octavia (tested on iso-conf production dev platform) on osa deployment in train version based on Ubuntu 18.04 in distro. So no, this is not a new OSA deployment | 08:32 |
| jrosser_ | "in distro" ? | 08:33 |
| Mouaa | our version of OSA has been in production for a few years. Unfortunately in distro mode yes | 08:34 |
| jrosser_ | oh that is unfortunate | 08:36 |
| Mouaa | Already discussed here even with several of you, I understood that it is not ideal but we cannot reinstall everything in source method at the moment (daily customers in thousands...) | 08:39 |
| jrosser_ | well fundamentally your issue is caused by `"msg": "Cloud default was not found."` | 08:44 |
| jrosser_ | that is going to need some more debugging | 08:48 |
| jrosser_ | seeing the output from `TASK [os_octavia : Add service users]` with -vvv or more verbose would be helpful | 08:49 |
| Mouaa | I know... but var "cloud: default" is used everywhere without problems, cf comparing to heat for example | 08:50 |
| jrosser_ | does it run against the utility container? | 08:50 |
| jrosser_ | is the python interpreter correct to pick up where the shade libs are installed? | 08:50 |
| jrosser_ | and so on | 08:51 |
| jrosser_ | train release allowed a mixture of python2 and python3, is that working properly for octavia? | 08:51 |
| jrosser_ | was the octavia role ever tested on the train branch for distro installations? | 08:52 |
| jrosser_ | here is the last patch we merged to the stable/train branch for os_octavia https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/690368 | 08:53 |
| jrosser_ | i don't see any distro jobs there | 08:53 |
| jrosser_ | if you have things working for heat then i would perhaps suggest comparing the verbose ansible output for heat against octavia so you can see if there are any differences | 08:55 |
| Mouaa | The Release note shown support for distro method normally: https://github.com/openstack/openstack-ansible-os_octavia/blob/stable/train/releasenotes/notes/openstack-distribution-packages-bc0d1d606a362ffc.yaml | 08:57 |
| Mouaa | I will try again to deploy octavia with more verbosity on tasks and paste the output to paste.opendev.org | 09:01 |
| damiandabrowski | Mouaa: i'd check if you have valid /root/.config/openstack/clouds.yaml on your octavia_service_setup_host(defaults to localhost) | 10:31 |
| damiandabrowski | for example, i remember the situation when this file was overridden by bifrost when I wanted to install it on the same host | 10:32 |
| Mouaa | Actually for testing, I just recreated the containers without replaying the os-octavia-install.yml playbook and the containers do not contain this file at this point | 10:42 |
| damiandabrowski | containers shouldn't contain this file, as I mentioned above, by default it should be present on deploy host | 10:43 |
| damiandabrowski | (service setup block is delegated there: https://opendev.org/openstack/openstack-ansible-os_octavia/src/branch/stable/train/tasks/service_setup.yml#L30 ) | 10:44 |
| *** ysandeep|afk is now known as ysandeep | 10:46 | |
| Mouaa | @damiandabrowski : this file doesn't exist on our deployer... | 10:47 |
| jrosser_ | damiandabrowski: wouldnt the service setup host usually be the utility container? | 10:55 |
| damiandabrowski | jrosser_: nowadays yes, but on train i believe it was localhost | 11:02 |
| damiandabrowski | let me double check it | 11:03 |
| damiandabrowski | ahh you are right, it's "{{ groups['utility_all'][0] }}" | 11:05 |
| damiandabrowski | i just checked master branch... | 11:07 |
| damiandabrowski | so, let's start from the beginning | 11:08 |
| jrosser_ | Mouaa: says that the deployment has worked for heat, for example | 11:08 |
| damiandabrowski | for stable/train i see this in octavia playbook: `octavia_service_setup_host: "{{ openstack_service_setup_host | default('localhost') }}"` | 11:08 |
| jrosser_ | but octavia fails on setting up the service user with `"msg": "Cloud default was not found."` | 11:08 |
| jrosser_ | most obvious thing is to check that the task is delegated to where we expect | 11:09 |
| jrosser_ | but there is no confirmation of that yet | 11:09 |
| damiandabrowski | omg, i just overcomplicated things for you guys...sorry, jrosser_ is right | 11:09 |
| *** dviroel is now known as dviroel}rover | 11:26 | |
| *** dviroel}rover is now known as dviroel|rover | 11:26 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Bind bmaas network to where ironic_api is deployed https://review.opendev.org/c/openstack/openstack-ansible/+/854173 | 11:39 |
| jrosser_ | jamesdenton: i started with a fresh AIO for ironic with SCENARIO=aio_lxc_ironic, theres no mention of the bmaas network in container networks for aio1 without that patch ^^ | 11:40 |
| Mouaa | @damiandabrowski, FYI, the file exist on utility container and seems no overrided (OSA install date) | 11:46 |
| jrosser_ | Mouaa: we still don't know if the failing task was delegated to the utility container though :( | 11:47 |
| jamesdenton | jrosser_ great! i will test here, too | 12:30 |
| jrosser_ | jamesdenton: i'm wondering what to do next, in my actual deployment we have ironic_api in a container, not metal | 12:31 |
| jrosser_ | but thats now how aio_lxc_ironic ends up being | 12:31 |
| jrosser_ | *not | 12:31 |
| jamesdenton | well, you could always migrate it out? :D | 12:32 |
| jrosser_ | of course :) | 12:32 |
| jrosser_ | i'm not sure i follow why the AIO is like it is, other than maybe "the networking is all a bit hard here, make it metal so it becomes easy" | 12:33 |
| jamesdenton | well, the env.d file makes it seem like ironic_api has been on metal for a while now, not just limited to aio, no? | 12:33 |
| jrosser_ | indeed | 12:33 |
| jamesdenton | and yes, maybe that was the original intend | 12:33 |
| jamesdenton | *intent | 12:34 |
| jamesdenton | a reluctance to add YetAnotherBridge™ | 12:34 |
| jrosser_ | it's quite challenging to make the callback from the node being provisioned to the ironic API work properly too | 12:34 |
| jrosser_ | as that would traditionally be the VIP | 12:34 |
| jamesdenton | it can be, yes. i avoid all of that by having the provisioning network be a routed network | 12:35 |
| jamesdenton | i don't really know what the 'reference arch' is there, though | 12:35 |
| jrosser_ | no, it's a bit unclear | 12:35 |
| jrosser_ | i get the feeling in a lot of these higher level services theres a bunch of handwaving and "everything just routes to everything else" | 12:36 |
| jamesdenton | there is a bit of that | 12:36 |
| jrosser_ | having said that we have made it work entirely in LXC with a non-routed bmaas network | 12:36 |
| jrosser_ | you can configure each conductor with a unique callback ip (it's own) on the bmaas network | 12:37 |
| jrosser_ | and the need to contact the VIP goes away completely | 12:37 |
| jamesdenton | which is OK if your provisioning network can reach bmaas | 12:37 |
| jrosser_ | right - and thats another place where we are not very clear on the reference architecture | 12:37 |
| jrosser_ | there are a lot of options for separate cleaning/whatever networks | 12:38 |
| jrosser_ | becasue in my mind provosioning network == bmaas but maybe i have not understood enough | 12:38 |
| jamesdenton | in any case, the provisioning network needs to be a neutron network, and maybe where bmaas gets tricky is you also have neutron dhcp agent and other things that need to connect to it. | 12:38 |
| jrosser_ | yes thats what we do, br-bmaas is a neutron network with neutron dhcp | 12:39 |
| jrosser_ | but also goes to ironic-api containers on eth15 | 12:39 |
| jamesdenton | you've got provisioning, cleaning, and inspection networks, which can all be the same... or different. not to mention the tenant network, which depending on what plugin you use, can be different from provisioning, too. in the generic case it's all likely one flat network | 12:39 |
| jamesdenton | the ironic-api container that may or may not exist? :D | 12:39 |
| jrosser_ | indeed - we have ironic multitenancy setup so the tenant network is separate | 12:40 |
| jamesdenton | but what is listening in ironic-api that's needed here? | 12:40 |
| jrosser_ | callback from IPA i think? | 12:40 |
| jamesdenton | how are you changing the switchport vlan to support different networks? | 12:40 |
| jrosser_ | networking-generic-switch | 12:41 |
| jamesdenton | that callback might be to the VIP? i don't recall | 12:41 |
| jamesdenton | ahh right on. you guys are using that? | 12:41 |
| jrosser_ | we are only as far as having this in the lab to work out wtf is going on with it all - hence all the patches for LXC stuff recently | 12:41 |
| jrosser_ | i think we broke a bunch of stuff moving neutron api to uwsgi though | 12:42 |
| jamesdenton | i'm on 25.0.0 and it's still OK | 12:42 |
| jrosser_ | my colleague was making a LP bug, i'll see if we submitted it yet | 12:49 |
| jrosser_ | short story is that we have many potential .conf files for the various neutron services which all need to be inserted into the service unit, more get added as you enable more neutron plugins | 12:50 |
| jamesdenton | FYI i think you have a duplicate here? https://review.opendev.org/c/openstack/openstack-ansible/+/854173 | 12:53 |
| jamesdenton | https://review.opendev.org/c/openstack/openstack-ansible/+/852174 | 12:54 |
| jrosser_ | argh | 12:55 |
| jamesdenton | damiandabrowski Just hit rabbitmq install issue, too. Looks like version may need to be bumped to 1:24.1.4-1? ./roles/rabbitmq_server/vars/debian.yml:38:_rabbitmq_erlang_version_spec: "{{ (rabbitmq_install_method == 'external_repo') | ternary('1:24.1.3-1', '1:22.*') }}" | 14:19 |
| damiandabrowski | yeah i was thinking about it, but it would leave all other Xena tags broken :/ Today I asked rabbitMQ team on slack why this version disappeared, but didn't get any reply yet | 14:20 |
| jamesdenton | truuuue. | 14:21 |
| jamesdenton | you're so considerate :) | 14:21 |
| damiandabrowski | haha :D unfortunately, I'm a bit afraid that at the end of the day, bumping erlan version will be our only option... :D | 14:23 |
| *** ysandeep is now known as ysandeep|dinner | 14:36 | |
| jrosser_ | damiandabrowski: is it still in the rabbitmq repo, or is it just cloudsmith that removed it? | 14:44 |
| damiandabrowski | i'm not sure if I understood You correctly. cloudsmith is an official rabbitmq repo. RabbitMQ team manages it, cloudsmith provides only hosting service | 14:45 |
| jrosser_ | ah right, i wasnt sure how that worked | 14:47 |
| jrosser_ | this also is not the first time | 14:47 |
| jrosser_ | jamesdenton: the ironic AIO is just fubar :( | 14:48 |
| damiandabrowski | PS. I just sent an email on their mailing list, let's hope we'll get an answer | 14:48 |
| damiandabrowski | https://groups.google.com/g/rabbitmq-users/c/olys-t2N5-Y | 14:48 |
| jrosser_ | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) | 14:48 |
| *** dviroel|rover is now known as dviroel|rover|lunch | 14:59 | |
| damiandabrowski | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Aug 23 15:00:38 2022 UTC and is due to finish in 60 minutes. The chair is damiandabrowski. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| damiandabrowski | ok, i don't have required privileges or what? :D | 15:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Bind http and tftp services to the bmaas network https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852122 | 15:01 |
| damiandabrowski | ah, now it works | 15:02 |
| damiandabrowski | #topic rollcall | 15:02 |
| jrosser_ | o/ hello | 15:02 |
| damiandabrowski | hey everyone! | 15:02 |
| damiandabrowski | #topic bug triage | 15:05 |
| damiandabrowski | looks like noonedeadpunk is keeping and eye on bugs while he's on vacation, but literally an hour ago a new bug has arrived | 15:07 |
| damiandabrowski | #link https://bugs.launchpad.net/openstack-ansible/+bug/1987405 | 15:07 |
| damiandabrowski | have you ever seen something similar? | 15:07 |
| jrosser_ | yes this is from my team | 15:07 |
| jrosser_ | switching neutron api to uwsgi has broken the way the config files are appended to the ExecStarts line | 15:08 |
| jrosser_ | depending on which neutron plugins you have there may need to be additional config files referenced | 15:08 |
| damiandabrowski | ahh ok, so perhaps we should patch either neutron or uwsgi role? | 15:16 |
| damiandabrowski | btw. do you remember the reason why we moved things behind uwsgi? | 15:17 |
| jrosser_ | i've not had chance to look at it yet | 15:17 |
| jrosser_ | i guess that by convention all of the API services are uwsgi, and neutron was one of the last | 15:17 |
| jrosser_ | though it seems we did not account for there being >=1 config file sometimes | 15:17 |
| damiandabrowski | let's take glance as an example, uwsgi only causes troubles there | 15:18 |
| damiandabrowski | some time ago i was thinking about disabling uwsgi for glance by default. Do you think it may be a good idea? | 15:18 |
| damiandabrowski | https://docs.openstack.org/glance/latest/admin/apache-httpd.html | 15:19 |
| damiandabrowski | "Glance project team recommends that Glance be run in its normal standalone configuration, particularly in production environments." | 15:19 |
| damiandabrowski | (i just realized that i got off topic a bit) | 15:21 |
| anskiy | I've also sent a patch to optionally disable uwsgi for mistral-api some time ago. | 15:33 |
| damiandabrowski | ah, so there may be more services with problems similar to glance... thanks for the info | 15:35 |
| damiandabrowski | #info on next meeting it may be worth to discuss what services should have uwsgi disabled by default | 15:36 |
| damiandabrowski | #topic office hours | 15:36 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Ironic role should pick the address services bind to https://review.opendev.org/c/openstack/openstack-ansible/+/854231 | 15:38 |
| damiandabrowski | as you may now, Xena branch has issues with rabbitmq repo because pinned erlang version disappeared. I posted a question about that on their mailing list. | 15:38 |
| damiandabrowski | #link https://groups.google.com/g/rabbitmq-users/c/olys-t2N5-Y/m/gCuZ8gDCAwAJ | 15:38 |
| jrosser_ | afaik the issue with glance is that the code is not written with uwsgi in mind | 15:38 |
| damiandabrowski | do we have any other topics to discuss? | 15:38 |
| damiandabrowski | jrosser_: exactly ;) | 15:39 |
| jrosser_ | but that doesnt automatically mean that the rest suffer the same | 15:39 |
| damiandabrowski | of course, i'm not trying to disable uwsgi for all services :D | 15:40 |
| opendevreview | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Declaratively set host aggregates https://review.opendev.org/c/openstack/openstack-ansible/+/854235 | 15:46 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Install rally only on first utility container https://review.opendev.org/c/openstack/openstack-ansible/+/854237 | 15:49 |
| damiandabrowski | ^ yesterday i've found a minor issue with rally | 15:50 |
| damiandabrowski | #endmeeting | 15:59 |
| opendevmeet | Meeting ended Tue Aug 23 15:59:45 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-08-23-15.00.html | 15:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-08-23-15.00.txt | 15:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-08-23-15.00.log.html | 15:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Ensure ironic inspector dhcp server listen address is defined https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852173 | 16:07 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Bind http and tftp services to the bmaas network https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852122 | 16:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Ensure ironic inspector dhcp server listen address is defined https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852173 | 16:08 |
| jrosser_ | jamesdenton: i got an ironic LXC AIO to deploy without error using all of https://review.opendev.org/q/topic:osa-ironic-tidy | 16:14 |
| jrosser_ | there was an error in this https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852122 | 16:14 |
| jamesdenton | thanks, i will check that out. i had both of mine fail | 16:14 |
| jrosser_ | and then removing some hardwired address here https://review.opendev.org/c/openstack/openstack-ansible/+/854231 | 16:14 |
| jrosser_ | so thats just as it was with ironic_api on metal and ironic-inspector in LXC | 16:15 |
| jrosser_ | i think perhaps it's worth a discussion at some point if we have the right structure here, and maybe some extra docs needed | 16:16 |
| jamesdenton | agreed. inspector also has two different working models, with one being more janky than the other | 16:16 |
| jamesdenton | s/janky/legacy | 16:17 |
| jrosser_ | right - we should double check that there are "sensible defaults" | 16:17 |
| jrosser_ | i notice also we are running simultaneously nginx + apache again here | 16:18 |
| jrosser_ | nginx in ironic and apache in keystone/horizon | 16:18 |
| jrosser_ | also nginx in the repo container | 16:18 |
| jamesdenton | thunderdome | 16:18 |
| jrosser_ | this could do with a clean up to be all apache | 16:18 |
| jamesdenton | if that's the preference, sure | 16:19 |
| jrosser_ | well, it was always switchable in keystone depending if you were doing federation or not | 16:19 |
| jrosser_ | and that made even less sense | 16:19 |
| jrosser_ | so keystone has become apache only to cover federaation and not-federation with an identical setup | 16:20 |
| jamesdenton | gotcha | 16:20 |
| jamesdenton | time to rebuild these vms | 16:21 |
| *** dviroel|rover|lunch is now known as dviroel|rover | 16:27 | |
| *** ysandeep|dinner is now known as ysandeep|out | 16:27 | |
| opendevreview | Merged openstack/openstack-ansible master: Attach bmaas network to ironic_api containers https://review.opendev.org/c/openstack/openstack-ansible/+/852174 | 18:26 |
| *** dviroel|rover is now known as dviroel|out | 22:31 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!