| *** chandankumar is now known as chkumar|rover | 05:52 | |
| *** arxcruz|ruck is now known as arxcruz | 08:10 | |
| mossblaser | zigo: I tried copying over the http-auto-chunked/http-chunked-input (and indeed the other settings) from the swift patchset into the glance uwsgi config (v2.0.20 btw) but this doesn't seem to fix things for Glance :( | 08:52 |
|---|---|---|
| *** mgoddard- is now known as mgoddard | 09:11 | |
| gokhani | Hi folks, how can I use my previously created letsencryp cert with OSA? I stucked rate limits :( | 10:01 |
| noonedeadpunk | gokhani I'm not sure there's a way unortunatelly :( | 10:04 |
| gokhani | noonedeadpunk: if I copy the certificates in /etc/letsencrypt/live/<domain> and disable create certs steps may be it will work | 10:10 |
| noonedeadpunk | oh well, with some manual steps it will yes | 10:10 |
| noonedeadpunk | you can even provide these certs as static ones... | 10:11 |
| noonedeadpunk | as if you just disable ssl, then endpoints likely will be just not serving https | 10:11 |
| gokhani | noonedeadpunk: you mean by static ones means like user provided ssl | 10:16 |
| noonedeadpunk | yup | 10:17 |
| noonedeadpunk | but again, question of renewal would be quite valid | 10:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Update ssh_keypairs role to fix module for Rocky Linux 8 https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/835152 | 11:01 |
| *** chkumar|rover is now known as raukadah | 12:01 | |
| mgariepy | good morning everyone | 12:43 |
| NeilHanlon | morning mgariepy :) | 12:55 |
| NeilHanlon | jrosser: w.r.t. osa-tests: think rocky is needed here? https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-ansible-env-prep.sh#L194 | 12:55 |
| opendevreview | Neil Hanlon proposed openstack/openstack-ansible-tests master: Update tests for Rocky Linux https://review.opendev.org/c/openstack/openstack-ansible-tests/+/835219 | 13:00 |
| opendevreview | Neil Hanlon proposed openstack/openstack-ansible-plugins master: Update ssh_keypairs role to fix module for Rocky Linux 8 https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/835152 | 13:00 |
| jamesdenton | spatel mgariepy hit this with OVN a couple of times, just a heads up if you see it: https://www.jimmdenton.com/neutron-ovn-private-chassis/ | 13:46 |
| mgariepy | hey jamesdenton :) i'll read that carefully :D | 13:47 |
| mgariepy | i haven't played much with ovn lately unfortunatly. | 13:50 |
| spatel | jamesdenton very interesting.. | 13:51 |
| spatel | I am running OVN in production and haven't seen any issue like that so far (I am using wallaby release) | 13:51 |
| jamesdenton | i'm not sure what the conditions are | 13:52 |
| spatel | feel like bug or race condition | 13:52 |
| spatel | did you report this issue ? | 13:53 |
| jamesdenton | no, but i referenced some bugs/patches that seem to be aware | 13:53 |
| spatel | ok | 13:54 |
| spatel | jamesdenton curious what kind of homelab you have and your electricity bill :) ? | 13:55 |
| jamesdenton | hah | 13:56 |
| spatel | I have started building my homelab using DL360 Gen9 1U servers | 13:56 |
| jamesdenton | i think my lab expense is ~$45-50/mo | 13:56 |
| spatel | I have 2 server but in future plan is to get more. | 13:56 |
| spatel | Do you keep them running all the time or just turn on/off when you need? | 13:57 |
| jamesdenton | it started as 1x controller and 1x compute that i left on continuously, and it has now morphed into 3x controller on ESXi and 3x hyperconverged ceph/compute and ~4-5 ironic nodes | 14:02 |
| jamesdenton | so there's 4 nodes on all the time, plus a mellanox SN2100 and a 1G switch | 14:02 |
| spatel | goodness | 14:15 |
| spatel | I am running lab only when i need it and then turn it off | 14:15 |
| spatel | running everything inside ESXi | 14:16 |
| jamesdenton | that's probably the better approach :D | 14:32 |
| spatel | jamesdenton also planning to put tesla solar panel to go green :) | 14:45 |
| spatel | then i can keep stuff running all the time (except winter hehe) | 14:45 |
| spatel | noonedeadpunk around? | 14:46 |
| spatel | or jrosser | 14:46 |
| jamesdenton | awesome | 14:46 |
| spatel | Getting this error and not sure why - got this error - https://paste.opendev.org/show/bzT7JMrwONnSza328XoO/ | 14:47 |
| noonedeadpunk | semi-around | 14:47 |
| noonedeadpunk | spatel: some fact gathering issue | 14:47 |
| noonedeadpunk | not happy about absent ansible_facts['processor_vcpus'] | 14:48 |
| spatel | odd.. | 14:48 |
| noonedeadpunk | yup.... | 14:49 |
| noonedeadpunk | what version is that? | 14:50 |
| spatel | wallaby 24.0.0 | 14:50 |
| spatel | i think xena sorry | 14:50 |
| spatel | Xena-24.0.0 release | 14:51 |
| gokhani | hi folks, is there any bug about mysql ansible collection on 22.1.0? I am getting errors when creating database. Error is "msg": "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required." | 14:51 |
| NeilHanlon | gokhani: what OS ? | 14:51 |
| gokhani | NeilHanlon: it is focal | 14:52 |
| NeilHanlon | would suggest to me that your ansible_python_interpreter is a bit wonky, or the module is just missing | 14:53 |
| NeilHanlon | is this a new, or an existing deployment? | 14:53 |
| gokhani | NeilHanlon: this is a new deployment | 14:57 |
| jrosser | gokhani: the db setup tasks should all be delegated to the utility host | 14:58 |
| jrosser | if something failed when deploying that and you moved onward rather than fix it, you would get that error | 14:58 |
| NeilHanlon | thanks jrosser, wasn't sure if it was a bootstrap problem | 14:59 |
| jrosser | there is a venv in the utility container which should have all the required libraries ready | 14:59 |
| jrosser | you should run with enough -vvv to see what the python interpreter is set to for the failing task | 15:00 |
| jrosser | then go to that place and check that the libraries are present | 15:01 |
| gokhani | jrosser: I see. yes you are right, I get errors when utility-install beacuse of network connection, but after solve connection problems, rerun playbook and everything is ok. | 15:06 |
| jrosser | gokhani: cool :) | 15:06 |
| jrosser | utility container is important | 15:06 |
| gokhani | jrosser: I think after failing playbook, I need to clean facts | 15:10 |
| jrosser | if it is more than 24hours since you deployed the utility container then there may not be any facts for it | 15:11 |
| jrosser | they have a timeout | 15:12 |
| gokhani | jrosser: now I checked the utility containers, openstack clients are missing in pip list | 15:13 |
| jrosser | it’s possible that we don’t re-gather them at the right point but you’d need to look at the log for your specific playbook run to see | 15:13 |
| jrosser | gokhani: did you check in the utility venv though? | 15:14 |
| jrosser | they are in /use/local/bin symlinked to the venv, so the regular host pip won’t see the python packages | 15:15 |
| gokhani | jrosser: yes, I checked in utility venv | 15:18 |
| gokhani | also checked python venv build log, it didn't try to install openstack clients | 15:20 |
| gokhani | maybe venv_rebuild=yes works for utility | 15:23 |
| gokhani | and yes venv_rebuild=yes solved the issue:) | 15:27 |
| spatel | jrosser any idea what i should or how to fix it - https://paste.opendev.org/show/bzT7JMrwONnSza328XoO/ | 15:29 |
| noonedeadpunk | ensure your facts are gathered? | 15:49 |
| noonedeadpunk | basically I believe they should be gathered on playbook startup https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-placement-install.yml#L18 | 15:50 |
| noonedeadpunk | ANd it's really interesting why they're not | 15:52 |
| noonedeadpunk | it doesn't seem that it should be delegated, so feels like facts in topic are just for placement container | 15:53 |
| spatel | same error now i am getting for glance container | 16:01 |
| spatel | everything was working month ago before i upgrade to xena | 16:01 |
| noonedeadpunk | Which sounds like exactly missing facts.... | 16:02 |
| noonedeadpunk | But I'm pretty sure that on X we have this all covered | 16:02 |
| noonedeadpunk | Except what we have changed - default for ANSIBLE_GATHER_SUBSET https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/openstack-ansible.rc#L36 | 16:03 |
| noonedeadpunk | BUT processor_vcpus are in hardware facts, so they're covered with extra task https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-glance-install.yml#L35 | 16:05 |
| noonedeadpunk | spatel: so question is - are you using default playbooks? and if yes, are you sure you're still on 24.0.0 for openstack-ansible repo? | 16:06 |
| spatel | noonedeadpunk what do you mean by default playbook? | 16:09 |
| spatel | https://paste.opendev.org/show/bfLUZVHABDq5L9HK8TJ8/ | 16:10 |
| noonedeadpunk | well, you could write own playbooks for example to run roles provided by osa :) | 16:12 |
| noonedeadpunk | spatel: and I was moe wondering about git status in /opt/openstack-ansible | 16:13 |
| spatel | On branch stable/xena | 16:15 |
| noonedeadpunk | ok, so would be great to understand if facts are actually gathered with task I mentioned and if include is processed as expected | 16:16 |
| spatel | noonedeadpunk one more thing i noticed its only throwing error when i use --tags common_mq | 16:17 |
| spatel | https://paste.opendev.org/show/bSvEVyGbc92eH1ygkl89/ | 16:17 |
| noonedeadpunk | ok, that;s smth | 16:17 |
| spatel | yesterday my rabbitMQ was upset so i tried to run that tags to rebuild users but stuck with that error :( | 16:18 |
| spatel | finally i create users manually | 16:18 |
| noonedeadpunk | Damn, I guess I know what the issue is | 16:18 |
| spatel | i knew it :) | 16:19 |
| spatel | you will fix something | 16:19 |
| noonedeadpunk | not sure yet how to fix though | 16:19 |
| noonedeadpunk | and need to prove idea in aio first | 16:19 |
| spatel | how does it working for you then with --tags common_mq | 16:19 |
| spatel | give me clue and i can try to reproduce.. | 16:20 |
| noonedeadpunk | so pre-tasks in playbook are not run when tags are being used | 16:21 |
| spatel | hmm | 16:21 |
| noonedeadpunk | so basically because of https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-glance-install.yml#L32 | 16:22 |
| noonedeadpunk | facts are not gathered down the line | 16:22 |
| spatel | But nova/neutron has no issue with tags common_mq | 16:23 |
| spatel | No error when running openstack-ansible os-nova-install.yml --tags common-mq | 16:23 |
| spatel | only placement/glance not happy with that | 16:23 |
| noonedeadpunk | which also means that backends are not disable when running with tags | 16:24 |
| noonedeadpunk | well, it's easy to workaround issue you see | 16:25 |
| noonedeadpunk | https://paste.openstack.org/show/brMDgheH7UjlRdzTlVrc/ | 16:26 |
| noonedeadpunk | but how to manage haproxy endpoints is interesting | 16:26 |
| spatel | oh wait why haproxy coming in picture ? | 16:27 |
| spatel | let me try that patch and see | 16:27 |
| NeilHanlon | noonedeadpunk: i think 835152 is ready to fix the CI fail you mentioned yesterday. I put in a change for the tests repo that it depends on. https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/835152 | 16:27 |
| noonedeadpunk | NeilHanlon: and reverted change jrosser did to run_tests.sh | 16:28 |
| noonedeadpunk | the idea is that when change merges to tests repo it will sync file it topic and produce bunch of these for each project | 16:29 |
| noonedeadpunk | so basically you should avoid patching run_tests.sh in 835152 manually | 16:31 |
| NeilHanlon | oh, whoops | 16:31 |
| spatel | noonedeadpunk that patch works no error now | 16:31 |
| NeilHanlon | didn't realize i did that | 16:31 |
| noonedeadpunk | well, that what diff says :) https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/835152/2..3 | 16:33 |
| opendevreview | Neil Hanlon proposed openstack/openstack-ansible-plugins master: Update ssh_keypairs role to fix module for Rocky Linux 8 https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/835152 | 16:33 |
| NeilHanlon | you know, I saw that diff and figured it was wrong 😂 | 16:33 |
| OutBackDingo_ | okay, # openstack-ansible setup-infrastructure.yml --syntax-check fails with cant find inventory.ini, yet there nothing that explains how to generate one | 16:57 |
| noonedeadpunk | spatel: I have several new workarounds now :) | 17:01 |
| spatel | +1 | 17:02 |
| spatel | noonedeadpunk let me know when its ready for testing | 17:31 |
| noonedeadpunk | it will be huge patch) | 18:02 |
| spatel | noonedeadpunk very interesting.. | 18:30 |
| spatel | why suddenly this issue showed up? | 18:30 |
| noonedeadpunk | but I spoilered it mostly) | 18:30 |
| spatel | i am curious why you didn't see this when you nuke your rabbitMQ? | 18:32 |
| spatel | or may be you never run --tags common_mq task | 18:32 |
| noonedeadpunk | I don't nuke rabbit at the first place :) | 18:38 |
| noonedeadpunk | Happened once in last 5 years :) | 18:38 |
| noonedeadpunk | (after which I added common-mq) | 18:38 |
| spatel | oh!! i do nuke time to time :) | 18:53 |
| spatel | may be because of scaling issue. I have 300 compute nodes in each cloud and sometime its so chatty that OOM out and kill rabbit PID | 18:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix extra facts gathering with tags https://review.opendev.org/c/openstack/openstack-ansible/+/835308 | 18:56 |
| noonedeadpunk | spatel: ^ | 18:56 |
| spatel | \o/ | 18:57 |
| noonedeadpunk | I bet for OOM we had some adjustment of score for it... | 18:57 |
| noonedeadpunk | Maybe mixing with galera... | 18:57 |
| noonedeadpunk | But under opinion that for both | 18:57 |
| spatel | Send me what score and where did you set.. | 18:58 |
| spatel | Yes we have rabbit/mysql running on same box and time to time getting oom | 18:58 |
| opendevreview | David Moreau Simard proposed openstack/openstack-ansible master: DNM: Test ara 1.5.8rc1 with openstack-ansible https://review.opendev.org/c/openstack/openstack-ansible/+/696634 | 19:02 |
| jrosser | OutBackDingo_: it’s not necessary to use syntax check usually | 20:36 |
| jrosser | you would not need an inventory.ini as OSA generates a dynamic inventory | 20:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!