| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Bump SHAs for 2023.1 https://review.opendev.org/c/openstack/openstack-ansible/+/923968 | 00:04 |
|---|---|---|
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_magnum stable/2024.1: Manage Magnum resources with the last play host https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/924033 | 07:03 |
| noonedeadpunk | cnilesh: question - you're installing on metal, without LXC, right? | 08:30 |
| noonedeadpunk | failed to work on your 2071952 yesterday - spawning sandbox now | 08:30 |
| opendevreview | Merged openstack/openstack-ansible-os_nova unmaintained/yoga: Update .gitreview for unmaintained/yoga https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/908481 | 10:15 |
| cnilesh | noonedeadpunk, yes yes , waiting for the workaround/patch | 10:47 |
| cnilesh | no_container: true | 10:48 |
| cnilesh | noonedeadpunk++ thank you sir | 10:51 |
| noonedeadpunk | yeah, will try to and let you know about results :) | 10:55 |
| cnilesh | noonedeadpunk++ thanks | 10:55 |
| cnilesh | noonedeadpunk, sorry for the late reply | 10:55 |
| f0o | anyone know what could cause this error? https://paste.opendev.org/show/bwXThB2b9a8b1POsi1Lv/ - I tripple checked resolution and connectivity, deployer and 10.20.0.11 have connectivity and all ssh-keys are trusted | 13:42 |
| noonedeadpunk | f0o: have you tried to `export ANSIBLE_LOCAL_TEMP=/tmp`? | 13:44 |
| f0o | exact same error just with /tmp/ instead of /root/.ansible/tmp/ | 13:45 |
| noonedeadpunk | and filesystem is not in RO I assume?:) | 13:46 |
| f0o | this is a fresh Ubuntu Jammy installation straight from the ISO - which is even weirder... | 13:46 |
| f0o | haha nop, first thing I checked | 13:46 |
| f0o | it's also not full | 13:46 |
| noonedeadpunk | and that's for any random task, or just some specific one? | 13:47 |
| f0o | I'm failing to setup-hosts.yml so its hard to say if it fails for other tasks too | 13:48 |
| noonedeadpunk | and fails pretty much at the very-very beginning? | 13:48 |
| f0o | if it's guaranteed to be on the 10.20.0.11 host then I can just create a new VM; it has notihng on it other than ssh-keys | 13:48 |
| f0o | yep | 13:48 |
| jrosser | noonedeadpunk: isnt the "None/None" at the end of that error a pointer to wrong version of ssh connection plugin? | 13:50 |
| jrosser | i seem to remember seeing similar when finding this https://github.com/openstack/openstack-ansible-plugins/commit/bbaf62e9233bd240da2bd3d613062cdeb9b5101e | 13:50 |
| noonedeadpunk | I frankly don;'t remember, but can be | 13:51 |
| jrosser | noonedeadpunk: also, kind of doh here https://github.com/MariaDB/mariadb-docker/issues/592 | 13:52 |
| noonedeadpunk | isn't it the same reason kinda? | 13:53 |
| jrosser | it's exactly the same | 13:53 |
| noonedeadpunk | as they right now not verifying only untrusted certs | 13:53 |
| noonedeadpunk | but not ones for wrong DNS | 13:53 |
| noonedeadpunk | and I guess in docker they're also trusted... | 13:53 |
| jrosser | f0o: what version of openstack-ansible are you using? | 13:54 |
| f0o | current master | 13:54 |
| jrosser | becasue you want to do development work? :) | 13:55 |
| f0o | sort of - still hunting down that OVS routing issue that I got | 13:55 |
| f0o | and want a reference AIO without my fancy stuff | 13:55 |
| noonedeadpunk | ah, master... | 13:56 |
| f0o | I can switch to stable/2023.2 that I got in prod | 13:56 |
| noonedeadpunk | I think on master you'd need that https://review.opendev.org/c/openstack/openstack-ansible/+/923951 | 13:56 |
| jrosser | try changing this to "master" https://github.com/openstack/openstack-ansible/blob/master/ansible-collection-requirements.yml#L14 | 13:56 |
| noonedeadpunk | if it wasn't failing :D | 13:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible collection versions https://review.opendev.org/c/openstack/openstack-ansible/+/923951 | 13:57 |
| jrosser | f0o: you are encountering the thing i linked before https://github.com/openstack/openstack-ansible-plugins/commit/bbaf62e9233bd240da2bd3d613062cdeb9b5101e | 13:57 |
| f0o | fuin :D | 13:57 |
| jrosser | but the version of the plugins collection you have there is pinned to a previous SHA, not tracking tip of master | 13:57 |
| noonedeadpunk | well, master is smth that's under active development. and it might work in CI, but not yet out of it | 13:58 |
| f0o | uhm is scripts/bootstra-ansible.sh broken for stable/2023.2 ? | 13:58 |
| f0o | https://paste.opendev.org/show/bg4rXuTgNFKVUeE9J4eu/ | 13:59 |
| noonedeadpunk | it should not be | 13:59 |
| f0o | why is wget segfaulting O_O | 13:59 |
| noonedeadpunk | Segmentation fault for wget? | 13:59 |
| noonedeadpunk | yeah | 13:59 |
| f0o | neat... I guess whatever ISO I grabbed is FUBAR | 14:00 |
| f0o | apt reinstall wget; wget google -> segfault xD | 14:00 |
| noonedeadpunk | cnilesh: so... I was not able to catch a horizon issue in my AIO vm for 2023.1 :( | 14:02 |
| jrosser | right yes so as we merged the bump to ansible 2.17 yesterday we definatly need the plugins sha fixing | 14:02 |
| noonedeadpunk | it's working pretty much nicely | 14:02 |
| cnilesh | noonedeadpunk, ops | 14:09 |
| cnilesh | but 2/3 time its reproduced here in my testbed | 14:10 |
| cnilesh | any inputs ? | 14:10 |
| noonedeadpunk | frankly - no idea where to look even | 14:10 |
| noonedeadpunk | it's some Django thing | 14:10 |
| noonedeadpunk | Have you checked apache log regarding errors? | 14:10 |
| cnilesh | humn, which django file any guess | 14:10 |
| noonedeadpunk | As I'd assume it should be reported | 14:10 |
| cnilesh | only the single line reported on the bz | 14:12 |
| noonedeadpunk | doh, and no stack trace... | 14:13 |
| cnilesh | no | 14:14 |
| noonedeadpunk | aha | 14:14 |
| noonedeadpunk | ok | 14:14 |
| noonedeadpunk | I guess I know the issue | 14:14 |
| noonedeadpunk | Frankly - I've deployed with SSL and that's why I don't see the issue | 14:14 |
| cnilesh | ;) | 14:14 |
| cnilesh | yup yup | 14:14 |
| noonedeadpunk | I have a guess now | 14:14 |
| cnilesh | with ssl no issues | 14:15 |
| cnilesh | check the teplates i shared on bz | 14:15 |
| cnilesh | i dnt think so thr is any issue in the template, | 14:15 |
| noonedeadpunk | I need to re-run all playbooks now to drop ssl :D | 14:16 |
| cnilesh | ;( | 14:17 |
| noonedeadpunk | ok, I've reproduced it | 14:46 |
| noonedeadpunk | cnilesh: try setting "horizon_external_ssl: False" and re-run os-horizon-install | 14:48 |
| noonedeadpunk | or better - openstack_external_ssl: False | 14:49 |
| cnilesh | noonedeadpunk, ok lte me run | 14:53 |
| cnilesh | noonedeadpunk, one thing I also noticed, if I rerun the os-octavia-install.yml , it redownload the ampora | 14:54 |
| cnilesh | need to skip this task if it is already in the glance | 14:54 |
| noonedeadpunk | yeah, probably. there's a variable to skip that iirc | 14:55 |
| cnilesh | noonedeadpunk, also , 2023.1 is installed on ubuntu22.04, while octavia is downloading focal i.e. 20.04 ampora image, we need to update that wll | 14:57 |
| jrosser | its not about the target OS really | 14:57 |
| jrosser | it is about having the right amphora version for the release of openstack | 14:58 |
| cnilesh | oh.... | 14:58 |
| noonedeadpunk | cnilesh: eventually, you're supposedto build own version of amphora images and update ir regularly | 14:58 |
| cnilesh | I thought it is aligned with OS | 14:58 |
| jrosser | nope, it boots as a VM | 14:59 |
| cnilesh | sure | 14:59 |
| noonedeadpunk | as the thing we have defined as "default" image has "test-only" in it's name: https://opendev.org/openstack/openstack-ansible-os_octavia/src/branch/master/defaults/main.yml#L316 | 14:59 |
| jrosser | so it coule be a different OS entirely to the cloud hosts | 14:59 |
| jrosser | cnilesh: the octavia project give instructions on how to build your own amphora https://docs.openstack.org/octavia/latest/admin/amphora-image-build.html | 15:00 |
| jrosser | but the ones we reference in the OSA role default are not intended for production use | 15:01 |
| noonedeadpunk | (though they're built in almost the same way) | 15:02 |
| jrosser | indeed | 15:02 |
| noonedeadpunk | ok, I'm really unsure if we should drop `horizon_external_ssl` or not alike to keystone | 15:35 |
| noonedeadpunk | (talking about https://opendev.org/openstack/openstack-ansible-os_keystone/commit/e8d0f0db5f623f3a2ebbacca6b237f16f7d27202) | 15:35 |
| noonedeadpunk | mainly due to that: https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/templates/horizon_local_settings.py.j2#L52-L60 | 15:36 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use openstack.osa.install_defaults role instead of vars_files https://review.opendev.org/c/openstack/openstack-ansible/+/923358 | 15:37 |
| jrosser | cnilesh: can you describe what you want to do with a "no ssl" deployment? is it correct that you have no https between haproxy and the backends, and no https between your browser and horizon? | 15:42 |
| noonedeadpunk | yeah - it's described here: https://bugs.launchpad.net/openstack-ansible/+bug/2071952 | 15:51 |
| noonedeadpunk | or well - https://paste.openstack.org/show/b5B8n0KLXBzWicWj0Ttn/ | 15:51 |
| jrosser | yeah, but if the intention is to then put something like f5 on the front of haproxy to make it ssl, i dunno if any of this is valid | 15:52 |
| noonedeadpunk | I think it's just a POC and there were series of reports related to SSL | 15:54 |
| noonedeadpunk | as self-gen ssl was put in front of haproxy, but also in front of galera (and I think rabbitmq?) which didn't worked out (as it was same SSL) | 15:54 |
| noonedeadpunk | so I think what happened - decision to somplify setup by dropping TLS... but dunno | 15:56 |
| jrosser | so there is some quite good explanation of the error here https://docs.djangoproject.com/en/5.0/ref/settings/#secure-proxy-ssl-header | 15:57 |
| noonedeadpunk | yup | 16:01 |
| noonedeadpunk | and that's why I thought of the keystone patch | 16:01 |
| noonedeadpunk | but frankly - I didn't understand fully the logic we should apply | 16:02 |
| noonedeadpunk | apparently - it's not ideal out of the box right now | 16:02 |
| noonedeadpunk | likely we also should drop https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/templates/openstack_dashboard.conf.j2#L45-L49 | 16:02 |
| noonedeadpunk | but keep SECURE_PROXY_SSL_HEADER ? | 16:03 |
| noonedeadpunk | so I can't say I understand 100% what we need to do in all of our usecases | 16:04 |
| noonedeadpunk | ie - no-tls, only haproxy tls, all tls... | 16:04 |
| noonedeadpunk | so no-tls is kinda easy - we should omit SECURE_PROXY_SSL_HEADER | 16:05 |
| noonedeadpunk | haproxy tls - likely add? But it also works nicely without it, just in case | 16:05 |
| noonedeadpunk | and then - full tls - we also don't need it? | 16:05 |
| jrosser | well https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/defaults/main.yml#L251-L252 | 16:05 |
| noonedeadpunk | I'm not sure that `horizon_secure_proxy_ssl_header` should exist at all... | 16:06 |
| noonedeadpunk | as it's used only in apache conf | 16:07 |
| noonedeadpunk | and we've jsut dropped same logic for keystone... | 16:07 |
| noonedeadpunk | so eventually, what I jsut did, set `horizon_external_ssl: false` while having tls on haproxy side. And I don't see any obvious issues... | 16:07 |
| noonedeadpunk | and it also works for no-tls case | 16:08 |
| noonedeadpunk | but I can be missing what is broken obviously | 16:08 |
| jrosser | i am wondering if there is some real legacy usecase that is confusing things | 16:09 |
| jrosser | https://opendev.org/openstack/openstack-ansible-os_horizon/commit/4283200534eafa444efd9bb408ddcb5c98a1d442 | 16:10 |
| jrosser | and just to make it more difficult it is haproxy > apache > uwsgi/django | 16:14 |
| jrosser | and at the same time we have maybe people trying to also point browser at the internal vip, that might be http or https | 16:14 |
| noonedeadpunk | it feels so, that there's legacy involved indeed | 16:16 |
| noonedeadpunk | but not sure what that usecase is | 16:16 |
| noonedeadpunk | as we never changed how haproxy behaved | 16:16 |
| noonedeadpunk | but changed haproxy<>apache | 16:16 |
| jrosser | no, but we may have confused things when adding ssl everywhere | 16:16 |
| jrosser | that we did not quite account for what was already in horizon setup | 16:16 |
| jrosser | as the patch i linked seems to suggest horizon was forever doing ssl on it's own | 16:17 |
| noonedeadpunk | 1 | 16:18 |
| noonedeadpunk | yeah as well as keystone | 16:18 |
| noonedeadpunk | it never worked for real though | 16:18 |
| jrosser | i feel we may be tripping over 2 from that list | 16:19 |
| jrosser | but question is - where do we want to add the X-Forwarded-Proto? | 16:20 |
| jrosser | django docs say this should be stripped from incoming requests at the lb, added only for incoming requests at the lb which were https | 16:22 |
| jrosser | but we also mess with this at apache, which could be wrong | 16:22 |
| jrosser | so this is kind of not aligned with what the django docs say, that we set the header unconditionally https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/templates/openstack_dashboard.conf.j2#L45-L49 | 16:26 |
| jrosser | but why "Your proxy sets the X-Forwarded-Proto header and sends it to Django, but only for requests that originally come in via HTTPS." | 16:31 |
| cnilesh | https://bugs.launchpad.net/openstack-ansible/+bug/1902585 | 17:59 |
| cnilesh | jrosser, yes no SSL at all | 18:11 |
| jrosser | cnilesh: look at the fix we made for uwsgi backlog, we made a place you can override that | 18:13 |
| cnilesh | jrosser++ thank you sir | 18:14 |
| cnilesh | also may I know the significance of #reserved_host_disk_mb = 2048 | 18:14 |
| cnilesh | #reserved_host_memory_mb = 2048 | 18:14 |
| cnilesh | #reserved_host_disk_mb = 2048 | 18:14 |
| cnilesh | #reserved_host_memory_mb = 2048 | 18:14 |
| jrosser | cnilesh: here is a useful tool for that kind of question https://codesearch.opendev.org/?q=reserved_host_disk_mb | 18:30 |
| cnilesh | jrosser, thank you | 18:31 |
| cnilesh | so mch | 18:31 |
| jrosser | so thats a config option for nova | 18:31 |
| jrosser | and you'd find the reference guide for that here https://docs.openstack.org/nova/latest/configuration/config.html | 18:32 |
| jrosser | you need to reserve enough ram on the compute host for everything else that you might run there | 18:32 |
| jrosser | otherwise you could end up either swapping or with OOM trouble when nova allocates all the ram it thinks it can to vm | 18:33 |
| jrosser | maybe you run monitoring agents, or converge ceph storage onto your computes, or gpu drivers.... so this may need tuning to be a suitable value for your deployment | 18:34 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use openstack.osa.install_defaults role instead of vars_files https://review.opendev.org/c/openstack/openstack-ansible/+/923358 | 19:23 |
| ccnilesh | noonedeadpunk++ jrosser++ thank you so much | 20:11 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!