| *** anskiy1 is now known as anskiy | 01:33 | |
| *** raukadah is now known as chandankumar | 04:51 | |
| *** ysandeep|PTO is now known as ysandeep | 05:38 | |
| noonedeadpunk | mornings | 06:23 |
|---|---|---|
| noonedeadpunk | we totally do have issue with logic in cinder-volume configuration :( | 06:24 |
| jrosser_ | morning | 06:38 |
| * jrosser_ reads ML | 06:39 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/yoga: Increase ControlPersist timeout to 300 seconds https://review.opendev.org/c/openstack/openstack-ansible/+/852107 | 06:40 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Increase ControlPersist timeout to 300 seconds https://review.opendev.org/c/openstack/openstack-ansible/+/852108 | 06:41 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Increase ControlPersist timeout to 300 seconds https://review.opendev.org/c/openstack/openstack-ansible/+/852108 | 06:41 |
| *** ysandeep is now known as ysandeep|afk | 06:52 | |
| noonedeadpunk | and I'm not really sure how we're gonna fix that | 07:08 |
| noonedeadpunk | ys, we can jsut disable active/active if no etcd is present, but that would break deployments in a way | 07:09 |
| noonedeadpunk | or we can forcefully install etcd when it's enabled | 07:09 |
| noonedeadpunk | but then install it only inside cinder_volume, which could be o bare metal as well... | 07:10 |
| jrosser_ | well also designate for a long time says you need a coordinator | 07:33 |
| jrosser_ | and we never did it for that either, which really should happen | 07:33 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Bind http and tftp services to the bmaas network https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852122 | 07:53 |
| noonedeadpunk | well, we kind of have etcd already in osa. With really low effort zookeeper can be added as well. | 07:55 |
| jrosser_ | i think i've had to completely re-implement common-tasks/dynamic-address-fact.yml inside the ironic role | 07:57 |
| noonedeadpunk | yeah, looksl ike that... | 08:00 |
| noonedeadpunk | to be clear -I was not pushing to replace dynamic-address-fact.yml was wondering if doing the octavia way was enough and what way we should go | 08:01 |
| noonedeadpunk | ironic looks more complicated I guess? | 08:01 |
| noonedeadpunk | not sure why though | 08:02 |
| opendevreview | Merged openstack/openstack-ansible-plugins stable/yoga: Fix gluster play_hosts https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/851379 | 08:29 |
| jrosser_ | i think that the difference in ironic is the templates need the IP address and also the interface name | 08:38 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server stable/ussuri: Use cloudsmith repo for rabbit and erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/850350 | 08:47 |
| noonedeadpunk | So if you say that's easier jsut to use dynamic-address-fact.yml - then let's just do that | 09:23 |
| noonedeadpunk | I really can't recall how did we fixed this nmap issue for centos 8.... | 09:24 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/9ae9e53c4a104a5bbbdf7ab9295dca99/log/job-output.txt#15990 | 09:24 |
| noonedeadpunk | considering we don't provide nmap version... | 09:25 |
| noonedeadpunk | were these jsut broken infra mirrors.... | 09:25 |
| jrosser_ | maybe dynamic-address-fact needs converting to a tiny role or module | 09:31 |
| jrosser_ | what is not so nice about what i have done (or what octavia role does) is that it needs to have container_networks be used inside the role | 09:32 |
| *** ysandeep|afk is now known as ysandeep | 09:33 | |
| jrosser_ | my patch for this to ironic is still broken on metal so i make an AIO anyway just now for it | 09:33 |
| admin1 | jrosser_ , i found that if the SSL expired and I had to only change the .pem and re-run haproxy playbook, that error popped up again ( failed 3 times for the 3 controllers and then was OK in the last run ) | 09:35 |
| jrosser_ | admin1: what error? you'll need to give some more context really | 09:38 |
| admin1 | right .. sorry .. its about haproxy complaining that the pem does not exist while in the 2nd run, it would have created it .. there were also 2 more people talking about this before .. | 09:41 |
| admin1 | i think i had filed a bug report for it .. searching .. | 09:41 |
| admin1 | else i will do a new one | 09:41 |
| jrosser_ | please include enough info to reproduce it | 09:42 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/yoga: Gather build hosts facts once. https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/849640 | 09:43 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/yoga: Run wheels build for each unique distro/arch https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/849004 | 09:43 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/yoga: Take account of lxc_apt_mirror in new debootstrap command https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/851387 | 10:07 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Remove ironic_server from inventory https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852199 | 10:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Bind http and tftp services to the bmaas network https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852122 | 10:21 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Ensure ironic inspector dhcp server listen address is defined https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852173 | 10:21 |
| opendevreview | Malin Roth proposed openstack/openstack-ansible-os_keystone master: Add PKCE method for OIDC https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/852390 | 10:23 |
| admin1 | jrosser_, this one https://bugs.launchpad.net/openstack-ansible/+bug/1973242 | 10:26 |
| admin1 | https://bugs.launchpad.net/openstack-ansible/+bug/1973242 -- updated with latest finding | 10:32 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Install only the required dhcp config files for inspector https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/852121 | 10:32 |
| jrosser_ | admin1: can you please add which certificate (yours or PKI role) and any vars that you have set for it? | 10:35 |
| jrosser_ | and public / internal endpoint etc | 10:36 |
| jrosser_ | also we need to see what the PKI role did when it was called from the haproxy role | 10:37 |
| admin1 | the vars are there. starting in the first line | 10:40 |
| admin1 | internal endppint is always an ip .. in the var .. external is also in the var .. | 10:41 |
| admin1 | nothing else used | 10:41 |
| admin1 | it works .. just that it takes 4 tries n+1 retries .. where n = number of controllers ( haproxy ) | 10:41 |
| jrosser_ | the log is really needed for this https://github.com/openstack/openstack-ansible-haproxy_server/blob/b4a564795b8f8c684010f49eb6e222586f1c5432/tasks/main.yml#L40-L57 | 10:42 |
| admin1 | what is the best way .. run all again with -vvvv ? | 10:45 |
| admin1 | and submit the file ? | 10:45 |
| jrosser_ | no need really for -vvvv, just to see if the PKI role copies in your certificate or not | 10:46 |
| jrosser_ | but it needs to be run in a way like it fails for you before | 10:47 |
| jrosser_ | oh well maybe i see whats wrong | 10:51 |
| jrosser_ | "cat: /etc/haproxy/ssl/haproxy_r2c2-172.29.236.9-ca.crt: No such file or directory | 10:51 |
| jrosser_ | `-ca.crt` thats important | 10:52 |
| jrosser_ | noonedeadpunk: do you use any self-supplied certificates for public endpoint? | 11:00 |
| noonedeadpunk | I do, yes | 11:00 |
| noonedeadpunk | we run Xena top though | 11:00 |
| noonedeadpunk | we actually run both for public and internal | 11:01 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/ussuri: Bump MariaDB version https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/851439 | 11:01 |
| jrosser_ | this looks like it assumes that the CA file is always given https://github.com/openstack/openstack-ansible-haproxy_server/blob/b4a564795b8f8c684010f49eb6e222586f1c5432/handlers/main.yml#L18 | 11:01 |
| jrosser_ | but this allows the CA to be optional https://github.com/openstack/openstack-ansible-haproxy_server/blob/b4a564795b8f8c684010f49eb6e222586f1c5432/vars/main.yml#L65-L75 | 11:02 |
| noonedeadpunk | well, that sounds like quite valid bug then | 11:11 |
| jrosser_ | only thing i can think of is to have two handlers, with conditions | 11:12 |
| noonedeadpunk | or do complex bash | 11:14 |
| noonedeadpunk | as it;s already a shell module | 11:14 |
| noonedeadpunk | so we can add `[[ if -f {{ haproxy_ssl_cert_path ~ '/haproxy_' ~ ansible_facts['hostname'] ~ '-' ~ item ~ '-ca.crt' }} ]]; then {{ haproxy_ssl_cert_path ~ '/haproxy_' ~ ansible_facts['hostname'] ~ '-' ~ item ~ '-ca.crt' }}; fi` ? | 11:15 |
| noonedeadpunk | or smth even better.... | 11:17 |
| agemuend_ | Hi jrosser_, we propose https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/852390 to be able to enable PKCE (RFC 7636) for federated identity. Its e.g. required for one of the European Science Clouds (EGI FedCloud) (https://docs.egi.eu/providers/cloud-compute/openstack/aai/#changes-in-apache-configuration). | 11:21 |
| jrosser_ | agemuend_: this one? https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/852390 | 11:23 |
| agemuend_ | Yes | 11:24 |
| jrosser_ | looks reasonable, the tests should finish soon | 11:26 |
| jrosser_ | is there a particular release you were wanting to use that with? | 11:27 |
| agemuend_ | We patched locally, would just be nice to have it in future | 11:31 |
| noonedeadpunk | agemuend_: commented jsut one small nit there. We can totally backport that actually | 11:34 |
| *** dviroel_ is now known as dviroel | 11:38 | |
| noonedeadpunk | ah, I know, we do provide ca-chain anyway | 12:04 |
| jrosser_ | i guess when you go buy a commerical cert then you just need the key/crt as the CA should be in all the systems anyway | 12:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Do not add cacert when it does not exist https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/852399 | 12:09 |
| noonedeadpunk | yeah, ca is, but chain is usually still needed | 12:09 |
| jrosser_ | admin1: ^ does this match what you are doing? | 12:11 |
| jrosser_ | i.e where is the intermediate CA cert for your cert/key coming from? | 12:12 |
| noonedeadpunk | I think it might be part of the certificate in bunch of cases... | 12:18 |
| noonedeadpunk | really depends on the input... | 12:19 |
| noonedeadpunk | so making CA file optional is valid thing I believe | 12:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Allow haproxy to bind on the interface https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/852039 | 12:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Allow haproxy to bind on the interface https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/852039 | 12:25 |
| admin1 | cert.pem has the cert, chain and root all in 1 file | 13:20 |
| noonedeadpunk | the question was if you have chain in .crt as you don't define haproxy_user_ssl_ca_cert | 13:23 |
| admin1 | i do not have a haproxy_user_ssl_ca_cert defined | 13:30 |
| admin1 | haproxy_user_ssl_ca_cert is a new variable introduced afaik | 13:31 |
| noonedeadpunk | I'm pretty sure it was there for quite a while... | 13:32 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic stable/yoga: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/846777 | 13:34 |
| jrosser_ | admin1: haproxy_user_ssl_ca_cert is there all the way back to queens at least | 13:46 |
| admin1 | is the error because haproxy_user_ssl_ca_cert is not there ? | 13:54 |
| admin1 | i will test | 13:55 |
| admin1 | and update | 13:55 |
| jrosser_ | it is because the code now expects that the CA cert is provided in it's own file | 13:55 |
| jrosser_ | but thats not great for backwards compatibility | 13:55 |
| jrosser_ | so you could try this patch out https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/852399 | 13:56 |
| opendevreview | Merged openstack/openstack-ansible master: Duplicate centos 8/9 logic for Rocky Linux 9 https://review.opendev.org/c/openstack/openstack-ansible/+/852181 | 14:07 |
| opendevreview | Merged openstack/openstack-ansible master: Remove ironic_server from env.d https://review.opendev.org/c/openstack/openstack-ansible/+/852182 | 14:07 |
| opendevreview | Merged openstack/openstack-ansible master: Remove neutron agents from ironic env.d file https://review.opendev.org/c/openstack/openstack-ansible/+/851699 | 14:19 |
| opendevreview | Merged openstack/openstack-ansible master: Do not create {hostname}-host_containers group as child of other groups https://review.opendev.org/c/openstack/openstack-ansible/+/851764 | 14:19 |
| opendevreview | Malin Roth proposed openstack/openstack-ansible-os_keystone master: Add PKCE method for OIDC https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/852390 | 14:25 |
| jrosser_ | noonedeadpunk: did you see the ML thread about heartbeat_in_pthread? | 14:53 |
| jrosser_ | i think we have some handling for this but there is talk of backporting a change which looks like it might affect our stable branches | 14:54 |
| noonedeadpunk | Well I saw thread but wasn't really following it. | 15:16 |
| jrosser_ | we handle the cases that it's needed to be False https://review.opendev.org/q/topic:bug%252F1961603 | 15:18 |
| jrosser_ | and they propose to switch the default to False which means instead we have to add many more patches to keep it as True where we want that | 15:19 |
| jrosser_ | and on stable branches as well | 15:19 |
| jrosser_ | i am uncertain if this is just laziness in RH wanting to flip it so they don't have to do any actual work in RHSOP | 15:20 |
| noonedeadpunk | do we really want to have that as true? | 15:20 |
| noonedeadpunk | I'm pretty sure that's the case | 15:20 |
| jrosser_ | but we already have patches to deal with this so it's not new news | 15:20 |
| jrosser_ | but changing the default would mean we would need to make loads of patches to have nothing change in existing deployments | 15:21 |
| mrf | Hi! https://paste.opendev.org/show/br1mjP6qVstCIrzMfAFp/ is not a minor error on the path? | 15:23 |
| noonedeadpunk | I mean - do we really track all changes in behaviour that's been backported? | 15:23 |
| jrosser_ | well maybe thats the thing, that this sort of thing is generally not backported | 15:23 |
| noonedeadpunk | mrf: should not really matter, it's minor imo | 15:23 |
| jrosser_ | which is why it's unfortunate that in the case of something that we actually have a workaround for someone proposes to mess with it | 15:24 |
| noonedeadpunk | jrosser_: I probably not understanding reall effect on deployments | 15:24 |
| jrosser_ | oh well the non uwsgi agents get wedged up and break | 15:24 |
| jrosser_ | given enough time / busyness | 15:24 |
| noonedeadpunk | but that's with var set to true? | 15:24 |
| noonedeadpunk | if it's set to false for everything - how that affect uwsgi? | 15:25 |
| jrosser_ | that i'm not so sure about | 15:25 |
| jrosser_ | there is "don't worry it'll all be OK" in the mailing list thread | 15:25 |
| noonedeadpunk | So our workaround is to not wedge services by setting new default they propose. And I was under impression, that disabling pthreads will actually be okeyish for uwsgi. Suboptimal, but okeyish | 15:26 |
| jrosser_ | yeah, could be - i don't know enough about it tbh | 15:28 |
| *** dviroel is now known as dviroel|lunch | 15:29 | |
| *** ysandeep is now known as ysandeep|out | 15:30 | |
| mrf | Is this path correct? certs/certs? cat: /etc/openstack_deploy/pki/certs/certs/haproxy_haproxy02-192.168.1.100.crt: No such file or directory" | 15:31 |
| jrosser_ | mrf there are certs/certs certs/csr and certs/private | 15:33 |
| jrosser_ | distinct from roots/..... | 15:33 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Check the service status during bootstrap against the internal VIP https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/852451 | 15:58 |
| *** dviroel|lunch is now known as dviroel | 16:38 | |
| admin1 | i have/had 2 cinder backends(ceph) , and i added 3rd one (nfs) and now I get a strange error: https://gist.githubusercontent.com/a1git/b0ddc0b718a3fb89d28b037d89f1583f/raw/34e0992495db81e7b591570946f4296981867a4e/gistfile1.txt | 18:00 |
| admin1 | Active-Active configuration is not currently supported by driver cinder.volume.drivers.nfs.NfsDriver | 18:01 |
| admin1 | is something active-active by default ? | 18:01 |
| jrosser_ | cinder can be active-active for rbd | 18:05 |
| admin1 | ok .. | 18:09 |
| admin1 | not sure how to solve it though .. i can try to move it in the first block :D | 18:10 |
| admin1 | get it load first, and then get to cinder and get to active active .. if it reads 1 by 1 or fifo type | 18:10 |
| spatel | how to protect accidental vm deletion? I can see lock feature but any other policy or workaround ? | 18:36 |
| spatel | can i create project admin user who has permission to create users/delete user for specific project? | 18:47 |
| admin1 | jrosser_, as per #openstack-cinder , its due to cluster=ceph on line 12 https://gist.github.com/a1git/fbf329c3027ea51278ef3c3c599d0dfa | 18:50 |
| admin1 | set by cinder_active_active_cluster: "{{ cinder_backend_rbd_inuse }}" | 18:53 |
| admin1 | to disable it is to set cinder_active_active_cluster: false ? | 18:54 |
| admin1 | spatel, do the vm creation using CI ? | 18:55 |
| admin1 | or gitlab or some form of api/automation instead of giving direct access ? | 18:55 |
| admin1 | or a create-only api role | 18:55 |
| jrosser_ | admin1: this is slightly unfair imho | 18:56 |
| jrosser_ | as you know openstack-ansible comes out-of-the-box with "sensible defaults" | 18:57 |
| jrosser_ | and then on top of that it lets you write literally any config you like into any of the services with overrides | 18:57 |
| admin1 | :) | 18:59 |
| admin1 | which is why we all love it | 18:59 |
| jrosser_ | but it means you have to do the homework on the services | 18:59 |
| jrosser_ | so it's totally possible to make bogus configs by trying to use combinations of things which are mutually exclusive | 19:01 |
| jrosser_ | openstack-ansible makes no claim to be able to resolve that for you | 19:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Check the service status during bootstrap against the internal VIP https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/852451 | 19:07 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: rabbitmq: default to using TLS for management user interface https://review.opendev.org/c/openstack/openstack-ansible/+/851380 | 19:24 |
| *** tosky_ is now known as tosky | 19:25 | |
| spatel | admin1 > | 19:39 |
| spatel | ? | 19:39 |
| *** dviroel is now known as dviroel|out | 21:22 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!