| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Accept both HTTP and HTTPS also for external VIP during upgrade https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/864785 | 00:27 |
|---|---|---|
| *** promethe- is now known as prometheanfire | 02:47 | |
| jrosser | noonedeadpunk: experiment with `add_host` https://paste.opendev.org/show/bN4fiVTIe6WJI8BTmEKV/ | 08:50 |
| damiandabrowski | morning | 09:04 |
| jrosser | morning | 09:28 |
| damiandabrowski | I have a question about TLS support for infrastructure services | 10:32 |
| damiandabrowski | 1. Does it make any sense to add TLS support to repo_server? | 10:33 |
| damiandabrowski | 2. Is it worth to add TLS support to memcached when `memcache_security_strategy = ENCRYPT` is being used? | 10:33 |
| noonedeadpunk | o/ | 10:53 |
| Mohaa | morning | 10:56 |
| noonedeadpunk | This failed so weirdly https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/875782 GPG check failed on metal but not on LXC... | 10:57 |
| Mohaa | Infra01 has been read-only! because the disk is full! Does journal-systemd rotate? | 11:02 |
| noonedeadpunk | Mohaa: it does but it's configurable - it can rotate based on timestaps or based on consumed diskspace | 11:08 |
| noonedeadpunk | https://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMaxUse= | 11:10 |
| Mohaa | umm, the disk was not the reason. I rebooted the server. After reboot it's working and only 9% of disk is populated. AppArmor has been blocking something! I'm t-shooting to find the reason | 11:12 |
| noonedeadpunk | o_O all haproxy patches passing after my intervention - weird... | 11:19 |
| noonedeadpunk | I was expecting at least smth to faild :D | 11:19 |
| noonedeadpunk | damiandabrowski: given that oslo.cache does support tls connection to memcached - I'd say we should encrypt it | 11:21 |
| noonedeadpunk | regarding repo server - well, there's nothing to protect to be frank, but given it's not too complex I think we can add tls support there as well. As main thing would be to fetch wheels from it. | 11:22 |
| noonedeadpunk | just for completeness of the feature | 11:22 |
| jrosser | so long as pip understands the system CA rather than just certifi | 11:23 |
| jrosser | but i think we already took care of that | 11:23 |
| jrosser | imho repo server is more about making sure the clients are all fine with it being TLS rather than difficulty with the repo server itself | 11:24 |
| noonedeadpunk | Yes, exactly | 11:32 |
| noonedeadpunk | We jsut use it in multiple places so harmonizing that might be a pita | 11:33 |
| jrosser | though actually - the internal VIP is TLS today? so maybe i talk nonsense here :) | 11:34 |
| jrosser | the clients use the vip.... so perhaps thats just not an issue | 11:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Simplify haproxy_service_configs structure https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/871188 | 11:35 |
| noonedeadpunk | Hm... I'm not sure that for repo we don't have some kind of exception... | 11:36 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/haproxy/haproxy.yml#L476-L486 | 11:36 |
| noonedeadpunk | So we don't have it behind tls as of today | 11:37 |
| noonedeadpunk | Which means it's also little sense to use tls for backends.... | 11:37 |
| noonedeadpunk | damiandabrowski: I'd finish main stuff that then took a look on repo to be frank. But at the end of the day we likely should harmonize this | 11:38 |
| noonedeadpunk | jrosser: but yes, all clients should be just fine with talking through TLS and be aware of our rootCA | 11:38 |
| damiandabrowski | noonedeadpunk: you're right, but last week when I was waiting for reviews in haproxy-separated-config changes i started adding TLS support to our services: https://review.opendev.org/q/topic:tls-backend+status:open | 11:42 |
| damiandabrowski | so i just wanted to clarify these things before i switch context again | 11:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Prepare haproxy role for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/875779 | 11:44 |
| Mohaa | I'm going to enable cinder-backup. | 12:38 |
| Mohaa | OSA does not support for third-part S3 solution. | 12:39 |
| Mohaa | It takes care for Ceph and swift, right? | 12:40 |
| Mohaa | I added this part to the end of `os-cinder-install.yml` and it worked in my lab, but not in the stage mode. I noticed it's because the package `net-tools` is not installed in the stage nodes (python has an error saying demand for the package). But why it needs deprecated net-tools? | 12:46 |
| jrosser | Mohaa: what do you mean about 3rd party S3 not supported? | 12:46 |
| Mohaa | the part added to the end of file: https://paste.opendev.org/show/bCVitOObcN7wi0GbL0LZ/ | 12:46 |
| jrosser | Mohaa: woah! https://docs.openstack.org/openstack-ansible/latest/reference/configuration/using-overrides.html#overriding-openstack-configuration-defaults | 12:48 |
| Mohaa | jrosser: we have DEL ECS storages that have S3 API, serving object storage service. | 12:48 |
| jrosser | you can already override every possible variable in all the service config files | 12:48 |
| jrosser | there should never be a need to modify the playbooks like that | 12:48 |
| jrosser | we use a special ansible module called `config_template` that allows you to inject extra config into any section of the service config files, even if there are not variables defined for those in the ansible roles | 12:49 |
| Mohaa | Wooooops! I forget everything after two weeks! | 12:50 |
| jrosser | so for cinder conf that would be this https://github.com/openstack/openstack-ansible-os_cinder/blob/master/defaults/main.yml#L397 | 12:50 |
| jrosser | and then if you need an extra package installing we should look at that too? | 12:52 |
| Mohaa | w8 | 12:56 |
| Mohaa | jrosser: https://paste.opendev.org/show/blKnyNWomHFenSmQFWAX/ ctrl+f for net_tools | 13:17 |
| jrosser | thats not really enough context around that failed task really | 13:19 |
| jrosser | though really i don't think that is at all to do with a missing package | 13:21 |
| jrosser | see how the same task runs OK against infra01 and infra03, and also what you see is `net_tools` in an ansible module path, not `net-tools` the apt package | 13:23 |
| jrosser | Mohaa: can you check that haproxy is running properly on infra02, and that `/var/run/haproxy.stat` is present on infra02? | 13:24 |
| noonedeadpunk | jrosser: add_host is indeed promising. I will test it out now wrt haproxy | 13:40 |
| jrosser | noonedeadpunk: yes the only downside is needing to use meta: to reset the inventory afterwards - that was a little slow | 13:41 |
| noonedeadpunk | well, it just re-execute dynamic_inventory | 13:42 |
| Mohaa | infra02:~# ls: cannot access '/var/run/haproxy.stat': No such file or directory | 14:05 |
| jrosser | Mohaa: that probably means that haproxy is not running properly on that node | 14:08 |
| jrosser | and i think is the cause of `No such file or directory` in your paste | 14:08 |
| Mohaa | Before running cinder installer, I run haproxy-install again and it was successful! | 14:11 |
| Mohaa | cinder installer output: http://sprunge.us/jMvCEX | 14:11 |
| jrosser | i'm not completely following that - there is still an error | 14:12 |
| jrosser | it is probably better to look at the service status and logs for haproxy rather than just running the playbook | 14:12 |
| Mohaa | +1 | 14:12 |
| Mohaa | It needed to restart haproxy on node 2! | 14:29 |
| *** lowercase is now known as Guest6298 | 14:48 | |
| *** lowercase_ is now known as lowercase | 14:48 | |
| Mohaa | I'm a bit confused about `cinder_cinder_conf_overrides: {}` | 14:59 |
| Mohaa | jrosser: cinder_cinder_conf_overrides: { < https://paste.opendev.org/show/bCVitOObcN7wi0GbL0LZ >} | 15:00 |
| Mohaa | this way ^? | 15:00 |
| jrosser | Mohaa: did you read the example here for nova? https://docs.openstack.org/openstack-ansible/latest/reference/configuration/using-overrides.html#overriding-openstack-configuration-defaults | 15:01 |
| jrosser | Mohaa: `cinder_cinder_conf_overrides` is a yaml dictionary where you describe the config file sections and keys/values you want to write | 15:11 |
| Mohaa | Yes, I'm reading the link again. I'm finding related options in the deployed cinder containers to insert them under `cinder_cinder_conf_overrides` | 15:13 |
| mgariepy | anyone knows it the RHEL variant fails from yesterday are fixed ? | 15:15 |
| mgariepy | hmm some merged at 8pm. so i guess the issues we had at 11am are fixed. | 15:17 |
| damiandabrowski | regarding https://bugs.launchpad.net/openstack-ansible/+bug/2007849 seems like the only thing that may prevent us from removing custom linear strategy plugin are magic variables: | 16:30 |
| damiandabrowski | https://github.com/openstack/openstack-ansible-plugins/blob/master/plugins/strategy/linear.py#L38-L55 | 16:30 |
| damiandabrowski | i used this playbook and keystone container has these variables available even without custom linear plugin: https://paste.openstack.org/raw/b7agdo7CIipLoFboQdYV/ | 16:31 |
| damiandabrowski | but do you think it's the right way to test it? | 16:32 |
| jrosser | damiandabrowski: did you see this in the comment? https://github.com/openstack/openstack-ansible-plugins/blob/master/plugins/strategy/linear.py#L42-L45 | 16:42 |
| damiandabrowski | yes, i'm just not sure if I understand it correctly, because cloudnull wrote on Friday: "I think the only thing that would actually need to be kept is the magic variable mapping" | 16:44 |
| damiandabrowski | so based on the comment you linked, i think we can drop it but i just wanted to double check that if you | 16:44 |
| jrosser | i'm wondering if it refers to things like this https://github.com/openstack/openstack-ansible-plugins/blob/master/plugins/connection/ssh.py#L29-L33 | 16:46 |
| jrosser | perhaps comment one of those out and then it would be possible to confirm that it is what controls that var being present | 16:47 |
| damiandabrowski | good idea, give me a sec | 16:47 |
| damiandabrowski | KeyError: 'Requested entry (plugin_type: connection plugin: ansible_collections.openstack.osa.plugins.connection.ssh setting: physical_host ) was not defined in configuration.' | 16:49 |
| damiandabrowski | yeah, it works exactly as you say | 16:49 |
| jrosser | excellent - so that means we can remove the whole linear strategy? | 16:50 |
| damiandabrowski | yup, i think so. There are 2 changes for that: | 16:51 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible/+/874482 | 16:51 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/874425 | 16:51 |
| jrosser | i wonder if this gets us any closer to mitogen working | 16:53 |
| jrosser | it was always some wierd thing with delegation that broke it before | 16:54 |
| damiandabrowski | ouh, never heard of it but looks promising | 16:59 |
| noonedeadpunk | we had attempted mitogen like... 3 years ago first time? | 17:03 |
| jrosser | i keep prodding at it :) | 17:05 |
| jrosser | just like arm | 17:05 |
| noonedeadpunk | hehe | 17:06 |
| jrosser | noonedeadpunk: do you rememeber why we don't yet update the openstack collection to 2.0.0? | 17:31 |
| cloudnull | I think the linear strategy still needs to provide https://github.com/openstack/openstack-ansible-plugins/blob/master/plugins/strategy/linear.py#L38-L55 - otherwise IDK if the ssh plugin will know what to do with the container tech options. But to be frank IDK if that's true anymore. | 17:33 |
| noonedeadpunk | I do - it needs openstacksdk newer then in u-c | 17:33 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-rabbitmq_server master: Do not use 'always' tag in inappropriate places https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/875971 | 17:34 |
| noonedeadpunk | And to bump openstacksdk there were some other blockers | 17:34 |
| jrosser | cloudnull: what do you think about the comment there - saying the docstring can provide the vars? | 17:34 |
| jrosser | we have `openstacksdk===1.0.1` in u-c now | 17:35 |
| cloudnull | I really have no idea :D I just remember that not working , however , my memory is OLD. | 17:36 |
| cloudnull | its been a long time since 2.5 - so it may all be good now. | 17:36 |
| cloudnull | in which case +1 delete the linear strat bits if at all possible. | 17:36 |
| cloudnull | I would be curious if mitogen could work without the strat, in which case the ssh plugin could be eliminated too, in-favor of mitogens native container connection capabilities. | 17:38 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use 2.0.0 release for ansible-collections-openstack https://review.opendev.org/c/openstack/openstack-ansible/+/873092 | 17:38 |
| jrosser | hrm so OVN doesnt log to the journal? | 17:51 |
| jrosser | i see things in /var/log/ovn/<random-files> | 17:51 |
| jamesdenton_ | it does not | 17:54 |
| *** jamesdenton_ is now known as jamesdenton | 17:54 | |
| jamesdenton | i have been unable to find a way to get the services to log to journal so far | 17:54 |
| noonedeadpunk | Also gluster does not | 18:00 |
| noonedeadpunk | jamesdenton: but can you provide syslog path or smth like that? | 18:01 |
| noonedeadpunk | As you can point logs to /dev/log which is kind of does the trick | 18:01 |
| jrosser | i do wonder what this is talking about too https://docs.openstack.org/neutron/latest/configuration/ovn.html | 18:04 |
| noonedeadpunk | I think neutron ovn agent? | 18:05 |
| noonedeadpunk | Not sure htough | 18:05 |
| noonedeadpunk | I think what we're looking for is smth like --syslog-method https://www.ovn.org/support/dist-docs/ovn-controller.8.html | 18:06 |
| jamesdenton | so that ovn.ini... i have not found anything that actually uses it | 18:08 |
| jamesdenton | many of the config options in there are shoehorned into ml2_conf.ini | 18:09 |
| jamesdenton | ovn-host and ovn-central use /etc/default/ files, and OVN_CTL_OPTS, but i'm not sure if there's an option for logging that can be put there | 18:10 |
| noonedeadpunk | isn't --syslog-method excatly the candidate for OVN_CTL_OPTS? | 18:13 |
| mgariepy | https://opendev.org/openstack/networking-ovn/commit/a6ff3490c4a1057e58cbeba8e2467d89f2c47593 | 18:27 |
| mgariepy | ovn.ini ? | 18:27 |
| noonedeadpunk | jrosser: regarding virt groups - you don't need to run add_host on localhost - you technically can do this as pre/post_tasks. But then if using serial it wil lbe executed independently for each run which is unfortunate :( | 19:16 |
| noonedeadpunk | but yeah, with localhost is looks nice and serial seems to work | 19:17 |
| spatel | jamesdenton we it comes to upgrade we should make sure ovn-controller get upgrade first before ovn central components like ovn-north/ovn-cental etc. | 19:57 |
| spatel | when* | 19:57 |
| spatel | Not sure if we should document that process or just ansible can handle that part | 19:58 |
| jamesdenton | hmmm, prob needs to be ansible but not sure what the order is now? | 20:05 |
| admin10 | i forgot .. what was the variable to whitelist the galera socket check source ip range | 20:11 |
| admin10 | found it .. | 20:12 |
| admin10 | galera_monitoring_allowed_source | 20:12 |
| admin10 | is it an array , or comma separated variables or a cidr ? | 20:13 |
| noonedeadpunk | admin10: it's cidrs separated by whitespace | 20:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Prepare service roles for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible/+/871189 | 20:41 |
| noonedeadpunk | jrosser: I indeed like this more ^ | 20:41 |
| admin10 | noonedeadpunk thanks | 20:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Prepare service roles for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible/+/871189 | 20:42 |
| jrosser | noonedeadpunk: it does look tidy | 20:50 |
| jrosser | noonedeadpunk: I am still not sure about the logic used to disable the temporary certbot backend - really where actually that is done….. | 20:51 |
| jrosser | because it needs to stay there - right up to the point that horizon is (possibly) installed | 20:52 |
| jrosser | but the playbooks may be run individually or in multiple runs so I’m not really seeing how the required present/absent state for the temporary certbot backend is decided in the horizon playbook | 20:53 |
| opendevreview | Merged openstack/openstack-ansible master: Do not use openstack.osa.linear strategy plugin https://review.opendev.org/c/openstack/openstack-ansible/+/874482 | 21:20 |
| damiandabrowski | jrosser: did you check latest changes(from last week) regarding certbot logic? i improved it before my vacation | 22:28 |
| damiandabrowski | now it's pretty simple, certbot backend is being enabled by haproxy playbook | 22:28 |
| damiandabrowski | and it's disabled by horizon playbook. We don't need any extra task to do that, we just add certbot service to horizon_haproxy_services with enabled=False | 22:29 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible/+/871189/16/inventory/group_vars/horizon_all.yml#50 | 22:29 |
| damiandabrowski | so to summarize: | 22:51 |
| damiandabrowski | - haproxy-install.yml enables certbot service if horizon is not deployed(yet) and LE is enabled with http-01 challenge | 22:51 |
| damiandabrowski | - horizon-install.yml disables certbot service(always) | 22:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!