| *** spatel_ has joined #openstack-ansible | 02:16 | |
| *** spatel_ is now known as spatel | 02:16 | |
| *** gyee has quit IRC | 02:20 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #openstack-ansible | 02:33 | |
| *** spatel has quit IRC | 03:08 | |
| *** macz_ has joined #openstack-ansible | 05:37 | |
| *** macz_ has quit IRC | 05:43 | |
| *** miloa has joined #openstack-ansible | 05:57 | |
| *** miloa has quit IRC | 06:12 | |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-role-pki master: Create CA and server certificates https://review.opendev.org/c/openstack/ansible-role-pki/+/788021 | 06:20 |
|---|---|---|
| *** rh-jlabarre has joined #openstack-ansible | 06:22 | |
| *** rh-jelabarre has quit IRC | 06:22 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Use PKI role to install CA certificates https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/790431 | 06:26 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Modernise TLS configuration https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/789789 | 06:33 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Use ansible-role-pki to generate SSL certificates https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/788032 | 06:33 |
| *** kleini has joined #openstack-ansible | 06:33 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server master: Use integrated tests for haproxy_server https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/790090 | 06:37 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server master: Use external PKI role to manage haproxy self-signed certificates https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/790078 | 06:37 |
| *** ChipOManiac has joined #openstack-ansible | 06:38 | |
| *** pto_ has joined #openstack-ansible | 06:47 | |
| *** pto has quit IRC | 06:50 | |
| *** pto_ has quit IRC | 06:51 | |
| *** tinwood has quit IRC | 06:54 | |
| *** tinwood has joined #openstack-ansible | 06:57 | |
| *** jhesketh has quit IRC | 07:08 | |
| *** tosky has joined #openstack-ansible | 07:09 | |
| *** pto has joined #openstack-ansible | 07:14 | |
| *** macz_ has joined #openstack-ansible | 07:17 | |
| *** andrewbonney has joined #openstack-ansible | 07:17 | |
| *** macz_ has quit IRC | 07:21 | |
| *** rpittau|afk is now known as rpittau | 07:26 | |
| noonedeadpunk | mornings | 07:57 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Rename whitelist and blacklist in role https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/784885 | 08:04 |
| ChipOManiac | noonedeadpunk morning. | 08:08 |
| *** macz_ has joined #openstack-ansible | 08:09 | |
| jrosser | morning | 08:11 |
| *** macz_ has quit IRC | 08:13 | |
| *** mgariepy has quit IRC | 08:15 | |
| admin0 | mornings | 08:17 |
| *** mgariepy has joined #openstack-ansible | 08:18 | |
| CeeMac | morning | 08:18 |
| openstackgerrit | Jonathan Herlin proposed openstack/openstack-ansible-os_tempest master: Rename whitelist and blacklist in role https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/784885 | 08:18 |
| jonher | morning | 08:19 |
| noonedeadpunk | thanks for fixing :) | 08:22 |
| *** jhesketh has joined #openstack-ansible | 08:23 | |
| *** pto has quit IRC | 08:29 | |
| *** dave-mccowan has joined #openstack-ansible | 08:31 | |
| *** pto has joined #openstack-ansible | 08:31 | |
| jonher | yeah i was waiting for centos things to be fixed before triggering another ci pipeline with just comments, but since you edited it now i just fixed the spelling | 08:36 |
| *** pto_ has joined #openstack-ansible | 08:37 | |
| *** pto has quit IRC | 08:40 | |
| noonedeadpunk | I hope they are now actually | 08:43 |
| noonedeadpunk | https://review.rdoproject.org/r/c/rdoinfo/+/33420 | 08:43 |
| jonher | oh ok :) we'll see if zuul agrees | 08:46 |
| *** sshnaidm_ has joined #openstack-ansible | 08:48 | |
| *** pto has joined #openstack-ansible | 08:50 | |
| *** sshnaidm has quit IRC | 08:50 | |
| *** pto has quit IRC | 08:51 | |
| *** pto has joined #openstack-ansible | 08:52 | |
| jrosser | noonedeadpunk: after setting the transport_url to a template with nova-manage, should we restart the services? | 08:55 |
| jrosser | i'm now unable to reproduce the nova-conductor SSL error here locally, maybe because i run the playbooks / restart things a a few times | 08:55 |
| jrosser | i.e we have no notify: stuff around this https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_db_setup.yml#L68-L99 | 08:56 |
| noonedeadpunk | jrosser: so, if it's not template - then service restart is not required. If it's template and you change config file then you need to restart iirc as template is evaluated on service startup | 08:58 |
| jrosser | i am still a bit confused about how this works tbh as we have a template with nova-manage and also transport_url in the config file | 08:59 |
| noonedeadpunk | so on startup nova parse transport_url from the config file. it sets local variables as of scheme, username, password, etc, based on the parsing result (like just named groups of regexp). and substitute template based on the stuf it parses from config file | 09:01 |
| jrosser | and this is becasue normally transport_url is in the DB for nova? | 09:05 |
| noonedeadpunk | no, it should be in both places actually anyway | 09:05 |
| jrosser | lol :) | 09:05 |
| noonedeadpunk | from cell_mappings it's used by conductor only, and all other services use config | 09:06 |
| noonedeadpunk | but template kind of allows to specify just in config and don't really care about db since it will be the same | 09:06 |
| *** rpittau is now known as rpittau|bbl | 09:07 | |
| jrosser | ok that makes sense | 09:07 |
| noonedeadpunk | otherwise, in case you change nova password, you have to update db as well with new credentials | 09:07 |
| noonedeadpunk | as conductor will try to connect with old ones | 09:07 |
| noonedeadpunk | (eventually that's how I found that all out) | 09:07 |
| noonedeadpunk | But, it parses only what you have for transport_url and don't really care about other options/sections of the config file | 09:08 |
| noonedeadpunk | so it takes just transport_url and apply regexp to it | 09:09 |
| noonedeadpunk | but I think I just don't really understood the question :( | 09:14 |
| *** pto has quit IRC | 09:19 | |
| jrosser | well in the simplest sense the question is wtf is happening here :) https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4b3/788032/6/check/openstack-ansible-deploy-aio_metal-ubuntu-focal/4b3e6f1/logs/host/nova-conductor.service.journal-08-05-13.log.txt | 09:20 |
| jrosser | and it's kind of tough because the logging is not really actually helpful for what has failed | 09:20 |
| jrosser | SSL version error could even be its still trying the non-ssl port for rabbitmq for example | 09:21 |
| noonedeadpunk | I think logging is not helpful as we don't really see how template has been filled... | 09:23 |
| *** sshnaidm_ is now known as sshnaidm | 09:28 | |
| noonedeadpunk | oh, that's actually change that implements it https://review.opendev.org/c/openstack/nova/+/578163/4/nova/objects/cell_mapping.py | 09:28 |
| noonedeadpunk | there's no log.debug there... | 09:29 |
| *** pto has joined #openstack-ansible | 09:30 | |
| noonedeadpunk | let me probably try to spawn up aio as well.... | 09:31 |
| jrosser | something makes me wonder if the logging isn't wired up right, not sure we ever see this https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L608-L610 | 09:43 |
| noonedeadpunk | I never saw that indeed | 09:43 |
| noonedeadpunk | I wonder if it expects debug to be set elsewhere | 09:44 |
| * jrosser asks in oslo channel | 09:47 | |
| *** macz_ has joined #openstack-ansible | 10:09 | |
| noonedeadpunk | it's a bit deserted though :( | 10:11 |
| *** macz_ has quit IRC | 10:14 | |
| *** rpittau|bbl is now known as rpittau | 10:19 | |
| *** jnamdar has joined #openstack-ansible | 10:19 | |
| *** Mr_Smurf has joined #openstack-ansible | 10:25 | |
| jnamdar | Hi everyone! I'm trying to spin up an AIO to checkout all of the supported openstack services in openstack-ansible. I was trying out the AIO doc in stable/ussuri and basically using a SCENARIO environnement variable with all the services (like aio_lxc_swift_etc...), but I keep getting into issues with some services like swift, ceph and others (some | 10:26 |
| jnamdar | of these issues are already logged on the launchpad). Has anyone got to setup such a thing (all of the services at once)? Would you recommend me to switch to the latest tag of openstack-ansible? FYI I got to install the default scenario without issues to try it out. | 10:26 |
| noonedeadpunk | I think we never tried using that with _all_ components | 10:27 |
| noonedeadpunk | also would be great to have some of the examples, as actually fixes might already be landed. | 10:28 |
| noonedeadpunk | imo swift doesn't make much sense when ceph is in place as they're going to conflict (and eventually implement the same functionality) | 10:29 |
| Mr_Smurf | sorry to bother you guys... but I'm stock trying to deploy ceph-rgw-install.yml. I'm missing something because no matter how much i run the openstack-hosts-setup.yml and setup-infrastructure.yml playbooks I don't get any containers. I just get the hostnames generated for them. | 10:30 |
| *** macz_ has joined #openstack-ansible | 10:30 | |
| Mr_Smurf | using stable/victoria | 10:31 |
| jnamdar | noonedeadpunk yes that's my thought as well about ceph. I don't really know how to disable it though, I did remove its .yml file from conf.d before launching the setup. For instance I encountered this bug among others https://bugs.launchpad.net/openstack-ansible/+bug/1879192 | 10:31 |
| openstack | Launchpad bug 1879192 in openstack-ansible "AIO with swift gnocchi and ceilometer will not install" [Undecided,New] | 10:31 |
| *** corbani has quit IRC | 10:32 | |
| noonedeadpunk | interesting. Actualy I somehow missed that bug :( | 10:33 |
| noonedeadpunk | well, personally I'd prefer ceph instead of the classic swift :) | 10:33 |
| *** macz_ has quit IRC | 10:34 | |
| noonedeadpunk | Mr_Smurf: so, when you run `./scripts/inventory-manage.py -g` you see all of the container names, but doesn't for `lxc-ls`? | 10:36 |
| noonedeadpunk | jnamdar: ceph basically shouldn't be deployed unless you provide it in scenario.... | 10:37 |
| Mr_Smurf | noonedeadpunk: yes | 10:37 |
| noonedeadpunk | let me try to spawn aio to catch the bug. Not an expert in swift :( | 10:38 |
| Mr_Smurf | noonedeadpunk: I am using an exteral ceph so I only want to deply the rgw service. I have configured ceph-rgw_hosts (ceph-osd_hosts and ceph-osd_hosts are set to {}) | 10:40 |
| noonedeadpunk | Mr_Smurf: ok, so you're missing only rgw container or all of them? | 10:41 |
| Mr_Smurf | noonedeadpunk: all of them | 10:41 |
| Mr_Smurf | noonedeadpunk: correction.. all of the rgw containers.. I have other containsers | 10:42 |
| Mr_Smurf | noonedeadpunk: the cloud is functional and I have lots of other containers but I can't get it to create the rgw containers | 10:43 |
| noonedeadpunk | oh, well. I think to create containers you would need to run setup-hosts.yml instead of the openstack-hosts-setup.yml (openstack-hosts-setup.yml is included in setup-hosts.yml though) | 10:43 |
| noonedeadpunk | ah, ok | 10:43 |
| noonedeadpunk | well, try doing this | 10:43 |
| Mr_Smurf | noonedeadpunk, I'll try that | 10:43 |
| noonedeadpunk | wait :) | 10:43 |
| noonedeadpunk | openstack-ansible playbooks/lxc-containers-create.yml --limit ceph-rgw,lxc_hosts | 10:44 |
| *** sshnaidm is now known as sshnaidm|afk | 10:46 | |
| jnamdar | noonedeadpunk yes I know right, I didn't provide it in the scenario env. variable so I don't know why it started installing | 10:48 |
| Mr_Smurf | noonedeadpunk: well when you say it... it's all obvious.. I'll run that :) Thanks | 10:48 |
| Mr_Smurf | noonedeadpunk: it looks like the containtser are beeing created now, thank you. | 10:50 |
| *** shyamb has joined #openstack-ansible | 11:14 | |
| noonedeadpunk | Mr_Smurf: ok, sweet :) | 11:14 |
| noonedeadpunk | jnamdar: maybe you've added manila? | 11:15 |
| jrosser | noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_log_levels | 11:17 |
| noonedeadpunk | o_O | 11:17 |
| noonedeadpunk | I'd expect inheriting debug = true tbh | 11:19 |
| noonedeadpunk | are ppl really expected to set all this for debug of wtf is going on? | 11:20 |
| jrosser | also like really? 'oslo.messaging=INFO', 'oslo_messaging=INFO' | 11:21 |
| jrosser | and i am wondering if i need the entire giant string just to override one of those :/ | 11:22 |
| noonedeadpunk | I guess just `oslo.messaging=INFO,oslo_messaging=INFO` | 11:22 |
| noonedeadpunk | well, depends if all of these have defaults... | 11:22 |
| Mr_Smurf | noonedeadpunk: now I just have to figure out how to prevent it from trying to generate fsid and use the one I have in my ceph cluster :) | 11:22 |
| noonedeadpunk | I think it's set with some variable... | 11:23 |
| Mr_Smurf | noonedeadpunk: yes.. I'm browsing documentaion... | 11:23 |
| noonedeadpunk | oh, like `generate_fsid: false` | 11:23 |
| noonedeadpunk | and `fsid: $UUID` | 11:24 |
| *** shyamb has quit IRC | 11:24 | |
| jrosser | i wonder if deploying radosgw with OSA but the rest of ceph externally is something tried before | 11:25 |
| Mr_Smurf | noonedeadpunk: thanks, generate_fsid: false was missing in my config | 11:25 |
| jrosser | that would be interesting to document what is needed | 11:26 |
| Mr_Smurf | I've not found any documentaion on that.. I can make some notes on what I'm doing | 11:26 |
| jrosser | that would be great, we have a section in the docs for various kinds of ceph integrations but not quite this scenario | 11:28 |
| jonher | ^ probably want to do the same soon. external ceph but rgw on OSA nodes | 11:29 |
| Mr_Smurf | jonher: then I can take notes and you can verify them before posting it in the documentation | 11:31 |
| jonher | sounds good | 11:32 |
| jnamdar | noonedeadpunk mmh yes I may have added manila in the scenario triggering a ceph vs swift conflict. Is there any doc that could point me to a setup with lots of services so I can avoid these kind of conflicts? | 11:43 |
| noonedeadpunk | um, no, I don't think we have one. but you can check https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/vars/main.yml for implicitly included scenarios | 11:45 |
| jnamdar | noonedeadpunk Oh I see, so if I use SCENARIO='aio_lxc_telemetry' for instance, that should be safe ? But 'aio_lxc_ironic_manila' would not? | 11:49 |
| noonedeadpunk | jnamdar: yeah, kind of | 11:55 |
| noonedeadpunk | once you said that, I think we kind of need the way to override this behaviour... | 11:56 |
| noonedeadpunk | as we were doing that mainly for CI purposes... | 11:56 |
| jrosser | i guess that SCENARIO is designed to cover a fixed set of things for automating CI and local development | 11:56 |
| Mr_Smurf | jonher: well, it did not work all the way... TASK [ceph-rgw : get keys from monitors] *** fatal: [host-ceph-rgw_container-36c58c34]: FAILED! => {"msg": "list object has no element 0"} | 11:57 |
| jrosser | if you want a more complex setup than that can accomodate you're probably at the point of having to understand how to create a custom config anyway | 11:57 |
| jnamdar | Before I started using SCENARIO, I was manually adding .yml files for each role in the conf.d folder, but that wasn't working as well (some config was missing) | 11:58 |
| jonher | Mr_Smurf my first guess would be yaml formatting seeing that error | 11:58 |
| jnamdar | @jros | 11:58 |
| jnamdar | @jrosser yeah, I think so too | 11:58 |
| jonher | oh i see, it tries to fetch from nodes that are not part of the deployment :) | 11:59 |
| Mr_Smurf | jonher: it might be looking in the list for OSA monitors that is not defined | 11:59 |
| noonedeadpunk | yeah, was just about to write that you still need to provide monitors for ceph-ansible, or you can probably jsut provide keys (can't recall exactly if they have option for that) | 12:01 |
| jrosser | there is documentation for that here https://docs.openstack.org/openstack-ansible-ceph_client/latest/config-from-file.html | 12:02 |
| jrosser | i think in this case becasue the mons are not part of the deployment you can't have ceph-ansible ssh to them, as they're not in the inventory | 12:04 |
| jrosser | so keys will need to be provided via the deployment host to the rgw | 12:05 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Add variables for rabbitmq ssl configuration https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/790037 | 12:07 |
| Mr_Smurf | jrosser: I'm running an strace to see if it is even looking in the ceph_keyrings_dir | 12:09 |
| jrosser | Mr_Smurf: there are very specific ansible tasks for that here https://github.com/openstack/openstack-ansible-ceph_client/blob/master/tasks/ceph_get_keyrings_from_files.yml | 12:11 |
| Mr_Smurf | jonher: yes, it's not even looking.. so I need to skip that task in some way | 12:11 |
| jrosser | so i thiink that an important thing first is that the condition here is met https://github.com/openstack/openstack-ansible-ceph_client/blob/master/tasks/ceph_auth.yml#L23-L24 | 12:11 |
| jrosser | it should be possible to debug most of this with adding maybe -vv to the openstack-ansible command | 12:12 |
| jrosser | and also add a few debug: var=<foo> tasks into the roles if you need to see some values at runtime | 12:13 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Add variables for rabbitmq ssl configuration https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/790037 | 12:14 |
| *** sshnaidm|afk is now known as sshnaidm | 12:19 | |
| *** ChipOManiac has quit IRC | 12:21 | |
| *** shyamb has joined #openstack-ansible | 12:23 | |
| *** macz_ has joined #openstack-ansible | 12:28 | |
| noonedeadpunk | oh my ^ | 12:30 |
| *** macz_ has quit IRC | 12:33 | |
| noonedeadpunk | jrosser: btw, have you seen that? It happens during parallel git clone http://paste.openstack.org/show/805279/ | 12:34 |
| noonedeadpunk | on focal (works nicely on the bionic at the same time) | 12:34 |
| noonedeadpunk | oh, well, this happens I think jsut during re-run... | 12:41 |
| noonedeadpunk | actually that's pretty logical... since we don't have tags for our repos... | 12:46 |
| noonedeadpunk | at least with depth=20 for sure | 12:46 |
| *** spatel has joined #openstack-ansible | 12:52 | |
| *** jpward has joined #openstack-ansible | 12:54 | |
| *** shyamb has quit IRC | 12:57 | |
| noonedeadpunk | some logic feels a bit weird to me... | 13:00 |
| noonedeadpunk | why don't we check for this for hash scenario? https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/library/git_requirements.py#L192 | 13:01 |
| *** macz_ has joined #openstack-ansible | 13:05 | |
| noonedeadpunk | and why we do all these fetching if we need to checkout to master... | 13:06 |
| *** macz_ has quit IRC | 13:09 | |
| *** pto has quit IRC | 13:15 | |
| *** dave-mccowan has quit IRC | 13:24 | |
| Mr_Smurf | jonher: I just commented out the part the copied the keys in the role file and then made new keys manually and placed in the containers | 13:35 |
| Mr_Smurf | jrosser: ^^ | 13:36 |
| jonher | ok we have done a takeover in a separate repo from OSA so i'll likely have mons available for that step, or configure the dir that it _should_ look in | 13:36 |
| Mr_Smurf | jonher: anyway I now hav a working set of rgw:s and a ceph cluster complaining about stray hosts but that is to be expected | 13:38 |
| noonedeadpunk | ok, parallel clone is kind of broken IMO.... | 13:43 |
| noonedeadpunk | Ie, i set `ok: [localhost] => (item={'name': 'os_ironic', 'scm': 'git', 'src': 'https://opendev.org/openstack/openstack-ansible-os_ironic', 'version': '109698e942a3c20a3e4d0fc66ffa94e741d2738e', 'trackbranch': 'master'}) ` | 13:43 |
| noonedeadpunk | but have `On branch master` with git status | 13:43 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: Prepare Train to EM https://review.opendev.org/c/openstack/openstack-ansible/+/790655 | 13:49 |
| *** gshippey has joined #openstack-ansible | 13:53 | |
| noonedeadpunk | oh, hm... it's actually working... just git status confuses a lot... | 13:57 |
| jrosser | noonedeadpunk: gshippey has been doing related work here recently for parallel clone, was there something to look at | 14:00 |
| * jrosser -ETO_MANY_MEETINGS | 14:00 | |
| noonedeadpunk | reset_to_version is really weird.... | 14:00 |
| noonedeadpunk | I mean even if we reset to tag, it would be still master, but diverged one... | 14:01 |
| noonedeadpunk | same applies to sha | 14:01 |
| *** sgautam has joined #openstack-ansible | 14:57 | |
| sgautam | Greeting everyone. | 15:02 |
| sgautam | I was trying to add the custom built service to the openstack stack though Openstack Ansible. I can see that repo in the repo_packages are public repo. Is there a way for us to use the private repo? | 15:02 |
| *** macz_ has joined #openstack-ansible | 15:06 | |
| noonedeadpunk | sgautam: Um, I guess as repo you can provide `git@github.com:mylogin/hello.git` ? | 15:06 |
| noonedeadpunk | eventually we leverage ansible's git module for cloning repos | 15:06 |
| jrosser | if you want to extend repo_packages, variables of the same pattern can go in user_variables.yml | 15:07 |
| noonedeadpunk | oh, right, I was talking more about roles... | 15:08 |
| noonedeadpunk | but eventually we use stuff from repo_packages as regular variables in our roles during venv build process | 15:09 |
| *** macz_ has quit IRC | 15:10 | |
| sgautam | So we might need to provide private key? As currently that project is private. I though the one provided in repo_packages are public repository. | 15:11 |
| *** spatel has quit IRC | 15:14 | |
| *** cyberpear has quit IRC | 15:15 | |
| *** spatel_ has joined #openstack-ansible | 15:19 | |
| *** spatel_ is now known as spatel | 15:19 | |
| noonedeadpunk | It feels for me I'm not sure about usecase you see? So you want to install extra service from private repo. But you would need to also have a role that will deploy it? | 15:22 |
| noonedeadpunk | or it should integrate with some openstack service? | 15:22 |
| noonedeadpunk | ie be part of nova or whatever? | 15:22 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Always use checkout for git_requirements library https://review.opendev.org/c/openstack/openstack-ansible/+/791011 | 15:24 |
| noonedeadpunk | jrosser: I come up to this ^ but not really sure if I've missed some valid usecase for git reset... I couldn't think about any tbh | 15:25 |
| jrosser | sgautam: you can also clone the repo to a local mirror if you have one and reference that, no authentication would be needed then | 15:26 |
| *** macz_ has joined #openstack-ansible | 15:26 | |
| *** gyee has joined #openstack-ansible | 15:26 | |
| jnamdar | noonedeadpunk : I was wondering which version and OS is the most tested for OSA? Would that be the latest OSA tag on Ubuntu 20 for instance ? | 15:27 |
| jnamdar | (I am including OSA roles as well like os_trove etc.) | 15:27 |
| evrardjp | One thing I regret since we split the roles into their own repo (rightfully so), is the complexity for ppl to understand how they can have their own roles. | 15:28 |
| jrosser | i never did find a nice pattern for own playbook to go with own role though | 15:29 |
| evrardjp | If you are interested, I still have an idea about that, but it would require the usual suspects for that to be around the table, and decide :) | 15:29 |
| noonedeadpunk | jnamdar: I think trove is valid actually only for master atm | 15:29 |
| jrosser | i think theres a chicken/egg about not being able to include_playbook unless you know the name ahead of time | 15:29 |
| evrardjp | jrosser: could you clarify why is that a problem? | 15:30 |
| noonedeadpunk | jnamdar: it should be pretty broken even for victoria | 15:30 |
| evrardjp | I am not sure to understand the problem :) | 15:30 |
| noonedeadpunk | but yes, that would be ubuntu focal | 15:30 |
| evrardjp | and hello folks | 15:30 |
| jrosser | well i guess at the end of setup_openstack you want to also run {{ user_playbooks }} | 15:31 |
| -openstackstatus- NOTICE: Any builds with POST_FAILURE result and no available logs between 11:41 and 14:41 UTC today were related to an authentication endpoint problem in one of our providers and can be safely rechecked now | 15:31 | |
| jrosser | and maybe end of setup_hosts and setup_infrastructure too | 15:31 |
| noonedeadpunk | evrardjp: I think in this case we're talking about really service, not the role. | 15:31 |
| noonedeadpunk | well, I think we can jsut include smth like user_playbook and not fail if it doesn't exist? | 15:32 |
| jrosser | yeah, well we have nice extensibility currently for roles with openstack_deploy/user_role_requirements.yml | 15:32 |
| jrosser | but no equivalent hook point for playbooks to deploy those extra roles you bring in | 15:32 |
| jrosser | evrardjp: ^ i think thats really the thing we miss to bring in a user defined service | 15:33 |
| noonedeadpunk | or can even check for existance of files in /etc/openstack_deploy/playbooks and just inlcude them | 15:34 |
| noonedeadpunk | shouldn't be really tough. but not sure how it's required though | 15:34 |
| jrosser | right, but adding stuff into that dir is nasty really from a version control POV | 15:34 |
| noonedeadpunk | as eventually you run setup-openstack sooooo rarely in prod | 15:34 |
| noonedeadpunk | um why so? git submodules? | 15:35 |
| evrardjp | noonedeadpunk: I have another idea , but you are closer :) | 15:35 |
| *** spatel has quit IRC | 15:35 | |
| jrosser | :) i think i try literally everything else before submodules | 15:35 |
| jnamdar | noonedeadpunk is there any board where I can check the testing results of these roles? | 15:35 |
| evrardjp | jrosser: we can blame the lack of ambition around python for this | 15:35 |
| noonedeadpunk | jrosser: ln -s ?:) | 15:36 |
| jrosser | that would do it :) | 15:36 |
| jrosser | with a gitignore entry | 15:36 |
| evrardjp | however, I feel like it OSA could have be smarter in terms of playbooks | 15:36 |
| jrosser | ideas welcome :) | 15:36 |
| evrardjp | When the HP folks finally wanted to collaborate with OSA, I loved their framework and extensibility | 15:37 |
| noonedeadpunk | I mean it doesn't matter much. But what I'd do is just to have another repo in /opt/ with playbooks/custom stuff | 15:37 |
| evrardjp | I quickly realised it was however too complex :) | 15:37 |
| evrardjp | noonedeadpunk: yeah, but it's annoying to say: run these 5 playbooks from there, then that one from here, then that 5 last ones from there | 15:38 |
| evrardjp | so you end up writing a script to do it, then you realise it's also annoying when something fails | 15:38 |
| evrardjp | but I guess we have to live with Ansible's nature :) | 15:38 |
| noonedeadpunk | evrardjp: well, for adding new compute we have that actually https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/add-compute.sh | 15:38 |
| evrardjp | yeah I am not surprised | 15:39 |
| evrardjp | :D | 15:39 |
| noonedeadpunk | where you can define pre and post tasks with env vars | 15:39 |
| noonedeadpunk | not ideal... | 15:39 |
| evrardjp | At some point, I wanted to have OSA without a-r-r | 15:40 |
| evrardjp | so it would force people to deal with osa tree, and deal with git. | 15:40 |
| noonedeadpunk | um..... | 15:41 |
| evrardjp | and to avoid the mess ups, I wanted to have all the roles as subtrees (intended to pull only) inside roles/ | 15:41 |
| evrardjp | so basically when you clone OSA, you have everything, but you will still need to do your homework in terms of branching, etc. | 15:41 |
| evrardjp | The benefit I saw was that people, contributin to the tree all the time, would be just one step away from contributing to OSA. | 15:42 |
| noonedeadpunk | well I was thinking more about collections I guess. | 15:42 |
| evrardjp | Oh those are old ideas , I don't mean they are relevant today :) | 15:43 |
| evrardjp | I just fixed my bouncer, saw light, so I just came by to say hello. | 15:43 |
| noonedeadpunk | And I guess what stopps from contributing upstream is not being aware how easy that could be and you are not unique with your usecase - it can be helpful for others | 15:44 |
| *** spatel_ has joined #openstack-ansible | 15:44 | |
| *** spatel_ is now known as spatel | 15:44 | |
| noonedeadpunk | so ppl continue to maintain stuff locally as they got used to with all that corporate stuff they've used before | 15:45 |
| evrardjp | so true | 15:45 |
| noonedeadpunk | it's more mindset thing I guess | 15:45 |
| evrardjp | yeah it was, and still is | 15:45 |
| evrardjp | glad we have good folks around this channel though! :) | 15:48 |
| evrardjp | how is everyone? | 15:49 |
| noonedeadpunk | doing babysteps I think ) | 15:49 |
| noonedeadpunk | yeah, you should come by more often to keep level of the channel :p | 15:50 |
| jrosser | theres good stuff happening, though keeping on top of operating system churn is kind of a big overhead | 15:50 |
| noonedeadpunk | especially when we're talking about centos | 15:51 |
| jrosser | support matrix feels barely sustainable sometimes | 15:51 |
| noonedeadpunk | oh! support matrix | 15:51 |
| jrosser | ah whoops :) | 15:51 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Always use checkout for git_requirements library https://review.opendev.org/c/openstack/openstack-ansible/+/791011 | 15:52 |
| evrardjp | jrosser: yes that's annoying. But I doubt anytime soon that ppl will move to a rolling release | 15:53 |
| evrardjp | I mean, I would totally love to see openstack on arch linux | 15:53 |
| evrardjp | you have latest python, so you also have the latest bugs | 15:53 |
| evrardjp | but it's also the right place to fix stuff and never care about doing an upgrade anymore | 15:53 |
| noonedeadpunk | well, I think from deployment tool prespective, rolling releases are not much better as they would require attention more often? | 15:54 |
| evrardjp | nevertheless, I see people not doing CI/CD with openstack, or having a very loose definition of _continuous_ :) | 15:54 |
| evrardjp | do they? | 15:54 |
| evrardjp | Assuming openstack is ready for those bits (which it isn't, let's be clear) | 15:54 |
| jrosser | stuff takes too long though | 15:55 |
| noonedeadpunk | well, you got new libvirt version, that nova has no idea of... | 15:55 |
| jrosser | like overnight we find rabbitmq is totally hosed for SSL | 15:55 |
| jrosser | and a whole cycle to deal with fixing that | 15:55 |
| noonedeadpunk | yeah | 15:55 |
| noonedeadpunk | :( | 15:55 |
| noonedeadpunk | well, let's sey we spent it for super proper fix imo | 15:55 |
| evrardjp | yeah it's a different mindset | 15:56 |
| noonedeadpunk | as we could do smth nasty really fast to cover that | 15:56 |
| evrardjp | I don't think it's a failure in either case, it's just a different approach :) | 15:56 |
| evrardjp | I know when I said that last time, ppl went out with their pitchforks | 15:57 |
| evrardjp | I am not sure why I am saying this again, maybe I am expecting to be chopped in bits this time instead? | 15:58 |
| evrardjp | haha | 15:58 |
| noonedeadpunk | Maybe you just like pain?:) | 15:58 |
| evrardjp | How long have you been PTL again? ;) | 15:58 |
| noonedeadpunk | um, year I guess? | 15:58 |
| evrardjp | You see, you like it too! | 15:59 |
| evrardjp | j/k ofc | 15:59 |
| noonedeadpunk | ah, no, just beginning of the second cycle! | 15:59 |
| noonedeadpunk | So I'm jsut trying to get used to it haha | 15:59 |
| evrardjp | hehe :) | 15:59 |
| evrardjp | yeah that makes sense | 15:59 |
| evrardjp | why is jrosser hiding now? | 16:00 |
| noonedeadpunk | And I do really terrible job in terms of recruting cores | 16:00 |
| jrosser | hmm? | 16:00 |
| evrardjp | jrosser: It's just that I am happy to see you here :) | 16:00 |
| noonedeadpunk | but remote events doesn't help here tbh | 16:00 |
| jrosser | evrardjp: nice to see you back around too! | 16:01 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix https://review.opendev.org/c/openstack/openstack-ansible/+/789376 | 16:01 |
| evrardjp | I am not really back, I am just stuck between a CI and meetings ;p | 16:01 |
| evrardjp | and FIPS stuff! | 16:01 |
| noonedeadpunk | evrardjp: oh, well, Walmart also aims to make pretty big thing during this cycle | 16:01 |
| evrardjp | noonedeadpunk: oh really? | 16:02 |
| evrardjp | They are still active here? | 16:02 |
| jrosser | we've added some cool things in the last few years, OIDC is pretty sorted, LetsEncrypt for HA | 16:02 |
| evrardjp | oh nice! | 16:02 |
| evrardjp | OIDC, that's quite fancy | 16:02 |
| evrardjp | so what you are using keystone domains with OIDC? | 16:02 |
| noonedeadpunk | https://specs.openstack.org/openstack/openstack-ansible-specs/specs/xena/protecting-plaintext-configs.html | 16:02 |
| jrosser | my team have OSA -> keycloak -> corporate IdP | 16:03 |
| evrardjp | nice! | 16:03 |
| jrosser | and we also got CLI working with OIDC auth | 16:03 |
| evrardjp | oh wow | 16:03 |
| evrardjp | that's very interesting! | 16:03 |
| evrardjp | Did you make a presentation on how this was done? | 16:03 |
| jrosser | it needs to launch a browser to auth and do the 2FA bit | 16:03 |
| evrardjp | and what it looks like? | 16:03 |
| evrardjp | this is awesome! | 16:03 |
| jrosser | but it returns a token into your shell and off you go | 16:03 |
| noonedeadpunk | we're looking into keycloak as well, for different thing atm, but hopefully will integrate with keystone one day as well | 16:04 |
| evrardjp | Keycloak is super well known | 16:04 |
| evrardjp | RH took the market there :D | 16:04 |
| evrardjp | jrosser: your corporate IdP is based on SAML or is it with OIDC too? | 16:04 |
| jrosser | it's both, forgerock stuff i think | 16:05 |
| evrardjp | I am just wondering how it works in terms of protocols behind this, and how it works on the openstack bits | 16:05 |
| evrardjp | I have to go, sadly! | 16:05 |
| noonedeadpunk | yeah, they did along with ipa... | 16:05 |
| evrardjp | I am super eager to hear about this too | 16:06 |
| noonedeadpunk | hope seeing you around:) | 16:06 |
| evrardjp | the castellan bits are also interesting btw :) | 16:06 |
| evrardjp | It's great to see you folks, and to see OSA in good hands. :) | 16:06 |
| jrosser | i think this is a public repo https://github.com/bbc/keystoneauth-oidc | 16:07 |
| noonedeadpunk | it is | 16:07 |
| jrosser | thats our fork with added support for PKCE which makes the CLI user-experience not ridiculous | 16:07 |
| noonedeadpunk | oh, so you actually auth services through oidc as well? | 16:07 |
| jrosser | otherwise you need to share a OIDC client-secret with CLI users which is really not cool | 16:08 |
| jrosser | yes, nothing except the corporate auth provider from an end-user perspective | 16:08 |
| jrosser | it's really minimised the overhead of user management | 16:08 |
| jrosser | like its zero, basically | 16:09 |
| noonedeadpunk | and you skip `service_setup.yml` I guess? since you pre-provision service users? | 16:10 |
| jrosser | oh hold on no..... | 16:11 |
| evrardjp | that would make a _great_ summit video | 16:11 |
| jrosser | external auth for users is all OIDC | 16:11 |
| noonedeadpunk | and internal still password? | 16:11 |
| jrosser | internal stuff for serivces is just as usual | 16:11 |
| noonedeadpunk | ah, I see | 16:11 |
| evrardjp | you could do service users though, I guess? and application passwords? | 16:12 |
| jrosser | and we need keycloak there to enhance the forgerock backend stuff which we could have used directly, but doesnt do fancy things like PKCE | 16:12 |
| noonedeadpunk | I just really looking into some solution not to make password rotations when ppl leave insane | 16:12 |
| *** rpittau is now known as rpittau|afk | 16:12 | |
| jrosser | i would seriously look at keycloak + vault + signed keys | 16:13 |
| jrosser | and try to just get rid of passwords | 16:13 |
| noonedeadpunk | and actually keycloak looks like solution | 16:13 |
| jrosser | managing ssh key add/remove is totally PITA so making that go away as well is really excellent | 16:13 |
| noonedeadpunk | yeah, that was my thought as well.... not enough time and not super high prio I guess now though... | 16:14 |
| noonedeadpunk | but super interesting and smth I want to do for sure | 16:15 |
| noonedeadpunk | well, ssh keys rotation could be perfectly done also with freeipa, and hosts are anyway in ldap... But other option is to delegate that to zuul CD part | 16:16 |
| noonedeadpunk | Since all keys are in repo jsut when you merge change - zuul rollout them | 16:17 |
| jrosser | it's also helpful for re-PXE'ing things, it's a kind of one-line bash late command to insert the CA into openssh config | 16:18 |
| jrosser | then anyone can ssh straight in the moment the host comes up | 16:18 |
| noonedeadpunk | yeah, agree | 16:18 |
| * jrosser really looking forward to ripping out os_nova key distribution for this | 16:19 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Always use checkout for git_requirements library https://review.opendev.org/c/openstack/openstack-ansible/+/791011 | 16:20 |
| noonedeadpunk | btw, talking about ssh keys, I think we should add https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/tasks/openstack_authorized_keys.yml#L28 "from" option to allow access only from deply host IPs | 16:21 |
| noonedeadpunk | but it's not place where we place ssh keys.... | 16:27 |
| jrosser | i wonder if that breaks anything like lsync or if we have properly other users for that | 16:28 |
| jrosser | what even is ssh_key_url? | 16:29 |
| jrosser | looks like opportunity to just delete that https://codesearch.opendev.org/?q=ssh_key_url | 16:30 |
| jrosser | noonedeadpunk: i have to head out for a bit bug not really finding whats breaking tempest in this https://7b87cf7610fc809332ed-10283f798a6aff54957655db10501caf.ssl.cf2.rackcdn.com/788031/7/check/openstack-ansible-deploy-aio_metal-debian-buster/df9f979/ | 16:40 |
| jrosser | interested if you can spot something | 16:40 |
| *** dwilde has joined #openstack-ansible | 17:10 | |
| noonedeadpunk | I also looked at ssh_key_url and feeled like probably some leftover from jenkins maybe? | 17:30 |
| noonedeadpunk | well, it's breaking old good WRONG_VERSION_NUMBER in conductor.... | 17:32 |
| noonedeadpunk | but feels like we should have url logged now... | 17:32 |
| noonedeadpunk | but don't see a thing.... | 17:34 |
| jrosser | oh! | 17:36 |
| jrosser | good spot | 17:36 |
| jrosser | becasue Connecting to AMQP server on 172.29.236.100:5672 | 17:36 |
| jrosser | thats not right | 17:36 |
| noonedeadpunk | that's wrong port? | 17:37 |
| jrosser | yeah, thats the non ssl one | 17:37 |
| jrosser | should be 5671 | 17:37 |
| noonedeadpunk | but we don't have that in config.... | 17:37 |
| noonedeadpunk | I tried half of the day to spawn 2 aio, but found so many other things on that road.... | 17:38 |
| *** MrClayPole has quit IRC | 17:38 | |
| jrosser | oh what, err, earlier in the same log is Connected to AMQP server on 172.29.236.100:5671 | 17:38 |
| noonedeadpunk | I think really depends on what connects | 17:39 |
| noonedeadpunk | I have a feeling that it's a cell that tries to connect to 5672 | 17:39 |
| jrosser | i think this is why i didnt see it before | 17:41 |
| jrosser | saw 5671 earlier in the log file and figured it was ok | 17:41 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix https://review.opendev.org/c/openstack/openstack-ansible/+/789376 | 17:42 |
| noonedeadpunk | I really can't imagine how to figure out whats wrong without chiming in nova code... | 17:43 |
| jrosser | even now with all the debugging on it's not really clear what thing is making that MQ connection | 17:44 |
| noonedeadpunk | yeah :( | 17:44 |
| jrosser | i would assume that 5672 is the default in some places, and this maybe means that we don't properly set a var somewhere | 17:47 |
| jrosser | though somehow it has the IP correct | 17:47 |
| *** MrClayPole has joined #openstack-ansible | 17:49 | |
| *** jnamdar has quit IRC | 17:54 | |
| *** andrewbonney has quit IRC | 17:56 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: Prepare Train to EM https://review.opendev.org/c/openstack/openstack-ansible/+/790655 | 17:57 |
| jrosser | noonedeadpunk: do you think we are missing {port} here https://github.com/openstack/openstack-ansible-os_nova/commit/c6d4c6207fa904f30e471c598884b7bce66cbc8f#diff-b3bafa1afef99e726b111b31a7802a47258c01212f425b191cd9c1cc87267ce7R95 | 18:17 |
| noonedeadpunk | ouch | 18:23 |
| noonedeadpunk | jrosser: sorry | 18:24 |
| jrosser | no worries :) | 18:24 |
| * jrosser makes patch | 18:24 | |
| *** dwilde has quit IRC | 18:24 | |
| noonedeadpunk | I think we should also add port for mysql as well... | 18:27 |
| jrosser | hmm, well i was looking in nova.conf and we don't specify the port | 18:27 |
| noonedeadpunk | indeed. And I can recall copy-pasting transport-url :( | 18:27 |
| noonedeadpunk | so stupid :( | 18:28 |
| noonedeadpunk | *copying database_connection and pasting as transport-url | 18:28 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Add variables for rabbitmq ssl configuration https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/790037 | 18:28 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Add port to transport_url database template https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/791038 | 18:28 |
| jrosser | we can maybe look at adding port for db too | 18:28 |
| jrosser | perhaps we have to add that to the url in the config file at the same time | 18:28 |
| jrosser | but this all has slightly mysterious behaviour w.r.t default values | 18:29 |
| noonedeadpunk | yeah, it's pretty weird overall... | 18:29 |
| noonedeadpunk | like `create_cell` doesn't even check properly for already existing template and just creates as much cells as you run the command, which would be exactly the same | 18:30 |
| noonedeadpunk | as they compare template with interpreted string... | 18:31 |
| jrosser | i guess thats kind of an error that we have a var in os_nova for nova_galera_port but it doesnt quite do what you expect | 18:31 |
| jrosser | anyway, enough for today, i've rechecked some stuff and will look again tomorrow | 18:32 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: Prepare Train to EM https://review.opendev.org/c/openstack/openstack-ansible/+/790655 | 18:36 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix https://review.opendev.org/c/openstack/openstack-ansible/+/789376 | 18:43 |
| noonedeadpunk | I think we need to merge that pki stuff and it would be good point for RC | 18:52 |
| jrosser | we have to patch all the roles that use rabbitmq | 18:53 |
| noonedeadpunk | well, yes | 18:54 |
| jrosser | but I think hopefully now the first set of core roles will work | 18:54 |
| noonedeadpunk | doh... no release id yet for bullseye... http://paste.openstack.org/show/805309/ | 18:59 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove Debain python workaround https://review.opendev.org/c/openstack/openstack-ansible/+/791042 | 19:01 |
| noonedeadpunk | yes | 19:02 |
| jrosser | if we want to do an rc soon then we should have a big push on merging stuff | 19:02 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Add Debian Bullseye support https://review.opendev.org/c/openstack/openstack-ansible/+/783606 | 19:02 |
| *** cyberpear has joined #openstack-ansible | 19:04 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Add Debian Bullseye support https://review.opendev.org/c/openstack/openstack-ansible/+/783606 | 19:05 |
| noonedeadpunk | and fix facts athering.... | 19:05 |
| noonedeadpunk | ok, yeah, will focus on reparing roles | 19:06 |
| noonedeadpunk | along with pushing ssl change then probably? | 19:06 |
| noonedeadpunk | rabbitmq ssl one | 19:06 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Include galera_devel into main https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/791045 | 19:14 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant master: Install mysql client libraries https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/777607 | 19:15 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant master: Install mysql client libraries https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/777607 | 19:15 |
| *** macz_ has quit IRC | 19:16 | |
| *** macz_ has joined #openstack-ansible | 19:32 | |
| *** snapdeal has joined #openstack-ansible | 19:41 | |
| *** Jeffrey4l has quit IRC | 19:49 | |
| *** Jeffrey4l has joined #openstack-ansible | 20:00 | |
| *** spatel has quit IRC | 20:01 | |
| *** macz_ has quit IRC | 20:08 | |
| *** macz_ has joined #openstack-ansible | 20:08 | |
| *** mcarden has joined #openstack-ansible | 20:37 | |
| *** openstackgerrit has quit IRC | 21:47 | |
| *** snapdeal has quit IRC | 22:15 | |
| *** tosky has quit IRC | 23:01 | |
| *** macz_ has quit IRC | 23:08 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!