| *** tosky has quit IRC | 00:22 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Update pip/setuptools/wheel to latest version https://review.opendev.org/c/openstack/openstack-ansible/+/770284 | 00:23 |
|---|---|---|
| openstackgerrit | Merged openstack/openstack-ansible stable/victoria: Disable magnum zero nodes test. https://review.opendev.org/c/openstack/openstack-ansible/+/780086 | 00:23 |
| *** tbarron has joined #openstack-ansible | 00:38 | |
| *** jamesdenton has quit IRC | 01:49 | |
| *** jamesden_ has joined #openstack-ansible | 01:51 | |
| *** spatel_ has joined #openstack-ansible | 01:51 | |
| *** spatel_ is now known as spatel | 01:51 | |
| *** spatel has quit IRC | 01:56 | |
| *** spatel_ has joined #openstack-ansible | 01:57 | |
| *** spatel_ is now known as spatel | 01:57 | |
| *** evrardjp has quit IRC | 03:33 | |
| *** evrardjp has joined #openstack-ansible | 03:33 | |
| *** chandankumar is now known as chkumar|ruck | 04:57 | |
| *** gyee has quit IRC | 05:10 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/victoria: Add Neutron Role Testing Overrides https://review.opendev.org/c/openstack/openstack-ansible/+/780458 | 05:28 |
| *** yasemind has joined #openstack-ansible | 05:35 | |
| *** miloa has joined #openstack-ansible | 06:47 | |
| *** miloa has quit IRC | 06:57 | |
| *** yasemind has quit IRC | 07:15 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/queens: Update get-pip url https://review.opendev.org/c/openstack/openstack-ansible-tests/+/781197 | 07:27 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests stable/queens: Update get-pip url https://review.opendev.org/c/openstack/openstack-ansible-tests/+/781197 | 07:27 |
| *** rpittau|afk is now known as rpittau | 07:51 | |
| *** jamesden_ has quit IRC | 08:15 | |
| *** jamesdenton has joined #openstack-ansible | 08:17 | |
| *** jawad_axd has joined #openstack-ansible | 08:29 | |
| *** tosky has joined #openstack-ansible | 08:37 | |
| *** mensis has joined #openstack-ansible | 08:57 | |
| *** andrewbonney has joined #openstack-ansible | 09:08 | |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server stable/train: Return CentOS 7 jobs to voting https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/780720 | 09:09 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron stable/victoria: Add integrated tests various Neutron deployment scenarios https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/780457 | 09:13 |
| noonedeadpunk | mornings | 09:16 |
| noonedeadpunk | jrosser: I can recall we discussed that we're missing hosts file management on the deploy host. Were any patches pushed by you regarding this? | 09:16 |
| jrosser | i've not done anything about that | 09:17 |
| noonedeadpunk | Just trying to recall where we ended up on this | 09:17 |
| noonedeadpunk | gotcha | 09:17 |
| jrosser | we've got them in DNS so it's not something affecting us directly | 09:17 |
| andrewbonney | I had one WIP but hadn't taken it further | 09:17 |
| jrosser | however it's still an issue | 09:17 |
| andrewbonney | https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/773868 | 09:17 |
| jrosser | excellent, andrewbonney++ | 09:18 |
| noonedeadpunk | yep. I just can recall seeing some patch, but thought maybe I dreamed :) | 09:18 |
| jrosser | uh urgh yes and ansible bug for extra excitement there | 09:19 |
| *** mensis has quit IRC | 09:20 | |
| andrewbonney | noonedeadpunk: I'll add something in to differentiate the blocks, but there's still a risk as a result that some delegation could hit the wrong deployment due to matching names | 09:23 |
| noonedeadpunk | Well, I look at my current deploy hosts, that were created during R, and all of them have /etc/hosts file provisioned... | 09:24 |
| noonedeadpunk | wait, not all... | 09:25 |
| ebbex | Is there a variable i can put at `neutron_git_install_branch: stable/{{ insert_variable_here }}` in my user_variables.yml so it applies for any branch I'm on? | 09:26 |
| noonedeadpunk | none I can recall | 09:27 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron stable/victoria: Fix conditional for ovs https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/779928 | 09:28 |
| noonedeadpunk | well... actually... you can set `neutron_git_install_branch: "{{ neutron_git_track_branch }}" | 09:29 |
| jrosser | relatedly, https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L22 | 09:30 |
| ebbex | noonedeadpunk: indeed, thanks :) | 09:30 |
| jrosser | thats correct on the stable branches | 09:31 |
| noonedeadpunk | I repeteadly forgot to update it... | 09:32 |
| jrosser | pip resolver stuff merged btw | 09:32 |
| noonedeadpunk | I mean `openstack_distrib_code_name` | 09:32 |
| jrosser | murano still has not merged the constraints changes which will make it more difficult to fix | 09:34 |
| noonedeadpunk | andrewbonney: maybe let's add a bool variable to manage deployhost hosts file and reno as known issue? | 09:34 |
| openstackgerrit | Merged openstack/openstack-ansible-haproxy_server master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/772011 | 09:34 |
| andrewbonney | Ok, will do | 09:35 |
| noonedeadpunk | yeah, murano is... in bad shape... | 09:39 |
| noonedeadpunk | wondering if we should drop it | 09:39 |
| noonedeadpunk | as I'm eventually not sure how fuctional murano is on it's own | 09:40 |
| noonedeadpunk | project if pretty deserted for years now | 09:40 |
| noonedeadpunk | but it's passing it's CI... | 09:40 |
| noonedeadpunk | I feel we just need steal devstack conf | 09:43 |
| jawad_axd | Hi all, how to make 'encrypted-volume(with barbican) creation from image' process fast?It is very very slow, since it copies and convert. Backend is 'ceph', although different pool for volumes. I have tried image-volume cache (https://docs.openstack.org/cinder/stein/admin/blockstorage-image-volume-cache.html) but didt notice any difference. Any comments on this? Thanks | 09:48 |
| jrosser | jawad_axd: is that possible to ever make fast? boot from volume on ceph is fast in the first place because it takes a snapshot (hopefully) of the original image, so all the blocks are pointing to the original data | 09:54 |
| jrosser | if you want to encrypt the root disk then it intuitively feels like you have to make an entire copy as it makes no sense to use a snapshot | 09:55 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-openstack_hosts master: Add hostname resolution to deploy host https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/773868 | 09:57 |
| noonedeadpunk | looking at devstack, I'm wondering what it would take to make an option to have public endpoints all on default ports distinguished by location. I guess it can be pretty easily done with ACLs? | 10:00 |
| jawad_axd | @jrosser By making process fast, I meant somehow using ceph snapshot/clone instead of conversion ;) . Boot from volume is fast..that is Ok. In my use case, I need to create encrypted root disk every time from image and boot from it, so I suppose, as you said, it will do entire copy for that. | 10:02 |
| jawad_axd | But its very slow..so need to stick with nova boot I guess, and forget about encrypted root disk. ;) | 10:03 |
| noonedeadpunk | youah, but the whole idea of encrypted drives is to use different encryption key for each of it. And that will for sure require full copy, since you can't jsut clone | 10:03 |
| jrosser | jawad_axd: you can always try to address the performance issue, if its CPU or network bound where the image conversion is done | 10:03 |
| openstackgerrit | Merged openstack/openstack-ansible stable/ussuri: Use nodepool epel mirror in CI for systemd-networkd package https://review.opendev.org/c/openstack/openstack-ansible/+/775636 | 10:03 |
| jrosser | or local disk speed to do the conversion, which is probably on the controller? | 10:04 |
| jrosser | it certainly wont scale very well though i'd have thought | 10:04 |
| jawad_axd | @noonedeadpunk thats right. There should be new encryption key for each instance, so it makes total sense to use entire copy. | 10:05 |
| jrosser | it's probably much much quicker to attach encrypted volumes to a VM though? | 10:06 |
| jawad_axd | @jrosser yes conversion is on controller.. and yes its very fast to attach encrypted vol. | 10:07 |
| jawad_axd | conversion rate is around 95MB/s ..and it takes around 10 minutes to create enc vol from image. | 10:10 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Fix murano role https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/781239 | 10:11 |
| *** gshippey has joined #openstack-ansible | 10:24 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/772014 | 10:39 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-galera_server master: Update galera_wsrep_sst_auth_user to use galera_root_user https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/781305 | 10:44 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-galera_server master: Update galera_wsrep_sst_auth_user to use galera_root_user https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/781305 | 10:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/772025 | 10:46 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/772010 | 10:47 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-galera_server master: Bump MariaDB version to 10.5.9 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/777088 | 10:50 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_server/+/772058 | 10:52 |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/772004 | 10:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/772033 | 10:59 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/780844 | 11:02 |
| noonedeadpunk | oh, nice, ironic is fixed | 11:02 |
| ebbex | jrosser: caught something after i approved; https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/781039 | 11:25 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/780859 | 11:26 |
| *** fanfi has quit IRC | 11:38 | |
| jrosser | ebbex: ahha - there is no btrfs in centos8, i think thats why i removed it completely | 11:40 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Remove references to unsupported operating systems https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/779738 | 11:47 |
| *** jawad_axd has quit IRC | 12:02 | |
| *** emmadavis has quit IRC | 12:04 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/780859 | 12:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/772051 | 12:18 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/780844 | 12:18 |
| noonedeadpunk | ok, so murano fails with `ERROR: Could not find a version that satisfies the requirement setuptools (from jsonschema)\nERROR: No matching distribution found for setuptools` o_O | 12:26 |
| noonedeadpunk | ok, but what's an issue here https://github.com/Julian/jsonschema/blob/v3.2.0/setup.cfg#L33 | 12:27 |
| noonedeadpunk | is it new resolver thing, that package is both in uc and requirements? | 12:27 |
| noonedeadpunk | it's not fixed in requirements though | 12:28 |
| noonedeadpunk | uh https://github.com/Julian/jsonschema/commit/d647d72f48a0fb7034de7a91bfb94953d80d49a8 | 12:28 |
| mgariepy | good morning everyone | 12:29 |
| noonedeadpunk | o/ | 12:32 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/victoria: Prevent neutron-l3-agent restart killing metadata service https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/781109 | 12:42 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/780844 | 12:49 |
| *** yolanda has quit IRC | 12:52 | |
| jamesdenton | good morning | 12:56 |
| noonedeadpunk | oh, jamesdenton, I was just looking at patch you've -W https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729 | 12:56 |
| jamesdenton | :o | 12:57 |
| noonedeadpunk | eventually it's not valid once we've dropped centos 7... | 12:57 |
| jamesdenton | ahh yes, in ubuntu there is a dpdk service that does some setup and it doesn't exist in centos | 12:57 |
| jamesdenton | so making openvswitch the dpdk service name is no good | 12:58 |
| *** pcaruana has quit IRC | 12:58 | |
| noonedeadpunk | but it's only for centos? | 12:59 |
| noonedeadpunk | or? | 12:59 |
| jamesdenton | well, the dpdk service itself doesn't apply to centos, but there may be some other thing needed. or not. i hadn't tested centos at the time | 13:00 |
| jamesdenton | but i can go through and verify things on focal and centos 8 | 13:00 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/vars/debian.yml#L32 we haver that set for debain/ubuntu | 13:00 |
| jamesdenton | right. no such service exists on centos | 13:00 |
| jamesdenton | and IIRC that service just does some dpdk driver shenanigans to help the user | 13:01 |
| noonedeadpunk | I just don't like https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729/5/tasks/providers/setup_ovs_dpdk.yml there | 13:01 |
| *** spatel_ has joined #openstack-ansible | 13:01 | |
| *** spatel_ is now known as spatel | 13:01 | |
| jamesdenton | doesn't really apply anymore, i guess | 13:03 |
| jamesdenton | now that centos 7 is done | 13:03 |
| noonedeadpunk | I'd say it supposed do be difference between ubuntu and centos | 13:04 |
| jamesdenton | prob right | 13:04 |
| jamesdenton | :D | 13:04 |
| LowKey | Hi, this might noob questions from me, how to remove manila from inventory ? before this, i'm install manila service, but now i want to remove it, i did remove the containers, and the clear the inventory using inventory-manage.py script . When I did run the playbooks, setup-openstack.yml, seem like it's still call task os-manila . http://paste.openstack.org/show/803691/ | 13:05 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_openrc master: Remove references to unsupported operating systems https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/781055 | 13:05 |
| jamesdenton | noonedeadpunk we can set a var in the respective distro vars file | 13:05 |
| noonedeadpunk | yeah | 13:06 |
| jrosser | LowKey: you need to remove the hosts from openstack_user_config as well i think, otherwise the manila services will get put back into the inventory | 13:07 |
| jamesdenton | which version of centos 8 is good to test with? is centos stream supported yet? | 13:07 |
| jrosser | LowKey: or remove the conf.d/ fragment if you used that method | 13:07 |
| LowKey | jrosser: yes, i remove manila on openstack_user_config and not use conf.d/ | 13:08 |
| LowKey | but it's seem still looking for it :( | 13:08 |
| jrosser | you need to get the ordering right | 13:09 |
| jrosser | remove from openstack_user_config | 13:09 |
| jrosser | then use inventory_manage to remove the inventory hosts | 13:09 |
| jrosser | unless there is a bug here as well | 13:10 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Convert remaining ansible_ variables to ansible_facts[] https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/780739 | 13:10 |
| mgariepy | https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L117-L118 < would it make sense to user "self" on each haproxy node if there is a galera service running on the haproxy node ? | 13:10 |
| mgariepy | anyone have done that? like to have the ctrl01 pointing to ctrl01_galera, and ctrl02 > ctrl02_galera etc.. ? | 13:11 |
| jrosser | it's not active/active | 13:11 |
| mgariepy | i know we are not using the galera cluster as active/active | 13:11 |
| mgariepy | but from the galera side they are all masters. | 13:12 |
| jrosser | don't we still need to cope with any one of the galera servers failing / maintainance | 13:13 |
| mgariepy | for normal ops. it works ok. | 13:14 |
| jrosser | means the active haproxy needs to be able to use any of the galeras | 13:14 |
| mgariepy | yes | 13:14 |
| mgariepy | i want them to use all the galera server. | 13:14 |
| mgariepy | i just think it would be better to prefer the one on the same host. | 13:15 |
| LowKey | jrosser: yes, wondering there is a bugs on ./inventory-manage.py , i still can see manila_data / manila_share on ./inventory-manage.py -G (when i did Group list) | 13:15 |
| jrosser | LowKey: have you deleted the groups? | 13:15 |
| mgariepy | currently all the haproxy have the 3 galera, like: ctrl01 then ctrl02 and ctrl03 as backup | 13:15 |
| mgariepy | if you want to reboot ctrl01, which have the keepalived ip | 13:15 |
| mgariepy | you need to maint all the ctrl01 service on the ctrl02 and 03, then move the ip from ctrl01 to the next one. | 13:16 |
| mgariepy | otherwise you end up with galera connection reset on the keepalived IP move , and then on the maint set for the ctrl01 containers. | 13:18 |
| LowKey | jrosser: how to remove the group? is it same with ./inventory-manage.py -r command ? | 13:18 |
| mgariepy | does it make sense? or i'm overthinking it ? | 13:21 |
| jrosser | you're trying to minimise the number of connection drops i think? | 13:27 |
| openstackgerrit | Merged openstack/openstack-ansible stable/train: Bump SHAs for stable/train https://review.opendev.org/c/openstack/openstack-ansible/+/780437 | 13:28 |
| mgariepy | yep. | 13:31 |
| mgariepy | i would like to make it easier to reboot a controller without any user noticing it. | 13:31 |
| mgariepy | when the controller is not the primary it's a lot less work to get it done. | 13:34 |
| noonedeadpunk | jamesdenton: sorry for dealy. no, it's not yet | 13:37 |
| jamesdenton | no problem. so centos 8 (non stream) is the way to go then | 13:38 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server master: Remove references to unsupported operating systems https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/781051 | 13:38 |
| jrosser | mgariepy: do we need some helpers to switch the state nicely through drain->maint? | 13:40 |
| mgariepy | the main issue is the DB. i think even if we do have a helper for that, when we change the FIP fron server 1 to server 2 all that soft transition will be lost. | 13:41 |
| jamesdenton | you need some connection mirroring | 13:41 |
| mgariepy | well that would be awesome but i don't really meant 1 hicup i just would prefer to only have one if possible. | 13:42 |
| mgariepy | i also like the simplicity of the current solution haha | 13:43 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add centos-8 support for ovs-dpdk https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729 | 13:51 |
| noonedeadpunk | jamesdenton: I've updated patch to match current needs ^ | 13:55 |
| noonedeadpunk | damn it, really can't still decide on the best approach with policies... | 13:55 |
| jamesdenton | just FYI, these tasks apply to that dpdk service that doesn't exist in centos: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_dpdk.yml#L25-L52 | 13:56 |
| jamesdenton | so there may need to be a different approach | 13:56 |
| jamesdenton | that is cross-platform, so to speak | 13:56 |
| noonedeadpunk | I'm so far from understanding dpdk architecture to be completely honest :( | 13:58 |
| noonedeadpunk | So I have no idea if some of these files might be used by thing that replaces dpdk service in centos | 13:58 |
| jamesdenton | for ubuntu that dpdk service is just a helper app of sorts | 13:59 |
| *** pcaruana has joined #openstack-ansible | 13:59 | |
| noonedeadpunk | considering patch suggested `hugetlbfs` group, I guess that some things might be shipped there differently | 13:59 |
| noonedeadpunk | if these tasks are really not needed - we can easily just move them to block with os_family condition | 14:00 |
| jamesdenton | at a minimum, yes. that said, the dpdk deployment as it is today does not completely set up the environment for dpdk due to lack of equivilant process | 14:01 |
| jamesdenton | not sure if spatel got it going in his environment or not. otherwise, i just need some time to deploy | 14:01 |
| noonedeadpunk | iirc he said it worked for him back then... but worth asking again:) | 14:02 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/780896 | 14:06 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use admin user instead of root for galera https://review.opendev.org/c/openstack/openstack-ansible/+/775684 | 14:09 |
| *** spatel has quit IRC | 14:16 | |
| ebbex | neutron_server_container-ea264010 -> 172.29.236.128]: FAILED! => {"changed": false, "msg": "file not found: /var/www/repo/os-releases/22.1.0/neutron-22.1.0-constraints.txt what's the obvious course of action I'm missing here? | 14:17 |
| noonedeadpunk | I think that maybe a result of messed up venv_rebuild | 14:19 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/780844 | 14:19 |
| noonedeadpunk | but this should have been fixed in 22.1.0 | 14:19 |
| noonedeadpunk | or smth went wrong with repo container or lsyncd there | 14:20 |
| *** spatel_ has joined #openstack-ansible | 14:21 | |
| *** spatel_ is now known as spatel | 14:21 | |
| spatel | jamesdenton i believe that is just for CentOS not for ubuntu (hugetlbfs) | 14:21 |
| jrosser | ebbex: a way to tell if it's lsync, is to wget the failing URL a few times, if you have 3 controller and it fails 2 out of 3 times it's on the build host but not lsync'd | 14:22 |
| spatel | but that is true we need to change group ownership to hugetlbfs otherwise it won't able to attach to dpdk socket | 14:22 |
| spatel | noonedeadpunk didn't we stop supporting centos-8 ? | 14:23 |
| spatel | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729 | 14:23 |
| noonedeadpunk | did we? | 14:24 |
| noonedeadpunk | I think we aim to switch to Stream | 14:24 |
| noonedeadpunk | But there are some issues with performance we saw for regular centos8 as well | 14:25 |
| spatel | do we have stream CI job or they are still using centos-8? | 14:25 |
| noonedeadpunk | still using centos8 | 14:25 |
| spatel | ok | 14:25 |
| noonedeadpunk | jrosser did huge job to get jobs for stream as well | 14:25 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/776226 | 14:26 |
| spatel | I have destroyed all my centos-8 lab so not sure how to validate that patch but if we really need then i can spin up for centos-8 to test dpdk patch | 14:26 |
| jamesdenton | spatel you switching to ubuntu? | 14:26 |
| spatel | Yes | 14:27 |
| spatel | now i hate centos :) | 14:27 |
| jamesdenton | hah | 14:27 |
| noonedeadpunk | lol | 14:27 |
| spatel | but if you want me to test out patches i can setup centos-8 i have lots of hardware in datacenter | 14:28 |
| noonedeadpunk | my exact feeling when I switched first couple of servers from centos to debian | 14:28 |
| spatel | I am not ready for debian :) | 14:28 |
| spatel | ubuntu is so easy to work.. no extra stupid repo required.. everything is included, just love it | 14:29 |
| LowKey | i'm using ubuntu :) | 14:30 |
| spatel | then you don't know the pain :) | 14:31 |
| *** d34dh0r53 has joined #openstack-ansible | 14:31 | |
| LowKey | spatel: i'm already in pain... | 14:32 |
| spatel | noonedeadpunk where are we stuck here, this is holding my other patch - https://review.opendev.org/c/openstack/openstack-ansible/+/777384 | 14:32 |
| spatel | LowKey love to know that | 14:32 |
| spatel | seem that patch stuck at centos-8 issue, why don't we set non-voting and move on if there is no easy fix. | 14:36 |
| jrosser | spatel: thats failed on focal, and at 28mins in it's bound to be a galera DB failure | 14:38 |
| jrosser | should recheck | 14:38 |
| spatel | i can see that | 14:38 |
| spatel | I want to setup come kind of CI-CD infra in my datacenter, what do you recommend ? jankins or zuul ? | 14:40 |
| spatel | jenkins* | 14:40 |
| spatel | to support my older cloud running on queens | 14:41 |
| noonedeadpunk | spatel: it feel like the DB issue we're seing randomly raising | 14:42 |
| noonedeadpunk | I think worth recheck | 14:43 |
| ebbex | I have {global,source}-constraints.txt and requirements.txt in all repo_containers, but no plain constraints.txt for neutron, and it's only missing for this project. the others (keystone,glance,cinder,etc) have the constraints.txt | 14:43 |
| spatel | hmm let me do recheck | 14:43 |
| ebbex | can i run something to force it into existance again, incase some task is skipped? | 14:44 |
| ebbex | -e 'venv_rebuild=true' ? | 14:47 |
| jrosser | the .txt files are generated on the build host as part of python_venv_build | 14:48 |
| jrosser | so delete/move them aside on the build host should be sufficient | 14:48 |
| * jrosser hesitates to advise venv_rebuild=true | 14:49 | |
| jamesdenton | YOLO | 14:50 |
| spatel | jrosser did we fixed that venv_rebuild issue right, which i had earlier between nova and neutron rebuild=true? | 14:50 |
| noonedeadpunk | well, I think we have fixed behaviour `venv_rebuild=true`? | 14:51 |
| ebbex | YOLO is right! :) Did the venv_rebuild, all requirements files deleted and generated over. | 14:51 |
| spatel | i think we did - https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773972 | 14:51 |
| jrosser | yes its fixed for anyone who's upgraded to those point releases | 14:53 |
| noonedeadpunk | it was 22.1.0 in log so... | 14:53 |
| noonedeadpunk | I hope it will end up well for ebbex | 14:53 |
| jrosser | well i thought "i'll just make openstack server rescue ....." work | 14:54 |
| jrosser | thats been an adventure taking 10x longer than i expected | 14:54 |
| noonedeadpunk | lol | 14:54 |
| noonedeadpunk | eventually, it works for ephemeral based | 14:54 |
| noonedeadpunk | (at least worked for me) | 14:55 |
| jrosser | i have ot working now for boot-from-volume too | 14:55 |
| jrosser | *it | 14:55 |
| noonedeadpunk | oh! | 14:55 |
| *** crazzy has joined #openstack-ansible | 14:55 | |
| noonedeadpunk | I hope you will push it for nova?:) | 14:55 |
| jrosser | openstack --os-compute-api-version 2.87 server rescue | 14:55 |
| jrosser | ^ secret stuff :) | 14:55 |
| noonedeadpunk | rly? that's super awesome! | 14:55 |
| noonedeadpunk | and why it took 10x longer?:) | 14:56 |
| noonedeadpunk | because of that or side effects?:) | 14:56 |
| jrosser | and i had to put hw_rescue_device: 'disk' and hw_rescue_bus: 'usb' on all my images | 14:56 |
| jrosser | oh theres a check in nova code, and the client libs pass a "2.0" api version unless you specify it to be at least the required microversion | 14:57 |
| noonedeadpunk | yeah, this thing I'm aware of | 14:57 |
| noonedeadpunk | it's so stupid but yeah | 14:57 |
| jrosser | also the default is to rescue using the same image the VM was booted from unless you pass --image | 14:58 |
| jrosser | which all sounds excellent, except the FSID ends up the same and you actually boot the original image with the rescue one attached | 14:58 |
| jrosser | rather than the other way round | 14:58 |
| jrosser | just took 10x longer as i had to unravel all of those traps for the unwary along the way | 14:59 |
| noonedeadpunk | doh | 14:59 |
| jrosser | docs suggest you can rescue using the same image but reality is different | 14:59 |
| jrosser | and it's *highly* confusing as it looks really correct | 15:00 |
| noonedeadpunk | so it really _has_ to be different image at the end of the day... | 15:02 |
| noonedeadpunk | yeah, can feel your pain | 15:03 |
| jrosser | ah /o\, cirros doesnt have usb storage module | 15:04 |
| jrosser | i'm looking for a self-serve setup so users can fix their own stuff in the case of losing ssh keys | 15:05 |
| LowKey | okey, no error anymore, after direct edit openstack_inventory.json | 15:07 |
| noonedeadpunk | jrosser: yeah, I get why you're doing this, as super struggling with the same thing | 15:07 |
| noonedeadpunk | as eventually other ways is to drop instance, mount volume to some rescue one, and create new one | 15:08 |
| noonedeadpunk | it's super not handy and time consuming | 15:08 |
| jrosser | i had some trouble with hw_rescue_bus=virtio not working properly | 15:08 |
| jrosser | hence using usb | 15:08 |
| noonedeadpunk | but eventuyally I didn't know that it's possible | 15:08 |
| jrosser | or scsi, whichever one makes it be a hard drive | 15:09 |
| noonedeadpunk | ah, well, 2.87 is U one... And last time I was looking at it was in T | 15:10 |
| jrosser | yeah, it's new | 15:11 |
| noonedeadpunk | super thanks for sharing this, TIL | 15:12 |
| jrosser | and the ssh keys lost problem is the trickiest, becasue the rescue image will get the old/useless public key from the metadata service | 15:13 |
| jrosser | cirros is ideal for this | 15:13 |
| jrosser | hmmm | 15:15 |
| noonedeadpunk | well, this can be solved by systemrescuecd image as well | 15:15 |
| noonedeadpunk | it's about procedure for users what should they do | 15:16 |
| jrosser | yes it can, i'm just going to check if uploading a new key in horizon results in the new key being in the rescue VM | 15:16 |
| jrosser | or if it's always the original one | 15:16 |
| noonedeadpunk | btw for ISO that might be hw_rescue_bus-cdrom or smth like that? | 15:16 |
| jrosser | probably booting from an iso on usb is good too? | 15:17 |
| jrosser | like dd'ing to a real usb stick | 15:17 |
| noonedeadpunk | maybe... | 15:17 |
| mnaser | o/ osa has some non voting jobs in gate, so would be grateful if we could clean those up =) | 15:21 |
| noonedeadpunk | I guess it might be harder then souds, because usually we have nv in gates when they're provided by the template and fails for specific repo | 15:25 |
| noonedeadpunk | but yeah, will look through what we can do | 15:25 |
| * jrosser mutters about dropping distro installs | 15:26 | |
| jrosser | i wonder if we need a new template without the distro jobs | 15:27 |
| jrosser | then this would be much easier in the job config | 15:27 |
| noonedeadpunk | I think distro jobs are in independant one? | 15:27 |
| noonedeadpunk | hm. or they are not | 15:28 |
| jrosser | this may be messy | 15:28 |
| noonedeadpunk | so, we have `openstack-ansible-deploy-aio_distro_metal-jobs` | 15:28 |
| jrosser | but then we are in wierd situations in places where like centos just is broken forever | 15:28 |
| noonedeadpunk | and `openstack-ansible-deploy-aio_metal-jobs` which do not involve distro | 15:29 |
| noonedeadpunk | centos in distro isn't broken on it's own - it's about rabbitmq and ssl? | 15:29 |
| noonedeadpunk | have we merged rabbitmq ssl thing? | 15:30 |
| jrosser | https://zuul.opendev.org/t/openstack/builds?pipeline=gate&result=FAILURE | 15:30 |
| jrosser | we have waaaay too much nv stuff there | 15:31 |
| noonedeadpunk | ok, so https://review.opendev.org/c/openstack/openstack-ansible/+/773377/1 merged | 15:32 |
| noonedeadpunk | why centos is failing then.... | 15:32 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/f6f8365662e0438fa0fefe3c08b567c2/log/job-output.txt#13096 | 15:34 |
| noonedeadpunk | `"/bin/bash: line 2: /usr/libexec/openstack: No such file or directory"` which is kind of fair | 15:35 |
| noonedeadpunk | we really need to write up collection for that.... | 15:35 |
| noonedeadpunk | *module | 15:35 |
| jrosser | do you know why it's failing | 15:35 |
| jrosser | i saw that and was not sure | 15:36 |
| noonedeadpunk | hm, not sure now... why `"{{ ansible_python['executable'] }}"` is /usr/libexec/python | 15:40 |
| noonedeadpunk | I'd say we broke it with https://opendev.org/openstack/openstack-ansible-os_cinder/commit/e89b8640657dfd14641990de8b7f9e1dd842fc27 | 15:58 |
| noonedeadpunk | but unlikely | 15:58 |
| *** d34dh0r53 has quit IRC | 16:12 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible version to 2.10.7 https://review.opendev.org/c/openstack/openstack-ansible/+/781505 | 16:14 |
| *** d34dh0r53 has joined #openstack-ansible | 16:16 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Install wget binary https://review.opendev.org/c/openstack/openstack-ansible/+/781506 | 16:16 |
| noonedeadpunk | btw talking about flags to the images, I tried out setting `hw_vif_multiqueue_enabled: true` to the images and this gave me ability to fill in 10gbps between isntances on different nodes. Eventually it gives to fill in all capacity that's available | 16:21 |
| *** gyee has joined #openstack-ansible | 16:21 | |
| * jrosser finds cirros does not understand xfs | 16:22 | |
| noonedeadpunk | doh, it's not indeed | 16:23 |
| jrosser | now i think i really use something else :) | 16:23 |
| jrosser | i changed the users key in horizon, but that doesnt affect old instanes for rescue | 16:23 |
| noonedeadpunk | I think key is stored in nova db in instance metadata | 16:25 |
| noonedeadpunk | and it will try to use it for the rescue as well | 16:25 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/780749 | 16:30 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Fix gnocchi_storage_driver condition https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/781513 | 16:50 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/780844 | 16:50 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/780869 | 16:53 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/780874 | 16:57 |
| *** d34dh0r53 has quit IRC | 16:58 | |
| *** d34dh0r53 has joined #openstack-ansible | 17:01 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Remove references to unsupported operating systems https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/781041 | 17:02 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/781516 | 17:04 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/781519 | 17:10 |
| *** rpittau is now known as rpittau|afk | 17:11 | |
| noonedeadpunk | jrosser: on centos distro ansible_python['executable'] is /usr/libexec/platform-python :( | 17:17 |
| jrosser | 8-O | 17:17 |
| noonedeadpunk | "openstack_service_setup_host_python_interpreter": "/usr/libexec/platform-python" | 17:17 |
| jrosser | and let me guess, theres nothing there at that path? | 17:17 |
| noonedeadpunk | no idea wtf tbh | 17:17 |
| noonedeadpunk | well, there is | 17:17 |
| jrosser | oh right | 17:17 |
| jrosser | just !='python' | 17:18 |
| jrosser | how does that even happen | 17:19 |
| noonedeadpunk | we jsut assume that `openstack_service_setup_host_python_interpreter | dirname` has openstack binary | 17:19 |
| noonedeadpunk | I guess new ansible version we've bumped? | 17:19 |
| *** d34dh0r53 has quit IRC | 17:19 | |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/tasks/cinder_backends.yml#L43 <- and things go wrong here | 17:20 |
| *** d34dh0r53 has joined #openstack-ansible | 17:20 | |
| *** jamesdenton has quit IRC | 17:23 | |
| *** jamesdenton has joined #openstack-ansible | 17:23 | |
| *** macz_ has joined #openstack-ansible | 17:23 | |
| noonedeadpunk | I have no ideas honestly except set it to /usr/bin/python3.... | 17:24 |
| noonedeadpunk | or symlink installed openstack binaries for utility there as well | 17:25 |
| noonedeadpunk | http://paste.openstack.org/show/803693/ | 17:25 |
| noonedeadpunk | platform-python is default system python for centos 8 | 17:44 |
| noonedeadpunk | /usr/libexec/ is not even in $PATH.... | 17:47 |
| *** jamesdenton has quit IRC | 17:55 | |
| *** jamesdenton has joined #openstack-ansible | 17:56 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon stable/train: Fix race condition in compression of static files https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/779470 | 18:04 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Hardcode service setup python interpreter https://review.opendev.org/c/openstack/openstack-ansible/+/781534 | 18:04 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: [DNM] https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/781535 | 18:10 |
| *** jawad_axd has joined #openstack-ansible | 18:14 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/781516 | 18:15 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/780869 | 18:19 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/781537 | 18:23 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Add hostname resolution to deploy host https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/773868 | 18:27 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/781538 | 18:31 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Switch default virtualenv to python3 https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/779678 | 18:35 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/781519 | 18:39 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/781538 | 18:40 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/781541 | 18:44 |
| *** jawad_axd has quit IRC | 18:47 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/781542 | 18:51 |
| *** trident has quit IRC | 19:01 | |
| *** andrewbonney has quit IRC | 19:03 | |
| *** priteau has quit IRC | 19:08 | |
| *** trident has joined #openstack-ansible | 19:19 | |
| *** spatel has quit IRC | 19:20 | |
| *** spatel_ has joined #openstack-ansible | 19:20 | |
| *** spatel_ is now known as spatel | 19:20 | |
| *** openstackgerrit has quit IRC | 19:34 | |
| *** zul has quit IRC | 19:43 | |
| *** sshnaidm is now known as sshnaidmoff | 21:21 | |
| *** sshnaidmoff is now known as sshnaidm|off | 21:21 | |
| *** openstackstatus has quit IRC | 21:26 | |
| *** openstackstatus has joined #openstack-ansible | 21:32 | |
| *** ChanServ sets mode: +v openstackstatus | 21:32 | |
| *** openstack has joined #openstack-ansible | 21:38 | |
| *** ChanServ sets mode: +o openstack | 21:38 | |
| *** openstack has joined #openstack-ansible | 22:03 | |
| *** ChanServ sets mode: +o openstack | 22:03 | |
| *** poopcat has quit IRC | 23:13 | |
| *** poopcat has joined #openstack-ansible | 23:15 | |
| *** poopcat has quit IRC | 23:21 | |
| *** macz_ has quit IRC | 23:28 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!