| *** gyee has quit IRC | 00:38 | |
| *** cp- has quit IRC | 01:36 | |
| *** cp- has joined #openstack-ansible | 01:36 | |
| *** cp- has quit IRC | 01:45 | |
| *** cp- has joined #openstack-ansible | 01:47 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #openstack-ansible | 02:33 | |
| *** miloa has joined #openstack-ansible | 05:24 | |
| *** SiavashSardari has joined #openstack-ansible | 05:53 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Return PyMySQL installation for distro installs https://review.opendev.org/c/openstack/openstack-ansible/+/786107 | 06:08 |
|---|---|---|
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Return PyMySQL installation for distro installs https://review.opendev.org/c/openstack/openstack-ansible/+/786107 | 06:10 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Return PyMySQL installation for distro installs https://review.opendev.org/c/openstack/openstack-ansible/+/786107 | 06:12 |
| *** tinwood has quit IRC | 06:22 | |
| *** tinwood has joined #openstack-ansible | 06:25 | |
| *** zbr has quit IRC | 06:30 | |
| *** zbr has joined #openstack-ansible | 06:32 | |
| *** macz_ has joined #openstack-ansible | 06:34 | |
| *** miloa has quit IRC | 06:34 | |
| *** PrinzElvis has quit IRC | 06:34 | |
| *** PrinzElvis has joined #openstack-ansible | 06:37 | |
| *** macz_ has quit IRC | 06:38 | |
| *** pto has quit IRC | 07:01 | |
| jrosser | morning | 07:09 |
| *** luksky has joined #openstack-ansible | 07:11 | |
| *** pto has joined #openstack-ansible | 07:12 | |
| *** andrewbonney has joined #openstack-ansible | 07:14 | |
| *** pto has quit IRC | 07:27 | |
| *** pto has joined #openstack-ansible | 07:38 | |
| *** miloa has joined #openstack-ansible | 07:40 | |
| *** miloa has quit IRC | 07:41 | |
| *** tosky has joined #openstack-ansible | 07:50 | |
| *** rpittau|afk is now known as rpittau | 07:55 | |
| zbr | o/ | 08:34 |
| *** macz_ has joined #openstack-ansible | 08:35 | |
| zbr | can someone explain me the history behind ansible-role-requirements.yml? apparently this seems to be unique to openstack and diverging from official ansible docs which only use requirements.yml | 08:36 |
| *** macz_ has quit IRC | 08:39 | |
| SiavashSardari | zbr openstack-ansible is kind of a wrapper for ansible commands and all features in general. maybe if you read https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/bootstrap-ansible.sh and https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml you can see that behind the | 09:09 |
| SiavashSardari | scene it is all pure ansible. | 09:09 |
| zbr | some kind of reimplementation of ansible-galaxy cli, in pure ansible, even if ansible-galaxy is part of ansible core. | 09:11 |
| zbr | i am not sure if the format is the same, but the logic seems very similar | 09:12 |
| noonedeadpunk | zbr: we don't publish our roles to galaxy and we clone them in parallel to save up time. But yes, that;'s pretty common to ansible-galaxy. Except I'm not sure if ansible-galaxy cli was implemented in ansible when ansible-role-requirements were introduced | 09:38 |
| noonedeadpunk | eventually we use ansible-galaxy cli for collections requirements https://opendev.org/openstack/openstack-ansible/src/branch/master/ansible-collection-requirements.yml | 09:40 |
| noonedeadpunk | but cloning roles with https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/library/git_requirements.py is so much faster then with ansible-galaxy cli.... | 09:41 |
| noonedeadpunk | But yes, technically we can I think move to usage of ansible-galaxy cli and merge these requirements files to just requirements.yml | 09:43 |
| zbr | galaxy ci can install both roles/collection and can work even without the galaxy server (no need to rely on it, common concern) | 09:56 |
| noonedeadpunk | main concern is clone speed at the moment | 09:56 |
| noonedeadpunk | (except historical) | 09:56 |
| zbr | is outside my power to improve galaxy tool performance | 09:57 |
| zbr | my concern was more about use of non standard filename, which breaks some tools that rely on standard. | 09:57 |
| zbr | for example both ansible-lint and molecule do recognize these files and can install the deps declared in them, but have no clue about the ansible-role-requirements.yml one | 09:58 |
| zbr | probably keeping the current custom installer is not a bad idea, but how about migrating it to use standard filenames? | 09:59 |
| noonedeadpunk | I've added discussion point to upcoming PTG :) | 10:00 |
| zbr | thanks. | 10:00 |
| noonedeadpunk | eventually for ansible-lint we prepare env in advance by clonning all roles | 10:00 |
| zbr | noonedeadpunk: yep, but this means you ask developer to do extra preparation, it will never work fine right out of a clone. | 10:03 |
| zbr | for example a tool like https://marketplace.visualstudio.com/items?itemName=zbr.vscode-ansible will never be able to recognize the deps on those repos. | 10:04 |
| noonedeadpunk | well, it won't anyway, as we are also using custom modules that we don't have time to trnsform into collections, that needs to be cloned as well along with setting proper ENV vars | 10:04 |
| noonedeadpunk | We have pretty long developer guide as well... Yeah, that's not ideal I know and hard for new ppl to do testing | 10:06 |
| zbr | the linter does recognize modules from both roles and collections, and also provides a mock feature to mock roles/modules/collections in case making them available is too hard. | 10:07 |
| noonedeadpunk | sounds quite good! | 10:07 |
| zbr | there are few cases where mocking is easy path forward, i do not recommend it as you lose some checks but it is very easy to use. | 10:07 |
| zbr | see https://github.com/ansible-community/ansible-lint/blob/master/.ansible-lint#L10-L18 | 10:08 |
| noonedeadpunk | I think main issue for us is lack of the time to get things done | 10:09 |
| noonedeadpunk | And in case they are just working at the moment - they are left for later | 10:10 |
| *** fresta has joined #openstack-ansible | 10:16 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Use uWSGI for trove-api https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/784202 | 10:18 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Update trove configuration https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/784571 | 10:18 |
| *** snapdeal has joined #openstack-ansible | 10:26 | |
| evrardjp | jrosser: I remember that some ppl wanted to use haproxy as a healthcheck for keepalived. So it made sense to bounce haproxy state to try to recover from split. I never used that bit in production, as I had my own haproxy configuration. Keepalived config was just a guideline, and it was expressed as such in the documentation, IIRC | 10:58 |
| evrardjp | If you were to ask me how to do it right, I would drop the notification script, I would change the healthchecks in keepalived | 10:59 |
| evrardjp | The simplest case is relying on VRRP, and if necessary act. | 11:01 |
| evrardjp | If I really have to keep the notify script, I would not bounce back the haproxy to an "up" state. I would just stop haproxy. This way you avoid split brain cases. However, due to how we counfigured all our load-balanced services, I am not sure it matters | 11:02 |
| evrardjp | In one of my PoC before I moved to my own haproxy role (due to requirements outside OSA's scope), I needed the bounce service because the addressed I pinged for health was behind haproxy. | 11:03 |
| evrardjp | That should explain the whole story I hope :) | 11:04 |
| *** snapdeal has joined #openstack-ansible | 11:05 | |
| snapdeal | Team, Need little help to understand what I have messed up .. So I am using Ubuntu 20 for deploying Victoria and utilizing Open vSwitch for neutron.. in netplan config I have created bridges like br-vlan , br-storag, br-mgmt and in openstack_user_config i have just given already created bridge interface name.. | 11:21 |
| snapdeal | once all setup was ready .. I can see ovs error as below : | 11:22 |
| snapdeal | Port br-vlan | 11:22 |
| snapdeal | Interface br-vlan | 11:22 |
| snapdeal | type: internal | 11:22 |
| snapdeal | error: "could not add network device br-vlan to ofproto (File exists)" | 11:22 |
| snapdeal | Does the OSA take care of bridge creation ? and my mistake was not to create bridge using netplan ? | 11:23 |
| SiavashSardari | snapdeal I'm not sure but check brctl show. maybe something in netplan went wrong | 11:26 |
| SiavashSardari | and there was a linuxbridge br befor ovs one | 11:27 |
| admin0 | snapdeal, how does your netplan config looks like ? | 11:27 |
| admin0 | pastebin/gist | 11:27 |
| admin0 | osa does not touch bridges .. but it would want to add containers to the right bridge | 11:28 |
| admin0 | so what you need to do is only have ovs in the compute/network nodes .. but not on the controllers | 11:28 |
| snapdeal | network: | 11:29 |
| snapdeal | version: 2 | 11:29 |
| snapdeal | renderer : networkd | 11:29 |
| snapdeal | ethernets: | 11:29 |
| snapdeal | enp6s0: | 11:29 |
| snapdeal | dhcp4: no | 11:29 |
| snapdeal | dhcp6: no | 11:29 |
| snapdeal | ens15: | 11:29 |
| snapdeal | dhcp4: no | 11:29 |
| snapdeal | dhcp6: no | 11:29 |
| snapdeal | eth5: | 11:29 |
| snapdeal | dhcp4: no | 11:29 |
| snapdeal | dhcp6: no | 11:29 |
| snapdeal | bridges: | 11:29 |
| snapdeal | #Primary Network bridge to connect with all containers | 11:29 |
| snapdeal | br-mgmt: | 11:29 |
| snapdeal | interfaces: [enp6s0] | 11:29 |
| snapdeal | addresses: [10.47.0.187/22] | 11:29 |
| snapdeal | SiavashSardari : I guess you are right,.... | 11:33 |
| snapdeal | brctl show show br-vlan as linux bridge and same is being added to ovs bridge | 11:33 |
| SiavashSardari | I think there is miss conf in your netplan | 11:34 |
| SiavashSardari | try something like this http://paste.openstack.org/show/804456/ and reboot your server | 11:36 |
| snapdeal | Okay .. This will make that ovs bridge instead of linux bridge ? | 11:37 |
| SiavashSardari | this way netplan will create ovs bridges | 11:37 |
| snapdeal | got it..let me try .. thx | 11:37 |
| SiavashSardari | yep | 11:37 |
| *** pto has quit IRC | 12:08 | |
| SiavashSardari | I wanted to add [cors] to gnocchi and keystone conf. I did the same thing for both and run the playbook with config tag. keystone worked fine but gnochhi failed. I found https://review.opendev.org/c/openstack/openstack-ansible/+/728052 and https://github.com/ansible/ansible/issues/57529 and run the playbook without tags and problem solved. but I | 12:13 |
| SiavashSardari | didn't understand what happens here. to my eyes keystone and gnocchi playbooks looks the same in that matter. | 12:13 |
| SiavashSardari | noonedeadpunk it was your patch, maybe you can help me understand what went wrong here? | 12:14 |
| noonedeadpunk | well, probably for keystone your facts were relevant, since you was running some other playbook that gathered facts for the containers recently | 12:16 |
| *** pto has joined #openstack-ansible | 12:17 | |
| noonedeadpunk | the issue here is that facts are expiring for ansible in 24 hours (that's default). You can disable facts expiration to overcome that. But then you will need to clean facts with any infra change manually | 12:17 |
| jrosser | snapdeal: have you read this about how OVS is intended to be used with openstack-ansible? https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch.html | 12:18 |
| noonedeadpunk | and when you run with tags, facts gathering is not happening, because play has specific tag | 12:18 |
| SiavashSardari | noonedeadpunk Thanks for explanation. that's what exactly happened here | 12:19 |
| mgariepy | to refresh the facts you can always run `ansible all -m setup` | 12:20 |
| noonedeadpunk | yeah | 12:20 |
| *** luksky has quit IRC | 12:20 | |
| mgariepy | you can also add `-f 100` to make it faster ! | 12:20 |
| *** luksky has joined #openstack-ansible | 12:20 | |
| noonedeadpunk | or not run all, but gnocchi_all :p | 12:21 |
| *** luksky has quit IRC | 12:21 | |
| SiavashSardari | mgariepy I did that. but nothing changed and running playbook with tag failed | 12:21 |
| *** luksky has joined #openstack-ansible | 12:21 | |
| SiavashSardari | noonedeadpunk to be exact I did your version =D | 12:21 |
| mgariepy | depends on the playbook you run.. | 12:21 |
| snapdeal | Jrosser: Yes.. I have gone through this earlier.. But missed to read the note that these bridges are created if network_interface is defined... So best would be to not add it in netplan.. and let the OSA crate OVS bridge for us.. | 12:21 |
| mgariepy | sometimes it does matter if it's all or only a subset. | 12:21 |
| noonedeadpunk | SiavashSardari: for pre-V deployments I have smth like that in my tree http://paste.openstack.org/show/804464/ | 12:22 |
| noonedeadpunk | and I run openstack-ansible test.yml -e hosts=gnocchi_all | 12:22 |
| noonedeadpunk | or smth like that | 12:23 |
| SiavashSardari | mgariepy I checked my history, I tried both and no luck either | 12:23 |
| jrosser | snapdeal: i think it's important to distinguish between how you deploy the networking/bridges on the controllers (usually 99% of the time this is never done with OVS) and what you do on compute/network nodes which may be OVS and auto-created bridges if you want for specific networks | 12:23 |
| SiavashSardari | noonedeadpunk Thanks for the hint. I remember I had a similar issue when I added some new computes too. what changed at V about this? | 12:25 |
| noonedeadpunk | the patch you mentioned has merged :) | 12:25 |
| SiavashSardari | it is backported to ussuri too | 12:26 |
| snapdeal | Jrosser: Controller and Network nodes are on same node here :) .. We directly used VLAN based network for each project ... But yes I understood you point .. | 12:27 |
| SiavashSardari | jrosser I did all of my bridges including controllers with ovs. I did that since Q I gueses and it works fine too | 12:28 |
| noonedeadpunk | oh, well, I didn't have U anywhere, so... | 12:29 |
| *** snapdeal has quit IRC | 12:30 | |
| *** macz_ has joined #openstack-ansible | 12:36 | |
| *** snapdeal has joined #openstack-ansible | 12:39 | |
| *** macz_ has quit IRC | 12:41 | |
| *** spatel_ has joined #openstack-ansible | 12:59 | |
| *** spatel_ is now known as spatel | 12:59 | |
| *** snapdeal has quit IRC | 13:13 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: [doc] Document how to use separate RabbitMQ cluster https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/784781 | 13:26 |
| *** arxcruz has quit IRC | 13:57 | |
| *** arxcruz has joined #openstack-ansible | 13:59 | |
| *** SiavashSardari has quit IRC | 14:15 | |
| jrosser | can anyone clueful with centos/dnf suggest what to do here http://paste.openstack.org/show/804473/ | 14:37 |
| jrosser | as far as i can see the mirrorlist is sensible http://paste.openstack.org/show/804475/ | 14:40 |
| mgariepy | jrosser, which version of centos 8 ? | 14:46 |
| mgariepy | the latest ? | 14:46 |
| jrosser | centos8-stream, booted from my image centos-stream-8-20201019.1-amd64.raw | 14:47 |
| jrosser | so new 'ish' | 14:48 |
| jrosser | actually its failing in the chroot made in lxc_hosts for the container image | 14:48 |
| *** macz_ has joined #openstack-ansible | 14:49 | |
| mgariepy | try to swicth to vault.centos.org | 14:49 |
| *** macz_ has quit IRC | 14:49 | |
| mgariepy | your curl does works for the latest and greatest they probably don't keep the old version there | 14:49 |
| mgariepy | ""enterprise""!! | 14:50 |
| *** macz_ has joined #openstack-ansible | 14:50 | |
| jrosser | mad thing is i can dnf update just fine from the system itself | 15:03 |
| jrosser | and the yum repo config appears the same | 15:03 |
| jrosser | but yes "enterprise" indeed | 15:03 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Use template URLs in Cell Mappings https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/786258 | 15:06 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump SHAs for master https://review.opendev.org/c/openstack/openstack-ansible/+/785800 | 15:07 |
| mgariepy | jrosser, maybe some caching on the host that are not present in the chroot ? | 15:09 |
| jrosser | feels like the chroot thats built is too new for the files we rsync across from the host, perhaps also some stuff missing from the chroot | 15:19 |
| *** d34dh0r53 has quit IRC | 15:20 | |
| *** d34dh0r53 has joined #openstack-ansible | 15:22 | |
| mgariepy | hmm sounds painful. | 15:26 |
| mgariepy | isn't rocky or alma linux available now ? | 15:27 |
| spatel | mgariepy i don't think rocky is read yet | 15:37 |
| spatel | Q2 2021 it will be available hope - https://rockylinux.org/news/community-update-february-2021/ | 15:38 |
| noonedeadpunk | I put more trust into almalinux tbh | 15:41 |
| noonedeadpunk | And I think it's already available | 15:42 |
| noonedeadpunk | But pretty sure that rhel lobby in OIF won't let creating alternative images in CI | 15:42 |
| jrosser | huh right it needs centos-stream-release centos-stream-repos in the chroot | 15:42 |
| jrosser | previously it was good enough to rsync /etc/yum.repos.d across, but aparrently not any longer | 15:43 |
| noonedeadpunk | doh..... | 15:43 |
| noonedeadpunk | well, probably that would be ok, if we had stream based host? | 15:44 |
| jrosser | finally i have side by side LXC AIO, one centos-8, one 8-stream | 15:44 |
| jrosser | oh this was all on the stream host | 15:44 |
| noonedeadpunk | whaaat | 15:44 |
| jrosser | yup | 15:44 |
| jrosser | was going to try to find WTF is making the stream one soooo slow for lxc | 15:45 |
| jrosser | but apart from a bit off diff on the output of sysctl i'm not really sure where to start | 15:45 |
| noonedeadpunk | (╯°□°)╯︵ soʇuǝɔ | 15:46 |
| jrosser | nice | 15:46 |
| noonedeadpunk | Well, I compared packages back then, but no luck there... | 15:47 |
| jrosser | this is getting kind of problematic now, we really need to drop the centos8 jobs asap | 15:47 |
| noonedeadpunk | So I have no idea what made that specific docker built working nicely... | 15:47 |
| spatel | jrosser what is the status of parallel repo issue? any pending action item there? | 15:57 |
| jrosser | i've not done anything | 15:58 |
| jrosser | theres two things mixed up | 15:58 |
| jrosser | if we think there is a bug with the parallel git clone, its possible | 15:58 |
| jrosser | but also really why we do shallow clones at all | 15:58 |
| * noonedeadpunk is going to try asking in #centos-stream or #centos-devel | 15:59 | |
| spatel | may be Git shallow clone lets you pull down just the latest commits, not the entire repo history. | 15:59 |
| spatel | may be that is our issue its not cloning entire repo | 16:00 |
| jrosser | as far as i know shallow clones can be really expensive for the git server to compute | 16:01 |
| spatel | I am curious why we hit that bug now not in past? | 16:01 |
| jrosser | the parallel clone stuff is fairly recent | 16:01 |
| spatel | hmm! if its not adding lots of value then why don't we remove it :) | 16:02 |
| noonedeadpunk | so parallel clone fails again because of the missing tree? despite we extended depth? | 16:02 |
| jrosser | spatel did an upgrade which didnt work, and i think thats another case where #commits could easily be >20 | 16:03 |
| spatel | noonedeadpunk if you want error log i can try to rebuild lab and collect data | 16:04 |
| jrosser | spatel: it should be reproducible in an AIO be checking out an old tag, bootstrap-ansible, a much newer one then re-bootsrap | 16:05 |
| jrosser | theres no need for complex setup to do that | 16:05 |
| jrosser | but you have to go from only having the old history to having one much newer, so needs to be a fresh build of an old tag | 16:05 |
| jrosser | imho i would rather remove the shallow clone | 16:06 |
| spatel | jrosser noonedeadpunk found error logs here its - http://paste.openstack.org/show/804358/ | 16:07 |
| jrosser | right, so it was always an error from the git command itself `git fetch --depth=20 --force\n stderr: 'fatal: error in object: unshallow 94d56d23b256071b223a85085fbfa4ad9212c972` | 16:07 |
| spatel | +1 agreed to remove complexity.. i would prefer wait then troubleshooting | 16:08 |
| jrosser | it's not a problem directly related to the parallel-ness of this | 16:08 |
| noonedeadpunk | spatel: but eventually code just fallbacks to old process in case parallel fails | 16:09 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml#L135-L149 | 16:09 |
| noonedeadpunk | but with depth we had a different output | 16:10 |
| spatel | yes it will fallback but with RED error message so always worried | 16:10 |
| noonedeadpunk | there tree wasn't found | 16:10 |
| jrosser | well `We've identified that this is a bug with the git.exe client when running `git repack -adfl` - removed (unreachable) objects are not removed from the `.git\shallow`, causing git to think it has an object during shallow fetch that it actually doesn't. Until the bug is fixed, the workaround is to use a regular fetch rather than a shallow one.` | 16:13 |
| jrosser | from vscode bugs | 16:13 |
| *** gyee has joined #openstack-ansible | 16:17 | |
| *** rpittau is now known as rpittau|afk | 16:21 | |
| noonedeadpunk | jrosser: was told to check out cloud image in https://quay.io/repository/centos/centos?tag=stream8&tab=tags as it's official source as well o_O | 16:22 |
| *** spatel has quit IRC | 16:22 | |
| jrosser | i wonder how we download that without needing docker | 16:24 |
| noonedeadpunk | not sure... | 16:26 |
| *** spatel_ has joined #openstack-ansible | 16:28 | |
| *** spatel_ is now known as spatel | 16:28 | |
| jrosser | i guess that there is another factor which is the infra centos8-stream image | 16:37 |
| jrosser | i'm going to time setup-infrastructure on centos8 + stream with locally built chroot, and the current downloaded/unpacked image we have today | 16:38 |
| *** andrewbonney has quit IRC | 17:59 | |
| *** ianychoi__ has quit IRC | 18:08 | |
| *** spatel has quit IRC | 18:15 | |
| *** spatel_ has joined #openstack-ansible | 18:19 | |
| *** spatel_ is now known as spatel | 18:19 | |
| jrosser | oh wow the dnf built chroot is spectacularly slow | 18:32 |
| jrosser | 35m11s for that vs 8m11s for the unpacked docker image as the rootfs | 18:32 |
| *** gshippey has quit IRC | 18:37 | |
| noonedeadpunk | is just simple lxc-attach and execute smth basic, like `w` is the same among them? | 19:38 |
| jrosser | that’s next I think, benchmark lxc attach and also ansible with/without the connection plugin | 20:09 |
| *** spatel has quit IRC | 20:15 | |
| *** luksky has quit IRC | 22:34 | |
| *** luksky has joined #openstack-ansible | 22:34 | |
| *** luksky has quit IRC | 22:40 | |
| *** mrda has quit IRC | 22:45 | |
| *** mrda has joined #openstack-ansible | 22:46 | |
| *** luksky has joined #openstack-ansible | 22:53 | |
| *** macz_ has quit IRC | 23:05 | |
| *** luksky has quit IRC | 23:18 | |
| *** tosky has quit IRC | 23:29 | |
| *** irclogbot_3 has quit IRC | 23:50 | |
| *** irclogbot_3 has joined #openstack-ansible | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!