| *** dviroel|out is now known as dviroel | 00:20 | |
| *** dviroel is now known as dviroel|out | 00:32 | |
| dmsimard | btw, stumbled about a talk in part about openstack-ansible from the ansible london meetup: https://www.meetup.com/Ansible-London/events/284846000/ it was recorded on youtube: https://youtu.be/VFwqUBlofZg?t=4345 | 01:03 |
|---|---|---|
| dmsimard | he gives glowing reviews and praises the channel for being great and helpful :D | 01:03 |
| *** ysandeep|out is now known as ysandeep | 05:10 | |
| jrosser | morning | 06:52 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_mount master: Add support for glusterfs mounts https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/837750 | 06:53 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Add facility to store repo contents on a remote mount https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837587 | 06:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 06:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 06:59 |
| jrosser | damiandabrowski[m]: following up on the ceph thing from yesterday `ceph-osd/focal-updates,now 17.1.0-0ubuntu3~cloud0 amd64 [installed,automatic]` <- this means it's a package from Quincy release of Ceph, not Pacific i think...... | 07:37 |
| jrosser | thats almost certainly not what we want at all | 07:37 |
| lowercase | Correct.. 17.1 indicates it is not even the stable release, but the beta one. 17.2.0 is the only stable version available atm, latest pacific release is 16.2.7 | 07:40 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 07:44 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 07:48 |
| lowercase | and wow, Quincy hasn't even been released for 24 hours yet. | 07:50 |
| lowercase | alright, going back to sleep. Stupid pager duty woke me up over a memory high alert. | 07:51 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add deployment wide variable for ceph apt pinning. https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/838653 | 07:58 |
| *** ysandeep is now known as ysandeep|lunch | 08:23 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Add deployment wide variable for ceph apt pinning https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 | 08:34 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 08:46 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 08:46 |
| damiandabrowski[m] | jrosser: I'm on the meeting right now so can't look much into ceph pinning atm, but as mgariepy said yesterday, apt pinning won't solve our issues, we should add ceph repo as well early in setup-hosts stage | 08:50 |
| jrosser | ok | 08:51 |
| jrosser | i made patches for the pins but i think we can add the repo to openstack_hosts | 08:52 |
| *** ysandeep|lunch is now known as ysandeep | 08:55 | |
| damiandabrowski[m] | what do You think about my previous idea about using ceph_client role to achieve it? | 08:56 |
| damiandabrowski[m] | "i only wonder where exactly do we want to run ceph_client role(probably we need to implement 'preinstall' tag as well to avoid installing ceph packages and only add repo/pins?). | 08:56 |
| damiandabrowski[m] | Only on lxc_hosts as 'lxc-templates' package triggers ceph-related packages(librbd1,librados2) to be installed which breaks things or maybe on all baremetal nodes? | 08:56 |
| damiandabrowski[m] | the other question is when do we want to run it? only when OSA manages ceph, so groups['ceph_all'] | length > 0?"" | 08:56 |
| noonedeadpunk | jrosser: just in case I commented on openstak_hosts patch | 09:00 |
| noonedeadpunk | I think we should adjust approach a bit for it to have more sense. As now we wil do pinning regardless of ceph being used or not at all which is weird kind of... | 09:00 |
| jrosser | i ave meetings now but we should discuss | 09:01 |
| noonedeadpunk | me too... | 09:03 |
| *** dviroel|out is now known as dviroel | 11:23 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki master: Allow to provide passphrase for keys https://review.opendev.org/c/openstack/ansible-role-pki/+/838713 | 11:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki master: Explicitly use community.crypto collection https://review.opendev.org/c/openstack/ansible-role-pki/+/838714 | 11:37 |
| mgariepy | hey | 11:48 |
| mgariepy | ping me when discussing the ceph pinning stuff :) | 11:48 |
| mgariepy | i'll try to be around tho i have a couple things to do away from my desk. | 11:49 |
| jrosser | mgariepy: you make a good point about needing to install the ceph repos early | 11:50 |
| jrosser | we need to decide if we always install the ceph.com repos alongside UCA | 11:51 |
| jrosser | otherwise we will get ceph 17.0 components *always* on lxc hosts due to dependancies | 11:51 |
| jrosser | or we always install ceph.com repo and pins then we get 16.x components everwhere as expected | 11:52 |
| jrosser | ^ seems we perhaps have no choice but to always install the ceph.com repo even for non ceph deploys, what does anyone else think? | 11:52 |
| jrosser | even today i wonder what will happen to a deployment with OSA integrated ceph, if there will be an accidental 16.x -> 17.x situation at upgrades | 11:54 |
| damiandabrowski[m] | the more I think about it, the more I lean towards setting ceph repo even on non-ceph deploys | 12:03 |
| damiandabrowski[m] | and for non-ceph deploys, `ceph_stable_release` variable still can be used to choose exact repo | 12:05 |
| noonedeadpunk | can anybody explain me why we need to pin ceph for non-ceph deployments?:) | 12:07 |
| damiandabrowski[m] | i think we don't need, need | 12:08 |
| damiandabrowski[m] | but it might be just more reasonable | 12:08 |
| noonedeadpunk | do we have some dependencies non-ceph that cones from uca and conflict ? | 12:08 |
| noonedeadpunk | So why we can't just define a variable in group_vars/ceph_all for that? | 12:09 |
| damiandabrowski[m] | what variable do You have in mind? | 12:09 |
| noonedeadpunk | then openstack_host will manage pinning early for members | 12:09 |
| noonedeadpunk | and for client - we can do this as we do now in ceph_client? | 12:09 |
| noonedeadpunk | or I'm missing here smth? | 12:09 |
| noonedeadpunk | damiandabrowski[m]: implemented by https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 :) | 12:10 |
| noonedeadpunk | but the one which won't have any default there :) | 12:10 |
| jrosser | lxc now pulls in librados | 12:10 |
| jrosser | and with only UCA you get ceph Q release parts | 12:10 |
| noonedeadpunk | whaaaaaaat | 12:10 |
| jrosser | yes that is the root of this issue | 12:11 |
| damiandabrowski[m] | lxc-templates->cloud-image-utils->qemu-utils->qemu-block-extra->librbd1/librados2 | 12:11 |
| jrosser | UCA now carries ceph 17.x packages | 12:11 |
| damiandabrowski[m] | so during lxc-templates package install, librbd1/librados2 is installed as a dependency | 12:11 |
| jrosser | and unless we do something about that *everywhere* then it's a mess, because of lxc | 12:11 |
| jrosser | or the other way to look at it, is we have UCA in places we don't need it | 12:12 |
| noonedeadpunk | lxc-templates doesn't _really_ depend on cloud-image-utils, does it? | 12:12 |
| jrosser | but thats not a solution for metal deploys | 12:12 |
| noonedeadpunk | At least https://packages.ubuntu.com/focal/lxc-templates says it's just a recommendation? | 12:12 |
| jrosser | we leave it up to deployers if they set up their hosts with no-recommends for apt though? | 12:13 |
| jrosser | anyway - i think this just highlights that we have a large ceph breakage risk due to not really being in control of if the packages come from ubuntu / uca / ceph.com | 12:13 |
| damiandabrowski[m] | noonedeadpunk: hmm, so the relation chain may look differently, but lxc-templates triggers librbd1/librados2 to be installed for sure | 12:14 |
| jrosser | damiandabrowski[m]: `APT::Install-Recommends "false";` | 12:15 |
| noonedeadpunk | um.... that sounds wrong to me damiandabrowski[m]... I kind of fail to see how that happens | 12:15 |
| damiandabrowski[m] | https://paste.openstack.org/raw/b9M4dqsvcn1ymXeksPgQ/ | 12:15 |
| damiandabrowski[m] | it's definitely one of these 3 packages, yesterday I checked it and I'm pretty sure it's lxc-templates | 12:16 |
| * mgariepy reading backlogs | 12:17 | |
| jrosser | well...... https://paste.opendev.org/show/bq2sVOHswt7H7RlBBzvi/ | 12:18 |
| jrosser | seems i don't get any ceph parts there | 12:18 |
| mgariepy | i guess qemu-utils is on uca | 12:19 |
| mgariepy | not installing recommends seems to be a good idea. but do we want to do that for all services ? | 12:20 |
| noonedeadpunk | for some reason I thought that by default you don't get recommendations installed? | 12:20 |
| jrosser | i am failing to reproduce this tbh | 12:20 |
| damiandabrowski[m] | yeah i thought that recommended packages are not installed by default but seems i was wrong | 12:21 |
| mgariepy | by default they are installed | 12:21 |
| noonedeadpunk | Also, if we're talking about qemu-utils, we install that only on computes. So then our only issue is ordering kind of... | 12:21 |
| mgariepy | should we pin librados2 librbd to lower priority on uca then ? | 12:22 |
| noonedeadpunk | well, indeed, it tries to install librbd and qemu-utils, pfff | 12:26 |
| mgariepy | not installing recommends will probably need quite a few patch to fix stuff. | 12:26 |
| noonedeadpunk | well, this can be fixed with 1 patch I believe :) | 12:27 |
| noonedeadpunk | But then we will likely be missing quite few packages... | 12:27 |
| mgariepy | we might broke some dependency | 12:27 |
| jrosser | noonedeadpunk: how does it do that? :) my fresh(ish) VM does not | 12:29 |
| jrosser | i tried the same command https://paste.opendev.org/show/bq2sVOHswt7H7RlBBzvi/ | 12:30 |
| noonedeadpunk | I believe mine does https://paste.openstack.org/show/bf7PfyxazKLLSMdjA43i/ | 12:30 |
| noonedeadpunk | maybe because lxc-templates already installed for you? | 12:30 |
| jrosser | yes you are right | 12:31 |
| jrosser | oh hold on | 12:34 |
| jrosser | this is becasue we moved to pin to yoga | 12:34 |
| jrosser | the yoga version of UCA brings in ceph 17.x packages http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/main/binary-amd64/Packages | 12:34 |
| jrosser | on my test vm don't have that change and still get 16.2.7 from `http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/xena/main amd64 Packages` | 12:35 |
| noonedeadpunk | yeah.... | 12:35 |
| miloa | Hi, we've got an openstack in openstack for our dev environment where we test our deployments with openstack-ansible. So we have to play with mtu. | 12:36 |
| noonedeadpunk | and moving pin sounds kind of reasonable... | 12:36 |
| mgariepy | still the pkg don't comes from the correct repo ? | 12:36 |
| noonedeadpunk | it kind of depends on definition of correct. | 12:36 |
| mgariepy | for lxc host without nova-compute librbd1 and librados2 should comes from focal-updates not uca | 12:37 |
| noonedeadpunk | I wonder if we can pin for ceph release name.... | 12:37 |
| miloa | Is there something for that in user variable for path_mtu and global_physnet_mtu like show in doc https://docs.openstack.org/neutron/yoga/admin/config-mtu.html | 12:37 |
| miloa | I did not find anything in os_neutron, but not sure if i missed something :) | 12:38 |
| noonedeadpunk | miloa: config overrides? | 12:39 |
| mgariepy | when ceph install it;'s repo it's pinned with a priority of 1000, what if we do simply set the priority of focal-update to 700 on lxc-hosts for librbd1/librados2 pkg? | 12:39 |
| jrosser | miloa: you can use neutron_neutron_conf_overrides to set global_physnet_mtu: | 12:39 |
| noonedeadpunk | miloa: like neutron_neutron_conf_overrides and neutron_ml2_conf_ini_overrides | 12:39 |
| noonedeadpunk | yah | 12:39 |
| jrosser | we get ceph things installed very early with lxc | 12:41 |
| miloa | jrosser, noonedeadpunk ok thanks a lot :) | 12:41 |
| noonedeadpunk | well another approach could be to prohibit installing ceph* from uca.... | 12:42 |
| mgariepy | for lxc hosts. | 12:42 |
| noonedeadpunk | I'd love to play with that bit tbh.... As maybe we can somehow set pin by release name of ceph which would be perfect... I kind of doubt we can though | 12:43 |
| jrosser | i think we can't, as thats determined by the UCA release | 12:44 |
| noonedeadpunk | yeah, fair | 12:44 |
| jrosser | and ceph.com has release specific repos, so that easier | 12:44 |
| jrosser | the most reliable approach is to always add ceph.com and always make it highest priority | 12:44 |
| *** ysandeep is now known as ysandeep|afk | 12:45 | |
| damiandabrowski[m] | +1 | 12:45 |
| mgariepy | or only when the host has also nova_compute or ceph group ? | 12:46 |
| noonedeadpunk | when: ceph_pkg_source | default('ceph') != distro ? | 12:46 |
| jrosser | or cinder too i think | 12:46 |
| noonedeadpunk | or glance | 12:47 |
| jrosser | :) | 12:47 |
| mgariepy | right lol | 12:47 |
| jrosser | * | 12:47 |
| noonedeadpunk | or manila | 12:47 |
| mgariepy | aio is a mess | 12:47 |
| mgariepy | haha | 12:47 |
| noonedeadpunk | it's even not about aio :) | 12:47 |
| mgariepy | well yeah, metal one also. | 12:47 |
| jrosser | well aio is broken - but i worry about real deployments | 12:48 |
| mgariepy | but i'm old and really like system container | 12:48 |
| jrosser | cinder-volume defaults to metal? | 12:48 |
| noonedeadpunk | it does | 12:48 |
| noonedeadpunk | checked it literally yestarday | 12:48 |
| jrosser | right so straight away the default config is in trouble with that vs. lxc | 12:48 |
| mgariepy | yep indeed. | 12:49 |
| noonedeadpunk | I quite frustrated about librbd dependancy for lxc-template | 12:49 |
| noonedeadpunk | that sounds weird and wrong for me | 12:49 |
| mgariepy | it's a recommends.. | 12:49 |
| damiandabrowski[m] | there's also few corner cases, where for ex. any node can become a part of ceph group during it's lifetime. In this case we'll also hit librbd1/librados2 downgrade error if we won't set ceph.com repos everywhere | 12:49 |
| noonedeadpunk | maybe we can just patch this specific task to avoid installing recomendations? | 12:49 |
| damiandabrowski[m] | not sure if we want to cover it though | 12:49 |
| noonedeadpunk | as out of lxc-templates we need jsut apparmor profile iirc | 12:50 |
| noonedeadpunk | but then indeed - we need to cover computes somehow.... | 12:50 |
| noonedeadpunk | then maybe we should also decrease prio of uca regarding ceph packages (to smth like 400) | 12:51 |
| mgariepy | installing all the repo early would generate a better state for pkg | 12:51 |
| mgariepy | uca do split ceph-osd and ceph-volume we might end up chasing packaging changes in the futures for uca/ceph if we pin certain pkg | 12:52 |
| noonedeadpunk | fair | 12:55 |
| noonedeadpunk | I just see some confusion for ppl that would find ceph repo in keystone container as example | 12:55 |
| mgariepy | indeed, | 12:56 |
| noonedeadpunk | So just trying to think about minimal thing we can do to avoid ceph everywhere and provide rleiable behaviour... | 12:56 |
| jrosser | maybe that says we have the UCA repo in too many places | 12:56 |
| mgariepy | we just need an equivalent to epel on ubuntu ;p | 12:57 |
| noonedeadpunk | yeah, I think we likely do have it | 12:57 |
| spatel | now you are telling centos is better :D | 12:57 |
| noonedeadpunk | IT IS NOT | 12:57 |
| noonedeadpunk | :p | 12:57 |
| mgariepy | LOL | 12:57 |
| spatel | I am not going back anyway :) | 12:58 |
| mgariepy | i've been burnt early with epel i do really hate it ;) | 12:58 |
| mgariepy | for which pkg do we need uca ? | 12:59 |
| noonedeadpunk | the most annoying thing with epel for me is that for each centos release package maintainers decide if they will build package for it. | 12:59 |
| noonedeadpunk | Even if package is present for latest fedora it doesn't mean it will appear for centos 8 or 9 | 12:59 |
| noonedeadpunk | which was quite surprise for me | 12:59 |
| noonedeadpunk | so no, thanks :) | 12:59 |
| damiandabrowski[m] | from my perspective, setting up ceph repo on all metal hosts may be the most reasonable option | 12:59 |
| mgariepy | is distro install dropped ? | 13:00 |
| noonedeadpunk | damiandabrowski[m]: even net nodes? | 13:01 |
| noonedeadpunk | so we're still back to group_vars kind of I believe... | 13:01 |
| noonedeadpunk | not yet | 13:01 |
| noonedeadpunk | basically we need ceph repo for lxc_hosts kind of... | 13:02 |
| mgariepy | distro install + uca yoga would be in the same situation i guess ? with a pkg downgrade ? or a pkg from uca. | 13:02 |
| jrosser | thankfully we never test that | 13:02 |
| mgariepy | LOL | 13:02 |
| mgariepy | if we don't test it it works ! :) | 13:03 |
| mgariepy | so we add the ceph repo to lxc_hosts so the gate is fixed but not copy the ceph-repo in the containers? | 13:04 |
| noonedeadpunk | um... I guess it won't... but not sure now once you asked :) | 13:05 |
| noonedeadpunk | and then for metal actually, lxc_hosts would be empty. | 13:06 |
| noonedeadpunk | damn it | 13:06 |
| damiandabrowski[m] | noonedeadpunk: I'm not saying it's the most optimal one :D but considering how complex it is to catch all scenarios where we need to fix it, adding this repo even on net nodes sounds reasonable | 13:07 |
| damiandabrowski[m] | i mean: it's only a repo, shouldn't make any harm(probably) :D | 13:07 |
| *** ysandeep|afk is now known as ysandeep | 13:08 | |
| noonedeadpunk | hm, so, uca has all ceph package under `Source: ceph` | 13:10 |
| noonedeadpunk | are we sure we can't jsut leverage priority of packages there? | 13:10 |
| noonedeadpunk | let me try to play with it a bit... | 13:10 |
| noonedeadpunk | I think I found smth | 13:19 |
| mgariepy | what is it ? :D | 13:20 |
| noonedeadpunk | Likely we can define `Package: src:ceph` but can't get it working yet :D | 13:23 |
| noonedeadpunk | At least it's said that in https://www.systutorials.com/docs/linux/man/5-apt_preferences/ | 13:23 |
| noonedeadpunk | Hm, or just set prio of ceph.com repo to 1001.... | 13:24 |
| noonedeadpunk | `causes a version to be installed even if this constitutes a downgrade of the package` | 13:25 |
| jrosser | but doesnt all the download.ceph.com packages also have `Source: ceph` ? | 13:28 |
| jrosser | or some kind of *source: ceph and repo: uca".... to reject packages | 13:29 |
| noonedeadpunk | you create a pin for a package that goes from repo | 13:29 |
| noonedeadpunk | so yeah - trying to source: ceph from uca | 13:29 |
| noonedeadpunk | but seems that you can't do that :( either you need to set prio for repo overall or if you want package - you should pin it to version | 13:31 |
| noonedeadpunk | I'm not sure you _really_ have to set `Package: * ` in preferences.d when you try to pin for repo... | 13:34 |
| jrosser | is this ok? https://paste.opendev.org/show/b2BBgbMc3rxyRaXDo90r/ | 13:36 |
| noonedeadpunk | Like I can google approach we need and reported as it's working.. | 13:36 |
| jrosser | i purge / reinsall librados2 and get this https://paste.opendev.org/show/bcPPxUsVNccvyLa2wnwB/ | 13:37 |
| noonedeadpunk | jrosser: Well, I can't get this working | 13:37 |
| noonedeadpunk | basically simple https://paste.opendev.org/show/bVeTlfXZDLxQAylcyt9F/ doesn't work either | 13:37 |
| noonedeadpunk | it does only when I set `Package: *` .... | 13:38 |
| jrosser | seems to work here using origin | 13:38 |
| jrosser | but i don't have the yoga uca though | 13:38 |
| noonedeadpunk | oh, huh | 13:39 |
| noonedeadpunk | So you're saying that I won't see change with apt policy but in fact it would be respected? | 13:39 |
| jrosser | see 'installed' and 'candiate' for these two https://paste.opendev.org/show/bmlZ5nRwnaXEz7P1pVS6/ | 13:41 |
| jrosser | i enabled the pin for the second one, commented out for the first | 13:41 |
| jrosser | and you see '444' as the priority for 16.2.7-0ubuntu0.21.04.1~cloud1 in the second apt-policy | 13:41 |
| noonedeadpunk | pfffff ) | 13:42 |
| noonedeadpunk | I was reading only line below lol | 13:42 |
| jrosser | which i guess shows that it adjusts the package prio rather than the repo | 13:42 |
| noonedeadpunk | and wasn't seing that for almost half an hour now :D | 13:42 |
| jrosser | doh /o\ | 13:42 |
| noonedeadpunk | yes, you're right | 13:43 |
| noonedeadpunk | so this should be easy fix I believe | 13:43 |
| noonedeadpunk | you want to push one? | 13:44 |
| jrosser | is it an adjustment to the patches i made before? | 13:44 |
| jrosser | like we hack this up https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 ? | 13:46 |
| noonedeadpunk | yeah, exactly ^ | 13:46 |
| noonedeadpunk | but again, maybe in this patch we should just introduce variable, or do some combine there | 13:47 |
| noonedeadpunk | as I see how ppl might want to avoid that or add more pinnings | 13:47 |
| noonedeadpunk | also I wonder if we should rely on meta? it's run each time role is included? | 13:48 |
| noonedeadpunk | I guess it is.... | 13:48 |
| noonedeadpunk | TIL that meta would respect variables that are defined in role defaults | 13:49 |
| noonedeadpunk | also we're not really consistent of how we're including apt_package_pinning - ie https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/master/tasks/install_apt.yml#L16 | 13:51 |
| noonedeadpunk | so somewhere call it from meta, somewhere just include role as usual | 13:53 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Add deployment wide variable for ceph apt pinning https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 | 13:58 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Add deployment wide variable for ceph apt pinning https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 | 13:58 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Prevent ceph packages installing from ubuntu-cloud-archive https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 | 13:59 |
| noonedeadpunk | jrosser: put one comment on that :) | 14:01 |
| jrosser | of course :) | 14:01 |
| jrosser | oh thats much better! | 14:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Prevent ceph packages installing from ubuntu-cloud-archive https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 | 14:02 |
| jrosser | i was a bit concerned about hardwiring the UCA hostname like that | 14:03 |
| jrosser | as we do provide places to override it, but passing into another role, ...... hmm | 14:03 |
| noonedeadpunk | We can Pin: release o=Canonical,n=focal-updates/yoga instead ofc... | 14:05 |
| noonedeadpunk | and get focal/yoga from vars... | 14:06 |
| noonedeadpunk | but indeed if url is overriden that could be anuthing as well... | 14:06 |
| noonedeadpunk | but yeah, `apt_repo_url` includes schema and path.... | 14:08 |
| noonedeadpunk | and there's no good way to parse that... | 14:13 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/wallaby: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/837297 | 14:15 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/xena: Use ansible roles corresponding to stable branches https://review.opendev.org/c/openstack/openstack-ansible-tests/+/837366 | 14:15 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/victoria: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/836335 | 14:15 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/victoria: Use ansible roles corresponding to stable branches https://review.opendev.org/c/openstack/openstack-ansible-tests/+/837368 | 14:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Do not deploy api-paste for CentOS distro deployment https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/838729 | 14:25 |
| *** ysandeep is now known as ysandeep|out | 14:30 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Cleanup distro vars https://review.opendev.org/c/openstack/openstack-ansible/+/837845 | 14:34 |
| mgariepy | dmsimard, thanks for sharing the meetup :) | 14:40 |
| noonedeadpunk | urgh, and now centos repos are broken..... | 14:49 |
| spatel | noonedeadpunk you mentioned about amqp bug in wallaby. does it part of 23.2.0 ? | 14:50 |
| noonedeadpunk | oh, yes, as well as issue with bootstrap of xena/wallaby... | 14:50 |
| spatel | Currently i am running 23.1.1 and having lots of issue related connection | 14:51 |
| spatel | i should be good to go with 23.2.0 correct? | 14:51 |
| spatel | Release note doesn't saying much so confirming with you - https://docs.openstack.org/releasenotes/openstack-ansible/wallaby.html | 14:52 |
| noonedeadpunk | spatel: 23.2.0 not including it. 23.2.1 likely does, but 23.2.2 includes that for sure | 14:56 |
| noonedeadpunk | basically you can just override `requirements_git_install_branch` to inlcude https://opendev.org/openstack/requirements/commit/887d45e86550bd5ffd25692f61063f78f85d7a2c | 14:56 |
| spatel | why 23.2.2 not showing up in release notes? | 14:57 |
| noonedeadpunk | itwas regular bump that updated upper constraints. | 14:58 |
| noonedeadpunk | fix wasn't part of osa change per say | 14:59 |
| spatel | oh! got it | 14:59 |
| spatel | Upgrade started | 15:21 |
| spatel | This always give me heart attack when i see RED screen - TASK [Clone git repos (parallel)] :) | 15:21 |
| spatel | noonedeadpunk is this safe to ignore? - https://paste.opendev.org/show/bjvOsaphPcjg5bCbqIu7/ | 15:24 |
| spatel | hmm ./scripts/bootstrap-ansible.sh failed | 15:24 |
| jrosser | there is an error with that SHA for ansible-sshd role | 15:25 |
| jrosser | idk why | 15:25 |
| jrosser | spatel: try updating the SHA for ansible-sshd to 59a3cb190f7913e32b044dc2733cfe7e46c4c99a | 15:27 |
| jrosser | oh | 15:27 |
| jrosser | sorry 2ba5bfee852aec5f699bf1c53193f7935beed825 | 15:27 |
| spatel | in ansible-role-requirements.yml file? | 15:28 |
| jrosser | yes - test lab right? :) | 15:28 |
| spatel | This is semi-production environment :) | 15:29 |
| spatel | why CI didn't catch this issue ? | 15:29 |
| jrosser | well, you can go look here and pick a SHA that is newer than the one which fails | 15:29 |
| jrosser | i have no idea | 15:29 |
| jrosser | https://github.com/willshersystems/ansible-sshd/commits/master | 15:29 |
| jrosser | maybe they force push to that repo, it's an external thing which other people can break | 15:30 |
| spatel | updated SHA and re-running script | 15:31 |
| *** dviroel is now known as dviroel|lunch | 15:31 | |
| spatel | no error this time all PASS | 15:32 |
| jrosser | we've seen the same thing in the past with the ceph-ansible repo | 15:32 |
| jrosser | no idea what they do to it to make it break though | 15:33 |
| spatel | hmm | 15:33 |
| jrosser | spatel: you can make a patch for that to fix the CI :) | 15:35 |
| spatel | Sure, i will do it | 15:35 |
| spatel | jrosser what do you guys use to speed up ansible? | 16:15 |
| noonedeadpunk | just in case I already generating bumps | 16:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Bump SHAa for Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/838754 | 16:17 |
| noonedeadpunk | jrosser: they jsut merge more then 20 commits since last bump | 16:17 |
| jrosser | yeah - lots of stuff | 16:17 |
| noonedeadpunk | I think I will add since for all repos..... | 16:18 |
| noonedeadpunk | as for ceph-ansible that has helped from what I can tell | 16:18 |
| jrosser | do we have a depends-on for that, as I don't think there is a ceph job run for that change | 16:30 |
| *** dviroel|lunch is now known as dviroel | 16:31 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Bump SHAs for Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/838758 | 16:43 |
| spatel | noonedeadpunk you already pushed ansible-sshd SHA so i don't need to cut the patch | 16:44 |
| spatel | openstack-ansible --forks 50 is it safe to run for speed up thing? | 16:46 |
| noonedeadpunk | If you have deploy host powerfull enough :) | 16:46 |
| spatel | its VM :D | 16:47 |
| spatel | setup-hosts.yml taking forever time :( | 16:47 |
| spatel | i have only 30 compute nodes | 16:48 |
| lowercase | its safe to use, but the more forks you spawn, the more cpu threads it will create and demand from your host. | 16:48 |
| spatel | i think 20 would be good enough then | 16:48 |
| lowercase | its going to take a lot of time regardless | 16:48 |
| spatel | I have kolla-ansible with 40 nodes production environment and somehow it takes few second to run any kind of ansible playbook. so i always think why OSA is slow :( | 16:50 |
| jrosser | you have to make appropriate use of tags really i think | 16:52 |
| jrosser | and if it doesnt work with tags, make bugs with a description | 16:52 |
| jrosser | it shouldnt be necessary to run vast amounts of the code just to make config updates | 16:53 |
| spatel | during upgrade process we can't use tags | 16:53 |
| spatel | 100% tag is best way | 16:54 |
| jrosser | well no, but you upgrades is a specific case | 16:54 |
| spatel | You know what let me run some test with fork and add more core in deployment host to see if that help in speed | 16:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Bump SHAs for Xena https://review.opendev.org/c/openstack/openstack-ansible/+/838762 | 16:59 |
| jrosser | noonedeadpunk: do you have a focal AIO? | 17:34 |
| jrosser | \o/ TASK [os_tempest : `Fail if tempest tests did not succeed] ok: [aio1_utility_container-f14d2839] => { "changed": false, "msg": All assertions passed"}` | 18:27 |
| jrosser | `lsb_release -c Codename:jammy` | 18:27 |
| mgariepy | nice :D | 18:34 |
| lowercase | it came out today | 19:31 |
| lowercase | ? | 19:31 |
| lowercase | i thought it was tomorrow | 19:31 |
| lowercase | it is tomorrow | 19:32 |
| jrosser | daily builds have been available for a while | 19:38 |
| jrosser | i used todays and a couple hours of hacking it passes tempest | 19:38 |
| *** dviroel is now known as dviroel|out | 22:27 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!