| holtgrewe | How do I add kernel parameters to the overcloud system OS kernel that boots after successful deployment? | 07:01 |
|---|---|---|
| jingvar | deployment or provision? | 07:10 |
| jingvar | you can use hook feature to add some actions before or past a kayobe command | 07:15 |
| holtgrewe | jingvar, what's the difference between deployment and provision? | 07:35 |
| jingvar | provison is OS installatuion, deployment is installation of openstack services etc - in my mind | 08:26 |
| jingvar | OS-operastion system | 08:26 |
| holtgrewe | jingvar, ah, then I want to add to the kernel command line on provisoning | 08:39 |
| holtgrewe | Another question. I have trouble understanding how setup of bonds works. Should they be setup on provisioning or on `kayobe overcloud host configure`? I have four ports in my node, ens3f0, ens3f1, ens4f0, ens4f1. ens3fX go into bond0 ens4fX go into bond1. After the provisioned node boots, I have eth0 ... eth4. I have the following kayobe configuration in group_vars/controllers/network-interfaces | 08:42 |
| holtgrewe | https://gist.github.com/holtgrewe/b26c3c77fe3a42ddde8d6c8f35091b16 | 08:42 |
| holtgrewe | ens3f0 aka eth0 is used for PXE boot and provisioning | 08:42 |
| holtgrewe | After running kayobe overcloud host configure, only /etc/sysconfig/network-scripts/ifcfg-eth0 remains but also remains configured with IP address etc. | 08:44 |
| jingvar | kayobe overcloud host configure - configure bond and other interfaces | 08:44 |
| holtgrewe | What would be needed is /etc/sysconfig/network-scripts/ifcfg-eth{0..4} with entries "MASTER=bond0" but they are missing... | 08:45 |
| jingvar | by default kayobe uses eth0,1.. etc | 08:45 |
| holtgrewe | Yes, but should my network configuration file here properly setup the bond? https://gist.github.com/holtgrewe/b26c3c77fe3a42ddde8d6c8f35091b16 | 08:47 |
| jingvar | look at https://github.com/jingvar/a-universe-from-nothing/commit/da0c7237e6cd60094d47a6db0e1781c0fd9b2c26 | 08:47 |
| jingvar | there one interface in bond because it for virtual env | 08:49 |
| holtgrewe | ah, the trick is to add it to the controller networks | 08:53 |
| holtgrewe | thanks! | 08:53 |
| *** dardelean is now known as manheim | 09:01 | |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: docs: stop installing kolla in quickstart https://review.opendev.org/c/openstack/kolla-ansible/+/817523 | 09:13 |
| opendevreview | Merged openstack/kolla-ansible master: docs: fix venv path "share/share" https://review.opendev.org/c/openstack/kolla-ansible/+/817432 | 09:15 |
| jingvar | How someone Xena in production? | 09:23 |
| jingvar | How- >Have | 09:24 |
| mgoddard | jingvar: easy | 09:26 |
| mgoddard | git checkout stable/xena | 09:26 |
| mgoddard | kolla-ansible upgrade | 09:26 |
| mgoddard | :D | 09:26 |
| mgoddard | the question is why... | 09:26 |
| mgoddard | oh, I see - How -> Have | 09:26 |
| jingvar | We have to use Victoria for some reasons, but world is changed :) and now we have to choose a release for production | 09:29 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/xena: Ubuntu: add upgrade jobs to gate https://review.opendev.org/c/openstack/kayobe/+/817536 | 09:36 |
| holtgrewe | Another question, is there an ETA for Rocky support in Kolla/Kayobe? | 09:36 |
| mgoddard | jingvar: normally we wait 3-6 months from openstack GA, but some people do upgrade on the day of release (e.g. vexxhost) | 09:47 |
| mgoddard | holtgrewe: I don't think so. Possibly yoga, but it depends who wants it, and how much | 09:47 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/xena: docs: Fix python-openstackclient package name and init-runonce path https://review.opendev.org/c/openstack/kolla-ansible/+/817264 | 09:50 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/wallaby: docs: Fix python-openstackclient package name and init-runonce path https://review.opendev.org/c/openstack/kolla-ansible/+/817265 | 09:51 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/victoria: docs: Fix python-openstackclient package name and init-runonce path https://review.opendev.org/c/openstack/kolla-ansible/+/817266 | 09:51 |
| holtgrewe | mgoddard, thanks for the answer. We're coming from an "academic" CentOS 7 HPC deployment and like the "slow" release cycles. We're hesitant to switch to stream for that. However, maybe baremetal-compute can be deployed with rocky images already? | 09:56 |
| EugenMayer | jingvar why not use Xena? Seems like a stable release to me on Openstack - did i miss something? | 09:59 |
| mgoddard | holtgrewe: oh sure, you can put what you like on the baremetal compute nodes | 10:01 |
| mgoddard | EugenMayer: it just depends on your appetite for risk, since it's a new release | 10:02 |
| *** mazzy5098813 is now known as mazzy509881 | 10:08 | |
| opendevreview | Merged openstack/kayobe-config-dev master: Synchronise job config with kayobe https://review.opendev.org/c/openstack/kayobe-config-dev/+/817478 | 10:10 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Fix octavia doesn't set subnet gateway_ip https://review.opendev.org/c/openstack/kolla-ansible/+/817267 | 10:10 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Fix octavia doesn't set subnet gateway_ip https://review.opendev.org/c/openstack/kolla-ansible/+/817268 | 10:13 |
| opendevreview | Merged openstack/kolla-ansible stable/victoria: Fix octavia doesn't set subnet gateway_ip https://review.opendev.org/c/openstack/kolla-ansible/+/817269 | 10:13 |
| *** mazzy5098814 is now known as mazzy509881 | 10:22 | |
| holtgrewe | mgoddard, thanks for the info, I'd still prefer to have just one distribution rolled out ... if there's a way to help with rocky linux support, I'd be glad to help | 10:31 |
| mgoddard | holtgrewe: great! here is the place to start: https://review.opendev.org/c/openstack/kolla-ansible/+/815104 | 10:32 |
| jingvar | mgoddard: thanks for you advise | 10:34 |
| holtgrewe | mgoddard, what is the actionable there? use the branch/apply the patch and try to roll out with Rocky here, then report/fix any bugs? | 10:36 |
| *** mazzy5098816 is now known as mazzy509881 | 10:37 | |
| jingvar | We use Rocky as host OS | 10:37 |
| mgoddard | holtgrewe: sure - review the change, apply it in a test environment, report back | 10:38 |
| mgoddard | holtgrewe: we won't be supporting rocky containers. I'd suggest using stream for now | 10:38 |
| jingvar | Are there a real dependeces for Stream packages? | 10:39 |
| holtgrewe | mgoddard, that's fine. I have no strong feelings on OS in containers but IMO it's helpful to have only one distribution on the bare metal. And there I prefer distributions that have point releases... Maybe I'm just getting old ;-) | 10:40 |
| jingvar | just there should be same kernels - IMHO | 10:42 |
| jingvar | https://review.opendev.org/c/openstack/kolla-ansible/+/815104 - it is pain, for adding support Rocky you have to add it into Ansible | 10:48 |
| holtgrewe | in case anyone has problems with IPA and dell servers/dell switches and LACP, I got rid of my problems by (1) switching to BIOS boot (apparently some traces of earlier manual UEFI Rocky deployment got stuck in the UEFI state...), (2) disabling the unused on-board NICs in BIOS, (3) manually enabling PXE boot *sigh*, and properly properly configuring lacp fallback with **short** fallback rates | 10:50 |
| holtgrewe | (realizing this channel is logged sending this to the future) | 10:50 |
| jingvar | sed -i "s/'OEL', 'Amazon', 'Virtuozzo', 'XenServer', 'Alibaba',/'OEL', 'Amazon', 'Virtuozzo', 'XenServer', 'Alibaba', 'Rocky',/g" \ | 10:51 |
| jingvar | /opt/kayobe/venvs/kayobe-control/lib/python3.6/site-packages/ansible/module_utils/facts/system/distribution.py | 10:51 |
| jingvar | and ofcourse change url to Rocky_cloud.qcow | 10:52 |
| *** mazzy5098818 is now known as mazzy509881 | 10:52 | |
| jingvar | holtgrewe: do you have LACP -ad, or fall back | 10:54 |
| jingvar | active-backup | 10:54 |
| holtgrewe | 802.3ad https://gist.github.com/holtgrewe/e78bcf06e53992bc48f29a57e9ff5f3a | 10:55 |
| holtgrewe | fallback | 10:55 |
| jingvar | I have a fun when I had deffernt LACP modes on switch and server | 10:55 |
| holtgrewe | if you don't use lacp rate fast then PXE will work but the network manager in IPA will not work | 10:56 |
| holtgrewe | at least in my hands | 10:56 |
| jingvar | it is on pxe interface boundle or data network | 10:57 |
| jingvar | we have 1 dedictated nic for pxe boot and bonds for tennants, storage etc networks | 10:59 |
| jingvar | without fallback | 10:59 |
| holtgrewe | I have just one bond for everything ;-) Coming from a classic HPC setup. | 11:00 |
| jingvar | it is bad idea | 11:02 |
| holtgrewe | how so? | 11:02 |
| jingvar | if you will have a loop in data networks - you lose control | 11:03 |
| jingvar | or some network issues | 11:04 |
| jingvar | imho minimal configuration is 1G and 10+10G | 11:04 |
| holtgrewe | true... well I "just" want to migrate away from our old xCAT based deployment | 11:05 |
| jingvar | understand | 11:06 |
| holtgrewe | I can probably rearchitecture later for the controllers/non-baremetal compute. | 11:08 |
| holtgrewe | but thanks for the pointer! | 11:08 |
| holtgrewe | besides, I have a couple of old blades in there that are connected via mezzanine cards and there I only have ILO+2x10GbE | 11:08 |
| jingvar | we have implemented raid for controllers, but there rottary disks 2T and initial raid sync is very slow | 11:12 |
| jingvar | I didn't dive into the issue, but maybe if we are obvious missed something | 11:13 |
| opendevreview | Verification of a change to openstack/kayobe-config-dev master failed: Define infra VMs for testing https://review.opendev.org/c/openstack/kayobe-config-dev/+/805239 | 11:34 |
| opendevreview | Verification of a change to openstack/kayobe master failed: CI: add Infra VM jobs https://review.opendev.org/c/openstack/kayobe/+/813048 | 11:34 |
| opendevreview | Merged openstack/kayobe stable/xena: Ubuntu: add upgrade jobs to gate https://review.opendev.org/c/openstack/kayobe/+/817536 | 11:47 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: docs: Fix python-openstackclient package name and init-runonce path https://review.opendev.org/c/openstack/kolla-ansible/+/817265 | 11:56 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: docs: Fix python-openstackclient package name and init-runonce path https://review.opendev.org/c/openstack/kolla-ansible/+/817264 | 11:57 |
| *** mazzy5098812 is now known as mazzy509881 | 11:58 | |
| *** amoralej|off is now known as amoralej | 12:49 | |
| *** stackedsax_ is now known as stackedsax | 13:43 | |
| *** kklimonda_ is now known as kklimonda | 13:45 | |
| *** gouthamr_ is now known as gouthamr | 13:45 | |
| *** snbuback_ is now known as snbuback | 13:45 | |
| *** bbezak_ is now known as bbezak | 13:45 | |
| *** johnsom_ is now known as johnsom | 13:46 | |
| *** headphoneJames_ is now known as headphoneJames | 13:46 | |
| *** r3ap3r_ is now known as r3ap3r | 13:55 | |
| holtgrewe | jingvar, do you use centos or ubuntu? | 15:17 |
| holtgrewe | I somehow get a ifcfg-bond0 on CentOS 8 stream with Type=Ethernet ... https://gist.github.com/holtgrewe/f58e189f00bb1a3625d0a04a25b6b1ff | 15:20 |
| holtgrewe | ah, I'm missing a link.. | 15:24 |
| jingvar | rocky | 15:27 |
| jingvar | on host | 15:27 |
| jingvar | centos in containers | 15:27 |
| jingvar | ifcfg-bond0 there should be minion and bond mode | 15:30 |
| jingvar | BONDING_OPTS="mode=802.3ad miimon=100 " | 15:31 |
| holtgrewe | Yes, I had a problem with my kayobe/controllers.yml file... | 15:33 |
| holtgrewe | thanks | 15:40 |
| holtgrewe | After enabling rocky, I have to redo "kayobe seed service deploy"? | 15:40 |
| holtgrewe | Hmm.. I set os_distribution: rocky but then I get "unauthorized: access to the requested resource is not authorized" when trying to redeploy bifrost | 15:44 |
| admiyo | I've tried a handful of different approaches to getting Kolla to install. Currently, I am on Debian ,as that seems to be what is best supported on my hardware. The deploy (all-in-one) fails fairly early on in with | 16:03 |
| admiyo | RUNNING HANDLER [common : Initializing toolbox container using normal user] | 16:04 |
| admiyo | the error seems to be ["Error response from daemon: Container 4b793477564edf83431872baa9bbc98278951edd5b2d851ab753d04703bbf4a0 is restarting, wait until the container is running"] | 16:04 |
| admiyo | known issue, or something new and unusual? This is a Wallaby install. I am super flexible about what I should target to install, so long as it has a high likelihood of actually working. | 16:05 |
| admiyo | issue seems to be Fluent D right now | 16:06 |
| admiyo | kolla/debian-source-fluentd:wallaby | 16:06 |
| admiyo | do I even need FLuent D for a minimal install? | 16:06 |
| holtgrewe | admiyo, I have no clue about kolla itself, but I'm using kolla through kayobe roughly following the docs and this here https://github.com/stackhpc/a-universe-from-nothing | 16:13 |
| holtgrewe | and there also is a video | 16:13 |
| holtgrewe | and that includes an all-in-one that is nicely packaged with libvirt based KVM | 16:13 |
| holtgrewe | and it nicely worked | 16:13 |
| holtgrewe | I then went from there to a deployment to bare metal. | 16:13 |
| admiyo | We found one issue, which is where we are geting docker | 16:34 |
| admiyo | instead of using theDebian package, our PXE pulls in the docker apt sources | 16:35 |
| EugenMayer | admiyo you should not install docker yourself at all | 17:36 |
| EugenMayer | admiyo let kolla do it all for you, thus you are not running into any issues. I use kolla with debina bullseye myself, eventhough in a multi node setup and it works just fine. I use debian for the deployer running kolla and also the compute/controller OS. | 17:37 |
| EugenMayer | admiyo i do though use ubuntu/source for the container os, since at least for me (Xena) debian was not available. Probably has been fixed in the meantime | 17:38 |
| EugenMayer | admiyo also under debian, i would use pyenv in any case. | 17:38 |
| EugenMayer | admiyo you find a lab setup here https://github.com/EugenMayer/openstack-lab, maybe interesting for you are those scripts run on the deployer (kolla host) https://github.com/EugenMayer/openstack-lab/tree/master/deploy - eventhough this one installs Xena you most probably have an working example here - just use 'all-in-one' instead of multinode | 17:40 |
| *** amoralej is now known as amoralej|off | 17:41 | |
| admiyo | EugenMayer, I'm running on AARCH64 (ARM) and I don't know what is causing the disconnect. I have been doing a lot of trial and error, and trying not to fight unnecessary battles. I had some issues with VENV, so I backed off it. But This is a vanilla machine, for nothing but running Kolla, and a venv seems like one level of inception more than I need | 19:00 |
| admiyo | I do see that the Kolla adds in the docker apt source. A Vanilla machine does not have that apt source file. | 19:01 |
| admiyo | Right now we're resetting the networking for the controller node as it seems like we had a disconnect with our tech. Cloud is easy. It is networking that is hard.... | 19:02 |
| EugenMayer | you are running openstack on ARM? ok, that's interesting! | 19:11 |
| EugenMayer | admiyo venv is the opposite of fighting problems or too much inception | 19:12 |
| EugenMayer | you garant. the right environment without compormises or polouting the system. My system if for kolla only too. Anyway, surely you choice | 19:13 |
| EugenMayer | admiyo if i would be you, i would first make a reproduceable deployment on amd64, e.g. using a lab like i did. When you got it all tighten up, move to arm and try to get it up and running there. You might be jumping too far now and cannot rely on anything you did, so you search everywhere and nowhere. Maybe a too big step at a time, IMHO | 19:15 |
| jingvar | intersting | 19:15 |
| EugenMayer | admiyo if i'am not mistaken those people here: https://cloudbase.it/openstack-on-arm64-part-1/ build the images from source | 19:19 |
| EugenMayer | are there ARM docker images by kolla right now? | 19:20 |
| admiyo | EugenMayer, Heh...yes, yes they do. On our machines. | 19:22 |
| admiyo | And yes, we are doing exactly what you said. | 19:22 |
| admiyo | We have two clusters. One is "production" which I can't touch right now, because I have no way to bring it back up. It aged out of support. The other we recently tore apart, and are modernizing now. | 19:23 |
| EugenMayer | is your other cluster also openstack based and also arm based? | 19:25 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 19:34 |
| admiyo | Yeah, its an older ARM hardware used for control plane, but it is ARM | 20:23 |
| opendevreview | Merged openstack/kayobe-config-dev master: Define infra VMs for testing https://review.opendev.org/c/openstack/kayobe-config-dev/+/805239 | 22:20 |
| opendevreview | Merged openstack/kayobe master: CI: add Infra VM jobs https://review.opendev.org/c/openstack/kayobe/+/813048 | 22:20 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 23:15 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!