| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 00:41 |
|---|---|---|
| ozzzo | EugenMayer: I'd try in the morning; the developers are probably sleeping right now | 00:58 |
| ozzzo | I know it's possible to run 1 controller with a few compute nodes but I haven't done it. What document did you follow to set it up? | 00:59 |
| EugenMayer | Good morngin. Does the kolla multi-node deployment require to run with more 3+ controllers or is this optional? It seems that after rebooting the controller (in 1 controller setup), the controller seems to have lost the state of (the compute instances running) and seems to not load several pages. I kind of suspect an issue with an quorum or such. | 08:03 |
| EugenMayer | Any hints? | 08:03 |
| EugenMayer | in the keystone logs i see: ion failed. -11 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'controller.cluster.kontextwork.net' ([Errno 111] Connection refused)") | 08:06 |
| EugenMayer | might be the galera cluster not working with one controller - quorum? | 08:06 |
| hrw | morning | 08:11 |
| hrw | EugenMayer: I do not know (I do kolla mostly) | 08:11 |
| hrw | mnasiadka, yoctozepto, mgoddard: some cleanups: https://review.opendev.org/q/topic:%22some-debian-cleanups%22+(status:open%20OR%20status:merged) | 08:11 |
| EugenMayer | sorry for all the fuzz, it seems it was just the disk size of the controller, which i did not happen to notice | 08:48 |
| EugenMayer | the disk was full and the side effect was all the reset | 08:48 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Debian: enable OVN images for AArch64 https://review.opendev.org/c/openstack/kolla/+/815946 | 09:22 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Debian: enable building Bifrost images https://review.opendev.org/c/openstack/kolla/+/814783 | 09:24 |
| hrw | both got support matrix update | 09:24 |
| mnasiadka | fantastic | 09:35 |
| hrw | yoctozepto: thanks for pointing to bifrost-base on aarch64 | 10:17 |
| hrw | yoctozepto: replied | 10:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: docs: weekly meetings page https://review.opendev.org/c/openstack/kolla/+/815494 | 10:33 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Debian: enable building Bifrost images on x86-64 https://review.opendev.org/c/openstack/kolla/+/814783 | 10:51 |
| hrw | yoctozepto: now it is even more up to date | 10:51 |
| mnasiadka | I don't think bifrost supports aarch64 at all, so we shouldn't worry. | 10:52 |
| EugenMayer | mnasiadka the issue i had is actually https://review.opendev.org/c/openstack/neutron/+/813408 - is this one build in into the docker images for xena right now? | 10:57 |
| EugenMayer | it seems to have been merged, but does that automatically mean it has been build upstream? | 10:57 |
| mnasiadka | yes, if the container image build data is after this merge - then it's included | 10:58 |
| mnasiadka | if we're talking about -source install_type | 10:59 |
| EugenMayer | yes, using source | 11:00 |
| hrw | mnasiadka: they do not. And there was no request to make it happen | 11:00 |
| hrw | mnasiadka: https://review.opendev.org/c/openstack/kolla/+/815946 | 11:03 |
| *** amoralej|off is now known as amoralej | 11:03 | |
| hrw | mnasiadka: and https://review.opendev.org/c/openstack/kolla/+/816175 | 11:04 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: gnocchi: do not install cradox https://review.opendev.org/c/openstack/kolla/+/816170 | 11:04 |
| hrw | and this one too ;D | 11:05 |
| EugenMayer | mnasiadka thank you | 11:08 |
| opendevreview | Merged openstack/kolla master: Debian: enable OVN images for AArch64 https://review.opendev.org/c/openstack/kolla/+/815946 | 12:33 |
| opendevreview | Merged openstack/kolla master: neutron-openvswitch-agent: openvswitch is already installed https://review.opendev.org/c/openstack/kolla/+/816175 | 12:33 |
| hrw | mgoddard: https://review.opendev.org/c/openstack/kolla/+/816170 passed zuul. can you s/-1/+2 now? | 12:42 |
| yoctozepto | mgoddard, mnasiadka: the remaining mariadb patch ~> https://review.opendev.org/c/openstack/kolla-ansible/+/814276 | 12:56 |
| EugenMayer | When deploying freezer with kolla (xena) it fails creating the freezer conf in the freezer task since the template uses `os_project_domain_name = {{ openstack_auth.domain_name }}` (domain_name) but it is not set: {"msg": "'dict object' has no attribute 'domain_name'"} | 13:57 |
| EugenMayer | checkin kolla-ansible/ansible/group_vars/all.yml domain_name is not set | 13:59 |
| EugenMayer | seems to be a xena regression: https://github.com/openstack/kolla-ansible/blob/stable/wallaby/ansible/group_vars/all.yml#L889 | 14:00 |
| EugenMayer | https://github.com/openstack/kolla-ansible/blob/stable/xena/ansible/group_vars/all.yml#L883 | 14:00 |
| frickler | EugenMayer: that was likely missed when switching to using system-scope | 14:04 |
| EugenMayer | frickler not sure what system-scope | 14:04 |
| EugenMayer | is | 14:04 |
| mnasiadka | missed because there's no freezer CI | 14:05 |
| mnasiadka | and tbh I don't know if freezer works at all | 14:05 |
| EugenMayer | mnasiadka you mean in xena or in general? | 14:06 |
| mnasiadka | EugenMayer: in general | 14:06 |
| EugenMayer | so you guys use other backup options i guess | 14:06 |
| mnasiadka | I don't do backups at all :) | 14:07 |
| mnasiadka | But I'm rather not a good example ;) | 14:07 |
| EugenMayer | i tried to fix it adding this to globals.yml https://gist.github.com/EugenMayer/2d64c44e07c98853dbe8feb6938b3e89 | 14:08 |
| frickler | EugenMayer: this patch introduced it https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 14:08 |
| frickler | EugenMayer: almost, domain_name="Default" | 14:08 |
| EugenMayer | frickler do you expect those template vars to work in globals.yml? | 14:09 |
| frickler | domain_id=default. and that should really be a UUID but isn't for historical reasons | 14:09 |
| frickler | EugenMayer: I'm not sure whether that would work. I'll do a patch for the freezer template | 14:10 |
| EugenMayer | will i could patch it in all.yml to for sure | 14:10 |
| frickler | EugenMayer: btw. did you make progress with submitting patches yourself? I remember there was some docs issue | 14:10 |
| EugenMayer | frickler actually i gave up since it was not working with SSH and i was really taken in by all the other things not working, so i had no bandwidth left. I registered, created a gerrit account, signed the contribution CLA and all this, but AFAIR gerrits cli did not work due to auth issues. I guess i will try at a later point in time | 14:12 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: Fix freezer config template https://review.opendev.org/c/openstack/kolla-ansible/+/816363 | 14:15 |
| frickler | mnasiadka: yoctozepto: ^^ quick fix for freezer | 14:16 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Ubuntu: add support for Apt repository configuration https://review.opendev.org/c/openstack/kayobe/+/816364 | 14:17 |
| EugenMayer | frickler thank you for the quick fix | 14:25 |
| EugenMayer | frickler if you are at it and interested, https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#preparation .. all kolla-ansible statements are missing -i to point at the inventor | 14:25 |
| EugenMayer | frickler i think your patch is not going to work. Using it, it will then fails 'TASK [service-ks-register : keystone | Creating services] with 'An exception occurred during task execution. To see the full traceback, use -vvv. The error was: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) | 14:44 |
| EugenMayer | (Request-ID: req-1e42c29b-213f-445d-8155-adc367f2a292) | 14:44 |
| EugenMayer | ' | 14:44 |
| EugenMayer | i first though it is an upgrade issue (redployment), so i deployed from scratch, but same issue | 14:45 |
| EugenMayer | oh | 14:45 |
| EugenMayer | you patched it the other way arround, i see. Using a static in the template. sorry, misunderstood it. will try | 14:46 |
| EugenMayer | if interesting for anybody. Tried a wallaby to xena upgrade. Worked without issue AFAIKS | 14:56 |
| jingvar | what is mariadb root account and password in kolla/passwords.yaml? | 15:02 |
| mnasiadka | jingvar: I think it's self explanatory? | 15:07 |
| EugenMayer | frickler mnasiadka with the template fix of fricker freezer installed and shows up in horizon. Cannot tell yet its fully functional, but it is their basically | 15:09 |
| *** amoralej is now known as amoralej|off | 15:13 | |
| jingvar | mnasiadka: toomany database_password for one file and me | 15:17 |
| mnasiadka | jingvar: each service has it's mariadb user and password, and there's root (mariadb ,,admin'') account | 15:17 |
| jingvar | I understand , just was blind | 15:37 |
| jingvar | Aborted connection 131071 to db: 'keystone' user: 'keystone' host: 'ctl02' (Got an error reading communication packets) | 15:37 |
| jingvar | faced with ^^^ | 15:37 |
| jingvar | and similar for other users | 15:38 |
| EugenMayer | max packet size? | 15:40 |
| yoctozepto | EugenMayer, frickler: let me know if you get freezer actually working; we've been getting only negative feedback for several cycles now | 15:47 |
| EugenMayer | yoctozepto cycles? you mean from customers? | 15:47 |
| jingvar | EugenMayer: default | 15:48 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Ubuntu: add support for Apt repository configuration https://review.opendev.org/c/openstack/kayobe/+/816364 | 15:48 |
| yoctozepto | EugenMayer: release cycles, like wallaby, xena, ... :-) users, not customers :D | 15:48 |
| EugenMayer | i see, sorry. Well i will know more tomorrow i guess. Installed it in my lab, looks fine, rolling out on the prod right now. Had to do a wallaby to xena update there first | 15:49 |
| jingvar | I 'm not sure that was before redeploy. we added software raid support, and redeploy the env, some controllers have raid sincing | 15:50 |
| jingvar | maybe disk IO is the reason, but fresh database is small , I think | 15:50 |
| yoctozepto | EugenMayer: no worries; have you actually tried the backup option? | 15:51 |
| EugenMayer | from xena? | 15:51 |
| EugenMayer | you mean the snapshot which is flagged as backup, right? | 15:51 |
| yoctozepto | I mean the functionalities of freezer | 15:52 |
| EugenMayer | not yet, not tried anything in that regard. Had to align it all first. Need to handle the massive slowdown on the compute first, maybe an issue with AMD and bullseye and the qemu version, not sure | 15:53 |
| EugenMayer | its an AMD Ryzen 9 5950X 16-Core Processor, usually handles heavy load, right now on VM with bullseye installing python3-pip, taking 3 mins to unpack | 15:54 |
| jingvar | max_allowed_packet | 16777216 | 15:54 |
| EugenMayer | if it is mac_allowed_packet, you should see it in the MySQL logs. Could be other things too though | 15:55 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Ubuntu: add support for Apt repository configuration https://review.opendev.org/c/openstack/kayobe/+/816364 | 15:57 |
| hrw | https://review.opendev.org/c/openstack/kolla/+/816170 waits for brave cores (and mgoddard) | 16:06 |
| mgoddard | hrw: you're saying I'm not a brave core? | 16:06 |
| hrw | mgoddard: you already looked at it in past so you are reviewer now | 16:06 |
| hrw | hm. nova-compute. how I msised it... | 16:11 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Debian: we want Openvswitch and ovn from backports https://review.opendev.org/c/openstack/kolla/+/816374 | 16:51 |
| hrw | this should handle it | 16:51 |
| EugenMayer | seems like the xena ubuntu source version of nove (qemu libvirt) does not properly support AMD Ryzen 9 5950X | 16:58 |
| EugenMayer | virtualization is extremly slow, using host-passthrough fixes it | 16:59 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Ubuntu: add support for Apt repository configuration https://review.opendev.org/c/openstack/kayobe/+/816364 | 17:23 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Ubuntu: add support for Apt repository configuration https://review.opendev.org/c/openstack/kayobe/+/816364 | 17:41 |
| opendevreview | Merged openstack/kolla master: gnocchi: do not install cradox https://review.opendev.org/c/openstack/kolla/+/816170 | 18:05 |
| EugenMayer | yoctozepto / frickler since you asked, i could create an action ( nova/nova/local fs storage), created an job but when starting the job it fails with `2021-11-02 19:33:12.997 7 ERROR freezer.scheduler.scheduler_job [-] Freezer client error: b'Critical Error: auth_url required to authenticate. Make sure to export | 18:34 |
| EugenMayer | OS_AUTH_URL=http://keystone_url:5000/v3\n' | 18:34 |
| EugenMayer | 2021-11-02 19:33:12.997 7 ERROR freezer.scheduler.scheduler_job [-] Job 0486acb71cfd4d4e904803cc9c835bc5 action backup failed after 1 tries | 18:34 |
| EugenMayer | ` | 18:34 |
| yoctozepto | EugenMayer: meh, expected to fail but not like this; I guess it lost auth_url? pastebin that service's config file | 18:40 |
| yoctozepto | and create a bug report for each issue | 18:40 |
| yoctozepto | we need them anyway ;p | 18:40 |
| EugenMayer | i have not clue what those config files are. You expect someone who tried this thing the first time, it never worked, to create perfect bundles of bug reports and to understand where the configuration, logs and containers run, are segmented and splitted. This is kinda weired :) | 18:41 |
| yoctozepto | EugenMayer: well, what can I do? (-: | 18:42 |
| EugenMayer | yoctozepto frickler: https://bugs.launchpad.net/freezer/+bug/1837028 that is the very same bug | 18:50 |
| EugenMayer | commented on the issue | 18:56 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 20:04 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Ubuntu: add support for Apt repository configuration https://review.opendev.org/c/openstack/kayobe/+/816364 | 20:47 |
| EugenMayer | trying to deploy glance on a dedicated node called "backups", when doing so it fails during prechecks : https://gist.github.com/EugenMayer/b3a59dff4c7d510c0fa105b20feaa8d1 | 23:00 |
| EugenMayer | i configured the inventory (multinode) using https://gist.github.com/EugenMayer/b3a59dff4c7d510c0fa105b20feaa8d1#file-multinode - what did i do wrong? | 23:02 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!