| *** ysandeep|out is now known as ysandeep | 03:45 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/845904 | 06:49 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/845904 | 06:50 |
| noonedeadpunk | so. most of upgrade jobs fail on tempest with process being killed | 06:55 |
| noonedeadpunk | /usr/local/bin/openstack-ansible: line 91: 131870 Killed ansible-playbook ${VAR1} "${@}" | 06:55 |
| jrosser_ | it’s not OOM is it? | 06:57 |
| noonedeadpunk | Well. Interesting is that it's really often ~2h when it's killed | 06:58 |
| noonedeadpunk | but yeah | 06:59 |
| noonedeadpunk | I think oom | 06:59 |
| noonedeadpunk | I can recall some patches to reduces processes, maybe we should backport smth... | 07:08 |
| noonedeadpunk | ok, see cloudkitty and mistral and both backported... | 07:11 |
| jrosser_ | neutron too if we make that uwsgi now? | 07:12 |
| noonedeadpunk | I think we have neutron already https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L107-L113 | 07:34 |
| noonedeadpunk | or not... | 07:35 |
| noonedeadpunk | maybe that's the reason actually | 07:36 |
| noonedeadpunk | that's really good point... | 07:36 |
| jrosser_ | this https://opendev.org/openstack/openstack-ansible/commit/6eb2dec708b41f3f8b8c61d4a916549205bfb562 | 07:36 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Reduce neutron WSGI workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/846117 | 07:37 |
| jrosser_ | i'm wondering if this is related to the problems we get with ansible 2.13 https://github.com/pallets/jinja/issues/1510 | 07:58 |
| noonedeadpunk | I haven't looked into 2.13 failures | 07:58 |
| jrosser_ | currently the version of jinja2 is held back in ansible-runtime venv by u-c | 07:59 |
| jrosser_ | yeah there are bigger things to fix this week i think | 07:59 |
| jrosser_ | our bionic jobs look unhappy https://zuul.openstack.org/builds?job_name=openstack-ansible-deploy-aio_metal-ubuntu-bionic&skip=0 | 08:22 |
| jrosser_ | urgh rabbitmq install trouble https://paste.opendev.org/show/bWr1wmqb2v4G4I1z61bh/ | 08:24 |
| noonedeadpunk | oh yes | 08:26 |
| noonedeadpunk | I wonder if we should also backport switching repos for rabbit... | 08:26 |
| jrosser_ | reformatted https://paste.opendev.org/show/btZwDwdi5Wj6AUj6Pimf/ | 08:26 |
| noonedeadpunk | As this issue is present at least for 3 days | 08:26 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826444 <- this one | 08:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/xena: Use cloudsmith repo for rabbit and erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/846120 | 08:29 |
| noonedeadpunk | oh, wait | 08:30 |
| noonedeadpunk | well, there's totally bionic mentioned | 08:33 |
| jrosser_ | do they have the versions we want though | 08:33 |
| noonedeadpunk | I see it here at least https://packages.erlang-solutions.com/erlang/debian/pool/ | 08:34 |
| jrosser_ | yes https://cloudsmith.io/~rabbitmq/repos/rabbitmq-server/packages/?q=distribution%3Aubuntu+AND+distribution%3Abionic+AND+version%3A3.9.8-1 | 08:35 |
| jrosser_ | what is actually breaking | 08:35 |
| noonedeadpunk | well, we don't use cloudsmith repo on Xena I guess | 08:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add mistral-extra to required projects in CI https://review.opendev.org/c/openstack/openstack-ansible/+/846135 | 08:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_mistral master: Add mistral-extra in the mistral venv https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/844049 | 09:00 |
| noonedeadpunk | jrosser_: let's also add mistral-extra to https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/defaults/repo_packages/openstack_services.yml | 09:05 |
| jrosser_ | https://review.opendev.org/c/openstack/openstack-ansible/+/844048 | 09:06 |
| jrosser_ | something wierd happening with magnum on centos https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/845993/ | 09:07 |
| jrosser_ | tempest appears to run without failures but gives rc=1 so the task fails | 09:08 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Forcefully remove Cinder v2 endpoints https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/845714 | 09:08 |
| noonedeadpunk | ah, ok | 09:11 |
| noonedeadpunk | regarding magnum - that looks suspicious https://zuul.opendev.org/t/openstack/build/676f976e87b3444399149a402ddb24a3/log/logs/openstack/aio1-utility/tempest_run.log.txt#51 | 09:13 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create master: Add ovs support for lxc containers. https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/837734 | 09:13 |
| noonedeadpunk | like it being killed during execution or smth | 09:13 |
| noonedeadpunk | and same test just fails for centos 9 https://zuul.opendev.org/t/openstack/build/48bf0c53a0cc4294a516aae6d2bcf349/log/logs/openstack/aio1-utility/tempest_run.log.txt#52 | 09:15 |
| jrosser_ | intereseting | 09:18 |
| jrosser_ | magnum API breaks https://zuul.opendev.org/t/openstack/build/676f976e87b3444399149a402ddb24a3/log/logs/host/magnum-api.service.journal-16-14-52.log.txt#2603-2612 | 09:21 |
| noonedeadpunk | I wonder if thats tempest plugin issue | 09:23 |
| noonedeadpunk | nah, it wasn;t changed since 2018 | 09:24 |
| opendevreview | Merged openstack/ansible-config_template master: Use ensure SafeDumper used yaml configs https://review.opendev.org/c/openstack/ansible-config_template/+/838141 | 09:25 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Use shlex_quote from ansible module_utils rather than SSH plugin https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845792 | 09:27 |
| noonedeadpunk | Hm, according to timestamps `ClusterTemplate fooo could not be found` is different from `OSError: write error` | 09:29 |
| noonedeadpunk | messages look even more interesting | 09:32 |
| noonedeadpunk | https://paste.openstack.org/show/bGPX0X4UqFg2D4Y0r3u5/ | 09:32 |
| noonedeadpunk | feels like oom as well tbh | 09:32 |
| noonedeadpunk | as basically it's glance that appears t obe dead | 09:33 |
| noonedeadpunk | Jun 16 01:36:09 aio1 glance-wsgi-api[49493]: 2022-06-16 01:36:09.464 49493 CRITICAL glance [req-f20f2697-06c0-4770-85d0-0e9581626f2a 6493ee21f61e4f03b23f314ce3990822 c239e4619618498ea1c785fa9acb014e - default default] Unhandled error: OSError: write error#0122022-06-16 01:36:09.464 49493 ERROR glance OSError: write error#0122022-06-16 01:36:09.464 49493 ERROR glance | 09:33 |
| noonedeadpunk | So OSError that magnum reports is also glance thing | 09:34 |
| noonedeadpunk | and placement reports same... | 09:34 |
| noonedeadpunk | hm | 09:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/wallaby: Use cloudsmith repo for rabbit and erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/846122 | 09:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Reduce m1.magnum flavor https://review.opendev.org/c/openstack/openstack-ansible/+/846142 | 09:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/846037 | 09:44 |
| opendevreview | Merged openstack/openstack-ansible master: Increase manila flavor size https://review.opendev.org/c/openstack/openstack-ansible/+/845998 | 09:54 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Fix glance mounts compatability https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/845767 | 09:55 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Remove mention of haproxy-endpoints role https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/845796 | 09:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update magnum tags and images for CI https://review.opendev.org/c/openstack/openstack-ansible/+/846145 | 10:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Increase manila flavor size https://review.opendev.org/c/openstack/openstack-ansible/+/846123 | 10:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Fixed dest typo in config_template https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/845993 | 10:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Reduce neutron WSGI workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/846117 | 10:14 |
| noonedeadpunk | tbh I'm surprised how good CentOS 9 is passing across roles | 10:15 |
| jrosser_ | yes, its worked out ok | 10:37 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Add support for role list https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690 | 10:38 |
| jrosser_ | noonedeadpunk: comment on here https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/845708 | 10:47 |
| noonedeadpunk | yep, thanks:) | 10:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_placement master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/845708 | 10:48 |
| *** ysandeep is now known as ysandeep|afk | 11:08 | |
| *** dviroel|afk is now known as dviroel | 11:18 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Delegate package import to setup host https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/846147 | 11:30 |
| noonedeadpunk | hopefully this will unblock murano... ^ | 11:31 |
| noonedeadpunk | exactly same issue as was with manilla | 11:31 |
| noonedeadpunk | not sure what to do with magnum though.... | 11:32 |
| noonedeadpunk | it just oom-ing | 11:37 |
| noonedeadpunk | oh, well... | 11:42 |
| noonedeadpunk | I think I found wtf | 11:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Limit magnum workers in CI https://review.opendev.org/c/openstack/openstack-ansible/+/846151 | 11:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Control amount of conductor workers https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/846153 | 11:52 |
| jrosser_ | ^ oh nice find | 11:57 |
| *** ysandeep|afk is now known as ysandeep | 12:18 | |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/846049 | 12:23 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/846013 | 12:25 |
| opendevreview | Merged openstack/openstack-ansible-os_tacker master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/846050 | 12:26 |
| opendevreview | Merged openstack/openstack-ansible-os_blazar master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/846015 | 12:26 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/845717 | 12:28 |
| opendevreview | Merged openstack/openstack-ansible-os_designate master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/846026 | 12:28 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/845903 | 12:29 |
| mgariepy | good morning | 12:32 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/846029 | 12:34 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/846043 | 12:35 |
| jrosser_ | hello mgariepy | 12:36 |
| mgariepy | anything urgent to review or fix ? | 12:37 |
| jrosser_ | there is this topic https://review.opendev.org/q/topic:osa%252Fservice_tokens though noonedeadpunk has some fixes in-progress for some of it | 12:38 |
| jrosser_ | and the remainder of https://review.opendev.org/q/topic:osa%252Fmanage_lb | 12:39 |
| noonedeadpunk | I guess main thing now is backport for neutron process limit | 12:39 |
| jrosser_ | looks like 846117 and 846120 are progressing well | 12:42 |
| noonedeadpunk | yup | 12:43 |
| noonedeadpunk | not really for W though... | 12:43 |
| noonedeadpunk | but anyway | 12:44 |
| jrosser_ | looks like a couple of jobs failing to upload logs again | 12:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Install murano client for distro installs https://review.opendev.org/c/openstack/openstack-ansible/+/846162 | 12:53 |
| noonedeadpunk | yup... | 12:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Delegate package import to setup host https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/846147 | 12:55 |
| noonedeadpunk | oh, nice to know that neutron was failing indeed because of missing nova patch | 12:58 |
| noonedeadpunk | for service_tokens I mean | 12:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/846163 | 13:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/846165 | 13:05 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/823009 | 13:21 |
| opendevreview | Merged openstack/openstack-ansible master: Bump SHA for pacemaker-corosync role https://review.opendev.org/c/openstack/openstack-ansible/+/845994 | 13:23 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server stable/xena: Use cloudsmith repo for rabbit and erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/846120 | 13:38 |
| mgariepy | wtf :/ https://zuul.opendev.org/t/openstack/build/529bbc921dd046698d1fa30fb7d7a6a7/log/job-output.txt#8483 | 13:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Define corosync interface for facts gathering https://review.opendev.org/c/openstack/openstack-ansible/+/846124 | 13:50 |
| noonedeadpunk | looks like just candidate for recheck.... | 13:51 |
| mgariepy | still weird. | 13:52 |
| noonedeadpunk | oh yes... it is.. | 13:57 |
| mgariepy | left a comment on : https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/846046 | 13:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/846046 | 14:02 |
| noonedeadpunk | yes, good catch, thanks | 14:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Create containers for rally/tempest independently https://review.opendev.org/c/openstack/openstack-ansible/+/840685 | 14:04 |
| *** ysandeep is now known as ysandeep|afk | 14:08 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Create containers for rally/tempest independently https://review.opendev.org/c/openstack/openstack-ansible/+/840685 | 14:15 |
| noonedeadpunk | I'm actually not sure how good this idea is overall now tbh ^ | 14:15 |
| noonedeadpunk | maybe we should just leave things as is with weird vars... | 14:15 |
| noonedeadpunk | As with that we add 2 containers at least, resource consumption with that as well | 14:16 |
| noonedeadpunk | and execution time | 14:16 |
| noonedeadpunk | Maybe I should look better in how to use single container for rally/tempest and if it's possbible... | 14:18 |
| noonedeadpunk | anyway this can be left for Z | 14:18 |
| *** ysandeep|afk is now known as ysandeep | 14:58 | |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/845904 | 15:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix deprecations handling in journal logs https://review.opendev.org/c/openstack/openstack-ansible/+/846189 | 15:30 |
| noonedeadpunk | Well, I'd say that we have everything required pushed. Hopefully, reducing neutron wsgi processes would help passing upgrade jobs | 15:39 |
| noonedeadpunk | as with that everything should jsut land | 15:39 |
| mgariepy | neutron worker is rechecking :/ | 15:40 |
| noonedeadpunk | yeah.... | 15:40 |
| *** ysandeep is now known as ysandeep|out | 15:58 | |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Delegate share type creation to setup host https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/845917 | 17:16 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/0e6a21621f3a40619c174c8f8dff9b30/log/logs/openstack/aio1_galera_container-2ac8c217/mariadb.service.journal-16-35-24.log.txt#278-280 | 17:24 |
| mgariepy | c9s error mariadb :/ | 17:25 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Reduce neutron WSGI workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/846117 | 17:47 |
| opendevreview | Merged openstack/ansible-config_template master: Release config-template to 1.3.0 https://review.opendev.org/c/openstack/ansible-config_template/+/845699 | 17:56 |
| noonedeadpunk | thanks mgariepy for rechecks) btw murano needs https://review.opendev.org/c/openstack/openstack-ansible/+/846162 and there're bunch of improvements in integrated repo for magnum | 18:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Remove mention of haproxy-endpoints role https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/845802 | 18:12 |
| noonedeadpunk | btw sorry, I found yours https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/840729 only when pushed another one :( | 18:13 |
| mgariepy | meh not an issue ;) | 18:13 |
| mgariepy | i didn't babysit it anyway :) | 18:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/846037 | 18:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Fixed dest typo in config_template https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/845993 | 18:20 |
| noonedeadpunk | btw for masakari I beleive we also need to land this https://review.opendev.org/c/openstack/openstack-ansible/+/846124 | 18:21 |
| noonedeadpunk | as their upgrade jubs fail here | 18:22 |
| mgariepy | yeah i saw that but i was kinda asking myself if the master one should merge first.. | 18:23 |
| mgariepy | but stuff are failing left and right today.. lots of weird issue.. | 18:23 |
| mgariepy | i wonder if we should collect : '/var/lib/mysql/mariabackup.backup.log' | 18:25 |
| mgariepy | as it would probably contains a bit of details on the reason why c9s has failed here : https://zuul.opendev.org/t/openstack/build/0e6a21621f3a40619c174c8f8dff9b30/log/logs/openstack/aio1_galera_container-2ac8c217/mariadb.service.journal-16-35-24.log.txt#275 | 18:26 |
| noonedeadpunk | ah, yes, you;re right.... | 18:34 |
| noonedeadpunk | I jsut backported it when there were jsut few jobs left on master.... | 18:34 |
| noonedeadpunk | as thought it will just pass... | 18:35 |
| mgariepy | yeah, random fails . | 18:35 |
| noonedeadpunk | well yes, good point that we should likely collect it. though I didn't thought we run backups in CI... | 18:36 |
| noonedeadpunk | actually should we? | 18:36 |
| noonedeadpunk | As I guess while backup run, mariadb goes to donor state and haproxy consider such backend as down iirc? | 18:37 |
| mgariepy | in this case trhe issue is the second galera node could receive the data and i have no clue why. | 18:39 |
| noonedeadpunk | ah, it's probably not backup but more initial sst sync | 18:40 |
| noonedeadpunk | yeah, makes sense then | 18:40 |
| noonedeadpunk | or well | 18:40 |
| noonedeadpunk | not failure, but that we actually don't run backups and I misread logs :) | 18:41 |
| noonedeadpunk | I really hope that upgrade jobs won't fail now.... | 18:41 |
| mgariepy | :/ https://zuul.opendev.org/t/openstack/build/be455b6fd24744e283d79e95e0122fb0/log/job-output.txt#13036 | 19:06 |
| mgariepy | what to i miss ? | 19:07 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/846046 | 19:24 |
| jrosser_ | ^ mgariepy | 19:24 |
| mgariepy | ho lol | 19:26 |
| mgariepy | i should not get up at 5 am too often :/ | 19:27 |
| jrosser_ | looks like openstacksdk 0.99.0 is now packaged in homebrew which will be fun for anyone using that on a mac to do ansible | 19:46 |
| opendevreview | Merged openstack/openstack-ansible-os_placement master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/845708 | 19:58 |
| mgariepy | network manager seems to be still causing issues for ironic :/ | 20:04 |
| mgariepy | on centso 9 stream | 20:06 |
| spatel | I am playing with ceph on single node, to mimic disk i did create loopdevice and put them in LVM so i can create lv but somehow when i run ceph-ansible it print device like this. where that dm-2 and dm-3 coming from? - https://paste.opendev.org/show/bEAp9gVnSF0AGxSZs4dW/ | 20:13 |
| spatel | that dm-2 and dm-3 causing error | 20:13 |
| jrosser_ | spatel: have you looked at how the OSA all-in-one sets up loopback devices for ceph? | 20:19 |
| spatel | not yet.. i may look at it | 20:19 |
| noonedeadpunk | doh.... | 20:22 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: Disable network manager on centos 9. https://review.opendev.org/c/openstack/openstack-ansible/+/846224 | 20:25 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: Stop NetworkManager on centos 9. https://review.opendev.org/c/openstack/openstack-ansible/+/846224 | 20:25 |
| noonedeadpunk | don't we delete networkmanager somewhere for centos 8? | 20:29 |
| noonedeadpunk | likely not... | 20:30 |
| mgariepy | i don't think so | 20:30 |
| mgariepy | it might fix ironic on centos 9 .. | 20:30 |
| mgariepy | consistency .. https://zuul.opendev.org/t/openstack/build/f46549448c3847cc9b1d8af18ff604bd/logs https://zuul.opendev.org/t/openstack/build/cbacc51ccfc348358061c9f4f8366b9a | 20:33 |
| noonedeadpunk | mgariepy: well, https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/846033 passed ? | 20:34 |
| noonedeadpunk | wonder how.... | 20:34 |
| mgariepy | maybe we should add 846224 as depend .. to see. | 20:35 |
| noonedeadpunk | yeah, I think it's good idea | 20:36 |
| noonedeadpunk | will you or should I? | 20:43 |
| mgariepy | do it | 20:44 |
| mgariepy | please :D | 20:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Allow redhat vars file to cover different RHEL derivatives https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/844021 | 20:44 |
| mgariepy | i think it's a race issue with the dhcp lease not being active when the pkg gets installed. | 20:45 |
| mgariepy | that make it passe. | 20:45 |
| noonedeadpunk | yeah might be... | 20:45 |
| mgariepy | anyway, see you tomorrow | 20:45 |
| mgariepy | my wife wants me to stop working haha | 20:46 |
| noonedeadpunk | yup:) have a good day :) | 20:46 |
| mgariepy | cya | 20:46 |
| noonedeadpunk | I'm in exact same situation hehe | 20:46 |
| jrosser_ | i did the directory creation inside the loop on purpose in here https://review.opendev.org/c/openstack/openstack-ansible/+/846189 | 20:49 |
| jrosser_ | agree it's ugly though | 20:49 |
| noonedeadpunk | should I return it back? | 20:50 |
| noonedeadpunk | I just tried to reduce actions inside loop for each log line... | 20:50 |
| noonedeadpunk | And indent was wrong there so... | 20:50 |
| noonedeadpunk | As I kind of didn't get purpose, except avoid directory creation if no deperecations | 20:51 |
| jrosser_ | yeah the logic is broken | 20:51 |
| jrosser_ | otherwise we end up with lots and lots of empty dirs to search through | 20:51 |
| jrosser_ | originally the only dirs that existed had actual deprectaions in them | 20:52 |
| noonedeadpunk | Ok, yes, I guess I just took metal as example... | 20:52 |
| noonedeadpunk | ANd based on that thought that meh... | 20:52 |
| jrosser_ | maybe we can fix that seperatley with how you also used the fd dict | 20:52 |
| noonedeadpunk | like drop empty dirs afterwards? | 20:53 |
| noonedeadpunk | I think it's doable relatively easily | 20:55 |
| noonedeadpunk | btw looking at result of https://zuul.opendev.org/t/openstack/build/4faf6a3dd1bb4aaaa572c96a54986337/logs I kind of feel that lxc containers are missing from run at all... | 21:00 |
| noonedeadpunk | or they're just not collected... | 21:00 |
| noonedeadpunk | ah, it's just not top level | 21:01 |
| *** dviroel is now known as dviroel|biab | 21:07 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix deprecations handling in journal logs https://review.opendev.org/c/openstack/openstack-ansible/+/846189 | 21:27 |
| noonedeadpunk | I made some weird thing.... | 21:28 |
| noonedeadpunk | but it worked in sandbox | 21:29 |
| jrosser_ | still some oom in telemetry jobs | 21:34 |
| noonedeadpunk | yeah | 21:34 |
| noonedeadpunk | well. telemetry was always tough to pass because of ooms | 21:35 |
| noonedeadpunk | And I guess we could miss some worker setting for smth | 21:35 |
| jrosser_ | more things related to memory going on too https://review.opendev.org/c/openstack/nova/+/846208 | 21:36 |
| noonedeadpunk | I'm kind of worried about `disabling the per thread arenas` tbh | 21:40 |
| noonedeadpunk | `Choosing a value MALLOC_ARENA_MAX is generally a tradeoff between performance and memory consumption.` | 21:43 |
| noonedeadpunk | These folks even did some benchmarking https://devcenter.heroku.com/articles/testing-cedar-14-memory-use | 21:44 |
| jrosser_ | i asked someone here about this who knows more python internals than me | 21:45 |
| jrosser_ | and they throught it wasnt so bad as there was not really proper threading | 21:46 |
| jrosser_ | unless you call out to some external library (numpy being good eample) which might have its own ideas about threads | 21:46 |
| noonedeadpunk | I think we should totally try it out in CI... | 21:47 |
| noonedeadpunk | And for prod I guess some rally will be helpfull to understand | 21:47 |
| jrosser_ | right - i'm done for today | 21:48 |
| jrosser_ | youve done some great debugging / fixing today btw | 21:48 |
| noonedeadpunk | I guess it's only env vars that could be set... | 21:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Reduce memory consumption in CI https://review.opendev.org/c/openstack/openstack-ansible/+/846228 | 21:58 |
| noonedeadpunk | let's see... Hope proxy would work... | 21:59 |
| jrosser_ | this may be similar to the ca certs stuff where I think we ended up having to do things in several places | 22:02 |
| jrosser_ | like environment, uwsgi and systemd iirc | 22:03 |
| jrosser_ | it was useful to look in /proc at the process environments to see if any of it was present in the right way | 22:06 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Wait for all volumes when starting glusterd.service https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845987 | 22:28 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Support service tokens https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/846040 | 23:02 |
| *** dviroel|biab is now known as dviroel | 23:07 | |
| *** dviroel is now known as dviroel|out | 23:16 | |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Remove mention of haproxy-endpoints role https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/845801 | 23:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!