*** ysandeep|out is now known as ysandeep|ruck | 04:02 | |
*** ysandeep|ruck is now known as ysandeep|ruck|afk | 04:38 | |
*** ysandeep|ruck|afk is now known as ysandeep|ruck | 05:08 | |
*** ysandeep|ruck is now known as ysandeep|ruck|afk | 06:18 | |
noonedeadpunk | ianw: that is neat, thanks! | 06:40 |
---|---|---|
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development https://review.opendev.org/c/openstack/openstack-ansible/+/847272 | 06:43 |
noonedeadpunk | jrosser_: btw I believe we have circular dependency here https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/848589 | 06:44 |
jrosser_ | noonedeadpunk: yeah that looks unfortunate | 06:57 |
jrosser_ | maybe we just remove all the jobs from os_horizon in the same patch and merge 848589 ASAP | 06:57 |
noonedeadpunk | well, yes, I guess so. But for 847272 there's some weird issue with distro installs at the very beginning. Which sounds more like some module issue tbh... Which is weird, as with that still Yoga packages are being used | 07:02 |
jrosser_ | "msg": "Invalid query params: region_id" | 07:04 |
jrosser_ | well - i was thinking for the distro jobs where does the openstacksdk come from | 07:06 |
jrosser_ | here https://github.com/openstack/openstack-ansible/blob/master/playbooks/utility-install.yml#L82-L96 | 07:07 |
jrosser_ | hmm python3-openstacksdk:amd64 (0.61.0-0ubuntu1~cloud1, automatic) | 07:08 |
*** ysandeep|ruck|afk is now known as ysandeep|ruck | 07:11 | |
noonedeadpunk | but utility should be setup with distro packages as well then.... but for this task I'm not sure it's utility version we should check or ansible venv? | 07:12 |
noonedeadpunk | but yeah, utility I guess indeed | 07:12 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Temporary disable horizon functional jobs https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/848693 | 07:14 |
noonedeadpunk | oh! I think it's because I set ansible-collections-openstack version to master | 07:15 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development https://review.opendev.org/c/openstack/openstack-ansible/+/847272 | 07:15 |
noonedeadpunk | But then likely everything else would fail | 07:16 |
noonedeadpunk | let's see | 07:16 |
jrosser_ | thats odd why only the distro job fails | 07:17 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Release pyscss constraints https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/848589 | 07:17 |
jrosser_ | i think the upgrade jobs were failing rally so we might have also something separate there to look at | 07:22 |
noonedeadpunk | upgrade jobs were failing in other place which I covered I believe | 07:26 |
noonedeadpunk | as they were trying to upgrade from Xena | 07:26 |
noonedeadpunk | or well, at least centos 9 was failing there | 07:28 |
noonedeadpunk | I was also wondering what are concerns regarding https://review.opendev.org/c/openstack/ansible-config_template/+/846391 ? | 07:29 |
jrosser_ | concerns? you mean why it's not merged yet? | 07:34 |
noonedeadpunk | Well, and no votes or comments:) | 07:35 |
noonedeadpunk | as we were discussing it several times I guess and there was no objections from anybody but no reviews either | 07:36 |
jrosser_ | oh well i think i concentrated on getting through the POST_FAILURES stuff and ignored everything that wasn't that or release patches | 07:36 |
noonedeadpunk | that's totally fair | 07:36 |
noonedeadpunk | I just started looking through review board and noticed it :p | 07:37 |
noonedeadpunk | can't say it's the only thing I noticed, but well | 07:37 |
noonedeadpunk | to be completely fair, I don't know if it will "just work" and dunno how to check except try out | 07:38 |
*** ysandeep|ruck is now known as ysandeep|lunch | 08:15 | |
damiandabrowski[m] | noonedeadpunk: I've found one disadvantage of making `tempest_plugins` a list of dicts. | 09:23 |
damiandabrowski[m] | when tempest_plugins is a dict, we can use combine() filter for tempest_extra_plugins which allows to override specific plugins. | 09:23 |
damiandabrowski[m] | when tempest_plugins is a list, we can just concatenate 2 lists together but it's not that smart as combine() | 09:23 |
damiandabrowski[m] | do we still want to make it a list? | 09:23 |
opendevreview | Merged openstack/openstack-ansible-os_horizon master: Release pyscss constraints https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/848589 | 09:36 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Return zuul integrated jobs https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/848732 | 09:47 |
noonedeadpunk | damiandabrowski[m]: well, it's fair point | 09:47 |
noonedeadpunk | but looking at _tempest_plugins seems there's nothing not configurable in it... | 09:48 |
damiandabrowski[m] | repo url / branch :D | 09:49 |
noonedeadpunk | but these are variables? | 09:49 |
noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/vars/main.yml#L56-L57 | 09:49 |
damiandabrowski[m] | ah, You're right | 09:50 |
noonedeadpunk | so if you want to change behaviour of anything that's pre-defined, you don't actually need to combine, as you have variables in place kind of... | 09:50 |
damiandabrowski[m] | ack | 09:52 |
noonedeadpunk | but dunno, maybe you're right. it's just a bit pita to deal with dicts when you actually don't need them | 09:52 |
noonedeadpunk | jrosser_: yeah, with old collection the only thing that is passing is distro job lol | 09:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development https://review.opendev.org/c/openstack/openstack-ansible/+/847272 | 09:56 |
*** ysandeep|lunch is now known as ysandeep|ruck | 10:16 | |
damiandabrowski[m] | do You think we can finally remove 'tests/' directory in openstack-ansible-os_* repos? I wonder what else should be cleaned up | 10:36 |
*** arxcruz is now known as arxcruz|brb | 10:59 | |
noonedeadpunk | I can't recall if we need that for docs/releasenotes? I guess not? | 11:03 |
damiandabrowski[m] | i think we don't | 11:11 |
jrosser_ | the only places that would need to keep it would be where we still have functional tests | 11:13 |
jrosser_ | and thats not very many at all now, probably non of the os_ roles | 11:13 |
noonedeadpunk | for sure we have that for config_template,systemd, qrouterd and maybe smth else | 11:14 |
opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_tempest master: Make _tempest_plugins a list of dicts. https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/848642 | 11:23 |
*** dviroel|biab is now known as dviroel | 11:29 | |
opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_tempest master: Allow to define extra tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/848643 | 11:30 |
damiandabrowski[m] | okok thanks | 11:34 |
noonedeadpunk | damiandabrowski[m]: with removing tests folder we should also cleanup tox.ini and drop run_tests.sh | 11:44 |
noonedeadpunk | I think it's also time to deprecate rsyslog roles | 11:45 |
damiandabrowski[m] | ack | 11:55 |
*** arxcruz|brb is now known as arxcruz | 12:03 | |
*** ysandeep|ruck is now known as ysandeep|brb | 12:52 | |
*** ysandeep|brb is now known as ysandeep|ruck | 13:01 | |
jrosser_ | damiandabrowski[m]: i think there is already a patch to os_ironic for a big cleanup to take as a starting point | 13:25 |
damiandabrowski[m] | ah this one? https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/824214 | 13:27 |
damiandabrowski[m] | thanks, i was just trying to figure out how everything works :D | 13:27 |
spatel | jamesdenton morning! | 13:55 |
spatel | In ovn multi-node deployment i have noticed our controller node doesn't install openvswitch so it doesn't act as network gateway for deployment. | 13:57 |
jamesdenton | mornin | 13:58 |
spatel | i have deployed multi-node ovn and noticed one of compute node is gw | 13:58 |
jamesdenton | there may be something in the playbooks setting 'chassis-is-gw' or whatever that is | 13:59 |
jamesdenton | https://github.com/openstack/openstack-ansible-os_neutron/blob/bbdd12561f4e5800e6504d14a7810fad8631a42b/tasks/providers/setup_ovs_ovn.yml#L23-L27 | 13:59 |
spatel | Yes chassis-is-gw tell you which node will be gateway but in OSA deployment we don't install openvswitch on infra node so how can infra node will be gateway? | 14:00 |
spatel | We need to install neutron-ovn-controller on infra nodes to deploy OVS switch | 14:01 |
spatel | am i missing something here? | 14:01 |
jamesdenton | https://github.com/openstack/openstack-ansible-os_neutron/blob/6c722ead673f1c80aee4f439bf5f6a41841e85a5/vars/distro_install.yml#L46-L48 | 14:03 |
jamesdenton | the controller needs to be a member of one of those groups | 14:03 |
jamesdenton | errr, one sec | 14:04 |
spatel | take your time because i need coffee :) | 14:05 |
spatel | all i can see we need to install neutron-ovn-controller on infra metal box which will make that node part of SDN and then you can set chassis-is-gw to route traffic. | 14:06 |
spatel | I have deployed 3 (infra) and 2 (computes) nodes deployment and found only compute nodes act as gateway. controller nodes doesn't have any OVN information to be part of SDN. | 14:08 |
jamesdenton | did you manipulate the env.d files as part of your deployment? | 14:10 |
jamesdenton | i think the 'ref arch' only has computes in the ovn controller role: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/env.d/nova_ovn.yml.example#L10 | 14:10 |
jamesdenton | and by ref arch, i mean the OSA arch | 14:10 |
jamesdenton | but that's definitely tunable | 14:11 |
spatel | yes i did env.d stuff | 14:11 |
spatel | what i need to do to deploy ovn-controller on infra nodes? | 14:11 |
spatel | we should make that deployment default instead compute nodes act as a gateway. (Let user decide if they want distributes FIP or not) | 14:12 |
mgariepy | with ovn it's kinda free tho. | 14:13 |
spatel | infra nodes should be gateway by default for ovn deployment in my opinion | 14:13 |
mgariepy | and nova-compute shouldn't? | 14:14 |
jamesdenton | you'll need to update the env.d files to make sure the infra nodes are members of the neutron_ovn_controller group; i don't have an example handy | 14:14 |
jamesdenton | i agree there needs to be a more graceful way to determine what is or isn't a gateway node | 14:16 |
spatel | nova-compute can be but let user decide what they want (everything is tunable in ovn but better out of box when someone deploy cloud infra should be gateway and later you can decide if you want compute to be gateway | 14:17 |
spatel | In OVN you have priority option to tell who will be gateway when you have multiple gateway so we can set infra node with high priority bit. | 14:19 |
jamesdenton | the original intent was to mirror DVR, where the compute is gateway, but reality is i don't really care. We just need to remove hardcoding from the playbooks to ensure we don't overwrite anything the user has done. So, that may mean not setting any gateway and leaving that to admin guide | 14:19 |
mgariepy | the admin guide might need some updating. | 14:20 |
jamesdenton | ^^ | 14:20 |
mgariepy | https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#optional-dvr-or-distributed-l3-routing | 14:20 |
spatel | we need more people to use OVN so we get better feedback :) | 14:21 |
spatel | I have started playing with ovn and my goal is to make it better with OSA | 14:22 |
anskiy | i don't think making infra node act as gateway by default is good idea: that's mixing control and data planes | 14:22 |
spatel | I have started building ovn-bgp-agent lab so hope we have that piece available in OSA | 14:22 |
mgariepy | if you have 2 vms with fips on 2 compute node, do they all go through the same compute host gw or they are each hosted on their compute node? | 14:23 |
spatel | I am looking at current deployment of osa (LXD based) where infra nodes are gateway. (Yes you can do whatever you want later) | 14:24 |
spatel | mgariepy i have create blog here on OVN non-distributed vs distributed FIPs - https://satishdotpatel.github.io/openstack-ansible-ovn-deployment-part2/ | 14:25 |
spatel | OVN is best solution for DVR style deployment and freaking easy. | 14:26 |
mgariepy | i guess ovn is the way to go now | 14:27 |
spatel | indeed, no need to worry about keepalived/namespace etc.. :) | 14:28 |
mgariepy | i have an ovs deploy that needs to be migrated to ovn. | 14:28 |
spatel | I am building FRR lab to test ovn-bgp-agent deployment to see how good it can fit with osa | 14:30 |
mgariepy | did you assist the talk about the migration on tripleO at the summit? | 14:38 |
*** ysandeep|ruck is now known as ysandeep|out | 14:39 | |
mgariepy | ho. just remembered while watching the talk you did told me already :) sorry | 14:44 |
noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:03 |
opendevmeet | Meeting started Tue Jul 5 15:03:30 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:03 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:03 |
opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:03 |
noonedeadpunk | #topic rollcall | 15:03 |
noonedeadpunk | hey o/ | 15:03 |
damiandabrowski[m] | hi! | 15:03 |
NeilHanlon | hiya! | 15:03 |
mgariepy | hey ! | 15:03 |
noonedeadpunk | #topic office hours | 15:06 |
noonedeadpunk | ok, so seems all our branches are now blocked because of rally | 15:06 |
*** dviroel is now known as dviroel|lunch | 15:07 | |
damiandabrowski[m] | :( | 15:08 |
noonedeadpunk | basically this is the fix https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848666 but upgrade jobs needs to be fixed first | 15:08 |
noonedeadpunk | And not sure how long we should backport it | 15:08 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally stable/victoria: Control rally-openstack installed version https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848718 | 15:08 |
noonedeadpunk | eventually I wonder, if we should not install rally in first run of upgrade jobs | 15:10 |
noonedeadpunk | as we don't run tempest | 15:11 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not install rally on first upgrade run https://review.opendev.org/c/openstack/openstack-ansible/+/848760 | 15:13 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Control rally-openstack installed version https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848666 | 15:14 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development https://review.opendev.org/c/openstack/openstack-ansible/+/847272 | 15:14 |
noonedeadpunk | I wonder if that would be enough to fix the issue:) | 15:14 |
noonedeadpunk | Other then that - stable 25.0.0 tag was pushed on 1st of July, but I think it's still hasn't been merged | 15:15 |
noonedeadpunk | I've already reminded release team about it | 15:15 |
damiandabrowski[m] | fingers crossed | 15:16 |
noonedeadpunk | then for master we had to set distro jobs to NV for now. As with openstacksdk 0.99 which is set in u-c we should use master ansible collections. Which does not work with older sdk.... | 15:17 |
noonedeadpunk | I expect things to be fixed only when packages for Z will be published and it will take a while | 15:18 |
noonedeadpunk | also jrosser_did good job on updating our grafana board https://grafana.opendev.org/d/8cbc9d7032/openstack-ansible?orgId=1&from=now-90d&to=now | 15:19 |
noonedeadpunk | which gives good idea about our CI state | 15:20 |
noonedeadpunk | I think we also can now try out shared queus in zuul. wdyt? | 15:20 |
damiandabrowski[m] | awesome, thanks jrosser! | 15:21 |
damiandabrowski[m] | not sure what prevented us to do that before, but I don't have any objections :D | 15:21 |
noonedeadpunk | well, I think if any task that is in gate fails, it leads to failure of all rest patches | 15:22 |
noonedeadpunk | if I got that correctly | 15:22 |
noonedeadpunk | and that sounds quite troublesome for me | 15:22 |
damiandabrowski[m] | ouh, yeah :/ | 15:23 |
noonedeadpunk | But I might be wrong in my understanding of the concept and how it works... | 15:23 |
noonedeadpunk | but rechecking each time when +w is set in incorrect order also nasty... | 15:24 |
jrosser_ | o/ hello | 15:31 |
* jrosser_ late | 15:31 | |
noonedeadpunk | I'd say in time:) | 15:32 |
NeilHanlon | Rocky 9 should be coming out "soon", so i'll probably start on that this week | 15:58 |
noonedeadpunk | that are good news! | 16:05 |
noonedeadpunk | just let us know where we can help | 16:06 |
noonedeadpunk | #endmeeting | 16:06 |
opendevmeet | Meeting ended Tue Jul 5 16:06:04 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:06 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-07-05-15.03.html | 16:06 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-07-05-15.03.txt | 16:06 |
opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-07-05-15.03.log.html | 16:06 |
*** dviroel|lunch is now known as dviroel | 16:22 | |
jrosser_ | has anyone ever implemented a plugin (like a keystone auth plugin) and have boilerplate for making all the stevedore plugin loading stuff work | 16:27 |
jrosser_ | RITICAL keystone [-] Unhandled error: stevedore.exception.NoMatches: No 'keystone.auth.password' driver found, looking for <my-plugin....> | 16:27 |
noonedeadpunk | well, I used to debug stevedore plugins behaviour in barbican. And the tricky thing there for me was how to discover plugin name. As it's not path to the file or class, it's another thing, so I had to print list of all found plugins to realize, that plugin I'm looking for named quite differently from what I expected or was documented | 16:38 |
noonedeadpunk | jrosser_: btw yeah, in https://review.opendev.org/c/openstack/openstack-ansible/+/848659 it's worth using haproxy_allowlist_networks | 16:43 |
jrosser_ | right i think i'm gettig mixed up with plugin names / namespaces / other things | 16:43 |
noonedeadpunk | or well, create haproxy_stick_table_allowlist_networks: "{{ haproxy_allowlist_networks }}" | 16:43 |
noonedeadpunk | it's super unobvious in stvedore for me. Even when I found actual name I failed to understand where it comes from.... | 16:44 |
noonedeadpunk | I commented in task. Join looks nice in test playbook - get result "http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 } !{ src 192.168.0.0/16 } !{ src 172.16.0.0/12 } !{ src 10.0.0.0/8 }" for multiple networks and "http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 } !{ src 192.168.0.0/16 }" for single one | 16:53 |
jrosser_ | noonedeadpunk: this was helpful https://pypi.org/project/entry-point-inspector/ | 17:17 |
jrosser_ | i made it work but omg the documentation is just useless | 17:17 |
jrosser_ | or the whole thing is so abstract it's impossible to get a grip on actually whats happening | 17:17 |
NeilHanlon | what in the world is a stevedore | 17:49 |
NeilHanlon | lol | 17:49 |
noonedeadpunk | well, I just putting print() somewhere in the code, as I didn't know about that tool, so.... | 18:59 |
noonedeadpunk | *I was just | 18:59 |
noonedeadpunk | not sure whos way was more challenging though hehe | 19:00 |
noonedeadpunk | doh, we have another circular dependency | 19:15 |
noonedeadpunk | for https://review.opendev.org/c/openstack/openstack-ansible/+/848760 and https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848666 | 19:16 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally stable/wallaby: Control rally-openstack installed version https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848721 | 19:18 |
noonedeadpunk | pff, and for backport since victoria it's erlang repo that is broken for bionic | 19:19 |
noonedeadpunk | that is annoying I must say | 19:24 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Control rally-openstack installed version https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848666 | 19:27 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not install rally on first upgrade run https://review.opendev.org/c/openstack/openstack-ansible/+/848760 | 19:29 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development https://review.opendev.org/c/openstack/openstack-ansible/+/847272 | 19:30 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Return upgrade jobs to voting https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848778 | 19:30 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development https://review.opendev.org/c/openstack/openstack-ansible/+/847272 | 19:31 |
noonedeadpunk | and we need to do the same for each damn branch.... | 19:33 |
spatel | I have haproxy related question - https://paste.opendev.org/show/bhM1FR1Y0fzOyaSPexiN/ | 20:04 |
jrosser_ | spatel: hit 35k what? | 20:36 |
spatel | we have loadtest tool develop for our customer application benchmarking, i am generating load to see how far i can go with haproxy | 20:37 |
jrosser_ | oh sure, but 35k bytes / kilograms / req/sec / concurrent clients ? :) | 20:38 |
spatel | I have 5 backend servers and 1 source ( 55k per backend server that means i can hit 250k concurrent connection but look like getting error at 35k related running out of ports) | 20:43 |
jrosser_ | i am sure we have talked about this before | 20:44 |
mgariepy | what are the ulimit for your haproxy ? | 20:45 |
mgariepy | https://www.freecodecamp.org/news/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27/ | 20:45 |
mgariepy | pretty sure haproxy can scale a lot. but it will need to have the correct config/tuning and so on. | 20:46 |
spatel | I have 10million | 20:48 |
spatel | I have 10million (ulimit + openfile + whatever tuning required by google search) | 20:48 |
spatel | If i add more IP in source then i can pass 50k without issue. by my math saying with single IP i should be able to do that | 20:49 |
spatel | I have 5 backend servers that means enough socket:ports available | 20:50 |
*** dviroel is now known as dviroel|afk | 21:59 | |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_rally master: Control rally-openstack installed version https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848666 | 22:03 |
*** dviroel|afk is now known as dviroel | 23:27 | |
*** dviroel is now known as dviroel|out | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!