| frickler | oh, interesting, so octavia is doing their own BGP setup instead of integrating with neutron? | 07:08 |
|---|---|---|
| grami[m] | Would that mean anycast could be an option in octavia to span over data centers? that would be a huge bonus for octavia. | 07:31 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/xena: Restart rsyslog from cloud-init https://review.opendev.org/c/openstack/octavia/+/837547 | 10:49 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Restart rsyslog from cloud-init https://review.opendev.org/c/openstack/octavia/+/837548 | 10:50 |
| opendevreview | Merged openstack/octavia stable/wallaby: Add fips jobs https://review.opendev.org/c/openstack/octavia/+/827900 | 11:25 |
| opendevreview | Merged openstack/octavia stable/wallaby: Use centos amphora image in the FIPS jobs https://review.opendev.org/c/openstack/octavia/+/831767 | 11:25 |
| johnsom | frickler there will still be integration points with neutron for ports and such, but the idea is to make the amphorae BGP participants | 13:22 |
| johnsom | grami[m] yes, it helps solve the cross AZ/datacenter HA issue if there is no shared L2 for the VIP. | 13:24 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Testing c9s job with new ssh key type https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/837597 | 16:38 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Testing c9s job with new ssh key type https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/837597 | 16:46 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Testing c9s job with new ssh key type https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/837597 | 17:35 |
| rm_work | johnsom: ever noticed a DB-side issue with members where they seem to be "fine" but the driver code can't convert the model to the provider dict? | 19:22 |
| rm_work | File "/opt/openstack/venv/octavia/lib/python3.7/site-packages/octavia/api/drivers/utils.py", line 445, in db_member_to_provider_member | 19:22 |
| rm_work | new_member_dict = member_dict_to_provider_dict(db_member.to_dict()) | 19:22 |
| rm_work | AttributeError: 'NoneType' object has no attribute 'to_dict' | 19:22 |
| rm_work | very interesting, looking for root cause now | 19:23 |
| rm_work | implies the db_member never came back from the DB but it is obviously THERE | 19:23 |
| johnsom | No, doesn't ring a bell for me. What action was going on? | 19:24 |
| rm_work | looks like just a pool GET O_o | 19:30 |
| rm_work | working on repro | 19:30 |
| opendevreview | Merged openstack/octavia stable/xena: Fix nft command line with negative priority values https://review.opendev.org/c/openstack/octavia/+/820536 | 19:31 |
| opendevreview | Merged openstack/octavia stable/xena: Add keepalive for redis-based taskflow boards https://review.opendev.org/c/openstack/octavia/+/825944 | 19:31 |
| johnsom | That doesn't make any sense an GET call won't call into the drivers, so wouldn't convert | 19:32 |
| rm_work | hmmm | 19:33 |
| rm_work | yeah looking at fully verifying, had ops folks forwarding me logs | 19:33 |
| gthiemonge | rm_work: we have this open story: https://storyboard.openstack.org/#!/story/2009128 | 19:44 |
| rm_work | ahhh interesting | 19:53 |
| rm_work | that could be what's happening | 19:53 |
| rm_work | that is the exact stack trace | 19:54 |
| rm_work | thanks gthiemonge I am much less confused now | 19:54 |
| rm_work | maybe can help you on the repro/debugging? | 19:54 |
| rm_work | though we're running... a shamefully old version I thinik | 19:55 |
| johnsom | A patch would be even more help. grin | 19:55 |
| rm_work | *I think, maybe ... Xena or something 😕 | 19:55 |
| rm_work | heh | 19:55 |
| rm_work | well can definitely replicate, will try to work out the bug | 20:22 |
| rm_work | i wonder if it's related to SQLAlchemy changes | 20:27 |
| rm_work | because it seems like the duplication must not be causing an exception now? | 20:27 |
| rm_work | oh nm i see | 20:41 |
| rm_work | de.columns is: ['member.uq_member_pool_id_address_protocol_port'] | 20:43 |
| rm_work | oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'e046177a-13b3-4321-810b-5f8fdda1194a-10.224.108.144-4443' for key 'member.uq_member_pool_id_address_protocol_port'") | 20:43 |
| rm_work | constraint_list = ['uq_member_pool_id_address_protocol_port'] | 20:44 |
| rm_work | de.columns = ['member.uq_member_pool_id_address_protocol_port'] | 20:44 |
| rm_work | I think that's a change in how SQLAlchemy returns the error | 20:44 |
| rm_work | it's including "member." | 20:44 |
| rm_work | updating that fixes the problem | 20:45 |
| rm_work | I wonder how many other places we're going to run into this -- and if the solution is to change the contraints string, or if there's some SQLAlchemy tweak | 20:46 |
| rm_work | johnsom / gthiemonge ^^ | 20:46 |
| johnsom | rm_work code link? | 20:47 |
| rm_work | https://github.com/openstack/octavia/blob/4a9de5b27a33a5521d0e0a1fdc4c63eb439d8149/octavia/api/v2/controllers/member.py#L124-L128 | 20:48 |
| johnsom | Thanks, context switching is hard. grin | 20:48 |
| rm_work | so we check if the de.columns that comes back is either column_list or constraint_list | 20:48 |
| rm_work | what comes back is 'member.uq_member_pool_id_address_protocol_port' | 20:49 |
| rm_work | but we are looking for 'member.uq_member_pool_id_address_protocol_port' | 20:49 |
| rm_work | err | 20:49 |
| rm_work | uq_member_pool_id_address_protocol_port' | 20:49 |
| rm_work | both mean the same thing but I think SQLAlchemy changed how they return that error | 20:49 |
| johnsom | Why do we even care? Why not just catch the DBDuplicateEntry and bounce a DuplicateMemberEntry | 20:50 |
| rm_work | I remember us making changes previously to deal with this, possibly in the other direction | 20:50 |
| rm_work | I think there could be some other cause? | 20:50 |
| johnsom | rm_work It seems like we are trying to be too smart with little to no value here | 20:52 |
| rm_work | I SWEAR I remember a CR up for changing this stuff | 20:55 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/listener.py#L336 | 20:55 |
| rm_work | wtf was it | 20:55 |
| rm_work | was it removing these from other places? | 20:55 |
| johnsom | Yeah, this rings a bell for me too | 20:55 |
| johnsom | Those are the only two places I see currently | 20:56 |
| rm_work | https://github.com/openstack/octavia/commit/44fb49b1286f589974c443393a05302604fdc525 | 20:56 |
| johnsom | lol, so, yeah | 20:57 |
| rm_work | Might have been an attempt to return a more useful error message? as it's not ALWAYS the ID | 20:58 |
| johnsom | rm_work I can push a patch for this if you are not already doing it | 20:58 |
| rm_work | obviously | 20:58 |
| rm_work | see the error we return is more useful | 20:58 |
| rm_work | `Another member on this pool is already using ip 10.224.108.144 on protocol_port 4443 (HTTP 409) (Request-ID: req-67c1aba7-9ada-45e3-b1b7-b5202fdab498)` | 20:58 |
| rm_work | instead it will return ... ??? something worse | 20:58 |
| johnsom | Yeah, but we can still return that info without trying to look into the exception raised from sqlalchemy | 20:58 |
| rm_work | differentiating from ID? | 20:59 |
| rm_work | could leave the "id" comparison and just else it | 20:59 |
| rm_work | rather than elif | 20:59 |
| johnsom | Who cares if it's ID or what column, it's still member with IP x and Port Y | 20:59 |
| rm_work | hmm well | 20:59 |
| rm_work | ok yeah I was expecting we use `IDAlreadyExists` for checking | 21:00 |
| rm_work | but we don't | 21:00 |
| johnsom | Plus, users can't specify their IDs, so.... | 21:00 |
| rm_work | it must only bubble up | 21:00 |
| johnsom | It would only matter if we added the backup/restore and allowed preserving the IDs | 21:01 |
| johnsom | An nothing catches that IDAlreadyExists | 21:01 |
| johnsom | Just bounces it to the user | 21:01 |
| opendevreview | Michael Johnson proposed openstack/octavia master: Fix duplicate object error messages https://review.opendev.org/c/openstack/octavia/+/837627 | 21:13 |
| johnsom | rm_work ^^^ | 21:13 |
| johnsom | Guess I should have run the tests to see if we were testing that stuff | 21:14 |
| opendevreview | Michael Johnson proposed openstack/octavia master: Fix duplicate object error messages https://review.opendev.org/c/openstack/octavia/+/837627 | 21:22 |
| johnsom | No test coverage, but an un-used variable pep8, so fixed that. | 21:22 |
| rm_work | yeah ok that's ... fair I think. the user error would need to be that anyway | 21:26 |
| johnsom | Seems to be some devstack borkage | 21:43 |
| johnsom | Or keystone. devstack is blowing up setting up keystone | 21:44 |
| johnsom | Hmm, no, pbr maybe. the other job failed installing bbq-client | 21:45 |
| opendevreview | Michael Johnson proposed openstack/octavia master: Fix duplicate object error messages https://review.opendev.org/c/openstack/octavia/+/837627 | 21:50 |
| johnsom | Yeah, everything is borked. lots-o-red on the zuul board | 21:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!