Thursday, 2021-01-28

openstackgerritBrian Haley proposed openstack/octavia-tempest-plugin master: Use py3.5 compatible tempest
haleybrm_work: as far as gates, the tempest ones are borked01:03
rm_workand you're working it?01:03
haleybpart was due to py2 going away, fixes merged for devstack but it's still failing, ^^ was just a random guess at debugging, but could be a dib issue?01:04
haleybmy middle name has been "gate failure"01:05
johnsomDo you have a link to a failed job?01:05
haleyb is barfing on it's feet01:05
haleybthe one-line change i did01:05
haleyball the scenario jobs are failing in all branches :(01:06
johnsomHmm, in that last zuul, one test failed, that was all01:06
rm_workhmm k01:06
haleybthe train and stein ones couldn't get a running devstack01:06
johnsomUssuri, the pip package DIB is installing is broken01:09
haleybjohnsom: so i guess i was on the right track with my last comment, i don't know how to fix that one01:10
*** sapd__x has joined #openstack-lbaas01:12
johnsomhaleyb Looks like you don't have to:
haleybYay!  just don't tell me that needs to go through the dib stable branches01:15
johnsomDIB is branchless01:15
johnsomWhich causes it's own problems, but...01:15
*** priteau has quit IRC01:16
haleybif only we knew a dib core...01:16
rm_workbut WHO could help us01:17
haleybi think the WHO is busy with COVID :-p01:18
* haleyb bugs out01:19
* johnsom is barely paying any attention as he is fighting ansible demons01:20
*** sapd__x has quit IRC02:07
*** armax has quit IRC02:09
*** nicolasbock_ has joined #openstack-lbaas02:21
*** nicolasbock has quit IRC02:21
*** irclogbot_2 has quit IRC02:21
*** nicolasbock_ is now known as nicolasbock02:21
*** irclogbot_3 has joined #openstack-lbaas02:23
*** rcernin has quit IRC02:34
*** rcernin has joined #openstack-lbaas02:36
openstackgerritBrian Haley proposed openstack/octavia-tempest-plugin master: DNM - testing if DIB change fixes gate
*** armax has joined #openstack-lbaas03:00
*** armax has quit IRC03:11
*** psachin has joined #openstack-lbaas03:37
*** vishalmanchanda has joined #openstack-lbaas04:22
*** lemko7 has joined #openstack-lbaas04:57
*** lemko has quit IRC04:58
*** lemko7 is now known as lemko04:58
*** lemko2 has joined #openstack-lbaas05:11
*** lemko has quit IRC05:12
*** lemko2 is now known as lemko05:12
*** rcernin has quit IRC06:18
*** rcernin has joined #openstack-lbaas06:22
*** gthiemonge has joined #openstack-lbaas06:30
*** gthiemonge has quit IRC06:31
*** gthiemonge has joined #openstack-lbaas06:31
*** gcheresh has joined #openstack-lbaas06:48
*** haleyb has quit IRC06:53
*** ccamposr__ has joined #openstack-lbaas07:04
*** ccamposr has quit IRC07:07
openstackgerritMerged openstack/octavia-dashboard master: Imported Translations from Zanata
*** haleyb has joined #openstack-lbaas07:37
*** rcernin has quit IRC07:42
openstackgerritArkady Shtempler proposed openstack/octavia-tempest-plugin master: Refactoring - add pool of ports to be used in tests Please ignore this patchset!!!
*** rcernin has joined #openstack-lbaas08:15
*** rpittau|afk is now known as rpittau08:27
*** haleyb has quit IRC08:41
openstackgerritArkady Shtempler proposed openstack/octavia-tempest-plugin master: Refactoring - add pool of ports to be used in tests
*** tkajinam has quit IRC08:58
rm_workugh lower-constraints failed on the pythonclient gate?09:02
gthiemongerm_work: I'm on it09:02
rm_workpylint version removed or something? O_o09:05
*** yamamoto has quit IRC09:06
gthiemongethe new pip dependency resolver...09:07
*** yamamoto has joined #openstack-lbaas09:13
cgoncalvesFYI, proposed octavia releases for train and victoria09:45
rm_workok, were there any outstanding patches we actually needed to get in?09:45
rm_workalso, are we close to saying that's the last release for Train? I should probably figure out the EOL stuff again for Stein at least09:46
rm_workcgoncalves: what version do you support in RH's distribution again?09:46
cgoncalvesrm_work, Train09:46
rm_workso Stein can EOL?09:46
cgoncalvesfine for us09:47
rm_workk, hopefully already EOL'd up to that... but I have a feeling maybe Rocky isn't09:47
cgoncalvesTrain is part of an LTS release though09:47
rm_workyeah that's fine09:47
cgoncalvesour release team pulls from branch so as long as we can merge patches is fine (even if not released)09:48
rm_workwell, EOL would archive the branch possibly...09:49
rm_workbut that's for another day to worry about, because as I said, just want to EOL up to stein :D09:49
rm_workah nice, rocky is EOL09:58
rm_workdo we have anything you think we NEED to merge in stein before an EOL?10:01
cgoncalvesrm_work, nothing comes to mind. there's only one stein patch open, very likely not going to merge ever:
rm_workyeah I am trying to put up the EOL patch but gerrit giving me problems (because of my system, not upstream)10:13
rm_workmy ssh agent doesn't get along with gerrit10:13
rm_workit tries too many incorrect identity files before it gets to the right one, so gerrit rejects my connections T_T10:13
cgoncalveshaving an identity crisis, uhm?! :)10:15
rm_workaha, got it10:16
rm_workset `IdentitiesOnly yes` in my ssh config for gerrit hosts10:16
rm_workI wonder if they will complain about an ML announcement <_<10:17
*** rcernin has quit IRC10:24
*** osmanlicilegi has quit IRC10:27
*** osmanlicilegi has joined #openstack-lbaas10:29
openstackgerritGregory Thiemonge proposed openstack/python-octaviaclient master: Fix lower-constraints job
rm_workah lower-constraints also broken in octavia master?10:39
rm_workI guess I should stop trying to merge stuff <_<10:40
gthiemongeoctavia master?10:40
rm_workwas trying to merge a bunch of changes in octavia10:40
rm_workxarlos: oh nm10:41
rm_worksorry bad ping10:41
rm_workbut, maybe it's just this patch10:41
rm_workwell, ussuri gates do appear to be broken tho10:42
rm_workbut I think we knew that10:42
xarloso/ :)10:42
rm_workahhh i see, Carlos posted why, it has another depends-on10:43
rm_workgthiemonge: looks like Carlos had something he wants on and then it'll be good to go too10:44
gthiemongeoh ok10:45
gthiemongeI missed that one10:45
rm_workoh lol, looks like that tags bug was already fixed by someone else:
rm_workgthiemonge: Michael also seems to have had a question here:
gthiemongeI reviewed it :D10:49
cgoncalvesgthiemonge, oh Greg... :D10:49
*** rcernin has joined #openstack-lbaas10:50
gthiemongeback in november, my brain doesn't work that far10:52
rm_workyeah sorry that's like... probably around the last time I actually did a review, too10:52
rm_workI have been otherwise occupied10:53
rm_worktrying to get back into things10:53
rm_workugh this one is hurting me
rm_workit's probably fine to do this? but so much SQL magic to unpack10:55
*** osmanlicilegi has quit IRC11:01
openstackgerritAdam Harwell proposed openstack/octavia master: Fix pools going into ERROR when updating the pool
*** osmanlicilegi has joined #openstack-lbaas11:05
rm_work^^ rebase11:06
*** rcernin has quit IRC11:08
openstackgerritGregory Thiemonge proposed openstack/octavia master: Add haproxy-source backend element in DIB
gthiemongeoops I haven't seen your last comments11:16
*** osmanlicilegi has quit IRC11:58
*** osmanlicilegi has joined #openstack-lbaas12:01
rm_workThe thing about ping check is a nit12:11
rm_workAnd I could be misremembering12:11
rm_workBleh yeah gates on master not working either I guess12:14
rm_workWill need to look tomorrow12:14
rm_workOr... In the morning/afternoon of today I guess :D12:14
gthiemongerm_work: we need to merge to fix the gates12:26
*** yamamoto has quit IRC12:29
*** yamamoto has joined #openstack-lbaas12:33
*** priteau has joined #openstack-lbaas12:43
*** ccamposr has joined #openstack-lbaas12:43
*** ccamposr__ has quit IRC12:45
*** osmanlicilegi has quit IRC12:50
*** osmanlicilegi has joined #openstack-lbaas12:55
*** yamamoto has quit IRC12:57
*** sapd__x has joined #openstack-lbaas13:01
openstackgerritCarlos Gonçalves proposed openstack/octavia master: Add ALPN support for TLS-enabled pools
*** parallax_ has joined #openstack-lbaas13:45
*** parallax has quit IRC13:45
*** parallax_ is now known as parallax13:45
*** sapd__x has quit IRC13:46
*** sapd__x has joined #openstack-lbaas13:46
*** vishalmanchanda has quit IRC13:51
*** ccamposr__ has joined #openstack-lbaas13:55
*** ccamposr has quit IRC13:57
openstackgerritKsawery Dziekoński proposed openstack/octavia master: Fix PK inspection and abstract raw SQL away to fix migrations on PostgreSQL
*** rcernin has joined #openstack-lbaas14:46
*** haleyb has joined #openstack-lbaas14:57
*** rcernin has quit IRC15:00
*** yamamoto has joined #openstack-lbaas15:02
*** yamamoto has quit IRC15:07
*** armax has joined #openstack-lbaas15:18
openstackgerritBrian Haley proposed openstack/octavia master: Bump oslo.log version to 4.3.0
*** psachin has quit IRC16:00
*** dosaboy has quit IRC16:41
*** dosaboy has joined #openstack-lbaas16:41
openstackgerritBrian Haley proposed openstack/octavia master: Bump oslo.log version to 4.3.0
openstackgerritBrian Haley proposed openstack/octavia master: Bump oslo.policy version to 2.1.0
haleybcgoncalves: i think we need to fix our tempest gates, it seems to have stalled though16:57
*** yamamoto has joined #openstack-lbaas17:07
*** gcheresh has quit IRC17:17
*** yamamoto has quit IRC17:23
rm_workgthiemonge: wait how does that fix the gates O_o17:55
*** rpittau is now known as rpittau|afk17:57
rm_workWell, I'm not sure how this fixes the gates still, but it seems to have passed? And i reviewed it and it seems good, so may as well give it a shot, lol18:11
rm_workOtherwise seems totally unrelated to the gate breakage18:11
*** sshnaidm|ruck has joined #openstack-lbaas18:45
sshnaidm|ruckhi, all, I have a question about octavia on devstack. I'm trying to setup a job in ansible openstack collections repo devstack based with octavia to check loadbalancer modules. I download amphora image, pass its location to devstack. Should I also upload it to cloud and tag or it should be done by devstack?
sshnaidm|ruckhope now it's a right channel)18:45
johnsomYep, right place now18:46
johnsomThe devstack plugin will build an image:
johnsomOnce that image is built, it will upload it to glance18:48
sshnaidm|ruckjohnsom, will it also tag?18:50
sshnaidm|ruckand if I provide OCTAVIA_AMP_IMAGE_FILE and download it before, then it should pick it up18:52
sshnaidm|ruckas I see from code..18:52
johnsomYeah, if it's already in glance18:53
sshnaidm|ruckjohnsom, oh, so if I download image it's on me to upload it to glance?19:01
johnsomYeah, if you don't want devstack to build it for you, yes.19:01
sshnaidm|ruckclear now, thanks johnsom19:08
haleybrm_work: that fixes a bug in the functional 'tips' job on master19:08
*** yamamoto has joined #openstack-lbaas19:21
haleybrm_work: i guess in more than that job now, for example, shows three failures due to alpn_protocols being present19:34
*** ccamposr has joined #openstack-lbaas19:36
rm_workugh ok i am really hating the new gerrit theme19:37
rm_workis there any way to go back?19:37
rm_workor is there anything that fixes it to actually show useful information?19:37
rm_workwoo, Stein EOL'd BTW :)19:38
*** ccamposr__ has quit IRC19:38
rm_workriding the Train to the future :P19:39
johnsomYeah, I'm not loving the new gerrit19:40
johnsomWait until you need to change the ACL on a gerrit group.19:40
rm_workit's like they took all the most useful information that I always look at first on a patchset and just *removed it*, then took the rest and removed any trace of spacing and distinct formatting so it all flows together into a giant pile of trash19:49
rm_worksorry, i haven't really looked at reviews in 3 months-ish, so i'm a bit late to the party here19:49
johnsomYeah, you have to hunt for it. Same for the job logs, etc. added a few layers of clicks to get to the goods19:50
rm_workwhere is the zuul job status??19:50
*** yamamoto has quit IRC19:50
rm_workthis is garbage19:50
rm_worki wonder if there's any way to hack it back with greasemonkey...19:51
johnsomlol, well there goes a week of your time19:51
*** jamesden_ is now known as jamesdenton19:53
*** itsjg has joined #openstack-lbaas19:58
*** rcernin has joined #openstack-lbaas20:03
openstackgerritMerged openstack/octavia master: Add ALPN support for TLS-enabled pools
openstackgerritBrian Haley proposed openstack/octavia master: Bump oslo.log version to 4.3.0
rm_workok, gonna start some rechecks, should pass now with that change merged?20:19
*** rcernin has quit IRC20:30
*** rcernin has joined #openstack-lbaas20:41
rm_workhaleyb: were you working on the stable branch gates too?20:51
*** zzzeek has quit IRC20:57
*** zzzeek has joined #openstack-lbaas20:57
*** spatel has joined #openstack-lbaas21:05
spateljohnsom: hey21:05
johnsomspatel Hi21:05
spatelquick question, on my controller node i am seeing 100 proc running of octavia-health-manager21:06
spateldo we need that many?21:06
spatelwhat is the best practice for small deployment?21:06
johnsomTake a look at the [health_manager] section in the config. I think some of the thread counts default to core count.21:07
johnsomrm_work might have some advice on a good number for the threads.21:08
spateli am planning to start with 8 proc and later i can add21:09
spatelmostly health_manager is just for pining amphora right?21:09
spatelor it does some heavy duty work?21:10
johnsomYeah, amps send a message every 10 seconds by default, each usually take .001 seconds to process21:10
spatelmy [health_manager] section doesn't have anything related thread so i believe its default21:10
rm_workyeah there's some math involved based on size of amphora fleet, number of HM hosts, etc21:10
johnsomThey also rotate across the health managers you have running21:10
spatellet me reduce count then21:10
spatelthanks for input let me start with small number at beginning21:11
johnsomWatch the log file, it will start whining at you if you set it too low. "THIS IS NOT GOOD", etc.21:13
spatel8 or 16 should be good for start21:14
spateljohnsom: am i missing something what is the setting to control it -
rm_workhealth_update_threads / stats_update_threads21:17
johnsomfailover_threads, health_update_threads, and stats_update_threads21:17
johnsomI would not touch failover21:18
rm_workfailover also has a low default (10)21:18
spatelThank you!21:18
spatelwhy default is None, there should be limit for Default otherwise someone can exploit it :)21:19
johnsomIt defaults to a value based on the CPU cores on the host21:20
rm_workthese threads shouldn't cause a huge problem as most will be asleep, and yes, will be capped to some reasonable(ish) scaling value21:20
johnsomYeah, they don't take up much space other than in your process listing21:21
spatelI am seeing 100 proc running and i have only 48 core cpu.21:22
rm_work48 physical cores or logical cores?21:22
johnsomThat seems right. ~48 for health_update_threads and ~48 for stats_update_threads21:23
rm_workah right, yes21:23
rm_work48 each :D21:23
rm_workplus master processes21:23
spateloh wait. i set it 16 and restart service and now its showing me 68 proc so look like not cpu core to core21:24
spatelI have 48 logical core :)21:25
spatellook like 4 proc per 1 thread21:25
spatelif i set 16 thread then proc count will be 6821:26
spatelthat is why i am seeing 10021:26
openstackgerritAdam Harwell proposed openstack/octavia master: Add default value for enabled column in l7rule table
spatelstats_update_threads = 4 / health_update_threads = 8   with that setting now i have 16 Proc running21:32
*** rcernin has quit IRC21:34
*** itsjg has left #openstack-lbaas21:40
*** yamamoto has joined #openstack-lbaas21:49
*** rcernin has joined #openstack-lbaas21:51
openstackgerritBrian Haley proposed openstack/octavia master: Bump oslo.policy version to 2.1.0
spatelrm_work: not sure what is the relation between stats_update_threads and health_update_threads22:13
rm_workthey are similar but separate queues22:13
rm_worktypically they probably will need to be similarly sized I think22:14
openstackgerritAdam Harwell proposed openstack/octavia master: Install HAProxy 2.x in CentOS amphora images
rm_work^^ rebased to kickstart tests again, spurious grenade failure AFAICT22:15
rm_worknova failed to boot a VM22:15
rm_workhaleyb: `ebtables v1.8.4 (nf_tables): table `broute' is incompatible, use 'nft' tool.`?22:22
rm_workhmm wait, maybe not the real error22:23
rm_workI hate the way DIB has so many red herrings22:23
haleybrm_work: from, funny i just saw the same thing22:24
johnsomTons of things are spewing that, it's not a real error.22:24
rm_workk yeah22:24
johnsomWell, it probably is, but it doesn't stop things22:24
rm_work`AttributeError: module 'pkg_resources' has no attribute 'get_distribution'`22:24
rm_workpip / setuptools issues?22:24
haleybthink it's a side-effect of something else.  it's like when the end of the log says the neutron agents didn't start, it's never them :)22:25
rm_work(trying to track down why all cent8 gates are failing)22:25
haleyboh, think i remember pinning something for that pkg one22:25
haleybor yeah, it's pip under the covers22:25
rm_worklooking at and every change has cent8 and scenario-amphora-v2 failing22:25
johnsomAttributeError: module 'pkg_resources' has no attribute 'get_distribution' <----- That is a real, blocking, error22:26
rm_workthe other one has this:22:27
rm_work2021-01-28 21:26:59.048957 | controller | [00;36mINFO alembic.runtime.migration [[00;36m-[00;36m] [01;35m[00;36mRunning upgrade  -> 1cea328f0f65, initial_logbook_details_tables[00m[00m22:27
rm_work2021-01-28 21:26:59.326529 | controller | [01;31mCRITICAL octavia-db-manage [[00;36m-[01;31m] [01;35m[01;31mUnhandled error[00m: TypeError: create_foreign_key() missing 3 required positional arguments: 'constraint_name', 'source_table', and 'referent_table'22:27
johnsomHmm, someone broke a migration?22:27
rm_workwell, looks like that is a dynamic FK handler22:28
*** yamamoto has quit IRC22:28
haleyb just merged with a migration, but the ids look ok22:31
*** spatel has quit IRC22:33
rm_workahhhh that migration isn't even ours22:34
rm_workthat's a base alembic migration22:34
johnsomWell, maybe it is. Ann added something like that to Octavia22:35
rm_worklook for `octavia-db-manage upgrade_persistence` in
rm_workyeah, that line in our plugin was added by Ann... in 2019?22:37
rm_workso I feel like this must be the result of something else changing outside of the original change? possibly bad assumptions were made?22:38
johnsomI don't think the migrations have changed in taskflow in a long while.22:38
johnsomYeah, no changes on the taskflow side22:40
rm_workso how could have caused an issue?22:41
rm_workdoes seem that this test was successful on previous merges22:42
rm_workadds a column to the pool table...22:42
haleybi don't know if it did, just that it was the last commit22:42
rm_workyeah, prior to that commit merging, it seems this test was passing22:42
*** yamamoto has joined #openstack-lbaas22:43
rm_workis this even supposed to be running?22:43
rm_workwhy does `octavia-db-manage upgrade_persistence` start trying to do alembic internal migrations?22:44
rm_workI think that is the issue22:44
johnsomLink to the job logs?22:44
rm_worklook for `octavia-db-manage upgrade_persistence` in
rm_workthe first thing the command tries to do is run an alembic internal migration22:45
rm_workif you look just above at the other db-manage we run, it doesn't run those at all, so it's not like they're a bookend thing like I assumed22:46
johnsomSo, alembic changed under us22:51
johnsom1.5.0 changed create_foreign_key()22:52
rm_workyeah the data they pass doesn't look AT ALL like what it's expecting22:52
rm_workAHHH taskflow22:53
johnsomYeah, Ann is running the migration from taskflow22:54
rm_worksorry i missed that in the path i posted earlier lol22:54
johnsomSo, if you fix:
johnsomPing me with a patch and I can review it.22:54
johnsomI wonder if we need to pin alembic on stable... hmmm22:54
rm_workhmmmmmmm k22:54
rm_workok, working on it22:56
johnsomrequirements bumped it 8 days ago22:57
johnsomOk, the one that broke it was probably 10 days ago, that was the bump to 1.5.022:59
*** AlexStaf has quit IRC22:59
rm_workwill this require a taskflow release and version bump on our side? <_<23:00
rm_workwould we be better off pinning for now?23:00
rm_workah nm, we can't23:00
rm_workbut also, not blocking gates, so it's not as big of a rush23:01
johnsomYeah, but I can help with that....23:01
openstackgerritBrian Haley proposed openstack/octavia master: Bump oslo.policy version to 2.1.0
openstackgerritBrian Haley proposed openstack/octavia master: Add common system and project policy check strings to constants
*** emccormick has quit IRC23:37
*** emccormick has joined #openstack-lbaas23:38

Generated by 2.17.2 by Marius Gedminas - find it at!