Thursday, 2020-07-30

*** born2bake has quit IRC00:09
*** mchlumsky has quit IRC00:25
*** mchlumsky has joined #openstack-lbaas00:26
*** wuchunyang has joined #openstack-lbaas02:17
*** wuchunyang has quit IRC02:35
*** sapd1 has joined #openstack-lbaas03:21
*** psachin has joined #openstack-lbaas03:36
*** ramishra has joined #openstack-lbaas03:52
*** wuchunyang has joined #openstack-lbaas04:00
*** gthiemonge has quit IRC04:02
*** gthiemonge has joined #openstack-lbaas04:03
*** wuchunyang has quit IRC04:04
*** mchlumsky has quit IRC04:06
*** mchlumsky has joined #openstack-lbaas04:09
*** mchlumsky has quit IRC04:32
*** mchlumsky has joined #openstack-lbaas04:33
*** takamatsu has quit IRC04:38
*** takamatsu has joined #openstack-lbaas04:38
*** wuchunyang has joined #openstack-lbaas04:39
*** ramishra has quit IRC04:59
*** ramishra has joined #openstack-lbaas05:04
*** ramishra has quit IRC05:14
*** ramishra has joined #openstack-lbaas05:15
*** armax has quit IRC05:18
*** gcheresh has joined #openstack-lbaas05:29
*** sapd1 has quit IRC05:32
*** wuchunyang has quit IRC05:41
*** vishalmanchanda has joined #openstack-lbaas05:49
cgoncalvesrm_work, is still open. are you going forward with a patch to u-c.txt requirements?06:47
*** maciejjozefczyk has joined #openstack-lbaas06:52
*** sapd1 has joined #openstack-lbaas07:04
*** rcernin has quit IRC07:32
*** ataraday_ has joined #openstack-lbaas07:50
*** KeithMnemonic has quit IRC08:23
openstackgerritCarlos Goncalves proposed openstack/octavia master: Allow amphorav2 to run without jobboard
rm_workcgoncalves: hmmmm08:27
*** rcernin has joined #openstack-lbaas08:28
rm_workso, technically we could use a different version because it's in our amphora requirements, right? <_<08:29
rm_workah nm, that's only separate for DIB stuff08:30
rm_workbut trying to think if there's a way to just override it in the amp build08:30
rm_workso we don't have to break neutron with a revert08:31
*** rcernin has quit IRC08:34
cgoncalvesrm_work, I'd avoid overriding it in the amp build08:38
cgoncalvesrm_work, no updates from your contact in Stockholm?08:38
rm_workhe's been offline08:38
rm_workno bouncer i guess :D08:39
rm_workwe're SURE we can't cap it inside octavia without requirements project freaking out?08:39
cgoncalvesrm_work, it is technically possible but.....08:40
rm_workit only even matters inside the amp...08:40
*** ccamposr has joined #openstack-lbaas08:40
rm_workwhere obviously neutron isn't installed08:40
rm_workwant to see if this passes08:42
openstackgerritAdam Harwell proposed openstack/octavia master: Blacklist pyroute 0.5.13
rm_workah sorry we changed this08:43
openstackgerritAdam Harwell proposed openstack/octavia master: Disallow pyroute 0.5.13
rm_workhabit T_T08:43
cgoncalvesrm_work, I think this will conflict with neutron as it requires >=0.5.13 but here octavia services block 0.5.13 and that is latest08:46
rm_workassuming they get installed in the same venv, does it still not use venvs?08:47
cgoncalvesrm_work, octavia *services* install libs in octavia/requirements.txt08:48
rm_workright i know08:48
rm_workdoes it not use venvs per service yet?08:48
cgoncalveserr, nope08:49
rm_worklike ... literally every real world install T_T lol08:49
rm_workwell if we disallow that version in global-requirements it'll just break neutron, which will break our gate too, right? lol08:50
cgoncalvesit might08:50
cgoncalvesok, lets hope the pyroute2 gets merged very soon08:51
*** spatel has joined #openstack-lbaas08:58
*** spatel has quit IRC09:04
openstackgerritCarlos Goncalves proposed openstack/octavia master: Allow amphorav2 to run without jobboard
*** ataraday_ has quit IRC09:35
*** stingrayza has quit IRC09:35
*** ataraday_ has joined #openstack-lbaas09:56
*** rcernin has joined #openstack-lbaas10:37
*** rcernin has quit IRC10:54
devfazhi there, just installed the updated (latest rocky) octavia version and but it there are still failures during failover
devfazanother output, may be related:
*** ZhuXiaoYu has joined #openstack-lbaas11:29
*** sapd1 has quit IRC11:46
*** spatel has joined #openstack-lbaas11:52
cgoncalvesdevfaz, hi. a workaround to this nova issue was released in Rocky 3.2.0 (backport patch:
cgoncalvesdevfaz, if the failover error occurred before the rocky update, try to fail over the load balancer again11:55
cgoncalvesdevfaz, alternatively you could clear the dns name information in the neutron port and re-run failover11:56
*** spatel has quit IRC11:56
*** ZhuJoseph has joined #openstack-lbaas11:56
*** ataraday_ has quit IRC11:57
*** ZhuXiaoYu has quit IRC11:59
devfazcgoncalves: we already have the mentioned patch in our octavia - just verified the code12:00
devfazcurrently octavia is failing over a lot of amphoras - i think i have to wait until octavia is idle again.12:01
cgoncalvesdevfaz, from your paste it seems more like you are on Stein release12:11
cgoncalvesunless you've made internal changes to Rocky12:11
devfazcgoncalves: no sorry, stable/stein - just upgraded days ago :(12:12
cgoncalvesok, that makes more sense12:12
devfazdoes this change anything?12:12
devfazso may the error be somewhere else?12:12
cgoncalvesnot really. the stein backport patch is and was released in 4.1.012:13
devfazcgoncalves: ok, so i checked for the line in the code and its there.12:14
devfazso which port do i have to unset the dns_name?12:14
*** servagem has joined #openstack-lbaas12:15
*** ZhuJoseph has quit IRC12:17
cgoncalvesdevfaz, yes12:17
cgoncalvesfrom your trace, port UUID is 2e65f86c-53b1-49ba-8dab-2f3bade281af12:18
devfazcgoncalves: worked! But should the above change not do this automatically?12:22
devfazcgoncalves: im having some more amphoras/loadbalancers, so if you would like to test something?12:23
cgoncalvesdevfaz, nice. could you please paste the full flow traceback? task FailoverPreparationForAmphora should have ran but appears it was not in your case...?12:25
*** ataraday_ has joined #openstack-lbaas12:32
cgoncalvesdevfaz, thank you. task FailoverPreparationForAmphora ran and succeed so I'd have expected the dns_name value in the neutron port to have been cleared. hmm.12:32
devfazcgoncalves: anything you want to test or should i try to clean my amphoras/lbs?12:35
cgoncalvesdevfaz, go for cleaning your amps12:36
devfazcgoncalves: i will try - thanks a lot for your help!12:37
cgoncalvesdevfaz, no problem. thank you for providing logs and predisposition to run experiments12:38
devfazcgoncalves: should i create a bugreport or will you (reopen)?12:41
*** tkajinam has quit IRC12:47
cgoncalvesdevfaz, feel free to create one. though I'm not sure it would be high priority since in train onwards we drop the port and create a new one12:50
*** mvorwerk has joined #openstack-lbaas12:50
devfazcgoncalves: good to know - so i will just try to upgrade asap12:50
cgoncalvesdevfaz, for reference these are the master and backport patches:
cgoncalvesit is available in stable/train branch but we have not yet cut a new train release12:52
*** born2bake has joined #openstack-lbaas12:52
devfazcgoncalves: another (different) issue ->
devfazcgoncalves: stack-trace:
cgoncalvesdevfaz, internal server error coming from the amphora. if you scroll up a bit you may find an error message13:04
cgoncalves{u'http_code': 500, u'error': u"a bytes-like object is required, not 'str'"}13:05
devfaz2020-07-30 12:57:38.653 14 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-8af8c463-596c-49db-aab4-c1aa507343a3 - 7b1d8988efca4ad696bc6670ca2b3c0f - - -] Amphora agent returned unexpected result code 500 with response {u'http_code': 500, u'error': u"a bytes-like object is required, not 'str'"}13:05
cgoncalvespy2 vs py313:05
cgoncalvesthere was someone just this week IIRC with the same issue13:05
devfazcgoncalves: we had a similar issue in the past, maybe got somehow removed from our branch.13:08
cgoncalvesdevfaz, that change never merged13:09
devfazcgoncalves: i know, we cherry-picked it years ago in our branch. Maybe it got lost or is causing this. Have to check.13:11
cgoncalvesdevfaz, could you get logs from the amphora? we need to check if it is the same error or different one13:12
cgoncalvesyou can compare it with the trace in!/story/200589813:14
devfazcgoncalves: not merged jet, isnt it - so using python2 in amphoras is the way to go?13:17
devfazcgoncalves: ignore me.. last comment.. merged..13:17
devfazincluded in train+13:17
cgoncalvesdevfaz, py3 issue fixed in stein+13:23
devfazcgoncalves: was this patched after stein was released? Asking because our amphora image is not jet using latest stable/stein, so maybe this is the reason for this.13:30
devfazthe output-log tells me the script is installing python3 in the amphora-image.13:30
cgoncalvesdevfaz, patch was released in stein 4.1.013:35
devfazcgoncalves: thx, so we dont have the fix in our amphora images.. build is running - again thx a lot for your help!13:37
*** TrevorV has joined #openstack-lbaas13:38
*** sapd1 has joined #openstack-lbaas13:42
*** psachin has quit IRC14:12
*** ataraday_ has quit IRC14:15
*** stingrayza has joined #openstack-lbaas14:21
*** stingrayza has quit IRC14:29
*** stingrayza has joined #openstack-lbaas14:30
*** mvorwerk has quit IRC14:40
*** armax has joined #openstack-lbaas15:23
*** gcheresh has quit IRC15:35
*** livelace has joined #openstack-lbaas15:52
johnsomOctavia team, I have just sent out an e-mail to the openstack-discuss mailing list proposing adding atarady_ and gthiemonge to the Octavia core reviewer team. Cores, please reply with your support or concerns.16:07
*** gcheresh has joined #openstack-lbaas16:08
*** gcheresh has quit IRC16:26
*** wuchunyang has joined #openstack-lbaas16:32
*** tow has quit IRC16:36
*** wuchunyang has quit IRC16:37
*** sapd1 has quit IRC17:29
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: WIP: Adjust scenario tests for NotImplemented skip
*** gcheresh has joined #openstack-lbaas18:02
*** livelace has quit IRC18:11
*** gcheresh has quit IRC18:29
*** livelace has joined #openstack-lbaas18:36
rm_workjohnsom: thoughts again today on pyroute2 issue?19:12
*** maciejjozefczyk has quit IRC19:13
*** maciejjozefczyk has joined #openstack-lbaas19:13
johnsomrm_work Caught me making lunch. Yeah, so with no progress on that I guess we need to revert as we can't exclude that version it our project right?19:25
rm_workseems so19:26
rm_worki think just add that version to the global exclusions list19:26
rm_worki doubt another release will happen with this broken19:27
rm_workah or yeah, revert the change that bumped it19:28
cgoncalveswe could go wild and run ".venv/pip install pyroute2==0.5.12" in
johnsomYeah, that is an idea19:29
*** maciejjozefczyk has quit IRC19:58
openstackgerritMichael Johnson proposed openstack/octavia master: Workaround broken pyroute2 0.5.13
johnsomGiving that a shot20:02
cgoncalvesI don't think you need to uninstall it first. install will override/reinstall20:03
cgoncalvesnit-picking on a temporary workaround xD20:03
johnsomyeah, I wasn't sure it would work reliably with just the reinstall, so I did what I know will work for sure.20:04
rm_workYeah that is what I was suggesting last night XD20:30
rm_work"cgoncalves: rm_work, I'd avoid overriding it in the amp build"20:30
rm_workchanged your tune today, eh? ;)20:30
openstackgerritMichael Johnson proposed openstack/octavia master: Fix accepting 'insert_headers' when unsupported
johnsom^^^ fixing bugs found by my new tempest scenario test work.20:34
johnsomYou could add insert_headers for TCP, etc.20:35
*** vishalmanchanda has quit IRC20:39
*** ianychoi has joined #openstack-lbaas20:41
*** TrevorV has quit IRC20:50
*** livelace has quit IRC21:14
*** born2bake has quit IRC21:59
*** tkajinam has joined #openstack-lbaas22:05
johnsomcgoncalves rm_work is passing. Can we get a review on it and if no one else is around we can single review merge that.22:29
rm_workwhy is cent8 failing? :D22:40
rm_workthis is relevant for me lol22:40
rm_work+2'd, looks like nova breaking booting anything22:42
rm_work(for the cent8 test failures)22:42
johnsomLooks like the hypervisor was broken on the centos8 nodepool instance: 2020-07-30T20:40:26.373867Z qemu-kvm: error: failed to set MSR 0xe1 to 0x022:44
johnsomqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.22:44
johnsomNone of the vms could start on that instance. Not even cirros22:45
johnsom nested-virt-centos-8-vexxhost-ca-ymq-1-001899958922:45
johnsommnaser If you have a minute to information gather for us, this host: nested-virt-centos-8-vexxhost-ca-ymq-1-0018999589 just blew up nested virt.  Would be interested if there was any info in the host logs, what OS and CPU it has.22:54
johnsomThe uglies are here:
johnsomI haven't seen that error before. It's different than the one we saw last year on other hosts.22:56
mnaserjohnsom: we recently bumped to qemu 5.x + libvirt 6.4.0 there i think -- but we're pending a kernel update that is likely happening today/tomorrow22:56
mnaserwhich helps a ton with this22:56
johnsomYeah, cool.22:56
johnsomThen let's not bother to dig deeper and see if *magic* happens22:57
mnaserdoes the cpu get logged somewhere in jobs22:57
mnaseri could swear devstack logged it22:57
johnsomYeah, but the lower level hypervisor can lie to us on that.22:57
mnaserbut knowing what lie you are given helps me a bit :)22:58
johnsomlol, ok, let me see if I can find it22:58
johnsomLooks like AMD EPYC 7402 24-Core Processor22:59
mnaserfound it, no those are the "right" hosts22:59
openstackgerritAdam Harwell proposed openstack/octavia master: Change amphora statistics to use deltas
*** rcernin has joined #openstack-lbaas23:13
openstackgerritAdam Harwell proposed openstack/octavia master: Refactoring amphora stats driver interface
rm_workrebased both on that fix so they can start running tests :D23:13
*** rcernin has quit IRC23:14
*** rcernin has joined #openstack-lbaas23:14

Generated by 2.17.2 by Marius Gedminas - find it at!