| *** born2bake has quit IRC | 00:09 | |
| *** mchlumsky has quit IRC | 00:25 | |
| *** mchlumsky has joined #openstack-lbaas | 00:26 | |
| *** wuchunyang has joined #openstack-lbaas | 02:17 | |
| *** wuchunyang has quit IRC | 02:35 | |
| *** sapd1 has joined #openstack-lbaas | 03:21 | |
| *** psachin has joined #openstack-lbaas | 03:36 | |
| *** ramishra has joined #openstack-lbaas | 03:52 | |
| *** wuchunyang has joined #openstack-lbaas | 04:00 | |
| *** gthiemonge has quit IRC | 04:02 | |
| *** gthiemonge has joined #openstack-lbaas | 04:03 | |
| *** wuchunyang has quit IRC | 04:04 | |
| *** mchlumsky has quit IRC | 04:06 | |
| *** mchlumsky has joined #openstack-lbaas | 04:09 | |
| *** mchlumsky has quit IRC | 04:32 | |
| *** mchlumsky has joined #openstack-lbaas | 04:33 | |
| *** takamatsu has quit IRC | 04:38 | |
| *** takamatsu has joined #openstack-lbaas | 04:38 | |
| *** wuchunyang has joined #openstack-lbaas | 04:39 | |
| *** ramishra has quit IRC | 04:59 | |
| *** ramishra has joined #openstack-lbaas | 05:04 | |
| *** ramishra has quit IRC | 05:14 | |
| *** ramishra has joined #openstack-lbaas | 05:15 | |
| *** armax has quit IRC | 05:18 | |
| *** gcheresh has joined #openstack-lbaas | 05:29 | |
| *** sapd1 has quit IRC | 05:32 | |
| *** wuchunyang has quit IRC | 05:41 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:49 | |
| cgoncalves | rm_work, https://github.com/svinota/pyroute2/pull/725 is still open. are you going forward with a patch to u-c.txt requirements? | 06:47 |
|---|---|---|
| *** maciejjozefczyk has joined #openstack-lbaas | 06:52 | |
| *** sapd1 has joined #openstack-lbaas | 07:04 | |
| *** rcernin has quit IRC | 07:32 | |
| *** ataraday_ has joined #openstack-lbaas | 07:50 | |
| *** KeithMnemonic has quit IRC | 08:23 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Allow amphorav2 to run without jobboard https://review.opendev.org/739053 | 08:25 |
| rm_work | cgoncalves: hmmmm | 08:27 |
| *** rcernin has joined #openstack-lbaas | 08:28 | |
| rm_work | so, technically we could use a different version because it's in our amphora requirements, right? <_< | 08:29 |
| rm_work | ah nm, that's only separate for DIB stuff | 08:30 |
| rm_work | but trying to think if there's a way to just override it in the amp build | 08:30 |
| rm_work | so we don't have to break neutron with a revert | 08:31 |
| *** rcernin has quit IRC | 08:34 | |
| cgoncalves | rm_work, I'd avoid overriding it in the amp build | 08:38 |
| cgoncalves | rm_work, no updates from your contact in Stockholm? | 08:38 |
| rm_work | he's been offline | 08:38 |
| rm_work | no bouncer i guess :D | 08:39 |
| rm_work | we're SURE we can't cap it inside octavia without requirements project freaking out? | 08:39 |
| cgoncalves | rm_work, it is technically possible but..... | 08:40 |
| rm_work | it only even matters inside the amp... | 08:40 |
| *** ccamposr has joined #openstack-lbaas | 08:40 | |
| rm_work | where obviously neutron isn't installed | 08:40 |
| rm_work | want to see if this passes | 08:42 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Blacklist pyroute 0.5.13 https://review.opendev.org/743926 | 08:42 |
| rm_work | ah sorry we changed this | 08:43 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Disallow pyroute 0.5.13 https://review.opendev.org/743926 | 08:43 |
| rm_work | habit T_T | 08:43 |
| cgoncalves | rm_work, I think this will conflict with neutron as it requires >=0.5.13 but here octavia services block 0.5.13 and that is latest | 08:46 |
| rm_work | assuming they get installed in the same venv, does it still not use venvs? | 08:47 |
| cgoncalves | rm_work, octavia *services* install libs in octavia/requirements.txt | 08:48 |
| rm_work | right i know | 08:48 |
| rm_work | does it not use venvs per service yet? | 08:48 |
| cgoncalves | err, nope | 08:49 |
| rm_work | like ... literally every real world install T_T lol | 08:49 |
| rm_work | RIP | 08:49 |
| rm_work | well if we disallow that version in global-requirements it'll just break neutron, which will break our gate too, right? lol | 08:50 |
| cgoncalves | it might | 08:50 |
| cgoncalves | ok, lets hope the pyroute2 gets merged very soon | 08:51 |
| *** spatel has joined #openstack-lbaas | 08:58 | |
| *** spatel has quit IRC | 09:04 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Allow amphorav2 to run without jobboard https://review.opendev.org/739053 | 09:08 |
| *** ataraday_ has quit IRC | 09:35 | |
| *** stingrayza has quit IRC | 09:35 | |
| *** ataraday_ has joined #openstack-lbaas | 09:56 | |
| *** rcernin has joined #openstack-lbaas | 10:37 | |
| *** rcernin has quit IRC | 10:54 | |
| devfaz | hi there, just installed the updated (latest rocky) octavia version and but it there are still failures during failover http://paste.openstack.org/show/796456/ | 11:23 |
| devfaz | another output, may be related: http://paste.openstack.org/show/796457/ | 11:24 |
| *** ZhuXiaoYu has joined #openstack-lbaas | 11:29 | |
| *** sapd1 has quit IRC | 11:46 | |
| *** spatel has joined #openstack-lbaas | 11:52 | |
| cgoncalves | devfaz, hi. a workaround to this nova issue was released in Rocky 3.2.0 (backport patch: https://review.opendev.org/#/c/678181/) | 11:54 |
| cgoncalves | devfaz, if the failover error occurred before the rocky update, try to fail over the load balancer again | 11:55 |
| cgoncalves | devfaz, alternatively you could clear the dns name information in the neutron port and re-run failover | 11:56 |
| *** spatel has quit IRC | 11:56 | |
| *** ZhuJoseph has joined #openstack-lbaas | 11:56 | |
| *** ataraday_ has quit IRC | 11:57 | |
| *** ZhuXiaoYu has quit IRC | 11:59 | |
| devfaz | cgoncalves: we already have the mentioned patch in our octavia - just verified the code | 12:00 |
| devfaz | currently octavia is failing over a lot of amphoras - i think i have to wait until octavia is idle again. | 12:01 |
| cgoncalves | devfaz, from your paste http://paste.openstack.org/show/796457/ it seems more like you are on Stein release | 12:11 |
| cgoncalves | unless you've made internal changes to Rocky | 12:11 |
| devfaz | cgoncalves: no sorry, stable/stein - just upgraded days ago :( | 12:12 |
| cgoncalves | ok, that makes more sense | 12:12 |
| devfaz | does this change anything? | 12:12 |
| devfaz | so may the error be somewhere else? | 12:12 |
| cgoncalves | not really. the stein backport patch is https://review.opendev.org/#/c/678180/ and was released in 4.1.0 | 12:13 |
| devfaz | cgoncalves: ok, so i checked for the line in the code and its there. | 12:14 |
| devfaz | so which port do i have to unset the dns_name? | 12:14 |
| devfaz | vrrp..? | 12:15 |
| *** servagem has joined #openstack-lbaas | 12:15 | |
| *** ZhuJoseph has quit IRC | 12:17 | |
| cgoncalves | devfaz, yes | 12:17 |
| cgoncalves | from your trace, port UUID is 2e65f86c-53b1-49ba-8dab-2f3bade281af | 12:18 |
| devfaz | cgoncalves: worked! But should the above change not do this automatically? | 12:22 |
| devfaz | cgoncalves: im having some more amphoras/loadbalancers, so if you would like to test something? | 12:23 |
| cgoncalves | devfaz, nice. could you please paste the full flow traceback? task FailoverPreparationForAmphora should have ran but appears it was not in your case...? | 12:25 |
| devfaz | cgoncalves: http://paste.openstack.org/show/796460/ | 12:27 |
| *** ataraday_ has joined #openstack-lbaas | 12:32 | |
| cgoncalves | devfaz, thank you. task FailoverPreparationForAmphora ran and succeed so I'd have expected the dns_name value in the neutron port to have been cleared. hmm. | 12:32 |
| devfaz | cgoncalves: anything you want to test or should i try to clean my amphoras/lbs? | 12:35 |
| cgoncalves | devfaz, go for cleaning your amps | 12:36 |
| devfaz | cgoncalves: i will try - thanks a lot for your help! | 12:37 |
| cgoncalves | devfaz, no problem. thank you for providing logs and predisposition to run experiments | 12:38 |
| devfaz | cgoncalves: should i create a bugreport or will you (reopen)? | 12:41 |
| *** tkajinam has quit IRC | 12:47 | |
| cgoncalves | devfaz, feel free to create one. though I'm not sure it would be high priority since in train onwards we drop the port and create a new one | 12:50 |
| *** mvorwerk has joined #openstack-lbaas | 12:50 | |
| devfaz | cgoncalves: good to know - so i will just try to upgrade asap | 12:50 |
| cgoncalves | devfaz, for reference these are the master and backport patches: https://review.opendev.org/#/q/I04cb2f1f10ec566298834f81df0cf8b100ca916c | 12:51 |
| cgoncalves | it is available in stable/train branch but we have not yet cut a new train release | 12:52 |
| *** born2bake has joined #openstack-lbaas | 12:52 | |
| devfaz | cgoncalves: another (different) issue -> http://paste.openstack.org/show/796462/ | 13:03 |
| devfaz | cgoncalves: stack-trace: https://pastebin.com/UTjR5rRU | 13:04 |
| cgoncalves | devfaz, internal server error coming from the amphora. if you scroll up a bit you may find an error message | 13:04 |
| cgoncalves | {u'http_code': 500, u'error': u"a bytes-like object is required, not 'str'"} | 13:05 |
| devfaz | 2020-07-30 12:57:38.653 14 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-8af8c463-596c-49db-aab4-c1aa507343a3 - 7b1d8988efca4ad696bc6670ca2b3c0f - - -] Amphora agent returned unexpected result code 500 with response {u'http_code': 500, u'error': u"a bytes-like object is required, not 'str'"} | 13:05 |
| devfaz | :) | 13:05 |
| cgoncalves | py2 vs py3 | 13:05 |
| cgoncalves | there was someone just this week IIRC with the same issue | 13:05 |
| devfaz | cgoncalves: we had a similar issue in the past, maybe https://review.opendev.org/#/c/480919/ got somehow removed from our branch. | 13:08 |
| cgoncalves | devfaz, that change never merged | 13:09 |
| devfaz | cgoncalves: i know, we cherry-picked it years ago in our branch. Maybe it got lost or is causing this. Have to check. | 13:11 |
| cgoncalves | devfaz, https://review.opendev.org/#/q/I6f5d95c5f875edda530f54ae72386d6495235ca6 | 13:12 |
| cgoncalves | devfaz, could you get logs from the amphora? we need to check if it is the same error or different one | 13:12 |
| cgoncalves | you can compare it with the trace in https://storyboard.openstack.org/#!/story/2005898 | 13:14 |
| devfaz | cgoncalves: not merged jet, isnt it - so using python2 in amphoras is the way to go? | 13:17 |
| devfaz | cgoncalves: ignore me.. last comment.. merged.. | 13:17 |
| devfaz | included in train+ | 13:17 |
| cgoncalves | devfaz, py3 issue fixed in stein+ | 13:23 |
| devfaz | cgoncalves: was this patched after stein was released? Asking because our amphora image is not jet using latest stable/stein, so maybe this is the reason for this. | 13:30 |
| devfaz | the output-log tells me the script is installing python3 in the amphora-image. | 13:30 |
| cgoncalves | devfaz, patch was released in stein 4.1.0 | 13:35 |
| devfaz | cgoncalves: thx, so we dont have the fix in our amphora images.. build is running - again thx a lot for your help! | 13:37 |
| *** TrevorV has joined #openstack-lbaas | 13:38 | |
| *** sapd1 has joined #openstack-lbaas | 13:42 | |
| *** psachin has quit IRC | 14:12 | |
| *** ataraday_ has quit IRC | 14:15 | |
| *** stingrayza has joined #openstack-lbaas | 14:21 | |
| *** stingrayza has quit IRC | 14:29 | |
| *** stingrayza has joined #openstack-lbaas | 14:30 | |
| *** mvorwerk has quit IRC | 14:40 | |
| *** armax has joined #openstack-lbaas | 15:23 | |
| *** gcheresh has quit IRC | 15:35 | |
| *** livelace has joined #openstack-lbaas | 15:52 | |
| johnsom | Octavia team, I have just sent out an e-mail to the openstack-discuss mailing list proposing adding atarady_ and gthiemonge to the Octavia core reviewer team. Cores, please reply with your support or concerns. | 16:07 |
| *** gcheresh has joined #openstack-lbaas | 16:08 | |
| *** gcheresh has quit IRC | 16:26 | |
| *** wuchunyang has joined #openstack-lbaas | 16:32 | |
| *** tow has quit IRC | 16:36 | |
| *** wuchunyang has quit IRC | 16:37 | |
| *** sapd1 has quit IRC | 17:29 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: WIP: Adjust scenario tests for NotImplemented skip https://review.opendev.org/714004 | 17:52 |
| *** gcheresh has joined #openstack-lbaas | 18:02 | |
| *** livelace has quit IRC | 18:11 | |
| *** gcheresh has quit IRC | 18:29 | |
| *** livelace has joined #openstack-lbaas | 18:36 | |
| rm_work | johnsom: thoughts again today on pyroute2 issue? | 19:12 |
| *** maciejjozefczyk has quit IRC | 19:13 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 19:13 | |
| johnsom | rm_work Caught me making lunch. Yeah, so with no progress on that I guess we need to revert as we can't exclude that version it our project right? | 19:25 |
| rm_work | seems so | 19:26 |
| rm_work | i think just add that version to the global exclusions list | 19:26 |
| rm_work | 0.5.13 | 19:27 |
| rm_work | i doubt another release will happen with this broken | 19:27 |
| rm_work | ah or yeah, revert the change that bumped it | 19:28 |
| cgoncalves | we could go wild and run ".venv/pip install pyroute2==0.5.12" in https://github.com/openstack/octavia/tree/master/elements/amphora-agent/post-install.d | 19:29 |
| johnsom | Yeah, that is an idea | 19:29 |
| *** maciejjozefczyk has quit IRC | 19:58 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Workaround broken pyroute2 0.5.13 https://review.opendev.org/744045 | 20:01 |
| johnsom | Giving that a shot | 20:02 |
| cgoncalves | I don't think you need to uninstall it first. install will override/reinstall | 20:03 |
| cgoncalves | nit-picking on a temporary workaround xD | 20:03 |
| johnsom | yeah, I wasn't sure it would work reliably with just the reinstall, so I did what I know will work for sure. | 20:04 |
| rm_work | Yeah that is what I was suggesting last night XD | 20:30 |
| rm_work | "cgoncalves: rm_work, I'd avoid overriding it in the amp build" | 20:30 |
| rm_work | changed your tune today, eh? ;) | 20:30 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix accepting 'insert_headers' when unsupported https://review.opendev.org/744047 | 20:34 |
| johnsom | ^^^ fixing bugs found by my new tempest scenario test work. | 20:34 |
| johnsom | You could add insert_headers for TCP, etc. | 20:35 |
| *** vishalmanchanda has quit IRC | 20:39 | |
| *** ianychoi has joined #openstack-lbaas | 20:41 | |
| *** TrevorV has quit IRC | 20:50 | |
| *** livelace has quit IRC | 21:14 | |
| *** born2bake has quit IRC | 21:59 | |
| *** tkajinam has joined #openstack-lbaas | 22:05 | |
| johnsom | cgoncalves rm_work https://review.opendev.org/#/c/744045/ is passing. Can we get a review on it and if no one else is around we can single review merge that. | 22:29 |
| rm_work | why is cent8 failing? :D | 22:40 |
| rm_work | this is relevant for me lol | 22:40 |
| rm_work | +2'd, looks like nova breaking booting anything | 22:42 |
| rm_work | (for the cent8 test failures) | 22:42 |
| johnsom | Looks like the hypervisor was broken on the centos8 nodepool instance: 2020-07-30T20:40:26.373867Z qemu-kvm: error: failed to set MSR 0xe1 to 0x0 | 22:44 |
| johnsom | qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed. | 22:44 |
| johnsom | None of the vms could start on that instance. Not even cirros | 22:45 |
| johnsom | nested-virt-centos-8-vexxhost-ca-ymq-1-0018999589 | 22:45 |
| johnsom | mnaser If you have a minute to information gather for us, this host: nested-virt-centos-8-vexxhost-ca-ymq-1-0018999589 just blew up nested virt. Would be interested if there was any info in the host logs, what OS and CPU it has. | 22:54 |
| johnsom | The uglies are here: https://zuul.opendev.org/t/openstack/build/148ce3fd294548bd82c5b24c7656cac6/log/controller/logs/libvirt/qemu/instance-00000001_log.txt | 22:55 |
| johnsom | I haven't seen that error before. It's different than the one we saw last year on other hosts. | 22:56 |
| mnaser | johnsom: we recently bumped to qemu 5.x + libvirt 6.4.0 there i think -- but we're pending a kernel update that is likely happening today/tomorrow | 22:56 |
| mnaser | which helps a ton with this | 22:56 |
| johnsom | Yeah, cool. | 22:56 |
| johnsom | Then let's not bother to dig deeper and see if *magic* happens | 22:57 |
| mnaser | does the cpu get logged somewhere in jobs | 22:57 |
| mnaser | i could swear devstack logged it | 22:57 |
| johnsom | Yeah, but the lower level hypervisor can lie to us on that. | 22:57 |
| mnaser | but knowing what lie you are given helps me a bit :) | 22:58 |
| johnsom | lol, ok, let me see if I can find it | 22:58 |
| mnaser | johnsom: https://1c5168ba037840fa5355-4c1d5fc7733900a07272f3d252a01439.ssl.cf2.rackcdn.com/744045/1/check/octavia-v2-dsvm-scenario-centos-8/148ce3f/zuul-info/host-info.controller.yaml | 22:59 |
| johnsom | Looks like AMD EPYC 7402 24-Core Processor | 22:59 |
| mnaser | found it, no those are the "right" hosts | 22:59 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Change amphora statistics to use deltas https://review.opendev.org/740815 | 23:13 |
| *** rcernin has joined #openstack-lbaas | 23:13 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Refactoring amphora stats driver interface https://review.opendev.org/737111 | 23:13 |
| rm_work | rebased both on that fix so they can start running tests :D | 23:13 |
| *** rcernin has quit IRC | 23:14 | |
| *** rcernin has joined #openstack-lbaas | 23:14 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!