| *** rcernin has quit IRC | 00:17 | |
| *** rcernin has joined #openstack-lbaas | 00:17 | |
| *** vesper11 has quit IRC | 03:45 | |
| *** vesper11 has joined #openstack-lbaas | 03:47 | |
| *** ricolin has joined #openstack-lbaas | 03:54 | |
| *** gthiemonge has quit IRC | 04:05 | |
| *** gthiemonge has joined #openstack-lbaas | 04:06 | |
| *** ccamposr has joined #openstack-lbaas | 05:02 | |
| *** ccamposr__ has quit IRC | 05:05 | |
| *** ccamposr__ has joined #openstack-lbaas | 05:09 | |
| *** ccamposr has quit IRC | 05:11 | |
| *** gcheresh has joined #openstack-lbaas | 05:29 | |
| *** ccamposr has joined #openstack-lbaas | 05:35 | |
| *** gcheresh has quit IRC | 05:35 | |
| *** ccamposr__ has quit IRC | 05:36 | |
| *** numans has joined #openstack-lbaas | 06:23 | |
| *** rcernin has quit IRC | 07:09 | |
| *** gcheresh has joined #openstack-lbaas | 07:44 | |
| *** tkajinam has quit IRC | 08:06 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 08:07 | |
| *** ccamposr__ has joined #openstack-lbaas | 08:16 | |
| *** tesseract has joined #openstack-lbaas | 08:16 | |
| *** ccamposr has quit IRC | 08:18 | |
| *** rpittau|afk is now known as rpittau | 08:33 | |
| *** ricolin has quit IRC | 09:16 | |
| kklimonda | how would I go about monitoring octavia services' state? For some OS components, they expose services API via /os-services endpoint (and some others use random endpoint with service-dependent response) but I don't see anything in octavia API docs about that | 10:01 |
|---|---|---|
| *** takamatsu has quit IRC | 10:05 | |
| *** takamatsu has joined #openstack-lbaas | 10:06 | |
| openstackgerrit | Merged openstack/octavia stable/rocky: Fix a potential race condition with certs-ramfs https://review.opendev.org/694825 | 10:29 |
| cgoncalves | kklimonda, hi. you could check if the API is HTTP 2xx responding. for other services, check if processes have port open with the message bus | 10:40 |
| kklimonda | cgoncalves: is this how others monitor octavia? are there examples I could take a look at? monitoring whether port to rabbitmq is open seems... flaky, or at least not as explicit as asking service for its status. | 10:43 |
| cgoncalves | kklimonda, I don't know how others do. TripleO does like so. have a look in tripleo-common/healthcheck/ | 10:45 |
| kklimonda | thanks :) | 10:45 |
| kklimonda | well, that makes my concept of using shared prometheus exporter a "little tricky" - thank god all OS services follow some common patterns :/ | 10:48 |
| kklimonda | but thanks nevertheless, at least I know where I stand | 10:48 |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Use VM as client in traffic operation scenario tests https://review.opendev.org/687562 | 10:53 |
| *** pcaruana has joined #openstack-lbaas | 11:14 | |
| *** openstack has joined #openstack-lbaas | 11:55 | |
| *** ChanServ sets mode: +o openstack | 11:55 | |
| *** servagem has joined #openstack-lbaas | 13:04 | |
| *** goldyfruit has quit IRC | 13:11 | |
| *** goldyfruit has joined #openstack-lbaas | 13:14 | |
| *** goldyfruit has quit IRC | 13:19 | |
| *** KeithMnemonic has joined #openstack-lbaas | 13:54 | |
| johnsom | The only service I see that has os-services is nova and it's used to disable scheduling. | 14:06 |
| *** KeithMnemonic has quit IRC | 14:33 | |
| *** TrevorV has joined #openstack-lbaas | 14:36 | |
| *** goldyfruit has joined #openstack-lbaas | 14:38 | |
| *** goldyfruit has quit IRC | 14:53 | |
| *** goldyfruit has joined #openstack-lbaas | 14:55 | |
| *** goldyfruit_ has joined #openstack-lbaas | 14:59 | |
| *** goldyfruit has quit IRC | 15:01 | |
| *** ccamposr has joined #openstack-lbaas | 15:43 | |
| *** ccamposr__ has quit IRC | 15:46 | |
| maciejjozefczyk | johnsom, hey! can we merge this one? I fixed nits you mentioned. https://review.opendev.org/#/c/676135/ | 15:52 |
| rm_work | yeah .... wtf? https://zuul.opendev.org/t/openstack/build/293129424067451291d67a12d76ec6f9/log/controller/logs/dib-build/amphora-x64-haproxy.qcow2_log.txt.gz#4700 | 15:53 |
| johnsom | maciejjozefczyk I can take a re-review over it. | 15:53 |
| rm_work | maciejjozefczyk: i doubt we can merge anything, gates have been fucked for a few days now | 15:53 |
| maciejjozefczyk | johnsom, thanks! | 15:53 |
| rm_work | it's prolly going to be a little bit T_T | 15:53 |
| maciejjozefczyk | rm_work, ough :( | 15:53 |
| johnsom | Yeah, the recent requirements release was .... less than good | 15:54 |
| rm_work | yeah this looks fine to me maciejjozefczyk, MAYBE we can merge tempest changes? not sure if they will break currently or not | 15:55 |
| johnsom | rm_work Yeah, I am puzzled on this last issue. It builds fine for me local. I can also run the egg_info command without issue | 15:55 |
| TrevorV | How do I, uh... delete a loadbalancer stuck in pending_create? | 15:55 |
| rm_work | TrevorV: wait for it to not be in pending | 15:55 |
| rm_work | TrevorV: did you kill the controller-worker in the middle of an operation? | 15:55 |
| TrevorV | I restarted services because of a bad configuration file, yeah | 15:56 |
| rm_work | yeah... then connect to the DB and change the status for that LB to ERROR | 15:56 |
| johnsom | Graceful restart should be fine, but if it kill -9 you might be hosed | 15:56 |
| TrevorV | Then i can just delete it? | 15:56 |
| rm_work | unfortunately if you interrupt the workers things break | 15:56 |
| rm_work | after you update the provisioning_status to ERROR, you can delete it as normal with the API, yes | 15:56 |
| *** devfaz has quit IRC | 15:56 | |
| *** devfaz has joined #openstack-lbaas | 15:57 | |
| cgoncalves | alternatively, you can run LB failover once the LB is on ERROR | 15:57 |
| rm_work | cgoncalves: it's never going to go to error, he killed the worker | 15:57 |
| rm_work | ah i guess once he manually does it, yeah | 15:58 |
| cgoncalves | yep | 15:58 |
| rm_work | though i still don't trust that so much if the LB NEVER finished creating | 15:58 |
| TrevorV | Well, nuked it either way. Thanks! | 15:58 |
| rm_work | i don't know that failover would pull it out of an incomplete create at the moment | 15:58 |
| TrevorV | In case anyone's curious... I'm testing simple octavia flows on power right now :D | 15:58 |
| rm_work | it could be missing all kinds of stuff | 15:58 |
| rm_work | ah cool | 15:59 |
| *** gcheresh has quit IRC | 16:00 | |
| johnsom | rm_work Do you have any ideas on that error? I haven't been able to walk through all of the logs yet, but on the surface that is not a helpful error | 16:01 |
| johnsom | I am listening to the Oslo virtual PTG at the moment | 16:01 |
| *** armax has joined #openstack-lbaas | 16:07 | |
| TrevorV | Am I supposed to create the "ssl-cert-snakeoil.pem" CA cert myself? I have yet to do anything SSL related in my testing here. | 16:11 |
| TrevorV | Sorry, I'm sure this is a stupid question... I'm really just trying to get a simple hello-world thing goin to verify that the individual basic components work. | 16:13 |
| cgoncalves | TrevorV, are you installing Octavia manually or via an installer like devstack/tripleo? | 16:14 |
| TrevorV | Via the packages in yum, currently, so manually | 16:14 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: [DNM] Testing current state of gates https://review.opendev.org/695941 | 16:14 |
| *** openstackstatus has joined #openstack-lbaas | 16:15 | |
| *** ChanServ sets mode: +v openstackstatus | 16:15 | |
| johnsom | Ok, going to freshen up my coffee and walk through the devstack log file to see if I can find something causing this other failure. | 16:16 |
| rm_work | yeah i'm looking too | 16:17 |
| cgoncalves | TrevorV, there's a WIP install guide for Ubuntu that might help you even though you're on an Yum distro. see https://review.opendev.org/#/c/672842/ | 16:18 |
| rm_work | i don't think we have the other log that the error message points us to (since it's inside the build process and we don't capture that) | 16:18 |
| cgoncalves | the devstack plugin is also a good install reference | 16:18 |
| TrevorV | cgoncalves its funny... I've got an ansible playbook that basically does all these steps currently... I'm double checking that I haven't quite missed something ha ha | 16:19 |
| TrevorV | And just arrived at the extra setup stuff, like the certs. thanks! | 16:19 |
| cgoncalves | TrevorV, there's a whole page doc page about certificates. if you need more info, I suggest https://docs.openstack.org/octavia/latest/admin/guides/certificates.html | 16:20 |
| TrevorV | Feels like overkill for a simple test of creating a loadbalancer, lol, but yeah, lemme dig in a bit more | 16:21 |
| cgoncalves | TrevorV, have you tried to deploy with TripleO? any lessons learned? :) | 16:21 |
| TrevorV | I don't wanna rant about things, lol. Can I just put an 'insecure' somewhere in the config to ignore the missing certs for a simple hello-world? | 16:26 |
| rm_work | in most cases yes, but not for the amphora communication, it's pretty essential | 16:30 |
| TrevorV | Alright. I'll figure this out then. | 16:30 |
| *** goldyfruit___ has joined #openstack-lbaas | 16:33 | |
| *** goldyfruit_ has quit IRC | 16:36 | |
| *** rpittau is now known as rpittau|afk | 16:40 | |
| johnsom | rm_work So looking at what happens around that time in my local image build, it might be the bump in pytest that is failing. I see they also bumped it in that UC change | 16:46 |
| johnsom | https://www.irccloud.com/pastebin/W2XWJIl7/ | 16:46 |
| *** goldyfruit_ has joined #openstack-lbaas | 16:55 | |
| *** goldyfruit___ has quit IRC | 16:57 | |
| rm_work | hmmm | 17:03 |
| johnsom | Not sure on that | 17:03 |
| rm_work | yeah the results of that DNM change are telling | 17:03 |
| rm_work | (everything is fucked) | 17:03 |
| johnsom | Seeing if I can inject something into setuptools/distutils to get more debug | 17:04 |
| rm_work | kk | 17:04 |
| rm_work | i just wish new things could stop breaking for a moment so we could be dealing with ONE issue at a time >_< | 17:08 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: DNM: Debug for distutils https://review.opendev.org/695947 | 17:11 |
| johnsom | Yeah, this has been at least three things breaking us all at once | 17:11 |
| *** ccamposr__ has joined #openstack-lbaas | 17:12 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: DNM: Debug for distutils https://review.opendev.org/695947 | 17:12 |
| *** ccamposr has quit IRC | 17:14 | |
| johnsom | rm_work Ok, it might be this: https://review.opendev.org/#/c/695823/ | 17:15 |
| rm_work | <_< | 17:16 |
| johnsom | Probably. That would make sense why it only breaks in the gates which use the mirrors | 17:16 |
| rm_work | ah i thought it was just in image-build gates | 17:17 |
| rm_work | the noop use mirrors too, no? | 17:17 |
| johnsom | But only DIB is installing that file | 17:18 |
| rm_work | ah | 17:18 |
| rm_work | should we recheck that? | 17:18 |
| rm_work | suse failed for who knows why | 17:18 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: DNM: Debug for distutils https://review.opendev.org/695947 | 17:21 |
| johnsom | Well, that patch will test our theory.... | 17:21 |
| johnsom | yeah, the suse job in DIB has been flakey for a long time | 17:22 |
| johnsom | At least we can jam in a work around until DIB releases if this is in fact the problem | 17:23 |
| *** takamatsu has quit IRC | 17:27 | |
| rm_work | wtf one of the voting tests already failed on the DIB CR | 17:31 |
| rm_work | two | 17:31 |
| rm_work | wtf is going on | 17:31 |
| rm_work | lol | 17:32 |
| rm_work | https://e78f58922a3ff4ca37c7-7f0e280361fb46971ab2d3794bb1099a.ssl.cf1.rackcdn.com/695823/2/check/dib-functests-xenial-python3/987ba3d/logs/apt-sources_test-sources.FAIL.log | 17:32 |
| rm_work | 2019-11-25 17:29:59.293 | File has unexpected size (1423910 != 1424218). Mirror sync in progress? [IP: 198.72.125.4 80] | 17:32 |
| *** takamatsu has joined #openstack-lbaas | 17:32 | |
| rm_work | fantastic :D | 17:32 |
| johnsom | lol, sigh | 17:34 |
| johnsom | I feel like the universe is telling us we should have taken the week off | 17:34 |
| rm_work | i have a feeling i'm going to be poking at this all through the holiday weekend | 17:45 |
| rm_work | so i might just take a nap while stuff rechecks | 17:45 |
| rm_work | >_< | 17:45 |
| johnsom | I will not really have access to look at stuff | 17:48 |
| *** tesseract has quit IRC | 17:57 | |
| johnsom | Darn, well, either my fix didn't work as expected, or that was not the issue | 18:00 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Use retry for AmphoraComputeConnectivityWait https://review.opendev.org/662791 | 18:03 |
| johnsom | Ok, it did fix it, but the upstream also does a remove that I didn't account for. | 18:04 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix gate issue with DIB https://review.opendev.org/695947 | 18:10 |
| johnsom | rm_work ^^^ that should fix that DIB issue temporarily until DIB releases. | 18:10 |
| johnsom | If you are disappearing today, let me know if you are cool with me +2/+W it in. | 18:11 |
| rm_work | yeah, merge away if it passes | 18:14 |
| rm_work | i'll be back in a couple hours | 18:14 |
| johnsom | Ok | 18:17 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix gate issue with DIB https://review.opendev.org/695947 | 18:18 |
| johnsom | Helps if I put the link in the commit message | 18:18 |
| rm_work | I bet I'll be back about the time tests actually pass :D | 18:31 |
| *** maciejjozefczyk has quit IRC | 19:01 | |
| johnsom | Hmm, so this will be interesting. The fix is working but grenade still blows up... | 19:33 |
| *** ccamposr has joined #openstack-lbaas | 19:41 | |
| *** ccamposr__ has quit IRC | 19:43 | |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Stop testing python 2 https://review.opendev.org/687370 | 19:46 |
| *** falencastro has joined #openstack-lbaas | 19:58 | |
| *** gcheresh has joined #openstack-lbaas | 20:06 | |
| *** servagem has quit IRC | 20:06 | |
| cgoncalves | DIB in the grenade job isn't picking up the change...? | 20:14 |
| cgoncalves | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bf3/695947/5/check/octavia-grenade/bf31ce2/logs/dib-build/amphora-x64-haproxy.qcow2.log | 20:14 |
| *** thobias_trevisan has joined #openstack-lbaas | 20:18 | |
| *** ataraday_ has joined #openstack-lbaas | 20:20 | |
| *** servagem has joined #openstack-lbaas | 20:23 | |
| *** colin- has joined #openstack-lbaas | 20:24 | |
| *** gcheresh_ has joined #openstack-lbaas | 20:28 | |
| *** gcheresh has quit IRC | 20:30 | |
| johnsom | Yeah, it checks out stable/train octavia which doesn't have my fix. I may pre-backport that fix so grenade will pass. | 20:38 |
| johnsom | I'm not sure why the pip version changed on the stable/train branch though, so I'm not sure why it's happening there. | 20:38 |
| johnsom | Maybe it's the packaged pip or devstack/grenade is installing a new version.... | 20:40 |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/train: Fix gate issue with DIB https://review.opendev.org/695982 | 20:44 |
| *** spatel has joined #openstack-lbaas | 21:15 | |
| openstackgerrit | Brian Haley proposed openstack/octavia-lib master: Stop testing python 2 https://review.opendev.org/693132 | 21:15 |
| *** spatel has quit IRC | 21:26 | |
| *** rcernin has joined #openstack-lbaas | 21:31 | |
| johnsom | lol, I bet there are a lot of very nervous admins today: "SSDs which were put into service at the same time will likely fail nearly simultaneously." https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00092491en_us | 21:36 |
| johnsom | My PSA for the day | 21:37 |
| johnsom | (public service announcement) | 21:37 |
| *** pcaruana has quit IRC | 21:38 | |
| *** gcheresh_ has quit IRC | 21:42 | |
| *** abaindur has joined #openstack-lbaas | 21:46 | |
| cgoncalves | grand! | 22:26 |
| johnsom | So it looks like my cherry pick of the DIB fix to train is working. Just five more jobs to finish. | 22:28 |
| johnsom | If someone else wants to +2 it great, otherwise I will +W once the tests finish | 22:28 |
| johnsom | https://review.opendev.org/#/c/695982 | 22:29 |
| *** KeithMnemonic has joined #openstack-lbaas | 22:41 | |
| *** TrevorV has quit IRC | 22:48 | |
| *** tkajinam has joined #openstack-lbaas | 23:09 | |
| *** abaindur has quit IRC | 23:10 | |
| *** abaindur has joined #openstack-lbaas | 23:11 | |
| *** abaindur has quit IRC | 23:11 | |
| *** abaindur has joined #openstack-lbaas | 23:11 | |
| *** abaindur has quit IRC | 23:12 | |
| *** abaindur has joined #openstack-lbaas | 23:13 | |
| *** ccamposr has quit IRC | 23:28 | |
| *** ccamposr has joined #openstack-lbaas | 23:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!