Friday, 2018-04-13

*** salv-orlando has joined #openstack-kuryr00:01
*** salv-orlando has quit IRC00:06
*** laddite378 has joined #openstack-kuryr00:24
*** laddite378 has quit IRC00:24
*** salv-orlando has joined #openstack-kuryr01:02
*** hongbin has joined #openstack-kuryr01:05
*** salv-orlando has quit IRC01:06
*** salv-orlando has joined #openstack-kuryr02:02
*** salv-orlando has quit IRC02:07
*** salv-orlando has joined #openstack-kuryr03:03
*** salv-orlando has quit IRC03:07
*** gcheresh_ has joined #openstack-kuryr04:01
*** salv-orlando has joined #openstack-kuryr04:04
*** salv-orlando has quit IRC04:08
*** hongbin has quit IRC04:11
*** caowei has joined #openstack-kuryr04:15
*** salv-orlando has joined #openstack-kuryr05:05
*** salv-orlando has quit IRC05:09
*** yboaron has joined #openstack-kuryr05:12
*** yboaron has quit IRC05:17
*** salv-orlando has joined #openstack-kuryr05:17
*** janonymous has joined #openstack-kuryr05:23
*** apuimedo has quit IRC06:43
*** janki has joined #openstack-kuryr06:49
dmelladodulek: loved your ascii at https://review.openstack.org/#/c/560951/07:07
dmelladoxD07:07
*** gcheresh_ has quit IRC07:08
*** jchhatbar has joined #openstack-kuryr07:09
*** janki has quit IRC07:12
openstackgerritDaniel Mellado proposed openstack/kuryr-tempest-plugin master: Skip service test  https://review.openstack.org/56114107:24
dulekdmellado: `dnf install cowsay`07:55
dmelladoheh07:55
dmelladodulek: in any case your 'rawr'07:56
dmelladomade it fail with pep807:56
dmelladoxD07:56
dmelladoand we're having odd issues with the service test so I'd say let's skip this for now to unlock the gates07:57
dulekdmellado: Uh, and Octavia still broken…07:57
dmelladoand debug it further afterwards07:57
dmelladoyep07:57
dmelladobut it's just that test being broken and I suspect it might get solved afterwards07:59
dmelladocheck my patch on07:59
dulekdmellado: So I have time until Zuul comments on 561141? Let's see if I'm able to debug that.07:59
dmelladohttps://review.openstack.org/#/c/561141/07:59
dmelladoyep08:00
dulekHm it looks transient.08:00
dmelladoyep08:01
dmelladothat's why I'd like to disable that one for now and have gcheresh investigate it08:01
*** garyloug has joined #openstack-kuryr08:01
dulekLooks like Kuryr creates the LB just fine.08:01
dulekI love this error message: http://logs.openstack.org/51/560951/1/check/kuryr-kubernetes-tempest-octavia/4aa8cb2/controller/logs/screen-o-hm.txt.gz#_Apr_12_19_54_59_16569408:03
dulek"THIS IS NOT GOOD"08:04
dmelladolooool08:04
dulekWhy not "I HAVE BAAAD FEELINGS ABOUT THIS"?08:04
dulekBasically what happens is Octavia cannot connect to Amphorae.08:04
dmelladodulek: if you follow tzumainn you'd say it's an EXCITING issue08:05
dmelladobut it's quite odd as I didn't see any changes on that as of lately, celebdor patch is still around on review08:05
dulekdmellado: Huh? It's a boring one! EXCITING was finding that infra's iptables rule that blocked traffic. Credits to celebdor.08:06
dulekdmellado: You've checked if Octavia's gates are fine?08:06
* dmellado needs a coffee08:07
dmelladolet's check that out08:07
* dulek needs to finally learn how to use infra's logstash.08:07
dmelladolol08:07
dmelladodulek: http://status.openstack.org/openstack-health/#/08:08
dulekdmellado: They're fine…08:08
dmelladohttp://status.openstack.org/openstack-health/#/g/project/openstack~2Foctavia08:08
dmelladoyeah08:08
dulekdmellado: Looks like their multinode jobs are on 50% failure ratio. I remember they're still using legacy jobs for tempest gates.08:09
dulekdmellado: Maybe the issue is in using non-legacy and their multinode does that and suffers the same?08:10
dmelladodulek: IIRC they're migrating that to zuulv3 native08:10
dmelladomultinode, I mean08:11
dmelladoyeah, it's all legacy for now08:11
dmelladobut that shouldn't affect the functionality08:11
dulekdmellado: Might be that they've fixed something in legacy only. Some configs or whatever.08:13
dmelladodulek: unless they 'fixed' anything on the devstack plugin related to the config08:13
dulekdmellado: I've wrote on #openstack-lbaas. If you know who we can ping there it'd useful. :)08:13
dmelladolegacy fixtures sould affect only on their gates08:13
dmelladooh, yeah08:13
dmelladolet me go there08:14
dulekdmellado: BTW - why gate failures always come in batches? We're fixing one and get another.08:17
dmelladodulek: yeah, the funny thing is that we were just so happy about being done with the infra work08:17
dmelladoand I need a coffee, for real08:19
dmelladoas I even forgot the decorators. before the skip_because08:19
dulekdmellado: File a bug along with skip patch. We don't want to forget about services being broken.08:22
dmelladoyep, makes sense08:23
celebdorThanks for the credits08:24
dmelladocelebdor: no worries, we do also have a lot of bad ones for you08:25
openstackgerritDaniel Mellado proposed openstack/kuryr-tempest-plugin master: Skip service test  https://review.openstack.org/56114108:26
*** yamamoto has quit IRC08:33
celebdorMeh08:33
celebdorDid the gate pass?08:34
celebdorI'm in some rh event08:34
dmelladoheh we do have an 'octaviagate' here08:34
dmelladocheck on the openstack-lbaas channel08:34
dmelladocelebdor: rh event? in bcn?08:44
*** yamamoto has joined #openstack-kuryr08:44
celebdorSome rh breakfast so the people get to to know who is who08:47
dmelladoyou just wanted to be lazy on friday xD08:47
*** yamamoto has quit IRC08:51
celebdorI just wanted to break isolation08:54
*** pcaruana has joined #openstack-kuryr09:12
dulekdmellado: Octavia's green on my recheck. The suggestion that it was VM or network performance issue makes sense now.09:22
dmelladodulek: huh09:28
dmelladoperformance?09:28
dmelladointeresting09:28
dulekdmellado: Yeah, folks noticed that it took Amphorae 25 seconds to answer a healthcheck.09:29
dulekWhich is ridiculous amount of time.09:29
dmelladoso then it looks like that issue was infra-blame09:31
dmelladobut it's interesting that it never showed up on octavia gates09:31
dmelladoso dulek09:33
dmelladolet's start the merge party09:33
dmelladofirst we'd need to get the devstack patch applied09:33
dulekdmellado: That would be best.09:33
dulekdmellado: Ah, no, still one Octavia gate failed. :(09:37
dmelladodulek: performance again?09:37
dmelladoif so that's deff *not* normal and we'd need to get in touch with infra-core09:38
dulekdmellado: I'm tempted to try increase timeout of both gate and waiting for connection in Tempest.09:38
dulekdmellado: I'm not 100% sure it's performance.09:38
dmelladodulek: could you try reproducing this on your 'favorite BM'?09:41
dmelladolet's skip the test while we investigate this09:41
dulekdmellado: I've already reviewed with +2 there.09:41
dmelladoawesome09:41
*** celebdor1 has joined #openstack-kuryr09:47
celebdor1dmellado: I tried running my patch in the server you lent me09:48
celebdor1and funnily enough, after creating the LB09:48
celebdor1it goes to error state and then offline09:48
celebdor1so it never gets to do the listener and so on09:48
dmelladoany logs on n-cpu?09:48
celebdor1I'll now look at it09:48
dmelladowe've been messing with the server quite a bit09:49
dmelladomaybe it's worth doing a clean install09:49
celebdor1dmellado: so much time :'(09:49
celebdor1it takes like an hour to stack09:49
celebdor1plus provision09:49
dmelladocelebdor1: just unstack, clean09:49
dmelladorm -rf ./opt/stack09:49
celebdor1ah, ok09:49
dmelladoand then go so it would reclone everything09:50
*** celebdor1 is now known as apuimedo09:50
dmelladoor RECLONE=True09:50
apuimedowhat have you been up to this morning?09:50
dmelladojust read the backlog09:50
dmelladodreaded gates09:50
dmelladodulek: + me = gatemasters09:50
dmelladoxD09:50
dmelladohow about you? I think your office is nicer, not so many salespersons xD09:51
apuimedoI think there is one09:58
apuimedoI don't have the backlog on the laptop09:59
apuimedocan you summarize in one line?09:59
dmelladoso, several gate issues due to the requirements on the tempest plugins being broken10:07
dmelladoalso, it looks like the infra might be having slowness issues so amphoraes are really slow and takes about half a minute to reply to the healthcheck10:08
dmelladothat makes the service goes haywire10:08
dmelladodulek: and me have several patches for fixing this and we'll investigate further10:08
dmelladoso, devstack patch10:08
dmelladohttps://review.openstack.org/#/c/560946/10:09
dmelladoafter this we'll start merging several patches to our kuryr-tempest-plugin10:09
dmelladoand we'll investigate the slowness issues10:09
apuimedodmellado: do you know what's going on with the RETRY exceeded crap?10:09
dmelladoRETRY?10:10
dmelladooh, on your patch10:10
apuimedoyeah10:10
dmelladoI can take a look if you can't figure it out10:10
dulekapuimedo: I was writing about this one.10:11
dulekLemme find it.10:11
dmelladodulek: devstack patch approved10:11
dulek\o/10:11
dulekapuimedo: Okay, I can't. But basically I suggest switching order of post_config and LIBS_FROM_GIT.10:12
dmelladooh, I do recall about that too10:12
dmelladojust swap those and see what happens10:12
dmelladoalso you can sacrifice a rooster if that doesn't help10:12
dmelladoxD10:12
dulekapuimedo: Because Ansible method that's supposed to convert this into local.conf failed.10:12
apuimedodulek: I already switched the order10:13
dulekapuimedo: At least that what I understood from logs.10:13
dulekOh?10:13
dulekI'm looking.10:13
* dmellado looks10:13
dulekapuimedo: Ah, crap.10:13
dulekapuimedo: http://logs.openstack.org/33/560433/6/check/kuryr-kubernetes-tempest-octavia/9fb2271/job-output.txt.gz#_2018-04-12_23_08_20_59154210:14
dulekapuimedo: Kinda cryptic, isn't it?10:14
dmelladodulek: that's quite weird10:15
dmelladolooks liek some kind of race condition10:15
dmelladoapuimedo: does this issue survive a recheck?10:15
dulekThis is the script doing that: https://github.com/openstack-dev/devstack/blob/master/roles/write-devstack-local-conf/library/devstack_local_conf.py10:16
apuimedodmellado: dulek: can you translate that to sw engineering from the infra language of mordor10:16
dulekapuimedo: This ansible module I've linked failed.10:16
dulekAnd it doesn't provide any logs. :(10:16
dmelladolemme check the octavia base you're having10:16
dmelladoit looks like it extends to all children10:17
apuimedodulek: I only see the link to devstack, not to ansible10:17
apuimedooh, right10:17
apuimedoit's ansible in devstack10:17
apuimedomaaaaaaan10:17
dulekapuimedo: Yup!10:17
apuimedolayers upon layers10:17
dulekapuimedo: Traversing all this is impossible without codesearch.openstack.org10:17
dulekSo I highly recommend it. :)10:17
dmelladohail the hound! xD10:17
apuimedodmellado: dulek: isn't this a bit bullshit?10:18
dulekHm?10:18
dmelladoman, did you delete irrelevant files again?10:18
apuimedoheh10:19
*** caowei has quit IRC10:19
apuimedothey suck10:19
apuimedoxD10:19
apuimedosorry about that I'll fix it10:19
dmelladohttp://paste.openstack.org/show/719134/10:19
apuimedodulek: dmellado: what I meant with the bullshit was. This module has no fucking documentation whatsoever10:19
dulekOh.10:19
apuimedoI thought it was a strong convention in ansible to document the modules on the same file so they get the docs nicely generated10:20
dulekOkay apuimedo, I think I got it.10:20
apuimedodulek: what is it?10:20
dulekapuimedo: Put it in devstack_localrc dict, not devstack_localconf10:20
dmelladooh, yeah10:21
dmelladoit's localrc10:21
dulekapuimedo: http://codesearch.openstack.org/?q=LIBS_FROM_GIT&i=nope&files=.*zuul.*&repos=10:21
apuimedodulek: which part? the LIBS_FROM_GIT or the neutron post config?10:21
apuimedoor both?10:21
dmelladothe libs from git10:21
dulekapuimedo: Yup, only LIBS_FROM_GIT.10:22
dmelladoi kinda missed that10:22
dmelladobut it ought to be there10:22
dmelladoyeah, I know about the documentation xD10:22
apuimedook10:24
dulekI'm not really sure what's the difference?10:24
apuimedodulek: it's on the documentation man10:24
dulekI thought localrc was long gone and local.conf is the new localrc10:24
apuimedoxD10:24
dmelladolol10:24
dmelladodulek: apuimedo seems to be related to the way it's parsed10:24
dulekUh, okay, whatever, let's get it fixed and Tempest-tested.10:25
dmelladoapuimedo: dulek: actually it's on the docs10:25
dmelladohttps://docs.openstack.org/devstack/latest/zuul_roles.html10:25
dmelladoxD10:25
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy  https://review.openstack.org/56043310:25
dulekdmellado: OH!10:25
apuimedodmellado: I was good to you and added the damned irrelevant files10:26
dmelladolol10:26
apuimedoso much trouble for something irrelevant10:26
dmelladodon't delete them again xD10:26
apuimedodmellado: I don't make promises on that10:26
ltomasbodulek, I'm getting this when installing with devstack (during kuryr-controller image creation): http://paste.openstack.org/show/719136/10:26
ltomasbodulek, have we raised any req that we need to include into the container?10:27
dmelladoltomasbo: if you make us fix another issue on devstack or the gates today we'll get you fired, I promise! xD10:27
ltomasbolol10:27
apuimedoltomasbo: or we will just move dmellado to the infra team10:27
dulekdmellado: Fired? That's an easy exit.10:27
apuimedosince it's where he spends all his time10:27
dmelladoLOL10:27
ltomasboxD10:28
dmelladodulek: we'll get ltomasbo under 'another manager'10:28
dulekltomasbo: Never seen this issue, but I fear the worst.10:28
dmelladoand you all know who I mean xD10:28
apuimedopsutil?10:29
dulekOkay, sorry.10:29
apuimedowhat the fuck10:29
dulekI've just laughed loud in my room.10:29
apuimedowait a fucking second10:29
dulekFUCK!10:29
apuimedodidn't I remove it?10:29
dulekOkay, I'm better now.10:29
dulekBasically - psutil was released an hour ago. xD10:29
dmelladodulek: LOL10:29
dulekThey must have a bug.10:29
apuimedodulek: hey, why the hell we have it on lower constraints?10:29
dmelladoagain, new releases breaking us? AGAIN x D10:30
apuimedoI removed that crap from our codebase a while ago10:30
apuimedoit was only a dep for like two days10:30
dulekapuimedo: Not sure, I'm checking.10:30
dmelladoltomasbo: you'll go under a manager where you can get bargains10:30
dmelladoxD xD xD10:30
ltomasbodmellado, I don't know of whom are you talking... but not sure I want to know...10:31
ltomasbodulek, apuimedo I'll test without the psutil req.10:31
*** dangtrinhnt has quit IRC10:32
*** janonymous has quit IRC10:33
dulekapuimedo: psutil must be downloaded by some other dep.10:33
dulekapuimedo: oslo.reports has it in requirements.10:35
dulekapuimedo: And oslo.reports was added before I've joined the team, not my fault! :D10:35
dulekltomasbo: https://github.com/giampaolo/psutil/issues/126810:36
dulekltomasbo: For now you can blacklist 5.4.410:36
ltomasboahh ok10:37
dmelladoon my env 5.4.3 gets installed10:39
dmelladowhat did you do ltomasbo ?10:39
dulekdmellado: Uh… It's in the container, I think it's installing without upper-constraints…10:39
dmelladodulek: hmmm probably yeah10:39
dmelladolet's check any another dep10:40
apuimedodmellado: https://imgur.com/a/9Y9Hv10:40
dmelladoapuimedo: LOL10:40
dulekapuimedo is using his Friday time correctly. :D10:40
dmelladoltomasbo: did you see that pic? xD10:40
apuimedodulek: devstack is slow man10:41
dmelladoltomasbo: could you share a pip freeze within the container?10:41
dulekdmellado: Just download kuryr/cni from DockerHub and check out. ;)10:41
dulekI've built it yesterday.10:41
ltomasboapuimedo, lol10:42
apuimedoI'm the resident GIMPer10:42
dmelladodulek: I'm just preparning ltomasbo for his new manager10:42
dmelladoxD10:42
apuimedodmellado: who's that gonna be?10:42
*** irenab has quit IRC10:42
dmelladoapuimedo: the one who has all the deals10:42
*** lihi has quit IRC10:43
*** snapiri has quit IRC10:43
*** oanson has quit IRC10:44
dulekOkay, you know what? I have some pierogis in the freezer and I'll just go to reheat and eat them. That should calm me down, because amount of broken stuff is too high for a single Friday.10:45
dmelladolol10:46
dmelladowhat is pierogis, dulek ?10:46
dulekdmellado: https://en.wikipedia.org/wiki/Pierogi :)10:47
dulekBTW this article seems to be pretty comprehensive. :D10:48
dulekltomasbo: BTW I've just built the container without issue.10:48
dmelladolol10:48
dmelladoeven legends10:48
ltomasbodulek, blacklisting 5.4.4, right?10:49
dmelladoltomasbo: nope10:51
dmelladocheck our conversation10:51
ltomasbodmellado, ??10:52
dmelladoit looks like the container might not be applying UC to its dependencies10:54
dmelladoif you have a working container could you paste a pip freeze out of it?10:54
dulekdmellado: I'm pretty sure it isn't.10:56
dulekltomasbo: Just for testing you can try adding `psutil!=5.4.4` into requirements.txt.10:56
dmelladodulek: then I wouldn't be getting psutil 5.4.3 out of a fresh tox10:56
dulekltomasbo: But I've just built container successfully. So I'm not sure what's happening.10:57
dulekdmellado: https://github.com/openstack/kuryr-kubernetes/blob/206c158d37060142d7d80d1be0564be10af576e7/cni.Dockerfile#L1310:57
dulekdmellado: That's why I'm pretty sure not UC are applied when creating the container.10:57
dulekdmellado: Because I simply haven't added that. :P10:57
dmelladodulek: lol10:57
dulekdmellado: We probably should.10:57
dmelladothen yeah, that's it10:57
dmelladojust modify the container adding -c there10:58
dulekBut still - I don't understand why I've just built the container and ltomasbo can't.10:58
dmelladojust like we did in the devstack patch10:58
dulekdmellado: Yup!10:58
ltomasbodulek, perhaps I'm missing some updates in my devstack10:59
ltomasbokuryr-kubernetes was just rebased, but I did not do the same for the other projects or devstack repo itself11:00
*** caowei has joined #openstack-kuryr11:03
dulekltomasbo: Naaah, shouldn't matter too much. Maybe besides requirements repo, but still.11:03
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to CNI Dockerfile  https://review.openstack.org/56119311:04
dmelladodulek: ^^11:04
dmelladothere you go xD11:04
ltomasbodmellado, what about the controller.Dockerfile?11:06
ltomasboxD seems dulek and I just replied the same...11:06
dulekdmellado: :D11:06
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles  https://review.openstack.org/56119311:07
dmelladolol11:07
dmelladoyeah, I missed it11:08
dmelladodone11:08
dmelladoxD11:08
apuimedodulek: the authors should be changed for the dockerfiles, you should pick over vikasc since he is no longer a maintainer of the containers11:09
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles  https://review.openstack.org/56119311:10
dmelladoltomasbo: dulek in the end I'll trust what shadower said11:10
dmelladotough crowd when it comes to reviews xD11:10
apuimedodmellado: you naughty, why did you remove the 'f' from rm -fr .git?11:10
apuimedoit is unrelated to the patch11:10
dmelladooh, did I?11:10
apuimedodmellado: you did11:11
apuimedohttps://review.openstack.org/#/c/561193/3/controller.Dockerfile11:11
apuimedoput it back!11:11
apuimedoI always like my 'rm' french11:11
dmelladoit was my vengeance for the irrelevant files11:11
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles  https://review.openstack.org/56119311:12
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles  https://review.openstack.org/56119311:21
dulekdmellado: You definitely need a coffee.11:21
dulekdmellado: See my comment. :D11:21
dmelladodulek: I know11:21
dmelladoI'm becoming even worse than apuimedo with these files11:22
dmelladoxD11:22
dmelladodulek: also, the allergy pills makes me sleepy xD11:22
dmelladoit's either sneeze or sleep11:22
dmelladoxD11:22
* dulek never had an allergy. Lucky guy!11:22
dulekdmellado: Anyway my pierogis came out of freezer all glued to each other and that haven't changed while cooking them. So I have multi-layered pierogi cake. Which sounds great, but edges are overcooked and the inside is still undercooked. :(11:24
dmelladoouch11:25
dmelladobut multilayered kinda recalls about openstack xD11:25
dulekdmellado: But I was supposed to calm down during lunch, not think about OpenStack! :D11:26
dmelladoLOL11:26
*** salv-orlando has quit IRC11:48
*** salv-orlando has joined #openstack-kuryr11:49
*** atoth has joined #openstack-kuryr11:51
*** salv-orlando has quit IRC11:53
*** salv-orlando has joined #openstack-kuryr12:02
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Modify maintainers of Dockerfiles  https://review.openstack.org/56120312:07
dulekltomasbo: Someone replied: https://github.com/giampaolo/psutil/issues/126812:12
dulekltomasbo: I completely don't understand why you have this issue. Maybe some cached image layers?12:13
ltomasbodulek, the fix they say is about python3 (pip3.4)12:16
dulekltomasbo: Whatever, fix is updating setuptools.12:17
dulekltomasbo: But when building you should have it fresh. So I really don't understand what happened.12:17
ltomasbodulek, yep, too much re-stacking I guess12:17
apuimedodulek: crispy and icy pierogi12:34
apuimedoyou are an innovator12:34
dulekapuimedo: WTF happened to that patch of yours again? :D12:43
dulekOctavia times out…12:43
dulekLBaaS is simply broken.12:43
dulekapuimedo: Okay, Octavia hangs on create_load_balancer_listener12:44
apuimedodulek: what?12:45
apuimedooh, of course12:45
apuimedobecause the damn amphora never goes active12:45
apuimedosame it happens to mainn's env12:45
apuimedo(and that one is created in heat, not in devstack)12:46
dulekYup, I know. But this sounds wrong to me.12:46
dulekLet's take a step back.12:46
dulekapuimedo: Here's the LBaaSV2 issue: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-lbaasv2/3baf69a/job-output.txt.gz#_2018-04-13_10_47_11_80166812:46
dulekI guess it's a simple mistake.12:47
dulekOkay, now Octavia.12:47
dulekI don't think we have the same issue in both DevStack and Mainn's env.12:48
dulekFirst is master, second is stable/queens, right?12:48
apuimedodulek: yup, that's a simple mistake12:48
dulekapuimedo: Also - folks from Octavia say that this warnings on connection to Amphorae are fine.12:49
dulekapuimedo: It takes a lot of time to boot it, it's expected.12:49
dulekImportant thing is that it comes online at the end: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-octavia/9412492/controller/logs/screen-o-cw.txt.gz#_Apr_13_11_11_18_70848512:49
dulekAt least that what I understand.12:49
apuimedodulek: that neutron-lbaas thing I fixed in another patchset and overwrote by mistake12:50
dulek:)12:50
apuimedodulek: is that the octavia worker log?12:52
duleko-cw12:54
dulekapuimedo: I don't anything about how Octavia works or is divided into services. D:12:54
apuimedoit is12:54
apuimedothat is the worker I guess12:54
apuimedothere's worker, health-manager and api12:55
apuimedoiirc12:55
dulekapuimedo: I start to think that error on your patch is different than the gate issue we have.12:55
dulekapuimedo: See - other gates failed in tempest, not on DevStack.12:55
*** caowei has quit IRC12:56
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy  https://review.openstack.org/56043312:58
apuimedodulek: well, does the gate issue happen in other patches?12:58
dulekapuimedo: Yup, Octavia gates are failing but due to timeout of tempest tests, not creating LB in DevStack.12:59
dulekapuimedo: And that issue on other patches is transient - happens on average on 1.5 gates out of 3.13:00
dulekapuimedo: Your has 100% hit ratio. ;)13:00
*** gcheresh_ has joined #openstack-kuryr13:00
dulekI'd bet python-openstackclient formats stuff differently than Neutron and we should monitor for LB status differently.13:01
apuimedodulek: I tried it manually13:09
apuimedoif you refer to the lb_state13:09
dulekapuimedo: I do.13:09
dulekOr maybe I'm not, just a sec.13:10
apuimedodulek: http://paste.openstack.org/show/719149/13:12
dulekapuimedo: It does wait in create_load_balancer_listener forever.13:12
apuimedodulek: cause the fucking LB doesn't go Active13:13
apuimedoit goes ERROR13:13
apuimedodoesn't it?13:13
dulekapuimedo: I might be able to check that. :D13:13
apuimedodulek: and operating status prolly goes "offline"13:13
apuimedosorry "OFFLINE"13:13
apuimedoit's nice because it's all uppercase13:14
dulekapuimedo: Shouldn't we then see anything on Octavia logs?13:14
dulekEh, a while ago it was possible to get the DB dump from gate logs.13:15
apuimedodid you check the health manager log?13:15
dulekapuimedo: I see that. This stream of WARNING happens always.13:16
dulekapuimedo: http://logs.openstack.org/60/556960/2/check/kuryr-kubernetes-tempest-daemon-octavia/f764a94/controller/logs/screen-o-cw.txt.gz13:17
dulekOh, wait that's not healthmanager. :P13:17
dulekapuimedo: The LB gets into ACTIVE: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-octavia/9412492/controller/logs/screen-o-cw.txt.gz#_Apr_13_11_11_18_70848513:18
dulek(this is from your patch now)13:18
dulekapuimedo: oscwrap loadbalancer show '' !!13:18
dulekapuimedo: You're not passing lb id.13:19
apuimedodamn... Really?13:19
apuimedopffff I knew refactoring into functions was a bad idea... I'm too accomodating to reviewers13:20
apuimedoxD13:20
dulekapuimedo: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-octavia/9412492/job-output.txt.gz#_2018-04-13_11_03_29_13490513:21
*** gcheresh_ has quit IRC13:21
dulekapuimedo: Ah13:21
dulekapuimedo: It'll be good now.13:22
dulekapuimedo: You've stopped passing algorithm there.13:22
dulekapuimedo: So last argument will be lb name and it'll be fine.13:22
dulekBut I've told you it's not Octavia. :D13:22
apuimedoalgorithm is for pool13:23
*** salv-orlando has quit IRC13:24
apuimedodulek: there's no way this is not from patchset 713:24
*** salv-orlando has joined #openstack-kuryr13:24
apuimedothere's patchset 8 for a while now with this fixed13:25
dulekapuimedo: It is from 7.13:25
dulekYes, that's what I've said, it should be fine now.13:25
apuimedobasically the issue was that the create_load_balancer_pool was copied from create_load_balancer_listener13:25
apuimedoand forgot to change the name13:25
apuimedobut I fixed that last night13:25
apuimedoand it was fixed on an earlier patchset13:25
dulekSure, sure. Glad we have one less Octavia issue.13:26
apuimedothen when I pushed for the LIBS_FROM_GIT today from another machine the error got pushed again13:26
apuimedobecause the method name issue I fixed from dani's server13:26
dulekapuimedo: Cut the number of machines. :D13:26
apuimedoand forgot to pull again from my laptop13:26
apuimedodulek: I'd like to13:26
*** salv-orlando has quit IRC13:28
dmelladoapuimedo: rebase hell xD13:29
dmelladoxD13:29
apuimedodmellado: and waiting an hour for each error detection does not help13:32
dmelladoyou tell me13:34
dmelladowe should go jaume's path13:34
dulekdmellado: 561141 is getting in now. Fingers crossed.14:04
dmelladodulek: yeah, even commented there14:05
dmelladoif it goes good this time we should be at least safe on the gates for now14:05
dulekThere's one fine thing about such a Friday. In the evening I'll go to the city center, find a nice crafty pub and first glass will taste great. ;)14:06
apuimedoI'll find a field of rocks and jump head first14:07
dulekapuimedo: BTW - your patch looks fine in the gate. It haven't caught dmellado patch disabling service test, so Octavia's failing, but DevStack installs fine.14:08
dmelladodulek: apuimedo in any case we'd need to investigate that after all this patch monkey balling14:08
dmelladodulek: lol, great plans xD14:08
apuimedodulek: why do we need to disable the service test with my fix?14:11
dulekapuimedo: Oh. Yeah, what's the point of testing your fix without that test…?14:11
dulekAwww. Anyway our gates with Octavia fail occasionally.14:11
dulekapuimedo: With amphorae returning 500.14:12
dulekapuimedo: And there are no logs for amphorae in gate results so we don't know what happens. \o/14:12
apuimedodulek: sorry if it's obvious... Why do they fail?14:12
dulekapuimedo: Answer above. :)14:12
apuimedoI feel like I may have missed something going on on #openstack-lbaas14:12
dulekapuimedo: We were discussing that in the morning. Like normal morning, not Catalan one.14:13
*** dougbtv_ has quit IRC14:19
apuimedoah14:21
apuimedodulek: and the result was?14:22
dulekapuimedo: Octavia folks were suspecting gate performance issue.14:22
dulekI've bought it in the morning, but not now.14:23
dulekapuimedo: http://logs.openstack.org/33/560433/8/check/kuryr-kubernetes-tempest-octavia/3416e38/controller/logs/screen-kuryr-kubernetes.txt.gz#_Apr_13_14_10_32_28069914:25
dulekapuimedo: There's an issue with LB Python code and Octavia.14:25
dulekapuimedo: We're getting 400.14:26
dulek"Unknown attribute for argument load_balancer.loadbalancer: tenant_id"14:26
dulekdmellado, apuimedo: Do we backport `docker exec` patch into stable/queens?14:30
dulekTechnically it's a bug and that venv thing wasn't really a good, portable idea.14:31
* dulek is asking in context of cleaning up upstream Dockerfiles.14:31
apuimedodulek: interesting!14:31
apuimedomaybe I missed something in the handler14:32
dulekapuimedo: I'd rather say APIs are not 100% compatible.14:32
dulekapuimedo: Oh, wait.14:32
dulekapuimedo: Shall it be project_id? :D14:32
apuimedodulek: probably14:33
* dulek checks if lbaas accepts that as well.14:33
dmelladoxD14:34
dmelladodulek: it'd be a good idea and an easy patch, so go for it14:34
dmelladoapuimedo: botched jobs again?14:34
*** kiennt26 has joined #openstack-kuryr14:34
apuimedodmellado: not sure yet14:34
apuimedobut likely14:34
apuimedoalright, we can move to use project_id14:36
apuimedohttps://developer.openstack.org/api-ref/network/v2/index.html#load-balancer-as-a-service-2-0-deprecated14:37
dulekapuimedo: I'm not sure how they've achieved that though - I don't see anything for project_id in https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/extensions/loadbalancerv2.py14:38
dulekBut okay, it's Neutron, it's magic.14:39
apuimedodulek: they have a decorator for this stuff14:40
apuimedoIIRC14:40
dulekapuimedo: Like I've said - M-A-G-I-C. ;)14:41
apuimedoI hate magic14:43
apuimedoin code14:43
dmelladoI hate magic in the gates14:46
dmelladothought https://review.openstack.org/#/c/561193/ had passed14:46
dmelladoand it didn't pick up a needed change so recheck again14:47
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy  https://review.openstack.org/56043314:47
dmelladodulek: apuimedo FYI openstack infra is doing odd things14:48
apuimedodmellado: now it says +1 on that patch you just said14:48
dmelladoapuimedo: yeah, I hit recheck14:48
dmelladoand it suddenly popped up there14:48
dmelladostating it took 1 hour14:48
dmelladoso it's weird14:48
apuimedodmellado: so it was my mistake in patchset 8 again. I thought octavia API being a superset and all would be true14:48
dmelladolooks like the publisher got stuck14:48
apuimedobut it is not14:49
openstackgerritMerged openstack/kuryr-tempest-plugin master: Skip service test  https://review.openstack.org/56114114:49
apuimedobecause lbaasv2 API accepts both tenant_id and project_id14:49
dmelladoand octavia doesn't14:49
dmelladoyeah14:49
apuimedonope14:49
dulek:)14:49
dmelladotenant_id has been legacy for quite a while, dude14:49
apuimedoliar liar pants on fire14:49
dulekIt's not too bad though.14:49
apuimedoanyway, no reason to keep using tenant_id14:49
dulekIt could be worse.14:49
apuimedoit *IS* part of the API dmellado14:49
*** openstackgerrit has quit IRC14:50
dmelladobut it *IS* recommended not to use it14:50
apuimedoyou either are a superset14:50
dmelladolol14:50
dmelladoopenstackgerrit went down14:50
apuimedoor a super fake14:50
dmelladosomething *IS* odd14:50
dmelladoxD14:50
apuimedodmellado: openstack is giving up on us14:50
dmelladodulek: what did you do? xD14:50
dulekdmellado: Gerrit's fine on my end.14:50
dmelladoopenstackgerrit (~openstack@review.openstack.org) has quit (Remote host closed the connection)14:51
dmelladosee ^^14:51
dmelladoxD14:51
dulekAh.14:55
dulekapuimedo: WTF, we were passing both tenant_id and project_id? :D15:04
dulekdmellado: And this Zuul result on your patch was probably result of my earlier recheck. I've did it because your patch haven't grabbed DevStack fix.15:04
dmelladoprobably yeah, it just took tooo long15:05
dmelladoand suddenly the gerrit went down so I had my concerns xD15:05
dmelladoin any case let's merge that one asap15:05
dmelladoand I'd be able to go for my beer on the weekend in a happy way15:05
*** livelace-link has joined #openstack-kuryr15:10
dmelladodulek: apuimedo would you do the honors?15:11
dmelladohttps://review.openstack.org/#/c/561193/15:11
dulekdmellado: Without waiting for your rechecks? :D15:12
dulekdmellado: Done.15:12
*** pcaruana has quit IRC15:12
dmelladodulek: yeah, it'll take too much time15:12
dmelladoxD15:12
dmelladoawesome15:12
dulekltomasbo: Ha, I've hit your psutil issue! Awesome!15:15
dmelladodulek: you too?15:15
*** gcheresh_ has joined #openstack-kuryr15:15
dmelladowell, it'd be fixed as soon as the patch gets merged15:15
dulekltomasbo: Happened on stable/queens, I guess setuputils are somehow capped there.15:15
dulekdmellado: Not on stable/queens, though we can backport.15:16
dulekdmellado: Oh wait, we can't. You haven't added bugreport. :D15:16
dmelladoyeah, that's my idea15:16
dmelladolol15:16
dmelladoxD15:16
dmelladono worries, we'll handle that15:16
dmelladobug + backport15:16
apuimedomissing bug report?!15:17
apuimedo*shame*15:17
dulekapuimedo: Shame on us, we gave him +2.15:17
dmelladoltomasbo: you write it, you like writing15:17
dmelladoshame on us all15:17
dmelladoxD15:17
dmelladodulek: how many issues did we have with the infra and so since yesterday?15:17
dmelladoI lost the count15:17
dulekOkay, going for a walk, will take a look here and +W anything passing Zuul in an hour or two.15:17
ltomasbodmellado, it is your patch! I have enough writing for this week!15:18
dmelladoltomasbo: LOL15:18
dulekdmellado: Too much. And Ocatvia one still hangs.15:18
dmelladoI was just kidding! xD15:18
apuimedoltomasbo: finish sf already!15:18
apuimedoxD15:18
ltomasboapuimedo, I think I'm done with writing, just remaining recording demos and prepare slides15:18
* ltomasbo fingers crossed15:19
dmelladoltomasbo: I'm pretty sure they'll break the testbed again15:19
dmelladoand we'll have to do more writing + reinstallation15:19
dmelladoI even had a nightmare about it xD15:19
ltomasboif we have the videos, I don't care anymore...15:19
dmelladoxD15:19
ltomasboapuimedo, dulek: I tried with octavia in my devstack and it fails too15:20
dmelladoloool dulek was fast enough to stop the merge15:20
dmelladoshame15:20
dmelladoxD15:21
ltomasboapuimedo, moving to lbaasv2, seems the subnet per namespace is working, just need to handle a couple of corner cases to support ports pool (remove subports when namespace is deleted)15:21
ltomasboapuimedo, and it will depend on your driver loading patch15:21
dmelladoltomasbo: what was the package name that you had the issue with?15:22
ltomasbopsutils15:22
ltomasbo(5.4.4)15:23
dmelladoack, thanks15:23
*** openstackgerrit has joined #openstack-kuryr15:25
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles  https://review.openstack.org/56119315:25
dmelladoI'll just +W this one, as it's just an amend in the code15:25
*** salv-orlando has joined #openstack-kuryr15:26
apuimedoltomasbo: slides?15:26
apuimedoltomasbo: which driver loading patch?15:26
apuimedodo you mean handler loading?15:27
dmelladoapuimedo: disregard that, SF stuff15:27
ltomasboapuimedo, slides for the final review meeting, we need to tell EC what we have done, and convince them we made good use of the money15:27
ltomasboapuimedo, yes, I meant handler loader (long week already...)15:27
apuimedook15:29
dmelladoltomasbo: 'convince them' actually sounds like we're ripping 'em off15:29
dmelladoxD15:29
apuimedoand you obviously aren't15:30
apuimedoyou fought the testbed gods15:30
ltomasboxD15:30
ltomasbowe did a lot of work actually15:30
ltomasbothough it was not well organized15:30
*** salv-orlando has quit IRC15:31
*** salv-orlando has joined #openstack-kuryr15:37
* dmellado goes off15:40
dmelladohave a nice weekend, folks!15:40
ltomasbohave a nice weekend!15:41
*** kiennt26 has quit IRC15:59
*** jchhatbar has quit IRC16:12
*** pcaruana has joined #openstack-kuryr16:15
*** jchhatbar has joined #openstack-kuryr16:20
*** apuimedo has quit IRC16:24
*** salv-orlando has quit IRC16:41
*** salv-orlando has joined #openstack-kuryr16:41
*** salv-orlando has quit IRC16:44
*** salv-orlando has joined #openstack-kuryr16:44
*** hongbin has joined #openstack-kuryr16:50
*** jchhatbar has quit IRC17:09
openstackgerritMerged openstack/kuryr-kubernetes master: Apply UC to Dockerfiles  https://review.openstack.org/56119317:15
*** atoth has quit IRC17:30
*** atoth has joined #openstack-kuryr17:32
dulekJust in case someone looks here over the weekend - Toni's patch looks green now and just waits for second +2: https://review.openstack.org/#/c/56043317:35
*** atoth has quit IRC18:11
*** apuimedo has joined #openstack-kuryr18:32
*** apuimedo has quit IRC18:57
*** gcheresh_ has quit IRC19:25
openstackgerritMerged openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy  https://review.openstack.org/56043319:54
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-tempest-plugin master: Revert "Skip service test"  https://review.openstack.org/56136422:08
*** garyloug has quit IRC22:28
*** hongbin has quit IRC23:13

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!