*** salv-orlando has joined #openstack-kuryr | 00:01 | |
*** salv-orlando has quit IRC | 00:06 | |
*** laddite378 has joined #openstack-kuryr | 00:24 | |
*** laddite378 has quit IRC | 00:24 | |
*** salv-orlando has joined #openstack-kuryr | 01:02 | |
*** hongbin has joined #openstack-kuryr | 01:05 | |
*** salv-orlando has quit IRC | 01:06 | |
*** salv-orlando has joined #openstack-kuryr | 02:02 | |
*** salv-orlando has quit IRC | 02:07 | |
*** salv-orlando has joined #openstack-kuryr | 03:03 | |
*** salv-orlando has quit IRC | 03:07 | |
*** gcheresh_ has joined #openstack-kuryr | 04:01 | |
*** salv-orlando has joined #openstack-kuryr | 04:04 | |
*** salv-orlando has quit IRC | 04:08 | |
*** hongbin has quit IRC | 04:11 | |
*** caowei has joined #openstack-kuryr | 04:15 | |
*** salv-orlando has joined #openstack-kuryr | 05:05 | |
*** salv-orlando has quit IRC | 05:09 | |
*** yboaron has joined #openstack-kuryr | 05:12 | |
*** yboaron has quit IRC | 05:17 | |
*** salv-orlando has joined #openstack-kuryr | 05:17 | |
*** janonymous has joined #openstack-kuryr | 05:23 | |
*** apuimedo has quit IRC | 06:43 | |
*** janki has joined #openstack-kuryr | 06:49 | |
dmellado | dulek: loved your ascii at https://review.openstack.org/#/c/560951/ | 07:07 |
---|---|---|
dmellado | xD | 07:07 |
*** gcheresh_ has quit IRC | 07:08 | |
*** jchhatbar has joined #openstack-kuryr | 07:09 | |
*** janki has quit IRC | 07:12 | |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Skip service test https://review.openstack.org/561141 | 07:24 |
dulek | dmellado: `dnf install cowsay` | 07:55 |
dmellado | heh | 07:55 |
dmellado | dulek: in any case your 'rawr' | 07:56 |
dmellado | made it fail with pep8 | 07:56 |
dmellado | xD | 07:56 |
dmellado | and we're having odd issues with the service test so I'd say let's skip this for now to unlock the gates | 07:57 |
dulek | dmellado: Uh, and Octavia still broken… | 07:57 |
dmellado | and debug it further afterwards | 07:57 |
dmellado | yep | 07:57 |
dmellado | but it's just that test being broken and I suspect it might get solved afterwards | 07:59 |
dmellado | check my patch on | 07:59 |
dulek | dmellado: So I have time until Zuul comments on 561141? Let's see if I'm able to debug that. | 07:59 |
dmellado | https://review.openstack.org/#/c/561141/ | 07:59 |
dmellado | yep | 08:00 |
dulek | Hm it looks transient. | 08:00 |
dmellado | yep | 08:01 |
dmellado | that's why I'd like to disable that one for now and have gcheresh investigate it | 08:01 |
*** garyloug has joined #openstack-kuryr | 08:01 | |
dulek | Looks like Kuryr creates the LB just fine. | 08:01 |
dulek | I love this error message: http://logs.openstack.org/51/560951/1/check/kuryr-kubernetes-tempest-octavia/4aa8cb2/controller/logs/screen-o-hm.txt.gz#_Apr_12_19_54_59_165694 | 08:03 |
dulek | "THIS IS NOT GOOD" | 08:04 |
dmellado | looool | 08:04 |
dulek | Why not "I HAVE BAAAD FEELINGS ABOUT THIS"? | 08:04 |
dulek | Basically what happens is Octavia cannot connect to Amphorae. | 08:04 |
dmellado | dulek: if you follow tzumainn you'd say it's an EXCITING issue | 08:05 |
dmellado | but it's quite odd as I didn't see any changes on that as of lately, celebdor patch is still around on review | 08:05 |
dulek | dmellado: Huh? It's a boring one! EXCITING was finding that infra's iptables rule that blocked traffic. Credits to celebdor. | 08:06 |
dulek | dmellado: You've checked if Octavia's gates are fine? | 08:06 |
* dmellado needs a coffee | 08:07 | |
dmellado | let's check that out | 08:07 |
* dulek needs to finally learn how to use infra's logstash. | 08:07 | |
dmellado | lol | 08:07 |
dmellado | dulek: http://status.openstack.org/openstack-health/#/ | 08:08 |
dulek | dmellado: They're fine… | 08:08 |
dmellado | http://status.openstack.org/openstack-health/#/g/project/openstack~2Foctavia | 08:08 |
dmellado | yeah | 08:08 |
dulek | dmellado: Looks like their multinode jobs are on 50% failure ratio. I remember they're still using legacy jobs for tempest gates. | 08:09 |
dulek | dmellado: Maybe the issue is in using non-legacy and their multinode does that and suffers the same? | 08:10 |
dmellado | dulek: IIRC they're migrating that to zuulv3 native | 08:10 |
dmellado | multinode, I mean | 08:11 |
dmellado | yeah, it's all legacy for now | 08:11 |
dmellado | but that shouldn't affect the functionality | 08:11 |
dulek | dmellado: Might be that they've fixed something in legacy only. Some configs or whatever. | 08:13 |
dmellado | dulek: unless they 'fixed' anything on the devstack plugin related to the config | 08:13 |
dulek | dmellado: I've wrote on #openstack-lbaas. If you know who we can ping there it'd useful. :) | 08:13 |
dmellado | legacy fixtures sould affect only on their gates | 08:13 |
dmellado | oh, yeah | 08:13 |
dmellado | let me go there | 08:14 |
dulek | dmellado: BTW - why gate failures always come in batches? We're fixing one and get another. | 08:17 |
dmellado | dulek: yeah, the funny thing is that we were just so happy about being done with the infra work | 08:17 |
dmellado | and I need a coffee, for real | 08:19 |
dmellado | as I even forgot the decorators. before the skip_because | 08:19 |
dulek | dmellado: File a bug along with skip patch. We don't want to forget about services being broken. | 08:22 |
dmellado | yep, makes sense | 08:23 |
celebdor | Thanks for the credits | 08:24 |
dmellado | celebdor: no worries, we do also have a lot of bad ones for you | 08:25 |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Skip service test https://review.openstack.org/561141 | 08:26 |
*** yamamoto has quit IRC | 08:33 | |
celebdor | Meh | 08:33 |
celebdor | Did the gate pass? | 08:34 |
celebdor | I'm in some rh event | 08:34 |
dmellado | heh we do have an 'octaviagate' here | 08:34 |
dmellado | check on the openstack-lbaas channel | 08:34 |
dmellado | celebdor: rh event? in bcn? | 08:44 |
*** yamamoto has joined #openstack-kuryr | 08:44 | |
celebdor | Some rh breakfast so the people get to to know who is who | 08:47 |
dmellado | you just wanted to be lazy on friday xD | 08:47 |
*** yamamoto has quit IRC | 08:51 | |
celebdor | I just wanted to break isolation | 08:54 |
*** pcaruana has joined #openstack-kuryr | 09:12 | |
dulek | dmellado: Octavia's green on my recheck. The suggestion that it was VM or network performance issue makes sense now. | 09:22 |
dmellado | dulek: huh | 09:28 |
dmellado | performance? | 09:28 |
dmellado | interesting | 09:28 |
dulek | dmellado: Yeah, folks noticed that it took Amphorae 25 seconds to answer a healthcheck. | 09:29 |
dulek | Which is ridiculous amount of time. | 09:29 |
dmellado | so then it looks like that issue was infra-blame | 09:31 |
dmellado | but it's interesting that it never showed up on octavia gates | 09:31 |
dmellado | so dulek | 09:33 |
dmellado | let's start the merge party | 09:33 |
dmellado | first we'd need to get the devstack patch applied | 09:33 |
dulek | dmellado: That would be best. | 09:33 |
dulek | dmellado: Ah, no, still one Octavia gate failed. :( | 09:37 |
dmellado | dulek: performance again? | 09:37 |
dmellado | if so that's deff *not* normal and we'd need to get in touch with infra-core | 09:38 |
dulek | dmellado: I'm tempted to try increase timeout of both gate and waiting for connection in Tempest. | 09:38 |
dulek | dmellado: I'm not 100% sure it's performance. | 09:38 |
dmellado | dulek: could you try reproducing this on your 'favorite BM'? | 09:41 |
dmellado | let's skip the test while we investigate this | 09:41 |
dulek | dmellado: I've already reviewed with +2 there. | 09:41 |
dmellado | awesome | 09:41 |
*** celebdor1 has joined #openstack-kuryr | 09:47 | |
celebdor1 | dmellado: I tried running my patch in the server you lent me | 09:48 |
celebdor1 | and funnily enough, after creating the LB | 09:48 |
celebdor1 | it goes to error state and then offline | 09:48 |
celebdor1 | so it never gets to do the listener and so on | 09:48 |
dmellado | any logs on n-cpu? | 09:48 |
celebdor1 | I'll now look at it | 09:48 |
dmellado | we've been messing with the server quite a bit | 09:49 |
dmellado | maybe it's worth doing a clean install | 09:49 |
celebdor1 | dmellado: so much time :'( | 09:49 |
celebdor1 | it takes like an hour to stack | 09:49 |
celebdor1 | plus provision | 09:49 |
dmellado | celebdor1: just unstack, clean | 09:49 |
dmellado | rm -rf ./opt/stack | 09:49 |
celebdor1 | ah, ok | 09:49 |
dmellado | and then go so it would reclone everything | 09:50 |
*** celebdor1 is now known as apuimedo | 09:50 | |
dmellado | or RECLONE=True | 09:50 |
apuimedo | what have you been up to this morning? | 09:50 |
dmellado | just read the backlog | 09:50 |
dmellado | dreaded gates | 09:50 |
dmellado | dulek: + me = gatemasters | 09:50 |
dmellado | xD | 09:50 |
dmellado | how about you? I think your office is nicer, not so many salespersons xD | 09:51 |
apuimedo | I think there is one | 09:58 |
apuimedo | I don't have the backlog on the laptop | 09:59 |
apuimedo | can you summarize in one line? | 09:59 |
dmellado | so, several gate issues due to the requirements on the tempest plugins being broken | 10:07 |
dmellado | also, it looks like the infra might be having slowness issues so amphoraes are really slow and takes about half a minute to reply to the healthcheck | 10:08 |
dmellado | that makes the service goes haywire | 10:08 |
dmellado | dulek: and me have several patches for fixing this and we'll investigate further | 10:08 |
dmellado | so, devstack patch | 10:08 |
dmellado | https://review.openstack.org/#/c/560946/ | 10:09 |
dmellado | after this we'll start merging several patches to our kuryr-tempest-plugin | 10:09 |
dmellado | and we'll investigate the slowness issues | 10:09 |
apuimedo | dmellado: do you know what's going on with the RETRY exceeded crap? | 10:09 |
dmellado | RETRY? | 10:10 |
dmellado | oh, on your patch | 10:10 |
apuimedo | yeah | 10:10 |
dmellado | I can take a look if you can't figure it out | 10:10 |
dulek | apuimedo: I was writing about this one. | 10:11 |
dulek | Lemme find it. | 10:11 |
dmellado | dulek: devstack patch approved | 10:11 |
dulek | \o/ | 10:11 |
dulek | apuimedo: Okay, I can't. But basically I suggest switching order of post_config and LIBS_FROM_GIT. | 10:12 |
dmellado | oh, I do recall about that too | 10:12 |
dmellado | just swap those and see what happens | 10:12 |
dmellado | also you can sacrifice a rooster if that doesn't help | 10:12 |
dmellado | xD | 10:12 |
dulek | apuimedo: Because Ansible method that's supposed to convert this into local.conf failed. | 10:12 |
apuimedo | dulek: I already switched the order | 10:13 |
dulek | apuimedo: At least that what I understood from logs. | 10:13 |
dulek | Oh? | 10:13 |
dulek | I'm looking. | 10:13 |
* dmellado looks | 10:13 | |
dulek | apuimedo: Ah, crap. | 10:13 |
dulek | apuimedo: http://logs.openstack.org/33/560433/6/check/kuryr-kubernetes-tempest-octavia/9fb2271/job-output.txt.gz#_2018-04-12_23_08_20_591542 | 10:14 |
dulek | apuimedo: Kinda cryptic, isn't it? | 10:14 |
dmellado | dulek: that's quite weird | 10:15 |
dmellado | looks liek some kind of race condition | 10:15 |
dmellado | apuimedo: does this issue survive a recheck? | 10:15 |
dulek | This is the script doing that: https://github.com/openstack-dev/devstack/blob/master/roles/write-devstack-local-conf/library/devstack_local_conf.py | 10:16 |
apuimedo | dmellado: dulek: can you translate that to sw engineering from the infra language of mordor | 10:16 |
dulek | apuimedo: This ansible module I've linked failed. | 10:16 |
dulek | And it doesn't provide any logs. :( | 10:16 |
dmellado | lemme check the octavia base you're having | 10:16 |
dmellado | it looks like it extends to all children | 10:17 |
apuimedo | dulek: I only see the link to devstack, not to ansible | 10:17 |
apuimedo | oh, right | 10:17 |
apuimedo | it's ansible in devstack | 10:17 |
apuimedo | maaaaaaan | 10:17 |
dulek | apuimedo: Yup! | 10:17 |
apuimedo | layers upon layers | 10:17 |
dulek | apuimedo: Traversing all this is impossible without codesearch.openstack.org | 10:17 |
dulek | So I highly recommend it. :) | 10:17 |
dmellado | hail the hound! xD | 10:17 |
apuimedo | dmellado: dulek: isn't this a bit bullshit? | 10:18 |
dulek | Hm? | 10:18 |
dmellado | man, did you delete irrelevant files again? | 10:18 |
apuimedo | heh | 10:19 |
*** caowei has quit IRC | 10:19 | |
apuimedo | they suck | 10:19 |
apuimedo | xD | 10:19 |
apuimedo | sorry about that I'll fix it | 10:19 |
dmellado | http://paste.openstack.org/show/719134/ | 10:19 |
apuimedo | dulek: dmellado: what I meant with the bullshit was. This module has no fucking documentation whatsoever | 10:19 |
dulek | Oh. | 10:19 |
apuimedo | I thought it was a strong convention in ansible to document the modules on the same file so they get the docs nicely generated | 10:20 |
dulek | Okay apuimedo, I think I got it. | 10:20 |
apuimedo | dulek: what is it? | 10:20 |
dulek | apuimedo: Put it in devstack_localrc dict, not devstack_localconf | 10:20 |
dmellado | oh, yeah | 10:21 |
dmellado | it's localrc | 10:21 |
dulek | apuimedo: http://codesearch.openstack.org/?q=LIBS_FROM_GIT&i=nope&files=.*zuul.*&repos= | 10:21 |
apuimedo | dulek: which part? the LIBS_FROM_GIT or the neutron post config? | 10:21 |
apuimedo | or both? | 10:21 |
dmellado | the libs from git | 10:21 |
dulek | apuimedo: Yup, only LIBS_FROM_GIT. | 10:22 |
dmellado | i kinda missed that | 10:22 |
dmellado | but it ought to be there | 10:22 |
dmellado | yeah, I know about the documentation xD | 10:22 |
apuimedo | ok | 10:24 |
dulek | I'm not really sure what's the difference? | 10:24 |
apuimedo | dulek: it's on the documentation man | 10:24 |
dulek | I thought localrc was long gone and local.conf is the new localrc | 10:24 |
apuimedo | xD | 10:24 |
dmellado | lol | 10:24 |
dmellado | dulek: apuimedo seems to be related to the way it's parsed | 10:24 |
dulek | Uh, okay, whatever, let's get it fixed and Tempest-tested. | 10:25 |
dmellado | apuimedo: dulek: actually it's on the docs | 10:25 |
dmellado | https://docs.openstack.org/devstack/latest/zuul_roles.html | 10:25 |
dmellado | xD | 10:25 |
openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy https://review.openstack.org/560433 | 10:25 |
dulek | dmellado: OH! | 10:25 |
apuimedo | dmellado: I was good to you and added the damned irrelevant files | 10:26 |
dmellado | lol | 10:26 |
apuimedo | so much trouble for something irrelevant | 10:26 |
dmellado | don't delete them again xD | 10:26 |
apuimedo | dmellado: I don't make promises on that | 10:26 |
ltomasbo | dulek, I'm getting this when installing with devstack (during kuryr-controller image creation): http://paste.openstack.org/show/719136/ | 10:26 |
ltomasbo | dulek, have we raised any req that we need to include into the container? | 10:27 |
dmellado | ltomasbo: if you make us fix another issue on devstack or the gates today we'll get you fired, I promise! xD | 10:27 |
ltomasbo | lol | 10:27 |
apuimedo | ltomasbo: or we will just move dmellado to the infra team | 10:27 |
dulek | dmellado: Fired? That's an easy exit. | 10:27 |
apuimedo | since it's where he spends all his time | 10:27 |
dmellado | LOL | 10:27 |
ltomasbo | xD | 10:28 |
dmellado | dulek: we'll get ltomasbo under 'another manager' | 10:28 |
dulek | ltomasbo: Never seen this issue, but I fear the worst. | 10:28 |
dmellado | and you all know who I mean xD | 10:28 |
apuimedo | psutil? | 10:29 |
dulek | Okay, sorry. | 10:29 |
apuimedo | what the fuck | 10:29 |
dulek | I've just laughed loud in my room. | 10:29 |
apuimedo | wait a fucking second | 10:29 |
dulek | FUCK! | 10:29 |
apuimedo | didn't I remove it? | 10:29 |
dulek | Okay, I'm better now. | 10:29 |
dulek | Basically - psutil was released an hour ago. xD | 10:29 |
dmellado | dulek: LOL | 10:29 |
dulek | They must have a bug. | 10:29 |
apuimedo | dulek: hey, why the hell we have it on lower constraints? | 10:29 |
dmellado | again, new releases breaking us? AGAIN x D | 10:30 |
apuimedo | I removed that crap from our codebase a while ago | 10:30 |
apuimedo | it was only a dep for like two days | 10:30 |
dulek | apuimedo: Not sure, I'm checking. | 10:30 |
dmellado | ltomasbo: you'll go under a manager where you can get bargains | 10:30 |
dmellado | xD xD xD | 10:30 |
ltomasbo | dmellado, I don't know of whom are you talking... but not sure I want to know... | 10:31 |
ltomasbo | dulek, apuimedo I'll test without the psutil req. | 10:31 |
*** dangtrinhnt has quit IRC | 10:32 | |
*** janonymous has quit IRC | 10:33 | |
dulek | apuimedo: psutil must be downloaded by some other dep. | 10:33 |
dulek | apuimedo: oslo.reports has it in requirements. | 10:35 |
dulek | apuimedo: And oslo.reports was added before I've joined the team, not my fault! :D | 10:35 |
dulek | ltomasbo: https://github.com/giampaolo/psutil/issues/1268 | 10:36 |
dulek | ltomasbo: For now you can blacklist 5.4.4 | 10:36 |
ltomasbo | ahh ok | 10:37 |
dmellado | on my env 5.4.3 gets installed | 10:39 |
dmellado | what did you do ltomasbo ? | 10:39 |
dulek | dmellado: Uh… It's in the container, I think it's installing without upper-constraints… | 10:39 |
dmellado | dulek: hmmm probably yeah | 10:39 |
dmellado | let's check any another dep | 10:40 |
apuimedo | dmellado: https://imgur.com/a/9Y9Hv | 10:40 |
dmellado | apuimedo: LOL | 10:40 |
dulek | apuimedo is using his Friday time correctly. :D | 10:40 |
dmellado | ltomasbo: did you see that pic? xD | 10:40 |
apuimedo | dulek: devstack is slow man | 10:41 |
dmellado | ltomasbo: could you share a pip freeze within the container? | 10:41 |
dulek | dmellado: Just download kuryr/cni from DockerHub and check out. ;) | 10:41 |
dulek | I've built it yesterday. | 10:41 |
ltomasbo | apuimedo, lol | 10:42 |
apuimedo | I'm the resident GIMPer | 10:42 |
dmellado | dulek: I'm just preparning ltomasbo for his new manager | 10:42 |
dmellado | xD | 10:42 |
apuimedo | dmellado: who's that gonna be? | 10:42 |
*** irenab has quit IRC | 10:42 | |
dmellado | apuimedo: the one who has all the deals | 10:42 |
*** lihi has quit IRC | 10:43 | |
*** snapiri has quit IRC | 10:43 | |
*** oanson has quit IRC | 10:44 | |
dulek | Okay, you know what? I have some pierogis in the freezer and I'll just go to reheat and eat them. That should calm me down, because amount of broken stuff is too high for a single Friday. | 10:45 |
dmellado | lol | 10:46 |
dmellado | what is pierogis, dulek ? | 10:46 |
dulek | dmellado: https://en.wikipedia.org/wiki/Pierogi :) | 10:47 |
dulek | BTW this article seems to be pretty comprehensive. :D | 10:48 |
dulek | ltomasbo: BTW I've just built the container without issue. | 10:48 |
dmellado | lol | 10:48 |
dmellado | even legends | 10:48 |
ltomasbo | dulek, blacklisting 5.4.4, right? | 10:49 |
dmellado | ltomasbo: nope | 10:51 |
dmellado | check our conversation | 10:51 |
ltomasbo | dmellado, ?? | 10:52 |
dmellado | it looks like the container might not be applying UC to its dependencies | 10:54 |
dmellado | if you have a working container could you paste a pip freeze out of it? | 10:54 |
dulek | dmellado: I'm pretty sure it isn't. | 10:56 |
dulek | ltomasbo: Just for testing you can try adding `psutil!=5.4.4` into requirements.txt. | 10:56 |
dmellado | dulek: then I wouldn't be getting psutil 5.4.3 out of a fresh tox | 10:56 |
dulek | ltomasbo: But I've just built container successfully. So I'm not sure what's happening. | 10:57 |
dulek | dmellado: https://github.com/openstack/kuryr-kubernetes/blob/206c158d37060142d7d80d1be0564be10af576e7/cni.Dockerfile#L13 | 10:57 |
dulek | dmellado: That's why I'm pretty sure not UC are applied when creating the container. | 10:57 |
dulek | dmellado: Because I simply haven't added that. :P | 10:57 |
dmellado | dulek: lol | 10:57 |
dulek | dmellado: We probably should. | 10:57 |
dmellado | then yeah, that's it | 10:57 |
dmellado | just modify the container adding -c there | 10:58 |
dulek | But still - I don't understand why I've just built the container and ltomasbo can't. | 10:58 |
dmellado | just like we did in the devstack patch | 10:58 |
dulek | dmellado: Yup! | 10:58 |
ltomasbo | dulek, perhaps I'm missing some updates in my devstack | 10:59 |
ltomasbo | kuryr-kubernetes was just rebased, but I did not do the same for the other projects or devstack repo itself | 11:00 |
*** caowei has joined #openstack-kuryr | 11:03 | |
dulek | ltomasbo: Naaah, shouldn't matter too much. Maybe besides requirements repo, but still. | 11:03 |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to CNI Dockerfile https://review.openstack.org/561193 | 11:04 |
dmellado | dulek: ^^ | 11:04 |
dmellado | there you go xD | 11:04 |
ltomasbo | dmellado, what about the controller.Dockerfile? | 11:06 |
ltomasbo | xD seems dulek and I just replied the same... | 11:06 |
dulek | dmellado: :D | 11:06 |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles https://review.openstack.org/561193 | 11:07 |
dmellado | lol | 11:07 |
dmellado | yeah, I missed it | 11:08 |
dmellado | done | 11:08 |
dmellado | xD | 11:08 |
apuimedo | dulek: the authors should be changed for the dockerfiles, you should pick over vikasc since he is no longer a maintainer of the containers | 11:09 |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles https://review.openstack.org/561193 | 11:10 |
dmellado | ltomasbo: dulek in the end I'll trust what shadower said | 11:10 |
dmellado | tough crowd when it comes to reviews xD | 11:10 |
apuimedo | dmellado: you naughty, why did you remove the 'f' from rm -fr .git? | 11:10 |
apuimedo | it is unrelated to the patch | 11:10 |
dmellado | oh, did I? | 11:10 |
apuimedo | dmellado: you did | 11:11 |
apuimedo | https://review.openstack.org/#/c/561193/3/controller.Dockerfile | 11:11 |
apuimedo | put it back! | 11:11 |
apuimedo | I always like my 'rm' french | 11:11 |
dmellado | it was my vengeance for the irrelevant files | 11:11 |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles https://review.openstack.org/561193 | 11:12 |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles https://review.openstack.org/561193 | 11:21 |
dulek | dmellado: You definitely need a coffee. | 11:21 |
dulek | dmellado: See my comment. :D | 11:21 |
dmellado | dulek: I know | 11:21 |
dmellado | I'm becoming even worse than apuimedo with these files | 11:22 |
dmellado | xD | 11:22 |
dmellado | dulek: also, the allergy pills makes me sleepy xD | 11:22 |
dmellado | it's either sneeze or sleep | 11:22 |
dmellado | xD | 11:22 |
* dulek never had an allergy. Lucky guy! | 11:22 | |
dulek | dmellado: Anyway my pierogis came out of freezer all glued to each other and that haven't changed while cooking them. So I have multi-layered pierogi cake. Which sounds great, but edges are overcooked and the inside is still undercooked. :( | 11:24 |
dmellado | ouch | 11:25 |
dmellado | but multilayered kinda recalls about openstack xD | 11:25 |
dulek | dmellado: But I was supposed to calm down during lunch, not think about OpenStack! :D | 11:26 |
dmellado | LOL | 11:26 |
*** salv-orlando has quit IRC | 11:48 | |
*** salv-orlando has joined #openstack-kuryr | 11:49 | |
*** atoth has joined #openstack-kuryr | 11:51 | |
*** salv-orlando has quit IRC | 11:53 | |
*** salv-orlando has joined #openstack-kuryr | 12:02 | |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Modify maintainers of Dockerfiles https://review.openstack.org/561203 | 12:07 |
dulek | ltomasbo: Someone replied: https://github.com/giampaolo/psutil/issues/1268 | 12:12 |
dulek | ltomasbo: I completely don't understand why you have this issue. Maybe some cached image layers? | 12:13 |
ltomasbo | dulek, the fix they say is about python3 (pip3.4) | 12:16 |
dulek | ltomasbo: Whatever, fix is updating setuptools. | 12:17 |
dulek | ltomasbo: But when building you should have it fresh. So I really don't understand what happened. | 12:17 |
ltomasbo | dulek, yep, too much re-stacking I guess | 12:17 |
apuimedo | dulek: crispy and icy pierogi | 12:34 |
apuimedo | you are an innovator | 12:34 |
dulek | apuimedo: WTF happened to that patch of yours again? :D | 12:43 |
dulek | Octavia times out… | 12:43 |
dulek | LBaaS is simply broken. | 12:43 |
dulek | apuimedo: Okay, Octavia hangs on create_load_balancer_listener | 12:44 |
apuimedo | dulek: what? | 12:45 |
apuimedo | oh, of course | 12:45 |
apuimedo | because the damn amphora never goes active | 12:45 |
apuimedo | same it happens to mainn's env | 12:45 |
apuimedo | (and that one is created in heat, not in devstack) | 12:46 |
dulek | Yup, I know. But this sounds wrong to me. | 12:46 |
dulek | Let's take a step back. | 12:46 |
dulek | apuimedo: Here's the LBaaSV2 issue: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-lbaasv2/3baf69a/job-output.txt.gz#_2018-04-13_10_47_11_801668 | 12:46 |
dulek | I guess it's a simple mistake. | 12:47 |
dulek | Okay, now Octavia. | 12:47 |
dulek | I don't think we have the same issue in both DevStack and Mainn's env. | 12:48 |
dulek | First is master, second is stable/queens, right? | 12:48 |
apuimedo | dulek: yup, that's a simple mistake | 12:48 |
dulek | apuimedo: Also - folks from Octavia say that this warnings on connection to Amphorae are fine. | 12:49 |
dulek | apuimedo: It takes a lot of time to boot it, it's expected. | 12:49 |
dulek | Important thing is that it comes online at the end: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-octavia/9412492/controller/logs/screen-o-cw.txt.gz#_Apr_13_11_11_18_708485 | 12:49 |
dulek | At least that what I understand. | 12:49 |
apuimedo | dulek: that neutron-lbaas thing I fixed in another patchset and overwrote by mistake | 12:50 |
dulek | :) | 12:50 |
apuimedo | dulek: is that the octavia worker log? | 12:52 |
dulek | o-cw | 12:54 |
dulek | apuimedo: I don't anything about how Octavia works or is divided into services. D: | 12:54 |
apuimedo | it is | 12:54 |
apuimedo | that is the worker I guess | 12:54 |
apuimedo | there's worker, health-manager and api | 12:55 |
apuimedo | iirc | 12:55 |
dulek | apuimedo: I start to think that error on your patch is different than the gate issue we have. | 12:55 |
dulek | apuimedo: See - other gates failed in tempest, not on DevStack. | 12:55 |
*** caowei has quit IRC | 12:56 | |
openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy https://review.openstack.org/560433 | 12:58 |
apuimedo | dulek: well, does the gate issue happen in other patches? | 12:58 |
dulek | apuimedo: Yup, Octavia gates are failing but due to timeout of tempest tests, not creating LB in DevStack. | 12:59 |
dulek | apuimedo: And that issue on other patches is transient - happens on average on 1.5 gates out of 3. | 13:00 |
dulek | apuimedo: Your has 100% hit ratio. ;) | 13:00 |
*** gcheresh_ has joined #openstack-kuryr | 13:00 | |
dulek | I'd bet python-openstackclient formats stuff differently than Neutron and we should monitor for LB status differently. | 13:01 |
apuimedo | dulek: I tried it manually | 13:09 |
apuimedo | if you refer to the lb_state | 13:09 |
dulek | apuimedo: I do. | 13:09 |
dulek | Or maybe I'm not, just a sec. | 13:10 |
apuimedo | dulek: http://paste.openstack.org/show/719149/ | 13:12 |
dulek | apuimedo: It does wait in create_load_balancer_listener forever. | 13:12 |
apuimedo | dulek: cause the fucking LB doesn't go Active | 13:13 |
apuimedo | it goes ERROR | 13:13 |
apuimedo | doesn't it? | 13:13 |
dulek | apuimedo: I might be able to check that. :D | 13:13 |
apuimedo | dulek: and operating status prolly goes "offline" | 13:13 |
apuimedo | sorry "OFFLINE" | 13:13 |
apuimedo | it's nice because it's all uppercase | 13:14 |
dulek | apuimedo: Shouldn't we then see anything on Octavia logs? | 13:14 |
dulek | Eh, a while ago it was possible to get the DB dump from gate logs. | 13:15 |
apuimedo | did you check the health manager log? | 13:15 |
dulek | apuimedo: I see that. This stream of WARNING happens always. | 13:16 |
dulek | apuimedo: http://logs.openstack.org/60/556960/2/check/kuryr-kubernetes-tempest-daemon-octavia/f764a94/controller/logs/screen-o-cw.txt.gz | 13:17 |
dulek | Oh, wait that's not healthmanager. :P | 13:17 |
dulek | apuimedo: The LB gets into ACTIVE: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-octavia/9412492/controller/logs/screen-o-cw.txt.gz#_Apr_13_11_11_18_708485 | 13:18 |
dulek | (this is from your patch now) | 13:18 |
dulek | apuimedo: oscwrap loadbalancer show '' !! | 13:18 |
dulek | apuimedo: You're not passing lb id. | 13:19 |
apuimedo | damn... Really? | 13:19 |
apuimedo | pffff I knew refactoring into functions was a bad idea... I'm too accomodating to reviewers | 13:20 |
apuimedo | xD | 13:20 |
dulek | apuimedo: http://logs.openstack.org/33/560433/7/check/kuryr-kubernetes-tempest-octavia/9412492/job-output.txt.gz#_2018-04-13_11_03_29_134905 | 13:21 |
*** gcheresh_ has quit IRC | 13:21 | |
dulek | apuimedo: Ah | 13:21 |
dulek | apuimedo: It'll be good now. | 13:22 |
dulek | apuimedo: You've stopped passing algorithm there. | 13:22 |
dulek | apuimedo: So last argument will be lb name and it'll be fine. | 13:22 |
dulek | But I've told you it's not Octavia. :D | 13:22 |
apuimedo | algorithm is for pool | 13:23 |
*** salv-orlando has quit IRC | 13:24 | |
apuimedo | dulek: there's no way this is not from patchset 7 | 13:24 |
*** salv-orlando has joined #openstack-kuryr | 13:24 | |
apuimedo | there's patchset 8 for a while now with this fixed | 13:25 |
dulek | apuimedo: It is from 7. | 13:25 |
dulek | Yes, that's what I've said, it should be fine now. | 13:25 |
apuimedo | basically the issue was that the create_load_balancer_pool was copied from create_load_balancer_listener | 13:25 |
apuimedo | and forgot to change the name | 13:25 |
apuimedo | but I fixed that last night | 13:25 |
apuimedo | and it was fixed on an earlier patchset | 13:25 |
dulek | Sure, sure. Glad we have one less Octavia issue. | 13:26 |
apuimedo | then when I pushed for the LIBS_FROM_GIT today from another machine the error got pushed again | 13:26 |
apuimedo | because the method name issue I fixed from dani's server | 13:26 |
dulek | apuimedo: Cut the number of machines. :D | 13:26 |
apuimedo | and forgot to pull again from my laptop | 13:26 |
apuimedo | dulek: I'd like to | 13:26 |
*** salv-orlando has quit IRC | 13:28 | |
dmellado | apuimedo: rebase hell xD | 13:29 |
dmellado | xD | 13:29 |
apuimedo | dmellado: and waiting an hour for each error detection does not help | 13:32 |
dmellado | you tell me | 13:34 |
dmellado | we should go jaume's path | 13:34 |
dulek | dmellado: 561141 is getting in now. Fingers crossed. | 14:04 |
dmellado | dulek: yeah, even commented there | 14:05 |
dmellado | if it goes good this time we should be at least safe on the gates for now | 14:05 |
dulek | There's one fine thing about such a Friday. In the evening I'll go to the city center, find a nice crafty pub and first glass will taste great. ;) | 14:06 |
apuimedo | I'll find a field of rocks and jump head first | 14:07 |
dulek | apuimedo: BTW - your patch looks fine in the gate. It haven't caught dmellado patch disabling service test, so Octavia's failing, but DevStack installs fine. | 14:08 |
dmellado | dulek: apuimedo in any case we'd need to investigate that after all this patch monkey balling | 14:08 |
dmellado | dulek: lol, great plans xD | 14:08 |
apuimedo | dulek: why do we need to disable the service test with my fix? | 14:11 |
dulek | apuimedo: Oh. Yeah, what's the point of testing your fix without that test…? | 14:11 |
dulek | Awww. Anyway our gates with Octavia fail occasionally. | 14:11 |
dulek | apuimedo: With amphorae returning 500. | 14:12 |
dulek | apuimedo: And there are no logs for amphorae in gate results so we don't know what happens. \o/ | 14:12 |
apuimedo | dulek: sorry if it's obvious... Why do they fail? | 14:12 |
dulek | apuimedo: Answer above. :) | 14:12 |
apuimedo | I feel like I may have missed something going on on #openstack-lbaas | 14:12 |
dulek | apuimedo: We were discussing that in the morning. Like normal morning, not Catalan one. | 14:13 |
*** dougbtv_ has quit IRC | 14:19 | |
apuimedo | ah | 14:21 |
apuimedo | dulek: and the result was? | 14:22 |
dulek | apuimedo: Octavia folks were suspecting gate performance issue. | 14:22 |
dulek | I've bought it in the morning, but not now. | 14:23 |
dulek | apuimedo: http://logs.openstack.org/33/560433/8/check/kuryr-kubernetes-tempest-octavia/3416e38/controller/logs/screen-kuryr-kubernetes.txt.gz#_Apr_13_14_10_32_280699 | 14:25 |
dulek | apuimedo: There's an issue with LB Python code and Octavia. | 14:25 |
dulek | apuimedo: We're getting 400. | 14:26 |
dulek | "Unknown attribute for argument load_balancer.loadbalancer: tenant_id" | 14:26 |
dulek | dmellado, apuimedo: Do we backport `docker exec` patch into stable/queens? | 14:30 |
dulek | Technically it's a bug and that venv thing wasn't really a good, portable idea. | 14:31 |
* dulek is asking in context of cleaning up upstream Dockerfiles. | 14:31 | |
apuimedo | dulek: interesting! | 14:31 |
apuimedo | maybe I missed something in the handler | 14:32 |
dulek | apuimedo: I'd rather say APIs are not 100% compatible. | 14:32 |
dulek | apuimedo: Oh, wait. | 14:32 |
dulek | apuimedo: Shall it be project_id? :D | 14:32 |
apuimedo | dulek: probably | 14:33 |
* dulek checks if lbaas accepts that as well. | 14:33 | |
dmellado | xD | 14:34 |
dmellado | dulek: it'd be a good idea and an easy patch, so go for it | 14:34 |
dmellado | apuimedo: botched jobs again? | 14:34 |
*** kiennt26 has joined #openstack-kuryr | 14:34 | |
apuimedo | dmellado: not sure yet | 14:34 |
apuimedo | but likely | 14:34 |
apuimedo | alright, we can move to use project_id | 14:36 |
apuimedo | https://developer.openstack.org/api-ref/network/v2/index.html#load-balancer-as-a-service-2-0-deprecated | 14:37 |
dulek | apuimedo: I'm not sure how they've achieved that though - I don't see anything for project_id in https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/extensions/loadbalancerv2.py | 14:38 |
dulek | But okay, it's Neutron, it's magic. | 14:39 |
apuimedo | dulek: they have a decorator for this stuff | 14:40 |
apuimedo | IIRC | 14:40 |
dulek | apuimedo: Like I've said - M-A-G-I-C. ;) | 14:41 |
apuimedo | I hate magic | 14:43 |
apuimedo | in code | 14:43 |
dmellado | I hate magic in the gates | 14:46 |
dmellado | thought https://review.openstack.org/#/c/561193/ had passed | 14:46 |
dmellado | and it didn't pick up a needed change so recheck again | 14:47 |
openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy https://review.openstack.org/560433 | 14:47 |
dmellado | dulek: apuimedo FYI openstack infra is doing odd things | 14:48 |
apuimedo | dmellado: now it says +1 on that patch you just said | 14:48 |
dmellado | apuimedo: yeah, I hit recheck | 14:48 |
dmellado | and it suddenly popped up there | 14:48 |
dmellado | stating it took 1 hour | 14:48 |
dmellado | so it's weird | 14:48 |
apuimedo | dmellado: so it was my mistake in patchset 8 again. I thought octavia API being a superset and all would be true | 14:48 |
dmellado | looks like the publisher got stuck | 14:48 |
apuimedo | but it is not | 14:49 |
openstackgerrit | Merged openstack/kuryr-tempest-plugin master: Skip service test https://review.openstack.org/561141 | 14:49 |
apuimedo | because lbaasv2 API accepts both tenant_id and project_id | 14:49 |
dmellado | and octavia doesn't | 14:49 |
dmellado | yeah | 14:49 |
apuimedo | nope | 14:49 |
dulek | :) | 14:49 |
dmellado | tenant_id has been legacy for quite a while, dude | 14:49 |
apuimedo | liar liar pants on fire | 14:49 |
dulek | It's not too bad though. | 14:49 |
apuimedo | anyway, no reason to keep using tenant_id | 14:49 |
dulek | It could be worse. | 14:49 |
apuimedo | it *IS* part of the API dmellado | 14:49 |
*** openstackgerrit has quit IRC | 14:50 | |
dmellado | but it *IS* recommended not to use it | 14:50 |
apuimedo | you either are a superset | 14:50 |
dmellado | lol | 14:50 |
dmellado | openstackgerrit went down | 14:50 |
apuimedo | or a super fake | 14:50 |
dmellado | something *IS* odd | 14:50 |
dmellado | xD | 14:50 |
apuimedo | dmellado: openstack is giving up on us | 14:50 |
dmellado | dulek: what did you do? xD | 14:50 |
dulek | dmellado: Gerrit's fine on my end. | 14:50 |
dmellado | openstackgerrit (~openstack@review.openstack.org) has quit (Remote host closed the connection) | 14:51 |
dmellado | see ^^ | 14:51 |
dmellado | xD | 14:51 |
dulek | Ah. | 14:55 |
dulek | apuimedo: WTF, we were passing both tenant_id and project_id? :D | 15:04 |
dulek | dmellado: And this Zuul result on your patch was probably result of my earlier recheck. I've did it because your patch haven't grabbed DevStack fix. | 15:04 |
dmellado | probably yeah, it just took tooo long | 15:05 |
dmellado | and suddenly the gerrit went down so I had my concerns xD | 15:05 |
dmellado | in any case let's merge that one asap | 15:05 |
dmellado | and I'd be able to go for my beer on the weekend in a happy way | 15:05 |
*** livelace-link has joined #openstack-kuryr | 15:10 | |
dmellado | dulek: apuimedo would you do the honors? | 15:11 |
dmellado | https://review.openstack.org/#/c/561193/ | 15:11 |
dulek | dmellado: Without waiting for your rechecks? :D | 15:12 |
dulek | dmellado: Done. | 15:12 |
*** pcaruana has quit IRC | 15:12 | |
dmellado | dulek: yeah, it'll take too much time | 15:12 |
dmellado | xD | 15:12 |
dmellado | awesome | 15:12 |
dulek | ltomasbo: Ha, I've hit your psutil issue! Awesome! | 15:15 |
dmellado | dulek: you too? | 15:15 |
*** gcheresh_ has joined #openstack-kuryr | 15:15 | |
dmellado | well, it'd be fixed as soon as the patch gets merged | 15:15 |
dulek | ltomasbo: Happened on stable/queens, I guess setuputils are somehow capped there. | 15:15 |
dulek | dmellado: Not on stable/queens, though we can backport. | 15:16 |
dulek | dmellado: Oh wait, we can't. You haven't added bugreport. :D | 15:16 |
dmellado | yeah, that's my idea | 15:16 |
dmellado | lol | 15:16 |
dmellado | xD | 15:16 |
dmellado | no worries, we'll handle that | 15:16 |
dmellado | bug + backport | 15:16 |
apuimedo | missing bug report?! | 15:17 |
apuimedo | *shame* | 15:17 |
dulek | apuimedo: Shame on us, we gave him +2. | 15:17 |
dmellado | ltomasbo: you write it, you like writing | 15:17 |
dmellado | shame on us all | 15:17 |
dmellado | xD | 15:17 |
dmellado | dulek: how many issues did we have with the infra and so since yesterday? | 15:17 |
dmellado | I lost the count | 15:17 |
dulek | Okay, going for a walk, will take a look here and +W anything passing Zuul in an hour or two. | 15:17 |
ltomasbo | dmellado, it is your patch! I have enough writing for this week! | 15:18 |
dmellado | ltomasbo: LOL | 15:18 |
dulek | dmellado: Too much. And Ocatvia one still hangs. | 15:18 |
dmellado | I was just kidding! xD | 15:18 |
apuimedo | ltomasbo: finish sf already! | 15:18 |
apuimedo | xD | 15:18 |
ltomasbo | apuimedo, I think I'm done with writing, just remaining recording demos and prepare slides | 15:18 |
* ltomasbo fingers crossed | 15:19 | |
dmellado | ltomasbo: I'm pretty sure they'll break the testbed again | 15:19 |
dmellado | and we'll have to do more writing + reinstallation | 15:19 |
dmellado | I even had a nightmare about it xD | 15:19 |
ltomasbo | if we have the videos, I don't care anymore... | 15:19 |
dmellado | xD | 15:19 |
ltomasbo | apuimedo, dulek: I tried with octavia in my devstack and it fails too | 15:20 |
dmellado | loool dulek was fast enough to stop the merge | 15:20 |
dmellado | shame | 15:20 |
dmellado | xD | 15:21 |
ltomasbo | apuimedo, moving to lbaasv2, seems the subnet per namespace is working, just need to handle a couple of corner cases to support ports pool (remove subports when namespace is deleted) | 15:21 |
ltomasbo | apuimedo, and it will depend on your driver loading patch | 15:21 |
dmellado | ltomasbo: what was the package name that you had the issue with? | 15:22 |
ltomasbo | psutils | 15:22 |
ltomasbo | (5.4.4) | 15:23 |
dmellado | ack, thanks | 15:23 |
*** openstackgerrit has joined #openstack-kuryr | 15:25 | |
openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Apply UC to Dockerfiles https://review.openstack.org/561193 | 15:25 |
dmellado | I'll just +W this one, as it's just an amend in the code | 15:25 |
*** salv-orlando has joined #openstack-kuryr | 15:26 | |
apuimedo | ltomasbo: slides? | 15:26 |
apuimedo | ltomasbo: which driver loading patch? | 15:26 |
apuimedo | do you mean handler loading? | 15:27 |
dmellado | apuimedo: disregard that, SF stuff | 15:27 |
ltomasbo | apuimedo, slides for the final review meeting, we need to tell EC what we have done, and convince them we made good use of the money | 15:27 |
ltomasbo | apuimedo, yes, I meant handler loader (long week already...) | 15:27 |
apuimedo | ok | 15:29 |
dmellado | ltomasbo: 'convince them' actually sounds like we're ripping 'em off | 15:29 |
dmellado | xD | 15:29 |
apuimedo | and you obviously aren't | 15:30 |
apuimedo | you fought the testbed gods | 15:30 |
ltomasbo | xD | 15:30 |
ltomasbo | we did a lot of work actually | 15:30 |
ltomasbo | though it was not well organized | 15:30 |
*** salv-orlando has quit IRC | 15:31 | |
*** salv-orlando has joined #openstack-kuryr | 15:37 | |
* dmellado goes off | 15:40 | |
dmellado | have a nice weekend, folks! | 15:40 |
ltomasbo | have a nice weekend! | 15:41 |
*** kiennt26 has quit IRC | 15:59 | |
*** jchhatbar has quit IRC | 16:12 | |
*** pcaruana has joined #openstack-kuryr | 16:15 | |
*** jchhatbar has joined #openstack-kuryr | 16:20 | |
*** apuimedo has quit IRC | 16:24 | |
*** salv-orlando has quit IRC | 16:41 | |
*** salv-orlando has joined #openstack-kuryr | 16:41 | |
*** salv-orlando has quit IRC | 16:44 | |
*** salv-orlando has joined #openstack-kuryr | 16:44 | |
*** hongbin has joined #openstack-kuryr | 16:50 | |
*** jchhatbar has quit IRC | 17:09 | |
openstackgerrit | Merged openstack/kuryr-kubernetes master: Apply UC to Dockerfiles https://review.openstack.org/561193 | 17:15 |
*** atoth has quit IRC | 17:30 | |
*** atoth has joined #openstack-kuryr | 17:32 | |
dulek | Just in case someone looks here over the weekend - Toni's patch looks green now and just waits for second +2: https://review.openstack.org/#/c/560433 | 17:35 |
*** atoth has quit IRC | 18:11 | |
*** apuimedo has joined #openstack-kuryr | 18:32 | |
*** apuimedo has quit IRC | 18:57 | |
*** gcheresh_ has quit IRC | 19:25 | |
openstackgerrit | Merged openstack/kuryr-kubernetes master: lbaasv2: Fix for deployments missing the proxy https://review.openstack.org/560433 | 19:54 |
openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-tempest-plugin master: Revert "Skip service test" https://review.openstack.org/561364 | 22:08 |
*** garyloug has quit IRC | 22:28 | |
*** hongbin has quit IRC | 23:13 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!