| *** k0da has quit IRC | 00:26 | |
| *** esberglu has joined #openstack-powervm | 01:36 | |
| *** esberglu has quit IRC | 01:36 | |
| *** esberglu has joined #openstack-powervm | 01:36 | |
| *** esberglu has quit IRC | 01:41 | |
| *** thorst has joined #openstack-powervm | 01:53 | |
| *** thorst has quit IRC | 01:57 | |
| *** tonyb_ is now known as tonyb | 02:40 | |
| *** jay1_ has joined #openstack-powervm | 03:21 | |
| *** jay1_ has quit IRC | 03:51 | |
| *** thorst has joined #openstack-powervm | 03:54 | |
| *** thorst has quit IRC | 04:00 | |
| *** k0da has joined #openstack-powervm | 04:50 | |
| *** k0da has quit IRC | 05:19 | |
| *** jay1_ has joined #openstack-powervm | 05:50 | |
| *** thorst has joined #openstack-powervm | 05:57 | |
| *** thorst has quit IRC | 06:02 | |
| *** k0da has joined #openstack-powervm | 07:00 | |
| *** thorst has joined #openstack-powervm | 07:58 | |
| *** thorst has quit IRC | 08:03 | |
| *** thorst has joined #openstack-powervm | 09:24 | |
| *** thorst has quit IRC | 09:29 | |
| *** k0da has quit IRC | 09:48 | |
| *** k0da has joined #openstack-powervm | 09:49 | |
| *** smatzek has joined #openstack-powervm | 10:59 | |
| *** jay1_ has quit IRC | 11:21 | |
| *** thorst has joined #openstack-powervm | 11:25 | |
| *** thorst has quit IRC | 11:29 | |
| efried | edmondsw The CI setup scripts run pvmctl; and pvmctl/pypowervm need to be version synced. | 11:40 |
|---|---|---|
| openstackgerrit | Merged openstack/nova-powervm master: Allow dynamic enable/disable of SRR capability https://review.openstack.org/399579 | 11:59 |
| edmondsw | efried ah, right. Ok, so I think we at least need a comment that explains that | 12:03 |
| efried | edmondsw Well, I thought it through some more, commented. | 12:03 |
| edmondsw | efried yes.. | 12:05 |
| edmondsw | efried I see you merged https://review.openstack.org/#/c/399579/ ... I thought we were thinking this didn't actually require any nova-powervm changes, in which case no need for a spec | 12:08 |
| edmondsw | did you think of something that needs to change? | 12:09 |
| efried | edmondsw I dig having a spec that says this didn't work before, and it's gonna work now. | 12:09 |
| *** jpasqualetto has joined #openstack-powervm | 12:09 | |
| efried | There's going to be a change required in nova-powervm - the requirements bump to pull in the new pypowervm - it's just that we're going to get it for free-ish from the bot. | 12:10 |
| *** thorst has joined #openstack-powervm | 12:10 | |
| edmondsw | I don't think I agree with that, but I didn't -1 it yesterday, so my fault... it's done now | 12:11 |
| *** esberglu has joined #openstack-powervm | 12:28 | |
| *** dwayne has quit IRC | 12:43 | |
| *** esberglu has quit IRC | 12:49 | |
| *** kylek3h has joined #openstack-powervm | 12:50 | |
| *** smatzek has quit IRC | 12:58 | |
| *** esberglu has joined #openstack-powervm | 13:05 | |
| *** mdrabe has joined #openstack-powervm | 13:15 | |
| *** smatzek has joined #openstack-powervm | 13:17 | |
| *** jay1_ has joined #openstack-powervm | 13:45 | |
| efried | esberglu I think the cherry-pick-rather-than-checkout change is busted. | 14:00 |
| efried | esberglu https://review.openstack.org/#/c/476204/ | 14:00 |
| esberglu | efried: Yeah I know why | 14:00 |
| efried | k | 14:00 |
| esberglu | efried: Well at least I think I do | 14:02 |
| esberglu | zuul only reports a merge failure when it can't automatically merge (i.e. you can't just click the rebase button) | 14:02 |
| esberglu | But I think the merge for cherry-pick may be different than that | 14:02 |
| efried | esberglu I *just* rebased that one. | 14:02 |
| efried | Oh | 14:03 |
| efried | I think I may know why. | 14:03 |
| efried | let me look at that output again... | 14:03 |
| efried | Yeah. | 14:03 |
| efried | esberglu We're trying to cherry-pick the last commit in a series. | 14:04 |
| efried | Without applying the ones leading up to it. | 14:04 |
| esberglu | efried: Oh duh. We even brought that up in IRC before | 14:04 |
| efried | If I were doing that locally, I would do it as: | 14:04 |
| efried | git review -d <the_change>; git rebase <main_branch> | 14:04 |
| esberglu | So without git review I think that would just be | 14:05 |
| efried | 'cept would probably have to add logic to clean out commits in the series that have merged (possibly at a different patch set) | 14:05 |
| esberglu | git checkout <change>, git rebase | 14:05 |
| efried | yeah, but to account for ^^, you may need some extra flaggage. | 14:06 |
| efried | I actually don't know how to solve the thing with the already-merged commits earlier in the chain. | 14:07 |
| esberglu | efried: I think we can just use that --keep-redundant flag | 14:07 |
| esberglu | Let me look at the rebase doc | 14:08 |
| efried | That'll only work if the commit in the chain is up to date with what merged. | 14:08 |
| esberglu | Surely there is a way to accomplish that, I'll check it out after the kickoff meeting is over | 14:11 |
| efried | esberglu Offhand, I think it will involve walking the commits backward until we get to the trunk, and somehow matching them up with the tip of the main branch. By Change-Id? | 14:12 |
| efried | esberglu In the meantime, prolly a good idea to revert that change. | 14:14 |
| esberglu | efried: 5516 | 14:15 |
| efried | esberglu Updated commit message, and +2 | 14:17 |
| esberglu | efried: thx | 14:18 |
| esberglu | merged | 14:18 |
| efried | Insta-recheck? | 14:18 |
| esberglu | Yep | 14:19 |
| esberglu | All changes to the powervm-ci repo are instant | 14:19 |
| efried | Loving that | 14:19 |
| esberglu | efried: Same | 14:19 |
| efried | I'm sure I'll get over it eventually, but it gives me a thrill every time. | 14:20 |
| esberglu | hahaha | 14:20 |
| *** dwayne has joined #openstack-powervm | 14:29 | |
| esberglu | efried: It actually will hit runs that have started before the change merges. Because powervm-ci doesn't get pulled until after the stack | 14:33 |
| esberglu | Wait no that isn't correct | 14:33 |
| esberglu | It gets pulled right before the stack | 14:34 |
| efried | "This change will take effect half an hour ago." | 14:35 |
| efried | That's amazing. | 14:35 |
| efried | Better than quantum. | 14:35 |
| esberglu | efried: thorst: Where's the source for this http://nova-powervm.readthedocs.io/en/latest/support-matrix.html | 15:13 |
| thorst | doc/source/support-matrix.rst | 15:13 |
| esberglu | Ah typo in my find command. Thanks | 15:14 |
| *** k0da has quit IRC | 15:21 | |
| *** jay1_ has quit IRC | 15:24 | |
| *** mdrabe has quit IRC | 15:53 | |
| *** mdrabe has joined #openstack-powervm | 15:57 | |
| edmondsw | efried I just proposed 5518 for what I was saying yesterday in 5511 | 16:53 |
| efried | ack | 16:53 |
| *** efried is now known as efried_afk | 16:56 | |
| edmondsw | esberglu efried what is sonar and why is it complaining about things that tox -epep8 allows? | 17:08 |
| esberglu | edmondsw: I believe its a set of rules like complexity checks, lines per file, etc. Not sure though | 17:12 |
| edmondsw | esberglu yeah, that's exactly what it is... I guess my question would be better worded, why do we do this if it doesn't match tox -epep8? | 17:12 |
| esberglu | edmondsw: No idea | 17:13 |
| edmondsw | they should be consistent, so you don't have to wait for jenkins to fail to go address something. | 17:14 |
| esberglu | edmondsw: Yeah that makes sense to me. But perhaps there is a reason for it. I'd ask in the novalink channel someone there probably knows | 17:18 |
| edmondsw | esberglu I hit this with pypowervm, so it's something we setup ourselves, not nova | 17:19 |
| esberglu | Yeah novalink slack channel | 17:20 |
| edmondsw | esberglu oh, sorry, I misread novalink as nova ;) | 17:20 |
| *** k0da has joined #openstack-powervm | 17:21 | |
| *** efried_afk is now known as efried | 17:42 | |
| esberglu | edmondsw: So what we really need to do regarding the convo in qa is use the devstack generated tempest.conf instead of our own. | 17:57 |
| esberglu | There's already a TODO for it | 17:57 |
| esberglu | But messing around with the local.conf files never goes smoothly | 17:58 |
| edmondsw | esberglu ah, that explains it | 17:58 |
| esberglu | So I've been putting it off since what we have is working | 17:58 |
| esberglu | IIRC the reason we had our own is that devstack didn't have enough conf options to do what we wanted with tempest | 17:58 |
| esberglu | Its getting closer, but there are still things missing | 17:58 |
| esberglu | So not only will we have to update our local.conf, we will likely have to create devstack patches/bugs as necessary | 17:59 |
| esberglu | Eventually I think the idea is to have a local.conf option for every tempest config setting | 17:59 |
| edmondsw | esberglu couldn't we let devstack create tempest's conf, and then add/update it as well? | 18:02 |
| edmondsw | opening devstack patches/bugs as well, sure, but in the meantime... | 18:02 |
| esberglu | edmondsw: We can just explicitly set tempest.conf values using the local.conf post config | 18:03 |
| esberglu | edmondsw: Here's what I will do. I will set up a stack and see what differs between their tempest.conf and ours | 18:04 |
| esberglu | Then set as much as possible using local.conf options | 18:04 |
| edmondsw | esberglu +1 | 18:04 |
| esberglu | Anything that doesn't have one we stuff in the post config for now, open a devstack bug, and either fix ourselves or wait for one | 18:04 |
| esberglu | Like they said in qa, they are REALLY easy patches | 18:05 |
| edmondsw | yeah | 18:06 |
| edmondsw | tx | 18:06 |
| efried | thorst esberglu mdrabe Any y'all interested in reviewing https://review.openstack.org/#/c/476695/ ? | 18:20 |
| efried | I won't blame you if you say no. | 18:20 |
| thorst | yikes | 18:20 |
| efried | ikr | 18:20 |
| esberglu | efried: I've been slowly working through it. About halfway down | 18:20 |
| efried | coo | 18:20 |
| efried | I'll hold off +W then. | 18:21 |
| efried | Dammit, how did we end up with a net LOC gain on this? | 18:21 |
| esberglu | efried: Really? Hasn't seemed that way so far | 18:22 |
| efried | i know | 18:22 |
| esberglu | Sure enough | 18:23 |
| efried | I mean, I've found a few | 18:23 |
| efried | But way more removals | 18:23 |
| thorst | efried: you added 301, removed 347 | 18:23 |
| efried | oh, I was reading it backwards? | 18:23 |
| thorst | I think so | 18:23 |
| esberglu | So did I lol. That makes more sense | 18:24 |
| efried | Yuh. Derp. | 18:24 |
| esberglu | efried: Seeing some "except Exception:" | 18:34 |
| esberglu | Thought we were trying to avoid that | 18:34 |
| esberglu | But perhaps that isn't within the scope of this change | 18:34 |
| efried | esberglu Yes, separate change TBD | 18:34 |
| *** jay1_ has joined #openstack-powervm | 18:35 | |
| esberglu | efried: Okay. 2 files left, but that's the only thing I have so far | 18:35 |
| esberglu | efried: LGTM | 18:36 |
| efried | esberglu Thanks! | 18:36 |
| esberglu | efried: Np. It appeared to be much more daunting than it really was | 18:36 |
| efried | Yeah, lotta files but relatively low LOC count. | 18:36 |
| edmondsw | efried good work there, tx | 18:38 |
| efried | Thanks for the thorough review | 18:38 |
| edmondsw | esberglu I'm having trouble stacking, looks like a keystone version discovery issue... have you seen that before? I'm basically using the OOT local.conf | 18:43 |
| mdrabe | "except Exception:" is bad? I just wrote that line | 18:44 |
| edmondsw | mdrabe not necessarily, but usually | 18:44 |
| edmondsw | you should catch specific exceptions that you expect, not everything, which could include both things you expect and things you didn't expect and didn't properly code to handle | 18:45 |
| edmondsw | or things you shouldn't handle, like an interrupt | 18:45 |
| mdrabe | Oh ok, thought there was some implicit behavior I wasn't aware of | 18:46 |
| edmondsw | one example of a case where you *should* do except Exception is if you need to make sure something you didn't expect doesn't kill your process | 18:46 |
| edmondsw | with suitable logging, of course, so you can figure out what it was and start expecting it in future | 18:47 |
| esberglu | edmondsw: Looking now | 18:48 |
| edmondsw | tx | 18:48 |
| esberglu | edmondsw: I've never ran devstack from a different directory than the other openstack projects | 18:53 |
| esberglu | Not sure if that matters at all | 18:53 |
| edmondsw | shouldn't | 18:53 |
| edmondsw | where are the keystone logs? | 18:53 |
| esberglu | The service isn't up yet so they don't exist | 18:54 |
| edmondsw | esberglu well it's trying to talk to the service, so it better be up... | 18:54 |
| esberglu | sudo journalctl -xea --unit devstack@ | 18:54 |
| esberglu | Then tab will show you what's been started | 18:54 |
| esberglu | Autocomplete | 18:54 |
| esberglu | Only dstat and etcd are up | 18:54 |
| esberglu | Looks like to me | 18:54 |
| edmondsw | ps -ef | grep keystone shows a bunch of wsgi processes for keystone | 18:55 |
| *** k0da has quit IRC | 18:55 | |
| esberglu | edmondsw: Maybe you have to add | 18:57 |
| esberglu | enable_plugin keystone git://git.openstack.org/openstack/neutron | 18:57 |
| esberglu | s/neutron/keystone/ | 18:57 |
| edmondsw | why wouldn't we have had to add that in our CI then? | 18:58 |
| esberglu | I was just thinking the same | 18:58 |
| edmondsw | I thought some of the base stuff like keystone you didn't need that for | 18:58 |
| esberglu | edmondsw: Yeah just spitballing here | 18:58 |
| edmondsw | esberglu appreciated :) | 18:58 |
| esberglu | Let me think for a while. I'm sure I've seen that error before. Can't remember why (and this could very well be something different) | 18:59 |
| esberglu | But it might come to me | 18:59 |
| esberglu | edmondsw: Perhaps something to do with WSGI_MODE=mod_wsgi? | 19:02 |
| esberglu | Could try commenting that out and restacking | 19:03 |
| edmondsw | esberglu worth a shot... | 19:06 |
| esberglu | edmondsw: That would be nice if it is the case. Might shed some insight as to what we need to do to remove that from CI | 19:06 |
| edmondsw | esberglu so commenting that line didn't fix the problem, but it did get keystone into journalctl so I can see the logs | 19:13 |
| edmondsw | and it looks like "from paste import deploy" is getting an ImportError because it can't find deploy | 19:14 |
| *** k0da has joined #openstack-powervm | 19:15 | |
| edmondsw | pip list shows that PasteDeploy 1.5.2 is installed... | 19:16 |
| efried | That thing again? | 19:17 |
| efried | Always pip & apt uninstall all paste crap before devstack. | 19:17 |
| edmondsw | efried yay | 19:19 |
| esberglu | edmondsw: efried: Ahh that's where I've seen. I knew it was familiar. We have a step for that in the undercloud deploy | 19:20 |
| esberglu | "pip install pastedeploy --upgrade --force-reinstall" | 19:20 |
| efried | Usually only see it first thing after fresh system install. | 19:20 |
| efried | If the CI node is installed from an image made after that's done, swhy you wouldn't have been seeing it lately. | 19:21 |
| edmondsw | yeah, this is a fresh system... stacking again now, but I expect that will fix it | 19:23 |
| openstackgerrit | Matt Rabe proposed openstack/nova-powervm master: WIP: Change NVRAM manager store to use uuid instead of instance object https://review.openstack.org/471926 | 19:23 |
| efried | edmondsw Until you hit the next thing. | 19:24 |
| edmondsw | right... I expect that will fix it, not devstack :) | 19:24 |
| esberglu | Yeah don't get your hopes too high when dealing with devstack, you'll very often be disappointed | 19:25 |
| edmondsw | esberglu I forgot to put the WSGI_MODE line back, so we'll see how this works with that commented... | 19:25 |
| edmondsw | I'm guessing it won't... | 19:25 |
| esberglu | edmondsw: In CI land you can still get it to stack. But it doesn't actually work | 19:26 |
| esberglu | So you may get though and end up unstacking anyways | 19:26 |
| esberglu | *through | 19:26 |
| edmondsw | esberglu do you remember what doesn't work, exactly? | 19:26 |
| esberglu | edmondsw: I just finished a stack and tempest run with that to see | 19:28 |
| edmondsw | efried if this paste thing is a known issue, has anyone tried to get it fixed in the NovaLink setup? | 19:28 |
| esberglu | edmondsw: I can upload the logs to the logserver from that | 19:28 |
| efried | edmondsw I don't think so. Not sure how simple it would be, though. I suspect it gets pulled in when you install some other dependency, like python-dev or whatever. | 19:29 |
| efried | And devstack seems to be the only thing that breaks on it - do you have any knowledge of pvc having trouble with it? | 19:30 |
| efried | Nothing else (like pvmctl) breaks) | 19:30 |
| openstackgerrit | Matt Rabe proposed openstack/nova-powervm master: WIP: Change NVRAM manager store to use uuid instead of instance object https://review.openstack.org/471926 | 19:30 |
| esberglu | edmondsw: http://184.172.12.213/manual/mod_wsgi/ | 19:31 |
| edmondsw | efried PowerVC will install everything it needs from debs, so it's not going to care if pip is messed up | 19:33 |
| edmondsw | esberglu wow, that's a lot of red | 19:34 |
| edmondsw | more of a burnt orange, I guess | 19:34 |
| esberglu | edmondsw: Yep | 19:34 |
| edmondsw | so... I don't want to bite that off today ;) | 19:34 |
| esberglu | If you scroll fast enough it makes you dizzy | 19:34 |
| esberglu | edmondsw: You know anything about mod_wsgi/uwsgi | 19:35 |
| edmondsw | a little... not too much | 19:36 |
| esberglu | edmondsw: The good thing is all of the errors are the same so once we figure it out we fix all of the orage | 19:37 |
| esberglu | orange | 19:37 |
| *** jay1_ has quit IRC | 19:53 | |
| edmondsw | esberglu <sigh> just one devstack problem after another... now hitting VolumeTypeCreateFailed: Cannot create volume_type with name lvmdriver-1 and specs {} | 20:12 |
| esberglu | edmondsw: I once spent an pretty much entire internship fighting devstack. So I know the struggle. I'll hop on quick | 20:13 |
| edmondsw | esberglu tx... this is with the WSGI_MODE uncommented again | 20:14 |
| esberglu | I miss screen... | 20:14 |
| esberglu | I find it hard to navigate the journalctl logs | 20:14 |
| edmondsw | I tried running the command that failed, and it worked for me... | 20:14 |
| edmondsw | timing issue, maybe | 20:14 |
| esberglu | Not seeing anything in the cinder-volume logs | 20:16 |
| esberglu | edmondsw: Have you been changing the local.conf before unstacking | 20:16 |
| edmondsw | yeah, I think I did that with the WSGI_MODE change | 20:16 |
| edmondsw | :( | 20:16 |
| esberglu | edmondsw: I don't think this issue is related to that. But I unstack also uses the local.conf | 20:17 |
| edmondsw | makes sense | 20:17 |
| esberglu | So if you change it before unstacking it might not clean up properly | 20:17 |
| esberglu | edmondsw: Honestly I would just try again. That works sometimes. I've tried to find better ways to debug devstack, but often it just turns into | 20:18 |
| esberglu | just trying stuff until it works | 20:18 |
| edmondsw | esberglu, this seems like a timing thing | 20:19 |
| edmondsw | yeah | 20:19 |
| edmondsw | I'll just try again | 20:19 |
| esberglu | Obviously still taking educated guesses, but trying to go through devstack to see whats going on is such a PITA its faster half the time | 20:20 |
| edmondsw | doesn't give me a warm fuzzy feeling about ever getting our CI stable... | 20:20 |
| esberglu | Eh. Usually once you get something that works it works for a while | 20:20 |
| esberglu | And if it breaks you at least have a timeframe for when the change was introduced | 20:20 |
| esberglu | But since you are independent I wouldn't even know where to start looking | 20:20 |
| edmondsw | but if we're hitting things like this, where there isn't any change, it was just some intermittent problem... | 20:21 |
| esberglu | edmondsw: I mean we won't ever get to the point where CI is passing everything. I think that this would fall into the acceptable failure category | 20:21 |
| esberglu | as long as it isn't common | 20:21 |
| esberglu | Devstack is also getting much better. Used to be that devstack would break us about once a week | 20:25 |
| efried | thorst Last chance at string cleanup patch | 20:41 |
| efried | https://review.openstack.org/#/c/476695/ | 20:41 |
| thorst | efried: I won't get to it today.... :-( | 20:42 |
| thorst | but I trust you | 20:42 |
| efried | Okay, merging. | 20:42 |
| *** smatzek has quit IRC | 20:47 | |
| openstackgerrit | Merged openstack/nova-powervm master: Clean up log messages https://review.openstack.org/476695 | 20:47 |
| esberglu | Once we get the generated tempest.conf going we should run against tempest as well | 20:54 |
| esberglu | Need to check what kind of capacity that will take but I think we will be fine | 20:54 |
| *** thorst has quit IRC | 21:12 | |
| *** kylek3h has quit IRC | 21:17 | |
| *** jpasqualetto has quit IRC | 21:27 | |
| *** esberglu has quit IRC | 21:27 | |
| *** thorst has joined #openstack-powervm | 21:46 | |
| *** thorst has quit IRC | 21:50 | |
| *** esberglu has joined #openstack-powervm | 21:52 | |
| *** esberglu has quit IRC | 21:57 | |
| *** k0da has quit IRC | 21:57 | |
| *** dwayne has quit IRC | 22:00 | |
| *** thorst has joined #openstack-powervm | 22:10 | |
| openstackgerrit | Matt Rabe proposed openstack/nova-powervm master: WIP: Change NVRAM manager store to use uuid instead of instance object https://review.openstack.org/471926 | 22:16 |
| *** thorst has quit IRC | 22:17 | |
| *** thorst has joined #openstack-powervm | 22:17 | |
| *** thorst has quit IRC | 22:22 | |
| *** mdrabe has quit IRC | 22:23 | |
| *** dwayne has joined #openstack-powervm | 23:06 | |
| *** edmondsw has quit IRC | 23:19 | |
| *** smatzek has joined #openstack-powervm | 23:30 | |
| *** smatzek has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!