| *** chas has joined #openstack-powervm | 00:24 | |
| *** chas has quit IRC | 00:29 | |
| *** jay1_ has joined #openstack-powervm | 00:35 | |
| *** smatzek has joined #openstack-powervm | 01:10 | |
| *** edmondsw has joined #openstack-powervm | 01:20 | |
| *** edmondsw has quit IRC | 01:25 | |
| *** thorst has joined #openstack-powervm | 01:46 | |
| *** thorst has quit IRC | 02:02 | |
| *** openstackgerrit has joined #openstack-powervm | 02:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/networking-powervm master: Updated from global requirements https://review.openstack.org/465572 | 02:11 |
|---|---|---|
| *** smatzek has quit IRC | 02:11 | |
| *** thorst has joined #openstack-powervm | 02:14 | |
| *** chas has joined #openstack-powervm | 02:24 | |
| *** chas has quit IRC | 02:30 | |
| *** thorst has quit IRC | 02:37 | |
| *** jay1_ has quit IRC | 02:42 | |
| *** jay1_ has joined #openstack-powervm | 02:55 | |
| *** edmondsw has joined #openstack-powervm | 03:08 | |
| *** edmondsw has quit IRC | 03:13 | |
| *** thorst has joined #openstack-powervm | 04:16 | |
| *** thorst has quit IRC | 04:18 | |
| *** chhavi has joined #openstack-powervm | 04:24 | |
| *** chas has joined #openstack-powervm | 04:25 | |
| *** thorst has joined #openstack-powervm | 04:26 | |
| *** chas has quit IRC | 04:30 | |
| *** thorst has quit IRC | 04:38 | |
| *** tjakobs has joined #openstack-powervm | 04:46 | |
| *** edmondsw has joined #openstack-powervm | 04:56 | |
| *** tjakobs has quit IRC | 04:56 | |
| *** edmondsw has quit IRC | 05:01 | |
| *** jay1_ has quit IRC | 05:21 | |
| *** jay1_ has joined #openstack-powervm | 05:33 | |
| *** thorst has joined #openstack-powervm | 05:38 | |
| *** thorst has quit IRC | 05:42 | |
| *** arunman has joined #openstack-powervm | 05:49 | |
| *** jay1_ has quit IRC | 05:56 | |
| *** thorst has joined #openstack-powervm | 06:09 | |
| *** jay1_ has joined #openstack-powervm | 06:19 | |
| *** chas has joined #openstack-powervm | 06:26 | |
| *** thorst has quit IRC | 06:27 | |
| *** chas has quit IRC | 06:30 | |
| *** edmondsw has joined #openstack-powervm | 06:45 | |
| *** edmondsw has quit IRC | 06:49 | |
| *** thorst has joined #openstack-powervm | 07:24 | |
| *** thorst has quit IRC | 07:29 | |
| *** thorst has joined #openstack-powervm | 08:25 | |
| *** chas has joined #openstack-powervm | 08:27 | |
| *** thorst has quit IRC | 08:29 | |
| *** chas has quit IRC | 08:31 | |
| *** k0da has joined #openstack-powervm | 08:35 | |
| *** thorst has joined #openstack-powervm | 09:26 | |
| *** thorst has quit IRC | 09:30 | |
| *** edmondsw has joined #openstack-powervm | 10:21 | |
| *** edmondsw has quit IRC | 10:25 | |
| *** thorst has joined #openstack-powervm | 10:27 | |
| *** chas has joined #openstack-powervm | 10:28 | |
| *** thorst has quit IRC | 10:31 | |
| *** chas has quit IRC | 10:33 | |
| *** smatzek has joined #openstack-powervm | 11:15 | |
| *** arunman has quit IRC | 11:43 | |
| *** thorst has joined #openstack-powervm | 11:58 | |
| *** edmondsw has joined #openstack-powervm | 11:58 | |
| *** kylek3h has quit IRC | 12:00 | |
| *** kylek3h has joined #openstack-powervm | 12:15 | |
| *** chas has joined #openstack-powervm | 12:29 | |
| *** chas has quit IRC | 12:33 | |
| *** kylek3h has quit IRC | 12:41 | |
| *** chas has joined #openstack-powervm | 12:45 | |
| *** efried has joined #openstack-powervm | 12:46 | |
| *** arunman has joined #openstack-powervm | 12:49 | |
| *** chas has quit IRC | 12:49 | |
| *** apearson has joined #openstack-powervm | 13:03 | |
| *** mdrabe has joined #openstack-powervm | 13:16 | |
| *** esberglu has joined #openstack-powervm | 13:32 | |
| *** jpasqualetto has joined #openstack-powervm | 13:34 | |
| *** jwcroppe has quit IRC | 13:38 | |
| esberglu | efried: thorst: edmondsw: The console tests don't work within the CI environment since pypowervm is non-local | 13:41 |
| edmondsw | esberglu ah, yes | 13:42 |
| thorst | hmmm. | 13:42 |
| efried | esberglu thorst Is there gonna be any way around that? | 13:42 |
| thorst | we could make a repeater of the repeater | 13:42 |
| efried | yay | 13:42 |
| thorst | but...that'd have...issues | 13:42 |
| thorst | I think we issue a mkvterm command for instance. | 13:43 |
| thorst | I think maybe step 1 is to create a mkvterm facade... | 13:44 |
| thorst | and then from that figure out how to tie that facade into the backing vnc | 13:44 |
| thorst | maybe we have a new 'pvm-ci' package that puts down that facade... | 13:45 |
| efried | And to be clear, all of this would be work purely to make these couple of tempest tests run properly. | 13:45 |
| thorst | heh, I think our team even made those tempest tests | 13:46 |
| esberglu | I guess I don't have enough background on this, but it seems like more work than it's worth for the time being? | 13:50 |
| thorst | I'm pretty sure mriedem said we needed to do it? | 13:51 |
| thorst | I'm not so sold its super tough TBH | 13:51 |
| thorst | but I'd need to brainstorm a bit more (otp atm unfortunately) | 13:51 |
| *** jwcroppe has joined #openstack-powervm | 13:53 | |
| *** apearson has quit IRC | 14:03 | |
| *** tjakobs has joined #openstack-powervm | 14:41 | |
| *** chas has joined #openstack-powervm | 14:45 | |
| *** thorst_ has joined #openstack-powervm | 14:49 | |
| *** chas has quit IRC | 14:50 | |
| *** thorst has quit IRC | 14:51 | |
| *** mdrabe has quit IRC | 14:54 | |
| *** mdrabe has joined #openstack-powervm | 15:01 | |
| efried | thorst_ edmondsw esberglu (adreznec mdrabe) -- https://etherpad.openstack.org/p/powervm-in-tree-todos | 15:06 |
| esberglu | efried: Nice. I might piggyback and add CI section there as well | 15:08 |
| edmondsw | efried excellent, thanks | 15:08 |
| esberglu | Unless you think a separate one would be better | 15:08 |
| efried | esberglu No, here. | 15:08 |
| esberglu | K good. I like it being in one place | 15:08 |
| efried | esberglu I already included the VNC tempest stuff (ll 6-11) | 15:08 |
| edmondsw | efried do we want this all visible publicly? | 15:09 |
| efried | edmondsw I don't see why not. | 15:09 |
| edmondsw | good... I'll look over it | 15:09 |
| efried | edmondsw I'm in favor of working openly as much as possible. If you see something specifically questionable, let me know. | 15:09 |
| edmondsw | yep | 15:10 |
| thorst_ | +2 to everything out there | 15:10 |
| efried | thorst_ esberglu Okay if I respond to mriedem's email and let him know we've got some challenges to overcome wrt the VNC tests? | 15:11 |
| mdrabe | efried: Anything more of a priority than others? Or should I just pick something? | 15:11 |
| efried | mdrabe Other than the stuff at the very top, nothing else really has implied priorities. | 15:12 |
| efried | mdrabe Some items may be blocked by others. | 15:12 |
| efried | E.g. we can't do SSP shared resource providers until the SSP change set is in and jaypipes' infra is all in place. | 15:13 |
| mdrabe | gotcha, I'll digest it | 15:13 |
| efried | mdrabe The lowest-hanging fruit is the "teeny changes" stuff starting on L24 | 15:14 |
| efried | Oh, thorst_ can I give you L29? | 15:14 |
| efried | Cause I don't know how to respond to that, but you probably do. | 15:14 |
| thorst_ | k | 15:17 |
| thorst_ | r.o.o down? | 15:17 |
| esberglu | Fine for me | 15:18 |
| efried | mdrabe esberglu Click the people icon in the top right of the etherpad and enter your nick. | 15:20 |
| efried | thx | 15:20 |
| edmondsw | efried I think I'm inclined to prioritize documenting our driver in the support matrix for Pike... thoughts? | 15:21 |
| efried | edmondsw Agree. | 15:21 |
| edmondsw | efried should we move that up to the top section? | 15:22 |
| efried | edmondsw Sure | 15:22 |
| efried | The stuff at the top doesn't have an order anymore (it did last night) so Ima take off the numbering. | 15:23 |
| efried | esberglu Can I recheck https://review.openstack.org/466768 yet? | 15:36 |
| esberglu | efried: I did just a little bit ago | 15:36 |
| efried | oh, okay, thanks. | 15:36 |
| esberglu | efried: 40 minutes into the run right now | 15:37 |
| *** k0da has quit IRC | 15:38 | |
| *** thorst_ has quit IRC | 15:39 | |
| *** thorst has joined #openstack-powervm | 16:00 | |
| *** thorst has quit IRC | 16:02 | |
| *** thorst has joined #openstack-powervm | 16:04 | |
| esberglu | efried: thorst: edmondsw: What's the plan for OVS support for CI? Are we trying to get both ovs and sea runs working simultaneously? | 16:22 |
| edmondsw | esbergu I thought that's what we'd said... work them both and see which one we get working first | 16:24 |
| esberglu | Yeah in-tree | 16:25 |
| esberglu | But what about OOT? | 16:25 |
| efried | I thought the first step was to get OVS CI working OOT at all. | 16:25 |
| efried | Then we can figure out if/how we're gonna do either/both. | 16:25 |
| esberglu | efried: Yeah I was just thinking about the necessary playbook changes long term. For the ovs ci poc it doesn't matter | 16:28 |
| efried | esberglu It's possible to use multiple net drivers at the same time, right? | 16:30 |
| efried | Yeah. | 16:30 |
| efried | Just not sure how to tell tempest that. | 16:30 |
| esberglu | efried: We can loop back on this discussion later. Don't even a working ovs local.conf yet | 16:32 |
| efried | nod. We'll need thorst involved anyway. | 16:33 |
| esberglu | efried: edmondsw: One of the vnc tests is failing the setup class with a port binding issue | 16:36 |
| esberglu | I put up a patch reverting the inclusion of those tests since they aren't working anyways | 16:37 |
| esberglu | 5337 | 16:37 |
| efried | esberglu Consistently? | 16:37 |
| efried | I thought we saw that issue intermittently anyawy | 16:37 |
| efried | anyway. | 16:37 |
| esberglu | efried: Yeah. There is still some wackiness going on with the in-tree CI with regards to networking | 16:38 |
| esberglu | networks interfering with other tests and things like that | 16:38 |
| efried | esberglu +2. How were you testing those anyway? Did you and thorst suss out the remotable business? | 16:38 |
| esberglu | I just added them to the whitelist manually and tested. Saw 0 failures and got ahead of myself and merged the patch before making sure they actually got ran | 16:40 |
| esberglu | Haven't done anything as far as getting them to work yet | 16:40 |
| efried | thorst edmondsw esberglu https://review.openstack.org/#/c/443189/ rebased, ready for internal +1s before I pester sdague/mriedem. | 16:46 |
| *** chas has joined #openstack-powervm | 16:46 | |
| efried | esberglu FYI, if it's just stupid Citrix XenServer CI that fails, you can say `xenserver: recheck` to save rechecking the world. | 16:50 |
| efried | The rest of the failures are non-voting (even the microsoft one at the bottom) | 16:50 |
| esberglu | efried: Yeah every CI has a specific recheck string (or at least are supposed to) | 16:51 |
| efried | right; just saw you did a full recheck on https://review.openstack.org/#/c/467656/ | 16:51 |
| *** chas has quit IRC | 16:52 | |
| esberglu | True | 16:53 |
| openstackgerrit | Merged openstack/nova-powervm master: Consume HttpNotFound https://review.openstack.org/466768 | 16:54 |
| openstackgerrit | Eric Larese proposed openstack/nova-powervm master: Clean up orphan tap devices on migrate https://review.openstack.org/466400 | 16:55 |
| *** arunman has quit IRC | 17:30 | |
| efried | edmondsw Were you going to take the support matrix? | 17:39 |
| efried | Getting ready to pick off another work item, wanted to make sure we didn't collide. | 17:39 |
| edmondsw | efried I will if it's still there when I free up, but that probably wouldn't be today | 17:40 |
| efried | edmondsw Okay. I think mebbe I'll do get_inventory() instead. | 17:40 |
| edmondsw | efried sounds good | 17:41 |
| *** apearson has joined #openstack-powervm | 17:48 | |
| *** jay1_ has quit IRC | 18:22 | |
| efried | esberglu Things are getting "read-only file system"... | 18:35 |
| efried | ...during devstack. | 18:35 |
| esberglu | link? | 18:35 |
| efried | https://review.openstack.org/#/c/461574/ | 18:36 |
| efried | both IT and OOT | 18:36 |
| efried | And saw it somewhere else earlier too. | 18:36 |
| esberglu | efried: Yeah other runs are getting flooded with failures as well | 18:38 |
| *** apearson has quit IRC | 18:38 | |
| efried | esberglu Never a dull moment in CIland. | 18:41 |
| *** chas has joined #openstack-powervm | 18:47 | |
| *** chas has quit IRC | 18:52 | |
| esberglu | efried: Looks like that happened to a set of runs all at the same time, hasn't happened since. Not really much info to go off of with only the console log | 18:59 |
| efried | esberglu Were all those runs perchance using the same neo? | 18:59 |
| efried | or the same vios? | 19:00 |
| efried | or the same SAN disks? | 19:00 |
| esberglu | I don't think the neo gets logged in the console log either. Let me see if I can piece it together from the jenkins jobs | 19:01 |
| *** k0da has joined #openstack-powervm | 19:03 | |
| efried | esberglu If it was transient, prolly not worth spending a lot of time nailing it down. | 19:04 |
| esberglu | efried: I think it's actually just bad patches | 19:04 |
| efried | esberglu Bad patches on what? | 19:04 |
| esberglu | https://review.openstack.org/#/c/462770/ | 19:04 |
| efried | Not the changes being tested | 19:05 |
| esberglu | https://review.openstack.org/#/c/462769/5 | 19:05 |
| *** apearson has joined #openstack-powervm | 19:05 | |
| esberglu | Those two related changes both hit it multiple times | 19:05 |
| esberglu | And they appear to be blowing everything up | 19:06 |
| *** chhavi has quit IRC | 19:06 | |
| efried | esberglu http://184.172.12.213/89/443189/23/check/nova-in-tree-pvm/dbd1e4f/console.html | 19:06 |
| efried | And that code is PRISTINE | 19:06 |
| esberglu | Dang | 19:07 |
| openstackgerrit | Eric Larese proposed openstack/nova-powervm master: Clean up orphan tap devices on migrate https://review.openstack.org/466400 | 19:31 |
| efried | thorst https://review.openstack.org/#/c/443189/23/nova/virt/powervm/tasks/storage.py You have any particular objection to renaming Connect/Disconnect to Attach/Detach throughout? | 19:41 |
| efried | Seems like more work than it's worth to me, but... | 19:41 |
| efried | Maybe in a followon, assigned to edmondsw ;-) | 19:42 |
| edmondsw | efried lol | 19:42 |
| edmondsw | too nit picky... not nit picky enough... ;) | 19:43 |
| efried | edmondsw can also tackle the "one space after a period" work item. | 19:43 |
| esberglu | efried: Looks like this is breaking us. Currently failed the jenkins check but it's W+1. Got time to help look? | 19:43 |
| esberglu | https://review.openstack.org/#/c/460177 | 19:43 |
| efried | looking. | 19:43 |
| edmondsw | efried I knew you'd love that one... | 19:43 |
| efried | I learned to touch-type in the 9th grade, and it was two spaces after a period. | 19:43 |
| edmondsw | efried yeah, same here... it's an adjustment | 19:44 |
| efried | I've been seeing this comment a lot lately, so I had to go look up the rationale. | 19:44 |
| edmondsw | efried I always get dinged on that, so sharing the love | 19:44 |
| efried | Distilled down, it's really "two spaces after a period if the font is fixed-width" | 19:44 |
| efried | ...which it is, in code. | 19:44 |
| efried | So I feel justified NOT changing my 25-year-ingrained habit. | 19:45 |
| esberglu | efried: Tons of | 19:45 |
| esberglu | "NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception." | 19:45 |
| esberglu | in n-cpu, looking in networking logs now | 19:45 |
| efried | But people seem to get pretty excited about it, and it's (barely) not worth the fight. | 19:45 |
| efried | esberglu Ugh, I remember we saw that intermittently earlier. Now we're seeing it all the time? | 19:46 |
| esberglu | On this commit chain yes | 19:46 |
| esberglu | Actually there are a ton of other errors in n-cpu too | 19:48 |
| esberglu | "ValidatorException: Insufficient available processing units on host for virtual machine 'pvm5-tempest-Ser-03f27746' (0.10 requested, 0.00 available)" | 19:48 |
| esberglu | I'm guessing that's whats actually hitting from this patch | 19:48 |
| esberglu | efried: If you look at _remove_deleted_instances_allocations in resource_tracker.py in that patch | 19:53 |
| esberglu | It should be logging a warning when deleting stale allocations | 19:53 |
| esberglu | I'm not seeing that warning anywhere in the logs | 19:54 |
| esberglu | So I think the allocations aren't being removed so it thinks there aren't any available hosts | 19:54 |
| efried | esberglu I don't understand how this could have changed it, though. | 19:58 |
| esberglu | Yeah I don't either. But we should be seeing that warning in the logs correct? | 19:59 |
| efried | esberglu Guess I would have to see the logs for a run with the code as it stood prior to this change set... | 20:00 |
| esberglu | I just looked at one I don't see it there either... | 20:01 |
| efried | esberglu Is the other one seeing the same error? | 20:01 |
| esberglu | efried: Nope | 20:02 |
| thorst | efried: no concern with the rename | 20:04 |
| *** smatzek has quit IRC | 20:12 | |
| *** edmondsw has quit IRC | 20:37 | |
| *** edmondsw has joined #openstack-powervm | 20:44 | |
| *** chas has joined #openstack-powervm | 20:48 | |
| *** chas has quit IRC | 20:52 | |
| *** smatzek has joined #openstack-powervm | 21:00 | |
| *** jwcroppe has quit IRC | 21:07 | |
| *** jwcroppe has joined #openstack-powervm | 21:08 | |
| *** chas has joined #openstack-powervm | 21:09 | |
| *** smatzek has quit IRC | 21:11 | |
| *** jwcroppe has quit IRC | 21:12 | |
| *** chas has quit IRC | 21:14 | |
| *** jpasqualetto has quit IRC | 21:16 | |
| *** jwcroppe has joined #openstack-powervm | 21:20 | |
| *** thorst has quit IRC | 21:22 | |
| *** jwcroppe has quit IRC | 21:25 | |
| *** chas has joined #openstack-powervm | 21:30 | |
| *** apearson has quit IRC | 21:32 | |
| *** chas has quit IRC | 21:34 | |
| *** chas has joined #openstack-powervm | 21:40 | |
| *** jwcroppe has joined #openstack-powervm | 21:43 | |
| *** chas has quit IRC | 21:54 | |
| *** chas has joined #openstack-powervm | 21:54 | |
| *** chas has quit IRC | 21:59 | |
| *** jwcroppe has quit IRC | 21:59 | |
| *** jwcroppe has joined #openstack-powervm | 21:59 | |
| *** mdrabe has quit IRC | 22:03 | |
| efried | esberglu http://184.172.12.213/56/467656/1/check/nova-out-of-tree-pvm/6bfb5fb/console.html two things here: | 22:04 |
| efried | One, not sure our patching strategy is working for non-master branches. | 22:04 |
| esberglu | Yeah definitely isn't. We don't do any checks for branches | 22:05 |
| efried | Two, we can remove the master patch from the patch list. | 22:05 |
| efried | cause it merged. | 22:05 |
| esberglu | efried: Just be aware that if we remove that from the patch list right now | 22:06 |
| esberglu | Any patches that aren't rebased such that the --strict patch is included | 22:06 |
| esberglu | Will fail | 22:06 |
| efried | I don't think so. | 22:06 |
| efried | Because it merged. | 22:07 |
| efried | uhm, maybe you're right. | 22:07 |
| esberglu | Yeah but CI checks out the change | 22:07 |
| esberglu | So it depends where the parent is | 22:07 |
| efried | Yeah, and it doesn't rebase it on master | 22:07 |
| esberglu | efried: I actually removed this patch earlier today and put it back on for that reason | 22:08 |
| esberglu | Was gonna remove it later on once there are less patches that aren't up to date | 22:09 |
| efried | okay, I dig it. | 22:09 |
| esberglu | We have green runs going though again finally. Not a good success rate but it's something | 22:10 |
| esberglu | Next week is gonna be tempest debug week | 22:10 |
| *** chas has joined #openstack-powervm | 22:15 | |
| efried | phew. | 22:18 |
| *** chas has quit IRC | 22:20 | |
| *** thorst has joined #openstack-powervm | 22:20 | |
| *** esberglu has quit IRC | 22:22 | |
| *** thorst has quit IRC | 22:23 | |
| *** esberglu has joined #openstack-powervm | 22:23 | |
| *** tjakobs has quit IRC | 22:23 | |
| *** esberglu has quit IRC | 22:28 | |
| *** edmondsw has quit IRC | 22:34 | |
| *** edmondsw has joined #openstack-powervm | 22:34 | |
| *** edmondsw has quit IRC | 22:39 | |
| *** thorst has joined #openstack-powervm | 22:53 | |
| *** chas has joined #openstack-powervm | 22:56 | |
| *** chas has quit IRC | 23:01 | |
| *** thorst has quit IRC | 23:08 | |
| *** k0da has quit IRC | 23:17 | |
| *** apearson has joined #openstack-powervm | 23:31 | |
| *** edmondsw has joined #openstack-powervm | 23:35 | |
| *** chas has joined #openstack-powervm | 23:37 | |
| *** edmondsw has quit IRC | 23:40 | |
| *** chas has quit IRC | 23:42 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!