| *** openstack has joined #openstack-powervm | 00:17 | |
| *** edmondsw has quit IRC | 00:19 | |
| *** BorD_ has joined #openstack-powervm | 00:24 | |
| *** thorst has joined #openstack-powervm | 00:26 | |
| *** thorst has quit IRC | 00:27 | |
| *** tonyb_ is now known as tonyb | 00:57 | |
| *** smatzek has joined #openstack-powervm | 01:32 | |
| *** thorst has joined #openstack-powervm | 01:38 | |
| *** thorst has quit IRC | 01:56 | |
| *** thorst has joined #openstack-powervm | 01:56 | |
| *** esberglu has quit IRC | 02:00 | |
| *** thorst has quit IRC | 02:00 | |
| *** edmondsw has joined #openstack-powervm | 02:03 | |
| *** edmondsw has quit IRC | 02:08 | |
| *** chhavi has joined #openstack-powervm | 03:02 | |
| *** smatzek has quit IRC | 03:05 | |
| *** edmondsw has joined #openstack-powervm | 03:51 | |
| *** edmondsw has quit IRC | 03:56 | |
| *** thorst has joined #openstack-powervm | 03:57 | |
| *** thorst has quit IRC | 04:01 | |
| *** chhavi has quit IRC | 04:13 | |
| *** mdrabe has quit IRC | 04:40 | |
| *** tjakobs has joined #openstack-powervm | 05:06 | |
| *** tjakobs has quit IRC | 05:26 | |
| *** chhavi has joined #openstack-powervm | 05:38 | |
| *** edmondsw has joined #openstack-powervm | 05:39 | |
| *** edmondsw has quit IRC | 05:44 | |
| *** thorst has joined #openstack-powervm | 05:58 | |
| *** thorst has quit IRC | 06:02 | |
| *** k0da has joined #openstack-powervm | 07:05 | |
| *** k0da has quit IRC | 07:21 | |
| *** edmondsw has joined #openstack-powervm | 07:27 | |
| *** edmondsw has quit IRC | 07:32 | |
| *** thorst has joined #openstack-powervm | 07:58 | |
| *** thorst has quit IRC | 08:03 | |
| *** k0da has joined #openstack-powervm | 08:45 | |
| *** edmondsw has joined #openstack-powervm | 09:15 | |
| *** edmondsw has quit IRC | 09:20 | |
| *** thorst has joined #openstack-powervm | 09:49 | |
| *** thorst has quit IRC | 09:59 | |
| *** thorst has joined #openstack-powervm | 10:17 | |
| *** smatzek has joined #openstack-powervm | 10:55 | |
| *** edmondsw has joined #openstack-powervm | 11:03 | |
| *** edmondsw has quit IRC | 11:08 | |
| *** thorst has quit IRC | 11:16 | |
| *** thorst has joined #openstack-powervm | 11:47 | |
| *** thorst has quit IRC | 11:52 | |
| *** thorst has joined #openstack-powervm | 12:08 | |
| *** edmondsw has joined #openstack-powervm | 12:15 | |
| *** efried has joined #openstack-powervm | 12:21 | |
| *** jpasqualetto has joined #openstack-powervm | 12:27 | |
| *** kylek3h has joined #openstack-powervm | 12:31 | |
| *** dwayne has quit IRC | 12:33 | |
| *** efried has quit IRC | 12:34 | |
| *** esberglu has joined #openstack-powervm | 12:37 | |
| *** esberglu has quit IRC | 12:39 | |
| *** esberglu has joined #openstack-powervm | 12:39 | |
| *** jpasqualetto has quit IRC | 12:40 | |
| *** mdrabe has joined #openstack-powervm | 12:42 | |
| *** esberglu has quit IRC | 12:44 | |
| *** jpasqualetto has joined #openstack-powervm | 12:50 | |
| *** esberglu has joined #openstack-powervm | 12:51 | |
| *** efried has joined #openstack-powervm | 12:53 | |
| *** jpasqualetto has quit IRC | 12:55 | |
| *** jwcroppe has quit IRC | 12:55 | |
| -openstackstatus- NOTICE: restarting gerrit to address slowdown issues | 12:56 | |
| esberglu | #startmeeting powervm_driver_meeting | 13:01 |
|---|---|---|
| openstack | Meeting started Tue Jun 20 13:01:24 2017 UTC and is due to finish in 60 minutes. The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:01 |
| openstack | The meeting name has been set to 'powervm_driver_meeting' | 13:01 |
| esberglu | #link https://etherpad.openstack.org/p/powervm_driver_meeting_agenda | 13:01 |
| efried | o/ | 13:02 |
| esberglu | Created an agenda etherpad. If everyone could start adding discussion items to topics prior to the meetings starting next week that would be great | 13:02 |
| esberglu | I will resend the event invites with the link included as well | 13:02 |
| efried | Let's do a roll call. That will determine what we can discuss today. | 13:03 |
| efried | thorst edmondsw chhavi yt? | 13:04 |
| efried | mdrabe ? | 13:04 |
| edmondsw | o/ | 13:04 |
| mdrabe | o/ | 13:05 |
| chhavi | yes, in call little slow | 13:05 |
| efried | It would be nice to have Jay here. | 13:05 |
| efried | The main thing I know of that we need to discuss is iSCSI code & testing. | 13:05 |
| esberglu | Alright lets get started | 13:06 |
| esberglu | #topic In-Tree Driver | 13:06 |
| esberglu | #link https://etherpad.openstack.org/p/powervm-in-tree-todos | 13:06 |
| edmondsw | I believe the support matrix changes merged | 13:06 |
| edmondsw | what is the status of the other half of the docs changes we need to make? | 13:07 |
| edmondsw | the hypervisors devref | 13:07 |
| esberglu | I've been putting that off to work CI. I guess I didn't think it was that time sensitive? | 13:07 |
| esberglu | If it is I can get it done | 13:07 |
| edmondsw | getting the CI working again has definitely been higher priority, but I think we want those doc changes to be completed with Pike | 13:08 |
| edmondsw | "completed" to the level of the support we have in the Pike code, that is | 13:08 |
| efried | Agreed | 13:09 |
| edmondsw | efried, I assume we have some time yet before they lock down that kind of change for Pike, yes? | 13:09 |
| esberglu | I'm out friday. So I will get a draft up before then and then fix any comments, etc. early next week. Sound good? | 13:09 |
| edmondsw | sounds good | 13:09 |
| efried | Yes, we have time for pike. | 13:10 |
| efried | https://releases.openstack.org/pike/schedule.html | 13:10 |
| edmondsw | so we have a few weeks there | 13:11 |
| efried | 7/27 is feature freeze, it would be nice to have this in before that. | 13:11 |
| edmondsw | yep | 13:11 |
| efried | though if it does slip, this isn't really a feature per se. | 13:11 |
| efried | I.e. no code impact. | 13:11 |
| efried | So not really crucial to get it in. | 13:12 |
| edmondsw | the string freeze may be the more important date | 13:12 |
| edmondsw | soft string freeze is the same week | 13:12 |
| esberglu | Anything else we need to discuss in-tree? | 13:14 |
| edmondsw | I need to send a note to the ML about https://review.openstack.org/#/c/471773/ ... still going over how I want to word that | 13:14 |
| efried | String freeze relates to translation, though, right? | 13:15 |
| edmondsw | efried right... and docs would be translated, no? | 13:15 |
| efried | Was about to ask. I don't know. | 13:15 |
| edmondsw | I assume yes | 13:15 |
| edmondsw | now that I think of it, though... maybe not? | 13:15 |
| esberglu | Either way lets just get it in by then | 13:15 |
| edmondsw | yeah | 13:15 |
| edmondsw | efried did you have anything to talk about IT? | 13:16 |
| efried | no | 13:16 |
| edmondsw | so next topic | 13:16 |
| esberglu | #topic OOT Driver | 13:17 |
| edmondsw | #topic Out-of-Tree Driver | 13:17 |
| edmondsw | anyone have anything to discuss OOT? | 13:17 |
| esberglu | I don't | 13:18 |
| edmondsw | I have a change up for the driver to use CONF.host, like we changed IT, but I was waiting for the CI to pass before pushing that | 13:19 |
| edmondsw | anybody else working on backporting IT changes to OOT? | 13:19 |
| efried | I need to do some testing of the power-off changes. | 13:19 |
| efried | Then merge the pypowervm side of that. | 13:19 |
| efried | Then coordinate a new pypowervm release with the iSCSI stuff, which we need to go into a bit. | 13:20 |
| efried | According to Hsien, the REST/core side of iSCSI stuff should be copacetic at this point. | 13:20 |
| efried | chhavi was teed up to talk to Jay and see whether the latest dev nvl+pypowervm had been tested with the OOT nova-powervm changes. | 13:21 |
| efried | Because we don't want to cut a new pypowervm release only to test iSCSI and find we need another one right away. | 13:21 |
| edmondsw | yep | 13:21 |
| efried | So those two - power-off & iSCSI - need to be tested out, then we cut pypowervm, then we funnel everything through g-r, then we merge the nova-powervm iSCSI change; and I'll also put up a change (IT & OOT) to use the new power-off method. | 13:22 |
| edmondsw | we should probably send an email since Jay hasn't been on the last few meetings | 13:22 |
| efried | chhavi Did you get a chance to talk to Jay today? | 13:22 |
| edmondsw | chhavi there are also still comments to address on https://review.openstack.org/#/c/467599/ | 13:24 |
| edmondsw | we should probably move on | 13:26 |
| edmondsw | #topic CI | 13:26 |
| edmondsw | esberglu, everything running now? | 13:26 |
| esberglu | CI is publishing and running, not passing | 13:26 |
| chhavi | yes I got the environment, | 13:26 |
| chhavi | need to load and verify | 13:27 |
| efried | Great chhavi. | 13:27 |
| efried | Do you think you'll know by tomorrow? | 13:27 |
| esberglu | Stable branches are passing | 13:28 |
| esberglu | OOT is failing 1 test | 13:28 |
| esberglu | OOT - http://184.172.12.213/63/475563/4/check/nova-out-of-tree-pvm/d0ae5a6/powervm_os_ci.html | 13:28 |
| esberglu | IT is failing a bunch, might have to do more local.conf work there | 13:28 |
| esberglu | IT - http://184.172.12.213/48/475448/7/check/nova-in-tree-pvm/e2b585a/powervm_os_ci.html | 13:28 |
| esberglu | And then some random stacks are failing with this | 13:29 |
| esberglu | SSLError - http://184.172.12.213/48/475448/7/check/nova-out-of-tree-pvm/3884cbb/console.html | 13:29 |
| esberglu | Planning on diving into those in that order today | 13:29 |
| esberglu | Other than that I have a bunch of small reviews for neo-os-ci and powervm-ci | 13:30 |
| esberglu | And I tested our config on devstack in the staging environment | 13:30 |
| esberglu | The latest patch there is ready to go | 13:30 |
| esberglu | Adding devstack to the silent pipeline I'm talking about | 13:31 |
| edmondsw | what does that mean, silent pipeline? | 13:31 |
| esberglu | It won't publish any results, but it will run on every patch to devstack | 13:32 |
| esberglu | We currently do that for neutron and ceilometer | 13:32 |
| *** jwcroppe has joined #openstack-powervm | 13:32 | |
| esberglu | The idea is that it should help identify breaking devstack changes ideally before they merge but in reality probably just faster | 13:33 |
| esberglu | Other than that just working through | 13:34 |
| esberglu | https://etherpad.openstack.org/p/powervm_ci_todos | 13:34 |
| esberglu | as able | 13:34 |
| esberglu | That's it for me, anyone have anything before we move on? | 13:34 |
| *** efried has quit IRC | 13:35 | |
| mdrabe | I'm still trying to figure out the 2.46.2 issue | 13:35 |
| esberglu | mdrabe: You planning on working on that today at all? | 13:36 |
| mdrabe | Yea I'm gonna run the bisect today to try and identify the commit | 13:36 |
| mdrabe | If it is with Jenkins | 13:36 |
| esberglu | Sweet. Just let me know when you need the env. back | 13:36 |
| mdrabe | Yup yup | 13:37 |
| esberglu | Next topic | 13:37 |
| esberglu | #topic Driver Testing | 13:37 |
| esberglu | Talked a little bit above, anything further discussion needed? | 13:38 |
| esberglu | Doesn't sound like it | 13:39 |
| esberglu | #topic Open Discussion | 13:39 |
| esberglu | Any final thoughts? | 13:40 |
| edmondsw | mdrabe thorst I need to talk to one or both of you about how nova-powervm gets pulled into PowerVC | 13:40 |
| edmondsw | so fyi, I'll be pinging | 13:41 |
| mdrabe | edmondsw DM me | 13:41 |
| edmondsw | yep | 13:41 |
| *** efried has joined #openstack-powervm | 13:42 | |
| esberglu | thanks for joining | 13:43 |
| efried | Hello? What'd I miss? | 13:43 |
| esberglu | efried: Nothing | 13:43 |
| esberglu | Unless you had something else | 13:43 |
| efried | Last thing I saw was the SSLError | 13:43 |
| *** jpasqualetto has joined #openstack-powervm | 13:43 | |
| efried | I apparently got disconnected but my IRC client didn't see fit to let me know about it. | 13:43 |
| efried | If we're done, I'll catch up with the meetbot. | 13:44 |
| edmondsw | nothing too significant... look for some CI patches that esberglu has proposed | 13:44 |
| efried | ack | 13:44 |
| edmondsw | I think we're done | 13:44 |
| esberglu | efried: Just click the link at the end of the meeting if you want to reread, but nothing big | 13:44 |
| esberglu | #endmeeting | 13:44 |
| openstack | Meeting ended Tue Jun 20 13:44:53 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:44 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-06-20-13.01.html | 13:44 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-06-20-13.01.txt | 13:44 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-06-20-13.01.log.html | 13:44 |
| edmondsw | efried do you know if it's possible to have nova accept both forms of its UIs... i.e. with and without tenant_id in the URL? | 13:46 |
| efried | UI or URI? | 13:46 |
| edmondsw | URI | 13:46 |
| edmondsw | I actually meant API... typing while in another meeting | 13:47 |
| efried | I don't know whether there's a level at which it will accept either form - though I imagine there probably is (deprecation cycle) - but note that if you're trying to guarantee one behavior or the other, you can specify a microversion. | 13:47 |
| edmondsw | I don't know if they did microversions with that | 13:48 |
| edmondsw | oddly enough | 13:48 |
| efried | Oh, I didn't get a chance to mention this, but I'll be attending bits of the TCC this week, so will have a bit less time on mainline work. | 13:48 |
| edmondsw | ack | 13:49 |
| efried | edmondsw You could ask that question in #openstack-nova or in the ML and get a pretty quick answer, I would think. | 13:50 |
| edmondsw | yeah, I just thought I'd try you first because I generally get no answer in #openstack-nova :) | 13:50 |
| edmondsw | I'll ask there in a bit | 13:50 |
| esberglu | efried: edmondsw: thorst: http://184.172.12.213/63/475563/4/check/nova-out-of-tree-pvm/d0ae5a6/powervm_os_ci.html | 14:27 |
| esberglu | https://github.com/openstack/neutron/blob/master/neutron/tests/tempest/api/admin/test_agent_management.py#L26-L31 | 14:27 |
| esberglu | This is failing because admin_client.list_agents() is only returning 1 agent | 14:27 |
| esberglu | And the test assumes that there are at least 2 agents, agents[0] and agents[1] | 14:28 |
| esberglu | Is this a bug there? I'm not sure what the dyn_agent is supposed to be referring to | 14:28 |
| esberglu | Figured I would see if any of you knew before I looked any deeped | 14:28 |
| esberglu | *deeper | 14:28 |
| edmondsw | not sure | 14:29 |
| *** tjakobs has joined #openstack-powervm | 14:33 | |
| *** thorst has quit IRC | 14:37 | |
| *** dwayne has joined #openstack-powervm | 14:42 | |
| *** thorst has joined #openstack-powervm | 14:42 | |
| *** thorst has quit IRC | 14:47 | |
| efried | esberglu Yeah, I don't know what kind of agents we're looking for here. | 15:02 |
| efried | Have you deeped yet? | 15:02 |
| esberglu | No I was stacking a test node to look into the in tree stuff | 15:03 |
| esberglu | About to | 15:03 |
| *** dwayne has quit IRC | 15:05 | |
| esberglu | efried: edmondsw: Brought this up in openstack-neutron | 15:17 |
| esberglu | They agreed that it was a bug with the test | 15:17 |
| edmondsw | cool | 15:17 |
| esberglu | I'm going to open a bug for it, then we can disable that test until it gets resolved | 15:17 |
| efried | esberglu Beaut. Are you planning to propose the fix? | 15:19 |
| efried | That would be good exposure. | 15:19 |
| esberglu | efried: I'm gonna get IT CI fixed first. But yeah if no one from neutron gets there before that point | 15:20 |
| efried | cool | 15:20 |
| *** thorst has joined #openstack-powervm | 15:33 | |
| *** thorst has quit IRC | 15:34 | |
| *** thorst has joined #openstack-powervm | 15:34 | |
| esberglu | efried: edmondsw: 5484 | 15:34 |
| efried | esberglu Please recheck https://review.openstack.org/#/c/474799/ when you think OOT is ready. (edmondsw FYI) | 15:34 |
| esberglu | efried: Should be ready as soon as 5484 merges | 15:34 |
| efried | esberglu Oh, if we were running more than one agent, it would work? | 15:35 |
| efried | We can run the pvm-sriov agent, no harm no foul. | 15:35 |
| esberglu | efried: Yeah we could probably just toss the sriov agent on | 15:35 |
| efried | That would be a local.conf change. | 15:35 |
| edmondsw | esberglu I thought it was just a single test | 15:35 |
| esberglu | edmondsw: It's failing the resource_setup for that group of tests | 15:35 |
| esberglu | So it's 5 | 15:35 |
| efried | Even if it was, I think adding the sriov agent so we can keep the tests is a better workaround. | 15:35 |
| edmondsw | esberglu oh, nm... it was in setup... right | 15:35 |
| esberglu | efried: I would like to test that out before we do it | 15:36 |
| esberglu | But I think that should work | 15:36 |
| edmondsw | esberglu how long will trying that take? | 15:37 |
| esberglu | So I say we just merge this for now. Then we can remove if the sriov agent bit works | 15:37 |
| efried | esberglu So you want to disable the tests for now, then test it the other way (with sriov enabled), then twiddle it? | 15:37 |
| efried | right, okay. | 15:37 |
| efried | +2 with that caveat. | 15:37 |
| esberglu | edmondsw: I would need to spawn, stack, and run tempest. I could just run those 5 tests though. Maybe an hour? | 15:37 |
| esberglu | But I would rather get IT working and come back to this | 15:37 |
| efried | Long as we don't lose sight of it. | 15:38 |
| efried | How many tests that we've put in the "further investigation" part of the skip list have we actually come back and re-enabled? | 15:38 |
| efried | I'm afraid of our CI tech debt curve trending ever upward. | 15:38 |
| edmondsw | +2 | 15:38 |
| esberglu | efried: Hopefully I should have time to start making progress with the addition of mdrabe | 15:39 |
| efried | okay. | 15:39 |
| esberglu | And the recent changes essentially getting rid of redeploys | 15:40 |
| edmondsw | efried I've been wondering if we should actually be disabling more in the CI, to improve performance | 15:40 |
| edmondsw | efried basically, only test things that have anything to do with us | 15:40 |
| efried | I don't think our runtime is long enough to be worth that, do you? | 15:40 |
| edmondsw | e.g. we've got tests disabled for identity v2 API... I see no value in ever re-enabling those | 15:40 |
| edmondsw | efried I don't know | 15:40 |
| edmondsw | how long is it? | 15:40 |
| efried | More to the point, how long is it compared to the other 3rd-party CIs. | 15:41 |
| efried | And it's on par. | 15:41 |
| efried | 45-90m, depending. | 15:41 |
| edmondsw | yeah, that seems long to me, regardless of what others take | 15:41 |
| edmondsw | I'm not saying we have to do that right now... but it's an idea worth mulling over | 15:41 |
| efried | I would be okay disabling tests that never hit the compute driver. | 15:42 |
| edmondsw | it wouldn't just help with perf... would also help with stability if we don't have to worry about those disabled things breaking us | 15:42 |
| efried | Not sure how many of those there are. | 15:42 |
| edmondsw | efried that's exactly what I'm saying | 15:42 |
| efried | But if it hits the compute driver at all, even if it's supposedly testing something unrelated, we should keep it. | 15:42 |
| edmondsw | absolutely | 15:42 |
| efried | Okay, sure, add it to the etherpad if it ain't there already. | 15:43 |
| edmondsw | will do | 15:43 |
| edmondsw | I added to https://etherpad.openstack.org/p/powervm_ci_todos | 15:45 |
| edmondsw | I just rechecked https://review.openstack.org/#/c/474799... let's see if it passes :) | 15:46 |
| esberglu | edmondsw: What you aren't confident after a week+ of failures ;) | 15:49 |
| *** k0da has quit IRC | 15:54 | |
| *** efried has quit IRC | 15:56 | |
| edmondsw | esberglu yeah... I want to see it :) | 16:04 |
| edmondsw | esberglu I'm curious about the answer to efried's question in 5480 | 16:05 |
| esberglu | edmondsw: I was gonna open one then add it to the comment before merging | 16:06 |
| edmondsw | tx | 16:06 |
| *** thorst has quit IRC | 16:09 | |
| *** mdrabe has quit IRC | 16:13 | |
| edmondsw | esberglu 5479... why does one file include devstack-pvm-ci-check and the other doesn't? | 16:14 |
| *** mdrabe has joined #openstack-powervm | 16:19 | |
| esberglu | edmondsw: Well we don't want devstack-pvm-ci-check to run every time there is a devstack patch | 16:22 |
| esberglu | edmondsw: I think what we really want is to | 16:22 |
| esberglu | a) get rid of either pvm-ci-check or noop-check-communication, they are essentially the same thing | 16:23 |
| esberglu | b) Only run whichever of those 2 we keep against the ci-sandbox project | 16:23 |
| edmondsw | and you intend to do that separately, I assume | 16:24 |
| esberglu | edmondsw: I left a comment (a) explained but before I realized we needed (b). Yes, I think a separate patch is more appropriate | 16:25 |
| *** thorst has joined #openstack-powervm | 16:25 | |
| edmondsw | esberglu ah, I see it | 16:26 |
| *** jwcroppe has quit IRC | 16:32 | |
| mdrabe | esberglu: Found the commit in Jenkins causing the problem ae402ba139df82f3d105b9e070444f1849543fce | 16:34 |
| mdrabe | I didn't see a bug in nodepool when I checked last week, so I think I'll have to create one, will probably follow up with a fix | 16:34 |
| edmondsw | esberglu are the CI failures in https://review.openstack.org/#/c/474799/ something you can explain? | 16:35 |
| edmondsw | the driver wasn't able to connect to NovaLink | 16:35 |
| edmondsw | so the compute service didn't come up properly | 16:35 |
| *** thorst has quit IRC | 16:35 | |
| esberglu | edmondsw: I'm confident that it's something in the patch. Hold on let me take a quick look | 16:36 |
| edmondsw | esberglu I sure hope not, and I would have expected problems with the patch to look differently | 16:36 |
| edmondsw | but you may be right | 16:36 |
| esberglu | edmondsw: Well not 100% confident. But I haven't seen this anywhere else OOT in the last couple days. And the only thing that has changed OOT is log publishing and disabling those 5 tests | 16:37 |
| esberglu | So unless this starts hitting other patches... | 16:37 |
| edmondsw | I'll kick off a recheck and see if it happens again | 16:37 |
| esberglu | edmondsw: Ok. I'll keep an eye out for it too | 16:38 |
| edmondsw | tx | 16:38 |
| *** thorst has joined #openstack-powervm | 16:52 | |
| *** thorst has quit IRC | 16:55 | |
| *** thorst has joined #openstack-powervm | 17:00 | |
| *** jpasqualetto has quit IRC | 17:02 | |
| *** efried has joined #openstack-powervm | 17:02 | |
| *** jwcroppe has joined #openstack-powervm | 17:03 | |
| *** jpasqualetto has joined #openstack-powervm | 17:11 | |
| *** openstackgerrit has joined #openstack-powervm | 17:14 | |
| openstackgerrit | Merged openstack/ceilometer-powervm master: Fix html_last_updated_fmt for Python3 https://review.openstack.org/474861 | 17:14 |
| efried | edmondsw I think in order to get any attention on https://review.openstack.org/#/c/472794/ you're going to have to either hit the ML or finish the UT and get rid of the WIP. | 17:18 |
| efried | Should do it sooner rather than later; the probability of getting non-feature patches through is going to be decreasing steadily between now and feature freeze. | 17:19 |
| efried | And yeah, yeah, I still owe you a comeback on https://review.openstack.org/#/c/471773/ | 17:19 |
| efried | I think last week I convinced myself that the algorithm you've worked up is roughly as simple as it's gonna get, which I'm pretty sad about. | 17:20 |
| efried | I think there's a couple of things we can/should do to make the code itself smaller, like combining the exception cases. | 17:20 |
| efried | Will comment accordingly. | 17:20 |
| *** jpasqualetto has quit IRC | 17:24 | |
| *** jpasqualetto has joined #openstack-powervm | 17:24 | |
| esberglu | efried: edmondsw: Enabling the sriov agent allowed those tests to pass. Patch will be up in a few | 17:29 |
| efried | cool | 17:29 |
| *** jwcroppe_ has joined #openstack-powervm | 17:35 | |
| edmondsw | efried yeah, I need to get back to that | 17:36 |
| *** jwcroppe has quit IRC | 17:37 | |
| esberglu | edmondsw: efried: 5486 | 17:38 |
| edmondsw | needs a comment | 17:40 |
| *** dwayne has joined #openstack-powervm | 17:41 | |
| efried | edmondsw Comments on https://review.openstack.org/#/c/471773/ | 17:42 |
| *** jwcroppe_ has quit IRC | 17:43 | |
| edmondsw | efried tx | 17:43 |
| esberglu | edmondsw: Done | 17:43 |
| esberglu | edmondsw: http://184.172.12.213/18/452818/7/check/nova-in-tree-pvm/59fdeb5/powervm_os_ci.html | 17:46 |
| esberglu | Not just your patch | 17:46 |
| edmondsw | esberglu unfortunately that doesn't mean it's not the same issue... that change was already merged in-tree | 17:47 |
| edmondsw | efried ^ | 17:47 |
| *** efried has quit IRC | 17:47 | |
| edmondsw | talking the CONF.host change | 17:48 |
| esberglu | edmondsw: Oh. That would make sense. Because this looks very similar to the current IT runs | 17:48 |
| esberglu | Oh wait this is an IT run | 17:48 |
| edmondsw | right | 17:48 |
| edmondsw | https://review.openstack.org/#/c/474799 is OOT | 17:48 |
| esberglu | edmondsw: Yeah still makes sense. I thought I linked an oot run in just now | 17:48 |
| esberglu | edmondsw: I'm gonna try a test run with that change reverted IT and see what happens | 17:49 |
| edmondsw | esberglu yeah | 17:50 |
| esberglu | edmondsw: You have a link handy for the IT change? | 17:51 |
| edmondsw | esberglu https://review.openstack.org/#/c/473975/ | 17:52 |
| esberglu | thx | 17:52 |
| *** dwayne has quit IRC | 17:54 | |
| *** thorst has quit IRC | 18:05 | |
| *** dwayne has joined #openstack-powervm | 18:14 | |
| *** chhavi has quit IRC | 18:18 | |
| *** jwcroppe has joined #openstack-powervm | 18:20 | |
| *** jwcroppe has quit IRC | 18:25 | |
| *** jwcroppe has joined #openstack-powervm | 18:38 | |
| *** jwcroppe has quit IRC | 18:44 | |
| *** thorst has joined #openstack-powervm | 18:58 | |
| *** mdrabe_ has joined #openstack-powervm | 18:59 | |
| *** mdrabe has quit IRC | 19:00 | |
| *** jwcroppe has joined #openstack-powervm | 19:03 | |
| *** jwcroppe has quit IRC | 19:06 | |
| *** thorst has quit IRC | 19:15 | |
| *** thorst has joined #openstack-powervm | 19:15 | |
| *** jwcroppe has joined #openstack-powervm | 19:18 | |
| *** thorst has quit IRC | 19:20 | |
| *** edmondsw has quit IRC | 19:29 | |
| *** jwcroppe has quit IRC | 19:37 | |
| *** k0da has joined #openstack-powervm | 19:48 | |
| *** edmondsw has joined #openstack-powervm | 20:00 | |
| *** jwcroppe has joined #openstack-powervm | 20:04 | |
| *** smatzek has quit IRC | 20:09 | |
| *** jwcroppe has quit IRC | 20:12 | |
| *** jwcroppe has joined #openstack-powervm | 20:13 | |
| *** k0da has quit IRC | 20:14 | |
| *** kylek3h has quit IRC | 20:16 | |
| *** jwcroppe has quit IRC | 20:17 | |
| *** mdrabe_ is now known as mdrabe | 20:18 | |
| *** efried has joined #openstack-powervm | 20:20 | |
| efried | edmondsw I'm gonna +W https://review.openstack.org/#/c/468565/ k? | 20:31 |
| edmondsw | efried please | 20:31 |
| edmondsw | efried I'm trying to stack on my neo and got "Failed to update apt repos, we're dead now", any ideas? | 20:33 |
| efried | Unstack. Run apt-get update and whack away at it until it succeeds. Then restack. | 20:33 |
| efried | edmondsw Is this on a system that successfully stacked before (since it was last reinstalled)? | 20:35 |
| edmondsw | no | 20:35 |
| efried | okay. Then your sources are probably busted. | 20:35 |
| edmondsw | efried yeah, there are a bunch of warnings about things appearing more than once | 20:35 |
| edmondsw | would that do it? | 20:36 |
| efried | Best bet is probably to replace stuff in /etc/apt/sources* with whatever's on your other neo that worked. | 20:36 |
| efried | No, duplicates shouldn't be breaking it. | 20:36 |
| *** smatzek has joined #openstack-powervm | 20:38 | |
| esberglu | edmondsw: In tree passed reverting the CONF commit | 20:39 |
| efried | whoah, I must have missed something. | 20:39 |
| efried | We talking about changing MTMS to CONF.host? | 20:39 |
| esberglu | efried: Yeah | 20:40 |
| openstackgerrit | Merged openstack/nova-powervm master: Remove volume_group_vios_name https://review.openstack.org/468565 | 20:40 |
| edmondsw | efried yeah... apparently that did break everything | 20:40 |
| edmondsw | trying to figure out why | 20:40 |
| efried | Do you still have a CI failure for that? | 20:40 |
| esberglu | efried: One sec | 20:41 |
| efried | ...as in, logs I can look at. | 20:41 |
| edmondsw | https://review.openstack.org/#/c/474799 | 20:41 |
| edmondsw | for one | 20:41 |
| efried | Yeah, I guess it would be anything. | 20:41 |
| efried | edmondsw Well, that one is OOT. | 20:41 |
| efried | I would prefer apples:apples | 20:41 |
| esberglu | http://184.172.12.213/48/475448/7/check/nova-in-tree-pvm/e2b585a/ | 20:41 |
| edmondsw | they're the same, but here's an IT: http://184.172.12.213/18/452818/7/check/nova-in-tree-pvm/59fdeb5/ | 20:41 |
| *** jwcroppe has joined #openstack-powervm | 20:46 | |
| *** jwcroppe has quit IRC | 20:51 | |
| *** jwcroppe has joined #openstack-powervm | 20:51 | |
| *** jwcroppe has quit IRC | 20:56 | |
| efried | edmondsw esberglu Here's the problem: https://review.openstack.org/#/c/473975/1/nova/virt/powervm/host.py@57 | 20:57 |
| efried | It would seem as though data['hypervisor_hostname'] needs to match something in the list get_available_nodes returns. | 20:58 |
| edmondsw | efried how did you figure that out? | 20:59 |
| efried | Logs & code | 20:59 |
| edmondsw | efried I figured... I was thinking more specifically :) | 20:59 |
| efried | Jun 20 08:05:35.660790 powervm-ci-powervm-devstacked-15470 nova-compute[6673]: DEBUG nova.compute.resource_tracker [None req-8d9e66da-cf03-4a01-ba08-c79df4328516 None None] Hypervisor/Node resource view: name=8247-21L*212A5CA free_ram=205056MB free_disk=917.44GB free_vcpus=7 pci_devices=None {{(pid=6673) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:747}} | 20:59 |
| efried | Note name=<mtms> | 20:59 |
| efried | Then... | 20:59 |
| efried | ComputeHostNotFound: Compute host powervm-ci-powervm-devstacked-15470 could not be found. | 20:59 |
| efried | From compute_node_get_all_by_host | 21:00 |
| edmondsw | I missed that name=<mtms> | 21:00 |
| efried | Which looks in resources['hypervisor_hostname'], where resources comes from the aforementioned. | 21:00 |
| efried | I'll put up a patch to confirm. | 21:00 |
| edmondsw | tx | 21:00 |
| esberglu | Sweet | 21:00 |
| edmondsw | efried patch will be IT, I assume? | 21:01 |
| efried | yeah | 21:01 |
| edmondsw | then we can just add to the existing OOT patch | 21:01 |
| efried | Yeah, you can patch that up in parallel if you like | 21:01 |
| esberglu | Once that merges all CI should be passing (how many times have I said that now?) | 21:01 |
| edmondsw | not rush on OOT since that hasn't merged yet | 21:01 |
| edmondsw | I'll wait and see exactly what you do IT so it matches | 21:02 |
| efried | edmondsw I'm not convinced this is the right way to go. | 21:03 |
| efried | Just gonna make sure it works. | 21:03 |
| efried | https://review.openstack.org/#/c/475926/ | 21:04 |
| efried | I'm thinking it might be best to get one of those values from the other. | 21:04 |
| efried | But I'm not sure how exactly that would work. | 21:04 |
| efried | Anyway, let's see if this flies first. | 21:04 |
| edmondsw | efried that's what I would have done | 21:04 |
| openstackgerrit | Merged openstack/networking-powervm master: Updated from global requirements https://review.openstack.org/472940 | 21:13 |
| openstackgerrit | Matthew Edmonds proposed openstack/nova-powervm master: Use CONF.host for powervm nodename https://review.openstack.org/474799 | 21:14 |
| esberglu | efried: edmondsw: https://github.com/powervm/powervm-ci/blob/master/devstack/prep_devstack.sh#L245-L249 | 21:15 |
| esberglu | Right now we have a count on discover_hosts but we don't fail if we reach the count | 21:15 |
| esberglu | Thinking that perhaps somehow the run could go through without that command succeeding in the timeframe | 21:16 |
| esberglu | But I would rather just fail there if we reach the timeout | 21:17 |
| edmondsw | esberglu do we even need the loop now that --strict is there? | 21:17 |
| esberglu | edmondsw: Yeah | 21:17 |
| esberglu | Because we restart the services right before this | 21:17 |
| *** smatzek has quit IRC | 21:17 | |
| *** efried has quit IRC | 21:18 | |
| esberglu | So we hit an issue where the command was "succeeding" but only because the service wasn't all the way up | 21:18 |
| edmondsw | I thought --strict blocked | 21:18 |
| esberglu | UURC | 21:18 |
| esberglu | IIRC | 21:18 |
| esberglu | Wait maybe not | 21:18 |
| esberglu | Yeah that's why | 21:19 |
| esberglu | Because the first discover_hosts might fail because the service isn't up | 21:20 |
| esberglu | So we keep retrying | 21:20 |
| esberglu | Until the service is up and it finds a host | 21:20 |
| esberglu | Or we hit the timeout | 21:20 |
| esberglu | s/"succeeding"/"failing"/ | 21:20 |
| *** jwcroppe has joined #openstack-powervm | 21:22 | |
| *** efried has joined #openstack-powervm | 21:22 | |
| efried | Stupid conference wifi resets after 24h. I lost some stuff there. | 21:27 |
| efried | I've got the change finished up and ready. Tempted to push it. | 21:27 |
| efried | edmondsw I'll wait for the results, I guess. But here's the UT delta FYI: http://paste.openstack.org/show/613220/ | 21:29 |
| efried | edmondsw Ugh, I should have caught this in the other change; should use self.flags from UT, not CONF. | 21:30 |
| efried | Mebbe I'll add that to mine. | 21:31 |
| edmondsw | efried yeah, go ahead | 21:32 |
| *** smatzek has joined #openstack-powervm | 21:32 | |
| edmondsw | I didn't even put the UT in the OOT patch, because it's essentially pointless and wasn't there before | 21:32 |
| efried | edmondsw "wasn't there before" is a good reason to put it in. | 21:32 |
| *** thorst has joined #openstack-powervm | 21:32 | |
| edmondsw | efried "essentially pointless" was my reason for not | 21:33 |
| efried | thorst Airport? | 21:33 |
| thorst | I'm inbetween connections | 21:33 |
| efried | I dunno how pointless it is. The UT I added proves we're getting it from CONF.host. | 21:33 |
| edmondsw | efried I didn't mean the UT you did... I meant the one I had made IT | 21:35 |
| *** kylek3h has joined #openstack-powervm | 21:38 | |
| edmondsw | efried I'm adding the UT you made for test_host to the OOT patch... it's a good change | 21:38 |
| efried | edmondsw Roger that. Let's get the CI passing both IT & OOT before we merge this time :) | 21:39 |
| edmondsw | efried shouldn't I be telling you that? ;) | 21:39 |
| edmondsw | :-P | 21:39 |
| efried | It wasn't *really* anyone's fault; it was a chain of miscues. I asked for core reviews while our CI was still failing; then mriedem asked me about the CI and I told him it was failing for unrelated reasons (which it was at the time - it wasn't getting past devstack). But then he merged it anyway, which was premature. | 21:40 |
| edmondsw | no worries, man | 21:40 |
| *** jpasqualetto has quit IRC | 21:40 | |
| esberglu | efried: Trying to push it off on me huh? lol | 21:40 |
| efried | Hah. No, I think mriedem is probably mostly to blame. | 21:40 |
| esberglu | Good call blame outside the team | 21:41 |
| efried | But I don't want to do *anything* that will encourage him to be even *less* eager to merge our stuff. | 21:41 |
| efried | As it is, if he notices, he'll be damn sure never to merge anything of ours again without our CI passing. | 21:41 |
| efried | esberglu I missed the discussion about --strict while my wifi was resetting, caught up on eavesdrop. I noticed that hole when you put up that change, but I figured you were okay letting it fail down the line. That said, I would also be fine with you failing it right there, which would be easier to diagnose. | 21:42 |
| esberglu | efried: Yeah your comment is why I eventually circled back | 21:43 |
| esberglu | The way it is now it will make it all the way through tempest with like test failures about cell mappings | 21:43 |
| esberglu | Which I hate | 21:43 |
| esberglu | *like 50 test failures | 21:44 |
| efried | esberglu Well, if we fail it right there, do we still get the syslogs so we'd be able to diagnose why the compute service didn't start? | 21:44 |
| esberglu | efried: Yep. The log output and publishing are post build steps that happen even on failure | 21:46 |
| efried | Great, then make it so. | 21:46 |
| esberglu | 5488 is already up for it | 21:46 |
| efried | BTW, I thought of a fairly easy way to isolate which tests would be candidates to skip because they don't hit our driver. | 21:46 |
| esberglu | efried: Sweet what's that | 21:47 |
| efried | Put up a patch where init_host raises an exception. | 21:47 |
| efried | See what passes. | 21:47 |
| efried | Not 100% sure if that would work. | 21:47 |
| *** thorst has quit IRC | 21:47 | |
| efried | Well, I'm sure anything that still passes would be legit to skip. | 21:47 |
| efried | But I'm not sure that hits everything we would want to skip. | 21:48 |
| esberglu | efried: At least something to start from. We can probably come up with some similar patch ideas and use the intersections to get a pretty decent list going | 21:49 |
| efried | Yeah. Anything more granular I would be concerned about missing tests we should have as we add function to our driver. | 21:49 |
| edmondsw | efried these CONF.host issues had me thinking the same thing :) | 21:49 |
| efried | Yeah, looking at that log, with CONF.host bustage, the compute service is basically worthless, though it actually does *run*. So anything that passes must not care about the comput host. | 21:50 |
| efried | It would be interesting to see the difference in passing tests between that and when the compute service just dies completely. | 21:51 |
| esberglu | efried: Weird your patch didn't get picked up by CI | 21:51 |
| esberglu | I ran a recheck | 21:51 |
| efried | Oh, then stop and let me post the final version. | 21:52 |
| esberglu | It's going this time | 21:52 |
| esberglu | efried: Go for it. When a new patch comes in it automatically aborts jobs on previous patches | 21:52 |
| efried | esberglu Done. | 21:52 |
| efried | edmondsw ^^ FYI | 21:52 |
| esberglu | efried: Oh you were just waiting to see if it worked so didn't want to kill it probably | 21:52 |
| efried | correct | 21:52 |
| efried | Cause I gotta bail here pretty soon. | 21:53 |
| efried | So I'm not going to be around to see the result tonight, but will check in a.m. | 21:53 |
| esberglu | efried: Sounds good have a good evening | 21:55 |
| edmondsw | esberglu efried I got a lot further stacking after fixing apt sources, but now got this: [ERROR] /home/neo/devstack/functions-common:602 git call failed: [git clone git://git.openstack.org/openstack/horizon.git /opt/stack/horizon --branch master] | 21:55 |
| efried | Try changing to github | 21:55 |
| edmondsw | that seems odd | 21:55 |
| edmondsw | hmm... if I can figure out where that came from | 21:55 |
| esberglu | edmondsw: We've had issues with g.o.o clones in the past. Had to change a bunch of them to github for CI | 21:56 |
| efried | edmondsw stackrc:GIT_BASE=${GIT_BASE:-git://git.openstack.org} | 21:56 |
| edmondsw | esberglu they're just inconsistent? | 21:56 |
| esberglu | Did you get some message about a bad index pack or something like that? | 21:56 |
| efried | Change to github.opestack.org I think | 21:56 |
| esberglu | edmondsw: Yeah they would just randomly fail sometimes. Othertimes would be getting like 10kB/s dl | 21:57 |
| esberglu | And we haven't noticed the same issue with github | 21:57 |
| efried | I don't think github.openstack.org is right - but that's the place to change it. | 21:57 |
| *** thorst has joined #openstack-powervm | 21:57 | |
| edmondsw | efried not just github.com? | 21:58 |
| efried | I don't remember. esberglu should know | 21:58 |
| esberglu | yeah github.com | 21:59 |
| *** smatzek has quit IRC | 21:59 | |
| edmondsw | esberglu if we've hit this a bunch, why don't we have this in the local.conf for the CI? | 22:00 |
| edmondsw | GIT_BASE=git://github.com | 22:00 |
| efried | We do | 22:00 |
| efried | Or if not in the local.conf, it's somewhere. | 22:01 |
| esberglu | We preclone all of the openstack repos as part of the image template build using github.com | 22:01 |
| edmondsw | efried it's not in local.conf... and seems like that would be the place to put it | 22:01 |
| esberglu | And I think devstack just uses the remotes set up from that | 22:01 |
| efried | shrug, I just know we have it in the CI somehow. I could find the change set(s) that did it, if necessary. | 22:02 |
| efried | But not today. | 22:02 |
| efried | TTYL. | 22:02 |
| efried | I'm out. | 22:02 |
| esberglu | With the exception of applying patches from openstack gerrit | 22:02 |
| *** dwayne has quit IRC | 22:02 | |
| edmondsw | esberglu didn't follow | 22:03 |
| edmondsw | we preclone everything, even stuff like horizon? | 22:04 |
| esberglu | We build the nightly image template. In that we clone all of the openstack projects (nova, neutron, nova-powervm, everything) so that we don't have to clone it on each ready node | 22:04 |
| esberglu | We then pull in any differences before stacking | 22:04 |
| edmondsw | I'm actually curious as to why horizon would get pulled down with our local.conf, since we shouldn't need horizon in a CI | 22:04 |
| *** thorst has quit IRC | 22:05 | |
| *** thorst has joined #openstack-powervm | 22:05 | |
| openstackgerrit | Matthew Edmonds proposed openstack/nova-powervm master: Use CONF.host for powervm nodename https://review.openstack.org/474799 | 22:06 |
| *** mdrabe has quit IRC | 22:06 | |
| esberglu | edmondsw: Umm I think there might be some horizon tests that we run? I can look into though | 22:06 |
| esberglu | edmondsw: I'm also heading out. Circle back on this topic tomorrow? | 22:07 |
| edmondsw | esberglu yep, have a good night | 22:07 |
| esberglu | You too | 22:07 |
| *** thorst has quit IRC | 22:07 | |
| *** esberglu has quit IRC | 22:07 | |
| *** esberglu has joined #openstack-powervm | 22:08 | |
| *** efried has quit IRC | 22:08 | |
| *** openstack has joined #openstack-powervm | 22:16 | |
| *** edmondsw has quit IRC | 22:23 | |
| *** esberglu has joined #openstack-powervm | 22:32 | |
| *** esberglu has quit IRC | 22:37 | |
| *** jwcroppe has joined #openstack-powervm | 22:43 | |
| *** jwcroppe has quit IRC | 22:43 | |
| *** tjakobs has quit IRC | 22:49 | |
| *** jwcroppe has joined #openstack-powervm | 22:52 | |
| *** dwayne has joined #openstack-powervm | 22:57 | |
| *** jwcroppe has quit IRC | 23:03 | |
| *** jwcroppe has joined #openstack-powervm | 23:04 | |
| *** jwcroppe has quit IRC | 23:09 | |
| *** tjakobs has joined #openstack-powervm | 23:52 | |
| *** tjakobs has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!