Thursday, 2017-02-16

« Wednesday, 2017-02-15 Index Friday, 2017-02-17 »
*** esberglu has joined #openstack-powervm00:15
*** thorst_ has joined #openstack-powervm00:19
*** thorst_ has quit IRC00:19
*** k0da has joined #openstack-powervm00:32
*** k0da has quit IRC00:42
*** thorst_ has joined #openstack-powervm00:46
*** thorst_ has quit IRC00:46
*** edmondsw has joined #openstack-powervm01:00
*** kriskend has joined #openstack-powervm01:00
*** edmondsw has quit IRC01:04
*** jwcroppe has quit IRC01:16
*** jwcroppe has joined #openstack-powervm01:16
*** jwcroppe has quit IRC01:20
*** kriskend has quit IRC01:23
*** jwcroppe has joined #openstack-powervm01:27
*** thorst_ has joined #openstack-powervm01:29
*** thorst_ has quit IRC01:32
*** esberglu has quit IRC01:34
*** thorst_ has joined #openstack-powervm02:17
*** thorst_ has quit IRC02:18
*** esberglu has joined #openstack-powervm02:38
*** thorst_ has joined #openstack-powervm02:38
*** thorst_ has quit IRC02:40
*** esberglu has quit IRC02:41
*** thorst_ has joined #openstack-powervm02:43
*** thorst_ has quit IRC02:43
*** thorst_ has joined #openstack-powervm02:46
*** thorst_ has quit IRC02:51
*** thorst_ has joined #openstack-powervm03:01
*** thorst_ has quit IRC03:03
*** mdrabe has joined #openstack-powervm03:08
*** mdrabe has quit IRC03:13
*** seroyer has joined #openstack-powervm03:33
*** seroyer has quit IRC04:26
*** esberglu has joined #openstack-powervm04:29
*** esberglu has quit IRC04:34
*** tlian has quit IRC04:34
*** thorst_ has joined #openstack-powervm05:04
*** thorst_ has quit IRC05:08
*** tjakobs has joined #openstack-powervm06:09
*** esberglu has joined #openstack-powervm06:23
*** esberglu has quit IRC06:28
*** edmondsw has joined #openstack-powervm06:29
*** edmondsw has quit IRC06:34
*** thorst_ has joined #openstack-powervm07:05
*** thorst_ has quit IRC07:10
*** tjakobs has quit IRC08:01
*** esberglu has joined #openstack-powervm08:08
*** esberglu has quit IRC08:13
*** thorst_ has joined #openstack-powervm08:25
*** thorst_ has quit IRC08:30
*** k0da has joined #openstack-powervm08:41
*** k0da has quit IRC08:50
*** openstackgerrit has quit IRC09:32
*** esberglu has joined #openstack-powervm09:58
*** esberglu has quit IRC10:03
*** thorst_ has joined #openstack-powervm10:26
*** thorst_ has quit IRC10:31
*** k0da has joined #openstack-powervm11:45
*** esberglu has joined #openstack-powervm11:47
*** esberglu has quit IRC11:52
*** seroyer has joined #openstack-powervm12:33
*** smatzek has joined #openstack-powervm12:41
*** esberglu has joined #openstack-powervm12:42
*** thorst_ has joined #openstack-powervm12:44
*** smatzek has quit IRC12:45
*** smatzek has joined #openstack-powervm12:45
*** edmondsw has joined #openstack-powervm12:46
*** esberglu has quit IRC12:47
*** seroyer has quit IRC13:02
*** seroyer has joined #openstack-powervm13:03
*** xia has joined #openstack-powervm13:04
*** tblakes has joined #openstack-powervm13:11
*** jpasqualetto has joined #openstack-powervm13:14
*** svenkat has joined #openstack-powervm13:15
*** jwcroppe has quit IRC13:15
*** esberglu has joined #openstack-powervm13:26
*** wangqwsh has joined #openstack-powervm13:28
*** jwcroppe has joined #openstack-powervm13:29
esberglu#startmeeting powervm_ci_meeting13:31
openstackMeeting started Thu Feb 16 13:31:36 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.13:31
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:31
openstackThe meeting name has been set to 'powervm_ci_meeting'13:31
thorst_o we have a meeting  :-)13:31
thorst_o/13:32
xiao/13:32
esbergluYeah the recurring meetings ended last week, didn't notice until today13:32
thorst_well, rock on.13:33
thorst_efried is still pretty sick.  I know he's got a doc's appointment13:34
thorst_adreznec: you in?13:34
esbergluWe can keep it quick if no one is around13:34
esberglu#topic In-tree CI13:34
esbergluIn-tree CI is now running on all nova patches silently13:35
thorst_how are run rates?  pass / fail percentages?13:35
esbergluPassing most of the time, but seeing some failures13:35
thorst_and I assume some are the cirros download thing that adreznec is going to fix?13:36
esbergluJust turned it on last night, so I haven't looked at numbers yet13:36
esbergluOr what tests are failing13:36
esbergluThat's the plan for today13:36
esberglu#action esberglu: Check out in-tree CI results13:36
wangqwshesberglu: was the destroy instance case included in ci test?13:37
thorst_esberglu: PM me the URL that the systems will use to download the cirros image13:37
thorst_when you are able...not urgent13:37
*** seroyer has quit IRC13:39
esbergluthorst_: Okay13:39
esbergluwangqwsh: Yeah there are multiple tests with spawn/delete13:39
thorst_sounds like good progress there and we're waiting for numbers throughout the day13:41
esbergluAny other comments/questions for in-tree?13:42
wangqwshdid you fix the port binding issue?13:42
wangqwshif the networking was failed to attach, the instance.host will not be set. it would cause the destroy can not identify the correct host.13:44
wangqwshi am not sure if you met this thing before.13:45
esbergluHmm okay. I will keep an eye on that when I look through the results today13:46
thorst_I don't remember where that was...13:46
thorst_that was the point of making that tempest test without networks.13:46
wangqwshwithout network? ok...13:47
thorst_yeah.  Its a supported use case13:48
wangqwshi got. i met this issue via nova boot command. I will try without network.13:48
thorst_I don't think it works from horizon UI13:49
thorst_but CLI it works13:49
esbergluDo we still want a test without networks? I thought we just wanted that when we couldn't get any other spawn tests working13:49
thorst_I think its dubious how they're working13:49
thorst_but I think we need efried around later to help answer that13:49
thorst_he's back in 2-3 hours13:49
esbergluAlright13:50
esberglu#topic out of tree driver13:51
esbergluNot much has changed with the out of tree13:51
esbergluExcept that now it won't run CI on in-tree patches13:52
thorst_I noticed less failures last night13:52
thorst_which made me happy13:52
thorst_cause that mailbox is at max capacity13:52
esbergluAnd we have a plan for the remote image failure13:53
esbergluOther than that I don't really have anything else for out of tree since efried isn't here13:54
thorst_yeah.  What level does OOT go back to?13:54
thorst_I was working with someone who was still deploying mitaka...do we run CI there/13:54
esbergluNo13:55
thorst_OK...good to know.13:55
esberglunewton through master13:55
thorst_interesting.13:55
esbergluwell we run CI, but we post a message saying mitaka isn't supported13:55
thorst_fair enough13:55
thorst_if they find anything, I may post fixes back there.  We'll see what happens13:55
*** mdrabe has joined #openstack-powervm13:56
esbergluThat's all I had for OOT. Anyone want to talk OSA CI plans? Or save that for another time13:57
thorst_well, I want a quick chat there.13:57
esberglu#topic OSA CI13:57
thorst_I'm thinking that OSA CI, we get started the week after PTG?13:57
thorst_or 'restart with big focus'13:57
thorst_:-)13:57
thorst_cause intree should be running by then.13:58
esbergluYep13:58
thorst_alright...good deal13:58
*** tlian has joined #openstack-powervm13:58
thorst_that was all I really wanted to understand.  When do we get heads down on that again13:58
esbergluYep. Finish in-tree, move to OSA13:59
thorst_good good13:59
esbergluThanks for joining. Any other items before I end it?14:00
thorst_nada14:00
wangqwshnope14:00
esberglu#endmeeting14:01
openstackMeeting ended Thu Feb 16 14:01:09 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_ci_meeting/2017/powervm_ci_meeting.2017-02-16-13.31.html14:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_ci_meeting/2017/powervm_ci_meeting.2017-02-16-13.31.txt14:01
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_ci_meeting/2017/powervm_ci_meeting.2017-02-16-13.31.log.html14:01
*** seroyer has joined #openstack-powervm14:03
*** burgerk has joined #openstack-powervm14:03
*** apearson has joined #openstack-powervm14:09
*** seroyer has quit IRC14:34
*** smatzek has quit IRC14:36
adreznecefried: thorst_ are we good to merge https://review.openstack.org/#/c/433927/ ?14:42
*** esberglu has quit IRC14:43
*** esberglu has joined #openstack-powervm14:44
*** burgerk has quit IRC14:45
thorst_adreznec: yes plz14:46
*** esberglu has quit IRC14:48
*** xia has quit IRC14:50
*** openstackgerrit has joined #openstack-powervm14:52
openstackgerritMerged openstack/nova-powervm master: Refactor power ops  https://review.openstack.org/43392714:52
*** esberglu has joined #openstack-powervm14:58
*** smatzek has joined #openstack-powervm15:00
*** seroyer has joined #openstack-powervm15:05
*** seroyer has quit IRC15:07
*** wangqwsh has quit IRC15:13
*** kriskend has joined #openstack-powervm15:13
*** burgerk has joined #openstack-powervm15:18
*** seroyer has joined #openstack-powervm15:21
*** tjakobs has joined #openstack-powervm15:45
*** jpasqualetto has quit IRC15:53
*** jwcroppe has quit IRC15:57
*** jwcroppe has joined #openstack-powervm15:59
*** jpasqualetto has joined #openstack-powervm16:05
adreznecesberglu: thorst_ FYI I found the patch we talked about yesterday... but I realize now that I wrote it to use nginx, which was fine before we were sharing the logserver and cache server nodes in the staging environment16:18
adreznecSince that already has a running apache setup...16:18
thorst_so can you add it to the apache server?16:19
thorst_:-)16:19
adreznecSo I'm going to have to redo that to fit in with the existing apache install...16:19
adreznecWell16:19
thorst_yay!16:19
adreznecI'm going to have to do more testing16:19
adreznecBecause now I can't really do much with the site configs16:19
adreznecWithout stepping on the logserver site config16:19
adreznecI can't just add it to the apache server though16:20
adreznecBecause we don't have one in the prod environment16:20
adreznecSo I still have to write stuff to set that up16:20
thorst_could just make a different server.16:20
* adreznec shrugs16:20
thorst_well, let me know if you want that16:23
*** dwayne has joined #openstack-powervm16:27
*** viclarson has joined #openstack-powervm16:32
viclarsonhi16:32
viclarsonguys16:32
viclarsonwe have a problem16:32
viclarsonif nvram_store set to none migration/resize/evacuation works fine but we lose odm16:33
viclarsonit's clear16:33
viclarsonif nvram_store set to swift it works too16:34
viclarsonbut if we have attached volumes16:34
*** jwcroppe has quit IRC16:35
thorst_viclarson: Yeah, that's kind of the OpenStack way.16:35
thorst_which is why we added that Swift store.16:35
thorst_because that saves the NVRAM for those operations16:35
thorst_a small swift store is all that is needed to keep that info around, the NVRAM data is pretty small16:35
viclarsonInstance rollback preformed due to: Unable to rebuild virtual machine on new host: Error is The device with UDID ... was not found on any of the  VIrtual I/O servers16:35
viclarsonmb we need to add some sleeps to code?16:36
viclarsonwhen nvram switched off migration with attached volumes works fine16:36
thorst_Cinder volumes?16:37
viclarsonwhat can be wrong?16:37
viclarsony16:37
thorst_we only keep the NVRAM for cinder volumes at the moment16:37
viclarsonvscsi16:37
thorst_not for nova ephemeral...16:37
viclarsoncinder volumes16:37
thorst_so I'm wondering if that is what you're hitting.16:37
thorst_boot from cinder?16:37
viclarsonboot from ssp16:38
viclarsonfrom epemera;16:38
viclarsonfrom epemeral16:38
thorst_what's your cinder device backend?16:38
viclarsonstorwize v700016:38
thorst_hmm...can you paste the full error to pastebin?16:38
viclarsonlater it's isolated environment not ez to copypaste16:40
viclarsonseveral minutes16:40
thorst_so basically what that error message is saying is...16:40
thorst_<sec, I'm trying to validate my theory>16:42
viclarsonhttp://pastebin.com/FQ5zZm0f16:43
thorst_OK - I think I can see how this happens.16:45
thorst_what version of the code are you on?16:46
thorst_Newton?  Ocata/master?16:46
viclarsonmitaka16:46
thorst_o, hmmm...16:46
viclarsony we like retro16:46
thorst_can you get a bug opened for us?16:47
thorst_https://bugs.launchpad.net/nova-powervm/16:47
thorst_efried: have you been following at all on this one by chance?16:47
viclarsony16:47
efriedthorst_ No.  Time stamp to roll back to?16:48
thorst_I think we need to discuss what's happening here.16:48
thorst_last 18 minutes16:48
viclarsonis it fixed in new releases?16:48
thorst_viclarson: don't think so.16:50
thorst_so here's what I think is happening16:50
thorst_I don't think we've run the discover_hdisk prior to reading the rebuild map16:51
thorst_viclarson: have you changed the nova_powervm driver at all?  The error you sent on pastebin isn't lining up with the stable/mitaka code16:53
thorst_https://github.com/openstack/nova-powervm/blob/stable/mitaka/nova_powervm/virt/powervm/driver.py#L135716:53
thorst_efried: I think the issue here is that for the vSCSI volumes we need to run a hdisk.discover first:  https://github.com/openstack/nova-powervm/blob/stable/mitaka/nova_powervm/virt/powervm/driver.py#L129416:54
thorst_thoguhts?  Am I way off base?16:54
openstackgerritTaylor Jakobson proposed openstack/nova-powervm master: Add imagecache information to devref  https://review.openstack.org/43435816:54
viclarsonyes i have changes16:55
viclarsonhttps://bugs.launchpad.net/nova-powervm/+bug/166540716:55
openstackLaunchpad bug 1665407 in nova-powervm "unable to migrate vm with attached volumes" [Undecided,New]16:55
*** mdrabe has quit IRC16:55
thorst_viclarson: should we add any of those changes into our driver so you don't have to carry patches??16:55
viclarson2 changes16:56
viclarson1) retry for vm deleting, it works unstable16:56
viclarson2) hypervisor_name == cfg.host instead of hypervisor reported to workaround horizon bug16:57
viclarsoni think you dont need this changes16:57
*** jwcroppe has joined #openstack-powervm16:58
efriedthorst_ I think you want to consult with mdrabe on this issue.16:59
*** nbante has joined #openstack-powervm16:59
thorst_hmm...that first one worries me17:00
thorst_we're definitely eager to see changes so you don't have to carry patches17:00
*** mdrabe has joined #openstack-powervm17:01
thorst_viclarson: let me chat with mdrabe this afternoon/tomorrow on the swift build_map and see what we can't do17:01
thorst_I unfortunately need to run for a bit.17:01
viclarsonwe also have problem in resize to same host (mb nova think vm stay active and never finish migration) but we just switched out resize to same host17:04
thorst_mdrabe: if you get a chance, please read backlog here (eavesdrop should have it)17:05
*** thorst_ is now known as thorst_afk17:05
mdrabeviclarson: You're hitting this error on evacuation? Or resize to another host?17:09
viclarsonhost.py data["hypervisor_hostname"] = ms_wrapper.mtms.mtms_str  changed to cfg.CONF.host17:09
viclarsonon resize17:09
viclarsonto another host17:09
mdrabeI don't _think_ the slot store should come into play there...17:10
mdrabeI saw the pastebin, do you by chance have the full stack trace in the compute logs resulting in this error:17:11
mdrabeError is The device with UDID 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODExMDRGMjAwMDAwMDAwMDAwMDY5NA== was not found on any of the Virtual I/O Servers.17:11
thorst_afkmdrabe: that's an hdisk UDID17:11
thorst_afkbut I'm afk17:11
viclarsondrive.py return [self.host_wrapper.mtms.mtms_str] changed to cfg.CONF.host17:11
mdrabeYea I think it's discover_hdisk that's failing17:11
viclarsoni will try17:12
mdrabedestination VIOSes maybe aren't zoned to the same backend17:12
viclarsoni think it worked when nvram_store was none17:13
viclarsonmb sleeps?17:14
mdrabeTough to say, need to figure out where that original error is coming from17:18
*** jwcroppe has quit IRC17:18
*** jwcroppe has joined #openstack-powervm17:18
esbergluthorst_: I'm looking through the in-tree CI failures. It seems like tests that should be failing every time are sometimes passing17:20
esbergluhttp://184.172.12.213/20/415520/12/silent/nova-in-tree-pvm/5bb5d5c/powervm_os_ci.html17:20
*** jwcroppe_ has joined #openstack-powervm17:20
viclarsondriver.py @retrying.retry( .. retry_on_instance_termination_failure ... ) on def _destroy17:20
esbergluFor example the ServerMetadataTestJSON tests17:20
esbergluThey are failing because the spawn a vm, create a port, then fail to bind the port to the vm17:21
esbergluBut when those tests pass, it doesn't look like any of that port stuff happens17:21
*** jwcroppe has quit IRC17:23
viclarsonplaced more logs17:33
viclarsonhttps://bugs.launchpad.net/nova-powervm/+bug/166540717:33
openstackLaunchpad bug 1665407 in nova-powervm "unable to migrate vm with attached volumes" [Undecided,New]17:33
viclarsonwow17:33
mdrabeviclarson: The volumes are migrated as well correct?17:33
viclarsonnot in pvmctl scsi list at any host17:35
mdrabeBut can you see duplicate volumes for the VMs being resized on the storage backend?17:36
mdrabeThe cinder logs would say if those volumes were cloned17:36
*** jpasqualetto has quit IRC17:42
viclarson19:26:38 unmapped from nlink-40w 19:26:41 mapped to nlink-38w17:43
efriedesberglu How often does test_rebuild_server_in_error_state fail?17:55
efriedthorst_afk adreznec https://review.openstack.org/#/c/433935/ passed CI this time around.  That test seems to fail the same way very frequently.  And I can't see anything wrong with it in the logs.17:57
thorst_afkyeah, I think its just timing out17:57
thorst_afkI'm of the mindset we should disable...17:57
thorst_afkbut still afk17:57
*** jpasqualetto has joined #openstack-powervm17:58
viclarsonis it the same? https://bugs.launchpad.net/nova-powervm/+bug/163766118:03
openstackLaunchpad bug 1637661 in nova-powervm "vscsi driver pre_live_migration_on_destination may not be handling hdisk discovery correctly" [Undecided,New]18:03
*** nbante has quit IRC18:38
*** nbante has joined #openstack-powervm18:41
efriedmdrabe ^^ ?18:44
viclarsonsleep() don't help18:44
efriedviclarson If thorst_afk's assessment is correct, it wouldn't.18:45
viclarsoni see discovering in logs18:45
efriedDoes it succeed?18:45
viclarson2017-02-16 19:26:44.888 29974 DEBUG nova_powervm.virt.powervm.volume.vscsi [req-77f4a5dc-01b4-4636-87fd-89954be65786 06134a41cd1b4ba583880e6a32ef5adc 55c738a50140429493b2ab75a02ea6e7 - - -] No ITLs for VIOS vio1-38w for volume 35d30504-6a77-4fb2-acab-286a9cd067ad. _discover_volume_on_vios /usr/lib/python2.7/dist-packages/nova_powervm/virt/powervm/volume/vscsi.py:21418:46
viclarson2017-02-16 19:26:44.889 29974 DEBUG nova_powervm.virt.powervm.volume.vscsi [req-77f4a5dc-01b4-4636-87fd-89954be65786 06134a41cd1b4ba583880e6a32ef5adc 55c738a50140429493b2ab75a02ea6e7 - - -] No ITLs for VIOS vio2-38w for volume 35d30504-6a77-4fb2-acab-286a9cd067ad. _discover_volume_on_vios /usr/lib/python2.7/dist-packages/nova_powervm/virt/powervm/volume/vscsi.py:21418:46
viclarsonno18:46
efriedviclarson Just keep the bug report updated with your findings.  It may be a day or two before thorst_afk/mdrabe can investigate.18:47
efriedmdrabe - Got a sec to discuss RemoveOptical and its inclusion in the in-tree driver?18:47
efriedI think...  I think we have to wait until we cut pypowervm 1.0.0.518:48
efriedand REST 1.0.0.5 for that matter.18:50
efriedesberglu adreznec Weigh in here?  We can pass CI cause we're pulling develop, but we can't merge that sucker without 1.0.0.5 in the requirements.18:51
*** smatzek has quit IRC18:52
esbergluefried: Yes I believe you are right19:04
adreznecYeah19:12
adreznecWe'll definitely need to wait until we cut the required version19:13
*** nbante has quit IRC19:15
esbergluefried: Did you see my post above about the in-tree CI failures?19:18
efriedesberglu Yeah, didn't look into it.19:18
efriedThis is kinda what I was thinking was going to bite us in general about spawns in-tree with early change sets.19:18
efriedI.e. we don't have networks, so how can we ever pass?19:19
esbergluYeah19:19
efriedAnd the answer seems to be: we only pass when the tests don't actually do any networking.19:19
efriedWhich happens... when?  Why?19:19
esbergluI stuck trying to figure out why the same test sometimes does networking and other times skips it19:28
*** smatzek has joined #openstack-powervm19:33
mdrabeefried: vopt removal in the in-tree driver you say?19:35
efriedmdrabe Yeah.19:35
mdrabeIs it a problem?19:35
efriedCan't put it in until in-tree requirement includes the RemoveOptical stuff, which won't exist until 1.0.0.519:35
efriedCurrently in-tree req is 1.0.0.419:36
efriedIt'll work in the CI cause we're loading up develop (and pvm-novalink 1.0.0.5)19:36
efriedBut if we merge, we break.19:36
efriedesberglu adreznec - When do we cut 1.0.0.5?19:36
mdrabeThat's okay by me19:36
adreznecefried: Next month, not sure on exact date...19:39
efriedk19:39
*** viclarson has quit IRC19:45
*** _-___-_ has joined #openstack-powervm20:02
efriedthorst_afk adreznec Can I get +2 on https://review.openstack.org/433935 ?20:21
*** _-___-_ has quit IRC20:27
*** jwcroppe_ has quit IRC20:28
*** jwcroppe has joined #openstack-powervm20:28
*** apearson has quit IRC20:31
*** apearson has joined #openstack-powervm20:34
thorst_afkefried: +220:46
adreznecthorst_afk: esberglu I kind of want to just give up on trying to share the logserver and cache server in the staging environment.20:58
adreznecEven after converting over to using apache the fact that the logserver uses apache with a custom site file server out of a non-standard root basically means I have to write code for staging that's completely different than the prod code20:58
esbergluI like having the environments consistent, so that is fine by me21:00
adreznecOk21:00
adreznecWe need another server for the staging environment then21:00
*** apearson has quit IRC21:01
*** apearson has joined #openstack-powervm21:03
adreznecthorst_afk: ^ how hard will it be to get something similar to the prod environment for that?21:06
openstackgerritEric Fried proposed openstack/nova-powervm master: In-tree backports: Task UT, Delete args  https://review.openstack.org/43511421:06
esbergluadreznec: Possible we could share?21:07
adreznecWasn't that our original plan21:07
adreznecBut we ran into issues with the playbooks for that?21:07
thorst_afkyeah, I thought they were sharing.  I can clone the VM if you want, just get a new hostname/ip if that's the case21:08
thorst_afkbut I'm afk21:08
efriedthorst_afk lacks true afk discipline.21:08
adreznecafk does not mean away from phone21:08
adreznec:P21:09
thorst_afk+2 (but I'm afk)21:09
esbergluadreznec: I think we would just have to hardcode the list of servers in the bind named.conf.options21:09
esbergluAnd we didn't want to do that21:09
adreznecRight21:09
openstackgerritMerged openstack/nova-powervm master: Synchronize power ops  https://review.openstack.org/43393521:09
*** jpasqualetto has quit IRC21:10
esbergluSo I guess it depends on whether or not we are okay with the hardcoded list21:11
esbergluCould redeploying the staging env potentially cause issues with the prod. env if they share?21:12
*** tblakes has quit IRC21:27
adreznecesberglu: Possibly... if anything went wrong it'd probably bust all dns searches and/or apt requests in the prod environment21:36
adreznecWhich would mean every run would fail21:36
adreznecSo yeah21:36
esbergluYeah so separate seems like a better idea21:37
efriedthorst_afk adreznec https://review.openstack.org/#/c/391288/ is rebased, updated with power refactors, and a couple other tweaks.  Ready for final pre-PTG review.21:42
efriedesberglu We're officially hard on the hook for ^^ to pass in-tree CI before next week.21:42
esbergluSaw that in the nova meeting21:46
efriedHah, OOT "succeeded".  That's good ;-)21:47
esbergluThat means we can merge now right!? lol21:49
esbergluLess failure emails now at least21:50
*** edmondsw has quit IRC21:52
*** edmondsw has joined #openstack-powervm21:53
*** edmondsw has quit IRC21:57
*** smatzek has quit IRC22:02
*** svenkat has quit IRC22:04
efriedthorst_afk adreznec esberglu https://review.openstack.org/#/c/427380/ (power on/off, reboot) rebased & ready.22:05
*** apearson has quit IRC22:11
*** smatzek has joined #openstack-powervm22:12
*** smatzek has quit IRC22:13
*** smatzek_ has joined #openstack-powervm22:13
esbergluefried: I think I know what is going on with the In-tree CI failures.22:19
esbergluOn the runs that succeed, it creates a vm. It then detects that no networks exist, so it doesn't to do any networking22:19
esbergluOn the runs that fail, it creates a vm. It then detects networks, except this time a network already exists. So it automatically tries to use that one22:19
esbergluWhich of course doesn't work because it's not supported yet22:19
esbergluSo we just have to figure out how to avoid that automatic network allocation22:19
*** apearson has joined #openstack-powervm22:19
efriedesberglu What's "it" in this case?22:21
esbergluSpawning22:22
efriedesberglu When you say, "this time a network already exists" -- where did it come from?  Why is it there?22:23
efriedthorst_afk adreznec esberglu https://review.openstack.org/#/c/409401/ (localdisk) rebased & ready.22:24
*** viclarson has joined #openstack-powervm22:32
viclarsonseems like it fails in slot_mgt.init_recreate_map() on vif.plug(). What happens if i skip this step?22:36
viclarsonseems like it fails in slot_mgt.init_recreate_map() on vif.plug(). What ll happen if i skip this step?22:37
efriedviclarson You won't be able to remote restart the VM ;-)22:38
*** smatzek_ has quit IRC22:38
thorst_afkefried: not rr, but rebuild22:38
thorst_afkrr is a 'powervm' term that means rebuild22:38
thorst_afkrebuild in openstack is also a reszie or a cold migration22:39
*** thorst_afk has quit IRC22:39
mdrabeviclarson The reason for the failure is because the destination host cannot discover volume 35d30504-6a77-4fb2-acab-286a9cd067ad22:40
viclarsonit can be discovered if switch out slot_mgr22:41
*** burgerk has quit IRC22:41
viclarsonchanging order of ConnectDisk and PlugVif can help?22:42
viclarsonok will wait22:43
esbergluefried: Looks like it's just failed network cleanup from other tests. So these tests aren't actually the problem22:46
esbergluIt's just a side effect from the others failing22:46
esbergluneutron.tests.tempest.api.admin.test_extension_driver_port_security_admin.PortSecurityAdminTests22:47
esbergluneutron.tests.tempest.api.admin.test_shared_network_extension.AllowedAddressPairSharedNetworkTest22:47
esbergluAre we cool with just disabling those neutron tests?22:47
*** efried has quit IRC22:51
*** efried has joined #openstack-powervm22:54
efriedesberglu Boo.  Does the same thing happen if we're running tests in parallel and a different test has created (but not yet cleaned up) a network?22:56
esbergluI haven't seen it yet. But could be possible22:57
*** k0da has quit IRC23:00
efriedesberglu So based on your observations, if we pass, we pass everything; but if we fail some, those failures trigger more failures.23:00
esbergluEvery failure of this nature has had 1 or both of those 2 neutron tests fail to clean up the network (which is essentially every other failure)23:01
*** dwayne has quit IRC23:02
esbergluSo if those are off I think we will see all passes pretty much23:02
esbergluAnd I don't think those neutron tests have much relevance to our in tree driver23:02
*** thorst_afk has joined #openstack-powervm23:03
*** mdrabe has quit IRC23:04
*** esberglu has left #openstack-powervm23:07
*** thorst_afk has quit IRC23:07
*** esberglu has joined #openstack-powervm23:07
efriedesberglu How often do those neutron tests fail?23:09
*** esberglu has quit IRC23:12
*** esberglu has joined #openstack-powervm23:21
*** tjakobs has quit IRC23:21
efriedesberglu How often do those neutron tests fail?23:23
*** esberglu has quit IRC23:24
*** esberglu has joined #openstack-powervm23:24
esbergluefried: those tests fail maybe 15% of the time23:26
esbergluRough estimate23:26
*** svenkat has joined #openstack-powervm23:28
*** viclarson has quit IRC23:28
*** apearson has quit IRC23:29
*** seroyer has quit IRC23:31
*** kriskend has quit IRC23:33
*** esberglu has quit IRC23:46
*** edmondsw has joined #openstack-powervm23:49
*** kriskend has joined #openstack-powervm23:52
*** edmondsw has quit IRC23:53
*** kylek3h has joined #openstack-powervm23:57
*** kylek3h_ has quit IRC23:58
« Wednesday, 2017-02-15 Index Friday, 2017-02-17 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!