Thursday, 2017-01-05

« Wednesday, 2017-01-04 Index Friday, 2017-01-06 »
*** kriskend_ has quit IRC00:02
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix  https://review.openstack.org/41631500:10
*** thorst_ has quit IRC00:10
*** jwcroppe has quit IRC00:27
*** jwcroppe has joined #openstack-powervm00:40
*** jwcroppe has quit IRC00:45
*** thorst_ has joined #openstack-powervm00:54
thorst_adreznec: new theory.  It looks like there is a sync power state right after the spawn.  It looks like we're returning powered off.01:32
thorst_sometimes.01:32
*** jwcroppe has joined #openstack-powervm01:47
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix  https://review.openstack.org/41631502:03
*** jwcroppe has quit IRC02:14
*** svenkat has joined #openstack-powervm02:58
*** tonyb has quit IRC03:05
*** tonyb has joined #openstack-powervm03:05
*** svenkat has quit IRC03:08
*** thorst_ has quit IRC03:14
*** thorst_ has joined #openstack-powervm03:15
*** thorst_ has quit IRC03:19
*** thorst_ has joined #openstack-powervm03:25
*** thorst_ has quit IRC03:25
*** jwcroppe has joined #openstack-powervm03:54
*** thorst_ has joined #openstack-powervm04:01
*** thorst_ has quit IRC04:01
*** jwcroppe has quit IRC04:05
*** tlian has quit IRC05:01
*** kotra03 has joined #openstack-powervm05:26
*** jwcroppe has joined #openstack-powervm05:43
*** jwcroppe has quit IRC05:54
*** thorst_ has joined #openstack-powervm06:02
*** thorst_ has quit IRC06:07
*** thorst_ has joined #openstack-powervm06:28
*** thorst_ has quit IRC06:33
*** jwcroppe has joined #openstack-powervm07:33
*** jwcroppe has quit IRC07:44
*** openstackgerrit has quit IRC07:50
*** jwcroppe has joined #openstack-powervm08:28
*** thorst_ has joined #openstack-powervm08:29
*** thorst_ has quit IRC08:34
*** jwcroppe has quit IRC08:37
*** jwcroppe has joined #openstack-powervm09:22
*** jwcroppe has quit IRC09:33
*** k0da has joined #openstack-powervm09:57
*** openstackstatus has quit IRC10:13
*** openstack has joined #openstack-powervm10:15
*** jwcroppe has joined #openstack-powervm10:17
*** jwcroppe has quit IRC10:27
*** thorst_ has joined #openstack-powervm10:30
*** thorst_ has quit IRC10:35
*** kotra03 has quit IRC11:44
*** smatzek has joined #openstack-powervm11:58
*** jwcroppe has joined #openstack-powervm12:07
*** jwcroppe has quit IRC12:15
*** thorst_ has joined #openstack-powervm12:46
*** openstackgerrit has joined #openstack-powervm13:01
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Switch to synchronous power on  https://review.openstack.org/41696913:01
*** edmondsw has joined #openstack-powervm13:05
*** wangqwsh has joined #openstack-powervm13:28
thorst_wangqwsh: I don't know that we'll have the meeting today...with esberlu out and efried's laptop overheated we'll be light on attendance13:34
thorst_adreznec: you around?13:34
*** svenkat has joined #openstack-powervm13:36
*** mdrabe has joined #openstack-powervm13:36
wangqwshthorst: yes, will we cancel meeting today?13:37
thorst_yeah, but I'm wondering if you've been able to turn off the publishing of the results to the patch sets for nova itself.13:38
thorst_since we're throwing in a lot of red until we get this fix13:38
thorst_(which hopefully I have today...)13:38
*** xia has joined #openstack-powervm13:39
wangqwshi read your mail, i did not turn off it.13:39
wangqwshok, let me try it13:39
thorst_we just want publishing of the results to the patch sets turned off - only for the nova project.  We still want nova-powervm to do everything, and we still want the nova patches to actually run and push logs up to the log server13:40
thorst_that way we can continue debugging, but we're not flooding the nova team with red.13:40
xiaHi everyone, I'm Yan Xia, wangqwsh's team member.13:41
thorst_Hey xia :-)13:41
thorst_pretty light group at the moment - others will be signing in after a few hours13:41
xiaHi thorst_ good to see u13:42
wangqwshsorry, where can i query the status for nova?13:42
thorst_wangqwsh: private messaged you the link since its on the IBM network13:43
thorst_all jobs are red now  :-)13:43
thorst_there is also this: http://ci-watch.tintri.com/project?project=nova13:43
wangqwshthx13:43
thorst_search for powervm in there...you can see how bad we're doing at the moment13:43
thorst_adreznec: I added the interfaces.template to https://ibm.ent.box.com/folder/1588132284213:56
*** jwcroppe has joined #openstack-powervm13:56
*** jpasqualetto has joined #openstack-powervm14:02
mdrabethorst_: About https://review.openstack.org/#/c/416969/1, so async PowerOn updates the VM to "powered on" then sync power states comes through and sets it to off if the job hasn't completed yet?14:03
*** tblakes has joined #openstack-powervm14:03
thorst_mdrabe: yep - and it appears to be destroying everything in CI now14:04
thorst_because we've had so many runs (think several thousand) without the novalinks being rebooted that its now slow enough of a job to happen14:04
mdrabeThat gonna be the same kinda thing for PowerOff?14:04
*** jwcroppe has quit IRC14:08
thorst_hmm...probably14:09
thorst_my job just failed...so lets see why14:09
adreznecthorst_: Crap, for some reason my phone didn't pop up the reminder for the CI meeting this morning and I totally spaced it. Reading on backscroll now, but ugh. Really surprised we hadn't hit this scenario before then14:10
*** efried has joined #openstack-powervm14:10
thorst_adreznec efried mdrabe: so it looks like my change worked...but now its failing on 28 other tests (I think they're new)14:12
thorst_crap just keeps piling in  :-)14:12
adrezneclink?14:12
adreznecnm, got it14:12
*** kriskend has joined #openstack-powervm14:13
adreznecThose keystone ones are definitely not new14:13
thorst_http://184.172.12.213/69/416969/1/check/nova-powervm-pvm-dsvm-tempest-full/bafeecc/powervm_os_ci.html14:13
thorst_well, they're not specific to my change14:13
efriedthorst_, how's the run time?14:14
thorst_and my change at least fixes the big three that I spent all yesterday on14:14
thorst_it was fine...finished without any noticable difference14:14
adreznec1h 05m 10s14:14
efriedexcellent - that's normal.14:14
thorst_and I had several take 1h 20m yesterday14:14
thorst_and some 50m14:14
thorst_so...not bad14:14
efriedcourse with failing tests, can't rely on that number14:14
thorst_can we shove this thing through?14:14
adreznecAnything <90m is fine imo14:14
thorst_ahh, crap...we can't because jenkins -1'd it.14:15
adreznecmhm14:15
thorst_and recheck (for some reason) doesn't pick up the change14:15
efriedYeah, we're gating our own stuff on our CI, right?14:15
thorst_which is awesome.14:15
adreznecrecheck is broken? since when?14:15
thorst_I noticed it yesterday14:15
thorst_recheck powervm - your patch isn't applied.14:15
thorst_I'll add a comment or something silly14:16
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Switch to synchronous power on  https://review.openstack.org/41696914:16
thorst_maybe we can push this through while our CI is running  (is that awful?  I just want to start getting some greens on future nova runs)14:17
efriedSo wait, I'm confused.  The change you're saying fixes the resize timing bug is this 'synchronous power on' one, not the "Check for instance state fix"?14:18
thorst_correct  :-)14:18
efriedAnd where were you seeing recheck problems?  And what were those problems?14:19
thorst_review.o.o is going real slow for me...trying to load it up to send you14:20
thorst_(stupid network)14:20
*** tlian has joined #openstack-powervm14:21
adreznecHmm14:23
adreznecYeah I just tried doing a regular recheck on another patch and I'm not seeing anything pop up in zuul14:23
*** smatzek has quit IRC14:24
adreznecAll right14:26
adreznecSo "recheck" is broken14:26
adreznecBut "recheck powervm" works14:26
adreznecAh14:27
adreznecefried: thorst_ I remember this now, see 465914:27
thorst_recheck powervm was what I was running14:28
adreznecWeird14:28
thorst_https://review.openstack.org/41631514:28
thorst_this is how slow review.o.o is for me right now14:28
thorst_it took all that time to load14:28
adreznecWorked for me on https://review.openstack.org/#/c/408758/214:28
adreznecRunning in Zuul right now14:28
thorst_it runs14:28
thorst_but it doesn't apply your patch14:28
thorst_see that patch set has 'thorst' comments on VERY common things14:29
thorst_and the logs have nothing about thorst in it14:29
adreznecHrm14:29
adreznecThat's disconcerting14:29
thorst_+214:29
*** wangqwsh_ has joined #openstack-powervm14:29
adreznecThe problem I'm having is that I don't see why a recheck would be any different in that case...14:30
adreznecIt just re-executes the same job14:30
thorst_once wangqwsh gets the nova publishes turned off...maybe he can investigate that14:31
efriedthorst_, so you're saying the results we got yesterday on that change set don't necessarily indicate that your fix didn't work, because the CI runs weren't including your patches??14:31
thorst_so if I publish a new version...it is fine14:31
thorst_if I recheck an existing one it doesn't pick up the change (at all)14:31
adreznecthorst_: FYI your new changeset failed14:32
adreznecHit ReadTimeoutError: HTTPSConnectionPool(host='pypi.python.org', port=443): Read timed out14:32
efriedDoesn't pick up the most recent patch set, or doesn't pick up the whole change set?14:32
adreznecMostly likely network issues14:32
*** wangqwsh has quit IRC14:32
*** wangqwsh_ is now known as wangqwsh14:32
thorst_efried: whole change set.14:32
efriedThat's bizarre.14:33
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Switch to synchronous power on  https://review.openstack.org/41696914:33
thorst_yuh14:33
adreznecI'm reading through the logs of the one I just rechecked14:33
adreznecTo see if that's the case there...14:33
thorst_super annoying when you're working at 10 PM on something breaking the entire CI14:33
adreznecWell, I'm seeing it do the git fetch of the change...14:34
thorst_weird...14:34
thorst_check logs...maybe I'm an idiot and it worked14:34
thorst_(or was frustrated and tired)14:34
thorst_10 PM is late for an old dude like me14:35
*** tblakes has quit IRC14:36
thorst_adreznec: In other news (while we wait...) should we put the link to the images in the wiki (https://wiki.openstack.org/wiki/PowerVM)14:39
thorst_I can do that unless you see a problem with it14:40
adreznecNope, seems fine.14:40
*** efried has quit IRC14:41
*** efried_ has joined #openstack-powervm14:41
*** tjakobs has joined #openstack-powervm14:42
adreznecAnd yeah, not sure what's going on here... I'm definitely seeing it say things like14:43
adreznec11:38:39 2017-01-04 16:38:46.706 | +functions-common:git_clone:566           [m git show --oneline14:43
adreznec11:38:39 2017-01-04 16:38:46.708 | +functions-common:git_clone:566           [m head -114:43
adreznec11:38:39 2017-01-04 16:38:46.708 | 8160bc3 Check for instance state fix14:43
adreznecIn the log14:43
thorst_weird...14:44
thorst_I guess next I'll try a recheck14:44
adreznecBut I'm also not seeing any of the logs you added showing up anywhere14:44
thorst_well, you've got one going14:44
adreznecIn n-cpu14:44
thorst_so I'm not blind at least.14:44
wangqwshthorst: jenkins job status would be reflected on gerrit for CI. only 'success' or 'failure'.  i think  there is no the third status...14:47
wangqwshif we do not want the 'red' in nova and run nova patches, one option is set the ci job for nova project is always 'success'.14:47
thorst_wangqwsh: I don't think we want to do that.  esberglu just recently turned that on...not sure how he did that14:49
thorst_adreznec may have more of an idea14:49
thorst_adreznec: FYI - https://wiki.openstack.org/wiki/PowerVM#Installation_Requirements14:49
thorst_bbiab14:49
*** tjakobs has quit IRC14:49
adreznecwangqwsh: There is another option, which is what we had before. Basically we don't vote/publish any results on the nova patches. We still run the jobs, but no comments get posted14:50
adreznecThat way we can still see the runs in our system14:50
thorst_yes please14:50
adreznecBut they don't get posted to gerrit14:50
*** thorst_ is now known as thorst_afk14:50
wangqwshok...let me try it14:51
*** jwcroppe has joined #openstack-powervm14:51
*** efried_ has quit IRC14:52
*** efried has joined #openstack-powervm14:52
*** xia has quit IRC14:53
*** xia has joined #openstack-powervm14:54
*** apearson has joined #openstack-powervm14:55
*** jwcroppe has quit IRC15:03
thorst_afkgeez...pypi must be REALLY slow...it has taken 45 minutes for us to get a stack through on this node.15:12
efriedLab network problem?  Which could also explain timeouts in test runs?15:13
thorst_afkwell, timeouts in test runs would all be within a given VM?15:16
thorst_afkthe timeouts I hit weren't timeouts...those three resize timeouts were because of that state change issue15:16
thorst_afknot sure about keystone or what not15:16
adreznecHuh15:17
adreznecwe should test from another lab15:17
thorst_afkI'll check the network in a bit too15:18
*** wangqwsh has quit IRC15:22
*** jwcroppe has joined #openstack-powervm15:22
*** smatzek has joined #openstack-powervm15:22
thorst_afknetwork doesn't seem too bad in POK...must be the backbone15:30
thorst_afkdo we have a file we can try to FTP down into POK?  So we can prove a bad path?15:30
*** tjakobs has joined #openstack-powervm15:32
adreznecthorst_afk: Could just try and grab something from rchgsa15:35
thorst_afknah, want non-IBM path15:35
thorst_afktrying ubuntu.com15:35
adreznecAh k15:35
*** kriskend_ has joined #openstack-powervm15:36
thorst_afkit's the DNS.15:36
thorst_afkdamnit all15:36
adreznecAgain? Ugh15:37
*** tblakes_ has joined #openstack-powervm15:38
*** kriskend has quit IRC15:39
thorst_afkwell, the backup DNS is taking 80ms (which we made primary)15:39
thorst_afkthe primary (which we abandoned) is taking .8 ms15:39
adreznecWe just can't win15:40
adreznecSucks that the lab infrastructure is so inconsistent15:40
thorst_afkso it only matters for the VMs themselves.15:41
*** tblakes_ is now known as tblakes15:41
thorst_afkadreznec: can we run our own DNS?15:45
thorst_afkthat backs to another...so we have a single server to update when we want all our CI slaves need a change/15:45
adreznecthorst_afk: Yeah, I suppose I could set up a dns cache if I had a VM to do it on15:49
thorst_afkadreznec: get me an IP and I'll get it set up?15:49
thorst_afk9.47.x.x something or other15:50
thorst_afkhow do we even change the IPs in the base image?15:53
thorst_afkI mean the DNS IP...15:53
*** thorst_afk is now known as thorst_15:53
adreznecthorst_: Well, we could recapture with the new IP (probably not ideal) or we could just do it in the capture script for the base image15:55
adreznecThe one nodepool runs on nightly capture15:55
*** xia has quit IRC15:55
thorst_adreznec: we could for an image rebuild tho15:55
*** tlian has quit IRC17:00
*** tlian has joined #openstack-powervm17:20
thorst_adreznec efried: Image built...its just saving to glance now17:32
*** tblakes has quit IRC17:43
*** kriskend__ has joined #openstack-powervm17:54
*** kriskend_ has quit IRC17:54
thorst_adreznec efried: ready nodes are respawning now.18:17
thorst_jobs are running...18:41
adreznecHmm18:47
adreznecOK18:47
thorst_stacking isn't exactly flying but...its not awful19:10
thorst_adreznec efried: 27 fails in my run20:20
adreznecthorst_: Which run20:26
adreznecI don't see anything new on 41698920:26
thorst_it's uploading20:26
thorst_check now20:27
thorst_all keystone failures20:27
adreznecHmm20:27
thorst_so we've got at least that going for us20:27
adreznecYep, same keystone ones20:27
thorst_adreznec: you able to dig in on that?20:32
thorst_I'm fried on CI crap atm20:32
adreznecLooking at a bit of OSA stuff quick, I can try and hop over to it after that20:34
thorst_wonder if we need to update a policy file...20:34
thorst_I bet that's it.20:34
thorst_I bet a new policy was added and we're using newton's devstack?20:34
adreznecWe should be using whatever devstack branch corresponds to the patch branch... aren't we?20:35
adreznece.g. nova master patch = devstack master branch, nova stable/newton = devstack stable/newton20:35
thorst_I thought esberglu pegged us to newton?20:36
thorst_not sure...20:36
adreznecI thought that was just for the undercloud20:36
adreznecHmm20:36
thorst_but it looks like a policy.json issue20:36
thorst_ooo, maybe20:36
thorst_well, its not a new permission...20:40
*** tblakes has joined #openstack-powervm20:41
efriedtest21:23
efriedtest21:23
*** smatzek has quit IRC21:29
thorst_efried: testing 1 221:36
thorst_adreznec: think this could be it?21:38
thorst_https://github.com/openstack-dev/devstack/commit/80b1d0ae7db263dada7fdc4d9d8190d0518b8f6c21:38
*** svenkat has quit IRC22:03
*** thorst_ has quit IRC22:04
*** apearson has quit IRC22:15
*** kriskend_ has joined #openstack-powervm22:23
*** kriskend__ has quit IRC22:23
*** dwayne__ has quit IRC22:45
*** edmondsw has quit IRC22:51
*** tjakobs has quit IRC23:30
*** tblakes has quit IRC23:31
*** svenkat has joined #openstack-powervm23:38
*** mdrabe has quit IRC23:50
*** jwcroppe has quit IRC23:58
*** jwcroppe has joined #openstack-powervm23:59
« Wednesday, 2017-01-04 Index Friday, 2017-01-06 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!