Thursday, 2017-02-16

« Wednesday, 2017-02-15 Index Friday, 2017-02-17 »
*** dct360 has quit IRC02:46
morganmordred: lol02:52
morgani think i'm gonna sit back and drink sparkling water before I make the last few "unpack things cause I moved" issues go away02:53
morganjust in time to unpack more things when Brie gets her stuff here :P02:53
*** gouthamr has quit IRC03:44
*** yfried has joined #openstack-shade06:08
*** yfried has quit IRC06:13
*** abregman has joined #openstack-shade06:55
*** yfried has joined #openstack-shade07:43
*** iogg has joined #openstack-shade07:58
*** yfried has quit IRC08:25
*** yfried has joined #openstack-shade08:26
*** abregman has quit IRC09:10
*** abregman has joined #openstack-shade09:16
*** openstackgerrit has quit IRC09:32
*** yfried has quit IRC10:25
*** cdent has joined #openstack-shade10:42
*** iogg has quit IRC11:29
*** cdent has quit IRC11:48
mordred\o/12:05
*** cdent has joined #openstack-shade12:16
*** iogg has joined #openstack-shade12:59
*** gouthamr has joined #openstack-shade13:15
*** iogg has quit IRC13:15
*** abregman is now known as abregman|mtg13:34
*** abregman|mtg has quit IRC13:34
*** gouthamr has quit IRC13:42
*** gouthamr has joined #openstack-shade13:43
*** openstackgerrit has joined #openstack-shade13:44
openstackgerritMonty Taylor proposed openstack-infra/shade master: Change request_id logging to match nova format  https://review.openstack.org/43491613:44
*** iogg has joined #openstack-shade13:47
openstackgerritMonty Taylor proposed openstack-infra/shade master: Change request_id logging to match nova format  https://review.openstack.org/43491613:48
openstackgerritMonty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data  https://review.openstack.org/43448113:48
*** jordanP has joined #openstack-shade13:48
*** abregman has joined #openstack-shade14:09
*** dct360 has joined #openstack-shade14:20
openstackgerritMerged openstack-infra/shade master: Fix several concurrent shade gate issues  https://review.openstack.org/43442815:14
openstackgerritJordan Pittier proposed openstack-infra/shade master: attach_volume should always return a vol attachment.  https://review.openstack.org/43496415:16
jordanPbtw mordred I wanted to say that both removing dependencies on python-*client and the usage of requests_mock is awesome. It's going to pay big times soon15:17
mordredjordanP: ++ I can't wait for both to be done15:18
mordredjordanP: it turns out the client libs make things way harder than they need to be :)15:18
jordanPyeah, I can imagine that's not the most sexy refactoring ever, but much needed15:18
mordredalso - patch looks great - looking forward to seeing how many of the functional tests fail because of the volume detach thing15:18
jordanPI ran them locally already15:19
mordredI mean the terrible race bug thing15:19
* mordred crosses fingers it just magically goes away15:19
jordanPah right.... I was trying to reproduce it but I had to fight with attach_volume and detach_volume first :)15:19
openstackgerritJordan Pittier proposed openstack-infra/shade master: attach_volume should always return a vol attachment.  https://review.openstack.org/43496416:10
*** abregman has quit IRC16:13
*** jordanP has quit IRC16:57
*** cdent has quit IRC17:54
*** iogg has quit IRC17:55
*** cdent has joined #openstack-shade18:32
*** cdent has quit IRC18:50
*** cdent has joined #openstack-shade18:50
openstackgerritMonty Taylor proposed openstack-infra/shade master: Change request_id logging to match nova format  https://review.openstack.org/43491618:54
openstackgerritMonty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data  https://review.openstack.org/43448118:54
morganmordred: how are we looking at fixing gate awkwardness stuff?19:12
morganmordred: i ask because i want to poke at things but happy to look if you need eyes at something19:12
clarkbmorgan: I ended up reviewing a change to fix the cirros image finding and nova microversions19:15
clarkbmorgan: if it hasn't merged yet that could probably use review19:15
mordredmorgan: gate awkwardness should be fixed/merged19:23
mordredmorgan: but the race-condition bug of "sometimes volumes don't detach all the way and the test fails when we try to delete the volume" in the functional test is still sometimes happening19:24
mordredmorgan: I added a wait loop - but that didn't fix it19:24
morganmordred: huh19:26
morganmordred: weeeeiiiiirrrrdddd19:26
mordredyah19:27
mordredmorgan: oh!19:28
mordredmorgan: I did not actually land that patch19:28
mordredmorgan: https://review.openstack.org/#/c/432770/19:28
mordredclarkb: ^^ if you have a sec19:28
morganLOL19:28
morganLGTM.19:29
morganpretty straight forward and has cinder folks support19:29
mordredyah19:30
mordredif we see the problem after this lands, then further investigation may be warranted19:30
morganmordred: ftr: i can only +1 in shade (so can't push that patch through for you)19:49
mordredmorgan: :) that's ok - I can19:50
morganmordred: hhee i know, just letting you know not to wait for me to +2/+A it19:50
morgan:P19:50
*** gouthamr has quit IRC20:24
openstackgerritMerged openstack-infra/shade master: Wait for volumes to detach before deleting them  https://review.openstack.org/43277020:31
* morgan rechecks some patches20:33
mordred\o/20:34
mordredmorgan: one of your patches I think has a legitimate fail20:34
mordredmorgan: like, py35 failed20:34
morganmordred: hm.20:39
mordredmorgan: btw - requests_mock has helped me find places where unicode handling in python 3 makes me want to die20:40
morganmordred: oomkiller20:40
morgan2017-02-16 17:42:57.720329 | [  643.686268] Out of memory: Kill process 1446 (python) score 967 or sacrifice child20:40
morgan2017-02-16 17:42:57.720393 | [  643.689871] Killed process 1446 (python) total-vm:7796916kB, anon-rss:7699784kB, file-rss:1636kB20:40
mordredwoot!20:40
morganlets see if it is happier with the recheck. might be transient oom issue20:40
morgansince the subsequent patch didn't fail20:40
*** gouthamr has joined #openstack-shade20:41
morganoh it did20:41
morganwell. crap on a stick20:41
morganhmmmmmmmmm20:41
morganam i reading that right... 8GB of data?20:41
morganin total-vm?20:42
morgan*blink*20:42
morganoh. uh20:42
morganhmmm..20:42
morganmordred: i bet we're leaking memory in requests_mock20:43
morganthere is no way ptyhon should be consuming ~7GB of total-vm there20:43
mordredmorgan: WOW20:43
morganLOL and it doesn't tell me which test caused the failure.20:45
morgan*facepalm*20:45
morganhttps://www.irccloud.com/pastebin/m0d95Ot7/20:45
morganI think i want to implement the pretty_tox wrapper for shade20:46
morganso we can see tests as they are ebing run20:46
mordredmorgan: mtreinish pointed me at stestr recently - have you seen it?20:50
mordredmorgan: (it's a side project he's been hacking on)20:50
morganostestr? or stestr?20:51
mordredhttps://github.com/mtreinish/stestr20:51
morganoh interesting20:51
openstackgerritMorgan Fainberg proposed openstack-infra/shade master: DNM- Pretty Tox addition to debug memory issues in py35  https://review.openstack.org/43510720:52
morganmordred: ftr, zuul passed py35 on the recheck, but the shade coverage failed and py27 failed on the other one20:54
morgansomething is definitely leaking memory20:54
morganand bad20:54
mordredmorgan: _excellent_20:54
morganlets see what pretty tox comes up with so i know what test it is20:54
morgani guess i could run a trimmed down VM limit locally too20:55
morganoooh think i found it20:56
mordredmorgan: yeah?20:56
* mordred excited20:57
morganyeah waiting for a timeout20:57
morganbut a test is spinning20:57
morganmordred: test_list_images_paginated21:00
morganmordred: spins on things like this for ever21:00
morganhttps://www.irccloud.com/pastebin/i1swPbO9/21:00
morganbut not always21:01
mordredmorgan: ooh - so like something in the registraiton isn't registering a new response for the pagination calls maybe? so it's just always matching and sending back the call which says that there is more pagination to be fetched?21:01
morganmaybe21:02
mordredyah - check it ...21:02
mordredthe payload returned for that21:02
mordredlists a next url21:02
mordredwhich is the same as the url that was fetched in that call21:02
morganthis was likely to fail regardless.21:03
mordredGET /v2/images?marker=91315d6c-a11a-4978-8309-31521bcf4ec6 ... next: /v2/images?marker=91315d6c-a11a-4978-8309-31521bcf4ec621:03
morganafaict21:03
morganit *sometimes* passes21:03
morganwhich is weird...21:03
morganbecause requests_mock should be super consistent21:03
morgansometimes it timesout21:04
mordredyah ... that's worrying21:05
mordredalso - requests_mock is one of those things that oh-man if there's a bug finding it will be _funnnnnnnn_21:05
morganyeah.21:05
morganwell at least i know how the matchers work now =/21:05
morgan:P21:05
morganare we missing a mock that says we hit the end?21:07
morganit's like "next" is still showing up over and over and over21:09
morgan*blink*21:09
morganwtf...21:09
morganoooooh21:09
morganmordred: i think that is missing complete_qs=True21:09
mordredmorgan: I agree with your assessment21:11
mordredmorgan: I am confused how this ever worked21:11
morganright?>21:11
morgani'm looking at the diff21:11
morgancan't see how it ever worked21:11
morganyeah w/o complete_qs=True it spins21:11
morganand timesout21:11
morganwtf.21:12
morganooh21:12
morganregister_uri was much much simpler21:12
morganregister_uris leverages a lot more of the power of request_mock21:12
morganand does bundled response_lists21:12
morganetc21:12
mordredah - yah21:12
mordredthat makes sense21:12
morganpreviously you just overwrite it21:12
morgani actually encode the QS as part of the key21:12
morganso we don't overwrite it21:13
morganit gets far far pickier about matching21:13
morganok easy to fix, but anywhere we use QS and we care, we need complete_qs=True now21:13
mordredalso - let's be honest - that's a LOT of conversions without any bugs in request_uris showing up21:13
morganvs a bit slopier21:13
mordred++21:13
mordredalso - still doesn't explain how this test could sometimes pass21:13
morgani think the matchers are a bit sloppy21:13
mordredbut maybe the answer to that is *shrug*21:14
morganit's likely a bug in requests_mock21:14
* morgan fixes21:14
openstackgerritMorgan Fainberg proposed openstack-infra/shade master: Convert use of .register_uri to .register_uris  https://review.openstack.org/43451621:15
morgan^21:15
morganthat should fix it21:15
openstackgerritMorgan Fainberg proposed openstack-infra/shade master: Convert test_object to use .register_uris  https://review.openstack.org/43453221:16
morganmordred: i think register_uris should look for '?' in the URI and error if complete_qs is not explicitly passed21:16
*** cdent has quit IRC21:16
mordredmorgan: ++21:16
mordredmorgan: btw - I updated tests to include rest call logging for novaclient (they pass in the novaclient logger to the Session so you have to configure logging for novaclient to get the ksa Session logging)21:16
morganso someone has to either say complete_qs=False (and damn near know wtf they are doing)21:16
morganor complete_qs=True21:16
mordredyup21:16
morgani'll get a patch up for register_uris that does that21:17
*** iogg has joined #openstack-shade21:23
morganmordred: tbh, i'm fairly pleased that we hit exactly 1 error in register_uris.21:26
openstackgerritMorgan Fainberg proposed openstack-infra/shade master: Require `complete_qs` if a query_string is in the mocked uri  https://review.openstack.org/43512721:36
morganmordred: ^21:36
morganmordred: though i think there is an edge case in register_uris, if you register a URI with complete_qs=False after you register a uri (same one) with complete_qs=True... i'm thinking it might result in weird behavior21:39
morganmordred: but we can cross that bridge if it happens, i don't expect it to tbh21:39
morganwe could also just default to complete_qs=True unless specifically set to false21:40
morgan[might be the most correct behavior for our use]21:40
mordredmorgan: yah - honestly, that's not a terrible idea21:43
mordredmorgan: I find it hard to imagine us needing to set it to False ever21:43
mordredmorgan: heya - sdague has a question in #openstack-dev I think you might be a good human to answer21:57
mordredmorgan: 21:56:00           sdague | stevemar / mordred / lbragstad is there some existing classes to easily fake out keystoneauth1 for testing it in other projects?21:57
mordredmorgan: nm - I got jamielennox in on it21:59
morganmordred: hehe, stopped lurking in -dev because it never really had any conversation in it22:01
mordredmorgan: amazingly enough it's been chatty like the last week or so22:01
*** gouthamr has quit IRC22:15
*** iogg has quit IRC22:42
*** jamielennox is now known as jamielennox|away23:02
*** cdent has joined #openstack-shade23:04
*** jamielennox|away is now known as jamielennox23:14
openstackgerritMerged openstack-infra/shade master: Actually normalize nova usage data  https://review.openstack.org/43448123:20
*** cdent has quit IRC23:47
openstackgerritMerged openstack-infra/shade master: Change request_id logging to match nova format  https://review.openstack.org/43491623:59
« Wednesday, 2017-02-15 Index Friday, 2017-02-17 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!