Thursday, 2014-08-07

« Wednesday, 2014-08-06 Index Friday, 2014-08-08 »
*** crc32 has quit IRC00:01
woodsterrm_work: we haven't begun to tune uwsgi for concurrency as of yet, but that should just affect availability nor break request processing. Pecan might also need to be configured for concurrency as well?00:03
rm_workI think it might be middleware related00:03
rm_worklike, the middleware mucks up multiple requests from the same client that are happening concurrently or something?00:04
rm_workthat seems to be what the Magneto team's fix would resolve00:04
rm_workwoodster: I am trying to determine where a similar fix might fit in the Barbican middleware00:04
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add support to Barbican for consumer registration  https://review.openstack.org/10784500:21
*** gyee has quit IRC00:22
rm_workk00:24
rm_worklet's see what happens00:24
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add support to Barbican for consumer registration  https://review.openstack.org/10784500:28
rm_workyep00:35
rm_workchellygel: yep, pretty sure this is it. let me know when you want to collect on your lunch :)00:37
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add support to Barbican for consumer registration  https://review.openstack.org/10784500:39
rm_workoh, shit… except what I need to do is make a new CR probably and put these changes in it, then rebase on top of that <_<00:39
*** bdpayne has quit IRC00:44
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add "Connection" HTTP header to response  https://review.openstack.org/11244400:47
openstackgerritA change was merged to openstack/barbican: Replace skipTest in favor of decorator  https://review.openstack.org/11237201:00
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add "Connection" HTTP header to response  https://review.openstack.org/11244401:02
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add support to Barbican for consumer registration  https://review.openstack.org/10784501:03
rm_workok, rebased the connection header fix, and added it as a dependency for consumer reg01:03
rm_workSHOULD be good to go01:03
rm_workdoes there need to be a bug created in Barbican just so we can close it with this CR?01:09
rm_workrechecking one more time just to be absolutely sure :P01:24
rm_worksee ya'll tomorrow01:24
openstackgerritChelsea Winfree proposed a change to openstack/barbican: Add Certificate Interface & Symantec Plugin  https://review.openstack.org/10719001:52
chellygelalee_afk, alee_, thanks -- fixed01:53
chellygelso happy it worked out rm_work :D01:57
openstackgerritArun Kant proposed a change to openstack/barbican: Adding keystone notification listener support  https://review.openstack.org/11081702:16
chellygelthanks alee_ :D02:18
*** woodster has quit IRC02:45
*** bdpayne has joined #openstack-barbican02:45
*** bdpayne has quit IRC02:55
openstackgerritAde Lee proposed a change to openstack/barbican: Add code to retrieve secrets metadata and data with transport key  https://review.openstack.org/10711203:02
*** bdpayne has joined #openstack-barbican03:12
*** alee_afk has quit IRC03:15
openstackgerritJohn Wood proposed a change to openstack/barbican: Replace hard-coded setup version setting  https://review.openstack.org/10958003:27
*** alee_afk has joined #openstack-barbican03:27
*** woodster_ has joined #openstack-barbican03:30
*** ayoung has quit IRC03:35
rm_workchellygel: :P03:36
chellygelhey rm_work  :D03:36
*** alee_ has quit IRC03:50
openstackgerritA change was merged to openstack/barbican-specs: Update spec to describe how CADF audit should be supported across keystone and barbican.  https://review.openstack.org/10641203:54
openstackgerritChelsea Winfree proposed a change to openstack/barbican: Add Certificate Interface & Symantec Plugin  https://review.openstack.org/10719004:23
*** bdpayne has quit IRC04:26
*** anteaya has quit IRC04:35
*** bdpayne has joined #openstack-barbican04:35
*** uberj has quit IRC04:36
*** uberj_ has joined #openstack-barbican04:36
*** anteaya has joined #openstack-barbican04:44
*** juantwo has quit IRC04:49
*** jaosorior has joined #openstack-barbican05:02
*** bdpayne has quit IRC05:02
*** bdpayne has joined #openstack-barbican05:26
*** bdpayne has quit IRC05:48
openstackgerritA change was merged to openstack/barbican: Add code to retrieve secrets metadata and data with transport key  https://review.openstack.org/10711205:49
*** alee has joined #openstack-barbican05:55
openstackgerritJuan Antonio Osorio Robles proposed a change to openstack/python-barbicanclient: Introduce cliff for cli framework  https://review.openstack.org/10758706:48
*** jamielennox is now known as jamielennox|away07:06
openstackgerritJuan Antonio Osorio Robles proposed a change to openstack/python-barbicanclient: Introduce cliff for cli framework  https://review.openstack.org/10758707:11
*** alee_afk has quit IRC07:20
*** arunkant has quit IRC08:34
*** arunkant has joined #openstack-barbican11:33
*** SheenaG1 has joined #openstack-barbican11:54
openstackgerritJuan Antonio Osorio Robles proposed a change to openstack/barbican: Remove remaining skipTest  https://review.openstack.org/11256711:55
openstackgerritJuan Antonio Osorio Robles proposed a change to openstack/barbican: Remove remaining skipTest  https://review.openstack.org/11256711:56
*** SheenaG11 has joined #openstack-barbican11:58
*** SheenaG1 has quit IRC11:58
*** juantwo has joined #openstack-barbican12:05
*** nkinder has quit IRC12:17
*** alee has quit IRC12:23
*** nkinder has joined #openstack-barbican12:24
*** SheenaG11 has quit IRC12:46
*** nkinder is now known as nkinder_away12:51
*** lecalcot has joined #openstack-barbican13:11
*** lecalcot has quit IRC13:22
*** alee has joined #openstack-barbican13:27
*** SheenaG1 has joined #openstack-barbican13:28
*** lecalcot has joined #openstack-barbican13:32
*** alee_ has joined #openstack-barbican13:42
*** SheenaG1 has quit IRC13:43
openstackgerritA change was merged to openstack/barbican: Remove remaining skipTest  https://review.openstack.org/11256713:49
*** lecalcot has quit IRC14:00
*** akoneru has joined #openstack-barbican14:03
*** ayoung has joined #openstack-barbican14:04
*** lecalcot has joined #openstack-barbican14:07
*** Kevin_Bishop has joined #openstack-barbican14:11
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add "Connection" HTTP header to response  https://review.openstack.org/11244414:13
openstackgerritAdam Harwell proposed a change to openstack/barbican: Add support to Barbican for consumer registration  https://review.openstack.org/10784514:14
*** samuelbercovici has joined #openstack-barbican14:16
*** lecalcot has quit IRC14:30
*** lecalcot has joined #openstack-barbican14:31
*** lecalcot has quit IRC14:32
*** crc32 has joined #openstack-barbican14:33
*** rellerreller has joined #openstack-barbican14:37
*** lecalcot has joined #openstack-barbican14:38
*** paul_glass has joined #openstack-barbican14:42
*** uberj_ has quit IRC14:50
*** uberj has joined #openstack-barbican14:51
*** samuelbercovici has quit IRC14:59
*** rellerreller has quit IRC15:00
*** rellerreller has joined #openstack-barbican15:01
*** paul_glass has quit IRC15:11
*** mdorman has joined #openstack-barbican15:16
*** SheenaG1 has joined #openstack-barbican15:18
*** rm_mobile has joined #openstack-barbican15:21
rm_mobileredrobot: fixed the fix, can you review again?15:21
*** paul_glass has joined #openstack-barbican15:22
rm_mobilewoodster_: same15:24
rm_mobilehttps://review.openstack.org/#/c/112444/15:24
rm_mobile<315:25
redrobotrm_mobile +215:26
rm_mobile<3<315:26
*** rellerreller has quit IRC15:33
*** lecalcot has quit IRC15:35
rm_mobileI am coming to appreciate the quality that does tend to emerge from the slow, painstaking multiple month review process15:36
rm_mobileIt's annoying from a "progress" perspective, but it does generate clean and effective code15:37
rm_mobileT_T15:37
rm_mobileAlso, thanks for putting up with my nagging :P15:37
*** jaosorior has quit IRC15:42
openstackgerritKevin Bishop proposed a change to openstack/barbican: First attempt at adding the symantecssl library  https://review.openstack.org/11014415:56
*** lisaclark2 has joined #openstack-barbican15:58
redrobotrm_mobile thank you for all the work... and not stabbing any of us yet..16:02
hockeynut"yet" :-)16:03
* rm_mobile puts away his shank16:03
hockeynutbarbican team breathes a collective sigh of relief16:04
redrobotI'm thinking about adding an ascii art castle to my email sig16:05
reaperhulkor, bear with me, don't have a sig at all16:06
redrobotno way!  that's what chat is for...  must...  flair...  email.16:06
*** lisaclark2 has quit IRC16:10
hockeynuthow did we come full circle from 300 baud text to high def graphics now back to asii art?16:13
rm_mobile"environmentalism"16:15
redrobotIt's so I can use it when I'm sending plaintext gpg-signed email ^_^16:15
rm_mobileAll that extra bandwidth is a waste of our planet's natural fiber-optic resources16:15
hockeynutsorry, I missed that last one while I was turning on lights in all of our unoccupied rooms16:16
*** bdpayne has joined #openstack-barbican16:18
*** lisaclark1 has joined #openstack-barbican16:22
redrobothttp://paste.openstack.org/show/91611/16:22
*** bdpayne has quit IRC16:30
hockeynutthere goes my monthly bandwidth16:33
hockeynutI like it, very barbicanny16:33
rm_mobileThat it a pretty sweet castle16:34
rm_mobile*is16:34
*** jaosorior has joined #openstack-barbican16:40
*** paul_glass has quit IRC16:45
openstackgerritArvind Tiwari proposed a change to openstack/barbican: Add more type in order post  https://review.openstack.org/8740516:56
*** atiwari has joined #openstack-barbican16:58
*** bdpayne has joined #openstack-barbican16:59
*** Kevin_Bishop has quit IRC17:01
*** bdpayne_ has joined #openstack-barbican17:14
*** lisaclark1 has quit IRC17:15
*** bdpayne has quit IRC17:16
*** lisaclark1 has joined #openstack-barbican17:20
*** lisaclark1 has quit IRC17:25
*** lisaclark1 has joined #openstack-barbican17:25
*** lisaclark1 has quit IRC17:25
*** paul_glass has joined #openstack-barbican17:31
*** juantwo_ has joined #openstack-barbican17:38
*** juantwo has quit IRC17:42
openstackgerritKaitlin Farr proposed a change to openstack/barbican: Adds store_secret_supports to secret_store  https://review.openstack.org/11038617:43
*** gyee has joined #openstack-barbican17:49
paul_glasshey hockeynut?17:51
hockeynutyessir17:51
paul_glassfor this skip on issue thing, should labelling an issue with an environment ignore the status of the issue?17:52
paul_glassor: when we specify an environment, does that imply the issue is open/closed in that environment?17:54
paul_glassor: am I totally misunderstand this?17:55
paul_glassmisunderstanding*17:55
hockeynutsorry - got scared off for a few mins18:13
hockeynutthe idea is to determine that the fix is available in environment X18:14
*** Kevin_Bishop has joined #openstack-barbican18:14
paul_glassokay.18:16
paul_glassso in JIRA, we tag with environment X18:17
paul_glassin the opencafe config file, we specify that this is environment X18:17
paul_glassand then the plugin grabs the issue, and if it sees environment X, it should run the test18:18
hockeynutu going to be in Austin tomorrow?18:19
paul_glassI wasn't planning on it, but I could be.18:19
hockeynutI think you've got the idea18:19
hockeynutthe whole idea is that we don't want to run a test when an issue's fix isn't available18:20
*** lecalcot has joined #openstack-barbican18:22
openstackgerritKevin Bishop proposed a change to openstack/barbican: First attempt at adding the symantecssl library  https://review.openstack.org/11014418:51
openstackgerritKaitlin Farr proposed a change to openstack/barbican: Adds store_secret_supports to secret_store  https://review.openstack.org/11038619:20
*** bdpayne_ has quit IRC19:24
openstackgerritChelsea Winfree proposed a change to openstack/barbican: Add Certificate Interface & Symantec Plugin  https://review.openstack.org/10719019:29
rm_workso when do you guys think this connection_handler fix will make it in? maybe middle of next week?19:36
rm_workI'm not honestly sure how much impact putting another middleware filter in your pipeline will affect things… it's not "non-impacting", at the least19:37
*** rm_mobile has quit IRC19:38
rm_workredrobot / woodster_ ^^19:41
*** bdpayne has joined #openstack-barbican19:49
chellygelrm_work, woodster_ is out and redrobot stepped away19:58
rm_workheh ok20:07
*** jraim__ is now known as jraim20:09
SheenaG1jraim: hey20:11
SheenaG1jraim: you're alive!20:11
jraimyep20:11
*** alee_ has quit IRC20:30
*** bubbva has quit IRC20:43
*** bubbva has joined #openstack-barbican20:43
openstackgerritA change was merged to openstack/barbican: Adds store_secret_supports to secret_store  https://review.openstack.org/11038620:46
openstackgerritKevin Bishop proposed a change to openstack/barbican: First attempt at adding the symantecssl library  https://review.openstack.org/11014420:47
*** bdpayne_ has joined #openstack-barbican21:01
*** bdpayne has quit IRC21:04
*** lecalcot has quit IRC21:18
*** lecalcot has joined #openstack-barbican21:18
*** alee has quit IRC21:19
*** lecalcot_ has joined #openstack-barbican21:20
*** lecalcot has quit IRC21:21
*** lecalcot_ has quit IRC21:21
*** lecalcot has joined #openstack-barbican21:21
*** juantwo_ has quit IRC21:21
*** akoneru has quit IRC21:29
*** atiwari has quit IRC21:34
redrobotpaul_glass http://www.meetup.com/Alamo-City-Python-Group/events/197962342/21:34
*** atiwari has joined #openstack-barbican21:39
rm_workredrobot:21:40
rm_work[14:36:48]  <rm_work> so when do you guys think this connection_handler fix will make it in? maybe middle of next week?21:40
rm_work[14:37:21]  <rm_work> I'm not honestly sure how much impact putting another middleware filter in your pipeline will affect things… it's not "non-impacting", at the least21:40
rm_work(you were away, I guess)21:40
redrobotrm_work just waiting on workflow... maybe woodster_ or reaperhulk can give it a push?21:41
reaperhulkwhat am I looking at21:41
rm_workah, i mean, does anyone else want to actually … analyze the impact it might have?21:41
redrobotreaperhulk https://review.openstack.org/#/c/112444/21:41
rm_workfor once I'm actually hesitant to just rush something through :P21:42
reaperhulka change from adam? -121:42
redrobotrm_work I mean, it's just ensuring the header is there no the way out, right?21:42
rm_workredrobot: yeah, but it's a new filter in the pipeline21:42
rm_workdunno21:42
rm_workI'm not super familiar with paste21:42
redrobotrm_work looked good to me when I saw it.21:42
rm_workalright21:42
rm_workthe license was another thing i wasn't clear on21:43
redrobotrm_work all of OpenStack is Apache v221:43
rm_workok, well21:43
rm_workhttps://review.openstack.org/#/c/92448/4/magnetodb/common/middleware/connection_handler.py21:43
rm_workCopyright 2014 Mirantis Inc.21:43
redrobottbh I don't know who the copyright holder should be?  I'm sure there's a clause about that in the contributor agreement21:44
reaperhulkwe have generally allowed the use of whatever company the contributor works for21:45
rm_workk, though I think technically magnetodb is Stackforge21:45
rm_worknot sure if it matters21:45
rm_workI'm just voicing my concern, if people think it21:45
reaperhulkhmm, okay, so trying to understand this change21:45
rm_work*it's fine, then cool :P21:45
reaperhulkIf you send a request with a connection header it copies it to the response21:46
reaperhulkand this fixes tempest madness?21:46
rm_workreaperhulk: yes21:46
reaperhulk...why does it fix tempest?21:46
rm_worki think it's not just tempest21:46
rm_worki think this COULD happen with multiple concurrent requests from users, technically21:46
rm_workthe middleware layer mucks up the responses or something21:47
rm_worknot 100% sure21:47
rm_workbut it definitely fixes the problem, I'll give it that21:47
reaperhulkI basically just want to understand this enough to know that I shouldn't be saying "pecan should fix this"21:47
rm_workreaperhulk: right21:48
rm_workOriginally when I saw this I said "oh, so it's a pecan problem"21:48
rm_workI am not convinced that the middleware thing isn't a workaround21:48
rm_workthat said, might we be ok with a workaround?21:48
rm_workwho is the pecan guy again? is he in this channel?21:49
reaperhulkryanpetrello :)21:49
rm_workcool21:49
rm_workmaybe he could shed some light21:49
rm_workryanpetrello: ^^21:49
reaperhulkI definitely don't have enough context to understand where this problem is really coming from21:50
rm_workif we ping him enough, on the first thursday of a month, during a fool moon, i heard he appears to grant wishes (that are related to API changes)21:50
rm_work*full moon21:50
rm_work… I am really bad with "patience"21:51
rm_workit gets worse when I get close to deadlines <_<21:51
*** atiwari has quit IRC21:54
rm_workreaperhulk: so are you thinking that we hold off until ryanpetrello has a chance to look and comment and possibly investigate doing the fix at the pecan layer? or telling us there is an easy way to already do this with a pecan setting? :P21:55
reaperhulkWell, copying a connection header from request to response smells like cargo culting without understanding the problem. Doesn't mean it is, but I'd like a logical and coherent reason why this resolves issues (preferably backed by an RFC)21:56
reaperhulk"it fixes it" is insufficient for me to understand where the fix should live :)21:56
reaperhulkFor instance, what value for connection triggers this problem? Any value? close only?21:57
rm_workprobably it is trying to use keep-alive or something?21:58
rm_workthough yeah, not totally sure21:58
rm_workthough that doesn't really make sense actually, these SHOULD be close connections21:59
rm_workah21:59
rm_workHTTP/1.1 applications that do not support persistent connections MUST include the "close" connection option in every message.22:00
rm_workhttp://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html22:00
rm_workguessing it's that22:00
rm_workreaperhulk: ^^22:00
rm_workso the fix is really just a hack, we're copying the Close connection header that someone else properly set :P22:00
reaperhulkOkay, does pecan not support persistent connections? Most things actually do. And if not, why wouldn't it set close on its own?22:01
rm_workyeah22:01
rm_workso the correct fix would really be to make sure that if we're not doing persistent connections, pecan needs to actually set that22:01
rm_workwhich actually makes sense for the error i was getting22:01
rm_workthe tempest REST client doesn't see "close" and then is confused when the connection is shut down :P22:02
reaperhulkit's also possible that a different wsgi server would add it (to my knowledge we're the only uwsgi users in devstack)22:04
reaperhulkRegardless, if we want to land this hack we can but we should prioritize fixing it upstream via whatever method makes the most sense. Right now it seems like we should talk to pecan but maybe they will redirect us.22:05
reaperhulkThen we can pull this hack when the right solution emerges22:05
rm_workI'm not 100% sure alright22:06
*** paul_glass has quit IRC22:06
*** bdpayne_ has quit IRC22:07
*** bdpayne has joined #openstack-barbican22:09
rm_worki'm ok with either, assuming that getting pecan to check it out and fix it doesn't take longer than Juno-3 code freeze :P22:10
rm_workwith ryanpetrello was here :(22:10
rm_work*wish22:10
reaperhulkwell let's hold off for now and see what we can learn tomorrow22:13
reaperhulkeither way we'll land something soon22:13
redrobotlanding rm_work patches is starting to sound like theo boy who cried wolf... :-P22:14
rm_workT_T22:15
rm_workthe one got as far as workflow :P22:15
dstufftTechincally you should send the Connection: close header on any response which comes before the server closes the connection22:17
dstufftjust closing the connection isn't concerned kosher afaik22:18
reaperhulkand since pecan doesn't actually control that (the wsgi server does) maybe it doesn't belong in pecan.22:19
dstufftthat's correct22:20
dstufftthe WSGI spec doesn't allow the WSGI consumer to set hop by hop headers22:20
dstufftwhich the Connection header is22:20
dstufftsetting the Connection header from within a WSGI app is wrong22:20
* dstufft has read both WSGI specs, and all the HTTP/1.1 specs recently22:21
dstufftmanaging the connection state is entirely within the bounds of the web server, not the wsgi app22:23
dstufftsame with transfer-encoding22:23
dstufftand other hop by hop headers22:23
reaperhulkSo then the correct fix here is to investigate why uwsgi isn't doing what we expect or switch to mod_wsgi inside devstack (which had a WIP CR a few months ago but got abandoned)22:26
reaperhulkbut for now I need to go down to the beach22:27
dstufftI only jumped in through the middle, what was the original problem?22:27
dstufftok22:27
reaperhulkproblem is tempest tests in the devstack gate are failing. apparently it is fixed by returning connection:close but why tempest is intolerant of not getting that or why uwsgi isn't sending it, etc are not something I understand right now22:28
reaperhulkrm_work can supply more info if you bug him ;)22:28
dstufftrm_work: can you tell me what failing looks like? ;)22:29
dstufft$2 says it's either state is leaking and closing the connection causes it to be cleaned up, or tempest doesn't properly handle persistent connections22:29
*** SheenaG1 has quit IRC22:33
openstackgerritKaitlin Farr proposed a change to openstack/barbican: Adds KMIPSecretStore and unit tests  https://review.openstack.org/10158222:34
*** rm_mobile has joined #openstack-barbican22:43
*** lecalcot has quit IRC22:44
*** lecalcot has joined #openstack-barbican22:44
*** Kevin_Bishop has quit IRC22:46
*** lecalcot has quit IRC22:49
*** ayoung has quit IRC22:53
*** AndChat|40521 has joined #openstack-barbican23:05
*** rm_mobile has quit IRC23:05
*** AndChat|40521 has quit IRC23:05
*** juantwo has joined #openstack-barbican23:09
rm_workah23:11
rm_workI was off for a bit, got locked out of my account by someone whacking on my keyboard while i wasn't at my computer (and it was locked)23:12
rm_workwhat did i miss23:12
*** SheenaG1 has joined #openstack-barbican23:14
rm_workdstufft: failure looks like a HTTP connection closed unexpectedly error23:14
rm_workor, i can get the real text...23:14
rm_workdstufft: error: [Errno 104] Connection reset by peer23:15
rm_workdstufft: did you actually see the fix CR?23:15
*** SheenaG11 has joined #openstack-barbican23:15
dstufftthe middleware?23:15
rm_workyes23:15
rm_workhttps://review.openstack.org/#/c/112444/3/barbican/api/middleware/connection_handler.py23:15
dstufftyea it's wrong23:15
rm_workit's … not EXPLICITLY setting Connection: close23:15
rm_workit's… passing the value on23:16
rm_workbut23:16
rm_workI don't *disagree* with you23:16
rm_worki'm researching uwsgi now23:16
*** SheenaG12 has joined #openstack-barbican23:17
dstufftthe fact it works at all is an accident, a WSGI server is not required to honor hop by hop headers sent by the app23:18
*** SheenaG1 has quit IRC23:19
*** SheenaG11 has quit IRC23:19
dstufftI should probably know this, but is it nginx + uwsgi?23:21
*** SheenaG12 has quit IRC23:21
*** jaosorior has quit IRC23:22
*** jamielennox|away is now known as jamielennox23:23
dstufftoh!23:23
dstuffthm23:23
rm_workno, devstack uses uwsgi directly23:23
rm_workas the webserver23:23
dstufftrm_work: does the tempest tests hammer it?23:23
rm_worka little bit23:23
rm_workyes23:23
rm_workthe issue is a timing one, definitely23:23
rm_workif we put sleep(2) into the tests before every HTTP hit, the problem goes away23:24
dstufftI wonder if it's a problem with the listen backlog23:24
rm_workdstufft: ANY help you can provide looking into this with me is very appreciated23:24
rm_workthis isn't really "my problem" but it seems I got stuck with it anyway23:24
rm_workand I feel a little like...23:24
*** mdorman has quit IRC23:25
rm_workhttp://24.media.tumblr.com/tumblr_lzaywwdqTC1qhgs46o1_500.jpg23:25
dstufftif tempest is holding open a bunch of connections, one per test, but isn't closing them, the socket backlog can fill up and the socket will return ECONNRESET23:25
rm_workhmm23:25
rm_workthat could be?23:25
rm_workis there a way we could verify that?23:26
dstufftif that's the cause the ECONNRESET will be returned before that connection ever has a chance to talk to uwsgi23:27
dstufftcould modify the tests to add some unique value to each HTTP hit, and the wsgi app to record what unique values it sees, and see if the ECONNRESET connections's unique value was ever seen by the server23:28
dstufftor could try just cranking up the backlog23:28
rm_workok, how would I do the latter?23:29
rm_workI have a system I can easily reproduce the problem on23:29
dstufft-l <something> on the uwsgi command line23:29
dstufftthat's a lowercase l23:29
dstufftaccording to this post the default is 10023:30
rm_workk23:30
dstufftthere might be a log message too23:30
dstufftsomeone said they had a log message saying -> your server socket listen backlog is limited to 100 connections.23:30
rm_workspinning up devstack with a patchset that was still vulnerable to the bug (pre-fix rebase)23:32
dstufftrm_work: are you familar with what the backlog setting does?23:34
rm_worknope23:36
rm_workyour server socket listen backlog is limited to 100 connections23:36
rm_workgetting that even with -l 1000023:36
rm_workthere doesn't appear to be any uwsgi config overriding that though (I guess the vassal INI could?)23:37
dstufft10000 might be greater than the max23:38
dstuffttry 1024 or 204823:38
rm_worki tried 1000 first23:39
rm_work;(23:39
dstufftoh23:39
dstuffthm23:39
dstufftthat's strange..23:39
dstufftI hate uwsgi23:39
dstufftconfusing bugger23:39
rm_worki don't see anything about "backlog" as an option in the uwsgi docs :(23:40
rm_workah, socket listen queue size23:41
dstufftyea23:42
dstufftrm_work: when linux gets a socket connection from a client, it puts it in a queue, a server pulls stuff off that queue and does stuff with it, the backlog is how big that queue is before the kernel starts rejecting connections23:44
rm_workk23:44
rm_workthat could be it23:44
rm_workif i could figure out how to actually make it respect my setting...23:44
reaperhulkit's possible that devstack has that queue set really low, but I forget what param that'd be in /proc23:45
dstufftIf I remember correctly, if those connections are being held open b/c they are persistent, then a limit of 100 means that we can have however many workers the uwsgi is setup for + 100 connections before it starts rejecting things23:45
rm_worki'm just running the uwsgi command that devstack uses23:45
rm_worki honestly don't think there are even a total of 100 connections *in* the tests23:45
rm_workbut23:45
rm_workmaybe23:45
dstufftI dunno :) it sounded similar to that kind of problem, but it may not be23:46
rm_workI wonder if it is possible to set that setting in the vassal configs23:46
dstufftrm_work: any messages like uWSGI listen queue of socket 3 full !!! (101/100) ***"23:46
dstufftin the log23:46
rm_workhmm23:47
dstufftreaperhulk: somaxconn23:47
rm_workhttp://logs.openstack.org/45/107845/19/check/gate-barbican-devstack-dsvm/ab5b005/logs/screen-barbican.txt.gz23:47
rm_workdon't SEE any23:47
rm_worknot sure how much of the logging is actually in here23:48
rm_workversus… routed elsewhere that i may not be able to see23:48
dstufftrm_work: what does cat /proc/sys/net/core/somaxconn mean?23:48
dstuffter23:48
dstufftsay23:48
dstufftI think that's right23:48
rm_work12823:48
reaperhulkheh, try setting -l 12723:48
rm_worklol23:48
reaperhulkI bet it will let you do that :)23:48
rm_worklol nope still 100 T_T23:49
rm_worki was totally with you though23:49
dstufftdid I ever mention computers are the worst23:49
rm_workit prints twice though, if you look at the log i linked23:49
rm_workwhich is a sample23:49
rm_workI guess once per vassal?23:50
rm_workwhich implies it's a per-vassal setting23:50
rm_workwhich makes me think maybe i can edit the vassal config to include this setting23:50
dstuffttry that23:50
rm_workbut the uwsgi docs are… <_<23:50
rm_workso i am not sure how23:50
dstufftyea23:50
dstufftI hate uwsgi23:50
dstufftit's the worst23:50
dstufftI mean, it's supposidly good software, and you can do a TON of things with it23:50
dstufftbut everytime I try to use it I get real angry23:51
reaperhulkwe need to do the mod_wsgi conversion for devstack anyway23:51
rm_workok i guessed and i was right23:51
rm_worklisten = #23:51
rm_work:P23:51
rm_workand it worked23:51
dstufft\o/23:51
rm_workso now i set it really high and retest...23:51
rm_workoh, it totally is locked to 128 heh23:52
rm_workawesome23:52
dstufftyou can change that23:52
reaperhulkecho 2048 > /proc/sys/net/core/somaxconn23:52
dstufftyea what reaperhulk said23:52
dstufftthis is linux, there's very little you can't do with some echo and /proc :D23:53
reaperhulkthen run tempest and we see what happens!23:53
rm_workyeah i did23:53
rm_worki'm familiar with /proc settings fortunately :P23:53
dstufftI have to afk for a few to give my daughter her infusion, will be back in 10-15 or so23:54
rm_workaaaand absolutely no difference23:54
rm_worktempest tests still borked23:55
rm_workkk thanks dstufft for the help :)23:55
rm_worktesting to see what the server actually sends me for headers23:55
rm_worklol so yeah23:56
rm_workuwsgi does NOT ever send a "close" connection header23:56
rm_workin fact23:56
rm_work* Connection #0 to host 127.0.0.1 left intact23:56
rm_workis what curl helpfully tells me23:56
dstufftwell23:56
dstufftuwsgi isn't RFC compliant than23:57
dstufftIIRC23:57
rm_workyeah, so23:57
dstufftunless it's a HTTP/1.0 connection23:57
rm_worki am trying to see if there's an option23:57
rm_work< HTTP/1.1 200 OK23:57
rm_workso nope23:57
* dstufft really goes now23:57
rm_workkk23:57
rm_workso reading this: http://uwsgi-docs.readthedocs.org/en/latest/HTTP.html#http-keep-alive23:58
rm_workimplies that I could do the opposite23:58
rm_workand put: add-header "Connection: close"23:58
rm_workbut i don't know if that means EVERY response would include that23:59
rm_workthough, even if every response did, i don't think there's anything at all in barbican using keep-alives >_<23:59
« Wednesday, 2014-08-06 Index Friday, 2014-08-08 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!