Monday, 2016-08-01

*** markvoelker has joined #openstack-security00:12
*** sdake has joined #openstack-security00:13
*** markvoelker has quit IRC00:17
*** sdake has quit IRC00:17
*** austin987 has quit IRC00:36
*** austin987 has joined #openstack-security00:40
*** austin987 has quit IRC00:47
*** austin987 has joined #openstack-security00:47
*** austin987 has quit IRC00:59
*** markvoelker has joined #openstack-security01:13
*** markvoelker has quit IRC01:17
*** austin987 has joined #openstack-security01:27
*** davidjd-gh has joined #openstack-security01:37
*** davidjd-gh has left #openstack-security01:37
openstackgerritRahul U Nair proposed openstack/bandit: Fixing jenkins failing on coverage reporting  https://review.openstack.org/34932902:55
unrahulgmurphy:  I have pushed a change for the bandit failing on tox02:56
unrahulfor Syntribos we had a similar issue and it was fixed by doing these changes.02:57
*** davidjd-gh has joined #openstack-security02:57
*** davidjd-gh has left #openstack-security02:57
unrahulgmurphy:  I am not getting any coverage failed errors locally, let's see what the bandit cores think.02:57
*** Nikolay_St has joined #openstack-security03:04
*** yuanying has quit IRC03:08
*** amitkqed has quit IRC03:38
*** amitkqed has joined #openstack-security03:38
gmurphythanks!03:38
*** tmcpeak has joined #openstack-security03:59
openstackgerritGrant Murphy proposed openstack/bandit: Add check for httpoxy vulnerability  https://review.openstack.org/34901504:50
*** tmcpeak has quit IRC05:02
*** Nikolay_St has quit IRC05:23
*** austin987 has quit IRC06:03
*** Nikolay_St has joined #openstack-security06:19
*** tkelsey has joined #openstack-security06:29
openstackgerritRahul U Nair proposed openstack/syntribos: Adding unique_id to tests  https://review.openstack.org/34528606:31
*** tkelsey has quit IRC06:33
*** Cormite has joined #openstack-security06:34
*** austin987 has joined #openstack-security06:36
*** tesseract- has joined #openstack-security06:44
*** austin987 has quit IRC06:48
openstackgerritRahul U Nair proposed openstack/syntribos: Adding additional fields to debug log  https://review.openstack.org/34708906:51
*** Cormite has quit IRC07:14
*** liverpooler has joined #openstack-security07:22
openstackgerritRahul U Nair proposed openstack/syntribos: Updating documentation  https://review.openstack.org/34938807:34
*** Cormite has joined #openstack-security07:53
*** yuanying has joined #openstack-security08:08
*** tkelsey has joined #openstack-security08:20
openstackgerritRahul U Nair proposed openstack/syntribos: Standardizing the way we diff signals  https://review.openstack.org/34940308:24
*** tkelsey has quit IRC08:33
openstackgerritOpenStack Proposal Bot proposed openstack/security-doc: Updated from openstack-manuals  https://review.openstack.org/34942609:04
*** eric_lopez has joined #openstack-security09:13
*** elo has quit IRC09:16
*** hyakuhei has joined #openstack-security09:33
*** hyakuhei has left #openstack-security09:37
openstackgerritMerged openstack/security-doc: Updated from openstack-manuals  https://review.openstack.org/34942610:20
*** hyakuhei has joined #openstack-security10:27
*** edmondsw has joined #openstack-security11:40
*** shohel has joined #openstack-security11:51
sigmavirusgmurphy: So I was talking to Twisted and they're going to fix that. I dont' know anyone around wsgiref11:59
*** sdake_ has joined #openstack-security12:02
openstackgerritMerged openstack/security-doc: Update links from Liberty to Mitaka  https://review.openstack.org/34569612:09
*** davidjd-gh has joined #openstack-security12:24
*** davidjd-gh has left #openstack-security12:24
*** markvoelker has joined #openstack-security12:27
gmurphysigmavirus: yeah this was mostly my question the other day. should this kind of rule be limited to specific version ranges? i'm not sure if any of the other rules (for example the xml ones) have been addressed now either.12:37
gmurphyso i can see value in having this rule right now. specifically for understanding if their's exposure to this vulnerability in your codebase. but as the vulnerability is patched it will become a false positive eventually.12:38
sigmavirusgmurphy: right12:40
sigmavirusI think yes?12:40
gmurphyok. i'm gonna have to think about how to do that then. i was hoping to sneak this one in under the blacklisting stuff… might not be able to do that now :-)12:44
*** sdake_ is now known as sdake12:48
*** Nikolay_St has quit IRC12:50
gmurphyi guess i could check twisted.version, and maybe sys.version for wsgiref12:52
*** _elmiko is now known as elmiko13:01
*** dikonoor has joined #openstack-security13:06
*** cleong has joined #openstack-security13:10
*** Nikolay_St has joined #openstack-security13:14
*** JAHoagie has joined #openstack-security13:15
*** dave-mcc_ has quit IRC13:19
*** dave-mccowan has joined #openstack-security13:19
*** dave-mccowan has quit IRC13:19
*** Nikolay_St has quit IRC13:25
openstackgerritMerged openstack/bandit: Fixing jenkins failing on coverage reporting  https://review.openstack.org/34932913:28
openstackgerritMerged openstack/bandit: Add check for httpoxy vulnerability  https://review.openstack.org/34901513:28
*** zul_ has quit IRC13:38
*** ametts has joined #openstack-security13:41
*** Nikolay_St has joined #openstack-security13:42
*** zul has joined #openstack-security13:43
*** liverpooler has quit IRC13:43
*** dave-mccowan has joined #openstack-security13:46
*** ametts has quit IRC13:51
*** ametts has joined #openstack-security14:00
*** unrahul has quit IRC14:04
*** amit213 has quit IRC14:04
*** fyxim has quit IRC14:04
*** aimeeu has quit IRC14:04
*** tpeoples has quit IRC14:04
*** DuncanT has quit IRC14:04
*** jraim has quit IRC14:04
*** serverascode has quit IRC14:04
*** fyxim has joined #openstack-security14:07
*** shohel has quit IRC14:15
*** dave-mccowan has quit IRC14:16
*** alejandro2 has joined #openstack-security14:16
*** alejandro2 has left #openstack-security14:16
*** dave-mccowan has joined #openstack-security14:17
*** aimeeu has joined #openstack-security14:17
*** austin987 has joined #openstack-security14:17
*** JAHoagie has quit IRC14:17
*** DuncanT has joined #openstack-security14:18
*** tpeoples has joined #openstack-security14:18
*** amit213 has joined #openstack-security14:19
*** jraim has joined #openstack-security14:19
*** serverascode has joined #openstack-security14:19
*** unrahul has joined #openstack-security14:19
*** tmcpeak has joined #openstack-security14:20
*** aastha has joined #openstack-security14:25
*** austin987 has quit IRC14:27
*** eric_lopez has quit IRC14:41
*** eric_lopez has joined #openstack-security14:49
*** dikonoor has quit IRC14:52
*** dave-mccowan has quit IRC15:09
*** dave-mccowan has joined #openstack-security15:13
*** dave-mccowan has quit IRC15:16
*** mdong has joined #openstack-security15:33
*** sdake has quit IRC15:34
*** dave-mccowan has joined #openstack-security15:43
*** Cormite has quit IRC15:43
openstackgerritMichael Dong proposed openstack/syntribos: CORS test now operates on test_resp instead of init_resp  https://review.openstack.org/34957815:53
openstackgerritMerged openstack/syntribos: Improving parser, adding unit tests  https://review.openstack.org/34803115:58
*** Guest15832 is now known as redrobot16:04
*** ccneill has joined #openstack-security16:07
*** markvoelker has quit IRC16:09
*** tesseract- has quit IRC16:14
*** vinaypotluri has joined #openstack-security16:29
*** tesseract- has joined #openstack-security16:46
*** tesseract- has quit IRC16:54
*** tesseract- has joined #openstack-security16:58
*** tesseract- has quit IRC16:58
*** sdake has joined #openstack-security17:02
*** markvoelker has joined #openstack-security17:05
*** sdake has quit IRC17:17
vinaypotluriccneill: i've assigned a task from trello "SECTEST-SYN Update documentation" where i have to figure out a way to add our documentation. Should it be something like a complete syntribos documentation or anything specific ?17:24
vinaypotluriex: http://docs.openstack.org/developer/nova/17:24
ccneillthe point is to publish the docs that we have in our "doc" folder to the OpenStack site17:27
ccneillI'm not sure how to kick off that process, or where that documentation would live17:27
ccneillbut I don't think we should be working on that yet - hence why I put the docs stuff in "Backlog" instead of "Doing"17:28
ccneillwe need to finish up the "doing" stuff first17:28
vinaypotluriokay17:28
ccneille.g. https://trello.com/c/Q2uhE3kI/114-sectest-syn-finish-revamping-cli-config17:28
ccneillsince that will ultimately change what we need to put in the docs17:29
vinaypotluricool17:30
vinaypotlurii'll work on "Doing" for now17:30
ccneillcool cool17:30
ccneillif there are any tasks that are assigned but don't have associated CRs, see if the asigned person is working on them, and if not you can re-assign those tasks to yourself17:30
vinaypotluriokay17:34
*** mhayden has quit IRC17:38
*** mhayden has joined #openstack-security17:40
*** markvoelker_ has joined #openstack-security17:53
*** markvoelker has quit IRC17:53
unrahulHey ccneill  I was thinking on how we should streamline template file naming/reading to avoid parsing issues and stuff..18:16
unrahulwhat if we  add a line on top of the template file, that the tool has to check and gets metadata from it.. rather than following a particular way of naming files..?18:17
*** markvoelker_ has quit IRC18:38
*** markvoelker has joined #openstack-security18:39
*** markvoelker has quit IRC18:39
*** Nikolay_St has quit IRC18:40
*** ccneill has quit IRC18:45
*** eric_lopez has quit IRC18:52
*** eric_lopez has joined #openstack-security19:01
*** ccneill has joined #openstack-security19:10
*** ccneill has quit IRC19:14
*** ccneill has joined #openstack-security19:15
ccneillunrahul: that might be a good approach19:17
ccneillunrahul: that gives us a lot more flexibility than trying to build it directly into the filename19:17
*** sdake has joined #openstack-security19:18
*** ametts has quit IRC19:30
*** ametts has joined #openstack-security19:36
unrahulyeah..I also feel so.. and that in a way leaves less room for users to make mistakes when they add/extend set of templates in syntribos.. where the file name is not of concern.. but a header should be there.. that says what it is..19:39
ccneillyeah.. I think a YAML metadata section would be handy-dandy19:41
unrahul+1 at the minimum an encoding   type header like # -*- coding: utf-8 -*-  , so like -*- method :get -*-19:45
mdongso my question is, what does that method metadata line in a request template get us that the request itself doesn’t already?19:47
mdongconsidering that the first word in a request template is the http method19:48
openstackgerritMerged openstack/syntribos: CORS test now operates on test_resp instead of init_resp  https://review.openstack.org/34957819:48
ccneillso I was thinking of it in terms of relationships to other request templates19:54
ccneille.g. associating all the methods for CRUDing a given resource together19:54
ccneillyou could also define what types of data the variables defined in the template accept19:55
*** ametts has quit IRC19:55
ccneillwithout having to make the template itself unreadable with a bunch of information jammed into every line (like the way we do CALL_EXTERNAL today)19:55
*** knangia has joined #openstack-security19:56
mdongre: CRUDing a given resource, that’ll only come up when we do request pipelining, and we can come up with a solution for pipelining that does not entail marking up our request templates any further19:57
mdongI think whatever metadata markup we add should be optional, because right now, request template generation without extensions is the least involved part of the Syntribos workflow19:58
ccneillagreed, not every request needs markup19:59
mdongit should be a selling point that Syntribos can just be pointed at a raw HTTP request file and work to some reasonable degree19:59
*** ametts has joined #openstack-security20:00
unrahulwell.. I think at lease there should be some way to ensure that the file that syntribos is processing is a request template, before reading the entire file  and ccneill  yup.. it can help us in organizing the templates  in a CRUD way, where it can be optional..20:01
unrahulat least*20:01
ccneillmdong: agreed that it should be possible to just use the raw HTTP request20:02
*** khanak has joined #openstack-security20:02
unrahulbecause ryt now its kind of open ended, the tool would try reading any file in the templates directory, what if its a huge binary file..or something, which really can be avoided in some other way.20:02
ccneillright.. so to an extent I think we want to use SOME file naming convention, even if that naming convention doesn't determine anything else about the file beyond "this is a request template"20:03
ccneillmaybe only loading files with a .template extension? .txt?20:04
mdongbut we could solve that problem with documentation as well, or just by passing a message to the user that the file currently being read is unparsable20:04
mdongbecause I think the real problem right now is that Syntribos just crashes when it gets an unparsable file, but there are ways of handling that gracefully20:05
ccneillmdong: what if I have 5 request templates open in VIM, and it creates .swp files for all of them in that dir? should we re-parse all of them and run tests against them, even though they should be the same as the 5 actual request templates?20:05
ccneillthere's nothing invalid about it, it's just wasted effort20:05
mdongI like the .template idea20:05
ccneilldoesn't really significantly change anything about how the templates work - just makes it clear that "THIS IS A TEMPLATE FILE"20:06
ccneillthat way you can put a README.md in your templates folder if you want, to explain the templates included, etc.20:06
mdongwe could also just, by default, ignore .swp, __MACOSX files, etc20:06
unrahulmdong:  okay, will this ever occur, syntribos is running in a ci env, tries to parse a huge file, takes up a lot of resources and slows down the entire ci job..? because as of now.. that can happen ryt..?20:06
ccneill¯\_(ツ)_/¯ that gets to be an endless game though20:06
unrahulmdong: whitelisting is always better than blacklisting..20:06
unrahuli like the idea of .templates..20:06
unrahulbandit also have something like that..ryt?20:07
mdongwell, how does Bandit handle the case of files it can’t read?20:07
mdongbecause for bandit, we just point it at a directory20:07
unrahuli am not sure.. but remember seeing.. a directory of files with some extension..20:08
mdongand invarably the directory will contain readmes and swap files20:08
mdongyet in the end it still only operates on the .py files20:08
unrahulwe could whitelist .templates files..20:08
unrahullike all templates should be files with .templates extension..20:09
ccneillmdong: my main thing is this - it takes almost no effort to give a file a given extension, but it takes a fair amount of effort for us to try to engineer around every OTHER kind of file that might exist there20:09
ccneillonly parsing .template files means that, if we get an invalid template, we got it because the user wrote a bad template file - not any other possible combination of weirdness20:10
unrahuli guess.. that can be a quick fix to some of the issues..20:10
ccneilla 50000MB .template file isn't randomly going to show up in that dir20:10
ccneilland a .DS_Store.template file isn't either20:10
ccneillvim will create derp.template.swp files20:11
mdongI like the .template idea, I just don’t want us to overengineer a solution to an edge case20:11
ccneilland users can put arbitrary READMEs, etc. in that folder as long as they don't name it ".template", which no other program uses20:11
ccneillwell.. maybe some program uses it, but it's not common20:11
unrahulmdong, ccneill  +1 agreed I also like the idea20:11
ccneillright, me neither - we parse ANYTHING that is a .template and attempt to treat it as a template20:12
ccneillwe don't do any crazy magic() to determine the content type, filesize to figure out if it's a "reasonable" size, etc.20:12
mdongthat sounds reasonable to me, but I’ll do a bit of research to see what bandit does when it encounters readme files and such20:13
mdongI suspect that it’s just looking at the file extension as well20:13
ccneillmdong: looks like it20:15
ccneillhttps://github.com/openstack/bandit/blob/master/tests/unit/cli/test_baseline.py#L2920:15
mdongyep20:15
mdonghttps://github.com/openstack/bandit/blob/master/bandit/core/manager.py#L16420:15
ccneill(didn't dig too far)20:15
ccneillah yep20:16
ccneillshould be a simple call to glob()20:16
ccneillbrb cig20:16
*** ametts has quit IRC20:36
*** ametts has joined #openstack-security20:38
*** JAHoagie has joined #openstack-security20:38
*** knangia has quit IRC20:48
*** sdake has quit IRC21:20
openstackgerritAastha Dixit proposed openstack/syntribos: Adding additional fields to debug log  https://review.openstack.org/34708921:22
*** khanak is now known as knangia21:26
*** cleong has quit IRC21:34
openstackgerritAastha Dixit proposed openstack/syntribos: Adding additional fields to debug log  https://review.openstack.org/34708921:44
openstackgerritCharles Neill proposed openstack/syntribos: Removes FuzzRequest from Syntribos  https://review.openstack.org/34711621:48
*** jamielennox|away is now known as jamielennox21:48
ccneilloof.. gonna try not to add anything else to my remove_fuzzrequest CR O:-)21:49
ccneill+571/-242 :X21:49
ccneillunrahul: added unittests for models, coverage says I got 100% coverage21:49
ccneill:D21:49
ccneillbrb21:49
*** ametts has quit IRC21:56
openstackgerritMichael Dong proposed openstack/syntribos: Fixed string check KeyError  https://review.openstack.org/34972322:12
*** sdake has joined #openstack-security22:13
unrahul:D whoa!22:13
unrahulccneill: yup saw that patch i was like.. i bet it was 100 lines less a minute ago..22:14
ccneill<_<22:24
ccneilladded a *bunch* of unit tests22:24
ccneilllol22:24
ccneillalso needed to make some changes to the RequestHelperMixin to handle URLs properly22:25
unrahulyup!, those unittests are one of the most complete ones we have.. I guess we would cross 58% overall with it..22:26
ccneillyeah.. we can probably do without 100% coverage, but I want us to get to at least 80-90%22:29
ccneillsome day22:29
unrahul*some day* :D22:30
ccneillyeah..22:30
ccneilllittle by little22:30
ccneillwe've gone from 8% to 58% pretty quickly :)22:30
openstackgerritMichael Dong proposed openstack/syntribos: Fixed string check KeyError  https://review.openstack.org/34972322:33
ccneillmdong: good catch! can't believe we missed that :(22:36
mdongyeah… added a unittest this time22:37
ccneillawesome22:37
*** davidjd-gh has joined #openstack-security22:37
*** davidjd-gh has left #openstack-security22:38
unrahulanother ++ to overall coverage22:38
ccneillunrahul, mdong: realized we haven't been naming unit test files for checks consistently22:42
ccneilldidn't think this was the CR to try and fix it in though22:43
ccneillbut we have one ("test_http_checks.py) that makes it explicit, and the rest don't22:43
ccneill¯\_(ツ)_/¯22:43
ccneillmaybe we make a tests/unit/checks folder at some point.. it doesn't matter much, just slightly more intuitive if you're looking for those particular tests22:44
mdongyeah, I noticed that. I think I prefer to have it explicit just because we have, for example, checks/ssl.py and tests/transport_layer/ssl.py22:44
ccneillright22:45
mdongwe can probably revisit this when we do unittests for the tests themselves22:45
unrahulyup.. we need to do something like that, things are confusing as it is.. no need for unit tests to add to that22:45
ccneillrenaming means we don't have to make a "tests/unit/tests" folder lol22:45
openstackgerritRahul U Nair proposed openstack/syntribos: Fixed bug RequestObject has no attribute `fuzz_request`  https://review.openstack.org/34973122:47
unrahulccneill: mdong syntribos was breaking for me.. because http parser was returning a new RequestObject each time (i guess),22:48
unrahulnot sure if you guys noticed..22:48
unrahulcould you guys check the CR i just posted..?22:49
ccneillunrahul: hmm, we should never call request.fuzz_request() though22:54
ccneilland there are no other "request_model_type"s22:54
ccneillthere's only RequestObject22:54
ccneillmaybe an outdated version of syntribos? I don't see any issues22:54
ccneilltry a "pip uninstall syntribos && pip install --upgrade -e ."22:55
ccneillgrep -r '\.fuzz_request' **/*.py22:55
ccneillsyntribos/tests/fuzz/base_fuzz.py:        fr = syntribos.tests.fuzz.datagen.fuzz_request(22:56
ccneillsyntribos/tests/fuzz/xml_external.py:            fr = syntribos.tests.fuzz.datagen.fuzz_request(22:56
ccneilltests/unit/test_datagen.py:            d for d in fuzz_datagen.fuzz_request(req, strings, "url", "ut")22:56
ccneilltests/unit/test_datagen.py:            d for d in fuzz_datagen.fuzz_request(req, strings, "data", "ut")22:56
ccneilltests/unit/test_datagen.py:                fuzz_datagen.fuzz_request(req, strings, "params", "ut"), 1):22:56
ccneillmake sure you pull too, since that parser change landed a little while ago22:56
ccneillargh, looks like Jenkins is overloaded again..22:59
unrahuldid that.. still having the same issue, AttributeError: 'RequestObject' object has no attribute 'fuzz_request'23:00
unrahul:/23:00
ccneillweird.. what file/line?23:00
unrahulsyntribos/syntribos/tests/fuzz/base_fuzz.py", line 121, in get_test_cases23:00
unrahulfr = xls.init_req.fuzz_request(23:01
unrahulcls*23:01
ccneilluhhhh23:01
ccneillhmmm...23:01
ccneillv_v23:01
*** sdake has quit IRC23:01
ccneillstrange...23:01
ccneillOH23:01
unrahulhttps://github.com/openstack/syntribos/blob/master/syntribos/tests/fuzz/base_fuzz.py#L12123:02
ccneillhttps://review.openstack.org/#/c/347116/7/syntribos/tests/fuzz/base_fuzz.py23:02
ccneillit's part of that giant CR :X23:02
ccneillI should've made the parser change dependent on the fuzzrequest change23:02
unrahul:D23:02
* ccneill 's bad23:02
unrahulso ryt.. now.. the master it breaking ..23:02
ccneillargh :(23:03
unrahulphew.. that was a weird side effect23:03
ccneillI haven't seen it failing yet..23:03
unrahulFor us it is breaking.. :/23:03
ccneillbut it should fail23:03
ccneillright23:03
ccneilljust not sure why I haven't seen it yet o_O23:03
*** sdake has joined #openstack-security23:03
unrahulcould you download the master and see if it is breaking.. ??23:04
unrahulyeah it should ryt..23:04
ccneillI mean.. it *should*, I think.. but I'm getting 0 errors O_o23:04
ccneillhmm23:04
unrahulwhoa..23:05
ccneilldoing an uninstall/reinstall..23:05
ccneillAH!23:05
ccneillthere we go23:05
ccneilllol23:05
unrahulhehe23:06
unrahul:D23:06
unrahulshould we merge the patch i had uploaded for now, as master is breaking?23:06
ccneillok, well.. I guess we should merge this to un-break master, and then when we get the bigger CR merged, it will revise it23:06
unrahulyup..23:06
unrahulthanks ccneill23:07
unrahulwas driving me crazy23:07
ccneillunrahul: np, +2'd23:07
ccneillhopefully Jenkins calms down a little now that it's the end of the work day...23:08
unrahulyeah,,23:08
ccneilland we get a +V soon23:08
*** elmiko is now known as _elmiko23:09
ccneillgonna head out, gotta go buy a new power cord for my personal laptop.. see y'all tomorrow o/23:10
unrahulyup c u ccneill !23:11
openstackgerritAastha Dixit proposed openstack/syntribos: Adding additional fields to debug log  https://review.openstack.org/34708923:21
*** davidjd-gh has joined #openstack-security23:29
*** davidjd-gh has left #openstack-security23:29
*** sdake has quit IRC23:31
*** edmondsw has quit IRC23:40
*** sdake has joined #openstack-security23:40
*** mdong has quit IRC23:46
*** ccneill has quit IRC23:46

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!