Wednesday, 2015-03-04

*** nkinder has joined #openstack-security00:09
*** fletcher has joined #openstack-security00:13
*** pdesai has quit IRC00:16
*** voodookid has quit IRC00:18
*** tmcpeak has joined #openstack-security00:18
*** tmcpeak has quit IRC00:20
*** markvoelker has quit IRC00:26
*** pdesai has joined #openstack-security00:28
*** pdesai has quit IRC00:28
*** bdpayne has quit IRC00:34
*** tmcpeak has joined #openstack-security00:40
*** tmcpeak has quit IRC00:42
*** ljfisher has quit IRC00:56
*** markvoelker has joined #openstack-security01:09
*** markvoelker has quit IRC01:14
*** salv-orlando has quit IRC01:20
*** jamielennox is now known as jamielennox|away01:59
*** browne has quit IRC02:08
*** bpokorny_ has quit IRC02:09
*** bpokorny has joined #openstack-security02:09
*** markvoelker has joined #openstack-security02:10
*** bpokorny has quit IRC02:14
*** markvoelker has quit IRC02:15
*** salv-orlando has joined #openstack-security02:20
*** markvoelker has joined #openstack-security02:27
*** markvoelker has quit IRC02:33
*** jamielennox|away is now known as jamielennox02:37
*** jamielennox is now known as jamielennox|away02:52
*** jamielennox|away is now known as jamielennox02:55
*** jamielennox is now known as jamielennox|away02:56
*** jamielennox|away is now known as jamielennox02:57
*** fletcher has quit IRC03:05
*** amrith is now known as _amrith_03:11
*** browne has joined #openstack-security03:12
*** salv-orlando has quit IRC03:21
*** markvoelker has joined #openstack-security03:28
*** _amrith_ is now known as amrith03:28
*** markvoelker has quit IRC03:33
*** markvoelker has joined #openstack-security04:29
*** mgagne has quit IRC04:31
*** markvoelker has quit IRC04:34
*** mgagne has joined #openstack-security04:45
*** mgagne is now known as Guest7571104:45
*** openstackgerrit has quit IRC04:46
*** openstackgerrit has joined #openstack-security04:52
*** markvoelker has joined #openstack-security05:31
*** jamielennox is now known as jamielennox|away05:32
*** openstack has joined #openstack-security05:36 [freenode-info] channel trolls and no channel staff around to help? please check with freenode support:
*** markvoelker has quit IRC05:36
*** salv-orlando has joined #openstack-security05:38
openstackgerritLeon Zachery proposed openstack/security-doc: Add reference links to Openstack Security Guide
*** salv-orlando has quit IRC06:29
*** markvoelker has joined #openstack-security06:32
*** markvoelker has quit IRC06:38
*** markvoelker has joined #openstack-security07:34
*** markvoelker has quit IRC07:39
*** openstackgerrit has quit IRC07:49
*** openstackgerrit has joined #openstack-security07:49
*** salv-orlando has joined #openstack-security08:15
*** markvoelker has joined #openstack-security08:35
*** markvoelker has quit IRC08:39
*** salv-orlando has quit IRC08:45
*** salv-orlando has joined #openstack-security08:45
*** openstack has joined #openstack-security15:28
*** amrith is now known as _amrith_15:44
*** dwyde has joined #openstack-security15:54
openstackgerritRobert Clark proposed stackforge/anchor: Multiple changes so that Anchor runs correctly
openstackgerritTim Kelsey proposed stackforge/anchor: Fixing several issues in Anchor startup
*** _amrith_ is now known as amrith16:44
*** bpokorny has joined #openstack-security16:45
*** tmcpeak has quit IRC17:06
*** amrith is now known as _amrith_17:12
*** tmcpeak has joined #openstack-security17:16
*** fletcher has joined #openstack-security17:25
fletchertmcpeak: heyo, around?17:26
tmcpeakfletcher: was just looking for you17:29
tmcpeakyou have your Bandit instance handy?17:29
fletcherboom, unfortunately I have to bounce in about 30 minutes but I think that should be enough time17:29
fletcherI do, yah17:29
tmcpeakyeah, that will be plenty17:29
tmcpeakcan you clone this, run Bandit against it and read the results:
*** edmondsw has quit IRC17:30
*** ukbelch has joined #openstack-security17:30
ukbelchhullo thar17:30
fletcherwith master I assume?17:30
tmcpeakukbelch: you'll be interested in this as well :)17:30
tmcpeakshouldn't matter17:31
ukbelchthat's why im here!17:31
tmcpeakbut for carity lett's use belch's checkin17:31
tmcpeakactually yeah, use Belch's check-in so we're all dealing with the same thing17:31
ukbelchwhat's been discussed already17:31
fletcherwell master has full line ranges right now, was the only thing Iw as thinking17:31
fletchernot much, we just started ukbelch17:31
tmcpeakukbelch: nothing, just asked him to clone Trove and run17:31
tmcpeakok, anything with full line ranges should be fine17:32
ukbelchok, so what's the use case for needing all line numbers for a given statement output?17:32
*** rkgudboy has joined #openstack-security17:32
tmcpeakwell before we get to that, I wanted to show the issue we're having with it17:33
fletcherok, it's running now17:33
ukbelchmenewhile, I can fix the json output issue, I figured there may be one there (hence the specific comment asking for the check :))17:33
fletcherThe use case is comparing against diffs in CI17:33
tmcpeakinteresting- ukbelch: I'm not seeing that issue on master17:34
fletcherSo for example, the example file I put in the comment17:34
tmcpeakis this something new your check-in introduced?17:34
fletcherthe popen spans 5,6,1017:34
tmcpeakok let me pull that up17:34
fletcherif somebody checks in something on line 10, that changes shell=True, i want to be able to catch that17:34
ukbelchyes, I refactored how the reporting worked to remove tuples17:34
fletcherjust having '5' doesn't help me17:34
tmcpeakright, yeah, I agree17:35
ukbelchahh, I get you17:35
tmcpeakin that case 5 isn't accurate, the issue spans the three lines17:35
fletcherright, yah17:35
tmcpeakpopen by itself isn't an issue17:35
ukbelchso, the issue there is because it's not the line itself getting checked, it's the function call17:35
tmcpeakukbelch: I'm not seeing that big ugly thing in trove output with amster17:36
ukbelchwhen the visitor functionality sees the popen, the tester is called, which checks the parameters being passed17:36
ukbelchso it records the last line visited before the test is called, which is the popen line17:36
tmcpeakyeah, current master output is more accurate though in that it shows all 3 lines17:37
fletcherIs there a specific file in this trove repo we can focus on that has the issue you guys were worried about? I'm getting a ton of output :/17:37
tmcpeakyeah, hang on, I'll dig it up17:37
ukbelchit would require a modification of the test itself to change that behaviour, sadly17:37
fletcherI really think the original behavior of line ranges was more accurate17:37
ukbelchthe code output should show all three lines17:37
fletcherbut i'll run it against this file and see what master outputs17:37
*** rkgudboy has quit IRC17:38
ukbelchwhat I do notice is that the json output doesn't show line numbers in it's code output17:38
fletcherYah, that was purposeful but I'm also not opposed to adding themn17:39
fletcherso i see the ouput from cfg.py17:39
ukbelchwhich means there's nothing to show all lines touched by that statement (especially as the newlines are compressed)17:39
fletcherI don't see any long line number lists17:39
tmcpeakyeah, ukbelch: this new output artifact seems to be related to your check in, old line number solution doesn't suffer from that17:40
tmcpeaksadly this new statement processing logic is introducing this large statement issue17:41
ukbelchlooking at your example output fletcher. it's only showing one newline between the first and second parameter to Popen17:41
tmcpeakfletcher: git fetch his patch and run against Trove and you'll see the issue17:41
ukbelchof course it is travis, that's the point :)17:41
tmcpeakyeah, old output was better though :/17:42
ukbelchbut yeah, need to pull my patch to see the extended output17:42
fletcherhmm, ukbelch, i think its just hiding in the center17:42
fletcher"subprocess.Popen([ 'ls',\n'-l'],\n        shell=True)\n"17:42
fletcheri see 3 \n17:42
ukbelchyes, but if the lines are 5, 6 and 10, shouldn't there be 5 \n?17:42
fletcherNo, it doesn't include blank lines17:43
fletcherother wise line range would be '5,6,7,8,9,10'17:43
ukbelchwhich I agree with, if it displayes line numbers for all code output17:43
ukbelchotherwise, actually mapping that to diffs or the actual code file could get confusing17:43
ukbelchbut, that's a small issue and a matter of taste17:44
fletcherhmm, so I'm totally fine with prepending the lines of code with the line number17:44
tmcpeakyeah, so fletcher: with the JSON diff thing, if somebody adds a comment, all line numbers are going to get shifted17:44
tmcpeakthis is a known problem, right?17:44
ukbelchit's easy enough to include the full line range in the json line_nums output17:44
fletcherI don't think it makes mapping to diffs harder. You just get "line touched" from the diff and do a "is this in line_range_list" of vuln17:44
ukbelchif that's the desired output17:44
*** browne has quit IRC17:45
tmcpeakI think I'm somewhat confused but that's par for the course :D17:45
tmcpeakso belch, this latest check-in with statement processing introduces a new problem with large statements we didn't used to have17:45
fletchertmcpeak: Yah, it's just getting line numbers from the AST17:45
fletcher(obviously, not being pedantic)17:45
ukbelchwhat new problem travis?17:46
tmcpeakwith the disgusting long statement output in Trove17:46
ukbelchgive me a sec to run it again with master, but ultimately it's rather fixing a problem in master, in that it's not printing the full context of some statements17:47
tmcpeakukbelch: can you provide an example of that please?17:47
ukbelchthe big messy output :)17:47
tmcpeakwell I'd argue we don't want that17:48
tmcpeakcan you show an example where the master output is less desirable than the new output?17:48
ukbelchgive me a second17:48
ukbelchI should be able to find a few17:49
ukbelchthe only case it currently works perfectly for, is function calls17:49
ukbelch>> Probable insecure usage of temp file/directory - ../trove/trove/tests/unittests/guestagent/[125]17:49
ukbelch125        output = "mysqld would've been started with the these args:\n"\17:49
ukbelchincorrect, as the issue isn't on that specific line, there's no temp directory here17:49
ukbelch>> Probable insecure usage of temp file/directory - ../trove/trove/tests/unittests/guestagent/[364]17:50
ukbelch364                restore_location='/tmp/somewhere')17:50
tmcpeaksure, doesn't include the full context17:50
ukbelchincorrect, as we don't know what the rest of the statement preceeding line 364 is17:50
ukbelch>> Probable insecure usage of temp file/directory - ../trove/trove/tests/unittests/guestagent/[295]17:50
ukbelch295        restr = RunnerClass(None, restore_location="/tmp",17:50
ukbelchincorrect, it's missing the rest of the parameters going into RunnerClass17:50
ukbelch>> Probable insecure usage of temp file/directory - ../trove/trove/tests/unittests/guestagent/[69]17:51
ukbelch69PREPARE = ("sudo innobackupex --apply-log /var/lib/mysql "17:51
ukbelchincorrect - isn't showing the temp directory part of the statement at issue17:51
ukbelchshoudl I go on?17:51
fletcherukbelch: looking at files now to gain context17:52
ukbelchthe whole point is, we had to add a fixed "n" parameter to output more lines, so we could see all the context of statements spread over multiple lines. The fix changes that need.17:52
ukbelchspitting out the entire mess of crappy config code is, as I see it, intended behaviour. Specifying the actual line number affected, thus being able to see which line in that statement is at fault, is therefore necessary17:53
ukbelchthe question is, if we can modify the tests so it outputs the correct line in the case of function calls17:54
tmcpeakis there any way to have the best of both worlds?  Since function tests output worked perfectly, leave the output as is for that, but in other contexts (like string) output the whole statement?17:54
ukbelchno, we can't really17:54
ukbelchthat's not how the reporting works. We can't pick and choose based on the statment type17:55
ukbelchit would be a minefield17:55
fletcherIn the end, more information is almost always better17:55
tmcpeakyeah.. I also agree that the arbitrary guess n lines around context solution was hacky17:55
fletcherBut all that did was print code17:55
tmcpeakfletcher: yeah, the problem is that in Trove they define this like 60 line list17:55
fletcherit didn't add those lines into any of the logic17:55
tmcpeakso the output is ridiculous17:55
ukbelchso, we can look to resolve two things. We can fix up the specific tests, and we can decide exactly what we want in the json output17:55
fletcherright, that seems like a bit of a corner case17:55
tmcpeakand the current result shows the problem is in all of those lines17:55
fletcherAh ok, so I get it17:56
tmcpeakfletcher: it is a corner case, but I'm concerned with the output being inaccurate17:56
tmcpeakit's saying there is an issue with all of those lines, when there really isn't17:56
ukbelchexactly, that config crap is very much a corner case17:56
ukbelchand it's entirely accurate17:56
fletcherA developer has to figure out which of 60 lines is causing teh erorr17:56
ukbelchyour problem is, it's verbose17:56
ukbelchthat's why we output the specific line number Fletcher :)17:57
tmcpeakwell, we could do statement line and issue line17:57
fletcherso can we have two fields17:57
tmcpeakoutput both17:57
ukbelchhe sees a block of code, see's line xx is the issue, and because we print line numbers they can hop straight to it17:57
fletcher"exact line number" "context range"17:57
fletcheror something17:57
tmcpeakyeah, that seems like it makes both cases happy17:57
fletcherright, which is fine for manul case17:57
fletcherbut problematic for automating17:57
ukbelchwe can have that for json, sure, if that's what we want. That's easy17:57
fletcherit's pointless to have json output if I have to reparse a string to pull out line numnbers17:58
tmcpeakno, just have them stored as a separate field as you said17:58
tmcpeakignore the field you don't care about17:58
fletcheryah, i agere with that17:58
*** ukbelch_ has joined #openstack-security17:58
ukbelch_for terminal output, outputting a huge list of numbers is entirely useless17:58
ukbelch_and confusing/distracting17:59
tmcpeakI also agree with that17:59
fletcherthat interests me, does that mean the Shell=True test will actually flag line 10 and return 5,6,10 - that would be awesome17:59
fletcheri agree with that terminal output17:59
ukbelch_potentially. I need to check if changing the context object within a test will affect the test output, but if it does it just means fixing the line number within the test itself18:00
ukbelch_right now we have two line related objects in the context. We have context['lineno'] and context['statement']['lineno']18:01
ukbelch_sorry, thats context['statement']['linerange']18:01
fletcherI've unfortunately got to leave for a few hours, but I'll be back on in a bit to discuss more. If we can ensure JSON output has [5,6,10] list for the pOpen warning I provided, I'll be happy :) fwiw18:01
tmcpeakok cool18:01
tmcpeaklater fletcher18:01
ukbelch_we can do that as an additional parameter, or replace the single number18:01
ukbelch_what would be your preference?18:01
tmcpeakadditional param is my preference18:02
tmcpeakadditional param which doesn't get output on terminal18:02
*** ukbelch has quit IRC18:02
ukbelch_completely separate things :)18:02
ukbelch_we store both already, and both are used in the reporting already (the range is used to pull the code lines, the specific number output to highlight the issue)18:03
ukbelch_it's just a case of sticking both in the json output :)18:03
tmcpeakok cool18:03
ukbelch_it's fletcher's use case, so I'll add it as an extra parameter for now, and he can comment when he gets back18:04
tmcpeaksounds good18:05
*** bdpayne has joined #openstack-security18:08
*** ukbelch_ has quit IRC18:11
*** dwyde has quit IRC18:11
*** ukbelch has joined #openstack-security18:11
*** ukbelch has quit IRC18:13
*** ukbelch has joined #openstack-security18:16
*** browne has joined #openstack-security18:17
*** logan2 has quit IRC18:18
*** ukbelch has quit IRC18:21
*** ukbelch has joined #openstack-security18:21
*** pdesai has joined #openstack-security18:45
*** pdesai has quit IRC18:54
*** _amrith_ is now known as amrith19:07
*** pdesai has joined #openstack-security19:32
*** pdesai has quit IRC19:35
*** dwyde has joined #openstack-security19:41
*** dave-mccowan has joined #openstack-security20:01
openstackgerritbruce-benjamin proposed openstack/security-doc: Added input about volume encryption feature
*** pdesai has joined #openstack-security20:21
openstackgerritbruce-benjamin proposed openstack/security-doc: Added input about volume encryption feature
openstackgerritDave Belcher proposed stackforge/bandit: Refactored AST processing
*** pdesai has left #openstack-security20:46
*** amrith is now known as _amrith_21:11
*** ukbelch has quit IRC21:22
*** ukbelch has joined #openstack-security21:42
*** ukbelch has quit IRC21:44
*** ukbelch has joined #openstack-security21:47
*** tmcpeak has quit IRC22:00
*** tmcpeak has joined #openstack-security22:13
*** jamielennox|away is now known as jamielennox22:14
*** tmcpeak has quit IRC22:15
*** _amrith_ is now known as amrith22:22
*** micael has joined #openstack-security22:23
*** micael has quit IRC22:26
*** ukbelch has quit IRC22:26
*** ukbelch has joined #openstack-security22:26
*** dwyde has quit IRC22:36
*** dwyde has joined #openstack-security22:50
*** bknudson has quit IRC22:50
*** browne has quit IRC22:55
*** browne has joined #openstack-security22:58
*** ljfisher has quit IRC23:21
*** ukbelch has quit IRC23:36
*** bknudson has joined #openstack-security23:52
openstackgerritDavid Wyde proposed stackforge/bandit: Refactor functional tests to clarify scoring

Generated by 2.14.0 by Marius Gedminas - find it at!