Thursday, 2015-08-06

*** hyakuhei has joined #openstack-security00:00
*** hyakuhei1 has quit IRC00:02
*** TimB28 has joined #openstack-security00:20
*** TimB28 has left #openstack-security00:21
*** elmiko has quit IRC00:29
*** curious_george has joined #openstack-security00:35
*** pdesai has quit IRC00:35
*** curious_george has quit IRC00:37
*** pdesai has joined #openstack-security00:58
*** pdesai has quit IRC01:02
*** browne has quit IRC01:03
*** pdesai has joined #openstack-security01:04
openstackgerritMerged openstack/bandit: Add info: License, Source, Bugs and Docs to README
*** pdesai has quit IRC01:08
*** sdake has joined #openstack-security01:27
*** yuanying has joined #openstack-security01:30
*** sdake_ has joined #openstack-security01:30
*** sdake has quit IRC01:34
*** pdesai has joined #openstack-security01:59
*** sdake_ has quit IRC02:09
*** markvoelker has quit IRC02:11
*** markvoelker has joined #openstack-security02:12
*** viraptor has joined #openstack-security02:40
*** dave-mccowan has joined #openstack-security02:40
*** browne has joined #openstack-security03:01
*** markvoelker has quit IRC03:04
*** pdesai has quit IRC03:26
*** pdesai has joined #openstack-security03:26
*** pdesai has quit IRC03:27
*** bpokorny has quit IRC03:34
*** dave-mccowan has quit IRC03:42
*** tmcpeak has quit IRC03:47
*** markvoelker has joined #openstack-security04:04
*** markvoelker has quit IRC04:09
openstackgerritStanislaw Pitucha proposed openstack/anchor: Implement new API format
openstackgerritStanislaw Pitucha proposed openstack/anchor: Move all plugins to stevedore
openstackgerritStanislaw Pitucha proposed openstack/anchor: Allow configurable signing backends
*** scorpion_17 has joined #openstack-security05:15
*** viraptor has quit IRC05:16
*** scorpion_17 has quit IRC05:19
*** scorpion_17 has joined #openstack-security05:19
*** scorpion_17 has quit IRC05:38
*** markvoelker has joined #openstack-security05:51
*** markvoelker has quit IRC05:56
*** hyakuhei_ has joined #openstack-security06:49
*** shohel has joined #openstack-security06:51
*** hyakuhei_ has quit IRC06:54
*** hyakuhei_ has joined #openstack-security06:59
*** hyakuhei_ has quit IRC07:05
*** hyakuhei_ has joined #openstack-security07:06
*** browne has quit IRC07:27
openstackgerritOpenStack Proposal Bot proposed openstack/security-doc: Updated from openstack-manuals
*** b10n1k has quit IRC07:37
*** hyakuhei_ has quit IRC07:42
*** alex_klimov has joined #openstack-security07:45
*** elo1 has joined #openstack-security07:48
*** elo has quit IRC07:50
*** markvoelker has joined #openstack-security07:52
*** elo1 has quit IRC07:53
*** sdake has joined #openstack-security07:54
*** elo has joined #openstack-security07:55
*** markvoelker has quit IRC07:56
*** alex_klimov has quit IRC07:57
*** elo has quit IRC08:06
*** elo has joined #openstack-security08:09
*** tkelsey has joined #openstack-security08:13
*** hyakuhei_ has joined #openstack-security08:15
*** alex_klimov has joined #openstack-security08:18
openstackgerritMerged openstack/security-doc: Updated from openstack-manuals
*** hyakuhei_ has quit IRC08:28
*** hyakuhei_ has joined #openstack-security09:15
*** hyakuhei_ has quit IRC09:18
*** hyakuhei_ has joined #openstack-security09:21
openstackgerritOpenStack Proposal Bot proposed openstack/security-doc: Updated from openstack-manuals
*** markvoelker has joined #openstack-security09:53
openstackgerritMerged openstack/security-doc: Updated from openstack-manuals
*** markvoelker has quit IRC09:57
*** hyakuhei_ has quit IRC10:08
*** yuanying has quit IRC10:09
*** hyakuhei_ has joined #openstack-security10:09
*** hyakuhei_ has quit IRC10:10
*** hyakuhei_ has joined #openstack-security10:24
*** hyakuhei_ has quit IRC10:25
openstackgerritStanislaw Pitucha proposed openstack/anchor: Stop mixing IPs and domains
*** markvoelker has joined #openstack-security10:54
*** markvoelker has quit IRC10:58
*** shohel has quit IRC11:03
*** shohel1 has joined #openstack-security11:03
*** sdake has quit IRC11:07
*** tjt263 has joined #openstack-security11:08
*** tjt263 has quit IRC11:16
*** shohel1 has quit IRC11:19
*** shohel has joined #openstack-security11:19
*** yuanying has joined #openstack-security11:24
*** tjt263 has joined #openstack-security11:24
*** yuanying has quit IRC11:29
openstackgerritMerged openstack/anchor: Simplify the tests
*** timkennedy has quit IRC11:59
*** timkennedy has joined #openstack-security11:59
*** markvoelker has joined #openstack-security12:16
*** edmondsw has joined #openstack-security12:36
*** zul has joined #openstack-security12:46
*** tmcpeak has joined #openstack-security12:49
*** elmiko has joined #openstack-security12:53
*** elmiko has quit IRC13:00
*** elmiko has joined #openstack-security13:00
*** dave-mccowan has joined #openstack-security13:01
*** dave-mcc_ has joined #openstack-security13:04
*** dave-mccowan has quit IRC13:07
*** yuanying has joined #openstack-security13:12
*** browne has joined #openstack-security13:15
*** yuanying has quit IRC13:17
*** singlethink has joined #openstack-security13:32
*** jmckind has joined #openstack-security13:34
*** bknudson has joined #openstack-security13:44
*** edmondsw has quit IRC13:46
*** sigmavirus24_awa is now known as sigmavirus2413:55
*** sdake has joined #openstack-security13:55
tmcpeaksigmavirus24: can you repro this:
openstackLaunchpad bug 1481922 in Bandit "'NoneType' object has no attribute '__getitem__'" [High,In progress] - Assigned to Sean McGinnis (sean-mcginnis)13:59
sigmavirus24Haven't tried yet13:59
tmcpeakbrowne: you?13:59
sigmavirus24I have suspicions though and I'm very very doubtful that changing how we use the decorator will actually fix that13:59
tmcpeaksmcginnis: interesting bug :)13:59
tmcpeaksigmavirus24: yeah, I'm not seeing how that helps either TBH13:59
tmcpeakthat decorator style is uniformly used elsewhere, right?14:00
tmcpeakhmm, actually it is not14:00
sigmavirus24Every other project in openstack uses `import module; ... @module.decorator\ndef foo():\n    pass'14:00
tmcpeakthat's the way it''s normally done ^14:01
sigmavirus24tmcpeak: right because we're not using the hacking check to ensure we only ever import modules14:01
tmcpeakfor the rest of Bandit at least14:01
sigmavirus24Using a module in an import shouldn't be a problem14:01
tmcpeakyeah, I agree it shouldn't be14:01
tmcpeakif I could repro this I'd go in with a debugger but I can't14:01
tmcpeakso yeah, that's the question of the day, why can't I14:02
brownei'll try to reproduce today.  but don't think i've seen that bug before14:03
tmcpeakeven pip install bandit in a fresh venv I can't repro14:04
*** jmckind has quit IRC14:05
sigmavirus24I'm cloning cinder to try it14:07
sigmavirus24So our "takes_config" decorator needs some good comments14:08
sigmavirus24I spent an hour last night on mental gymnastics udnerstanding it14:08
*** jmckind has joined #openstack-security14:08
sigmavirus24(I was on my phone at a friend's house ignoring a boring conversation)14:08
tmcpeakahh yeah, it's probably never been touched since I wrote it in September14:09
tmcpeaksigmavirus24: yeah, you're right, that's pretty sketch14:11
sigmavirus24I mean14:11
sigmavirus24I figured out what the branching was doing14:11
sigmavirus24but I took a lot of looking at other (less obvious) uses of takes_config14:11
sigmavirus24and to be fair, while we can use @takes_config or @takes_config('config_section') I think they should be separate decorators for simplicity's sake14:12
tmcpeakthat would be easier to understand14:12
*** sdake has quit IRC14:14
*** voodookid has joined #openstack-security14:19
tmcpeaksmcginnis: you around?14:22
tmcpeakI'd like to figure out how our envs are different :)14:23
*** voodookid has quit IRC14:24
*** edmondsw has joined #openstack-security14:28
sigmavirus24tmcpeak: it shouldn't ever be possible for _config to be None, right?14:32
tmcpeaksigamvirus24: no, it comes from bandit config file14:34
tmcpeakunless you can't find the config file, but Bandit goes nuts if that happens14:34
*** voodookid has joined #openstack-security14:39
*** yaya has joined #openstack-security14:50
*** yaya has quit IRC14:59
*** shohel has quit IRC15:00
*** yuanying has joined #openstack-security15:00
*** yuanying has quit IRC15:05
smcginnistmcpeak: Here now.15:05
smcginnistmcpeak: Yeah, weird one.15:05
smcginnistmcpeak: No idea why what I proposed in the patch makes any difference, but it does.15:06
sigmavirus24smcginnis: what version of bandit on what os?15:06
smcginnissigmavirus24: Looks like it's bandit 0.13.0 running on Ubuntu 14.01.15:07
sigmavirus2414.01 or 14.04?15:07
smcginnissigmavirus24: Sorry, you are right. Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-45-generic x86_64)15:07
sigmavirus24hmm I can repro this on OSX too15:07
smcginnissigmavirus24: Good, it's not just me. ;)15:08
smcginnissigmavirus24: Can you see if my change fixes it on OSX? Would be good to see if that's different.15:08
sigmavirus24So self.config.get_option(test._takes_config) is returning None15:10
sigmavirus24and test._takes_config is corrrect it seems15:10
*** bpokorny has joined #openstack-security15:10
*** zul has quit IRC15:10
*** yaya has joined #openstack-security15:23
sigmavirus24so smcginnis, tox -e bandit in cinder uses tools/bandit.yaml15:26
sigmavirus24Which has no try_except_pass section15:26
sigmavirus24Because it has no such section we return None from self.config.get_option('try_except_pass')15:26
sigmavirus24when we return None, we pass that into the try_except_pass function15:26
sigmavirus24I haven't tested your patch yet, but I see no reason why it would change this behaviour15:27
smcginnissigmavirus24: I don't know why it makes a difference, but it does.15:27
smcginnissigmavirus24: So any time a plugin is added to bandit, we will need to update cinder's bandit.yaml for each one?15:27
*** sdake has joined #openstack-security15:31
tmcpeaksmcginnis: yeah, that's a downside to our current setup.  One of our planned changes involves moving module config into separate configs which will be more static15:31
smcginnistmcpeak: That will be good.15:32
tmcpeakalthough when did we merge try pass15:32
tmcpeakCinder config must be very old15:32
smcginnistmcpeak: The patch to add it was out there for a while.15:32
sigmavirus24I would argue that we can handle this appropriately15:32
sigmavirus24And I have an idea15:33
sigmavirus24If something takes config and there is no config15:33
sigmavirus24We shouldn't run it15:33
sigmavirus24Or we should take some kind of default config15:33
tmcpeaksigmavirus24: yeah, that sounds pretty damn sensible :)15:33
smcginnisI know there were other cores that had some reservations with adding bandit at this point.15:33
smcginnisIf there is going to be an ongoing need to track updates I think there will be even more pushback.15:33
tmcpeaksmcginnis: it should be fine, you'll still detect lots of issues15:34
tmcpeakthis particular plugin won't run15:34
smcginnisSo anything we can do to make that not necessary will be goodness.15:34
smcginnistmcpeak: True15:34
smcginnisJust doesn't look good if it starts off with a long list of exceptions. ;)15:34
tmcpeakyeah, I think sigmavirus24's solution is good here15:34
sigmavirus24that said15:34
tmcpeaksigmavirus24: if you get that patch up I'll push a new Bandit with it today15:34
sigmavirus24Do we report the fact that we're skipping a test somehow?15:35
tmcpeaksigmavirus24: yeah, I'm sure I saw that elsewhere.  I think one simple warning up front is sufficient15:35
sigmavirus24Also this code is getting super complex around this stuff =/15:36
tmcpeakwhich code?15:36
tmcpeaksmcginnis: AFAIK Cinder isn't even running any Bandit gate yet, let alone a voting one15:37
sigmavirus24tmcpeak: the code in bandit.core.tester15:37
sigmavirus24I'll be picking this apart for a refactor later15:37
smcginnistmcpeak: Correct.15:37
sigmavirus24Now is not the time15:37
tmcpeaksigmavirus24: sounds good15:38
smcginnistmcpeak: We just merged support for running it with tox.15:38
*** singlethink has quit IRC15:38
smcginnisNot in gate, just on demand for whoever is interested.15:38
tmcpeakI think there are several issues here:   1) new Bandit and Cinder Bandit merged at roughly the same time, so new Bandit wasn't tested with Cinder   2) as sigmavirus24 mentions, our current procedure for dealing with missing config is stupid   3) we need better automation in Bandit to make sure that the projects that are consuming it aren't broken in some way with new Bandit versions15:39
sigmavirus24Agreed with tmcpeak15:40
sigmavirus24I really want to set-up reverse gates for projects that are consuming bandit15:40
tmcpeak1) is a fluke,  2) sigmavirus24 is going to work on right now,  3) is something I'm going to move up to the top of the stack15:40
sigmavirus24Just confirmed that my fix works15:40
tmcpeaksigmavirus24: great, yeah, that makes sense15:41
sigmavirus24smcginnis: what was that bug number?15:41
sigmavirus24nevermind found15:41
openstackgerritIan Cordasco proposed openstack/bandit: Skip a test if it requires config but none is found
sigmavirus24oh I forgot to add a warning to it15:42
*** dwyde has joined #openstack-security15:45
openstackgerritIan Cordasco proposed openstack/bandit: Skip a test if it requires config but none is found
sigmavirus24tmcpeak: bknudson Daviey ^15:55
sigmavirus24Also we should make a bug to send all warnings.warn's to the logs15:55
bknudsonsigmavirus24: do all tests have config?15:56
sigmavirus24bknudson: not really15:56
sigmavirus24some tests  takea config but the config may not config them15:56
tmcpeaksigmavirus24: awesome, thank you15:57
tmcpeaklooking now15:57
*** yaya has quit IRC16:00
tmcpeaksigmavirus24: looks good, is there a simple way of doing a unit test?16:02
sigmavirus24not at the moment really16:03
sigmavirus24I can imagine a way to test it16:03
sigmavirus24But it won't be pretty and I really don't have bandwidth for it right now16:03
sigmavirus24We could register a check that takes a bogus config name16:03
sigmavirus24And assert it isnt' called16:03
sigmavirus24And/or that warnings.warn is called16:04
sigmavirus24That said, if we refactor that entire method, testing it would be easier16:04
tmcpeakyeah ok cool16:04
*** alex_klimov has quit IRC16:04
tmcpeakI'm happy to skip for now16:04
tmcpeakthis could be part of more comprehensive unit testing later16:04
tmcpeaksigmavirus24: so you've confirmed this fixes the issue?16:04
tmcpeakbrowne, tkelsey: reviewsies?16:06
tkelseytmcpeak: I'm still devstacking like mad, but i'll take a look16:08
tmcpeakok thanks16:08
sigmavirus24tmcpeak: it fixes in tox for me16:10
sigmavirus24tmcpeak: also someone can write a test as a follow on like I described16:10
tmcpeaksigmavirus24: unit test?16:13
tmcpeakI'm incline to punt for now, it seems like a strange place to draw the line on starting to do comprehensive testing16:13
sigmavirus24tmcpeak: I agree wholeheartedly with bknudson's comment16:14
sigmavirus24Don't get me wrong16:14
sigmavirus24I'm just head's down on a product fire16:14
*** yaya has joined #openstack-security16:14
tmcpeakI agree with his comment also16:14
*** sdake has quit IRC16:14
tmcpeakif we had testing we can discover stuff like this before bug reports16:14
tmcpeaksomething else to tackle at the midcycle16:15
*** pdesai has joined #openstack-security16:19
*** pdesai has quit IRC16:22
openstackgerritMerged openstack/bandit: Skip a test if it requires config but none is found
Davieytmcpeak: stop making those of us that aren't going jelly that we are missing the midcycle.16:25
*** hyakuhei1 has joined #openstack-security16:26
*** singlethink has joined #openstack-security16:28
*** hyakuhei has quit IRC16:29
tmcpeakDaviey: oh yeah, :(16:33
sigmavirus24== Daviey16:33
tmcpeakwe'll do something less fun like implementing unit tests ;)16:33
sigmavirus24I'll be in SATX that week with my team16:33
*** jmckind has quit IRC16:39
*** malacostraca has joined #openstack-security16:39
*** malacostraca has left #openstack-security16:40
tmcpeakok cool, looks like we're live on that change16:48
tmcpeak(sigh) Bandit 0.13.1 time :(16:48
*** yuanying has joined #openstack-security16:48
*** singleth_ has joined #openstack-security16:49
*** hyakuhei has joined #openstack-security16:51
tmcpeaksigmavirus24: is this right? 291 [0.. /usr/local/lib/python2.7/site-packages/bandit/core/ UserWarning: "hardcoded_tmp_directory" has been skipped due to missing config "hardcoded_tmp_directory".16:51
tmcpeak  '"{1}".'.format(test.__name__, test._takes_config)16:51
sigmavirus24yeah that's how all warnings print16:51
sigmavirus24like I said16:52
tmcpeakoh you did? okies16:52
sigmavirus24we need code to redirect warnings.warn to logging.info16:52
sigmavirus24sorry I'm talking inf ragments right now16:52
sigmavirus24warnings.warn isn't ideal but it provides us with a guarantee that we will only see the warning once16:52
sigmavirus24rather than spamming logs16:52
sigmavirus24we just need an extra bit of code to take things from warnings.warn and put them in logs instead of stderr16:53
*** singlethink has quit IRC16:53
sigmavirus24there's example code on a urllib3 issues iirc16:53
*** yuanying has quit IRC16:53
tmcpeakI'm happy with this for now16:54
tmcpeakit's better than what we had16:54
tmcpeakbtw, now with move of tmp files to config, now older projects don't have the tmp file test anymore it seems16:55
tmcpeakbknuson ^16:55
tmcpeakbknudson even ^16:55
sigmavirus24import logging16:56
*** gmurphy has left #openstack-security16:56
tmcpeakahh ok great16:57
tmcpeakI'll add that16:57
tmcpeakthanks sigmavirus2416:57
Davieytmcpeak: No other complaints or comments from users on 0.13.0 ?16:59
tmcpeakDaviey: not that I've heard17:00
*** gmurphy has joined #openstack-security17:00
openstackgerritMerged openstack/security-doc: Adding file permissions section
openstackgerritMerged openstack/security-doc: Updating missing link in object storage section
openstackgerritMerged openstack/security-doc: Trying to add numbers and orders to commands
Davieytmcpeak / bknudson: bandit is listed here..-
tmcpeakDaviey: oh cool17:25
*** sdake has joined #openstack-security17:33
*** salv-orlando has quit IRC17:35
*** markvoelker has quit IRC17:44
*** yaya has quit IRC17:44
*** dwyde has quit IRC17:44
*** zul has joined #openstack-security17:56
hyakuheiRighto, meeting done, time for 2.5 hour drive home :)17:58
tmcpeakcool thanks hyakuhei17:58
*** hyakuhei has quit IRC17:58
browneDaviey: wait, so how to upper contraints work?  Who maintains that?17:58
brownedoes it just scan PyPi?17:58
elmikowhere is hyakuhei1 at, scotland.... ;)17:59
Davieybrowne: bot18:00
Davieybrowne: I think it daily looks for changes on pypi, then proposes them18:01
browneDaviey: ah, ok thx18:01
Davieyelmiko: I'd hate to think the speed that hyakuhei1 would have to drive at to go from Scotland to Wales in 2.5 hours!18:01
*** yaya has joined #openstack-security18:02
elmikoDaviey, it's my ignorance, the isles just never seem that big. or maybe i drive too fast lol18:03
Davieyelmiko: At least you can find us on a map... You are the 1% :)18:04
elmikoi have had awesome times visiting england and scotland. we're planning to come back next feburary18:04
elmikowe nearly planned to move there last time we visited18:04
Davieyi'm still looking for those parts.18:04
Davieyright. time to go o/18:05
*** markvoelker has joined #openstack-security18:05
elmikolater Daviey18:05
*** tkelsey has quit IRC18:22
*** yuanying has joined #openstack-security18:37
*** salv-orlando has joined #openstack-security18:37
tmcpeaksigmavirus24: you around?18:40
sigmavirus24sort of18:40
sigmavirus24what's up?18:40
tmcpeakeven with that change we get this:18:41
tmcpeak[tester]WARNING/usr/local/lib/python2.7/site-packages/bandit/core/ UserWarning: "try_except_pass" has been skipped due to missing config "try_except_pass".18:41
tmcpeak  '"{1}".'.format(test.__name__, test._takes_config)18:41
tmcpeakspecifically the .format is in there18:41
tmcpeaknot sure why18:41
tmcpeakany ideas?18:41
tmcpeakor should I googles18:41
*** yuanying has quit IRC18:41
sigmavirus24that's very weird18:41
tmcpeakyeah, my thoughts also18:42
sigmavirus24might not be able to use .format18:42
tmcpeakalso this: [general_hardcoded_password]WARNING/usr/local/lib/python2.7/site-packages/bandit/plugins/ UserWarning: Could not substitute '%(site_data_dir)s' to a path with a valid word_list file18:42
tmcpeak  warnings.warn(e.message)18:42
*** dwyde has joined #openstack-security18:44
*** salv-orlando has quit IRC18:44
tmcpeakI'll play with it18:44
*** b10n1k has joined #openstack-security18:51
*** jmckind has joined #openstack-security18:53
*** yaya has quit IRC19:05
*** dwyde has quit IRC19:05
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
tmcpeaksigmavirus24, browne: ^19:21
tmcpeaksigmavirus24: good point, I'll move it19:28
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
*** yaya has joined #openstack-security19:31
*** singleth_ has quit IRC19:32
tmcpeaksigmavirus24: back at you19:33
tmcpeakalso browne: looksies?19:33
tmcpeakI'd like to get 0.13.1 out the door before I bounce for the weekend at 4:30 mountain time19:34
tmcpeaklooks like I introduced a crapload of pep8 problems anyways19:38
*** yaya has quit IRC19:38
tmcpeaksmcginnis: what's the py3 happy way of doing it?19:39
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
*** singlethink has joined #openstack-security19:43
smcginnistmcpeak: Oh, sorry, meant to include that in the comment. val = six.text_type(message)19:46
*** salv-orlando has joined #openstack-security19:47
tmcpeaksmcginnis ok, how about the formatting approach I took?19:48
smcginnistmcpeak: I'll take a look at the latest.19:48
tmcpeaksmcginnis: great, thank you19:48
smcginnistmcpeak: Looks good!19:49
tmcpeakbrowne, sigmavirus24: around?19:50
*** salv-orlando has quit IRC19:51
browneyep, back from lunch19:54
tmcpeakwould love to have mergies soon19:56
brownethis is needed for 0.13.1?19:57
tmcpeakbrowne: yep19:57
tmcpeakDaviey: yeah, I was thinking about utils, but honestly it gets monkey patched once at log initialization and then it's good for the rest of time19:57
tmcpeakmain seemed appropriate19:57
tmcpeakutils is more for things that will get used in multiple places19:57
tmcpeakat least in my mind19:58
brownetmcpeak: could you tag it with a bug, since this is important for a respin19:58
Davieytmcpeak: Well, either way.. It crossed my mind yo try and move all the functions out of to try and make that file simpler...19:58
tmcpeakDaviey: yeah, probably time for a clean19:58
tmcpeakbrowne: no, but I can file one :)19:58
tmcpeaksigmavirus24's earlier change closed abug19:58
Davieytmcpeak: Leave it there.. and for the future we'll refactor a bunch of it i guess19:58
tmcpeakthis just makes the output nicer19:58
Davieyrefactoring WITH better unit testing :)19:59
tmcpeakyep yep19:59
Davieybrowne: I'm not sure i agree with opening bugs for bugs sake TBH.20:00
brownesure i get that, but in this case we're doing a 0.13.1 just for this or was mostly for sigmavirus24 fix?20:01
sigmavirus24more so for the fix I tossed in20:01
sigmavirus24if "this" is warnings.warn stuff then this is to make sure that we don't have super ugly output20:01
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
tmcpeakbrowne: done20:02
tmcpeakbrowne: mostly the sigmavirus24 fix20:02
tmcpeakbtw, sigmavirus24 is a lot to type out, from now on it's going to be sv24 I think ;)20:02
browneyeah, i wish my IRC client was smarter20:03
sigmavirus24tab completion is your friend tmcpeak20:03
Davieytmcpeak: here, have a sug<tab>20:03
tmcpeakoh, lol20:03
tmcpeakwow, I never knew I had that20:03
browneoh crap, that does work.20:03
DavieyXmas come early for tmcpeak20:03
browneme too20:03
tmcpeakmy life just got 8-9% better20:03
DavieyAlthough, I am displeased with how many nicks start with t here.20:03
Davieytmcpeak: Are you using telnet/netcat to IRC? If so, switching to a proper client will make your life ~3% better.20:05
tmcpeakDaviey: no ways, I use Adium20:07
tmcpeakterminals scare me20:07
tmcpeakthe little duck jumps up and down when I get called out20:07
tmcpeakthat pleases me20:07
DavieyWOW.. I'd never even seen Adium before20:08
tmcpeakit's legit20:08
smcginnis+1 irssi :)20:08
tmcpeaklol, this thing20:09
sigmavirus24Daviey: weechat or bust =P20:09
DavieyAdium looks like Microsoft Chat -
smcginnisWe could make it really interesting and start talking about text editors. :D20:09
* sigmavirus24 doesn't have time for that20:09
tmcpeakirssi do you even HTML bro?
browneAdium is unfortunately the best free IRC client on mac OS (that i know of)20:09
brownei keep wishing for something more like Slack20:10
tmcpeakahh I haven't used slack yet, is it good?20:10
brownetmcpeak: its amazing20:10
*** daemontool_ has joined #openstack-security20:10
tmcpeakwow, that's high prize for a chat client20:10
sigmavirus24browne: textual is often available for free20:11
sigmavirus24it's nothing (visually) like Slack but it does work well and gives me growl notifications which is all I care about20:11
brownesigmavirus24:  yeah, think i tried others, but they didn't support proxies20:11
tmcpeakbrowne, sigmavirus24, Daviey, etc etc etc new rev here:
sigmavirus24that said, when I dno't care if people are pinging me, I use irssi or weechat20:11
tmcpeakbtw, has Zuul given up on life again?20:11
DavieyZuul has been super sketchy the last few weeks20:12
tmcpeakyeah man20:12
*** marzif has quit IRC20:12
sigmavirus24Go figure, a CI system needs disk space to work20:12
tmcpeakit's out of disk?20:13
tmcpeaksomebody plug in some damn thumb drives, get this thing rolling again20:13
DavieyThere is actually a spec' to allow people to contribute nodes to the CI pool.20:14
DavieyAnd another Spec to drop Jenkins and make Zuul everything.20:14
tmcpeakvoluntarily or no? ;)20:14
tmcpeakI could see a future where you click the wrong link and next thing you know you're running Zuul jobs20:15
Davieytmcpeak: I jokingly offered my old machine, and I was thrown back a serious spec.  FML.20:15
brownetmcpeak: should i still be seeing this20:15
brownebandit/plugins/ UserWarning: Using relative path for word_list: ./wordlist/default-passwords20:15
browne  % word_list_path)20:15
tmcpeakhmm, lol no20:16
DavieyThat is probably my fault20:16
tmcpeakbut I see it too20:16
tmcpeaklet me fix that too20:16
brownetmcpeak:  all i did was checkout your patch and do tox -r20:16
sigmavirus24yeah those should all be covered by the monkey patching of warnings module20:16
Davieyi imported warnings in the plugin, that is why20:16
DavieyDoes it make sense to convert that to an exception and convert it to a warning in
tmcpeakbrowne: crap20:18
tmcpeakI think it isn't because of the entry points?20:19
tmcpeakdoes this make sense?20:19
*** alex_klimov has joined #openstack-security20:19
tmcpeakDaviey: yeah, I think it does20:19
DavieyYeah, warnings isn't a global monkey patch.. so my local import is using the real one20:20
tmcpeakDaviey, sigmavirus24, browne: ok how about this20:20
tmcpeakI'll implement BanditPluginException20:20
tmcpeakand raise that in the hardcoded password plugin20:20
tmcpeakcatch that when I run the plugins20:21
sigmavirus24I'm confused20:21
tmcpeakand if I see it use the monkey patched warning?20:21
tmcpeaksigmavirus24: I think because of stevedore plugins are not covered under the warnings monkey patch done in main20:21
sigmavirus24I wonder if we can do the monkey patch immediately in, i.e., right after we define the function with which we're monkey patching the warnings module20:22
sigmavirus24I don't think that'll change anything20:22
sigmavirus24I get it20:22
sigmavirus24we need to monkey patch warnings before other modules import it?20:22
tmcpeaksigmavirus24: you'd think I'm already doing that20:23
tmcpeakI'm monkey patching immediately in main20:23
sigmavirus24yeah but we import the plugin manager thing right?20:23
*** yaya has joined #openstack-security20:23
sigmavirus24that loads all plugins auto-magically I think20:23
tmcpeaksigmavirus24: logger comes first20:24
tmcpeaklogger = _init_logger(debug)20:25
tmcpeak    # By default path would be /etx/xdg/bandit, we want system paths20:25
tmcpeak    os.environ['XDG_CONFIG_DIRS'] = '/etc:/usr/local/etc'20:25
tmcpeak    extension_mgr = _init_extensions()20:25
*** yuanying has joined #openstack-security20:25
sigmavirus24tmcpeak: I'm not sure20:26
* sigmavirus24 also doesn't totally have his head here right now20:26
*** jmckind_ has joined #openstack-security20:28
*** yuanying has quit IRC20:29
Davieytmcpeak: Try implementing our own exception and see if you can make that a warning in bandit.py20:30
DavieyThat is probably a more graceful way of handling it anyway IMO20:30
bknudsona warning specific to bandit would be better20:30
sigmavirus24warnings do not actually catch exceptions though20:31
Davieyno, that isn't what i mean20:31
sigmavirus24Just making sure we're all on the same page20:31
*** jmckind has quit IRC20:32
Davieysigmavirus24: So we throw an exception in the plugin.. which bubbles up to where we do the filtering as a warning20:33
DavieySo reinterpret the exception as a warning20:33
*** yaya has quit IRC20:33
DavieyThat would work?20:33
*** jmckind_ has quit IRC20:33
DavieyJust thinking about it... I find it amazing that warnings module doesn't JFDI for us.20:34
tmcpeakyeah.. just struggling over that myself20:38
tmcpeakException -> warning doesn't feel nice20:38
bknudsonyou can make warnings raise the exception20:38
*** jmckind has joined #openstack-security20:38
Davieybknudson: that is the wrong way around, isn't it?20:42
Davieythe only thing we want warnings for is to filter to stop repeating20:43
bknudsonare you wondering how to log an exception? it's log.exception('whatever')20:43
Davieyno, no20:43
tmcpeakI believe I'm an idiot20:45
Davieybknudson: for i in range(1,10): log.exception('I only want to see this error once and not 10 times FFS!')20:45
sigmavirus24bknudson: we're using warnings so we'll only see a message once20:46
sigmavirus24I don't get this exception/warning stuff though20:46
bknudsonoh, sure... and you can control that per exception, I think.20:46
DavieySeems our very own Doug Hellman has written a book about this.20:50
tmcpeakwell I went nuts here20:50
tmcpeakturns out all of that stuff I said about the monkey patch not working was complete bs20:50
Daviey^^ doing something like that is what i was suggesting ^^20:51
tmcpeakDaviey: real quick, what is it we actually want to output here?20:52
tmcpeak    raise RuntimeError("Could not substitute '%(site_data_dir)s' "20:52
tmcpeak                       "to a path with a valid word_list file")20:52
*** salv-orlando has joined #openstack-security20:52
Davieytmcpeak: Just inform the user that we couldn't find a dictionary file20:53
Davieybefore we were silently ignoring this, and therefore doing no check but appearing to pass20:53
tmcpeakok I'm just going to say that20:54
* Daviey needs more Gin20:55
tmcpeakI just chased my tail for 30 mins for literally zero reason20:55
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
tmcpeakship it20:56
tmcpeakbrowne, sigmavirus24: approvies please20:56
tmcpeakI need to go far far away from a computer apparently20:56
tmcpeaksite_data_dir is apparently a function20:57
tmcpeakprinting it isn't working20:57
tmcpeakmonkey patch for warnings works fine with plugins20:58
Davieytmcpeak: wait, what was the fix for this discussion ?20:58
Davieywhat caused the fugly issue?20:58
tmcpeakwe were trying to %s print a function20:58
tmcpeakthe solution is the original warning monkey patch20:59
tmcpeakit works fine20:59
Davieywe are dumb.21:00
*** salv-orlando has quit IRC21:00
Davieynice catch tho.21:00
tmcpeakyeah, dumb :(21:00
tmcpeaksigmavirus24, browne: ok mergies for real21:01
Davieytmcpeak: I did put in the detail incase someone was scratching their head as to what was wrong in the config.. but i don't think it matters21:02
browneok, checking21:02
tmcpeakyeah, we can add it back later21:03
brownei still see the same thing21:03
brownebandit/plugins/ UserWarning: Using relative path for word_list: ./wordlist/default-passwords21:03
browne  % word_list_path)21:03
tmcpeakoh crap21:03
tmcpeakoh man21:04
tmcpeakthat's it21:04
tmcpeakI think I need to retire21:04
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
tmcpeakbrowne: try that ^21:05
brownetmcpeak:  will do21:06
brownetmcpeak:  no dice.  and now its worse.  I get assertion failure in the tests21:08
tmcpeakoh man21:08
tmcpeakok, well, this is going to require more time21:09
tmcpeakI don't really want to push this last minute and go away for the weekend, lest something breaks really bad21:09
tmcpeakso I guess 0.13.1 is going to have to wait21:09
Davieytmcpeak: I was busy when the earlier discussion happened about needing this .1.. but just to check, it isn't OMG emergency.. just annoying?21:10
tmcpeakwish I could repro your issue21:10
tmcpeakDaviey: yeah, Cinder (which isn't using it in a gate) gets a lot of exceptions when running because they don't have a profile which includes try,pass,except and others21:11
tmcpeaksigmavirus24 fixed the issue, but now we're having nasty formatting printing21:11
Davieyhow come Cinder is using the latest release?21:12
openstackgerritTravis McPeak proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
tmcpeakbrowne: sorry, one last time?21:12
tmcpeakDaviey: they don't have an upper pinned, so it just pulls the latest21:13
tmcpeakthat's the way the majority of projects are set up21:13
Davieytmcpeak: Hmm, o21:13
tmcpeakbrowne: it probably doesn't work21:14
brownetmcpeak: ha, should i stop?21:14
tmcpeakbrowne: go ahead21:14
tmcpeakif it works I'll do a little happy dance21:14
Davieytmcpeak: Unit tests fail at least :)21:14
Davieyno happy dance :(21:14
tmcpeakstill failing21:15
Davieyoh, lemme try the recent one21:15
brownetmcpeak: is it failing for you?21:15
tmcpeakbrowne: what's failing?21:15
tmcpeakthe output looks correct but unit tests are failing21:15
tmcpeakand I don't see your warning so I can't tell if it's fixed21:16
tmcpeakmy unit test is failing in a way that doesn't seem to have anything to do with this21:16
Davieytmcpeak: sorry, last commit fixed unit tests21:16
Davieybut, still the same crappy output21:16
Davieybandit/plugins/ UserWarning: Using relative path for word_list: ./wordlist/default-passwords "".format(word_list_path))21:16
brownetmcpeak: is this a Ubuntu only problem?  i still see the UserWarning in the output, but the test passed now21:16
tmcpeakok, so the monkey patch is not working21:17
tmcpeakas I initially suspected21:17
tmcpeakI also have broken unit tests on my system somehow21:18
tmcpeakFAIL: tests.test_config.ConfigTests.test_find_configs21:18
tmcpeakAssertionError: 3 not less than 321:18
tmcpeakI'm sure we fixed this21:19
*** jmckind has quit IRC21:19
tmcpeaksmcginnis: you still around?21:19
*** dwyde has joined #openstack-security21:19
tmcpeakI'm tempted to just fix this properly when I'm not feeling rush21:20
tmcpeakdon't want to bang out some half-assed fix21:20
sigmavirus24We already have ugly warning printing21:20
sigmavirus24so what's more of it? =P21:20
tmcpeaksigmavirus24: yeah, well that's an alternative21:20
tmcpeakI guess your fix is better than not having it21:21
tmcpeakshould I push 0.13.1 with that?21:21
tmcpeakat least a warning is better than spammed exceptions21:21
tmcpeakreleases are cheap21:21
tmcpeaksigmavirus24, browne, Daviey, bknduson votes21:21
tmcpeakrelease with sigmavirus24's fix now, smooth it out Monday or wait until Monday, release once and have it nice?21:22
*** markvoelker has quit IRC21:22
tmcpeakI'll go with release now and re-release Monday21:22
Davieytmcpeak: Gate is almost idle over the weekend21:23
DavieyI'd wait until Monday21:23
tmcpeakyeah, this isn't even a gate issue21:23
tmcpeakthis is just a cinder devs playing around with Bandit user experience issue21:23
tmcpeakhmm actually21:23
tmcpeakKeystone would probably see this in their gate as well, but it shouldn't cause it to fail21:23
Davieytmcpeak: I'm having a quick play with an idea.. but don't block on me21:23
tmcpeakbknudson: are you around?21:24
bknudsontmcpeak: y, where would I go?21:24
tmcpeakbknudson: hmm nevermind21:25
tmcpeakwanted to make sure this 13 didn't break Keystone but it looks like it did not21:25
tmcpeakso yeah, in that case I'm happy to wait until Monday to do anything21:25
tmcpeakbknudson - basically the issue is that the tmp file test now requires config, which your profile does not have21:25
tmcpeakwhich apparently causes copious errors to be blasted in output21:26
bknudsonwhy aren't we running the tmp test?21:26
*** dave-mcc_ has quit IRC21:26
bknudsonwhat's the test name?21:26
tmcpeakbknudson: no, you are21:27
tmcpeaklol wait what21:27
tmcpeakyou aren't21:28
tmcpeakwhy not21:28
bknudsonwhen was it added?21:28
brownebecause earlier bandit versions had a bug where the password list couldn't be found21:28
tmcpeakoh interesting21:28
tmcpeakbknudson: the gate isn't running it21:29
browneso goes back to my question, what version of bandit should the projects assume21:29
tmcpeakit's only in the verbose profile21:29
tmcpeakthat's very intersting21:29
bknudsonat this point projects have to assume 0.10.121:29
bknudsonsince that's what's in g-r21:29
tmcpeakno but, >=0.10.1, right?21:29
tmcpeaknot actual 10.121:30
bknudsonright, they have to work with 0.10.121:30
tmcpeakbknudson: your gate is running 0.1321:30
tmcpeakoh right21:30
bknudsonand assume that later versions are compatible21:30
tmcpeakthey have to be compatible with 0.10.121:30
brownewhat scenario would they not get 0.13, if they tested internally with their own PyPi mirror that's old?21:30
bknudsonbandit might be installed via rpm or deb or whatever21:31
bknudsonnot everyone uses pypi21:31
bknudsonbecause it's not secure21:31
brownetrue, although i don't think there are packages yet.  ok, so should we bump the g-r here soon?  we've made a lot of fixes21:32
bknudson(that's the rumor anyways)21:32
tmcpeakbrowne: yeah, definitely21:32
browneso when 0.13.1 is up, push g-r to 0.13.1 and then update projects?21:33
*** E7D4A1B8 has joined #openstack-security21:33
tmcpeakI really want to build this automation too to make sure we don't break projects21:33
browneok sounds good to me unless there are concerns21:33
tmcpeakthat's going to be my #1 priority at midcycle21:34
brownetmcpeak: yeah, that would be really nice.  but we can do manually for now21:34
browne#2 priority is probably unit tests21:34
tmcpeakmanual is prone to errors and it sucks21:34
brownethe maintenance of maintaining a separate bandit.yaml in each project also sucks hard21:35
tmcpeakyeah, definitely21:35
*** E7D4A1B8 has quit IRC21:36
*** yaya has joined #openstack-security21:36
*** E7D4A1B8 has joined #openstack-security21:36
Davieytmcpeak: did we notice the Cinder noise issue, or did Cinder dev's?21:38
smcginnisDaviey: You mean the exception messages? That was me.21:39
* smcginnis is a cinder dev21:39
Davieysmcginnis: Ah, ok - thanks21:40
smcginnisCausing trouble wherever I go. :)21:40
*** b10n1k has quit IRC21:44
*** zul has quit IRC21:45
sigmavirus24bknudson: I trust that's trolling (about PyPI not being secure021:52
bknudsonsigmavirus24: tmcpeak was complaining about it during the ossg meeting21:52
* sigmavirus24 wasn't around for the meeting21:53
*** zul has joined #openstack-security21:55
*** dwyde has quit IRC21:57
*** salv-orlando has joined #openstack-security21:59
*** yaya has quit IRC21:59
Davieytmcpeak: So, raising an exception in the plugin and then wrapping it in try/except in and raising it as a warning seems to do it?21:59
tmcpeaksigmavirus24: yeah, PyPI wasn't really built for enterprise uses22:00
tmcpeakdstufft has his hands full trying to make it better but legacy PyPI is super legacy22:00
tmcpeakwe got brute force prevention added to it yesterday22:00
tmcpeakDaviey: yeah, well in one of those cases we want to warn and keep going, and in the other we don't22:01
Davieytmcpeak: Right, if UserWarning is raised, catch it and convert it to a warning.warn22:01
tmcpeakDaviey: yeah, that should work22:02
DavieyUserWarning isn't a show stopper but something that should carry on22:02
tmcpeakif we're taking our time though, I'd like to actually implement two of our own exceptions - BanditPluginError and BanditPluginWarning22:02
tmcpeakto be more explicit22:02
tmcpeakbut yeah, UserWarning is fine22:02
Daviey$ git diff | grep BanditWarning22:03
Daviey+class BanditWarning(Exception):22:03
Daviey+                    raise utils.BanditWarning("Using relative path for word_list: {0}"22:03
tmcpeakperfect :)22:03
tmcpeakDaviey: if you (or anybody else) want to take over that set I had going, that would be awesome22:03
sigmavirus24Daviey: banditwarning should inherit from Warning22:05
sigmavirus24Not exception22:05
Davieytmcpeak: You are 99% there, i don't want to steal your glory.. Just the last stretch now22:05
tmcpeaksteal away22:06
tmcpeakI shun glory22:06
Davieysigmavirus24: Ah, good thinking22:06
sigmavirus24I shun sleep22:06
Davieysigmavirus24: can you 'raise' a Warning?22:06
sigmavirus24Daviey: you're not supposed to22:06
tmcpeakyeah, can you?22:07
sigmavirus24warnings.warn('message', BanditWarning)22:07
Davieyah right22:07
Davieywait.. but we can't try/except a warning22:08
*** salv-orlando has quit IRC22:08
sigmavirus24I don't understand why you want to22:08
DavieyBut i suppose the redfintion of warnings cn be imported per plugin22:09
Davieyrather than using core warnigns module22:09
tmcpeakDaviey: yeah, but that sucks22:09
tmcpeakI'd prefer the raise approach22:09
Davieysigmavirus24: Do you have a better direction?22:09
bknudsonif you set warnings filter to "error" it'll raise the exception22:09
edmondsw so the issue smcginnis found, , is because bandit.yaml is missing something?22:10
openstackLaunchpad bug 1481922 in Bandit "'NoneType' object has no attribute '__getitem__'" [High,Fix committed] - Assigned to Sean McGinnis (sean-mcginnis)22:10
sigmavirus24I don't understand what you're all doing so22:10
sigmavirus24edmondsw: correct22:10
Davieywell then we are warning-treated-as-exception, catching-and-convert-back-to-monkey-warning22:10
Davieywhich is even nastier IMO22:10
edmondswsigmavirus24 what exactly is missing?22:10
tmcpeakedmondsw: yeah, so basically browne proposed the config for Bandit a while ago, since then we added new plugins which require config and the new profile didn't get them22:10
tmcpeakedmondsw: I'll dig up a link22:10
edmondswtmcpeak, tx22:11
tmcpeakthis block:  and this block:
sigmavirus24edmondsw: try_except_pass was what smcginnis was seeing22:11
edmondswsame here22:11
smcginnisedmondsw: Yeah, outdated config that caused problems for missing new plugins.22:12
*** yuanying has joined #openstack-security22:13
edmondswok, I think I understand... need to add try_except_pass:\n  check_typed_exception: True to the yaml22:16
*** yuanying has quit IRC22:17
*** edmondsw has quit IRC22:20
*** bknudson has quit IRC22:23
Davieytmcpeak: sweet, that worked...22:24
Daviey$ bandit -r bandit/22:24
Daviey[bandit]INFOusing config: /usr/local/etc/bandit/bandit.yaml22:24
Daviey[bandit]INFOrunning on Python 2.7.622:24
Daviey[general_hardcoded_password]WARNINGUsing relative path for word_list: ./wordlist/default-passwords22:24
*** singlethink has quit IRC22:32
*** zul has quit IRC22:35
Davieytmcpeak: So actually, just monkey patching in is enough.. This works,
Davieytmcpeak: should i push that up?22:42
Davieyactually, it seems it can be removed from aswell22:44
*** markvoelker has joined #openstack-security22:50
* Daviey pushes up.. feel free to revert.22:51
openstackgerritDave Walker proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
*** voodookid has quit IRC23:03
*** yuanying has joined #openstack-security23:14
browneDaviey: looks much better!23:14
brownethe only thing that's odd is how these warning messages are printed in the middle of the file count23:15
browne790 [0.. [tester]WARNING"hardcoded_tmp_directory" has been skipped due to missing config "hardcoded_tmp_d23:15
brownebut that's a nit23:15
*** yuanying has quit IRC23:18
*** alex_klimov has quit IRC23:19
*** yuanying has joined #openstack-security23:23
*** Daviey has quit IRC23:25
*** Daviey has joined #openstack-security23:26
*** salv-orlando has joined #openstack-security23:27
openstackgerritStanislaw Pitucha proposed openstack/anchor: Implement new API format
openstackgerritStanislaw Pitucha proposed openstack/anchor: Move all plugins to stevedore
openstackgerritStanislaw Pitucha proposed openstack/anchor: Move sample config for tests to one place
openstackgerritStanislaw Pitucha proposed openstack/anchor: Allow configurable signing backends
*** bknudson has joined #openstack-security23:35
*** sdake has quit IRC23:38
openstackgerritDave Walker proposed openstack/bandit: Capture warnings for missing plugins or config in normal logging
Davieysigmavirus24: Can you take a quick look.. /me wants to go to bed. :)23:46

Generated by 2.14.0 by Marius Gedminas - find it at!