Friday, 2015-07-17

*** salv-orlando has joined #openstack-security00:01
*** openstack has joined #openstack-security00:04
openstackgerritNathaniel Dillon proposed openstack/security-doc: Adding section to compute chapter  https://review.openstack.org/20282500:05
*** salv-orlando has quit IRC00:06
*** y_sawai has joined #openstack-security00:18
*** y_sawai has quit IRC00:19
*** hyakuhei1 has quit IRC00:28
*** hyakuhei has joined #openstack-security00:28
*** sdake has joined #openstack-security00:30
*** sigmavirus24 is now known as sigmavirus24_awa00:37
*** barra204 has quit IRC00:39
*** tmcpeak has quit IRC01:10
*** bpokorny has quit IRC01:32
*** hyakuhei has quit IRC01:37
*** hyakuhei has joined #openstack-security01:39
*** elo1 has joined #openstack-security01:50
*** elo has quit IRC01:54
*** dave-mccowan has quit IRC02:18
*** y_sawai has joined #openstack-security02:48
*** y_sawai has quit IRC02:49
*** hyakuhei has quit IRC02:49
*** hyakuhei has joined #openstack-security02:50
*** y_sawai has joined #openstack-security02:52
*** elo1 has quit IRC03:01
*** y_sawai_ has joined #openstack-security03:01
*** y_sawai has quit IRC03:04
*** tmcpeak has joined #openstack-security03:13
*** tmcpeak has quit IRC03:13
*** tmcpeak has joined #openstack-security03:15
*** hyakuhei has quit IRC03:17
*** hyakuhei has joined #openstack-security03:19
*** sdake has quit IRC03:23
*** sdake has joined #openstack-security03:33
*** y_sawai_ has quit IRC03:48
*** y_sawai has joined #openstack-security03:48
*** y_sawai has quit IRC03:49
*** y_sawai has joined #openstack-security03:49
*** y_sawai has quit IRC03:54
*** hyakuhei has quit IRC03:55
*** hyakuhei has joined #openstack-security03:55
*** misc has quit IRC04:17
*** misc has joined #openstack-security04:25
*** y_sawai has joined #openstack-security04:30
*** y_sawai has quit IRC04:36
*** sicarie is now known as sicarie_away04:46
*** sdake has quit IRC04:56
*** hyakuhei has quit IRC05:05
*** hyakuhei has joined #openstack-security05:09
*** browne has quit IRC05:16
*** tmcpeak has quit IRC05:19
*** browne has joined #openstack-security05:21
*** Daviey_ has joined #openstack-security05:22
*** Daviey has quit IRC05:23
*** browne has quit IRC05:27
*** markvoelker has joined #openstack-security05:41
*** markvoelker_ has joined #openstack-security05:44
*** markvoelker has quit IRC05:45
*** y_sawai has joined #openstack-security05:52
*** hyakuhei has quit IRC06:06
*** ig0r_ has joined #openstack-security06:08
*** hyakuhei has joined #openstack-security06:11
*** ig0r_ has quit IRC06:45
*** ig0r_ has joined #openstack-security06:49
*** y_sawai has quit IRC07:07
openstackgerritAndreas Jaeger proposed openstack/security-specs: Housekeeping  https://review.openstack.org/20290407:09
openstackgerritAndreas Jaeger proposed openstack/security-specs: Sync with global requirements  https://review.openstack.org/20290607:12
*** kutija_ has quit IRC07:25
*** kutija has joined #openstack-security07:26
*** alex_klimov has joined #openstack-security07:28
*** salv-orlando has joined #openstack-security07:39
openstackgerritAndreas Jaeger proposed openstack/security-doc: Setup RST Security Guide  https://review.openstack.org/20291708:03
*** salv-orlando has quit IRC08:17
*** hyakuhei1 has joined #openstack-security08:18
*** hyakuhei has quit IRC08:18
*** markvoelker_ has quit IRC08:42
*** markvoelker has joined #openstack-security08:57
*** markvoelker has quit IRC09:02
*** markvoelker has joined #openstack-security09:12
*** Daviey_ is now known as Daviey09:14
*** markvoelker has quit IRC09:17
*** hyakuhei1 has quit IRC09:24
*** hyakuhei has joined #openstack-security09:25
*** markvoelker has joined #openstack-security09:26
*** tkelsey has joined #openstack-security09:30
*** markvoelker has quit IRC09:31
*** hyakuhei has quit IRC09:35
*** hyakuhei has joined #openstack-security09:37
*** markvoelker has joined #openstack-security09:41
*** markvoelker has quit IRC09:45
*** rmarathu has joined #openstack-security09:50
*** Nospheratos has joined #openstack-security09:52
*** markvoelker has joined #openstack-security09:55
*** markvoelker has quit IRC10:00
openstackgerritAndreas Jaeger proposed openstack/security-doc: Setup RST Security Guide  https://review.openstack.org/20291710:01
*** markvoelker has joined #openstack-security10:07
*** markvoelker has quit IRC10:12
*** markvoelker has joined #openstack-security10:22
*** Nospheratos has quit IRC10:29
*** openstackgerrit has quit IRC10:31
*** openstackgerrit has joined #openstack-security10:31
*** markvoelker has quit IRC10:32
*** markvoelker has joined #openstack-security10:37
*** sdake has joined #openstack-security10:39
*** markvoelker has quit IRC10:42
*** hyakuhei has quit IRC10:50
*** hyakuhei has joined #openstack-security10:51
*** markvoelker has joined #openstack-security10:51
*** markvoelker has quit IRC10:56
*** markvoelker has joined #openstack-security11:06
*** markvoelker has quit IRC11:11
*** rmarathu has quit IRC11:17
*** kcaj has joined #openstack-security11:19
*** markvoelker has joined #openstack-security11:19
*** markvoelker has quit IRC11:24
*** sdake has quit IRC11:32
*** markvoelker has joined #openstack-security11:32
*** markvoelker has quit IRC11:44
*** rmarathu has joined #openstack-security11:49
*** markvoelker has joined #openstack-security11:55
*** markvoelker has quit IRC12:00
*** markvoelker has joined #openstack-security12:09
*** hyakuhei has quit IRC12:10
*** hyakuhei has joined #openstack-security12:11
*** markvoelker has quit IRC12:13
*** markvoelker has joined #openstack-security12:16
*** markvoelker has quit IRC12:21
*** edmondsw has joined #openstack-security12:23
*** markvoelker has joined #openstack-security12:24
*** markvoelker has quit IRC12:32
*** sdake has joined #openstack-security12:34
*** rmarathu has quit IRC12:39
*** markvoelker has joined #openstack-security12:39
*** markvoelker has quit IRC12:43
*** markvoelker has joined #openstack-security12:53
*** markvoelker has quit IRC12:58
*** rmarathu has joined #openstack-security12:59
*** browne has joined #openstack-security13:00
*** rmarathu has quit IRC13:05
*** markvoelker has joined #openstack-security13:05
*** markvoelker_ has joined #openstack-security13:07
*** markvoelker has quit IRC13:09
*** markvoelker_ has quit IRC13:34
*** tmcpeak has joined #openstack-security13:41
tkelseyo/ tmcpeak13:42
tmcpeakyo13:42
*** markvoelker has joined #openstack-security13:49
*** dave-mccowan has joined #openstack-security13:54
*** markvoelker has quit IRC13:54
*** markvoelker has joined #openstack-security13:59
*** sigmavirus24_awa is now known as sigmavirus2414:06
*** markvoelker has quit IRC14:07
*** fubi has quit IRC14:12
*** markvoelker has joined #openstack-security14:14
*** markvoelker has quit IRC14:18
*** hyakuhei1 has joined #openstack-security14:23
*** hyakuhei has quit IRC14:23
*** timkennedy has quit IRC14:24
*** markvoelker has joined #openstack-security14:28
*** sicarie_away is now known as sicarie14:31
*** markvoelker has quit IRC14:33
openstackgerritMerged openstack/anchor: Add tests for CA read failures  https://review.openstack.org/20187714:35
*** voodookid has joined #openstack-security14:36
*** markvoelker has joined #openstack-security14:50
tmcpeaksigmavirus24, browne, chair6, Daviey, tkelsey: https://bugs.launchpad.net/bandit/+bug/147568114:52
openstackLaunchpad bug 1475681 in Bandit "System wide pip install fails on some platforms" [Undecided,New]14:52
Davieyuho14:52
tmcpeakI saw something similar when working with one of our devs yesterday14:52
tmcpeakonly way I can think this could happen is if it can't find plugins14:53
Davieytmcpeak: have you done a bisect?14:53
tmcpeakso I think at a minimum we need to scream very loudly if it can't find any plugins14:53
tmcpeakif it can't find plugins we can probably just print a big loud error message and exit14:53
tmcpeakno point in running without plugins14:53
tmcpeaknow the real question is why it can't find plugins in this case14:53
Davieytmcpeak: "report any findings for some projects." .. since last week, bandit exits if it can't find any plugins14:53
tmcpeakahh ok, that hasn't been pushed into PyPI yet14:54
DavieyYeah.. only landed last week14:54
tmcpeakDaviey: ok awesome14:54
tmcpeaknow new question is why can't it find the plugins ;)14:54
browneyep, i've seen this before.  when it can't find the plugins14:54
tmcpeakseems to happen more frequently on Ubuntu...14:55
tmcpeakgmurphy as well as the dev I was talking to see it on Ubuntu systems14:55
Daviey$ bandit .14:55
Daviey[bandit]INFOusing config: /home/dave/openstack/old/bandit/bandit/config/bandit.yaml14:55
Daviey[bandit]INFOrunning on Python 2.7.614:55
Daviey[bandit]ERRORCould not find any tests to apply, please check the configuration.14:55
Daviey$ echo $?14:55
Daviey214:55
browneyou don't need vagrant to reproduce14:55
tmcpeakDaviey: ok awesome, so that part's handled14:56
tmcpeakbrowne: well I probably need vagrant to reproduce since I don't run Ubuntu14:56
browneoh ok.  i always use Ubuntu14:56
Davieytmcpeak: The finding of plugins may well change if we change to a stevedore based plugin infrastructure14:56
*** markvoelker has quit IRC14:57
*** markvoelker_ has joined #openstack-security14:57
*** markvoelker_ has quit IRC14:57
browneDaviey: is there a stevedore patch in flight?14:57
tmcpeakDaviey: true14:57
*** markvoelker has joined #openstack-security14:57
Davieybrowne: I started toying around with it.. but I am now writing a spec for it14:57
Davieytmcpeak: So i have seen something similar in my local env.  When i am using bandit bin from my tox -e py27 environment, i need to symlink the plugin directory into the site-packages under tox env.14:59
*** dwyde has joined #openstack-security15:00
tmcpeakinteresting15:00
tmcpeakso where is your Bandit bin and where are the plugins?15:00
tmcpeakrather where is your bandit.py15:00
Davieybandit/.tox/py27/bin/bandit15:01
DavieyDoing an strace it ONLY looks for plugins in bandit/.tox/py27/lib/python2.7/site-packages/bandit/plugins15:01
tmcpeakoh, I failed to correctly read your original message15:01
tmcpeakhmmmmm...15:02
sigmavirus24hm15:02
* sigmavirus24 is doing vagrant up still15:02
tmcpeakisn't bandit/.tox/py27/bin/bandit itself a symlink to something?15:02
*** timkennedy has joined #openstack-security15:02
*** bknudson has joined #openstack-security15:03
sigmavirus24in retrospect, I could have just spun up a cloud server faster15:03
* sigmavirus24 shrugs15:03
DavieyThere is a ./lib/python2.7/site-packages/bandit.egg-link which points to the git root, and SHOULD be respected.. but the plugin finding doesn't respect it15:03
brownei'm trying to reproduce now on my ubuntu15:03
sigmavirus24tmcpeak: should be15:03
tmcpeaksigmavirus24: how does flake8 handle this?15:04
tmcpeakno sense coming up with a solution when we can just steal one15:04
sigmavirus24so ... all of our plugins are 3rd party packages anyway15:04
sigmavirus24so basically we use pkg_resources (stevedore without all the convenience) and import from our entry-point15:04
Davieytmcpeak: The bandit executable is a pbr generated script, which DOES respect the egg.link file15:04
Davieyhttp://paste.openstack.org/show/383932/15:05
sigmavirus24But I suspect that we're not packaging things correctly15:05
sigmavirus24which I have a different way of verifying15:05
sigmavirus24I suspect that setuptools isn't grabbing bandit/plugins15:05
sigmavirus24because it doesn't know that it should15:05
tmcpeaksigmavirus24: I think you're right15:05
Davieysigmavirus24: I'm not sure it is that...15:05
sigmavirus24Daviey: that's my first suspicion15:05
sigmavirus24I haven't confirmed it yet15:05
tmcpeakDaviey: what's this output from?15:06
sigmavirus24also, keep in mind, tox installed != pip installed15:06
Davieytmcpeak: that pastebin is the bandit executable15:06
Davieysigmavirus24: I'm almost certain this issue is caused because we have a cack handed plugin discovery15:06
Davieywhich doesn't respect egg link files15:07
sigmavirus24egg-link is when you do "python setup.py install" which is not what pip does15:07
sigmavirus24that's what tox does15:07
tmcpeakDaviey: that sounds right too, extra points for "cack handed"15:07
sigmavirus24pip does not make eggs anymore15:07
sigmavirus24(hasn't in a long long time)15:07
browneericwb@ericwb-virtual-machine:~/bandit$ time .tox/py27/bin/bandit -r ../nova/15:08
browne[bandit]INFOusing config: /home/ericwb/bandit/bandit/config/bandit.yaml15:08
browne[bandit]INFOrunning on Python 2.7.615:08
browne[bandit]ERRORCould not find any tests to apply, please check the configuration.15:08
brownereal0m0.383s15:08
browneuser0m0.340s15:08
brownesys0m0.040s15:08
Davieytmcpeak: Sorry, British-ishm..15:08
tmcpeakI like British-ism15:08
tmcpeaks15:08
*** bpokorny has joined #openstack-security15:08
tmcpeakok, browne: so you can reproduce15:08
tmcpeakwould you mind putting in a "import pdb; pdb.set_trace()"15:08
Davieybrowne: strace -f .tox/py27/bin/bandit -r ../nova/ 2>&1 | grep plugins15:09
tmcpeakand then doing "os.path.abspath(os.curdir)15:09
tmcpeakwould be good to know where Bandit is actually executing from and where the Bandit plugins are in relation to that15:09
*** timkennedy has quit IRC15:09
DavieyI think browne is seeing the same thing i described15:10
tmcpeakor Daviey, anybody that has the issue in front of them,15:10
sigmavirus24I can confirm that bandit/plugins is included15:10
Davieytmcpeak: Anywhere particular, or just in the bandit executable ?15:11
tmcpeakas close to main as you can get15:11
Davieyok15:12
browneericwb@ericwb-virtual-machine:~/bandit$ strace -f .tox/py27/bin/bandit -r ../nova/ 2>&1 | grep plugins15:12
browneopenat(AT_FDCWD, "/home/ericwb/bandit/.tox/py27/lib/python2.7/site-packages/bandit/plugins", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)15:12
tmcpeakok cool, so that's as expected, it's looking for the plugins basically in the same place it would if it was a source install15:12
tmcpeaknow the question is where are the plugins actually installed15:12
tmcpeakor are they15:13
*** sdake has quit IRC15:13
sigmavirus24oh15:13
sigmavirus24I know what the problem is15:13
tmcpeaksick - what you got?15:13
brownebandit is not installed here: ~/bandit/.tox/py27/lib/python2.7/site-packages15:13
sigmavirus24so on Ubuntu apt installs to /usr/lib/python2.7/dist-packages15:13
sigmavirus24pip installs to /usr/local/lib/python2.7/dist-packages15:13
Davieybrowne: can you do, ls -al /home/ericwb/bandit/.tox/py27/lib/python2.7/site-packages/bandit*15:13
sigmavirus24for some reason, we're looking in teh wrong place15:13
Daviey^^ i bet there is an egg link file15:13
tmcpeakwho is doing apt though?15:14
sigmavirus24http://paste.openstack.org/show/383960/15:14
browneDaviey: right, no bandit there15:14
browneubuntu typically goes to dist-packages15:14
Davieybrowne: but there is an egg link file called bandit?15:14
sigmavirus24browne: they both go to dist-packages but one is /usr/lib and the other is /usr/local/lib15:14
browneDaviey: nope, no bandit dir of any kind15:15
sigmavirus24ther is no .egg-info15:15
Davieysigmavirus24: I think this is the wrong road you are going down... python (outside of tox) should search all PYTHONPATH's15:15
brownesigmavirus24: pip installed goes to /usr/local/lib, whereas deb packages go to /usr/lib15:15
sigmavirus24Daviey: look at my paste and the original bug15:16
sigmavirus24bandit installed with pip, when you strace that, you get one attempt to open "bandit/plugins" in /usr/lib/python2.7/dist-packages15:16
sigmavirus24but bandit, installed by pip, goes in /usr/local/lib/python2.7/dist-packages15:16
DavieyYes, and i am saying it shouldn't matter15:16
DavieyThis is because of our clumsy plugin discovery15:17
browneericwb@ericwb-virtual-machine:~/bandit/.tox$ find |grep bandit15:17
browne./py27/lib/python2.7/site-packages/bandit.egg-link15:17
browne./py27/bin/bandit15:17
openstackgerritMerged openstack/anchor: Add tests for higher coverage  https://review.openstack.org/20146415:17
Davieyif i put foobar.py in /usr/local/lib or /usr/lib/.. python should search the entire path and find the first one that matches that name, right?15:18
tmcpeakdoes anybody know of a way to reproduce this on the mac without Vagrant, or am I barking up the wrong tree?15:18
browneDaviey: correct15:18
sigmavirus24tmcpeak: Mac's don't do this silly separation of logic15:18
tmcpeak;)15:18
tmcpeakhence why I've never seen the problem15:18
brownetmcpeak: just use vmware fusion to spin up a ubuntu.  :)15:19
Davieyright, so browne confirmed the same behaviour as me... the executable 'bandit' CORRECTLY finds the bandit lib.. but later fails to find the plugin as it isn't respecting ./py27/lib/python2.7/site-packages/bandit.egg-link15:19
sigmavirus24It's because of https://github.com/openstack/bandit/blob/8c7c1e53c56a299ca893ef9fde8f6d120341495f/bandit/core/constants.py#L3715:19
tmcpeakyeah, I've got a VM, so once I do that 'pip install bandit'?15:19
sigmavirus24We just need to convert the rest of the stuff to use the entry-points that we have available15:19
sigmavirus24That will fix all of this15:19
sigmavirus24tmcpeak: if you have pip installed, yes15:19
gmurphytmcpeak: yeah vagrant isn't necessary just ubuntu vm15:20
tmcpeakok cool15:20
brownetmcpeak: just follow instructions by Ian in https://review.openstack.org/#/c/199548/.  see comment PS415:20
Davieysigmavirus24: right, that is the clumsy bit i was talking about15:20
Davieynot respecting egg-link files15:20
sigmavirus24So we already have logic to load plugins from entry-points15:21
sigmavirus24We just need to add the entry-points for our own plugins and stop that silly looking for plugins in a specific directory :D15:22
browne++15:22
DavieyWell we could do that... or just switch to stevedore :)15:22
tmcpeaksigmavirus24, Daviey, browne: yeah, that sounds sensible15:22
tmcpeakDaviey: stevedore is already being used for extensions15:22
DavieyThe workaround i've been doing is: mkdir -p $(pwd)/.tox/py27/lib/python2.7/site-packages/bandit15:22
Davieyln -s $(pwd)/bandit/plugins /home/dave/openstack/old/bandit/.tox/py27/lib/python2.7/site-packages/bandit/plugins15:23
Davieytmcpeak: I mean using it for plugins15:23
sigmavirus24Daviey: we already use stevedore15:23
DavieyFor plugins?15:23
sigmavirus24We allow third party plugins through stevedore15:23
sigmavirus24yes15:23
sigmavirus24We just never registered our own through the  entrypoints for stevedore to find15:23
DavieyAh, so we need to use entry-points for OUR plugins then?15:23
Davieyright15:23
sigmavirus24yes15:23
tmcpeakyep15:24
sigmavirus24Add them to setup.cfg, remove that other hackery, confirm you still have your plugins, send review, merge it, cut 0.12.115:24
Davieysigmavirus24: Do you want to JFDI, or should i?15:24
tmcpeakJFDI, forget the spec15:24
Davieyyeah15:24
tmcpeakwe all know this is what we need15:24
DavieyWho is doing it?15:25
tmcpeakI don't want specs to trip us up, just used for coordination for when ideas might be contentious15:25
DavieyYeah, this isn't contentious. :)15:25
tmcpeakDaviey: are you willing to?15:25
sigmavirus24Daviey: if you have the time, go ahead15:25
sigmavirus24Otherwise, I'll get to it sometime tonight15:25
tmcpeakI'm under-the-gun ATM, but I could get to it next week15:26
DavieyWell, it is Friday afternoon and I've peaked from doing primary work.. So.. i could give it a gander15:26
tmcpeakif somebody can do it faster than that, that would be awesome15:26
tmcpeakDaviey: great!15:26
tmcpeakthanks man15:26
tmcpeakthe effort you save in writing the spec might be enough to get it merged :P15:26
Davieyyeahhah15:27
tmcpeakbrowne, Daviey, sigmavirus24: thanks for the swarming15:27
brownetmcpeak: np15:28
Davieynp15:28
sigmavirus24Daviey: feel free to ping me here for a +215:28
sigmavirus24I know what it /should/ look like15:28
sigmavirus24And I'm happy to test it on my vm when it's done to make sure it works as expected15:28
tmcpeakyeah, I'll watch for your change as well and move reviews to the top of the queue15:28
openstackgerritMerged openstack/anchor: Make sure X509_NAME lives long enough  https://review.openstack.org/20207815:28
*** sdake has joined #openstack-security15:28
browneI can help with the other +2 today.  i have a local env to test also15:29
tmcpeaksweet!15:29
tmcpeakgood stuff guys15:29
brownetmcpeak: but cutting 0.12.1 is all you. :)15:30
tmcpeakfor sure15:30
sigmavirus24tmcpeak: gets to test drive openstack/releases for ossg15:30
sigmavirus24=P15:30
tmcpeaklol15:30
sigmavirus24assuming the release managers manage bandit releases too15:30
tmcpeakthey haven't until now, but it might change now that we're in the big umbrella or whatever it's called15:31
sigmavirus24I guess not15:31
sigmavirus24tumbrella (tent + umbrella)15:31
sigmavirus24tentrella?15:31
sigmavirus24umbtenta?15:31
tmcpeakwe're in the big tentrella now15:31
openstackgerritMerged openstack/anchor: Add test for robots file  https://review.openstack.org/20187915:33
*** timkennedy has joined #openstack-security15:35
DavieyBeing in the big tetrapack means that centralized release management isn't a thing.. right?15:41
*** rol01340 has joined #openstack-security15:42
tmcpeakDaviey: yeah, nobody has cornered me yet though ;)15:43
*** rol01340 has left #openstack-security15:44
openstackgerritTim Kelsey proposed openstack/bandit: Improving SQL Injection detection  https://review.openstack.org/20264615:54
tmcpeakI just had a not so fun thought - this release actually might break gates15:55
tmcpeaklet's say a project has been thinking they had no issues because Bandit was running with no plugins.  Now all of a sudden it finds the plugins and boom, fails15:55
tmcpeakI think we actually need to validate that we aren't generating new findings in the existing project gates before we push :\15:55
tkelseytmcpeak: we need to use that cool script the get the projects using it and the config in use, then we need to run it locally15:55
tmcpeakit's a longshot that projects have been running incorrectly this whole time, but if they are it would really suck15:56
tkelseyif we find unexpected stuff, we need to let the project know before pushing15:56
tmcpeaktkelsey: yeah, for sure15:56
tkelseyim sure it will be fine so long as we give a heads up15:56
tkelseyand maybe even a patch or two if we find stuff15:57
tmcpeakyeah15:57
tkelseypriority testing goes to Keystone, since bknudson has been a hero and adopted it early15:58
tmcpeakyeah, I think they're the only voting gate still too15:59
tkelseycool15:59
tmcpeakso technically nothing would break15:59
tmcpeakwould still be good to check them all though15:59
tkelseyyeah16:00
openstackgerritTim Kelsey proposed openstack/bandit: Improving SQL Injection detection  https://review.openstack.org/20264616:04
*** Windir has joined #openstack-security16:11
*** Windir has quit IRC16:11
*** Windir has joined #openstack-security16:13
*** alex_klimov has quit IRC16:25
*** hyakuhei1 has quit IRC16:49
*** hyakuhei has joined #openstack-security16:52
*** sigmavirus24 has quit IRC16:52
*** sigmavirus24 has joined #openstack-security16:53
*** elo has joined #openstack-security16:53
sigmavirus24tmcpeak: so16:54
*** browne has quit IRC16:54
tmcpeaksigmavirus24: wassup16:54
sigmavirus24If I understand correctly, installing and using from within tox shouldn't be an issue for us16:54
*** dwyde has quit IRC16:54
sigmavirus24That should also be how most of the projects consume bandit, no?16:54
tmcpeaksigmavirus24: yeah, most are using a tox bandit setup16:56
*** sigmavirus24 has quit IRC16:57
*** sigmavirus24 has joined #openstack-security17:00
tmcpeaksigmavirus24: oh, I see what you're saying.. yeah, you're right.  If they're using tox nothing should change17:10
sigmavirus24tmcpeak: I'll verify it17:10
sigmavirus24* against trove17:10
tmcpeaksigmavirus24: awesome, thank you17:10
tmcpeakmy favorite17:11
sigmavirus24tmcpeak: I shouldn't need to since gmurphy said it worked fine inside a virtualenv in the bug17:11
sigmavirus24But just to be extra paranoid17:11
sigmavirus24=P17:11
tmcpeakyeah, we ended up solving the problem for the internal dev I was talking to yesterday with a venv17:12
sigmavirus24oh17:13
sigmavirus24I forgot tox always installs the project17:13
sigmavirus24=(17:13
sigmavirus24installing trove is taking a while17:13
sigmavirus24=P17:13
sigmavirus24-_-17:13
tmcpeakinstalls?17:13
tmcpeakit does?17:14
sigmavirus24by default tox will do `python setup.py install` (roughly)17:14
tmcpeakhuh17:14
tmcpeakI learn something new17:14
tmcpeak'eryday17:14
openstackgerritTim Kelsey proposed openstack/bandit: Improving SQL Injection detection  https://review.openstack.org/20264617:14
sigmavirus24lol17:16
sigmavirus24install_command = python -c 'print("foo")' {packages}17:16
sigmavirus24that gets around installing trove lol17:16
sigmavirus24And using it in tox works just fine17:17
sigmavirus24Apply http://paste.openstack.org/show/384218/ to trove's tox.ini and compare to bandit installed globally and you'll see that it works17:18
sigmavirus24So17:18
sigmavirus24we need not work about breaking anyone17:18
sigmavirus24(You only need the install_command line if you want this to go quickly and you don't want to install mysql or any of the other -dev packages)17:19
openstackgerritTim Kelsey proposed openstack/bandit: Improving SQL Injection detection  https://review.openstack.org/20264617:19
sigmavirus24Daviey: how goes?17:20
tmcpeaksigmavirus24: awesome17:20
Davieysigmavirus24: I'm iterating over the plugins OK.. but trying to getmembers is proving a little bit of a pain17:21
sigmavirus24Daviey: if you throw a wip up, I'll pull it and see if I can offer help. I'm on lunch at the moment so no one can give me crap for not working on our product =P17:21
tmcpeak;)17:23
openstackgerritDave Walker proposed openstack/bandit: Initial Stevedore work  https://review.openstack.org/20317417:24
Davieysigmavirus24: https://review.openstack.org/20317417:24
Davieyuh, damn bot.. making lives easier17:24
sigmavirus24Daviey: you don't need to reinvent the extension manager17:25
sigmavirus24We have that loading things already17:25
tmcpeaklegit cow ascii17:25
Davieysigmavirus24: Yeah.. I was trying to make it as unintrusive change as possible to start with17:26
sigmavirus24Daviey: https://github.com/openstack/bandit/blob/master/bandit/core/extension_loader.py#L2717:26
DavieyOIC17:26
sigmavirus24also17:27
sigmavirus24let's keep the cowsay in that commit message17:27
sigmavirus24In fact17:27
*** tkelsey has quit IRC17:27
sigmavirus24Let's add cowsay to every commit message17:27
sigmavirus24I'll make a hacking check17:27
sigmavirus24It'll enforce cowsay17:27
tmcpeakhaha17:27
DavieyI'm sure a git post commit hook to just add it17:27
DavieyIt isn't too dissimilar to the Change-ID hook17:28
Davieyif first_line contains "[WIP]": do_cowify() ; done17:28
Davieysigmavirus24: If you want to just push onwards with this, i won't be upset17:29
sigmavirus24let me see17:29
Davieyerr, stevedore.. not cowsay17:29
sigmavirus24oh17:29
sigmavirus24i want to work on cowsay now though =P17:29
tmcpeaklol - I see I'm in good company "ooh - shiny!"17:30
sigmavirus24oh I see17:30
Davieyhttps://drfatani.files.wordpress.com/2014/05/i-love-shiny-things.png17:30
sigmavirus24so Daviey when using stevedore17:30
sigmavirus24You'll get the functions back directly17:30
sigmavirus24getmembers was looking for functions defined int he module17:30
sigmavirus24which means that's totally unnecessary now17:30
Davieysigmavirus24: Right.. but they all retured None17:30
DavieyI tried that17:30
sigmavirus24So you don't need to muck around with gemembers or inspect anymore17:31
Davieyfunctions_list: {'xml': None, 'crypto_request_no_cert_validation': None, 'general_bind_all_interfaces': None, 'injection_shell': None, 'exec': None, 'blacklist_imports': None, 'try_except_pass': None, 'injection_paramiko': None, 'blacklist_calls': None, 'general_hardcoded_password': None, 'injection_wildcard': None, 'secret_config_option': None, 'general_bad_file_permissions': None, 'asserts': None, 'injection_sql': None, 'exec_as_root': No17:31
Davieysigmavirus24: see i tried to replace functions_list with, functions_list = dict((x.name, x.obj) for x in available_plugins) ?17:32
*** dwyde has joined #openstack-security17:32
DavieyThat _should_ IIUC have given me all the functions from the plugins, no ?17:32
sigmavirus24so extension_loader.MANAGER does the right things for you because you want invoke_on_load=True iirc17:32
DavieyYeah, that blew up17:32
sigmavirus24at least17:32
sigmavirus24that works well for the formatters we have defined17:32
sigmavirus24x.plugin isn't it?17:33
Davieysigmavirus24: with it set to True, i got - W Could not load 'crypto_request_no_cert_validation': 'module' object is not callable17:33
sigmavirus24yeah17:33
sigmavirus24L95 in bandit/core/result_store.py17:34
DavieySo.. it started getting complicated.. So i thought for phase 1, it might make sense to reuse the same workflow and do inspect.getmembers17:34
sigmavirus24Daviey: I don't want to steal this from you, hopefully that's enough to push you in the right direction17:34
Davieysigmavirus24: I honestly wouldn't be upset if you took this..17:35
sigmavirus24hm17:36
sigmavirus24oh sorry you do want invoke_on_load=False17:36
sigmavirus24the extension_loader is doing it right =P17:36
sigmavirus24Daviey: okay17:36
DavieyI mean, i'm sure i'll have something working eventually.  but i do need to EoD shortly.17:37
sigmavirus24sure17:38
sigmavirus24On my team we hand things off internationally when sprinting on stuff17:38
sigmavirus24We'll just do a handoff here ;)17:38
tmcpeakawesome17:40
Davieynice!17:43
DavieyOkay, weekend is starting.. I'll check in later o/17:49
sigmavirus24Have a good weekend Daviey!17:50
tmcpeakDaviey: awesome, have a good one17:52
*** browne has joined #openstack-security17:52
*** bpokorny_ has joined #openstack-security18:02
*** bpokorny has quit IRC18:03
*** markvoelker has quit IRC18:32
openstackgerritIan Cordasco proposed openstack/bandit: Register plugins included as entry-points  https://review.openstack.org/20317418:51
sigmavirus24gmurphy: tmcpeak ^ should fix it18:51
sigmavirus24Daviey was like 99% of the way there18:51
chair6+70, -104 .. most of the + is config .. that commit is a thing of beauty :)18:56
*** bknudson has quit IRC19:00
tmcpeaksigmavirus24: awesome!19:03
tmcpeakreviewsies coming now19:04
tmcpeakbrowne: can you please test too (I still can't repro the problem ;) )19:04
*** bpokorny_ has quit IRC19:07
*** bpokorny has joined #openstack-security19:07
tmcpeaksigmavirus24: my only complaint is the actual section where we're loading plugins is tough to read19:08
tmcpeakcan we get comments in there?19:08
tmcpeaklink breaks19:08
tmcpeaklogical separation, etc19:08
tmcpeakI'm not sure what setup.cfg supports19:08
*** bpokorny_ has joined #openstack-security19:19
*** bpokorny has quit IRC19:19
*** edmondsw has quit IRC19:24
*** sdake has quit IRC19:32
sigmavirus24tmcpeak: It should support comments19:33
sigmavirus24tmcpeak: do you mean line-breaks like having things across multpile lines?19:33
sigmavirus24*multiple19:33
tmcpeaksigmavirus24: no, just blank lines in between things19:33
tmcpeaklike19:33
sigmavirus24Sure19:33
sigmavirus24I might add comments too adding references to the file19:34
sigmavirus24I mean19:34
sigmavirus24That's inherent in the entry-point, but it won't hurt19:34
tmcpeakyeah, we can start with breaking them down by actual file,19:34
sigmavirus24Also if people go grepping for where that file is used, they'll see that ideally19:34
sigmavirus24In case people don't know where to add their new checks19:34
sigmavirus24SInce they won't be auto-loaded by module anymore19:34
sigmavirus24(That's more of a developer consideration than anything else)19:35
tmcpeaksorry, where which file is used/19:35
tmcpeak?19:35
sigmavirus24e.g., people looking where 'bandit/plugin/asserts.py'19:35
tmcpeakoh right19:35
sigmavirus24someone's like "I added a check! Why doesn't it magicappear"19:35
tmcpeakyeah, I expect some such confusion initially19:35
openstackgerritIan Cordasco proposed openstack/bandit: Register plugins included as entry-points  https://review.openstack.org/20317419:42
sigmavirus24tmcpeak: ^19:42
sigmavirus24Also fixed two typos in function names that I noticed19:42
tmcpeaksigmavirus24: awesome19:44
* tmcpeak looks19:44
*** ig0r_ has quit IRC19:47
tmcpeaksame dev from yesterday is having a hard time even in a virtualenv19:50
tmcpeak:|19:50
tmcpeakconfusion abounds19:50
sigmavirus24weird19:53
sigmavirus24did they try blowing away the virtualenv and rebuilding it?19:53
sigmavirus24the great thign about this vagrant box is that /vagrant/ is bandit's repo on my laptop19:54
sigmavirus24so I just pip uninstall -y bandit; pip install /vagrant19:54
* sigmavirus24 doesn't trust pip install -e /vagrant19:54
*** dave-mccowan has quit IRC20:07
* sigmavirus24 is trying it out in a venv anyway20:09
brownetmcpeak: will test now20:10
brownelove the cowsay20:10
*** timkennedy has quit IRC20:10
sigmavirus24browne: the cowsay is all Davey20:11
browneworks!20:14
*** dave-mccowan has joined #openstack-security20:16
tmcpeakbrowne: awesome20:16
browneanyone else want to look before i merge?20:18
tmcpeakbrowne: if it looks good to you, mergeys ;)20:19
browneok will do20:19
tmcpeakawesome, thank you20:20
tmcpeakfalse alarm on the dev having trouble in a virtualenv20:32
chair6excellent20:35
sigmavirus24tmcpeak: good to hear20:39
sigmavirus24It worked fine in a virtualenv for me20:39
sigmavirus24but I didn't want to be /That Guy/20:39
tmcpeakI've been that guy all day :P20:39
sigmavirus24Funny thing is that I realized I never sent this patch20:43
sigmavirus24And was thinking about it yesterday20:43
sigmavirus24"Hm, I should get to that. It should have been a follow-on of the original stevedore stuff"20:44
*** elo has quit IRC20:44
sigmavirus24So, sorry. I guess I should have done this sooner =P20:44
chair6it's okay, we know you're busy being a virus and all20:45
sigmavirus24lol20:45
chair6quite liking where bandit is heading towards.. input from all you folks is really appreciated, whether you "should have done this sooner" or not :)20:46
chair6i missed the discussion the other day about bandit specs20:46
chair6i personally am inclined to avoid using the specs repo for now, and instead just use launchpad blueprints20:47
sigmavirus24To be honest, I had thought about making something like bandit as a plugin for flake8 for a year now20:47
tmcpeakyeah, but then we can end up with confusion like we had with the statement buffer stuff20:47
sigmavirus24never comfortable enough to mark things as insecure20:47
* sigmavirus24 thinks he missed that confusion20:48
chair6heh, yeah, it's kinda nice having that come from a separate tool sigma20:48
chair6tmcpeak howso?  don't we just write what we plan to do in the blueprint form instead of the spec form?20:48
tmcpeakoh, I'm lumping blueprint and spec together20:49
tmcpeakI don't know the difference20:49
chair6(i have very little exposure to the spec side of things, so am arguing against something i know little about.. lazily trying to adding another workflow to my workflows)20:49
chair6heh, seems we might both be ill-informed then :)20:49
tmcpeakI'm generally lumping "written plans before we write code" together20:50
tmcpeaker "written plans before we write big code"20:50
chair6http://specs.openstack.org/ .. this is specs20:50
chair6we have a security-specs repo, that we could theoreticaly add bandit specs to20:50
tmcpeakok, so maybe something huge like on the Bandit 1.0 level would deserve a spec20:50
tmcpeakthis look pretty broad20:51
chair6heres an example of a swift spec .. http://specs.openstack.org/openstack/swift-specs/specs/in_progress/at_rest_encryption.html20:51
sigmavirus24So it depends as far as other projects are concerned20:51
sigmavirus24Big features tend to have specs that have associated blueprints20:51
sigmavirus24The blueprint is more to allow us to track the work in launchpad20:51
sigmavirus24The spec has all the technical details and discussion of the feature and renders nice things for others on specs.o.o20:51
tmcpeakthe main use-case I'm concerned with is soliciting feedback prior to making a change people might disagree with20:52
sigmavirus24Using both is actually what upstream projects do as far as I know20:52
chair6ahhh, cool .. so maybe we ahve a single 'spec' that describes what bandit hopes to become, then use whitepad to track individual features/changes against that spec?20:52
sigmavirus24tmcpeak: right, that's where specs make sense20:52
tmcpeakahh ok20:52
chair6lol, s/whitepad/blueprint/20:52
tmcpeakso multi-threaded for example20:52
sigmavirus24blueprints can track dependencies too which specs don't20:52
tmcpeakI think some are in favor and some aren't, right?20:52
tmcpeakthat could be a spec20:52
*** dwyde has quit IRC20:52
tmcpeakand then we can comment-war back and forth and come to consensus20:52
tmcpeakbefore we write code20:52
sigmavirus24I was going to write a spec. There's already a bp for that20:52
tmcpeakis that the general idea?20:53
sigmavirus24Or have that + code so that you can see the implementation along side the description20:53
sigmavirus24Yeah that's kind of how other projects do it20:53
tmcpeakok cool, that makes sense20:53
tmcpeakI think the other issue we're running into is synchronizing - not everybody is hanging out in IRC at the same time or at all20:53
sigmavirus24Right20:53
sigmavirus24Glance team has that issue20:53
sigmavirus24We have glance cores who are only ever on IRC for our weekly meeting or when they need to get a hold of someone20:54
*** dwyde has joined #openstack-security20:54
sigmavirus24Apropos of nothing, do we want a separate bug team for bandit to manage bugs?20:54
sigmavirus24So chair6 isn't the only bug supervisor?20:55
chair6heh, that's probably just from when i set launchpad up20:57
chair6yeah, good plan .. i can create a team that has current cores in it for a start20:57
sigmavirus24apparently as an ossg member I can change that20:58
sigmavirus24=P20:58
sigmavirus24I just looked and saw an edit button20:58
chair6.. oh, or you can :)20:58
sigmavirus24I was thoroughly confused20:58
sigmavirus24You can do it20:58
sigmavirus24I should be ansible-ing20:58
tmcpeakoh ansible land20:58
tmcpeakit's been a while since I've come across anything I feel more polarized about then Ansible - it's so magical when it works and fills me with hate when it doesn't20:59
*** edmondsw has joined #openstack-security20:59
sigmavirus24tmcpeak: this is why you run everything with -vvvvvvvv21:00
tmcpeakhaha yeah21:03
*** y_sawai has joined #openstack-security21:12
chair6okay, "Bandit Core" has been created on launchpad and populated21:14
tmcpeaksweet21:14
chair6i've set the 'driver' for the bandit project to it, as well as the bug supervisor21:14
sigmavirus24awesome!21:22
*** dave-mccowan has quit IRC21:27
*** dave-mccowan has joined #openstack-security21:28
*** dave-mccowan has quit IRC21:33
openstackgerritNathaniel Dillon proposed openstack/security-doc: Adding section to compute chapter  https://review.openstack.org/20282521:45
openstackgerritMerged openstack/bandit: Register plugins included as entry-points  https://review.openstack.org/20317422:03
*** y_sawai has quit IRC22:19
*** edmondsw has quit IRC22:28
*** dwyde has quit IRC22:39
*** sicarie has quit IRC22:48
* Daviey checks in22:59
Davieysigmavirus24: Thanks for taking that, nicely done.22:59
*** voodookid has quit IRC23:00
*** hyakuhei1 has joined #openstack-security23:03
*** hyakuhei has quit IRC23:03
chair6the best bit about it was the cowsay, just for the record23:11
Davieychair6: Well, that was my contribution.. So i'll take all the credit.23:13
chair6rightly so23:13
Davieysigmavirus24: I see why it wasn't working for me now.. I was lacking the function name in setup.cfg23:14
DavieyWe do claim to be able to support multiple functions per plugin.. which this doesn't do.. We might need to add Class support.23:16
*** dave-mccowan has joined #openstack-security23:16
*** y_sawai has joined #openstack-security23:19
*** tmcpeak has quit IRC23:21
chair6isn't that what's happening with (say) the xml stuff?  multiple functions per plugin?23:21
chair6for example:23:21
chair6    lxml_function_calls = bandit.plugins.xml:lxml_function_calls23:21
chair6    etree_celement_import = bandit.plugins.xml:etree_celement_import23:21
chair6    etree_element_import = bandit.plugins.xml:etree_element_import23:21
chair6i guess it depends on what you mean when you say 'plugin'23:22
*** y_sawai has quit IRC23:24
*** freerunner has joined #openstack-security23:25
DavieySo it is..23:26
*** bpokorny_ has quit IRC23:30
chair6groovy :)23:47
chair6happy weekend, y'all23:47

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!