| *** JAHoagie has quit IRC | 00:33 | |
| *** sdake has quit IRC | 00:36 | |
| *** sdake has joined #openstack-security | 00:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/security-doc: Updated from openstack-manuals https://review.openstack.org/359505 | 01:21 |
|---|---|---|
| *** JAHoagie has joined #openstack-security | 01:29 | |
| openstackgerrit | Charles Neill proposed openstack/syntribos: Removing "config-dir", modifying "config-file" https://review.openstack.org/348589 | 01:30 |
| *** salv-orlando has joined #openstack-security | 01:40 | |
| *** salv-orlando has quit IRC | 01:51 | |
| *** ccneill has quit IRC | 02:07 | |
| *** knangia has quit IRC | 02:11 | |
| openstackgerrit | Rahul U Nair proposed openstack/syntribos: Standardizing the way we diff signals https://review.openstack.org/349403 | 02:26 |
| *** murphy_zhao has quit IRC | 02:46 | |
| *** sdake has quit IRC | 02:46 | |
| *** sdake has joined #openstack-security | 02:49 | |
| *** salv-orlando has joined #openstack-security | 02:50 | |
| *** yuanying_ has joined #openstack-security | 02:51 | |
| *** yuanying has quit IRC | 02:52 | |
| *** yuanying has joined #openstack-security | 02:53 | |
| *** salv-orlando has quit IRC | 02:53 | |
| *** yuanying has quit IRC | 02:55 | |
| *** yuanying has joined #openstack-security | 02:56 | |
| *** yuanying_ has quit IRC | 02:57 | |
| *** JAHoagie has quit IRC | 03:07 | |
| *** aasthadi has joined #openstack-security | 03:36 | |
| *** yuanying has quit IRC | 03:48 | |
| *** yuanying has joined #openstack-security | 03:51 | |
| *** salv-orlando has joined #openstack-security | 03:58 | |
| *** bigdogstl has joined #openstack-security | 04:01 | |
| *** bigdogstl has quit IRC | 04:05 | |
| *** salv-orlando has quit IRC | 04:09 | |
| *** aasthadi has quit IRC | 04:09 | |
| *** aasthadi has joined #openstack-security | 04:11 | |
| *** JAHoagie has joined #openstack-security | 04:33 | |
| *** dave-mccowan has quit IRC | 04:49 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/security-doc: Updated from openstack-manuals https://review.openstack.org/359505 | 04:52 |
| *** salv-orlando has joined #openstack-security | 04:54 | |
| *** ibmchas has joined #openstack-security | 05:06 | |
| *** aasthadi has quit IRC | 05:58 | |
| *** rcernin has joined #openstack-security | 06:23 | |
| *** JAHoagie has quit IRC | 06:36 | |
| *** aasthadi has joined #openstack-security | 06:46 | |
| *** ibmchas has quit IRC | 06:50 | |
| *** tesseract- has joined #openstack-security | 06:56 | |
| *** murphy_zhao has joined #openstack-security | 07:01 | |
| *** aasthadi has quit IRC | 07:49 | |
| *** aasthadi has joined #openstack-security | 07:51 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack/bandit: Clean imports in code https://review.openstack.org/359659 | 07:52 |
| *** aasthadi has quit IRC | 07:55 | |
| *** woodster_ has quit IRC | 08:19 | |
| *** terri has quit IRC | 08:23 | |
| *** pcaruana has joined #openstack-security | 08:29 | |
| *** terri has joined #openstack-security | 08:32 | |
| *** dikonoor has joined #openstack-security | 08:45 | |
| openstackgerrit | Emma Foley proposed openstack/security-doc: Updated from openstack-manuals https://review.openstack.org/359505 | 09:30 |
| *** vinaypotluri has quit IRC | 09:31 | |
| *** salv-orlando has quit IRC | 09:40 | |
| *** salv-orlando has joined #openstack-security | 09:41 | |
| *** shohel has joined #openstack-security | 09:58 | |
| openstackgerrit | Merged openstack/security-doc: Updated from openstack-manuals https://review.openstack.org/359505 | 10:06 |
| *** aastha has quit IRC | 10:17 | |
| *** sdake has quit IRC | 10:28 | |
| *** chas has joined #openstack-security | 10:32 | |
| *** chas is now known as ibmchas | 10:33 | |
| *** dikonoor has quit IRC | 10:44 | |
| *** d0ugal has quit IRC | 10:59 | |
| *** d0ugal has joined #openstack-security | 11:00 | |
| *** salv-orlando has quit IRC | 11:25 | |
| *** salv-orlando has joined #openstack-security | 11:25 | |
| *** zhihui has quit IRC | 11:43 | |
| *** sigmavirus|away is now known as sigmavirus | 11:51 | |
| *** zhihui has joined #openstack-security | 11:51 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/security-doc: Updated from openstack-manuals https://review.openstack.org/359822 | 11:58 |
| *** zhihui has quit IRC | 12:16 | |
| *** markvoelker has joined #openstack-security | 12:25 | |
| *** sangress_ has joined #openstack-security | 12:26 | |
| openstackgerrit | Merged openstack/security-doc: Updated from openstack-manuals https://review.openstack.org/359822 | 12:34 |
| *** sangress_ has left #openstack-security | 12:36 | |
| *** edmondsw has joined #openstack-security | 12:50 | |
| *** zul has quit IRC | 12:51 | |
| *** markvoelker has quit IRC | 12:53 | |
| *** jass93 has quit IRC | 12:57 | |
| *** salv-orlando has quit IRC | 13:07 | |
| *** dave-mccowan has joined #openstack-security | 13:15 | |
| *** _elmiko is now known as elmiko | 13:17 | |
| *** woodster_ has joined #openstack-security | 13:23 | |
| *** openstackgerrit has quit IRC | 13:26 | |
| *** openstackgerrit has joined #openstack-security | 13:26 | |
| *** singlethink has joined #openstack-security | 13:32 | |
| *** sdake has joined #openstack-security | 13:34 | |
| *** sdake_ has joined #openstack-security | 13:37 | |
| *** sdake has quit IRC | 13:40 | |
| *** cleong has joined #openstack-security | 13:46 | |
| *** aasthadi has joined #openstack-security | 13:51 | |
| *** aastha has joined #openstack-security | 13:56 | |
| *** mvaldes has joined #openstack-security | 14:09 | |
| *** JAHoagie has joined #openstack-security | 14:12 | |
| *** JAHoagie has quit IRC | 14:13 | |
| *** nkinder has joined #openstack-security | 14:18 | |
| *** knangia has joined #openstack-security | 14:37 | |
| *** edtubill has joined #openstack-security | 14:55 | |
| openstackgerrit | Khanak Nangia proposed openstack/security-doc: Updated OSSN-0073 Added information about Horizon dashboard leaks https://review.openstack.org/357328 | 15:04 |
| *** vinaypotluri has joined #openstack-security | 15:14 | |
| *** hockeynut has joined #openstack-security | 15:16 | |
| *** salv-orlando has joined #openstack-security | 15:18 | |
| *** aasthadi has quit IRC | 15:21 | |
| *** aasthadi has joined #openstack-security | 15:21 | |
| *** salv-orlando has quit IRC | 15:25 | |
| *** sdake_ has quit IRC | 15:25 | |
| *** hockeynu_ has joined #openstack-security | 15:28 | |
| *** shohel has quit IRC | 15:28 | |
| *** hockeynut has quit IRC | 15:29 | |
| *** singlethink has quit IRC | 15:31 | |
| *** hockeynu_ has quit IRC | 15:36 | |
| *** sdake has joined #openstack-security | 15:36 | |
| *** diazjf has joined #openstack-security | 15:38 | |
| *** ibmchas has quit IRC | 15:40 | |
| *** hockeynut has joined #openstack-security | 15:40 | |
| *** salv-orlando has joined #openstack-security | 15:48 | |
| *** aasthadi has quit IRC | 15:48 | |
| *** ccneill has joined #openstack-security | 15:51 | |
| vinaypotluri | does anyone know how to join barbican threat analysis meeting ? | 15:53 |
| redrobot | vinaypotluri we'll get a google hangout going and post the link here in a few minutes | 15:53 |
| vinaypotluri | cool | 15:54 |
| *** tmcpeak has joined #openstack-security | 15:55 | |
| redrobot | vinaypotluri | 15:59 |
| redrobot | https://hangouts.google.com/call/pxpx2sxjmjcxho4zqnsvx2h7lme | 15:59 |
| *** dikonoor has joined #openstack-security | 16:01 | |
| tmcpeak | https://etherpad.openstack.org/p/barbican-threat-analysis | 16:04 |
| *** dg_____ has joined #openstack-security | 16:04 | |
| dg_____ | http://openstack-security.github.io/collaboration/2016/04/26/threat-analysis-process.html | 16:05 |
| hyakuhei | The hangout is full! | 16:05 |
| hyakuhei | Have fun suckaz! | 16:05 |
| hyakuhei | dg_____ redrobot ^^^ | 16:05 |
| tmcpeak | rob you're already connected twice | 16:07 |
| tmcpeak | hyakuhei: try again | 16:07 |
| tmcpeak | https://hangouts.google.com/call/t6cwfc2dcjh7zb637hyibjx4oee | 16:09 |
| tmcpeak | ^ | 16:09 |
| ccneill | glad we saved part of the TA for remote - definitely a learning experience | 16:10 |
| tmcpeak | ccneill: lol | 16:10 |
| hyakuhei | RAGE. | 16:10 |
| tmcpeak | dg_____: hyakuhei redrobot | 16:10 |
| vinaypotluri | aastha: https://hangouts.google.com/call/t6cwfc2dcjh7zb637hyibjx4oee | 16:11 |
| redrobot | woodster_ you might be able to join the new hangout https://hangouts.google.com/call/t6cwfc2dcjh7zb637hyibjx4oee | 16:12 |
| dg_____ | https://etherpad.openstack.org/p/barbican-threat-analysis | 16:12 |
| hyakuhei | ^^ yup | 16:13 |
| woodster_ | redrobot: no luck on that one. No biggy though, I'll follow up with the etherpad afterwards. | 16:14 |
| aastha | i am not able to join either | 16:18 |
| aastha | will check the etherpad though | 16:19 |
| *** rcernin has quit IRC | 16:29 | |
| *** hockeynu_ has joined #openstack-security | 16:32 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack/bandit: Clean imports in code https://review.openstack.org/359659 | 16:33 |
| *** hockeynut has quit IRC | 16:35 | |
| *** tesseract- has quit IRC | 16:35 | |
| *** mdong has joined #openstack-security | 16:36 | |
| *** tmcpeak1 has joined #openstack-security | 16:37 | |
| *** tmcpeak has quit IRC | 16:38 | |
| *** hockeynut has joined #openstack-security | 16:45 | |
| *** hockeynu_ has quit IRC | 16:45 | |
| *** singlethink has joined #openstack-security | 16:51 | |
| *** dg_____ has quit IRC | 17:00 | |
| *** mdong_ has joined #openstack-security | 17:02 | |
| *** mdong has quit IRC | 17:03 | |
| *** mdong_ is now known as mdong | 17:03 | |
| openstackgerrit | Michael Dong proposed openstack/syntribos: Memoized extension functions https://review.openstack.org/358870 | 17:08 |
| *** hockeynut has quit IRC | 17:10 | |
| openstackgerrit | Michael Dong proposed openstack/syntribos: Memoized extension functions https://review.openstack.org/358870 | 17:10 |
| *** hockeynut has joined #openstack-security | 17:12 | |
| *** rcernin has joined #openstack-security | 17:24 | |
| *** dikonoor has quit IRC | 17:28 | |
| ccneill | unrahul: can you mute while you guys are chatting amongst yourselves? | 17:41 |
| hyakuhei | unrahul ^^^ | 17:41 |
| unrahul | hey guys sorry forgot to mute | 17:43 |
| *** tmcpeak has joined #openstack-security | 18:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/security-doc: Updated from openstack-manuals https://review.openstack.org/360034 | 18:12 |
| *** hockeynut has quit IRC | 18:13 | |
| *** tmcpeak1 has quit IRC | 18:13 | |
| *** diazjf has quit IRC | 18:15 | |
| *** tmcpeak1 has joined #openstack-security | 18:16 | |
| *** tmcpeak has quit IRC | 18:17 | |
| openstackgerrit | Rahul U Nair proposed openstack/syntribos: Adding a script to generate README.rst from docs https://review.openstack.org/358818 | 18:17 |
| *** hyakuhei has quit IRC | 18:30 | |
| *** hyakuhei has joined #openstack-security | 18:41 | |
| *** hyakuhei has quit IRC | 18:41 | |
| *** hyakuhei has joined #openstack-security | 18:41 | |
| *** hyakuhei has quit IRC | 18:41 | |
| *** hyakuhei has joined #openstack-security | 18:41 | |
| *** sdake has quit IRC | 19:20 | |
| *** diazjf has joined #openstack-security | 19:22 | |
| openstackgerrit | Michael Dong proposed openstack/syntribos: fixed results error and failure counting https://review.openstack.org/359463 | 19:23 |
| mdong | ccneill, unrahul: The last patch I had merged broke master - it threw an exception when you keyboardinterrupt | 19:25 |
| mdong | this latest patch should fix that, can I get a review on it real quick? | 19:26 |
| mdong | https://review.openstack.org/359463 | 19:26 |
| ccneill | noticed that yesterday, I'll take a look | 19:26 |
| ccneill | mdong: looks like there's an error on line 157 now | 19:33 |
| ccneill | TypeError: print_log_path_and_stats() takes exactly 3 arguments (2 given) | 19:33 |
| mdong | d’oh | 19:37 |
| mdong | actually, just realized that this patch needs more work. Should I throw up a quick 1 linepatch that just fixes master while I work on this one or can it wait? | 19:41 |
| ccneill | I can do a separate bugfix real quick | 19:43 |
| tmcpeak1 | fixing master is always good :) | 19:43 |
| unrahul | i have a patch for it | 19:46 |
| unrahul | that i had abandoned | 19:46 |
| unrahul | I could unabandon it | 19:46 |
| unrahul | ? | 19:46 |
| unrahul | ccneill: mdong https://review.openstack.org/#/c/359472/ | 19:47 |
| mdong | oh, I didn’t see that one | 19:47 |
| ccneill | unrahul: can you unabandon a patch? if so, that's good with me | 19:48 |
| unrahul | yeah.. i pushed it....and then i was reviewing ur changes so that u are taking care of it | 19:48 |
| unrahul | yup already did | 19:48 |
| *** hockeynut has joined #openstack-security | 19:48 | |
| ccneill | wait.. | 19:49 |
| ccneill | nvm | 19:49 |
| *** fduthilleul has joined #openstack-security | 19:49 | |
| ccneill | search was being weird on gerrit | 19:49 |
| ccneill | +2'd | 19:49 |
| mdong | so I just noticed that when we’re printing out results, the result that’s being printed after every template, is actually the overall results object, containing failures across all templates | 19:50 |
| ccneill | er | 19:51 |
| ccneill | :X | 19:51 |
| mdong | that is, the result object isn’t being cleared or anything after each template, and the entire thing is being printed every time | 19:51 |
| ccneill | yeah, that's not good | 19:53 |
| mdong | there’s not really a way for us to actually clear the results object, so we could either take a subset of the results object after each template and print only that | 19:53 |
| mdong | or we could just only print the results at the end of the run | 19:53 |
| ccneill | honestly I'm okay with us not printing it after every template | 19:53 |
| mdong | yeah that’s what I’m leaning towards too | 19:54 |
| ccneill | it messes with the -o flag too | 19:54 |
| ccneill | I thini | 19:54 |
| ccneill | think* | 19:54 |
| mdong | yeah, i guess it would | 19:54 |
| ccneill | it spits out multiple json objects | 19:55 |
| ccneill | so it's not actually a valid JSON file | 19:55 |
| ccneill | :/ | 19:55 |
| ccneill | I also realized that using a URL as a key to JSON is probably not a great idea, since it makes using tools like jq hard | 19:55 |
| *** fduthilleul has left #openstack-security | 19:56 | |
| ccneill | failures could be a list, with objects inside them that have a url: key | 19:56 |
| openstackgerrit | Merged openstack/syntribos: Fixed a trivial bug in keyboard interrupt part https://review.openstack.org/359472 | 19:56 |
| ccneill | that way the hierarchy is predictable the whole way down, and you can write scripts using jq to analyze the output at the end | 19:57 |
| mdong | that would be a non-trivial change, just because of the way the json formatter is written | 19:57 |
| ccneill | yeah, I figured as much :( which is why I haven't pushed for us to fix it right now | 19:58 |
| mdong | but if you could put the schema you’re thinking into a gist, I can start looking at it | 19:58 |
| unrahul | what should be the key you are thinking of ccneill | 20:00 |
| unrahul | the template_name ? | 20:00 |
| ccneill | I'll mock something up real quick in a gist | 20:00 |
| *** hockeynut has quit IRC | 20:01 | |
| *** hockeynut has joined #openstack-security | 20:03 | |
| *** mdong has quit IRC | 20:23 | |
| ccneill | unrahul / mdong: https://gist.github.com/cneill/a511451284a0c5f33295477150bd94d4 | 20:25 |
| unrahul | ccneill this look good.. way better.. , I have a question, do we need to have some sort of key: value for failures.. so that parsing them based on some identifier would be easier..? | 20:28 |
| unrahul | each failure that is* | 20:29 |
| *** mdong has joined #openstack-security | 20:29 | |
| unrahul | also , we can add the diff/comparison of signals to the result as well.. i suppose..? | 20:30 |
| mdong | sorry, did you just post the schema? if you did, can I get the link again? | 20:31 |
| *** knangia has quit IRC | 20:31 | |
| unrahul | *adding my comments to the gist as well * | 20:31 |
| unrahul | mdong: https://gist.github.com/cneill/a511451284a0c5f33295477150bd94d4 | 20:31 |
| mdong | thanks | 20:31 |
| unrahul | +1 | 20:31 |
| ccneill | unrahul: you mean including the test number maybe? | 20:35 |
| ccneill | unrahul: +1 on signal diff | 20:35 |
| unrahul | yeah.. something like that.. so that we can identify it .. easily.. or u know.. communicate better.. "the failure ## : | 20:36 |
| unrahul | what do u think? | 20:36 |
| ccneill | I think adding test number would definitely make it easier to search based on the CLI output | 20:36 |
| unrahul | +1 ccneill | 20:37 |
| ccneill | not sure if we need a separate "failure #" too | 20:37 |
| ccneill | I think just the test ID should be sufficient | 20:37 |
| unrahul | no.. the test id should suffice | 20:37 |
| ccneill | so for the signal diff.. not sure if there's much point actually since the signals in each holder are already listed | 20:37 |
| ccneill | I guess we could but I think it's probably clear enough as it is for now | 20:38 |
| ccneill | as we add more checks/signals maybe it would be more useful to have the diff if the list gets too long | 20:38 |
| *** nkinder has quit IRC | 20:39 | |
| mdong | so, are the test IDs supposed to be unique across templates? | 20:39 |
| mdong | this is unrelated to the schema | 20:39 |
| mdong | because right now they’re not | 20:39 |
| unrahul | hmm. ,, i am wondering.. how to make it easier to parse the failures if there was too many.. | 20:40 |
| unrahul | not sure what is the ryt way | 20:40 |
| unrahul | parse it so that .. reading cli would be easier. | 20:40 |
| *** edtubill has quit IRC | 20:41 | |
| ccneill | well with jq you could just do jq ".failures[].test_id=1234" I think | 20:41 |
| *** diazjf has quit IRC | 20:41 | |
| ccneill | I think we just want to print the results one time at the end of execution instead of between every template | 20:42 |
| ccneill | makes it easier to handle writing to a file instead | 20:42 |
| unrahul | yup.. i agree.. so that the results are at one place.. all togther.. and yes as u said, makes it easier to use tools like jq | 20:43 |
| ccneill | mdong: interesting that they're not unique across templates.. I thought the way we were handling it they would be? | 20:44 |
| mdong | nah, right now they’re being reset every time run_all_tests is called, which is once per template | 20:44 |
| mdong | and they’re not being associated with the test object either | 20:44 |
| ccneill | ahhhh just noticed that | 20:44 |
| mdong | so if we wanna use those to index into the results output, we’d need to change that | 20:45 |
| ccneill | thought the template loop was happening inside run_all_tests | 20:45 |
| ccneill | probably good to associate with the test object | 20:45 |
| ccneill | so there should be like a | 20:46 |
| ccneill | last_test_id= in run(), you pass that to run_all_tests, and return the final test_id at the end | 20:47 |
| ccneill | or.. | 20:47 |
| ccneill | nah, scratch that | 20:47 |
| ccneill | put current_test_id=1000 as a class variable in Runner, associate the current value with each test object, then increment it | 20:50 |
| ccneill | https://github.com/openstack/syntribos/blob/master/syntribos/runner.py#L205 | 20:50 |
| ccneill | so that would be cls.current_test_id += 5 | 20:50 |
| ccneill | https://github.com/openstack/syntribos/blob/master/syntribos/runner.py#L202 | 20:50 |
| ccneill | that would be removed | 20:51 |
| ccneill | and you'd add test_class.test_id = cls.current_test_id before line 205 | 20:52 |
| ccneill | ¯\_(ツ)_/¯ I think that sounds sane? | 20:52 |
| mdong | yeah, looks about right to me | 20:52 |
| unrahul | +1 | 20:53 |
| *** austin987 has joined #openstack-security | 20:57 | |
| *** diazjf has joined #openstack-security | 21:02 | |
| *** edtubill has joined #openstack-security | 21:10 | |
| *** salv-orl_ has joined #openstack-security | 21:17 | |
| *** salv-orlando has quit IRC | 21:20 | |
| mdong | so, er, associating the test_id in the results is gonna be a somewhat involved task | 21:22 |
| mdong | because the json formatter flattens the list of [(test, [issues])] into a a list of [issues] | 21:23 |
| mdong | that is, it takes result.failures, which has all the information about the test, and then just basically throws away the test itself, and only operates on the issues | 21:24 |
| mdong | this is done because different test objects can raise the exact same issue, and we wanted to remove the redundancy in the output | 21:25 |
| ccneill | ah | 21:26 |
| mdong | point is, there’s not a 1 to 1 matching of issues and tests that generate them, so do we print multiple test_id’s for every issue? | 21:26 |
| ccneill | hmm | 21:26 |
| ccneill | test_ids? | 21:26 |
| ccneill | i.e., we accumulate the list of test_ids and append them to the output | 21:26 |
| mdong | I mean, is that useful? | 21:26 |
| ccneill | sure, that way you can still use something like jq to filter by test_id | 21:27 |
| ccneill | would just have to check "contains" instead of "equals" | 21:27 |
| ccneill | might get a little overwhelming in terms of human readability | 21:28 |
| mdong | yeah that’s what I’m afraid of | 21:28 |
| ccneill | hmm.. :/ | 21:28 |
| mdong | also, it would require either we change how the formatter works entirely, or have test_ids as a class parameter for each issue, which has its own problems | 21:29 |
| ccneill | I think having that searchability is definitely desirable from a | 21:30 |
| mdong | namely, if two issues have the same data, we want to just consider them as the same issue. Right now the duplicate issue is just thrown away | 21:30 |
| ccneill | right | 21:30 |
| mdong | but if we associate test_ids with each issue, we’d have to merge them | 21:30 |
| ccneill | I think as the one piece of information that ties together the CLI output, the debug log, and potentially the results log, the test_id is useful enough to warrant the change though | 21:31 |
| *** vinaypotluri has quit IRC | 21:31 | |
| ccneill | makes correlating things much easier | 21:32 |
| ccneill | I was having a hard time during our keystone testing mapping findings from the results log to the debug log | 21:32 |
| ccneill | especially after losing the CLI output after running the tool multiple times, `cat`ing the results log, etc. | 21:33 |
| mdong | yeah, fair enough | 21:33 |
| ccneill | it shouldn't be THAT many, right? I mean, less than the number of payloads at least | 21:33 |
| ccneill | ¯\_(ツ)_/¯ | 21:33 |
| ccneill | we'll see | 21:33 |
| ccneill | worst case scenario, we can add some documentation explaining how to filter the results a bit with jq so it's not so hard to read manually | 21:35 |
| *** nkinder has joined #openstack-security | 21:35 | |
| *** mvaldes has quit IRC | 21:35 | |
| mdong | so, do we want a unique failure_id? | 21:36 |
| mdong | because we can’t use test_ids as a unique identifier for an issue | 21:36 |
| mdong | ccneill unrahul ^ | 21:37 |
| ccneill | hmm.. | 21:37 |
| unrahul | eeh. would it be too many ids..? | 21:38 |
| ccneill | maybe we print out failure_ids after each template | 21:38 |
| ccneill | so CLI output has test_id (for searching in debug log) and failure_id (for searching results log) | 21:39 |
| ccneill | not sure how intuitive that is though.. | 21:39 |
| ccneill | I think we want both pieces of info though if we go that route | 21:39 |
| ccneill | like, if we JUST have failure IDs, it becomes hard to use the debug log to debug non-failed requests | 21:40 |
| unrahul | or should it be some string, based on context, like test type or something.. i am not sure. | 21:40 |
| ccneill | but it's more natural for us to use a single failure_id in the results log than tacking all the test_ids on there | 21:40 |
| ccneill | we could put the failure IDs in the debug log too | 21:40 |
| ccneill | so then, you run the tool, check results.json, see the failure_id you want to investigate, search the debug log for that failure_id | 21:42 |
| ccneill | hmm.. I guess the test id really isn't THAT useful if we have the failrue id | 21:43 |
| ccneill | failure* | 21:43 |
| *** austin987 has quit IRC | 21:43 | |
| unrahul | :/ may be we can avoid failure ids.. and keep thigns simple. | 21:45 |
| mdong | if I had to choose, I’d just go with failure_ids only | 21:45 |
| mdong | but is there much value in having failure_id and test_ids? | 21:45 |
| ccneill | yeah, I'm leaning that way.. if all we get in the results log is a test_id, it's harder to debug that particular failure | 21:46 |
| *** vinaypotluri has joined #openstack-security | 21:46 | |
| unrahul | yeah .. thats a point.. | 21:46 |
| ccneill | only utility for test_id if we have failure_ids in all 3 locations (cli, results, debug), is if you're watching the CLI output and you want to debug a particular test | 21:46 |
| ccneill | but since we're talking about only using failure_id in the results output and not adding test_id, it's useless as soon as you lose the CLI output | 21:46 |
| ccneill | so I'd be okay with us just using failure_id/issue_id/whatever for now | 21:47 |
| ccneill | and seeing how that serves us | 21:47 |
| *** cleong has quit IRC | 21:48 | |
| *** sdake has joined #openstack-security | 21:48 | |
| mdong | sounds good to me | 21:48 |
| unrahul | +! | 21:48 |
| ccneill | kewl | 21:48 |
| ccneill | hmm.. how to generate the failure_id though | 21:49 |
| ccneill | since the aggregation happens in the formatter | 21:49 |
| ccneill | I'm wondering if we should do the aggregation outside the formatter so that we don't have two separate states | 21:50 |
| ccneill | like, why build up the giant "results" object only to clobber it | 21:50 |
| *** sdake_ has joined #openstack-security | 21:50 | |
| mdong | as in, why have a results object that holds different information than what the json formatter ultimately operates on? | 21:51 |
| ccneill | right | 21:52 |
| unrahul | should we discuss more on this on the standup or during our weekly meeting, i am not able to put all these into perspective.. :o | 21:52 |
| ccneill | it seems like that adds an additional level of complexity, making sure that you report the right stats when you "squish" multiple issues into one | 21:52 |
| ccneill | unrahul: I'm basically proposing that we don't do anything in formatters beyond pretty-print the information that is included in the global "results" object | 21:53 |
| mdong | so we can’t get away from the results object, simply because unittest requires it. Originally, I was aggregating everything using the results object itself, but unittest forces certain requirements to the data it contains | 21:53 |
| *** nkinder has quit IRC | 21:53 | |
| ccneill | arrrrrghhhh | 21:53 |
| mdong | namely, the results object holds tests, but we care about issues | 21:53 |
| * ccneill shakes fist at unittest | 21:53 | |
| ccneill | we're doing all the stats calculations/etc. ourselves at this point.. | 21:54 |
| unrahul | :| | 21:54 |
| ccneill | wonder what, exactly, we're getting from unittest | 21:54 |
| mdong | so instead of trying to wrangle the difference between tests and issues in the same object, I had the results care about tests, and the formatter care about issues | 21:54 |
| *** sdake has quit IRC | 21:54 | |
| unrahul | will this lead to us removing unittest as a whole.? | 21:54 |
| mdong | or rather I guess originally there wasn’t a formatter at all | 21:54 |
| ccneill | unrahul: maybe, but probably not right now lol | 21:55 |
| mdong | but point is, yeah, blame unittest | 21:55 |
| ccneill | sigh.. | 21:55 |
| * ccneill just wants to start over sometimes | 21:55 | |
| ccneill | lol | 21:55 |
| ccneill | let me look at the results/etc. code for a minute, I haven't worked with that stuff in a bit | 21:56 |
| *** mdong has quit IRC | 21:56 | |
| unrahul | :D.. feels like how karver was called in to fix something.. and bascially the entire software stack was rewritten in silicon valley | 21:56 |
| ccneill | you just start pulling on one thread... :P | 21:57 |
| *** mdong has joined #openstack-security | 21:57 | |
| mdong | woah, just got kicked off IRC | 21:58 |
| ccneill | o_O | 21:58 |
| ccneill | weird | 21:58 |
| unrahul | hehee.. >> | 21:58 |
| mdong | idk if my messages went through, so I’ll just repost them | 21:58 |
| mdong | mdong: though, with that being said, if we just decide we don’t care about holding onto the test objects at all | 21:58 |
| mdong | [4:57pm] mdong: and we really don’t use the test objects themselves once they’ve been added to the result object | 21:58 |
| mdong | [4:57pm] mdong: at least I don’t think | 21:59 |
| mdong | [4:57pm] mdong: we could pretty easily modify the results object to just hold test.failures instead of (test, test.failures) | 21:59 |
| ccneill | hmm | 21:59 |
| ccneill | yeah, if we're not using the test object after that, I'm in favor of tossing it | 22:00 |
| ccneill | buuut.. that might be our mechanism for doing stuff like request pipelining | 22:00 |
| ccneill | v_v | 22:00 |
| ccneill | maybe not | 22:00 |
| ccneill | actually, no, screw that - we'll figure out some other way to handle that lol | 22:00 |
| mdong | yeah…if we depend on unittest’s results object to do request pipelining, then we should find another way to do request pipelining | 22:01 |
| *** sdake_ has quit IRC | 22:01 | |
| ccneill | yeah lol | 22:01 |
| unrahul | :D | 22:03 |
| *** sdake has joined #openstack-security | 22:03 | |
| *** tmcpeak1 has quit IRC | 22:05 | |
| mdong | so we could do the aggregating within the result object itself, but it would require basically throwing away most of the data unittest wants to shove in there | 22:05 |
| mdong | so addFailure(self, test, err) - we throw away the test, throw away the err, and just keep test.failures | 22:05 |
| ccneill | looking to see if we end up doing anything with test | 22:07 |
| *** mdong has quit IRC | 22:07 | |
| ccneill | we already throw err away in addFailure it looks like | 22:07 |
| ccneill | not sure what err is | 22:07 |
| ccneill | ah | 22:08 |
| *** mdong has joined #openstack-security | 22:08 | |
| ccneill | stacktrace | 22:08 |
| mdong | er, disconnected from IRC again | 22:08 |
| mdong | missed what you said just now | 22:08 |
| ccneill | we already get rid of err | 22:08 |
| ccneill | https://docs.python.org/3.1/library/unittest.html#unittest.TestResult.addFailure | 22:08 |
| mdong | yeah | 22:08 |
| ccneill | it's stacktrace data from an exception | 22:08 |
| ccneill | but that exception in our case is an AssertionError every time (I think) | 22:09 |
| ccneill | if it's actually a failrue, otherwise it's an error | 22:09 |
| ccneill | right? | 22:09 |
| mdong | yeah | 22:09 |
| ccneill | hm | 22:10 |
| ccneill | looks like we use self.results.getDescription(test) on line 39 of the json formatter... | 22:10 |
| ccneill | not sure if we could do that another way.. | 22:10 |
| ccneill | (for errors) | 22:11 |
| ccneill | we already throw tests away for failures | 22:11 |
| mdong | basically that’s responsible for this line in the output | 22:11 |
| mdong | "test": "tearDownClass (syntribos.tests.fuzz.buffer_overflow.domains_get.template_BUFFER_OVERFLOW_HEADERS_buffer-overflow.txt_str3_model2)" | 22:11 |
| ccneill | right | 22:11 |
| mdong | we could just do like test.__name__ | 22:12 |
| ccneill | we could just put that string in the issue | 22:12 |
| mdong | or really | 22:12 |
| mdong | we could even leave errors as it is | 22:12 |
| mdong | cause we only care about aggregating the issues | 22:12 |
| ccneill | right | 22:12 |
| mdong | so originally, for the very first implementation, the aggregation was actually done in the result | 22:13 |
| mdong | was there a reason we moved it to the json formatter that we might be forgetting? | 22:13 |
| ccneill | I think that was before my time :X | 22:13 |
| ccneill | at least before I got heavily involved | 22:13 |
| ccneill | I think it makes sense from a general best-practices perspective though | 22:13 |
| mdong | well, I can’t remember and I wrote the thing lol | 22:14 |
| ccneill | "formatter" should not mean "thing that creates an entirely new structure" but "thing that tries to meaningfully output existing structure" | 22:14 |
| ccneill | in my mind | 22:14 |
| ccneill | it *may* have had something to do with stats aggregation | 22:14 |
| ccneill | using unittest's stats | 22:14 |
| ccneill | but we track all that ourselves now I believe | 22:14 |
| ccneill | I could be totally wrong, but that's all I can think of | 22:15 |
| mdong | ok, I can take a crack at this while I’m mucking about with the schema anyway | 22:16 |
| *** dave-mccowan has quit IRC | 22:19 | |
| ccneill | good news: it will make writing formatters easier O:-0 | 22:20 |
| ccneill | O:-) | 22:20 |
| *** rb2016 has joined #openstack-security | 22:24 | |
| *** markvoelker has joined #openstack-security | 22:29 | |
| openstackgerrit | Michael Dong proposed openstack/syntribos: changes to runner and result https://review.openstack.org/359463 | 22:29 |
| *** aasthadi has joined #openstack-security | 22:30 | |
| openstackgerrit | Michael Dong proposed openstack/syntribos: Added config file improvements https://review.openstack.org/358856 | 22:32 |
| *** jass93 has joined #openstack-security | 22:33 | |
| *** hockeynut has quit IRC | 22:35 | |
| openstackgerrit | Michael Dong proposed openstack/syntribos: Added config file improvements https://review.openstack.org/358856 | 22:39 |
| openstackgerrit | Michael Dong proposed openstack/syntribos: changes to runner and result https://review.openstack.org/359463 | 22:40 |
| mdong | ccneill: can you put this discussion on that etherpad you had up for syntribos irc discussion? | 22:41 |
| ccneill | yep | 22:41 |
| ccneill | good call | 22:41 |
| mdong | anyway, I’m out | 22:44 |
| mdong | see yall | 22:44 |
| *** mdong has quit IRC | 22:44 | |
| *** singlethink has quit IRC | 22:51 | |
| *** aasthadi has quit IRC | 22:51 | |
| *** rcernin has quit IRC | 22:53 | |
| *** aasthadi has joined #openstack-security | 22:54 | |
| *** bigdogstl has joined #openstack-security | 22:54 | |
| *** zigo has quit IRC | 22:56 | |
| *** sdake has quit IRC | 22:57 | |
| *** zigo has joined #openstack-security | 22:57 | |
| *** diazjf has quit IRC | 22:57 | |
| *** bigdogstl has quit IRC | 22:59 | |
| *** elmiko is now known as _elmiko | 23:03 | |
| *** markvoelker has quit IRC | 23:04 | |
| *** sdake has joined #openstack-security | 23:07 | |
| openstackgerrit | Rahul U Nair proposed openstack/syntribos: Adding a script to generate README.rst from docs https://review.openstack.org/358818 | 23:14 |
| *** aasthadi has quit IRC | 23:15 | |
| openstackgerrit | Rahul U Nair proposed openstack/syntribos: Standardizing the way we diff signals https://review.openstack.org/349403 | 23:16 |
| unrahul | ccneill: I have made the changes for the two patches.. and have uploaded a new one | 23:17 |
| ccneill | cool, I'll check them out | 23:17 |
| *** edmondsw has quit IRC | 23:20 | |
| ccneill | +1'd both | 23:21 |
| unrahul | thanks charles.. | 23:21 |
| unrahul | I am working on the sanitzing part.. probably will push a patch today.. | 23:21 |
| ccneill | holy crap, my "excluded tests" CR is going on 30 hours now | 23:21 |
| unrahul | whats the deal ryt | 23:22 |
| unrahul | is not like next week is the summit | 23:22 |
| unrahul | wonder if they had reduced the number of instances processing the jobs | 23:23 |
| ccneill | I think it was because we did +1 and +1worflow at the same time, then +2 | 23:23 |
| unrahul | oh.. hmm... that is kinda wierd | 23:23 |
| unrahul | i gave a workflow now | 23:23 |
| unrahul | lets see if it is pushed now | 23:23 |
| unrahul | vinaypotluri: can you please review the patches 349403 and 358818 | 23:24 |
| openstackgerrit | Merged openstack/syntribos: Fixes a bug in "excluded tests" https://review.openstack.org/353784 | 23:25 |
| unrahul | yay!! | 23:26 |
| unrahul | got merged | 23:26 |
| unrahul | ccneill: | 23:26 |
| ccneill | boom! | 23:26 |
| ccneill | gonna try to finish up my config-dir/config-file patch today | 23:27 |
| unrahul | +1 | 23:27 |
| ccneill | can someone sum up the topics in the discussion I just added to this etherpad | 23:27 |
| ccneill | https://etherpad.openstack.org/p/syntribos-discussion | 23:27 |
| ccneill | eh, I'll come back to it later | 23:30 |
| *** aasthadi has joined #openstack-security | 23:30 | |
| *** hockeynut has joined #openstack-security | 23:30 | |
| ccneill | if everyone's busy | 23:30 |
| *** aasthadi has quit IRC | 23:40 | |
| openstackgerrit | Merged openstack/syntribos: Standardizing the way we diff signals https://review.openstack.org/349403 | 23:42 |
| openstackgerrit | Merged openstack/syntribos: Adding a script to generate README.rst from docs https://review.openstack.org/358818 | 23:42 |
| ccneill | boooom | 23:42 |
| unrahul | whoa! | 23:43 |
| unrahul | hey ccneill I have summed up the discussion as I understood in the etherpad.. when you get time kindly check it out and update.. | 23:43 |
| ccneill | cool, thanks unrahul | 23:51 |
| ccneill | looks good | 23:51 |
| openstackgerrit | Charles Neill proposed openstack/syntribos: Refresh readme https://review.openstack.org/360123 | 23:56 |
| ccneill | unrahul: just realized, I don't think the README has been updated since we made the modifications to the source rst files.. :S | 23:56 |
| ccneill | also looks like the links in the "Details" section aren't working on Github :/ https://github.com/openstack/syntribos | 23:57 |
| ccneill | I'll add to that CR I just submitted | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!