00:10:51 #startmeeting CongressTeamMeeting 00:10:52 Meeting started Thu Mar 10 00:10:51 2016 UTC and is due to finish in 60 minutes. The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:10:53 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 00:10:56 The meeting name has been set to 'congressteammeeting' 00:11:19 thinrichs: I tested congress for Mitaka release and accepted my presentation. 00:11:38 https://www.openstack.org/summit/austin-2016/summit-schedule/events/7199 00:11:53 that's highlight. 00:12:20 Congratulations! 00:12:56 I'm looking forward to it. 00:13:31 The title will be slightly changed because of some reasons, but the topic of it won't be changed. 00:14:12 While we're on the topic of Austin, I requested 3 working rooms just like at the last summit. 00:14:29 But there are 11 more teams requesting space this summit, so we'll see if we can get all those rooms. 00:14:43 There's supposedly more room in Austin than Tokyo. 00:15:26 ramineni_: want to give a status update? 00:16:12 thinrichs: yes 00:16:27 thinrichs: I also did some testing around, and raised some patches for bugs reported 00:17:18 ramineni_: any noteworthy bugs? 00:17:24 thinrichs: now our tempest should be stable as propably all the outstanding tempest changes are merged now 00:17:29 masahito: same question to you 00:18:19 ramineni_: great! Thanks for keeping on top of that. Gate failures due to tempest changes can really hurt the team's overall productivity. But you've kept us moving along! 00:18:21 thinrichs: no, didnt find any critical ones as such 00:19:52 Moving on then. 00:19:56 ekcs: progress report? 00:20:14 Been doing testing. I encountered 500 errors on policy creation & deletion, I couldn't reproduce them once I restarted congress. 00:20:14 Could be a database connection issue. I want to investigate further at some point, but probably not a priority right now. 00:21:34 No clue as to how to replicate? That sounds like it could be a race condition. 00:22:01 We changed around quite a bit of code at the API layer, so there could easily be a subtle bug. 00:22:15 I don’t think it’s race condition because it was very consistent. Until I restarted congress then it’s not there. 00:22:15 Did you find the problem in the logs so we know where the error happened? 00:22:22 Yes. 00:22:42 I see the exception and stack trace in logs. 00:23:34 The bugs are here if anyone interested: https://bugs.launchpad.net/bugs/1554707 00:23:34 Launchpad bug 1554707 in congress "500 when deleting policy" [Undecided,Invalid] 00:23:42 https://bugs.launchpad.net/bugs/1554712 00:23:43 Launchpad bug 1554712 in congress "create policy fails" [Undecided,Invalid] 00:25:04 The first one looks like potentially a real problem. Like a type mismatch. 00:25:39 Trying to figure out why that would disappear after a restart 00:26:23 Did you try inserting/deleting elements into the database directly to see if you could replicate? 00:26:24 Second one (create) I can see how that would disappear because db connection re-established and synchronize no longer fails. 00:26:41 first one (delete) I don’t understand either why it would go away. 00:27:13 thinrichs: no. Not sure what you’re suggesting. 00:28:04 Both the 500s are caused by the same error: that event.target is a string but is expected to be an object. 00:28:05 thinrichs: say create policy. then delete it from db. then delete policy from congress? 00:29:20 thinrichs: Yup. 00:29:30 I can dig further if that’s desirable. 00:29:32 ekcs: these feel like real bugs (probably the same real bug). 00:30:00 ekcs: I'll mark these as critical so we make sure to figure out what's happening 00:30:10 ekcs: it'd be great if you could dig into it a bit. 00:30:14 got it. 00:30:50 Maybe even just manually looking through the code path for create/delete policy to figure out how that event.target might not be an object. 00:31:52 ok 00:32:08 ekcs: off the top of my head, I'm surprised there's any mention of events when we are creating/deleting policies 00:32:32 ekcs: definitely ping me if you want help 00:32:52 ok got it. 00:33:18 I'll do a quick update. 00:33:44 Worked a bit on one of the bugs ramineni_ found 00:33:54 That code seems more or less ready... 00:34:16 At least for more review 00:34:20 #link https://review.openstack.org/#/c/289650/ 00:34:44 Also last week I did the release of Mitaka3 for both the server and client. 00:35:28 Been waiting to do a round of testing til this first batch of bug fixes goes in. 00:35:53 I think that's about it. 00:36:23 Let's open up for discussion. 00:36:26 #topic open discussion 00:36:31 Anyone have anything? 00:36:34 I have a quick update 00:36:51 I've been spending most of my time getting the OPNFV Brahmaputra release out the door, and automating the Congress install on it. 00:36:54 bryan_att: meant to ask if you had resolved that glacne problem 00:37:10 I sent logs about the issue I reported about glance images not showing up in the glance table. 00:37:33 In my Ravello deploy the same issue did not occur. I will try to repeat it on my NUC deploy using the JOID and Apex OPNFV installers. Also the list of issues I reported earlier will be re-tested and if they are still there, send to the list individually. 00:37:52 (I have a working install of OPNFV B on Ravello with Congress and my test webapp for it. I'll be using that for demos and testing while I'm at ONS next week.) 00:38:07 My Congress install script is now totally automated, though still in bash. I'm getting Puppet training so I can implement it in Puppet. That will be the common install tool I will focus on across OPNFV installer projects. 00:38:13 that's about it 00:38:20 bryan_att: Lots of progress! 00:38:31 bryan_att: I saw your email with the logs about glance. 00:38:42 masahito, ramineni_: did either of you get a chance to look at the logs? 00:38:59 Nothing stood out to me as pointing toward a solution. 00:39:10 if it reappears I will send additional logs 00:39:23 bryan_att: that'd be great! 00:39:32 I took a glance the log. 00:39:56 It looks like the client works well. 00:40:13 yes, just doesn't send any image rows! 00:40:24 Everything but that. :) 00:40:36 looks like sync didnt happen yet? 00:41:04 I tried the test after a few hours and the same result 00:41:14 That could easily do it. But I thought we would pull the data immediately if we hadn't already pulled it. 00:42:01 bryan_att: ya, did you see any errors in congress log while polling data? 00:42:19 no, none that stood out but I will look closer 00:42:41 ohok 00:42:58 The #1 unique thing here is that I am running Congress in an LXC container on the main OpenStack controller node. 00:43:11 Off the top of my head I don't remember this: do we have the --trace option available for the row list command? 00:43:37 That would at least print out the search it's trying to do. 00:43:50 Though maybe for a basic datasource list query that won't help much. 00:44:04 Oh, but I just remembered we have a writeup on how to debug these kinds of problems… 00:44:18 I'll try that (--trace) also. 00:44:46 #link https://congress.readthedocs.org/en/latest/troubleshooting.html#datasource-troubleshooting 00:44:51 sorry for taking so much time - thanks for the ideas and help! 00:44:52 That's for the datasource one in particular. 00:45:19 bryan_att: our pleasure. It's wonderful getting feedback from real users!! 00:45:46 trying to get there... 00:46:37 Any other topics for discussion? 00:46:57 thinrichs: Back to the event.target, do you know if they are meant to be strings or objects or either? 00:46:58 I see this comment in agnostic: 00:47:00 def _update_obj(self, events, theory_string): 00:47:01 """Apply events. 00:47:03 Checks if applying EVENTS is permitted and if not 00:47:04 returns a list of errors. If it is permitted, it 00:47:06 applies it and then returns a list of changes. 00:47:07 In both cases, the return is a 2-tuple (if-permitted, list). 00:47:08 Note: All event.target fields are the NAMES of theories, not 00:47:09 theory objects. theory_string is the default theory. 00:47:10 """ 00:48:35 My best guess right now is they’re always meant to be strings. the failure occors because of a simple mistake of trying to use it as an object. But the line is reached so rarely. 00:48:37 I imagine the comment is correct: that event.target is a string. Though quite possibly later we destructively modify that field and turn it into an object. 00:49:20 ekcs: would make sense. What I don't understand is why update_obj is called during a create_policy. 00:49:34 reached so rarely because policy creation/teletion doesn’t lead to disabled events normally. 00:49:34 I wonder if there's a cascading delete: delete the policy results in deleting all the rules in the policy. 00:49:41 ok that’s helpufl. iall look forther. 00:50:00 Last thoughts? 00:51:14 thinrichs: another quci question. 00:51:50 ekcs: ok 00:52:19 which code section is responsible for ordering the evaluation of rules so that eg negated literals are only evaluated when ground. 00:52:58 is it in topdown? 00:53:34 Definitely not topdown. We do it statically. 00:53:49 I think it's in agnostic.py:Runtime._update_obj_datalog 00:53:58 You'll see the update_would_cause_errors 00:54:16 hmm ok. thanks. 00:54:36 Actually… it's the update() call in that function, 00:54:43 which is implemented in nonrecursive.py, 00:54:56 which then calls compile.reorder_for_safety 00:54:57 ok. 00:55:57 Thanks all! 00:56:00 #endmeeting