15:02:00 <jimbaker> #startmeeting craton
15:02:01 <openstack> Meeting started Mon Nov 14 15:02:00 2016 UTC and is due to finish in 60 minutes.  The chair is jimbaker. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:02 <vikasc> sorry jimbaker
15:02:04 <openstack> The meeting name has been set to 'craton'
15:02:12 <jimbaker> vikasc, np
15:02:23 <sigmavirus> Good morning, all!
15:02:36 <palendae> o/
15:02:39 <jimbaker> sigmavirus, hi, thanks for a good conversation on #craton about cachine
15:02:41 <jimbaker> caching
15:03:33 <jimbaker> we can spend a few minutes this morning on this discussion, although it's of course fairly openended from the server perspective
15:03:43 <jimbaker> but from the client, it's straightforward
15:04:34 <palendae> Looks like. Invalidating will be the biggest concern now, but that's easy ;)
15:04:37 <jimbaker> palendae, thanks for joining!
15:05:08 <jimbaker> palendae, it's only one of the so-called hardest problems in computer science
15:05:14 <palendae> jimbaker: ;)
15:05:17 <sigmavirus> palendae: so, i'm going to work on adding caching headers to the server
15:05:23 <jimbaker> palendae, http://martinfowler.com/bliki/TwoHardThings.html
15:05:30 <palendae> jimbaker: I know, I was making a joke
15:05:34 <sigmavirus> CacheControl will then be used on the client side and will take care of invalidating thing sintelligently (as it does)
15:05:58 <palendae> sigmavirus: Ok, cool
15:06:08 <jimbaker> yeah, i love the joke. especially the off by one part
15:06:41 <jimbaker> but, hey, i am sort of in this field a bit too deep ;)
15:06:47 <jimbaker> it's rubbed off
15:07:09 <jimbaker> sulo, around?
15:07:17 <jimbaker> syed__, hi
15:07:24 <sulo> o/
15:07:30 <sulo> hi everyone
15:07:40 <jimbaker> ok, looks like we have our quorum
15:07:58 <syed__> Hi  guys
15:08:32 <sulo> hi syed__
15:08:45 <jimbaker> first, it's good to be back. i guess i have missed the last 3 of these meetings, between travel and vacation
15:09:41 <jimbaker> so let's get started with a quick standup of what we plan to be working on this week; and especially review progress
15:10:00 <jimbaker> sigmavirus, you want to go first?
15:10:06 <sigmavirus> Sure
15:10:36 <sigmavirus> I'm going to having Flask provide proper cache-control headers to users, once that's in place, I'll rework my existing client work to rely on the much better (and more appropriate) cachecontrol library
15:10:56 <sigmavirus> I'll start a review to add that library to global-requirements now so that we can maybe have that resolved by the time the client work is ready to be done
15:11:12 <sigmavirus> </update>
15:11:47 <jimbaker> sigmavirus, +1. what specific logic are you planning for the server side - just expiry time for the time being?
15:13:07 <sigmavirus> jimbaker: no, thinking of providing ETags
15:13:13 <sigmavirus> Those are more reliable in my experience
15:13:23 <sulo> probably add mod time
15:13:41 <sigmavirus> sulo: you mean, "Last-Modified" so we can use "If-Modified-Since"?
15:13:49 <sulo> yeah
15:14:05 <sulo> etag woudl be difficult to do/ more server work ?
15:14:12 <jimbaker> sigmavirus, sure, i think we would certainly prefer etags. just a concern of whether we can get quick/easy versioning out of what we have already
15:14:37 <jimbaker> vs something simple like TTL based approaches
15:15:14 <sulo> probalby the best etag would be to add some sort of hash but then we'd have to compue that ...
15:15:59 <sulo> ttl bases sutff probably would be bad ?
15:16:00 <sigmavirus> we can start with Last-Modified, if that's what you all want
15:16:05 <jimbaker> sulo, i started looking at a range of options that exist, such as NDB and its memcached support - supposedly we do get a version, not certain what it would mean for making it easy going forward
15:16:19 <sigmavirus> So, to be clear, I'm not doing caching between the DB and the API
15:16:25 <sulo> sigmavirus: +1
15:16:27 <sigmavirus> That's far more work than is necessary here
15:16:33 <jimbaker> sigmavirus, yeah, agreed here
15:16:39 <sigmavirus> I'm only providing cache-control headers as defined in the RFC to the clients
15:16:41 <sulo> yeah that should be something different
15:16:47 <sulo> yeah agreed
15:17:13 <jimbaker> we want to get the client to follow the server's direction - we can improve the server for its consistency with the database over time - a number of options
15:17:38 <jimbaker> sigmavirus, ok, so that sounds great
15:17:46 <palendae> From my perspective, I'm not too concerned about the underlying implementation, just that there are fewer roundtrips
15:18:19 <jimbaker> palendae, agreed - we should not get too worked up about server performance just yet :)
15:18:59 <jimbaker> there are so many things we can do to improve that, and we can take advantage of a variety of approaches for a specific setup
15:19:28 <jimbaker> sulo, what is this week looking like for you?
15:20:00 <sulo> So i just started looking at functional testing
15:20:28 <sulo> i have a bunch of reviews in place (i need to update the filter pr .. which i will do later today)
15:20:46 <sulo> but will probalby get the functional testing going this week
15:21:17 <jimbaker> sulo, sounds good about the filter update. i think that's pretty much ironed out in those comments, in terms of removing in support + using ?vars=x:y
15:21:41 <sulo> yeah, i think this makes sense
15:22:07 <jimbaker> sulo, i expect we we will want to use some variant on jsonpath for the equiv of the in operator; and probably just pass complex queries in the request
15:22:21 <jimbaker> but deferring means we can figure that out :)
15:22:35 <sulo> yeah agreed, we can do that as impovement work ..
15:23:05 <jimbaker> sulo, re functional testing - so this will be using docker-py?
15:23:28 <sulo> yeah i think
15:23:34 <sulo> i think so
15:23:50 <jimbaker> sulo, ack, follows what we have been doing manually, so it seems to make sense
15:24:27 <sulo> i have a question
15:24:34 <jimbaker> sulo, did you write up your barcelona observations for openstack-dev? (i still need to do mine, although i did speak about it for an OSIC meeting)
15:24:42 <jimbaker> sulo, ?
15:24:48 <sulo> sigmavirus might know .. what happens on infra side if we choose to use docker based tests
15:25:06 <sulo> jimbaker: i have not
15:25:15 <sulo> i can do that if required
15:25:22 <syed__> Just a general question
15:25:38 <syed__> Dont we need tempest aswell? Since eventually this gets integrated in openstack
15:25:52 <jimbaker> sulo, no worries, it was in the action items from last week that sigmavirus assigned you - but why don't i take responsibility for writing that email for us jointly
15:25:56 <jimbaker> sort of my job :)
15:26:42 <jimbaker> syed__, it's a great question - our perspective has been no, because we are not exactly like other openstack services
15:26:44 <sulo> syed__: i dont think so ...but i am pinging someone to find out
15:26:44 <sigmavirus> sulo: I don't know what happens on the infra side, but I think you have to do most of the orchestration yourself
15:27:04 <sulo> sigmavirus: something like 3party thing ?
15:27:19 <sigmavirus> syed__: so tempest is also a test framework, we can use tempest's libraries to craft our own tempest like tests, and that wouldn't be a bad idea
15:27:33 <sigmavirus> sulo: no, more like running specific commands to run the functional tests on a VM provided by infra
15:27:43 <sulo> sigmavirus: ah that we can do
15:27:48 <jimbaker> sigmavirus, thanks for that clarification
15:27:53 <sulo> sigmavirus: our vm is container here ..
15:28:07 <sulo> sigmavirus: do they allow for us to sipin up docker during out test ?
15:28:19 <sigmavirus> sulo: I think, yes
15:28:24 <sulo> sigmavirus: cool, thx
15:28:26 <jimbaker> sulo, which should be quite lightweight - just really requires that docker daemon is available
15:28:32 <sulo> yeah
15:28:39 <sulo> pretty much thats all our req is for that
15:28:44 <syed__> Thanks sigmavirus
15:30:09 <palendae> sulo: They definitely allow spinning up lxc containers, so Docker's probably allow
15:30:11 <palendae> allowed*
15:30:18 <jimbaker> syed__, the one piece where we see integration with openstack that we should test explicitly here: maybe testing against AIO builds with ansible (but that's really integration testing); and perhaps testing against some prebuilt nova setup when we implement virtualized variables
15:30:21 <sulo> palendae: cool thats good to know
15:30:25 <sigmavirus> Idk how kolla would test on openstack CI otherwise
15:30:26 <palendae> Guessing Kolla spins up docker instances too
15:30:42 <sulo> sigmavirus: palendae: yeah good point
15:30:50 <jimbaker> yeah, this is pretty essential for actually testing openstack type stuff. one would think :)
15:31:06 <sigmavirus> jimbaker: docker? yeah, no
15:31:24 <sigmavirus> Craton is the only project that has a dockerfile and uses it as part of development afaik
15:31:29 <jimbaker> just a question of how flexible the specifics are, and then how much we can abuse :)
15:31:41 <jimbaker> sigmavirus, so that's the abuse part
15:32:52 <jimbaker> anyway, we will find out more. i'm not terribly worried about getting a docker daemon setup, given that we should be able to get a VM in the testing process. it just might be easier
15:33:02 <jimbaker> and something we can just build oin
15:33:03 <jimbaker> on
15:33:29 <sulo> jimbaker: what
15:33:49 <sulo> which vm ?
15:33:51 <jimbaker> sulo, with respect to running our tests on openstack infra
15:34:32 <jimbaker> sulo, please ignore these concerns for now - i don't see any impact on what we are trying to do in terms of a dockerized way to do functional testing
15:35:35 <jimbaker> sulo, sound good?
15:35:38 <sulo> yes
15:35:51 <sulo> thats all from my side
15:36:02 <jimbaker> sulo, thanks!
15:36:48 <jimbaker> palendae, good meeting with the OSA team last week in terms of figuring out OSA integration. anything more you want to discuss about what you plan to be working on this week?
15:37:18 <palendae> jimbaker: I'm going to try to push forward on our refactorings and high level APIs
15:37:46 <jimbaker> which means being able to put in some db backend in, like craton
15:38:17 <palendae> Yeah, I think we'll need to get further along with the separations, but we're close
15:38:45 <palendae> There are some reviews from stevelle up now that are putting our existing implementations behind our agreed APIs
15:38:56 <jimbaker> it seems like the merge approach means that it's really easy for craton as it is now
15:39:11 <jimbaker> we will find out shortly enough, which is good
15:40:36 <jimbaker> sulo, quick question - what's the status of being able to get at containers in the rest api? i would assume this is just using the host api, right? (since addressable)
15:40:50 <sulo> yep
15:40:59 <sulo> device_type=container
15:41:07 <jimbaker> cool, just wanted to verify that assumption, and yeah, device_type
15:41:22 <jimbaker> docs anyone? ;)
15:41:26 <jimbaker> we will get there
15:41:31 <sulo> this reminds me
15:41:45 <sulo> we dont verify device_type .. perhaps we should
15:41:58 <sulo> what i mean is .. you can put device_type=<anything>
15:42:04 <jimbaker> sulo, in terms of a defined set of device_types?
15:42:08 <sulo> yeah
15:42:12 <jimbaker> maybe this should be a config option
15:42:38 <jimbaker> could have a separate db table, seems more than what we need
15:42:57 <jimbaker> don't want to hardcode however
15:43:06 <sulo> i mean this could be as siimple as validating it to few predefined string
15:43:23 <jimbaker> right, but it would be enumerated in the craton config
15:43:24 <sulo> jimbaker: it can be in jsonschema vlidation schema
15:43:32 <sulo> jimbaker: yeah can do that
15:43:48 <jimbaker> sulo, jsonschema seems too hardcoded to me
15:44:05 <jimbaker> jsonschema should just know it's a string
15:44:10 <sulo> jimbaker: how do you mean, the schema is hardcoded
15:44:46 <sulo> anyway .. its not too important how we do it
15:44:47 <jimbaker> sulo, because it seems like a valid config option for users
15:44:53 <sulo> yeah
15:45:01 <sulo> we can make it a config option
15:45:03 <jimbaker> sulo, cool
15:45:15 <sulo> lets create isuue for this for now
15:45:24 <sulo> we can disscuss more when we get to it
15:45:25 <jimbaker> #action sulo add issue for device_type configuration
15:46:13 <jimbaker> syed__, what are plans for this week?
15:46:17 <jimbaker> your plans?
15:46:53 <syed__> So i will be working on crud devices to users, plus working on writing appropriate test and schema entries in client as per our current schema for regions, cells, hosts etc
15:47:10 <jimbaker> syed__, +1
15:47:18 <jimbaker> we will also follow up on RBAC this week
15:47:26 <syed__> sure, looking forward towards that
15:48:07 <jimbaker> me: work on blueprints (including RBAC with syed__); and finalize the resolution_order branch that's almost ready on my laptop
15:48:47 <jimbaker> also i promised harlowja a fix on listener support in taskflow, so i will get that in
15:48:59 <jimbaker> as time: revisit the taskflow stuff i did prior to the summit
15:49:26 <jimbaker> almost got that working, but ran out of time
15:50:23 <jimbaker> so that was a nice discussion for the week. we will check in with jovon when he get in the office
15:50:53 <syed__> Sounds good to me
15:50:57 <jimbaker> palendae, sigmavirus, sulo, syed__, any other topics we should cover for today?
15:51:12 <sulo> https://blueprints.launchpad.net/craton/+spec/craton-notifications
15:51:23 <sulo> just going to drop that here for everyone to looksee
15:51:45 <sulo> in case someone wants to pick it up and run with it
15:51:57 <jimbaker> sulo, ahh interesting. what are your thoughts on this implementation? it's very much related to RBAC given governance
15:52:42 <jimbaker> i do see this as a separable task from say following the db's write ahead log
15:53:20 <sulo> yeah .. need to check if there is already a notification thing in oslo
15:53:22 <jimbaker> sulo, there's also the governance concern that we discussed last week re keeping variable versions around
15:53:30 <sulo> ah yes
15:53:37 <sulo> thats a separate bp i need to create
15:53:47 <jimbaker> sulo, i think we can do that in the context of the blame system
15:54:23 <sulo> yeah .. the versioning could be
15:54:32 <jimbaker> btw, in my branch, i move blame such that it's a method on anything that mixes in variables
15:55:01 <jimbaker> so could readily add a parameter to the method to get old versions, etc
15:55:36 <sulo> sounds good to me
15:56:45 <jimbaker> sulo, re notification - assuming everything goes through the python api, vs directly against the db, this should be straightforward. maybe a mixin or just an addition to the craton base?
15:57:10 <jimbaker> much like create, modified timestamp support
15:57:33 <jimbaker> anyway we should certainly see if there's something preexisting in oslo that could support.\
15:58:07 <jimbaker> ok, time to wrap up the meeting
15:58:12 <jimbaker> any other questions?
15:58:15 <jimbaker> concerns?
15:58:16 <syed__> i am good
15:58:19 <syed__> thanks jimbaker
15:58:45 <jimbaker> cool, let's just move over to #craton
15:58:50 <jimbaker> thanks everyone!
15:58:54 <jimbaker> #endmeeting