17:01:44 <docaedo> #startmeeting app-catalog
17:01:44 <openstack> Meeting started Thu Jun 25 17:01:44 2015 UTC and is due to finish in 60 minutes.  The chair is docaedo. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:47 <ativelkov> o/
17:01:48 <openstack> The meeting name has been set to 'app_catalog'
17:02:00 <docaedo> Who do we have today?
17:02:42 <kzaitsev_mb> pong
17:03:05 <kfox1111> hi
17:03:21 <docaedo> OK
17:03:38 <docaedo> #topic Horizon panel to browse app catalog
17:04:04 <docaedo> kfox1111 do you want to lead this? I'll share the link to the CORS change in a sec
17:04:42 <kfox1111> sure.
17:05:18 <kfox1111> so I was watching the summit video on writing a horizon plugin and it looked very very easy, so I took a stab at adapting the demo plugin to work with the app catalog.
17:05:25 <kfox1111> demo video here: https://www.youtube.com/watch?v=N3onNfjf4uY
17:06:11 <kfox1111> right now it only supports the heat resources but the others should be easy to add as well.
17:06:50 <kfox1111> It also needs a local copy of the yaml file converted to a json document. We need to enable CORS support on apps.openstack.org and have json versions of the yaml files there in order to pull from.
17:06:59 <kfox1111> docaedo's been working on that.
17:07:06 <docaedo> To help support that, I have this patch to enable CORS, and to copy the YAML contents to JSON files when the repo is updated: https://review.openstack.org/#/c/194875/
17:07:47 <kfox1111> the prototype ui is available here: https://github.com/kfox1111/horizon branch angular-table-demo if anyone wants to play with it.
17:08:24 <kfox1111> It needs to be split up into a plugin and a set of patches against horizon. stock horizon didn't quite have enough features to make it work out of the box.
17:08:38 <kzaitsev_mb> docaedo: so it's going to be a static json file, that's regenerated every time you update the app-catalog
17:08:41 <kzaitsev_mb> right?
17:08:51 <kfox1111> for now, thats probably the easiest way forward.
17:08:55 <docaedo> kzaitsev_mb: yes for now
17:09:08 <kfox1111> longer term, it might be nice to store the json docs in something like elasticsearch in order to make faceting/searching work nicely.
17:09:46 <docaedo> next topic I want to talk about touches on a bigger picture thing, which is goign away from individual yaml files into something that easily supports more assets (but not 'till we're doing talking about this)
17:10:23 <kzaitsev_mb> docaedo: that's actually a correct approach for short-to-mid term imo =)
17:11:04 <docaedo> can the horizon plugin be extended to kick off a glance create-image pretty easily?
17:11:09 <kzaitsev_mb> kfox1111: I think I'm ok with a separate plugin, although in my opinion it creates a situation, where a user has 2 ways to perform the same task.
17:11:49 <docaedo> kzaitsev_mb: I think for app catalog to be easy to use from horizon, that's an OK problem
17:12:28 <kfox1111> docaedo: I think so. might need a patch similar to the heat one.
17:12:32 <docaedo> because for instance, if there's a panel that can browse app catalog contents, and give you a click for "get this glance image", techincally they could also browse the catalog directly, get the URL, and import from glance
17:12:52 <ativelkov> docaedo: I think everything which is available in app catalog should be put into Glance
17:13:13 <ativelkov> including Heat and murano artifacts
17:13:24 <docaedo> ativelkov: how would a heat template be put into glance? you mean glance v3?
17:13:28 <kfox1111> kzaitsev_mb: I think amazon, google, and apple's app stores all have that. a local tool and a website you can browse.
17:13:31 <ativelkov> docaedo: yes
17:13:57 <ativelkov> kfox1111: agree
17:14:07 <kzaitsev_mb> I still think, that a less obtrusive way to perform the task (give user access to catalog assets) is to hook up to specific forms (import heat template, import glance image, etc.)
17:14:09 <kfox1111> the local tool is more convenient. the website helps when you don't have that tool handy and your bored. :)
17:14:16 <kzaitsev_mb> like, give a separate source.
17:14:25 <kfox1111> the workflow to the user is backwards though.
17:14:34 <kfox1111> you have to know you want a heat template, then go looking for one.
17:14:48 <kfox1111> with the central ui, you go look for an app, and go launch it.
17:15:01 <kfox1111> you don't have to care what engine the develoepr decided to implement it with.
17:15:04 <docaedo> regarding glance, I think ultimately that's worth working towards, but that's definitely further down the road
17:15:29 <ativelkov> docaedo: well, it is not THAT further down :) V3 is already in trunk
17:15:38 <kfox1111> like most things in horizon, I think both ui's can be supported.
17:15:45 <docaedo> a horizon plugin could give people a flexible way to pull in catalog assets right now (and could even be back-ported to Juno for instance)
17:16:15 <kfox1111> docaedo: I'm afraid with the horizon changes that are going on right now, that might be difficult.
17:16:26 <kfox1111> It was easy to write the plugin using the angular stuff, but thats brand new.
17:16:29 <kzaitsev_mb> kfox1111: yep. I still haven't found time to play with this idea =( I hope I'll have some next week, and show you a mock-up. =)
17:16:34 <docaedo> ah right, that's true
17:16:40 <kfox1111> it could be rewritten whith their older stuff to be more portable, but alot more work. :/
17:16:55 <kfox1111> kzaitsev_mb: ok. cool.
17:17:09 <kfox1111> with the angular stuff coming about,
17:17:27 <ativelkov> I just don't want anybody to reinvent the wheel: Glance has most of the features for app catalog already present, and the rest are in the roadmap. I think the app-catalog team should take part in its design, so these two things aren't misaligned
17:17:39 <kfox1111> we should be able to make our code widgety, and then embed it in glance's ui, or heat or murano, prefiltered out to that service.
17:17:58 <docaedo> at least regarding where assets land locally, a horizon plugin doesn't have to be opinionated about that, should be able to put heat in glance if the environment supports it
17:17:59 <kfox1111> ativelkov: Yeah. I think thats a good idea.
17:18:34 <docaedo> I think the mission of glance artifact repository overlaps with app catalog, but they're not aiming to be the same thing
17:18:36 <kzaitsev_mb> kfox1111: now, you're talking! that's exactly what I thought, regarding widgets and integration.
17:19:23 <kfox1111> kzaitsev_mb: I think both workflows can be enabled by that. each of the services can get their filtered view, but we can provide an overall view too so the user can pick the workflow that meets their needs best.
17:19:39 <ativelkov> There are two things to consider: Glance may serve as App-Catalog backend (thus providing search, tags, versions, artifact type extensions, etc, etc) and it may be the "local thing"
17:20:27 <docaedo> ativelkov: agreed on glance being good choice for backend (will talk about that when we switch to next topic)
17:20:49 <docaedo> kfox1111: is there anything I or others can do to help you with the horizon stuff ATM?
17:21:14 <kfox1111> that would make for a nice "install" thing too. it copies bits from the remote catalog to glance locally.
17:21:36 <ativelkov> kfox1111: exactly!
17:21:49 <kfox1111> If anyone's interested in hooking it up to glance/murano and has some time, that would be cool.
17:21:58 <kfox1111> the cors stuff would really help too.
17:22:24 <kfox1111> we probably really need to meet with the glance team and figure out how we settle on tags, versons, etc.
17:22:42 <kfox1111> if we can share all of that, it would make the downlaod from global -> local really smooth.
17:22:44 <kzaitsev_mb> kfox1111: I'm going to try and help you with murano =) (Since I'm a murano developer anyway =))
17:22:55 <kzaitsev_mb> and probably glance too
17:23:02 <ativelkov> kfox1111: I am here as part of the Glance Team
17:23:06 <kfox1111> kzaitsev_mb: awesome. :)
17:23:20 <ativelkov> (and the main driver of artifacts/v3 btw)
17:23:20 <kfox1111> ativelkov: cool. :)
17:24:39 <docaedo> OK, moving on?
17:24:43 <kfox1111> so where to hook in is at:
17:24:44 <kfox1111> https://github.com/kfox1111/horizon/blob/angular-table-demo/openstack_dashboard/dashboards/project/app_catalog/templates/app_catalog/index.html
17:24:50 <kfox1111> see the button.
17:25:21 <kfox1111> The other place: https://github.com/kfox1111/horizon/blob/angular-table-demo/openstack_dashboard/dashboards/project/app_catalog/views.py
17:25:24 <kfox1111> is where the data gets pulled in.
17:25:51 <kfox1111> no, wait.. scratch that second one...
17:26:24 <kfox1111> its here: https://github.com/kfox1111/horizon/blob/angular-table-demo/openstack_dashboard/static/dashboard/project/app_catalog/app_catalog.js
17:26:56 <kfox1111> so we need to get it pulling from the other yaml files as well, and from apps.openstack.org.
17:27:22 <kfox1111> then update the button from "Launch" to "Install" for Murano and Glance entries.
17:27:36 <kzaitsev_mb> kfox1111: so you just cloned horizon and decided to skip all the bustle with plugins for now =)
17:27:52 <kfox1111> though I'm still thinking we probably want to split this catalog into "applications" and "components"
17:28:11 <kfox1111> where things that are simply installable go in components, and things that are imediately launchable go in applications.
17:28:18 <docaedo> next topic touches on multiple yaml files, and multiple asset types :)
17:28:31 <kfox1111> kzaitsev_mb: yeah. I think we can worry about pulling out the plugin bits later.
17:28:40 <kfox1111> horizon itself doesn't quite have that figured out yet for angular.
17:29:06 <kfox1111> I just cloned their angular demo plugin branch.
17:29:10 <kfox1111> then tweaked it a bit.
17:30:01 <kfox1111> docaedo: Ok. does yaql schema support subtypes?
17:30:26 <docaedo> #topic Next steps to adding additional asset types
17:30:41 <docaedo> Near term, I want to take the following steps (within two weeks)
17:30:41 <docaedo> Update website JS to handle pulling existing assets from a single YAML instead of three
17:30:42 <kfox1111> in xml schema for example, you can have one schema, that has generic types, and then have seperate schema files depending on the parent type.
17:30:44 <docaedo> Combine all three YAMLs into a single one (will require one new field, asset_type)
17:30:47 <docaedo> Make sure this change does not impact Murano (I don't think it would?)
17:31:04 <docaedo> This would give us a foundation for then creating new asset types, and changing the website
17:31:07 <docaedo> to handle them.
17:31:09 <docaedo> Outside this effort (medium term) kfox1111 I think wants to explore using elasticsearch to h
17:31:12 <docaedo> old asset list
17:31:16 <kfox1111> so... is murano shipping any code that actually uses the static files yet?
17:31:48 <kzaitsev_mb> kfox1111: what do you mean static files?
17:31:52 <kfox1111> because these url's are becoming... api for lack of a better word.
17:31:55 <docaedo> AFAIK no, I believe when you paste in the application name, murano then goes to the object store to pull it
17:32:07 <kfox1111> I mean, if we moved the yaml file on the website, would it break any existing customers?
17:32:29 <ativelkov> which yaml files?
17:32:36 <ativelkov> the app definitions?
17:32:36 <kzaitsev_mb> yaml — not at all. murano relies on the binary files and paths to binary files
17:32:54 <kfox1111> for example, http://apps.openstack.org/static/glance_images.yaml
17:33:14 <kfox1111> so its not using http://apps.openstack.org/static/murano_images.yaml directly?
17:33:32 <ativelkov> nope
17:33:43 <kzaitsev_mb> say if you have app a.zip that says it needs app b.zip it would try to download it from the same server .../apps/b.zip
17:33:54 <ativelkov> each murano package has its own list of dependencies
17:33:54 <kzaitsev_mb> kfox1111: true
17:34:25 <docaedo> so changing the YAML will not impact murano, but changing the object store would
17:34:34 <kzaitsev_mb> docaedo: correct
17:34:57 <kfox1111> ok. so then merging without having a process that pulls back out the murano stuff back into a seperate yaml file would be ok.
17:34:59 <docaedo> combining into a single yaml is safe, and then from there lets us start working on the website to handle that better
17:35:14 <kfox1111> +1.
17:35:14 <docaedo> also makes the horizon stuff easier (single json to grab)
17:35:29 <kfox1111> ativelkov: does the glance artifact stuff already have metadata defined for versions, tags, etc?
17:35:36 <ativelkov> kfox1111: yes
17:35:37 <kfox1111> yeah.
17:35:49 <docaedo> Longer term, I think glance should be used for this (as it would provide the functionality w
17:35:52 <docaedo> e need), but it's not an easy swap in - but luckily ativelkov can help scope that effort :)
17:36:02 <kfox1111> since we're starting to talk about reworking the stuff in the catalog, maybe we should review glance's metadata and adopt what we can?
17:36:03 <docaedo> (ugh, sorry for ugly paste)
17:36:20 <ativelkov> docaedo: do you want a PoC which runs the API for app-catalog on Glance V3?
17:36:27 <kfox1111> ativelkov: Have a link with docs or a spec for the metadata?
17:36:52 <kfox1111> that would be interesting. :)
17:36:57 <docaedo> kfox1111: I'm fine with looking at that sure, but I'm thinking this change is VERY simple, and just gives us an analog of what we have today
17:37:11 <docaedo> seeing a PoC of ap catalog on gance would be rad
17:37:13 <ativelkov> kfox1111: https://github.com/openstack/glance-specs/blob/master/specs/kilo/artifact-repository.rst
17:37:21 <kfox1111> docaedo: yeah. not saying we have to do both at the same time. just getting the ball rolling.
17:37:52 <docaedo> #action docaedo to create blueprint for the first phase of this effort
17:38:03 <kfox1111> ativelkov: Thanks.
17:38:06 <ativelkov> (its a kilo spec so it's a bit outdated, by in general the schema remain the same)
17:38:15 <kfox1111> any plans for artifact dependencies?
17:38:20 <ativelkov> Already there
17:38:48 <kfox1111> ah I see it.
17:38:59 <ativelkov> scalar dependencies (X depends on Y) and list dependencies (X depends on [A,B,C])
17:39:25 <kfox1111> do you have an import mechanism finished yet?
17:39:40 <ativelkov> static only for now, i.e. dependency is established between two particular artifacts (by id), no query-based ones yet
17:39:55 <ativelkov> kfox1111: nope, there is an unfinished PoC
17:40:02 <ativelkov> import is planned for L cycle
17:40:07 <kfox1111> by id? hmm...
17:40:17 <kfox1111> uuid or some other identifier like murano uses?
17:40:22 <ativelkov> uuid
17:40:29 <kfox1111> bummer. k.
17:40:55 <ativelkov> dynamic dependencies will use name + version combination
17:41:05 <kfox1111> so when importing, there needs to be a process to map some generic name to a specific one.
17:41:08 <ativelkov> including Name<=version
17:41:13 <kfox1111> is that slotted for L too?
17:41:47 <ativelkov> yup. It is relatevely simple, but the performance may suck, as there is no way to pre-cache the transitive dependencies
17:42:22 <kfox1111> yeah.
17:42:25 <ativelkov> eg. if X depends on Y and Y depends on Z, then we'll have two DB queries to fetch all the dependencies of X if they are dynamic
17:42:46 <ativelkov> static dependencies are denormalized, so the whole graph may be fetched in one queiry
17:42:50 <ativelkov> query*
17:43:05 <kfox1111> you could denormalize the dynamic ones in a cache?
17:43:27 <ativelkov> yes, sure
17:44:40 <kfox1111> is glance itself planning on implementing a launch ui for artifacts so that if you are browsing, say a heat template, you can hit "launch" and it will start up the workflow?
17:45:04 <ativelkov> kfox1111: no, glance is a catalog of stuff, it does not launch anything
17:45:15 <kfox1111> well, it does today for images.
17:45:26 <kfox1111> just wondering how that will work with other types of artifacts.
17:45:28 <ativelkov> that's nova who launches them
17:45:50 <kfox1111> yes, but in horizon, you can view the images, and hit launch next to the image, and it kicks of f the nova workflow.
17:46:05 <kfox1111> just wondering if the glance ui team is going to do the same with other artifacts.
17:46:11 <ativelkov> right, but that is the UI of this particular artifact type (images)
17:46:12 <kfox1111> cause that overlaps with what I've been doing.
17:46:14 <docaedo> good question and I was just going to ask that too
17:46:28 <ativelkov> we din't plan to have artifact-specific UIs
17:46:50 <docaedo> so if there is a non-image artifact in glance, does it have to be viewed somewhere else?
17:46:58 <kfox1111> ok. so the app catalog can/should still do that peice then.
17:47:52 <ativelkov> Our understanding was that the project which is going to consume that particular type is going to own the appropriate UI
17:48:12 <kfox1111> thats true.
17:48:21 <ativelkov> i.e. Murano Dashboard uses glance client to retrieve murano artifacts, but it is up to Murano to render the appropriate UI
17:48:32 <kfox1111> but the glance image pain can kick that other ui to start.
17:48:55 <ativelkov> Glance wil (traditionally) own the Images
17:48:58 <kfox1111> or are other artifact types hidden from the user?
17:49:23 <ativelkov> In Images view only the artifacts of type "Image" will be shown
17:49:41 <ativelkov> probably we will have some "admin view" which will display all the artifacts regardless of their type
17:49:49 <ativelkov> but that is not designed yet
17:49:51 <kfox1111> ok. so every project that wants to store their artifact type need to provide glance artifact management ui for their parts?
17:50:23 <ativelkov> first of all they will have to provide a glance plugin which defines the schema of their artifact
17:50:35 <kfox1111> ok. that makes sense.
17:50:38 <ativelkov> fields, dependencies, BLOBs etc
17:51:04 <ativelkov> So, that is the project which owns the schema (via the plugin), not the Glance
17:51:05 <kfox1111> then do you use glance cli to load the artifact types, glance artifact-list to find them, etc?
17:51:13 <ativelkov> right
17:51:25 <ativelkov> OR you may use project's own CLI
17:51:49 <docaedo> ativelkov: what do you think ETA would be for seeing a PoC? Just want to know when to circle back to it - can check status weekly anyway
17:52:16 <ativelkov> I.e. python-muranoclient references python-glanceclient to access artifacts but defines its own type-specific set of CLI commands
17:52:30 <kfox1111> just trying to figure this out from the heat templates perspective.  so you'd create a glance artifact, load it with glance artifact-create, and then heat ui would need to be extended to support finding heat artifacts in the catalog?
17:52:51 <kfox1111> hmm... ok.
17:52:57 <ativelkov> docaedo: the POC itslef is an easy part (next week I think), will just need to think about the loader from your yaml's
17:53:09 <ativelkov> kfox1111: yes
17:53:23 <docaedo> ativelkov: great! well I'll be on IRC to help however I can
17:53:50 <ativelkov> BTW, we had a PoC before the first LoC of app-catalog was created :)
17:53:53 <kfox1111> ativelkov: sounds very interesting.
17:54:48 <docaedo> nice! I think there's no question app catalog was needed, and a few different folks had slightly different interpretations in progress
17:55:01 * kfox1111 nods
17:55:06 <docaedo> I'm looking forward to hopefully getting all of the best considerations rolled into this together
17:55:23 <docaedo> So, moving on to open conversation?
17:55:34 <kfox1111> ativelkov: whats the status with the glance artifacts and the indexing service?
17:55:40 <kfox1111> since its already doing elasticsearch?
17:56:27 <ativelkov> Well, we've been in good contact with Travis, I was doing the reviews of it while that code was still part of Glance
17:56:53 <ativelkov> We plan to have plugin to Searchlite which will index all the artifacts in its elastic search
17:57:08 <kfox1111> interesting.
17:57:16 <ativelkov> It's not started yet, as we want to finish the higher-priority stuff first
17:57:17 <docaedo> just a few minutes left BTW
17:57:27 <kfox1111> so the backend for the global app catalog might be able to just be glance + searchlite,
17:57:39 <ativelkov> That's my vision of it
17:57:42 <kfox1111> with a horizon plugin and a website wrapped around it,
17:57:48 <docaedo> I'm excited to see the glance stuff moving along, and it really will be probably the perfect backend
17:57:54 <kfox1111> and a tool to import artifacts from glance to glance.
17:58:08 <ativelkov> Eventually we want to get a federation of catalogs
17:58:15 <ativelkov> with app.openstack.org being a root entity
17:58:24 <kfox1111> cool. yeah.
17:58:28 <ativelkov> and particular cloud deployments acting as leafs
17:58:34 <kfox1111> I was hoping to see something yum/apt like.
17:58:39 <ativelkov> yeah
17:59:13 <ativelkov> Then think about signing the artifact, chains of trusts... yeah, lot's of work :)
17:59:28 <kfox1111> yup.
17:59:50 <docaedo> ok we're out of time - thanks for this conversation, looking forward to the next steps!
17:59:59 <kfox1111> yup. me too. :)
18:00:08 <ativelkov> Thank you guys
18:00:10 <kfox1111> and it does sound like we still need the ui.
18:00:18 <kfox1111> just gota figure out how to plug it into murano/glance. :)
18:00:30 <docaedo> that won't be too hard :)
18:00:33 <docaedo> #endmeeting