20:00:16 <robcresswell> #startmeeting horizondrivers
20:00:17 <openstack> Meeting started Wed Apr 20 20:00:16 2016 UTC and is due to finish in 60 minutes.  The chair is robcresswell. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:21 <openstack> The meeting name has been set to 'horizondrivers'
20:00:25 <david-lyle> o/
20:00:31 <lcastell> o/
20:00:33 <ediardo> o/
20:00:40 <ankur-gupta-f> o/
20:00:49 <bpokorny> o/
20:00:52 <tsufiev> o/
20:01:08 <robcresswell> Hi all
20:01:20 <tqtran> [=_=]/
20:01:38 <robcresswell> Just one notice to start
20:01:38 <jlopezgu_> Hi
20:01:51 <robcresswell> #topic Summit schedule
20:01:58 <robcresswell> #link https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Horizon%3A
20:02:13 <robcresswell> This is the Horizon room topic if you need to refer back, but wanted to bring it up anyway
20:02:13 <TravT> o/
20:02:39 <robcresswell> So make sure you attend the sessions :D
20:02:43 <david-lyle> we're tagged in a talk titled "fail." so that's promising
20:02:52 <robcresswell> Yeah I saw that
20:03:05 <robcresswell> Very encouraging
20:03:11 <david-lyle> I'm sure they mean it in the best possible way
20:03:42 <robcresswell> Any other notices before I move on to blueprints?
20:04:32 <kenji-i> o/
20:04:35 <robcresswell> #topic Blueprint Review
20:04:45 <robcresswell> There's several bps on the agenda
20:04:50 <robcresswell> #link https://wiki.openstack.org/wiki/Meetings/HorizonDrivers#Agenda_for_2016-04-20_2000_UTC
20:05:26 <robcresswell> #link https://blueprints.launchpad.net/horizon/+spec/operation-history-log
20:05:55 <robcresswell> kenji-i: If you want to add anything to the explanation, go ahead :)
20:06:24 <robcresswell> ducttape_: This sounds similar to a feature you mentioned before?
20:06:38 <kenji-i> robcresswell: okay, thanks :)
20:06:55 * ducttape_ reads up on that one
20:07:12 * tsufiev is curios how many operators assign different accounts to different people
20:07:22 <robcresswell> Basically its about logging the actions that users take within Horizon
20:08:03 <kenji-i> robcreddwell: yes, this fieature to log an operation on Horizon
20:08:32 <tyr_> this sounds similar to an audit logging request we've been looking at. Instead of adding to our logs, it inserts Keystone WSGI middleware to log any API requests to Horizon server
20:08:40 <ducttape_> that would be somewhat helpful.  I would expect larger deployments to also have something like splunk aggregating all the actions too
20:09:21 <david-lyle> kenji-i: this seems like something that should be outside Horizon
20:09:26 <ducttape_> some of the need for that, would be for having a view into api calls made by admins - cli or within horizon
20:09:36 <david-lyle> I don't want to pass the buck, but it misses all API and CLI calls
20:09:49 <tqtran> doesnt ceilometer does this to some degree?
20:10:00 <david-lyle> so you only get part of the story
20:10:16 <david-lyle> nova has the events log which we show
20:10:17 <tyr_> see also: http://docs.openstack.org/developer/keystonemiddleware/audit.html
20:10:18 <ducttape_> not sure to what degree ceilometer would help here tqtran, maybe a bit
20:11:11 <kenji-i> I think in some cases, there are required to log an access log.
20:11:20 <ducttape_> thats the logging part, yes tyr_, but then you need to aggregate all the logs everywhere, combine them, then have a powerful ui to query them
20:11:38 <kenji-i> not api call but operation by user
20:11:38 <tyr_> smells like search ;)
20:11:57 <ducttape_> it's splunk, in a nutshell
20:12:10 <lhcheng> there's more standard auditing, use pycadf
20:12:20 <lhcheng> we already have audit middleware for that
20:12:46 <lhcheng> the audit can be pushed to a queue or something for recording purposes
20:12:49 <ducttape_> but that requires operators to consolidate / aggregate logs etc , right lhcheng ?
20:13:22 <lhcheng> yeah, up to the operator to determine what they want to do with the audit log
20:13:50 <ducttape_> so, back to the bp at hand, I think it is somewhat useful, but it will be a partial solution
20:14:07 <ducttape_> right ?
20:14:12 <robcresswell> Depends if we want to carry a partial solution in Horizon (and even if we should carry that logic at all) ?
20:14:18 <tqtran> if you want to add a panel to support this view, thats entirely possible and welcomed. but consolidating/aggregating logs is beyond what horizon can do
20:14:35 <tyr_> I agree. Auditing Horizon is a good step...but only part of the problem. Often cloud actions occur outside of the UI.
20:14:36 <ducttape_> right tqtran, this much is certain
20:15:21 <ducttape_> I would see this more of a contrib panel, if operators use splunk  (or other alternatives)
20:15:23 <tyr_> I think we should first focus on logging rest calls to the Horizon server in a way that is consistent with the REST calls to other services...before worrying too much about a UI to expose those logs.
20:16:11 <ducttape_> so the other side should be logging it tyr_, not to say that horizon logging is great - there is room for improvement
20:16:39 <ducttape_> but if I have to aggregate all of cinder/glance/neutron etc logs anyways, I'll just use that location
20:16:42 <david-lyle> this is basically turning up the log level
20:16:47 <david-lyle> we can already do that
20:17:37 <david-lyle> not entirely sure where the userid fits in though, TBH
20:17:37 <ducttape_> I think there is value in this type of feature, but you will need to assume a certain way to aggregate and gather all the logs
20:18:22 <ducttape_> also, if horizon is setup behind a load balancer, you'd need to show all the actions from all the horizon servers etc
20:18:24 <tsufiev> osprofiler library stores messages about API calls in mongodb (one of possible backends)
20:18:46 <ducttape_> that would work tsufiev
20:18:50 <david-lyle> tsufiev: you're not running osprofiler in production are you?
20:18:58 <tsufiev> but it was designed with a slightly different goal in mind...
20:19:08 <david-lyle> all the time
20:19:21 <tsufiev> david-lyle, not yet, and I don't know if it could be switched on _permanently_
20:19:38 <david-lyle> that's a pretty faulty audit solution then :P
20:19:45 <tsufiev> a great amount of data, it would be
20:20:14 <robcresswell> Yeah we're getting off topic with osprofiler
20:20:14 <tsufiev> but I'll ask rally folks if they though about that use-case
20:20:23 <ducttape_> should we just say that this bp needs more info on how the data is stored / aggregated ?
20:20:39 <robcresswell> Yeah I've just been scanning the patch itself
20:20:56 <tyr_> at a high-level, I'd like a Horizon logging solution to be consistent with other services.
20:21:07 <david-lyle> my thought is if auditing isn't done outside of Horizon, it's too incomplete to be useful
20:21:41 <ducttape_> +1
20:21:46 <lhcheng> there's already auditing solution in place, why are we re-inventing the auditing solution?
20:22:14 <ducttape_> that is a partial solution lhcheng, but its the best start we have
20:22:42 <lhcheng> yeah, we could improve and contribute on that. but let's not start another project :)
20:23:01 <david-lyle> big tent requires more projects, it's a pyramid scheme
20:23:04 <ducttape_> ok, so lhcheng would like to start another project, I second this motion
20:23:14 <robcresswell> Great
20:23:15 <david-lyle> we don't make money without more new projects
20:23:19 <lhcheng> lol
20:23:56 <david-lyle> excuse me multi-level marketing
20:24:21 <robcresswell> I quite liked the contrib idea. If there is some value in holding this, we could hold it in contrib and see if it can extended beyond that for a more complete solution.
20:24:21 <ducttape_> ground floor level opportunities
20:24:46 <ducttape_> yep robcresswell.  I think that patch is going to have issues when you have multiple horizons too
20:24:47 * r1chardj0n3s is wondering what he's stumbled in on ;-)
20:25:14 <robcresswell> ducttape_: Agreed, but I'm trying to keep the bp review seperate from implementation.
20:25:26 <robcresswell> Otherwise we'll be here all day.
20:25:31 <robcresswell> night
20:26:15 <robcresswell> I'd say approve for now, and see where it goes. Anyone strongly against that idea?
20:26:22 <ducttape_> wfm
20:27:23 <robcresswell> #info https://blueprints.launchpad.net/horizon/+spec/operation-history-log Approved, Marked Low Priority.
20:27:36 <robcresswell> #link https://blueprints.launchpad.net/horizon/+spec/django-debug-toolbar
20:28:23 <robcresswell> Oh, this is mine. Its a pretty common django tool. Also adds a fairly detailed template profiler for load times.
20:28:31 <ducttape_> could that thing just be turned on when debug is turned on ?
20:28:34 <david-lyle> I think this piece should a documented add on
20:28:41 <ducttape_> kinda like the css preview page ?
20:28:58 <david-lyle> the show() method is called for every request
20:29:06 <tsufiev> robcresswell, does django debug toolbar profile templates as well?
20:29:10 <david-lyle> 19 times when launch instance is invoked
20:29:12 <robcresswell> ducttape_: Right now its disabled when DEBUG is False, but also has an explicit bool control too. Called ENABLE_DEBUG_TOOLBAR iirc.
20:29:27 <tsufiev> it seems as there are 2 debugging/profiling features in this bp
20:29:34 <ducttape_> thanks
20:29:49 <robcresswell> tsufiev: Its an add on to the debug toolbar. So, they are interlinked.
20:30:15 <tsufiev> okay, so the the toolbar is a prerequisite for template profiler
20:30:48 <tsufiev> like the idea
20:31:19 <tsufiev> I guess the most difficult part is to push 2 new dependencies to global requirements ;)?
20:31:28 <robcresswell> The question is, as David mentioned, whether we just mention it and leave it all commented out, or provide the code and toggle it on a boolean
20:31:51 <robcresswell> I'm obviously biased :)
20:31:57 <r1chardj0n3s> it's pretty easy to just pip install over a venv, right?
20:32:06 <david-lyle> you have three-four logical checks each time show is called
20:32:19 <tsufiev> robcresswell, what if made a switch at Developer dashboard?
20:32:27 <tsufiev> *we made
20:33:02 <david-lyle> tsufiev: cookie driven rather than settings
20:33:27 <david-lyle> well that still doesn't eliminate the overhead of checking on every request
20:33:44 <tsufiev> david-lyle, ack
20:33:50 <tyr_> I like that it would be easy to enable...but I wouldn't want it on by default in debug mode. Also not certain about asking deployers to include this developer only dependency.
20:34:06 <robcresswell> Its disabled by default, even in debug mode.
20:34:12 <ducttape_> ah good point tyr_, that extra prereq is bad
20:34:15 <david-lyle> you could even script enabling it
20:34:17 <david-lyle> in tools
20:34:56 <david-lyle> pip install 2-packages, append onto the local settings
20:34:57 <ducttape_> the problem is that people trying to run horizon in prod are going to have some python dependancy on a library they have no interest in running with
20:35:19 <david-lyle> ducttape_: I'm for not including it and just having a scripted solution to enable it
20:35:32 <robcresswell> Sure
20:35:35 <david-lyle> that's where I was headed, if unclear
20:35:47 <ducttape_> that would be ideal, or even to have some developer requirements.txt thing
20:35:57 <david-lyle> but how you inject the show method I don't know yet
20:36:17 <tsufiev> what if we placed it into contrib/ and changed contrib/ dependencies regulations?
20:36:29 <david-lyle> oh, nevermind
20:36:35 <david-lyle> it's adding an app, forgot
20:36:46 <david-lyle> so yeah, a contrib panel would be fine
20:37:03 <david-lyle> err, not panel, but script
20:37:10 <tsufiev> I mean, for example, contrib/ apps don't have to work with global requirements out of the box and may require additional deps to be installed
20:37:16 <tyr_> bring it in as a FEATURE plugin?
20:37:20 <robcresswell> So as long as its conditional, the idea is fine, is the general consensus?
20:37:49 <ducttape_> I think so robcresswell, make it pluggable and avoid the global requirements change
20:37:52 <robcresswell> Reluctant to presume with my own bp, but we should move on to the next item.
20:37:52 <tyr_> flip Disabled=False in its enabled file and away you go.
20:37:53 <tsufiev> +1
20:37:57 <david-lyle> tyr_: something like that
20:38:22 <robcresswell> Great. So will approve and leave a comment about ensuring its optional, as are its dependencies
20:38:26 <robcresswell> Sound good?
20:38:29 <ducttape_> +1
20:38:32 <r1chardj0n3s> +1
20:38:44 <tyr_> +1
20:38:48 <tsufiev> ack
20:39:02 <david-lyle> perfect
20:39:10 <hurgleburgler> +1
20:39:28 <robcresswell> #https://blueprints.launchpad.net/horizon/+spec/django-debug-toolbar Approved, marked Low
20:39:31 <robcresswell> ugh
20:39:40 <robcresswell> #info https://blueprints.launchpad.net/horizon/+spec/django-debug-toolbar Approved, marked Low
20:39:55 <robcresswell> #link https://blueprints.launchpad.net/horizon/+spec/horizon-use-quotas-api
20:40:01 <robcresswell> ducttape_: You're up
20:40:04 <ducttape_> ok, so some background
20:40:30 <ducttape_> horizon uses the 'limits" api for lots of stuff, mainly around the pie charts on overview pages and checking quotas
20:40:46 <ducttape_> the limits api is old, but reliable
20:41:10 <ducttape_> cinder has now introduced this ability to have multiple storage types
20:41:22 <r1chardj0n3s> it's also incredibly slow :/
20:41:26 <ducttape_> so deployment might have SSD / HDD etc
20:41:38 <ducttape_> it gets worse r1chardj0n3s  ;)
20:42:14 <tsufiev> ducttape_, is the 'usage' feature a part of limits API or quotas API?
20:42:15 <ducttape_> if I want to show users how much cinder quota they have, I need to no longer call limits api, instead I need the quotas call
20:42:36 <ducttape_> the actual "you used this much" is returned in limits - but NOT in quotas call
20:42:37 <tsufiev> I saw some horrendous workarounds for usage in horizon code
20:42:59 <tsufiev> I think it was exactly cinder
20:42:59 <ducttape_> this all is a giant mess, indeed
20:43:04 <robcresswell> Quota management in general is something that gets mentioned a lot too
20:43:04 <david-lyle> usage is fairly inaccurate too
20:43:21 <robcresswell> So it seems like it could use an improvement in many regards
20:43:24 <david-lyle> or at least was
20:43:35 <ducttape_> we have the need to show users "you have this much available in ssd" and "this much in hdd" etc space
20:43:47 <david-lyle> I applaud ducttape_ for volunteering
20:43:49 <ducttape_> but to do this, it means making quotas calls
20:43:58 <david-lyle> that is code no one ever wants to touch
20:44:27 <ducttape_> and when I switch to quotas, then I need to make subsequent calls to summarize "oh - you used 500GB of SSD" etc
20:44:32 <tsufiev> david-lyle, I made one attempt some time ago and still regret that :)
20:44:44 <ducttape_> and that slowness r1chardj0n3s mentioned becomes more worser
20:44:51 <ducttape_> perhaps even worserest
20:45:04 <r1chardj0n3s> (just try limits with 10s of thousands of servers :/)
20:45:38 <hurgleburgler> yikes
20:45:44 <tsufiev> ducttape_, the proper solution to that problem is a cross-project initiative, isn't it?
20:45:47 <ducttape_> so I'm stuck in my current position, I need to start making quota calls, and the work I need to do should probably be pushed back upstream
20:46:01 <david-lyle> tsufiev: funny you should mention that
20:46:13 <tsufiev> I mean, quotas have to be fixed in every affected service
20:46:15 <david-lyle> there happens to be a quotas working group
20:46:22 <ducttape_> but I loathe trying to make everything quotas, because that seems pretty horrid too
20:46:31 <robcresswell> The blueprint itself seems too small given the scope then; aren't we talking about an effort per-project, rather than a general "migrate API" bp?
20:47:05 <ducttape_> it could be.   right now I am also asking which other projects have this aspect - to be able to have random new quotas show up
20:47:22 <david-lyle> well, we migrated to limits from quotas (ducttape_ drove that), migrating back via ducttape_ seems like proper punishment
20:47:23 <ducttape_> like I think glance and nova / neutron might be fixed
20:47:41 <ducttape_> the irony is not lost on me
20:47:57 <ducttape_> evidently I was a very bad person in my former life
20:48:09 <david-lyle> I just want to highlight for the new comers :)
20:48:11 * ducttape_ is not sure about this life, either
20:48:31 <hurgleburgler> lol
20:48:37 <tsufiev> :D
20:49:01 <david-lyle> what is the permission model on the quota APIs ?
20:49:02 <tsufiev> I'm still missing the point why come back to quotas from limits if quotas were so bad :)
20:49:13 <ducttape_> so the bp is there / filed.... and I'll push what I can under review in there for more feedback, but this quotas stuff and having a quotas working group sounds like waiting on the government to fix my problems
20:49:23 <david-lyle> would we have to do something like limits for non-admins and quotas for admins?
20:49:55 * david-lyle hasn't checked
20:49:57 <robcresswell> ducttape_: Yeah I'm just wondering whether we can build this up to a better scope for people to contribute. The bp in its current state isnt ideal
20:50:00 <ducttape_> quotas provides a richer set of features tsufiev
20:50:27 <tsufiev> ah, okay
20:50:46 <ducttape_> as an example, we have users that log into horizon, and say "I failed to create a volume, but it says I have 100GB left"
20:51:09 <ducttape_> and we say "you only have 100GB of HDD storage, you are out of SSD storage"
20:51:28 <ducttape_> and the customer says "why is the chart telling me this / confusing me? "
20:51:36 <robcresswell> I think: expand the bp to detail what needs to be done, and we can use the summit to talk about the work further, since it sounds like you're not available to implement the entire thing yourself ;)
20:52:09 <ducttape_> I'm excellent at creating bps that I can't complete.  challenge accepted
20:52:13 <ankur-gupta-f> \o/
20:52:26 <david-lyle> looks like quota show calls are open
20:52:51 <tsufiev> I would gladly implement the bp (if that's possible)
20:53:05 <robcresswell> The trap worked!
20:53:07 <tsufiev> well, I meant that pkarikh would gladly implement it :)
20:53:11 <robcresswell> I mean, well volunteered tsufiev
20:53:12 <ducttape_> I'll take all the help I can get :D
20:53:12 <david-lyle> SCATTER
20:53:21 <tsufiev> since he's interested in quotas fixing
20:53:40 <ducttape_> my next bp on the list today is not as bad, I promise
20:53:49 <robcresswell> ducttape_: I'm gonna mark the bp as Drafting until it can be expanded further
20:53:59 <ducttape_> sounds good, thanks robcresswell
20:54:12 <robcresswell> #info https://blueprints.launchpad.net/horizon/+spec/horizon-use-quotas-api Needs more work, but the idea is generally approved of.
20:54:15 <robcresswell> :D
20:54:28 <robcresswell> #link https://blueprints.launchpad.net/horizon/+spec/admin-neutron-l3-agent
20:54:30 <ducttape_> so neutron agent lists - good times.
20:54:54 <ducttape_> we have a hypervisor page, which is a good analogy of what I am after for this
20:55:13 <ducttape_> this lists all the places where nova is running / available to host vms
20:55:47 <ducttape_> well, neutron has a similar list of agents it has.   like "these are places that host dhcp servers"  "these are places that host routers"
20:56:09 <ducttape_> as admins, we have to use the cli to navigate that right now
20:56:26 <ducttape_> but it would be very helpful to show some of that info in the UI
20:56:27 <david-lyle> would this be added to the table in system information?
20:56:53 <ducttape_> I would see this as a new neutron panel in the admin area, if it really got flushed out
20:57:21 <ducttape_> right now I have a small patch up, that just shows the location(s) where the neutron router resides
20:57:45 <ducttape_> but really, this is a pretty rich environment for a lot of interesting stuff
20:58:04 <ducttape_> if I was in tsufiev's shoes - this is where to volunteer
20:58:23 <ducttape_> the api is pretty well know, there is a need from admins for a view into this stuff, etc
20:58:23 <tsufiev> this time I abstain :)
20:58:24 <david-lyle> it's not linked to the bp
20:58:35 <robcresswell> Looks like a good idea, but needs filling out more so people can pick up on it ducttape_
20:58:38 <ducttape_> https://review.openstack.org/#/c/306619/
20:58:48 <ducttape_> #link https://review.openstack.org/#/c/306619/
20:59:04 <ducttape_> ok, yep agree robcresswell
20:59:23 <david-lyle> ducttape_: http://pasteboard.co/kSwaEFE.png
20:59:35 <david-lyle> I could see hosts listed in that table
20:59:40 <david-lyle> just a thought
20:59:50 <robcresswell> thanks ducttape_
20:59:54 <david-lyle> or is there a lot more information you're going to add?
20:59:57 <ducttape_> yep, makes sense david-lyle
21:00:11 <robcresswell> #info https://blueprints.launchpad.net/horizon/+spec/admin-neutron-l3-agent needs to be expanded
21:00:12 <ducttape_> I could see the ability to drill down on those items too david-lyle
21:00:22 <robcresswell> We're at time; will add final item to next meeting
21:00:32 <robcresswell> Remember, no meetings next week because of summit
21:00:35 <ducttape_> like you know the l3 agent is running on daves machine - what other routers are living there???
21:00:49 <ducttape_> thanks all
21:00:53 <robcresswell> Lets continue in Horizon chat. Thanks all
21:00:56 <robcresswell> #endmeeting