14:00:16 <tobberydberg> #startmeeting publiccloud_wg
14:00:17 <openstack> Meeting started Thu May 23 14:00:16 2019 UTC and is due to finish in 60 minutes.  The chair is tobberydberg. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:20 <openstack> The meeting name has been set to 'publiccloud_wg'
14:00:35 <tobberydberg> o/
14:00:58 <witek> hi
14:01:00 <ncastele> \o/
14:02:03 <ncastele> Lot of stuff and many different topics in the brainstorming :o
14:02:27 <tobberydberg> Hope you are all doing well! Lets wait a few minutes before we get started to see if more people are on their way in :-)
14:02:40 <tobberydberg> Yes ncastele  .... a lot of them indeed :-)
14:03:48 <tobberydberg> Just to have it said, like your comments there, pretty much aligned with my personal thinking here as well
14:04:38 <tobberydberg> So, are there more people in here at this point interested in the billing topic?
14:04:43 <tobberydberg> mnaser ?
14:04:50 <mnaser> o/ :>
14:04:59 <tobberydberg> +1
14:06:04 <tobberydberg> Don't see anyone else from the "comments list" that are online, so lets start
14:06:17 <tobberydberg> #topic 1. Joint development effort billing
14:07:39 <tobberydberg> So, the plan for this initiative to get going was to start to collect some raw ideas until this meeting
14:08:01 <tobberydberg> There are a lot of ideas and thoughts around this topic in the etherpad
14:08:09 <tobberydberg> #link https://etherpad.openstack.org/p/publiccloud-sig-billing-implementation-proposal
14:09:35 <tobberydberg> So, today the plan is to discuss these ideas and see if we can find a smart way to move forward, limit the scope to something that we all things are a good first step etc
14:10:31 <ncastele> I see multiple topics: collect data, store data, customize, aggregate, display, price data, bill data (invoice)
14:10:35 <tobberydberg> Reading all the comments, I believe it will be really important to limit the scope first of all
14:10:46 <ncastele> +1
14:11:28 <tobberydberg> yes, agree, and I guess we will touch a lot of them in some sense in our limitation
14:12:21 <ncastele> collecting and storing looks pretty obvious and required, imo
14:12:33 <tobberydberg> In my mind, what I would love to see as a end result would be something that can respond to a api call and return the cost for a resource/project/etc
14:12:59 <tobberydberg> that definitely includes those parts ncastele
14:13:37 <tobberydberg> Also, I would love to be able to use tools already implemented and that we can collaborate with
14:13:49 <ncastele> I could agree, but one of the issue if we start to think about cost, then we talk about prices, currency, taxes, catalog, etc.
14:14:17 <ncastele> (which is a bit pain in the ***)
14:14:48 <witek> tobberydberg: is CloudKitty fulfilling your requirements?
14:14:49 <tobberydberg> I added "Limitation of scope" at the etherpad where we can put our suggested limitation
14:16:21 <tobberydberg> ncastele yes, agree...but that would be a nice end goal for us and for endusers. RAW cost ... nothing to do with taxes, nothing with discounts etc etc
14:17:10 <tobberydberg> just the raw cost based on usage and a cost per unit of some kind
14:17:43 <tobberydberg> It might be that is to much for the scope
14:17:54 <tobberydberg> witek No, it does not
14:19:00 <ncastele> it means that the first iteration should handle the definition of product/catalog/prices, or a way to have some kind of middleware that can call an external service to get the price/cost of a resource
14:19:11 <tobberydberg> It's been some time since I looked into it now though ... but for example it supports billing plan for flavors, that is not what we bill for today
14:19:59 <witek> I thought they can define flexible billing models, but I haven
14:20:08 <ncastele> to be honest, in OVH we already have a platform where we define the catalog/prices/special stuff, and I don't want to synchronize this tool with some other table/database in OpenStack
14:20:09 <witek> haven't looked at it myself
14:20:25 <tobberydberg> Yea, I guess if that were to happen some configuration of which resources to bill for in which unit to which price will be needed
14:21:08 <ncastele> yes
14:21:38 <tobberydberg> ncastele Well, I would like to have good support in openstack for users that would like to be able to do that
14:22:10 <ncastele> As a first iteration, I definitely think it's easier to focus on collecting/storing data, with the goal of displaying quantities
14:22:13 <tobberydberg> For me personally, I would be super happy with a good reliable backend that in real time can give me the aggregated usage
14:22:34 <ncastele> Then, from quantities/usage, we can iterate to something that can handle prices/cost
14:22:35 <tobberydberg> can agree on that as well ncastele
14:23:02 <tobberydberg> Thought, ideas from anyone else here?
14:24:15 <witek> I would recommend Monasca for collecting/storing, but I'm biased :)
14:24:19 <tobberydberg> Would ceilometer  agent work as the collector?
14:25:00 <tobberydberg> can Monasca collect all events, usage etc in real time at this point?
14:25:37 <ncastele> I don't have enough knowledge on ceilometer/monasca to confirm stuff about it
14:26:04 <witek> not yet, we can collect system usage metrics, libvirt, ovs metrics
14:26:08 <ncastele> But I have strange usecases that we can challenge with ceilometer/monasca to see if it fits
14:26:09 <witek> any prometheus metrics
14:26:16 <witek> and ceilometer metrics
14:26:37 <tobberydberg> mnaser do you have insight in that?
14:26:52 <mnaser> I wanted to share something on that
14:27:00 <tobberydberg> please do
14:27:06 <mnaser> (but google is rough)
14:27:34 <tobberydberg> You will just have to write if all you self then :-)
14:29:09 <mnaser> (trying to look for it)
14:29:33 <mnaser> its a prometheus exporter
14:29:40 <mnaser> that actually captured the project id and user id
14:29:49 <mnaser> https://github.com/zhangjianweibj/prometheus-libvirt-exporter
14:29:50 <mnaser> there
14:29:58 <mnaser> its actually pretty neat
14:30:32 <mnaser> zhangjianweibj seems to work at inspur too who contribute to openstack often
14:31:18 <ncastele> *written in Go* > Ok we can use it <3
14:31:21 <tobberydberg> So this would be a kind of replacer for what ceilometer does today? Collecting the data? Or, also the storing of data?
14:32:57 <witek> mnaser: the project you've posted seems to be one man effort, zhangjianweibj was evaluating different projects, I've seen him contributing to Monasca as well
14:33:19 <mnaser> it is an exporter than prometheus can scrape
14:33:30 <mnaser> witek: it could be a start, that was the idea.. it's an approach
14:33:37 <mnaser> just wanted to throw it out here
14:35:03 <ncastele> it's mainly oriented for nova instances, we need to focus also on other components (volumes, snapshots, storage, etc.)
14:35:38 <ncastele> Regarding the first iteration, it could be a good idea to scope the resources/services we want to collect/store from
14:35:43 <tobberydberg> Yea, implementations of any kind to be able to be something official in the openstack community, it should be written in python, but the interesting parts are the approach as mnaser says
14:36:12 <mnaser> I felt it was more interesting as an overall approach
14:36:22 <mnaser> personally in my experience, we need something that is sharded
14:36:29 <mnaser> it's a catch 22
14:36:49 <tobberydberg> saw that Linaro had one as well, as well as Cananoical
14:37:12 <mnaser> machines polling their own instances is nice because it scales out .. but ends up all in one system trying to handle saving it all which melts things down
14:37:27 <mnaser> machines storing state on their own is not reliable
14:38:26 <ncastele> it has to be aggregated somewhere so we can easily manage/compute/display through API
14:38:44 <tobberydberg> So, not being able to use anything for the collection that is already implemented will increase the scope a bit :-)
14:38:45 <mnaser> so I think the idea here is two step: 1) collection, 2) (auto-?)aggregation
14:39:16 <mnaser> I do think we all share a common pain point
14:39:17 <tobberydberg> 1)collection and storing :-)
14:39:29 <mnaser> which is: I need to know how many seconds/minutes/hours this resource has consumed
14:39:34 <mnaser> that is *mostly* what we care about..
14:39:58 <ncastele> +1
14:40:05 <tobberydberg> yes. I was first thinking of using ceilometerm but implement a smarter way of storing the data, but that might be hard to accomplish ?
14:40:26 <tobberydberg> agree with that mnaser
14:40:37 <mnaser> historically openstack telemetry never gave us that
14:40:43 <ncastele> (and I also care about Swift usage but it's completely a different topic)
14:40:50 <mnaser> it was just _data_ without what we want which is raw numbers
14:41:08 <witek> I think on collection side the best approach is for services to instrument their code themselves and expose in the standardized way
14:41:09 <mnaser> well ceilometer stores into gnocchi now which includes several stores
14:41:29 <tobberydberg> yea, WHAT to measure and collect I guess will be another toic, but definitely as swift as well =)
14:41:33 <witek> they can be then scraped by prometheus, pushed to Monasca or Gnocchi
14:41:53 <mnaser> HONESTLY I'm a bit in favour of letting prometheus do the scraping
14:42:14 <mnaser> it's a big project, it's got people behind it, no need for us to try and just make things happen in ceilometer when the world has found a 'better' alternative
14:42:20 <mnaser> its also ~cLoUd nAtIvE~
14:42:31 <witek> https://prometheus.io/docs/practices/instrumentation/
14:42:50 <tobberydberg> Like that approach a lot
14:42:51 <witek> that's what prometheus advises about collecting best metrics
14:43:20 <tobberydberg> will it be able to scrape data for all resources that exists in OpenStack space?
14:43:23 <mnaser> now for me the issue remains: <collector> <= <prometheus> => <??? storage ???> <=> <?? exposed api ??>
14:43:43 <mnaser> the storage part is an implementation detail imho, people will decide to implement whatever they want depending on their operational experience
14:44:02 <mnaser> but the tooling to get $data => $hours_consumed .. that's the hard part which would drive the scope down
14:45:39 <witek> tobberydberg: that's the point, the exporter won't be able to collect all the data, the services are best suited to provide the valuable information, they should instrument their code
14:45:59 <ncastele> from what I understood, prometheus will collect metrics from the infrastructure, and we plan to base the usage/hours from those metrics, right ?
14:46:11 <ncastele> Not sure following well the part linked to prometheus
14:46:25 <tobberydberg> My experience with prometheus is pretty limited unfortunate
14:47:24 <tobberydberg> mnaser Are you of another opinion when it comes to the ability of being able to use prometheus for collecting all the data we need?
14:47:36 <tobberydberg> (you seam to have some experience with this)
14:47:53 <mnaser> I like prometheus, it can also push data to remote stores natively
14:48:08 <ncastele> what information prometheus will bring to us regarding the usage ? cpus used, ram used ?
14:48:10 <witek> exporter implements the so called `black box` monitoring, view from outside of the application
14:48:18 <tobberydberg> I mean, as you said witek, involving all teams to do this for us would of course super, but don't see that to happen
14:48:31 <mnaser> https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage
14:48:52 <mnaser> imho I'd be inclined on actually having exporters live outside projects
14:49:05 <mnaser> the velocity and release cycle of openstack would kill if we wanted to add a new thing to monitor :)
14:49:40 <witek> but is requested from different parties on a regular basis
14:49:51 <witek> recently from Tim Bell on openstack-discuss
14:50:48 <mnaser> requested != going to happen imho :p
14:50:54 <mnaser> and again, release cycles would hurt us a lot
14:50:57 <mnaser> esp with back port policies :)
14:51:13 <tobberydberg> yea, it will probably not happen
14:51:36 <mnaser> need to get $x metered but running stein? push a patch to train and it won't be back portable to stein because it will involve adding a new feature
14:51:39 <mnaser> which breaks our stable policy
14:52:26 <witek> well, as a starting point we could collect the list services and metrics we're interested in for billing
14:52:37 <mnaser> yeah. imho an openstack collector would be eneat
14:52:44 <mnaser> auto-detects running services and gets all the info
14:53:05 <mnaser> in https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage -- is there a remote storage endpoint that's of preference for folks here.. maybe trying to see if we can all agree on something (so we can work on tooling that talks to it)
14:53:12 <tobberydberg> So, time is sooooon up for todays meeting unfortunate so would be good to summarize a bit and find out next steps
14:53:15 <ncastele> just want to challenge, with an example
14:53:28 <ncastele> if we base the usage on metrics
14:53:36 <ncastele> (just want to be sure to understand)
14:54:02 <ncastele> what happen when a server is suspended for example ? Does it stop to send metrics ?
14:54:54 <mnaser> no, prometheus should still report it, with a state
14:55:07 <mnaser> in the tooling, we can decide to provide # of hours grouped by state
14:55:21 <mnaser> x hours active, y hours shutoff, z hours shelved
14:56:31 <ncastele> Okay. I'm not enough aware of prometheus and what metrics/information it can bring to us to understand properly this discussion, will need to have a look
14:56:53 <tobberydberg> So my suggestion here, we all think about this approach and see if that would be possible working solution, as well as thinking about what remote storage endpoint that would be preferable to see if we can come to a agreement on that
14:57:35 <tobberydberg> If we do that, should we try to schedule a new meeting before next schedule meeting for the group?
14:57:51 <tobberydberg> Or, is it good to have a new meeting in 2 weeks?
14:58:01 * mnaser feels more often is good
14:58:46 <ncastele> as we are bootstraping the topic, we should meet more often in the coming weeks if we want to achieve something
14:58:55 <ncastele> (especially with the summer break that is coming :( )
14:58:56 <tobberydberg> Ok, can agree on that.
14:59:29 <tobberydberg> next wednesday at the same time?
14:59:57 <witek> wednesday or thursday?
15:00:17 <witek> oh, thursday is holiday for me
15:00:46 <tobberydberg> Preferably wednesday for me, public holiday in sweden next thursday, and I know I'm not at home... :-)
15:01:14 <tobberydberg> yea, guess that goes for most of europe
15:01:16 <ncastele> would love to have it Tuesday as wednesday/thursday is holidays, but I can find a way to be there if it's on wednesday
15:01:45 <tobberydberg> mnaser ... tuesday wednesday ?
15:02:17 <mnaser> I will be at CERN next week
15:02:29 <mnaser> but after that it should be okay
15:02:43 <tobberydberg> Please put some thought around this area until next time
15:03:32 <tobberydberg> Lets say Tuesday 1400 UTC next week, after that we use the timeslot we have for this meeting in two weeks
15:03:49 <witek> thanks tobberydberg
15:04:08 <tobberydberg> Thanks a lot for today folks! See you all next week, and have fun at CERN mnaser :-)
15:04:32 <mnaser> thanks for hosting tobberydberg
15:04:34 <mnaser> thaaanks
15:04:40 <ncastele> thanks everyone :)
15:05:16 <tobberydberg> #endmeeting