| *** jamesmcarthur has joined #openstack-publiccloud | 02:41 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** gtema has joined #openstack-publiccloud | 06:58 | |
| *** hberaud|gone is now known as hberaud | 07:03 | |
| *** witek has joined #openstack-publiccloud | 07:34 | |
| *** hberaud is now known as hberaud|lunch | 10:00 | |
| *** gtema_ has joined #openstack-publiccloud | 10:42 | |
| *** gtema has quit IRC | 10:42 | |
| *** hberaud|lunch is now known as hberaud | 11:08 | |
| *** gtema_ has quit IRC | 12:05 | |
| *** trident has quit IRC | 12:21 | |
| *** trident has joined #openstack-publiccloud | 12:26 | |
| *** gtema_ has joined #openstack-publiccloud | 12:27 | |
| *** gtema_ has quit IRC | 13:32 | |
| *** ncastele has joined #openstack-publiccloud | 13:49 | |
| *** ricolin has joined #openstack-publiccloud | 13:57 | |
| tobberydberg | o/ | 14:00 |
|---|---|---|
| ncastele | hi there | 14:02 |
| tobberydberg | Hi ncastele | 14:05 |
| tobberydberg | I wonder if all the rest are on an early summer vacation =) | 14:05 |
| *** jamesmcarthur has joined #openstack-publiccloud | 14:05 | |
| ncastele | Summer vacation, in the beginning of June ? :o | 14:05 |
| ncastele | Too early, I'm not ready yet to see people on vacation :D | 14:06 |
| tobberydberg | National holiday in Sweden today, but not many swedes in these meetings anyway ;-) | 14:06 |
| tobberydberg | Hehehe...we have like 28 degrees and sunny the last couple of days ... hot as hell for Sweden | 14:06 |
| tobberydberg | In a lot of ways I'm ready for vacation ;-) | 14:07 |
| ncastele | Two days of storm and rain in France. Can we switch ? | 14:07 |
| tobberydberg | Hahaha, shit ... well, not sure, think I keep our weather for a few more days to be honest ;-) | 14:07 |
| *** wondra has joined #openstack-publiccloud | 14:08 | |
| tobberydberg | So, are we anymore here for todays meeting? mnaser? | 14:08 |
| mnaser | i will jump in and out, tc meeting right now (once a month) | 14:09 |
| tobberydberg | wondra ? zhipeng ? | 14:09 |
| wondra | Hi! | 14:09 |
| tobberydberg | ok ok, totally fine of course ... just would like to be a few here before kicking it off | 14:09 |
| tobberydberg | welcome wondra | 14:10 |
| mnaser | i'll def be in and out | 14:10 |
| tobberydberg | So, lets start then | 14:10 |
| tobberydberg | #startmeeting publiccloud_wg | 14:10 |
| openstack | Meeting started Thu Jun 6 14:10:45 2019 UTC and is due to finish in 60 minutes. The chair is tobberydberg. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:10 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:10 |
| *** openstack changes topic to " (Meeting topic: publiccloud_wg)" | 14:10 | |
| openstack | The meeting name has been set to 'publiccloud_wg' | 14:10 |
| tobberydberg | Simple agenda this time | 14:11 |
| ncastele | Yep | 14:11 |
| tobberydberg | #topic 1. Joint development effort billing | 14:11 |
| *** openstack changes topic to "1. Joint development effort billing (Meeting topic: publiccloud_wg)" | 14:11 | |
| tobberydberg | notes from last meeting is here: #link http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-05-28-14.05.log.html | 14:12 |
| ncastele | homeworks have been done in the etherpad, and I'm quite happy we both wrote the same stuff | 14:12 |
| ncastele | https://etherpad.openstack.org/p/publiccloud-sig-billing-implementation-proposal | 14:12 |
| tobberydberg | Yea, some parts are there for sure ... might be more there ... would be good if more people fill in their needs as well so we can get a full list of the needs | 14:13 |
| ncastele | +1 | 14:13 |
| tobberydberg | But yes, pretty much the same | 14:13 |
| tobberydberg | I guess we both feel that we need something that fires events as well as continues reports .... scraping method potentially | 14:14 |
| *** gtema_ has joined #openstack-publiccloud | 14:16 | |
| ncastele | Scraping is a way to go, but can lead to an impact on control plane performances | 14:16 |
| tobberydberg | What I was thinking about is that we basically need a tool or a set of tools that can handle all form of metrics, not only the metrics for the resources we bill for, but for things that are for general information/statistic purposes as well | 14:17 |
| tobberydberg | well ... depends on how that scraping is done I would assume | 14:17 |
| ncastele | what do you mean by things for general information ? | 14:17 |
| tobberydberg | could be security groups and rules for instance | 14:18 |
| tobberydberg | SOME might bill for that (we do not), but it would still be of interest for me to see the usage of that | 14:18 |
| tobberydberg | Hard to track such thing today since neutron doesn't store anything deleted in the databse | 14:19 |
| ncastele | Yes | 14:19 |
| tobberydberg | barbican stuff is another thing | 14:19 |
| wondra | What do you mean by scraping? | 14:19 |
| wondra | API requests? | 14:19 |
| ncastele | there are no retention at all in neutron db ? | 14:19 |
| ncastele | API requests, or nice SQL queries on database | 14:20 |
| wondra | Nice? :-) | 14:20 |
| tobberydberg | not what I know of, no (on the retention) | 14:20 |
| wondra | What about the usage of ceilometer events as I suggested? That is a log of the history in itself. | 14:20 |
| tobberydberg | wondra I guess you can doing scraping in different forms ... but one can be db queries, you can also scrape each compute node for its resources ... | 14:21 |
| ncastele | I'm not enough into ceilometer to ensure it can answer all our needs | 14:21 |
| tobberydberg | What I hear from people, the ceilometer agents are pretty stable in that sense, but still, people do not fully rely on it anyone | 14:22 |
| tobberydberg | *anyway | 14:22 |
| wondra | Our billing is based on that. We did it back in Kilo and will have to rework it due to an API change for Ocata. | 14:22 |
| tobberydberg | usually having something else as well and then comparing the results | 14:23 |
| wondra | You get events about every entity along with details, like the owner, size, etc. | 14:23 |
| tobberydberg | yes, which is good | 14:23 |
| ncastele | Is it easy to enhance the content of an event with custom information depending of the service ? | 14:23 |
| tobberydberg | the storage of that data and possibility to query it is hard to work with though | 14:24 |
| wondra | Dunno. But you can choose what you store from the notification which is being sent by the particular OpenStack project in the events pipeline.yaml | 14:24 |
| wondra | https://docs.openstack.org/ceilometer/ocata/events.html | 14:24 |
| wondra | Querying it would be done with the Panko API, which we haven't learned yet. Still on the old Ceilometer one. We basically query it every day on midnight for every tenant and compute the usage. | 14:25 |
| tobberydberg | It's been a little bit to long since I looked into all the side projects around ceilometer, but listening to people it seams not to work very well | 14:26 |
| *** hberaud is now known as hberaud|school-r | 14:26 | |
| wondra | How does CloudKitty work anyway? Doest it query Panko or Gnocchi? | 14:26 |
| tobberydberg | gnocchi being outside of openstack makes it worse | 14:27 |
| tobberydberg | think it uses gnocchi | 14:27 |
| witek | they don't rely on a specific DB from what I know | 14:27 |
| ncastele | Cloudkitty offers drivers for prometheus/ceilometer/gnocchi | 14:27 |
| witek | they can use Gnocchi, Monasca, Prometheus | 14:27 |
| tobberydberg | ok ok | 14:28 |
| witek | :) | 14:28 |
| tobberydberg | are you folks basing your billing today on all native openstack services? | 14:29 |
| tobberydberg | we do not, now days not at all using openstack telemetry services for that (unfortunately) | 14:30 |
| ncastele | for the collecting part, yes, we are relying on ceilometer (in a heartbeat way, not in an event way) | 14:30 |
| ncastele | for instances | 14:30 |
| ncastele | For volumes, we have some sql queries that are pushing usage data into ceilometer | 14:31 |
| tobberydberg | ok, so ceilometer doesn't work for that? | 14:31 |
| ncastele | but it's a scrapping way (scrapping hypervisor, scrapping our ceph clusters, etc.) | 14:31 |
| ncastele | ceilometer does the work because we scrap each 5 minutes, but we bill by hour, meaning that we accept we can lose some points | 14:32 |
| mnaser | oh i didn't know cloudkitty can use prometheus TIL | 14:32 |
| tobberydberg | ok. I would say that we need something that is down at seconds level of usage for all type of resources | 14:33 |
| mnaser | i think if we forget scraping and bring evented monitoring, we dont have to track things per second anymore, because you can easily introspect things "what time did it start and end" | 14:33 |
| ncastele | but when we come to seconds billing, it's not possible for us to use the actual system because for this precision, we need to base on events/on precise dates and time | 14:33 |
| ncastele | +1 mnaser | 14:34 |
| tobberydberg | mnaser totally agree | 14:34 |
| mnaser | then it's $bill_in_whatever_increment_you_want | 14:34 |
| tobberydberg | byes | 14:34 |
| witek | seconds meaning 1s, 10s or 30s ? | 14:35 |
| tobberydberg | do we need the scraping for other purposes? | 14:35 |
| tobberydberg | 1s | 14:35 |
| ncastele | seconds meaning 1s :D | 14:35 |
| tobberydberg | or even less than seconds event based will give you | 14:35 |
| wondra | I believe that we do it for Floating IPs. Cinder is fine with cinder-volume-audit in cron, but Neutron does not have that. | 14:36 |
| witek | the actual question is if such fine resolution is really useful for billing? | 14:36 |
| ncastele | it depends, if we trust at 100% our event based collecting, then we do not need scrapping. But scrapping can consolidate the event base process | 14:36 |
| tobberydberg | I would say CPU load etc etc will need scraping if we sould like to cover those bits as well | 14:36 |
| *** hberaud|school-r is now known as hberaud | 14:36 | |
| wondra | To clarify - most openstack components issue the .exist event, which allows you to find entities that you missed the .start notifications for. Floating IPs do not have it. | 14:37 |
| tobberydberg | so same question again, if events is enough, is ceilometer reliable to use in its current shape and form? | 14:38 |
| wondra | Billing by CPU load - the white unicorn of public clouds? | 14:39 |
| tobberydberg | (in just collecting the "metrics") | 14:39 |
| wondra | Dunno. I'm 4 releases behind. | 14:39 |
| ncastele | Dunno either | 14:39 |
| tobberydberg | mnaser what is you feeling there? | 14:39 |
| wondra | Eh, actually 6. Damn. | 14:39 |
| tobberydberg | wondra I was thinking just to have the same source of data to stuff that are interesting to visualise for users ... not necessarily bill for it =) | 14:40 |
| tobberydberg | wondra just go directly to train this fall ;-) | 14:41 |
| wondra | Having visualisations in our customer portal would be nice. I've got example code from a bachelor's thesis for Gnocchi. | 14:41 |
| tobberydberg | +1 | 14:42 |
| wondra | With our main product being VPS, we don't need it in Horizon. | 14:42 |
| mnaser | i think it's probably easier to start building the structure for the billable stuff | 14:43 |
| tobberydberg | The biggest issue that I see here moving forward will be the storage and real time query bits | 14:43 |
| mnaser | imho | 14:43 |
| mnaser | well, we think of things as a resource, that's what ceilometer did at the time, resource X (could be cinder uuid) which starts created_at $x and deleted_at $y (or none) | 14:43 |
| mnaser | and then we have a tool that says "query how many seconds has resource X been used in the period A => B" and it can do all the math | 14:44 |
| mnaser | i have a lot of that code | 14:44 |
| tobberydberg | event driven only, and I think that will cover the billing bits | 14:45 |
| tobberydberg | so, using ceilometer for that mnaser? Reading off the rabbit queue? | 14:46 |
| mnaser | nope, reading the db :\ | 14:46 |
| tobberydberg | neutron? | 14:46 |
| mnaser | not billing anything for that | 14:46 |
| tobberydberg | ok. I totally agree that reading o the database is very easy for a lot of the resources, but not for all | 14:47 |
| tobberydberg | object storage is another thing that will be har to track that way | 14:47 |
| tobberydberg | dynamic size stuff | 14:48 |
| mnaser | well i was thinking for that sort of thing, parsing logs might be the way to go | 14:48 |
| mnaser | logs are pretty much events of download/upload i guess | 14:49 |
| wondra | Reading the DB is bad. There's no clear contract. Any OpenStack project can change it and break the code. | 14:49 |
| tobberydberg | true | 14:49 |
| ncastele | agree with wondra, the db contract is not stable. But it's the easiest and less impacting way to read data | 14:52 |
| tobberydberg | So, just a few more minutes left of todays meeting, would be good to sum it up and find action points until next meeting | 14:52 |
| ncastele | I tried a PoC on reading from Nova APIs as an admin, and I quickly reached some limits | 14:52 |
| wondra | But the deployment tools are trailing the OpenStack releases. Maybe a billing product that reads the database could only exist for 1 year old releases... | 14:53 |
| tobberydberg | #action All - continue to define what we really need to track | 14:53 |
| tobberydberg | what else? more clear suggestions on collection method of events? DB queries? whatnot? | 14:55 |
| tobberydberg | Should we try to book a meeting for next week as well to keep up the pace before all vacations? | 14:55 |
| tobberydberg | I'm happy to take a meeting same time next week | 14:56 |
| wondra | Not against. I have said most of what I know, though. | 15:00 |
| tobberydberg | well, time is up for today ... any opinions on above before we close down? | 15:01 |
| tobberydberg | Thanks wondra ... have you added your needs of metrics to the etherpad as well? | 15:01 |
| witek | good understanding of what exactly has to be collected is important | 15:01 |
| wondra | yes, I have. | 15:02 |
| tobberydberg | yes, I think so too and that is what we need to base tooling etc on to be successful | 15:02 |
| tobberydberg | +1 wondra | 15:02 |
| tobberydberg | Ok. So I'll end todays meeting. I send out a reminder for a meeting next week, and we continue with once per week for a few more weeks and see what we can get out of that | 15:03 |
| tobberydberg | Thanks for today folks! | 15:03 |
| witek | thanks, I'm in vacation, see you in 3 weeks | 15:04 |
| tobberydberg | #endmeeting | 15:05 |
| *** openstack changes topic to "New meeting time!! Thursday odd weeks at 1400 UTC in this channel!!" | 15:05 | |
| openstack | Meeting ended Thu Jun 6 15:05:17 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:05 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-06-06-14.10.html | 15:05 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-06-06-14.10.txt | 15:05 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-06-06-14.10.log.html | 15:05 |
| ncastele | thanks for the meeting (sorry for the end of the meeting, I was a bit on an emergency on the infra :o ) | 15:05 |
| ncastele | Goal for next meeting is to precise our needs for collecting, got it :) | 15:06 |
| *** wondra has quit IRC | 15:12 | |
| *** hberaud is now known as hberaud|gone | 16:19 | |
| *** witek has quit IRC | 16:32 | |
| *** ricolin has quit IRC | 16:54 | |
| *** mrhillsman is now known as openlab | 17:02 | |
| *** openlab is now known as codebauss | 17:05 | |
| *** codebauss is now known as openlab | 17:13 | |
| *** openlab is now known as codebauss | 17:14 | |
| *** codebauss is now known as openlab | 17:15 | |
| *** openlab is now known as codebauss | 17:16 | |
| *** codebauss is now known as mrhillsman | 17:24 | |
| *** jamesmcarthur has quit IRC | 17:33 | |
| *** gtema_ has quit IRC | 17:50 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 18:56 | |
| *** jamesmcarthur has quit IRC | 19:12 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 19:14 | |
| *** jamesmcarthur has quit IRC | 19:55 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 19:56 | |
| *** ncastele has quit IRC | 20:02 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 20:27 | |
| *** jamesmcarthur_ has joined #openstack-publiccloud | 20:29 | |
| *** jamesmcarthur has quit IRC | 20:31 | |
| *** jamesmcarthur_ has quit IRC | 21:03 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 21:05 | |
| *** jamesmcarthur_ has joined #openstack-publiccloud | 21:16 | |
| *** jamesmcarthur has quit IRC | 21:18 | |
| *** jamesmcarthur_ has quit IRC | 21:36 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 21:37 | |
| *** jamesmcarthur has quit IRC | 21:46 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 23:13 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 23:17 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 23:45 | |
| *** jamesmcarthur has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!