Wednesday, 2014-10-08

*** jeromatron has joined #magnetodb00:42
*** keith_newstadt has joined #magnetodb01:02
*** keith_newstadt has quit IRC01:07
*** keith_newstadt has joined #magnetodb01:08
*** jeromatron has quit IRC01:21
*** vivekd has joined #magnetodb03:23
*** jeromatron has joined #magnetodb03:35
*** rushiagr_away is now known as rushiagr03:37
*** rushiagr is now known as rushiagr_away03:51
*** rushiagr_away is now known as rushiagr04:36
*** jeromatron has quit IRC05:10
*** ajayaa has joined #magnetodb05:15
*** ajayaa has quit IRC06:16
*** jeromatron has joined #magnetodb06:22
*** ajayaa has joined #magnetodb06:29
*** jeromatron has quit IRC06:53
*** k4n0 has joined #magnetodb07:08
ajayaaHi. Why are the notifications of priority  'debug' not sent to amqp?07:17
ajayaaWhen an item is added to a table, a notification of priority debug is generated, which is not sent to the amqp at all.07:18
ajayaa*without any log or warning07:19
ajayaaikhudoshyn ^^07:19
*** romainh has joined #magnetodb07:49
*** rushiagr is now known as rushiagr_away09:12
ikhudoshynajayaa, hm, looks like smth with notification->logging driver. it is configured in *.conf09:13
ikhudoshyncwang is the one who implemented it09:14
ikhudoshynnow i'm working on moving to oslo.notify but patch is in progress09:14
*** rushiagr_away is now known as rushiagr09:19
ikhudoshynoh, I see09:21
ikhudoshynho particular reason for that from our side09:22
ikhudoshynmagnetodb.openstack.common.* is a code common for all OS projects09:22
ikhudoshyninitially it was just copy-pasted from one project to another09:23
ikhudoshynnow it is extracted as set of oslo.* projects09:23
ikhudoshynwe're working on moving our stuff to use it09:23
ikhudoshynthe only idea that comes to my mind is they try to save MQ from being flooded by numerous debug messages09:24
openstackgerritIllia Khudoshyn proposed a change to stackforge/magnetodb: Add 'last_updated' to 'table_info'
ajayaaikhudoshyn, Even I thought the same. But some of the operations in mdb are sent with debug priority and some are sent with info priority.09:27
ajayaafor e.g., operations such as put_item, query are debug level.09:27
ajayaaIf we are going to od billing via ceilometer these things need to reach message queue.09:28
ikhudoshynin my notification patch i replaced all notifications with debug level to use info09:28
ajayaaikhudoshyn, cool. :)09:29
ikhudoshyni really hope it will get in eventually09:29
ikhudoshynbtw that fact about ingoring debug ones -- great to know, thanks09:29
ikhudoshyni guess not mavy of us were aware of it09:30
ikhudoshyn* many09:30
ajayaaikhudoshyn, Are there any other consumers of notifications other than ceilometer?09:33
ajayaaI feel like, notifications in current format does not help too much with ceilometer.09:34
ikhudoshynas for notification format, oslo.notify provides sort of a spec for notification format09:35
ikhudoshynwe're just to use it09:35
ikhudoshynkinda OS-way))09:35
ajayaaikhudoshyn, the content of notification.09:35
ikhudoshynwhat do you mean?09:35
ajayaa* I should have put in the above way.09:35
ajayaafor e.g. If a user does a query, then message contains query parameters and user and project.09:36
ikhudoshynhere is the spec09:36
ikhudoshynand event_type09:36
ajayaaIt should also contain how many rows were returned or how many bytes were returned.09:37
ikhudoshynwell I don't see any reason why not to do so))09:37
ikhudoshynif u think we need (or you need it) - just raise the discussion, register BP..09:38
ikhudoshynit's open source any way09:38
ajayaaokay. I will come up with the specs and blueprint.09:39
ajayaaikhudoshyn, Would it not depend on your patch?09:39
ikhudoshynwell it shouldn't09:40
ikhudoshynthey won't be able to get merged, but it's ok09:40
ajayaaokay. Thanks.09:40
ikhudoshynwhen team approves it it sometimes requires to get rebased first, but it's a normal process09:41
ajayaaI am familiar with rebase process. I have gone through it several times. :)09:41
ikhudoshynsure) but sometimes it annoys so much))09:42
*** isviridov_away is now known as isviridov09:42
isviridovHello ikhudoshyn, ajayaa09:47
ajayaaHi isviridov,09:48
isviridovJust looked at you coversation.09:48
ajayaaWhat so you think?09:49
ikhudoshynisviridov, o/09:49
isviridovWe are not sending the message on every event, because it will flood the queue.09:49
isviridovIt was discussed the following workagounds09:49
isviridov1. implement notification driver what will agregate the messages09:50
ajayaaisviridov, who is the intended consumer of messages?09:50
isviridov2. use monitorign api for usage data09:51
ajayaaand what do we aim to achieve via sending messages?09:51
isviridovajayaa, it is expeccted to be celiometer or any other billing of monitoring system.09:52
ajayaaisviridov, Let's say that I want to know the amount of bytes user has read already.09:52
ajayaaIf we don't send notifications related to query and scan to message bus, then these kind of answers are impossible know.09:53
isviridovNotifications sound good. But how many notification are you going to send?09:54
ajayaaRight now we can send one notifications per api call, but going forward if the load on message bus increases considerably we can think of sending these notifications at some time interval.09:55
openstackgerritAndrei V. Ostapenko proposed a change to stackforge/magnetodb: Application log rotation
ajayaaFor e.g. if you do a query, then we can send a notification containing the number of bytes the user has read.09:57
isviridovWe have to think about the load from the beginning. Even now we have installation what process up to 40k requests per second. It means 40k messages in the queue for celiometer. Do you think it is possible to handle such a stream by celiometer now?09:59
ajayaaisviridov, I am not sure. We need to ask ceilometer guys about it. But scalability of rabbitmq could be mitigated by using zeromq.10:01
ajayaaBut using monitoring api does not give a complete idea of usage information.10:02
isviridovNot sure that it is true for celiometer ;)10:02
ajayaaIt just gives you number of items and size of table.10:02
ajayaaisviridov, How do you measure read request by an user?10:03
isviridovajayaa, what stops you from extending it? And add counters you need?10:03
isviridovajayaa, we don't. What the goal?10:03
ajayaaisviridov, The goal is to measure everything user is consuming.10:04
ajayaaNot just size of tables or number of items.10:04
isviridovajayaa, let us start from the list and set the priority for each metric.10:06
openstackgerritAndrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request
isviridovI believe the ML is good place for such discussion.10:06
ajayaaisviridov, okay.10:06
isviridovI think we can implement everything with internal counters and exposing it via monitoring api or/and sending agregated information to queue. But with taking into account the expected load, it woun't be so easy as just send message on each request10:08
ajayaaisviridov, I agree with you.10:09
ajayaaisviridov, We need to decide what to send via message bus and what to expose via monitoring api.10:10
*** romainh has quit IRC10:13
isviridovBut anyhow we have to clarify celiometer's capacity as well.10:14
ajayaaisviridov, Yes. Also, We need to quantify the work-load of magnetodb as well.10:16
ajayaaAs you said you have a setup of mdb which is serving 40K rps.10:16
ajayaaWhat is the maximum work-load we are expecting mdb to take on?10:17
isviridovIt should be designed to handle 100K per second10:19
rushiagrisviridov: 'it' == 'magnetodb'??10:22
isviridovrushiagr, yeap10:29
rushiagrisviridov: I have a problem with that :) When cassandra scales infinitely, it would be very weird to provide a scheme on top of it which doesn't scale that much10:30
rushiagrisviridov: why would people use magnetodb then, if we're not assuring them that they don't need to care when their load increases when they start getting more customers/traffic spikes? :)10:31
isviridovHello rushiagr10:31
rushiagrisviridov: hey10:32
isviridovrushiagr, ideally we will have linear scalability and theoretically infinity capacity of requests per second and stored items as well.10:34
isviridovBut we have to have some figure in mind to make better decisions.10:34
openstackgerritAndrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request
isviridovrushiagr, what are your expectations from mdb?10:36
rushiagrisviridov: I think to make better decisions, we should target scalability as much as cassandra can provide10:37
rushiagrisviridov: that should be our ultimate goal10:37
rushiagrisviridov: sorry, in a meeting. Will come back soon to carry on the discussion10:38
isviridovrushiagr, looking forward10:39
*** romainh has joined #magnetodb10:51
rushiagrisviridov: I'm back11:04
rushiagrisviridov: maybe I interpreted your statement wrongly. What I understood was that you said that mdb should target 100k ops per second only11:05
rushiagrisviridov: in the process of scaling linearly, we will need to scale mdb nodes as well11:06
rushiagrisviridov: again, I have not read through the discussion which brought this topic up11:06
isviridovrushiagr, it was a figure to show the ammount of expected load11:07
rushiagrisviridov: but probably it is just a simple thing, like putting multiple mdb nodes behind a load balancer11:07
rushiagrisviridov: this is the short-term goal right? longer term we might want mdb to scale beyond this too..?11:08
isviridovrushiagr, mdb is linear scalable now. But there is always practical limit, what really depends on work load11:09
rushiagrisviridov: you mean to say practical limit on the operations the virtual machine on which MDB is running can perform?11:10
*** openstack has joined #magnetodb14:14
openstackgerritAndrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request
*** rushiagr_away is now known as rushiagr14:21
openstackgerritAndrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request
*** jeromatron has joined #magnetodb14:47
*** jeromatron has quit IRC15:07
*** jeromatron has joined #magnetodb15:07
openstackgerritIllia Khudoshyn proposed a change to stackforge/magnetodb: Use oslo.notify for notifications
*** ajayaa has quit IRC15:51
*** jeromatron has quit IRC15:52
*** k4n0 has quit IRC16:00
*** rushiagr is now known as rushiagr_away16:01
*** ajayaa has joined #magnetodb16:10
*** jeromatron has joined #magnetodb16:56
*** romainh has left #magnetodb16:59
*** ajayaa has quit IRC17:13
*** jeromatron has quit IRC17:25
*** jeromatron has joined #magnetodb17:45
*** jeromatron has quit IRC18:37
*** jeromatron has joined #magnetodb18:38
*** charlesw has quit IRC21:30
*** vnaboychenko has joined #magnetodb22:22
*** jeromatron has quit IRC22:45

Generated by 2.14.0 by Marius Gedminas - find it at!