*** elynn has joined #senlin | 00:09 | |
*** elynn has quit IRC | 00:34 | |
*** Qiming has quit IRC | 00:47 | |
*** flwang has joined #senlin | 01:31 | |
*** Qiming has joined #senlin | 01:45 | |
*** yanyanhu has joined #senlin | 01:45 | |
*** elynn has joined #senlin | 01:52 | |
*** wei1 has joined #senlin | 01:53 | |
*** wei1 has quit IRC | 01:57 | |
flwang | Qiming: ping | 01:59 |
---|---|---|
yanyanhu | flwang, feilong? | 02:05 |
Qiming | hello sir | 02:09 |
Qiming | :) | 02:09 |
elynn | morning senlin :) | 02:09 |
yanyanhu | morning | 02:10 |
yanyanhu | hi, Qiming, have talked with rally team. I think the first patch for basic support of senlin will be merged after their 4.0 release is done | 02:10 |
yanyanhu | and they also hope us to add those plugins into rally repo | 02:11 |
Qiming | they are not synced with openstack release? | 02:11 |
Qiming | what does it mean by a 4.0 release | 02:12 |
yanyanhu | no I guess | 02:12 |
yanyanhu | oh, sorry, 0.4.0 | 02:12 |
yanyanhu | looks like their release was still not done till yesterday | 02:13 |
Qiming | oh, I see | 02:13 |
yanyanhu | so senlin support will not be included in that release | 02:13 |
Qiming | there is no deliverables registered in openstack/releases project | 02:13 |
Qiming | not a big deal | 02:14 |
yanyanhu | so, they use different release mode from us? | 02:14 |
yanyanhu | ok | 02:14 |
yanyanhu | so I think this patch will be used for test of gate job https://review.openstack.org/307170 | 02:14 |
yanyanhu | and finally those plugins will be delivered to rally I guess | 02:14 |
yanyanhu | just leave jobs inside senlin repo | 02:15 |
Qiming | seems they are still doing independent releases | 02:15 |
Qiming | http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml#n3695 | 02:15 |
yanyanhu | oh, I see | 02:15 |
yanyanhu | I recalled they joined big tent a little bit earlier than us? | 02:16 |
Qiming | yes | 02:16 |
yanyanhu | ok. I will keep working on this and talk with them if needed | 02:17 |
Qiming | btw, patch 307170 makes little to no sense to me | 02:17 |
yanyanhu | yes? | 02:17 |
Qiming | it is doing exactly what we were trying to avoid | 02:17 |
yanyanhu | oh, you mean the duplicated code? | 02:17 |
Qiming | yes | 02:17 |
yanyanhu | yes. But looks like there is no other way to do so... | 02:18 |
yanyanhu | since tempest scenario in rally is not for this purpose | 02:18 |
Qiming | why we cannot share the code? | 02:18 |
yanyanhu | and those plugins will finally go to rally side, so we don't need to maintain them in senlin | 02:19 |
yanyanhu | you mean with tempest test cases defined in senlin? | 02:19 |
Qiming | for example, https://review.openstack.org/#/c/307170/2/rally-jobs/plugins/senlin_plugin.py, the _create_cluster() function is doing nothing new | 02:19 |
yanyanhu | that's because rally doesn't support benchmarking using tempest scenario. Only verification support is provided | 02:19 |
Qiming | if we are adding a 'health_threshold' property to a cluster | 02:20 |
yanyanhu | so if we want to benchmark something like cluster_create/cluster_delete individually, we need to define them as individual actions | 02:20 |
Qiming | there are too many places to sync now | 02:20 |
yanyanhu | that's true... | 02:20 |
Qiming | when the code wasn't updated, it becomes outdated, not accurate, or even wrong | 02:21 |
yanyanhu | yes, but I guess if we want to benchmarking senlin in that fine granularity using rally, we may have to do so | 02:21 |
yanyanhu | actually we don't implement too much cases using rally I think | 02:22 |
yanyanhu | much less then cases we defined in tempest plugin | 02:22 |
Qiming | line 71-90 | 02:22 |
Qiming | for example | 02:22 |
*** zzxwill__ has quit IRC | 02:22 | |
*** zzxwill__ has joined #senlin | 02:23 | |
Qiming | function _create_cluster was defined as just a generic function | 02:23 |
yanyanhu | yes | 02:23 |
Qiming | it has too many parameters and most of them have default values | 02:23 |
yanyanhu | you mean we should remove those default values? | 02:23 |
Qiming | it looks like a new fork of the senlinclient library | 02:24 |
Qiming | if we are benchmarking things, we'd better have a good plan, what are the cases we care | 02:24 |
yanyanhu | yes, since user can only customize the parameter that defined in the action function when performing benchmarking | 02:24 |
Qiming | what are the cases that are representative | 02:24 |
Qiming | and ... this plugin is using its own client, instead of invoking the APIs directly | 02:26 |
Qiming | is that the expectation, or I'm missing something | 02:26 |
yanyanhu | yes, that is also following rally's osclient | 02:26 |
Qiming | so the benchmarking is about an end-to-end scenario, right? | 02:26 |
yanyanhu | Qiming, yes | 02:27 |
yanyanhu | I think the time counting is with the granularity of that function | 02:27 |
Qiming | it is not about the service time between the moment API receiving a request to the moment the action is done | 02:27 |
yanyanhu | so includes all interaction though client | 02:28 |
yanyanhu | yes, it's not for senlin's action granularity | 02:28 |
Qiming | it is counting REST call time, SDK processing time, keystone authentication delay, middleware processing time, all those stuff | 02:28 |
yanyanhu | yes | 02:28 |
Qiming | so it has nothing to do with tempest then | 02:29 |
yanyanhu | I guess if we want to collect time data in that level, we need some checkpoint inside senlin? | 02:29 |
Qiming | em, don't think so | 02:29 |
Qiming | you can get an idea by checking the logs already | 02:30 |
Qiming | all log entries have timestamps | 02:30 |
yanyanhu | yes, but that's not the way we should use I guess | 02:30 |
Qiming | then we should use tempest for that purpose | 02:31 |
yanyanhu | benchmark should be done with treat test object as blackbox I guess? | 02:31 |
Qiming | leave rally outside of this test | 02:31 |
Qiming | I have no idea what rally is trying to do | 02:31 |
Qiming | if you are testing the performance by invoking senlinclient | 02:31 |
Qiming | you are doing an end-to-end performance test | 02:32 |
Qiming | it is about your cloud deployment as a whole | 02:32 |
yanyanhu | the ability we want to leverage from rally is statistics and result collecting and visualization | 02:32 |
Qiming | yes | 02:32 |
Qiming | but if you have hard-coded the invocation by bringing in senlinclient | 02:33 |
yanyanhu | if we want to test every detail of our own service, I guess we need more work to do by ourselves | 02:33 |
yanyanhu | inside senlin to collecting those time data | 02:33 |
Qiming | the only thing it can do is for end-to-end benchmarking | 02:33 |
yanyanhu | I see | 02:33 |
Qiming | this line: @atomic.action_timer("senlin.list_clusters") | 02:33 |
yanyanhu | so it's not for benchmarking senlin's service performance | 02:34 |
Qiming | is it just a label or something? | 02:34 |
Qiming | exactly | 02:34 |
Qiming | that is my concern | 02:34 |
yanyanhu | oh, this label will be shown in test results | 02:34 |
Qiming | I'm not against rally, but we have to be sure what we are doing | 02:34 |
flwang | Qiming: i have question about your api-ref | 02:34 |
flwang | i saw you copied some code from nova | 02:34 |
Qiming | it could be: @atomic.action_timer("senlin.make_clusters_dance_blah_blah") | 02:34 |
Qiming | yes, flwang | 02:35 |
Qiming | those code, I suppose will eventually go back to api-site | 02:35 |
flwang | but nova does that change is because swagger doesn't support micro version | 02:35 |
Qiming | it is there only to make the tox job happy, and to generate the same look-n-feel of api doc locally | 02:36 |
flwang | does senlin has the same issue? | 02:36 |
Qiming | flwang, we just added micro-versioning support | 02:36 |
Qiming | we believe it is very very important to keep API versioned | 02:36 |
Qiming | so we will, have the same problem | 02:36 |
Qiming | I hate the fact that the doc team is "swaggering" this way and that | 02:37 |
Qiming | I had a WIP for swagger locally, not committed | 02:37 |
Qiming | then I learned from Anne, that won't be the plan any more | 02:38 |
flwang | sorry? | 02:38 |
Qiming | they are more leaning towards a (short-sighted) hack | 02:38 |
flwang | do you mean swagger won't the plan? | 02:38 |
Qiming | this one? http://lists.openstack.org/pipermail/openstack-dev/2016-March/090659.html | 02:39 |
flwang | i don't think so. | 02:39 |
Qiming | hile we're not completely giving | 02:39 |
Qiming | up on Swagger, we're also recognizing the engineering effort to maintain | 02:39 |
Qiming | and sustain those efforts isn't going to magically appear. | 02:39 |
flwang | based on my understanding, swagger still could be the way we will go for | 02:40 |
Qiming | I don't think docs team has technical arsenal to drive this | 02:40 |
Qiming | I don't like the way Sean is doing either | 02:40 |
flwang | Qiming: i will forward a mail between anne and me | 02:41 |
Qiming | okay | 02:41 |
flwang | personally, i would like each team just work out a swagger json and api-site will extract them and build it | 02:41 |
flwang | anne said it should work, and i'm going to do that | 02:41 |
*** elynn_ has joined #senlin | 02:41 | |
Qiming | Ideally, the whole community should make a decision | 02:42 |
Qiming | I was creating some json using the fairy-slipper thing I guess? | 02:42 |
flwang | Qiming: btw, are you guys still interested in using zaqar in senlin? | 02:42 |
Qiming | flwang, definitely | 02:43 |
Qiming | but ... I tried ping you several times | 02:43 |
flwang | is there a bp/spec to track it? | 02:43 |
flwang | Qiming: sorry, pls blame timezone | 02:43 |
Qiming | to report that zaqar functional tests break | 02:43 |
flwang | i owe you a beer :) | 02:43 |
flwang | functional test? | 02:43 |
Qiming | there are many api return code doesn't match | 02:43 |
Qiming | tox -e functional | 02:43 |
Qiming | Charles has been looking into zaqar recently | 02:44 |
Qiming | although the whole team is keen to know what we can achieve during Newton | 02:44 |
*** elynn has quit IRC | 02:45 | |
flwang | Qiming: i think we don't support tox -e functional | 02:45 |
flwang | we use tox -e integration | 02:45 |
Qiming | We are not tracking this using a BP yet, it is documented in a TODO.rst file so we won't forget | 02:46 |
Qiming | okay, you are using 'integeration' to run functional tests then | 02:47 |
Qiming | http://git.openstack.org/cgit/openstack/senlin/tree/TODO.rst#n77 | 02:47 |
flwang | based on the name 'Zaqar queue based receiver.' it could be very easy i think | 02:48 |
flwang | you just need create a queue and post message to the queue, done | 02:48 |
Qiming | if it is a receiver | 02:48 |
Qiming | it means senlin will create a "listener" or "subscriber" | 02:49 |
flwang | if you can let me know what's a 'receiver' in senlin, i can give you some ideas :) | 02:49 |
Qiming | it is an abstraction, with different flavors | 02:49 |
Qiming | by default, when you do senlin reiver-create, you will get a webhook, to which you can HTTP POST a signal | 02:50 |
Qiming | we want to extend that to receiver-create -t message, so the caller can get a queue name/address/id or something | 02:50 |
flwang | ah, aodh just done a similar thing | 02:50 |
flwang | https://review.openstack.org/#/c/264309/ | 02:51 |
Qiming | when another alert system is posting things into that queue, senlin will get notified | 02:51 |
flwang | so senlin is the 'subscriber'? | 02:51 |
Qiming | yes | 02:52 |
flwang | does it need an auth? | 02:52 |
flwang | token i mean | 02:52 |
Qiming | in our context, we want to enable some (or any) senlin operations to be triggered by external alerts | 02:52 |
Qiming | for webhook, no | 02:53 |
Qiming | for zaqar, that is a good question | 02:53 |
flwang | if so, it should be easy | 02:53 |
flwang | let me repeat the user case | 02:53 |
flwang | 1. user create a queue in zaqar named A | 02:54 |
flwang | 2. create a subscription in zaqar on the queue A and the 'subscriber' is an url of senlin, so that senlin can be notified | 02:54 |
flwang | 3. post messages the queue A | 02:54 |
flwang | 4. senlin is notified | 02:54 |
flwang | is it correct? | 02:54 |
Qiming | in step 1, we don't want to bother a user to do queue creation | 02:55 |
flwang | Qiming: no worries, zaqar support lazq queue | 02:55 |
flwang | lazy | 02:55 |
flwang | you don't have to create the queue explicitly | 02:56 |
Qiming | step 1 will be invoked by senlin internally | 02:56 |
flwang | so we even don't need the step1 | 02:56 |
Qiming | the queue name 'A' is something the event alarm system need to know | 02:56 |
flwang | ok, it could be a pre-defined patten i think | 02:57 |
Qiming | exactly | 02:57 |
Qiming | then when senlin-receiver is successful, it means senlin is saying: okay, I have a queue for you to post your request, come on | 02:58 |
Qiming | just like it used to say: okay, I have a webhook for your to invoke, let me know if you have a request | 02:59 |
flwang | then, the question is when zaqar notify senlin, the subscriber is a link/endpoint of senlin, right? | 03:00 |
flwang | does the endpoint need auth? | 03:00 |
Qiming | what are the formats you support for a subscriber? | 03:01 |
Qiming | Can I just look into the queue directly? | 03:01 |
Qiming | when are user wants to post a message, I was expecting he/she was authenticated by zaqar | 03:02 |
Qiming | s/are/a/ | 03:03 |
flwang | when you create a subscriptoin, the post body is 'subscriber', 'ttl' and 'options' | 03:04 |
flwang | subscriber could be a http/https, email | 03:04 |
flwang | yes, you can look into the queue for sure | 03:04 |
Qiming | ... so for senlin, it is just another webhook | 03:04 |
Qiming | not triggered by the monitoring system directly | 03:05 |
flwang | TBH, i'm a little bit confused | 03:05 |
flwang | so what's the 'subscriber' look like? | 03:05 |
flwang | it's another 3rd party URL or a senlin's endpoint? | 03:06 |
Qiming | I'm not sure I understand what a 'subscriber' means | 03:07 |
Qiming | okay, let's get back to the webhook use case | 03:07 |
flwang | 'subscriber' is a URL or email address | 03:07 |
Qiming | right | 03:08 |
flwang | so when there is a message posted to the queue, zaqar will forward the message to post it to the URL or send it to the email address | 03:08 |
Qiming | if what I should (have to) provide you is a senlin endpoint | 03:08 |
Qiming | for zaqar to invoke | 03:08 |
Qiming | that is no difference for us, as a webhook | 03:08 |
Qiming | a webhook is something like this: http://{host:port}/v1/webhooks/{webhook_id}/trigger?V=1 | 03:09 |
Qiming | that would be the so-called subscriber to a zaqar queue | 03:10 |
Qiming | a monitoring service/system can invoke that URL directly | 03:10 |
Qiming | thus eliminating the need of zaqar in between | 03:10 |
flwang | so user(3rd party) send notification to the webhook? | 03:11 |
Qiming | yes | 03:12 |
flwang | and the webhook call senlin? | 03:12 |
Qiming | the webhook is a senlin endpoint | 03:12 |
Qiming | when you are doing a HTTP POST to that URI, an action will be triggered on a specific cluster | 03:13 |
flwang | does the webhook need auth to access? | 03:13 |
Qiming | no | 03:13 |
Qiming | unless I tell you where it is | 03:13 |
Qiming | there is no listing or showing API related to webhook | 03:14 |
flwang | i see | 03:14 |
Qiming | webhook_id is a UUID | 03:14 |
Qiming | note that in "http://{host:port}/v1/webhooks/{webhook_id}/trigger?V=1" there is a "V=1" | 03:14 |
Qiming | that means a version 1 protocol | 03:15 |
Qiming | when security concern rises, we can move to V=2, for example, while requiring more parameters on the URL telling me who you are | 03:15 |
flwang | ok, so based on my understanding, even zaqar is a good thing for this case, it's not senlin's business, right? | 03:16 |
Qiming | it was not what we have imagined | 03:16 |
flwang | so what you were looking for? | 03:17 |
Qiming | when creating a receiver, we hope we can create something other than webhooks | 03:17 |
Qiming | because webhooks are exposing the controller's IP address | 03:17 |
Qiming | that is dangerous | 03:17 |
flwang | ok, if that's the key | 03:18 |
Qiming | instead, we are looking to something like 'senlin receiver-create -c my_cluster -a CLUSTER_SCALE_OUT -f message' | 03:18 |
flwang | then we could meet somewhere | 03:18 |
Qiming | then 'senlin receiver-show' | 03:18 |
flwang | let me repeat and pls fix it | 03:18 |
flwang | 1. senlin receiver-create -c my_cluster -a CLUSTER_SCALE_OUT -f message will return a queue name to user(3rd party) | 03:19 |
Qiming | you will get something like '{'receiver': {'id': xxx, 'channel': {'queue': {....}}}' | 03:19 |
Qiming | yes | 03:19 |
flwang | and meanwhile, senlin will create the subscription with his webhook | 03:19 |
flwang | user(3rd party) can't access the subscriptions, so they can't know the webhook url(ip) | 03:20 |
Qiming | when creating that receiver, senlin has invoked zaqar to create a queue, and senlin will listen on that queue | 03:20 |
flwang | 2. user(3rd party) send messages to the queue | 03:20 |
flwang | senlin is notified | 03:20 |
flwang | senlin don't have to listen | 03:21 |
flwang | senlin will be notified | 03:21 |
Qiming | ... why we have to mix webhooks and message queues? | 03:21 |
Qiming | they are different kinds of 'receivers' to senlin | 03:21 |
flwang | ok, sorry | 03:21 |
Qiming | when a user decides that a webhook is appropriate, we don't need man-in-the-middle | 03:21 |
flwang | i seeeeeeeeeeeeeeeeee | 03:22 |
Qiming | if they believe a messaging based notification is more appropriate, they will abandon webhook | 03:22 |
flwang | so for the zaqar case, you don't want to(have to) be notified | 03:22 |
flwang | you just want to listen/poll the queue, right? | 03:22 |
Qiming | it is a pretty harsh binary ... this assumption may be totally invalid, ;) | 03:22 |
Qiming | yes | 03:23 |
flwang | i see | 03:23 |
Qiming | just like another piece of code where we are listening to oslo.messaging | 03:23 |
Qiming | we need to know the topic and some other parameters before doing so | 03:23 |
Qiming | (we were trying to listen to oslo.messaging so that we know a VM we created is down ...) | 03:24 |
flwang | Qiming: ok, if that's what you guys want, super easy i think | 03:28 |
Qiming | :) | 03:29 |
Qiming | thanks | 03:29 |
flwang | i was confused just because i always think you want to be notified instead of listening/polling | 03:29 |
Qiming | very interested in seeing a strawman running, then we can figure out how to solve the authentication problem, if any | 03:29 |
flwang | and i'm sure you guys will be benefited from queue | 03:29 |
Qiming | hoho | 03:30 |
Qiming | nice | 03:30 |
flwang | may i know what's the message looks like? | 03:30 |
flwang | like an alarm? | 03:30 |
Qiming | a json body | 03:30 |
Qiming | something like that | 03:30 |
flwang | i mean the content | 03:31 |
flwang | ok, then queue is good for you guys | 03:31 |
flwang | imagine a scenario | 03:31 |
flwang | you got 2 alarms and they are different, what does senlin to do? | 03:31 |
Qiming | it could be as simple as this: {"parameters": {"count": 3, "strict": false}} | 03:32 |
Qiming | this is a parameter to a CLUSTER_SCALE_IN operation | 03:32 |
Qiming | all optional | 03:32 |
flwang | i mean an alarm received at 13:01 say cpu usage is 95%, and alarm got at 13:05 say cpu usage is 60% | 03:32 |
Qiming | yes, users decide what parameters they will pass in the above body | 03:33 |
flwang | i mean senlin could be benefited from a queue system | 03:33 |
flwang | so that it can make some decisions more smart | 03:33 |
Qiming | :) | 03:33 |
Qiming | we are trying to make senlin stupid | 03:33 |
flwang | instead of processing each message even they're confict | 03:33 |
Qiming | leaving the decisions to users | 03:33 |
flwang | ok, then skip my words :) | 03:34 |
Qiming | users will say when 13:01 say cpu usage is 95%, they will do CLUSTER_SCALE_OUT by 50%, they instruct the monitoring system to do so | 03:34 |
Qiming | senlin will check the request parameters and do what it was asked | 03:35 |
flwang | ok | 03:35 |
Qiming | 5 minutes later, another alarm comes in | 03:35 |
flwang | i'm just thinking aloud, i'm a newbie for senlin | 03:35 |
Qiming | senlin will say ... no, you just said that 95%, isn't you? | 03:35 |
Qiming | I'm not interested, you can try tell me something new at 13:11 | 03:36 |
Qiming | your cluster is in a 'cool_down' period | 03:36 |
flwang | Qiming: that's what i'm talking about i think | 03:36 |
flwang | senlin could do it more smart | 03:36 |
Qiming | right, it is a tricky area | 03:37 |
flwang | instead of doing anything | 03:37 |
flwang | at least, it could be a config or something like that | 03:37 |
Qiming | we want to build the mechanisms layer, make it robust and flexible | 03:37 |
Qiming | then we can work on the fancy things, ;) | 03:37 |
flwang | fair enough | 03:38 |
Qiming | and ... we are already making some of those higher level decisions abstracted and exposed to users as Policies | 03:38 |
Qiming | always hesitate when coming up with something smart, :P | 03:39 |
Qiming | it is sometimes poison, :) | 03:39 |
Qiming | flwang, speaking of the API tihng | 03:45 |
Qiming | I'm wondering if we should use swagger or something alike | 03:45 |
Qiming | team has proposed some ideas about API schema validation | 03:46 |
Qiming | checking parameter's data type, value range, making sure required parameters are there and unknown parameters are rejected | 03:46 |
Qiming | I was thinking about this last night | 03:47 |
Qiming | going the swagger way would benefit both the API documentation and schema validation | 03:47 |
*** elynn_ has quit IRC | 04:19 | |
flwang | Qiming: yep, and we're using json schema | 04:23 |
flwang | https://github.com/openstack/zaqar/blob/master/zaqar/api/v2/request.py | 04:23 |
*** elynn_ has joined #senlin | 04:45 | |
*** elynn_ has quit IRC | 04:49 | |
*** elynn_ has joined #senlin | 04:50 | |
yanyanhu | hi, Qiming, can I directly revert a patch in gerrit UI? | 05:44 |
yanyanhu | will remove this change https://review.openstack.org/#/c/303923/ | 05:44 |
openstackgerrit | Yanyan Hu proposed openstack/senlin: Revert "Add Rally plugin for Senlin tempest" https://review.openstack.org/307565 | 05:46 |
*** elynn_ has quit IRC | 06:26 | |
*** yanyanhu has quit IRC | 06:26 | |
*** Qiming has quit IRC | 06:29 | |
*** Qiming has joined #senlin | 08:00 | |
xuhaiwei | Qiming, available now? | 08:02 |
Qiming | yes | 08:08 |
xuhaiwei | shall we do it on phone or irc? | 08:09 |
Qiming | let's try phone call now | 08:10 |
xuhaiwei | ok | 08:10 |
Qiming | we are in xinhui's office | 08:10 |
Qiming | rehearsaling the deep dive presentation | 08:10 |
xuhaiwei | again, the password, sorry about it | 08:11 |
Qiming | Participant Code: 21575371 | 08:12 |
Qiming | sorry I dropped dialing in again | 08:32 |
xuhaiwei | can you enter again? | 08:32 |
*** Qiming has quit IRC | 09:04 | |
*** Qiming has joined #senlin | 10:25 | |
*** Qiming has quit IRC | 10:37 | |
*** Qiming has joined #senlin | 12:11 | |
*** Qiming_ has joined #senlin | 12:14 | |
*** Qiming has quit IRC | 12:16 | |
*** elynn has joined #senlin | 12:18 | |
*** xuhaiwei has quit IRC | 12:19 | |
*** elynn has quit IRC | 12:28 | |
cschulz | meeting? | 13:04 |
Qiming_ | hi, cschulz, it is cancelled, sorry for the late notice | 13:27 |
cschulz | Figured that since noticed you were working on presentations | 13:30 |
Qiming_ | sorry for that, :) | 13:31 |
cschulz | Hope it is going well | 13:31 |
*** openstackstatus has joined #senlin | 13:38 | |
*** ChanServ sets mode: +v openstackstatus | 13:38 | |
-openstackstatus- NOTICE: We have recovered one of our cloud providers, but there is a huge backlog of jobs to process. Please have patience until your jobs are processed | 13:41 | |
*** zzxwill__ has quit IRC | 14:32 | |
*** zzxwill__ has joined #senlin | 14:33 | |
*** Qiming_ has quit IRC | 15:38 | |
*** zzxwill__ has quit IRC | 17:22 | |
*** dnievas has joined #senlin | 18:58 | |
dnievas | hi guys | 18:59 |
*** flwang has left #senlin | 21:32 | |
*** zzxwill__ has joined #senlin | 22:05 | |
*** zzxwill__ has quit IRC | 22:35 | |
*** Qiming has joined #senlin | 23:10 | |
*** Qiming has quit IRC | 23:25 | |
*** dnievas has quit IRC | 23:27 | |
*** xuhaiwei has joined #senlin | 23:36 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!