| *** salmon_ has quit IRC | 00:35 | |
| *** dshulyak_ has joined #openstack-solar | 07:01 | |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Implement timelimit mechanism based on tracking of tasks https://review.openstack.org/270766 | 07:50 |
|---|---|---|
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add implementation of puller with pool of threads and tracking of tasks https://review.openstack.org/270765 | 07:50 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add hack to start/end session after each retry in locking https://review.openstack.org/271180 | 07:50 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Zerorpc worker for orchestration modules https://review.openstack.org/269166 | 07:50 |
| dshulyak_ | hello | 07:53 |
| dshulyak_ | pigmej: i tested lock patch with sqlite, and the problem i was fixing here (https://review.openstack.org/#/c/271180/) is different | 07:54 |
| dshulyak_ | i will try to use read_uncommited pragma, or disable transactions, after all riak doesnt provide any, so i think it is better to have compliant interface | 07:55 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Use sqlite with autocommit=True https://review.openstack.org/271180 | 08:21 |
| *** salmon_ has joined #openstack-solar | 08:56 | |
| pigmej | dshulyak_: yeah that problem that you hit is different on sqlite. | 09:00 |
| pigmej | the problem is that I enabled transactions, because otherwise sqlite is very very slow | 09:00 |
| pigmej | it starts each db action in separate transaction | 09:00 |
| pigmej | btw, hey dshulyak_ :) | 09:01 |
| dshulyak_ | i have a test where i am creating 15 resources and then executing usual workflow, it takes the same time with/wo enabled autocommit | 09:02 |
| dshulyak_ | but i can restore patch with commit/begin in the middle of retry section | 09:03 |
| pigmej | weird | 09:03 |
| pigmej | hmm, but wait did you disable transaction.start ? and commit ? | 09:03 |
| pigmej | ah you idd | 09:05 |
| pigmej | hmm, weird, because for me sqlite without explicit transaction was terrible slow | 09:06 |
| pigmej | and they even mention it in faq and other stuff afair | 09:06 |
| pigmej | that with autocommit=True, insert performance is kinda bad | 09:07 |
| pigmej | yeah it is: http://www.sqlite.org/faq.html#q19 | 09:07 |
| pigmej | dshulyak_: that's why I enabled transactions :) | 09:07 |
| pigmej | good everything works with latest vagrant images :) | 11:14 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Implement timelimit mechanism based on tracking of tasks https://review.openstack.org/270766 | 12:01 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add implementation of puller with pool of threads and tracking of tasks https://review.openstack.org/270765 | 12:01 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Support configurable retries on orchestration layer https://review.openstack.org/272036 | 12:01 |
| salmon_ | https://etherpad.openstack.org/p/solar-kolla_todo | 12:01 |
| pigmej | dshulyak_: online? | 12:21 |
| dshulyak_ | eys | 12:21 |
| pigmej | do you have other idea for that sqlite? | 12:21 |
| dshulyak_ | i can try pragma_uncommited.. | 12:22 |
| dshulyak_ | with another database we could use nested transcations afaiu, but not with sqlite… | 12:23 |
| pigmej | yeah, BUT maybe instead we could have separate DB for locks ... | 12:24 |
| pigmej | with autocommit=True or without transactions.... | 12:24 |
| pigmej | because autocommit=True == performance fuckup | 12:24 |
| dshulyak_ | separate db? | 12:24 |
| pigmej | yup | 12:25 |
| pigmej | somehow hack sqlite client to open separate DB next to primary one | 12:25 |
| pigmej | just for locks | 12:25 |
| pigmej | and back to this problem 'later' | 12:25 |
| dshulyak_ | hm, maybe lets just begin new transcation when lock will be finally acquired | 12:26 |
| dshulyak_ | basically we can do anything before lock finally acquired | 12:27 |
| dshulyak_ | it is like first step in scheduler | 12:27 |
| pigmej | hmm, yeah that's also possibility | 12:27 |
| pigmej | BUT then it should be outside lock part | 12:27 |
| dshulyak_ | it needs to be before retry | 12:28 |
| dshulyak_ | commit/begin before each retry, and one when lock is acquired | 12:28 |
| dshulyak_ | idk how to make 1st part outside lock part | 12:29 |
| pigmej | hmm, right. | 12:31 |
| pigmej | then you may be right :) | 12:32 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Improve fixtures for functional tests https://review.openstack.org/272065 | 12:33 |
| dshulyak_ | ok, i will do that next | 12:34 |
| pigmej | dshulyak_: btw we need to merge your worker asap | 12:34 |
| pigmej | :) | 12:34 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add implementation of puller with pool of threads and tracking of tasks https://review.openstack.org/270765 | 13:24 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Re-start sqlite session before retry and after succesful acquire https://review.openstack.org/271180 | 13:24 |
| salmon_ | guys please read this https://etherpad.openstack.org/p/mos-dev-annoucement-solar-0.0.1 | 14:03 |
| salmon_ | dshulyak_: pigmej bogdando ^ | 14:04 |
| bogdando | salmon_, "Stevedore support, which allows to easily add advanced input types, handlers and transports" did we document this as how-to? | 14:06 |
| bogdando | ditto to "Composer files support was improved" | 14:06 |
| pigmej | bogdando: it's stevedore, so it's already docummented by stevedore authors ;D | 14:07 |
| bogdando | may be you should put direct docs links to these, looks very interesting | 14:07 |
| pigmej | but no we don't have dev documentation | 14:07 |
| bogdando | pigmej, we need examples in Solar how-to | 14:07 |
| salmon_ | bogdando: ok, I will add | 14:07 |
| pigmej | bogdando: we don't have any dev docs :( | 14:07 |
| bogdando | otherwise it looks like something good but unknown | 14:07 |
| bogdando | try to put from resources side | 14:08 |
| bogdando | how one who's creating a new resource could leverage these two new things | 14:08 |
| bogdando | the rest LGTM | 14:08 |
| salmon_ | ok, lunch first :) | 14:19 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add init script for solar-worker https://review.openstack.org/270665 | 14:52 |
| dshulyak_ | i would change celery/zeromq and riak messages - for riak we can say that we have separate interface for working with db layer, and currently we can use riak/sql backend, and we will develop something similar for orchestration and will be able to use any kind of transport and plug-in/plug-out scheduler/other executors logic if necessary | 14:56 |
| dshulyak_ | salmon_: ^ | 14:56 |
| dshulyak_ | ah sorry you said about db, but i think it is better to emphasize abstract inteface | 14:57 |
| dshulyak_ | guys, please review this changes - https://review.openstack.org/#/q/status:open+project:openstack/solar+branch:master+topic:rebase_zero | 14:59 |
| salmon_ | dshulyak_: just update the description there | 15:10 |
| dshulyak_ | ok | 15:10 |
| salmon_ | dshulyak_: in announcement not in review :) | 15:13 |
| dshulyak_ | yeah :) added my changes | 15:23 |
| pigmej | dshulyak_: does lock fix work for you ? | 15:23 |
| dshulyak_ | i checked it on friday, it removed problems with siblings - so yes | 15:26 |
| pigmej | ok, so we need merge it :) | 15:26 |
| pigmej | or review :D | 15:27 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Remove duplicatd code in orchestration/scheduler https://review.openstack.org/272130 | 15:27 |
| dshulyak_ | hm, so you dont like my init scrip | 15:30 |
| pigmej | no :( | 15:30 |
| pigmej | we dislike daemonization in python :) | 15:30 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Remove duplicated code in orchestration/scheduler https://review.openstack.org/272130 | 15:32 |
| dshulyak_ | idk why but with new changes to lock - test with 10 reosurces - 8 sec on sqlite - ~ 12 on riak | 15:35 |
| pigmej | well sqlite is fast :P | 15:37 |
| pigmej | especially that it uses cache so it works like inmemory cache | 15:37 |
| pigmej | also serialization is probably faster :) | 15:37 |
| pigmej | don't worry, I know how to speduup riak one :) | 15:37 |
| dshulyak_ | pigmej: why you added https://review.openstack.org/#/c/271446/4/solar/dblayer/solar_models.py ? | 15:39 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Removed global ansible library https://review.openstack.org/272135 | 15:40 |
| dshulyak_ | in the case of conflict it is possible that lockers wont be equal to data[‘lockers’] ? or why? | 15:41 |
| pigmej | we return data that db have alreayd | 15:41 |
| pigmej | so, when there are 2 concurrent processes, DB will know that | 15:41 |
| pigmej | (riak in fact) | 15:41 |
| pigmej | so I update local state with state that is visible for DB | 15:42 |
| pigmej | :) | 15:42 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Removed global ansible library https://review.openstack.org/272135 | 15:43 |
| dshulyak_ | i didnt get it, you are updating with (latest) lockers and then merging returned value from db and your own lockers | 15:43 |
| dshulyak_ | i mean when/why they can be different | 15:44 |
| dshulyak_ | ok, i’m going afk, see you tomorrow, or maybe later | 15:48 |
| *** dshulyak_ has quit IRC | 15:48 | |
| salmon_ | ciao | 16:04 |
| pigmej | the problem is that when I update then we can have siblings conflict | 16:04 |
| *** salmon_ has quit IRC | 17:23 | |
| *** dshulyak has joined #openstack-solar | 17:58 | |
| *** dshulyak has quit IRC | 18:12 | |
| *** openstackgerrit has quit IRC | 19:02 | |
| *** openstackgerrit has joined #openstack-solar | 19:02 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!