| *** tzn has joined #openstack-solar | 00:23 | |
| *** tzn has quit IRC | 00:28 | |
| *** tzn has joined #openstack-solar | 01:24 | |
| *** tzn has quit IRC | 01:29 | |
| *** tzn has joined #openstack-solar | 02:25 | |
| *** tzn has quit IRC | 02:31 | |
| *** tzn has joined #openstack-solar | 04:27 | |
| *** tzn has quit IRC | 04:32 | |
| *** tzn has joined #openstack-solar | 06:29 | |
| *** tzn has quit IRC | 06:34 | |
| *** dshulyak_ has joined #openstack-solar | 07:25 | |
| *** tzn has joined #openstack-solar | 07:30 | |
| *** tzn has quit IRC | 07:34 | |
| *** tzn has joined #openstack-solar | 08:31 | |
| *** openstackgerrit has quit IRC | 08:32 | |
| *** openstackgerrit has joined #openstack-solar | 08:33 | |
| *** tzn has quit IRC | 08:36 | |
| *** salmon_ has joined #openstack-solar | 09:11 | |
| openstackgerrit | Merged openstack/solar: Fixed OSError during concurrent TempFileHandler init https://review.openstack.org/265260 | 09:25 |
|---|---|---|
| pigmej | salmon_: all except torrent are ok | 09:25 |
| salmon_ | we will see ;) | 09:25 |
| pigmej | torrent is: | 09:26 |
| pigmej | 1) not rebased with master | 09:26 |
| pigmej | 2) some weird things happen there | 09:26 |
| salmon_ | I know, it didn't work 3 days ag o :P | 09:31 |
| pigmej | I'm not having keyerror;p | 09:31 |
| pigmej | for some reason it just doesn't work properly in container.. | 09:35 |
| *** tzn has joined #openstack-solar | 09:36 | |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Fixed torrent transport https://review.openstack.org/263823 | 09:39 |
| pigmej | I have no idea currently what's wrong with it inside container, It works flawlessly outside | 09:40 |
| openstackgerrit | Merged openstack/solar: Use stevedore for transports management https://review.openstack.org/265323 | 09:53 |
| salmon_ | pigmej: rebase https://review.openstack.org/#/q/project:openstack/solar ! | 09:57 |
| pigmej | salmon_: doing | 10:03 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Standarized props names in ssh transports https://review.openstack.org/263303 | 10:11 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Ansible Playbook uses now solar_transports https://review.openstack.org/263305 | 10:11 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Implements ansible_playbook_local handler https://review.openstack.org/263306 | 10:11 |
| pigmej | salmon_: rebased | 10:12 |
| pigmej | Ok, I'm going to check why the heck torrent is not working in container | 10:36 |
| *** tzn has quit IRC | 10:58 | |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Raise better error when requested transport is not found https://review.openstack.org/265770 | 11:06 |
| *** salmon_ has quit IRC | 11:22 | |
| *** salmon_ has joined #openstack-solar | 11:23 | |
| dshulyak_ | pigmej: is it possible to detect 2i support in a different way https://review.openstack.org/#/c/259127/1/solar/dblayer/riak_client.py ? | 12:22 |
| pigmej | dshulyak_: "depends" :) | 12:24 |
| pigmej | thinking | 12:24 |
| pigmej | there are several ways to do so but I wonder what wil be the best there | 12:24 |
| dshulyak_ | i wasnt able to find any info in bucket properties | 12:24 |
| pigmej | dshulyak_: we have this problem only in delete ? | 12:25 |
| dshulyak_ | pigmej: yes, bitcask backend used only for single model, which is Lock | 12:27 |
| pigmej | ok, so let's make it like this now but maybe add todo there ? | 12:27 |
| pigmej | because get_keys is really slow | 12:27 |
| pigmej | and it's kinda "the most stupid method for whole cluster" | 12:27 |
| pigmej | it scans whole key space, and then filters keys for bucket... | 12:28 |
| pigmej | so maybe instaed please make sure that we will not execute it on 2i enabled buckets ? | 12:28 |
| pigmej | afair riak client throws something like 'secondary index is not supported with ... backend' right? | 12:28 |
| pigmej | does it make sense for you dshulyak_ ? | 12:29 |
| pigmej | you could use also your bucket_properties dict | 12:29 |
| pigmej | or how it was called | 12:29 |
| dshulyak_ | pigmej: i will check message, but what about bucket_properties? there is no information about 2i support | 12:30 |
| pigmej | somewhere is the info about backend used for this bucket | 12:33 |
| pigmej | only one with 2i is leveldb | 12:33 |
| pigmej | I cannot find my code which did it right now | 12:33 |
| pigmej | but I meant your bucket_properties dict in python code | 12:33 |
| *** tzn has joined #openstack-solar | 12:42 | |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Implement riak based lock mechanism for orchestration https://review.openstack.org/259127 | 12:47 |
| openstackgerrit | Merged openstack/solar: Standarized props names in ssh transports https://review.openstack.org/263303 | 12:48 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Ansible Playbook uses now solar_transports https://review.openstack.org/263305 | 12:52 |
| pigmej | dshulyak_: we will be able to use your cool playbook handler :)) | 12:52 |
| pigmej | salmon_: I have to admit, that it's different than ansible_template but it's more clever :) | 12:52 |
| dshulyak_ | pigmej: you are talking about ansible_local? | 12:53 |
| dshulyak_ | or ansible_playbook but with our transports? | 12:53 |
| pigmej | dshulyak_: yup :) | 12:53 |
| pigmej | the second | 12:53 |
| pigmej | :) | 12:53 |
| pigmej | ansible_local is basically your thingy | 12:53 |
| pigmej | dshulyak_: We will have ansible + ansible_local and ansible_playbook + ansible_playbook_local | 12:56 |
| pigmej | _local means it's running via ansible ssh thing, non local means it uses our transports | 12:56 |
| dshulyak_ | it is better to remove all this _local, i personnaly prefer it, but it is not compliant with all other handlers | 12:57 |
| pigmej | dshulyak_: we need it :( | 12:57 |
| dshulyak_ | where? | 12:57 |
| pigmej | to play with node which doesn't have *yet* all stuff that solar needs | 12:58 |
| pigmej | salmon_: do you remember the exact use case ? | 12:58 |
| dshulyak_ | what the difference with plain ssh transport? | 12:59 |
| pigmej | that you need ansible on remote node | 12:59 |
| pigmej | and puppet | 12:59 |
| salmon_ | pigmej: as you wrote, we need to be able to use any node | 13:00 |
| pigmej | for _local you need ansible without having packages installed | 13:00 |
| pigmej | salmon_: yeah but it started somehere | 13:00 |
| dshulyak_ | yes, thats the difference, but then bash handler should be used | 13:00 |
| pigmej | dshulyak_: yeah but then you code ansible in bash :D | 13:01 |
| dshulyak_ | it is not good that there is 2 very different ways | 13:01 |
| pigmej | we had some discussion about this | 13:01 |
| pigmej | somewhere is even a document... | 13:01 |
| pigmej | dshulyak_: the problem is that this bash handler would be complicated | 13:01 |
| pigmej | https://docs.google.com/document/d/1wEmO24syNq3W-cCJ-m9nfmxKXUnMXW6stHacEH4tZRc/edit | 13:02 |
| pigmej | dshulyak_: there you have a doc | 13:02 |
| pigmej | that's though follow up | 13:02 |
| salmon_ | in the end ansible_playbokk remote/local should only stay | 13:04 |
| pigmej | salmon_: I would even say only 'remote' version | 13:04 |
| pigmej | local should be only used internally or "I know what I'm doing mode" | 13:04 |
| salmon_ | :) | 13:05 |
| pigmej | you tried to write pure bash version for installing something in poznan | 13:05 |
| salmon_ | yup | 13:05 |
| pigmej | on early december ;P | 13:05 |
| salmon_ | yeah, ansible local is better approach to use ;) | 13:06 |
| pigmej | well, let's say it another, it's EASIER :D | 13:06 |
| salmon_ | :) | 13:06 |
| dshulyak_ | is there any changes in the workflow to bootstrap solar? | 13:08 |
| dshulyak_ | http://paste.openstack.org/show/483420/ | 13:08 |
| pigmej | dshulyak_: "some" would be easiest answer | 13:08 |
| pigmej | you need new image | 13:08 |
| dshulyak_ | can i just install all necessary software? | 13:08 |
| pigmej | or | 13:08 |
| pigmej | https://bpaste.net/show/3a21c9981ddc | 13:08 |
| pigmej | + | 13:09 |
| pigmej | https://bpaste.net/show/5605a60ffeee | 13:09 |
| pigmej | should be enough | 13:09 |
| pigmej | and pip install -e . obviously :) | 13:09 |
| dshulyak_ | i wanted to test with sqlite actually | 13:09 |
| dshulyak_ | why do i need this overrdie? | 13:09 |
| pigmej | if you want sqlite then you don't need it | 13:11 |
| pigmej | we switched by default to sqlite because os infra doesn't use riak | 13:12 |
| pigmej | etc | 13:12 |
| pigmej | so by default there is sqlite, and we export that override to adjust stuff for vagrant or fuel-devops stuff | 13:12 |
| dshulyak_ | hm, then what is changed? | 13:12 |
| pigmej | ? | 13:13 |
| dshulyak_ | "some" would be easiest answer | 13:13 |
| salmon_ | dshulyak_: path to db for sqlite may be wrong | 13:13 |
| salmon_ | solar_db: sqlite:////tmp/solar.db | 13:14 |
| salmon_ | does it exist ? | 13:14 |
| dshulyak_ | yes | 13:14 |
| pigmej | dshulyak_: that's error from docker container? | 13:14 |
| dshulyak_ | no | 13:14 |
| salmon_ | strange | 13:15 |
| dshulyak_ | ah okay | 13:15 |
| dshulyak_ | with riak i have more descriptive error | 13:15 |
| dshulyak_ | solar.core.resource.repository.ResourceNotFound: Resource definition {'repo': 'templates', 'resource_name': 'nodes', 'version_sign': '>=', 'version': None} not found | 13:15 |
| pigmej | I saw similar in docker or when I used sqlite in and outside riak | 13:15 |
| pigmej | docker* | 13:15 |
| pigmej | it's not related | 13:15 |
| pigmej | dshulyak_: disk space end ? | 13:15 |
| dshulyak_ | so i need to create repositories i think | 13:15 |
| pigmej | well peewee error was unrelated to this | 13:16 |
| salmon_ | wow, so strange error :D | 13:16 |
| pigmej | salmon_: that ResourceNotFound is ok | 13:16 |
| pigmej | it's because no repositories | 13:16 |
| pigmej | dshulyak_: | 13:16 |
| pigmej | you need to link | 13:17 |
| pigmej | - file: src=/vagrant/resources dest=/var/lib/solar/repositories/resources state=link | 13:17 |
| pigmej | - file: src=/vagrant/templates dest=/var/lib/solar/repositories/templates state=link | 13:17 |
| pigmej | or just execute `solar repository import resources` and then the same for templates | 13:17 |
| salmon_ | with link | 13:17 |
| pigmej | even add -l there | 13:17 |
| pigmej | yup | 13:17 |
| pigmej | :) | 13:17 |
| dshulyak_ | yeah, with sqlite it is still the same and i have a lot of disk space there.. | 13:20 |
| dshulyak_ | and unit tests are ok | 13:20 |
| pigmej | remove sqlite database then and check again | 13:22 |
| pigmej | because it looks like some issue with sqlite db itself (that IO error) | 13:22 |
| pigmej | maybe it's owned by root? | 13:22 |
| pigmej | dshulyak_: https://review.openstack.org/#/c/263305/ could you review it? | 13:53 |
| pigmej | dshulyak_: your PR is still in progress, right? | 14:02 |
| dshulyak_ | pigmej: will do, i still didnt test it with riak strong consistency turned on, but overall i think it will stay this way | 14:04 |
| pigmej | you mean? | 14:05 |
| pigmej | you want to have locks with non strong consistent bucket on riak ? | 14:05 |
| dshulyak_ | pigmej: yes, the way we are using it currently - one physical node and n_val=1, afair all operations on 1 vnode are sequential and therefore we will always know if lock is acquired by another thread | 14:12 |
| pigmej | dshulyak_: yeah but we're *not* going to recommend one node riak cluster | 14:13 |
| pigmej | riak is cluster system, please don't craete solution which works only on one node | 14:13 |
| dshulyak_ | it doesnt matter | 14:13 |
| dshulyak_ | :) | 14:13 |
| pigmej | whole idea of using riak was to have "distributed" stuff | 14:13 |
| pigmej | it matters | 14:13 |
| dshulyak_ | how exactly? | 14:13 |
| pigmej | with distributed env you will not have siblings on write | 14:13 |
| pigmej | or you want ot have n_val = 1 for locking? | 14:14 |
| dshulyak_ | yes, we have n_val=1 right now | 14:14 |
| dshulyak_ | why i wont have siblings with >1 node ? | 14:15 |
| pigmej | you will have them but not on write | 14:15 |
| pigmej | (not always) | 14:15 |
| dshulyak_ | ah ok, i got it | 14:15 |
| pigmej | I'm not sure though how n_val=1 would behave in distributed env, we will probably voliate all good practices.... ;d | 14:16 |
| dshulyak_ | with strongly consistent bucket i expect to have RiakError on write (according to docs), and in _acquire method i am checkin sibglings error both on write and read | 14:16 |
| pigmej | yeah but for read it's uselless | 14:16 |
| pigmej | because concurrent operation is already in progress (2) | 14:16 |
| pigmej | if A created lock, and B did it too, then C will notice siblings on read | 14:17 |
| pigmej | but both A and B will not know about that situation | 14:17 |
| pigmej | so, C will wait but A and B will continue | 14:17 |
| openstackgerrit | Merged openstack/solar: Raise better error when requested transport is not found https://review.openstack.org/265770 | 14:18 |
| dshulyak_ | if A will create lock, and B will create lock with pw=quorum, will i notice siblings on write or on read after this write? | 14:20 |
| pigmej | no | 14:20 |
| pigmej | :( | 14:20 |
| dshulyak_ | no what? | 14:20 |
| pigmej | you will not notice it on write | 14:21 |
| pigmej | you may but you may not | 14:21 |
| pigmej | https://aphyr.com/posts/285-call-me-maybe-riak then go to section named "Strict quorum" | 14:21 |
| pigmej | and check if you meant that or not :) | 14:22 |
| pigmej | I'm not sure what will happen with n_val=1 because I don't know now how this n_val selection works exactly | 14:22 |
| pigmej | but if I would need to bet, I would say that A and B will succeed, and none of them will know about this situation | 14:23 |
| pigmej | and after some time, riak will create siblings for this. | 14:24 |
| dshulyak_ | you are talking about n_val=1 with more than one node in cluster? | 14:24 |
| pigmej | because I'm pretty sure that first node which will receive write with n_val will be success | 14:24 |
| pigmej | yup | 14:24 |
| pigmej | with one node riak is probably sequential | 14:25 |
| pigmej | I say "proabably" because I never ever tested this case :) | 14:25 |
| dshulyak_ | i would expect that with n_val=1 and pw=1 - one of the concurrent writes will fail, i will try to test today/tommorow, also that part from aphyr blog isnt very clear, what the point of primary vnode if riak will try to preserve write even if primary is not avaiable :) ? | 14:31 |
| pigmej | yup | 14:34 |
| pigmej | riak will try to preserve writes because it's how dynamo works | 14:34 |
| pigmej | BUT n_val=1 is interesting case, maybe it will somehow work with that | 14:34 |
| pigmej | the lowest n_val that I used in production was 3 | 14:35 |
| pigmej | so.. :) | 14:35 |
| pigmej | dshulyak_: could you in meantime look on my handler patches ? If these are ok for you please merge, then I will continue my last thing at this part :) | 14:35 |
| dshulyak_ | done | 14:38 |
| dshulyak_ | also while in MV i started doing some stuff with zeromq based worker | 14:40 |
| pigmej | yeah you mentioned it :) | 14:42 |
| pigmej | I'm not sure if we need zmq but yeah let's see it someday | 14:44 |
| pigmej | we also need to create that 'small' worker | 14:44 |
| pigmej | which will be some inproc stuff but it will be easy probably :) | 14:44 |
| pigmej | dshulyak_: thanks for reviews! | 14:44 |
| dshulyak_ | i though about separating scheduler/worker code and communication code, and then for each communication method there will be separate client | 14:45 |
| pigmej | yeah | 14:50 |
| pigmej | I had similar idea / thoughts | 14:50 |
| pigmej | then scheduler could be *anything* and worker will be scheduler independent | 14:51 |
| pigmej | (with some limitations obviously) | 14:51 |
| openstackgerrit | Merged openstack/solar: Ansible Playbook uses now solar_transports https://review.openstack.org/263305 | 14:53 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Implements ansible_playbook_local handler https://review.openstack.org/263306 | 15:15 |
| pigmej | there was merge conflict dshulyak_ :) | 15:15 |
| dshulyak_ | anyone remembers what is the fix for this - mount: unknown filesystem type 'vboxsf' ? | 15:41 |
| pigmej | never had it | 15:41 |
| pigmej | so I can't help :) | 15:41 |
| dshulyak_ | ok, whatever it was update helped | 15:56 |
| pigmej | ;) | 16:24 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Preserve original order of keys in CreatedResources.like https://review.openstack.org/265917 | 16:40 |
| pigmej | dshulyak_: sorry for that mistake :D | 16:41 |
| dshulyak_ | which mistake? | 16:43 |
| dshulyak_ | with order :D ? | 16:43 |
| pigmej | that I don't preserve order :D | 16:43 |
| pigmej | yeah | 16:43 |
| pigmej | I coded that part though but I forgot to use it | 16:44 |
| pigmej | I commented PR too, just use list, tuple instead | 16:44 |
| pigmej | wouldn't it be better ? | 16:44 |
| pigmej | (it's already coded) | 16:44 |
| dshulyak_ | i used it at first, but then i will iterate again in __init__ | 16:45 |
| salmon_ | I think, I tested it only with riak | 16:45 |
| pigmej | yup | 16:46 |
| pigmej | only riak uses it | 16:46 |
| pigmej | ah no openstack too | 16:46 |
| pigmej | ok screw it, it's ok with this ordered dict | 16:46 |
| pigmej | Don't change it dshulyak_ :) | 16:46 |
| pigmej | dshulyak_: anyway, how did you catch it ? | 16:49 |
| dshulyak_ | 10.0.0.3 on solar-dev2, 10.0.0.4 - solar-dev1 | 16:49 |
| pigmej | oh interesting | 16:50 |
| dshulyak_ | and deployment hanged | 16:50 |
| pigmej | ost ? | 16:50 |
| dshulyak_ | yes | 16:50 |
| openstackgerrit | Merged openstack/solar: Preserve original order of keys in CreatedResources.like https://review.openstack.org/265917 | 16:51 |
| pigmej | strange | 16:51 |
| pigmej | I tested ost loles tested it too, ;D | 16:51 |
| salmon_ | yeah, interesting | 16:51 |
| pigmej | anwyay, torrent inside docker is bad idea | 16:51 |
| pigmej | it just needs some ports open / exposed, I don't want to hardcode some stuff for our docker container | 16:51 |
| pigmej | can we instead write in readme that torrent is incompatible with docker now ? | 16:52 |
| dshulyak_ | this is same problem with PYTHONHASHSEED | 16:52 |
| dshulyak_ | or maybe we can write simple init.d script | 16:52 |
| pigmej | salmon_: can we test it with devops ? | 16:53 |
| dshulyak_ | fuel migrated to centos7, so we dont depend on docker that much | 16:53 |
| pigmej | maybe we could somehow add some generic post test things that checks something in DB ? | 16:53 |
| salmon_ | pigmej: with fuel-devops we are using containers | 16:53 |
| pigmej | dshulyak_: well, if we will get rid of celery + redis... | 16:53 |
| pigmej | salmon_: I'm talking about 2 things in the same time ;D | 16:53 |
| pigmej | dshulyak_: too :D | 16:54 |
| dshulyak_ | i simplified celery start up a lot - celery worker -A solar.orchestration.runner -P gevent -c 1000 -Q system_log,celery,scheduler | 16:54 |
| pigmej | salmon_: anyway, torrent example works when firt torrent seed is started outside docker | 16:54 |
| pigmej | I don't want to run this docker with privileged mode or something... | 16:54 |
| pigmej | dshulyak_: well the problem is that we have different paths in container and outside | 16:55 |
| pigmej | but yeah the simpler the better | 16:58 |
| pigmej | dshulyak_: https://review.openstack.org/#/c/263306/ can you do workflow +1 again ? :) It needed rebase :) | 17:00 |
| dshulyak_ | done | 17:01 |
| pigmej | thx :) | 17:01 |
| openstackgerrit | Merged openstack/solar: Implements ansible_playbook_local handler https://review.openstack.org/263306 | 17:04 |
| *** dshulyak_ has quit IRC | 17:39 | |
| *** dshulyak_ has joined #openstack-solar | 17:56 | |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Fixed torrent transport https://review.openstack.org/263823 | 18:12 |
| pigmej | salmon_: ^ torrent works | 18:12 |
| pigmej | I fixed port range to 6881-6981 and I exposed it in compose | 18:13 |
| pigmej | https://bpaste.net/show/7f5138a3e9f4 | 18:13 |
| pigmej | :) | 18:13 |
| *** dshulyak_ has quit IRC | 18:48 | |
| *** tzn has quit IRC | 19:01 | |
| *** tzn has joined #openstack-solar | 21:26 | |
| -openstackstatus- NOTICE: Gerrit is restarting to resolve java memory issues | 22:06 | |
| openstackgerrit | Merged openstack/solar: Fixed torrent transport https://review.openstack.org/263823 | 22:09 |
| *** tzn has quit IRC | 22:40 | |
| *** salmon_ has quit IRC | 23:29 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!