| *** salmon_ has joined #openstack-solar | 08:12 | |
| *** dshulyak_ has joined #openstack-solar | 08:13 | |
| dshulyak_ | hello | 08:24 |
|---|---|---|
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Implement timelimit mechanism based on tracking of tasks https://review.openstack.org/270766 | 08:30 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add implementation of puller with pool of threads and tracking of tasks https://review.openstack.org/270765 | 08:30 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Zerorpc worker for orchestration modules https://review.openstack.org/269166 | 08:30 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add hack to start/end session after each retry in locking https://review.openstack.org/271180 | 08:30 |
| openstackgerrit | Dmitry Shulyak proposed openstack/solar: Add hack to start/end session after each retry in locking https://review.openstack.org/271180 | 08:32 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Fix session in CLI on Debian systems https://review.openstack.org/271182 | 08:38 |
| pigmej | dshulyak_: https://review.openstack.org/#/c/271180/ ? | 09:16 |
| dshulyak_ | pigmej: yeah, i added it to show what the problem, im looking for a better solution | 09:17 |
| dshulyak_ | there is 2 problems | 09:17 |
| pigmej | BUT I still don't get it | 09:17 |
| pigmej | session_start ,session_end are noops three | 09:18 |
| pigmej | there | 09:18 |
| dshulyak_ | well they are not :) | 09:18 |
| dshulyak_ | begin/commit ? | 09:18 |
| pigmej | still, they should change nothing | 09:19 |
| pigmej | at least for lock | 09:19 |
| dshulyak_ | lets say we have 2 conccurent updates, A and C | 09:19 |
| pigmej | and I still don't get the problem | 09:19 |
| dshulyak_ | A acquired lock first | 09:19 |
| dshulyak_ | C tried to acquire it, and entered retry loop | 09:19 |
| dshulyak_ | A released lock | 09:19 |
| dshulyak_ | C wont see that A released it | 09:19 |
| dshulyak_ | i can reproduce it each time | 09:20 |
| pigmej | why C won't see it? | 09:20 |
| pigmej | ah, sqlite | 09:20 |
| pigmej | + transaction isolation :D | 09:20 |
| dshulyak_ | yeah | 09:20 |
| dshulyak_ | documentation says that sqlite provides serializable isolation for different db connections | 09:21 |
| pigmej | yeah, :( | 09:21 |
| pigmej | well, we could remove transactions | 09:21 |
| pigmej | BUT sqlite is slow then | 09:21 |
| pigmej | because it will issue each operation in separate transaction | 09:22 |
| dshulyak_ | i wanted to experiment with this or select for update, but it seems there is no support for select for update | 09:22 |
| dshulyak_ | in sqlite | 09:22 |
| dshulyak_ | or read uncommited pragma :) | 09:22 |
| pigmej | yeah | 09:22 |
| dshulyak_ | and with riak - the problem was that lock is not covering everything that we need, it is released before all data is saved, so another transacation can read old values, and later we will get siblings error | 09:24 |
| dshulyak_ | i will probably use it as a decorator of some kind | 09:24 |
| pigmej | well, then don't release too early :) | 09:24 |
| dshulyak_ | thats the plan | 09:27 |
| pigmej | ;D | 09:28 |
| pigmej | dshulyak_: but nice catch :) | 09:32 |
| pigmej | dshulyak_: the problem is simple | 10:09 |
| pigmej | we need somehow include and access to ./library from transports | 10:09 |
| pigmej | for both, installed package (give pbr some love...), and from `pip install -e .` | 10:10 |
| pigmej | ok | 10:11 |
| pigmej | ->> from pkg_resources import Requirement, resource_filename | 10:11 |
| pigmej | ->> f = resource_filename(Requirement.parse('solar'), 'library') | 10:11 |
| pigmej | ->> f | 10:11 |
| pigmej | '/vagrant/library' | 10:11 |
| pigmej | this thing works | 10:11 |
| pigmej | ;D | 10:11 |
| pigmej | salmon_, dshulyak_ ^ | 10:11 |
| pigmej | can you guys execute python -c "from pkg_resources import Requirement, resource_filename; print resource_filename(Requirement.parse('solar'), 'library')" ? | 10:12 |
| salmon_ | /home/salmon/projects/mirantis/solar/library | 10:12 |
| pigmej | perfect :) | 10:13 |
| salmon_ | hmm, wait L( | 10:13 |
| salmon_ | /home/salmon/.virtualenvs/test_solar/lib/python2.7/site-packages/library | 10:13 |
| salmon_ | this one | 10:13 |
| pigmej | hmm | 10:13 |
| salmon_ | ls /home/salmon/.virtualenvs/test_solar/lib/python2.7/site-packages/library | 10:13 |
| salmon_ | ls: cannot access /home/salmon/.virtualenvs/test_solar/lib/python2.7/site-packages/library: No such file or directory | 10:13 |
| pigmej | yeah manifest needs to be hanged probably | 10:14 |
| pigmej | chceking | 10:14 |
| dshulyak_ | shouldnt MANIFEST.in capture that? | 10:16 |
| pigmej | I changed it `recursive-include library/*` and it changes nothing | 10:17 |
| pigmej | hmm | 10:17 |
| pigmej | 7.7Gpip-PwIwOd-build WTF ? | 10:17 |
| pigmej | oh, when you build package better ensure bootstrap is empty ... :d | 10:20 |
| dshulyak_ | can you explain what needs to supported? pip install -e - just creates symlink to solar | 10:28 |
| pigmej | dshulyak_: `pip install .` | 10:28 |
| salmon_ | pip install solar | 10:28 |
| pigmej | cd to another dir and try to locate library | 10:28 |
| salmon_ | test package is here https://testpypi.python.org/pypi/solar/0.0.1.dev2438 | 10:28 |
| pigmej | salmon_: it's broken, incorrect manifest no pbr files entry | 10:29 |
| pigmej | salmon_: `pip install .` | 10:29 |
| pigmej | is also a way for testing it (just remeber to change directory then) | 10:29 |
| salmon_ | you can push your package | 10:29 |
| pigmej | yeah but it just adds more complexity :) | 10:29 |
| pigmej | https://github.com/openstack-dev/pbr/blob/master/pbr/tests/testpackage/setup.cfg#L33 | 10:32 |
| pigmej | but sadly it changes nothing :) | 10:32 |
| pigmej | http://docs.openstack.org/developer/pbr/#files | 10:35 |
| pigmej | dshulyak_: any progress? | 10:49 |
| dshulyak_ | no | 10:49 |
| dshulyak_ | sdist is doing something weird, it just copies everything thats is under git control | 10:56 |
| pigmej | ok I know how to do it without pbr | 10:56 |
| pigmej | dshulyak_: yeah thanks to our beloved friend 'pbr' ;P | 10:57 |
| dshulyak_ | have you saw this https://github.com/openstack-dev/pbr/blob/master/pbr/tests/testpackage/setup.cfg ? | 10:59 |
| pigmej | dshulyak_: I have | 11:01 |
| pigmej | sadly this thing have only 'package data" | 11:01 |
| pigmej | but these data_files are completely not working for me | 11:01 |
| pigmej | or maybe they are but I can't access them at all | 11:02 |
| dshulyak_ | :) | 11:02 |
| pigmej | and also I have seen that in tests they are just checking if globbing works correctly | 11:02 |
| pigmej | but I can't locate tests where they actually check something inside that data_files | 11:03 |
| pigmej | dshulyak_: https://bitbucket.org/thesheep/hatta/src/bc60f89edecb1326114e7f4bd8e65ed9000b7158/setup.py?at=default&fileviewer=file-view-default#setup.py-21 | 11:04 |
| pigmej | + https://pythonhosted.org/setuptools/setuptools.html#non-package-data-files | 11:05 |
| dshulyak_ | i dont know wtf - but when i am trying to create a package from this test_package - data-files are empty | 11:09 |
| dshulyak_ | ok, it worked out with testpackage | 11:13 |
| pigmej | oh | 11:14 |
| pigmej | where are the files located ? | 11:14 |
| dshulyak_ | and setup.py install works as well | 11:15 |
| pigmej | cool | 11:16 |
| pigmej | whree are the data-files located ? | 11:16 |
| pigmej | or how you did that ? | 11:17 |
| pigmej | :) | 11:17 |
| dshulyak_ | yeah, but thats test package.. | 11:19 |
| dshulyak_ | and i am feeling stupid because i cant reproduce it with solar :) | 11:19 |
| pigmej | :D:D | 11:19 |
| pigmej | add include library * to manifest | 11:20 |
| pigmej | sorry | 11:20 |
| pigmej | include library/*.py | 11:20 |
| dshulyak_ | i added graft to manifest, according to doc - it should be like include whole dir | 11:20 |
| pigmej | I checked all options from doc and no changes so... | 11:21 |
| pigmej | https://bpaste.net/show/74ff97560839 | 11:21 |
| pigmej | this is also not working for some reason | 11:21 |
| dshulyak_ | ok, good news - sdist works | 11:26 |
| dshulyak_ | ah crap thats coincedence | 11:26 |
| pigmej | ;D | 11:29 |
| pigmej | wtf | 11:35 |
| pigmej | I really don't know what's going on there | 11:35 |
| pigmej | I feel really stupid now | 11:35 |
| dshulyak_ | well, sdist includes library in the tar | 11:37 |
| dshulyak_ | but thats not because of data-files | 11:37 |
| dshulyak_ | and we need to exclude everything else in MANIFEST | 11:37 |
| pigmej | that's another story | 11:37 |
| pigmej | but including library in sdist is part of a success | 11:38 |
| pigmej | the second part is 'how the fuck access it' | 11:38 |
| dshulyak_ | yeah, i cant find it after installation.. :) | 11:38 |
| pigmej | maybe we will move that dir | 11:38 |
| pigmej | into solar package ? | 11:38 |
| pigmej | because then it's easy | 11:39 |
| pigmej | python -c "from pkg_resources import Requirement, resource_filename; print resource_filename('solar', 'library')" | 11:39 |
| pigmej | I also know how to include that modules only for resource | 11:40 |
| pigmej | so resource could include module that it needs | 11:40 |
| pigmej | but setuptools/pbr tandem... wtf | 11:40 |
| *** openstackgerrit has quit IRC | 11:47 | |
| *** openstackgerrit has joined #openstack-solar | 11:47 | |
| pigmej | so.... I have no idea, I checked all even stupid ideas that came to my mind | 11:53 |
| pigmej | and... | 11:53 |
| pigmej | nothing | 11:53 |
| dshulyak_ | pigmej: /etc/library/ | 11:53 |
| dshulyak_ | keystone_service.py mysql_user.py README.md | 11:53 |
| dshulyak_ | :D | 11:53 |
| pigmej | ok, now try in virtualenv :P | 11:54 |
| dshulyak_ | ls .tox/venv/xxY/ | 11:57 |
| dshulyak_ | keystone_service.py mysql_user.py README.md | 11:57 |
| dshulyak_ | data_files = xxY = library/* | 11:57 |
| dshulyak_ | hm | 11:57 |
| dshulyak_ | do you have any idea how we use this :) ? | 11:58 |
| dshulyak_ | pigmej: ^ | 11:58 |
| pigmej | ? | 12:00 |
| pigmej | dshulyak_: ? | 12:01 |
| dshulyak_ | we can copy data files into some place like /var/lib/solar_library | 12:01 |
| pigmej | we can't because then you require root access | 12:02 |
| dshulyak_ | yeah, with venv it will be a problem | 12:02 |
| pigmej | can we just move that library into solar/ and then later remove it ? | 12:02 |
| dshulyak_ | but we also can copy them in any place relatively to venv root | 12:02 |
| dshulyak_ | but then i dont know how to access it | 12:03 |
| pigmej | yeah and that's the prbole mD: | 12:03 |
| pigmej | problem* | 12:03 |
| pigmej | well I can how to access venv things | 12:05 |
| pigmej | but it will not work on non venv setup ;D | 12:07 |
| pigmej | salmon_: u there? | 12:13 |
| salmon_ | yup | 12:13 |
| pigmej | dshulyak_: I propose to move library into solar package | 12:13 |
| pigmej | then we can easily locate it | 12:13 |
| dshulyak_ | +1 | 12:13 |
| pigmej | python -c "from pkg_resources import Requirement, resource_filename; print resource_filename('solar', 'library')" | 12:13 |
| pigmej | because otherwise it's like wtf | 12:13 |
| dshulyak_ | i cant find another way | 12:13 |
| dshulyak_ | i wonder why they decided to copy it relatively to venv root | 12:14 |
| pigmej | me too | 12:14 |
| dshulyak_ | it seems useless | 12:14 |
| pigmej | yeah | 12:15 |
| pigmej | salmon_: any thoughts? | 12:16 |
| salmon_ | whatever works. We will remove `library` before next release | 12:17 |
| pigmej | k | 12:17 |
| pigmej | I will take care about that then | 12:18 |
| pigmej | i will also rename library to ansible_library to avoid confusion | 12:18 |
| salmon_ | ok | 12:20 |
| pigmej | hmm | 12:20 |
| pigmej | i forgot how to include subirectory now :( | 12:21 |
| pigmej | ok works nvm :) | 12:21 |
| salmon_ | where solar-celery script will be installed? | 12:31 |
| salmon_ | pigmej: ^ | 12:31 |
| pigmej | salmon_: you mean ? | 12:32 |
| pigmej | that entry point ? | 12:32 |
| salmon_ | after pip install solar, how can I want to start celery | 12:32 |
| salmon_ | grr | 12:33 |
| *** openstackgerrit has quit IRC | 12:33 | |
| salmon_ | How can I start celery :) | 12:33 |
| *** openstackgerrit has joined #openstack-solar | 12:33 | |
| pigmej | hmm | 12:33 |
| pigmej | wait not now :) | 12:34 |
| salmon_ | :P | 12:34 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Moved library to solar/ansible_library https://review.openstack.org/271269 | 12:37 |
| pigmej | salmon_: ^ | 12:37 |
| openstackgerrit | Merged openstack/solar: Fix session in CLI on Debian systems https://review.openstack.org/271182 | 12:43 |
| pigmej | salmon_: I have patch for 'solar-celery' script in pbr BUT it will conflict with ansible_library thingy | 12:46 |
| pigmej | I will make them dependent, because I don't want to fix conflicts then :) | 12:47 |
| salmon_ | sure | 12:47 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Added solar-celery script to setup.cfg https://review.openstack.org/271275 | 12:48 |
| pigmej | salmon_: ^ done | 12:48 |
| pigmej | what else do we need ? | 12:48 |
| pigmej | hmm, zuul dead :( | 12:51 |
| salmon_ | I will test it now :) | 12:53 |
| pigmej | shit now pep8 check reaches ansible files... | 12:57 |
| salmon_ | :D :D :D | 12:57 |
| pigmej | can we ignore whole file ? | 12:57 |
| salmon_ | nope :( | 12:57 |
| pigmej | without #noqa in first line ? | 12:57 |
| salmon_ | we can not | 12:58 |
| pigmej | ... | 12:58 |
| pigmej | so... | 12:58 |
| pigmej | salmon_: it will complain about licence for these files... | 12:58 |
| salmon_ | shiiiiit | 12:59 |
| salmon_ | :/ | 12:59 |
| pigmej | hacking can't ignore whole files ? | 12:59 |
| salmon_ | nope | 12:59 |
| salmon_ | I had the same problem with ply | 12:59 |
| pigmej | super package | 12:59 |
| pigmej | so... | 13:00 |
| pigmej | we have no solution then ... | 13:00 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Moved library to solar/ansible_library https://review.openstack.org/271269 | 13:05 |
| pigmej | ok we can ignore directory :D | 13:05 |
| salmon_ | cool | 13:06 |
| dshulyak_ | error in setup command: 'tests_require' must be a string or list of strings containing valid project/version requirement specifiers | 13:21 |
| dshulyak_ | sdist throws this error for you? | 13:21 |
| dshulyak_ | i removed -r requirements.txt on my local env | 13:22 |
| salmon_ | dshulyak_: how do you run it? | 13:26 |
| dshulyak_ | salmon_: python setup.py sdist | 13:26 |
| salmon_ | no errors | 13:27 |
| pigmej | me too | 13:27 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Added solar-celery script to setup.cfg https://review.openstack.org/271275 | 13:30 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: More clear info about key/pw change in FAQ https://review.openstack.org/271286 | 13:36 |
| dshulyak_ | pigmej: that script is kinda useless outside of vagrant env | 13:36 |
| dshulyak_ | or no | 13:36 |
| salmon_ | dshulyak_: why? works for me on local env | 13:36 |
| pigmej | we changed it :) | 13:36 |
| salmon_ | dshulyak_: we changed it a bit | 13:37 |
| salmon_ | :) | 13:37 |
| pigmej | dshulyak_: that script was "broken" yesterday :D | 13:37 |
| pigmej | in several places;D | 13:37 |
| dshulyak_ | what you changed? | 13:37 |
| dshulyak_ | i thought i added vagrant user there | 13:37 |
| dshulyak_ | but no | 13:37 |
| salmon_ | hardcoded paths and running command in variables | 13:37 |
| pigmej | https://review.openstack.org/#/c/270917/ + https://review.openstack.org/#/c/271059/ | 13:37 |
| pigmej | yeah that running command was cool ;D | 13:38 |
| dshulyak_ | i think i reused it from somewhere :) whats wrong with it? | 13:39 |
| dshulyak_ | btw this wont take any effect in init.d - export SOLAR_CONFIG_OVERRIDE=${SOLAR_CONFIG_OVERRIDE:-/.solar_config_override} | 13:39 |
| dshulyak_ | all init.d scripts started in isolated environment | 13:39 |
| pigmej | dshulyak_: that's why it's like that :) | 13:39 |
| pigmej | dshulyak_: https://review.openstack.org/#/c/271059/1/utils/solar-celery command execution was in START, STOP | 13:40 |
| dshulyak_ | yeah, i understood, but whats wrong with it? | 13:40 |
| pigmej | that you always executed start/stop and then choosen action | 13:40 |
| salmon_ | you always was running start and stop | 13:40 |
| pigmej | '$(...)' is execution | 13:41 |
| dshulyak_ | hm, really? | 13:41 |
| pigmej | yup ;d | 13:41 |
| salmon_ | yup | 13:41 |
| salmon_ | https://bpaste.net/show/8bb27a73c5f5 :) | 13:44 |
| pigmej | :) | 13:44 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Update installation instructions https://review.openstack.org/271295 | 13:47 |
| dshulyak_ | are u sure about that one? i cant reproduce it | 13:49 |
| openstackgerrit | Merged openstack/solar: Moved library to solar/ansible_library https://review.openstack.org/271269 | 13:49 |
| dshulyak_ | they are not executed | 13:49 |
| dshulyak_ | or they are, but not printed? | 13:49 |
| dshulyak_ | ah yes, they just not printed | 13:50 |
| pigmej | the result is stored in variable | 13:51 |
| dshulyak_ | yeah, i tested with touch | 13:51 |
| salmon_ | yup, this is why I din't notice it at the first time :) | 13:51 |
| pigmej | me too | 13:52 |
| pigmej | I was mad about celery multi that | 13:52 |
| pigmej | s/that//g | 13:52 |
| salmon_ | it looks like we will be able to release solar today :) | 13:56 |
| pigmej | do we miss something ? | 13:56 |
| salmon_ | Exception: Database connection string is empty, please ensure that you set config path correctly | 13:56 |
| salmon_ | we do :P | 13:56 |
| salmon_ | I do | 13:56 |
| salmon_ | in docs | 13:56 |
| pigmej | anyway, I need to eat something, brb ! | 13:58 |
| salmon_ | me too | 13:58 |
| openstackgerrit | Merged openstack/solar: More clear info about key/pw change in FAQ https://review.openstack.org/271286 | 14:04 |
| pigmej | back | 14:28 |
| openstackgerrit | Merged openstack/solar: Added solar-celery script to setup.cfg https://review.openstack.org/271275 | 14:44 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Update installation instructions https://review.openstack.org/271295 | 14:44 |
| pigmej | salmon_: is it ready ? | 14:45 |
| salmon_ | pigmej: yup | 14:47 |
| pigmej | I posted commetns;p | 14:48 |
| salmon_ | pigmej: where do you want to link it? | 14:49 |
| salmon_ | pigmej: I replied | 14:51 |
| pigmej | I replied too | 14:53 |
| pigmej | ;D | 14:53 |
| pigmej | salmon_: are you going to upload new changeset? | 15:02 |
| salmon_ | yup | 15:02 |
| dshulyak_ | pigmej: i remember that you had one more idea about locks and riak buckets, yes? | 15:18 |
| pigmej | yup | 15:19 |
| dshulyak_ | it seems that i can reproduce faulty behaviour with current implementation, 2 or 3 threads will think that they acquired lock, and then one of them will fail with siblings error | 15:20 |
| pigmej | hmm, can you make small example where it fails ? | 15:20 |
| dshulyak_ | i have functional test, it starts to fail with scale > 7 | 15:21 |
| pigmej | but it contains the hacks ? | 15:21 |
| pigmej | the session_* things etc? | 15:21 |
| pigmej | then it alters behaviour of whole lock and dblayer | 15:21 |
| pigmej | I would expect that it breaks stuff | 15:21 |
| dshulyak_ | i tried with and without session_* hacks | 15:22 |
| pigmej | can you chcek with riak n_val=1? | 15:22 |
| pigmej | maybe it's because of transaction isolation etc | 15:22 |
| dshulyak_ | i am talking about riak n_val=1 | 15:22 |
| dshulyak_ | with sqlite it seems ok now | 15:22 |
| pigmej | hmm | 15:23 |
| pigmej | can you show small example ? | 15:23 |
| pigmej | without whole worker logic? | 15:23 |
| pigmej | because there is literally no place where lock can fail IF backend is working properly | 15:23 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Update installation instructions https://review.openstack.org/271295 | 15:23 |
| pigmej | it just adds elements to list, and checks +1 and -1 | 15:23 |
| dshulyak_ | i will try to implement some example, but i cant guarantee that i will be able to reproduce... | 15:25 |
| pigmej | dshulyak_: well, I will look into your code too | 15:25 |
| pigmej | BUT I don't know how this lock implementation could be wrong (if used correctly) | 15:25 |
| pigmej | it's like one big list with sum of values | 15:26 |
| pigmej | maybe there is some issue ... but I don't know ;) | 15:26 |
| dshulyak_ | pigmej: i was also wondering - https://github.com/openstack/solar/blob/master/solar/dblayer/solar_models.py#L1246 - it doesnt change riak_object.data, but only sibling | 15:26 |
| pigmej | data reads from siblings | 15:26 |
| pigmej | :) | 15:27 |
| dshulyak_ | no | 15:27 |
| dshulyak_ | i checked | 15:27 |
| dshulyak_ | data in sibling and riak_object is different | 15:27 |
| dshulyak_ | let me recheck to avoid confustion :) | 15:27 |
| pigmej | what data ? | 15:28 |
| pigmej | dshulyak_: https://github.com/basho/riak-python-client/blob/master/riak/riak_object.py#L27 | 15:29 |
| pigmej | riak_object.data is content_property | 15:29 |
| pigmej | content_property always reads and sets everything in riak_object.siblings[0]\ | 15:29 |
| pigmej | you can also find the same method: http://docs.basho.com/riak/latest/dev/using/conflict-resolution/python/#More-Advanced-Example | 15:30 |
| dshulyak_ | i see | 15:30 |
| pigmej | https://bpaste.net/show/8a58ef96717f | 15:34 |
| pigmej | wrr not like that | 15:34 |
| pigmej | waiat | 15:34 |
| pigmej | I wonder how to make test which matches your usage | 15:34 |
| pigmej | dshulyak_: | 15:36 |
| pigmej | https://bpaste.net/show/0571a3b188c3 | 15:36 |
| pigmej | if I would be able to acquire lock several times, then that returned list (after map) should have some value != 1, isn't it? | 15:36 |
| dshulyak_ | yeah, thats clear | 15:37 |
| pigmej | and I have always 1 | 15:37 |
| pigmej | I'm unable to acquire the same lock twice | 15:38 |
| dshulyak_ | same with this example | 15:39 |
| pigmej | you mean ? | 15:39 |
| pigmej | you can't get lock twice too with this example ? | 15:39 |
| dshulyak_ | i mean i have the same result with this test | 15:39 |
| pigmej | the same == broken or the same as me ? | 15:39 |
| dshulyak_ | same as you | 15:39 |
| pigmej | I changed concurrency to 100 and the same | 15:40 |
| pigmej | (you may need to increase lock timeout) | 15:40 |
| pigmej | maybe it's not lock issue itself but somehow cache / dblayer fuckup | 15:40 |
| pigmej | hmm, are you using multiple process ? | 15:42 |
| dshulyak_ | with higher concurency it fails with RuntimeError sometimes, but thats ok | 15:43 |
| dshulyak_ | retries etc | 15:43 |
| dshulyak_ | single process | 15:44 |
| pigmej | dshulyak_: do you use `with` or acquire release by hand ? | 15:44 |
| dshulyak_ | the cache is cleared in session_end | 15:44 |
| pigmej | yeah but I don't have session_end "hack" | 15:45 |
| pigmej | lock cleans session itself by deleting only what's needed | 15:45 |
| pigmej | afair | 15:45 |
| dshulyak_ | ? what do you mean | 15:45 |
| pigmej | https://github.com/openstack/solar/blob/master/solar/dblayer/locking.py#L67 | 15:45 |
| pigmej | lock object shouldn't be cached at all | 15:45 |
| pigmej | hmm | 15:46 |
| pigmej | is there any way taht you acquire the same lock twice from the same identity ? | 15:46 |
| dshulyak_ | what do oyu mean by session_end hack? | 15:46 |
| dshulyak_ | no, identity is different | 15:46 |
| pigmej | dshulyak_: you showed today some session_end/session_start in locking | 15:46 |
| pigmej | for sqlite | 15:46 |
| dshulyak_ | thats for sqlite | 15:46 |
| pigmej | ok not related then at all | 15:47 |
| dshulyak_ | behaviour with riak is the same with this hack and without | 15:47 |
| pigmej | dshulyak_: do you use `with` or `acquire` `release` by hand? | 15:47 |
| dshulyak_ | with | 15:47 |
| dshulyak_ | the code in scheduler is the same | 15:47 |
| pigmej | then wtf ... | 15:47 |
| pigmej | dshulyak_: can you add print in the else clause if [identity, 1, stamp] in lk.lockers: ? | 15:49 |
| pigmej | locking.py:127 | 15:49 |
| pigmej | add else and log.debug please :) | 15:49 |
| pigmej | just in case, you *shouldnt* reach that else | 15:50 |
| dshulyak_ | ok | 15:51 |
| pigmej | if it did reach that else.... then conflict resolution in another thread removed it | 15:51 |
| pigmej | HMMMMMMMMMMMMMMMMMMMMMMMMMMM | 15:52 |
| pigmej | [1, 1, 1, 1, 1, 1, 2, 1, 1, 1] | 15:52 |
| pigmej | dshulyak_: ^ | 15:53 |
| pigmej | https://bpaste.net/show/e73a5f21c207 | 15:53 |
| pigmej | it crashed once and now it doesn't want to :( | 15:55 |
| openstackgerrit | Merged openstack/solar: Update installation instructions https://review.openstack.org/271295 | 15:56 |
| dshulyak_ | maybe the amount of switches is important somehow | 15:56 |
| pigmej | yup | 15:56 |
| pigmej | it seems so | 15:56 |
| pigmej | [[4, 1, u'540b1e08-b8fb-492d-bac2-67c2b8c3d35d'], [7, 1, u'da5246da-ba29-467e-85af-3a042915f3d9'], [7, -1, u'da5246da-ba29-467e-85af-3a042915f3d9'], [6, 1, u'65035f55-8145-4cd2-a5db-8be45bb61994'], [0, 1, u'3d75b254-7b5a-470c-8de0-611355219d77'], [3, 1, u'ec0da7fa-e1fb-42c7-90f9-fa80e77c99c5'], [6, -1, u'65035f55-8145-4cd2-a5db-8be45bb61994']] | 15:56 |
| dshulyak_ | because i should have a lot of them :) | 15:56 |
| pigmej | `lockers` when it crashed | 15:56 |
| pigmej | and it's bad | 15:57 |
| pigmej | at least I have some debug info now... | 15:58 |
| pigmej | dshulyak_: it happens to you often or rarely ? | 16:00 |
| dshulyak_ | every testrun with concurrency > 7 | 16:00 |
| pigmej | :| | 16:01 |
| pigmej | something special is in your code ;D | 16:01 |
| pigmej | Ok I know where we *may* have problem :) | 16:04 |
| dshulyak_ | where? | 16:05 |
| pigmej | in some secret place :P | 16:05 |
| pigmej | https://github.com/openstack/solar/blob/master/solar/dblayer/locking.py#L124 | 16:06 |
| dshulyak_ | :D | 16:06 |
| pigmej | this may be not working as expected | 16:06 |
| pigmej | I don't remember why I added it though | 16:06 |
| dshulyak_ | secret like parent in gevent local :) ? | 16:06 |
| pigmej | ;D | 16:06 |
| pigmej | no :P | 16:06 |
| dshulyak_ | for example if i have such lines in my log: | 16:08 |
| dshulyak_ | ./report:DEBUG:solar:+++ A 139899849609456 | 16:08 |
| dshulyak_ | ./report:DEBUG:solar:+++ A 139899852235792 | 16:08 |
| dshulyak_ | ./report:DEBUG:solar:+++ R 139899849609456 | 16:08 |
| dshulyak_ | ./report:DEBUG:solar:+++ R 139899852235792 | 16:08 |
| pigmej | what is +++ ? | 16:08 |
| dshulyak_ | it means that one of this threads will fail with SiblingsError | 16:08 |
| dshulyak_ | +++ is just for grep | 16:08 |
| pigmej | k | 16:08 |
| dshulyak_ | A - after acquire, R - before release | 16:09 |
| pigmej | ok I know what's the problem :) | 16:09 |
| pigmej | I think I know | 16:09 |
| dshulyak_ | good, because i kinda stuck with this :) | 16:09 |
| pigmej | ;D | 16:09 |
| pigmej | can you for now switch to RLock from gevent.lock | 16:09 |
| pigmej | ? | 16:09 |
| pigmej | then you will be safe | 16:09 |
| pigmej | and you can continue work :) | 16:09 |
| pigmej | just add Lock to _acquire in _CRDTishLock | 16:09 |
| dshulyak_ | i can switch to sqlite for other things, thats not a problem | 16:10 |
| pigmej | it works by accident for sqlite | 16:10 |
| pigmej | :D | 16:10 |
| pigmej | I mean, it's probably the same reason why it usually works for me, and crashes like one per ~50 executions | 16:11 |
| pigmej | dshulyak_: it would be cool if you could hardcode lock name | 16:13 |
| pigmej | you would help me with debugging this then | 16:13 |
| pigmej | please hardcode lock name, make at least one "working" lock situation, and then please try to use it again | 16:13 |
| dshulyak_ | you want me to use it from different sessions? | 16:14 |
| dshulyak_ | test sesssions | 16:15 |
| pigmej | well, instead of lock being random | 16:15 |
| pigmej | set it for always the same | 16:16 |
| pigmej | https://github.com/openstack/solar/blob/master/solar/dblayer/locking.py#L104 I want it always enter this | 16:16 |
| pigmej | not https://github.com/openstack/solar/blob/master/solar/dblayer/locking.py#L108 | 16:16 |
| dshulyak_ | it is not that easy to hardcode a lockname because of random buckets | 16:20 |
| dshulyak_ | not random - generated bucket names | 16:20 |
| pigmej | then acquire it once before :) | 16:20 |
| pigmej | if it's hard skip it | 16:20 |
| dshulyak_ | removed bucket patching for Lock, so do you want to see logs or something specific? | 16:22 |
| dshulyak_ | pigmej: ^ | 16:22 |
| pigmej | make sure it doesn't enter taht from_dict in concurrent | 16:23 |
| pigmej | bug is https://github.com/openstack/solar/blob/master/solar/dblayer/locking.py#L106-L110 | 16:23 |
| pigmej | probably :) | 16:23 |
| dshulyak_ | pigmej: im still able to catch siblings error, even if it doesnt enter that place | 16:26 |
| pigmej | oh :( | 16:26 |
| pigmej | you make me sad :( | 16:26 |
| pigmej | http://wp.patheos.com.s3.amazonaws.com/blogs/wwjtd/files/2013/06/sadpanda.png | 16:27 |
| pigmej | dshulyak_: you destroyed my good feeling today :( | 16:28 |
| pigmej | https://bpaste.net/show/9bee6bc18d03 | 16:50 |
| pigmej | dshulyak_: ^ | 16:50 |
| pigmej | a bug is there :) | 16:50 |
| pigmej | it creates lock from scratch 2 times | 16:50 |
| pigmej | I need to find out why it does not realize it later even with my recent fixes | 16:50 |
| dshulyak_ | i dont know, i didnt hit from_dict in my test | 16:53 |
| pigmej | ... | 16:54 |
| *** dshulyak_ has quit IRC | 17:08 | |
| *** dshulyak_ has joined #openstack-solar | 17:29 | |
| pigmej | ok probably fixed dshulyak_ :D | 17:30 |
| dshulyak_ | what was the problem? | 17:30 |
| pigmej | well, for sure my mistake :P | 17:30 |
| pigmej | :D | 17:31 |
| pigmej | well, wrong assumption in one place | 17:31 |
| pigmej | salmon_: do we want this in release or not? | 17:31 |
| pigmej | dshulyak_: https://bpaste.net/show/98f87bfa50fb | 17:34 |
| pigmej | it solves for me locking problems | 17:38 |
| dshulyak_ | can you please push the change to gerrit? | 17:41 |
| pigmej | it's WIP | 17:42 |
| pigmej | I can but I will give myself -1 | 17:42 |
| pigmej | ;P | 17:42 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Fix locking problems https://review.openstack.org/271446 | 17:43 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Fix locking problems https://review.openstack.org/271446 | 17:44 |
| pigmej | dshulyak_: ^ | 17:44 |
| dshulyak_ | i see | 17:44 |
| pigmej | my favourite part is `I may be not locking` :P | 17:45 |
| pigmej | dshulyak_: just tell me that it works; P | 17:51 |
| -openstackstatus- NOTICE: Restarting zuul due to a memory leak | 17:51 | |
| dshulyak_ | pigmej: yeah :) it seems to work, no siblings error, but now my test fails because of timeout | 17:52 |
| pigmej | yeah it may be a bit slower | 17:53 |
| pigmej | but we have now some not necessary saves there, so I will fix them on monday | 17:53 |
| pigmej | but yeah, good news that it works ... :) | 17:53 |
| salmon_ | pigmej: what? | 17:54 |
| pigmej | salmon_: ? | 17:54 |
| salmon_ | what we want in release?> | 17:54 |
| pigmej | https://review.openstack.org/#/c/271446/ | 17:55 |
| salmon_ | pigmej: does it affect examples? | 17:55 |
| pigmej | well, we have a bug in locking | 17:56 |
| pigmej | https://bugs.launchpad.net/solar/+bug/1537154 | 17:56 |
| openstack | Launchpad bug 1537154 in Solar "Fix CRDTishLock problem in some cases" [Critical,In progress] - Assigned to Jedrzej Nowak (jnowak) | 17:56 |
| dshulyak_ | i think it will be hard to reproduce with celery, because of sqlite | 17:56 |
| pigmej | yeah, but still there are some chances | 17:57 |
| salmon_ | Known issues: deploy sometimes failes? :) | 17:57 |
| pigmej | well, if it will crash, then you need to solar resource clear_all | 17:57 |
| salmon_ | I never hit it, what's the probabilty of hitting it? | 17:58 |
| pigmej | who knows | 17:58 |
| salmon_ | you should :P | 17:59 |
| pigmej | i never hit it before today | 17:59 |
| pigmej | BUT now I know one script that reproduces it pretty much everytime | 17:59 |
| salmon_ | for now, I only care about examples | 18:00 |
| pigmej | doesn't matter | 18:00 |
| pigmej | the bug is in locking, in theory we can fuckup orchestration state | 18:00 |
| pigmej | the chances are rare because as dshulyak_ correctly stated, sqlite is already a limiting factor there | 18:02 |
| salmon_ | we will add it to known issues | 18:11 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Fix locking problems https://review.openstack.org/271446 | 18:12 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Fix locking problems https://review.openstack.org/271446 | 18:14 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Update usage docs https://review.openstack.org/271454 | 18:17 |
| salmon_ | pigmej, dshulyak_ ^ I hope it's last commit | 18:17 |
| pigmej | salmon_: can you remove these <uid> ? | 18:18 |
| pigmej | and keep 'last' instead ? | 18:18 |
| salmon_ | sure | 18:18 |
| pigmej | because that <uid> looks ugly | 18:18 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Update usage docs https://review.openstack.org/271457 | 18:20 |
| salmon_ | grr, again | 18:20 |
| pigmej | :D:D | 18:21 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Update usage docs https://review.openstack.org/271454 | 18:22 |
| salmon_ | this one Update usage docs Change-Id: I1f61e2e272aba4dddd1fecb792be1e1b5de4bfd8 | 18:22 |
| salmon_ | https://review.openstack.org/#/c/271454/ | 18:22 |
| pigmej | solar connect node1 mariadb_service # it will deploy mariadb_service on node1 | 18:24 |
| pigmej | why ? | 18:24 |
| salmon_ | ehs | 18:24 |
| salmon_ | # it will mark service to run on node1 | 18:24 |
| salmon_ | better? | 18:25 |
| pigmej | yup | 18:26 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Update usage docs https://review.openstack.org/271454 | 18:27 |
| salmon_ | done | 18:27 |
| pigmej | we will see :P | 18:28 |
| salmon_ | :D | 18:29 |
| pigmej | dshulyak_: u ok with it? | 18:29 |
| pigmej | ok he did +2 on previous changeset | 18:30 |
| pigmej | salmon_: "merged" | 18:31 |
| salmon_ | ok, lat one test | 18:34 |
| salmon_ | *last | 18:34 |
| salmon_ | jenkins dead again? | 18:44 |
| pigmej | -openstackstatus- NOTICE: Restarting zuul due to a memory leak | 18:44 |
| pigmej | :P | 18:44 |
| salmon_ | ah | 18:44 |
| pigmej | salmon_: well that's perfect excuse, | 18:44 |
| pigmej | "We wanted to release on friday, but jenkins was dead, so we did not release" :P | 18:44 |
| salmon_ | > why you didn't release? | 18:45 |
| salmon_ | - because Java | 18:45 |
| pigmej | :D:D | 18:48 |
| pigmej | it's even better ;d | 18:48 |
| pigmej | message for jenkins http://orig08.deviantart.net/464c/f/2010/149/a/7/sad_panda_chibi_by_mongrelssister.png | 18:54 |
| openstackgerrit | Merged openstack/solar: Update usage docs https://review.openstack.org/271454 | 19:30 |
| salmon_ | yupi | 19:30 |
| pigmej | I will not even comment this ;p | 19:30 |
| openstackgerrit | Lukasz Oles proposed openstack/solar: Version 0.1.0 https://review.openstack.org/271492 | 19:35 |
| salmon_ | pigmej: ^ | 19:35 |
| pigmej | oh | 19:37 |
| pigmej | I hope it will not take hour now... | 19:38 |
| salmon_ | :( | 20:05 |
| pigmej | hwat :> | 20:10 |
| pigmej | it may take hour ;D | 20:10 |
| openstackgerrit | Merged openstack/solar: Version 0.1.0 https://review.openstack.org/271492 | 20:17 |
| pigmej | finally ;D | 20:18 |
| pigmej | salmon_: | 20:18 |
| salmon_ | yupi | 20:18 |
| salmon_ | ok, taggin a release | 20:18 |
| salmon_ | ! [remote rejected] 0.1.0 -> 0.1.0 (prohibited by Gerrit) | 20:30 |
| salmon_ | :( | 20:30 |
| salmon_ | why | 20:30 |
| salmon_ | pigmej: dshulyak_ ideas? | 20:31 |
| salmon_ | can we only push signed tags? https://github.com/openstack-infra/project-config/blob/master/gerrit/acls/openstack/solar.config#L9 | 20:32 |
| salmon_ | I do not have gpg | 20:33 |
| pigmej | hmm | 20:37 |
| pigmej | I don't have gpg too | 20:37 |
| pigmej | salmon_: hmm, I think it's not aboug GPG | 20:41 |
| pigmej | I think it's about group solar-release | 20:41 |
| salmon_ | https://review.openstack.org/#/admin/groups/1218,members | 20:45 |
| salmon_ | solar-core is part of solar-release | 20:45 |
| *** dshulyak_ has quit IRC | 20:50 | |
| *** dshulyak_ has joined #openstack-solar | 20:50 | |
| pigmej | I'm not even in solar-release :P | 20:51 |
| salmon_ | you are as a part of solar-core | 20:52 |
| pigmej | right | 21:08 |
| salmon_ | https://github.com/openstack/solar/tree/v0.1.0 :) | 21:09 |
| pigmej | cooooooooooooool :) | 21:12 |
| salmon_ | https://pypi.python.org/pypi/solar | 21:17 |
| pigmej | btw, | 21:18 |
| pigmej | 11 downloads in the last day | 21:18 |
| pigmej | 11 downloads in the last week | 21:18 |
| pigmej | 451 downloads in the last month | 21:18 |
| pigmej | :D:D | 21:18 |
| salmon_ | wuuut? | 21:19 |
| pigmej | pypi bug ;p | 21:19 |
| salmon_ | dshulyak_: finally we can merge your changes. No more rebases ;) | 21:45 |
| pigmej | you mean workre? | 21:46 |
| pigmej | worker ? | 21:46 |
| salmon_ | anything what can be merged, but yes basically worker | 21:47 |
| pigmej | mooooooooooooooonday | 21:47 |
| salmon_ | I didn't say today :P | 21:48 |
| dshulyak_ | hah | 21:48 |
| pigmej | o hello dshulyak_ :) | 21:49 |
| dshulyak_ | yeah, today, please review | 21:49 |
| salmon_ | :D | 21:49 |
| pigmej | I refuse! | 21:49 |
| pigmej | ;P | 21:49 |
| pigmej | https://review.openstack.org/#/c/271180/ this is not needed anymore | 21:51 |
| dshulyak_ | i’m not sure, the problem was that there was old data | 21:54 |
| dshulyak_ | i will test on monday probably | 21:54 |
| pigmej | k | 21:55 |
| salmon_ | please test now :P | 21:57 |
| pigmej | salmon_: eeee nope | 22:04 |
| *** dshulyak_ has quit IRC | 22:38 | |
| *** salmon_ has quit IRC | 22:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
