| *** hieulq has quit IRC | 00:46 | |
| *** hieulq has joined #openstack-barbican | 00:47 | |
| *** rm_work has quit IRC | 01:08 | |
| *** rm_work has joined #openstack-barbican | 01:09 | |
| *** daidv has joined #openstack-barbican | 01:27 | |
| *** namnh has joined #openstack-barbican | 01:27 | |
| *** daidv has quit IRC | 01:33 | |
| *** namnh has quit IRC | 01:36 | |
| *** dave-mcc_ has joined #openstack-barbican | 01:43 | |
| *** annp has joined #openstack-barbican | 01:45 | |
| -openstackstatus- NOTICE: The freenode network is currently the target of automated spam attacks, we have enabled temporary restrictions on targetted OpenStack channels which requires users to be logged on to NickServ. If you see spam in your channel, please report it in #openstack-infra. Thanks. | 01:46 | |
| *** dave-mcc_ has quit IRC | 02:02 | |
| *** ChanServ sets mode: +r | 02:52 | |
| *** annp has quit IRC | 03:03 | |
| *** ssathaye has quit IRC | 05:55 | |
| *** namnh has joined #openstack-barbican | 06:37 | |
| *** namnh has quit IRC | 06:37 | |
| *** namnh has joined #openstack-barbican | 06:38 | |
| *** hieulq has quit IRC | 06:47 | |
| *** hieulq has joined #openstack-barbican | 06:48 | |
| *** alee has quit IRC | 07:33 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack/barbican master: [WIP] Implement OVO [4] https://review.openstack.org/500244 | 09:07 |
|---|---|---|
| *** namnh has quit IRC | 10:52 | |
| *** pbourke has quit IRC | 11:43 | |
| *** raildo has joined #openstack-barbican | 12:21 | |
| *** namnh has joined #openstack-barbican | 13:32 | |
| *** namnh has quit IRC | 14:00 | |
| *** namnh has joined #openstack-barbican | 14:01 | |
| *** namnh has quit IRC | 14:02 | |
| *** namnh has joined #openstack-barbican | 14:02 | |
| *** ChanServ sets mode: -r | 14:04 | |
| *** alee has joined #openstack-barbican | 14:24 | |
| *** abishop has joined #openstack-barbican | 14:39 | |
| *** abishop has quit IRC | 14:44 | |
| *** cleong has joined #openstack-barbican | 14:49 | |
| *** testtest_ has joined #openstack-barbican | 15:26 | |
| *** corey_ has joined #openstack-barbican | 15:26 | |
| *** cleong has quit IRC | 15:27 | |
| *** testtest_ has quit IRC | 15:28 | |
| *** nkinder has quit IRC | 15:54 | |
| *** namnh has quit IRC | 16:36 | |
| *** catintheroof has joined #openstack-barbican | 16:38 | |
| *** david_8 has joined #openstack-barbican | 16:45 | |
| *** catintheroof has quit IRC | 16:45 | |
| *** david_7 has quit IRC | 16:49 | |
| *** catintheroof has joined #openstack-barbican | 17:23 | |
| *** catintheroof has quit IRC | 17:30 | |
| *** catintheroof has joined #openstack-barbican | 18:36 | |
| *** catintheroof has quit IRC | 18:36 | |
| *** catintheroof has joined #openstack-barbican | 18:36 | |
| *** catintheroof has quit IRC | 18:36 | |
| *** namnh has joined #openstack-barbican | 18:37 | |
| *** namnh has quit IRC | 18:42 | |
| *** catintheroof has joined #openstack-barbican | 18:56 | |
| *** corey_ is now known as cleong | 19:16 | |
| *** ssathaye has joined #openstack-barbican | 19:17 | |
| EmilienM | alee: can you join #tripleo please? | 19:32 |
| *** dave-mccowan has joined #openstack-barbican | 20:12 | |
| *** namnh has joined #openstack-barbican | 20:38 | |
| alee | dave-mccowan, hey | 20:39 |
| alee | dave-mccowan, hey - got a barbican problem to troubleshoot --- have some time to help? | 20:41 |
| dave-mccowan | sure | 20:41 |
| alee | dave-mccowan, cool | 20:42 |
| alee | dave-mccowan, so - in tripleo, I start up barbican-api and the keystone-listener and barbican-worker | 20:42 |
| *** namnh has quit IRC | 20:42 | |
| alee | dave-mccowan, when I started them all at the same time , I ended up in db deadlocks | 20:43 |
| alee | dave-mccowan, now I end up with the occassional deadlock | 20:43 |
| alee | dave-mccowan, http://logs.openstack.org/53/502553/5/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/7b65eb1/logs/subnode-2/var/log/containers/httpd/barbican-api/barbican_wsgi_main_error.log.txt.gz | 20:44 |
| alee | dave-mccowan, trying to figure out whats going on | 20:46 |
| alee | dave-mccowan, the deadlock is happening when we populate the db with secret stores as per multile secret store setup | 20:46 |
| alee | dave-mccowan, https://github.com/openstack/barbican/blob/master/barbican/plugin/util/multiple_backends.py#L192 | 20:47 |
| alee | dave-mccowan, is it possible that both the barbican-api and the barbican-worker go through this code? | 20:47 |
| *** raildo has quit IRC | 20:48 | |
| dave-mccowan | do you think a process is not cleaning-up after a write? or is a process holding one lock, while waiting for another resource? | 20:51 |
| alee | dave-mccowan, the latter | 20:51 |
| alee | dave-mccowan, I think two processes are trying to access the db concurrently and stepping over each other | 20:52 |
| alee | dave-mccowan, if both worker and barbican-api go thorugh the same code -- maybe they step on each other trying to make the same insert? | 20:53 |
| dave-mccowan | if there's only one resource, it wouldn't deadlock, would it? | 20:54 |
| alee | dave-mccowan, I'm just guessing , but it does seem like that code needs some kind of semaphore around it to make sure it only gets executed by a single thread | 20:55 |
| alee | or process | 20:55 |
| alee | dave-mccowan, I'm trying to figure out how the worker starts up -- does it actually execute that code? | 21:00 |
| *** ssathaye has quit IRC | 21:08 | |
| alee | dave-mccowan, so that code is eventually triggered by setup_database_engine_and_factory() in repositories,py | 21:10 |
| *** salmankhan has joined #openstack-barbican | 21:11 | |
| alee | dave-mccowan, which is executed in app.py, retry_scheduler.py and server.py | 21:12 |
| alee | dave-mccowan, basically I think all of worker, notifier and api-server | 21:13 |
| alee | dave-mccowan, and concurrent access doing the same thing causes deadlock somehow .. | 21:14 |
| alee | dave-mccowan, how can we ensure only one goes through at a time? | 21:15 |
| dave-mccowan | alee the deadlock is always inserting into the secret_stores table? | 21:19 |
| alee | dave-mccowan, in the cases where I have seen it - yes | 21:20 |
| *** ssathaye has joined #openstack-barbican | 21:20 | |
| *** catintheroof has quit IRC | 21:20 | |
| alee | dave-mccowan, not saying it wont happen elsewhere --- but have not seen it elsewhere | 21:20 |
| alee | dave-mccowan, we have not seen this in devstack because more likely than not 1) we dont have listener and worker in devstack (2) we aren't testing multi-secret store config | 21:26 |
| *** cleong has quit IRC | 21:26 | |
| alee | dave-mccowan, we probably need to fix both of those things | 21:26 |
| alee | dave-mccowan, I do know that I did not start seeing these errors when I added the listener and worker | 21:27 |
| *** salmankhan has quit IRC | 21:28 | |
| dave-mccowan | the multiple backend code is the freshest, so it makes sense new issues would be there. | 21:29 |
| alee | dave-mccowan, so how to fix -- some sort of file lock? | 21:30 |
| alee | dave-mccowan, or some kind of lock at the db layer .. | 21:30 |
| alee | dave-mccowan, do we implement locks anwyhere in the barbican code? | 21:31 |
| dave-mccowan | i was just seaching db deadlocks. the typical cause is doing things in a different order. as long as each thread does the inserts in the same order, it shouldn't lock. | 21:31 |
| dave-mccowan | the sql code should be managing the locks for us. | 21:31 |
| alee | dave-mccowan, but its the same insert being done by multiple processes .. | 21:32 |
| *** openstackgerrit has quit IRC | 21:33 | |
| dave-mccowan | the table should be locked during the insert. a problem could occur if inserts are done to both project table and secret store table, but in a different order for two threads. | 21:34 |
| *** salmankhan has joined #openstack-barbican | 21:43 | |
| alee | dave-mccowan, so whatya think? | 21:45 |
| dave-mccowan | alee i think a timing conflict between project table and secret store table. since the project_id is a foreign key of secret_store, a process needs to lock both tables to insert into the secret store. maybe one of the threads is still locking the project table, then goes to initialize the secret store, while another process starts with the secret store. | 21:48 |
| dave-mccowan | i'd add more logging and then recreate. | 21:54 |
| alee | dave-mccowan, its not straightforwrad to recreate -- being a timeing thing and all | 22:03 |
| *** salmankhan has quit IRC | 22:20 | |
| *** namnh has joined #openstack-barbican | 22:39 | |
| *** namnh has quit IRC | 22:43 | |
| *** catinthe_ has joined #openstack-barbican | 23:08 | |
| *** catinthe_ has quit IRC | 23:13 | |
| *** tonyb has quit IRC | 23:26 | |
| *** tonyb has joined #openstack-barbican | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!