Tuesday, 2014-08-19

*** oomichi has joined #openstack-oslo00:00
openstackgerritA change was merged to openstack/oslo.messaging: Port to Python 3  https://review.openstack.org/10800700:11
*** yamahata has joined #openstack-oslo00:18
*** harlowja is now known as harlowja_away00:29
*** tsekiyam_ has quit IRC00:30
openstackgerritA change was merged to openstack-dev/pbr: Teach pbr about post versioned dev versions.  https://review.openstack.org/8044900:44
openstackgerritA change was merged to openstack/oslo.utils: Make strutils.mask_password more secure  https://review.openstack.org/11461400:45
*** dims has joined #openstack-oslo00:54
*** harlowja_away is now known as harlowja00:56
*** praneshp has quit IRC01:09
jogodhellmann: https://bugs.launchpad.net/nova/+bug/134945201:13
jogodhellmann: any way to make oslo executils timeout if no response01:14
*** bknudson has quit IRC01:23
*** shakamunyi has joined #openstack-oslo01:34
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare: Support for pluggable Datastore selection  https://review.openstack.org/11455101:38
*** oomichi has quit IRC01:48
openstackgerritzhangjl proposed a change to openstack/oslo.messaging: Message was sended to wrong node with use zmq as rpc_backend  https://review.openstack.org/8493801:56
*** oomichi has joined #openstack-oslo01:57
openstackgerritamrith proposed a change to openstack/oslo.concurrency: Mask passwords in exceptions and error messages  https://review.openstack.org/11465601:58
*** arnaud__ has quit IRC02:01
*** noelbk has quit IRC02:02
*** noelbk has joined #openstack-oslo02:02
*** alexpilotti has quit IRC02:09
*** morganfainberg is now known as morganfainberg_Z02:14
*** inteq has joined #openstack-oslo02:19
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare: Support for pluggable Datastore selection  https://review.openstack.org/11455102:22
openstackgerritamrith proposed a change to openstack/oslo-incubator: Mask passwords in exceptions and error messages  https://review.openstack.org/10941702:26
*** Krast_ has quit IRC02:26
*** Krast has joined #openstack-oslo02:26
openstackgerritamrith proposed a change to openstack/oslo-incubator: Mask passwords in exceptions and error messages  https://review.openstack.org/10941702:30
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare: Support for pluggable Datastore selection  https://review.openstack.org/11455102:34
*** arnaud has quit IRC02:35
openstackgerritamrith proposed a change to openstack/oslo.concurrency: Mask passwords in exceptions and error messages  https://review.openstack.org/11465602:35
*** dims has quit IRC02:35
*** arnaud has joined #openstack-oslo02:41
*** dims has joined #openstack-oslo02:42
*** dims has quit IRC02:47
*** jecarey has joined #openstack-oslo03:06
*** shakamunyi has quit IRC03:09
*** dims has joined #openstack-oslo03:12
*** dims has quit IRC03:17
*** markmcclain has quit IRC03:24
*** amotoki has joined #openstack-oslo03:33
openstackgerritzhangjl proposed a change to openstack/oslo.messaging: Message was sended to wrong node with use zmq as rpc_backend  https://review.openstack.org/8493803:34
*** shakamunyi has joined #openstack-oslo04:07
*** harlowja is now known as harlowja_away04:08
*** SridharG has joined #openstack-oslo04:14
*** harlowja_at_home has joined #openstack-oslo04:57
*** harlowja_at_home has quit IRC04:59
*** arnaud has quit IRC05:00
*** dims has joined #openstack-oslo05:01
*** harlowja_at_home has joined #openstack-oslo05:04
*** dims has quit IRC05:06
*** Krast has quit IRC05:08
*** shakamunyi has quit IRC05:13
*** arnaud has joined #openstack-oslo05:26
*** inteq has quit IRC05:27
*** arnaud has quit IRC05:31
*** k4n0 has joined #openstack-oslo05:36
*** morganfainberg_Z is now known as morganfainberg05:41
openstackgerritAndreas Jaeger proposed a change to openstack/oslo.concurrency: Initial translation setup  https://review.openstack.org/11460805:45
*** SridharG has quit IRC05:47
*** ildikov has quit IRC05:58
*** arnaud has joined #openstack-oslo06:04
*** Krast has joined #openstack-oslo06:14
*** SridharG has joined #openstack-oslo06:15
*** arnaud__ has joined #openstack-oslo06:16
openstackgerritJoshua Harlow proposed a change to openstack/taskflow: Initial shift to having backends provide storage impls  https://review.openstack.org/11442206:17
openstackgerritJoshua Harlow proposed a change to openstack/taskflow: Initial shift to having backends provide storage impls  https://review.openstack.org/11442206:18
openstackgerritOpenStack Proposal Bot proposed a change to openstack/oslo.vmware: Imported Translations from Transifex  https://review.openstack.org/11473206:22
*** harlowja_at_home has quit IRC06:23
*** Krast has quit IRC06:28
*** mrda has quit IRC06:30
*** mrda has joined #openstack-oslo06:33
*** arnaud has quit IRC06:40
*** arnaud__ has quit IRC06:40
*** flaper87|afk is now known as flaper8706:46
*** noelbk has quit IRC06:54
*** noelbk has joined #openstack-oslo06:54
*** noelbk has quit IRC06:55
*** noelbk has joined #openstack-oslo06:55
*** ildikov has joined #openstack-oslo06:58
*** ihrachyshka has joined #openstack-oslo06:59
*** noelbk has quit IRC07:00
*** ihrachyshka has quit IRC07:10
*** ildikov has quit IRC07:15
*** ildikov_ has joined #openstack-oslo07:15
openstackgerritJoshua Harlow proposed a change to openstack/taskflow: Initial shift to having backends provide storage impls  https://review.openstack.org/11442207:25
*** AAzza_afk is now known as AAzza07:29
openstackgerritBogdan Dobrelya proposed a change to openstack/oslo.messaging: Should not send replies for cast messages  https://review.openstack.org/11501007:35
*** pblaho has joined #openstack-oslo07:42
*** noelbk has joined #openstack-oslo07:46
openstackgerritAnn Kamyshnikova proposed a change to openstack/oslo.db: ModelsMigrationsSync:Override compare_server_default for MySQL  https://review.openstack.org/11520307:47
*** stannie has joined #openstack-oslo07:48
openstackgerritBogdan Dobrelya proposed a change to openstack/oslo.messaging: Should not send replies for cast messages  https://review.openstack.org/11501007:48
*** noelbk has quit IRC07:55
*** noelbk has joined #openstack-oslo07:56
*** noelbk has quit IRC08:05
*** ihrachyshka has joined #openstack-oslo08:07
*** ihrachyshka has quit IRC08:13
*** ihrachyshka has joined #openstack-oslo08:13
*** Krast has joined #openstack-oslo08:22
*** gpocente1 is now known as gpocentek08:27
*** gpocentek has joined #openstack-oslo08:27
openstackgerritMa Wen Cheng proposed a change to openstack/oslo.messaging: Disable Topic name used as Queue name for QPID consumers  https://review.openstack.org/11380808:34
openstackgerritJulien Danjou proposed a change to openstack/oslo-incubator: log: add missing space in error message  https://review.openstack.org/11521308:41
openstackgerritAlexei Kornienko proposed a change to openstack/oslo.messaging: Moved all tests to unit folder  https://review.openstack.org/11513608:51
openstackgerritAlexei Kornienko proposed a change to openstack/oslo.messaging: Added functional test for RPC  https://review.openstack.org/11513708:51
*** yamahata has quit IRC08:57
*** ihrachyshka_ has joined #openstack-oslo08:59
*** ihrachyshka has quit IRC09:02
openstackgerritAlexei Kornienko proposed a change to openstack/oslo.messaging: Moved all tests to unit folder  https://review.openstack.org/11513609:02
openstackgerritAlexei Kornienko proposed a change to openstack/oslo.messaging: Added functional test for RPC  https://review.openstack.org/11513709:02
*** dims has joined #openstack-oslo09:35
*** dims has quit IRC09:39
*** AAzza is now known as AAzza_afk09:40
*** alexpilotti has joined #openstack-oslo09:45
*** ujjain has quit IRC09:46
*** ujjain has joined #openstack-oslo09:48
*** ujjain has quit IRC09:51
*** ujjain has joined #openstack-oslo09:51
*** pblaho is now known as pblaho|afk10:00
*** linkid has joined #openstack-oslo10:05
*** pblaho|afk is now known as pblaho10:29
*** pcm_ has joined #openstack-oslo10:30
openstackgerritBogdan Dobrelya proposed a change to openstack/oslo.messaging: Should not send replies for cast messages  https://review.openstack.org/11501010:31
openstackgerritamrith proposed a change to openstack/oslo.concurrency: Log stdout, stderr and command on execute() error  https://review.openstack.org/11465910:44
openstackgerritamrith proposed a change to openstack/oslo.concurrency: Handle a failure on communicate()  https://review.openstack.org/11466410:45
*** ihrachyshka_ is now known as ihrachyshka11:04
*** dims has joined #openstack-oslo11:17
*** pblaho has quit IRC11:50
*** dims has quit IRC11:56
*** Krast has quit IRC12:02
*** gordc has joined #openstack-oslo12:09
*** dims has joined #openstack-oslo12:09
*** pblaho has joined #openstack-oslo12:14
*** pcm__ has joined #openstack-oslo12:22
*** pblaho has quit IRC12:24
*** pcm_ has quit IRC12:26
*** kgiusti has joined #openstack-oslo12:26
*** pblaho has joined #openstack-oslo12:27
openstackgerritOleksii Chuprykov proposed a change to openstack/oslo.db: Add olostest in requirements  https://review.openstack.org/11526912:27
openstackgerritOleksii Chuprykov proposed a change to openstack/oslo.db: Add oslotest in requirements  https://review.openstack.org/11526912:35
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare Driver - Ignore datastore in maintenance mode  https://review.openstack.org/11527212:35
*** dims has quit IRC12:37
*** dims has joined #openstack-oslo12:38
*** jeckersb_gone is now known as jeckersb12:38
*** pblaho has quit IRC12:48
*** markmc has joined #openstack-oslo12:49
*** dtantsur has joined #openstack-oslo12:53
*** pblaho has joined #openstack-oslo12:56
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare Driver - Ignore datastore in maintenance mode  https://review.openstack.org/11527212:57
dtantsurHi folks! Anyone mind reviewing relatively easy patch to oslo-incubator https://review.openstack.org/#/c/113978 ? I'm sorry for bothering you, but it's blocking 2 patches to python-ironicclient. Thank you12:58
*** pblaho is now known as pblaho|meeting13:00
*** amotoki has quit IRC13:03
*** bknudson has joined #openstack-oslo13:12
*** AAzza_afk has quit IRC13:15
*** AAzza_afk has joined #openstack-oslo13:15
*** mriedem has joined #openstack-oslo13:15
*** AAzza_afk is now known as AAzza13:16
openstackgerritVlad Okhrimenko proposed a change to openstack/oslo.db: Add get_non_innodb_tables() to utils  https://review.openstack.org/10899413:16
openstackgerritVlad Okhrimenko proposed a change to openstack/oslo.db: Add get_non_innodb_tables() to utils  https://review.openstack.org/10899413:17
*** miqui has joined #openstack-oslo13:19
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare Driver - Ignore datastore in maintenance mode  https://review.openstack.org/11527213:20
*** ildikov_ has quit IRC13:23
*** jecarey has quit IRC13:28
*** AAzza has quit IRC13:30
*** pblaho|meeting is now known as pblaho13:31
*** zzzeek has joined #openstack-oslo13:32
*** pblaho has quit IRC13:34
*** AAzza_afk has joined #openstack-oslo13:36
*** AAzza_afk is now known as AAzza13:36
*** pblaho has joined #openstack-oslo13:37
*** ildikov_ has joined #openstack-oslo13:38
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare: Support for pluggable Datastore selection  https://review.openstack.org/11455113:39
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare Driver - Ignore datastore in maintenance mode  https://review.openstack.org/11527213:39
*** oomichi has quit IRC13:42
*** HenryG_ has joined #openstack-oslo13:46
*** HenryG has quit IRC13:47
*** zzzeek has quit IRC13:48
*** zzzeek has joined #openstack-oslo13:50
*** linkid has left #openstack-oslo13:50
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare Driver - Ignore datastore in maintenance mode  https://review.openstack.org/11527213:53
amrithdims, are you there?13:54
dimsamrith: pong13:54
amrithhi dims13:55
amrithI'm going to have to do another merge of strutils.py13:55
amrithfrom oslo.utils into oslo.concurrency13:55
amrithsince it appears that o.c has its copy of strutils.py13:55
amrithwhich means I have to pick up some more encodeutils changes13:55
amrithI'll pull that in later today13:55
zzzeekhi viktors , i need a little more guidance re: your comments on https://review.openstack.org/#/c/110170/, thanks!13:55
amrithare you at #troveday by any chance?13:55
amrithYorikSar ^^^ ... pl see above (heads-up). this is to pick up the change I submitted in oslo-incubator which just merged into oslo.utils.13:57
amrithdims ^^^13:57
dimsamrith: did not feel right to pop into to mid summit w/o participating even a little bit :)13:58
dimsamrith: ack (encodeutils)13:58
amrithsorry #troveday is a day long trove related event (cambridge marriott). feel free to drop by ...13:58
amrithmid-cycle is tomorrow through Friday13:58
amrithif you want to stop by #troveday today, that's not a problem (free all day event on openstack trove).13:59
amrithcocktail party in the evening ;)13:59
*** mriedem has left #openstack-oslo13:59
*** mriedem has joined #openstack-oslo13:59
dimsamrith: sounds great. i'll try14:00
amrithdims, I'll push changes later today once I can figure out the lineage of the strutils.py that ended up in oslo.concurrency. looking forward to meeting you if you can make it.14:01
amriththx14:01
viktorszzzeek: hi!14:02
zzzeekhey14:02
viktorsas for your questions14:02
*** amrith is now known as amrith_14:02
YorikSaramrith: Hi.14:03
viktorszzzeek: oslo.db have no test_connect_fail() test method, so foolks from Ironic made it itself14:03
YorikSaramrith: I think you should land patch to oslo.utils, then port to incubator, and then update it in concurrency using usual incubator update process.14:04
viktorsthey can use either our is_backend_avail() method or his - nevermind14:04
zzzeekviktors: from what i can tell , that test only tests the _is_backend_avail() function they made right there14:04
zzzeekviktors: we have plenty of tests for that function in oslo.db14:04
zzzeekviktors: if their function works, it returns False - success!14:04
viktorszzzeek: my note was - this patch will break Ironic's unittests, because they uses self.FIXTURE.* attributes14:04
ihrachyshkazzzeek: hey! so long... :) do you know how to reach anyone from those who maintain sqlalchemy-migate on stackforge to check and merge (and release) my patches that are needed for mysql-connector14:04
ihrachyshka?14:04
YorikSaramrith: I'm trying to figure out if we actually need strutils there and how can we switch to oslo.utils.14:05
zzzeekihrachyshka: i dont know whos doing sqlalchemy-migrate14:05
zzzeekviktors: OK….so  since that test in ironic seems to do nothing useful can I patch them to remove it ?14:05
zzzeekviktors: doesnt seem like a big deal ?14:05
viktorszzzeek: it's not a big deal - to use is_backend_avail() from oslo.db, we need remove self.FIXTURE.* attributes usage14:05
zzzeekviktors: ok14:06
viktorsyou can run locally Ironic unittest with custom oslo.db to see this exception14:07
*** HenryG_ is now known as HenryG14:07
zzzeekviktors: Ok how are things like this timed, we like to fix all consuming projects before oslo.db is released, right?   eg. this doesnt hold up the patch itself14:07
*** markmcclain has joined #openstack-oslo14:07
ihrachyshkazzzeek: ah, ok... I thought that you've passed authority for the codebase to someone. was it technically an independent fork to stackforge space?14:08
zzzeekihrachyshka: last i knew, domen kozar was the maintained14:08
zzzeekmaintainer14:08
zzzeekihrachyshka: also, I never wrote sqlalchemy-migrate.  I gave them some patches around sqlalchemy 0.6 but that was it14:09
ihrachyshkazzzeek: ouch. I was misleaded then. :) ok, thanks anyway.14:10
viktorszzzeek: it's would be really nice to run projects unittests on each oslo.db patch to avoid of this issues. dhellmann has some ideas how to do it14:10
zzzeekviktors: yeah that is dhellmann stuff for sure :)   im still fully confused by incubators and all that14:11
* zzzeek just writes the code14:11
viktors:)14:11
*** oomichi has joined #openstack-oslo14:12
viktorszzzeek: you can look at this spec - https://review.openstack.org/#/c/95885/14:13
*** shakamunyi has joined #openstack-oslo14:16
*** oomichi has quit IRC14:17
*** zzzeek has quit IRC14:17
*** zzzeek_ has joined #openstack-oslo14:17
zzzeek_im having real network probs today14:17
zzzeek_viktors: looks good to me14:17
zzzeek_viktors: so i need to know what else you want to see on that patch, it changed existing functionality in-place so what new tests do you need ?14:18
viktorszzzeek_: I talked about tests from that spec )14:19
viktorszzzeek_: unittests looks good for me, I just don't want to break another projects14:20
viktorsagain :)14:20
zzzeek_viktors: what spec?  https://review.openstack.org/#/c/95885/6/specs/enhance-cross-test-gate-job.rst ?14:20
viktorsyes14:20
zzzeek_viktors: OK sorry I’m trying to get at, what do you need on https://review.openstack.org/#/c/110170/ to approve, just that it runs with other projects ?14:20
zzzeek_viktors: e.g. you want me to patch to ironic first ?14:21
viktorszzzeek_: yes14:21
viktorszzzeek_: or add some backward compatibility stuff to your patch14:21
zzzeek_oh “OS” == “openstack”14:21
zzzeek_viktors: i think they dont want to use that acronym14:21
zzzeek_I thought you meant “open source"14:21
viktorszzzeek_: oh, sorry for this confusion14:22
*** jkraj has joined #openstack-oslo14:25
*** tsekiyama has joined #openstack-oslo14:29
*** celttechie has joined #openstack-oslo14:30
*** shakamunyi has quit IRC14:32
openstackgerritVlad Okhrimenko proposed a change to openstack/oslo.db: Add get_non_innodb_tables() to utils  https://review.openstack.org/10899414:36
openstackgerritVlad Okhrimenko proposed a change to openstack/oslo.db: Add get_non_innodb_tables() to utils  https://review.openstack.org/10899414:36
*** ildikov_ has quit IRC14:38
*** celttechie has quit IRC14:45
*** pblaho is now known as pblaho|afk14:46
*** pblaho|afk is now known as pblaho14:47
*** shakamunyi has joined #openstack-oslo14:47
*** jecarey has joined #openstack-oslo14:48
*** markmcclain has quit IRC14:52
*** markmcclain has joined #openstack-oslo14:53
*** zzzeek_ has quit IRC14:55
*** zzzeek has joined #openstack-oslo14:55
*** celttechie has joined #openstack-oslo14:57
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare: Support for pluggable Datastore selection  https://review.openstack.org/11455115:08
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/oslo.vmware: VMWare Driver - Ignore datastore in maintenance mode  https://review.openstack.org/11527215:09
*** markmcclain has quit IRC15:12
*** alexpilotti has quit IRC15:18
*** flaper87 is now known as flaper87|afk15:19
*** ihrachyshka has quit IRC15:22
openstackgerritMichael Bayer proposed a change to openstack/oslo.db: Reorganize DbTestCase to use provisioning completely  https://review.openstack.org/11017015:32
openstackgerritMichael Bayer proposed a change to openstack/oslo.db: Use testr instance provisioning to lazily create databases  https://review.openstack.org/11048615:32
openstackgerritMichael Bayer proposed a change to openstack/oslo.db: Add transactional test support to provisioning  https://review.openstack.org/11315315:32
zzzeekviktors: I just restored those names and confirm taht ironic’s tests now pass15:32
*** praneshp has joined #openstack-oslo15:39
*** dtantsur is now known as dtantsur|brb15:45
viktorszzzeek: thanks! I'll look at patch15:48
*** AAzza is now known as AAzza_afk15:49
*** mriedem has quit IRC15:55
*** AAzza_afk is now known as AAzza15:58
*** dims has quit IRC15:59
*** dims has joined #openstack-oslo15:59
*** SridharG has quit IRC16:00
*** celttechie has quit IRC16:02
*** mriedem has joined #openstack-oslo16:03
*** dims has quit IRC16:04
*** alexpilotti has joined #openstack-oslo16:04
*** ildikov has joined #openstack-oslo16:05
*** arnaud has joined #openstack-oslo16:05
*** alexpilotti has quit IRC16:06
*** alexpilotti has joined #openstack-oslo16:06
openstackgerritOleksii Chuprykov proposed a change to openstack/oslo.db: Added check to see whether oslotest is installed  https://review.openstack.org/11526916:14
*** pblaho has quit IRC16:15
*** viktors is now known as viktors|afk16:16
*** arnaud has quit IRC16:24
*** AAzza is now known as AAzza_afk16:28
bnemecamrith_: Let's not patch strutils in oslo.concurrency.  Let's just convert oslo.concurrency to use oslo.utils.16:31
*** markmcclain has joined #openstack-oslo16:34
*** AAzza_afk is now known as AAzza16:36
*** jkraj has quit IRC16:47
*** markmcclain has quit IRC16:49
*** arnaud has joined #openstack-oslo16:54
*** stannie has quit IRC16:54
*** markmcclain has joined #openstack-oslo16:55
*** sheeprine has quit IRC16:56
*** harlowja_away is now known as harlowja16:56
*** sheeprine has joined #openstack-oslo16:57
*** ihrachyshka has joined #openstack-oslo16:58
*** sheeprine has quit IRC17:00
*** sheeprine has joined #openstack-oslo17:00
*** mriedem1 has joined #openstack-oslo17:02
*** arnaud has quit IRC17:02
*** mriedem has quit IRC17:03
*** AAzza is now known as AAzza_afk17:09
*** cburgess has joined #openstack-oslo17:09
jogocburgess: o/17:09
jogodhellmann: so cburgess and I have been looking at https://bugs.launchpad.net/nova/+bug/134945217:09
jogocburgess: I like jarlow's gist17:10
jogohttps://gist.github.com/harlowja/9c35e443dfa136a4f96517:10
cburgessYeah I just saw that.17:10
cburgessThough he is using a lock rather then a semaphore. Not sure if that hits the same bug.17:11
cburgessI'm trying to find the original eventlet bug that vishy linked now. Digging through my logs.17:11
jogocburgess: awesome17:11
cburgessOK so... the original bug that vishy things we are hitting is https://bitbucket.org/eventlet/eventlet/issue/137/use-of-threading-locks-causes-deadlock17:12
jogobnemec: paybe you can help us, we have a oslo/eventlet bug17:12
jogobnemec:  maybe*17:12
jogobnemec: that is causing random nova-compute deadlocks because ... FAIL17:12
cburgessIt looks like one or both of these changes https://github.com/openstack/nova/commit/a0bcd7b90c38b104cb278223679cedf5cc11c74c, https://github.com/openstack/nova/commit/4b6ea1e1b87241918baf21a16817417226a9fc62 is resulting in us now trying to hold a threading.semaphore and an eventlet one at the same time.17:13
openstackgerritgordon chung proposed a change to openstack/oslo-incubator: add deprecation note to middleware  https://review.openstack.org/11534917:13
cburgessThe theory is that this is causing a deadlock preventing the eventlet reactor from poping a thead and doing work.17:14
cburgessIn effect deadlocking the entire eventlet stack.17:14
*** dtantsur|brb is now known as dtantsur17:14
jogocburgess: can you update the bug with that17:15
cburgessjogo: Doing it now. ;)17:15
jogocburgess: excellent17:15
jogocburgess: so not sure how to procede because oslo doesn't want to be eventlet specific or whatever that is17:15
* bnemec continues to hate lockutils17:15
cburgessjogo: I was guessing based upon the comments from the original commits. Let me find the original patches.17:16
jogobnemec: yeah, IMHO oslo trying to move away from eventlet was a mistake17:16
jogobnemec: not because eventlet is good, but because its tricky and too easy to get wrong17:17
cburgessjogo: Ouch thats above my pay grade :)17:17
jogobnemec: well wrather the weay oslo tried to move away17:17
jogorrather*17:18
* jogo hasn't had his coffee yet17:18
cburgessjogo: I just updated the ticket with the info. Also the original oslo commits that we *think* are causing the problem are here  https://review.openstack.org/#/c/43895/  https://review.openstack.org/#/c/54581/17:18
cburgessjogo: I think we need to write a test case to prove this one way or another. harlowja seems has a good start. I don't like the idea that we have to run a full tempest test against nova to see the deadlock.17:19
jogobnemec: so how do you think we can move forward and try to resolve this bug?17:19
jogocburgess: excellent idea17:19
bnemecjogo: Yeah, I guess I'm not surprised that mixing locks would cause issues.17:20
cburgessjogo: If we can run something locally or even with a similiar usage pattern at least then we know what the issue is.17:20
jogobnemec: so this bug manifests itself when nova-compute and nova-network are running on the same node17:20
bnemecdhellmann: YorikSar: ^ related to the oslo.concurrency graduation.17:20
harlowjacburgess ya, to easy to do crazy things when u dont monkey patch threading17:20
harlowjaespecially with libraries we don't control...17:20
harlowjaanyone uses a lock anywhere, and it dies17:20
cburgessharlowja: Oh yeah for sure. I suppse we could patch threading but we intentionally don't do that in nova right now.17:20
jogocburgess: why don't we?17:21
harlowjai guess u just never hope a library that gets pulled in tries to use a lock :-/17:21
cburgessjogo: Good question, that was actually going to be my next question. Let me pull up the code where we do the monkeypatch.17:21
cburgessjogo: https://github.com/openstack/nova/blob/master/nova/cmd/__init__.py#L4317:22
cburgessWe only do it if we are using the debugger.17:22
harlowjacburgess any idea why that is? whats special about the debugger17:22
harlowjaseems like u just increased the number of things u have to debug when u turn off threading being monkey patched (isn't that causing more issues when debugging, lol)17:23
cburgessharlowja: No clue, been that way for a while though, I think we even did it that way in essex, I know it was like that in grizzly.17:23
jogothe debugger should be off by default17:23
jogothe debugger is for remote debugging17:24
cburgessThe original eventlet bug goes away if we do thread=False17:24
*** markmcclain has quit IRC17:25
harlowjaa http://en.wikipedia.org/wiki/Heisenbug lol17:25
*** markmcclain has joined #openstack-oslo17:25
jogocburgess: so I am a bit confused, I thought thread=False means we don't monkey patch threading, in which case https://gist.github.com/harlowja/9c35e443dfa136a4f96517:26
jogocburgess: so are you saying not setting thread=False is triggering a different bug?17:26
cburgessjogo: The original bug from comstud indicated that thread=False causes the bug to go away.17:27
cburgesshttps://bitbucket.org/eventlet/eventlet/issue/137/use-of-threading-locks-causes-deadlock17:27
cburgessjogo: Granted we aren't even sure thats the bug we arer hitting. Its just a theory. This is why I think we need to write an indepdent test case kind of like harlowja started with the gist.17:28
jogocburgess: but then we can hit the issue that harlowja has17:28
jogocburgess: gotcha17:28
cburgessjogo: Yeah catch22, damned if we do, damned if we don't.17:28
harlowjaya, run that gist, program locks17:28
cburgessThis is why I'm a but confused as to exactly whats going on and why we are trying to mix threading. Like bnemec we are almost guaranteed to it issues doing that.17:29
cburgessOK that wasn't english17:30
cburgessI need more coffee.17:30
cburgessSo to be clear, we are guessing at the issue. it seems similiar to the original eventlet issue comstud found but it could just be another issue when mixing threading implementations.17:30
jogocburgess: right17:30
*** SridharG has joined #openstack-oslo17:31
cburgessjogo: So... I'll see if I can create a tets case today. This might be tricky given we might need multiple processes holding multiple locks.17:31
bnemecSo, what we might need for lockutils is a way to conditionally use eventlet semaphores even if we aren't monkeypatched?17:32
cburgessbnemec: Yeah we might need an eventlet version and a non-eventlet version.17:33
harlowjaand pray that no other library (like logging uses a lock and calls into any eventlet patched function via any way)17:34
cburgessharlowja: Thats actually how comstud foud the originaly issue, python logging.17:34
harlowjacburgess ya, its rightly using a lock somewhere inside of it17:34
* harlowja reminds me of a conversation yesterday with a guy at work, about how he was saying python should of never introduced the threading concepts at all, lol17:35
bnemecHmm, but that would only potentially solve the debugger case where we monkeypatch thread=False.17:35
cburgessOK well let me see if I can find a way to reproduce this in a simple test case. Something we can all see and work on.17:35
bnemecFor the other case lockutils will already be using an eventlet semaphore.17:35
jogocburgess: good luck!17:35
cburgessjogo: Random question... we aren't running with the debugger during tempest testing are we? Just curious.17:38
cburgessAs we are sure its disabled so we aren't calling thread=False right?17:38
jogocburgess:  I  don't think so17:38
jogocburgess:  99% sure17:38
cburgessk17:38
*** shakamunyi has quit IRC17:39
*** AAzza_afk is now known as AAzza17:41
*** dims has joined #openstack-oslo17:53
dimshi amrith_: i made it17:56
openstackgerritOpenStack Proposal Bot proposed a change to openstack/oslo.messaging: Updated from global requirements  https://review.openstack.org/11536318:06
*** shakamunyi has joined #openstack-oslo18:11
*** viktors has joined #openstack-oslo18:14
*** dtantsur is now known as dtantsur|afk18:22
*** mriedem has joined #openstack-oslo18:23
*** mriedem1 has quit IRC18:24
*** shakamunyi has quit IRC18:30
*** AAzza is now known as AAzza_afk18:35
*** dims has quit IRC18:37
*** shakamunyi has joined #openstack-oslo18:44
*** AAzza_afk is now known as AAzza18:48
*** SridharG has quit IRC18:51
*** dims has joined #openstack-oslo19:00
*** dims has quit IRC19:00
openstackgerritArnaud Legendre proposed a change to openstack/oslo.vmware: Bypass vCenter when uploading to datastores  https://review.openstack.org/11463419:01
*** dims has joined #openstack-oslo19:01
*** dims has quit IRC19:05
*** harlowja has quit IRC19:13
*** harlowja has joined #openstack-oslo19:13
*** zzzeek has quit IRC19:15
*** ihrachyshka has quit IRC19:16
openstackgerritA change was merged to openstack/oslo.messaging: Fix error in example of an RPC server  https://review.openstack.org/10841019:17
*** markmcclain has quit IRC19:17
*** zzzeek has joined #openstack-oslo19:17
*** AAzza is now known as AAzza_afk19:24
*** ildikov has quit IRC19:24
openstackgerritJoshua Harlow proposed a change to openstack/taskflow: Add an example which shows how to use the run_iter cancelling  https://review.openstack.org/11538219:24
*** tpatil has joined #openstack-oslo19:29
tpatilHi, Can someone please review this patch https://review.openstack.org/#/c/103186/? This issue is very critical especially when you are restarting (sending SIGHUP signal) nova-compute service.19:33
*** ihrachyshka has joined #openstack-oslo19:35
*** ildikov_ has joined #openstack-oslo19:39
openstackgerritKen Giusti proposed a change to openstack/oslo.messaging: An initial implementation of an AMQP 1.0 based messaging driver  https://review.openstack.org/7581519:42
*** dhellman_ has joined #openstack-oslo19:46
*** tpatil has quit IRC19:50
*** markmcclain has joined #openstack-oslo19:50
*** freyes has joined #openstack-oslo19:55
*** dhellman_ has quit IRC19:59
*** kgiusti has left #openstack-oslo20:02
openstackgerritJoshua Harlow proposed a change to openstack/taskflow: Add an example which shows how to use the run_iter throw  https://review.openstack.org/11538220:04
*** gordc has quit IRC20:11
*** gordc has joined #openstack-oslo20:12
*** markmc has quit IRC20:21
*** dims has joined #openstack-oslo20:37
*** viktors has quit IRC20:40
*** dims has quit IRC20:45
*** shakamunyi has quit IRC20:46
*** miqui has quit IRC20:49
openstackgerritIhar Hrachyshka proposed a change to openstack/oslo-incubator: Add mock cleanup function to each test case  https://review.openstack.org/9546520:51
*** celttechie has joined #openstack-oslo21:34
*** gordc has quit IRC22:07
*** gordc has joined #openstack-oslo22:08
*** ihrachyshka has quit IRC22:14
*** HenryG_ has joined #openstack-oslo22:17
*** HenryG has quit IRC22:19
*** markmcclain has quit IRC22:26
openstackgerritArnaud Legendre proposed a change to openstack/oslo.vmware: Bypass vCenter when uploading to datastores  https://review.openstack.org/11463422:28
*** jecarey has quit IRC22:29
*** harlowja has quit IRC22:39
*** harlowja has joined #openstack-oslo22:39
*** zzzeek has quit IRC22:42
*** mriedem has left #openstack-oslo22:46
*** zzzeek has joined #openstack-oslo22:47
*** jeckersb is now known as jeckersb_gone22:57
*** arnaud has joined #openstack-oslo22:59
*** bknudson has quit IRC23:01
*** shakamunyi has joined #openstack-oslo23:02
*** gordc has quit IRC23:02
*** zzzeek_ has joined #openstack-oslo23:08
*** zzzeek has quit IRC23:08
*** zzzeek_ is now known as zzzeek23:08
*** zzzeek has quit IRC23:10
*** shakamunyi has quit IRC23:18
openstackgerritArnaud Legendre proposed a change to openstack/oslo.vmware: Bypass vCenter when uploading to datastores  https://review.openstack.org/11463423:21
*** sheeprine has quit IRC23:25
*** sheeprine has joined #openstack-oslo23:25
*** sheeprine has quit IRC23:25
*** sheeprine has joined #openstack-oslo23:25
*** shakamunyi has joined #openstack-oslo23:33
*** ajo_ has joined #openstack-oslo23:34
*** celttechie_ has joined #openstack-oslo23:34
*** sheeprine has quit IRC23:36
*** celttechie has quit IRC23:36
*** ajo has quit IRC23:36
*** sheeprine has joined #openstack-oslo23:36
*** sheeprine has quit IRC23:36
*** sheeprine has joined #openstack-oslo23:36
*** HenryG_ is now known as HenryG23:42
openstackgerritJoshua Harlow proposed a change to openstack/taskflow: Initial shift to having backends provide storage impls  https://review.openstack.org/11442223:45
*** tsekiyam_ has joined #openstack-oslo23:46
*** tsekiyama has quit IRC23:50
*** tsekiyam_ has quit IRC23:50
*** celttechie_ has quit IRC23:53
*** yamahata has joined #openstack-oslo23:53
*** markmc has joined #openstack-oslo23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!