Friday, 2014-04-25

*** eghobo has quit IRC00:07
*** eghobo has joined #openstack-ironic00:07
*** newell has quit IRC00:07
*** eguz has joined #openstack-ironic00:08
*** eghobo has quit IRC00:12
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions  https://review.openstack.org/9023300:22
*** eguz has quit IRC00:23
*** zdin0bot has joined #openstack-ironic00:35
*** tteggel_ has joined #openstack-ironic00:38
*** tteggel has quit IRC00:48
*** tteggel_ is now known as tteggel00:48
*** zdin0bot has quit IRC00:52
*** zdin0bot has joined #openstack-ironic00:52
*** zdin0bot has quit IRC00:53
*** zdin0bot has joined #openstack-ironic00:53
*** yongli has joined #openstack-ironic01:00
*** rloo has quit IRC01:08
*** nosnos has joined #openstack-ironic01:33
*** killer_prince has quit IRC01:42
*** eghobo has joined #openstack-ironic01:47
*** eghobo has quit IRC01:57
*** zdin0bot has quit IRC01:59
*** zdin0bot has joined #openstack-ironic02:00
*** early has quit IRC02:26
*** yfujioka has joined #openstack-ironic02:29
*** early has joined #openstack-ironic02:31
*** yfujioka has quit IRC02:34
*** zdin0bot has quit IRC02:35
*** yongli has quit IRC02:35
*** yongli has joined #openstack-ironic02:49
*** coolsvap has joined #openstack-ironic02:52
*** eghobo has joined #openstack-ironic02:56
*** eghobo has quit IRC03:09
*** zdin0bot has joined #openstack-ironic03:13
*** harlowja is now known as harlowja_away03:13
*** zdin0bot has quit IRC03:15
*** zdin0bot has joined #openstack-ironic03:33
*** flwang__ has quit IRC03:36
*** nosnos has quit IRC03:45
*** coolsvap has quit IRC03:54
*** eghobo has joined #openstack-ironic04:12
*** coolsvap has joined #openstack-ironic04:13
raiesgreghaynes:04:15
*** rameshg87 has joined #openstack-ironic04:45
*** lazy_prince has joined #openstack-ironic04:45
*** nosnos has joined #openstack-ironic04:53
*** matsuhashi has joined #openstack-ironic04:54
*** matsuhashi has quit IRC05:24
*** zdin0bot has quit IRC05:26
*** pradipta_away has quit IRC05:26
*** zdin0bot has joined #openstack-ironic05:26
*** sabah has joined #openstack-ironic05:28
*** zdin0bot has quit IRC05:31
*** matsuhashi has joined #openstack-ironic05:40
*** max_lobur has joined #openstack-ironic05:47
*** coolsvap1 has joined #openstack-ironic05:48
*** coolsvap has quit IRC05:48
*** coolsvap1 is now known as coolsvap|afk05:49
*** coolsvap|afk is now known as coolsvap05:50
*** sabah has quit IRC06:02
*** ifarkas has joined #openstack-ironic06:04
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/8850806:07
*** matsuhashi has quit IRC06:14
*** vkozhukalov has joined #openstack-ironic06:15
greghaynesraies: HAI06:17
*** matsuhashi has joined #openstack-ironic06:24
*** Mikhail_D_ltp has joined #openstack-ironic06:36
*** tatyana_ has joined #openstack-ironic06:49
*** tatyana has quit IRC06:51
*** tatyana_ is now known as tatyana06:51
*** rwsu has quit IRC06:57
*** foexle has joined #openstack-ironic06:59
openstackgerritRamakrishnan G proposed a change to openstack/ironic: Add IloPower module for IloDriver  https://review.openstack.org/9029407:00
*** nosnos has quit IRC07:02
*** eghobo has quit IRC07:02
*** vigneshvar_ has joined #openstack-ironic07:03
*** viktors|afk is now known as viktors07:13
*** matsuhashi has quit IRC07:15
*** matsuhashi has joined #openstack-ironic07:15
*** yuriyz has joined #openstack-ironic07:16
*** ndipanov has joined #openstack-ironic07:26
*** matsuhashi has quit IRC07:28
*** matsuhashi has joined #openstack-ironic07:29
*** tatyana has quit IRC07:33
*** vkozhukalov has quit IRC07:37
*** sabah has joined #openstack-ironic07:50
*** martyntaylor has joined #openstack-ironic07:56
*** BadCub has quit IRC07:59
*** mkerrin has quit IRC07:59
*** matsuhashi has quit IRC08:06
*** mkerrin has joined #openstack-ironic08:06
*** matsuhashi has joined #openstack-ironic08:07
*** derekh has joined #openstack-ironic08:09
*** romcheg has quit IRC08:10
*** romcheg has joined #openstack-ironic08:10
*** nosnos has joined #openstack-ironic08:11
*** Mikhail_D_ltp has quit IRC08:13
*** foexle has quit IRC08:25
*** foexle has joined #openstack-ironic08:25
*** lsmola has joined #openstack-ironic08:29
*** dguerri is now known as _dguerri08:30
*** _dguerri is now known as dguerri08:30
*** lucasagomes has joined #openstack-ironic08:32
*** dguerri is now known as _dguerri08:35
*** _dguerri is now known as dguerri08:35
*** matsuhashi has quit IRC08:36
*** nosnos has quit IRC08:46
*** romcheg has quit IRC08:50
*** foexle_ has joined #openstack-ironic08:51
Mikhail_D_wkG'morning Ironic :)08:51
*** romcheg has joined #openstack-ironic08:53
*** foexle has quit IRC08:54
*** zdiN0bot has joined #openstack-ironic08:56
*** Alexei_987 has joined #openstack-ironic08:56
Mikhail_D_wkcomstud: Hi! I have a question concerning the removal of `@ objectify` from `dbapi` ... When do you plan to do this, of course, if you plan? And how do you evaluate the work in the chain patches `Get rid object model `dict` methods`?08:58
comstudMikhail_D_wk: I'm slowly working on it when I can between other things09:00
*** romcheg has quit IRC09:00
comstudThe next thing on my list is removing it from 'get_nodeinfo_list'09:00
*** zdiN0bot has quit IRC09:00
comstudBut that requires implementing a NodeList object in objects/node.py09:00
comstudand some support into Node to partially fill a Node and allow lazy-loading of other attributes09:01
comstudI may be working on that tomorrow (my time tomorrow :)  it's 2am right now.)09:01
comstudMikhail_D_wk: I'm not sure I understand the 2nd question.. what do you mean by 'evaluate' ?09:02
comstudAre you asking how I'm labeling my patches (commit msgs).. or are you asking about the side effects it can cause?09:03
comstudIf you mean the latter, nothing really changes.. because at the same time, I'm swapping out calls in the code: dbapi.get_node() becomes Node.get(), etc... still the code still gets a Node object.09:04
*** vkozhukalov has joined #openstack-ironic09:04
comstud(in the case of get_nodeinfo_list, I'll be replacing dbapi.get_nodeinfo_list calls with: NodeList.get(attributes=attributes, filters=filters), etc..09:05
comstudNot sure if that answered your question :)09:05
Mikhail_D_wkcomstud: If you seen them, in the right direction I did these patches :)09:06
comstudThe object model ones?09:06
comstudI saw.. I left a comment on one of them in the past09:07
*** romcheg has joined #openstack-ironic09:07
comstudThe tests you're changing under ironic/tests/db don't actually need to change...09:07
comstudBut it's okay if they do. Sqlalchemy supports both dict and attribute notation09:07
Mikhail_D_wkI'm sorry to answer for so long09:07
comstud(Just like our Node object does ATM)09:07
comstudno problem09:08
Mikhail_D_wk:)09:08
comstudI think I would skip modifying the test of ironic/tests/db/*09:08
comstudMy work is going to make them return SQLAlchemy models or dicts09:08
comstudBut...09:08
comstudThe rest of our tests should really use .<field_name> notation, as you've been fixing09:09
comstuderr09:09
*** yongli has quit IRC09:09
comstudI think I would skip modifying the rest* of ironic/tests/db/*09:09
comstudAn example of my change:09:10
comstudget_nodeinfo_list returns a list of [object.Node] right now09:10
*** jistr has joined #openstack-ironic09:10
comstuder09:10
comstudthat's a bad a example09:10
comstudit doesn't do that09:10
comstudhehe09:10
Mikhail_D_wk))09:10
comstudI changed get_node() from returning a object.Node.. to returning the SQLAlchemy model09:11
comstudChanging the tests doesn't break anything because you can SQLAlchemyModel['foo'] OR SQLALchemyModel.foo09:11
comstudIt's possible that I'll be making DBAPI return dicts, however09:11
comstudwhich is why I would stop converting *those* tests09:11
Mikhail_D_wkyes, I saw this09:12
comstudok09:12
comstudMy goal is to not have any use of 'dbapi' other than in objects/*.py09:12
comstud(there may be a few edge cases)09:12
comstudDid I answer your questions?09:13
Mikhail_D_wkclear :) thanks for explanation :)09:14
Mikhail_D_wkyeah :)09:14
Mikhail_D_wkcomstud: And you said you already 2am?09:15
comstudyeah, 2am09:15
comstudtime for bed, I think :)09:16
Mikhail_D_wkThen good night! :)09:16
comstudthanks!  laters..09:16
* comstud &09:16
Mikhail_D_wkGood thinking! :)09:16
lucasagomescomstud, ping re task, node09:18
lucasagomescomstud, https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L71009:19
lucasagomesI think I would need a big clean up before that09:19
lucasagomessome other part of the code assumes that node is being passed as well09:19
*** athomas has joined #openstack-ironic09:20
Mikhail_D_wkHi, lucasagomes :)09:21
lucasagomesMikhail_D_wk, morning :D09:22
*** radsy has joined #openstack-ironic09:27
*** radsy has joined #openstack-ironic09:27
*** rameshg87 has quit IRC09:42
*** vigneshvar_ has quit IRC09:44
*** matsuhashi has joined #openstack-ironic09:47
*** rameshg87 has joined #openstack-ironic09:49
*** radsy has quit IRC09:49
*** nosnos_ has joined #openstack-ironic09:52
*** zdiN0bot has joined #openstack-ironic09:56
*** zdiN0bot has quit IRC10:00
*** matsuhashi has quit IRC10:11
*** matsuhas_ has joined #openstack-ironic10:14
*** athomas has quit IRC10:17
*** athomas has joined #openstack-ironic10:23
*** matsuhas_ has quit IRC10:30
openstackgerritRamakrishnan G proposed a change to openstack/ironic: Add IloDriver and its utils  https://review.openstack.org/8950010:32
*** matsuhashi has joined #openstack-ironic10:42
*** matsuhashi has quit IRC10:43
openstackgerritRamakrishnan G proposed a change to openstack/ironic: Add IloPower module for IloDriver  https://review.openstack.org/9029410:53
*** rameshg87 has left #openstack-ironic10:53
*** dguerri is now known as _dguerri10:57
*** subah has joined #openstack-ironic10:59
*** sabah has quit IRC10:59
*** nosnos_ has quit IRC11:03
*** subah has quit IRC11:04
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic-python-agent: Updated from global requirements  https://review.openstack.org/8872211:16
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-ironicclient: Updated from global requirements  https://review.openstack.org/8924411:23
*** coolsvap is now known as coolsvap|afk11:29
*** romcheg has quit IRC11:45
*** lucasagomes is now known as lucas-hungry11:48
*** jdob has joined #openstack-ironic12:30
*** lazy_prince has quit IRC12:36
*** lucas-hungry is now known as lucasagomes12:56
openstackgerritDmitry Tantsur proposed a change to openstack/ironic: Implement more robust caching for master images  https://review.openstack.org/8538712:59
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Add ManagementInterface  https://review.openstack.org/8606313:01
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Remove 'node' parameter from the VendorPassthru interface  https://review.openstack.org/9036313:09
*** linggao has joined #openstack-ironic13:14
*** romcheg has joined #openstack-ironic13:26
*** mkerrin has quit IRC13:28
*** zdin0bot has joined #openstack-ironic13:33
NobodyCamgood morning ironic, says the man making coffee13:33
NobodyCamTGIF!13:33
zdin0botNobodyCam: morning :-)13:33
NobodyCammorning zdin0bot13:34
NobodyCamlucasagomes: Did you see my ping yesterday?13:35
NobodyCam*CORES* review jam monday 8:00am <-13:35
lucasagomesNobodyCam, morning13:36
lucasagomesNobodyCam, oh ah sure13:36
lucasagomesworks for me13:36
romchegMorning all!13:39
romchegdevananda: Congrats!13:40
lucasagomesNobodyCam, btw, did you test the oslo.messaging?13:40
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Remove 'node' parameter from the VendorPassthru interface  https://review.openstack.org/9036313:41
anteayacongratulations devananda13:42
lucasagomesmorning romcheg anteaya13:42
*** jbjohnso has joined #openstack-ironic13:42
anteayalucasagomes: morning13:43
anteayaI forgot what it felt like to have breakfast with the family13:43
anteaya:D13:43
anteayamorning romcheg13:43
anteayamorning NobodyCam13:43
*** mdenny has joined #openstack-ironic13:48
NobodyCamlucasagomes: I got my test env working late last night .. may test failed ... :( and it was to late for me to dig in to why13:50
NobodyCamlooking now13:50
lucasagomesNobodyCam, ah no hurry, just to know13:51
lucasagomesthanks for testing it out :)13:51
NobodyCammorning anteaya :)13:51
NobodyCam:) we've missed you13:52
NobodyCam:-p13:52
*** max_lobur has quit IRC13:52
anteayaI've missed you too13:52
NobodyCam*(BLUSH)*13:52
anteayaalways glad to come home13:52
anteaya:D13:52
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: set_console_mode to not overwrite node.last_error  https://review.openstack.org/9037613:56
*** martyntaylor has quit IRC14:00
*** Shrews has quit IRC14:03
*** martyntaylor has joined #openstack-ironic14:04
*** Shrews has joined #openstack-ironic14:04
*** dtantsur has quit IRC14:06
comstudlucasagomes: ah, ok. i think maybe we clean up task and node args later then. don't want to hold up the feature14:10
comstudalthough i see you put up a patch14:11
lucasagomescomstud, right, I submitted a new for the first patch of the series14:11
comstudok14:11
*** viktors has quit IRC14:11
lucasagomeswant tot see the feedback of that one before changing the others14:11
comstudyeah14:11
lucasagomescomstud, can u take a quick look?14:11
comstudreviewing now14:11
comstudyesp14:11
lucasagomescomstud, I think the db thing maybe should go to another patch, it's not really related to those changes14:12
lucasagomesthat's better to be refactored at once for each module, instead partially do it on this patch14:12
comstudhehe, your patches are going to conflict with mine14:12
comstudI'll finish reviewing yours, if you look at one of mine? :)14:12
lucasagomescomstud, sure14:12
comstudhttps://review.openstack.org/#/c/89328/14:12
NobodyCamlucasagomes: wow - http://paste.openstack.org/show/Jlw0gfSuy40v5GPdpE77/14:16
NobodyCamnot sure thats anthing from your patch... retaeting14:16
NobodyCamretesting even :-p14:16
comstudhrm14:17
comstudthat's generally bad14:18
*** jgrimm has joined #openstack-ironic14:19
comstudlucasagomes: https://review.openstack.org/#/c/90363/2/ironic/tests/drivers/test_ipminative.py,unified14:24
comstudlucasagomes: were these tests passing the wrong arguments before?!14:24
yuriyzmorning/evening Ironic14:24
lucasagomescomstud, they were...14:24
lucasagomes:(14:24
comstudnice14:24
lucasagomesthey were passing node instead of task14:24
lucasagomesNobodyCam, looking14:24
comstudwel, only node.  not task and node14:24
comstudhehe ok14:25
yuriyzlucasagomes, why code 202 here https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/node.py#L9614:25
comstudseems like those aren't tested quite right if it passed14:25
lucasagomesyuriyz, yeah that's async, shouldn't be 202 :( I will fix14:25
lucasagomesyuriyz, didn't have time to reply on the review, thanks for that14:26
lucasagomescomstud, all the spawn_after is getting _spawn_worker as it first parameter14:26
lucasagomescomstud, should we try to move this _spawn_worker logic to the taskmanager?14:27
comstudlucasagomes: deva wanted to keep it in the Manager14:28
comstudI think I agree with it being in the Manager as well14:28
comstudthis way.. you could still use the task_manager.spawn_worker with say.. 'eventlet.spawn'14:29
lucasagomesyeah for me it looks a bit out of the place as-is14:29
lucasagomesyeah14:29
comstudin case you want to spawn but not use the worker pool14:29
lucasagomesthe manager is the resposable for starting the async job, so it should have control over the thread_pool as well14:29
lucasagomesyeha14:29
comstudyeah14:29
lucasagomesI will add a comment about it, cause at least for me it feels a bit out of place to have the conductormanager controlling the pool, but the task being started by the taskmanager14:30
comstudwell, the taskmanager code is generic14:31
comstudit doesn't care if you use the pool or not.. i kinda like it that way14:31
comstudbut yeah, leave your opinion14:31
comstudin the review14:31
lucasagomesack14:31
anteayaNobodyCam: you're awesome14:32
NobodyCam????14:32
comstudI can move it if there's consensus.. but this is what deva and i discussed14:32
NobodyCamoh the email :-014:32
lucasagomescomstud, right, the patch lgtm I will give it a try here14:32
comstudlucasagomes: you can think of this in the reverse... the worker pool right now doesn't require that you use a task.  there could be a need to use the worker pool without having a task14:33
comstudlucasagomes: in which case, the worker pool should stay in the manager14:33
comstudhehe14:33
lucasagomeshmm14:33
lucasagomesright yeah I can see the benefits14:33
lucasagomesesp the flexibility of passing spawn_after(eventlet.spawn... as u mentioned there14:34
comstudyeah14:34
comstudthere could be a case in the future where we always want to spawn a thread to finish a task, bypassing the pool14:34
comstudi'm not sure what case that would be :)  but who knows14:34
lucasagomesheh yeah... well it sounds like some premature optimization... but I'm not totaly against it in this case14:35
comstudnod14:35
comstudi think the manager maybe wanting to use the worker pool without a task is more of a reason to keep it in the manager14:36
comstudeven if we don't have a case for that right now, either.14:36
lucasagomes+214:36
lucasagomes+114:36
comstud:)14:36
lucasagomesI think I like it more after you explained the reasons why the pool should stay in the manager14:36
comstudi can think of a use case immediately for that14:37
comstudwhich is periodic tasks14:37
comstudfor example, right now...14:37
comstuda couple of periodic tasks loop through a bunch of Nodes14:37
comstudlock and spawn workers14:37
comstudA limit of how many workers it uses was added to 1 of them..14:38
comstuder, I guess only one of them does this (_check_deploy_timeouts)14:38
comstudbut..14:38
comstudit might be better if that periodic task just stole 1 worker14:38
comstudand did all of the locking of each node within that worker14:38
comstudinstead of this weird hack to try to prevent it from eating 'x' workers14:39
comstudbut anyway.. that's a separate issue14:39
comstudbut if we changed that, we would end up spawning a worker with no task14:39
lucasagomescomstud, heh yeah, having a limit for the number of workers that can be spawned by a periodic task is a bit hacky indeed14:43
lucasagomesanyhoo, I'm going to give ur patch a go here14:43
comstudi'm going to propose a patch for that shortly. :)14:43
comstudworking on it now.. kinda bugs me14:43
lucasagomesheh14:43
lucasagomesgreat14:43
comstudwe'll see what ppl think in the review14:43
lucasagomescomstud, about that vendor_passthru... as part of the management interface series I'm removing all that vendorpasstrhu stuff from the ipmitool and ipmitnative drivers14:45
comstudah right14:45
lucasagomesso my patches conflicts heh yay14:46
lucasagomesI didn't want to add more patches to that management interface series, need to get some merged firtst14:46
lucasagomesthe linear dependency is getting really long14:46
comstudyeah i saw14:46
comstudi was working backwards thinking it was going to be small14:46
comstuduntil I got like 5 patches back14:46
comstudhehe14:46
lucasagomesyeah14:47
NobodyCamwe need to land those14:47
lucasagomesNobodyCam, +1000000014:47
lucasagomeshah14:47
NobodyCamhow many cores do we have on line atm?14:47
lucasagomesNobodyCam, I put another patch up today for the first of the serie... will wait reviews on that to start fixing the others14:48
NobodyCamlol14:48
lucasagomesNobodyCam, 3 I think? me u and romcheg14:48
openstackgerritDmitry Tantsur proposed a change to openstack/ironic: Consider Glance checksum when caching master images  https://review.openstack.org/9039014:49
NobodyCamlucasagomes: hummm getting stranger and strnger14:52
NobodyCamhttp://paste.openstack.org/show/FqGf0GLZDcYh5Ki5B6mj/14:52
lucasagomesNobodyCam, ouch14:52
lucasagomesNobodyCam, doesn't seems related to the oslo.messaging but hmmm14:52
lucasagomesweird anyway14:52
NobodyCam2014-04-25 14:48:49.494 4221 ERROR wsme.api [-] Server-side error: "Node a4a4b637-a58d-4496-bf0b-b42f5d02b97d is locked by host undercloud-undercloud-khowqq7mhuco.novalocal, please retry after the cur14:54
NobodyCamrent operation is completed.14:54
NobodyCamwe have to do something about the random nature of this failure14:54
lucasagomesyeah, we should retry those14:58
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Cleanup mock patch without `with` part 2  https://review.openstack.org/7325614:58
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Cleanup mock patch without `with` part 3  https://review.openstack.org/8653614:58
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Cleanup mock patch without `with` part 1  https://review.openstack.org/7322314:58
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Get rid of the newline "\"  https://review.openstack.org/6679314:58
NobodyCamour gate jobs will never be stable14:58
NobodyCam:(14:58
NobodyCambbt... brb14:59
*** zdiN0bot1 has joined #openstack-ironic15:00
*** zdiN0bot1 has quit IRC15:04
*** foexle_ has quit IRC15:06
lucasagomes:/15:08
*** zdin0bot has quit IRC15:08
*** zdin0bot has joined #openstack-ironic15:08
*** dtantsur has joined #openstack-ironic15:11
jrolldevananda: congrats :)15:12
jrollhappy friday ironic :)15:12
*** BadCub has joined #openstack-ironic15:14
*** stephenpearson has joined #openstack-ironic15:16
NobodyCamhappy friday jroll15:16
NobodyCamlucasagomes: when you tested the messaging patch did you attempt to spin up more then one node?15:26
lucasagomesjroll, happy friday15:26
lucasagomesNobodyCam, hmmm not that I remember15:26
lucasagomesNobodyCam, I can try to give it a go15:26
NobodyCamI ask because I in my testing seed deploys undercloud but under cloud is not deploying overcloud15:26
NobodyCami am going to retest with out hte patch15:27
lucasagomesright15:27
lucasagomeslemme give it a try as well15:27
*** newell has joined #openstack-ironic15:29
NobodyCamneed to rebuild images for the test... will take a few min15:29
*** kathay has joined #openstack-ironic15:31
*** martyntaylor has quit IRC15:40
*** martyntaylor has joined #openstack-ironic15:43
*** shakamunyi has joined #openstack-ironic15:44
agordeevjroll: JayF: russell_h: morning :) wdyt about hw discovery improvement? -> https://etherpad.openstack.org/p/ipa-discovery-ext15:45
NobodyCammorning agordeev :) happy friday15:46
agordeevNobodyCam: morning! happy friday to you too15:46
agordeevalso morning for everyone else15:47
agordeevhave a nice TGIF15:47
Mikhail_D_wkgood morning/afternoon/evening all! :)15:47
agordeevMikhail_D_wk: morning15:47
NobodyCammorning Mikhail_D_wk15:47
ShrewsHas anyone asked to have Ironic added to the subscription topic categories for the openstack-dev ML?15:48
*** martyntaylor has left #openstack-ironic15:48
Shrewsguess not  :)15:50
NobodyCam:-p15:50
openstackgerritRussell Haering proposed a change to openstack/ironic: Drivers may expose a top-level passthru API  https://review.openstack.org/8191915:52
lucasagomesNobodyCam, deployed two machines http://paste.openstack.org/show/77105/15:55
NobodyCamw00t.. did you deploy them at the same time. or do one then then other15:56
lucasagomesNobodyCam, one after the other15:57
NobodyCamack15:57
* kathay looks at NobodyCam16:00
kathaysyn-ack16:00
*** zdiN0bot1 has joined #openstack-ironic16:00
NobodyCam:-p16:01
*** stephenpearson has quit IRC16:03
*** stephenpearson has joined #openstack-ironic16:03
*** eghobo has joined #openstack-ironic16:05
*** zdiN0bot1 has quit IRC16:05
NobodyCammorning kathay :-p16:08
devanandag'morning, all16:08
NobodyCamgood morning devananda :)16:09
*** matty_dubs is now known as matty_dubs|lunch16:20
lucasagomesmorning devananda16:28
lucasagomesNobodyCam, devananda when you get some time, can you take a look at https://review.openstack.org/#/c/86063/ ?16:29
lucasagomesI fixed somethings that comstud pointed out, but I'd like ur opnion about it16:30
lucasagomeswhether I should not pass the 'node' parameter for those methods16:30
comstudya, I'd like to clean up where we pass both 'task' and 'node', since there's no reason to send node separately (it's in the task)16:31
devanandaexactly16:31
NobodyCamI thought we tackled that already16:32
*** athomas has quit IRC16:32
comstudno, it's all over the place16:32
comstud:)16:32
comstudlucas started this: https://review.openstack.org/#/c/90363/  based on my feedback16:32
devanandalucasagomes: I have a concern adding NotSupportedByDriver16:33
comstudbut there's a number of other cases after that still16:33
lucasagomesdevananda, right16:33
devanandalucasagomes: if drivers' functionality differ enough within core/common interfaces that we need to add that, we're exposing an implementation detail to users that should be abstracted16:33
devanandawe shouldn't be doing that16:34
lucasagomesdevananda, hmm, problem is that some drivers implement get current bootdev, some other doesn't16:35
lucasagomesso I wanted to have a specific errors for those who doesn't support it16:35
devanandalucasagomes: that exc isn't used in this patch (probably used in follow on..)16:35
comstud$ egrep -R 'task, (node|task\.node)' ironic | wc -l16:35
comstud13816:35
devanandalucasagomes: is the intent for it to indicate that an interface isnt supported, or that a given method of an interface isn't supported?16:35
comstudheh16:35
lucasagomesdevananda, yeah it was being used in the Fake driver, but I think I changed it after rloo review16:35
JoshNangagordeev: i like the improved data, which could be a new hardware manager. the driver saves whatever you send it to the DB. as far as what level of hardware discovery, i'm not sure i see much reason to add it.16:35
lucasagomesdevananda, a given method of the interface isnt supported16:36
devanandalucasagomes: right. i think that violates the interface contract16:36
devanandalucasagomes: if a driver implements a core or common interface, it must implement all of it16:36
lucasagomesdevananda, they all support setting the device, but not getting16:36
devanandalucasagomes: and we have invalidparameter for eg. "that boot device isn't valid on this hardware"16:37
JoshNangagordeev: basically, you add all your hardware managers to the config file, build the agent, and whatevers configured in hardware.get_manager() sends it data as part of its initial checkin, it gets saved to the db, yay!16:37
lucasagomesdevananda, hmm16:37
devanandalucasagomes: what doesn't support getting it?16:37
lucasagomesdevananda, https://github.com/stackforge/pyghmi/blob/master/pyghmi/ipmi/command.py#L125-L12616:37
*** yuriyz has quit IRC16:38
lucasagomesipmitool also I don't think it has a method to actually get the current bootdev16:38
devanandaheh, ok16:39
devanandajbjohnso: comments ^ ?16:39
devanandalucasagomes: returning NULL or '' would seem better than an exception there16:39
lucasagomesdevananda, right, we can have a boot_devices.NONE = None16:40
lucasagomesand I can return that in case I can't get it16:40
devanandahmm16:40
jrollagordeev: gah, forgot to look yesterday. will definitely try to get you feedback soon16:41
devanandaat least that's my initial reaction. it's not not-implemented. if the hardware lacks support, that's not the driver's fault.16:41
lucasagomesdevananda, would be better to return something like 'uknown'16:42
lucasagomes?16:42
devanandalucasagomes: probably16:42
devanandain mysql, that's what NULL/NONE means :)16:42
lucasagomesdevananda, right, ipmitool has a None options for boot dev (since we are basing our options in the ipmitool ones)16:43
devanandaright16:44
devanandabrb16:44
*** athomas has joined #openstack-ironic16:45
lucasagomesdevananda, ack will make some changes in the patch then, cheers for the feedback16:45
*** harlowja_away is now known as harlowja16:45
*** derekh has quit IRC16:47
*** mkerrin has joined #openstack-ironic16:48
*** igor has joined #openstack-ironic16:59
*** ndipanov has quit IRC16:59
lucasagomescomstud, 138 :(17:00
*** igor__ has quit IRC17:00
lucasagomescomstud, yeah it's all over the place, the problem is also that some helper functions get boths methods as well, e.g https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/pxe.py#L47217:01
*** zdiN0bot1 has joined #openstack-ironic17:01
comstudlucasagomes: yeah17:04
lucasagomesget both paramters*17:05
devanandaback17:05
*** zdiN0bot1 has quit IRC17:06
*** eghobo has quit IRC17:07
*** openstackstatus has quit IRC17:07
*** eghobo has joined #openstack-ironic17:08
*** openstackstatus has joined #openstack-ironic17:08
NobodyCamwb17:10
*** stephenpearson has quit IRC17:10
kathaygood morning, NobodyCam   :)17:15
NobodyCam:)17:15
comstudlucasagomes, devananda: so17:16
comstudlucasagomes, devananda: I guess a lot of these calls need to keep 'node' because it's possible you have locked multiple nodes17:16
comstudand you don't know which one in the task you're dealing with17:16
lucasagomescomstud, I gotta look deeper into it, but I dunno about any of those calls that are multinode17:17
comstudwell17:17
comstudthe problem is the helper methods should probably not assume17:17
comstudalthough I kinda dislike the multiple-node-locking in the first place17:18
comstudhehe17:18
lucasagomesyeah17:18
lucasagomesif the third one failed to get the lock should u unlock the others ? etc...17:18
comstudnod, right now must succeed locking.. others are unlocked and exception re-raised if not17:19
comstuder, all must succeed17:19
comstudbut, i'm not sure this is used anywhere17:19
devanandaright. well, planned feature that we moved away from and all17:19
devanandaoriginal intent of the task manager was to support synchronizing operations on >1 node, but we've pushed that logic much higher up the stack17:20
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Add ManagementInterface  https://review.openstack.org/8606317:20
comstudnod17:20
comstudSo we can remove multiple node support in task manager and clean this all up?17:20
devanandaat this point, i don't think we have anything functionally operationg on >1 node at a time17:21
comstudI think that's required in order for us to fully ditch passing both 'task' and 'node'17:21
devanandaso yes, it can probably be cleaned up...17:21
devanandathough17:21
lucasagomes:)17:21
devanandathere are some (specialized?) hardware types where it may still be handy17:21
devanandaeg, dense compute with 4 SOCs on a single blade/cartridge17:21
devanandacertain components are shared between them17:22
devanandaso eg, flashing firmware on one node will affect the other 317:22
lucasagomesdevananda, like operations that is done to one of the nodes will affect the others shared?17:22
lucasagomeshmm17:22
devanandayes17:22
comstudyou'll need some decent changes I think to support that17:22
devanandaright17:22
comstudessentially any operation would have to for node in task.resources:17:22
comstud   call_method(task, node)17:22
comstudwherever you need17:22
devanandaexactly. that was the original intent I had with task17:23
lucasagomesright, maybe leaving it fells into the premature optimization thing? since we would need many other parts to change to actually get those working17:23
devanandaso we should still rip out node as a separate param :)17:23
comstudmight be more simple to just wrap TaskManager with a TasksManager when you need it :)17:23
comstudlist of tasks vs task with list of nodes17:23
openstackgerritDavid Shrewsbury proposed a change to openstack/ironic: Implement instance rebuild in nova.virt.driver  https://review.openstack.org/9042917:24
NobodyCamgrrrrrrr... Node cd950b2f-c437-4fd5-bc68-560726aca7d6 is locked by host undercloud-undercloud-iztcbftbhsul.novalocal, please retry after the current operation is completed.17:24
NobodyCam:-p17:24
comstudanyway, ok, cool17:24
lucasagomesNobodyCam, same error w/o oslo.messaging?17:25
*** zdin0bot1 has joined #openstack-ironic17:26
NobodyCamI have gotten that with both... I believe that is a real random error. just happen to be power syncing17:26
*** vkozhukalov has quit IRC17:26
Shrewswoohoo! go, go shrews3  http://paste.openstack.org/show/77116/17:27
* Shrews crosses fingers and toes17:27
*** zdin0bot has quit IRC17:27
jbjohnsodevananda, let me look17:27
*** russellb is now known as rustlebee17:28
jbjohnsodevananda, so ipmitool doesn't support getting the device?17:28
NobodyCambrb17:29
jbjohnsolucasagomes, oh, let me explain that comment17:30
jbjohnsolucasagomes, it's not by device that 'getting' is a problem17:30
jbjohnsohistorically, I've seen systems that don't implement getting *or* setting17:31
jbjohnsounder the covers, it's impossible at the IPMI layer to support set and not get of that parameter17:31
jbjohnsowell, guess it's not impossible, but man that would be some weird work to make it one way...17:31
*** romcheg1 has joined #openstack-ironic17:32
jbjohnsoI have seen some that errored on any boot device request, and I've seen some that happliy except it, but the respective boot firmware does not honor it17:32
*** matty_dubs|lunch is now known as matty_dubs17:32
jbjohnsobut I see now that ipmitool doesn't actually front end getting the current setting..17:33
lucasagomesthat's a good feeback17:33
jbjohnsoraw 0 9 5 0 017:33
jbjohnsothat would be the ipmitool way to get the data17:33
jbjohnso 01 05 80 00 00 00 0017:33
jbjohnsothe return code is implicitly 0 in that case...17:33
lucasagomesyeah, well, I gotta dig more into the ipmitool, but I didn't find the option the last time I tried17:33
jbjohnsoand then you have to convert and bitmask the values17:34
jbjohnsoor chassis bootparam get 517:35
jbjohnso   - Boot Device Selector : Force PXE17:36
lucasagomesright will give it a try :D17:36
JayFjbjohnso: +1 I've seen similar weirdness with setting boot devices17:37
jrollso while we're talking ipmi stuff17:37
jbjohnsodevananda, fwiw, I updated for ipmi 2.0 rev 1.1, though the most interesting bits (lan cfg params) aren't done at all17:37
jrollI have a question17:37
devanandajbjohnso: gotcha. so most likely, either the BMC supports both or neither, and we may or may not even know (eg, silent accept-and-ignore is possible)17:37
jbjohnsodevananda, correct.  The BMC is nothing more than 'remember these bits'17:38
jbjohnsodevananda, it's really BIOS/UEFI that asks BMC for those bits, and if it don't bother, well then the obvious happens17:38
jbjohnsobut that's a problem of using a less than thorough vendor, if it doesn't work in a silent fashion, tht's between a user and his vendor17:39
jbjohnsodevananda, btw, how's your broadcom nic firmware going ;)17:39
devanandajbjohnso: right. so if getbootdev returns None -- that probably means the BMC just doesn't know.17:39
*** kathay has quit IRC17:39
jbjohnsodevananda, it means 'no specific guidance'17:39
jrollwe have a vendor that has a bug in set boot device - they recommend sending a sequence of raw bytes before sending the bootdev command. is that possible to upstream? maybe make it some kind of config option?17:39
jbjohnsodevananda, I call that 'default'17:39
jbjohnsojrist, what is the sequence of raw bytes?17:39
jbjohnsojroll, err what is the sequence of raw bytes?17:40
jrollI don't remember off hand17:40
jrollsec17:40
jristjbjohnso: :)17:40
jbjohnsonow everyone knows exactly how many characters I type before I hit 'tab'17:40
devanandajroll: my guess is, that would fit well within pyghmi but not ironic17:40
jristI was just going to spout off some random bytes17:40
jrolllol17:40
jrolldevananda: hmm17:40
jrollwe're using ipmitool atm17:40
jbjohnsoright, but if I see the bytes, I might be able to understand what they are getting at17:40
jrollJoshNang couldn't get pyghmi working with our hardware iirc17:40
jrolloh sorry, deva distracted me17:41
jroll00 08 03 0817:41
jrolljbjohnso: ^17:41
JayFthat's the combination to my luggage!17:41
jbjohnsook, one moment17:41
comstudI just used that joke last night.17:41
jrollheh17:41
JoshNang:)17:42
devanandajbjohnso: so one of my req's (i may have mentioned this to you a while back) to getting pyghmi as the default power driver in ironic17:42
jbjohnsoI'm combing through the spec as we speak (to keep with the theme)17:42
devanandajbjohnso: is going to be automated testing on hardware17:42
NobodyCamready kafka17:42
jbjohnsoset system boot options..17:42
JoshNangjroll: i didn't debug at all why pyghmi wasn't working on our hardware. i tried, it didn't immediately work (think it had problems initiating a connection), and ipmitool filled all our needs, so i left it there.17:43
jrollJoshNang: heh17:43
jrollok17:43
jbjohnsojroll, ok, pyghmi already does taht17:43
jrolljbjohnso: sends those bytes with the bootdev command?17:43
jbjohnsojroll, yep17:43
jrollhuh.17:43
jrollawesome17:44
jbjohnsojroll, it disables a very annoying feature of set boot device17:44
jbjohnso'don’t automatically clear boot flag valid bit if Chassis Control command not17:44
jbjohnsoreceived within 60-second timeout (countdown restarts when a Chassis17:44
jbjohnsoControl command is received)'17:44
jbjohnsoI hate that behavior and hate that it is supposed to be default17:44
jbjohnsoso pyghmi says 'screw it' and already does that17:44
jbjohnsoso I agree with your vendor17:44
jbjohnsowell, it might be a bug still yet17:45
jrollheh17:45
jrollcool17:45
jbjohnsothey may expire the timer near end of post instead of waiting for chassis reset17:45
*** lucasagomes is now known as lucas-dinner17:45
jrollit's definitely a bug, it messes with the sata controller somehow. but anyway. cool.17:46
jrollthanks :)17:46
jbjohnsoanyhow, pyghmi better work or else someone (me) has failed.  Unless it's a certain dell model in which case I'll hate them for a blatant security violation17:46
jrolllol17:46
jrollyeah, I'll poke at it today17:46
jrolland file bugs as needed17:46
jbjohnsocan't remember which one it was, but they made man-in-the-middle prevention impossible in some firmware on some model17:46
jrolllovely17:46
jrollthanks dell :P17:46
jbjohnsoI hope that day is long gone though17:47
jbjohnsoIf you do see a failure, and wouldn't mind capturing it and divulging the password used for the session captured17:47
jbjohnsowith tcpdump or wireshark17:47
jbjohnsothen I can decrypt it and tell roughly what might be weird17:48
jrollsure thing17:48
jbjohnsoI tried even avoiding the stuff that demands 'intelplus' in ipmitool by being sneaky17:48
*** Hefeweizen has quit IRC17:50
jbjohnsojroll, generally, I can make sense of 'weird random byte streams' in the context of ipmi at least17:52
jbjohnsomy goal is to have reasonable behavior out of the box without configuration if at all possible.  I'm annoyed at lan v lanplus and intelplus and so on17:52
jrolljbjohnso: cool, I'm glad one of us can make sense of that :)17:53
jbjohnso*SO FAR* I've been able to avoid most of the drivers of those options through careful planning...17:53
jbjohnsowell, 'careful' may be charitable..17:53
*** hemna has joined #openstack-ironic17:53
jrollheh17:54
jbjohnsojroll,         response = self.raw_command(netfn=0, command=8, data=(3, 8))17:54
jbjohnsosee, it was already in there...17:54
jbjohnsoeasiest feature request ever17:54
jrollI believed you :)17:54
* jroll goes off to test17:55
jbjohnsoI'll claim that I considered the request so important, I actually time travelled (backwards) to fix it17:55
jbjohnsotime travelling forwards is too easy...17:55
jbjohnsowell, at the rate of 1 s/s at least17:56
jrolllol17:56
*** hemna has quit IRC17:57
jbjohnsodevananda, fyi, I am able to pull all the sensor data with pyghmi at the moment17:58
jbjohnsodevananda, and provided a 'health' function to get to the 'what is actually wrong' bit of the output (if anything)17:58
jbjohnsoI still need to do the inventory and eventlog bits, but my next stop is socket pooling and maybe multiprocessing pooling...17:58
devanandajbjohnso: inventory and console/event log would be great17:59
jbjohnsodevananda, yeah, the biggest thing about console is the security model you want18:00
jbjohnsodevananda, since I assume you don't want to be in the middle of a highly stateful connection...18:00
devanandajbjohnso: nope. and fwiw, i think we need console *log* more than an interactive console18:01
devanandajbjohnso: but others may disagree18:01
JayFI disagree18:01
devanandasee :)18:01
NobodyCamdevananda: not I18:01
jbjohnsodevananda, well, confluent offers both18:01
JayFAn interactive console is super useful to recovering busted machines without needing administrator intervention18:01
jbjohnsoit just requires some authentication scheme for the live console18:01
JayFso a user consuming a node from Ironic can fix their machine if, say, they boot a kernel on it that bricks it :)18:01
jbjohnsothe plain text console is indeed plain text, but there is a binary metadata file that provides extra awesomeness18:01
NobodyCamlog would be awesome for the gate tests where you can not do interav=ctive18:01
NobodyCaminteractive even18:02
JayFinteractive > log > nothing18:02
jbjohnsoerr the console log18:02
devanandaJayF: well, if they deploy a bad kernel, they should just deploy another good kernel18:02
jbjohnsodevananda, what happens if they flash that HP provided broadcom firmware ;)18:02
JayFdevananda: I mean more in the sense of a user, in an operating system, doing something like "apt-get dist-upgrade", and the resulting kernel not booting18:02
jbjohnsoheh18:02
jbjohnsoanyway18:02
devanandajbjohnso: right... heh.18:02
jbjohnsohttps://www.youtube.com/watch?v=G_lDaktYnsQ18:03
NobodyCamlucas-dinner: with out patch :( http://paste.openstack.org/show/ZZh2uWojDvFsr9d9AEZN/18:03
jbjohnsothat's how confluent can do console18:03
devanandaJayF: "nova rescue $INSTANCE"18:03
jbjohnsoand there is logging in that18:03
NobodyCamretesting with the patch18:03
devanandaJayF: to PXE boot the instance with a known-good kernel so they can mount / and fix their grub conf (or what ever)18:03
devanandaJayF: no need for SOL18:03
jrolljbjohnso: another feature request: allow ctrl-c to interrupt ipmi operations :)18:03
JayFHmm. That sounds awesome. I wonder if we support that in the Rackspace cloud...18:03
JayFmy Openstack knowledge is shaded a bit more red and black than it should be18:04
jrolljbjohnso: for example if they hang18:04
devanandaJayF: hah. i dunno, but it's part of nova's API and should be easy enough to implement in ironic18:04
jbjohnsojroll, ctrl-c doesn't interrup something?18:05
jrolljbjohnso: I have a get_power() that hung and ctrl-c doesn't help :/18:05
jrollI'm in the python shell fwiw18:05
jbjohnsodevananda, easiest thing is to give console.  Even if there is a 'better' way, it's easier to do what is familiar18:05
jbjohnsojroll, hmm, but you were able to make the object ok?18:05
NobodyCambrb... quick walkies18:05
jbjohnsojroll, what about the ipmitcl.py util?18:05
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions  https://review.openstack.org/9023318:06
jrolljbjohnso: yeah, I did: set_bootdev(), set_power(), get_power(), set_power(), get_power() #hung18:06
jbjohnsodevananda, besides, there are also vendors that don't make some things available except via F118:06
jrolljbjohnso: I haven't tried that (nor was I aware of it)18:06
jbjohnsowhich is in no way a good situation, but a console can alleviate the bad...18:06
jrollI think this is just a random bmc oddity18:06
jbjohnsojroll, let me see what happens18:06
jrollwe've had plenty of those even with ipmitool18:06
jrolljbjohnso: hm?18:07
jbjohnsooh, I'm doing the python interpreter18:07
jbjohnsoto see if I can see the same thing18:07
jrolloh18:07
jbjohnsoI actually never used it from the interpreter directly, believe it or not18:07
jrollI mean, I'm definitely inclined to blame our hw for the lockup, but ctrl-c wasn't working yeah18:07
jrollha18:07
jrollheh, it did end up booting18:08
jbjohnsoset_power to what?18:08
jrollseems to wfm18:08
jrollI flipped it off, then on18:09
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions  https://review.openstack.org/9023318:09
jrollthen get_power after on hung18:09
jrollbut it did boot18:09
jbjohnsojroll, shared port configuration?18:09
jbjohnso(wondering if the on caused link state change to have an outage...)18:09
jrollthe OOB NIC also has a 1g NIC, yeah18:10
jbjohnsothough that should raise an exception18:10
jrollwe *usually* don't see a hang using ipmitool18:10
jrollidk18:10
* jroll kicks $vendor18:10
jbjohnsowondering if it's specifically related to the timing of the power cycle and network conditions...18:11
jbjohnsoa session that is valid but loses networking  for a blip should still timeout or recover...18:11
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Don't log MACAlreadyExists in Manager.update_port  https://review.openstack.org/9043618:11
jbjohnsojroll, but it did seem to at least generally 'work as advertised'?18:11
jrollyeah18:11
jrollyep :)18:11
JayFAh so the difference between CLI ipmitool and pyghmi in this case is that the session is persistent?18:11
jrollJayF: yeah18:11
jrollalthough how ironic uses pyghmi is one session per command18:12
jbjohnsoJayF, yeah, the session hangs on for dear life for the lifetime of the object18:12
jbjohnsoincidentally, it also caches SDR to memory per session18:12
jrolljbjohnso: fwiw it worked this time, so who knows18:12
jbjohnsowell, SDR is fetched the first time on demand, but then retained18:12
devanandajroll: your OOB and IB NICs share a connection?18:13
JayFWell, for now.18:13
jbjohnsoso the first 'get_health' is significantly longer than a follow up 'get_health' on the object18:13
jrolldevananda: not really18:13
jrolldevananda: they won't18:13
devananda18:10:07 < jroll> the OOB NIC also has a 1g NIC, yeah18:13
jrollthey do today18:13
jrollright18:13
devanandaah18:13
jrollwe also have other NICs on the box18:13
devanandajroll: yay security18:13
jrollthat 1g should be disabled18:13
devanandaright18:13
jrollright now it is not18:13
devanandak18:13
jroll:)18:14
jrollso here's a thing18:14
jrollsay a BMC gets in a state where it needs a reset18:14
jbjohnsojroll, was the command perhaps longer than 60 seconds at the prompt?18:14
jbjohnsojroll, in the interpreter, while at the prompt, our keepalive isn't running and I think the bmc got mad...18:15
jrollwill ironic end up putting the node in maintenance mode or?18:15
jrolljbjohnso: ahhh :)18:15
jbjohnsojroll, I just went back and my object needed to be ctrl-ced a lot before it gave up18:15
devanandajroll: atm, i think it will, because sync_power_state will fail repeatedly18:15
jbjohnso^C^C^C^C^C^C^CTr18:15
jbjohnso'Tr' is the Traceback from KeyboardInterrupt18:15
jrollhuh18:16
jrolldevananda: ok18:16
jbjohnso>>> cmd.get_health()18:16
jbjohnso{'badreadings': [{'name': 'Mezz Exp 2 Fault', 'imprecision': None, 'value': None, 'states': ['Critical'], 'health': 2, 'units': '', 'type': 'Add-in Card'}], 'health': 218:16
jrollcool18:16
jbjohnsonow if I wait long enough, I bet it gets hung up...18:16
*** zdin0bot1 has quit IRC18:17
jbjohnsoI'll have to leave it at prompt for 60 seconds or so...18:17
*** zdin0bot has joined #openstack-ironic18:17
jbjohnsoI wouldn't have noticed this because usually the event loop is running...18:17
matty_dubslinggao: About to +1 your patch. But, funny story... Was testing ipmitool sol activate command over ssh, vs. in shellinaboxd, to make sure it worked before testing your patch18:19
matty_dubsssh also binds to ~?, and ~. disconnects you from an ssh session apparently18:19
matty_dubsTook me a little bit to figure out why I kept losing my ssh connection when testing.18:19
linggaoHi matty_dubs, yes. I encountered that problem before. :)18:20
linggaoI'll add -v flag to the ipmitool commnad if debug is enabled per Yuriy's comment. So please wait...18:21
jbjohnsomatty_dubs, that's why 'confetty' doesn't bind '~' for the escape sequences ;)18:21
matty_dubslinggao: Sure thing18:21
matty_dubsjbjohnso: Yay! ;)18:22
jbjohnsoI love it when I play 'I forgot how many hops I sshed through, guess the number of ~s'18:22
matty_dubsIt's been years since I remembered the ssh escape sequences, and I was a couple levels deep in ssh sessions, so I was completely baffled when they all just closed while testing this18:23
jbjohnsojroll, btw, ctrl-c once did work when I waiting for it to catch... I don't know18:23
jbjohnsowell, I'll not pay too close attention, since I'm pretty sure that leaving the object idle at the interactive prompt isn't a significant use case..18:23
jbjohnsootherwise I'd start doing things that could wreak havoc with eventlet/non-eventlet environments...18:24
openstackgerritlinggao proposed a change to openstack/ironic: Support serial console access  https://review.openstack.org/6410018:33
matty_dubsIs pep8 going to gripe about line 430? https://review.openstack.org/#/c/64100/24..25/ironic/drivers/modules/ipmitool.py18:35
openstackgerritlinggao proposed a change to openstack/ironic: Support serial console access  https://review.openstack.org/6410018:35
matty_dubsThat was fast! ;)18:36
linggaomatty_dubs, I forgot to run pep8. This new one should pass the Jekins.18:37
lucas-dinnerNobodyCam, :(18:37
NobodyCamlucas-dinner: testing again with patch18:37
linggaomatty_dubs, thanks a lot for testing.18:37
matty_dubsMy pleasure18:37
*** athomas has quit IRC18:38
adam_goh finally18:43
adam_gcheck-tempest-dsvm-virtual-ironic SUCCESS in 23m 30s (non-voting)18:43
NobodyCamadam_g: \o/18:43
devanandaadam_g: woot!18:44
adam_gshould we make that voting at some point in the near future, if it proves stable over the next week or so?18:44
NobodyCamadam_g: ++18:45
*** comstud is now known as bearhands18:49
devanandaadam_g: yes18:49
devanandaadam_g: we should make 2 changes: a) voting on ironic and python-ironic, b) move out of experimental for devstack, devstack-gate, and nova18:49
adam_gdevananda, yeah.. im also going to add it to DIB. since a change there earlier this week broke our deployment ramdisk18:53
*** zdiN0bot1 has joined #openstack-ironic19:03
*** vkozhukalov has joined #openstack-ironic19:06
*** zdiN0bot1 has quit IRC19:07
*** datajerk has quit IRC19:10
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions  https://review.openstack.org/9023319:13
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions  https://review.openstack.org/9023319:13
devanandaadam_g: ++19:15
openstackgerritChris Krelle proposed a change to openstack/ironic: Fix for tripleO undercloud gate tests DO NOT MERGE  https://review.openstack.org/8552919:16
openstackgerritChris Krelle proposed a change to openstack/ironic: Workaround race with Neutron port updates  https://review.openstack.org/8436119:16
devanandaNobodyCam: still testing https://review.openstack.org/#/c/88307/ ?19:16
NobodyCamyes :(19:16
*** zdin0bot1 has joined #openstack-ironic19:17
*** zdin0bot has quit IRC19:17
NobodyCami forgot to save the images and overwrote them. :( so needed to rebuild19:17
NobodyCamwhich is not all that quick atm19:17
NobodyCamits runing right now fyi19:18
*** datajerk has joined #openstack-ironic19:23
*** zdin0bot1 has quit IRC19:25
*** zdin0bot has joined #openstack-ironic19:25
adam_ghttps://review.openstack.org/#/c/90452/19:28
adam_gdevananda, ^19:28
*** zdin0bot has quit IRC19:31
*** igor has quit IRC19:36
*** zdin0bot has joined #openstack-ironic19:45
*** jistr has quit IRC19:45
*** zdin0bot has quit IRC19:46
openstackgerritA change was merged to openstack/ironic: Spawn support for TaskManager and 2 locking fixes  https://review.openstack.org/8932819:46
*** harlowja is now known as harlowja_away19:49
*** shakamunyi has quit IRC19:50
openstackgerritJim Rollenhagen proposed a change to openstack/ironic-python-agent: Remove Heartbeat-Before check from heartbeat  https://review.openstack.org/9045719:57
*** harlowja_away is now known as harlowja20:09
JayFPassthru just got approved! \o/20:13
*** jdob has quit IRC20:15
*** jdob has joined #openstack-ironic20:15
*** ifarkas has quit IRC20:20
*** jdob has quit IRC20:24
*** jdob has joined #openstack-ironic20:24
openstackgerritA change was merged to openstack/ironic: Drivers may expose a top-level passthru API  https://review.openstack.org/8191920:28
*** igor has joined #openstack-ironic20:30
openstackgerritJim Rollenhagen proposed a change to openstack/ironic-python-agent: Remove Heartbeat-Before check from heartbeat  https://review.openstack.org/9045720:31
jrollGheRivero: around?20:34
*** igor has quit IRC20:34
devanandabearhands: what'd happen if a compute node is deleted in nova, while there are instance(s) on it?20:36
bearhandshm20:36
bearhands'nothing' ?20:36
bearhandsthe instances would be like orphaned, I guess20:37
bearhandsI think you can still delete them20:37
devanandabut nova wont try to delete them automatically?20:37
NobodyCamhe's utc +/- 1???20:37
devanandathat seems odd :)20:37
bearhandswell20:37
bearhandsi don't think there's anything that deletes the node in nova20:37
bearhandsi think i saw recently someone was adding an admin API call for it20:38
devanandahttp://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n545920:38
bearhandsbut20:38
GheRiverojroll: pong20:38
bearhandsyeah20:38
bearhandsso that would be for baremetal only20:38
GheRiverodevananda: congrats!20:38
bearhands(and now ironic, I guess)20:38
devanandabearhands: if eg. nova.virt.ironic.driver:get_available_nodes() returns an empty list suddenly, the rt would delete everything20:38
devanandayea20:38
devanandaGheRivero: thanks!20:38
bearhandsdevananda: right, don't do that20:39
devanandaheh20:39
jrollGheRivero: hey, wanted to let you know I just addressed your question here https://review.openstack.org/#/c/88722/ :)20:39
bearhandsthat's kind of like...20:39
bearhandsyou keep some state on your openstack instances. you write something that polls nova-api for your instances20:39
bearhandsand destroy the data if you see the instances gone20:40
jrollGheRivero: sorry I missed that before20:40
openstackgerritJay Faulkner proposed a change to openstack/ironic-python-agent: Use systemd-nspawn instead of docker for image  https://review.openstack.org/9046520:40
bearhandsand nova-api randomly returns [] ?20:40
devanandahttps://review.openstack.org/#/c/88590/20:40
devanandaNobodyCam: please take a moment to look at my comment ^20:40
bearhands(we actually had this problem pre-openstack :)20:40
GheRiverojroll: thanks. will review it soon!20:40
devanandabearhands: that's pretty much what i thought. thanks :)20:40
NobodyCamoh20:41
NobodyCamrevoving /me's vote20:41
NobodyCamremoving even20:41
bearhandsdevananda: at least here, I don't think much badness happens... the node will get recreated20:41
bearhandswhen it shows up again20:42
devanandabearhands: and the instance that was on it and associated with the previous compute node's id?20:42
openstackgerritJim Rollenhagen proposed a change to openstack/ironic-python-agent: Check configdrive size before writing to partition  https://review.openstack.org/8939020:43
bearhandsdevananda: instance would still be around.. instance is associated with host only by its name20:43
bearhandswell20:43
bearhandsinstance.host, instance.node == strings20:43
devanandaah, there's no FK or reference from instance to compute_node ?20:44
bearhandsnot that i'm aware of20:44
devanandak20:44
devanandaso anyway, it's not a good idea, but maybe not as bad as it looks20:44
bearhandsright20:44
bearhands"don't do that"... but I think nova will handle it mostly okay.20:44
bearhandswe could make nova a little smarter20:45
bearhandssay20:45
bearhandsdon't delete the node unless it's been gone for 'x' minutes.20:45
bearhandsor whatever20:45
*** openstackstatus has quit IRC20:45
devanandaor delete the node if ironic returns valid data not including that node20:45
devananda(but dont do anyhting hasty when ironic doesn't return any data eg. due to network failure)20:46
bearhandsor that, perhaps.. maybe an emptylist would be a warning that.. .maybe something else bad is going on20:46
bearhands:)20:46
*** openstackstatus has joined #openstack-ironic20:46
bearhandsi don't see much reason to delete the entry immediately20:46
devanandaraise an exception seems better to me. it'll be logged, and there shouldn't be any silent db changes as a result afaict20:47
bearhandsscheduling will not use it after x seconds20:47
bearhandsif it stops updating20:47
bearhandsalthough20:47
bearhandsi guess that may not be true with baremetal/ironic20:47
bearhandssince the nova-compute service entry would be 'up'20:47
devanandaright20:47
bearhandswe could fix that20:47
devanandaand >1 n-cpu will be reporting the same "host" name and the same set of nodes20:47
bearhandsi think there's a hypervisor_mainteannce or something20:48
* bearhands checks20:48
devanandawe're going to really need to sort the >1 n-cpu issues in atlanta20:48
bearhandshm, there's no way to label a ComputeNode entry as 'disabled'.20:48
devanandabearhands: adam_g has proposed to fake that by setting its resource counts to 020:49
devanandahttps://review.openstack.org/#/c/88476/420:49
bearhandsyeah20:49
*** zdiN0bot has joined #openstack-ironic20:49
bearhandsi thin i suggested the opposite...20:49
devanandao.020:49
bearhandsfaking its full20:49
devanandaheh20:49
bearhandswell20:49
bearhandskind of the same thing20:49
bearhands:)20:49
devanandaiirc you said "dont delete it"20:50
devanandaor someone did20:50
bearhandsdon't recall that20:50
bearhandswell20:50
*** openstackstatus has quit IRC20:50
bearhandsi might have said that, because that's true20:50
devanandabut yea, setting it to have 0 available resources (whether by current,max=0 or current=max) seems the right way20:50
devanandaand of those two, (current,max=0) seems better because it is not lying to resource trackers about total consumed resources20:51
devanandawhereas setting current=max (eg, while the node is in maintenance mode) is going to skew any resource usage calculation20:51
bearhands2014-04-17 13:06:24     <comstud>       devananda: ya, that's what I was going to say.. show them to compute manager as fully utilized20:51
bearhands2014-04-17 13:10:43     <comstud>       which is why i think you still present it.. but fully utilized (as it would be with an instance)20:51
bearhandsheh20:51
devanandabecause there's no tenant using it20:51
*** openstackstatus has joined #openstack-ironic20:51
bearhandsyeah20:51
bearhandsi agree20:51
* bearhands watches the rain come down20:52
devanandaspeaking of the weather ... it's awesomely sunny outside... i'm going for a walk shortly :)20:54
*** jgrimm has quit IRC20:54
bearhandsdevananda: sunny? where are you at?20:55
devanandaSunny Seattle!20:55
devanandalol20:55
bearhandslol i see20:55
bearhandseast sfbay here.. and it just poured... for about 60 seconds20:55
devanandaheh20:55
bearhandsor, since i'm from the midwest, I would just call it normal 'rain'20:56
bearhandsrain here to people seems to mean even a slight mist20:56
*** ChanServ sets mode: +v openstackstatus20:56
*** romcheg1 has quit IRC20:57
devanandacopying the chatlog to that review for later reference20:57
bearhandscools20:57
bearhandsi'm looking it over20:57
*** linggao has quit IRC20:58
-openstackstatus- NOTICE: Gerrit will be unavailable for a few hours starting at 1600 UTC on Monday April 28th for an upgrade. See https://wiki.openstack.org/wiki/GerritUpgrade20:59
devanandahttps://review.openstack.org/#/c/87396/ needs another +221:00
devanandaNobodyCam: you had previously looked at it, have time for another look today?21:00
* NobodyCam looking21:00
bearhandsdevananda: i'm not sure that 'get_available_nodes' should filter out NOSTATE. that could make a node 'disappear'21:04
*** matty_dubs is now known as matty_dubs|gone21:04
NobodyCamdevananda: have you tested this?21:04
NobodyCam:)21:04
bearhandsas long as get_availble_resources checks NOSTATE (like you suggest), then we won't schedule to it.21:04
bearhandsbut anyway, left that feedback21:04
devanandabearhands: NOSTATE indicates a node that was just registered with ironic whose power state has not been fetched from the BMC yet21:05
devanandabearhands: that'll either convert to a real state (ON, OFF) or ERROR after some retries21:05
bearhandsya, but say.. something/someone screws up and sets it back to NOSTATE?21:05
devanandahmm21:05
bearhandsi don't see harm in it still being presented to nova21:05
bearhandsbut with 0 resources21:06
devanandai'd say, make it unschedulable -- but dont delete the compute record21:06
devanandaright21:06
devanandathought that's what i was suggesting21:06
bearhandsright21:06
bearhandswell you did, in get_available_resources()21:06
devanandawait. is this deleting compute records for things in ERROR state?21:06
bearhandsbut it's also filtered out in get_available_nodes() right now21:06
* devananda rereads21:06
bearhandsin this patch21:06
bearhandsi think get_available_nodes() should just reutrn the full node_list we get from ironic21:06
bearhandsleave the unschedule magic completely to get_available_resources()21:07
devanandaooh right21:07
*** jbjohnso has quit IRC21:07
bearhandshttps://review.openstack.org/#/c/88476/4/ironic/nova/virt/ironic/driver.py,unified line 29121:07
devanandayea, i would remove all the state check in get_available_nodes21:07
bearhandscools, left it as feedback, just wanted to make sure i wasn't on crack21:08
devanandajust return all nodes. let the scheduler see them, even if they have 0 resources21:08
devanandaside note21:08
devanandawe could pass the node.maintenance property (and others) up as compute node properties, then add a FilterScheduler to match on it21:09
devanandaexclude things that way, instead of using the resource counts21:09
bearhandsi was looking at that21:09
devanandathere was a session proposal to add that21:09
bearhandsi commented on it yesterady or so21:09
devanandafor hardware model # or something21:09
bearhands127     # extra_resources is a json string containing arbitrary21:09
bearhands 128     # data about additional resources.21:09
bearhands 129     extra_resources = Column(Text)21:09
bearhandswas thinking that could be a place to put it21:10
NobodyCamdevananda: atm its looking like I'll be +2'ing 88307 in about 5 min21:10
bearhandsit's not really a 'resource', but.. there's not a real good other place to put it21:10
bearhandsmaybe 'stats' IS the right place (which is where resources go today)21:10
devanandaNobodyCam: awesome21:10
boris-42bearhands btw for storing json21:10
boris-42bearhands there is a interesting way in sqlalchemy21:10
devanandabearhands: stats ++21:10
boris-42bearhands something like https://github.com/stackforge/rally/blob/master/rally/db/sqlalchemy/types.py#L8921:11
bearhandsI'd almost prefer just adding a 'maintenance' column to this table21:11
bearhands:)21:11
bearhandsbut anyway21:11
boris-42bearhands so sqalclhemy will automatically do load() and dump()21:11
boris-42bearhands and track status dict() was changed21:11
boris-42bearhands to update column21:11
bearhandsboris-42: Yeah, I saw this somewhere else, also21:12
boris-42bearhands I think docs of sqlalchemy lol=)21:12
bearhandsobjects covers up most of the need for this21:12
boris-42bearhands heh I am afraid a bit about performance of such approach21:12
bearhandssince they should be your interface to dbapi, they loads/dumps and deal with stuff21:12
boris-42bearhands I mean objects21:13
bearhandswhich approach?21:13
bearhandswell21:13
bearhandsI'm afraid of using more things in sqlalchemy.. for performance21:13
bearhands:)21:13
boris-42bearhands but it's next ORM21:13
bearhandsThe json conversion has to happen somewhere21:13
boris-42bearhands ORM on ORM21:13
bearhandsI'd prefer it to happen in objects, I think21:13
bearhandssomewhat, but sqlalchemy sucks, man21:13
boris-42bearhands but we still use it =)21:14
boris-42bearhands and I don't think that sqla sucks=)21:14
bearhandsYeah, but I have an alternative to it... mysqldb direct.21:14
bearhandswhich won't have this21:14
boris-42bearhands it will21:14
bearhandsunless i implement it within my mysqldb impl21:14
boris-42bearhands cause OpenStack uses SQL in wrong way21:14
bearhandswell, i won't argue that21:14
boris-42bearhands so but these are different things21:15
bearhandsbut even using it the correct way, sqlalchemy performance sucks way worse than using mysqldb directly.21:15
boris-42bearhands okay with this agree21:15
bearhandsi have the numbers stashed away somewhere to prove it21:15
boris-42bearhands I remember21:15
boris-42=)21:15
bearhandsnod21:15
bearhandstherefore, I'd prefer to use less from sqlalchemy, not more21:15
bearhands:)21:15
boris-42bearhands so someday you are guys going to get rid of using it hardcoding direct usage of it inside objects?21:16
bearhands(and yeah, i know, objects is basically a mini-orm on top of an orm in our case :)21:16
bearhandsboris-42: I'm not sure I would go that far... We need to support more than mysql21:16
bearhandsBut I could see that 'mysql' ends up meaning to use 'mysqldb' instead of sql-a21:17
boris-42bearhands so the mini-orm on orm will be worse perfromance=)21:17
bearhandsright, but we don't have to use the ORM21:17
bearhandswe could use core21:17
bearhandsand it's really not any less efficient, actually, the way we're doing things right now21:18
bearhandsi mean, tell me how objects doing json vs sql-a doing json is different21:18
boris-42bearhands same same=)21:19
bearhandshehe21:19
boris-42bearhands https://review.openstack.org/#/c/43151/21:19
boris-42bearhands yep that could be hidden inside objects21:19
boris-42bearhands but I don't see big difference to make it in db.api21:19
bearhandsyeah i saw this patch a long time ago21:19
bearhandswe've now eliminated 'stats' being in a separate table, which solves a lot of the problems the original query had21:20
boris-42bearhands yep that makes totally sense21:20
boris-42bearhands btw there is a lot of issue21:20
boris-42bearhands with joins21:20
bearhandsbut ya, this points out where sql-a is slower than it should be21:21
boris-42bearhands like this one http://pavlovic.me/rally/glance_list.html21:21
bearhandsglance joins all of the properties21:21
boris-42bearhands in cinder & nova the same21:21
bearhandswhich are key/value table21:21
boris-42bearhands not key/value21:21
bearhandsi mean, i assume it joins21:21
boris-42bearhands key/value/created_at/deleted_at/updated_at21:21
bearhandswell, image_properties is technically name/value rows21:21
bearhandsright21:21
devanandaok, really going for a walk before the sun hides again21:21
devanandabbiab21:21
bearhandsdevananda: cools, i'm out soon also21:21
bearhandsenjoy21:21
boris-42bearhands soo21:22
bearhandsboris-42: yeah, it's extra bad21:22
boris-42bearhands yep dates are terrible21:22
boris-42bearhands in joins)21:22
boris-42bearhands so nova/cinder the same story21:22
boris-42bearhands with volumes and instances21:22
*** zdiN0bot has quit IRC21:23
NobodyCamexport DIB_REPOLOCATION_diskimage_builder=https://review.openstack.org/openstack/diskimage-builder21:23
NobodyCamgah ww21:23
bearhandsgotta run, laters all.. have a good weekend21:26
NobodyCamhave a good weekend bearhands21:26
*** jdob has quit IRC21:29
openstackgerritJay Faulkner proposed a change to openstack/ironic-python-agent: Use systemd-nspawn instead of docker for image  https://review.openstack.org/9046521:29
*** igor has joined #openstack-ironic21:30
openstackgerritAdam Gandelman proposed a change to openstack/ironic: Decouple state inspection and availability check  https://review.openstack.org/8847621:31
*** vkozhukalov has quit IRC21:32
*** igor has quit IRC21:35
lucas-dinnerdevananda, NobodyCam thanks for the reviews :D21:41
lucas-dinnerI saw that the oslo.messaging was approved as well and failed to merge I will fix it21:42
NobodyCamlucas-dinner: what are you doing up :p lol21:43
lucas-dinnerNobodyCam, lol I was watching some series21:43
lucas-dinnerwas a bit tired to go out21:44
NobodyCam:)21:44
NobodyCamI'll reapprove21:44
lucas-dinner:) cheers, just fixing the conflicts21:46
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Port to oslo.messaging  https://review.openstack.org/8830721:47
*** coolsvap|afk has quit IRC21:47
NobodyCam:)21:48
*** eguz has joined #openstack-ironic21:55
NobodyCamstepping away for a bit will re +a after mr J is done21:57
*** eghobo has quit IRC21:59
*** zdiN0bot has joined #openstack-ironic22:00
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions  https://review.openstack.org/9023322:06
openstackgerritA change was merged to openstack/ironic: nova.virt.ironic passes ephemeral_gb to ironic  https://review.openstack.org/8739622:23
devanandabearhands: if you're still around, 84823 needs a rebase. looks like it fell off my (and everyone's) queue22:23
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Adding a reference driver for the agent  https://review.openstack.org/8479522:24
openstackgerritA change was merged to openstack/ironic-python-agent: Use systemd-nspawn instead of docker for image  https://review.openstack.org/9046522:25
*** igor_ has joined #openstack-ironic22:30
devanandaShrews: 88403 also needs a rebase, but yea, that should totally land. I didn't see it before ...22:33
*** igor_ has quit IRC22:34
devanandaSpamapS: on the swap partition patch, any thoughts on whether that'll impact tripleo?22:38
SpamapSdevananda: swap shmap22:41
devanandaSpamapS: heh :) well, it changer partition order22:41
SpamapSdevananda: We might be working around bugs in block device mappings that might cause us pain. Not sure.22:41
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Sync oslo logging  https://review.openstack.org/8734622:44
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Remove 'fake' driver from default enabled drivers  https://review.openstack.org/8871122:44
*** zdiN0bot has quit IRC22:46
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions  https://review.openstack.org/9023322:59
*** zdiN0bot has joined #openstack-ironic23:00
*** tatyana has joined #openstack-ironic23:05
*** eguz has quit IRC23:22
*** eghobo has joined #openstack-ironic23:23
*** derekh has joined #openstack-ironic23:24
*** dkehn_ is now known as dkehnx23:30
*** igor has joined #openstack-ironic23:30
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Adding a reference driver for the agent  https://review.openstack.org/8479523:32
*** igor has quit IRC23:35
*** derekh has quit IRC23:52
*** shakamunyi has joined #openstack-ironic23:55
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic: Updated from global requirements  https://review.openstack.org/8923423:58
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic-python-agent: Updated from global requirements  https://review.openstack.org/8872223:58
*** mdenny has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!