Tuesday, 2015-01-06

openstackgerritJosh Gachnang proposed openstack/ironic: Improve IPA client library  https://review.openstack.org/11111800:02
*** GheRivero has quit IRC00:03
*** hemna has joined #openstack-ironic00:11
*** ryanpetrello has quit IRC00:15
*** GheRivero has joined #openstack-ironic00:15
*** andreykurilin has quit IRC00:20
openstackgerritSteven Dake proposed openstack/ironic-specs: Override boot options via glance property  https://review.openstack.org/14423500:24
*** hemna has quit IRC00:27
*** ijw has joined #openstack-ironic00:29
*** hemna has joined #openstack-ironic00:30
*** ijw has quit IRC00:30
*** ijw has joined #openstack-ironic00:31
*** ChuckC has quit IRC00:32
*** ijw has quit IRC00:33
NobodyCamok gots to drop from the internets for a few...00:33
*** Masahiro has joined #openstack-ironic00:33
*** hemna has quit IRC00:34
*** david-ly_ has joined #openstack-ironic00:36
*** Masahiro has quit IRC00:38
*** david-ly_ has quit IRC00:41
*** hemna has joined #openstack-ironic00:42
*** hemna_ has joined #openstack-ironic00:47
*** hemna has quit IRC00:48
*** zhidong has joined #openstack-ironic00:48
*** ryanpetrello has joined #openstack-ironic00:49
*** achanda has joined #openstack-ironic00:50
*** achanda has quit IRC00:55
*** achanda has joined #openstack-ironic00:56
*** Masahiro has joined #openstack-ironic00:56
*** achanda has quit IRC01:01
*** ChuckC has joined #openstack-ironic01:03
*** hemna_ has quit IRC01:06
*** hemna_ has joined #openstack-ironic01:07
*** GheRivero has quit IRC01:10
*** GheRivero has joined #openstack-ironic01:11
*** GheRivero has quit IRC01:20
*** hemna_ has quit IRC01:24
*** hemna_ has joined #openstack-ironic01:25
*** GheRivero has joined #openstack-ironic01:27
*** chenglch has joined #openstack-ironic01:29
*** nosnos has joined #openstack-ironic01:50
*** penick has joined #openstack-ironic01:53
*** penick_ has joined #openstack-ironic01:54
*** hemna_ has quit IRC01:54
*** hemna_ has joined #openstack-ironic01:55
*** penick has quit IRC01:57
*** penick_ is now known as penick01:57
*** rwsu has quit IRC02:03
*** hemna_ has quit IRC02:06
*** ChuckC has quit IRC02:07
*** hemna_ has joined #openstack-ironic02:07
*** GheRivero has quit IRC02:09
*** ryanpetrello has quit IRC02:14
*** GheRivero has joined #openstack-ironic02:15
openstackgerritRuby Loo proposed openstack/ironic-specs: Make sure the filename is the same as the blueprint name  https://review.openstack.org/14508802:15
*** ChuckC has joined #openstack-ironic02:18
*** eghobo has quit IRC02:27
openstackgerritJosh Gachnang proposed openstack/ironic: Improve IPA client library  https://review.openstack.org/11111802:30
*** yjiang5 has left #openstack-ironic02:46
*** chenglch|2 has joined #openstack-ironic02:58
*** vinbs has joined #openstack-ironic03:01
*** chenglch has quit IRC03:01
*** dlaube has quit IRC03:11
*** hemna_ has quit IRC03:15
*** pensu has joined #openstack-ironic03:21
*** jerryz has joined #openstack-ironic03:32
*** nosnos has quit IRC03:34
*** Haomeng has joined #openstack-ironic03:34
*** Haomeng|2 has quit IRC03:34
*** ryanpetrello has joined #openstack-ironic03:38
*** pensu has quit IRC03:40
*** chenglch|2 has quit IRC03:42
*** david-lyle has joined #openstack-ironic03:48
openstackgerritJay Faulkner proposed openstack/ironic-python-agent: Allow use of multiple simultaneous HW managers  https://review.openstack.org/14319303:49
*** penick has quit IRC03:51
*** Nisha has joined #openstack-ironic03:52
*** penick has joined #openstack-ironic03:54
*** ryanpetrello has quit IRC03:55
openstackgerritJay Faulkner proposed openstack/ironic-python-agent: Allow use of multiple simultaneous HW managers  https://review.openstack.org/14319303:58
*** ramineni has joined #openstack-ironic03:59
*** naohirot has joined #openstack-ironic04:04
naohirotGA ironic!04:05
*** rloo has quit IRC04:07
*** pensu has joined #openstack-ironic04:14
*** david-lyle is now known as david-lyle_afk04:25
*** nosnos has joined #openstack-ironic04:29
*** rameshg87 has joined #openstack-ironic04:30
Nisha#join openstack-meeting-304:36
*** chenglch has joined #openstack-ironic04:36
*** dlaube has joined #openstack-ironic04:36
*** pensu has quit IRC04:40
*** hshiina has joined #openstack-ironic04:43
*** penick has quit IRC04:43
*** lsmola has quit IRC04:45
*** pcrews has quit IRC04:50
devanandahi folks!04:55
devanandaalmost meeting time04:55
Haomengdevananda: ok04:59
*** sdake has quit IRC04:59
*** lsmola has joined #openstack-ironic05:00
*** lintan has joined #openstack-ironic05:01
*** eghobo has joined #openstack-ironic05:07
*** chenglch has quit IRC05:22
*** k4n0 has joined #openstack-ironic05:24
*** k4n0 has quit IRC05:25
*** rakesh_hs has joined #openstack-ironic05:26
*** eghobo has quit IRC05:26
*** pensu has joined #openstack-ironic05:27
*** pradipta_away is now known as pradipta05:29
*** eghobo has joined #openstack-ironic05:30
jrollNobodyCam: imagine how you might suspend an ironic instance for non-payment, you don't want to destroy data in case they do end up paying, right?05:55
NobodyCamjroll: https://review.openstack.org/#/c/107778/05:56
jrollNobodyCam: can you link me to somewhere I can download extra time in the day, though?05:57
jroll:P05:57
jrollI'm aware of this one05:57
jrollbut ty :)05:57
rameshg87NobodyCam, why was it dropped anyway ?06:04
devanandajroll: snapshot :)06:05
jrolldevananda: I may or may not get that joke(?)06:05
jrollor do you mean, snapshot an ironic instance06:06
jrollbecause wat06:06
devanandajroll: yes. reboot into IPA, compress data, archive to separate storage06:06
jrollhmm, that may be doable.06:06
jrollyeah06:06
jrollhow do you do that in a sane fashion with 3TB of data? :)06:06
russell_hswift06:07
devanandaalso, 107778 was abandoned because it used a conf option to ALWAYS soft-off06:07
devanandarather than per-request06:07
jrollrussell_h: I bet our swift group would love for us to push/pull 3tb at once06:07
russell_heh, they'd survive06:07
devanandajroll: imagine if someone snapshotted a bare metal swift node into swift ...06:07
jrollheh06:07
russell_hjroll: the real problem would be when we do it 30x at once06:08
devanandathere are clearly some things you shouldn't do06:08
jrollright.06:08
devanandabut, cold-migrate an instance like taht? way cool06:08
jrollsuper cool06:08
lintanjroll: I like the idea to take a snapshot of instance06:08
devanandaboot IPA on two nodes, stream the disk from one to the other, update ports and so on, reboot06:08
russell_hsnapshots are very much on our internal wish list06:08
russell_hat least of boot disks06:08
* jroll learned something today ^06:09
lintanWe also need this in our environment, yes boot disks06:09
jroll:P06:09
jrollyeah, this is super interesting06:10
* jroll wants 'nova suspend' first06:10
russell_hjroll: I wrote some thing that was archiving all the data cloud monitoring collected to swift06:10
jrollor a 'nova stop' that doesn't unplug a computer06:10
devanandajroll: nova pause ?06:10
russell_hjroll: by the time they added compression it was uploading a 500MB JSON file every few seconds06:11
jrolldevananda: maybe06:11
jrolldevananda: 'a way to cut off instance access without a hard power off'06:11
russell_hunfortunately for my street cred it turns out JSON compresses really well06:11
jrollwe have a hack for this today but it isn't awesome06:11
jroll(kill the ports)06:11
devanandajroll: hm. that's just "dom.suspend()"06:12
devanandaprobably not the right thing06:12
jrolldevananda: eh? ironic doesn't support anything like this06:12
devanandaalso, i have no idea how to do that generically for hardware06:12
jrollwell, soft power off would be a start06:12
devanandanope06:12
devanandathat's TOTALLY different06:12
jrolldunno how generic that would be, AMT probably doesn't support it06:13
jrollwhy06:13
devanandayou get all kinds of local state changes when you ask an OS to shut down06:13
russell_hnutty question, can an iLO dump the contents of RAM?06:13
jrollnova stop && nova lock06:13
jrollsure06:13
jrollbut like06:13
jrollyou can't suspend hardware.06:13
russell_h(or any other high-end-ish BMC)06:13
jrollit's the closest you can get imo06:13
jroll(with any hardware I'm aware of, my knowledge is not vast)06:14
devanandarussell_h: sure, but you can feel safe because it doesn't have a documented API for that06:14
russell_hlmao06:14
lintanmaybe we should not think about the snapshot of memory at first?06:14
devanandarussell_h: sorry, it's late, my humor is showing :)06:14
jrollread: drinks are poured06:14
russell_hdevananda: its cool, I expect the same is true of Intel AMT crap06:14
russell_hwhich is even safer because its barely documented at all06:15
jrollbecause it's for... desktops.06:15
jrollat any rate06:15
devanandalaptops too06:15
jrollI think soft power off is a good first step06:15
jrolldevananda: !servers, I should say06:16
devanandajroll: I agree. nothing wrong with supporting soft power off in a generic (and optional) way06:16
devanandakeeping in mind that some power managers may not implement it06:16
devanandaeg, a PDU06:16
jrollyeah06:16
devanandaso calls to eg. "ironic node-set-power-state --soft off" need to error appropriately in that case06:17
jrollI'd also love 'nova stop' to default to stop and handle that error appropriately06:17
jrolldefault to soft*06:18
devanandaalso: ssh $host "sudo shutdown now"06:18
devanandajroll: with appropriate timeouts and falling back to --hard, yes, we should do that06:18
jrolloh, have you heard our funny story about "sudo shutdown"?06:18
devanandanope06:18
jrollso nova has a loop that checks power status for instances, right?06:19
jroll(keep in mind we do force_power_state_sync=false06:19
jrollso someone shuts an instance down, ironic notices and records power_state=off06:19
jrollnova sees that and decides it needs to call the stop() api06:19
jrollwhich calls virt_driver.stop()06:20
jrollso then it's stopped in nova, powered off in ironic06:20
jrollnova start should work fine in this case, but rackspace doesn't expose that06:21
jrollso theoretically you're stuck, but turns out nova reboot will turn it back on06:21
jrollbut it took some time to figure that out06:21
jrollI guess it's funnier if you dealt with the wtf :P06:22
devanandaoh, why don't you expose nova start? :p06:23
russell_hstill trying to figure that out06:23
jrolldon't ask me06:23
russell_hI'm not convinced anyone knows06:23
devanandathat I can evan ask that question makes me really sad06:23
jrollI live in a cell, I don't own any nova-api nodes :P06:23
russell_hmy guess: since stop isn't exposed it never occured to anyone to expose start06:23
jrollrussell_h: that just transforms the question :P06:24
*** killer_prince is now known as lazy_prince06:24
russell_hyeah but thats an easier question06:24
russell_hmy guess on that one: people thought stopping a server meant they don't get billed06:24
jrolltrue06:25
russell_h5 hours until SpaceX launch06:25
jrollapi_response = 'you will still be billed' + api_response06:25
russell_hwho is pumped06:25
jrollheh, staying up for that?06:25
russell_hnot sure yet06:25
jrollor waking up early?06:25
russell_hI plan to be awake somehow06:25
jrollgotta be ready to trade06:25
*** rameshg87 has quit IRC06:26
russell_hSpaceX is private :(06:26
jrolloh.06:26
jrollbut elon musk confidence changes must affect tesla amirit06:26
*** rameshg87 has joined #openstack-ironic06:26
russell_hprobably06:26
russell_hElon lost me a lot of money today06:26
russell_hI mean it wasn't his fault06:27
russell_hbut between SCTY and TSLA06:27
russell_hand everything else06:27
russell_hrough day06:27
jrollthat's what you get for going all in on TSLA06:27
jroll:P06:27
*** chenglch has joined #openstack-ironic06:28
* jroll signs off for the night, see everyone tomorrow06:29
*** jiangfei has quit IRC06:41
*** jiangfei has joined #openstack-ironic06:42
jiangfeican ironic deploy in docker, someone kown about it ?06:56
Haomengjiangfei: good idea, I think the points is networking06:57
jiangfeiHaomeng: hm,  do you mean neutron-dhcp-server06:59
Haomengjiangfei: yes, not sure if our ovs and neutron agents can be run in docker06:59
Haomengjiangfei: I mean the docker networking env, if the os running in docker can receive the physical traffic, such as the baremetal dhcp discovery request07:00
jiangfeiHaomeng:  i see some news about: 'The docker driver does not work with neutron'07:01
Haomengjiangfei: yes07:01
Haomengjiangfei: so we just put ironic api and conductor into docker, that is not very useful:)07:02
Haomengjiangfei: but it is easy for installation07:02
jiangfeiHaomeng:yes07:03
*** dlaube has quit IRC07:25
*** ifarkas has joined #openstack-ironic07:26
*** chlong has quit IRC07:49
*** chlong has joined #openstack-ironic07:49
*** chlong has quit IRC07:51
*** chlong has joined #openstack-ironic07:53
*** teju has joined #openstack-ironic07:58
*** vinbs_ has joined #openstack-ironic08:01
*** vinbs has quit IRC08:02
*** vinbs_ is now known as vinbs08:02
*** Nisha has quit IRC08:31
*** Nisha has joined #openstack-ironic08:38
*** eghobo has quit IRC08:43
*** hshiina has left #openstack-ironic08:48
*** jistr has joined #openstack-ironic08:58
*** jiangfei has quit IRC08:59
*** jiangfei has joined #openstack-ironic08:59
*** dtantsur|afk is now known as dtantsur09:00
*** jcoufal has joined #openstack-ironic09:03
*** jcoufal has quit IRC09:03
dtantsurMorning09:04
*** derekh has joined #openstack-ironic09:08
*** erwan_taf has joined #openstack-ironic09:09
*** athomas has joined #openstack-ironic09:12
*** zhidong has quit IRC09:21
*** Masahiro has quit IRC09:27
*** Masahiro has joined #openstack-ironic09:27
*** Masahiro has quit IRC09:31
*** Masahiro has joined #openstack-ironic09:37
openstackgerritRamakrishnan G proposed openstack/ironic-specs: New driver interface for RAID configuration  https://review.openstack.org/13589909:38
*** Masahiro has quit IRC09:39
Nishadtantsur, morning09:39
NishaHappy new year09:39
dtantsurNisha, hi! Happy new year (if applies :)09:40
Nishadtantsur,  :)09:41
Nishadtantsur, hope you had wonderful vacations09:41
erwan_tafhey dtantsur09:41
dtantsuryeah, pretty awesome :)09:42
dtantsurerwan_taf, o/09:42
dtantsurNisha, and you?09:42
Nishadtantsur, yes me too09:42
Nishadtantsur, i saw your +2 on the discovery spec but it is not yet approved.... any reason?09:43
Nishanow it has 2 +209:43
Nishaand 2 +109:43
Nishadtantsur,  and no comments so far09:43
dtantsurNisha, just wanted other folks to have a look. I'm afraid I've influenced it too much :)09:43
Nishahehe09:43
dtantsurbut as nobody jumped in, I think it's fine for me to approve it now09:44
Nisha:)09:44
dtantsur+A09:46
Nishadtantsur, thanks09:46
dtantsurNisha, thank you for all your efforts :)09:47
*** GheRivero has quit IRC09:48
Nishadtantsur,  thanks for all the help on this09:48
Nishajust hope code also goes in as soon as i post it09:48
Nishai think i need to wait for states code to get merged frst?09:48
*** GheRivero has joined #openstack-ironic09:48
*** Masahiro has joined #openstack-ironic09:50
openstackgerritMerged openstack/ironic-specs: Discover node properties using new target inspect with node-set-provision-state  https://review.openstack.org/10095109:52
*** Masahiro has quit IRC09:53
*** andreykurilin has joined #openstack-ironic09:53
*** GheRivero has quit IRC09:53
*** GheRivero has joined #openstack-ironic09:53
dtantsurNisha, yeah, we should. anyway I think you get propose some peaces already.09:57
*** andreykurilin has quit IRC09:57
*** Masahiro has joined #openstack-ironic09:57
dtantsur* get = can09:57
*** andreykurilin has joined #openstack-ironic09:58
dtantsur* peace = piece09:58
* dtantsur gets more coffee09:58
Nishadtantsur, :)09:58
Nishawill do09:58
Nishathanks09:58
Nishaits merged09:58
*** Masahiro has quit IRC10:00
*** alexpilotti has joined #openstack-ironic10:02
*** vinbs has quit IRC10:04
*** jcoufal has joined #openstack-ironic10:04
*** Masahiro has joined #openstack-ironic10:09
*** vinbs has joined #openstack-ironic10:10
*** Masahiro has quit IRC10:14
*** vdrok has joined #openstack-ironic10:15
*** pradipta is now known as pradipta_away10:15
*** Nisha has quit IRC10:19
*** GheRivero has quit IRC10:20
*** naohirot has quit IRC10:26
*** igordcard has joined #openstack-ironic10:29
*** igordcard has quit IRC10:29
openstackgerritDmitry Tantsur proposed stackforge/ironic-discoverd: Store introspection result in the local database  https://review.openstack.org/14500810:30
dtantsurifarkas, ^^^10:30
*** GheRivero has joined #openstack-ironic10:31
*** vinbs has quit IRC10:31
openstackgerritDmitry Tantsur proposed stackforge/ironic-discoverd: Introduce __version__ and __version_info__  https://review.openstack.org/14519210:34
gilliardHi Can someone help me understand what I ought to do about https://review.openstack.org/#/c/142782/ ?10:42
ifarkasdtantsur, will take a look in that10:43
gilliardI'm planning on splitting up the get_image_info method into two, for prepare() and clean_up() - but that won't fix the related issue raised at https://bugs.launchpad.net/ironic/+bug/140544810:44
*** chenglch has quit IRC10:46
gilliardTo fix that case, I think we need to know whether we're doing a deploy or a rebuild. But both cases are handled by the same code path.10:46
*** GheRivero has quit IRC10:46
dtantsurgilliard, I think these 2 bugs/cases can be handled separately...10:47
*** GheRivero has joined #openstack-ironic10:47
gilliarddtantsur: The bugs are marked as being duplicates, which I sort-of agree with. Although they describe separate problems.10:48
gilliard2 patches with a partial-bug each?10:48
dtantsuryep10:48
gilliardOK. Thanks.10:49
openstackgerritDmitry Tantsur proposed openstack/ironic-specs: In-band hardware properties inspection via ironic-discoverd  https://review.openstack.org/13560510:50
gilliardI'm still not sure what approach to take with the 2nd one though. The image cache needs to be told whether it needs to flush the kernel/ramdisk (rebuild) or if it can reuse them10:51
*** Masahiro has joined #openstack-ironic10:54
openstackgerritMerged stackforge/ironic-discoverd: Introduce __version__ and __version_info__  https://review.openstack.org/14519210:56
*** Masahiro has quit IRC10:58
*** ramineni has quit IRC11:00
*** Masahiro has joined #openstack-ironic11:00
*** pelix has joined #openstack-ironic11:04
*** lucasagomes has joined #openstack-ironic11:05
lucasagomesmorning11:18
lucasagomesjust to have an idea, who is willing to go to the mid-cycle meetup in grenoble?11:18
*** rameshg87 has quit IRC11:22
dtantsurlucasagomes, I'd like to, but I doubt I'll be sent :)11:23
*** GheRivero has quit IRC11:24
*** GheRivero has joined #openstack-ironic11:24
lucasagomesdtantsur, ack, I'm asking to ask devananda to try to find out the number of participants11:25
*** naohirot has joined #openstack-ironic11:27
*** bradjones has quit IRC11:28
*** bradjones has joined #openstack-ironic11:30
*** andreykurilin has quit IRC11:31
openstackgerritNaohiro Tamura proposed openstack/ironic: Add iRMC Driver and its iRMC Power module  https://review.openstack.org/14490111:43
*** pensu has quit IRC11:45
*** Masahiro has quit IRC11:48
* dtantsur short walk, brb11:53
openstackgerritMerged stackforge/ironic-discoverd: Store introspection result in the local database  https://review.openstack.org/14500811:57
*** pradipta_away is now known as pradipta12:03
openstackgerritNaohiro Tamura proposed openstack/ironic-specs: iRMC Virtual Media Deploy Driver for Ironic  https://review.openstack.org/13486512:26
openstackgerritNaohiro Tamura proposed openstack/ironic-specs: iRMC Management Driver for Ironic  https://review.openstack.org/13602012:35
*** ryanpetrello has joined #openstack-ironic12:45
*** Masahiro has joined #openstack-ironic12:49
*** dprince has joined #openstack-ironic12:51
*** Masahiro has quit IRC12:54
*** teju has left #openstack-ironic12:59
*** pensu has joined #openstack-ironic13:04
*** rakesh_hs has quit IRC13:06
dtantsurlucasagomes, heh, didn't notice that I have to be back on Feb 6th for DevConf here in Brno to talk about Ironic :) midcycle is Feb 3-5, right?13:10
lucasagomesdtantsur, yes, mid-cycle is 3-513:11
lucasagomesdevconf cool!13:11
dtantsurlucasagomes, ask mgmt to bring you for devconf, let's have midcycle afterparty ;)13:12
*** pradipta is now known as pradipta_away13:16
lucasagomesdtantsur, that would be awesome! I would love to13:16
lucasagomesI will see if I can go, it's pretty close :)13:17
*** vdrok has quit IRC13:21
*** athomas has quit IRC13:22
openstackgerritMatthew Gilliard proposed openstack/ironic: Check glance for new kernel and ramdisk on rebuild  https://review.openstack.org/14278213:24
openstackgerritMatthew Gilliard proposed openstack/ironic: Check glance for new kernel and ramdisk on prepare  https://review.openstack.org/14278213:24
openstackgerritLucas Alvares Gomes proposed openstack/ironic: PXE/ISCSI: Call partprobe after partioning the disk  https://review.openstack.org/14283813:38
*** athomas has joined #openstack-ironic13:39
*** jjohnson2 has joined #openstack-ironic13:39
openstackgerritNisha Agarwal proposed openstack/ironic-specs: Discover node properties for iLO drivers  https://review.openstack.org/10300713:52
*** Nisha has joined #openstack-ironic13:54
*** rloo has joined #openstack-ironic13:56
*** Masahiro has joined #openstack-ironic14:00
*** pensu has quit IRC14:03
*** nosnos has quit IRC14:04
*** Masahiro has quit IRC14:05
*** naohirot has quit IRC14:11
openstackgerritDmitry Tantsur proposed stackforge/ironic-discoverd: Make database a required configuration option  https://review.openstack.org/14523114:20
dtantsurlucasagomes, do you want me to update https://review.openstack.org/#/c/144769 for you (if you agree with comments)?14:22
*** kragniz has quit IRC14:22
* lucasagomes checks14:22
lucasagomesdtantsur, ah wait, because we might not need it14:22
dtantsurack14:22
*** kragniz has joined #openstack-ironic14:22
*** lazy_prince is now known as killer_prince14:28
*** vdrok has joined #openstack-ironic14:30
*** GheRivero has quit IRC14:42
*** GheRivero has joined #openstack-ironic14:49
*** vipul has quit IRC14:52
*** vipul has joined #openstack-ironic14:52
*** jistr has quit IRC14:56
*** jistr has joined #openstack-ironic14:57
*** hypnosb has quit IRC14:59
*** hypnosb has joined #openstack-ironic14:59
*** krtaylor has quit IRC15:03
NobodyCammorning Ironic15:04
*** zz_jgrimm is now known as jgrimm15:05
*** krtaylor has joined #openstack-ironic15:11
NobodyCamdo we have a way to force a images download. I has some folks that are replacing the image with a new image but seems like the orginal image is cached and not downloading the new image15:20
*** rameshg87 has joined #openstack-ironic15:21
lucasagomesNobodyCam, morning15:29
lucasagomesNobodyCam, the final user image? or ramdisks?15:30
lucasagomesnot sure if we actually look whether the image have been updated or not15:30
lucasagomesmaybe worth looking at the image_cache.py module15:30
* lucasagomes 1 sec, going for lunch with my fam15:30
lucasagomesbbl15:30
*** lucasagomes is now known as lucas-hungry15:30
*** MattMan has quit IRC15:31
*** MattMan has joined #openstack-ironic15:31
NobodyCammorning lucas-hungry15:32
jrollmorning all :)15:34
jrollNobodyCam: don't we cache the image by glance uuid?15:35
jrollafaik you can't change an image and keep the same uuid15:35
jrollvia glance api anyway15:35
gilliardWe cache by <node-uuid>/<label> where label is 'disk' or 'kernel' etc15:40
gilliardhttps://github.com/openstack/ironic/blob/master/ironic/drivers/modules/image_cache.py#L7815:40
gilliardSo if the node uuid doesn't change, the cache  doesn't seem to download new images. EG during nova rebuild.15:41
NobodyCamgilliard: morning jroll15:42
NobodyCamgah15:42
gilliardmorning :)15:42
NobodyCamjroll: morning15:42
NobodyCamgilliard: morning too15:42
jrollgilliard: it seems to use the glance id15:42
jrollhttps://github.com/openstack/ironic/blob/master/ironic/drivers/modules/image_cache.py#L101-10215:42
jrollmaster_file_name is the glance uuid there15:43
gilliardyes, but we return early if the dest_path exists a few lines later.15:43
jrollor at least it caches images by uuid, then links as you say15:43
*** jcoufal_ has joined #openstack-ironic15:44
jrollmmm.15:44
jrollnice one15:44
jrollis there a bug filed for this?15:44
NobodyCamahh so unlinking would allow it wee the new uuid?15:44
jrollwe should check if that path exists *and* links to the correct image15:44
Nishahi, i am facing the issue spoken of in http://lists.openstack.org/pipermail/openstack-dev/2014-September/046094.html on 14.04 Ubuntu also now with latest devstack15:45
Nishadoes anyone know the workaround?15:45
NobodyCammorning Nisha15:45
jrollif someone writes up a bug, I'm going to throw up a quick patch15:45
NishaNobodyCam, morning15:45
NobodyCam++ sudo ovs-vsctl -- --if-exists del-port15:46
NobodyCamovs-tap1 -- add-port br-int ovs-tap1 tag=15:46
Nishayes15:46
NobodyCamhummm15:46
*** jcoufal has quit IRC15:47
Nishai just updated the devstack15:47
Nishaand it is failing at this point15:47
ShrewsNisha: 12.04?15:48
Nishano 14.0415:48
Shrewsoh, hrm15:48
*** rwsu has joined #openstack-ironic15:48
ShrewsNisha: did you try recloning everything?15:48
Nishayes15:48
Nishai removed old stack15:48
Nishai cloned devstack freshly and ran stack15:48
*** Masahiro has joined #openstack-ironic15:49
rameshg87JayF, hi15:50
Nishaactually issue is happening here15:50
openstackgerritMatthew Gilliard proposed openstack/ironic: Check glance for new kernel and ramdisk on prepare  https://review.openstack.org/14278215:50
Nisha2015-01-06 14:09:46.265 | +++ sudo ip netns exec qdhcp-6a65f545-1d41-4dab-8a12-1702965c0d72 ip link list15:50
Nisha2015-01-06 14:09:46.297 | Cannot open network namespace "qdhcp-6a65f545-1d41-4dab-8a12-1702965c0d72": No such file or directory15:50
openstackgerritJim Rollenhagen proposed openstack/ironic: Check if image link points to the correct image  https://review.openstack.org/14525415:50
gilliardjroll: yes there's a bug https://bugs.launchpad.net/ironic/+bug/140150115:51
*** subscope has quit IRC15:51
jrollgilliard: I would think that patch I just made should fix this, but I haven't tested15:51
jrollthough I see you're doing it, and very differently15:52
*** BertieFulton has joined #openstack-ironic15:52
rlooNobodyCam, gilliard, jroll: is this bug related? https://bugs.launchpad.net/bugs/140544815:52
jrollrloo: "This bug report is a duplicate of:  Bug #1401501: rebuild with new image does not update kernel and ramdisk images."15:52
jrollI'd say yes ;)15:53
rloojroll: ha ha. I didn't actually read it all.15:53
NobodyCamNisha: looks like some thing is not set: that error is comming from https://github.com/openstack-dev/devstack/blob/master/lib/ironic#L46915:53
NobodyCamthat should be set up on line 45915:54
*** Masahiro has quit IRC15:54
jrollNobodyCam: now I'm wondering if this isn't sleeping long enough or something https://github.com/openstack-dev/devstack/blob/master/lib/ironic#L45515:55
jrolleither that or ovs-vsctl changed the output format15:55
jrollgilliard: at any rate, are you willing to give my patch a test?15:56
jrollhttps://review.openstack.org/14525415:56
openstackgerritMerged openstack/ironic-python-agent: Skip failing execute tests when using /tmp noexec  https://review.openstack.org/14429415:57
NobodyCambrb15:57
gilliardjroll: Oh certainly. You mean https://review.openstack.org/14525415:58
jrollyes15:58
gilliardwill do15:58
jrollwithout going too deep, I *think* that should fix it15:58
jrollI can also test it later on sometime15:58
gilliardMy patch should be a partial-fix as well.16:00
gilliardI don't think they conflict though. I'm not 100% sure that they're duplicates really.16:00
jrollthe bugs or the patches?16:01
gilliardI don't think the 2 patches conflict, and I don't think the bugs are duplicates.16:02
gilliard2 different problems, we have to fix both before nove rebuild will work.16:02
gilliardmy patch is https://review.openstack.org/14278216:03
gilliard BTW16:03
jrollthey look like the same bug16:03
jrollright, I see it16:03
gilliardFirstly, we didn't check if the kernel or ramdisj had changed for an image. Secondly, we didn't update the cache, even if we did check.16:04
gilliard2 sides of the same bug :)16:04
jrollohhhhhhhh16:04
jrollI see :)16:05
gilliardwas I talking nonsense? I am having coffee-flow problems.16:05
jrollsame :P16:05
jrollgilliard: left a quick note there16:07
jrolldidn't do a full review, though16:07
NobodyCamback16:09
NishaNobodyCam, i re-checked everything. IMO, i have set everything16:10
Nishai have tried it 6-7 times already and it is failing at same point16:10
gilliardjroll: saw, thanks16:10
jrollNisha: it must be something with your environment as the gate is still working and uses the same code :)16:11
NobodyCamwow lag16:11
jrollNisha: I'd recommend just grabbing a fresh install of ubuntu and starting again16:11
NobodyCamNisha: can you run "sudo ovs-vsctl show |grep ${tapdev} -A1 -m1 | grep tag | cut -d':' -f2 | cut -b2-" from th command line?16:11
rameshg87gilliard, jroll, i have a question regarding https://review.openstack.org/#/c/142782/416:12
NishaNobodyCam, that line is after the failing command line16:12
jrollNobodyCam:16:12
jroll15:50:22           Nisha | 2015-01-06 14:09:46.265 | +++ sudo ip netns exec qdhcp-6a65f545-1d41-4dab-8a12-1702965c0d72 ip link list16:12
jroll15:50:22           Nisha | 2015-01-06 14:09:46.297 | Cannot open network namespace "qdhcp-6a65f545-1d41-4dab-8a12-1702965c0d72": No such file or directory16:12
rameshg87gilliard, jroll, why don't we just clear off instance_info['kernel'] and instance_info['ramdisk'] before doing a rebuild ?16:12
rameshg87gilliard, jroll, looks to me that should solve the problem16:12
Nishathe command line failing needs to be passed to make next line work16:13
NobodyCamahh16:13
jrollrameshg87: that's a great point16:13
*** mikedillion has joined #openstack-ironic16:13
gilliardrameshg87: how do we tell if we're doing a rebuild?16:13
rameshg87jroll, and if we are lucky, we should still hit the image_cache even after clearing the instance_info16:14
*** mikedillion has quit IRC16:14
jrollgilliard: just always clear it when building16:14
rameshg87gilliard, i guess here https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L95416:14
jrollgilliard: (if it's there)16:14
jrollif it's a new build, it won't be there and thus won't hurt16:14
*** jistr has quit IRC16:15
*** jistr has joined #openstack-ironic16:16
NishaNobodyCam, at gate is the devstack updated>16:16
*** jistr has quit IRC16:17
*** jistr has joined #openstack-ironic16:17
jrollNisha: yes16:17
NobodyCamNisha: ya16:17
*** erwan_taf has quit IRC16:18
Nisha:(16:18
jrollNisha: really, wipe the machine and run devstack with a fresh install of ubuntu16:19
jrollsometimes weird things get in there that break things16:19
jrollthe gate runs everything on a fresh install partially for this reason16:19
Nishajroll, ok16:19
Nishatrying one last time :(16:19
jrollNisha: are you using RECLONE=true?16:20
Nishayes i set it because it was not happening16:20
jrollok16:20
Nishajroll, should we use it always or sometimes?16:21
Nishameans set it to true always or occasionally16:21
jrollyou should use it when you want to update dependencies like neutron16:21
jrollso, periodically should be fine16:21
Nishak16:21
vdrokrameshg87, kernel and ramdisk in instance_info are used now here https://review.openstack.org/#/c/136741/16:24
vdrokhi all16:24
gilliardhi vdrok16:24
vdrokhi gilliard !16:24
rameshg87vdrok, hi16:25
vdrokrameshg87, so if they're cleared on rebuild i'll have save them to some other place16:25
jrollvdrok: where do they come from there? the API?16:26
rameshg87vdrok, hmm .. that's true ..16:26
NobodyCammorning vdrok :)16:26
vdrokjroll, hi, they should be specified in instance_info when using not glance image16:26
jrollright, ok16:26
vdrokmorning NobodyCam :)16:26
jrollmaybe they should have glance:// prepended if from glance16:26
jrolland then if they start with glance://, delete and re-fetch16:27
gilliardWherever else they come from might have the same issue.16:27
vdrokyup, glance:// or no prefix if its just uuid16:27
rameshg87vdrok, but how do you relate your use-case with rebuild ?16:27
rameshg87vdrok, your use-case of non-glance image reference doesn't consider nova16:28
jrollrameshg87: on rebuild, the user would change instance_info then issue rebuild16:28
vdrokrameshg87, yes, as i understood its part of the effort to make ironic more standalone16:28
rameshg87vdrok, i mean today you cannot give any thing other than glance image from nova16:28
vdrokrameshg87, yup16:29
rameshg87jroll, yeah i agree16:29
rameshg87jroll, vdrok, but my point was change nova rebuild to clear off instance_info['kernel16:29
jrollthe point is to (help) not use nova :P16:29
jrolloh16:29
jrollohhh.16:29
vdrokah, right16:29
rameshg87jroll, vdrok, it doesn't affect your use-case16:30
vdrokrameshg87, sorry, didn't get it :)16:30
jrollrameshg87: but you can rebuild without nova16:30
jrollbut yeah that's a good start16:30
vdrokrameshg87, understand now16:30
rameshg87jroll, but why would you rebuild without nova ? :)16:30
jrollrameshg87: why not, maybe somebody wrote a lightweight nova sort of thing16:30
jrolland uses the api for build/rebuild16:30
rameshg87jroll, hmm okay16:31
jrollit is possible :)16:31
rameshg87jroll, :)16:31
rameshg87vdrok, and i did give a try of your code with ilo driver16:32
rameshg87vdrok, i faced one issue which i have commented16:33
rameshg87vdrok, fixing that works perfectly fine :)16:33
vdrokrameshg87, yup, saw your comments16:34
rameshg87vdrok, and i had one question16:34
vdrokrameshg87, i'm on vacation now, will  update next monday :)16:34
vdrokrameshg87, sure16:34
rameshg87vdrok, sure :) have a happy vacation then ..16:34
rameshg87vdrok, how do you see file://xxx working with multiple conductors ?16:34
vdrokrameshg87, thanks :)16:34
vdrokrameshg87, i thought about that16:35
rameshg87vdrok, are we expecting that the same file be present on all the conductors ?16:35
vdrokrameshg87, but it seems to be out of ironic scope16:35
vdrokrameshg87, you may set up some share16:35
vdrokrameshg87, if you don't then it should be on every conductor16:35
rameshg87vdrok, hmm okay ..16:36
*** BertieFulton has left #openstack-ironic16:36
rameshg87jroll, JayF talked about hardware manager today in the meeting16:36
rameshg87jroll, wanted to talk to you or JayF regarding that16:36
NobodyCambrb16:37
jrollrameshg87: go ahead, JayF isn't online yet but can read back16:37
rameshg87jroll, how do we actually put another hardware manager into agent16:38
rameshg87jroll, is there some mechanism available for that ?16:38
*** dlaube has joined #openstack-ironic16:38
rameshg87jroll, looking at code i see we recognise all hardware manager using stevedore16:38
jrollrameshg87: use entry points16:39
* jroll finds an example16:39
jrollrameshg87: https://github.com/rackerlabs/onmetal-ironic-hardware-manager/blob/master/setup.cfg#L18-2016:39
jrollrameshg87: and then just pip install that when building the agent16:39
*** pcrews has joined #openstack-ironic16:40
rameshg87jroll, ah okay .. i get it now16:41
Nishareviewers, https://review.openstack.org/#/c/103007/2616:41
jrollrameshg87: you could even use that repo as a template for your manager16:41
rameshg87jroll, i can build a new hardware manager and add an entry point in the same namespace16:41
jrollyep :)16:42
jrollrameshg87: and you could write your own dockerfile that inherits from the upstream dockerfile16:42
jrolland then just build that16:42
rameshg87jroll, would be much more easy if i use dib to build :)16:43
rameshg87jroll, just add one more element and build it with ironic-agent16:43
jrollrameshg87: I guess16:43
jrollthat will also require at least 4gb of ram16:44
rameshg87jroll, yeah 3gb i guess16:44
gilliardThanks for the discussion about image caches. I'm out now but will put up new patchset tomorrow.16:44
jrollrameshg87: right... docker is straightforward too16:45
jrollrameshg87: here's a simplified version of our downstream dockerfile: https://gist.github.com/jimrollenhagen/5f0d0369e06acd5f3aa816:45
rameshg87jroll, looks even simpler infact :)16:46
jrollthe "FROM upstream" inherits from the dockerfile in tree16:46
rameshg87jroll, okay16:47
rameshg87jroll, but what if someone has a hetergenous environment16:47
rameshg87jroll, so they will have one ramdisk built with all the hardware managers for his environment16:47
jrollrameshg87: it's designed to allow that16:48
rameshg87jroll, but i see it looking for next hardware manager only if it raises NotImplementedError16:48
rameshg87jroll, lines 391-399 https://review.openstack.org/#/c/143193/13/ironic_python_agent/hardware.py16:49
jrollrameshg87: right, per function16:49
jrollso let's take erase_device for example16:49
jrollwhen you call erase_device, it will go through each hw manager until one is found that does not raise NotImplementedError16:49
jrollso if you had a "fancy pci-e flash drive" hardware manager16:50
jrollit might check if the device to be erased is a fancy pci-e flash drive16:50
jrollif not, raise NotImplementedError16:50
jrollif so, erase it and return16:50
rameshg87jroll, yeah i got your point16:50
rameshg87jroll, but may be NotImplementedError doesn't seem to be right word  :)16:50
jrollrameshg87: put that feedback on the review then :)16:51
rameshg87jroll, sure :)16:51
jrollperhaps another exception we can catch as well16:51
rameshg87jroll, yeah16:51
*** jerryz has quit IRC16:51
rameshg87jroll, so if i had to add a raid configuration method there16:51
rameshg87jroll, i would add it to HardwareManager class16:52
jrollrameshg87: it would make sense for that hw manager to raise NotImplementedError for something like 'get_cpus'16:52
jrollright16:52
rameshg87jroll, have an abstract metho raise NotImplementedError16:52
rameshg87jroll, and add my own hardware manager which implements it and then bundle it along and create a new agent ramdisk16:52
rameshg87jroll, correct ?16:53
jrollyep :)16:53
rameshg87jroll, great :)16:53
rameshg87jroll, that serves my use-case then16:53
jrollgreat :)16:53
rameshg87jroll, thanks i will go through the code tomorrow and check ..16:53
jrollrameshg87: cool, good luck :)16:54
* rameshg87 thinks it is time to go to sleep16:54
rameshg87goodnight ironic :)16:54
*** rameshg87 has quit IRC16:54
jrollnight16:54
NobodyCambrb16:55
*** lucas-hungry is now known as lucasagomes16:59
JayFjroll: thanks for fielding that ? for me :)17:00
jrollnp17:00
*** ifarkas has quit IRC17:04
*** eghobo has joined #openstack-ironic17:06
*** vdrok is now known as vdrok_afk17:09
*** mikedillion has joined #openstack-ironic17:13
*** pensu has joined #openstack-ironic17:15
NobodyCammorning JayF :)17:15
JayFmorning17:15
*** jcoufal_ has quit IRC17:18
devanandamorning, all17:20
jrollhiya devananda17:21
*** jistr has quit IRC17:22
NobodyCammorning devananda17:23
lucasagomesdevananda, jroll, JayF  morning17:26
JayFmorning17:26
jroll\o lucas17:26
*** mikedillion has quit IRC17:29
NobodyCamdevananda: have you had a chance to look over rloo's comment(s) on https://review.openstack.org/#/c/139217?17:30
*** MattMan has left #openstack-ironic17:38
*** Masahiro has joined #openstack-ironic17:38
*** Masahiro has quit IRC17:43
*** Nisha has quit IRC17:46
*** athomas has quit IRC17:51
*** ijw has joined #openstack-ironic17:53
* NobodyCam makes a fresh pot of coffee17:57
*** penick has joined #openstack-ironic17:58
*** eghobo has quit IRC18:01
*** derekh has quit IRC18:04
openstackgerritRuby Loo proposed openstack/ironic-specs: Check that filename is same as blueprint name  https://review.openstack.org/14508818:09
openstackgerritAlex Weeks proposed openstack/ironic-specs: Add pluggable metrics backend for Ironic and IPA  https://review.openstack.org/13717118:12
aweeksNobodyCam: ^ I think I addressed your comments18:13
aweeksAnd, if any non-Rackspace people could take a look at https://review.openstack.org/#/c/137171/ it would be very much appreciated18:13
NobodyCamaweeks: :) will take a look in a few18:14
aweeksthanks18:14
*** ijw has quit IRC18:23
NobodyCamaweeks: have you already started on the shared metrics reporting library?18:24
aweeksNobodyCam: yes, currently its here: https://github.com/rackerlabs/metricslogger/pull/1/files18:25
JayFNobodyCam: It's running in our production environment, although not split out fully yet :)18:25
aweeksI'm hoping to push another big update to the PR today18:25
JayFoh wow you got further splitting that out than I realized18:25
JayFnice18:26
aweeksJayF: yeah, refactoring the config functions to be more sane, and the tests18:26
NobodyCamgah whats up with the lag today :(18:26
*** jistr has joined #openstack-ironic18:30
*** dtantsur is now known as dtantsur|afk18:31
*** jistr has quit IRC18:31
*** jistr has joined #openstack-ironic18:32
mjturekhey does anyone know a good guide for provisioning physical hardware using devstack with ironic enabled? Been stuck on getting neutrondhcp/tftp working with my external node18:37
mjturekneutron's *18:37
Shrewsmjturek: there was a devstack patch up to make that easier, but I don't have a link handy. Dealing with a dead laptop right now... :(18:40
mjturekahhhh, thanks Shrews. I'll search launchpad a bit!18:41
*** pelix has quit IRC18:47
*** killer_prince has quit IRC18:47
*** killer_prince has joined #openstack-ironic18:47
*** killer_prince is now known as lazy_prince18:48
Shrewsmjturek: i can't find it, but this recent review seems to indicate that is already merged: https://review.openstack.org/141966  That's all I know about it, though.18:48
mjturekShrews, thanks! I'll see what I can get out of it :)18:49
devanandamjturek: sdake was in here asking the same thing yesterday. and several other folks were discussing it before the break18:49
devanandait's definitely an area where our docs need to be improved18:49
*** datajerk has quit IRC18:50
mjturekdevananda gotcha, maybe digging through the logs would be useful then18:50
NobodyCammjturek: if your going through the process now. I would love to review a doc update patch :)18:51
NobodyCam<shameless pulg>18:51
mjturekNobodyCam, I would be more than happy to help out with documenting the process :)18:51
mjturekjust trying to get through it successfully first :-P18:51
NobodyCamofc :)18:52
devanandaperhaps an etherpad where everyone working on that shares notes -- as a starting point?18:52
mjturekdevananda, sounds like a great idea18:52
NobodyCamwehave https://etherpad.openstack.org/p/IronicDocumentationTasks18:53
mjturekhow about a section there then?18:53
NobodyCamthat would work for /me18:53
devanandahow about https://etherpad.openstack.org/p/ironic-devstack-real-hardware18:54
NobodyCamahh another ether pad to keep track of ...lol18:55
devanandahah18:55
devanandanever mind - doc is probably better :)18:55
NobodyCamheheheh18:55
*** ijw has joined #openstack-ironic18:55
mjturekdevananda, NobodyCam, okay cool :) I'll drop notes there as I go18:55
NobodyCammjturek: awesome Thank you :)18:56
mjturekNobodyCam, no problem! always looking to contribute18:56
NobodyCam(and i am okay with another etherpad)18:56
*** ijw_ has joined #openstack-ironic18:58
*** datajerk has joined #openstack-ironic18:58
*** ijw has quit IRC19:01
*** penick has quit IRC19:02
NobodyCambrb quick walkies time19:03
*** ijw_ has quit IRC19:04
*** penick has joined #openstack-ironic19:06
*** eghobo has joined #openstack-ironic19:08
*** erwan_taf has joined #openstack-ironic19:09
*** dkingshott has joined #openstack-ironic19:11
openstackgerritJay Faulkner proposed openstack/ironic-python-agent: Allow use of multiple simultaneous HW managers  https://review.openstack.org/14319319:11
*** vdrok_afk has quit IRC19:11
JayFI updated https://review.openstack.org/#/c/143193/ in response to code review comments; would love additional eyes19:22
*** eghobo has quit IRC19:23
*** ryanpetrello_ has joined #openstack-ironic19:25
*** Masahiro has joined #openstack-ironic19:27
*** ryanpetrello has quit IRC19:27
*** ryanpetrello_ is now known as ryanpetrello19:27
NobodyCamahh TY JayF you get the recheck on rloo's patch :)19:29
* rloo looks up from reviewing. huh? ;)19:30
*** eghobo has joined #openstack-ironic19:30
*** Masahiro has quit IRC19:31
NobodyCamlol your title check patch needed to be rechecked19:32
rlooNobodyCam: just noticed. is that error / time out happening a lot?19:32
NobodyCamI haven't seen a it... but I maybe behind the curve19:33
rlooNobodyCam: ok, will try to keep an eye out. I think I may have seen it happen last night.19:33
lucasagomesjroll, fyi https://review.openstack.org/14523519:36
lucasagomestested locally19:36
NobodyCamoh neat-oh19:36
JayFrloo: Might I request a world-famous Ruby-review on https://review.openstack.org/#/c/143193/19:38
JayF:P19:38
rlooNobodyCam: that is Bug 1326813 - for the timeout thing19:38
rlooJayF: not sure. let me see...19:38
*** david-lyle_afk is now known as david-lyle19:39
rlooJayF: hmm. I haven't yet (I don't think) reviewed any agent code. Should I start now?19:39
JayFThat would probably be a good place to start.19:42
NobodyCamJayF: you happen to have a free minute to take a look at https://review.openstack.org/#/c/13921719:42
NobodyCamlucasagomes: ^^^^^^ also19:42
JayFIt touches a little bit of everything19:42
JayFopening it in a tab NobodyCam19:42
lucasagomesack19:43
JayFitsatrap! State machine code! Pull the ripcord!19:43
JayF:P19:43
*** mriedem has joined #openstack-ironic19:43
NobodyCamlol19:43
devanandaadam_g: recent grenade-sideways failure, which seems to be the result of a kernel panic while starting the deploy ramdisk -- http://logs.openstack.org/60/144760/1/gate/gate-grenade-dsvm-ironic-sideways/5d08c02/logs/ironic-bm-logs/baremetalbrbm_2_console.txt.gz19:43
mriedemdevananda: i'm unfamiliar with http://logs.openstack.org/60/144760/1/gate/gate-grenade-dsvm-ironic-sideways/5d08c02/logs/ironic-bm-logs/baremetalbrbm_2_console.txt.gz - can you point out what specifically you're looking at19:44
devanandamriedem: this is the kernel log of a VM used by devstack & ironic to "mock" bare metal behavior within the gate env19:44
devanandamriedem: scroll to the bottom - the problem should then be obvious19:44
adam_gyea, finding something to fingerprint for that failure is tough19:44
mriedem[  422.148000] Kernel panic - not syncing: Fatal exception in interrupt19:44
adam_gthose console logs aren't in logstash19:44
devanandaadam_g: gah :(19:45
mriedemis there any way to get this back into the ironic logs as an error?19:45
devanandathere's nothing in the screen logs to fingerprint -- because there's no error there, aside from a timeout19:45
mriedemright19:45
devanandamriedem: no, because it's outside of what Ironic can see19:45
devanandaironic knows the node is powered on -- it is monitoring that19:45
devanandaand all the PXE env is correctly prepared -- we could watch for that, but it would pass, since the node has already PXE booted19:46
devanandaironic is left waiting for the OS and init scripts within the booted ramdisk to *do* something -- namely, call back to ir-api19:46
devanandawe could adjust the timeout on this19:46
devanandaso it fails faster19:46
mriedemwhat's the timeout today?19:46
devanandaor add these kernel logs to logstash19:46
JayFa console log type functionality could possibly see this19:47
devananda30 minutes19:47
mriedemi'm not sure how easy it is to add the kernel logs to logstash, would be a question for clarkb probably, i know it's easier if they have oslo log formatting19:47
JayFbut not likely automatically detect it19:47
*** hypnosb has quit IRC19:47
devananda 537 # Timeout (seconds) for waiting callback from deploy ramdisk.19:47
devananda 538 # 0 - unlimited. (integer value)19:47
devananda 539 #deploy_callback_timeout=180019:47
mriedembut syslog is indexed so i don't see why this couldn't be19:47
*** hypnosb has joined #openstack-ironic19:48
devanandamriedem: it's not the devstack host's syslog or kernel log, though19:48
mriedemdevananda: ok, the tempest build timeout is 5 minutes :)19:48
mriedemdevananda: oh these are remote systems?19:48
devanandaheh. so easy change there -- tune down the timeout for devstack-gate19:48
devanandamriedem: no. they're special VMs created by devstack and enrolled with Ironic19:49
mriedemok19:49
devanandathey behave like remote systems19:49
mriedemwas figuring that since the logs are published we should be able to index them19:49
NobodyCambrb19:49
devanandafor all intents and purposes, once tempest starts exercising Ironic and Nova, these are remote systems19:49
adam_gthere is/was another issue with nodes provisioning, i thought we had a fingerprint to catch nodes failing to reach an ACTIVE state19:50
mriedemi think the node reaches active, it's the instance deploy that fails if i'm understanding the logs19:51
mriedemor is instance == node in ironic speak?19:51
adam_glet me look19:51
devanandamriedem: instance == what's on the node. node == the hardware19:51
mriedemk, that's what i thought19:52
devanandamriedem: the node reached POWER_ON. but it was still waiting-for-callback19:52
mriedemso not active19:52
devanandawhich means the deploy ramdisk did not POST back to ir-api19:52
devanandait hadn't even copied the instance image to the node yet19:52
mriedemb/c of said kernel panic19:52
devanandaquite far from active19:52
adam_gah19:52
adam_ghttps://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries/1393099.yaml19:52
adam_gwe're failing in a nova test, so we're not tracking ironic state at all (on the client/tempest side at least)19:53
devanandathat would catch it19:53
adam_gno19:53
adam_gDetails: (ServerAddressesTestJSON:setUpClass) Server 4e4b8db2-9b2c-470b-a1d9-028696199413 failed to reach ACTIVE status and task state "None" within the required time (300 s). Current status: BUILD. Current task state: spawning.19:53
devanandaooh. there's no "Timed out waiting for node" in this log19:54
adam_gthat existing query hits against scenario tests, which do some tracking of ironic states19:54
mriedemright, nothing really good to track on19:54
devanandaright - tempest timeout hit first19:54
mriedemfwiw this is the bug i reported https://bugs.launchpad.net/ironic/+bug/140806719:55
devanandaadam_g: what do you think of lowering deploy_callback_timeout within Ironic for this (and possibly many other) jobs?19:55
devanandapossibly all of them?19:55
devananda30 minutes is way longer than it should take, even on most hardware19:55
JayF+119:56
adam_gdevananda, to fail faster? yeah, sounds doable19:56
mriedemdevananda: adam_g: we could just set deploy_callback_timeout lower in devstack-gate for tempest runs19:56
mriedembut if 30 min isn't a sane default, i guess you can change it in the code19:57
openstackgerritMerged openstack/ironic-specs: Check that filename is same as blueprint name  https://review.openstack.org/14508819:57
devanandaI suspect a production-sane default here is still going to be longer than what we want in d-g19:57
*** ijw has joined #openstack-ironic19:57
mriedemi'd think so too19:57
devanandaeg, 10 min is reasonable in production, but not in d-g19:57
devanandabecause some hardware POSTs slowly19:58
NobodyCami was just about to say 10 min19:58
adam_gseems the callback timeout in devstack/tempest should be lower than the nova boot timeout19:58
devanandaIIRC, we picked 30 as the default to be on the safe side19:58
adam_gso we get an ironic failure before the nova19:58
devanandaadam_g: ++19:58
mriedemso < 300, sure19:58
JayFdevananda: I'd be a bigger fan of that timeout being more like 15m than 10m by default :)19:58
mriedemadam_g: devananda: could be like 4 min in d-g19:58
JayFdevananda: although it'll obviously vary wildly by size of image written and such19:58
adam_ghmm19:59
devanandaJayF: no. this is tiemout waiting for the first call back19:59
devanandaJayF: not end of provisioning19:59
adam_gtheres no active_timeout or similar, eh?19:59
adam_gi guess there can't be20:00
devanandait should be POST + deploy ramdisk transfer and boot time20:00
JayFoh wow20:00
JayFokay20:00
devanandaadam_g: we dont, but should, have timeouts for each state transition20:00
devanandaJayF: just double checked -- this timeout only applies to DEPLOYWAIT today20:01
adam_gdevananda, but as JayF mentioned, that timeout depends largely on the size of images being booted20:01
devanandashouldn't20:02
devanandahardware POST time >> time to send a 1GB file over http20:02
*** ijw has quit IRC20:03
devanandaalso, IPA's image is not 1GB20:03
devanandaand the iscsi/pxe image is even smaller20:03
devanandaI need to run. bbi3020:04
adam_gsorry, i mean a timeout on reaching ACTIVE20:04
*** ijw has joined #openstack-ironic20:04
NobodyCamthe longest post i can recall traacking was like 7 or 8 minutes20:04
NobodyCamadam_g: thats nova's time out or ironics?20:04
adam_gNobodyCam, i dont know if nova has a timeout, but tempest has an active timeout it uses when booting instance.20:06
adam_gNobodyCam, but my concern is about having timeouts for ironic state transitions, in addition to exisitng callback timeout20:06
*** dprince has quit IRC20:09
NobodyCamhumm20:09
NobodyCamhow would that impace the long running agent folks20:10
NobodyCamimpact even20:10
devanandaNobodyCam: shouldnt20:13
devanandathats not a transition20:14
NobodyCambut wouldn't we see it as a node in deploy wait?20:14
devanandatoday? I dunno20:14
devanandawith the new FSM? nope20:14
devanandathere's soooo much stuff that is waiting for that :(20:15
NobodyCam:(20:15
* devananda really goes AFK now20:15
* NobodyCam alsosteps afk 20:18
rlooJayF: there are a few specs that I want to look at. I think/try to look at 143193 by the end of Thurs.20:20
*** erwan_taf has quit IRC20:21
adam_gmriedem, https://review.openstack.org/#/q/status:open+branch:master+topic:callback_timeout,n,z20:24
*** jistr has quit IRC20:25
mriedemadam_g: thanks20:26
mriedemcommented20:26
openstackgerritVictor Howard proposed openstack/ironic: display error logging should be improved  https://review.openstack.org/14477120:27
* NobodyCam is back 20:30
JayFrloo: it's not a spec; it's actual code, and tbh I'd love to have it landed before then :)20:32
*** pensu has quit IRC20:32
rlooJayF: yeah, I know it's code. Just that I'm behind on reviewing the specs. So this is a high priority thing for you?20:32
JayFrloo: it's blocking something I'd like to be able to do downstream (tear apart our big hardware manager into smaller pieces that possibly others could consume)20:33
rlooJayF: hmm. I'll try to get to it tomorrow then. Since it isn't code I'm familiar with, I need a bigger chunk of time to get up to speed/review.20:34
*** sambetts has quit IRC20:36
adam_gJayF, re IPA+devstack, is the time between a 'nova boot' and ironic powering the machine on typically <60 sec? with a cold cache20:36
JayFadam_g: I don't have a good feel for that; tbh20:37
adam_gder, i suppose i could just go look at gate logs20:37
JayFadam_g: I'd suspect so, though. It's only ~200mb transferred locally20:37
*** ijw has quit IRC20:37
*** sambetts has joined #openstack-ironic20:37
adam_gyeah, looks about 30 sec20:39
JayFThis is a change with 2x +2 from Rackers on IPA, but we wanted an outside opinion before landing it: https://review.openstack.org/#/c/141957/ -- basically adds a really sweet standalone mode to IPA, whereby you can spin up an agent without an ironic to issue it commands for functional testing20:41
*** andreykurilin has joined #openstack-ironic20:43
*** pcaruana is now known as pcaruana|afk|20:45
openstackgerritJosh Gachnang proposed openstack/ironic: Improve IPA client library  https://review.openstack.org/11111820:46
jroll19:32:33            rloo | NobodyCam: just noticed. is that error / time out happening a lot? <- hp cloud is having networking issues, fyi21:01
rloojroll: thx. there's a bug for it. 1326813. Guess it happens now and then :-(21:02
jrollyeah, networks are hard21:03
JayF++21:03
*** ijw has joined #openstack-ironic21:08
*** ijw has quit IRC21:14
NobodyCamdoh21:15
*** Masahiro has joined #openstack-ironic21:16
*** Masahiro has quit IRC21:20
*** Marga_ has joined #openstack-ironic21:21
*** russellb has joined #openstack-ironic21:27
*** eghobo has quit IRC21:41
* NobodyCam may need to run out for a bit..21:43
*** russellb has left #openstack-ironic21:53
*** ryanpetrello_ has joined #openstack-ironic21:55
*** ryanpetrello has quit IRC21:56
*** ryanpetrello_ is now known as ryanpetrello21:56
*** penick has quit IRC21:56
*** eghobo has joined #openstack-ironic21:57
*** Marga_ has quit IRC21:57
*** Marga_ has joined #openstack-ironic21:57
Shrewsrloo: i gave https://review.openstack.org/139217 a second +2, but didn't approve. i want to give you time to make sure you're comfortable with it21:57
rlooShrews: let me look. it was awhile ago. I think it was due to changing the logic a bit though?21:58
*** Marga_ has quit IRC21:59
*** Marga_ has joined #openstack-ironic21:59
rlooShrews: wrt 139217, am I wrong about the > vs >= at line 1479? https://review.openstack.org/#/c/139217/9/ironic/conductor/manager.py22:08
openstackgerritDevananda van der Veen proposed openstack/ironic: Hide oslo.messaging DEBUG logs by default  https://review.openstack.org/14536122:17
*** jjohnson2 has quit IRC22:18
JayFdevananda: ^ related: https://www.youtube.com/watch?v=9xLW-1c836k22:18
*** penick has joined #openstack-ironic22:18
devanandaright22:19
NobodyCamdevananda: nice catch22:21
Shrewsrloo: sorry, went to the store real quick...22:29
Shrewsrloo: yes, i believe devananda is correct here22:29
rlooShrews: no worries22:29
rlooShrews: ok, I'm not going to go through it to convince myself.22:30
Shrewsrloo: the heart of it is, count is incremented before the power state check, but not checked until AFTER that check22:33
Shrewsso it will get one more power state check before it goes over the config value22:33
Shrewsbut, yeah, a bit terse code there22:34
Shrewsdevananda: i *thought* there was a fix to oslo.messaging for that22:35
Shrewsdevananda: i'd rather just bump the version there, if so22:35
* Shrews looks22:36
JayFShrews: we shouldn't bump our version independent of global reqs, right?22:36
ShrewsJayF: yeah. i would expect someone to bump it in global reqs though22:37
adam_gShrews, fix committed, but not released yet https://bugs.launchpad.net/designate/+bug/139925722:37
JayFfor https://review.openstack.org/#/c/137926/ do we care that the "H" in github is not capitalized?22:37
JayFI'd like to land it.22:37
Shrewsadam_g: thx22:37
Shrewsi guess it won't hurt to go ahead and land it if the new oslo package isn't out22:39
Shrewsdevananda: that linked bug # is, erm, off?22:40
rlooJayF: wrt 137926, if that's the only reason? I don't care about the H. It can be updated afterwards.22:42
Shrewsthat seems like a silly reason to hold it back22:43
rlooShrews: unless we could get sued or something? but even so, fix it later.22:44
openstackgerritOpenStack Proposal Bot proposed openstack/ironic-python-agent: Updated from global requirements  https://review.openstack.org/14233422:45
NobodyCamlol22:48
rlooShrews: sorry, to be clear about 139217 wrt the counts. "devananda is correct" that there is a bug there, so this patch fixes that bug, right? And we're fine if it isn't mentioned that it was fixed in the commit?22:54
NobodyCamrloo: i'm okay with it just because I don't want to rebase the entire patch set :-p22:56
*** ryanpetrello has quit IRC22:56
NobodyCamwounder if just editing the commit message thru the web interface would work?22:56
rlooNobodyCam: so if the behaviour changes for someone because of this, and it isn't mentioned in the commit, and no bug is opened for it, is it OK? Sorry, just being devil's advocate here.22:57
Shrewsrloo: you raise a valid point about it not being in the commit msg. but we can edit that thru the iface22:58
ShrewsNobodyCam: are you trying that?22:58
NobodyCamlol was just about to...22:58
rlooShrews, NobodyCam: thx :-)22:59
NobodyCambut have not yet22:59
Shrewsif we can help deva push this along, we should22:59
NobodyCamyep22:59
NobodyCamthis may require a full rebase but...22:59
rlooI still have a big problem with the code anyway; it doesn't handle the target-state properly or something.22:59
rlooI added a patch to the end of the series, with updated unit tests to show how it fails when it shouldn't fail.23:00
Shrewsrloo: oh?23:00
Shrewsah, see it now23:01
*** mriedem has left #openstack-ironic23:01
NobodyCamrloo: - corrects minor issue with retry loop  .. work?23:01
rlooNobodyCam: no. sec.23:02
openstackgerritMerged openstack/ironic-specs: Add support for VirtualBox WebService.  https://review.openstack.org/13792623:02
rlooNobodyCam: devananda sez: 'the "max_retries" option is acting as "max_tries" -- in other words, set the option to 3, and it will attempt to sync the power state 3 times, not 4.'23:02
rlooNobodyCam: that's what he fixed.23:03
Shrewsi.e., it retried one more time than it was supposed to23:03
Shrewsrloo: i'll look at your fsm patch tomorrow. thx for pointing that out23:04
rloodevananda also fixed something else, but I am fine not mentioning it because you're going to ask me to word it for you and I'm too tired ;)23:04
*** Masahiro has joined #openstack-ironic23:04
openstackgerritJim Mankovich proposed openstack/ironic-specs: Support per Driver Sensor Meters  https://review.openstack.org/13007423:05
NobodyCamrloo: - corrects minor issue with max_retries running an extra attemp  .. better?23:05
NobodyCamor - corrects issue with max_retries running an extra attemp  .. better?23:05
NobodyCamor - corrects issue with max_retries running an extra attempt  .. :-p23:06
openstackgerritChris Krelle proposed openstack/ironic: Refactor async helper methods in conductor/manager.py  https://review.openstack.org/13921723:07
rlooNobodyCam: yeah23:07
NobodyCamand now the rebasing starts23:07
NobodyCamlol I should have done it locally23:08
*** Masahiro has quit IRC23:09
rloodinner time. ciao.23:10
NobodyCamnight rloo23:10
NobodyCam:)23:10
*** rloo is now known as rloo_afk23:11
*** alexpilotti has quit IRC23:12
*** eghobo has quit IRC23:13
openstackgerritChris Krelle proposed openstack/ironic: Enable async callbacks from task.process_event()  https://review.openstack.org/14086823:14
openstackgerritMerged openstack/ironic-python-agent: Add standalone mode for IPA  https://review.openstack.org/14195723:17
*** jgrimm is now known as zz_jgrimm23:18
openstackgerritChris Krelle proposed openstack/ironic: Convert check_deploy_timeout to use process_event  https://review.openstack.org/14086923:18
openstackgerritChris Krelle proposed openstack/ironic: Convert drivers to use process_event()  https://review.openstack.org/14088323:20
NobodyCamrloo_afk: should I rebase 14230323:20
*** ijw has joined #openstack-ironic23:20
*** hypnosb has quit IRC23:20
*** ChuckC has quit IRC23:20
*** aweeks has quit IRC23:20
*** zer0c00l has quit IRC23:21
*** yuriyz has quit IRC23:21
*** lynxman has quit IRC23:21
*** NobodyCam has quit IRC23:21
*** soren has quit IRC23:21
*** TheJulia has quit IRC23:21
*** med_ has quit IRC23:21
*** aignatov has quit IRC23:21
*** kylestev has quit IRC23:21
*** ekarlso- has quit IRC23:21
*** marios has quit IRC23:21
*** TheJulia has joined #openstack-ironic23:21
*** NobodyCam has joined #openstack-ironic23:21
*** lynxman has joined #openstack-ironic23:21
*** yuriyz has joined #openstack-ironic23:21
*** aweeks has joined #openstack-ironic23:21
NobodyCamhello?23:21
NobodyCamwow that was strange23:21
*** zer0c00l has joined #openstack-ironic23:22
jrollyay netsplits23:22
*** ChuckC has joined #openstack-ironic23:22
*** med_ has joined #openstack-ironic23:22
*** aignatov has joined #openstack-ironic23:22
*** kylestev has joined #openstack-ironic23:22
*** ekarlso- has joined #openstack-ironic23:22
*** marios has joined #openstack-ironic23:22
NobodyCamIrssi: Join to #openstack-ironic was synced in 64 secs23:23
NobodyCamwow supper slow23:23
JayFNet is splitting23:24
NobodyCamlol23:24
NobodyCami'm not going to rebase the rloo last patch in the FSM set as the first line of the commit message is NOT TO BE MERGED23:24
*** soren has joined #openstack-ironic23:26
*** zz_jgrimm has quit IRC23:26
NobodyCamShrews: commit message updated on the first of the fsm patches: https://review.openstack.org/#/c/13921723:27
ShrewsNobodyCam: re- +2'd23:28
NobodyCam:) I had a +223:28
NobodyCambut now I've edited it23:29
*** zz_jgrimm has joined #openstack-ironic23:29
NobodyCamlol23:29
NobodyCamdevananda: gots a sec to toss a +1 on 139217 ... just to show your ok with the external (/me) edit...23:29
*** penick has quit IRC23:34
*** ijw has quit IRC23:47
openstackgerritJay Faulkner proposed openstack/ironic-python-agent: Allow use of multiple simultaneous HW managers  https://review.openstack.org/14319323:51
NobodyCamShrews: would you remove your -1 if I updated the bug on the commit message?23:54
NobodyCamfor 14536123:55
devanandaNobodyCam: looks like rloo_afk removed her -1 but didn't + it ?23:55
ShrewsNobodyCam: yup23:55
*** hypnosb has joined #openstack-ironic23:55
NobodyCamdevananda: I'm just about to fix the bug number on 145361...23:56
NobodyCamdevananda: on 139217? ya I just updated the commit message and rebased the set .. but rloo had dinner while I did that :-p23:56
*** naohirot has joined #openstack-ironic23:56
devanandahang on ...23:56
ShrewsNobodyCam: what is the proper bug?23:57
NobodyCamhttps://bugs.launchpad.net/designate/+bug/139925723:57
devanandaoh, hah. yes, wrong bug23:57
devanandasomehow I copied the wrong ##23:57
devanandathanks23:57
*** ryanpetrello has joined #openstack-ironic23:57
NobodyCam:-p23:58
devanandaI was wondering why it the bot didn't post a comment on the bug ...23:58
*** dkingshott has quit IRC23:58
openstackgerritChris Krelle proposed openstack/ironic: Hide oslo.messaging DEBUG logs by default  https://review.openstack.org/14536123:58
NobodyCamfor edits like that the web gui thing isn't half bad I must say23:59
openstackgerritMerged openstack/ironic-python-agent: Updated from global requirements  https://review.openstack.org/14233423:59
ShrewsNobodyCam: heh, for some reason, i thought there was a diff bug. thx23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!