Friday, 2018-08-10

*** gkadam has joined #openstack-kayobe05:12
*** egonzalez has joined #openstack-kayobe06:58
*** priteau has joined #openstack-kayobe08:14
*** ktibi has joined #openstack-kayobe08:22
*** mgoddard has joined #openstack-kayobe08:28
mgoddardmorning kayobians08:38
verdurinMorning. Did mangle EFI booting yesterday, though there was something missing from the IPA image, so the node was stuck in a loop.08:43
mgoddardwell that's progress. Any idea what is missing from IPA?08:44
mgoddardwhich IPA images are you using?08:44
verdurinAbout to start a meeting, so I'll check later. Saw it go past in the logs, something hardware-related.08:46
verdurinWas just using the image built by kayobe.08:48
mgoddardI think the default images are coreos, although we typically build centos-based IPA images locally08:50
mgoddardrequires some config in etc/kayobe/ipa.yml08:50
ktibimgoddard, kayobe use centos by default, no coreOS09:00
ktibiverdurin, I tried to use uefi on a platform but I abandoned for legacy boot :/ I use ipxe.09:03
*** dougsz_ has joined #openstack-kayobe09:04
mgoddardktibi: default is coreos:
mgoddardif you don't build the image. The default image built is CentOS09:09
ktibimgoddard, yes I talked about build09:09
ktibihum you make me think maybe I'll try coreOS with my node which not work during inspection09:11
ktibibecause I use devuser with centos not work. I build myself the IPA and copy in bifrost container, no work too09:11
ktibiI can't get the LOG for see why inspection fail09:11
*** ktibi has quit IRC09:23
*** ktibi has joined #openstack-kayobe09:33
verdurinmgoddard: here is the introspection error
verdurinThe node was listed as enrolled on the seed, but clearly the error meant it kept trying10:16
mgoddardverdurin: did you add extra-hardware to ipa_collectors_extra in ipa.yml?10:17
verdurinmgoddard: I think I copied over the default one to the config path, so I could get it to recognise the build images = True setting10:21
verdurinmgoddard: there's nothing in ipa_collectors_extra10:23
mgoddardverdurin: hmm, is extra-hardware present in the kernel command line arguments of the default PXE config?10:28
mgoddardargument name would be ipa-inspection-collectors10:31
verdurinmgoddard: ah it is, yes. I copied that from elsewhere. Trying with that removed now10:31
mgoddardverdurin: that can be useful, but requires a ramdisk built with the python-hardware package10:32
verdurinYes, I'd been using that with TripleO10:32
mgoddardwe use to install it10:32
verdurinAnd unlike with the deployment image, it looks like there is a mechanism for including the extra elements10:44
mgoddardit uses stackhpc.os-image-elements10:45
verdurinIt's now trying to set the node name to a switch interface picked up via LLDP: and failing...10:50
mgoddardhmm, yeah that's not going to work10:53
mgoddardkayobe's probably a bit too opinionated there, we should change the default10:55
mgoddardalthough if your switch supports sending the interface description via LLDP it is a neat discovery trick10:56
verdurinmgoddard: grand, I'll try that.10:56
mgoddardnormally it requires setting an option on the switch globally or per interface10:56
mgoddardverdurin: the early stages of use are very useful, before you become accustomed to the idiosyncracies of the software. If you have any feedback it would be appreciated10:59
mgoddardnot now necessarily10:59
verdurinmgoddard: I've been trying to keep notes of everything10:59
mgoddardverdurin: perfect11:00
*** dougsz_ has quit IRC11:50
*** dougsz_ has joined #openstack-kayobe12:54
*** egonzalez has quit IRC13:03
verdurinIs there a way of using generic ipmitool with the iLO devices on this hardware?13:44
mgoddardI imagine so, but you might need to configure the server to allow IPMI13:45
verdurinI've already done that.13:48
verdurinJust need to add the values in kayobe-config/etc/kayobe/inventory/group_vars/all/bmc ?13:48
mgoddardare you running 'overcloud inventory discover'?13:50
verdurinNOt yet.13:50
verdurinLooking at this because when introspection finishes I noticed that it's unable to power off the node.13:50
verdurinAnd that failure is reported in the introspected node's record.13:51
mgoddardwhich failre?13:51
verdurinThe power management failure.13:51
mgoddardhave you configured the ipmi username & password?13:51
mgoddardipmi_username, ipmi_password in etc/kayobe/bmc.yml13:52
verdurinI had configured the introspection IPMI details, I'm now looking at doing so more generally, hence the question13:52
mgoddardthese will be applied to nodes during inspection via rules13:52
verdurinOkay. Referring to the earlier question, it's a bit confusing having these settings in multiple places.13:53
mgoddardmost settings for kayobe are in etc/kayobe/*.yml13:54
mgoddardI'd put the bmc config in etc/kayobe/bmc.yml rather than etc/kayobe/inventory/group_vars/all/bmc13:55
verdurinIt does explain the precedence here, admittedly
mgoddardit takes a bit of getting used to, but there is a method to the madness13:57
mgoddardopen to suggestions of course13:58
verdurinI keep getting caught out because (e.g.) I run 'ag ipmi_username' then pick what seems the likeliest one.13:58
mgoddardI guess the thing to keep in mind is that kayobe tends to set global defaults in kayobe/ansible/group_vars/all/<file>, and these variables can be set in kayobe-config/etc/kayobe/<file>.yml14:00
mgoddardthere are exceptions, and sometimes you need to use inventory host_vars/group_vars when variables need to vary between hosts/groups14:01
verdurinHmm. It's still not picking up the IPMI credentials.14:18
mgoddardare the inspector rules correct14:18
verdurinThey're basically the same ones you pasted earlier, with LLDP included because I fixed that on the switch.14:21
mgoddardcan you check the rules in inspector?14:21
mgoddardyou can use python-ironic-inspector-client with OS_TOKEN=fake OS_URL=http://localhost:505014:22
verdurinFirst rule is 'Set IPMI driver_info if no credentials'14:26
mgoddarddoes it have the correct credentials?14:28
mgoddardopenstack baremetal introspection rule show <rule>14:28
verdurinNo, neither username nor password are in there.14:31
verdurinBoth are set in etc/kayobe/bmc.yml and I rebuilt the image after doing so.14:31
mgoddardthe rules get registered during seed service deploy14:32
mgoddardyou could short cut via14:33
mgoddardkayobe playbook run ansible/seed-introspection-rules.yml14:33
mgoddardit should re-register the rule14:33
verdurinYes, looks like that's made some changes.14:38
mgoddardyou should be able to reprocess the inspection data14:39
mgoddardopenstack baremetal introspection reprocess <node>14:39
mgoddardverdurin: have you read It covers our typical cloud discovery setup14:42
verdurinmgoddard: I hadn't seen that. Will read it properly. Sadly this pilot hardware is HPE, so I can't use all your nice stuff.14:44
verdurinThe production hardware will be Dell with Dell management switches, albeit running Cumulus.14:45
mgoddardcumulus on dell has come up for us before, although haven't tried it yet14:45
verdurinMarvellous - power management works now, so I can add the remaining nodes.14:47
verdurinThanks for your forebearance mgoddard14:47
mgoddardI'm hoping that from this I can generate some better config docs for the seed side of things14:49
mgoddardno problem verdurin - this is one of the hardest parts of getting a system up and running, and it's inevitably the first thing you do, when you're least familiar14:50
verdurinmgoddard: I think this is the fourth different OpenStack installation I've used, but who's counting...14:52
verdurin(installation method)14:52
mgoddardstill looking for 'the one'?14:53
verdurinI quite liked TripleO but debugging was just too much of a mither.14:54
mgoddardthat has been my (limited) experience with it14:54
mgoddardonce you get to know the patterns used in kolla ansible, I think it works quite well14:55
mgoddardprovides a lot of control14:55
mgoddardseen this:
verdurinI very much like Ansible, so it's an excellent base on which to build.14:56
verdurinDid skim through that. Should take a proper look.14:57
verdurinmgoddard: incidentally, the reprocessing of introspection data appears to require some extra configuration that I'm currently missing.15:08
verdurin"Inspector is not configured to store data. Set the [processing] store_data configuration option to change this"15:11
verdurinNot that it really matters.15:11
mgoddardoh, my bad15:14
mgoddardit requires
mgoddarddidn't make it into queens15:14
mgoddardyou can cherry pick & rebuild the container image if it's useful to you15:14
*** gkadam has quit IRC16:06
*** dougsz_ has quit IRC16:58
*** mgoddard has quit IRC17:01
*** ktibi has quit IRC17:29
*** mgoddard has joined #openstack-kayobe17:30
*** priteau has quit IRC17:52
*** mgoddard has quit IRC18:30
*** priteau has joined #openstack-kayobe19:47
*** gkadam has joined #openstack-kayobe19:47
*** priteau has quit IRC20:12
*** gkadam has quit IRC22:34

Generated by 2.15.3 by Marius Gedminas - find it at!