| opendevreview | Steve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros https://review.opendev.org/c/openstack/ironic/+/815392 | 00:11 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros https://review.opendev.org/c/openstack/ironic/+/815392 | 03:19 |
| opendevreview | Steve Baker proposed openstack/ironic master: Add missing mode setting on pxe created directories https://review.opendev.org/c/openstack/ironic/+/809294 | 04:40 |
| MikeCTZA | morning ironic | 05:36 |
| opendevreview | Mahnoor Asghar proposed openstack/python-ironicclient master: Made `baremetal <command> --help` display command specific help. https://review.opendev.org/c/openstack/python-ironicclient/+/816098 | 06:20 |
| iurygregory | good morning janders MikeCTZA and Ironic o/ | 07:02 |
| MikeCTZA | hoping today to try get past my ironic issues ... but then again ... I've said that about 5 days in the past | 07:02 |
| arne_wiebalck | Good morning janders iurygregory MikeCTZA and Ironic! | 07:10 |
| janders | hey iurygregory MikeCTZA arne_wiebalck and Ironic o/ | 07:10 |
| iurygregory | morning arne_wiebalck o/ | 07:10 |
| MikeCTZA | arne_wiebalck: happy workiversary | 07:10 |
| iurygregory | yeah, congrats on 15 years :D | 07:11 |
| iurygregory | . | 07:19 |
| iurygregory | wrong window :D | 07:19 |
| arne_wiebalck | MikeCTZA: iurygregor: Thanks! | 07:20 |
| dtantsur | good morning folks! | 07:55 |
| janders | hey dtantsur o/ | 08:05 |
| iurygregory | morning dtantsur | 08:11 |
| MikeCTZA | arne_wiebalck: ideas where to start looking? I run IPA from the ramdisk and it fails to perform inspection, on controller in inspector logs I see it cant match with "Look up error: Could not find a node for attributes" (then the MACs) | 08:14 |
| dtantsur | MikeCTZA: you need to create a node with at least one MAC so that inspector can match the data with the node | 08:19 |
| dtantsur | (or with IPMI address, not sure it applies to you) | 08:20 |
| MikeCTZA | so the openstack baremetal port create is the MAC of the server NIC not the BMC correct? | 08:20 |
| rpittau | good morning ironic! o/ | 08:22 |
| janders | hey rpittau o/ | 08:23 |
| iurygregory | morning rpittau o/ | 08:23 |
| dtantsur | MikeCTZA: correct | 08:23 |
| rpittau | hey janders iurygregory :) | 08:24 |
| MikeCTZA | hmmm dtantsur | 08:27 |
| MikeCTZA | https://www.irccloud.com/pastebin/sZX3LtIC/ | 08:27 |
| MikeCTZA | this is what I'm seeing, not sure what I'm missing or dont have correctly setup, as I assume it would work, I've had to do a few other changes to get that to be the case too | 08:28 |
| dtantsur | MikeCTZA: IPMI addresses don't seem to match? | 08:28 |
| dtantsur | you have a node with ipmi_address=10.102.30.31 MAC ec:f4:bb:d5:a6:10 | 08:29 |
| MikeCTZA | yes I have to nodes sorry I pasted a bit mixed up | 08:29 |
| dtantsur | also... why are you even running inspection at this point? | 08:30 |
| MikeCTZA | I also have 10.102.30.29 with MAC: ec:f4:bb:d5:a6:3a | 08:30 |
| dtantsur | (sorry if I missed the previous context - just out of PTO) | 08:30 |
| MikeCTZA | I thought I needed to, maybe I dont | 08:31 |
| dtantsur | you have nodes in available, do you even expect inspection to be run? | 08:31 |
| dtantsur | let's take a step back: what are you trying to achieve now? | 08:31 |
| MikeCTZA | OK so ... I'm basically trying to get ironic working in anyway | 08:31 |
| MikeCTZA | we use kolla ansible for deployment, Ive had to change a few things to suit our networtking but that seems to be OK for now | 08:32 |
| MikeCTZA | so if nodes are available then should I see resources with I do a hypervisor show on the BM node? | 08:32 |
| MikeCTZA | I was trying to deploy a node and get no host avail so maybe I'm looking in the wrong areas | 08:33 |
| dtantsur | okay, so inspection is not where you want to end up | 08:33 |
| MikeCTZA | ah OK, so its me being a bit confused then | 08:34 |
| dtantsur | I'm not 100% familiar with how Kolla sets up inspector, but usually a so called "PXE filter" is used in ironic-inspector to prevent its DHCP from responding to normal nodes | 08:34 |
| dtantsur | have you modified anything around this part? | 08:34 |
| MikeCTZA | that was what I had to modify ... the inspector was bound to wrong IP which I sorted, and also had a filter of dnsmasq, if I have that I just get MAC,ignore in the "mac files" so I had to comment that out | 08:35 |
| MikeCTZA | then I can get the BM boxes to PXE OK | 08:35 |
| MikeCTZA | I also had some DNS issues in the IPA RAMdisk but if I dont need inspection then that becomes a mute point I suspect | 08:36 |
| dtantsur | MikeCTZA: well, the disabled PXE filter is what prevents nodes to always doing inspection | 08:37 |
| dtantsur | (I wonder if you should just disable inspector completely for now, if Kolla allows it) | 08:38 |
| dtantsur | the confusing point: there are 2 dnsmasqs: one from neutron, one from inspector | 08:38 |
| dtantsur | the PXE filter is used to prevent the latter from conflicting with the former | 08:38 |
| dtantsur | but if you have no use for inspection (you knows MAC addresses and don't want to collect anything advanced), you may just disable it? | 08:38 |
| MikeCTZA | I'm also not totally sure about my openstack network config when it comes to creating the baremetal node line | 08:39 |
| MikeCTZA | I'll stop the inspector container for now then, we know the MAC and nodes so its not like they are dynamic or we get often we have pretty static setup | 08:39 |
| dtantsur | MikeCTZA: don't forget to stop its dnsmasq as well | 08:40 |
| MikeCTZA | I had forgotten that one so stopped that as well now | 08:40 |
| MikeCTZA | so ... now ... when a BM node boots and PXEs, that should then load the IPA and do anything? or is there now no need for that part? | 08:41 |
| dtantsur | yeah, so, now the node should boot off the neutron's dnsmasq, boot IPA, write the image and reboot into it | 08:41 |
| MikeCTZA | easier said that done ... I suspect - LOL, thats assuming I have all other things correct network wise | 08:42 |
| MikeCTZA | I'm still finding my feet and trying to understand ironic so sorry if it seems I'm a bit of a noob (cause I am with ironic) | 08:43 |
| MikeCTZA | OK so as I suspected could happen I didnt PXE boot at all now | 08:46 |
| MikeCTZA | I think I'm missing some openstack network bits in the mix now which I havent put back | 08:47 |
| dtantsur | check neutron logs for errors, maybe it fails to bind the port? | 08:49 |
| dtantsur | also tcpdump is your friend :) | 08:49 |
| MikeCTZA | so dtantsur I have ironic-dnsmasq/dnsmasq.conf:dhcp-range=10.102.55.100,10.102.55.140 ... will that now not be applicable as I've stopped ironic-dnsmasq - how will I then PXE/get an IP? | 08:58 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 08:58 |
| MikeCTZA | I'm guessing I get from neutron from what you said above as there are 2 dnsmasqs | 08:59 |
| dtantsur | MikeCTZA: if I understand Kolla right (you may check with them), it will not be applied. You need to set up your subnet in Neutron. | 08:59 |
| MikeCTZA | like this https://www.irccloud.com/pastebin/NaFGVsIe/ | 09:00 |
| MikeCTZA | like that? | 09:00 |
| dtantsur | I assume you know what you're doing with VLANs? | 09:00 |
| dtantsur | also I noticed that your previous paste had physnet1 for the ironic port, here it is physnet2 | 09:00 |
| dtantsur | is it intentional? | 09:00 |
| MikeCTZA | yeah was my iterations of testing on 2 boxes, physnet2 is the one in the ml2_conf.ini with network_vlan_ranges = physnet1:2007:2007,physnet2 | 09:01 |
| MikeCTZA | I'm all over trying stuff to get this going ... | 09:02 |
| dtantsur | I guess we should summon mgoddard at this point, I'm not strong in advanced networking topics | 09:02 |
| MikeCTZA | I'm not our main networking guy in this but think I'm doing things correctly, but thats still the area I'm concerned about | 09:03 |
| MikeCTZA | dtantsur: so if we use inspection (which we dont need to have), it will then use the inspector and ironic_dnsmasq, if you aren't using it it will use neutron's dnsmasq to get the IP (which it will get from the openstack subnet I created earlier)? | 09:18 |
| dtantsur | there is some history behind inspector. neutron cannot do what it needs, this is why inspector ships its own dnsmasq | 09:21 |
| dtantsur | it is used during inspection; then the PXE filter is used to disable it while deploying/cleaning | 09:22 |
| MikeCTZA | OK, so if a box is available in openstack baremetal list then you must have ,ignore set in the inspector so it doesnt try do anything with it and then its up to another role (neutron) to handle things | 09:23 |
| dtantsur | correct | 09:24 |
| rpittau | dtantsur: I asked infra about building/publsihing debian on arm, they have different arm-based nodes supplied by linaro and osuosl, capacity and utilization is reported at https://grafana.opendev.org/d/S1zTp6EGz/nodepool-linaro and https://grafana.opendev.org/d/EoBNqFXMk/nodepool-osuosl | 10:14 |
| rpittau | should be look for us, current usage doesn't seem high | 10:14 |
| rpittau | s/look/good | 10:14 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Build and publish arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 10:30 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Build and publish arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 10:32 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Build and publish arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 10:33 |
| dtantsur | rpittau: \o/ | 11:10 |
| dtantsur | this is amazing | 11:11 |
| janders | master ARM is on! | 11:15 |
| MikeCTZA | am I correct in saying that inspection is a voluntary thing and you dont need to do that as long as the node gets marked as available in openstack you are good? | 11:37 |
| janders | see you tomorrow Ironic o/ | 11:51 |
| opendevreview | Merged openstack/ironic stable/wallaby: Fix various issues in the anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/815870 | 11:52 |
| opendevreview | Verification of a change to openstack/ironic bugfix/18.1 failed: Fix various issues in the anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/815871 | 11:52 |
| arne_wiebalck | MikeCTZA: I think that is largely correct. As mentioned, inspection may do some things for you (like port creation) which you have to do yourself otherwise, but I think in general you do not have to inspect. | 13:00 |
| MikeCTZA | OK cool thank arne_wiebalck got a meeting tmrw with the others here (who you "met") for us to brainstorm what we think we need to do to get things working (or to try at least)as we are missing something or have some config issues with our setup | 13:02 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Bump pip for tinyipa to 21.3.1 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/814894 | 13:24 |
| rpittau | if anyone has a minute, the updates for pip in ipa-builder are kind of important as they fix a bunch of stuff, please have a look -> ttps://review.opendev.org/c/openstack/ironic-python-agent-builder/+/814894 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815944 | 13:25 |
| opendevreview | Verification of a change to openstack/ironic bugfix/18.1 failed: Fix various issues in the anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/815871 | 13:55 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Associate pip version to python version in DIB based images https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815944 | 14:44 |
| arne_wiebalck | Is there an API way to move nodes when they are stuck in "deploying" ? | 14:58 |
| dtantsur | no, we don't have a way to break lock | 14:58 |
| dtantsur | only by restarting a conductor | 14:58 |
| arne_wiebalck | dtantsur: thanks, will restarting be enough? | 14:58 |
| dtantsur | should be | 14:58 |
| arne_wiebalck | dtantsur: ty | 14:58 |
| arne_wiebalck | dtantsur: works, thanks again! | 15:01 |
| dtantsur | I guess I should add it to the troubleshooting faq | 15:01 |
| dtantsur | hmm, we do have it there https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#why-are-my-nodes-stuck-in-a-ing-state | 15:11 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Document commands to leave WAIT states https://review.opendev.org/c/openstack/ironic/+/816365 | 15:15 |
| arne_wiebalck | dtantsur: for ^^, does undeploy really work by itself? Isn't it that you need to get the node into "deploy failed" first by restarting the conductor, and then undeploy? | 15:48 |
| dtantsur | arne_wiebalck: nope | 15:48 |
| dtantsur | furthermore, restarting a conductor does not abort "wait" states | 15:48 |
| dtantsur | (maybe I should document this as well) | 15:49 |
| *** sshnaidm_ is now known as sshnaidm | 15:49 | |
| arne_wiebalck | oh, yeah ... so the text should probably not say "cleaning" then but "cleaning, i.e. state clean_wait" since there is also a state "cleaning" | 15:50 |
| dtantsur | This is a part of the section on wait, but I'll clarify | 15:50 |
| arne_wiebalck | yes, you're are correct | 15:50 |
| arne_wiebalck | but I was reading this out of context and others may do as well ("deploying can be aborted" sounds like "a node in state 'deploying' ..." | 15:51 |
| arne_wiebalck | ) | 15:51 |
| dtantsur | yep, updating | 15:51 |
| arne_wiebalck | thanks! | 15:52 |
| * dtantsur is waiting for `git review` | 15:52 | |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Document commands to leave WAIT states https://review.opendev.org/c/openstack/ironic/+/816365 | 15:52 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Build and publish arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 15:54 |
| NobodyCam | Good morning Openstack Folks.. Happy not Monday | 17:12 |
| rpittau | hey NobodyCam, thanks :) | 17:17 |
| rpittau | and I guess I'm leaving | 17:17 |
| rpittau | see ya tomorrow! o/ | 17:17 |
| *** pmannidi is now known as pmannidi|AFK | 17:34 | |
| opendevreview | Merged openstack/ironic master: Document commands to leave WAIT states https://review.opendev.org/c/openstack/ironic/+/816365 | 17:45 |
| NobodyCam | night rpittau | 17:45 |
| arne_wiebalck | Image sizes: Do we have a size (range) where we know that a bios will have issues? Does this differ from bios to bios? Is there a theoretical maximum we know of? (I have seen issues with images as small as 420MB compressed, but I am not clear on whether this was due to a broken BIOS or just pretty close to a limit.) | 18:10 |
| dtantsur | I'm not aware of that, we have too few reports | 18:11 |
| dtantsur | o/ | 18:11 |
| arne_wiebalck | we may not know the limit(s), but since we have looked into shrinking the image several times during the last years (e.g. by removing packages, adding new compression libs), this seems to be a recurring theme | 18:25 |
| NobodyCam | hey hey arne_wiebalck | 19:51 |
| NobodyCam | would you be willing to toss +1 on https://review.opendev.org/c/openstack/nova/+/813263 | 19:51 |
| NobodyCam | if you agree with the patch logic of course! | 19:51 |
| opendevreview | Steve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros https://review.opendev.org/c/openstack/ironic/+/815392 | 21:01 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!