| opendevreview | Steve Baker proposed openstack/ironic master: Add platform:rpm shim, grub packages to bindep https://review.opendev.org/c/openstack/ironic/+/815386 | 01:27 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/ironic master: Write master grub config on startup https://review.opendev.org/c/openstack/ironic/+/815580 | 01:28 |
| opendevreview | Steve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros https://review.opendev.org/c/openstack/ironic/+/815392 | 01:28 |
| opendevreview | Steve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros https://review.opendev.org/c/openstack/ironic/+/815392 | 02:47 |
| opendevreview | HanGuangyu proposed openstack/ironic master: Add description to the mod_wsgi part https://review.opendev.org/c/openstack/ironic/+/814916 | 05:19 |
| arne_wiebalck | Good morning, Ironic! | 06:35 |
| janders | hey arne_wiebalck o/ | 06:38 |
| dtantsur | morning folks | 06:55 |
| arne_wiebalck | hey janders dtantsur o/ | 07:04 |
| janders | hey dtantsur o/ | 07:05 |
| rpittau | good morning ironic! o/ | 07:19 |
| rpittau | second breakfast, brb :) | 08:29 |
| dtantsur | is rpittau a hobbit? :) | 08:38 |
| rpittau | :D | 09:01 |
| rpittau | dtantsur: now I remember the main problem with tinycore aarch64, the only port available is for RPi | 09:22 |
| dtantsur | ugh | 09:35 |
| dtantsur | does it matter for us? | 09:35 |
| rpittau | well, it's like starting from zero | 09:45 |
| rpittau | the port is an installer for raspberry pi, it's a bootloader that installs tinycore on RPi | 09:45 |
| rpittau | you have kernel and everything but it's tied to the RPi cpus and components | 09:45 |
| dtantsur | okay, so in ARM world there is no such thing as "generic aarch64"? | 09:47 |
| rpittau | not for tinycore AFAIK | 09:49 |
| rpittau | other distributions have something generic, like Fedora or Debian | 09:49 |
| dtantsur | .... | 09:50 |
| dtantsur | should we take a stab at an ipa-builder check job then? | 09:51 |
| dtantsur | I wonder what the RAM requirements will be to actually run such an image | 09:51 |
| rpittau | you mean with DIB ? not sure about the size, probably similar to x86_64 ? | 09:54 |
| dtantsur | yeah | 09:56 |
| rpittau | I'll fire up a check job with DIB using debian, let's see how it goes | 09:59 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 10:03 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 10:10 |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: Fix error messages in burnin code https://review.opendev.org/c/openstack/ironic-python-agent/+/815630 | 10:40 |
| *** dviroel|rover|afk is now known as dviroel|rover | 10:50 | |
| arne_wiebalck | TheJulia: dtantsur: https://opendev.org/openstack/ironic-python-agent/blame/commit/be3882162e432b292af9e787661661a05fa7d11a/ironic_python_agent/extensions/image.py#L292 was introduced for 'iscsi' ... do we know if this is needed for 'direct'? | 12:00 |
| arne_wiebalck | What I do know is that it is not sufficient for iscsi, but before submitting a patch I would like to understand if this is an issue for direct as well. | 12:01 |
| arne_wiebalck | (not sufficient means: _rescan_device() is not sufficient to make sure the device nodes are there) | 12:02 |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: Respect global parameters when downloading a configdrive https://review.opendev.org/c/openstack/ironic-python-agent/+/815448 | 12:05 |
| opendevreview | Riccardo Pittau proposed openstack/metalsmith master: Update pep8 test requirements https://review.opendev.org/c/openstack/metalsmith/+/814543 | 12:53 |
| dtantsur | arne_wiebalck: I'm not sure how this is related to iscsi? | 13:26 |
| dtantsur | I mean, it was "vital for iscsi" as TheJulia wrote in the commit message, but this code is used in all deploy methods | 13:27 |
| dtantsur | (and there is no more iscsi code in-tree) | 13:27 |
| TheJulia | I suspect it would be advisable to keep it in general becasue the case we were hitting was we were not spotting a change or loading a change to the OS before taking the next step | 13:39 |
| TheJulia | And... the Os doesn't always load changes if locks get involved | 13:40 |
| arne_wiebalck | dtantsur: this was my question since the initial commit said "needed for iscsi" | 13:51 |
| arne_wiebalck | dtantsur: the comment was removed, but the code stayed | 13:51 |
| dtantsur | needed for iscsi, but added to the generic code | 13:51 |
| dtantsur | do you think it should or should not be there? | 13:51 |
| arne_wiebalck | dtantsur: this is what I am trying to figure out :) | 13:52 |
| arne_wiebalck | on iscsi, it is not enough | 13:52 |
| arne_wiebalck | I am debugging this a the moment and have added "blockdev --rereadpt" which seems to work (tbc) | 13:52 |
| arne_wiebalck | but when I read the comment, I was wondering if the scan is needed for direct at all | 13:53 |
| dtantsur | hard to tell 100% | 13:53 |
| arne_wiebalck | https://usercontent.irccloud-cdn.com/file/EMwhwvgp/couldnotinstallbootloader.png | 13:53 |
| TheJulia | it may be okay to remove... *but* it is like the last line of defense for hte logic to make sure the information is present | 13:54 |
| arne_wiebalck | this is the error rate I saw and how is stopped when I added the blockdev command | 13:54 |
| dtantsur | I'd not remove it unless we're 100% sure it's not needed | 13:54 |
| arne_wiebalck | this is all on iscsi still | 13:54 |
| arne_wiebalck | dtantsur: ++ | 13:54 |
| arne_wiebalck | my patch will be to extend it actually | 13:54 |
| arne_wiebalck | _rescan_device() | 13:54 |
| TheJulia | ahh, okay | 13:54 |
| TheJulia | sounds good to me | 13:54 |
| arne_wiebalck | I will do some more tests, like trying to figure out if only iscsi, if blockdev is helping, if reproducable on direct, ... | 13:55 |
| arne_wiebalck | it is not fatal, but creates errors in our logs | 13:56 |
| arne_wiebalck | "I will be back" :) | 13:56 |
| arne_wiebalck | (this may take some days since it is a race) | 13:57 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 13:59 |
| janders | see you on Monday Ironic o/ | 14:14 |
| TheJulia | o/ | 14:14 |
| janders | (we have a surprise state holiday on Friday - moved from August due to COVID) | 14:15 |
| TheJulia | enjoyu | 14:15 |
| TheJulia | s/yu/y/ | 14:15 |
| janders | thank you TheJulia | 14:16 |
| TheJulia | bfournie: you around? | 14:16 |
| bfournie | Hi TheJulia | 14:25 |
| TheJulia | You mentioned someone reproduced the UEFI firmware memory loading issue observed on lenovo gear, was that with like a usb drive? | 14:27 |
| bfournie | TheJulia: trying to recall this... was this reproduction in-house do you know? | 14:29 |
| TheJulia | I think so | 14:30 |
| TheJulia | I'm going back and forth with jarrod johnson @ lenovo on a few different topics at the moment | 14:31 |
| *** ricolin_ is now known as ricolin | 14:40 | |
| bfournie | TheJulia: so I haven't found any more notes on Lenovo besides whats in 1990590 regarding the size of the ramdisk that iurygregory had added (comment 41) | 14:48 |
| TheJulia | heh, its the same hardware | 14:52 |
| TheJulia | model wise | 14:52 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 14:52 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 14:53 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 15:37 |
| tzumainn | hi! I've been testing metalsmith a bit recently, and I was wondering - if you deploy and specify a network, would it make sense for the created port to be created with the device_owner set to baremetal:none? | 15:47 |
| tzumainn | without that, it seems as if the networking-ansible driver won't work | 15:48 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Always include the oslo_log log file in ramdisk logs https://review.opendev.org/c/openstack/ironic-python-agent/+/815861 | 15:58 |
| arne_wiebalck | I see the same mount /boot/efi error with direct, so _rescan_disk() is needed but not enough :) | 16:02 |
| TheJulia | arne_wiebalck: good to know | 16:03 |
| arne_wiebalck | it is not failing all the time, it is a race again | 16:03 |
| arne_wiebalck | with the special files showing up too late | 16:03 |
| TheJulia | tzumainn: not sure, I mean, metalsmith development was a few years back at this point and I'm not sure exactly what your trying to achieve with doing such. | 16:03 |
| TheJulia | arne_wiebalck: oh jeeze :( | 16:04 |
| arne_wiebalck | I will try to confirm if 'blockdev --rereadpt' is enough to fix it | 16:04 |
| arne_wiebalck | I am worried it was just fixed by the additional time of the call ... | 16:05 |
| tzumainn | TheJulia, well, the issue is that without that setting the networking-ansible driver doesn't work | 16:05 |
| tzumainn | you end up with this message: "networking_ansible.ml2.mech_driver [req-39e37b77-eedf-4e04-be67-0f5521c42905 5a84b63362ce4bc8b7127fc05c762ec3 75073346\ | 16:05 |
| tzumainn | 211449fbba47ab6b0ddc56db - default default] Port 2153caef-19df-4160-951c-1328abae415d has device_owner: and vnic_type: baremetal which is not supported \ | 16:05 |
| tzumainn | by networking-ansible, ignoring." | 16:05 |
| TheJulia | wtf | 16:06 |
| TheJulia | I'd fix networking-ansible, tbh | 16:06 |
| TheJulia | it *shouldn't* care about device owner | 16:06 |
| tzumainn | TheJulia, ah, okay - I noticed that ironic now sets the device owner to baremetal:none, so I was wondering if that was just the new standard | 16:06 |
| TheJulia | arne_wiebalck: legitimate concern :( | 16:06 |
| TheJulia | tzumainn: no idea, I don't remember that change | 16:07 |
| arne_wiebalck | TheJulia: I will find out :) | 16:07 |
| rpittau | dtantsur: the debian arm64 ipa ramdisk is building -> https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 16:07 |
| rpittau | need to do one change to ipa-builder first | 16:07 |
| dtantsur | \o/ | 16:08 |
| dtantsur | one comment inline | 16:09 |
| rpittau | good, we won't need the new pkgs, just new pip | 16:09 |
| rpittau | dtantsur: ^ | 16:09 |
| dtantsur | ah, ok | 16:09 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build arm64 debian based ipa ramdisk https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/815815 | 16:10 |
| rpittau | going to split now, see you tomorrow! o/ | 16:10 |
| dtantsur | c u rpittau | 16:13 |
| opendevreview | Ruby Loo proposed openstack/ironic stable/xena: Fix various issues in the anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/815808 | 16:16 |
| opendevreview | Ruby Loo proposed openstack/ironic stable/wallaby: Fix various issues in the anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/815870 | 16:16 |
| dtantsur | rloo: probably bugfix/18.1 as well if you don't mind | 16:17 |
| rloo | dtantsur: sure, thx for mentioning that cuz it wasn't in my radar! | 16:17 |
| opendevreview | Ruby Loo proposed openstack/ironic bugfix/18.1: Fix various issues in the anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/815871 | 16:18 |
| arne_wiebalck | bye everyone o/ | 16:19 |
| TheJulia | iurygregory: you around? | 16:26 |
| opendevreview | Ruby Loo proposed openstack/ironic bugfix/18.1: Minor updates to anaconda doc https://review.opendev.org/c/openstack/ironic/+/815872 | 16:29 |
| opendevreview | Ruby Loo proposed openstack/ironic stable/wallaby: Minor updates to anaconda doc https://review.opendev.org/c/openstack/ironic/+/815873 | 16:29 |
| iurygregory | TheJulia, yup (not working day) but I'm here =) | 16:31 |
| rloo | the backports for the anaconda fixes have conflicts cuz it updates docs that don't have those ^^ changes. Figured better to backport the doc changes too so they are the same. | 16:31 |
| TheJulia | up for a really quick question? | 16:31 |
| iurygregory | yup =) | 16:31 |
| iurygregory | go ahead | 16:31 |
| TheJulia | iurygregory: do you remember what drove us to do https://github.com/openstack/ironic-python-agent/commit/b6210be196fea271b2c49f89d3e1638517c1198c#diff-2ac3e3f56d389db15b980c3471cb49965be083d4221667e009a4916595ef8a40R243 | 16:32 |
| TheJulia | we *think* this is causing major issues on lenovo sr650 machines | 16:32 |
| TheJulia | it is basically the only uefi patch we never backported to 16.1 | 16:33 |
| iurygregory | oh wow! | 16:33 |
| iurygregory | let me check things | 16:33 |
| TheJulia | and 16.1 worked just fine | 16:33 |
| dtantsur | you mean, grub-install plainly does not work on UEFI any more? | 16:34 |
| TheJulia | hmmm | 16:34 |
| TheJulia | dtantsur: well, that has been a thing for a little while now | 16:35 |
| iurygregory | TheJulia, I think the idea was to remove any booty entry that was duplicated | 16:35 |
| dtantsur | ah, you mean that line, not the whole patch #facepalm | 16:35 |
| dtantsur | well, I can assure you that duplicated labels can cause a lot of issues | 16:35 |
| iurygregory | we could have duplicated labels and this would remove | 16:35 |
| opendevreview | Merged openstack/ironic master: Fix various issues in the anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/814087 | 16:35 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Always include the oslo_log log file in ramdisk logs https://review.opendev.org/c/openstack/ironic-python-agent/+/815861 | 16:36 |
| TheJulia | Yeah, it is a conundrum | 16:36 |
| TheJulia | err, actually it did make it into 16.1 | 16:38 |
| TheJulia | freaky | 16:38 |
| TheJulia | but the code path utilization changed to leverage it and dedupe, since it was ironic1 previously | 16:38 |
| iurygregory | ouch D: | 16:39 |
| TheJulia | I'm thinking of moving the delete before the add | 16:41 |
| TheJulia | since which would re-establish the boot order eplicitly | 16:41 |
| iurygregory | so we would change the regex we need to use I think | 16:50 |
| iurygregory | we would remove any entry with *ironic*? | 16:50 |
| TheJulia | well, we use ironic if we dont' have a properly formatted default | 16:51 |
| dtantsur | and any fixed path on a removable media (thank you shim) | 16:51 |
| TheJulia | separate patch ;) | 16:52 |
| TheJulia | actually | 16:52 |
| iurygregory | "efibootmgr: ** Warning ** : Boot0004 has same label ironic" this was the warning we could get after adding the new entry | 16:52 |
| TheJulia | any ideas on identifying removable media from the list? | 16:53 |
| dtantsur | btw we're discussion it downstream with derekh and bfournie, and we think to engage the shim team | 16:54 |
| dtantsur | ironic can work around it, but the same issue affects pretty much any CoreOS ISO (and maybe more) | 16:54 |
| dtantsur | if we want to fix it in ironic, we need to do it on *any* action: inspection, cleaning, deploy | 16:54 |
| dtantsur | otherwise the machine won't reboot after the 1st action | 16:54 |
| iurygregory | wow | 16:55 |
| dtantsur | yeah, so probably evaluate_hardware_support or something like that | 16:55 |
| dtantsur | in other news, bloody CoreOS IPA takes more than 10 minutes to even start IPA in a nested VM | 16:58 |
| derekh | I still don't see how ironic can deal with this, any BM server with a boot entry pointing to the CDROM will cause the instruction to boot from CDROM to fail, so we can't run IPA to clean it (unless using PXE) | 16:58 |
| dtantsur | derekh: if the record is there, we cannot do anything. if it's just been created on this very start-up, we can purge it. | 16:58 |
| derekh | Also its described here https://storyboard.openstack.org/#!/story/2008763 | 16:58 |
| derekh | But its coreos (in the case I'm looking at) what started up and created the entry, what purges it ? | 17:00 |
| dtantsur | derekh: IPA when it starts (it won't help AI obviously) | 17:00 |
| dtantsur | I do agree that the real fix belongs somewhere between shim and coreos | 17:01 |
| derekh | ok | 17:02 |
| *** derekh is now known as derekh_afk | 17:04 | |
| * TheJulia gets totally derailed | 17:14 | |
| * dtantsur wonders if TheJulia is a Train | 17:15 | |
| TheJulia | dtantsur: I've already kind of discussed it with valpertha, it is wired in assertion, but maybe it won't if there is no csv file? | 17:15 |
| dtantsur | yeah, this is why I'm saying that the fix may be in coreos | 17:15 |
| TheJulia | dtantsur: no, trying to get more solar panels is a major headache | 17:15 |
| dtantsur | one would expect the opposite, given the climate situation.. | 17:16 |
| TheJulia | like... people don't listen and I know exactly what is required | 17:16 |
| TheJulia | we're at lowering our expectations point and we'll do the rest of the electrical work ourselves point | 17:16 |
| dtantsur | :( | 17:16 |
| dtantsur | if somebody has time today, my CoreOS IPA work could really use https://review.opendev.org/c/openstack/ironic-python-agent/+/815861/ and https://review.opendev.org/c/openstack/ironic-python-agent/+/815651 | 17:17 |
| dtantsur | good evening folks o/ | 17:18 |
| dtantsur | for those in the USA: I'm out tomorrow afternoon and on Monday, so see you on Tuesday | 17:19 |
| TheJulia | :) | 17:19 |
| NobodyCam | Top of the morning Ironic folks. | 17:37 |
| NobodyCam | I heard a while back (week or so back) about a SkyLake PXE issue? is there a bug I can follow along at home??? | 17:45 |
| TheJulia | bfournie: ^^ I'm drawing a blank on what the issue was actually leaning towards, do you remember? | 17:50 |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| -opendevstatus- NOTICE: mirror.bhs1.ovh.opendev.org filled its disk around 17:25 UTC. We have corrected this issue around 18:25 UTC and jobs that failed due to this mirror can be rechecked. | 18:44 | |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Delete EFI boot entry duplicate labels first https://review.opendev.org/c/openstack/ironic-python-agent/+/815899 | 18:56 |
| lmcgann | hey, im trying to control a node via the serial console but Im having some issues. if I telnet to the console url given by 'console show' and provide the node, I can see the output of the node booting. However, I dont seem to be able to control the node. And prompts about login dont appear as they usually do in the web console. Am I misunderstanding something? | 19:33 |
| TheJulia | lmcgann: so... console has to be directed to ttyS0 so could they not be on the image? | 20:33 |
| lmcgann | TheJulia: youre saying that I am able to read the output the bios makes but once the image boots, something in the image makes the OS not hook up to the right tty? | 20:42 |
| TheJulia | for serial, yes | 20:42 |
| cvstealth | When an instance is deployed on a node that is using a port group does the interface mapping always have to be calculated as user-data on the instance? Poking around the configdrive and in the metadata service with an instance in this configuraton I didn't see where any port/ip information was being made available. | 21:12 |
| TheJulia | cvstealth: I believe so, becaues it would come from the neutron ports. Nova has code internally to try and reconcile this as ports can still be in progress | 21:25 |
| TheJulia | it is supposed to be done at some point, but I'm not sure for bonded ports and how they get represented in neutron | 21:26 |
| cvstealth | TheJulia: In that case do you have to know what node your instance is going to land on before launching it? | 21:29 |
| TheJulia | cvstealth: not really, that is supposed to be resolved in placement | 21:31 |
| TheJulia | part of the reason netorking-baremetal feeds information into neutron if I understand the purpose correctly | 21:32 |
| *** dviroel|rover is now known as dviroel|rover|afk | 21:33 | |
| cvstealth | It looked like the configdrive contents were being generated prior to the deploy ramdisk being run then the networking-baremetal service was only inserting the neutron ports for the port group after the deploy phase. It read as a chicken and egg condition where I can't generate the user-data with the bonding configuration without know the mac address of the node that the instance was going to | 21:37 |
| cvstealth | land on. | 21:37 |
| TheJulia | for some reason that does sound right and I guess that is likely a legitimate egg because there is no way to really determine what NIC will get attached upon deployment and we still update the information neutron has | 22:07 |
| TheJulia | You may just be in the uncanny valley of port binding with bonds. I'm wishing sam betts was still around these parts, he would know off the top of his head | 22:08 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
