Monday, 2021-12-20

opendevreviewSteve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros  https://review.opendev.org/c/openstack/ironic/+/81539200:00
opendevreviewSteve Baker proposed openstack/ironic master: WIP use privsep to copy for loader_file_paths  https://review.opendev.org/c/openstack/ironic/+/82136300:00
opendevreviewSteve Baker proposed openstack/bifrost master: Support PXE network boot with grub  https://review.opendev.org/c/openstack/bifrost/+/80722000:25
*** pmannidi|Gone is now known as pmannidi\AFK00:34
*** pmannidi\AFK is now known as pmannidi|AFK00:34
opendevreviewSteve Baker proposed openstack/ironic master: WIP use privsep to copy for loader_file_paths  https://review.opendev.org/c/openstack/ironic/+/82136303:40
*** amoralej|off is now known as amoralej07:01
*** akahat is now known as akahat|ruck07:25
arne_wiebalckGood morning, Ironic!08:07
holtgrewe^gonegood morning08:09
*** holtgrewe^gone is now known as holtgrewe08:09
jingvarmorning08:12
rpittaugood morning ironic! o/08:18
jandersgood morning arne_wiebalck holtgrewe jingvar rpittau and Ironic o/08:36
rpittauhey janders :)08:36
arne_wiebalckhey janders holtgrewe jingvar rpittau o/08:36
rpittauhey arne_wiebalck :)08:36
holtgrewearne_wiebalck: \o -- I think I'm 99% there regarding the UEFI image. However it turns out that the root file system is mounted as readonly . /etc/fstab says `LABEL=root_fs / xfs defaults 0 1` and /pro/cmdline is `BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.49.1.el7.x86_64 root=LABEL=root_fs ro console=tty0 crashkernel=auto net.ifnames=0 console=ttyS0 console=tty0 console=ttyS0,115200 no_timer_check 08:50
holtgrewenofb nomodeset gfxpayload=text rd.auto=1` dmesg says `XFS (md126p3): ro->rw transition prohibited on unknown (0x4) ro-compat filesystem`.08:50
holtgreweHeh, maybe it's caused by ipa using CentOS 8 and me trying to use CentOS 7... and v8 is using a too recent features.08:55
opendevreviewAija Jauntēva proposed openstack/ironic bugfix/19.0: Fix redfish update_firmware for newer Sushy  https://review.opendev.org/c/openstack/ironic/+/82224109:10
arne_wiebalckholtgrewe: o/ your commandline does not have/need the raid auto-assemble?09:10
opendevreviewAija Jauntēva proposed openstack/ironic stable/xena: Fix redfish update_firmware for newer Sushy  https://review.opendev.org/c/openstack/ironic/+/82224209:10
arne_wiebalckholtgrewe: that should only be a problem the other way round: a c7 IPA installing a cs8 image09:11
holtgrewearne_wiebalck: actually it's my local Ubuntu 20 machine buidling the image.09:13
holtgrewearne_wiebalck: OK, can confirm that I got it working. If I want to create some tutorial-style documentation for what it takes to build an image, where is the right place to put it? This repository? https://opendev.org/openstack/diskimage-builder ?09:37
arne_wiebalckholtgrewe: I think it should go to the Ironic docs09:39
arne_wiebalckholtgrewe: https://opendev.org/openstack/ironic/src/branch/master/doc/source09:39
arne_wiebalckholtgrewe: we do not have a place yet for "best pratices / tips and tricks", but it is on my to-do list in fact09:40
arne_wiebalckholtgrewe: I think your recipe will help others a lot09:41
arne_wiebalck holtgrewe: it should probably be on the same level as "Troubleshooting FAQ"09:50
arne_wiebalckholtgrewe: I can try to put sth then you can add (if that sounds ok?). Not sure  I manage before the break, though.09:51
holtgrewearne_wiebalck: I have some experience with sphinx from other projects. No worries, I need to figure out 1-2 other things before.09:53
holtgrewecloud-init does not work yet and no "centos" user is created by default...09:54
* dtantsur rubs his eyes10:07
holtgreweAh, centos is created by cloud-init...10:09
dtantsurholtgrewe: re docs: I think updating https://docs.openstack.org/ironic/latest/user/creating-images.html is a good idea10:37
holtgrewedtantsur: OK, will look into this. It will probably take me until Thursday to get to it, though.10:38
dtantsurholtgrewe: it's a holiday season, don't rush :)10:39
dmelladodtantsur:  o/ I come to bother again, sorry xD10:40
dtantsurhey hey dmellado 10:40
dmelladoso, trying bifrost w/centos8 it afils when building the image10:40
dmelladoso not sure if this is a known issue10:40
dmelladohttps://paste.openstack.org/show/811765/10:40
dmelladoI tried setting export DIB_INSTALLTYPE_pip_and_virtualenv=package10:40
dmelladodidn't really make a difference10:41
dtantsurdmellado: do you see where the actual error is? I cannot find it10:41
rpittau I was going to ask the same10:41
dmelladome neither, that's why I thought it may be something obvious that I didn't understand10:42
dmelladoxD10:42
dtantsurdmellado: could it be that it just failed downloading the base image?10:42
dmelladohttps://paste.openstack.org/show/811766/10:42
dmelladothat's the full start10:42
dmelladostderr seems empty10:42
dtantsurdmellado: try running the same command manually?10:43
dmelladoI'll try to run the command manually and see10:43
dmelladoyeahj10:43
dmelladoa-ha10:44
dmelladonetwork10:44
dmelladoit's always dns 10:44
dmellado2021-12-20 10:43:49.941 | Getting /home/dmellado/.cache/image-create/CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2.tgz.lock: Mon Dec 20 04:43:49 CST 202110:44
dmellado2021-12-20 10:43:49.942 | Fetching Base Image10:44
dmellado2021-12-20 10:43:49.951 |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current10:44
dmellado2021-12-20 10:43:49.951 |                                  Dload  Upload   Total   Spent    Left  Speed10:44
dmelladoxD10:44
dtantsurit's never going to be not DNS10:45
dmelladoanyway, progressing, so let's see10:45
dmelladoyah, works10:46
dmelladosome network issue, now that it's cached it'll probably work10:46
rpittaucurrently in CI with dib built images I'm seeing this error  OSError: [Errno 101] ENETUNREACH but haven't had the time to investigate further10:46
rpittauthis happens on both debian and centos10:47
dmelladodoes it happens on the dib build phase?10:47
rpittauno, during deployment, the image builds fine10:47
rpittauthe patch is https://review.opendev.org/c/openstack/bifrost/+/82122310:47
rpittauthat fixes the issue with pip from source10:47
* dmellado fetching10:48
dmelladothanks!10:48
dtantsurfolks, we don't have the weekly meeting today, do we?10:48
rpittaudtantsur: we don't10:48
*** sshnaidm|afk is now known as sshnaidm11:07
dmelladodtantsur: also, is there any quick var or setting to avoid using /root/ for the image creation? that partition is quite full11:15
dtantsurdmellado: I suspect we run DIB with sudo..11:16
dmelladoprobably11:16
dmelladoI guess I'll have to check on the dib if there is any setting to use $whatever path11:17
dtantsuryep11:17
dmelladoon the good side, I'm working on another PR to improve a few things on the dnsmasq.j2 file xD11:17
dtantsurnice!11:18
* dmellado thinking about being cheap and just symlinking /httpboot11:21
dmelladoxD11:21
opendevreviewMerged openstack/bifrost master: Add support for root filesystem UUID customisation  https://review.opendev.org/c/openstack/bifrost/+/81978511:47
dmelladohttps://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/redhat-common/bin/extract-image#L4612:27
dmelladohmm so do you happen to know whether you can modify the TEMP file from bifrost?12:28
dtantsurdmellado: just set TMP_DIR?12:28
dmelladonvm, I'M BLIND12:29
dmelladotime for coffee12:29
dmelladoxD12:29
dtantsurcoffee++12:30
dmelladocaveat: my kid did wake me up at 6am12:30
dtantsurouch :(12:35
dmellado2021-12-20 12:45:04.605 | > Error: Unable to find a match: scsi-target-utils12:45
dmelladodoes this ring a bell when building the image?12:45
dmelladorpittau: does this happen on CI?12:45
dmelladoSeems like an old bug....12:46
dmelladohttps://bugs.launchpad.net/tripleo/+bug/186442712:46
dmelladois this a regression on you folks?12:49
rpittauMmm I don't see it in ci12:49
rpittauThe only issue in seeing is a network connection where the ipa ramdisk can't reach ironic 12:49
dmelladohttps://paste.openstack.org/show/811769/12:51
dmelladoseems that tgt gets converted to scsi-image-utils then boom12:51
rpittaudmellado: I'll check after lunch, I'm on my mobile at the moment :)12:53
dmelladono worries, thanks in advance!12:53
*** amoralej is now known as amoralej|lunch13:06
dtantsurwhy are we even trying to install it, I wonder13:06
rpittaudmellado: that's from running dib directly ?13:22
rpittaufrom what I can see the problem is in dib, it doesn't take into account that there's no scsi-target-utils for cnetos 813:24
rpittauso it's the same bug13:27
rpittauI believe "deploy-tgtadm" should not be included anymore13:29
holtgrewearne_wiebalck: is there a way to specify the target image size when using diskimage-builder based on an existing qcow2? My image build crashes like this https://pastebin.com/1Wf7B5ty13:35
holtgreweblock device config yaml is here https://paste.openstack.org/show/811771/13:36
dmelladorpittau: both running dib or bifrost, but seems to somehow have worked the 2nd time, let me ehcck13:48
opendevreviewDmitry Tantsur proposed openstack/ironic master: Automatically configure enabled_***_interfaces  https://review.opendev.org/c/openstack/ironic/+/82090914:11
*** amoralej|lunch is now known as amoralej14:12
jamesdentonhi dtantsur - do you know in what scenario (if any) where the inspection_network specified in ironic.conf is leveraged for in-band inspection?14:30
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] ImageCache: respect Cache-Control: no-store  https://review.opendev.org/c/openstack/ironic/+/82232914:47
dtantsurjamesdenton: hi, in this: https://docs.openstack.org/ironic/latest/admin/inspection.html#managed-inspection14:49
jamesdentonthanks - is managed triggered with 'openstack baremetal node inspect' vs 'openstack baremetal introspection start'? Just trying to determine the difference there. I have both methods working, but trying the former is using the ilo inspection interface (which doesn't appear to use network) and the latter doesn't seem to manipulate networking14:52
dmelladorpittau: I don't really see how can modify that in dib from bifrost14:54
dtantsurjamesdenton: well, it's a bit more subtle14:54
dtantsur'baremetal introspection start' is a low-level ironic-inspector command that always uses unmanaged inspection14:54
jamesdentonok14:55
dtantsur'baremetal node inspect' uses the configured inspection interface, which can be inspector14:55
dtantsuryour node clearly has 'ilo' inspection interface configured, which is out-of-band inspection14:55
dtantsurwhat you need is $ baremetal node set --inspect-interface inspector <node>14:55
dtantsuror put it as default_inspect_interface to your ironic.conf14:55
dtantsurthen `baremetal node inspect` will use ironic-inspector14:56
dtantsurjamesdenton: ^^14:56
rpittaudmellado: you can't, that is embedded in dib https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/ramdisk/element-deps14:57
jamesdentongotcha. i did try that, but get a 'notsupported' error, like this:  openstack.exceptions.NotSupported: The baremetal-introspection service for 10.20.0.11:None exists but does not have any supported versions. So i must be missing some configuration, or maybe a mismatch in interfaces14:57
jamesdentonthank you for the tips - i'll dig around on my side and see whats up14:59
rpittaudmellado: for centos8 that should probably be change into targetcli and iscsi-initiator-utils15:01
dtantsurjamesdenton: this means that ironic cannot talk to ironic-inspector. something misconfigured in keystone or in the [inspector] section.15:15
jamesdentonyes - i found ironic-inspector API wasn't UP due to a local issue with dnsmasq. I've fix that, thank you. 15:16
dmelladorpittau: but where is that change supposed to be around?15:16
dmelladoI assume in diskimagte-builder directly15:16
dmelladobut not sure about where the change is15:16
rpittaudmellado: yeah, probably the best place would be the centos element15:17
rpittaudmellado: this is where the pkg tgt gets remapped for redhat-based systems https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/redhat-common/bin/map-packages#L8515:19
jamesdentondtantsur do you know if networking-generic-switch is supported with inspection?15:20
dmelladoOh, I was there but the WARNING really scared me xD15:20
dmellado # This global list has been deprecated by the pkg-map element.  New15:20
dmellado# package mappings should go in pkg-map files inside each element. 15:20
dtantsurjamesdenton: should work IMO15:20
rpittaudmellado: yes, the  correct remap is done using the pkg-map element15:20
dtantsuryou'll need to make sure you have all the information configured on the ports in advance15:20
jamesdentonright, ok15:21
rpittaudmellado: probably here https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/redhat-common/pkg-map or in the centos/centos-minimal element15:28
dmelladorpittau: I see15:32
dmelladoI'll make a PR15:32
dmelladobut then there's this thing which creeps me out15:32
dmelladoalso busybox is not there? 15:32
dmellado2021-12-20 15:27:42.762 | Running install-packages install.15:32
dmellado2021-12-20 15:27:42.762 | + dnf -v -y install busybox15:32
dmellado2021-12-20 15:27:42.888 | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync15:32
dmellado2021-12-20 15:27:42.888 | DNF version: 4.7.015:33
dmellado2021-12-20 15:27:42.889 | cachedir: /tmp/yum15:33
dmellado2021-12-20 15:27:42.896 | User-Agent: constructed: 'libdnf (CentOS Linux 8; generic; Linux.x86_64)'15:33
dmellado2021-12-20 15:27:42.997 | repo: using cache for: appstream15:33
dmellado2021-12-20 15:27:43.009 | appstream: using metadata from Fri Dec 10 13:03:48 2021.15:33
dmellado2021-12-20 15:27:43.010 | repo: using cache for: baseos15:33
dmellado2021-12-20 15:27:43.022 | baseos: using metadata from Fri Dec 10 13:03:32 2021.15:33
dmellado2021-12-20 15:27:43.022 | repo: using cache for: extras15:33
dmellado2021-12-20 15:27:43.022 | extras: using metadata from Fri Dec 10 12:56:25 2021.15:33
dmellado2021-12-20 15:27:43.022 | Last metadata expiration check: 4:41:49 ago on Mon Dec 20 10:45:54 2021.15:33
dmellado2021-12-20 15:27:43.194 | No match for argument: busybox15:33
dmellado2021-12-20 15:27:43.199 | Error: Unable to find a match: busybox15:33
dmelladoxD15:33
dmelladoooops, I  meant to paste the paste15:33
dmelladonot this15:33
dmelladohttps://paste.openstack.org/show/811773/15:33
dmelladosorry, wrong paste xD15:33
holtgrewearne_wiebalck: I'm now quite happy with my CentOS 7 image. The next step forward for me would be to make this work for rocky linux... This appears to be a bit more involved, and there already is a patch here https://review.opendev.org/c/openstack/diskimage-builder/+/80580015:38
holtgreweSorry, is this even the right place to ask?15:38
holtgreweIt looks like the commenters have reservations against supporting RockyLinux for hosting openstack itself.15:39
holtgreweAll I want is to build a RockyLinux image for my baremetal compute hosts...15:39
opendevreviewMerged openstack/ironic bugfix/19.0: Fix redfish update_firmware for newer Sushy  https://review.opendev.org/c/openstack/ironic/+/82224115:40
dtantsurholtgrewe: #openstack-dib is a somewhat better channel for this15:41
dtantsurI know TheJulia has +2 rights on DIB, but that's it15:41
dmelladono, no busybox on centos815:41
dmelladoaweeesome15:41
dmelladoxD15:41
holtgrewedtantsur: thanks15:41
rpittaudmellado: you found out yourself :)15:41
arne_wiebalckholtgrewe: nice, congrats!15:41
arne_wiebalckholtgrewe: TFW when the instantiated node starts pinging is still great , even after all these instance creations :)15:42
dmelladois busybox *really* needed? xD15:42
dtantsurdmellado: I'm really curious how you ended up with busybox..15:43
dtantsurwhat's the command you're running?15:43
arne_wiebalckholtgrewe: I think the comments are more on the how than on the why FWIS. None the less, adding it to DIB means maintenance work, so I guess the question is whether there is sufficient demand. 15:44
arne_wiebalckholtgrewe: feel free to comment on the ticket that you would like/use it, I  guess there will be others to use it eventually 15:45
dmelladodtantsur: dib uses it for creating the centos8 image xD15:45
dmelladoramdisk-image-create -o /httpboot/deployment_image.qcow2 -t qcow2 centos vm enable-serial-console simple-init15:45
dmelladojust that xD15:45
holtgrewearne_wiebalck: I see. I was looking into using the centos element for installing rocky but it's missing python3...15:45
dtantsurdmellado: okay, don't use ramdisk-image-create15:46
dtantsurhonestly, we may even ask them to remove it15:46
dtantsur*always* use disk-image-create15:46
dmelladoack, thought it was going to be faster15:46
dmelladoswitching to disk-image-builder now15:46
dtantsurholtgrewe: a completely different approach would be to ask Rocky folks to publish images with a UEFI partition and linux-firmware included :)15:47
arne_wiebalckdtantsur: ++15:47
dtantsurDIB is cool, but it has a tiny team that maintains it15:48
holtgrewedtantsur: push it up up upstream :-D15:48
holtgreweIn the worst case, I'll go for maintaing my own elements.15:48
dtantsurwell, yes :)15:48
dtantsuryeah, that's an option as well15:48
dtantsurright now https://rockylinux.org/cloud-images is only about AWS. this is pretty bad :)15:49
rpittauI was looking at rocky linux as well but I stopped when I ended up there :)15:49
dtantsuraha, https://download.rockylinux.org/pub/rocky/8.5/images/ has a genericcloud image15:50
dtantsurI wish they linked to it somewhere15:50
rpittauand that became interesting again15:50
holtgreweSo far you (the OpenStack) team appear to be the most friendly and approachable folks... anyway, your tools look to be well extensible15:50
holtgreweSo I can just help myself.15:50
dtantsurI'm glad you have good experiences here :)15:51
jamesdentonthanks for the help dtantsur -- it took the right combination of things to get it going. Namely, using ipmi driver (not iLo) and (maybe) require_managed_boot=True15:52
dtantsurjamesdenton: sounds like a good start15:53
dtantsuryou can look into the iLO driver if you need any advanced features from it15:53
jamesdentoni don't, really. I just was trying to avoid flat networking for inspector15:53
jamesdentonwhich, seems like it's possible with the right configuration :)15:54
dtantsurjamesdenton: I definitely tried it myself a couple of years ago (with networking-ansible, which is pretty similar to networking-generic-switch)15:54
jamesdentontrying to get OSA's documentation and workflow in place15:54
jamesdentoni just need to figure out why this node is picking up a rogue IP via DHCP - i think i've got some cruft to cleanup15:56
jamesdentonthanks again15:56
opendevreviewDmitry Tantsur proposed openstack/ironic bugfix/19.0: Configure CI for bugfix/19.0  https://review.opendev.org/c/openstack/ironic/+/82233715:58
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent bugfix/8.3: Configure CI for bugfix/8.3  https://review.opendev.org/c/openstack/ironic-python-agent/+/82233816:01
rpittauI'm starting to wonder if the issue with dibipa integration job in bifrost is the python version16:01
dtantsurmmm?16:02
rpittauwe use debian in dibipa integration, with python 3.916:02
rpittauI just saw another error that may be related to that: Could not get real physical RAM from lshw: Expecting ',' delimiter: line 20 column 8 (char 477): json.decoder.JSONDecodeError: Expecting ',' delimiter: line 20 column 8 (char 477)16:03
rpittauin tinyipa that does not happen (python 3.6)16:03
dtantsurrpittau: sounds rather like something with lshw16:04
rpittaualso the issue with the network unreachable is quite strange, I can reproduce it locally16:04
rpittaudtantsur: yeah could be some lshw change16:04
dtantsur# lshw -quiet -json | jq .16:05
dtantsurparse error: Expected separator between values at line 11, column 816:05
dtantsurit produces invalid JSON on bullseye.....16:05
rpittauyay ?16:05
dtantsurabsolutely cursed16:05
dtantsurthis is version 02.18.85-0.716:06
rpittauon fc35 we have 02.19 and it works fine16:07
dtantsurFYI zigo ^^^16:07
dtantsurrpittau: I don't know what to suggest, to be honest. debian needs to fix their package.16:09
rpittauyeah, nothing we can really do there16:12
rpittauI'll go back bumping my head on the network issue16:12
dtantsur++16:12
dtantsuryeah, the output is complete garbage16:14
holtgreweIt looks to me that my machine is not properly cleaned but boots something else via DHCP... do I *need* a cleaning network or would that also work with flat network mode?16:14
dtantsurholtgrewe: you can run cleaning on the same network as everything else16:15
dtantsurIronic needs to know your cleaning_network because it needs to create ports in Neutron while cleaning16:15
holtgrewedtantsur: OK... then it must be something else.16:16
dtantsurholtgrewe: if you have a conflicting DHCP server, it may be an issue16:16
dtantsurI mean, DHCP is okay, but a PXE server is a no-go16:16
holtgreweWell, IT is failure after failure until something succeeds and then back to failure ;-)16:16
dtantsur:D16:16
arne_wiebalckholtgrewe: there is also https://docs.openstack.org/ironic/latest/user/creating-images.html I just realised16:30
*** amoralej is now known as amoralej|off16:33
dtantsurrpittau: tried reporting it to the debian's tracker, I hope it didn't go to /dev/null :)16:35
rpittaudtantsur: fingers crossed :)16:36
dtantsurrpittau, zigo, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=100202516:37
zigodtantsur: It's well known, however it's fixed in Unstable (and backports).16:52
rpittausee ya tomorrow folks! o/17:03
arne_wiebalckrpittau: o/17:05
dtantsurzigo: yeah, but we don't use unstable..17:13
dtantsuranyway, see you tomorrow everyone17:13
opendevreviewMerged openstack/ironic stable/xena: Fix redfish update_firmware for newer Sushy  https://review.opendev.org/c/openstack/ironic/+/82224217:19
opendevreviewMerged openstack/ironic-tempest-plugin master: [trivial] add python 3.9 in classifier  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/82166417:19
arne_wiebalckbye everyone o/17:26
admiyoRunning two Kolla setups, one old and working and one new and not working.  What is supposed to respond to a DHCP request when provisioning a node?  I thought it was ironic-dnsmasq, but that seems to get an entry in   /etc/dnsmasq/dhcp-hostsdir/0c:42:a1:49:df:4c that just reads  0c:42:a1:4a:2c:ec,ignore18:20
admiyoAnd that is the same on both clusters.  So, on the working cluster, what process should then handle the DHCP request?18:21
admiyonova_compute_ironic  ?18:25
stevebaker[m]good morning19:56
zigodtantsur: I tried to get the stable team to allow uploading the version of unstable in stable, but the problem is the new gtk3 build-dep ... :/22:25
zigodtantsur: Would it help if I upload to backports?22:25

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!