Monday, 2021-03-01

« Sunday, 2021-02-28 Index Tuesday, 2021-03-02 »
*** tosky has quit IRC00:11
*** dsneddon_ has joined #openstack-ironic00:23
*** dsneddon_ has quit IRC00:28
*** dsneddon_ has joined #openstack-ironic00:29
*** dsneddon_ has quit IRC00:35
*** zzzeek has quit IRC00:35
*** dsneddon_ has joined #openstack-ironic00:36
*** zzzeek has joined #openstack-ironic00:36
*** dsneddon_ has quit IRC00:57
*** dsneddon_ has joined #openstack-ironic01:54
*** dsneddon_ has quit IRC01:59
*** dsneddon_ has joined #openstack-ironic02:12
*** dsneddon_ has quit IRC02:17
*** rcernin has quit IRC02:19
*** dsneddon_ has joined #openstack-ironic02:19
*** dsneddon_ has quit IRC02:24
*** dsneddon_ has joined #openstack-ironic02:28
*** rcernin has joined #openstack-ironic02:35
*** dsneddon_ has quit IRC02:40
*** dsneddon_ has joined #openstack-ironic02:47
*** dsneddon_ has quit IRC02:52
*** dsneddon_ has joined #openstack-ironic02:53
*** dsneddon_ has quit IRC02:59
*** dsneddon_ has joined #openstack-ironic03:02
*** dsneddon_ has quit IRC03:07
*** dsneddon_ has joined #openstack-ironic03:09
*** uzumaki has joined #openstack-ironic03:34
*** Xuchu has joined #openstack-ironic03:58
*** mkrai has joined #openstack-ironic04:03
*** rcernin has quit IRC04:14
*** rcernin has joined #openstack-ironic04:21
*** dsneddon_ has quit IRC04:26
*** uzumaki has quit IRC04:51
*** rcernin has quit IRC05:01
*** rcernin has joined #openstack-ironic05:08
*** dsneddon_ has joined #openstack-ironic06:09
*** uzumaki has joined #openstack-ironic06:37
*** moshiur has joined #openstack-ironic06:48
*** mkrai has quit IRC06:59
*** mkrai has joined #openstack-ironic06:59
*** rcernin has quit IRC07:04
*** dsneddon_ has quit IRC07:14
*** moshiur has quit IRC07:14
*** moshiur has joined #openstack-ironic07:15
*** mkrai has quit IRC07:18
moshiurHi everyone, I'm trying to build the IPA image with OpenSUSE as a base. Running "ironic-python-agent-builder -o my-ipa-opensuse opensuse -v" is failing, saying that07:22
moshiurFailed to enable unit, unit openstack-ironic-python-agent.service does not exist.07:22
moshiurBut I saw earlier that it was installed. Is there something that I'm missing? Here is short the snippet of the log.07:22
moshiurhttp://paste.openstack.org/show/803039/07:22
*** k_mouza has joined #openstack-ironic07:23
*** k_mouza has quit IRC07:27
arne_wiebalckGood morning, ironic!07:34
arne_wiebalckmoshiur: Is using OpenSUSE as the base a must-have for you?07:38
*** uzumaki has quit IRC07:40
moshiurYes I need to use OpenSuse as the base image. I installed and added necessary packages and all the steps were passing. Now I am almost at the end of the building process. But getting this error. Thanks in advance.07:45
*** rcernin has joined #openstack-ironic07:48
*** rcernin has quit IRC07:53
arne_wiebalckmoshiur: I think there were issues with building IPA images with OpenSUSE in the past. rpittau|afk is probably the best to talk to.07:55
*** Qianbiao has joined #openstack-ironic08:00
moshiurDo I ping rpittau or continue in the group chat?08:04
*** rpittau|afk is now known as rpittau08:04
rpittaugood morning ironic! o/08:04
*** fmuyassarov has joined #openstack-ironic08:06
*** k_mouza has joined #openstack-ironic08:10
moshiurHi rpittau.08:11
rpittauhey moshiur08:12
arne_wiebalckmoshiur: group chat (so that everyone profits :))08:12
moshiurI am getting an error form IPA image building with openSuse, and I have added the logs in my previous messages. arne_wiebalck asked me to talk to you regarding this issue08:12
arne_wiebalckhey rpittau o/08:13
rpittauhey arne_wiebalck :)08:13
rpittauyeah, I was reading the backscroll08:14
*** k_mouza has quit IRC08:14
rpittaumoshiur: the ironic-python-agent-ramdisk element provided in ipa-builder is very centos/redhat centered, not sure when it was tested last time with suse08:19
rpittauanyway, to get the service name it uses svc-map, maybe the mapping is not done well on suse08:20
*** tosky has joined #openstack-ironic08:20
rpittaumoshiur: https://docs.openstack.org/diskimage-builder/latest/elements/svc-map/README.html08:20
jandersgood morning arne_wiebalck rpittau moshiur and Ironic o/08:21
rpittauhey janders :)08:21
*** rcernin has joined #openstack-ironic08:21
*** uzumaki has joined #openstack-ironic08:23
*** mkrai has joined #openstack-ironic08:24
moshiurgood morning janders08:25
*** rcernin has quit IRC08:26
rpittaumoshiur: as you can see under dib/ironic-python-agent-ramdisk/svc-map there's no definition for the service in suse/opensuse08:28
rpittauI'm pretty sure the service name is different08:28
rpittauyou may want to try and add that there08:28
openstackgerritVerification of a change to openstack/ironic failed: [trivial] Remove default parameter from execute  https://review.opendev.org/c/openstack/ironic/+/77734808:29
moshiurThanks rpittau, any idea what could be the service name in OpenSuse?08:31
*** uzumaki has quit IRC08:35
moshiurrpittau: there are some service name mentioned in the SVC-MAP readme file. is it the correct service name.08:35
moshiurdo I need to use also openstack-ironic-python-agent?08:36
rpittaumoshiur: the service names in the map file are for redhat/centos and debian08:36
rpittauthen you have a default08:36
rpittauyou need to add one for suse as explained in the svc-map docs08:37
rpittauotherwise it will use the default, which it doesn't seem to be correct08:37
moshiurok will try that. Thanks agin rpittau.08:38
moshiurI will give update.08:38
arne_wiebalckhey janders o/08:43
*** lucasagomes has joined #openstack-ironic08:50
*** rcernin has joined #openstack-ironic08:55
*** dougsz has joined #openstack-ironic08:55
*** ociuhandu has joined #openstack-ironic08:56
*** rcernin has quit IRC09:00
*** uzumaki has joined #openstack-ironic09:05
*** ociuhandu has quit IRC09:10
*** rcernin has joined #openstack-ironic09:11
*** uzumaki has quit IRC09:14
*** uzumaki has joined #openstack-ironic09:14
*** ociuhandu has joined #openstack-ironic09:15
*** rcernin has quit IRC09:16
*** uzumaki has joined #openstack-ironic09:20
*** derekh has joined #openstack-ironic09:31
*** k_mouza has joined #openstack-ironic09:53
openstackgerritwushiming proposed openstack/ironic master: remove unicode from code  https://review.opendev.org/c/openstack/ironic/+/77796910:07
*** jawad_axd has joined #openstack-ironic10:21
moshiurHi rpittau , the issue is solve by adding "suse:ironic-python-agent" to svc map.. :). it creates "my-ipa-opensuse.d"  "my-ipa-opensuse.initramfs". But unable to create the .kernel file.10:30
uzumakiGood morning folks! arne_wiebalck janders iurygregory o/10:31
moshiurrpittau: failed to detect the OS in 99-ramdisk-create file. now I am debugging the file.10:32
rpittaumoshiur: that's probably again because the dib element for ironic-python-agent is written for centos/rhel and debian/ubuntu10:33
jandersgood morning uzumaki o/10:34
uzumakihow you doing janders ?10:34
*** ociuhandu has quit IRC10:36
moshiurrpittau: I have the same opinion like you. I will figure it out. thanks10:36
*** ociuhandu has joined #openstack-ironic10:38
uzumakiWhat's the difference between CLUSTER_PROVISIONING_IP and PROVISIONING_IP in the ironic env file? (for metal3 bmo) they both seem to belong to the same prov. subnet but i'm confused10:38
arne_wiebalckrpittau: is the issue moshiur is struggling with something we should add to our docs? I think this is not the first time we get questions about using  opensuse as the base for the IPA image, and that may help with future questions :)10:39
*** ociuhandu has quit IRC10:43
rpittauarne_wiebalck: it's somwthing that was partly already discussed10:44
rpittauthe support for suse/opensuse was left out on purpose for different reasons, mainly because the support on diskimage-builder is experimnetal, not active10:44
*** rcernin has joined #openstack-ironic10:51
iurygregorygood moring janders arne_wiebalck rpittau uzumaki and Ironic o/10:51
rpittauwe could mention the svc-map element in the docs, but I'd rather not go into details for each distribution, that is something that is addressed on DIB side10:51
rpittauhey iurygregory :)10:51
arne_wiebalckrpittau: ok, thanks!10:51
arne_wiebalckhey iurygregory o/10:52
uzumakihow you doing iurygregory o/10:52
iurygregoryo/10:52
uzumakiWhat's the difference between CLUSTER_PROVISIONING_IP and PROVISIONING_IP in the ironic env file? (for metal3 bmo) they both seem to belong to the same prov. subnet but i'm confused10:52
*** mkrai has quit IRC10:56
uzumakihow I'm wondering how do I setup ironic authentication (say tls/basic)? can't seem to find any documentation on it10:58
*** fmuyassarov has quit IRC11:07
*** ociuhandu has joined #openstack-ironic11:15
*** ociuhandu has quit IRC11:19
openstackgerritMerged openstack/ironic master: [trivial] Remove default parameter from execute  https://review.opendev.org/c/openstack/ironic/+/77734811:27
*** k_mouza has quit IRC11:27
*** k_mouza has joined #openstack-ironic11:28
*** ociuhandu has joined #openstack-ironic11:31
*** rcernin has quit IRC11:31
rpittaulightning fast review :) -> https://review.opendev.org/c/openstack/ironic-inspector/+/77331611:33
iurygregory+W11:34
iurygregorytook 39sec (because gerrit is slow lol )11:34
rpittau:)11:34
*** rcernin has joined #openstack-ironic11:38
openstackgerritMerged openstack/ironic-python-agent master: Allow clean_configuration to run against full-device arrays  https://review.opendev.org/c/openstack/ironic-python-agent/+/77785411:42
iurygregoryuzumaki, I have the feeling they are the same O.o (I can't find any docs that would explain the difference..)11:44
*** ociuhandu has quit IRC11:44
uzumakiiurygregory, I know.. I just set them both to the same value and setup Ironic, nothing blew up.. Let's see if provisioning a node messes something up11:45
openstackgerritMerged openstack/ironic-inspector master: Add release version to release notes  https://review.opendev.org/c/openstack/ironic-inspector/+/77331611:45
iurygregoryuzumaki, yeah, you were able to figure out how to setup the auth?11:46
uzumakinope, I setup without any auth and i have a feeling ironic will start complaining pretty soon, when i'm provisioning something11:48
iurygregoryIt should support noauth11:52
*** rcernin has quit IRC11:55
*** ociuhandu has joined #openstack-ironic11:56
*** rcernin has joined #openstack-ironic12:00
*** ociuhandu has quit IRC12:01
janderssee you tomorrow Ironic o/12:11
iurygregorybye janders o/12:11
*** ociuhandu has joined #openstack-ironic12:12
*** yoctozepto has quit IRC12:17
*** yoctozepto has joined #openstack-ironic12:17
*** ociuhandu has quit IRC12:18
*** rcernin has quit IRC12:21
*** derekh has quit IRC12:23
*** derekh has joined #openstack-ironic12:25
*** ociuhandu has joined #openstack-ironic12:45
*** ociuhandu_ has joined #openstack-ironic12:47
*** ociuhandu has quit IRC12:50
*** mkowalski has quit IRC12:59
*** mkowalski has joined #openstack-ironic13:00
*** hoonetorg has joined #openstack-ironic13:03
*** uzumaki has quit IRC13:06
*** uzumaki has joined #openstack-ironic13:06
*** rh-jlabarre has joined #openstack-ironic13:12
uzumakiiurygregory, yup, noauth works fine13:14
*** anuradha1904 has joined #openstack-ironic13:25
*** ociuhandu_ has quit IRC13:26
*** ociuhandu has joined #openstack-ironic13:27
*** ociuhandu has quit IRC13:27
*** ociuhandu has joined #openstack-ironic13:30
*** ociuhandu has quit IRC13:35
iurygregoryif any core has some time https://review.opendev.org/c/openstack/sushy-tools/+/776660 =)13:40
TheJuliaDone, now to see if I don't spill my coffee this time13:47
TheJuliaDo we want to do a review jam this morning?13:48
TheJuliaAlso, good morning everyone!13:48
iurygregorygood morning TheJulia =)13:49
rpittaugood morning TheJulia :)13:49
QianbiaoGood morning TheJulia13:50
TheJuliaSorry for not being around on Thursday13:51
iurygregoryno worries!13:51
TheJuliaFamily stuff :(13:52
iurygregoryfamily 1st =)13:52
TheJuliaI've told many people the same thing :)13:53
openstackgerritMerged openstack/ironic-python-agent master: Use try_execute from ironic-lib  https://review.opendev.org/c/openstack/ironic-python-agent/+/77759113:54
TheJuliaI guess a review jam to try and go through outstanding items makes sense?13:54
iurygregoryrbac? =)13:55
mnaserGood morning Ironic — I’ve been having fun with software raid + UEFI and I have not been able to get a successful boot (mostly getting an error about grub device not found md127,3)13:55
mnaserAnd because of that it obviously doesn’t boot13:56
TheJuliahttps://review.opendev.org/c/openstack/ironic-lib/+/777594 is a really quick13:56
iurygregoryI think you spent the weekend trying to figure out this right mnaser ?13:56
TheJuliaiurygregory: likely good as well13:56
TheJuliaI'll start the call in a moment and we can keep it semi-freeform13:56
rpittausounds good13:57
TheJuliamnaser: where exactly are you seeing thta error13:57
TheJulia?13:57
mnaserI use dib to build the images and include block-device-efi and booting it without RAID works just fine so my issue is around booting EFI + software raid13:57
rpittaummm13:58
iurygregorygoing to get some water, will join in a few13:58
TheJuliamnaser: we will likely need insight into the raid configuration your trying to use13:58
mnaser"{'type': 'CommandExecutionError', 'code': 500, 'message': 'Command execution failed', 'details': 'Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.\nCommand: chroot /tmp/tmp_rk2i70n /bin/sh -c "grub2-install"\nExit code: 1\nStdout: \'\'\nStderr: "Installing for x86_64-efi platform.\\ngrub2-install: error: ../grub-core/kern/disk.c:237:disk `md127,3\' not13:58
mnaserfound.\\n".'}"13:58
TheJuliamnaser: whate version of the ramdisk are you using?13:58
TheJuliagah13:58
TheJuliayeah, there is your problem in general13:59
mnaserThat is showing up on the conductor — though it seems to not fail and move forward. I am running tip of Victoria agent (but Ussuri control plane)13:59
mnaserI definetly think that my image is the issue14:00
TheJuliain that case I think you should have the uefi loader fixes14:00
*** uzumaki has quit IRC14:01
TheJuliabut it means IPA is not finding the uefi loaders most likely14:01
mnaserMy OS is on the third partition on my image for what it’s worth14:01
mnaserOk I assume it’s not an issue when we’re not using RAID because my image already has all the EFI stuff14:02
*** lmcgann has joined #openstack-ironic14:02
TheJuliareview jam is happening on https://meetpad.opendev.org/ironic14:06
*** beekneemech is now known as bnemec14:08
TheJuliamnaser: if you can grab a full ironic-python-agent log saved to the conductor, it should provide insight. It is a bit of a weird and careful dance to try and get UEFI+RAID setup14:10
mnaser TheJulia wait the IPA logs get uploaded to the conductor?!?14:11
TheJuliamnaser: yes14:11
mnaserthat’s literally the most useful insightful thing ever. I did not know about this14:11
arne_wiebalckTheJulia: UEFI and RAID should work fine ... I hope :)14:12
mnaserIt is definitely tricky … I’ve slowly progressed through it14:12
mnaserhttps://docs.openstack.org/ironic-python-agent/latest/admin/troubleshooting.html#where-can-i-find-the-ipa-logs14:13
mnaseri guess it may be useful to add it there, cause im not sure where it shows up on the conductor14:14
arne_wiebalckmnaser: should not be much more tricky than BIOS and RAID14:14
arne_wiebalckmnaser: RAID is tricky :-D14:14
mnaseryeah RAID on boot drive exceptionally so14:14
mnaserthis is what i am running with https://www.irccloud.com/pastebin/F3FVuPat/14:14
mnaserand my dib command has been: `DIB_DEV_USER_PWDLESS_SUDO="yes" DIB_DEV_USER_AUTHORIZED_KEYS=/root/.ssh/authorized_keys disk-image-create -p mdadm centos vm block-device-efi dhcp-all-interfaces devuser -o centos-8-devuser-mdadm`14:15
*** rloo has joined #openstack-ironic14:16
arne_wiebalckI usually have system partition (for the image) and a payload partition ... this is why I did not notice the cleaning up whole devices issue14:16
mnaserarne_wiebalck: i did have a look at the CERN blog post too :)14:18
arne_wiebalck:)14:18
mnaserwhat my final goal is: 2x 240G drives as boot, 2x 960G drives as 'scratch' space14:18
arne_wiebalckoh, nice14:19
mnaserin raid-0, but i had some other issues with two devices right away14:19
arne_wiebalckI only have 2 drives in total14:19
mnaserso simplified it a little bit down14:19
openstackgerritPavlo Shchelokovskyy proposed openstack/ironic master: Allow ansible deploys to be fast-tracked  https://review.opendev.org/c/openstack/ironic/+/77800514:19
mnaseri don't seem to find any of the posted conductor logs though14:20
arne_wiebalckso, I usually have two RAID-1 partitions plus two RAID-1 or RAID-0 partitions14:20
arne_wiebalckdepending on the use case14:20
openstackgerritPavlo Shchelokovskyy proposed openstack/ironic master: Allow ansible deploys to be fast-tracked  https://review.opendev.org/c/openstack/ironic/+/77800514:21
arne_wiebalckmnaser: this is manual cleaning, no? are there logs stored on cleaning ... I thought this was only on deploy ... I may be wrong ... I usually check the logs on the node itself :)14:23
mnaserarne_wiebalck: well the cleaning / create_configuration RAID part works just fine14:23
mnaserthis is deploy time when grub2 gets installed where it blows up14:23
arne_wiebalckmnaser: ah, ok, true .. my bad14:23
arne_wiebalckmnaser: there should be logs! :-D14:24
mnaserarne_wiebalck: i think once i find out where the logs are, i will update https://docs.openstack.org/ironic-python-agent/latest/admin/troubleshooting.html#where-can-i-find-the-ipa-logs14:24
*** mjturek has joined #openstack-ironic14:25
*** uzumaki has joined #openstack-ironic14:28
*** derekh has quit IRC14:38
openstackgerritMerged openstack/sushy-tools master: New option to prevent libvirt defining boot order  https://review.opendev.org/c/openstack/sushy-tools/+/77666014:38
*** derekh has joined #openstack-ironic14:38
*** tzumainn has joined #openstack-ironic14:43
TheJuliawell, the conundrum is we shouldn't be running grub-install14:45
TheJuliaever.14:45
*** paras333 has joined #openstack-ironic14:46
*** ayoung has joined #openstack-ironic14:48
openstackgerritRiccardo Pittau proposed openstack/ironic stable/ussuri: [DNM] Test CI ussuri  https://review.opendev.org/c/openstack/ironic/+/77800714:54
*** sdanni has joined #openstack-ironic14:56
rpittauooook I think TheJulia you're right the latest ipa ramdisk published last week broke ironic ussuri CI14:59
* TheJulia quickly updates the agenda15:01
rpittauwhich is kind of weird though15:01
TheJulia#startmeeting ironic15:01
openstackMeeting started Mon Mar  1 15:01:22 2021 UTC and is due to finish in 60 minutes.  The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
TheJuliarpittau: it happens15:01
iurygregoryo/15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
TheJuliao/15:01
*** openstack changes topic to " (Meeting topic: ironic)"15:01
openstackThe meeting name has been set to 'ironic'15:01
rpittauo/15:01
ajyao/15:01
Qianbiaoo/15:01
rpioso\o15:01
arne_wiebalcko/15:01
bdoddo/15:01
*** stendulker has joined #openstack-ironic15:01
erbarro/15:01
stendulkero/15:01
TheJuliaOur agenda is on the wiki as always.15:01
TheJulia#link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting15:01
TheJuliaIt doesn't look like there is much to specifically talk about so I suspect today will largely be status updates.15:02
TheJulia#topic Announcements / Reminders15:02
*** openstack changes topic to "Announcements / Reminders (Meeting topic: ironic)"15:02
TheJulia#info This week is ?R-6?, meaning we need to release Sushy/Ironic-lib/metalsmith this week.15:03
TheJulia#info Outstanding sushy/ironic-lib patches that are in the gate, so if all goes well we should be able to take care of that tomorrow15:03
TheJulia#info Next Week, python client library releases are due.15:03
rlooo/15:03
TheJuliaDoes anyone have anything to announce or remind us of this week?15:04
iurygregoryI've sent the email you asked last week TheJulia http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020774.html =)15:04
TheJulia\o/15:04
iurygregoryI will add a doodle so people can choose slots so I can do the 101 XD15:05
TheJulia++15:05
TheJuliaThank you iurygregory15:05
iurygregorynp!15:05
TheJulia\o/ now we can skip action items from the prior week15:05
iurygregory<insert happy dance>15:05
TheJuliaWell, seems like there are no more announcements to be made this week.15:07
TheJuliaSince we already covered the one action item, I guess we can review status15:09
TheJulia#topic Review subteam status reports15:09
*** openstack changes topic to "Review subteam status reports (Meeting topic: ironic)"15:09
TheJulia#link https://etherpad.opendev.org/p/IronicWhiteBoard15:09
TheJuliaStarting at line 29715:09
TheJuliait... *looks* like we're good on NVMe support15:10
rpittauyeah, changes merged, probably some follow up/docs next but the big part is done15:11
TheJuliajanders: Question for when your up: Do we need any doc changes. I'm assuming yes.15:11
TheJuliaarne_wiebalck: I see you +2'ed the node db history change15:12
TheJuliaarne_wiebalck: does it look like we might be able to make it for some of that work this cycle?15:12
arne_wiebalckTheJulia: yes, I had a look earlier15:12
arne_wiebalckTheJulia: erm ... :-)15:13
arne_wiebalckTheJulia: kaifeng is really doing the work tbh15:13
TheJuliaYeah15:13
TheJuliaI don't think it will make it in Wallaby15:13
arne_wiebalckthat seems tight, I agree15:14
TheJuliaWell, a good Xena feature then :)15:14
arne_wiebalckso, we keep it out for now?15:14
arne_wiebalckI mean the DB change?15:14
arne_wiebalckdoes it make sense to merge it now when all the rest is still in the making?15:14
TheJuliaI *suspect* it is likely okay it not preferred to get the db change in15:14
TheJuliabut15:15
TheJuliawithout the whole api surface we don't know how it will all stack yet15:15
arne_wiebalckI would think so, too15:15
* TheJulia notes the status for redfish raid15:15
rloo++ please don't merge the DB change w/o the rest :)15:15
TheJuliaiurygregory: I guess privsep is also moving to xena?15:15
arne_wiebalckright, but adding sth non-complete has  a higher risk that we introduce sth we need to change later15:16
arne_wiebalckrloo: ++15:16
TheJuliayeah15:16
TheJuliaokay, then we punt it to Xena then15:16
iurygregoryTheJulia, yeah, rpittau was able to help me a bit (but I don't think we should land only initial support..)15:16
TheJuliaiurygregory: I concur, noting.15:16
* TheJulia marks the json to yaml stuff as done15:17
TheJuliaajya: thank you for the update onc onfig molds15:17
TheJuliarpittau: zer0c00l: would review jaming on the anaconda deployment interface make sense?15:18
arne_wiebalckTheJulia: ++15:19
rpittauTheJulia: probably15:19
* TheJulia adds lazy loading node attributes to ironic-week-prio15:19
TheJuliaAre we good to move on to priorities for the coming week?15:20
rpittaulet's15:20
arne_wiebalckTheJulia: I was about to propose that, ty.15:20
TheJulia#topic Deciding on priorities for the coming week15:20
*** openstack changes topic to "Deciding on priorities for the coming week (Meeting topic: ironic)"15:20
arne_wiebalckShall we remove lines 571-595?15:20
arne_wiebalckOr even to 609?15:21
TheJulia#link https://review.opendev.org/q/hashtag:ironic-week-prio+status:open15:21
openstackgerritMerged openstack/sushy master: Follow up TaskMonitor refactor  https://review.opendev.org/c/openstack/sushy/+/77773315:21
TheJuliaarne_wiebalck: from the whiteboard?15:22
arne_wiebalckyep15:22
iurygregorywe can also filter if the patch has verified +1 =)15:22
iurygregoryI can paste the link in the whiteboard15:22
TheJuliaarne_wiebalck: I'm not sure I understand why15:22
TheJuliaPlease keep in mind, i was MIA part of last week and I've been mentally occupied with family stuff as of recent15:23
*** uzumaki has quit IRC15:23
*** uzumaki has joined #openstack-ironic15:23
arne_wiebalckTheJulia: The bare metal bit serves no purpose, does it?15:23
arne_wiebalck*bare metal SIG15:23
TheJuliaAhh, Interesting15:23
TheJuliamy line numbers were off15:24
arne_wiebalckAnd the redfish profiles is duplicated in the Wallaby priorities.15:24
TheJuliaSeems reasonable15:24
TheJuliaIf you wouldn't mind, go ahead15:24
arne_wiebalckrpioso: any objections?15:24
rpiosoarne_wiebalck: Nope15:25
* TheJulia removes merged items from the list15:25
TheJuliaAny new items to add that people are aware of to the review priorites for the week?15:25
arne_wiebalckrpioso: I will check one more if we need anything for "historical reasons" :)15:26
arne_wiebalck*once15:26
*** mkrai has joined #openstack-ironic15:26
rpiosoarne_wiebalck: Thank you :-)15:26
arne_wiebalckrloo: I updated on the DB loading patch, pls have  a look if that is what you had in mind15:27
rlooarne_wiebalck: ++ on my list for today/soon :)15:28
arne_wiebalckrloo: ty15:28
TheJuliaany objections to adding the anaconda deploy stuffs to the priority15:29
arne_wiebalcksounds good!15:29
*** Qianbiao has quit IRC15:29
*** moshiur has quit IRC15:31
TheJuliaI've added a few items under new items15:33
TheJuliaAny objections to these?15:33
rpittaushould be fine15:33
TheJuliaOkay, I can add the tag to those15:34
openstackgerritMerged openstack/ironic-lib master: Explicit execute arguments  https://review.opendev.org/c/openstack/ironic-lib/+/77759415:34
TheJuliaAre we good to proceed?15:35
iurygregory++15:35
TheJuliaNext up would be Baremetal SIG15:35
rpittauonwards!15:35
TheJulia#topic Baremetal SIG15:35
*** openstack changes topic to "Baremetal SIG (Meeting topic: ironic)"15:35
TheJuliaarne_wiebalck: anything to note this week?15:35
arne_wiebalckWe have iurygregory next week on the "Ironic Prometheus Exporter"15:35
openstackgerritMerged openstack/python-ironicclient master: Add tests for 'baremetal port create' command  https://review.opendev.org/c/openstack/python-ironicclient/+/33288015:35
TheJuliaAwesome15:35
arne_wiebalckAnd a discussion on PTG input from operators ... maybe :)15:36
TheJuliaOhh, Interesting15:36
TheJuliaHave we started the twitter advertising campaign ?15:36
arne_wiebalckI started with a mail today, no twitter yet.15:36
arne_wiebalckMaybe we could also ask the foundation?15:36
TheJuliaokay, I need to dig up the operators group twitter tag15:36
TheJulia++15:37
iurygregorywithout link for zoom in the tweet =)15:37
TheJuliaThat is an awesome idea15:37
arne_wiebalckThe Large Deployment SIG moved to talks as we do.15:37
TheJuliaiurygregory: yeah, no tweeting zoom links :(15:37
TheJuliaI saw that, which they did tweet15:37
arne_wiebalckThey had 30 participants.15:37
TheJuliaNICE!15:37
iurygregoryawesome!15:37
arne_wiebalckQuite a good outcome I think15:37
TheJuliaWell, I guess we can proceed to Open Discussion then :)15:38
TheJuliahttps://twitter.com/osopsmeetup15:38
TheJulia#topic Open Discussion15:39
*** openstack changes topic to "Open Discussion (Meeting topic: ironic)"15:39
TheJuliaSo what things to discuss?15:39
rloohow beautiful are crickets?15:41
rpittaunot a discussion topic (it could become one though), tinyipa ramdisk in ussuri is broken15:41
TheJuliacrickets are awesome... sometimes.15:42
TheJuliaHow badly broken are we talking?15:42
rpittauthe image was rebuild and published using tinycore 11.1, which we don't support in ussuri because the CI is based on bionic (kernel too old)15:43
openstackgerritMerged openstack/sushy master: Initial support for secure boot databases  https://review.opendev.org/c/openstack/sushy/+/77339715:43
rpittauthat means all the jobs that somehow need to access the ramdisk with chroot won't work15:44
rpittaufor example ironic-standalone15:44
iurygregorycan't we pin the image we will use?15:44
iurygregoryO.o15:44
rpittauiurygregory: we do already15:44
TheJuliarpittau: because we're building the image?15:45
TheJuliain those jobs, i.e. ironic-standalone?15:45
rpittaufunny enough, if we were building the image we would be fine15:45
iurygregorywoot?!15:45
rpittauthe problem is that we download the image and then we chroot into it to modify it15:45
rpittauand we download the ussuri one that was rebuilt with 11.115:45
TheJuliarpittau: where do we modify it at?15:45
rpittauhttps://945835ad432887103cff-08f6dc72eccd53018dd797f31cdfe820.ssl.cf1.rackcdn.com/777430/1/check/ironic-standalone/7ab00ea/job-output.txt15:46
JayFrpittau: do we need to publish an updated ussuri ramdisk using master IPA-builder?15:46
rpittauwe access it to modify the configure ssh15:46
iurygregorymaybe we can ping the job to publish the image to an old IPA-builder...15:46
iurygregoryoverride-checkout15:46
rpittauJayF: we should not use ipa-builder master, we pinned jobs to 2.2.0 but somehow it got published with something more recent15:46
iurygregoryO.o15:47
TheJuliaso that job is building it15:47
iurygregory*magic*15:47
rpittauwhat happened is that a change merged on the 28th15:47
JayFah understand, problem is in the other direction15:47
rpittauand triggered a rebuild15:47
* JayF wonders if new pip resolver made it not-respect our package limits15:47
rpittauI'm not sure is on us to be honest15:47
TheJuliaoh15:47
TheJuliait is adding an ssh key15:47
TheJuliaugh15:47
rpittauyeah15:48
TheJuliaWhy do we need this?15:48
rpittauit's to test if we can access the server, it's a tempest test if I'm not mistaken15:49
rpittaubut still, the problem is the image publishing with the wrong version15:49
iurygregoryironic-python-agent-build-image-tinyipa will publish the image right?15:49
iurygregoryops publish-openstack-python-branch-tarball15:50
rpittauiurygregory: on our side we publish with ipa-builder15:50
TheJuliaMaybe instead of downloading, we just let the image build it completely for the job itself? At least to unblock the main ci for ussuri?15:50
TheJuliaFeels like this quickly becomes a chicken/egg problem15:50
rpittaubut there's an automated job that publish the images, I think it's on infra side15:50
iurygregoryon ipa we have     post:15:51
iurygregory      jobs:15:51
iurygregory        - ironic-python-agent-build-image-tinyipa15:51
iurygregory        - ironic-python-agent-build-image-dib-centos815:51
JayFI will echo the concern from rpittau that we not only fix CI; we fix the image for users who may be consuming built IPA images for Ussuri.15:51
rpittauJayF: yeah, even if it's tinyipa15:51
TheJuliaJayF: agree15:51
JayFIt being TinyIPA is a good point... that actually makes me worry less about it15:51
JayFhonestly, other than for internal CI use, we'd probably be better off not publishing TinyIPA images at all15:52
rpittauJayF: I think there's no choice :)15:52
TheJuliaYeah, we need the images published because if we buitl every time we would have a higher failure rate due to some of the clouds due to elongated job times and connections initated.15:53
TheJuliaso basically we know the build did something wrong when it came time to publish15:55
rpittauwe can probably see if pinning on the build image jobs work15:55
TheJuliabut we don't know what exactly until we find the logs?15:55
JayFI did want to make one request before open discussion shut down -- https://review.opendev.org/c/openstack/ironic-specs/+/777172 is a spec I've written about removing the need for conductor->IPA communication for the direct driver. I'd appreciate at least one non-VzM core to give it a look over if possible -- I'll be implementing this (downstream) over the next two weeks and would like to15:55
JayFmake sure what I build will be in the ballpark of an acceptable upstream design15:55
rpittauthe ipa jobs that iurygregory mentioned are related to master ipa-builder, so pinning them should work15:55
iurygregoryyep15:56
TheJuliapinning on the stable branch correct?15:56
iurygregoryyes15:56
TheJuliasounds like a plan then15:56
JayFIf master IPA-builder can't build a working image for ussuri, it makes me wonder if that needs to be a repo that starts getting branched like our others.15:56
TheJuliasounds like somthing that needs to be double check-ed elsewhere, fwiw15:56
rpittauon ipa-builder tag 2.2.015:56
TheJuliaI thought we had to code to auto-checkout the right version on branches15:57
rpittaunot for ipa-builder15:57
TheJuliaJayF: added to the ironic-week-prio list15:57
JayFTheJulia: Thanks!15:57
iurygregorybranches would be good, tag would also work (we probably need to have documentation with the versions...)15:58
TheJuliaiurygregory: well, there is central docs for that with releases15:58
*** k_mouza has quit IRC15:59
*** k_mouza has joined #openstack-ironic15:59
iurygregoryI will try to take a look at it15:59
TheJuliaokay16:00
TheJuliaOkay, anything else to discuss today?16:02
rpittauI have a patch ready to go for IPA ussuri to pin ipa-builder to 2.2.0, 2 minutes will be up16:02
iurygregoryrpittau, ack16:02
TheJuliaokay. Thanks everyone!16:02
* iurygregory brb lunch time16:03
TheJulia#endmeeting16:04
*** openstack changes topic to "Bare Metal Provisioning | Status: http://bit.ly/ironic-whiteboard | Docs: http://docs.openstack.org/ironic/ | Bugs: https://storyboard.openstack.org/#!/project_group/75 | Contributors are generally present between 6 AM and 12 AM UTC, If we do not answer, please feel free to pose questions to openstack-discuss mailing list."16:04
openstackMeeting ended Mon Mar  1 16:04:03 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:04
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ironic/2021/ironic.2021-03-01-15.01.html16:04
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ironic/2021/ironic.2021-03-01-15.01.txt16:04
openstackLog:            http://eavesdrop.openstack.org/meetings/ironic/2021/ironic.2021-03-01-15.01.log.html16:04
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent stable/ussuri: Pin version of ipa-builder when publishing image  https://review.opendev.org/c/openstack/ironic-python-agent/+/77802116:05
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent stable/ussuri: Pin version of ipa-builder when publishing image  https://review.opendev.org/c/openstack/ironic-python-agent/+/77802116:08
rpittauI'm wondering if we should do that in ipa-builder side16:09
rpittaummmm16:09
*** frigo has joined #openstack-ironic16:10
rpittaulet's see how it goes and then we'll see16:11
openstackgerritMerged openstack/ironic master: Switch iLO and iRMC to the new secure boot framework  https://review.opendev.org/c/openstack/ironic/+/77012216:16
openstackgerritVerification of a change to openstack/python-ironicclient failed: Implement client commands for node indicators  https://review.opendev.org/c/openstack/python-ironicclient/+/77656916:17
openstackgerritMerged openstack/ironic master: Initial Project scoped tests  https://review.opendev.org/c/openstack/ironic/+/77245116:18
*** frigo has quit IRC16:24
*** frigo has joined #openstack-ironic16:26
*** stendulker has quit IRC16:26
frigohello Ironic!16:27
TheJuliagreetings16:27
frigoI would be interested in having s3 support here https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/image_utils.py#L173 - do you think this is interesting for the community ?16:28
TheJuliamnaser: did you find the ramdisk log?16:28
mnaserTheJulia: i did not ^^'16:28
TheJuliafrigo: sounds reasonable for some16:28
mnaserwe do run it inside k8s with osh, so it is possible that we may need to tweak a few things16:29
TheJuliamnaser: /var/log/ironic/deploy_logs I think16:29
TheJuliayeah, your going to need to look inside the containers then16:29
mnaserhttps://github.com/openstack/ironic/blob/8604f84fd7bda4e30d3f07005c4901f3662303a7/releasenotes/notes/collect-deployment-logs-2ec1634847c3f6a5.yaml16:29
TheJuliasince this is last resort debugging stuffs16:29
iurygregoryTheJulia, if you have some time today https://review.opendev.org/c/openstack/releases/+/778023 =)16:30
TheJuliaokay, slightly off on the path :)16:30
mnaserTheJulia: so the fact that this doesn't fail the deployment16:30
mnaseris probably why im not seeing anything show up16:30
TheJuliaugh...16:30
TheJuliayeah16:30
mnaseri will flip it to 'always'16:30
TheJulia++16:30
mnaseralso, while i wait for that to megrge, i will update the ipa docs16:30
mnaseras they don't mention that you can get the logs there16:30
TheJulia<316:30
frigoTheJulia I'm going to code a little something, probably it will end up adding a dependency to boto3 and a few ifs in the file. Not sure it will meet your quality standards (also I opened https://storyboard.openstack.org/#!/story/2008644 but really, s3 is what I wished)16:32
TheJuliafrigo: fwiw, if you would add an RFE to storyboard.openstack.org against openstack/ironic, it seems reasonable enough to approve.16:32
*** mkrai has quit IRC16:32
TheJuliaerrr16:32
TheJuliahmm16:32
*** mkrai has joined #openstack-ironic16:32
TheJuliaoh, good, boto3 is APL216:32
TheJuliafrigo: seems like your ahead of me this morning16:33
arne_wiebalckmnaser: https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#retrieving-logs-from-the-deploy-ramdisk16:33
mnaserarne_wiebalck: ah, i was looking at the IPA logs16:33
mnasers/logs/docs/16:34
mnaseri will link to that16:34
arne_wiebalckmnaser: thanks!16:34
frigowill do. THANKS!16:36
* TheJulia goes and figures out something for breakfast16:37
openstackgerritMerged openstack/bifrost master: Add basic upgrade jobs  https://review.opendev.org/c/openstack/bifrost/+/75037616:37
openstackgerritMohammed Naser proposed openstack/ironic-python-agent master: Added comment about IPA logs being uploaded to Ironic  https://review.opendev.org/c/openstack/ironic-python-agent/+/77803116:39
*** frigo has quit IRC16:39
mnaseri cant really come up with great language so eh ^16:39
TheJuliamnaser: maybe s/on/upon/ ?16:40
mnaserTheJulia: repushing!  gerrit is slooow16:41
openstackgerritMohammed Naser proposed openstack/ironic-python-agent master: Added comment about IPA logs being uploaded to Ironic  https://review.opendev.org/c/openstack/ironic-python-agent/+/77803116:41
arne_wiebalckmnaser: I think IPA logs on the conductor is only true for deployment, not for cleaning. TheJulia ?16:41
arne_wiebalckmnaser: While for cleaning the rest of the section applies.16:42
TheJuliait is both afaik16:42
TheJuliabecause the logs don't have an instance_uuid in the name :)16:42
mnaserwell now that i have it set to always and i am about to rm an instanec which will be autocleaned16:43
mnaseri will let you know :p16:43
arne_wiebalckTheJulia: mine do ... maybe I misconfigured things :)16:45
arne_wiebalckTheJulia: like: only on deploy16:45
mnaserarne_wiebalck: cause you have an excellent cloud that never fails to clean? :-P16:45
arne_wiebalckmnaser: ha! I wish I knew how often I ran "maintenance unset, manage, provide" in the past years!16:46
mnaserthe true test of patience16:47
TheJuliaarne_wiebalck: legitimate failures or bugs?16:47
arne_wiebalckTheJulia: there are legitimate failures? :-P16:48
TheJuliaWell, everything is a defect until diagnosed :)16:48
arne_wiebalckTheJulia: legitimate: none working BMCs, RAID cleaning errors, PXE timeouts, boot issues, ...16:49
TheJulia:(16:49
*** mkrai has quit IRC16:51
arne_wiebalckit seems introspection logs go there as well16:51
mnaserTheJulia / arne_wiebalck: /var/log/ironic/deploy# ls => 2d795742-bce8-47a7-86e6-6b8e3567c8ea_cleaning_2021-03-01-16-50-52.tar.gz16:51
TheJulia\o/16:51
mnaseri have ipa-debug=1 too16:52
mnasertime to get some real debug output!!!16:52
arne_wiebalckI don't have these files.16:52
mnaseri did just set it  to =always16:52
TheJuliathe type naming, cleaning or deploy was added in victoria I believe16:52
* TheJulia look sat the number of hours of calls tomorrow and cringes16:53
arne_wiebalckoh, that would explain it as well16:53
arne_wiebalckmnaser: which option are you referring to?16:53
*** gyee has joined #openstack-ironic16:53
mnaserdeploy_logs_collect: always16:53
* arne_wiebalck goes and checks the config ...16:53
arne_wiebalckyep, I have that16:54
arne_wiebalckwas the sending of cleaning logs added separately?16:55
mnasernope but maybe its an ipa thing?16:55
mnasertime to read the code while i wait for pxeboot16:55
arne_wiebalckI seriously only have deploy logs ... I can tell as I have one per hour and that is from Rally16:55
TheJuliaNow... where in the world do I need to restart my brain this morning16:56
mnaserarne_wiebalck: i assume you use autocleaning?16:56
mnaserarne_wiebalck: https://github.com/openstack/ironic/commit/d31e71a736e18c4ec403d411bd1514808e9529a7 ?16:57
mnaserlooks like that was victoria16:57
mnasermaybe backported to tip of train16:57
mnaserhttps://review.opendev.org/c/openstack/ironic/+/73077116:57
mnasermaybe just dont have that commit? :)16:58
openstackgerritMerged openstack/bifrost master: Make DHCP inventory handling more flexible  https://review.opendev.org/c/openstack/bifrost/+/76792916:58
*** lucasagomes has quit IRC16:59
rpittaugood night! o/17:00
*** rpittau is now known as rpittau|afk17:00
mnaseroh i got another error this time17:02
mnaserinteresting17:02
mnaserTheJulia / arne_wiebalck: http://paste.openstack.org/show/803111/17:03
mnaserwait,17:03
mnaserpaste trimmed it17:04
openstackgerritJulia Kreger proposed openstack/ironic master: secure-rbac - minor follow-up for project scoped tests  https://review.opendev.org/c/openstack/ironic/+/77803317:04
mnaserTheJulia / arne_wiebalck: https://gist.githubusercontent.com/mnaser/d272c65868fa73a060e34650c9ebecd3/raw/93733579b6f9e3d79cd48e9bc4c6343d56ada56a/gistfile1.txt17:05
TheJuliaumm... what partition table is inside the image?17:06
TheJuliaoh wait17:06
TheJuliahjmm17:06
mnaserman did i mess up and put the wrong rootfs uuid17:07
mnaserfor that imgae17:07
mnaser`cat journal  | grep ironic-python-agent | grep tmp_vqwaf14` only shows umount but no mounts17:08
mnasermaybe the failure handling is unmounting thing that may have never been mounting at that point17:08
mnasercause it likely failed before mounting17:09
TheJuliaCould not create partition 3 from 34 to 1126433\nCould not change partition 3's type code to ef00!\nUnable to set partition 3's name to 'uefi-holder-0'!\nError encountered; not saving changes.17:09
mnaseryeah i wasnt running into that with another image i was using before17:09
mnaseri could reuse that other image which was giving me the couldnt install but it still seems.. related17:10
openstackgerritDerek Higgins proposed openstack/ironic-python-agent master: Increase the memory limit for qemu-img  https://review.opendev.org/c/openstack/ironic-python-agent/+/77803517:10
mnaserTheJulia: `sgdisk -n 0:34s:+550MiB -t 0:ef00 -c 0:uefi-holder-0 /dev/sda` -- shouldn't this run against /dev/md127 ?17:11
mnaserfull transparency: i have no _no clue_ how anything about this UEFI stuff works, its dark magic17:11
TheJuliaso uefi partitions can't be on the raidset17:11
TheJuliaso it should be the base devices17:12
TheJuliamnaser: if you want, we can jump on meetpad real quick and I can give you a 10,000 ft view of how it works17:12
mnaserTheJulia: i can be free in a minute or two but i am attending the k8s / tc meeting thingy we've had setup at 12:3017:13
TheJuliaI'll be quick17:13
TheJuliaConnected, join https://meetpad.opendev.org/ironic when ready17:14
iurygregoryI love the 30sec to send the patch17:19
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: Align the indicators API with the spec  https://review.opendev.org/c/openstack/ironic/+/77654017:19
openstackgerritMerged openstack/ironic master: Update oslo.policy requirement to version 3.6.2  https://review.opendev.org/c/openstack/ironic/+/77430717:25
*** frigo has joined #openstack-ironic17:28
TheJuliaiurygregory: heh17:28
TheJuliaarne_wiebalck: figured it out, it is use of MAX + software raid + UEFI17:28
TheJuliaarne_wiebalck: so mnaser is going to dial it back a little to like 220G and retry17:28
arne_wiebalckTheJulia: there is a bug?17:28
TheJuliaLikely17:29
arne_wiebalck"nice" !17:29
TheJuliaIPA should likely just subtract 550G from MAX17:29
arne_wiebalckoh!17:29
TheJuliaerr17:29
TheJulia550MB17:29
arne_wiebalckisn't the start sector set accordingly?17:29
TheJuliaactually, ~552 to preserve spacing17:29
arne_wiebalckhmm, with no partitions ...17:30
TheJuliait doesn't look like it offhand17:30
arne_wiebalckthat is totally possible17:30
TheJuliaalso with gpt, the partition can be anywhere on the disk physically17:30
arne_wiebalckright, but it is created upfront at the start IIRC17:30
arne_wiebalckthere should always be a 550MB partition to which the bootloader is relocated17:31
TheJuliaLooks like we're trying to do it in _install_grub2's entire process17:31
arne_wiebalckbut: I think I never tested UEFI RAID with a single MAX partition17:31
TheJuliaand there is no other partition on the disk from what the log suggests17:31
arne_wiebalckbut I do use MAX for the second one all the time17:32
arne_wiebalckthinking about it, even MAX creates partitions17:32
arne_wiebalckshould, it *should* create partitions :-D17:33
TheJuliabut not uefi holder partitions I think :)17:34
frigoout of curiosity, did you guys try some hardware RAID lately? I managed to use the ilo driver for that, but the process was far from smooth17:37
arne_wiebalckTheJulia: mnaser: there should be "Creating partition on..." log lines, and these start too early on the disk?17:40
TheJuliafrigo: I personally have not tried it, but if you'e got a bug report we can let the ilo devs know17:44
TheJuliaarne_wiebalck: I think he is pre-creating the raid set17:45
TheJuliaI *think*17:45
arne_wiebalckpre-creating?17:45
TheJuliausing a cleaning setep17:45
TheJuliaI think. He is on a TC call now, so I'm sure he will respond later17:46
arne_wiebalckthis should be the normal way to do it17:46
mnaserTheJulia: yes, using clean_configuration, erase_devices_metadata, create_configuration17:46
mnaserso just did that, and then now provide and trying again17:46
TheJuliak17:47
openstackgerritMerged openstack/ironic master: Add support to manage certificates in iLO  https://review.opendev.org/c/openstack/ironic/+/76057317:47
arne_wiebalckfor all devices in UEFI mode the start of the partition should be after 550MB17:47
iurygregoryspeaking of iLO we need to probably talk about their CI =X17:47
TheJuliaiurygregory: noted on the etherpad and I talked to stendulker this morning17:47
iurygregoryTheJulia, awesome!17:47
TheJuliasecure-rbac truth: When we tighten down controls, we need not forget that some people will runt he old policy matrix for a while. :\17:47
arne_wiebalckmnaser: it would be interesting to see the "Creating partition on ...  " debug messages. From re-reading the code I would think the start should only be after 550MB, even if there is only 1 partition with MAX.17:49
* arne_wiebalck wonders if we have a test for this ...17:49
mnaserarne_wiebalck: from the cleaning?17:49
mnaseri just ran it right now but using 220, dont have output for MAX17:50
arne_wiebalckmnaser: yes, when it creates the RAID17:50
arne_wiebalckmnaser: are there "Creating partition on ...  " messages?17:50
frigocool TheJulia. I opened https://bugs.launchpad.net/diskimage-builder/+bug/1916346 for the initial ramdisk creation, and https://storyboard.openstack.org/#!/story/2008643 for the sum firmware update (although sut was what I really want but hey...) I guess I open a new story ? basically needed the controller to first move from HBA to raid (which is 117:50
openstackLaunchpad bug 1916346 in diskimage-builder "proliant-tools installation fails, miss dependencies" [Undecided,New]17:50
frigocommand to run, but not supported yet in there), then I need to refer to the controller by its whole full name "Smart Array P840ar in Slot 0 (Embedded)" which is quite inconvenient (especially this does not show up in the introspection data), then the physical disks MUST be referred to their names like "1I:1:1", you cannot use any "hint" as the doc17:50
frigosays.. then the volume_name is ignored, but then.. hey it works. So here's that :)17:50
*** dougsz has quit IRC17:53
mnaserhttps://www.irccloud.com/pastebin/yir84mdI/17:55
mnaserarne_wiebalck: ^17:55
mnaserso.. we .. do taht already?17:55
mnaserbut maybe not big enough?17:55
arne_wiebalckthis is what I would have expected17:56
arne_wiebalckfor MAX this should go to the end of the device17:56
arne_wiebalck(instead of 220GB, ofc)17:56
arne_wiebalck550MB should be big enough ... I think we took this from one of the UEFI authors :)17:57
mnaserso it would be like17:57
mnaser239GiB 551MiB with max right?17:57
mnaseror something along those lines17:57
arne_wiebalckparted may be able to take -117:58
arne_wiebalckhttps://github.com/openstack/ironic-python-agent/blob/4a22c887f8f30ce18b23e3a7b4761e71512be7a5/ironic_python_agent/hardware.py#L201117:58
arne_wiebalck-1 it is17:59
mnasereeek17:59
mnaserInstalling for x86_64-efi platform.\ngrub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..\ngrub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..\ngrub2-install: error: efibootmgr: not found.\n".: ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader17:59
mnaserto device /dev/md127 failed with Unexpected error while running command.17:59
arne_wiebalckoh, yeah!17:59
*** derekh has quit IRC18:01
mnaserTheJulia / arne_wiebalck: this is where i am at now -- https://gist.githubusercontent.com/mnaser/d3ed5de939349f4a15a17e0bc8cc0686/raw/f0d48565065385aaa1bbd10a93a3b7d3ccdaa1ad/gistfile1.txt18:01
mnaser`grub2-install: error: efibootmgr: not found.`18:02
mnaseri wonder if that is maybe related?18:02
iurygregoryO.o18:02
iurygregorymissing the efibootmgr in the image? O.o18:03
arne_wiebalckmnaser: it is :-D18:03
mnaseranother detail, so image needs mdadm *and* efibootmgr18:03
iurygregory:D18:03
* iurygregory runs (since I was the one to add efibootmgr code)18:03
TheJuliamnaser: yes, you'll want those....18:03
mnasertime to now add another package and we go agane!18:04
iurygregoryyou can do it :D18:04
TheJuliamnaser are you using ironic-pyton-agent-builder?18:04
arne_wiebalckmnaser: you build the image yourself?18:04
openstackgerritJulia Kreger proposed openstack/ironic master: Project Scoping Node endpoint  https://review.opendev.org/c/openstack/ironic/+/77392418:04
openstackgerritJulia Kreger proposed openstack/ironic master: [doc] Add initial system-scoped text to secure-rbac  https://review.opendev.org/c/openstack/ironic/+/77804418:04
* TheJulia thoughts we made the efibootmgr install by default18:04
mnaseri use ipa-builder but i think the problem here is that target image is lacking efibootmgr18:04
mnaser(i.e. my dib built centos)18:04
arne_wiebalckTheJulia: it is not the IPA image18:05
arne_wiebalckmnaser: exactly18:05
iurygregoryTheJulia, I think we did ...18:05
iurygregoryarne_wiebalck, oh gotcha =)18:06
mnaserdisk-image-create -p mdadm -p efibootmgr centos vm block-device-efi dhcp-all-interfaces devuser -o centos-8-devuser-mdadm-efibootmgr18:06
* arne_wiebalck shamelessly plugs https://techblog.web.cern.ch/techblog/post/bios_uefi_cloud_image/ which mentions this is needed in a UEFI image for physical nodes18:06
* TheJulia laughs that she somehow names portgroups progtroups18:06
arne_wiebalckTheJulia: as in: send the progtroups!18:06
TheJuliaarne_wiebalck: yes18:07
TheJuliaarne_wiebalck: very yes18:07
arne_wiebalckTheJulia: :-D18:07
iurygregoryprogtroups is nice =D18:09
arne_wiebalckmnaser: there are some more tools needed: grub2-efi-x64, grub2-efi-x64-modules, efibootmgr, and efivar-lib18:11
arne_wiebalckmnaser: I remember it took me a while to get this working with our image18:11
arne_wiebalckbye everyone, see you tomorrow o/18:13
iurygregorybye arne_wiebalck o/18:16
*** k_mouza has quit IRC18:18
*** k_mouza_ has joined #openstack-ironic18:18
*** uzumaki has quit IRC18:18
*** uzumaki has joined #openstack-ironic18:19
*** k_mouza_ has quit IRC18:19
*** uzumaki has quit IRC18:23
mnasergood bye, take care arne_wiebalck !18:23
*** frigo has quit IRC18:35
iurygregorywe probably need a release note https://review.opendev.org/c/openstack/ironic-python-agent/+/778035 right TheJulia ?18:40
TheJuliayeah, likely18:42
TheJuliawe're hitting that in other places18:42
TheJuliaand tl;dr a bug needs to get filed against qemu-img trying to use memory for as if it was a VM18:42
iurygregoryI will tell derek about it18:43
TheJuliabasically we tried to ramp things up to 3+gb and still see random failures18:44
iurygregorywow18:45
iurygregoryO.o18:45
iurygregorythis was in downstream testing?18:46
*** dsneddon has joined #openstack-ironic18:57
mnaserok, progress18:59
mnaseri'm now stuck on boot but we're past the boot problems lol18:59
openstackgerritJulia Kreger proposed openstack/ironic master: Project Scoping Node endpoint  https://review.opendev.org/c/openstack/ironic/+/77392418:59
openstackgerritJulia Kreger proposed openstack/ironic master: Port/Portgroup project scoped access  https://review.opendev.org/c/openstack/ironic/+/77546518:59
mnaserhttps://usercontent.irccloud-cdn.com/file/51HTQR8M/image.png18:59
mnaseri do have `mdadm` installed on that system18:59
mnaserbut i wonder if the initramfs doesnt contain it19:00
mnaserso its failing to properly boot?19:00
mnaseri'm about like 99% sure this is the case19:51
mnaserIPA doesnt seem to run something like update-initramfs?19:51
openstackgerritArun S A G proposed openstack/ironic master: Add anaconda configuration and template  https://review.opendev.org/c/openstack/ironic/+/77216619:52
openstackgerritArun S A G proposed openstack/ironic master: Add anaconda support in the pxe boot driver  https://review.opendev.org/c/openstack/ironic/+/77589119:52
openstackgerritArun S A G proposed openstack/ironic master: Validate the kickstart template and file before use  https://review.opendev.org/c/openstack/ironic/+/77744819:52
openstackgerritArun S A G proposed openstack/ironic master: Add agent_state and agent_status params to heartbeat  https://review.opendev.org/c/openstack/ironic/+/77805819:52
*** stevebaker has joined #openstack-ironic20:01
*** jawad_axd has quit IRC20:01
*** jawad_axd has joined #openstack-ironic20:07
*** uzumaki has joined #openstack-ironic20:07
*** jawad_axd has quit IRC20:10
TheJuliaiurygregory: yes20:11
TheJuliamnaser: no, it doesn't. We expect the ramdisk in the image would have everything required.20:11
TheJuliamnaser: also, the initramfs stuff looks at the running kernel at times if memory serves.20:12
mnaserTheJulia: cool, I am now working on rebuilding an image with the regenerate dracut element20:12
openstackgerritJulia Kreger proposed openstack/ironic master: Volume targets/connectors Project Scoped RBAC  https://review.opendev.org/c/openstack/ironic/+/77631420:16
openstackgerritJulia Kreger proposed openstack/ironic master: Project scope driver vendor pass-through  https://review.opendev.org/c/openstack/ironic/+/77676720:16
openstackgerritJulia Kreger proposed openstack/ironic master: Follow-up on project scoped trait tests  https://review.opendev.org/c/openstack/ironic/+/77676820:16
iurygregoryPTG  https://april2021-ptg.eventbrite.com20:19
*** k_mouza has joined #openstack-ironic20:20
*** k_mouza has quit IRC20:25
*** mgagne has joined #openstack-ironic20:42
*** anuradha1904 has quit IRC20:54
*** paras333 has quit IRC21:00
mnaserTheJulia, arne_wiebalck: finally, i got  a booted instance21:10
mnaseri will surely write up something about this21:10
mnaseri do think i see a bug21:10
mnaserif you don't have erase_devices, i think you end up with a new EFI partition on every single deploy21:10
arne_wiebalckmnaser: \o/21:10
mnaserhttps://www.irccloud.com/pastebin/A9f6Wv63/21:11
mnasersee multiple 550M EFI system fs21:11
mnaseri havent had erase_devices just to make things faster21:11
arne_wiebalckyou mean on all disks?21:11
arne_wiebalckor multiple on each disk?21:11
arne_wiebalckah, sorry, didn't see I could scroll ... reading ...21:12
arne_wiebalckinteresting21:14
mnaserso i think that was with every deploy attempt21:14
arne_wiebalckonly the RAID creation should create partitions21:14
arne_wiebalckdid you run the RAID clean step before every deploy?21:15
arne_wiebalck(so the RAID creation)21:15
mnaserarne_wiebalck: no, only once, the rest was normal cleaning21:15
*** paras333 has joined #openstack-ironic21:16
arne_wiebalckmnaser: hmm ... normal cleaning should remove everything and only the RAID clean step should create EFI partitions21:16
mnaserarne_wiebalck: https://github.com/openstack/ironic-python-agent/blob/319efe2c2d323817db86388c47bf06e5c6ff269c/ironic_python_agent/extensions/image.py#L552-L554 leading to https://github.com/openstack/ironic-python-agent/blob/319efe2c2d323817db86388c47bf06e5c6ff269c/ironic_python_agent/extensions/image.py#L391-L45521:16
arne_wiebalckoh, true: the clean step only leaves space and deploy populates it21:17
mnaserand that maybe explains that MAX might have worked fine21:18
arne_wiebalckthis is different from the initial BIOS mode where all was set before deploy21:18
mnaserbut on the *second* deploy, it fails21:18
mnaserand right now it doesnt fail cause i have 220G and its a 240G disk so it doesnt fail21:18
arne_wiebalcknot sure if this is a real bug if you do not clean between deploys ... I think we expect this ... need to think about it21:20
*** paras333 has quit IRC21:20
arne_wiebalckmaybe it should complain rather than create a sequence of EFI partitions21:20
TheJuliamnaser: eek, yeah, that is a nice bug21:20
mnaseri mean either reuse it if it exists -- or complain there's something already there21:21
TheJuliawell, we shouldn't try to re-use it I guess21:21
TheJuliait seems wrong that it is already there21:21
TheJulialike, entirely wrong :(21:21
arne_wiebalckwell, there was no cleaning21:21
mnaserso i should figure out why my erase_devices take 5evr then i guess21:22
mnaseri disabled that because it ended up taking 4 hours for some reason21:22
TheJuliamnaser: nvme?21:22
arne_wiebalckshred21:22
mnaserno, not nvme on those systems21:22
mnaseri was hoping that it would use the SED on those SSDs arne_wiebalck21:22
mnaserand if that fails it shreds instead? i thought that was the normal behaviour21:22
TheJuliamnaser: SATA or SAS?21:22
arne_wiebalckyeah, hope is one thing, logging in and checking top is another ;)21:23
mnaser2x SSDSC2KB24 and 2x SSDSC2BB9621:23
mnaserp sure its sata21:23
TheJuliahmm21:23
mnaserarne_wiebalck: well now armed with my cleaning logs21:23
mnaseri should be able to gather more info on why/what21:23
TheJuliathey should be flagging secure erase21:23
TheJuliayeah, let us know21:23
TheJuliathere *are* some cases where it is not possible, and some controllers strip security out and other stuff21:24
TheJuliabut yeah, that doesn't sound like it is the case here21:24
arne_wiebalckyeah, I have seen shred where I expected secure erase as well21:24
mnasermy last thing to fix :)21:24
TheJuliaeven secure erase is also sometimes not instant21:24
arne_wiebalckTheJulia: oh, no?21:24
mnaser^ yeah but i mean, probably not 4 hours not instant?21:25
arne_wiebalckTheJulia: I thought this is just dropping the key21:25
TheJuliaarne_wiebalck: some sata disks secure erase means just keyrotate and unalloc the space21:25
TheJuliaothers it means zero everything out21:25
arne_wiebalckTheJulia: oh!21:25
TheJuliaand if it is a sata disk which prefers zeroing itself out.... espescialy a spinning one21:25
TheJuliayou can get the idea there :(21:25
TheJuliamnaser: nah, most secure erases are like ~2 minutes a drive21:26
TheJuliahdparm should even tell you what the method would be21:26
mnaseroh then i can run that when im sshed here21:26
arne_wiebalck4 hours is too long to zero 240GB, no?21:26
mnaserwell21:26
mnaser2x 240G and 2x 960G :)21:26
mnaser2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.21:27
mnaser4min for SECURITY ERASE UNIT. 4min for ENHANCED SECURITY ERASE UNIT.21:27
mnasersudo hdparm -I /dev/sda (2 min is 240G, 4 min is 960G)21:27
TheJuliaYeah21:27
TheJuliasounds like they are not unlocking or something21:27
TheJuliawell now that your saving the agent logs!21:27
mnaseryessss21:28
mnaserand i can very happily take a break knowing this all works21:28
TheJulia\o/21:28
mnaserand i can leave some sort of thing for others to follow :D21:28
arne_wiebalckmnaser: that makes sense: 160MB/s21:28
* TheJulia fires up some loud music and looks into the wonderful world of allocations21:28
* arne_wiebalck calls it a day 21:30
*** rcernin has joined #openstack-ironic21:33
*** uzumaki has quit IRC21:42
TheJuliatempest's connectivity check makes me watnt to scream21:50
TheJuliatesttools.matchers._impl.MismatchError: 'PING 10.0.100.187 (10.0.100.187): 56 data bytes\n64 bytes from 10.0.100.187: seq=0 ttl=64 time=25.154 ms\n64 bytes from 10.0.100.187: seq=1 ttl=64 time=0.752 ms\n64 bytes from 10.0.100.187: seq=2 ttl=64 time=0.658 ms\n64 bytes from 10.0.100.187: seq=3 ttl=64 time=0.729 ms\n\n--- 10.0.100.187 ping statistics ---\n4 packets transmitted, 4 packets received, 0% packet21:50
TheJulialoss\nround-trip min/avg/max = 0.658/6.823/25.154 ms\n' matches Contains(' bytes from 10.0.100.187')21:50
*** rcernin has quit IRC21:54
*** rcernin has joined #openstack-ironic21:55
*** rcernin has quit IRC21:55
*** rcernin has joined #openstack-ironic21:55
iurygregoryenrrrr22:05
iurygregoryI remember you fixed something like this before no? O.o22:06
iurygregorythis is in stable/branch?22:06
*** lmcgann has quit IRC22:10
TheJuliatzumainn: you around?22:17
erbarrHello, are Third-Party CI accounts required to be part of the Third-Party CI gerrit group that blocks emails being sent from gerrit to reviewers when results get posted as a comment? If not, could I remove Dell's account from that group?22:18
TheJuliaiurygregory: yeah, tempest is branchless so I dunno. but yeah22:18
TheJuliaerbarr: I'd ask in #opendev22:19
tzumainnTheJulia, hi!22:20
iurygregoryI can only add this "=(" I wish there was an easy way to fix things in tempest for stable branches hehe22:21
erbarrTheJulia: I had a chat with fungi on infra,he basically said that is up to the project, I don't want to spam though, so, if you guys prefer not to I totally understand22:21
TheJuliatzumainn: I think I figured it out22:22
TheJuliaiurygregory: I mean, there shouldn't be a stable branch22:22
TheJuliaand it failed saying that it succeeded22:22
TheJuliawhich is totally a 8(::::: moment22:23
iurygregory"failed saying that it succeeded" ?!22:23
TheJuliaiurygregory: it says it failed because the ping worked22:23
TheJuliaerbarr: so help me understand, what exactly are you trying to achieve and why? is the service account getting comments like crazy or something?22:24
iurygregoryhttps://media.giphy.com/media/xlqFxnsFeyuu2pDp4j/giphy.gif22:25
TheJuliatzumainn: I'm guessing create_restricted for allocations is going to be system scoped only, would you concur or see it another way22:25
TheJuliaiurygregory: exactly22:25
TheJuliamy cat is spread out across my redhat laptop and using my phone as a pillow22:25
iurygregorythe cat is going to submit a patch? :D22:26
erbarrTheJulia, no, so I was wondering why emails were not being sent when the comments with results were added and I ended up finding that is due to the Third-Party CI group in gerrit, rpioso was wondering if we could get email notifications when results get posted. I can always do it on my own though if need be22:26
fungierbarr: TheJulia: to clarify, the "Third-Party CI" group in gerrit is basically just a mechanism to silence e-mail notifications for comments from any accounts put in it. not all third-party ci systems are in there, so if you want ironic's change subscribers receiving e-mail notifications from a particular ci system, it can be removed from that group22:26
iurygregoryTheJulia, today I've discovered how it feels when the corgi doesn't let you work :D22:27
TheJuliaerbarr: fungi: I have no preference either way, but I already get way too many emails that I can't keep track of so it might be good to add to the monday meeting agenda and inquire22:28
TheJuliabecause, others may feel way differently22:28
TheJulia*if* they are not already getting emails22:28
erbarrTheJulia, sounds good to me :)22:28
* TheJulia files for email bankruptcy daily22:29
tzumainnTheJulia, I *think* so, although - let's say we wanted project members to have access to create_restricted so they could use allocations for nodes that they leased; would that imply that all project members would also have to be system members?22:29
TheJuliatzumainn: it would imply any group member in the entire deployment could create an allocation22:29
TheJuliathey would't need to be system as well, the scope is just a different operating context22:30
tzumainnoh right I mixed that up22:30
tzumainnthen yeah, that makes perfect sense to me - I don't see why or even how create_restricted could be project scoped22:30
TheJuliaYeah, I'm thinking just SYSTEM_MEMBEr22:31
TheJuliasince that allows overriding of owner22:31
TheJuliayay, 2 allocation tests working22:36
*** rcernin has quit IRC22:39
TheJuliaand now only 122:39
TheJuliaI think it is time to go exercise22:41
tzumainnif you'd like another pair of eyes, just let me know22:41
*** rcernin has joined #openstack-ironic22:42
TheJuliatzumainn: looks like I've got the tests where I need them, I'll just start patching in the filtered view tomorrow and the policy checking as I'm expecting it (since it doesn't seem to be working with the tests I've stubbed out when comparing to the state of the policy rule)22:42
*** paras333 has joined #openstack-ironic23:05
*** paras333 has quit IRC23:09
jandersgood morning Ironic o/23:10
jandersTheJulia saw your comment regarding doc requirements for NVMe cleaning. Looking at it now, might have some questions shortly.23:11
openstackgerritJay Faulkner proposed openstack/ironic-specs master: No Conductor to IPA Communication spec  https://review.opendev.org/c/openstack/ironic-specs/+/77717223:23
jandersTheJulia IRT NVMe cleaning docs it seems to me I only need to update these:23:38
jandershttps://opendev.org/openstack/ironic/blame/branch/master/doc/source/admin/cleaning.rst#L27623:38
jandersand23:38
jandershttps://opendev.org/openstack/ironic/blame/branch/master/doc/source/admin/cleaning.rst#L298 (these now need to cover both NVMe secure erase and ATA secure erase and the appropriate recommendations ). Does this sound about right?23:38
TheJuliajanders: it does23:39
janders(my understanding is that config reference, sample configs and release notes will auto-update based on what's in code)23:39
jandersthanks TheJulia! I will try to sort this out later today. I've got a demo to prepare for this (due later today) and I'm on Metal Support roster this week so worst case it will be tomorrow. Thanks for the reminder.23:40
janderss/for/on23:41
TheJuliaYour understanding is correct, thanks!23:41
jandersTheJulia while we're on this topic, while testing / preparing demo I made some observations which I find quite valuable. I'm testing on node that has 2xSSDs, 8xHDDs and 2xNVMes. To make the nvme_erase code run, I set erase_devices priority to 99. Fired off cleaning, and.... three days later....23:45
TheJuliaRutro23:46
jandersTheJulia I then found a way of switching off all the non-NVMe storage in BIOS and now it's more like three minutes than three days. However I think it's not completely unusual for operators to have hybrid storage like this.23:46
TheJuliaHow big are the spinners?23:47
janders1TB23:47
TheJuliaOh!23:47
TheJuliaBy default we don't enable parallel erasure23:47
TheJuliaYou can toggle that if you want :)23:48
TheJuliaI think that is a fair thing to enable in ironic wallaby23:48
jandersTheJulia conclusion: perhaps it would be worthwhile investigating what would be a good approach to handle configs like? Maybe an extra clean step that allows mix and match along the lines of metadata_erase the HDDs yet nvme_erase the NVMes and maybe add a new feature for blkdiscard'ing the SSDs?23:48
jandersTheJulia I have a sneaking suspicion it might be hard to implement in Wallaby but maybe we should look into this for Xena? What do you think?23:49
TheJuliaWell, by default it is a serialized operation23:49
TheJuliaWe have had the code to do it in parallel, and most people that use spinners for things like ceph need zeroed out disks regardless23:50
jandersyeah parallel zeroing would probably drop it from ~70 to ~7hours23:51
jandersI might actually test this at the end after making the demo and before returning the hardware23:51
jandersbut I still wonder if it would be good to have an optional time-optimised storage cleaning where it's secure if reasonably quick, otherwise it sacrifices security for speed. So nvme formatting NVMes, blkdiscarding SSDs and metadata_clean of HDDs. If I were an operator tasked running hybrid kit, I would seriously consider using this.23:53
iurygregorymorning janders o/23:53
jandershey iurygregory o/23:53
« Sunday, 2021-02-28 Index Tuesday, 2021-03-02 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!