| TheJulia | I think that is it, I think I have positve feedback on the second patch | 01:10 |
|---|---|---|
| vanou | Thanks. And not positive on first patch? | 01:15 |
| TheJulia | No, the first patch is basically a feature, but we can discuss it I guess during the weekly meeting | 01:21 |
| TheJulia | It is so windy here birds are flying backwards | 01:22 |
| TheJulia | NobodyCam: ^ | 01:23 |
| vanou | Thanks. I'll attend next weekly meeting and discuss it. | 01:24 |
| opendevreview | Steve Baker proposed openstack/ironic master: Respond to rpc requests on stop until hash ring reset https://review.opendev.org/c/openstack/ironic/+/872601 | 03:21 |
| opendevreview | Steve Baker proposed openstack/ironic master: On rpc service stop, wait for node reservation release https://review.opendev.org/c/openstack/ironic/+/872834 | 03:21 |
| opendevreview | Steve Baker proposed openstack/ironic master: Expose conductor online boolean for accurate alive https://review.opendev.org/c/openstack/ironic/+/873660 | 03:21 |
| TheJulia | looks like our ci jobs are broken | 05:42 |
| TheJulia | grenade specifically https://paste.openstack.org/show/bqCNtDiOPmaL3PLSK4wP/ | 05:42 |
| TheJulia | I'll look in the morning before I get on calls, but if anyone beats me to it, it would be much appreciated | 05:47 |
| rpittau | good morning ironic! o/ | 09:14 |
| rpittau | fungi: thanks! and happy travels :) | 09:15 |
| rpittau | welllll of course the linters fail :/ | 09:37 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Update shim-signed name for ubuntu jammy https://review.opendev.org/c/openstack/bifrost/+/874650 | 09:52 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Update shim-signed name for ubuntu jammy https://review.opendev.org/c/openstack/bifrost/+/874650 | 10:32 |
| dtantsur | TheJulia: morning! do you think https://storyboard.openstack.org/#!/story/2010537 could be fixed with https://opendev.org/openstack/sushy/commit/2c8229046c6ca8aa342e028c96aaa066f517eb9f? | 10:51 |
| dtantsur | cc janders | 10:51 |
| * janders is looking | 10:55 | |
| janders | just noting ajya was heavily involved in fine-tuning the patch helping me structure it so that it is less likely to break future iDRAC firmware | 10:57 |
| janders | (I take it as a possible slight endorsement) | 10:57 |
| dtantsur | yeah, and I think we tested 6.10 downstream recently? | 10:58 |
| dtantsur | unrelated: ouch https://github.com/unbit/uwsgi/commit/5838086dd4490b8a55ff58fc0bf0f108caa4e079 | 11:02 |
| dtantsur | (may not be the end of the world, but needs keeping an eye) | 11:02 |
| dtantsur | TheJulia, janders, okay, apparently, iDRAC may provide fields in the settings, which are not actually settable. sigh. | 12:30 |
| iurygregory | morning Ironic | 13:47 |
| iurygregory | dtantsur, QA downstream was going to test 6.10.* (I saw something in past weeks related to this) | 13:56 |
| TheJulia | dtantsur: that issue is definitely fixed by that patch | 14:02 |
| TheJulia | dtantsur: and one of the operators verified it who chats in here as well | 14:02 |
| dtantsur | TheJulia: apparently, not completely... at least per one operator I'm talking to | 14:03 |
| TheJulia | oh, no | 14:03 |
| TheJulia | uhhh | 14:03 |
| TheJulia | uhh.... | 14:03 |
| TheJulia | one moment | 14:03 |
| TheJulia | hmm, no that is it | 14:04 |
| TheJulia | I had similar thing, where boot mode could not be set on 6.10, and it was the patch you listed | 14:06 |
| TheJulia | and we had independent verification after dell came back and said "hey, this patch" | 14:06 |
| dtantsur | TheJulia: the claim to have tested 4.1.5 with this patch.. | 14:07 |
| TheJulia | github says 4.4.2 4.4.1 have the patch | 14:09 |
| TheJulia | but 4.1.5 has the commit as well | 14:10 |
| rpittau | hi all, due to a recent added test in sushy-tools unfortunately the RDO RPM can't be compiled anymore in CS8 in xena and yoga, see https://review.rdoproject.org/r/c/openstack/sushy-tools-distgit/+/47114 | 14:17 |
| rpittau | just wondering if it's worth to "fix" it or not considering that we stopped supporting Python 3.6 in sushy-tools 8 months ago | 14:17 |
| TheJulia | afaik, only usage is community and it is the testing tools which we *really* don't support anyway | 14:19 |
| TheJulia | I'd tag out the version on stable branches and move on... personally. | 14:19 |
| rpittau | TheJulia: yep, that's what RDO did and I agree on that | 14:21 |
| iurygregory | TheJulia, great job in the IPE patch o/ | 14:26 |
| TheJulia | iurygregory: thanks | 14:26 |
| TheJulia | testing was the hardest part... *insert unhappy look at random list return order from prometheus* | 14:27 |
| iurygregory | yeahh | 14:39 |
| dtantsur | TheJulia: ufff, the person has retested with 4.1.5, it works correctly. | 14:39 |
| TheJulia | \o/ | 14:41 |
| dtantsur | This has been haunting me for a long time, I think I finally understand it now: https://storyboard.openstack.org/#!/story/2010603 | 14:48 |
| TheJulia | yes, I have deja vu | 14:51 |
| TheJulia | as in I've seen it myself and I think we've discussed it previously | 14:52 |
| TheJulia | but maybe the final save was not working in my cases, and in your case maybe the speed of API query might just be high enough | 14:52 |
| TheJulia | well, I haven't had a case or a issue where it has come up recently | 14:52 |
| TheJulia | but yeah, should be a thing as described | 14:53 |
| dtantsur | I'm definitely hitting it in my local testing | 14:58 |
| dtantsur | oh lol, we even have a unit test for the wrong behavior? | 15:00 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Do not move nodes to CLEAN FAILED with empty last_error https://review.opendev.org/c/openstack/ironic/+/874784 | 15:12 |
| rpittau | I think we have to look at ansible-collections-openstack >= 2.0 for bifrost, I'll ahve to pin keystone client to make CI pass | 16:15 |
| rpittau | sorry python-openstackclient, not keystoneclient | 16:17 |
| dtantsur | oh, the time has come? | 16:23 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Update shim-signed name for ubuntu jammy https://review.opendev.org/c/openstack/bifrost/+/874650 | 16:25 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: CI fixes and workarounds https://review.opendev.org/c/openstack/bifrost/+/874650 | 16:26 |
| rpittau | extremely convoluted explanation in the commit message says yes | 16:27 |
| rpittau | after CI is happy, I'm going to have a look | 16:27 |
| opendevreview | Mark Goddard proposed openstack/networking-generic-switch master: Add Fake device type https://review.opendev.org/c/openstack/networking-generic-switch/+/873098 | 16:32 |
| opendevreview | Mark Goddard proposed openstack/networking-generic-switch master: Add ngs-stress test script https://review.opendev.org/c/openstack/networking-generic-switch/+/874789 | 16:32 |
| JayF | bifrost is so freakin' cool | 17:03 |
| TheJulia | blame 3 glasses of redbreast | 17:04 |
| opendevreview | Mark Goddard proposed openstack/networking-generic-switch master: Fake: support adding a random sleep and injecting failures https://review.opendev.org/c/openstack/networking-generic-switch/+/874793 | 17:17 |
| rpittau | good night! o/ | 17:20 |
| espenfl | Hi there. I have started some work in DIB (specifically the bootloader element) to fix a few issues we discovered when trying to make serial consistent. However, during the process I found something else which I will try to explain to get some feedback on. Heads up; I am a beginner on DIB, Ironic, Bifrost and all things OpenStack related, so I suspect it is my understanding | 17:20 |
| espenfl | that is off here. We are running DIB on Ubuntu 20.04, so it detects the debian family and installs for instance grub-pc. This again dumps files in /etc/grub.d and when running grub2-mkconfig, the content of /etc/default/grub is not included. I have inspected the final generated grub config file to verify this, also in the final image. Some distros include the default grub in | 17:20 |
| espenfl | the 00-header file, but that does not seem to be the case here. The 50-bootloader file in the bootloader element modifies /etc/default/grub, but that does not seem to be included. Any comments or hints here would be awesome. Thanks. | 17:20 |
| opendevreview | Mark Goddard proposed openstack/networking-generic-switch master: Add ngs-stress test script https://review.opendev.org/c/openstack/networking-generic-switch/+/874789 | 17:21 |
| JayF | I think /etc/default/grub, in grub2, is used to do variable substitution in the larger config, yeah? | 17:21 |
| JayF | it doesn't get injected whole-hog | 17:21 |
| TheJulia | yes | 17:23 |
| TheJulia | blend of grub.conf files or bls entries | 17:23 |
| TheJulia | I believe ubuntu is still grub config files with a gruenv | 17:29 |
| TheJulia | grubenv | 17:29 |
| espenfl | right, I agree | 17:50 |
| espenfl | is there a variable at 50-bootloader that lets me know which family I am in? | 17:51 |
| espenfl | is there a variable at 50-bootloader that lets me know which distro I am in? | 17:51 |
| JayF | I'm not sure; I'm not an ubuntu user and really your problem, I think, boils down to a distro behavior concern | 17:53 |
| JayF | I wonder if they'd know in an ubuntu support channel... | 17:53 |
| * JayF not trying to be unhelpful just don't want to mislead with inaccurate info | 17:53 | |
| espenfl | Yes, I agree absolutely and thanks for the comments. I am a bit unsure how to approach it now, but will try to source more info and go from there. At least the 50-bootloader in its current states, modifying the /etc/default/grub for e.g. the serial command is not picked up when runing DIB on Ubuntu. | 17:55 |
| JayF | Hmm. Makes me wonder if DIB is not ever regenerating grub configs during build | 17:56 |
| espenfl | I guess question number one I need to get answered (elsewhere) is if all the distros we would support use a vanilla setup or not. Then we tailor from there. DIB does run grub2-mkconfig/grub-mkconfig on build. | 17:56 |
| JayF | ack | 17:57 |
| espenfl | grub-mkconfig does have some limitations. While adding extra custom menu entries to the end of the list can be done by editing /etc/grub.d/40_custom or creating /boot/grub/custom.cfg, changing the order of menu entries or changing their titles may require making complex changes to shell scripts stored in /etc/grub.d/. This may be improved in the future. In the meantime, | 17:57 |
| espenfl | those who feel that it would be easier to write grub.cfg directly are encouraged to do so (see Booting, and Shell-like scripting), and to disable any system provided by their distribution to automatically run grub-mkconfig. | 17:57 |
| JayF | most of my knowledge of diskimage-builder is of it in ramdisk mode, to build e.g. IPA images | 17:57 |
| JayF | I've not used it much for making disk images | 17:57 |
| espenfl | argh...sorry for the copy paste dump | 17:57 |
| espenfl | https://opendev.org/openstack/diskimage-builder/src/commit/174089a6a5f1ff4628a9d103e4328a1b1298e241/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L123 | 17:58 |
| espenfl | This was the one I wanted to paste, where DIB runs the mkconfig | 17:58 |
| JayF | https://opendev.org/openstack/diskimage-builder/src/commit/174089a6a5f1ff4628a9d103e4328a1b1298e241/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L102 | 17:59 |
| espenfl | thanks a lot, I will do some more investigations and suggest something on gerrit | 17:59 |
| JayF | That looks like it might be overriding whatever you're setting manually? | 17:59 |
| * JayF not sure he understands all your inputs / desired outputs though | 17:59 | |
| espenfl | Not sure I understand that question, but to make a very specific example: This line https://opendev.org/openstack/diskimage-builder/src/commit/174089a6a5f1ff4628a9d103e4328a1b1298e241/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L104 is not picked up during the run at | 18:03 |
| espenfl | https://opendev.org/openstack/diskimage-builder/src/commit/174089a6a5f1ff4628a9d103e4328a1b1298e241/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L123. So in the final grub.cfg there is nothing given for serial. Also for instance | 18:04 |
| espenfl | https://opendev.org/openstack/diskimage-builder/src/commit/174089a6a5f1ff4628a9d103e4328a1b1298e241/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L88 is also not picked up and the final grub.cfg only have terminal_output/terminal_input defined as console only. | 18:04 |
| JayF | How are you testing this? Just literally looking at the content of the image locally? | 18:05 |
| JayF | Just making sure e.g. grub handling code in Ironic isn't coming into play | 18:05 |
| espenfl | both by doing printouts of the DIB run, and inspecting the /boot in the final image | 18:05 |
| JayF | ack | 18:06 |
| JayF | then I'm honestly not certain what could be going on | 18:06 |
| JayF | when I see behavior like this, I often run with DIB_TRACE enabled | 18:06 |
| JayF | and comb through the results | 18:06 |
| JayF | many times it has to do with elements injecting things at the wrong time in the build process | 18:06 |
| espenfl | Me neither, seems strange honestly, thanks for the DIB_TRACE suggestion. Will try this and continue investigations. | 18:06 |
| JayF | I'll note there is an #openstack-dib | 18:06 |
| JayF | but in all honestly same folks will likely be helping you either place | 18:07 |
| espenfl | thanks, did not know | 18:07 |
| JayF | so feel free to keep hanging out with us or go over there :D | 18:07 |
| espenfl | okey, thanks...I feel Bifrost, Ironic, IPA, DIB etc. are all somewhat represented here | 18:07 |
| JayF | this channel is for the openstack bare metal program, which includes all those things except DIB :D | 18:09 |
| JayF | but it's all ... artificial lines | 18:09 |
| JayF | we're all openstack | 18:09 |
| JayF | just trying to make things work :D | 18:10 |
| TheJulia | ... and we have cores who are also cores on DIB | 18:14 |
| TheJulia | is it friday yet?!? | 18:14 |
| * TheJulia has a low spoon count right now | 18:14 | |
| espenfl | great | 18:21 |
| *** JasonF is now known as JayF | 19:51 | |
| TheJulia | espenfl: so, out of curiosity, is ubuntu moving to BLS ? | 19:55 |
| TheJulia | because on non-BLS, if my memory is serving me correctly, it is more harmful to run mkconfig | 19:55 |
| TheJulia | than not to | 19:55 |
| TheJulia | hmm, there is also preservation logic in there | 19:56 |
| TheJulia | I thought mkconfig was supposed to read /etc/default/grub | 20:00 |
| * TheJulia wonders if we're in a flavor of grub issue territory | 20:00 | |
| JayF | TheJulia: that's why I suggested asking the ubuntu folks; some of the described behavior didn't match my expectations for grub (which have mostly been set by gentoo and rhel-based OSes) | 20:01 |
| JayF | and bluntly, ubuntu is sometimes just a little weird in ways you wouldn't expect | 20:02 |
| TheJulia | I could see gentoo having a mix of the patches since grub is effectively forked by distro these days | 20:02 |
| JayF | https://github.com/gentoo/gentoo/tree/master/sys-boot/grub/files nothing too intrusive | 20:03 |
| TheJulia | but starting with grub 2.06 which I *Think* is from the rh bootloader | 20:11 |
| TheJulia | well, from their mirror | 20:11 |
| TheJulia | mirror is the wrong word, it is their working copy on the bootloader devs repo someplace on github | 20:11 |
| JayF | When given a choice, most of the time gentoo will choose similar options to fedora; we carry the same set of gnome-terminal patches, the default kernel configs are based on fedora, etc | 20:13 |
| TheJulia | yeah, ubuntu has a n entirely different fork hosted on launchpad | 20:13 |
| TheJulia | which imports stuff in targeted fashion it looks like | 20:13 |
| TheJulia | at least, that is what it *looks* like, it has been a while since I've dug through lp hosted stuffs | 20:14 |
| TheJulia | looks like some stuff got synced up back in 2021 | 20:15 |
| espenfl | Yes, I will try to investigate these things in more details, implement, test and report back. Not sure about the BLS as I am no Ubuntu person (on Arch most of the time), but it does not seem like it from initial investigations. | 21:10 |
| JayF | espenfl: we've been recently burned (Ironic as a project) by those grub incompatabilities betweeen rh-based/deb-based stuff recently; so we're a little extra attuned to it :) | 21:12 |
| JayF | TheJulia: https://canonical.com/careers/4388367 canonical is hiring for grub maintainers :) | 21:38 |
| JayF | TheJulia: that's the answer I got from a friend there when I asked them to unfork grub (jokinglyish) | 21:38 |
| TheJulia | Oh… my | 21:38 |
| JayF | so go find some Ironic ally to go take that job and we'll be set LOL | 21:39 |
| stevebaker[m] | espenfl: I've spent quite some time in 50-bootloader, but only on red hat based distros. Feel free to add me to reviews or whatever | 21:47 |
| TheJulia | ++ | 21:47 |
| JayF | if stevebaker[m] ever pretends to not know something, don't believe 'em | 21:48 |
| JayF | ;) | 21:48 |
| JayF | rpittau: I'm looking at > [ironic][QA][horizon][release][OpenStackSDK] Cycle With Intermediary without recent deliverables | 21:49 |
| JayF | rpittau: I think we still have actions to take there? | 21:49 |
| stevebaker[m] | I have an ulterior motive here, I'm looking for a volunteer to allow dib to build Secure Boot capable ubuntu/debian | 21:49 |
| TheJulia | double check and trigger some releases before the 2nd it seems | 21:49 |
| JayF | yeah I think it's scoped into the work he's already picked up | 21:50 |
| TheJulia | ack | 21:51 |
| espenfl | stevebaker[m]: will do | 21:51 |
| espenfl | Also, I think I got more details that are relevant. Ubuntu cloud images are used and which, in its default settings source /etc/default/grub, /etc/default/grub.d/50-cloudimg-settings.cfg and then /etc/default/grub.d/init-select.cfg. And GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0" and GRUB_TERMINAL=console is set in 50-cloudimg-settings.cfg. No wonder the stuff in | 21:56 |
| espenfl | /etc/default/grub is gone. | 21:56 |
| espenfl | one option would be to inject the changes into something in /etc/default/grub.d that are picked the last instead of /etc/default/grub | 21:57 |
| espenfl | https://sources.debian.org/src/grub2/2.06-8/debian/patches/default-grub-d.patch/ | 22:03 |
| stevebaker[m] | espenfl: red hat grub does not have that patch. Fun fact: the rpm spec repo builds grub by downloading upstream then applying 323 patches | 22:14 |
| espenfl | stevebaker[m]: interesting covers it I guess :) | 22:16 |
| espenfl | do I have a way to describe what is going on here a bit better in an issue? should I just open a new bug report and dump what I have discovered so far? eventually I will try to fix it, but I think it would be nice to get the facts out there so that others also can verify or comment | 22:17 |
| TheJulia | They are pretty good, afaik, about making sure they are on the GitHub repo, but effectively it is forked out with no central maintainer anymore | 22:18 |
| TheJulia | stevebaker[m]: what upstream? | 22:19 |
| stevebaker[m] | TheJulia: https://ftp.gnu.org/gnu/grub/grub-2.06.tar.xz | 22:21 |
| TheJulia | oh wow | 22:22 |
| TheJulia | that is surprising | 22:22 |
| espenfl | So I guess there are two approaches to fixing this on the DIB end, either inject a file in `/etc/default/grub.d` that is guaranteed to be picked last, or let `init-select.cfg` source it if possible. Also, we need to know if we are on Ubuntu (maybe we also need to know if we use the specific cloud images). | 22:28 |
| espenfl | Any thoughts or comments on this? | 22:29 |
| JayF | I like the first option you present | 22:31 |
| JayF | it's less likely to bitrot over time | 22:31 |
| espenfl | Yes, agreed. More info on the table. Amazon recommends modifying the 50-cloudimg-settings.cfg file: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/grub.html | 22:34 |
| espenfl | maybe we should also check if there are any files in `/etc/default/grub.d` in case that patching is reverted sometime in the future (also, I guess for regular Ubuntu images, this is not used, given its name) | 22:37 |
| JayF | espenfl: I'd suggest putting some of your findings in an etherpad (https://etherpad.openstack.org/p/any-string-you-choose-to-use-to-name-your-etherpad) just to aggregate them and make it easier to follow along for folks troubleshooting | 22:48 |
| JayF | I know most of this context will have flowed out of my brain like water by the time I revisit tomorrow :D | 22:48 |
| JayF | good luck and thank you for sharing your journey with us o/ | 22:48 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!