| opendevreview | Merged openstack/metalsmith master: metalsmith_instances module, pass hostname in reserve https://review.opendev.org/c/openstack/metalsmith/+/845857 | 00:34 |
|---|---|---|
| opendevreview | Merged openstack/metalsmith master: metalsmith_instances module, unprovision by hostname https://review.opendev.org/c/openstack/metalsmith/+/846531 | 00:34 |
| TheJulia | hmm, there still may be some sort of issue with networking but it doesn't quite make sense | 00:36 |
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Add iweb to the use tinyipa on list https://review.opendev.org/c/openstack/ironic/+/847396 | 00:43 |
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Add iweb to the use tinyipa on list https://review.opendev.org/c/openstack/ironic/+/847396 | 01:03 |
| *** undefined_ is now known as Guest3112 | 04:09 | |
| opendevreview | Merged openstack/ironic master: CI: Add iweb to the use tinyipa on list https://review.opendev.org/c/openstack/ironic/+/847396 | 04:13 |
| *** akahat is now known as akahat|ruck | 04:39 | |
| *** undefined_ is now known as Guest3115 | 05:51 | |
| rpittau | good morning ironic! o/ | 06:17 |
| rpittau | mmm still seeing some random issues in ipa failing to connect to ironic | 07:32 |
| rpittau | if anyone has a chance would really love a review on https://review.opendev.org/c/openstack/ironic-python-agent/+/775391 :) | 07:34 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use bifrost on centos9 integration job https://review.opendev.org/c/openstack/ironic/+/846155 | 07:57 |
| opendevreview | Martin André proposed openstack/ironic stable/ussuri: Fix installation of virtualbmc with py36 in devstack https://review.opendev.org/c/openstack/ironic/+/847548 | 08:53 |
| opendevreview | Martin André proposed openstack/ironic stable/train: Fix installation of virtualbmc with py36 in devstack https://review.opendev.org/c/openstack/ironic/+/847436 | 08:54 |
| janders | hey rpittau o/ | 09:07 |
| rpittau | hey janders :) | 09:07 |
| rpittau | alright I think we need to pin neutron as we did for swift in ipa bugfix/8.3 | 09:09 |
| rpittau | probably doing that in ironic bugfix/19.0 will benefit all the connected bugfix branches | 09:10 |
| rpittau | we need 2 patches at least, one in ironic and one in ipa, going to do that now before lunch | 09:11 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/19.0: Pin neutron to stable/yoga for bugfix/19.0 ironic-base job https://review.opendev.org/c/openstack/ironic/+/847550 | 09:15 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.3: Remove swift stable/yoga pin https://review.opendev.org/c/openstack/ironic-python-agent/+/847552 | 09:17 |
| dtantsur | TheJulia: when updating "use tinyipa list", keep in mind that the same jobs run on IPA-builder and DIB. we may miss regressions this way. | 09:18 |
| dtantsur | good morning folks | 09:18 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.3: Fix discovering WWN/serial for devicemapper devices https://review.opendev.org/c/openstack/ironic-python-agent/+/847158 | 09:18 |
| rpittau | hey dtantsur :) | 09:18 |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Prevent clear_job_queue failures on older iDRACs https://review.opendev.org/c/openstack/ironic/+/846859 | 10:22 |
| opendevreview | Merged openstack/ironic master: Use bifrost on centos9 integration job https://review.opendev.org/c/openstack/ironic/+/846155 | 10:44 |
| dtantsur | rpittau, could you check https://review.opendev.org/c/openstack/ironic/+/847388 please? kinda urgent for us. | 11:19 |
| janders | see you next week Ironic, have a great weekend everyone o/ | 11:41 |
| janders | ajya when you're back next week I'd very much appreciate your feedback on revised https://review.opendev.org/c/openstack/ironic/+/846859 (just working on the unit test now) | 11:42 |
| rpittau | bye janders o/ | 11:58 |
| rpittau | dtantsur: I'm reviewing it now | 12:00 |
| dtantsur | TheJulia: morning! when you're online, could you check https://review.opendev.org/c/openstack/ironic/+/847388 please? It's critical for us. | 12:31 |
| kubajj | Hey everybody, I feel quite bad for asking so many questions, but I am still struggling with the IPA setup. I am able to update the ramdisk on a node by following the tutorial from dtantsur, but I am probably building it wrong or something as after updating it with an unedited IPA built from source the nodes are not able to clean (they timeout). | 12:31 |
| dtantsur | Don't worry about question kubajj, this is why we're here | 12:32 |
| dtantsur | Check the node's virtual console (virsh console, virt-manager, whatever you can get). | 12:32 |
| dtantsur | It's possible that your devstack used a lightweight image, and now you're using a heavy one | 12:32 |
| dtantsur | You need at least 3G RAM on the testing nodes - verify that | 12:33 |
| * dtantsur hunts for lunch, brb | 12:33 | |
| TheJulia | dtantsur: oh, and we have once and I remember patching it. The bottom line is we have to balance our testing to enable us to deliver. :\ Feels awful at the same time. | 12:55 |
| TheJulia | dtantsur: sipping some coffee, the requested patch is on the screen | 12:56 |
| *** undefined__ is now known as rcastillo | 13:08 | |
| *** undefined_ is now known as rcastillo | 13:09 | |
| TheJulia | looks like we have some cache tuning stuffs to do with sqlalchemy 2.0... but out of the box the performance looks better | 13:14 |
| TheJulia | like.. mind numbingly fast | 13:15 |
| TheJulia | in CI | 13:15 |
| TheJulia | I think I've figured out the networking issue | 13:42 |
| TheJulia | but not how to fix it | 13:42 |
| TheJulia | tl;dr dnsmasq stops talking to it | 13:43 |
| TheJulia | and the lease expires | 13:43 |
| TheJulia | and boom | 13:44 |
| TheJulia | networkmanager also doesn't log the lease disappearing | 13:44 |
| jungleboyj | TheJulia: Good morning. Random question for you ... | 14:00 |
| jungleboyj | We are running into challenges with booting special boot images from Red Hat for edge deploys. Is Ironic able to handle mounting/booting special images? | 14:01 |
| TheJulia | jungleboyj: is it coreos images? | 14:03 |
| jungleboyj | Checking. I know that I have gotten CoreOS support into Confluent. That is how we are automating OpenShift install. | 14:05 |
| jungleboyj | Missed getting to see all of you in Berlin by the way. :-( | 14:06 |
| TheJulia | dtantsur: https://storyboard.openstack.org/#!/story/2010109 | 14:08 |
| TheJulia | that is what is murdering us in the gate | 14:09 |
| TheJulia | jungleboyj: so the answer really depends on the image and which type, there are numerous types. | 14:10 |
| kubajj | If I want to use the local.conf from https://docs.openstack.org/ironic/latest/contributor/dev-quickstart.html#ironic but want to use the heavy IPA, is there anything else that I need to change other than IRONIC_RAMDISK_TYPE from tinyipa to dib? When I compile the IPA I want to update the nodes with, how do I know which OS I should use? (I assume that centos at the very end of | 14:13 |
| kubajj | https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/846609/1/doc/source/contributor/index.rst specifies that I node will run on CentOS.) | 14:13 |
| jungleboyj | TheJulia: Right. We have support for Core OS. This apparently sounds like something more than just Core OS. Working on getting more details. | 14:13 |
| jungleboyj | Started getting pinged overnight about this and not sure the context yet. | 14:14 |
| TheJulia | jungleboyj: ok, running out for breakfast | 14:14 |
| TheJulia | bbiab | 14:15 |
| jungleboyj | TheJulia: Enjoy! | 14:15 |
| TheJulia | if anyone has suggestions of how to tell network manager to sit down and be quiet, lmk! | 14:15 |
| dtantsur | TheJulia: so, tl;dr, a Stream 9 regression? :( | 14:26 |
| dtantsur | kubajj: IRONIC_VM_SPECS_RAM=1024 is something you should not use in this case | 14:27 |
| dtantsur | 1G is enough for tinyIPA, but not for DIB builds | 14:28 |
| kubajj | dtantsur: Thanks. What about the IRONIC_BUILD_DEPLOY_RAMDISK=False. Can I keep that? | 14:29 |
| dtantsur | kubajj: yes. you're going to replace it with your ramdisk anyway, right? | 14:29 |
| kubajj | dtantsur: | 14:30 |
| kubajj | Yes, that's the plan | 14:30 |
| dtantsur | then you don't need devstack to build one for you | 14:30 |
| kubajj | But doesn't it say that if I want to build one from source, I should set it to True? | 14:31 |
| dtantsur | kubajj: if you want devstack to build you one - yes. | 14:32 |
| dtantsur | but you'll need to jump through some steps to make it contain your change | 14:32 |
| kubajj | dtantsur: So does it build one for me if I don't change anything? | 14:33 |
| dtantsur | kubajj: IRONIC_BUILD_DEPLOY_RAMDISK=True asks devstack to build you a ramdisk of the IRONIC_RAMDISK_TYPE type | 14:35 |
| dtantsur | the value of False causes a pre-built ramdisk to be downloaded | 14:35 |
| kubajj | dtantsur: I see, thanks | 14:36 |
| TheJulia | dtantsur: I’m guessing a feature in stream9 or some insanity. After breakfast I’ll dig. It could be something DIB is doing | 14:46 |
| TheJulia | dtantsur: if I’m remembering the dhcp spec… networkmanager is way out of line | 14:47 |
| TheJulia | And the server is totally in the right | 14:47 |
| TheJulia | And this will break things all over the place If it is coming from the base images | 14:48 |
| opendevreview | Verification of a change to openstack/ironic master failed: No deploy_kernel/ramdisk with the ramdisk deploy and no cleaning https://review.opendev.org/c/openstack/ironic/+/847388 | 14:55 |
| dtantsur | of course.. | 14:56 |
| opendevreview | Verification of a change to openstack/ironic master failed: No deploy_kernel/ramdisk with the ramdisk deploy and no cleaning https://review.opendev.org/c/openstack/ironic/+/847388 | 15:14 |
| rpittau | bye everyone, have a great weekend! o/ | 15:17 |
| opendevreview | Merged openstack/sushy master: Update jobs names https://review.opendev.org/c/openstack/sushy/+/838662 | 15:22 |
| opendevreview | Mark Goddard proposed openstack/ironic-python-agent master: Warn when smartctl not found https://review.opendev.org/c/openstack/ironic-python-agent/+/847591 | 15:49 |
| opendevreview | Mark Goddard proposed openstack/ironic-python-agent master: Warn when smartctl not found https://review.opendev.org/c/openstack/ironic-python-agent/+/847591 | 16:01 |
| opendevreview | Baptiste Jonglez proposed openstack/networking-generic-switch master: Do not make actual device changes in bind_port() https://review.opendev.org/c/openstack/networking-generic-switch/+/847592 | 16:03 |
| TheJulia | dtantsur: I think we're going to need someone who groks the internals of networkmanager to fix this... it is almost surely a bug and I'm not seeing anything that could allow us to route around short of teaching IPA to kill networkmanager | 17:36 |
| TheJulia | which does not seem ideal | 17:36 |
| dtantsur | :( | 17:36 |
| TheJulia | fundimentally, it keeps retrying every sixty seconds, and I don't understand why it can't settle | 17:37 |
| TheJulia | which cascades from there. It also knows the lease is 1 day too | 17:37 |
| TheJulia | which is what makes it absurd, and there are no signs of the carrier changing. | 17:37 |
| TheJulia | well, I take that back, the kernel does log a change | 17:38 |
| TheJulia | in the VM | 17:38 |
| TheJulia | none in the host... | 17:39 |
| dtantsur | huh | 17:39 |
| TheJulia | but even then, it renews after that | 17:40 |
| TheJulia | its only when it tries to renew again 60 seconds later that things start going sideways | 17:40 |
| TheJulia | I did initially find what seemed like a similar issue with the in-networkmanager client | 17:41 |
| TheJulia | except, we don't use that client | 17:41 |
| TheJulia | we use dhclient because that doesn't cause the line carrier to be bounced completely | 17:41 |
| TheJulia | because networkmanager is modeled around wifi | 17:42 |
| opendevreview | Julia Kreger proposed openstack/metalsmith stable/yoga: CI: Extend timeouts for RAX https://review.opendev.org/c/openstack/metalsmith/+/847446 | 17:44 |
| TheJulia | dtantsur: well, there is a thought. it is e1000 on bios.... | 17:51 |
| TheJulia | on uefi we wouldn't use it I suspect | 17:51 |
| dtantsur | there was a story behind it, I don't remember any more | 17:54 |
| jeyoung | Hey all, can I get some help getting this merged? https://review.opendev.org/c/openstack/sushy-tools/+/842385 Looks like the failure is unrelated to the code change? | 18:02 |
| dtantsur | jeyoung: I think we dropped the 3.6 job, recheck away | 18:03 |
| dtantsur | on this positive note, I'm leaving for PTO. see you in a bit more than a week! | 18:07 |
| dtantsur | o/ | 18:07 |
| TheJulia | dtantsur: enjoy! | 18:12 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Add ipa-tox-examples job to gate https://review.opendev.org/c/openstack/ironic-python-agent/+/844003 | 18:25 |
| JayF | https://review.opendev.org/c/openstack/ironic-python-agent/+/842605 is there any reason this can't merge right now? Just making sure, since it's related to a zuul deprecation, that we didn't have specific timing in mind for landing it -> https://review.opendev.org/c/openstack/ironic-python-agent/+/842605 | 18:47 |
| JayF | lol, URL sandwich, at least you can't miss the change I'm talking about :) | 18:47 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Discover what is going on with dhcp https://review.opendev.org/c/openstack/ironic/+/847601 | 18:56 |
| JayF | Hmm. Is it possible to have two Ironic nodes with the same name? | 18:56 |
| JayF | I remember that was possible in some Ironic clusters I've run; but frankly IDK if that was a downstreamism or not | 18:56 |
| JayF | just wondering if we need to consider that for https://review.opendev.org/c/openstack/ironic/+/845856 | 18:57 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Docs: specify what to do with the created images https://review.opendev.org/c/openstack/ironic/+/847136 | 18:58 |
| JayF | TheJulia: I'm sorry if CI is still broken, I just kicked a bunch of jobs b/c I thought we had fixed it /o\ | 19:05 |
| * JayF doing his friday review sweep thru ironic/ipa/ipa-builder | 19:05 | |
| TheJulia | JayF: no worries, some of it will actually help, but we do seem to have a legit issue with networkmanager | 19:06 |
| TheJulia | :( | 19:06 |
| TheJulia | JayF: two ironic nodes... same name.... I think it was possible, albeit not advised, any restart would kill in flight stuff for both | 19:06 |
| TheJulia | there is no hash ring delineation so you'd also get double the work | 19:07 |
| opendevreview | Merged openstack/metalsmith stable/wallaby: metalsmith list handle missing port https://review.opendev.org/c/openstack/metalsmith/+/847103 | 19:07 |
| JayF | So it sounds like the answer is maybe "it's already a little bugged" which means I'm going to review that change to "please test to make sure we're not making this more bugged" | 19:08 |
| JayF | because if it's possible; it shouldn't break things | 19:08 |
| TheJulia | stevebaker[m]: ^ accidently approved out of order, but the others wer ein flight last I checked | 19:08 |
| JayF | even if it's a barely sensible thing, because I suspect the downstreams I've worked on are not the only places that would have automation applying metadata (like node.name) to stuff | 19:08 |
| TheJulia | oh | 19:08 |
| TheJulia | duplicate node names | 19:08 |
| TheJulia | uhhhhh | 19:08 |
| TheJulia | ohhhh | 19:08 |
| TheJulia | I think they have to be unique | 19:09 |
| * TheJulia checks | 19:09 | |
| JayF | that's what I'm askin' about, yeah | 19:09 |
| JayF | I know in the other two big clusters I worked on, it wasn't unique, but both were heavily patched | 19:09 |
| TheJulia | yeah, it has a unique constraint | 19:09 |
| JayF | awesome, ty for validing that, I'll approve that change now then | 19:10 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/db/sqlalchemy/alembic/versions/3ae36a5f5131_add_logical_name.py#L32 | 19:10 |
| JayF | I was looking thru the object models for the relationship/contstraint | 19:10 |
| TheJulia | sorry, I thought youmeant conductor hostnames | 19:10 |
| JayF | searching the migrations was big brained | 19:10 |
| TheJulia | eh, it happens | 19:10 |
| TheJulia | you can actually do things without the migrations | 19:10 |
| TheJulia | just... yeah. your code has to know how to handle everything independently | 19:10 |
| opendevreview | Merged openstack/metalsmith stable/yoga: metalsmith list handle missing port https://review.opendev.org/c/openstack/metalsmith/+/847097 | 19:27 |
| opendevreview | Merged openstack/metalsmith stable/yoga: metalsmith_instances module, pass hostname in reserve https://review.opendev.org/c/openstack/metalsmith/+/847098 | 19:27 |
| opendevreview | Merged openstack/metalsmith stable/yoga: metalsmith_instances module, unprovision by hostname https://review.opendev.org/c/openstack/metalsmith/+/847099 | 19:27 |
| TheJulia | \o/ | 19:47 |
| TheJulia | now xena... go xena | 19:47 |
| opendevreview | Verification of a change to openstack/ironic master failed: No deploy_kernel/ramdisk with the ramdisk deploy and no cleaning https://review.opendev.org/c/openstack/ironic/+/847388 | 19:53 |
| opendevreview | Merged openstack/metalsmith stable/xena: metalsmith list handle missing port https://review.opendev.org/c/openstack/metalsmith/+/847100 | 20:39 |
| opendevreview | Merged openstack/ironic-python-agent master: Remove importlib-metadata from requirements https://review.opendev.org/c/openstack/ironic-python-agent/+/846782 | 20:46 |
| opendevreview | Verification of a change to openstack/ironic bugfix/19.0 failed: deploy_utils: only check glance for image properties kernel/ramdisk https://review.opendev.org/c/openstack/ironic/+/844473 | 20:47 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Warn when smartctl not found https://review.opendev.org/c/openstack/ironic-python-agent/+/847591 | 22:47 |
| opendevreview | Merged openstack/metalsmith stable/wallaby: metalsmith_instances module, pass hostname in reserve https://review.opendev.org/c/openstack/metalsmith/+/847104 | 23:44 |
| opendevreview | Merged openstack/metalsmith stable/wallaby: metalsmith_instances module, unprovision by hostname https://review.opendev.org/c/openstack/metalsmith/+/847105 | 23:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!