| opendevreview | Julia Kreger proposed openstack/ironic master: RBAC: Fix allocation check https://review.opendev.org/c/openstack/ironic/+/905038 | 01:09 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Change to enforced policy by default https://review.opendev.org/c/openstack/ironic/+/902009 | 01:09 |
| opendevreview | Merged openstack/ironic-lib master: Drop lower-constraints.txt (again) https://review.opendev.org/c/openstack/ironic-lib/+/904993 | 02:45 |
| TheJulia | Any thoughts on removing ironic-inspector-tempest-managed-non-standalone ? It is failing a ton, nobody has fixed it. It looks like the base image, configured to rely upon netboot. Given we'e basically deprecated inspector itself at this point, it just seems like something we don't really need to be spinning resources. | 04:56 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Remove ironic-inspector-tempest-managed-non-standalone https://review.opendev.org/c/openstack/ironic/+/905057 | 05:04 |
| TheJulia | If someone that knows the inner config of the metal3-integration job could take a look at https://zuul.opendev.org/t/openstack/build/a0d63ddc5a8c4aa2b1acfd900df55489 It seems entirely unrelated to changing the policy default for RBAC since it no-ops in that config. All sign points to disk space for the namespace? | 05:05 |
| rpittau | good morning ironic! o/ | 08:02 |
| rpittau | TheJulia: re metal3-integration job: from a quick look it seems like the issue is indeed related to the lack of disk space | 08:27 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Uplift default Ansible version to 8.x https://review.opendev.org/c/openstack/bifrost/+/903950 | 08:33 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Uplift default Ansible version to 8.x https://review.opendev.org/c/openstack/bifrost/+/903950 | 08:35 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/23.1: Handle LLDP parse Unicode error https://review.opendev.org/c/openstack/ironic/+/904860 | 08:40 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Make inspection URL optional if the collectors are provided https://review.opendev.org/c/openstack/ironic-python-agent/+/904026 | 09:09 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: DNM api debugging https://review.opendev.org/c/openstack/ironic/+/905113 | 12:44 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Make inspection URL optional if the collectors are provided https://review.opendev.org/c/openstack/ironic-python-agent/+/904026 | 12:45 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Switch the dibipa jobs to Redfish https://review.opendev.org/c/openstack/bifrost/+/905114 | 12:49 |
| dtantsur | rpittau: I'm not super fond of ^^^, but I don't see other options | 12:50 |
| rpittau | mmm alright :/ | 13:24 |
| TheJulia | good morning | 14:04 |
| TheJulia | rpittau: ack, okay. I guess it is good to see the job fail again on another job change entirely unrelated. | 14:06 |
| rpittau | TheJulia: oooook that's not good for me :D | 14:16 |
| * dtantsur is curious how many people actually test iPXE with IPv6 | 14:17 | |
| TheJulia | it is used quite a bit, we just only have one job running it in our CI, why? | 14:19 |
| TheJulia | rpittau: sorry :( | 14:19 |
| rpittau | TheJulia: no worries! what's the other failing change? | 14:20 |
| dtantsur | TheJulia: I see a customer bug which seems to boil down to iPXE sending a malformed host header | 14:20 |
| dtantsur | without brackets around the IP | 14:20 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: DNM: Change policy to enforce only new policy https://review.opendev.org/c/openstack/ironic-inspector/+/905119 | 14:20 |
| TheJulia | dtantsur: interesting, could it be the uefi network stack doing it? | 14:21 |
| dtantsur | Who knows? Possibly? | 14:22 |
| TheJulia | Just thinking it gets a ton of networking stuffs from that, and it seems rather... odd | 14:22 |
| dtantsur | Does it use HTTP implementation from the stack? | 14:22 |
| TheJulia | I think it might... | 14:23 |
| dtantsur | Could explain why we don't see it more often | 14:23 |
| TheJulia | so, it does not, it does most of it from what I'm seeing | 14:24 |
| TheJulia | what header/field is malformed? | 14:24 |
| dtantsur | Host | 14:24 |
| TheJulia | https://github.com/ipxe/ipxe/blob/master/src/net/tcp/httpcore.c#L601-L655 | 14:27 |
| dtantsur | I'm looking at that too, but it's not trivial to understand if the host if bracketed anywhere | 14:28 |
| TheJulia | it doesn't reformat it I suspect it was geared around v4/v6 or an FQDN, and they never had anyone have anything which did any sort of host header header check. Looks like http_connect in httpconn.c just copies whatever is in the field | 14:30 |
| dtantsur | They do have tests for this case https://github.com/ipxe/ipxe/blob/master/src/tests/uri_test.c#L586-L596 | 14:30 |
| TheJulia | heh, they are expecting to get the url with brackets | 14:30 |
| TheJulia | which is valid | 14:30 |
| dtantsur | Could it be that we're not supplying it correctly? but then how does it work at all? | 14:33 |
| TheJulia | That is a distinct possibility. I would sort of expect code to go "oh, I need to drop the brackets" tooso I'm wondering if it is just hidden c library smarts when it gets linked | 14:35 |
| TheJulia | There is also the interface binding which is occuring, not sure what all that is about | 14:35 |
| dtantsur | daaaaamn | 14:40 |
| TheJulia | find something major? | 14:40 |
| dtantsur | yeah, it's our fault. we don't escape the host in one case.. which should not really be hit, but it is | 14:40 |
| TheJulia | doh | 14:40 |
| dtantsur | s/escape/put in brackets/ you got it | 14:40 |
| dtantsur | https://github.com/metal3-io/ironic-image/blob/main/ironic-config/inspector.ipxe.j2#L8 | 14:40 |
| TheJulia | doh | 14:41 |
| dtantsur | my only question is why they hit unmanaged inspection | 14:43 |
| TheJulia | wrong addresses? | 14:46 |
| dtantsur | just 3G of logs to download, and I'll probably know.. | 14:47 |
| TheJulia | wheeeeeee | 14:47 |
| dtantsur | yep, you're right, wrong bootMACAddress | 15:01 |
| dtantsur | TIL people use these commands https://docs.openstack.org/python-ironic-inspector-client/latest/cli/index.html#list-interface-data so we probably need to migrate them too | 15:06 |
| dtantsur | Meanwhile, do I really want to know why the dibipa jobs have both glean and cloud-init on the ramdisks? maybe I do, maybe I don't? | 15:16 |
| TheJulia | short answer, I've been trying to fix that, and I need to have a long discussion with you about it in general | 15:16 |
| dtantsur | \o/ | 15:16 |
| TheJulia | the longer answer is cloud-init is not explicitly excised in builds/images when it comes from OS vendors | 15:18 |
| * dtantsur watches a situation where the whole inventory becomes a *KEY* in a **kwargs dictionary and slowly loses his mind | 15:23 | |
| dtantsur | haha, missing content-type makes ironic go insane | 15:30 |
| TheJulia | can you blame it? | 15:38 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Add missing headers to the inspection callback https://review.opendev.org/c/openstack/ironic-python-agent/+/905126 | 15:39 |
| dtantsur | given the state of my own sanity? not really :) | 15:39 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Support several API and Inspector URLs https://review.opendev.org/c/openstack/ironic-python-agent/+/903999 | 15:45 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Make inspection URL optional if the collectors are provided https://review.opendev.org/c/openstack/ironic-python-agent/+/904026 | 15:45 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: Change policy to enforce only new policy https://review.opendev.org/c/openstack/ironic-inspector/+/905119 | 15:54 |
| TheJulia | rpittau: so, do we have an idea on the forward path w/r/t the metal3-integration job? | 15:59 |
| JayF | TheJulia: we often set it -nv when it's broken like this for ironic-unrelated reasons, I'd +2A such a patch | 16:03 |
| JayF | TheJulia: I'll note they just enabled e2e tests on all PRs in metal3, per slack over there, so I am suspect that could be causing infrastructural issues | 16:03 |
| rpittau | TheJulia: I'm not sure what changed, I also see it's not always failing | 16:03 |
| TheJulia | Indeed, but if there is a quick fix there is no real reason to NV it if we can just fix it quickly | 16:03 |
| rpittau | JayF: | 16:03 |
| rpittau | I don't think that impacts our CI though | 16:04 |
| JayF | I assumed it might be running on the same machine(s) | 16:04 |
| rpittau | oh ok, no it's running on opendev CI | 16:04 |
| TheJulia | Shouldn't, could it be location on disk being used? aren't some of the machines out there only like 40 or 60gb, and then there are ones with a giant /opt ? | 16:04 |
| rpittau | this actually passed today https://zuul.opendev.org/t/openstack/build/bfee7cf06545478781baa9e3c0fd0bed | 16:04 |
| rpittau | TheJulia: yeah, I was wondering that too | 16:05 |
| TheJulia | okay, Then in that case I'll rev my dnm path once the patch on inspector to do the same finishes running | 16:05 |
| TheJulia | and just kind of live with it for now as an intermittent failure | 16:06 |
| rpittau | the disk space required seems also variable | 16:06 |
| rpittau | just checked a couple of failed jobs | 16:06 |
| TheJulia | that seems... weird | 16:07 |
| clarkb | github is having an outage/had a recent outage. Many of the recent failures are due to failures accessing github resources | 16:07 |
| rpittau | clarkb: yes, that's the other failures, but not clear why suddenly we're seeing the disk space issue too, or how can be related | 16:08 |
| clarkb | ok without links to where you see the disk is full (or info on which path has no space and where the job ran) it is hard for me to help beyond that. Just noticed that a handful of the latest failurse appear related to the github outage | 16:16 |
| rpittau | clarkb: thanks for the info, we saw the failures in various changes, for example https://zuul.opendev.org/t/openstack/build/776964071b3f44bcb3d5efaeee7a31c6 the directory involved is /shared which should not be a separate partition | 16:18 |
| clarkb | rpittau: can you link to where you see /shared is full? | 16:19 |
| clarkb | that job ran on ovh-bhs1 which has a single partition and filesystem. | 16:19 |
| clarkb | that filesystem should be roughly 80GB large iirc | 16:20 |
| rpittau | clarkb: sure, and sorry, I got the wrong change! | 16:20 |
| rpittau | in the ironic logs here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a0d/902009/7/check/metal3-integration/a0d63dd/controller/before_pivoting/ironic.log | 16:20 |
| rpittau | Failed to deploy instance: Disk volume where '/shared/tftpboot/master_images' is located doesn't have enough disk space. Required 446 MiB, only 225 MiB available space present. | 16:21 |
| clarkb | note you can link directly to that message like this: https://zuul.opendev.org/t/openstack/build/a0d63ddc5a8c4aa2b1acfd900df55489/log/controller/before_pivoting/ironic.log#1551 | 16:23 |
| rpittau | clarkb: yep, thanks | 16:24 |
| clarkb | that job ran on rax-ord which as two disks. One for / and then another that is typically mounted under /opt | 16:24 |
| clarkb | However the job is responsible for partitioning and formating a filesystem on the ephemeral disk and mounting it so I can't say for certain if /opt is a different fs in that particular job. The / disk is about 20GB and the epeheramal disk should be around 80GB iirc | 16:24 |
| clarkb | looks like these jobs collect system info at the end. May want to include `df` output | 16:25 |
| rpittau | clarkb: yes, I will do that | 16:26 |
| rpittau | teh space there can quickly decrease as we have a lot of images and containers runnning | 16:26 |
| rpittau | clarkb: thanks! | 16:26 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Add df logs to metal3 integration job https://review.opendev.org/c/openstack/ironic/+/905133 | 16:29 |
| clarkb | looking at https://zuul.opendev.org/t/openstack/build/a0d63ddc5a8c4aa2b1acfd900df55489/log/controller/system/docker-images.txt some of those images look fairly large. The ironic and sushy-tools in particular stand out to me. I think our (opendev and zuul) python images tend to be closer to 200-300MB. One exception is zuul-executor where we install multiple versions of | 16:30 |
| clarkb | ansible and for reasons ansible is quite large | 16:30 |
| clarkb | that said free space on / is going to be less than 20GB so optimizing the image size may not solve this particular problem | 16:30 |
| rpittau | clarkb: this images come from the metal3 project and they follow different rules, for example they're based on CentOS Stream 9 | 16:31 |
| clarkb | hopefully they are doing multistage builds and installing wheels into the final production images | 16:32 |
| rpittau | clarkb: it's actually a mix of things, but wheels are indeed installed in the final image | 16:33 |
| clarkb | rpittau: taking a quick glance the sushy-tools image appears to be based on debian and has two versions of python installed (3.9 and 3.11) as well as git and gcc. I suspect there is a fair bit of pruning that can be done | 16:40 |
| clarkb | but again I'm not sure that will address the job failing issue. However, it does seem like a fairly bloated image | 16:40 |
| rpittau | clarkb: oh yeah, I was thinking about the main ironic image | 16:40 |
| clarkb | I suspect they aren't doing a multistage build and installing from binary packages (wheels) which is why you end up with git and gcc installed. I have no idea why you need two versions of python | 16:41 |
| rpittau | clarkb: which dockerfile are you looking at though? we're basing sushy-tools on python3.9 https://github.com/metal3-io/ironic-image/blob/main/resources/sushy-tools/Dockerfile | 16:43 |
| clarkb | rpittau: I'm not looking at a dockerfile. I am running bash in quay.io/metal3-io/sushy-tools and using du to inspect disk consumption | 16:44 |
| JayF | rpittau: is that a version where apt is going to install reccomends by default? | 16:44 |
| JayF | rpittau: if so, that explains ~everything clarkb has seen | 16:44 |
| rpittau | clarkb: that image may be outdated | 16:44 |
| clarkb | rpittau: thats the exact image in your job? https://zuul.opendev.org/t/openstack/build/a0d63ddc5a8c4aa2b1acfd900df55489/log/controller/system/docker-images.txt#14 | 16:45 |
| rpittau | mmm alright then something's not right | 16:45 |
| clarkb | oh though I guess the image above it is a local image with the same id so maybe built locally and tagged that way? | 16:45 |
| rpittau | the size of the sushy-tools image there is way too big | 16:46 |
| rpittau | anyway I don't think this is the root cause for the disk space issue, but good to know, thanks clarkb | 16:47 |
| rpittau | good night! o/ | 17:07 |
| *** jlvillal_ is now known as jlvillal | 17:13 | |
| opendevreview | Julia Kreger proposed openstack/ironic master: Change to enforced policy by default https://review.opendev.org/c/openstack/ironic/+/902009 | 19:43 |
| TheJulia | so two relatively quick and easy bug fixes I'd like to get some eyes on: https://review.opendev.org/c/openstack/ironic/+/905022 and https://review.opendev.org/c/openstack/ironic/+/905038 basically issues encountered when changing the default policy enforcement over to the new RBAC policy | 19:45 |
| JayF | I looked at 905038 last night and it made no sense, it makes perfect sense now | 19:48 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Disable legacy RBAC policy by default. https://review.opendev.org/c/openstack/ironic/+/902009 | 19:48 |
| JayF | temporal nature of code review lol | 19:48 |
| TheJulia | lol | 19:49 |
| JayF | +2 both of those :) | 19:49 |
| TheJulia | it is true! | 19:49 |
| JayF | lets make sure the release note for swapping the defaults | 19:49 |
| JayF | is really really loud | 19:49 |
| JayF | I haven't looked at that change yet so maybe it is but just saying it out loud :D | 19:49 |
| JayF | like jay in a loud bar with a headcold trying to get someone's attention on the other side of the room loud lol | 19:50 |
| TheJulia | oh, both changes are very loud release notes | 19:51 |
| TheJulia | why is jay in the irc bar with a headcold though? | 19:51 |
| JayF | jay in a bar in general would be B.C. in general anyway lol | 19:55 |
| JayF | I haven't been in a bar that wasn't attached to somewhere like an Applebees or BWW in so long lol | 19:55 |
| TheJulia | .... We started to visit speakeasys | 19:55 |
| TheJulia | speaking of... | 19:55 |
| * TheJulia feels a need to meet the editor | 19:56 | |
| JayF | I used up my liver's capacity for alcohol in my 20s, so while speakeasys are cool I have to make sure my drinks are easy lol | 19:56 |
| TheJulia | makes tons of sense | 19:56 |
| JayF | alright, gotta go walk the dog while there's a lull in the wind/rain | 19:57 |
| TheJulia | ++ | 19:59 |
| JayF | I'd like to suggest someone who has knowledge of bifrost take a look at this and verify we don't have any dangling dependencies https://review.opendev.org/c/openstack/governance/+/877132/3/reference/projects.yaml | 21:40 |
| JayF | I'm 99.9999% sure we're OK, but don't trust myself to know what I may not know | 21:40 |
| TheJulia | yeah, none of that was used afaik | 21:43 |
| JayF | https://review.opendev.org/c/openstack/governance/+/905145/1/reference/projects.yaml | 21:45 |
| JayF | wrong link, but I think you're right anyway | 21:45 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Basic support for OVN VTEP switches https://review.opendev.org/c/openstack/ironic/+/900568 | 23:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!