| opendevreview | Merged openstack/ironic master: Fix system scoped manageable node network failure https://review.opendev.org/c/openstack/ironic/+/905022 | 04:43 |
|---|---|---|
| opendevreview | Merged openstack/ironic master: RBAC: Fix allocation check https://review.opendev.org/c/openstack/ironic/+/905038 | 04:43 |
| opendevreview | Merged openstack/networking-baremetal master: Do not try to bind port when we can't https://review.opendev.org/c/openstack/networking-baremetal/+/903252 | 04:55 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.2: Fix system scoped manageable node network failure https://review.opendev.org/c/openstack/ironic/+/905086 | 05:20 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: Fix system scoped manageable node network failure https://review.opendev.org/c/openstack/ironic/+/905087 | 05:21 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.2: RBAC: Fix allocation check https://review.opendev.org/c/openstack/ironic/+/905088 | 05:21 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: RBAC: Fix allocation check https://review.opendev.org/c/openstack/ironic/+/905089 | 05:22 |
| rpittau | good morning ironic! o/ | 07:48 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Make inspection URL optional if the collectors are provided https://review.opendev.org/c/openstack/ironic-python-agent/+/904026 | 07:55 |
| rpittau | JayF: re bifrost deps: from what I can see and just to confirm what TheJulia said, I don't think bifrost is impacted | 07:57 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [WIP] Deprecate inspector support https://review.opendev.org/c/openstack/bifrost/+/905192 | 08:01 |
| rpittau | the issue with disk space in metal3 integration job is really odd, but at least now we have confirmation that it's indeed / getting full https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_853/905133/1/check/metal3-integration/8532d95/controller/system/df--h.txt | 08:55 |
| rpittau | if we can increase / to 60 GB should be fine | 08:56 |
| rpittau | still I don't get what changed in terms of size to provoke this issue | 08:56 |
| dtantsur | yeah, it's odd indeed | 08:56 |
| rpittau | in this particular change we just miss few MB, 14 to be precise https://zuul.opendev.org/t/openstack/build/8532d952cc6843ed8488462c2366ac40/log/controller/before_pivoting/ironic.log#1548 | 08:58 |
| rpittau | anyway, let's see if we can increase / partition first to make the job pass | 08:58 |
| opendevreview | Merged openstack/ironic-python-agent master: Remove unnecessary egg_info options https://review.opendev.org/c/openstack/ironic-python-agent/+/904054 | 09:04 |
| opendevreview | Merged openstack/networking-baremetal master: Remove deprecated pbr options https://review.opendev.org/c/openstack/networking-baremetal/+/904058 | 09:25 |
| opendevreview | Merged openstack/ironic-python-agent master: Remove deprecated pbr options https://review.opendev.org/c/openstack/ironic-python-agent/+/904055 | 09:32 |
| opendevreview | Merged openstack/ironic bugfix/23.1: Handle LLDP parse Unicode error https://review.opendev.org/c/openstack/ironic/+/904860 | 09:34 |
| opendevreview | Merged openstack/sushy master: Allows System to access VirtualMedia in Sushy https://review.opendev.org/c/openstack/sushy/+/904463 | 10:02 |
| opendevreview | Merged openstack/metalsmith master: CI: Ask ironic devstack to set node owner https://review.opendev.org/c/openstack/metalsmith/+/905012 | 10:47 |
| rpittau | dtantsur: I proposed to reduce the size of sushy-tools, that should mitigate the issue https://github.com/metal3-io/ironic-image/pull/473 | 11:29 |
| rpittau | thanks clarkb for spotting the image size :) | 11:30 |
| dtantsur | nice! | 11:44 |
| edebeste | Hi all, is ironic able to target bluefield dpus to provision OS images to? | 11:54 |
| dtantsur | TheJulia: ^^ | 11:54 |
| dtantsur | rpittau: I wonder if we can stop installing libvirt-python from source. bookworm has 9.0.0, which is reasonably recent. | 12:07 |
| rpittau | right, that would probably save us some more space | 12:07 |
| rpittau | I'll add that to the change | 12:07 |
| dtantsur | yeah, not having gcc will help a ton as well | 12:10 |
| rpittau | mmm or maybe not, it went up to 1GB :D | 12:12 |
| rpittau | the python3-libvirt package is bringing in a ton of other packages | 12:12 |
| rpittau | but I can probably remove gcc after installation | 12:14 |
| rpittau | we're down to 350MB, hopefully CI will be happy about it :) | 12:20 |
| dtantsur | rpittau: you probably don't need gcc at all any more? | 12:22 |
| dtantsur | python stuff has to be installed from wheels (if it's not, we're going to have issues) | 12:22 |
| dtantsur | ugh, their networking is quite unstable now | 12:57 |
| rpittau | dtantsur: we need gcc because of the libvirt binding | 13:24 |
| rpittau | AFAIK libvirt python needs to be compiled as it's tied to the architecture and to the libvirt package installed in the OS | 13:31 |
| opendevreview | Merged openstack/ironic master: Make bandit voting on check and gate https://review.opendev.org/c/openstack/ironic/+/879498 | 13:31 |
| dtantsur | rpittau: hmm, no, the debian's python3-libvirt must play well with their libvirt | 13:32 |
| dtantsur | actually, pip install is riskier in this sense | 13:32 |
| rpittau | dtantsur: yes, but the pkg has so many devependencies that the image size is 1GB | 13:33 |
| rpittau | basically most of the image is taken by libvirt + python-libvirt when installing using apt | 13:33 |
| dtantsur | wow, that's crazy | 13:35 |
| dtantsur | okay, let's at least try to uninstall gcc afterwards | 13:36 |
| opendevreview | Jake Hutchinson proposed openstack/bifrost master: Bifrost NTP configuration https://review.opendev.org/c/openstack/bifrost/+/895691 | 13:38 |
| TheJulia | edebeste: so... the answer is sort of complicated, depending on the version of bluefield because, aiui, there are non-upstream patches in the mix from nvidia's ubuntu images and if that card has a real bmc, or if the bmc is just a VM in the main OS on the card. | 14:10 |
| TheJulia | edebeste: I've pondered adding a bfb-install clean step to kind of simplify that, but given rshim updates are only an nvidia image thing, that just seems more like resetting the bluefield's OS to what nvidia ships step, which is not really what most want. | 14:12 |
| edebeste | TheJulia: Thanks. Appreciate the response. I ask because I know my group is currently investigating it's use for a deployment and are eyeing maas instead due to their claim for bluefield support. I need to do more research into things myself as well. New to bluefield | 14:14 |
| TheJulia | edebeste: ahh, fun. So the bottom line is basically ubuntu is the only OS nvidia supports on bluefields | 14:15 |
| TheJulia | I know folks who have used ironic "and it just works" for their scenario with bluefields (i.e. a real BMC on the cards, and bluefield2s, not bluefield 3 cards) | 14:17 |
| TheJulia | but only a slim selection of cards have the real BMC | 14:17 |
| * TheJulia wonders if she should just hammer out a bfb-install clean step for the "hell of it" | 14:18 | |
| * dtantsur wonders if he should just hammer his laptop and run to the woods to live with owls for the "hell of it" | 14:29 | |
| edebeste | TheJulia: Thanks, that insightful | 14:30 |
| * TheJulia wonders if she should be sending dtantsur a beverage | 14:33 | |
| dtantsur | That could help :) | 14:37 |
| dtantsur | Some time ago, we agreed that it's better to have a per-node external_http_url rather than global external_http_url_v6 used for nodes with IPv6 BMC. I'm now seriously starting questioning this decision. | 14:41 |
| dtantsur | The problem I have right now is: the caller does not know the correct IP address. I don't know how I populate driver_info[external_http_url] without somehow learning Ironic's networking. | 14:42 |
| dtantsur | JayF: I think you was the most against it back in the days, or am I confusing something? | 14:42 |
| TheJulia | why can't an IP be allocated, assigned a DNS name, and ironic be informed of that DNS name? | 14:43 |
| JayF | I certainly don't know if I had some sort of problem with it in the past, but I do know that the need for this feature implies really really strange or bad network design | 14:43 |
| TheJulia | or IPs with A and AAAA records | 14:43 |
| dtantsur | First and foremost, that will need to push DNS settings to the BMC | 14:43 |
| JayF | For basically the reason that Julia is laying out in detail | 14:43 |
| dtantsur | nothing strange at all, just some BMC having only one networking stack, while Ironic being dual-stack | 14:44 |
| TheJulia | most people's BMC's I've seen have had dns configured actually | 14:44 |
| dtantsur | yeah, but it may not be your cluster's DNS | 14:44 |
| TheJulia | either through static reservations or actual config, since vendors now prefer software updates to be auto-magic | 14:44 |
| dtantsur | 8.8.8.8 won't help me much | 14:44 |
| TheJulia | wasn't there a initial state dns requirement for openshift back in the days? | 14:45 |
| dtantsur | I'm quite sure we don't add the OpenShift's internal DNS to BMCs | 14:45 |
| dtantsur | I'm equally sure that many customers will never allow that | 14:45 |
| dtantsur | (also, don't get me started about DNS in kubernetes) | 14:46 |
| dtantsur | (also, I have a bootstrap issue here - there is no openshift yet at all) | 14:47 |
| TheJulia | dtantsur: I thought the requirement was external to the cluster dns which was working before the cluster launched | 14:48 |
| TheJulia | overall, fair though | 14:48 |
| TheJulia | I just worry we're making a mistake by not embracing DNS, but there are valid reasons not to, just... duplicating parameters because of v6's presence is also NotGreat | 14:49 |
| TheJulia | err, NotGreat™ | 14:50 |
| dtantsur | I'm afraid "you need a valid DNS record for your temporary bootstrap VM on your laptop" is not a viable thing I can ask | 14:51 |
| TheJulia | I've heard crazier requirements in the past | 14:51 |
| TheJulia | :) | 14:51 |
| dtantsur | Dual-stack is generally a weird place. It has been added as an afterthought both to OpenStack and Kubernetes (including OCP and Metal3). | 14:51 |
| dtantsur | TheJulia: I'm not in the position to set arbitrary rules... | 14:52 |
| TheJulia | oh yeah | 14:53 |
| TheJulia | Always an afterthought :( | 14:53 |
| TheJulia | and the closer you get to the metal, the more people just expect it to be your problem and just make it all work | 14:54 |
| dtantsur | yea | 14:54 |
| * TheJulia should send hjensas some booze | 14:55 | |
| dtantsur | And v6-only BMCs are not entirely insane.. they just don't play well with v4-primary dualstacks | 14:55 |
| TheJulia | yup | 14:55 |
| dtantsur | and then there is this aspect of metal3's container that self-determine their networking | 14:55 |
| TheJulia | I'd almost prefer a giant "is this a v4 or a v6 bmc knob" | 14:55 |
| dtantsur | so, Ironic knows its IP addresses, but the callers (BMO, whatever) do not. | 14:56 |
| TheJulia | and by default, just treat the variables like you have FQDN | 14:56 |
| TheJulia | and you could kind of "guess" based upon the input | 14:56 |
| TheJulia | unless they are operating a well run/configured dual stack environment | 14:56 |
| TheJulia | and give you dns names to start | 14:56 |
| dtantsur | It absolutely does not help that bloody gopls takes GiB's of memory on the bloody openshift-installer repo | 15:00 |
| opendevreview | Merged openstack/ironic master: Fix versions in release notes https://review.opendev.org/c/openstack/ironic/+/904103 | 15:10 |
| opendevreview | Jake Hutchinson proposed openstack/bifrost master: Bifrost NTP configuration https://review.opendev.org/c/openstack/bifrost/+/895691 | 15:20 |
| JayF | Yeah, makes sense that bootstrapping is a problem. | 15:35 |
| JayF | I will say, none of those networks sound like the kinda ones I'd want deployed in my environment, but I can easily see how organic growth gets someone there :/ | 15:35 |
| dtantsur | JayF, TheJulia, so, is a combination of external_http_url_v6 with a driver_info[bmc_ip_version] something worth putting my thoughts in or is it still rather meh from your perspective? | 15:59 |
| JayF | my preference would be driver_info[bmc_ip_version] | 16:00 |
| dtantsur | it's not really either/or: we need a 2nd configuration for the alternative version | 16:00 |
| JayF | I keep trying to figure out some way to magic the other half | 16:00 |
| JayF | but you can't | 16:00 |
| dtantsur | or something like external_http_url_per_version | 16:00 |
| dtantsur | ironic is not self-conscious enough to determine its own networking :) | 16:01 |
| JayF | honestly, it makes me wish we had split urls | 16:01 |
| dtantsur | split? | 16:01 |
| JayF | external_http_host { v4: 1.2.3.4, v6: 1.2.3.4 } | 16:01 |
| JayF | er lol | 16:01 |
| JayF | external_http_host { v4: 1.2.3.4, v6: ::11 } | 16:01 |
| JayF | external_http_path "/baremetal/v1" | 16:01 |
| JayF | that sort of thing | 16:01 |
| dtantsur | gotcha (needs a port as well) | 16:02 |
| JayF | yeah, and I'm not saying we should change | 16:02 |
| JayF | I'm just saying at least the config would be more sensible if it was structured that way | 16:02 |
| JayF | provide one dns value or two ips for the host | 16:02 |
| JayF | You have real world problems, you should try to solve them, I'm just trying to hammer it into a generalized shape and it may not be possible :D | 16:03 |
| opendevreview | Jake Hutchinson proposed openstack/bifrost master: Bifrost NTP configuration https://review.opendev.org/c/openstack/bifrost/+/895691 | 16:07 |
| TheJulia | I do sort of agree with the sentiment overall, but I also feel like there is a line around asserting DNS servers to BMCs remotely which is both a feature, and something we should never do unless we're explicitly asked to do it | 16:15 |
| TheJulia | and there didn't seem to be a ton of interest around asserting things and stuff to the bmc networking configuration wise when we discussed it at the PTG | 16:16 |
| clarkb | rpittau: you might also want to consider a multistage build. The first stage can install all your build deps and produce wheels for everything then the second stage will install the wheels and not need the extra bits like git and gcc and all the resulting intermediate build artifacts | 16:19 |
| clarkb | rpittau: if you are interested in that you can look at https://opendev.org/opendev/system-config/src/branch/master/docker/python-builder (stage 1) and https://opendev.org/opendev/system-config/src/branch/master/docker/python-base (stage 2) and how to consume them https://opendev.org/opendev/system-config/src/branch/master/docker/ircbot/Dockerfile | 16:21 |
| rpittau | clarkb: yep, that's what we do in other cases | 16:21 |
| TheJulia | Regarding the two rbac related bug fixes, both of which have merged against master (Thanks!), I've backported them to 2023.1. I'm not sure there is really a need to backport further. | 16:29 |
| rpittau | good night! o/ | 17:01 |
| Sandzwerg[m] | hmm, was there any request for encryption at rest for BMC credentials when they get stored on the node level? | 17:05 |
| TheJulia | Sandzwerg[m]: there has been some mild expression of desire, but nobody has ever stepped forward to work on it. | 17:12 |
| Sandzwerg[m] | I see. Assumed something like this. | 17:12 |
| TheJulia | given the API also doesn't expose password value, that has also seemed reasonable for most folks, some folks have mused integrating with external services, but we also use them a lot for things like nodes supporting ipmi because we don't cache the credentials in ram. | 17:14 |
| TheJulia | The only case where credentials might get cached in ram is with redfish sushy clients | 17:15 |
| TheJulia | Since we cache those, largely due for their sessions with the established credentials, but they need to be able to re-authenticate when the session expires. | 17:16 |
| Nisha_Agarwal | TheJulia, | 17:18 |
| Nisha_Agarwal | hi | 17:18 |
| TheJulia | Hi Nisha, how are you today? | 17:18 |
| Nisha_Agarwal | TheJulia, doing good :) How are you? | 17:18 |
| TheJulia | I'm alright, any response from your firmware team? | 17:19 |
| Nisha_Agarwal | TheJulia, I am seeing something different...wanted to ask on that...Since RHOSP is kolla based, which ip address is associated with ironic conductor container | 17:20 |
| Nisha_Agarwal | In other words, i dont see the ip address of the undercloud in the client details | 17:21 |
| Nisha_Agarwal | of the session | 17:21 |
| Nisha_Agarwal | hence asking | 17:21 |
| TheJulia | Do you have an HTTP proxy in place? | 17:21 |
| Nisha_Agarwal | Yes | 17:21 |
| TheJulia | ohhh... | 17:21 |
| TheJulia | that might be the issue then | 17:21 |
| Nisha_Agarwal | proxy was required for undercloud setup itself...else podman doesnt work | 17:22 |
| TheJulia | Anyway, where are you looking and not seeing | 17:22 |
| Nisha_Agarwal | Ok will paste the output | 17:22 |
| TheJulia | ok | 17:22 |
| TheJulia | I'm trying to remember which ironic is used in that case, I guess I need to go look at the logs | 17:26 |
| Nisha_Agarwal | for example i see the ip address as "ClientOriginIPAddress": "10.79.90.44" | 17:28 |
| Nisha_Agarwal | when i do GET on the session uri for a particular session | 17:28 |
| Nisha_Agarwal | and the undercloud ip starts with 128* | 17:28 |
| TheJulia | i guess that is a header your proxy is injecting? | 17:28 |
| Nisha_Agarwal | ok could be...i need to check that but in that case how will it work in devstack env | 17:30 |
| Nisha_Agarwal | proxy is used there also | 17:30 |
| Nisha_Agarwal | the client origin ip looks to be proxy ip... | 17:33 |
| TheJulia | Okay, that makes sense then | 17:33 |
| Nisha_Agarwal | The firmware team Connection: close just closes the network socket and doesnt invalidate the session | 17:39 |
| Nisha_Agarwal | and a session key expiry is by default 24 hrs | 17:39 |
| Nisha_Agarwal | TheJulia, ^^^ | 17:40 |
| TheJulia | Interesting | 17:40 |
| TheJulia | I wonder if we're hitting something with the proxy then | 17:40 |
| Nisha_Agarwal | may be...when i use curl then the ClientOriginIpAddress is the system IP | 17:41 |
| TheJulia | dtantsur: do you have a good example for the networkData field in metal3 ? | 17:42 |
| TheJulia | specifically with openshift | 17:42 |
| dtantsur | TheJulia: for the final instance, not for the ramdisk? I don't think I do, we don't use this feature. | 17:42 |
| TheJulia | ramdisk is what I'm looking for | 17:44 |
| dtantsur | TheJulia: point 4 https://docs.openshift.com/container-platform/4.14/installing/installing_bare_metal_ipi/ipi-install-expanding-the-cluster.html#preparing-the-bare-metal-node_ipi-install-expanding | 17:45 |
| TheJulia | perfect, thanks! | 17:45 |
| TheJulia | I'm guessing userData is likely just a string of json? | 17:47 |
| TheJulia | which would align with the config drive in ironic's api? | 17:47 |
| dtantsur | yeah | 17:47 |
| dtantsur | TheJulia: note that openshift does *not* use the Ironic's network_data. Also note that this uses the preprovisioningNetworkData field, which is CoreOS-specific. | 17:48 |
| TheJulia | ack | 17:48 |
| TheJulia | hmmmmm | 17:48 |
| Nisha_Agarwal | i can see when i used sushy , then the ClientOriginIPAddress is proxy ip | 17:49 |
| TheJulia | dtantsur: oh, interesting, so it looks like some of the docs exclude preprovisioningNetworkData | 17:50 |
| dtantsur | In practice, it may mean that you need to provide the network config twice: for the ramdisk like shows in the document and for the instance through the normal networkData | 17:51 |
| TheJulia | yup | 17:51 |
| TheJulia | Nisha_Agarwal: where exactly are you seeing that, since it is not something we log or report on afaik | 17:53 |
| Nisha_Agarwal | TheJulia, i have put a pdb in sushy where it gets the authtoken | 18:00 |
| Nisha_Agarwal | where we populate the auth token and the session_uri | 18:01 |
| TheJulia | is it in the header responses back on the request? | 18:01 |
| TheJulia | err, response header | 18:01 |
| Nisha_Agarwal | So i just get that session uri and used a GET call on that session uri | 18:01 |
| Nisha_Agarwal | in that GET output we can see the field as "ClientOriginIPAddress": "1.1.1.1" | 18:02 |
| Nisha_Agarwal | So this curl -gikx "" -H 'X-Auth-Token:3vFTqTQRTpORP8seu0ikAQ==' -X GET https://<ip>/redfish/v1/SessionService/Sessions/Administrator75516825a7ad4051a0be59aa865a69a7 | 18:03 |
| Nisha_Agarwal | gives us the response as | 18:03 |
| Nisha_Agarwal | {"@odata.etag": "\"30e0fdb421510233428419775960592b\"", "@odata.id": "/redfish/v1/SessionService/Sessions/Administrator75516825a7ad4051a0be59aa865a69a7", "@odata.type": "#Session.v1_7_0.Session", "ClientOriginIPAddress": "1.1.1.1", "CreatedTime": "2024-01-10T17:57:23Z", "Id": "Administrator75516825a7ad4051a0be59aa865a69a7", "Name": "User Session Administrator75516825a7ad4051a0be59aa865a69a7", "Oem": {"Hpe": {"@odata.type": | 18:03 |
| Nisha_Agarwal | "#HpeH3Session.v1_0_0.HpeH3Session", "InitialRole": "Administrator"}}, "SessionType": "Redfish", "UserName": "Administrator"} | 18:03 |
| Nisha_Agarwal | Here the ClientOriginIPAddress for sushy in RHOSP17.1 is proxy ip | 18:04 |
| TheJulia | Ahh, okay, your directly curling the IP from the session list | 18:04 |
| TheJulia | well, curling the url of the session | 18:04 |
| Nisha_Agarwal | i checked once in non-RHOSP env for the auth token created by sushy...and did the same above i saw ClientIPAddress as the system IP | 18:05 |
| Nisha_Agarwal | for non-RHOSP i can test more but for RHOSP env i have seen this now many times | 18:05 |
| TheJulia | which makes sense, it inherently doesn't get the environment variable for the proxy | 18:06 |
| Nisha_Agarwal | and hence sure with sushy it adds the IP of the proxy, while in curl it adds the IP of system | 18:06 |
| TheJulia | I think it is more a matter of the remote BMC is recording the IP where the session originates | 18:06 |
| TheJulia | the post to create the session is really quite simple with two fields in response | 18:07 |
| TheJulia | a token and a uri | 18:07 |
| Nisha_Agarwal | And i think thats where sushy token is not getting validated... | 18:07 |
| Nisha_Agarwal | hmmm i will speak to firmware team once more...but not so sure...as of now everything is hit and trial | 18:08 |
| TheJulia | do you mean if the IP somehow changes, you think the redfish endpoint is invalidating the request | 18:08 |
| TheJulia | ? | 18:08 |
| Nisha_Agarwal | Yes...looks like | 18:08 |
| TheJulia | That seems like a weird feature | 18:08 |
| TheJulia | but would explain it entirely if there is a transparent proxy that picks up the request | 18:08 |
| TheJulia | or even a proxy cluster | 18:08 |
| TheJulia | I remember when I was at HPE, all browser requests could end up with with like one of seven IPs | 18:09 |
| Nisha_Agarwal | hmmm... | 18:09 |
| Nisha_Agarwal | wierd thing is that i didnt see this for non-RHOSP env | 18:09 |
| TheJulia | That was also a very long time ago | 18:09 |
| TheJulia | because it likely picks up your shell enviornment proxy environment variables | 18:10 |
| TheJulia | where as in the container, there are none | 18:10 |
| Nisha_Agarwal | for non-RHOSP where proxy is still there, i see the ClientIP is system ip address | 18:10 |
| TheJulia | Yeah, which makes sense, because there is no proxy IP | 18:10 |
| Nisha_Agarwal | proxy is there in both | 18:10 |
| Nisha_Agarwal | if there is no proxy in container then it shouldnt pick up the proxy ip when i use sushy | 18:11 |
| TheJulia | it could be writing the X-Forwarded-For header to ClientIP in the session json/tracking | 18:11 |
| Nisha_Agarwal | but actually when it has the proxy ip , i guess thats where it is failing.... | 18:11 |
| Nisha_Agarwal | will check with firmware team and see if they can crack anything with this | 18:12 |
| TheJulia | can you unset your proxy environment variables and see what happens? | 18:12 |
| Nisha_Agarwal | using sushy? | 18:12 |
| TheJulia | yeah | 18:12 |
| TheJulia | That would explain the basic fundamental operating difference | 18:13 |
| Nisha_Agarwal | yes i ran on etime after unsetting the proxy, and i can see the system ip | 18:20 |
| Nisha_Agarwal | in clientip | 18:21 |
| TheJulia | your machine, or the undercloud? | 18:21 |
| Nisha_Agarwal | undercloud | 18:21 |
| Nisha_Agarwal | from non-RHOSP it always adds the system IP | 18:21 |
| Nisha_Agarwal | even with proxy being there | 18:22 |
| TheJulia | but sushy sessions from the undercloud you get the proxy IP as the ClientIP? | 18:22 |
| Nisha_Agarwal | yes | 18:23 |
| Nisha_Agarwal | even if i assume its firmware issue, i should see the same behaviour irrespective of RHOSP and non-RHOSP | 18:24 |
| TheJulia | Eh, not if the input is different | 18:24 |
| Nisha_Agarwal | input is same...on the same system | 18:24 |
| Nisha_Agarwal | with curl, on undercloud we see the systemIP in the ClientIP | 18:25 |
| Nisha_Agarwal | i will check with firmware team also on this...But request you to check if some network setting on RHOSP side can solve this | 18:26 |
| Nisha_Agarwal | network/or general setting | 18:26 |
| Nisha_Agarwal | we tested 4 scenarios: | 18:27 |
| Nisha_Agarwal | RHOSP Undercloud <--> RHOSP SDFLEX BM target ---> fails with attribute missing error | 18:28 |
| Nisha_Agarwal | RHOSP Undercloud <--> Devstack SDFLEX BM target ---> fails with attribute missing error | 18:28 |
| Nisha_Agarwal | Devstack System <--> RHOSP SDFLEX BM target ---> fails with attribute missing error | 18:28 |
| Nisha_Agarwal | Devstack Sytem <--> Devstack SDFLEX BM target ---> doesn't fail | 18:28 |
| Nisha_Agarwal | so this is for sure that the issue is seen only in RHOSP env | 18:29 |
| Nisha_Agarwal | and ClientIP thing can be one of the possible reason... | 18:30 |
| TheJulia | The wait,s o you have two different targets? | 18:31 |
| Nisha_Agarwal | nopes...these are all the scenarios we tested to root cause the error | 18:32 |
| TheJulia | so your saying firing up devstack on rhel, it fails? | 18:32 |
| Nisha_Agarwal | nopes | 18:32 |
| Nisha_Agarwal | i said if u have a devtsack setup | 18:33 |
| Nisha_Agarwal | and try to do simple two steps, | 18:34 |
| Nisha_Agarwal | create sushy object and then do get_system | 18:34 |
| TheJulia | so I'm trying to understand your different scenarios, and I guess I'm sure of what you mean by "RHOSP Undercloud <--> Devstack SDFLEX BM target" | 18:34 |
| Nisha_Agarwal | here i meant that from RHOSP undercloud we tried to deploy(or simply use python shell to root cause) a Overcloud Baremetal target | 18:35 |
| Nisha_Agarwal | here the Overcloud Baremetal target is SuperdomeFlex series of servers | 18:36 |
| TheJulia | so how is that different from "RHOSP Undercloud <--> RHOSP SDFLEX BM target" ? | 18:36 |
| Nisha_Agarwal | the same series of servers which are not part of RHOSP subnet while they are part of devstack subnet, we do not hit the issue when we try the same tests from devstack subnet to devstack subnet. | 18:37 |
| Nisha_Agarwal | BM target is the target system which we want to certify | 18:37 |
| TheJulia | so the targets are just different groups of servers? | 18:38 |
| Nisha_Agarwal | wait...i think u r confused | 18:39 |
| TheJulia | Very confused, which is why I'm asking questions trying to clarify | 18:39 |
| Nisha_Agarwal | undercloud is normal HPE server(Not SuperDome Flex system) | 18:40 |
| Nisha_Agarwal | in Overcloud servers we have 3 SuperDomeFlex servers(SuperdomeFlex, SuperDomeFlex 280 and HPE ScaleupCompute 3200) | 18:41 |
| Nisha_Agarwal | we can hit the issue on all of these servers from undercloud | 18:41 |
| TheJulia | And nodes accessed from the undercloud gets a proxy IP as the ClientIP of the session record on the BMC? | 18:43 |
| Nisha_Agarwal | Yes | 18:43 |
| TheJulia | Okay, and the devstack setup, is the proxy being leveraged? | 18:44 |
| TheJulia | and in those cases the ClientIP is the devstack node? | 18:44 |
| Nisha_Agarwal | yes it is there in devstack also | 18:46 |
| Nisha_Agarwal | but the ClientIP is the systemIP even though the proxy is there | 18:46 |
| TheJulia | because those processes launch with the environment variables which include a proxy parameter | 18:46 |
| Nisha_Agarwal | hmmmm | 18:46 |
| TheJulia | The container launched processes shouldn't have them aiui | 18:46 |
| Nisha_Agarwal | if we add the baremetal IP to no_proxy, proxy has no effect, right? | 18:47 |
| TheJulia | it should not be used then | 18:47 |
| TheJulia | although, it sounds like we've got a transparent proxy somewhere in the mix | 18:47 |
| Nisha_Agarwal | hmmm not sure...will do some more experiments tomorrow | 18:48 |
| Nisha_Agarwal | may be something to do with proxy | 18:48 |
| Nisha_Agarwal | and thats leading to all thi | 18:48 |
| TheJulia | That is what it really does sound like | 18:48 |
| TheJulia | since it gets the ip even without it being explicitly configured or being in the environment variables | 18:48 |
| Nisha_Agarwal | will do some more experiments tomorrow and then ask the engineer to trigger deploy with no_proxy | 18:49 |
| TheJulia | and if that IP changes due to infrastucture config or a load balancer of some sort, and the "bmc" invalidates the request then, that would explain everything, really | 18:49 |
| TheJulia | okay | 18:49 |
| Nisha_Agarwal | infratstructre config means RHOSP config? | 18:49 |
| TheJulia | no, like your switches/routers | 18:50 |
| TheJulia | well, routers, really | 18:50 |
| Nisha_Agarwal | ok | 18:50 |
| TheJulia | I'm not sure we have explicit proxy passing capability in rhosp | 18:50 |
| Nisha_Agarwal | hmmm | 18:51 |
| TheJulia | so when reproducing the issue, you get an AccessError with dmitry's patch right? | 19:07 |
| Nisha_Agarwal | Yes | 19:07 |
| TheJulia | okay | 19:08 |
| TheJulia | That is good | 19:08 |
| Nisha_Agarwal | but that patch is not yet there in our RHOSP as i think then we need to do undercloud setup again | 19:10 |
| Nisha_Agarwal | we jsut checked manually and did see the correct error message | 19:11 |
| TheJulia | ok | 19:14 |
| TheJulia | gah, I never did service/adminy access to ironic-insector | 23:43 |
| *** jph4 is now known as jph | 23:48 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!