| opendevreview | Verification of a change to openstack/ironic-inspector stable/xena failed: CI: Zuul no longer respects queue https://review.opendev.org/c/openstack/ironic-inspector/+/860162 | 00:02 |
|---|---|---|
| opendevreview | Merged openstack/ironic-lib stable/xena: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-lib/+/860173 | 00:46 |
| opendevreview | Merged openstack/ironic-lib stable/wallaby: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-lib/+/860174 | 01:13 |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/train failed: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860190 | 01:27 |
| opendevreview | Merged openstack/ironic-inspector stable/yoga: CI: Zuul no longer respects queue https://review.opendev.org/c/openstack/ironic-inspector/+/860161 | 01:28 |
| opendevreview | Verification of a change to openstack/ironic-inspector stable/wallaby failed: CI: Zuul no longer respects queue https://review.opendev.org/c/openstack/ironic-inspector/+/860163 | 01:28 |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860186 | 02:46 |
| opendevreview | Verification of a change to openstack/ironic-python-agent bugfix/8.1 failed: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860194 | 02:46 |
| opendevreview | Merged openstack/ironic-python-agent stable/ussuri: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860189 | 04:38 |
| opendevreview | Merged openstack/ironic-python-agent stable/victoria: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860188 | 04:38 |
| opendevreview | Merged openstack/ironic-python-agent stable/yoga: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860185 | 04:38 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/8.3: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860195 | 04:39 |
| arne_wiebalck | Good morning, Ironic! | 06:27 |
| jssfr | Good Morning, arne_wiebalck! | 06:28 |
| arne_wiebalck | hey jssfr o/ | 06:28 |
| janders | good morning arne_wiebalck jssfr and Ironic o/ | 09:37 |
| opendevreview | Nisha Agarwal proposed openstack/ironic master: Fixes anaconda deploy for PXE boot https://review.opendev.org/c/openstack/ironic/+/860055 | 10:13 |
| janders | ajya big thank you for your insights in the review of https://review.opendev.org/c/openstack/sushy/+/856597. I've done further testing on this and will share some of the results soon. Meanwhile - you mentioned that iDRACs would not allow update for immediately-settable properties and hence the Settings resource would reject such updates. Did you | 10:13 |
| janders | have a particular machine in mind and if so, what was it? (while testing, I wanted to have a look at the iDRAC implementation but the R640 I have doesn't seem to present a Settings Resource) | 10:13 |
| ajya | janders: Settings will be added in future release. If you want something particular to check, I can do it. Or can leave iDRAC out for now and I'll keep an eye on the patch to see if there is nothing conflicting. | 10:20 |
| dtantsur | oh, I've just read the comment... the redfish situation is getting crazier each day | 10:26 |
| ajya | dtantsur: it looks like a new pattern that sushy hasn't encountered before. Ofc it would be easier if all settings could be updated in one place. | 10:27 |
| dtantsur | janders: on the machine we're looking at, are the properties returned in GET of the settings resource (i.e. /Pending)? | 10:27 |
| dtantsur | ajya: same question for you re Dell machines ^^ | 10:27 |
| dtantsur | if NO in both cases, we can use that as an indicator | 10:27 |
| ajya | dtantsur: no, they are not, and this could be the way to determine if these settings are patchable through Settings resource or not | 10:28 |
| dtantsur | okay, this is great! now we need to confirm the same on the machine janders is dealing with | 10:28 |
| dtantsur | janders: and I wonder if we still have a way to verify the same with the machine from https://storyboard.openstack.org/#!/story/2009871 | 10:30 |
| ajya | note there is 3rd property, not usually used, BootSourceOverrideMode that is patchable through Settings resource | 10:30 |
| dtantsur | I think we do use it | 10:30 |
| dtantsur | meaning, we may need to patch 2 resources | 10:31 |
| ajya | it's in the code, but looking at the actual request it is not used for iDRAC | 10:31 |
| ajya | it depends on the settings I guess | 10:31 |
| dtantsur | isn't it legacy vs UEFI? | 10:31 |
| ajya | yup | 10:31 |
| dtantsur | then we do use it in case the requested boot mode does not match the current | 10:31 |
| janders | dtantsur here is the GET from the Settings URI on the Lenovo in question: https://paste.openstack.org/show/bzwXNobL7iF5kyd3G0UG/ | 10:32 |
| ajya | and yes, in the end there would be 2 requests - 1 for active resource (BootSourceOverrideTarget, BootSourceOverrideEnabled) and another for Settings/Pending (BootSourceOverrideMode). For now backward compatibility is still there for iDRAC to patch all through active resource. | 10:32 |
| dtantsur | janders: sweet. Then we need to fetch the settings resource, see what is there. What is not, patch directly. | 10:32 |
| janders | I've done some digging in this today and where I am puzzled is it would seem to me that the Settings URI could be a generic mechanism of checking what changes are requested but not yet active | 10:33 |
| dtantsur | ajya: are there any ordering concerns? i.e. the active resource must be patches before the pending settings? | 10:33 |
| janders | but - if I try to make other changes (e.g. change Secure Boot settings) they don't end up there | 10:33 |
| janders | it seems the only thing that ever uses the /Pending/ URL is boot sequence changes | 10:33 |
| ajya | janders: yes, once you patch something you will see future values there until applied (e.g., on reboot), but for now we are not using that in anyway in sushy afaik | 10:34 |
| ajya | if not.. could be implementation differences, but I'll double check on iDRAC | 10:34 |
| janders | yet - 1) it takes effect immediately and 2) the setting is visible there which it shouldn't according to the table you referenced IIUC | 10:35 |
| dtantsur | janders: we should not use the settings resource for any fields that are not in its GET representation | 10:36 |
| dtantsur | I think that's the gist of this conversation? | 10:36 |
| ajya | dtantsur: I don't think the order of PATCHing matter here. For active resource the changes are applied immediately, Settings resource requires reboot. We want to have both changes already made at the point of rebooting to avoid rebooting twice (1st for Settings to change legacy/UEFI), 2nd for pxe/virtual media boot. | 10:38 |
| janders | dtantsur, I'm not disagreeing just still not clear to me how this is supposed to work when implemented correctly | 10:38 |
| dtantsur | ajya: okay, good. Another concern: do we need a reboot between attaching a virtual media and configuring boot settings? | 10:39 |
| janders | another finding I wanted to share: | 10:39 |
| dtantsur | ... the latter via the settings resource, I mean | 10:39 |
| janders | Nokia's settings resource: https://review.opendev.org/c/openstack/sushy/+/830553/13/sushy/tests/unit/json_samples/settings-nokia.json | 10:39 |
| janders | Lenovo SE450: https://review.opendev.org/c/openstack/sushy/+/856597/9/sushy/tests/unit/json_samples/settings-lenovo-se450.json#4 | 10:39 |
| janders | note the SupportedApplyTimes field present in the Lenovo and not in the Nokia | 10:39 |
| ajya | dtantsur: I'd say no, we already do it it one go | 10:40 |
| dtantsur | ack, thanks | 10:40 |
| dtantsur | janders: I don't think we're actually using SupportedApplyTimes in this context? | 10:40 |
| dtantsur | we can assume OnReset since we're going to reset anyway | 10:41 |
| janders | I was more thinking whether the presence and value of this field can indicate which scenario from the table we're dealing with | 10:41 |
| janders | (the table = section 9.10 Settings resource, Table 23) | 10:42 |
| janders | but - with the approach you proposed it may not matter | 10:43 |
| dtantsur | ¯\_(ツ)_/¯ | 10:43 |
| janders | now I wonder how this would (or wouldn't) work on the Nokia | 10:44 |
| dtantsur | ¯\_(ツ)_/¯ *2 | 10:45 |
| dtantsur | janders: do we have any chances of getting back to whoever complained about them? | 10:45 |
| janders | I don't believe I have access to one of these machines anymore but from the samples it looks like they may have no reference to BootSequenceOverride in the Settings URI BUT still expect us to make the change there: https://review.opendev.org/c/openstack/sushy/+/830553/13/sushy/tests/unit/json_samples/settings-nokia.json | 10:45 |
| janders | yes, I can do that (and maybe get access back( | 10:46 |
| janders | dtantsur ajya do you know what determines when Properties appear in the Settings URI? | 10:46 |
| janders | I'm curious about the difference between this: | 10:46 |
| * dtantsur does not | 10:46 | |
| janders | https://review.opendev.org/c/openstack/sushy/+/856597/9/sushy/tests/unit/json_samples/settings-lenovo-se450.json#4 | 10:46 |
| janders | and this (same, just gathered now): https://paste.openstack.org/show/bzwXNobL7iF5kyd3G0UG/ | 10:47 |
| janders | both are from the Lenovo | 10:47 |
| janders | I will collect the first one again from the live system to make sure it's the same thing | 10:48 |
| opendevreview | Nisha Agarwal proposed openstack/ironic master: Fixes anaconda deploy for PXE boot https://review.opendev.org/c/openstack/ironic/+/860055 | 10:49 |
| janders | OK, answering my own question: the json file is an extract from the System representation, not a response to a GET against the Settings URI | 10:50 |
| ajya | janders: if they can be applied immediately and no reboot is required, or application time cannot be delayed, e.g., in the maintenance timeframe | 10:50 |
| janders | dtantsur I will try to get the GET response against /redfish/v1/Systems/Self/SD from the Nokia to validate the direction we want to take | 10:52 |
| ajya | introduction of Settings allows some control over ApplyTime, have TaskMonitor URL in response to watch for result, though here for sushy it's not must have, as ApplyTime supports OnReset only anyway, and don't need to watch for TaskMonitor as it's sufficient to successfully boot IPA to continue. | 10:53 |
| janders | my current understanding (please correct me if I am wrong) is: if the Nokia lists the Boot Sequence related properties under /SD then we can take the approach we discussed (checking what is listed under Settings URI). And if not we need to re-assess. Correct? | 10:54 |
| ajya | agree | 10:55 |
| ajya | if the don't list and still require to use /SD, then it's different behavior compared to Lenovo and iDRAC | 10:56 |
| janders | I will try to get the person I was working with on the Nokia fix to run that GET query and then we'll know | 10:57 |
| janders | Will keep you posted | 10:57 |
| dtantsur | ++ | 11:02 |
| janders | ajya dtantsur: also - do you think this is worth a mention at the PTG? | 11:08 |
| dtantsur | janders: depending on which audience you want to reach | 11:09 |
| janders | dtantsur my first thought was sharing our findings and comparing notes with fellow contributors who may have run into this (TL;DR - discussions among devs) | 11:10 |
| iurygregory | morning Ironic | 11:29 |
| *** ravlew is now known as Guest2256 | 11:38 | |
| janders | hey iurygregory | 11:39 |
| iurygregory | janders, o/ | 11:40 |
| *** ravlew5 is now known as ravlew | 11:52 | |
| ajya | janders: atm don't feel a need for a PTG topic, especially, if solution will be found | 11:59 |
| janders | ajya ACK | 12:00 |
| TheJulia | good morning | 13:06 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix allocations default table type https://review.opendev.org/c/openstack/ironic/+/860198 | 13:09 |
| iurygregory | morning TheJulia | 15:00 |
| TheJulia | dtantsur: JayF: fyi https://review.opendev.org/c/openstack/ironic/+/860198 should fix things up. I'm not sure if we want to go to the point of putting in mysql specific migration code and ultimately alter table on those things could be not fun or just not matter. | 15:17 |
| dtantsur | TheJulia: do we need a migration? or do we assume that creating a database with 4-bytes utf-8 was impossible to begin with? | 15:39 |
| TheJulia | *well* | 15:39 |
| TheJulia | allocations, was likely Latin1 when created on a number of systems | 15:40 |
| JayF | dtantsur: TheJulia: iurygregory: https://review.opendev.org/c/openstack/releases/+/860125 any feelings on Elod's comment about doing minor bumps instead of patch bumps? | 15:41 |
| JayF | I'm fairly certain patch bumps are the right move; but am happy to take input if others have an opinion | 15:41 |
| * TheJulia sighs | 15:41 | |
| TheJulia | I'd go patch bumps, but I won't be able to look at Elod's comments until later | 15:42 |
| dtantsur | have we really added a requirement? oO | 15:43 |
| TheJulia | oh, driver fix which was already a openstack requirement in general | 15:43 |
| dtantsur | which one? | 15:44 |
| dtantsur | a new upper cap in https://opendev.org/openstack/ironic/commit/6c0152afa141d05ee28cba81178622021574ae17#diff-7503dc13ce10d004ddbfcd349619d074d68513e6 does not seem a huge deal to me | 15:45 |
| dtantsur | but if we really ADDED a requirement, it's a big ugh | 15:45 |
| dtantsur | ah, wait, in the end | 15:45 |
| dtantsur | JayF: my feeling is that it has to be reverted per stable policy | 15:45 |
| dtantsur | so you probably don't want me to comment on that release request ;) | 15:46 |
| TheJulia | security related, fwiw | 15:46 |
| JayF | dtantsur: I said basically the same thing, and nack'd that change at one point, but it ended up going through for reasons | 15:46 |
| JayF | if it's security, that's probably why | 15:46 |
| dtantsur | if we let it slip in, we should stop pretending we care about the stable policy, at least when it comes to requirements | 15:46 |
| JayF | I believe I was told we already ignored it for *added* requirements (vs changing the version of an existing one) | 15:47 |
| JayF | I'm gonna be honest, I'm OK with reverting it if that change is a problem | 15:47 |
| dtantsur | this does not match my perception | 15:47 |
| TheJulia | it was already in g-r, fwiw. | 15:47 |
| dtantsur | g-r has nothing to do with this | 15:47 |
| dtantsur | there are no obligation for downstreams to package or ship everything that is in g-r | 15:47 |
| JayF | the only reason I'd be hesitant to revert it is that most of the downstream packagers, aiui, pull from git by default | 15:48 |
| JayF | in which case they've had this in tree, or available in tree, for a while | 15:48 |
| dtantsur | I'd probably not revert the whole patch, but I'd consider seriously 1) whether we really need a new package just to parse a version, 2) a written policy for this sort of cases from now on | 15:49 |
| TheJulia | realistically, we should just move it to driver-requirements.txt instead | 15:49 |
| dtantsur | it does not really change anything | 15:49 |
| TheJulia | That makes this whole thing moot since it is for a driver, at least in my view | 15:49 |
| dtantsur | the stable policy applies to driver-requirements just as well | 15:49 |
| JayF | dtantsur++ that seems like the solution that has the most grace. Don't back out what's already there and consider it a lesson learned. | 15:50 |
| TheJulia | That seems like that flys in contridiction to the what the TC has blessed as valid for hardware drivers | 15:50 |
| dtantsur | TheJulia: yeah, but that's what we've been telling driver developers over and over again, including on this patch. | 15:50 |
| dtantsur | like, we prevented them from bumping the scciclient version for this very reason | 15:51 |
| JayF | I think requirements.txt vs d-requirements.txt is ... a semantic difference? I'm concerned about how/what changes for developers and bluntly, having it in requirements.txt at least means for the usual case it's "easy" (pip install --upgrade ironic would pull in new packaging) | 15:51 |
| JayF | s/developers/deployers/ | 15:51 |
| TheJulia | ... also that would have broken things horrifically in really bad ways | 15:51 |
| JayF | there is a question of "what should policy be for moving forward" but right now, we are here. How do we move forward from here? | 15:52 |
| dtantsur | I wonder if all we need is to replace version.parse() with map(int, version_string.split('.', 2)[:3]) | 15:52 |
| TheJulia | https://openinfrafoundation.formstack.com/forms/trackchairnominations2023 <-- if anyone is interested | 15:52 |
| JayF | Options seem to be: 1) [Re]move requirement by either moving to d-requirements or removing use of packaging.version 2) Consider it a mistake, reiterate policy on list but move forward 3) Hard revert, potentially breaking people that have pulled from git (and leaving the contributor who backported it with more work) | 15:52 |
| TheJulia | dtantsur: that is a good possibility actually, if they aren't using the comparison operators.... | 15:53 |
| * TheJulia doesn't remember at this point | 15:53 | |
| JayF | honestly factoring out packaging.version seems like a really, really clean fix | 15:53 |
| dtantsur | TheJulia: you can compare tuples | 15:53 |
| JayF | bluntly; just check the license of packaging; if it's compatable lift the method we need if there's anything more complex than just comparing tuples | 15:53 |
| TheJulia | I feel like I did that previously and got bit, but that was *ages* ago | 15:54 |
| * TheJulia goes back to meeting | 15:54 | |
| dtantsur | this task is complex in the generic case, but we only need one specific package | 15:54 |
| dtantsur | and one specific action\ | 15:54 |
| * dtantsur goes to the driver school, c u tomorrow | 15:54 | |
| JayF | This is made so much more complex by my inability to test this code | 15:55 |
| JayF | so I'm loathe to move forward with a change in a driver I don't have hardware *or* third party CI to test | 15:55 |
| JayF | vanou: ^ It looks like we need to solve a problem introduced by the new packaging requirement in the backported iRMC change. | 15:56 |
| JayF | vanou: We have to factor out that packaging dependency or revert the patch IMO | 15:56 |
| iurygregory | JayF, looking now... | 16:09 |
| JayF | iurygregory: scrollback from here to when I pinged you is all relevant too | 16:09 |
| iurygregory | reading | 16:09 |
| iurygregory | ok so the problem is that we added "packaging>=16.5 # Apache-2.0" right? | 16:15 |
| iurygregory | or is related to the bump in scciclent python-scciclient>=0.8.0,<0.13.0? | 16:16 |
| iurygregory | or both? | 16:16 |
| JayF | ->>> ok so the problem is that we added "packaging>=16.5 # Apache-2.0" right? | 16:16 |
| JayF | that | 16:16 |
| JayF | My current action is likely to be trying to factor out the packaging dep and getting vanou to test it, since I have no hardware to test with | 16:17 |
| JayF | but my todo list hasn't shrunk in days so I'd be thrilled if there was another option (or volunteer to fix it lol) | 16:18 |
| opendevreview | Pierre Riteau proposed openstack/ironic-inspector master: Add missing space to configuration help message https://review.opendev.org/c/openstack/ironic-inspector/+/860278 | 16:22 |
| TheJulia | I think a refactor is likely best/easiest, plate is semi-overflowing, I'd <3 if someone could pick it up fixing it | 16:23 |
| JayF | packaging is dual apache/bsd licensed | 16:24 |
| JayF | I'm literally going to lift the method we're using out if it's separatable | 16:24 |
| JayF | with a comment | 16:24 |
| iurygregory | ok, that makes sense to me if we can get rid of the dep | 16:28 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/yoga: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860281 | 16:41 |
| JayF | ^^ lets see if CI likes that at all | 16:43 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/yoga: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860281 | 16:44 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/yoga: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860281 | 17:17 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix allocations default table type https://review.opendev.org/c/openstack/ironic/+/860198 | 17:19 |
| TheJulia | dtantsur: fixed the mispelling ^ | 17:20 |
| iurygregory | JayF, tks I will keep an eye on CI | 17:41 |
| JayF | Gonna be blunt; if folks want ironic-inspector EM branches to keep going back as far as Ironic brnaches; someone is going to need to dedicate time to repairing the CI | 17:58 |
| JayF | it seems to be in extremely rough shape compared to ironic/ipa | 17:59 |
| JayF | I think the bugfix/10.7 and bugfix/10.9 branches there are missing the requirements pin, for instance | 17:59 |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector bugfix/10.7: CI: Various required fixes https://review.opendev.org/c/openstack/ironic-inspector/+/860171 | 18:03 |
| JayF | https://review.opendev.org/c/openstack/ironic-inspector/+/860170 looks broken in a crazy way; like maybe a requirement got pulled from pypi or somehow broke? | 18:06 |
| * TheJulia looks at email, sighs, and needs to now open jira | 18:19 | |
| TheJulia | JayF: 8| | 18:20 |
| * JayF just emailed list, with notice of intent-to-EOL branches from queens-victoria across all Ironic projects | 18:21 | |
| iurygregory | JayF, I totally forgot because of downstream priorities, we had a thread about closing stable branches that are older than ussuri (train .....) | 18:23 |
| iurygregory | from downstream perspective I think RH would be ok with Ussuri (RHOSP is based on this version if I recall) | 18:24 |
| JayF | I'm working from what I see is going on right now. I'd rather re-send a notice. Plus it serves as a one week buffer before that hits my todo list again :| | 18:24 |
| iurygregory | oh no, I think is wallaby | 18:24 |
| TheJulia | ahh | 18:24 |
| TheJulia | I see what is going on | 18:24 |
| TheJulia | tox.ini is referencing master | 18:24 |
| JayF | iurygregory: If you'll respond on-list, I'd appreciate it. We cannot maintain too-old-branches without people volunteering to take active maintenance. | 18:24 |
| TheJulia | for constraints | 18:24 |
| iurygregory | yeah ++ | 18:24 |
| iurygregory | will do that | 18:24 |
| JayF | And like, we usually do a bazaar model of "someone will fix it" | 18:24 |
| JayF | my digging in the last two days have told me, no, we won't fix it for all of them | 18:25 |
| TheJulia | iurygregory: osp 16.x is train, 17.x is wallaby | 18:25 |
| JayF | and maybe need a single person to take ownership of it | 18:25 |
| iurygregory | right | 18:25 |
| iurygregory | we agreed in 2) Closing all stable branches pre-train. | 18:25 |
| TheJulia | we need to close older bugfix branches that are not needed anymore as well | 18:26 |
| JayF | So theoretically I could do q/r/s nowish, but I'd rather do the whole set in batch so I'll just wait for the ML thread | 18:26 |
| JayF | TheJulia: that is on my list too | 18:26 |
| JayF | TheJulia: we've just like, abandoned them but never set EOL | 18:26 |
| JayF | and I'm pretty sure there's no process to make them EOL | 18:26 |
| JayF | so more manual work for releases team || more tooling for them to make | 18:26 |
| TheJulia | there is not the same contract around openstack releases for bugfix branches | 18:27 |
| JayF | of course not, but I'd rather that final sha be on a tag or something rather than just that disappearing forever | 18:27 |
| JayF | we have to delete the branches for openstack release tooling/zuul to be happy, right? | 18:27 |
| TheJulia | agreed, although for inspector, it looks like they rarely had backports and the prior commit was the .gitreview update | 18:27 |
| * JayF is unclear about this process -- if it exists. | 18:27 | |
| TheJulia | we have to get rid of branches with queue in them | 18:28 |
| TheJulia | basically | 18:28 |
| JayF | I don't know what you mean by that? | 18:28 |
| TheJulia | well, the queue zuul label, so fixing a branch with no other changes since it was first cut, doesn't mean we should actually fix it, the orignal tag is out there, we just tag the released version as the last thing | 18:28 |
| JayF | i don't know if it's from staring at my screen too long without a break, but I still don't follow | 18:29 |
| TheJulia | maybe it might be easier to speak words than type? | 18:29 |
| TheJulia | go nom something, we can talk after my 12:30-1:30 meeting if your not back before then | 18:29 |
| JayF | TheJulia: https://meet.google.com/gkn-mmqp-bvy?authuser=1 | 18:29 |
| JayF | if you have 5 now | 18:30 |
| JayF | I'd rather do it now | 18:30 |
| TheJulia | I do | 18:30 |
| * JayF is going to order food for pickup | 18:30 | |
| opendevreview | Jay Faulkner proposed openstack/ironic bugfix/19.0: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860141 | 19:49 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/xena: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860142 | 19:49 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/xena: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860142 | 19:50 |
| opendevreview | Jay Faulkner proposed openstack/ironic bugfix/18.1: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860143 | 19:50 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860144 | 19:50 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: Stable only: Factor out addition of packaging lib https://review.opendev.org/c/openstack/ironic/+/860144 | 19:51 |
| opendevreview | Verification of a change to openstack/ironic-inspector stable/xena failed: CI: Zuul no longer respects queue https://review.opendev.org/c/openstack/ironic-inspector/+/860162 | 20:22 |
| JayF | the ironic-inspector-tempest-managed-non-standalone job on ^ that branch seems to be a "coin flip" in terms of if it passes | 20:28 |
| TheJulia | there was some neutron issues which felt like a complete coin flip | 20:59 |
| JayF | I'm tempted to just make that job non-voting? | 21:04 |
| JayF | Xena is in maintenance until 4/23 so probably not OK? | 21:04 |
| TheJulia | yeah, likely not, I think it is dhcp issues overall | 21:26 |
| JayF | I'm going to recheck this to try to get in the queue fix :/ not ideal but I don't have time to dig more deeply today | 21:29 |
| TheJulia | ack | 21:31 |
| JayF | it does look like that stable patch is OK, potentially | 21:31 |
| JayF | I'd prefer someone with iRMC to take a look and confirm; but it's passing tests | 21:31 |
| JayF | wallaby is not, but that's because the patch it modifies hasn't landed in wallaby yet | 21:32 |
| TheJulia | they should be on in a few hours | 21:32 |
| opendevreview | Julia Kreger proposed openstack/ironic-lib stable/yoga: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-lib/+/860172 | 21:38 |
| JayF | TheJulia: <3 ty | 21:50 |
| TheJulia | I think the issue on stable/victoria is diskimage-builder is just pulling such an old version in, it doesn't know how to grok the centos cloud iamges mirror page | 21:52 |
| JayF | oooh | 21:52 |
| JayF | I will note that V is one of the stable branches I proposed retirement of on the list | 21:52 |
| JayF | so unless you feel strongly about fixing it and keeping it going; you might want to point your attention to newer branches | 21:53 |
| TheJulia | I do not, honestly | 21:54 |
| TheJulia | Just seems odd | 21:54 |
| TheJulia | so ironic-inspector failed because the ramdisk ran out of ram | 21:57 |
| janders | good morning Ironic o/ | 21:57 |
| JayF | o | 21:57 |
| JayF | o/ | 21:58 |
| TheJulia | yoga ipa as of last build wsa 416 mb, xena 416, master/zed 380 | 21:59 |
| TheJulia | wallaby 421 | 21:59 |
| TheJulia | centos8 images specifically | 22:00 |
| JayF | We've clearly done a good job of reducing size over time. What's the right fix for the older branches? Given the infrequency of runs; I'd be tempted to suggest just raising the RAM on the nodes if possible? | 22:00 |
| * TheJulia senses somethign landed and made things go boom | 22:00 | |
| JayF | Those values are much larger than usual (?) | 22:00 |
| TheJulia | oh yeouch | 22:01 |
| JayF | I'm asking, are they? | 22:01 |
| * JayF is lost in the conversation | 22:01 | |
| TheJulia | so, New Centos GeneicCloud image for 8 landed on 9/13 and is 100mb larger than we anticipate | 22:01 |
| TheJulia | so, it is a little bigger than we expect, but we've historically been trying to keep them smaller | 22:02 |
| TheJulia | oh, well... hmm | 22:02 |
| TheJulia | the image that dropped in january is 1.4GB | 22:02 |
| JayF | It seems weird that a stable OS (8 is released, right?) would have that big of a jump | 22:02 |
| TheJulia | so same size roughly | 22:02 |
| TheJulia | 8-stream | 22:02 |
| JayF | yep yep I forgot about the stream stuff, that's right | 22:03 |
| TheJulia | at some point they started shipping cockpit | 22:03 |
| TheJulia | and firmware itself grows tons | 22:03 |
| JayF | So what do we do, then? | 22:03 |
| TheJulia | looking | 22:03 |
| TheJulia | downloading an image to take a look inside | 22:04 |
| TheJulia | and figure out how much ram the jobs will need | 22:04 |
| TheJulia | so approx 1gb of ram for just the extracted fs, so basically 2.5gb | 22:12 |
| TheJulia | https://github.com/openstack/ironic/blob/stable/wallaby/zuul.d/ironic-jobs.yaml#L49 | 22:13 |
| JayF | so that means it should be large enough to work already, then, but it's not? | 22:17 |
| JayF | or just needs a little more headroom? | 22:17 |
| TheJulia | umm... hmm | 22:20 |
| TheJulia | https://www.irccloud.com/pastebin/RZQwdsXi/ | 22:20 |
| TheJulia | I think we're not in raw streaming rode | 22:20 |
| TheJulia | mode | 22:20 |
| TheJulia | ... what is it attempting to deploy | 22:20 |
| JayF | so if it's not streaming, then it's using an extra gig of space on disk | 22:22 |
| JayF | and the failures make sense | 22:22 |
| TheJulia | ehh.. but it looks like it shoudl be cirros | 22:28 |
| TheJulia | the image is 15.2MB | 22:31 |
| * TheJulia is confused | 22:31 | |
| TheJulia | it passed on recheck | 22:36 |
| TheJulia | this is a new one for me | 22:36 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector stable/ussuri: CI: Zuul no longer respects queue https://review.opendev.org/c/openstack/ironic-inspector/+/860165 | 22:42 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/train: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860190 | 22:51 |
| TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f2b/860187/1/check/ipa-tempest-uefi-redfish-vmedia-src/f2b4ecb/controller/logs/ironic-bm-logs/node-0_console_2022-10-03-23%3A12%3A25_log.txt <--- wut.... | 22:56 |
| TheJulia | 8| | 22:56 |
| * TheJulia ponders dinner | 22:59 | |
| JayF | dhcp issue sounds like it fits the bill | 23:00 |
| JayF | I haven't read one of these logs in SO long lol | 23:00 |
| JayF | TheJulia: | 23:01 |
| JayF | search for 72.333403 | 23:01 |
| JayF | it's missing ifupdown and something is expecting it | 23:01 |
| JayF | lines starting: | 23:01 |
| JayF | > [ 72.333403] systemd[382]: dhcp-interface@eth1.service: Failed to execute command: No such file or directory | 23:01 |
| * JayF needs to eod | 23:01 | |
| TheJulia | ditto | 23:03 |
| TheJulia | resume tomorrow | 23:03 |
| janders | TheJulia I'm tidying up backports of verify steps fixes and packaging them downstream. I spotted this comment: https://review.opendev.org/c/openstack/ironic/+/851950/comments/df52d7d9_e3c71721. If I click on the change-id in gerrit and that sends me back to the same patch, that means the patch hasn't been backported, right? (in which case I will | 23:11 |
| janders | backport it now) | 23:11 |
| TheJulia | that is correct | 23:12 |
| janders | ty! | 23:12 |
| janders | TheJulia do you think https://review.opendev.org/q/Icd8c5378469887962ff32eea2f38697c539f7e95 needs to go all the way back to Xena as well? (I'm kind of handling the two fixes together as they address the same bug at different levels) | 23:31 |
| opendevreview | Jacob Anders proposed openstack/ironic stable/yoga: Modify do_node_verify to avoid state machine stuck https://review.opendev.org/c/openstack/ironic/+/860306 | 23:54 |
| opendevreview | Jacob Anders proposed openstack/ironic bugfix/20.2: Modify do_node_verify to avoid state machine stuck https://review.opendev.org/c/openstack/ironic/+/860307 | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!