| opendevreview | Merged openstack/ironic-specs master: Quickfix: Correct rendering level of UEFI boot rec https://review.opendev.org/c/openstack/ironic-specs/+/900481 | 00:29 |
|---|---|---|
| opendevreview | Merged openstack/bifrost master: [CI] Disable new ansible lint rule breaking gate https://review.opendev.org/c/openstack/bifrost/+/900487 | 01:33 |
| opendevreview | Steve Baker proposed openstack/ironic master: [WIP] Replace cinderclient usage with openstacksdk https://review.opendev.org/c/openstack/ironic/+/900265 | 01:45 |
| gmann | For RBAC: we are progressing to new defaults enabled by default. Nova, Glance, Neutron and might be a few more projects already did and all CI testing running is with their new defaults | 02:49 |
| gmann | plan is to 1. implement the new default RBAC for project 2. disable those new defaults by default for at least one cycle they are available 3. enable new defaults by defaults. | 02:50 |
| gmann | these are the steps written in goal documents, let me know if more clarification needed and I can update the same there | 02:50 |
| gmann | JayF: here is I am tracking RBAC work, or let me know if any queries on that https://etherpad.opendev.org/p/rbac-goal-tracking | 02:51 |
| TheJulia | Okay cool, looks like I can change the default and maybe drop some CI jobs then | 02:54 |
| JayF | oh, cool, I should've checked the tc tracker, of course | 04:50 |
| gmann | TheJulia: JayF: untill old defaults are not removed make sure you run the old defaults jobs also. to make sure we do not break users running/enabling the old defaults | 05:20 |
| gmann | in nova, neutron etc I run a single old defaults job and rest all CI run on new defaults as they are enabled on service as well as devstack side by default | 05:21 |
| gmann | example https://github.com/openstack/nova/blob/b64ecb0cc776bd3eced674b0f879bb23c8a4b486/.zuul.yaml#L751 | 05:21 |
| TheJulia | Yeah, that would be obvious to do, we can just tie into an existing job easily | 05:24 |
| rpittau | good morning ironic! o/ | 07:32 |
| dtantsur | TheJulia: the non-standalone job will hopefully be gone with the inspector merging in Ironic | 09:38 |
| masghar | I also see: ironic-inspector-tempest, ironic-inspector-tempest-managed-non-standalone, ironic-inspector-tempest-rbac-scope-enforced | 09:55 |
| opendevreview | Derek Higgins proposed openstack/ironic master: Ensure enable_netboot_fallback writes out pxe config on adopt. https://review.opendev.org/c/openstack/ironic/+/811989 | 10:15 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Fix key-order ansible errors https://review.opendev.org/c/openstack/bifrost/+/900511 | 10:17 |
| iurygregory | good morning Ironic | 11:55 |
| rpittau | TheJulia, JayF, stevebaker[m], I would really love a review on https://review.opendev.org/c/openstack/ironic/+/894918 when you have a moment, it has already a +2, thanks! :) | 12:14 |
| adam-metal3 | hello Ironic, | 13:50 |
| adam-metal3 | what would your opinion about a new feature that would change block discovery process during the "os install devvice selection" , basically my idea is to not prioritize the wwn and serial numbers but instead match against all wwn and serial available from both udev adm and lsblk I have made a first version of this feature and sent downstream to test it on real multipath rad cards and such | 13:53 |
| adam-metal3 | https://github.com/Nordix/ironic-python-agent/commit/68c8ca2c367f94f78e55fe3e920863061417f59a | 13:53 |
| adam-metal3 | I intendt to improve a bit on the logic further and maybe introduce some additional unit tests and then upstream it | 13:54 |
| adam-metal3 | My motivation is that downstream has a hard time to supply the correct serials many times as they or customer gets the serial info from different sources and things get mixed up somewhere between the user, the driver, the udev, or somewhere and it is just a pain on a diverse set of hws with a lot of different users | 13:57 |
| adam-metal3 | ofc I would make a launchpad issue also jsut wanted to get some feedback first | 13:58 |
| opendevreview | Merged openstack/ironic-prometheus-exporter master: Document new LP bug tracker https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/900460 | 14:00 |
| dtantsur | adam-metal3: I don't have immediate objections, but I'd be curious to see the LP issue with an explanation why it's needed (ideally some examples) | 14:01 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Add link to LP bug tracker https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/900446 | 14:03 |
| adam-metal3 | dtantsur: great, sure I will try to collect some examples | 14:03 |
| TheJulia | Yeah, a few examples would help. I could definitely see the information getting corrupted as passed from source to source through humans. I guess this is why we have other qualifiers, but if your trying to use many disks... then it becomes difficult and you sort of then want to default to something simpler | 14:22 |
| TheJulia | well, more exact and simple at the same time | 14:23 |
| adam-metal3 | There is both the question of human error but I am receiving reports from time to time about situations where distros show different serials , we had in the past issue with wwn numbers where those were mixed up on some versions of udev, I have got reports where BMC was showing diffrent format of serial or just part of it same with bios so the main issue is always how things are represented on one abstraction level compared to | 14:27 |
| adam-metal3 | an other an where do folks take the data from (source of truth) | 14:27 |
| adam-metal3 | but I will try to get concrete examples | 14:28 |
| TheJulia | eww | 14:29 |
| TheJulia | yeah, because some BMCs get protocol translation in the middle | 14:29 |
| TheJulia | well, protocol translation between device, and actual bmc | 14:29 |
| TheJulia | ... We've seen something similar where the BMC was providing one serial number, but the device was reporting something entirely different | 14:30 |
| dtantsur | but I guess we cannot reconcile that in IPA since we don't know what the BMC provides? | 14:30 |
| TheJulia | I think iurygregory even got them to pull the device and what it was stamped with was something entirely different | 14:30 |
| TheJulia | no, we can | 14:30 |
| TheJulia | that gets shipped back likely over an i2c bus from intermediate controller devices | 14:31 |
| iurygregory | I'm lost | 14:31 |
| TheJulia | err | 14:31 |
| TheJulia | can't | 14:31 |
| iurygregory | let me read for context | 14:31 |
| TheJulia | we can't intercept/understand the i2c bus device stream/contents since it is for the BMC and under the BMC's control and even then it is up to the BMC to report accurate data | 14:31 |
| dtantsur | yeah, so an IPA side patch won't help there | 14:32 |
| TheJulia | iurygregory: you had a case like 6-9 months ago with a serial number being totally wrong | 14:32 |
| dtantsur | I guess adam-metal3 have a different c ase | 14:32 |
| opendevreview | Merged openstack/ironic stable/wallaby: Align iRMC driver with Ironic's default boot_mode https://review.opendev.org/c/openstack/ironic/+/866628 | 14:32 |
| TheJulia | likely, or at least similar enough we might be able to guard/defend against it | 14:32 |
| iurygregory | yeah I seem to remember this one | 14:32 |
| adam-metal3 | currently I am approaching this from the high level, I would like to collect wider range of serials and wwns to have a better chance to find the one the user is looking for , IMO as I have mentioned issues/misunderstanding/format missmatch can originate from different abstraction levels so I plan to look for more to find more if that makes sense | 14:35 |
| TheJulia | I think you mentioned something like they finally pulled the disk and it was an entirely separate serial or something crazy | 14:35 |
| TheJulia | adam-metal3: I think that could make sense, we've seen some fields get moved/assumed when say a device doesn't offer one value which is expected by the tool | 14:36 |
| TheJulia | adam-metal3: I guess what I'm wondering is how many such cases are actually things we can really defend against if it is happening at lower levels. It might not be all the time, OS differences sounds like some fun bug somewhere along the way | 14:41 |
| adam-metal3 | TheJulia: as far as I have seen, in most cases if both lsblk and udev adm data was combined the needed serial or wwn was there in one of the fields so I have focused on this approach | 14:44 |
| TheJulia | Cool | 14:45 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Fix unit tests broken by olso.utils https://review.opendev.org/c/openstack/ironic/+/900533 | 15:06 |
| JayF | Looks like a couple IPA bugfix branches were inadvertantly recreated, I'm going to remedy the issue after validating all the commits are same | 15:40 |
| JayF | https://opendev.org/openstack/ironic-python-agent/src/branch/bugfix/8.6 and https://opendev.org/openstack/ironic-python-agent/src/branch/bugfix/9.2 | 15:41 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Fix unit tests broken by olso.utils https://review.opendev.org/c/openstack/ironic/+/900533 | 15:41 |
| JayF | should not exist per https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/EWBWFHS4XF6R55SZXNO7GE7GM64QMW4E/ | 15:41 |
| JayF | and I've found open changes against them in gerrit which explain why they were reopened | 15:42 |
| JayF | 8.6 was apparently never retired; I see no evidence of it being tagged | 15:45 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Make sqlalchemy-2x job voting again https://review.opendev.org/c/openstack/ironic/+/900537 | 15:46 |
| JayF | so looks like those retirements were never properly completed, from that email | 15:46 |
| JayF | I'm not sure who executed on that; but either way I'm going to clean it up now | 15:47 |
| JayF | Looks like our bugfix branches, which were retired, got recreated | 15:54 |
| JayF | rpittau: dtantsur: I am -2 to using bugfix branches in metal3, as mcuh as I can be in that repo, until we fix the dang release automation | 15:56 |
| JayF | I'm done chasing this | 15:56 |
| dtantsur | JayF: how is it related to release automation? | 15:57 |
| JayF | the branches got recreated by release automation | 15:57 |
| JayF | bceause they were cut by relase automation | 15:57 |
| JayF | at least that's the hypothesis at the time | 15:57 |
| JayF | and I've spent more time than I think anyone realizes trying to keep these things sane | 15:57 |
| JayF | we should have never implemented a new release type without automating it, that shortcut has been a hole in the bottom of my time bucket every second I've been PTL | 15:58 |
| dtantsur | I'm still struggling to understand. If the next time some automation screws up stable branches, will you suggest never releasing Ironic ever again? | 15:59 |
| JayF | Years ago, a subset of Ironic contributors decided to create a new release type, unique to Ironic, with releases cut automatically and no retirement strategy | 15:59 |
| dtantsur | Sorry for caring about Ironic outside of OpenStack.. | 16:00 |
| JayF | I, during my tenure as PTL, at the urging of CI management people in infra, trying to resolve zuul config errors, have tried to work out a retirement strategy for them multiple times | 16:00 |
| rpittau | JayF: besides the -2, what would be the alternative toi bugfix branches then ? | 16:00 |
| JayF | dtantsur: I don't care about the release timing. I care about us doing unautomated things and feeling like I've been stuck with the leftover work created by that lack of automation. | 16:00 |
| JayF | That is the root cause and any other comment is coming from a place of frustration and should've been swallowed until I was less frustrated | 16:01 |
| dtantsur | Still unclear. The part that has never been automated was actual releases, not branching. | 16:01 |
| TheJulia | wouldn't the thing to do, instead of jumping to conclusions and taking stances, to first understand precisely *how* the branches got recreated | 16:02 |
| TheJulia | How can we prevent a thing we don't fully understand if there is disconnect why the tooling recreated | 16:02 |
| dtantsur | At least as far as I know, branch creation get automated. If we did not retire them properly and the automation ended up re-running.. yeah, it's our screw-up. | 16:02 |
| TheJulia | if it was indeed the tooling | 16:02 |
| JayF | TheJulia: I've been on that ride literally *3* different times in 18 months. | 16:02 |
| TheJulia | Has a root cause been determiend then in the tooling that resulted in re-creation? | 16:03 |
| JayF | Each time, a different one | 16:03 |
| TheJulia | Is there a write-up? Are there details which can be drawn from to understand? | 16:04 |
| JayF | and each time, I'm the only one who noticed it happened, despite it being in all public ironic CI and zuul config error dashboards | 16:04 |
| JayF | TheJulia: both times it was "bugs" in the release tooling, depending on perspective; I have a hard time seeing them as bugs because we're using them in ways they weren't really designed for | 16:04 |
| JayF | the other time I think it was literally I did a thing wrong the first time I ran the commands | 16:05 |
| dtantsur | The visibility problem is real, but not really specific to bugfix branches. We're notoriously bad at learning that stable jobs are not passing. | 16:05 |
| JayF | which is what I was trying to fix when this injected itself into my workstream. again. | 16:05 |
| TheJulia | I'm still wondering why we have stable jobs | 16:05 |
| JayF | hence the frustration :) | 16:05 |
| TheJulia | well | 16:05 |
| dtantsur | Frankly, if a stable branch is suddenly gone, I won't notice until you come to me screaming :) | 16:05 |
| TheJulia | periodic stable jobs | 16:05 |
| TheJulia | dtantsur: and I suspect that is how it should be | 16:05 |
| dtantsur | Periodic jobs are only useful if there is visibility into them. | 16:05 |
| TheJulia | bingo | 16:06 |
| dtantsur | In OpenShift, we have a whole team that watching the jobs as their full time occupation | 16:06 |
| JayF | All of this is redirecting from the original issue though; which is we have these unique things to ironic; which needs unique maintenance work done to keep them alive *and/or* a project to improve automation, and AFAICT I'm the only one, other than those who experience the results directly (people who are punished by zuul config errors and wasted CI resources) who has this | 16:07 |
| JayF | problem on their radar | 16:07 |
| TheJulia | So lets take a step back here, there are issues, lets try and sort through the what and get a plan of action which is mutually agreeable. Some of this means spreading some context and determining the right path | 16:07 |
| JayF | and seeing bugfix branches being used by metal3, and getting additional use and investment without that automation backfill happening was really disheartening to me | 16:07 |
| dtantsur | There is so far absolutely nothing unique in the things that have been discussed | 16:08 |
| TheJulia | JayF: you have mentioned "improve automation" a few times, perhaps it is time to bring us to that water | 16:08 |
| JayF | For !bugfix Ironic branches and releases; we update a yaml file, automation handles it. This is for creation, releasing, and deletion. | 16:08 |
| TheJulia | your providing a requirement, not the context anyone needs to act on it | 16:09 |
| JayF | For bugfix branches, automation does releases and branch creation, but retirement is completely manual (and has, as mentioned, a habit of being reverted) | 16:09 |
| TheJulia | okay, a bit more context | 16:09 |
| TheJulia | what are the other details around this, you mentioned bugs? | 16:09 |
| TheJulia | are there patches/changes/fixes people can look at? | 16:09 |
| JayF | these branches remaining around cause them to be processed by zuul, causing errors in CI system, pain for infra team, and if they aren't erroring in zuul, they're likely running failing periodic jobs | 16:09 |
| TheJulia | so lets fix the root issue, not focus on the after effects | 16:10 |
| rpittau | so the issue is the retirement? stop? | 16:10 |
| JayF | There needs to be a project-level attention paid to this: we need to figure out how to properly get bugfix branches represented in release automation, and have that automation support it | 16:10 |
| TheJulia | rpittau: we still get configuration errors | 16:10 |
| TheJulia | rpittau: because config cannot be a static set in stone state | 16:10 |
| dtantsur | Configuration errors are coming from the un-retired jobs, no? | 16:10 |
| dtantsur | ehh, s/jobs/branches/ | 16:11 |
| TheJulia | branches | 16:11 |
| JayF | Gerrit basically uses repo:branch as a primary key | 16:11 |
| JayF | infra admins have made it clear to TC that indef growing branch counts are unsustainable | 16:11 |
| rpittau | alright, so I ask again, is the retirement the main issue here? | 16:11 |
| rpittau | or better, keeping the retired branches as that | 16:12 |
| TheJulia | seems like it, the lack of support for automated process in retirement seems to result in resurrection... somehow?!? | 16:12 |
| JayF | that is basically my assumption, yes | 16:12 |
| JayF | I suspect our half-use of it is causing some of the pain | 16:12 |
| * TheJulia still wants an answer to the somehow and sort of wonders if they got re-created in the recent administrative actions in preparation for upgrade | 16:13 | |
| JayF | rpittau: yes-ish? but I think if you zoom out, it's more that, we're using the half of release automatino that works, and it's sorta making the whole thing more wonky than if we had it actually supported or didn't use it at all (I really dislike the second option) | 16:13 |
| TheJulia | I guess the challenge also is, how are things getting executed that we possibly end up in additional branches, is state just being re-asserted automatically somewhere that we don't know the process | 16:13 |
| JayF | The reason I don't suspect that, TheJulia, is simply because we don't have stable/mitaka and friends popping up | 16:13 |
| TheJulia | JayF: do you have links to the release tooling code that parses the yaml files ? | 16:14 |
| TheJulia | out of curiosity, just trying to get all the context out | 16:14 |
| JayF | https://opendev.org/openstack/releases | 16:14 |
| dtantsur | it has to be in.. yeah, here | 16:14 |
| TheJulia | well, does it *have* to be? | 16:14 |
| JayF | it is in there, openstack_releases/ | 16:14 |
| TheJulia | so, who is going to own this challenge before us? | 16:15 |
| dtantsur | looking at deliverables/mitaka/ironic.yaml, for instance, it does NOT have a record for stable/mitaka | 16:15 |
| dtantsur | it's possible that we need to remove any records for deleted branches | 16:15 |
| TheJulia | ohhhh | 16:15 |
| TheJulia | is that in the git history? | 16:15 |
| TheJulia | would be funny if this is just rooted in process knowledge | 16:16 |
| TheJulia | or sad | 16:16 |
| TheJulia | or lolsob | 16:16 |
| JayF | it's not funny, there was no process | 16:16 |
| TheJulia | definitely lolsob | 16:16 |
| JayF | I created one, some people cursorily-ack'd it | 16:16 |
| JayF | This is basically me having to redo something I worked on for a while if this is the answer, I don't see any humor in it | 16:16 |
| TheJulia | I think to dmitry's point, is there a paring process we were just unaware of in the care and feeding | 16:16 |
| dtantsur | huh, pike does have a branch record | 16:17 |
| TheJulia | but no branch | 16:17 |
| JayF | I suspect there's something like, if we have a *-eol tag set, don't recreate the stable/* | 16:17 |
| JayF | release automation has a lot of tricks like that | 16:18 |
| JayF | which is why bugfix branches not having like, real "support" in there is tricky, it's not a general purpose tool so much as it's made to work for openstack-y stuff | 16:18 |
| dtantsur | there is definitely some logic that is checking for -eol | 16:18 |
| rpittau | the only thing that I can say is that I'll look into it during this cycle, as the main promoter of the bugfix branches I think it's fair | 16:20 |
| rpittau | I've looked into that for the creation automation, I guess I can do the same for the retirement | 16:20 |
| JayF | To be clear, I don't love/hate/care about bugfix branches at all, but I have been ... seeing a lot of the places where we have a lot of status-quo-goop built up | 16:21 |
| TheJulia | rpittau: if you need an extra set of eyes, please feel free to reach out | 16:21 |
| JayF | and this one hit me unexpectedly this morning | 16:21 |
| JayF | did not mean to go at it so aggressively :) thank you for helping direct it in a good way | 16:21 |
| TheJulia | Frustration is totally understandable, and sometimes is needed to get the point across | 16:22 |
| dtantsur | ++ | 16:22 |
| dtantsur | rpittau: we need to see what tag-releases job does.. for that, we need to find where it's defined, which is not trivial with zuul | 16:22 |
| TheJulia | we just need to fix the root issue, which is going to take some learning and understanding | 16:22 |
| JayF | dtantsur: rpittau: I suggest, engage releases team, make sure they want a piece of this before you just submit a PR :) | 16:22 |
| rpittau | no problem, I'll set a launchpad bug as reminder | 16:22 |
| rpittau | and yeah, it's not super easy but it's there somewhere :) | 16:22 |
| TheJulia | dtantsur: I *think* that gets defined in opendev's zuul config, but it has been literal ages since I've looked | 16:23 |
| JayF | dtantsur: rpittau: I'm 99% sure the answer is "yes", but an email to them beforehand would probably go a really, really long way | 16:23 |
| dtantsur | UI to the rescue https://zuul.opendev.org/t/openstack/job/tag-releases | 16:24 |
| TheJulia | \o/ | 16:24 |
| rpittau | heh that's cheating :D | 16:24 |
| TheJulia | I need a good quote justifying perceptions around cheating as just using a tool | 16:24 |
| TheJulia | alas, I have noen | 16:24 |
| TheJulia | none | 16:24 |
| dtantsur | ~/scripts/release-tools/process_release_requests.sh - need to find that | 16:25 |
| rpittau | dtantsur: https://opendev.org/openstack/project-config/src/branch/master/roles/copy-release-tools-scripts/files/release-tools/process_release_requests.sh | 16:25 |
| dtantsur | nice | 16:26 |
| * JayF afk for a bit | 16:26 | |
| TheJulia | hmmm https://opendev.org/openstack/project-config/src/branch/master/roles/copy-release-tools-scripts/files/release-tools/process_release_requests.py | 16:27 |
| clarkb | fwiw as noted in the opendev channel for this particular instance the hash for bugfix/8.1 does not match the hash of the 8.1.0 tag that is configured for it in the releases dir | 16:27 |
| clarkb | instead we suspect that an open change on the branch prevented deletion from occuring (gerrit won't allow you to dleete a branch with open changes) | 16:27 |
| clarkb | specifically this change https://review.opendev.org/c/openstack/ironic-python-agent/+/861062 | 16:28 |
| TheJulia | oh, abandoned after deleted | 16:28 |
| clarkb | which is still a problem with not having things properly automated as it allows for humans to miss things like that | 16:28 |
| JayF | yeah, 8.1 is the "weird" case, but don't let that distract from the myriad others that were deleted and came back | 16:28 |
| TheJulia | so cleanup/retirement absolutely needs to be automated down to a flag *or something* that fails CI | 16:28 |
| TheJulia | "go clean up your open changes" before the delete goes through | 16:29 |
| dtantsur | clarkb: I cannot really find the place where the automation decided not to create branch (e.g. for old stable branches). How does it decide that? | 16:29 |
| clarkb | dtantsur: you would hae to ask the openstack release team | 16:29 |
| dtantsur | k | 16:29 |
| clarkb | maybe they have a list of active series and they don't process files for inactive branch series | 16:30 |
| clarkb | but I don't know for sure. I'm not super clued into their processes around branch management and tagging | 16:30 |
| dtantsur | Ah, well, found the answer https://opendev.org/openstack/project-config/commit/306e2381ecabb8251345dac76c160462225726a2 | 16:31 |
| dtantsur | And see, it does account for bugfix branches even. But -eol tags is not a thing that we do. | 16:31 |
| dtantsur | rpittau: maybe something we should ask the release team is if it's okay to remove branch records from their deliverable files | 16:32 |
| dtantsur | alternatively, add an explicit flag and respect it in the project-config tooling | 16:33 |
| JayF | I totally do EOL tags | 16:34 |
| JayF | Go check the remote. I'm not at my PC anymore to show | 16:34 |
| dtantsur | hmm, yeah, I see a bunch of them | 16:35 |
| JayF | I would totally buy that somehow I'm not matching their format and matching the format fixing the problem would be a good short-term solution to at least make the manual process work | 16:35 |
| JayF | And given the process is manual, it wouldn't surprise me if the tags have some minor inconsistency as well | 16:35 |
| JayF | But I actually have to disconnect from my phone now so that I can drive. Somehow I don't think IRC and driving are a good combination 😂 | 16:36 |
| TheJulia | hmm, that grep seems suspect to me | 16:36 |
| TheJulia | in the change dtantsur linked | 16:36 |
| rpittau | That wouldean changing bugfix branch names at the end of the supported period adding -eol | 16:36 |
| clarkb | note there is no eol tag for bugfix/8.1 | 16:36 |
| clarkb | while I don't think it was recreated anymore by the release automation if we simply delete the branch I suspect it would be given ^ | 16:36 |
| clarkb | would also need to add the eol tag first | 16:37 |
| rpittau | We should probably do a test and talk to the release team in the meantime | 16:38 |
| dtantsur | We could probably find that job run that re-created the branches (it has to be on a patch that modifies the same yaml) | 16:38 |
| dtantsur | if we're talking about 8.1, it has to be something modifying deliverables/xena/ironic-python-agent.yaml | 16:38 |
| TheJulia | there also does not appear to be an eol tag for 8.6 | 16:39 |
| TheJulia | oh, nvmd | 16:40 |
| TheJulia | see it was never retired | 16:40 |
| dtantsur | do we haver a list of branches that got re-created? | 16:40 |
| * dtantsur snack, brb | 16:40 | |
| rpittau | I really need to drop now, but I'm happy to keep looking into it in the next days | 16:41 |
| rpittau | see you tomorrow o/ | 16:41 |
| TheJulia | for ipa, it is presently reporting 8.1, 8.3, 8.6, 9.0, 9.2, 9.3, 9.5, 9.6 as open bugfix branches | 16:42 |
| TheJulia | eols for 6.6, 8.0, 8.4 | 16:43 |
| TheJulia | so this is a consistency/process problem as well | 16:44 |
| TheJulia | which maybe just means we need an explicit "this is retired" flag in the yaml | 16:45 |
| TheJulia | I guess, since rpittau sent the email in May, we can't really continue digging without understanding process | 16:47 |
| TheJulia | since it also speaks in the future of deletion | 16:47 |
| * TheJulia eats something for breakfast and begins the great tab hunt to find the document she was just working on | 16:53 | |
| dtantsur | TheJulia: okay, so the branches that did get an eol tag have not been re-created | 17:01 |
| dtantsur | and the ones without an eol tag ended up re-appearing | 17:01 |
| dtantsur | which is consistent with how I'm reading the tooling. probably a procedural mistake on our side? | 17:01 |
| TheJulia | that is kind of what I'm wondering | 17:05 |
| TheJulia | and has me wondering if maybe the right path is just to automate the rest of it so it not possible to make | 17:05 |
| dtantsur | 1) make an eol tag, 2) push the eol tag, 3) delete the branch, 4)..., 5) profit? | 17:06 |
| clarkb | 2.5) abandon all open changes on the branch | 17:06 |
| TheJulia | vi release/ironic/project.yaml, edit file, CI kicks it back if 2.5 is not met, then it makes a tag, deletes branch.... sock gnomes come in, take the socks, then profit | 17:07 |
| dtantsur | right | 17:07 |
| TheJulia | That way we're not relying upon humans to execute more than one step | 17:08 |
| TheJulia | Love you all, but we are after all, only human | 17:08 |
| TheJulia | ... (and I forget things ALL the time....) | 17:08 |
| * dtantsur is an owl | 17:08 | |
| * TheJulia suspects dtantsur is instead a shape shifter | 17:08 | |
| * dtantsur can neither confirm nor deny | 17:08 | |
| TheJulia | :) | 17:08 |
| TheJulia | so to sort of wrap the circle on our semi-post-mortem here, we need to sync up with rpittau, we can do that in the morning, and if we have consensus on make it busy-contributor-proof, then ping the release team, see if they are onboard, and do the needful | 17:10 |
| dtantsur | ++ | 17:10 |
| TheJulia | cool cool, in that case, I'm going to go back to trying to hammer out this policy document | 17:10 |
| * TheJulia gets out a forge, a hammer, and a piece of paper | 17:10 | |
| opendevreview | Merged openstack/ironic-python-agent master: improve multipathd error handling https://review.opendev.org/c/openstack/ironic-python-agent/+/898209 | 17:31 |
| JayF | Who is an owl? Who? Who? | 18:12 |
| TheJulia | oh my | 18:12 |
| TheJulia | don't tell me we have two Owls now.... | 18:13 |
| dtantsur | whoooooo | 18:13 |
| TheJulia | Bartender! Bring me something strong! | 18:13 |
| JayF | I'll note also rpittau TheJulia that elod welcomed the idea of a patch for real support when I mentioned it in that channel | 18:13 |
| TheJulia | Then likely we just JFDI, lets sync in the morning and have the tooling do it for us | 18:14 |
| dtantsur | We've been to a cool zoo not far from here, they had a small tower that you could enter (no fences, nothing), and inside near the roof were a couple of barn owls just sitting there and JUDGING YOU | 18:14 |
| TheJulia | they are super judgey | 18:15 |
| TheJulia | and also cute and friendly | 18:15 |
| * TheJulia had a one who was her friend and sat in her lap when she was like 11 | 18:15 | |
| dtantsur | yep, and these were the dark ones, particularly judgemental | 18:15 |
| dtantsur | awwwww, that's cute! | 18:15 |
| TheJulia | (imprinted on humans, the owl liked watching television) | 18:15 |
| TheJulia | ((My father was a trained/registered/licensed raptor handler, so we got a transport request once and the owl escaped and spent time moving around inside the car while we were heading to the rehab center for it to live it's best life)) | 18:16 |
| TheJulia | stupidly gentle bird too | 18:17 |
| dtantsur | heh, interesting | 18:17 |
| TheJulia | My father may have also had a thing about taking a golden eagle through a drive through | 18:17 |
| * dtantsur is trying to imagine | 18:19 | |
| TheJulia | think giant eagle barely able to fit into a jeep, looking at the person trying to take an order | 18:20 |
| dtantsur | mm, nice, and what did it order usually? | 18:20 |
| dtantsur | :D | 18:20 |
| TheJulia | it didn't, my father did :) | 18:20 |
| dtantsur | damn, that would be quite a scene | 18:21 |
| dtantsur | "double espresso for me and an orange juice for my human, thank you" | 18:21 |
| JayF | TheJulia: so we're related then, actually | 18:24 |
| JayF | TheJulia: Faulkner's original root -> Falconer | 18:24 |
| JayF | at least, mythically so | 18:24 |
| TheJulia | JayF: I didn't know that! | 18:24 |
| * JayF adopts TheJulia's dad | 18:24 | |
| TheJulia | My father was doing it for inspiration for art | 18:24 |
| JayF | oh of course, he does those nice sketches, I've seen them before | 18:25 |
| TheJulia | He mostly does dogs/horses at this point | 18:26 |
| TheJulia | I have some of the birds somewhere around here, they just don't go with my style | 18:27 |
| TheJulia | Anyone interested in reviewing https://review.opendev.org/c/openstack/sushy-tools/+/896963 ? | 18:28 |
| TheJulia | might help if I set the hashtag and not the topic | 18:29 |
| JayF | I'll look but usually don't +2 sushy* repos unless nobody else is on it | 18:29 |
| TheJulia | jay, fixing the same on your bug deputy role doc | 18:29 |
| JayF | lolsob | 18:29 |
| TheJulia | see, I said there would be a lolsob at some poitn | 18:29 |
| JayF | we have a :lolsob: in our slack at work, it's one of my favorite emojis; it expresses a common emotion in software dev and operations | 18:30 |
| * dtantsur only has sobsob downstream today... | 18:32 | |
| dtantsur | I'll continue sobbing tomorrow, who whooo | 18:32 |
| TheJulia | :( | 18:32 |
| TheJulia | dtantsur: you could always hack in support to leverage boot progress indicators! | 18:33 |
| * TheJulia just rechecked the sushy patch | 18:33 | |
| TheJulia | (as therapy, of course) | 18:33 |
| dtantsur | if I ever stop handling escalations for a change... | 18:33 |
| TheJulia | oh, http boot uri needs to merge first | 18:33 |
| TheJulia | doh | 18:33 |
| TheJulia | now to cd ~/code/ironic && git checkout head~1 && git pull --ff-only && git stash pop | 18:37 |
| *** tosky_ is now known as tosky | 18:43 | |
| * TheJulia suddenly wonders if virtualpdu might be subject to eventlet shenanagains | 18:43 | |
| TheJulia | oh no, it works! | 18:43 |
| TheJulia | hmm, networking, ssh timed out | 18:45 |
| * JayF inspects it for bad monkeypatching | 18:45 | |
| TheJulia | it actually worked when tested in ci, just the overall job failed | 18:46 |
| JayF | no eventlet at all in virtualpdu | 18:46 |
| TheJulia | so, the issue is ipxe failed to boot to disk | 18:47 |
| TheJulia | libvirt presents a virtio device | 18:49 |
| TheJulia | that seems... not bios bootable | 18:50 |
| TheJulia | yeah, our xml is getting mushed into virtio land | 18:52 |
| TheJulia | Crazy thought, what if we stop running basicbaremetalops on snmp driver | 18:53 |
| TheJulia | and just run the ramdisk test? | 18:53 |
| TheJulia | we don't *really* need to do a full deploy to prove out the power control works | 18:53 |
| JayF | yes, that's exactly the kinda optimization I was thinking about when I proposed the ptg session | 18:54 |
| JayF | that's not a crazy thought, it's so sane it just seems that way :D | 18:55 |
| TheJulia | okay then | 18:58 |
| TheJulia | oh what did I name that est | 18:58 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Change snmp job to not use a focal node https://review.opendev.org/c/openstack/ironic/+/893824 | 19:05 |
| TheJulia | That might work, I hope | 19:06 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix unit tests broken by olso.utils https://review.opendev.org/c/openstack/ironic/+/900533 | 19:13 |
| TheJulia | okay | 19:17 |
| TheJulia | JayF: looks like it failed due to ioapic initialization failure which has a 1 in 100 failure rate in general with the the kernel | 19:19 |
| TheJulia | s/the the/the/ | 19:19 |
| JayF | ah; I checked the node console, saw it was killed mid-deploy, rechecked it | 19:20 |
| TheJulia | yeah, it paniced | 19:21 |
| TheJulia | https://www.reddit.com/r/linuxmasterrace/comments/g1291w/colonel_panic/ | 19:21 |
| TheJulia | lol | 19:21 |
| TheJulia | seems like an awful group on reddit | 19:22 |
| JayF | I do not generally participate in any of those ... named communities | 19:23 |
| TheJulia | I only occassionaly find them via google | 19:23 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix unit tests broken by olso.utils https://review.opendev.org/c/openstack/ironic/+/900533 | 19:26 |
| opendevreview | Merged openstack/ironic master: Fix unit tests broken by olso.utils https://review.opendev.org/c/openstack/ironic/+/900533 | 22:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!