| JayF | I'm running these on my beefy local desktop | 00:45 |
|---|---|---|
| JayF | so maybe I can just try and find out tomorrow | 00:45 |
| TheJulia | JayF: for what it is worth, last year I ran it until I had like 123k rows in my local db... it didn't take tooooo long | 01:08 |
| TheJulia | but yeah | 01:08 |
| JayF | I mean, I usually work off the laptop and use the desktop when I need beef, this is exactly what it's for | 01:09 |
| JayF | maybe if 100k doesn't take too long, I'll run it for some exceedingly extreme value of N until my office gets toasty lol | 01:09 |
| TheJulia | ... yeah... it did increase the home office temp a bit | 01:10 |
| stevebaker[m] | a lock torture test would be interesting | 01:13 |
| JayF | I'm just running stuff that Julia has already written tbh | 01:18 |
| JayF | but I'd be all ears for an idea on how to do a lock torture test, whatever that'd mean | 01:18 |
| ashinclouds[m] | You’d need multiple completing threads…. | 01:36 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix idrac-redfish RAID controller mode conversion https://review.opendev.org/c/openstack/ironic/+/855872 | 07:38 |
| opendevreview | Merged openstack/tenks master: [CI] Gate on CentOS Stream 9 and Rocky Linux 9 https://review.opendev.org/c/openstack/tenks/+/856859 | 08:51 |
| * dtantsur is curious if he really needs to start reminding the core team (incl. 1 former and 2 candidate PTLs) how requirements work or better just give up | 09:32 | |
| dtantsur | jk, I've given up long ago.. | 09:36 |
| opendevreview | Merged openstack/tenks master: [CI] Remove CentOS 7 support leftovers https://review.opendev.org/c/openstack/tenks/+/856860 | 09:41 |
| opendevreview | kamlesh chauvhan proposed openstack/ironic-tempest-plugin master: Add iDRAC Redfish configuration molds test https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/855710 | 09:51 |
| opendevreview | SONG SHUKUN proposed openstack/ironic master: Add support auth protocols for iRMC https://review.opendev.org/c/openstack/ironic/+/857035 | 10:02 |
| opendevreview | Merged openstack/tenks master: [CI] Optimise ovs setup https://review.opendev.org/c/openstack/tenks/+/856861 | 10:11 |
| opendevreview | Merged openstack/tenks master: [ansible-lint] Restore sanity https://review.opendev.org/c/openstack/tenks/+/856892 | 10:11 |
| opendevreview | Merged openstack/tenks master: [ansible-lint] Unignore jinja[spacing] https://review.opendev.org/c/openstack/tenks/+/856893 | 10:11 |
| opendevreview | Merged openstack/tenks master: [CI] Ignore .ansible-lint in the main jobs https://review.opendev.org/c/openstack/tenks/+/856896 | 10:11 |
| opendevreview | Merged openstack/tenks master: [CI] Name the Debian version https://review.opendev.org/c/openstack/tenks/+/856897 | 10:11 |
| iurygregory | good morning Ironic | 12:15 |
| iurygregory | dtantsur, because of the bump in sushy version for zed? .-. | 12:39 |
| dtantsur | iurygregory: yup. sorry if I sound a bit overly sarcastic, but this goes against the agreed requirements approach. | 12:39 |
| dtantsur | now, we could change the approach and say we just bump version at the end of each cycle. this would even solve some problems. but we have never said it? | 12:40 |
| dtantsur | and I think I'm spending too much time being The Guardian of Requirements here :D | 12:40 |
| iurygregory | dtantsur, no need to be sorry, I do understand that normally we don't have an upper bound for requirements, just trying to understand how bad is to bump (I know it means that "we don't support versions lower than X) | 12:43 |
| dtantsur | it's not "bad", it's inconsistent | 12:43 |
| dtantsur | we're sending a message to downstreams that is not quite true: ironic can no longer work with sushy<4.3.0 | 12:44 |
| dtantsur | imagine somebody has a regression (etags anyone) in 4.0. So far they can stick to 3.2, but now we've officially disallowed that | 12:44 |
| iurygregory | hummm | 12:44 |
| ashinclouds[m] | Well if you’ve got super micro hardware | 12:44 |
| dtantsur | and again, it's partly a question of policy. if we say "you MUST use the latest version of sushy/ironic-lib from the release series", it's fair | 12:44 |
| dtantsur | but it's not what we say | 12:45 |
| ashinclouds[m] | We almost need we recommend and then we require as different things | 12:45 |
| dtantsur | right | 12:45 |
| dtantsur | requirements is what we require | 12:45 |
| dtantsur | normally, requirements are not bumped for bug fixes because of cascading effect on downstreams | 12:45 |
| dtantsur | another example: imagine proliantutils doing sushy>=3.0,<4.0 for some reason. oops? | 12:45 |
| dtantsur | (I'm not justifying having upper caps, but they do happen) | 12:46 |
| dtantsur | I also see one big advantage of what we've done with 4.3: if we realize we need newer sushy features on stable/zed, we're not screwed | 12:47 |
| dtantsur | so, I don't object making it a new policy, as long as it's a policy | 12:47 |
| dtantsur | (and then we probably need to do it for everything in driver-requirements too) | 12:48 |
| * TheJulia starts making coffee | 12:49 | |
| dtantsur | yep, it's not a good pre-coffee discussion ;) | 12:49 |
| iurygregory | totally not, I'm still a bit slow since I only had 1 mug of coffee :D | 12:49 |
| iurygregory | reading carefully to understand all the points | 12:49 |
| TheJulia | You know, has one of the major vendors done their typical end of cycle d-r bump yet? | 12:50 |
| iurygregory | this is a good question... | 12:51 |
| dtantsur | a good indeed. if we make it a policy, we can just propose the change for them. | 12:51 |
| iurygregory | the scenario where I see it's good to bump is specially when we drop support for some python version (in the case the min requirement we have doesn't work properly with new python) | 12:56 |
| dtantsur | it's probably fair, but only has a strong impact on the 2->3 transition | 12:56 |
| iurygregory | yeah | 12:56 |
| iurygregory | normally for sushy we are able to backport things and we release for other stable branches | 12:57 |
| iurygregory | maybe we just couldn't backport some of the SMC fixes? (at least from what I remember we backported most of them) | 12:58 |
| TheJulia | wasn't one of the big things related to etags? | 13:03 |
| * TheJulia lacks complete understanding of what was needed for SMC in sushy | 13:04 | |
| * TheJulia is also waiting for the coffee to cool a little bit | 13:04 | |
| dtantsur | I was under impression we backported the fixes | 13:51 |
| dtantsur | this was another cause of my surprise by that bump | 13:52 |
| dtantsur | https://review.opendev.org/q/Idb2d6bee96bd23cfc4fb8f014661231766998781 backported | 13:53 |
| dtantsur | https://review.opendev.org/q/I021d1116613e2b35352ca9749f8bc8a84bf655a1 backported | 13:53 |
| dtantsur | https://review.opendev.org/q/I7ea0f7ef856cc47f1d4ea334717bfb4577d0c093 backported | 13:54 |
| dtantsur | what fixes do we miss? | 13:54 |
| dtantsur | or maybe we just haven't released them? | 13:54 |
| dtantsur | (I do think we released yoga 4.1.2 with this fix) | 13:58 |
| iurygregory | https://storyboard.openstack.org/#!/story/2009865 | 14:16 |
| iurygregory | this was the story | 14:16 |
| iurygregory | https://review.opendev.org/q/I384baf5c27a531be0c4b78e11e1214d6371fa8be | 14:16 |
| iurygregory | but we did backport to xena and wallaby... | 14:16 |
| iurygregory | maybe they are using other version and we didn't backport? | 14:17 |
| TheJulia | oh migrations... why are you so broken on my machine :( | 14:18 |
| dtantsur | it's not an accident that the words "migrations" and "migraine" look similar | 14:49 |
| TheJulia | blah | 14:58 |
| JayFisTethered | Big downside to self-hosting your IRC client bastion: when internet is out, my irc is out :( | 15:18 |
| dtantsur | oops | 15:22 |
| JayFisTethered | I'm trying to dig on a bifrost failure, but I'm not super experienced in troubleshooting CI for bifrost | 15:49 |
| JayFisTethered | https://zuul.opendev.org/t/openstack/build/062f0c9f6a764cce869b6a3f597c6889/log/logs/bifrost.log#14801 is this the failure being complained about? | 15:50 |
| JayFisTethered | (for https://review.opendev.org/c/openstack/bifrost/+/849247 ) | 15:50 |
| dtantsur | JayFisTethered: correct | 15:54 |
| dtantsur | if you scroll a bit down: | 15:54 |
| dtantsur | "You must correct your GRUB install devices before proceeding:", | 15:54 |
| dtantsur | "", | 15:54 |
| dtantsur | " DEBIAN_FRONTEND=dialog dpkg --configure grub-pc", | 15:54 |
| dtantsur | " dpkg --configure -a", | 15:54 |
| dtantsur | "dpkg: error processing package grub-pc (--configure):", | 15:54 |
| dtantsur | " installed grub-pc package post-installation script subprocess returned error exit status 1", | 15:54 |
| dtantsur | which is puzzling... | 15:54 |
| JayFisTethered | Yeah; looked like an upstream distro break to me, potentially | 15:54 |
| JayFisTethered | nothing in that PR could break it afaict | 15:54 |
| dtantsur | https://forum.openmediavault.org/index.php?thread/38006-recent-updates-and-upgrades-can-fail-when-updating-to-grub-pc-2-02-dfsg1-20-deb1/ | 15:55 |
| dtantsur | jesus | 15:55 |
| JayFisTethered | aw hell | 15:55 |
| JayFisTethered | We should see if there's a bug against this behavior | 15:56 |
| JayFisTethered | that's a big break for stable (Right? bullseye is stable, right?) | 15:56 |
| dtantsur | yeahh.. | 15:56 |
| JayFisTethered | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=535239 | 15:57 |
| JayFisTethered | that's not exactly it | 15:57 |
| JayFisTethered | https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=grub-pc | 15:58 |
| dtantsur | we may need to bring it to the infra | 16:01 |
| JayF | o/ | 16:03 |
| JayF | I'm back to normal | 16:03 |
| dtantsur | \o | 16:03 |
| JayF | I mean, this seems like a bifrost-specific break, no? | 16:04 |
| dtantsur | JayF: we could do something like https://github.com/turnkeylinux/tracker/issues/1579#issuecomment-892221912, but it's probably not the right thing to do on bare metal | 16:04 |
| dtantsur | JayF: no, it's CI-specific. I guess grub in the CI is not set up correctly. | 16:04 |
| JayF | oh, it's erroring *on the machine bifrost is running* that it can't install grub | 16:05 |
| dtantsur | yes | 16:05 |
| dtantsur | see my explanation on #opendev now | 16:06 |
| * TheJulia glares at test_migrations.py | 16:28 | |
| opendevreview | Julia Kreger proposed openstack/ironic master: Phase 2 - SQLAlchemy 2.0 compatability https://review.opendev.org/c/openstack/ironic/+/857516 | 16:33 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP Phase 3 - SQLAlchemy 2.0 Compatability https://review.opendev.org/c/openstack/ironic/+/857932 | 16:33 |
| TheJulia | blarg | 16:33 |
| TheJulia | so... still got some sporadic warnings appearing on test_migrations.py's run, but getting there | 16:34 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP Phase 3 - SQLAlchemy 2.0 Compatability https://review.opendev.org/c/openstack/ironic/+/857932 | 16:44 |
| TheJulia | okay, that takes care of a couple more | 16:44 |
| TheJulia | w/r/t sqlalchemy stuffs, as much as I'd love to see them merge for Zed, I don't think we should given how much risk that has with it being *that close* to the DB | 16:50 |
| TheJulia | although if openstack wide is shipping the minimum sqlalchemy version at 1.4, we could eventually backport them | 16:51 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Do not install grub2 and shim on the host system https://review.opendev.org/c/openstack/bifrost/+/857935 | 16:55 |
| dtantsur | JayF: let's see if ^^ works | 16:55 |
| JayF | aight, I'll look once tests finish | 16:56 |
| dtantsur | o/ | 17:02 |
| TheJulia | goodnight dtantsur | 17:30 |
| * JayF about to do the 100k nodes benchmarks | 17:48 | |
| JayF | curious how long it'll take just to insert the data lol | 17:49 |
| TheJulia | mysql insert is slow.... | 18:16 |
| TheJulia | soooooooo | 18:16 |
| TheJulia | :) | 18:16 |
| JayF | > Created 100000 nodes in 1630.1096534729004 seconds. | 18:24 |
| JayF | honestly, not that terrible. | 18:25 |
| JayF | Now I'm going to see if my PC melts as I setup the benchmark lol | 18:25 |
| JayF | current master: > Took 121.13843488693237 seconds to return all 100000 nodes via nodes API call pattern. | 18:31 |
| JayF | with phase 1+2 patches cherry picked on top: > Took 121.2013418674469 seconds to return all 100000 nodes via nodes API call pattern. | 18:31 |
| JayF | TheJulia: ^^ If it's slower, it's not slower-enough to matter | 18:31 |
| JayF | .07s difference when getting 100k nodes lol | 18:32 |
| TheJulia | lol | 18:51 |
| TheJulia | 825/nodes/sec is blisteringly fast | 18:52 |
| JayF | I mean, I can multithread the API calls | 18:56 |
| JayF | or add even more nodes | 18:56 |
| JayF | but I'm fairly convinced there's no slowdown on the python side | 18:56 |
| TheJulia | yeah | 18:56 |
| JayF | part of why I picked my beefy desktop to run these is to try and remove mysql from the equation... this should all fit in ram so it's all about single threaded python perf | 18:56 |
| TheJulia | 100k nodes really exposes it if there is anything | 18:56 |
| JayF | and like, it's reproducably very slightly slower with the changes | 18:56 |
| JayF | ooooh | 18:56 |
| JayF | I wonder if I should try with your changes /and sqlalchemy 2.0/ | 18:57 |
| JayF | I wonder if I should try with your changes /and sqlalchemy 2.0/ | 18:57 |
| JayF | will that work now/ | 18:57 |
| TheJulia | should 2.0.0b1 is out | 18:57 |
| JayF | it's not in pypi | 18:59 |
| JayF | but I'll try and get it via git | 19:00 |
| JayF | TheJulia: it doesn't work at all against sqla 2.0 :( | 19:03 |
| JayF | 2022-09-15 12:03:38.968 1581244 CRITICAL ironic [-] Unhandled error: sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper[Node(nodes)], expression "relationship("List['NodeTrait']")" seems to be using a generic class as the argument to relationship(); please state the generic argument using an annotation, e.g. "traits: Mapped[List['NodeTrait']] = | 19:04 |
| JayF | relationship()" | 19:04 |
| TheJulia | WOOOT! | 19:04 |
| JayF | sarcastic woot, I presume? lol | 19:04 |
| JayF | you want me to put this somewhere :( | 19:04 |
| JayF | like in a bug or a PR comment or something | 19:04 |
| TheJulia | yeah, please | 19:04 |
| JayF | tell me where and it shall be done | 19:05 |
| TheJulia | interestingly enough.. I did that | 19:05 |
| TheJulia | at least, I think i did | 19:05 |
| JayF | do I need the phase 3 patch, too? | 19:05 |
| TheJulia | no good ideas of where at the moment | 19:05 |
| TheJulia | nah, that is all test_migrations | 19:05 |
| TheJulia | and horribly broken still | 19:05 |
| JayF | do you want me to dig this? | 19:06 |
| TheJulia | maybe comment on the change | 19:06 |
| JayF | I've steered clear of actual engineering b/c I figured the coordination would be more pain than I could provide value | 19:06 |
| JayF | https://gist.github.com/jayofdoom/cb132e3d39213eb79b195e849bfbcebb and it's a comment on the phase2 patch | 19:07 |
| JayF | TheJulia: I'm going to try and clear this error if I can | 19:10 |
| JayF | and at least comment on your pr how to fix | 19:10 |
| JayF | yeah. it's in the new traits relationship stuff added in phase1, but I'm not sure why/how | 19:20 |
| TheJulia | yeah... oddly enough in 1.4 it works like a champ | 19:29 |
| TheJulia | and it was added in 1.4 for 2.0 | 19:29 |
| TheJulia | hi mysql, why you hate me. | 19:29 |
| TheJulia | err, mariadb | 19:29 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Phase 1 - SQLAlchemy 2.0 Compatability https://review.opendev.org/c/openstack/ironic/+/856336 | 19:30 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Phase 2 - SQLAlchemy 2.0 compatability https://review.opendev.org/c/openstack/ironic/+/857516 | 19:30 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP Phase 3 - SQLAlchemy 2.0 Compatability https://review.opendev.org/c/openstack/ironic/+/857932 | 19:30 |
| TheJulia | oh, so I think I know whati s going on with the mapping | 19:31 |
| TheJulia | after re-reading it again | 19:31 |
| TheJulia | fixing | 19:39 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Phase 1 - SQLAlchemy 2.0 Compatability https://review.opendev.org/c/openstack/ironic/+/856336 | 19:40 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Phase 2 - SQLAlchemy 2.0 Compatability https://review.opendev.org/c/openstack/ironic/+/857516 | 19:40 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP Phase 3 - SQLAlchemy 2.0 Compatability https://review.opendev.org/c/openstack/ironic/+/857932 | 19:40 |
| JayF | oooh exciting | 19:40 |
| JayF | oooooooh | 19:41 |
| JayF | that makes so much sense | 19:41 |
| JayF | benchmark is not going big kaboom! | 19:43 |
| TheJulia | \o/ | 19:43 |
| TheJulia | rumor has it, it should be even *faster* | 19:43 |
| JayF | well, data will has it in about 100 seconds give or take lol | 19:44 |
| TheJulia | 15% increase, nice | 19:44 |
| JayF | no, I mean, if we wait it'll be data and not rumor | 19:45 |
| JayF | I have no results yet | 19:45 |
| TheJulia | oh! | 19:45 |
| TheJulia | okay | 19:45 |
| JayF | also was thinking of refactoring this benchmark tool a little | 19:45 |
| TheJulia | by all means | 19:45 |
| JayF | to make it not really assume you have /etc/ironic/ironic.conf and the like lol | 19:45 |
| JayF | just adding some cli flags to set the database url would make it containerizable I think | 19:45 |
| TheJulia | Yeah, it was a quick and dirty "i need to exercise these lower layers" code | 19:45 |
| JayF | > Took 117.68264269828796 seconds to return all 100000 nodes via nodes API call pattern. | 19:46 |
| TheJulia | slightly better! | 19:46 |
| JayF | that's a real improvement | 19:46 |
| JayF | /home/jay/current-master.txt:Took 121.13843488693237 seconds to return all 100000 nodes via nodes API call pattern. | 19:46 |
| JayF | /home/jay/phase2-with-sqla2.txt:Took 117.68264269828796 seconds to return all 100000 nodes via nodes API call pattern. | 19:46 |
| JayF | /home/jay/with-patches.txt:Took 121.2013418674469 seconds to return all 100000 nodes via nodes API call pattern. | 19:46 |
| JayF | to re-summarize for people who don't want to read above | 19:46 |
| JayF | sqla 1.4 w/patches: small perf hit, barely measurable | 19:47 |
| TheJulia | like... 28/nodes/second faster | 19:47 |
| JayF | sql 2.0 w/patches: improvement of 28/nodes/second | 19:47 |
| JayF | that's stellar | 19:47 |
| * TheJulia dances | 19:48 | |
| TheJulia | anyway, I need to go pickup my car and go have my blood drawn | 19:48 |
| JayF | this is baller nice work TheJulia | 19:48 |
| iurygregory | wow nice results \o/ | 19:50 |
| TheJulia | I'm likely going to try to wrap phase 3 up tomorrow | 19:50 |
| JayF | I think given these results it should be safe to merge into zed if we want; but I might suggest it'd be wise to wait for Antelope | 19:51 |
| TheJulia | I *think* I know what is going on with it, but I can't get it to play nicely with the temporarly table namings locally since it ends up in absurdly long keys and then the engine explodes for some reason | 19:51 |
| JayF | if for no other reason than to give CI an entire cycle to beat on it and expose any edges we might have missed | 19:51 |
| TheJulia | yeah, I'm leaning more towards Antelope personally | 19:51 |
| TheJulia | I did already notice a couple failures which made me think locking in CI, but it is one of those "the right things happened.... maybe something else happend to the VM I can't see that drove it to occur" | 19:52 |
| TheJulia | like an IO pause | 19:52 |
| TheJulia | ... one of the two VMs definitely had a pause occur long enough to timeout a rabbit message | 19:53 |
| iurygregory | and we know that CI always explodes close to the release :D | 19:53 |
| TheJulia | exactly | 19:55 |
| JayF | came across this in RFE review: https://storyboard.openstack.org/#!/story/1300939 | 21:50 |
| JayF | we don't even have any translations committed, do we? | 21:50 |
| JayF | This seems to me like a "it's important enough to do" or "just close the ticket and admit we won't" | 21:50 |
| TheJulia | wow that is an oldie | 21:53 |
| TheJulia | we have some translations | 21:54 |
| TheJulia | maybe not | 21:55 |
| TheJulia | okay, we had translations | 21:56 |
| TheJulia | I think it makes sense to close because the argument can be made that translating content can be a breaking behavior change | 21:56 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Document existence of non-production "fake" driver https://review.opendev.org/c/openstack/ironic/+/857984 | 22:05 |
| JayF | the problem with me doing bug triage is that I stop in the middle to fix them sometimes ^ :P | 22:08 |
| JayF | That should get like, content review -- if there are further use cases for the fake driver, I'm unfamiliar with them, but we shoudl include them | 22:08 |
| JayF | https://storyboard.openstack.org/#!/story/1427923 this is another one that I'm kinda shocked if it still exists, but in this case, if it does we should consider fixing it | 22:10 |
| *** bodgix3 is now known as bodgix | 22:12 | |
| TheJulia | it does unfortunately, although it is a rare operation and I think you functionally need system/owner admin rights | 22:18 |
| JayF | we have an entire cycle worth of these kind of "it's bad but not horrible" bugs | 22:19 |
| * TheJulia wonders why not see if outreachy interns might be interested | 22:20 | |
| TheJulia | In at least the smaller ones as long as we don't force them through a spec | 22:20 |
| JayF | Eh, maybe. None of it is crazy hard but we're not talking low hanging fruit on most of those | 22:21 |
| TheJulia | it is that or we accept they are by design but not ideal | 22:21 |
| TheJulia | yeah, you give them one thing, not multiple | 22:21 |
| TheJulia | but yeah | 22:21 |
| JayF | I think we're better off to accept it and close the bug if we're not going to fix them | 22:21 |
| JayF | if we want the bugtracker to be useful, we can't keep aspirational rfes-that-are-hiding-in-bugs kinda things like that in there never moving | 22:22 |
| JayF | (or we need a tool that allows better exclusion of such bugs) | 22:22 |
| TheJulia | agreed | 22:22 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Document existence of non-production "fake" driver https://review.opendev.org/c/openstack/ironic/+/857984 | 22:50 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Document existence of non-production "fake" driver https://review.opendev.org/c/openstack/ironic/+/857984 | 22:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!