| rpittau | good morning ironic! o/ | 06:59 |
|---|---|---|
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Add inspection hooks https://review.opendev.org/c/openstack/ironic/+/892661 | 07:48 |
| *** Continuity__ is now known as Continuity | 08:03 | |
| Continuity | Morning o/ | 08:04 |
| *** Continuity__ is now known as Continuity | 10:24 | |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch stable/xena: CI: Removing job queue https://review.opendev.org/c/openstack/networking-generic-switch/+/897023 | 11:13 |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch stable/wallaby: CI: Removing job queue https://review.opendev.org/c/openstack/networking-generic-switch/+/897024 | 11:13 |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch stable/victoria: CI: Removing job queue https://review.opendev.org/c/openstack/networking-generic-switch/+/897025 | 11:13 |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch stable/ussuri: CI: Removing job queue https://review.opendev.org/c/openstack/networking-generic-switch/+/897026 | 11:13 |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch stable/train: CI: Removing job queue https://review.opendev.org/c/openstack/networking-generic-switch/+/897027 | 11:14 |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch stable/pike: CI: Removing job queue https://review.opendev.org/c/openstack/networking-generic-switch/+/897028 | 11:14 |
| iurygregory | good morning Ironic o/ | 11:15 |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch stable/pike: CI: Removing job queue https://review.opendev.org/c/openstack/networking-generic-switch/+/897028 | 11:49 |
| * TheJulia raises eyebrows re branches | 12:54 | |
| TheJulia | zigo: yes, you can, the inherent challenge is if your in a higher security environment, you need to change the network, which was the original root question if you've tried it out | 12:55 |
| zigo | Well, I'm still stuck at the level of before the last summit, where the Ironic client doesn't use the correct endpoint, so my Ironic setup isn't complete ... :/ | 12:56 |
| TheJulia | ugh :( | 13:08 |
| frickler | mnasiadka: iirc JayF was working on retiring at least the older n-g-s branches, so the above should not be necessary | 13:08 |
| mnasiadka | ah okI'll abandon - no problem ;) | 13:08 |
| TheJulia | zigo: I don't entirely remember at this point, but didn't it seem like like rooted in keystoneauth1 ? | 13:09 |
| zigo | TheJulia: It was hard to debug, because it failed in a generator, so I didn't really know how to go forward... | 13:09 |
| TheJulia | :\ | 13:09 |
| zigo | IMO, the issue was in ironicclient. | 13:09 |
| TheJulia | zigo: was there a bug filed? | 13:09 |
| zigo | I'm sorry I had no time to move forward ... | 13:09 |
| TheJulia | I can't look this week, I'm driving to San Francisco for a conference | 13:10 |
| zigo | No bug filled, because it wouldn't have been valuable enough with enough details. | 13:10 |
| TheJulia | .. and giving that talk | 13:10 |
| TheJulia | zigo: ahh | 13:10 |
| TheJulia | hmmm. | 13:10 |
| TheJulia | I don't remember enough of what was going on to try and hunt | 13:10 |
| TheJulia | unfortunately | 13:10 |
| zigo | But in short, it was because I had the Ironic endpoint at /<SOMETHING> instead of :<someport>/ (ie: root of the http). | 13:10 |
| frickler | mnasiadka: see e.g. https://review.opendev.org/c/openstack/releases/+/895707 | 13:11 |
| zigo | TheJulia: I'll try to re-start my PoC, which is easy to setup, as I scripted absolutely all. | 13:11 |
| zigo | It starts Ironic in Debian and all the setup is done with Puppet... | 13:11 |
| mnasiadka | frickler: sure, done ;) | 13:12 |
| TheJulia | zigo: the openstack puppet modules? | 13:13 |
| TheJulia | zigo: I actually have an idea of where to look, but won't have time really for a few days | 13:14 |
| dtantsur | So, our scale testers managed to choke Ironic to the extent it only returns NoFreeConductorWorker | 13:34 |
| dtantsur | I wonder how realistic is having a limit of 100 workers. Has anyone tried raising it? | 13:35 |
| dtantsur | cc TheJulia, JayF ^^ | 13:35 |
| TheJulia | I have heard of some folks running towards 200 | 13:35 |
| TheJulia | But haven't thought about it honestly | 13:37 |
| TheJulia | wheee 3.5 hours of driving to go | 13:38 |
| dtantsur | wheeee | 13:40 |
| dtantsur | deploying 500-600 nodes in parallel, well, no wonder it has issues... | 13:42 |
| TheJulia | wheeeeeeee | 13:42 |
| rpittau | good luck TheJulia :) | 13:43 |
| iurygregory | wondering if arne_wiebalck tested deploying this amount in parallel :D | 13:43 |
| opendevreview | Merged openstack/ironic-prometheus-exporter master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/896048 | 13:43 |
| iurygregory | good luck TheJulia | 13:43 |
| TheJulia | I've heard of folks doing such huge deploys... with numerous conductors and an awesome backend database cluster of super magic scale | 13:44 |
| iurygregory | super magic is the answer | 13:44 |
| dtantsur | All we have is 1 conductor backed by SQLite :D | 13:45 |
| iurygregory | wondering why in the past things used to work lol | 13:46 |
| dtantsur | I suspect it was non-converged flow | 13:46 |
| dtantsur | I also suspect we need to do something about heartbeats.. | 13:46 |
| iurygregory | hummmm | 13:47 |
| TheJulia | how so? I can guess and all :) | 13:47 |
| dtantsur | 500 nodes heartbeating may prevent the conductor from doing anything else | 13:48 |
| opendevreview | Baptiste Jonglez proposed openstack/networking-generic-switch master: Introduce NGS agent design https://review.opendev.org/c/openstack/networking-generic-switch/+/897047 | 13:49 |
| iurygregory | but this would happen in both flows now? non-converged and the converged one? | 13:50 |
| dtantsur | no heartbeats in the non-coverged flow. it's just ramdisk deploy. | 13:50 |
| iurygregory | oh ok! | 13:50 |
| TheJulia | dtantsur: well, yeah :) If we kick back a "no free conductor" error to it, then we likely ought to make sure we have appropriate hold downs in the agent | 13:51 |
| dtantsur | Hmm the tester claims to had been running the converged flow before. Interesting. | 13:53 |
| opendevreview | Merged openstack/ironic-inspector master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/ironic-inspector/+/896045 | 13:53 |
| JayF | I wonder if the various misplaced locks somehow were acting as a rate limit or bottleneck. | 13:54 |
| JayF | And now with those gone, we'll try as hard as we can to deploy everything immediately, even in a case where that might not be wise | 13:54 |
| dtantsur | Possibly | 13:55 |
| TheJulia | looks like we do have an opportunity to teach the agent to be a little smarter, fwiw | 13:56 |
| dtantsur | I wonder if the thread number must be larger than max_concurrent_deploy | 13:56 |
| dtantsur | and then we can use max_concurrent_deploy to tell BMO to go retry | 13:56 |
| TheJulia | oh yes, every yes | 13:56 |
| dtantsur | right. max_concurrent_deploy defaults to 250, the number of threads - to 100 | 13:56 |
| TheJulia | concurrent is across the board/entire deployment wise for a scaled deployment | 13:57 |
| dtantsur | fair enough | 13:57 |
| * dtantsur needs to remember what happens when `max_concurrent_deploy is hit | 13:57 | |
| TheJulia | maximum upper bounds of deploys in flight | 13:58 |
| TheJulia | I think we error on the deploy state change request | 13:58 |
| dtantsur | ConcurrentActionLimit is a generic HTTP 500? | 13:58 |
| TheJulia | I believe so yes | 13:59 |
| dtantsur | hmm, inconveniet | 13:59 |
| dtantsur | TheJulia, do you remember why it was not derived from TemporaryFailure? | 13:59 |
| TheJulia | no specific reason | 14:01 |
| TheJulia | I'd treat it as a bug at this point if we want to change it | 14:01 |
| TheJulia | *also* | 14:01 |
| rpittau | going to catch a flight, see you tomorrow! o/ | 14:01 |
| TheJulia | the concurrent action limit was in mainly abuse prevention, so TemporaryFailure is likely better | 14:02 |
| iurygregory | safe travels rpittau o/ | 14:02 |
| dtantsur | (logs with thousands of clusters look scary) | 14:02 |
| TheJulia | ugh | 14:02 |
| iurygregory | yes it does | 14:02 |
| dtantsur | I've just had my archive manager crashing on trying to read this tar.gz | 14:02 |
| iurygregory | specially when the files were rotated every 50MB.. =) | 14:02 |
| TheJulia | ugh | 14:02 |
| iurygregory | oh wow, this hasn't happen to me yet | 14:03 |
| TheJulia | It has happened to me a few times | 14:03 |
| TheJulia | I've gotten multi gigabyte log dumps from customres | 14:03 |
| dtantsur | iurygregory: must-gather for just the openshift-machine-api namespace is 2.5G | 14:03 |
| TheJulia | archie manager chokes hardcore | 14:04 |
| iurygregory | WTH | 14:04 |
| TheJulia | You need to manually extract for your own sanity | 14:04 |
| dtantsur | that's a monstrous testing lab. they repeatedly lunch thousands of OCP clusters for 10-15 hours. | 14:04 |
| iurygregory | only for that namespace?! JESUS | 14:04 |
| iurygregory | yeah, the 502 bug I'm working is on the same lab if I recall | 14:05 |
| dtantsur | yeah, I think I've extracted the relevant bits | 14:05 |
| dtantsur | thinking what I could need more | 14:05 |
| dtantsur | (thanks god vim can handle 100M text files - normal editors choke much before) | 14:05 |
| iurygregory | truth ^ | 14:06 |
| dtantsur | I must admit, I'm excited about this bug. I love to see Ironic tested at such a scale! | 14:08 |
| TheJulia | dtantsur: if you want to change the base class to temporaryfailure, consider me +2 | 14:11 |
| dtantsur | on it | 14:11 |
| TheJulia | Can't spend much more time talking about it today, I need to hit the road soon if I want to get into San Francisco before lunch time | 14:11 |
| dtantsur | Sure, no worries. I need to stare at the logs, maybe play with numbers. | 14:12 |
| TheJulia | enjoy | 14:12 |
| TheJulia | :) | 14:12 |
| * TheJulia packs up laptop and goes to find breakfast and resume the drive | 14:26 | |
| iurygregory | safe travel TheJulia =) | 14:29 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Fix the HTTP code for reaching max_concurrent_deploy: 503 instead of 500 https://review.opendev.org/c/openstack/ironic/+/897050 | 14:34 |
| dtantsur | btw anyone who wants to drive forward scale testing Ironic should review https://review.opendev.org/c/openstack/sushy-tools/+/875366/ | 14:38 |
| dtantsur | this can allow us to test this sort of scale without being able to create many thousands of VMs | 14:38 |
| dtantsur | hold on, what the hell is this? https://opendev.org/openstack/ironic/src/branch/master/ironic/conf/conductor.py#L216-L217 | 14:44 |
| dtantsur | this seems... far-reaching | 14:45 |
| iurygregory | I think it was for the effort to support multi-arch or something | 14:45 |
| iurygregory | trying to remember | 14:45 |
| JayF | dtantsur: deploy_ramdisk_by_arch | 14:45 |
| dtantsur | I know why the new options were added, I see no point in deprecating the old ones | 14:45 |
| dtantsur | it's just new busy-work for 99% of operators | 14:45 |
| JayF | There was consensus in the reviews for that change that deprecation was the right move; with an extremely long tail | 14:45 |
| dtantsur | jesus.. okay | 14:46 |
| JayF | basically try to drive new opers to the *_by_arch but we don't have a timeline for actually pulling the deprecated value | 14:46 |
| JayF | let me put it this way: I would not be upset if this value existed for a long, long, long time in a deprecated state to avoid that exact busywork you are concerned of | 14:46 |
| dtantsur | okay, that makes sense | 14:46 |
| JayF | but we don't wanna have new opers using the more coarse setting with a fine grained one existing | 14:46 |
| dtantsur | Has anyone pondered if we need bootloader_by_arch? I don't know the answer off the hand | 14:47 |
| JayF | That's a pretty astute observation. | 14:47 |
| JayF | the answer is almost certain yes. dang | 14:48 |
| dtantsur | Possibly no. EFI partitions are arch-independent, aren't they? | 14:48 |
| JayF | they are? | 14:48 |
| dtantsur | haha | 14:48 |
| dtantsur | welllll... I think so? | 14:48 |
| JayF | I honestly have very little experience with ... standards-compliant, server quality arm | 14:48 |
| JayF | Is there any ... affordable arm hardware with some kind of BMC and efi support? | 14:49 |
| dtantsur | On the other hand, operators may not wish to include all possible boot loaders in all possible ISOs | 14:49 |
| dtantsur | JayF, if you find one, tell me. I've been looking for it for some time. | 14:49 |
| TheJulia | They are arch independent | 14:56 |
| TheJulia | They have separate files by arch name | 14:56 |
| dtantsur | yeah, but it may not be desirable to have really large bootloaders | 14:57 |
| dtantsur | * images | 14:57 |
| JayF | so not crucial, but a nice to have | 14:58 |
| JayF | another low-hanging-fruit thing that has clear examples on how to implement | 14:58 |
| * JayF adds to his list to write it up for LP | 14:58 | |
| JayF | I've been focusing on trying to find nuggets like this and write up h/t low-hanging-fruit bugs in LP for them | 14:58 |
| TheJulia | Re by arch stuff, mixed arch shops have used it | 14:58 |
| JayF | I *try* to put enough context to make them doable by anyone who knows how to submit patches to openstack | 14:58 |
| JayF | oh hey it's time | 15:00 |
| iurygregory | yes | 15:00 |
| JayF | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Oct 2 15:00:23 2023 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| JayF | Welcome to the weekly Ironic meeting. | 15:00 |
| iurygregory | o/ | 15:00 |
| dtantsur | o/ | 15:00 |
| JayF | A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. | 15:00 |
| JayF | #topic Announcements/Reminder | 15:00 |
| TheJulia | o/ | 15:00 |
| JayF | #info As always; please add hashtag:ironic-week-prio to patches ready for review; and prioritize your reviews on such patches. | 15:01 |
| JayF | #topic Review Action Items | 15:01 |
| JayF | I had an action to cut another release of NGS; I have not yet as it never made it onto my todo list; my bad. | 15:02 |
| JayF | I'll get that done as soon as we're done | 15:02 |
| JayF | #action JayF Carryover: release another NGS | 15:02 |
| JayF | #topic Bobcat release: October 4th, 2023 | 15:03 |
| JayF | Other than that refreshed action; is there anything else we need to do/consider/etc? | 15:03 |
| JayF | Thank you all for another awesome Ironic release! \o/ I'll be on OpenInfra Live on Thursday talking about it; if you want me to highlight something specific please say so. | 15:03 |
| iurygregory | \o/ | 15:04 |
| TheJulia | Not that I’m aware of, likely a reminder wrt the PTG and etherpad | 15:04 |
| JayF | ack; good callout | 15:04 |
| JayF | That means I need to have a time for the BM SIG by then | 15:04 |
| JayF | which leads into... | 15:04 |
| JayF | #topic PTG Planning for PTG October 23-27, 2023 | 15:05 |
| JayF | I'll be honest, the time I had set aside to take care of this before the meeting was eaten by an emergency Friday; but for now my plan is the following: | 15:05 |
| JayF | I'm working off the assumption of similar/identical availability to last PTG | 15:05 |
| JayF | 1300-1700 UTC main window, 2200-2400 APAC friendly if needed. Any objections or additional input? | 15:05 |
| JayF | I will be sending this to the mailing list as well for feedback. | 15:05 |
| TheJulia | I think that is reasonable | 15:06 |
| JayF | My intention will be to start labelling topics in PTG etherpad with times (or 'do we need to talk about this?' in a couple of cases I suspect) | 15:06 |
| JayF | I'll make noise in here when I do so if someone wants to async feedback with me; the goal will be to have a schedule nailed down for time slots before Thursday, and hopefully have topics<->timeslots nailed down by EOW | 15:07 |
| TheJulia | I advise against aggressive compression, some of the topics can be split as well | 15:07 |
| JayF | ack | 15:07 |
| TheJulia | Also, operator feedback being good, we should be prepared for emergent items | 15:08 |
| JayF | maybe like, try to keep the top level slots to a larger theme | 15:08 |
| TheJulia | ++, and just be flexible | 15:08 |
| JayF | I think we always are | 15:08 |
| dtantsur | ++ | 15:08 |
| JayF | PTG schedules for Ironic are ... suggestive at best :D | 15:08 |
| JayF | Going to move on? | 15:08 |
| JayF | #topic Review Ironic CI status & update whiteboard if needed | 15:08 |
| TheJulia | We also need to schedule a 3rd party ci session during the PTG | 15:09 |
| JayF | That is on the list :) | 15:09 |
| TheJulia | Cool cool | 15:09 |
| TheJulia | I resume driving then | 15:09 |
| JayF | the list is ... huge | 15:09 |
| JayF | If you've (not Julia-you, anyone-listening-you) not looked at it, please do | 15:09 |
| JayF | I have no knowledge about CI status and the like | 15:09 |
| JayF | I think we're doing OK? Julia fixed some metalsmith pain around SDK/Ansible shenanigans | 15:09 |
| JayF | Looks like folks have similar opinions to CI as me :) outta sight outta mind | 15:11 |
| JayF | #info No RFEs to review, skipping agenda item | 15:11 |
| JayF | #topic Open Discussion | 15:11 |
| dtantsur | yeah, I'm not aware of issues | 15:11 |
| JayF | any open discussion items? | 15:11 |
| dtantsur | Does anything speak against increasing heartbeat interval? | 15:12 |
| JayF | Hmm. | 15:12 |
| JayF | we used to document it loudly | 15:12 |
| TheJulia | dtantsur: eh, mixed feelings. I’d teach agent to handle the too busy case first | 15:13 |
| JayF | oh, you're asking a different question than I thought you were? | 15:13 |
| JayF | heartbeat interval will not help you for performance when doing 500+ deploys | 15:13 |
| JayF | because agent will check in as soon as it's done with an item | 15:13 |
| JayF | regardless of hb interval, as an optimization | 15:13 |
| dtantsur | yeah, I know | 15:13 |
| dtantsur | but at this scale, even periodic check-ups can be an issu3e | 15:13 |
| JayF | This seems like a combination of edge cases? Extreme scale + fast track + simultaneous deploys? Just making sure I understand | 15:14 |
| JayF | ? | 15:14 |
| JayF | I guess extreme scale isn't really a good term, it's more about pushing so many deploys at once | 15:14 |
| dtantsur | I don't think fast track is even relevant; just huge scale | 15:14 |
| JayF | I think it's: huge scale *on one conductor* | 15:15 |
| JayF | which traditionally isn't even something we've optimized for, yeah? | 15:15 |
| TheJulia | Guys, we’re in meeting… | 15:15 |
| JayF | this is open discussion | 15:15 |
| JayF | I think this counts :) | 15:15 |
| TheJulia | Oh, didn’t realize we moved to open | 15:15 |
| TheJulia | Sorry! | 15:15 |
| iurygregory | it happens =) | 15:15 |
| JayF | I'm just trying to understand the problem space/edges here... if it's single conductor/single process I'm not super surprised there are edges here, and incresing perf in that case is good (and handling being on the edges of our scale) | 15:16 |
| JayF | dtantsur: I suggest writing this up in an LP bug, with details about the edges, and potentially if we can brainstorm specific "scaling up simultaneous deploys w/single-process ironic" maybe have a PTG session about it | 15:18 |
| dtantsur | hmm, good point | 15:18 |
| JayF | I know for a fact you can do 500 simul deploys with Ironic with a scaled up cluster, conductor groups, all the trimmings :D | 15:19 |
| JayF | doing it on a single-process will be an extremely valuable exercise in seeing how Ironic handles queuing tasks and being patient under heavy load | 15:20 |
| JayF | it'll be fun and will be a nice robustness exercise, similar to how the sqlite locking fixes were, except hopefully these will be 500x less painful :D | 15:20 |
| JayF | I'm going to close the meeting, this can just be chat in channel | 15:20 |
| JayF | ty all o/ | 15:20 |
| JayF | #endmeeting | 15:20 |
| opendevmeet | Meeting ended Mon Oct 2 15:20:52 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:20 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-10-02-15.00.html | 15:20 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-10-02-15.00.txt | 15:20 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-10-02-15.00.log.html | 15:20 |
| JayF | heh it just occcurred to me | 15:22 |
| JayF | we literally *remove the queue* for single process Ironic | 15:22 |
| JayF | lol | 15:22 |
| JayF | Action item actioned https://review.opendev.org/c/openstack/releases/+/897054 NGS 7.3.0: addl release for Bobcat | 15:32 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Delay heartbeat after a failure and handle Unavailable https://review.opendev.org/c/openstack/ironic-python-agent/+/897055 | 15:41 |
| dtantsur | opinions? | 15:41 |
| dtantsur | hmm, maybe it's actually already handled by the event.wait | 15:42 |
| dtantsur | Just as a data point, Ironic received around 100 heartbeat requests PER SECOND | 15:49 |
| dtantsur | I nearly think we need a separate thread pool for heartbeats | 15:54 |
| iurygregory | the logic does make sense to me | 16:01 |
| iurygregory | trying to understand the part event.wait you mentioned | 16:02 |
| dtantsur | iurygregory: see the loop just above. it uses wait() on an event with a minimal wait for 5 seconds already | 16:03 |
| iurygregory | oh the while not self.stop_event.wait ? | 16:03 |
| dtantsur | yeah | 16:03 |
| iurygregory | right, make sense | 16:04 |
| dtantsur | We also have an issue in our thread pool handling that essentially doubles the capacity... | 16:05 |
| TheJulia | 100/second is super impressive | 16:09 |
| shermanm | I actually had a question related to large (100-500) instance launches on a single conductor. Most of the time seemed to get taken up by hashing and converting the glance image from qcow2->raw? | 16:13 |
| dtantsur | you can easily hit that (not my case though) | 16:14 |
| dtantsur | so, we're doing a weird thing. we let eventlet create 100 threads. so far so good. then we let futurist park 100 more requests that just wait for these 100 threads. | 16:14 |
| dtantsur | this is my fault: I think I misunderstood the futurist API back then. | 16:14 |
| dtantsur | I don't believe letting requests sit and wait is a good idea in any case. I'd rather have 200 threads trying to run. | 16:15 |
| shermanm | (sorry, I also don't need to interrupt the threading/heartbeat discussion) | 16:16 |
| dtantsur | no worries. you're right: image conversion is very heavy. shouldn't block the threads, but will otherwise take a ton of time and CPU | 16:17 |
| * dtantsur hopes he didn't scare them away :( | 16:18 | |
| shermanm | mostly I was trying to figure out what combination of configuration triggered such a conversion on the host, as I was under the impression that you could send a qcow2 (or raw) image to the agent directly, and not have the controller do this conversion | 16:20 |
| dtantsur | you can, you need to set force_raw to False, keeping in mind that your qcow2 conversion will happen in memory on the node | 16:22 |
| shermanm | ahh, thank you | 16:24 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Bump workers_pool_size to 300 and remove queueing of tasks https://review.opendev.org/c/openstack/ironic/+/897071 | 16:44 |
| dtantsur | the case of a release note being larger than the change ^^^ JayF and TheJulia, your opinions especially appreciated | 16:44 |
| JayF | I renew my request for documenting the conditions in a bug; it's difficult for me to consume that sorta information in the IRC firehose and without all the context it's hard to review the change | 16:45 |
| dtantsur | JayF, it's not per se a fix for that bug, more of a logical problem in our defaults | 16:45 |
| JayF | wow that is like, two lines and 50 lines of release note LOL | 16:45 |
| dtantsur | I'll WIP it until I can double-check all assumptions there. but please review anyway. | 16:47 |
| JayF | the changes look OK, but also are the sort of thing that could have unintended side effects | 16:47 |
| JayF | I wanna see the CI run on it, and I'm curious what it'd do different under load but we don't have a good way to test that upstream | 16:48 |
| JayF | if we are going to land a change like that; top of the cycle is the 100% ideal place to land it so if it did cause weirdness we'd find it | 16:48 |
| JayF | (not the default change; the no-more-queuing) | 16:48 |
| dtantsur | I feel like futurist is a mess... | 16:48 |
| dtantsur | I think my assumptions may be right. Will see what the CI says. | 16:58 |
| dtantsur | The more I think about it, the more I'd like to backport the no-more-queuing | 17:12 |
| dtantsur | Queuing means we're keeping TaskManager instances cached somewhere inside futurist. Potentially for a long time. | 17:13 |
| dtantsur | Potentially blocking threads that are actually running. Ugh. | 17:13 |
| JayF | locked, unlocked, or both? | 17:13 |
| JayF | Are taskmanagers always a lock? | 17:13 |
| dtantsur | Only with shared=False | 17:14 |
| dtantsur | But we use that quite often. And then go into spawn_worker. | 17:14 |
| JayF | ah | 17:14 |
| dtantsur | Ha, the tested has found an issue in their configuration. Maybe it's not too bad in the end :D | 17:18 |
| dtantsur | * the tester | 17:18 |
| dtantsur | (they claim to have tested the same thing successfully in the past, so who knows) | 17:18 |
| dtantsur | Anyway, time for dinner. It's a public holiday tomorrow, so talk to you on Wednesday. | 17:19 |
| JayF | Does this make sense as a statement for the Bobcat update? | 18:28 |
| JayF | > Ironic has more thoroughly tested, and resolved many issues related to single conductor mode with SQLite database backend. This is the configuration utilized by Metal3. | 18:28 |
| TheJulia | I would reword the the very first part somehow | 19:09 |
| TheJulia | Maybe flip it, "Ironic resolved issues reported by the Metal3 project related to SQLlite database utilization." | 19:11 |
| JayF | oooooh | 19:11 |
| JayF | I like that a *lot* better | 19:11 |
| JayF | I know it sounded weird, making them the actor fixes it | 19:11 |
| TheJulia | yeah | 19:13 |
| TheJulia | and also emphasises the feedback loop and the power of it | 19:14 |
| JayF | I have a *very* bad habit of slipping into passive voice much too often | 19:14 |
| TheJulia | https://usercontent.irccloud-cdn.com/file/uKuKXHp4/IMG_1782.JPG | 19:18 |
| TheJulia | My awesome hotel room view right now | 19:19 |
| JayF | Whoa I didn't know they were redoing the ferry building tower (that is what it is, yeah?) | 19:19 |
| JayF | I used to work at 2nd@Folsom, I'd often walk straight down Folsom to right under the bay bridge and just look at the water after a long day | 19:19 |
| JayF | (and you got to get on your muni train at the first stop guaranteeing a seat but that was merely a bonus) | 19:20 |
| TheJulia | JayF: yeah | 19:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
