| cardoe | rpittau: could we land https://review.opendev.org/c/openstack/python-ironicclient/+/937628 before you make a client release? | 00:15 |
|---|---|---|
| cardoe | we might also need an epoxy release for the formatter fix that JayF... I can't remember if there was a workaround? | 00:15 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Create service plugin to configure security groups https://review.opendev.org/c/openstack/networking-generic-switch/+/949834 | 04:25 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Support security groups in netmiko base class https://review.opendev.org/c/openstack/networking-generic-switch/+/949835 | 04:25 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Add security group support to netmiko_cisco_nxos https://review.opendev.org/c/openstack/networking-generic-switch/+/952314 | 04:25 |
| opendevreview | Verification of a change to openstack/networking-generic-switch master failed: Move _is_port_supported, _is_port_bound to utils https://review.opendev.org/c/openstack/networking-generic-switch/+/949831 | 06:11 |
| opendevreview | Verification of a change to openstack/networking-generic-switch master failed: Improve test coverage of NetmikoSwitch commands https://review.opendev.org/c/openstack/networking-generic-switch/+/949832 | 06:11 |
| rpittau | good morning ironic! o/ | 07:35 |
| rpittau | cardoe: yeah, I was waiting for that and something else | 07:35 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/sushy-tools master: Add PATCH support for Redfish DateTime fields in Manager resource https://review.opendev.org/c/openstack/sushy-tools/+/950925 | 07:42 |
| opendevreview | Matt Anson proposed openstack/bifrost master: Add support for aarch64 hardware https://review.opendev.org/c/openstack/bifrost/+/952329 | 07:43 |
| opendevreview | Merged openstack/python-ironicclient master: Add CLI support for migrated inspection rules https://review.opendev.org/c/openstack/python-ironicclient/+/937628 | 07:43 |
| queensly[m] | Good morning | 07:45 |
| abongale | Good Morning Ironic! | 08:26 |
| opendevreview | Harald Jensås proposed openstack/ironic master: Include inspector conf groups in sample conf/docs https://review.opendev.org/c/openstack/ironic/+/952338 | 08:52 |
| opendevreview | Harald Jensås proposed openstack/ironic master: Include inspector conf groups in sample conf/docs https://review.opendev.org/c/openstack/ironic/+/952338 | 08:54 |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Add Tempest tests for inspection rules in Ironic https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/951761 | 09:48 |
| opendevreview | Pierre Riteau proposed openstack/ironic master: Add missing space to log statement https://review.opendev.org/c/openstack/ironic/+/952344 | 10:22 |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Add Tempest tests for inspection rules in Ironic https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/951761 | 11:22 |
| opendevreview | Takashi Kajinami proposed openstack/ironic-ui master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/ironic-ui/+/952352 | 11:59 |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Add Tempest tests for inspection rules in Ironic https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/951761 | 12:33 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/sushy-tools master: Add unit tests for set_datetime and get_datetime methods https://review.opendev.org/c/openstack/sushy-tools/+/950925 | 14:20 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/946741 this person has been trying to get this in for a while, and I don't think they fully grok the idea of a "follow up" coming before their change lands. They've jumped through a bunch of hoops for us though, and it'd be nice if we landed it. | 14:29 |
| JayF | Also automated cleaning via runbook https://review.opendev.org/c/openstack/ironic/+/945259 is ready for review and I'd love to get this landed before my sabbatical | 14:29 |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Add Tempest tests for inspection rules in Ironic https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/951761 | 14:30 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Automated cleaning by runbook https://review.opendev.org/c/openstack/ironic/+/945259 | 14:32 |
| JayF | apparently I had one comment-related PR feedback outstanding :D | 14:32 |
| cardoe | I +2'd 946741, JayF. | 14:35 |
| cardoe | TheJulia: can take a peek as well since she's had feedback as well. | 14:35 |
| JayF | yep, I was not going to instaland | 14:36 |
| cardoe | But I agree the person has done everything we've asked of them. | 14:36 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Remove non-abstract methods from HWM https://review.opendev.org/c/openstack/ironic-python-agent/+/951977 | 14:38 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Remove non-abstract methods from HWM https://review.opendev.org/c/openstack/ironic-python-agent/+/951977 | 14:38 |
| cardoe | cid: you got cycles for https://review.opendev.org/c/openstack/openstacksdk/+/945700 rebase? | 14:45 |
| cardoe | Was gonna try to get them to land that for flamingo | 14:45 |
| TheJulia | I might be able to look at it later tdoay | 14:45 |
| TheJulia | runbooks, that is | 14:45 |
| cid | cardoe, I will rebase that before EOD | 14:46 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Remove non-abstract methods from HWM https://review.opendev.org/c/openstack/ironic-python-agent/+/951977 | 14:47 |
| cardoe | bifrost has a bunch of stuff in the pipeline that seems to have a handful of reviews.... like https://review.opendev.org/c/openstack/bifrost/+/946724 seems like an easy +2/+W | 14:48 |
| JayF | I'm basically trying to refresh all my patches and get the relevant ones landed (or landable) and abandon what needs abandonment | 14:48 |
| cardoe | I think we wanna land https://review.opendev.org/c/openstack/ironic/+/946843 before flamgio? | 14:49 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: [docs] Automated cleaning by runbook https://review.opendev.org/c/openstack/ironic/+/951901 | 14:49 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: PTG: Move CI job descriptions into yaml https://review.opendev.org/c/openstack/ironic/+/946584 | 14:53 |
| JayF | the amount I don't want to rebase ^ again is much :) | 14:53 |
| JayF | cardoe: yes, but I think we were going to let it (snmp deprecation) sit until like, milestone 3 just to give that team time. I know NobodyCam has made slight reference to the fact they wanna fix it. | 14:55 |
| JayF | cardoe: that being said, I'm OK with landing it before I go on sabbatical if nobody else is likely to remember | 14:55 |
| * cid gently pings cardoe, that I just rebased the patch. | 15:03 | |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Added a FAQ on CI debugging, starts with network https://review.opendev.org/c/openstack/ironic/+/946589 | 15:07 |
| JayF | dtantsur: bwahaha your comment https://review.opendev.org/c/openstack/ironic/+/941611/1/ironic/tests/unit/common/test_pxe_utils.py is dead on, and that preexisting method is misimplemented. Made those restored-tests passed ... and broke a bunch of badly mocked one | 15:37 |
| JayF | dtantsur: very good catch, ty, I'll have a revision up soon which will fix both issues | 15:37 |
| JayF | we had tests in the redfish module enforcing the unwanted behavior; so something got missed in code review when all that landed :| | 15:41 |
| opendevreview | Merged openstack/bifrost master: Update ironic.conf.j2 template. https://review.opendev.org/c/openstack/bifrost/+/951832 | 16:15 |
| clif | so how flake-y are tests in gerrit these days? I have some failures on my PR from yesterday but so far it looks like the failures are unrelated to my change | 16:18 |
| JayF | can you link it? | 16:20 |
| JayF | I'll take a quick look | 16:20 |
| clif | https://review.opendev.org/c/openstack/ironic/+/952300 | 16:20 |
| JayF | generally we try to figure out why even if not related; but we have a couple of things that are a bit random | 16:20 |
| JayF | mainly related to ordering of things as devstack comes up | 16:20 |
| clif | and is there a way to jump straight to the failure listed in zuul? | 16:22 |
| JayF | I usually just ^f FAILURE | 16:23 |
| JayF | but in those logs from zuul... it's troubleshooting an entire cloud | 16:23 |
| JayF | usually the test failure, and in one case yours looked like this, is basically "we didn't deploy the thing" | 16:23 |
| JayF | and you have to check it | 16:23 |
| JayF | the other with the missing neutron processes are the existing race we've been dealing with for a couple of weeks | 16:23 |
| clif | but, for instance, it seems to break out the stats for OK, Changed, and Failure | 16:24 |
| JayF | someone is probably going to have to coordinate with devstack-repo-proper to fix that race, and I don't have the time or mental capacity to get it done now :) | 16:24 |
| clif | there's really no way to click on "failure" and have it take me to what it thinks the failure is? | 16:24 |
| JayF | Click on gerrit -> click Zuul Summary tab at mid-screen -> look for FAILURE jobs, click the link | 16:26 |
| clif | so does your comment cause a recheck? | 16:26 |
| JayF | yes | 16:26 |
| clif | yea I do that, then on the zuul.opendev.org page it shows "158 OK | 93 Changed | 1 Failure" | 16:27 |
| JayF | once you're in the link; the truncatedoutput you see is basically the ansible run output | 16:27 |
| JayF | which usually isn't useful UNLESS it indicates a devstack setup failure | 16:27 |
| JayF | clif: zoom? | 16:28 |
| clif | I have a meeting in 2 min | 16:28 |
| JayF | ack; after that. it's easier to just go over on a video call | 16:28 |
| clif | sounds good | 16:28 |
| JayF | but essentially for most failures it's not trivial, you have to dig into the controller/logs/ folders and try to ID what happened | 16:28 |
| JayF | in this case, I checked ironic-bm-logs/ (consoles from our fake-baremetal) and noticed that node-1 was killed mid-boot, which indicates some kinda weird, likely environmental failure | 16:29 |
| clif | yeah I've had a fair amount of practice pouring through logs like that, its just the UI in zuul seems to indiciate it "knows" whats failed | 16:29 |
| JayF | or timeout (and given the job was running so long, I think bad performance+timeout) | 16:29 |
| JayF | so those tempest tests running | 16:29 |
| JayF | the "1" we usually care about | 16:29 |
| JayF | is "openstack server create --flavor=fake-baremetal" more or less | 16:29 |
| JayF | which is the end to end | 16:29 |
| JayF | so when it fails, it's basically like "well SOMETHING failed!" | 16:30 |
| JayF | artifacts/Unit Test Report will give you an HTML summary of hte actual tests | 16:30 |
| JayF | which will indicate > ironic_tempest_plugin.tests.scenario.test_baremetal_basic_ops.BaremetalBasicOps < as the only test :D | 16:30 |
| JayF | > tempest.exceptions.BuildErrorException: Server c2d265fe-c26b-4d29-a2ba-11cab5c1a074 failed to build and is in ERROR status is spicier than usual | 16:31 |
| JayF | > Jun 10 21:39:10.784136 np0041084029 ironic-conductor[105848]: ERROR ironic.conductor.utils [None req-252975b7-9919-45fd-b113-e80749c083e5 None None] Timeout reached while waiting for callback for node 6aa6650d-f99d-4948-ab08-318943e04a70 | 16:31 |
| JayF | and there's the smoking gun for it being a performance issue, that combined with the informatino that in the console, it didn't have an error surrounding IPA, just the instance itself went kaput | 16:32 |
| JayF | we could check, I guess, to make sure sushy-tools isn't broken, but at some point you have to stop digging or you get to the molten core of the earth :D | 16:32 |
| JayF | dtantsur: you still around, by any chance? Trying to figure out if our desired expected behavior in the driver_info[deploy_ramdisk] (but not deploy_kernel) case is: an exception, or graceful fallback to less specific values | 17:15 |
| JayF | dtantsur: AFAICT the behavior currently is mixed depending on what driver you use :( | 17:16 |
| JayF | so unifying that behavior is sorta what https://review.opendev.org/c/openstack/ironic/+/941611/ is turning into | 17:16 |
| JayF | meaning: we never use driver_info[ramdisk] with config[kernel]; but we would use config[kernel] and config[ramdisk] in a case where driver_info[ramdisk] (without kernel) was set | 17:17 |
| JayF | existing behavior in pxe_utils: throw exception if we see driver_info[ramdisk] but not driver_info[kernel] | 17:18 |
| JayF | existing behavior in redfish driver: use driver_info[ramdisk] and config[kernel] together (wrong) | 17:18 |
| opendevreview | Harald Jensås proposed openstack/networking-baremetal master: Clean up baremetal agents on node delete https://review.opendev.org/c/openstack/networking-baremetal/+/945500 | 17:49 |
| dtantsur | JayF: by chance, somewhat around. My expectation was to disallow mixed sources. | 18:09 |
| JayF | disallow is ambiguous | 18:12 |
| JayF | disallow can mean ignore them as invalid | 18:12 |
| JayF | or it can mean actively ert | 18:12 |
| JayF | **err | 18:12 |
| TheJulia | brraaaains | 18:12 |
| dtantsur | blow up with "driver_info[kernel] missing" | 18:12 |
| TheJulia | Am I allowed to declare irc bankruptcy for the morning? | 18:21 |
| dtantsur | I shall permit that | 18:22 |
| TheJulia | \o/ | 18:22 |
| dtantsur | JayF: my IRC is not stable, so in case the response did not come through: blow up with "driver_info[kernel] missing" | 18:23 |
| JayF | when I change the redfish to that behavior it's going to make a lot of people maybe breaky | 18:23 |
| dtantsur | I'd not guess "a lot of", but if you're concerned, we can go through deprecation.. | 18:24 |
| * TheJulia wonders if there is a tl;dr of the dsicussion | 18:24 | |
| JayF | 10:15:48 those like 5 lines and the pr | 18:24 |
| TheJulia | so no molten cores?! | 18:25 |
| * TheJulia likes the magnetosphere.... | 18:25 | |
| TheJulia | ahh, I wouldn't expect disjointed-ness | 18:26 |
| TheJulia | but yeah, I could see someone doing it | 18:26 |
| TheJulia | seems like the total exception of use and not the rule | 18:27 |
| JayF | I might make a flag to enable/disable behavior just with a warning | 18:27 |
| JayF | to allow people to not be instabroke | 18:27 |
| TheJulia | Yeah, we tend not to be ones to instrabreak folks/things | 18:28 |
| TheJulia | Instabreaks should be reserved for Insta-Wow, the "break randomly at any time in most subtle and not so subtle ways" project | 18:29 |
| TheJulia | Wait, the Sham-Wow folks might still exist, that would be awful then | 18:31 |
| * TheJulia expects at least one person will get the reference | 18:34 | |
| JayF | you're going to love my high quality config validation | 18:36 |
| JayF | TheJulia: how crazy do we need to get with cheroot benchmarking? you think node lists are sufficient, or do we need some creates too | 18:40 |
| TheJulia | I think lists are sufficient | 18:40 |
| JayF | tl;dr from early testing: it's a negligible difference that's not really statistically significant; if it is, it's in favor of the new code at least for longer queries | 18:40 |
| TheJulia | Writes to the DB are just stupidly resource locking | 18:41 |
| JayF | I think there's a little bit more overhead, but the larger the limit the faster it gets | 18:41 |
| TheJulia | yeah | 18:41 |
| TheJulia | I think you just covered the base "lets do something" sanity check just so we can make sure we aren't taking anything too hard | 18:41 |
| JayF | for 1000 notes, ~1.7s for cheroot, ~1.8 for eventlet | 18:41 |
| JayF | at 10k nodes, cheroot wins by about a tenth more | 18:42 |
| JayF | for single gets it's neck and neck with cheroot more likely to lose | 18:42 |
| JayF | so I think we're seeing overhead of creating a thread and little else | 18:42 |
| TheJulia | I wonder what will happen once the monkey patch disappears | 18:42 |
| JayF | that will be the real pain | 18:44 |
| JayF | but for now, I am +2 to moving forward | 18:44 |
| TheJulia | cool cool | 18:46 |
| JayF | well, except the minor comments outstanding | 18:47 |
| opendevreview | Harald Jensås proposed openstack/networking-baremetal master: Clean up baremetal agents on node delete https://review.opendev.org/c/openstack/networking-baremetal/+/945500 | 19:29 |
| opendevreview | Clif Houck proposed openstack/ironic master: Report provisioned state where appropiate when raising InvalidState* Exceptions https://review.opendev.org/c/openstack/ironic/+/952300 | 19:40 |
| TheJulia | JayF: Looks like I'll be taking a look tomorrow, sort of reaching the end of my spoons earlier than I expected | 20:25 |
| JayF | no big deal | 20:25 |
| TheJulia | iurygregory: a tomorrow question: any progress with idrac10? | 20:25 |
| opendevreview | Verification of a change to openstack/ironic master failed: trivial: revise text from mutlinode job changes https://review.opendev.org/c/openstack/ironic/+/951766 | 20:38 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/sushy-tools master: Add PATCH support for Redfish DateTime fields in Manager resource https://review.opendev.org/c/openstack/sushy-tools/+/950925 | 20:43 |
| opendevreview | cid proposed openstack/ironic master: [WIP] Eventlet: Migrate API & JSON-RPC to cheroot https://review.opendev.org/c/openstack/ironic/+/951054 | 23:38 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!