| iurygregory | snmp-pxe job seems unhappy today... a lot of failures | 00:57 |
|---|---|---|
| iurygregory | https://zuul.opendev.org/t/openstack/builds?job_name=ironic-tempest-ramdisk-bios-snmp-pxe&project=openstack/ironic | 00:57 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 02:08 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 02:20 |
| opendevreview | Kaifeng Wang proposed openstack/sushy master: PCIeDevice and PCIeFuntion https://review.opendev.org/c/openstack/sushy/+/947047 | 05:24 |
| opendevreview | Kaifeng Wang proposed openstack/sushy master: PCIeDevice and PCIeFuntion https://review.opendev.org/c/openstack/sushy/+/947047 | 05:28 |
| opendevreview | Verification of a change to openstack/ironic master failed: Clear `last_error` on power match/sync https://review.opendev.org/c/openstack/ironic/+/955432 | 06:50 |
| rpittau | TheJulia: thanks for checking! I was lost :/ | 06:56 |
| rpittau | good morning ironic! happy friday! o/ | 06:57 |
| opendevreview | Merged openstack/ironic-python-agent master: Hint at sector sizes when reporting an invalid written image https://review.opendev.org/c/openstack/ironic-python-agent/+/954498 | 07:10 |
| queensly[m] | Good morning o/ | 08:02 |
| abongale | Good morning Ironic o/ | 08:22 |
| masghar | Thanks for taking a look at the grenade job, TheJulia | 08:41 |
| mumesan[m] | Good morning! | 10:50 |
| FreemanBoss[m] | o/ | 10:50 |
| iurygregory | I'm wondering if neutron broke us "neutron-metadata-agent: no process found" :eyes: | 11:42 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/186edb85a98c4e2bb06c615e8bece800/log/controller/logs/screen-q-ovn-metadata-agent.txt this doesn't look good :D | 11:43 |
| opendevreview | Mithun Krishnan Umesan proposed openstack/networking-generic-switch master: Autogenerate list of NGS compatible devices https://review.opendev.org/c/openstack/networking-generic-switch/+/955798 | 12:02 |
| iurygregory | I saw some ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Permission denied) in the logs, wondering if is a neutron side issue or if we need to fix something in how we call ovs... | 12:22 |
| rpittau | iurygregory: I think this is a question for neutron directly, either way | 12:39 |
| rpittau | they can probably at least clarify the metadata agent behavior | 12:39 |
| TheJulia | regarding the neutron job, the l3 agent is failing and ultimately the upgrade test fails on verification of l3 traffic movement | 13:01 |
| TheJulia | err, grenade job | 13:02 |
| * TheJulia starts the coffee sipping | 13:02 | |
| iurygregory | rpittau, agree, I've asked in their channel already | 13:07 |
| TheJulia | https://review.opendev.org/c/openstack/neutron/+/945497 <-- here's our problem :\ | 13:19 |
| iurygregory | so, do we need to bump versions on our side? | 13:19 |
| iurygregory | this is the grenade problem right? | 13:19 |
| TheJulia | no, re-consulting the code | 13:20 |
| TheJulia | so, at a glance it looks like neutron's goes sideways when 1.4 is requested | 13:21 |
| rpittau | yay | 13:23 |
| iurygregory | <insert it's fine gif> | 13:28 |
| TheJulia | https://review.opendev.org/c/openstack/neutron/+/945497/8/neutron/agent/rpc.py#105 | 13:30 |
| TheJulia | I added a comment | 13:30 |
| TheJulia | I think we need to basically add logic to downgrade the version | 13:31 |
| TheJulia | and or just revert | 13:31 |
| TheJulia | but, this is a super long standing bug | 13:31 |
| TheJulia | so if memory serves, grenade intentionally puts neutron in a half upgraded state | 13:33 |
| TheJulia | and we're seeing the sign of that | 13:33 |
| TheJulia | lets shift the discussion over to neutron, if there is a open bug someplace then should be relatively easy to write a fix | 13:34 |
| iurygregory | TheJulia, I'm going to push a patch to move the snmp job to non-voting | 14:06 |
| TheJulia | iurygregory: I'm already working on one for it and the grenade job | 14:06 |
| iurygregory | oh ok | 14:07 |
| TheJulia | since fixing it won't be quick | 14:07 |
| iurygregory | happy to review when you finish | 14:07 |
| TheJulia | okay, shouldn't be much longer, I'm doing the needful paperwork w/r/t neutron and grenade | 14:12 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 14:24 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 14:26 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 14:29 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 14:35 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 14:38 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: mark failing jobs non-voting pending fixes/removal https://review.opendev.org/c/openstack/ironic/+/955916 | 15:07 |
| TheJulia | iurygregory: ^^^ | 15:10 |
| TheJulia | took a little while since it doesn't seem like an actual code fix to neutron is needed but our job config is out of date | 15:10 |
| iurygregory | TheJulia, ack | 15:14 |
| iurygregory | going to grab lunch, will review when I'm back | 15:14 |
| TheJulia | ok | 15:14 |
| opendevreview | Nahian Pathan proposed openstack/ironic master: Reduce API calls when collecting sensor data with redfish https://review.opendev.org/c/openstack/ironic/+/955484 | 15:15 |
| cardoe | TheJulia: do we need to patch neutron to fix the RPC issue? | 15:20 |
| TheJulia | no | 15:20 |
| TheJulia | the logs show the upgrade sequence is being violated | 15:21 |
| TheJulia | i.e. rpc stuffs not getting restarted before end rpc services | 15:21 |
| cardoe | All of a sudden I see the stream of convo in neutron now | 15:21 |
| TheJulia | end services are getting restarted which is why it is blowing up | 15:21 |
| opendevreview | Nahian Pathan proposed openstack/sushy master: Support expanded Chassis and Storage for redfish https://review.opendev.org/c/openstack/sushy/+/955211 | 15:33 |
| cardoe | So need some ideas here with the sushy mocking in ironic | 16:53 |
| cardoe | So I'm trying to avoid duplicating code and I'd really also like to take out as much mocking as possible cause I swear everytime I fix a bug it's cause someone wrote the mock and passed the tests but the mock doesn't operate like the actual code/interface. | 16:53 |
| TheJulia | oh yes, that is totally a thing | 16:54 |
| cardoe | So we mock get_members() for Sushy but we also mock the summary property. The summary property is something we put in sushy that just calls get_members() and does some stuff. | 16:54 |
| TheJulia | it would be easier to look at it, fwiw | 16:55 |
| cardoe | The issue is that if I try to use real sushy objects we've got this cache_it() thing in Sushy which interferes with the tests. The cached value is returned back in some cases instead of changed data. | 16:55 |
| cardoe | https://review.opendev.org/c/openstack/ironic/+/955536 | 16:55 |
| cardoe | First file... test_inspect.py | 16:55 |
| cardoe | I'm gonna duplicate and clean it up so it works. | 16:57 |
| cardoe | But wondering if we should move away from using summary in Ironic | 16:57 |
| TheJulia | but because we're mocking parts where there is still underlying sushy code we're calling, we're getting odd results | 16:58 |
| TheJulia | because we're actually then testing sushy in ironic | 16:58 |
| TheJulia | not testing ironic with sushy | 16:58 |
| cardoe | Or if we should do something like running a sushy emulator with real data and letting the tests hit a real redfish endpoint | 16:58 |
| TheJulia | eh, no | 16:58 |
| TheJulia | thats just a lot of complexity | 16:58 |
| cardoe | yeah | 16:58 |
| TheJulia | and then what are we *really* testing | 16:58 |
| cardoe | Well that's a great question | 16:59 |
| cardoe | We're trying to test that the "shape" of the data we upload in the inspector is the right shape. | 16:59 |
| cardoe | Which goes back to my other idea around the shape. | 17:01 |
| cardoe | If we used json schema and validated it, all these tests don't matter. | 17:02 |
| TheJulia | we're testing that the inspect method does what we expect with certian data and cases we get from the driver, the issue is likely the shape of our sushy interactions are wrong for testing, or we're testing at a too high level when we need to decompose it | 17:02 |
| TheJulia | ugh | 17:05 |
| TheJulia | init_system_mock wow | 17:05 |
| TheJulia | hmm | 17:06 |
| iurygregory | omg CI is busted? POST_FAILURE | 17:17 |
| cardoe | So yeah TheJulia you see what I mean... we're not really testing anything good. But the question is... what do we want to test? | 19:18 |
| cardoe | I think that nailing down what inspection data should look like would go a long way. | 19:19 |
| cardoe | I mean I just saw a patch from John Garbutt that points out a field we're not correctly populating. | 19:19 |
| TheJulia | Well, each driver and usage is going to do what is felt to be right. That is going to vary by driver and it should. It’s like the redfish driver should test/represent based upon what it knows. The challenge is it’s being done as a giant monolithic thing and likely need to be decomposed into smaller/distinct chunks | 20:20 |
| TheJulia | Get data, do needful to data, save/commit data | 20:20 |
| TheJulia | regarding our work to eradicate eventlet: Please review this blog post https://github.com/OpenStackweb/ironic-website/pull/62/files | 20:40 |
| opendevreview | Merged openstack/ironic master: Drop remaining reference to openSUSE https://review.opendev.org/c/openstack/ironic/+/955581 | 20:57 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!