| rm_work[m] | bah I wish search worked better in Element... | 02:31 |
|---|---|---|
| rm_work[m] | has there been discussion here that I should find and read regarding https://bugs.launchpad.net/nova/+bug/2106073 ? | 02:32 |
| rm_work[m] | TheJulia: not sure which side of the fence this is going to end up falling on (assuming the nova folks haven't completely washed their hands of it, which they may have), but it is affecting us apparently T_T | 02:33 |
| rm_work[m] | from reading the thread it seems like possibly one of those things that is a major architectural issue, not just "someone go fix it quick" | 02:34 |
| rm_work[m] | but have been trying to think of ways to actually get back to contributing upstream, and maybe could convince folks here to let me spend cycles on this if there's momentum to actually fix it | 02:34 |
| JayF | rm_work[m]: https://review.opendev.org/c/openstack/ironic/+/946378 (+the next one in that chain for a larger fix/improvement) | 02:42 |
| rm_work[m] | ah ok cool, so well in hand | 02:43 |
| rm_work[m] | ... was that linked on there and I just missed it | 02:43 |
| rm_work[m] | almost assuredly yes | 02:43 |
| rm_work[m] | or maybe not. anywho, +2 already, nice | 02:44 |
| rm_work[m] | I wonder if I can get that pulled down and test it in our lab... | 02:45 |
| rm_work[m] | assuming it is ready for testing? I see the followup is still -1 but that could be anything without spending a bit more time diving in | 02:46 |
| rpittau | good morning ironic! o/ | 05:21 |
| AmarachiOrdor[m] | Good Morning rpittau, Good morning everyone | 05:37 |
| arne_wiebalck | Good morning, Ironic! | 06:28 |
| abongale | Good Morning, Ironic! | 08:21 |
| queensly[m] | Good morning | 08:47 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Default ansible to version 10.x https://review.opendev.org/c/openstack/bifrost/+/948245 | 09:26 |
| freemanboss[m] | Good morning everyone | 10:03 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: fix: portgroup create with uuid for idempotency https://review.opendev.org/c/openstack/python-ironicclient/+/949032 | 13:15 |
| TheJulia | good morning | 13:38 |
| TheJulia | rm_work[m]: yeah, I'm working on it since we have customers who do all sorts of spine/leaf configurations which could likely run up against this. It actually explains some of the reports of issues we've not really understood over the past couple of years | 13:39 |
| TheJulia | which is unfortunate, but yeah. Anyhow, it is not a fun issue and I'm really hopeful to have a fully testable fix this week. Unfortunately nova sends Ironic pre-mastered configuration drives which means we really need to unpack/extract the contents, patch, and repackage | 13:41 |
| TheJulia | Wheeeee ironic networking meeting in like 20 | 13:42 |
| TheJulia | err, that is an old meeting, not the new meeting | 13:57 |
| * TheJulia consumes more coffee | 13:57 | |
| cardoe | networking meeting? | 14:12 |
| cardoe | isn't that in an hour and 45? | 14:13 |
| cardoe | So I had a stupid idea with OVN router flavors and creating a generic bare metal flavor as a reference. Not sure if that should live in networking-generic-switch or if I should submit that against Neutron proper. | 14:15 |
| TheJulia | Yeah, it is | 14:37 |
| TheJulia | I had a bad invite on my calendar from ages ago | 14:37 |
| TheJulia | which got me confuzzled | 14:37 |
| shermanm | can I ask where the ironic networking meeting schedule is written down? | 14:39 |
| shermanm | on https://etherpad.opendev.org/p/ironic-networking it still says Jan 15th | 14:39 |
| cardoe | https://etherpad.opendev.org/p/ironic-standalone-networking is the new one I got in the invite. | 14:45 |
| shermanm | cardoe: thanks! I remember now that we'd planned on two "tracks" for networking , standalone and non- | 14:55 |
| TheJulia | shermanm: for integrated cases I think we're going to try and take the specs process to try and effectuate the needful | 15:12 |
| shermanm | thejulia: +1, just making sure I wasn't totally out of the loop :) | 15:14 |
| TheJulia | https://meet.google.com/rct-jvgq-qrw?authuser=0 | 16:00 |
| opendevreview | Elod Illes proposed openstack/bifrost unmaintained/zed: [CI][UM-only] Remove not defined kolla-ansible job https://review.opendev.org/c/openstack/bifrost/+/949057 | 16:51 |
| TheJulia | alegacy: template is https://opendev.org/openstack/ironic-specs/src/branch/master/specs/template.rst | 17:01 |
| cardoe | I hope my feedback helped and didn’t detract. | 17:44 |
| JayF | honestly it's good you brought up what you did | 17:54 |
| JayF | a good check to ensure we're not gobbling up all the scope in the universe | 17:54 |
| cardoe | I just see some of the asks around NGS stuff like making it an agent that knows the state of the switches, etc. | 18:04 |
| cardoe | And my thought is that there's open source tools for this written by people that mess with those devices day in and day out. | 18:05 |
| mnasiadka | Hello | 18:05 |
| mnasiadka | TheJulia: Is there an option to get some re-review on this IPA patch https://review.opendev.org/c/openstack/ironic-python-agent/+/816685 ? | 18:05 |
| cardoe | The better approach would likely be to tell people to run one of those things and then make it easy to get the requested operation fired off to that system. | 18:05 |
| TheJulia | cardoe: I think you were making a really good point, thanks! | 18:06 |
| alegacy | cardoe: ya, I do like the idea of being able to point a driver to an existing automation system. Can you remind me of the ones you mentioned? I was scribbling down notes as you were talking... I got autobot, dev42, netbox... correct me if I'm in the weeds. | 18:14 |
| opendevreview | Merged openstack/ironic master: Add an index on ports.node_id https://review.opendev.org/c/openstack/ironic/+/948431 | 18:18 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Revert "Add an index on ports.node_id" https://review.opendev.org/c/openstack/ironic/+/949068 | 18:36 |
| JayF | cid and I were doing more performance testing | 18:37 |
| JayF | and we are now reasonably convinced that the preexisting CONSTRAINT `ports_ibfk_1` FOREIGN KEY (`node_id`) REFERENCES `nodes` (`id`) is providing an index somehow? | 18:37 |
| JayF | I may not fully understand all the moving parts; but a "for node in nodes; dbapi.get_port_by_node_id(node.id)" actually gets slightly slower with the index | 18:37 |
| JayF | and an explain pre-running migration seems to show that we're arleady using an index | 18:38 |
| JayF | TheJulia: dtantsur: ^ sanity check? | 18:38 |
| TheJulia | whenever you apply a constraint, it requires the db to maintain an index | 18:38 |
| JayF | yeah | 18:39 |
| JayF | so the index is a noop/dupe | 18:39 |
| JayF | and the revert should be landed :( | 18:39 |
| TheJulia | So, technically. The key is if your doing any sort of join or ... sorry searching for the word in the context switch | 18:39 |
| TheJulia | join or... query with additional parameters you'll want to match it if possible | 18:39 |
| JayF | oh, I get what you're saying | 18:40 |
| TheJulia | i.e. if your doing "give me the ports which match this node *and* some other value, idealy you want a composite index | 18:40 |
| JayF | if we have a query that gets address/name/extra from port | 18:40 |
| TheJulia | The index, should match what you query | 18:40 |
| TheJulia | well, query to *match* | 18:40 |
| JayF | an index of node_id -> portid/address/name/extra would be valuable | 18:40 |
| JayF | yes? | 18:40 |
| TheJulia | Somewhat, the DB inherently indexes the pkey + any other single column indexed field | 18:41 |
| JayF | is the action for now to revert that simple index the right one though? | 18:41 |
| JayF | even if there may be /an/ index that improves things, this likely isn't it | 18:41 |
| JayF | I feel like for a db migration there's a timer | 18:42 |
| TheJulia | if your doing select from blah where field_a='a_value' and field_b != NULL; then you'll want the index composited on field_a and field_b | 18:42 |
| JayF | yep, I think that matches my understanding | 18:42 |
| JayF | which means a fkey port.node_id -> node.id | 18:42 |
| TheJulia | which is different from select field_a, field_b | 18:42 |
| JayF | and an index port.node_id -> node.id | 18:42 |
| JayF | is the same thing | 18:42 |
| JayF | and we'd need a better index that uses the methods/calls in question | 18:42 |
| TheJulia | well, fkey reference is the direct link, but yeah | 18:42 |
| TheJulia | I guess, I'm talking theory and resulting practice, what is the driving observation? | 18:44 |
| JayF | the driving observation is I can't find a benchmark this improves | 18:44 |
| JayF | or a DB query this changes the explain on | 18:44 |
| JayF | and the explain showing me the fkey/contraint existed pre-index is a red flag | 18:44 |
| JayF | cid: https://opendev.org/openstack/networking-baremetal/src/branch/master/networking_baremetal/agent/ironic_neutron_agent.py#L221 is what the report was timing out at high scale | 18:46 |
| JayF | but it's not even doing a join in the DB, it's just getting all the ports and mapping later | 18:46 |
| JayF | I don't think there's anything we can do DB-side to speed that up | 18:46 |
| TheJulia | Query patterning is also a death by a thousand cuts, The key is to do as much as you can as close to the db and not go back to the db | 18:46 |
| TheJulia | which is kind of the challenge I'm trying to understand | 18:47 |
| dxterslab | stevebaker[m]: do you know of any redfish API that provides a websocket for SOL access? | 18:47 |
| JayF | yeah the tl;dr on the root ask in that ironic_client.get_ports(detail=True) call is timing out in networking-baremetal in some cases | 18:47 |
| JayF | but looking at the query, I don't think we can do crap on the ironic side to speed it up | 18:47 |
| TheJulia | dxterslab: afaik, no. from what I've seen is some just point at ipmi with the suggestion to take that path instead | 18:47 |
| JayF | it really is a shard/conductor group awareness thing potentially needed | 18:47 |
| TheJulia | so its a joined query thing to do it, right now we're likely checking/reconciling it upfront | 18:48 |
| TheJulia | err | 18:48 |
| TheJulia | after the fact, not upfront | 18:48 |
| dxterslab | TheJulia: yeah... the noVNC approach looks to be non-trivial at all and probably proxying IPMI to websocket will be easier | 18:48 |
| TheJulia | somebody did that, but its really just not ideal and web socketing it is apparently awkward | 18:49 |
| TheJulia | (and they didn't post the code, insert grumpy upstream maintainer comments) | 18:49 |
| TheJulia | Hey, folks... Any thoughs on dialing back logging on keystoneauth.identity.v3.base ? | 18:50 |
| dxterslab | TheJulia: what challenges do you for see that I might not thought about? | 18:51 |
| TheJulia | dxterslab: the comments which were made with me on a call were "this is awkward and painful", but the folks built a third party api/surface to facilitate it separately | 18:52 |
| dxterslab | TheJulia: I guess I get to FAFO then lol | 18:53 |
| TheJulia | sorry :( | 18:53 |
| TheJulia | I like the idea of having a proxy there for it, but $something does need to sit in that middle ground | 18:53 |
| TheJulia | also, apparently ipmitool doesn't have support for the flow control on the ipmi protocol | 18:54 |
| TheJulia | so, super easy to stall out interactions | 18:54 |
| TheJulia | (I know some folks that want to use serial command multiplexing for $reasons) | 18:55 |
| dxterslab | right now i am looking to just support a single session and no multiplexing but vnc has presented to be more challenging than expected | 18:55 |
| TheJulia | understandable | 18:56 |
| dxterslab | TheJulia: where the comments based on ipmitool or pyghmi? i am planning to use pyghmi to do this | 19:06 |
| TheJulia | ipmitool, I'd avoid pyghmi since it is pretty much a single maintainer thing at this point | 19:07 |
| TheJulia | and... really only gets tested/developed against lenovo gear | 19:07 |
| JayF | pyghmi is great 90% of the time and nobody runs it because of the 10% | 19:07 |
| dxterslab | reassuring :D! | 19:08 |
| opendevreview | Merged openstack/ironic master: Trivial: Fix gap in microversion sequence https://review.opendev.org/c/openstack/ironic/+/948829 | 19:13 |
| rm_work[m] | TheJulia: thanks for your work on that! when you're good to go if you want a tester, you can ping me, i'll see if we can pull that down in our Lab env | 20:46 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Inspection throws exception on CPU-less systems https://review.opendev.org/c/openstack/ironic/+/949090 | 21:18 |
| opendevreview | Julia Kreger proposed openstack/ironic master: provide host_id to neutron early on https://review.opendev.org/c/openstack/ironic/+/946378 | 21:23 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Patch configdrive metadata https://review.opendev.org/c/openstack/ironic/+/946677 | 21:23 |
| TheJulia | rm_work[m]: if you want to give the two patches I just posted a spin, by all means. I still need to write a bunch of tests for the latter but the same basic code just seemed to work when I short circuited validation on network_data.json out of nova | 21:24 |
| TheJulia | I just revised the commit message to lay out slight metadata differences, likely also just highlighted *another couple bugs* | 21:33 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Patch configdrive metadata https://review.opendev.org/c/openstack/ironic/+/946677 | 21:33 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Inspection throws exception on CPU-less systems https://review.opendev.org/c/openstack/ironic/+/949090 | 21:33 |
| opendevreview | Merged openstack/ironic-python-agent master: netutils: Use ethtool ioctl to get permanent mac address https://review.opendev.org/c/openstack/ironic-python-agent/+/946562 | 21:53 |
| opendevreview | Merged openstack/ironic master: Revert "Add an index on ports.node_id" https://review.opendev.org/c/openstack/ironic/+/949068 | 22:48 |
| opendevreview | Merged openstack/ironic master: Remove tags from README https://review.opendev.org/c/openstack/ironic/+/948479 | 22:48 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!