| arne_wiebalck | Good morning, Ironic! | 07:26 |
|---|---|---|
| rpittau | good morning ironic! o/ | 08:05 |
| iurygregory | good morning Ironic | 11:19 |
| opendevreview | Merged openstack/sushy stable/xena: Retry on iDRAC SYS518 errors for all requests https://review.opendev.org/c/openstack/sushy/+/868778 | 11:25 |
| opendevreview | Merged openstack/sushy stable/wallaby: Retry on iDRAC SYS518 errors for all requests https://review.opendev.org/c/openstack/sushy/+/868779 | 11:25 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/xena: Fix volume deletion on newer iDRACs https://review.opendev.org/c/openstack/sushy/+/869313 | 11:32 |
| rpittau | so futures does not compile on py3.10 , and we can't move jobs to jammy because we depend on virtualpdu that depends on futures (still uses python 2) | 12:04 |
| rpittau | is virtualpdu under our umbrella ? | 12:04 |
| rpittau | probably not, development stopped roughly 3-4 years ago | 12:07 |
| rpittau | anyway you can see the failure here https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b05/869052/7/check/ironic-tempest-wholedisk-bios-snmp-pxe/b0558b5/job-output.txt | 12:07 |
| opendevreview | Riccardo Pittau proposed openstack/sushy master: Remove setuptools workaround https://review.opendev.org/c/openstack/sushy/+/869971 | 12:23 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/wallaby: Fix volume deletion on newer iDRACs https://review.opendev.org/c/openstack/sushy/+/869934 | 12:34 |
| opendevreview | Denys Mishchenko proposed openstack/sushy-tools master: Save previous hd boot device when setting Pxe https://review.opendev.org/c/openstack/sushy-tools/+/869379 | 12:51 |
| opendevreview | kamlesh chauvhan proposed openstack/ironic stable/zed: Fix "'NoneType' object is not iterable" in RAID https://review.opendev.org/c/openstack/ironic/+/869935 | 12:52 |
| iurygregory | rpittau, virtualpdu is not under our umbrella afaik =) | 12:52 |
| opendevreview | kamlesh chauvhan proposed openstack/ironic stable/yoga: Fix "'NoneType' object is not iterable" in RAID https://review.opendev.org/c/openstack/ironic/+/869936 | 13:02 |
| kubajj | hello everybody | 13:21 |
| kubajj | dtantsur: | 13:22 |
| kubajj | dtantsur: after the api change, I would like to come back to the swift issue you pointed out. If I recall correctly, it was that the data does not get deleted when the node does? | 13:23 |
| stephenfin | TheJulia: You should be able to revert the other changes made in https://review.opendev.org/c/openstack/ironic-inspector/+/869198/ now. I fixed that bug in tox also https://github.com/tox-dev/tox/pull/2849 | 13:23 |
| dtantsur | kubajj: correct | 13:23 |
| kubajj | dtantsur: is there anything else that would be missing from the changes I did? How difficult do you think it would be to allow fetching introspection data from Redfish or iDRAC (I think you mentioned this at some point). | 13:26 |
| dtantsur | kubajj: it's not hard as in "complicated architecture", but it will require you to do some intensive reading on Redfish :) how much time do you have left on the project? | 13:28 |
| kubajj | dtantsur: I need to submit on the 24th of March, but by then I need to write up ~ 40 pages and also do some evaluation (do you reckon I could run some performance comparisons or there would not be any difference - I think you mentioned that having inspector and ironic separate had impact on metal3?) | 13:30 |
| dtantsur | kubajj: it does have an impact, but so far we didn't get rid of inspector completely, so we cannot measure that. what else could you do for evaluation? | 13:31 |
| kubajj | dtantsur: so in short, I think I have 2-4 weeks of development if I don't want to stress about it | 13:31 |
| kubajj | dtantsur: my supervisor suggested that I could do a quantitative one and ask some of you questions about what I did | 13:32 |
| dtantsur | okay. the next step would be to extract the generic code to store introspection data and put it outside of inspector.py (for other drivers to use) | 13:32 |
| dtantsur | (I'll be happy to answer any questions) | 13:32 |
| dtantsur | kubajj: also (and this is totally my fault), we have placed https://opendev.org/openstack/ironic/src/branch/master/ironic/conf/inspector.py#L42-L52 in the wrong section. It should be generic for all inspection methods. | 13:33 |
| dtantsur | we'll need to move them with deprecation | 13:33 |
| kubajj | dtantsur: should I move it in a dedicated follow-up or can I do it with the deletion of data on node deletion | 13:35 |
| kubajj | dtantsur: I guess a dedicated one would be more clean | 13:35 |
| dtantsur | agreed | 13:36 |
| kubajj | dtantsur: about the next step, this is still on the ironic side, right? just extract the methods I created from ironic/drivers/modules/inspector.py right? | 13:38 |
| dtantsur | yep. just so that other drivers don't need to import inspector.py. I think we have a suitable utils module | 13:39 |
| dtantsur | yeah https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/inspect_utils.py | 13:40 |
| dtantsur | kubajj: ^^ | 13:40 |
| kubajj | dtantsur: also, where should I move the CONF parameters to? I assume some new file for all the inspection methods? | 13:40 |
| dtantsur | let me think | 13:41 |
| dtantsur | kubajj: my first reaction is to great [inspection], but I wonder if people are going to confuse it with [inspector] | 13:41 |
| dtantsur | we also have [conductor], which is a garbage bin for all options we never cared to sort better :D | 13:42 |
| dtantsur | kubajj: [inventory] is a good candidate, what do you think? then we can also shorted the option names. | 13:45 |
| kubajj | dtantsur: yeah, sounds great | 13:46 |
| kubajj | dtantsur: could you please have a look at https://review.opendev.org/c/openstack/ironic/+/866876 once you have a minute as I will need to change all the occurrences of the parameter in there which would be a lot easier if it was merged | 13:47 |
| dtantsur | right | 13:49 |
| opendevreview | Riccardo Pittau proposed openstack/sushy master: Remove setuptools workaround https://review.opendev.org/c/openstack/sushy/+/869971 | 13:53 |
| dtantsur | kubajj: left comments. I've spotted an issue with API versions and also marked the places to move around in your next patch. | 13:55 |
| rpittau | so this should bring virtualpdu into Python 3 era and let us use jammy -> https://review.opendev.org/c/x/virtualpdu/+/869983 | 13:57 |
| rpittau | probably needs more work | 13:57 |
| dtantsur | I wonder who has +2 there | 13:58 |
| rpittau | I'm trying to find out | 13:58 |
| kubajj | dtantsur: should I change all of that in the same patch or do some of it in a follow up? I guess I can just do all of it (including the move to [inventory] there, right?) | 13:58 |
| dtantsur | rpittau: maybe mgoddard | 13:58 |
| rpittau | roup virtualpdu-core | 13:58 |
| dtantsur | kubajj: no, let's do a follow-up at this point | 13:58 |
| dtantsur | the API version fix is very small, will be easy to re-review | 13:58 |
| dtantsur | otherwise we're risking to get stuck with it for longer | 13:59 |
| kubajj | dtantsur: Is the API version fix the comment on #1978? | 13:59 |
| dtantsur | kubajj: together with #2056 | 14:00 |
| kubajj | dtantsur: Oh, so I should remove the error (the whole else block) and the add the check behind allow_node_history. On it now | 14:01 |
| dtantsur | right | 14:01 |
| kubajj | dtantsur: any clue why it would return 200 for the old version test after I changed it? It should check that the minor version is >= 1.81 right? | 14:15 |
| dtantsur | kubajj: I suspect you may move creation of NodeInventoryController to _subcontroller_map | 14:17 |
| dtantsur | maybe otherwise lookup does not happen | 14:17 |
| kubajj | dtantsur: I see. That might be the problem | 14:18 |
| dtantsur | (so, from line 2006 to line 2033) | 14:18 |
| mgoddard | dtantsur, rpittau: no +2 for me there | 14:19 |
| dtantsur | hmmmmm | 14:19 |
| rpittau | dtantsur, mgoddard, I found the group | 14:19 |
| rpittau | https://review.opendev.org/admin/groups/66d40bfe0feebc21c0edccea9b495cfff3f5960a,members | 14:19 |
| mgoddard | should it be under ironic project governance? | 14:19 |
| dtantsur | we could check with them | 14:20 |
| rpittau | I can send an email to them, at least to explain the situation | 14:20 |
| dtantsur | ++ | 14:24 |
| * TheJulia tries to wake up | 14:32 | |
| TheJulia | I don't remember getting any responses from the virtualpdu folks when I reached out when I was PTL :( | 14:32 |
| kubajj | dtantsur: do I need to tell it somewhere that inventory is valid once it is in the _subcontroller_map? I am getting loads of 400 errors | 14:32 |
| dtantsur | kubajj: I don't think so.. could you show me the diff? | 14:33 |
| kubajj | dtantsur: https://paste.opendev.org/show/bVd7CKpwuoiPo0n3R34Q/ | 14:34 |
| * TheJulia tries to wake up | 14:35 | |
| dtantsur | kubajj: it's a bit suspicious that you removed 'else', but not 'if' | 14:35 |
| dtantsur | that's the only thing I can spot | 14:35 |
| dtantsur | are there any error messages? | 14:36 |
| kubajj | https://www.irccloud.com/pastebin/Xeymx7we/ | 14:37 |
| kubajj | even after I remove the if | 14:37 |
| * dtantsur blinks | 14:37 | |
| dtantsur | kubajj: TheJulia has a suggestion on line 1859, maybe you're missing it? | 14:38 |
| TheJulia | something written using the subcontroller pattern? | 14:38 |
| dtantsur | I have.. opinions... about our subcontrollers :) | 14:39 |
| dtantsur | TheJulia: it's about the inventory API and passing node_ident through | 14:39 |
| kubajj | dtantsur: same errors after I added it | 14:41 |
| kubajj | I remember this was an issue when I was implementing it. | 14:43 |
| kubajj | dtantsur: could I not call the not api_utils.allow_node_inventory() from the controller itself? Might solve these problems. But would still need the error message there. | 14:44 |
| rpittau | need to split, I'm out tomorrow, see you on monday! o/ | 14:55 |
| JayF | dtantsur: rpittau: I'll look into virtualpdu from a governance standpoint and see if I can get anywhere | 15:00 |
| opendevreview | Vincent proposed openstack/bifrost master: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/870017 | 15:26 |
| opendevreview | Vincent proposed openstack/bifrost master: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/870017 | 15:32 |
| JayF | TheJulia: iurygregory: So, once I remove Aija from sushy-core, there are only two remaining members; the only one of whom is active is iurygregory | 16:41 |
| JayF | Do we anticipate continuing to need/desire a separate sushy-core group? | 16:41 |
| JayF | because sushy-core will just be identical to ironic-core | 16:42 |
| dtantsur | I would love to.. but I don't anticipate | 16:42 |
| JayF | yeah; that's sorta where I land | 16:42 |
| JayF | I'm going to make sushy-core into a group only inherited from ironic-core; it's likely easier to leave it that way than to remove the separate group for now | 16:42 |
| JayF | but it might be valuable to do a pass and consolidate; I think we have more complexity in our number of core groups than we need given the number and nature of contribution now | 16:43 |
| JayF | running ironic-inspector bugfix branch cleanups now | 16:44 |
| TheJulia | That aligns with what we've generally done, so sounds good to me | 16:45 |
| JayF | complete | 16:46 |
| JayF | bugfix branch cleanup is complete | 16:46 |
| JayF | hopefully we won't be the #1 star in zuul-config-errors anymore lol | 16:46 |
| JayF | inspector and IPE both have active branches that still need ci love | 16:50 |
| JayF | https://etherpad.opendev.org/p/IronicZuulConfErrs is a curated list of the repos we are responsible for that are still misconfigured in zuul | 16:50 |
| TheJulia | at least we can now see the wheat from the chaff | 17:04 |
| JayF | yeah exactly, I am surprised we actually only had that small of a number of broken things still | 17:04 |
| opendevreview | kamlesh chauvhan proposed openstack/ironic stable/xena: Fix "'NoneType' object is not iterable" in RAID https://review.opendev.org/c/openstack/ironic/+/870025 | 17:41 |
| jrosser | does anyone know how to specify the baremetal api version in clouds.yml? | 18:18 |
| NobodyCam | Good Morning OpenStack Folks... Happy New years | 18:18 |
| JayF | NobodyCam: o/ | 18:45 |
| JayF | jrosser: I don't even know what clouds.yml refers to in this context | 18:46 |
| NobodyCam | Morning JayF | 18:46 |
| NobodyCam | \0 | 18:46 |
| jrosser | JayF: like a yaml version of openrc where your credentials go when using the cli or sdk | 18:46 |
| JayF | ah, yeah I haven't used that much sorry | 18:47 |
| jrosser | ansible ironic modules are hard to use :( | 18:47 |
| JayF | there are some docs up about that, pending landing | 18:48 |
| JayF | were those yours (?) | 18:48 |
| * JayF finds the link | 18:48 | |
| jrosser | no that’s for deploying ironic with OSA | 18:48 |
| jrosser | if you want to then use ansible to define all your baremetal nodes post deployment then there are ansible modules for that | 18:49 |
| JayF | oh; you're trying to integrate ironic and ansible in a different way | 18:49 |
| JayF | I don't have specific tips about that, ansible is a giant blind spot for me :( sorry | 18:50 |
| jrosser | but ironic seems to want system_scope:all where no other arrive does (on Y at least) and then there is no version negotiation between sdk and ironic api which leads to very surprising results | 18:50 |
| jrosser | *no other service | 18:50 |
| JayF | what specific API things are you trying to do with ansible+ironic? | 18:50 |
| JayF | we have a system role and project role; the service role is up for review now | 18:51 |
| JayF | but the difference between system:admin and service:admin is extremely minimal aiui | 18:51 |
| jrosser | create my nodes with this for example https://docs.ansible.com/ansible/latest/collections/openstack/cloud/baremetal_node_module.html | 18:51 |
| jrosser | that fails on both token scope and api version as far as we can see | 18:52 |
| JayF | creating nodes will require you to use a system:admin I believe; you could use the service:admin for that in the future | 18:52 |
| JayF | creation/deletion of nodes is one of the most sensitive parts of Ironic's API | 18:52 |
| jrosser | right, but my naive expectation was that the admin token that I can use for everything else would be also good here | 18:53 |
| JayF | TheJulia: you have a sec? You'd know a lot more about the state of RBAC across services in Y to help jrosser | 18:53 |
| TheJulia | o/ | 18:53 |
| JayF | jrosser: I think you're literally running a release dead in the middle of the RBAC work across services, and I think that might be some of the complexity | 18:53 |
| JayF | but I'm not sure | 18:53 |
| TheJulia | how goes it | 18:54 |
| jrosser | so eventually we make a specific system scope token just to use with ironic and now we can at least authenticate | 18:54 |
| TheJulia | okay, so your enforcing scope with yoga? | 18:54 |
| jrosser | but it took a lot of debug to find out what was going on | 18:54 |
| jrosser | I am not sure | 18:55 |
| TheJulia | if it is complaining about wanting system scope, then you are most likely | 18:55 |
| jrosser | the rbac thing is pretty opaque to me as an operator / deployer | 18:55 |
| TheJulia | ahh | 18:55 |
| TheJulia | okay | 18:55 |
| TheJulia | so lets see | 18:55 |
| jrosser | even as core on OSA I have almost no clue about it :( | 18:55 |
| JayF | jrosser: I think "no clue about RBAC" is the default state for most of us ;) | 18:56 |
| jrosser | anyway, Z looked different again in the policy definition | 18:56 |
| TheJulia | so, ironic as a traditionally admin scoped service, we initially went in the direction relying upon the system scope which was for the direct management of the system, originally | 18:56 |
| TheJulia | that since changed | 18:56 |
| TheJulia | yeah, it has evolved some, z has manager role as well | 18:57 |
| jrosser | I think now I know what kind of token I need I’m happy with that and I’lol write docs for OSA about it | 18:57 |
| jrosser | but the api version I don’t know what to do about | 18:57 |
| TheJulia | what do you mean? | 18:57 |
| jrosser | when we use the openstack.cloud.baremetal_node ansible module it appears to use an early api version | 18:58 |
| jrosser | for example moving the node directly to available | 18:58 |
| * TheJulia raises a huge eyebrow | 18:58 | |
| jrosser | different state machine behaviour I think | 18:58 |
| * TheJulia pulls up the module code | 18:59 | |
| jrosser | well it’s not just the module, it’s the interaction between sdk and api primarily I think | 18:59 |
| jrosser | the module will be quite thin I think | 18:59 |
| TheJulia | yeah, that sounds like a huge issue | 18:59 |
| TheJulia | what sdk version do you have? | 19:00 |
| jrosser | so I wanted to set the equivalent of OS_BAREMETAL_API_VERSION in the auth parameters to the module | 19:00 |
| jrosser | not sure right now, just travelling but I can look later | 19:01 |
| TheJulia | the sdk has changed a *ton* since the module was originally written | 19:01 |
| jrosser | yes it has | 19:02 |
| jrosser | so maybe another question would be should I expect auto negotiation if I have a new enough sdk? | 19:02 |
| JayF | in context of an ansible module; auto negotiation seems nondesirable | 19:03 |
| JayF | although maybe better than using an API microversion from 2015 :P | 19:04 |
| TheJulia | eh... I need to check the sdk code | 19:04 |
| TheJulia | but one oddball thing about RBAC, is you can't use the api version to try and bypass RBAC | 19:04 |
| jrosser | oh no that’s not what I want to do | 19:04 |
| TheJulia | but... defaulting to old versions, it shouldn't be doing afaik | 19:04 |
| TheJulia | oh, good :) | 19:04 |
| jrosser | just automate what I would otherwise do with the cli | 19:05 |
| TheJulia | ack | 19:05 |
| TheJulia | ahh, sort of like what we do with bifrost | 19:05 |
| JayF | OS_API_VERSION=before_auth # checkmate, security :P | 19:05 |
| * TheJulia momentarilly steps away to try and get into a bios setup over ipmi sol | 19:06 | |
| TheJulia | so yeah, the max version it should auto-negotiate is 1.76 | 19:08 |
| TheJulia | at least with latest sdk | 19:08 |
| TheJulia | same for yoga | 19:08 |
| TheJulia | ugh.... | 19:12 |
| jrosser | hmm I will check what sdk we have later | 19:13 |
| jrosser | it might be <0.99 because it’s been “fluid” beyond that | 19:13 |
| JayF | TheJulia: there's an ipmi command to boot into bios next boot | 19:14 |
| JayF | TheJulia: as with all IPMI commands, you have a .0005% chance it'll do what's on the tin | 19:14 |
| TheJulia | JayF: my half crashed/bricked idrac wouldn't accept it | 19:14 |
| TheJulia | wheeeeeeeeeee | 19:14 |
| TheJulia | jrosser: maybe... looks like the versioning stuff has been heavily rewritten so the sdk maintainers might have decided to take a different path | 19:15 |
| jrosser | is there an easy way I can check what api version I end up with? | 19:15 |
| TheJulia | no, but what exactly are you seeing error/response wise when your trying to use the module? | 19:16 |
| jrosser | well that the state ends up immediately as available | 19:17 |
| jrosser | then the next step to set up ports or whatever all blows up as you can’t do that in available state | 19:17 |
| TheJulia | ahhh | 19:17 |
| jrosser | so translating what we do on the cli to the equivalent set of ansible modules does not end with success | 19:18 |
| jrosser | and debugging what was going on shows an early api version | 19:20 |
| TheJulia | hmm | 19:21 |
| TheJulia | that is ddefinitely an early api version | 19:21 |
| jrosser | we have a similar thing with nova where to create a resource class we have to specify a new enough api version | 19:21 |
| TheJulia | so we may have never seen this in bifrost because we create the nodes with the ports upfront in one command invocation | 19:23 |
| jrosser | with the cli? | 19:23 |
| TheJulia | module | 19:23 |
| jrosser | ah interesting | 19:23 |
| TheJulia | https://github.com/openstack/bifrost/blob/master/playbooks/roles/ironic-enroll-dynamic/tasks/main.yml#L32 | 19:23 |
| TheJulia | it doesn't have the ability to set all of the parameters, but it does create the ports | 19:50 |
| * TheJulia raises eyebrow at server now saying fans have failed | 19:54 | |
| jrosser | TheJulia: it is definitely constrained to openstacksdk<0.99.0\ | 20:01 |
| TheJulia | happier universe with 1.x? | 20:09 |
| jrosser | I will try tomorrow - it’s getting late now | 20:12 |
| jrosser | though 1.x as it’s own issues, and * else works, like images, flavours, users, projects blah blah so I’m hesitant to just use the latest | 20:13 |
| jrosser | plus I think there is coupling between the ansible collection version and sdk version | 20:14 |
| * NobodyCam waves at TheJulia | 20:29 | |
| TheJulia | o/ NobodyCam | 20:29 |
| NobodyCam | :) | 20:30 |
| TheJulia | I *do* remember some weirdness there, but I've not been in the middle of that in a long time | 20:30 |
| jrosser | i right it is worse than i thought https://github.com/openstack/ansible-collections-openstack#branches-and-non-backward-compatibility-%EF%B8%8F | 20:42 |
| TheJulia | https://thumbs.gfycat.com/AdmiredEarlyDegus-mobile.mp4 | 21:41 |
| TheJulia | That is how my brain responds to the sdk right now | 21:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!