| stevebaker[m] | TheJulia: ok, I've got a series of 3 reviews which are hopefully uncontroversial. Starting here https://review.opendev.org/c/openstack/diskimage-builder/+/824647 | 01:26 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/ironic-python-agent-builder stable/wallaby: Preliminary work to support CentOS 9 Stream https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823365 | 01:39 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent-builder stable/wallaby: [DNM] testing dib-centos9 experimental job https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/824136 | 01:40 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent-builder stable/wallaby: Preliminary work to support CentOS 9 Stream https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823365 | 03:45 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent-builder stable/wallaby: Replace genisoimage with xorriso https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823367 | 03:45 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent-builder stable/wallaby: [DNM] testing dib-centos9 experimental job https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/824136 | 03:45 |
| arne_wiebalck | Good morning, Ironic! | 07:39 |
| *** pmannidi is now known as pmannidi|brb | 07:39 | |
| janders | hey arne_wiebalck o/ | 09:32 |
| janders | Happy Friday | 09:32 |
| arne_wiebalck | hey janders o/; | 09:36 |
| dtantsur | morning ironic. finally Friday? :) | 10:09 |
| janders | dtantsur yes! | 10:11 |
| arne_wiebalck | dtantsur: mgoddard: I just tried https://review.opendev.org/c/openstack/ironic/+/823723 and it breaks cs8 on software RAID as suspected | 11:08 |
| arne_wiebalck | dtantsur: mgoddard: not sure if other cs8 WDIs would work, but ours does if Ironic ignores the bootloader installation error | 11:10 |
| arne_wiebalck | dtantsur: mgoddard: the actual error is that the cs8 grub2-install does not have secureboot support and then raises an error | 11:12 |
| mgoddard | arne_wiebalck: thanks for confirming. So what's the path forward? Revert then make the check more strict? | 11:15 |
| arne_wiebalck | dtantsur: mgoddard: options I see include: a) move to efibootmgr for s/w RAID, b) make the behavior configurable, e.g. whether or not to ignore errors or have a list of errors to ignore, c) revert :) | 11:16 |
| arne_wiebalck | ordered by preference, inversely ordered by complexity :-D | 11:17 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Update idrac-redfish export configuration step https://review.opendev.org/c/openstack/ironic/+/816816 | 11:18 |
| mgoddard | arne_wiebalck: I have time available to revert :D | 11:18 |
| arne_wiebalck | mgoddard: :-D :-D | 11:19 |
| arne_wiebalck | IIRC, stevebaker[m] was looking at efibootmgr some months ago in a different context and also assessed that it should be possible to move to efibootmgr | 11:19 |
| arne_wiebalck | for s/w RAID | 11:19 |
| arne_wiebalck | and tbh, we do call grub2-install, which fails, but we do not need it anyway ... seems like sth we should fix | 11:20 |
| arne_wiebalck | now, what I do not know if "it works" is b/c of our image or if that would break also others | 11:21 |
| arne_wiebalck | I think mnaser was also using cs8 images on top of s/w RAID | 11:21 |
| dtantsur | I guess the question is how quickly we can fix it | 11:24 |
| dtantsur | we cannot leave master broken for too long | 11:24 |
| dtantsur | if it's going to take time, I'd rather have us ignore the error again for software RAID | 11:24 |
| iurygregory | good morning | 11:24 |
| dtantsur | hey iurygregory, happy Friday | 11:25 |
| iurygregory | happy friday dtantsur o/ | 11:27 |
| * iurygregory hopes he won't need to spend half day in a meeting :D | 11:27 | |
| dtantsur | ouch | 11:27 |
| arne_wiebalck | dtantsur: it'd be great if we had someone else confirm it is a breaking change | 11:27 |
| dtantsur | arne_wiebalck: we don't have a wide choice of people who use software RAID and are ready to test master changes | 11:28 |
| dtantsur | I think we can take your word on it | 11:28 |
| arne_wiebalck | dtantsur: ok | 11:28 |
| arne_wiebalck | dtantsur: I guess we have time until the release before we need to take a revert decision | 11:29 |
| arne_wiebalck | (and do no backports for now) | 11:29 |
| dtantsur | arne_wiebalck: well, we should keep master working | 11:29 |
| dtantsur | people may be consuming a random commit (e.g. if they use bifrost) | 11:30 |
| arne_wiebalck | hmm, ok | 11:30 |
| arne_wiebalck | mgoddard: I guess master before your patch caused issues for you (and triggered the patch) ? | 11:32 |
| mgoddard | it was broken in that a deployment failure was silently ignored | 11:39 |
| mgoddard | so the machine failed to boot | 11:39 |
| dtantsur | I suspect our software RAID code actually relies on that command to fail | 11:40 |
| mgoddard | in my case I was using a non-UEFI image with UEFI enabled | 11:40 |
| dtantsur | aka "command failed successfully" :) | 11:40 |
| arne_wiebalck | or "Errror: SUCCESS" | 11:43 |
| arne_wiebalck | *Error | 11:43 |
| arne_wiebalck | mgoddard: your failed deployment was not with s/w RAID, I assume? | 11:43 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Do not make password files world-readable https://review.opendev.org/c/openstack/bifrost/+/824700 | 11:51 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Tighten permissions for PXE directories https://review.opendev.org/c/openstack/bifrost/+/824144 | 11:56 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Change the TFTP directory to /var/lib/tftpboot https://review.opendev.org/c/openstack/bifrost/+/823552 | 11:58 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Tighten permissions on keystone directories https://review.opendev.org/c/openstack/bifrost/+/824702 | 12:03 |
| mgoddard | arne_wiebalck: correct | 12:42 |
| *** rcastillo|rover is now known as rcastillo | 13:23 | |
| arne_wiebalck | yet another option then would be to error out only when not s/w RAID | 13:26 |
| arne_wiebalck | but we're making the code more and more convoluted | 13:26 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Start Bifrost Architecture documentation https://review.opendev.org/c/openstack/bifrost/+/824719 | 14:17 |
| dtantsur | TheJulia: I hope you'll like this ^^^ | 14:17 |
| dtantsur | this publishes a lot of tribal knowledge | 14:22 |
| TheJulia | I’ll try to look today | 14:30 |
| TheJulia | Woke. Up with migraine | 14:30 |
| dtantsur | ouch. get back to bed, it's Friday :) | 14:32 |
| TheJulia | Still in. Light hurts | 14:32 |
| bkranendonk | hi all, does inspector has some kind of policy engine built in? it keeps denying me from retrieving inspector rules: Failed retrieving Inspector rule 35535433-15bd-5a74-907c-96ecb3113276: ClientError('Access denied by policy') | 14:35 |
| bkranendonk | or is this keystone policy based? | 14:35 |
| bkranendonk | cant find any info on this, and dont have keystone policy.json enabled | 14:36 |
| TheJulia | bkranendonk: policy is embedded into the code | 14:37 |
| TheJulia | bkranendonk what user/rights are you attempting to use ? | 14:37 |
| bkranendonk | user admin, project admin (default admin user created by kolla-ansible) | 14:38 |
| TheJulia | bkranendonk: system admin or member of the bare metal project and an admin in it if memory serves | 14:40 |
| TheJulia | At least, I think | 14:40 |
| TheJulia | The policy is strict since it is an admin only service | 14:41 |
| TheJulia | Give me 5, making coffee | 14:41 |
| bkranendonk | ok, does the policy check on source IPs/cors stuff aswell? | 14:41 |
| bkranendonk | I can however update Node objects properties | 14:42 |
| TheJulia | Coffee https://usercontent.irccloud-cdn.com/file/yfR7Yggi/IMG_0256.JPG | 14:44 |
| TheJulia | yeouch, monitors are bright | 14:46 |
| TheJulia | bkranendonk: no, it does not use cors or ips | 14:46 |
| TheJulia | ironic's policy is a little loser if memory serves due to backwards compatability | 14:46 |
| TheJulia | but lets see, what just inspector has | 14:46 |
| bkranendonk | alright. strange thing is that inspector debug shows me that the received os_auth data is all null :P | 14:48 |
| bkranendonk | enforce: rule="introspection" creds= (...)project_domain_id": null, "project_id": null,(...) | 14:48 |
| bkranendonk | might be doing something wrong at the client/kolla/ansible side, will check | 14:49 |
| TheJulia | yeah | 14:50 |
| TheJulia | I wonder if it is failing to lookup the ID | 14:50 |
| TheJulia | so, by default for the new RBAC model it is https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/policy.py#L38 and until it is enforced by default https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/policy.py#L61 is the rule | 14:51 |
| TheJulia | it falls back to | 14:51 |
| TheJulia | wow, migraine is so bad complete sentences are not a thing at the moment | 14:51 |
| TheJulia | I would check the config for inspector to make sure it is valid for talking to keystone to validate tokens | 14:52 |
| bkranendonk | thanks! i however found the issue to be an ansible module that is not ingesting the os_auth | 14:53 |
| bkranendonk | so it just tries to auth with all null types, well yeah; thats not gonna work :) | 14:53 |
| bkranendonk | TheJulia: thanks for your help again, much appreciated | 14:54 |
| TheJulia | bkranendonk: no problem, hopefully there is a patch someplace now :) | 15:00 |
| TheJulia | or will be soon! | 15:00 |
| arne_wiebalck | TheJulia: do you remember with which release the new default policy was introduced? | 15:01 |
| TheJulia | Wallaby, however the enforcement of the new default has not been turned on yet | 15:14 |
| TheJulia | and likely won't be for a while | 15:14 |
| TheJulia | since TC wants everyone to move at the same time | 15:15 |
| TheJulia | Which makes sense, since it is such a massive change | 15:15 |
| arne_wiebalck | since I just moved to Wallaby, the Puppet module created a policy file but I don't think it contains the default you pointed to above | 15:16 |
| TheJulia | I greatly dislike the puppet module | 15:17 |
| arne_wiebalck | heh | 15:19 |
| arne_wiebalck | there is also json vs yaml, but I guess the format change is independent from the content change? | 15:21 |
| TheJulia | it should all be yaml at this point | 15:22 |
| TheJulia | but the puppet maintainers have hard resistance to *any* change | 15:23 |
| TheJulia | without it being opt-in | 15:23 |
| arne_wiebalck | the wallaby module created a json file in /etc/ironic-inspector/policy.json ... (need to check if our config demands this somewhere) | 15:24 |
| arne_wiebalck | yep, it does | 15:25 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Tighten permissions on keystone directories https://review.opendev.org/c/openstack/bifrost/+/824702 | 15:29 |
| TheJulia | yeah, they should be creating json and actually they shouldn't *need* to create a policy file at all. | 15:37 |
| dtantsur | dear uwsgi, y u h8 me | 15:52 |
| TheJulia | dtantsur: what now? | 15:53 |
| TheJulia | and is it eventlet? | 15:54 |
| dtantsur | trying to make it run from the right user in bifrost | 15:54 |
| dtantsur | currently keystone runs from the nginx user, not from keystone | 15:54 |
| TheJulia | common group perhaps? | 15:54 |
| dtantsur | yeah, but I'm trying to avoid nginx being able to read keystone configuration | 15:56 |
| TheJulia | ahh | 15:57 |
| dtantsur | it's quite a bloody mess | 15:57 |
| TheJulia | the only way then is to actually proxy to a separate process. I think | 15:57 |
| dtantsur | to be able to drop privileged, uwsgi needs to be started as root | 15:57 |
| dtantsur | but then /run/uwsgi is created as root 0700 | 15:57 |
| dtantsur | and uwsgi cannot create sockets in it | 15:58 |
| dtantsur | meh | 15:58 |
| dtantsur | I'll figure it out | 15:58 |
| TheJulia | you could pre-create the socket | 16:00 |
| dtantsur | or even just the directory | 16:00 |
| TheJulia | its just a fifo or lifo buffer right? | 16:00 |
| TheJulia | well, really just fifo | 16:00 |
| dtantsur | actually, I can simplify the heck out of everything now that we don't have keystone-admin | 16:01 |
| dtantsur | but then it won't be backportable | 16:01 |
| TheJulia | I think the last time I was creating a buffer for a process it was for mongodb's local connections or something whacky like that | 16:02 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Tighten permissions on keystone directories https://review.opendev.org/c/openstack/bifrost/+/824702 | 16:07 |
| dtantsur | okay, trying this | 16:07 |
| * TheJulia would like the migraine to finish going away | 16:08 | |
| TheJulia | any ironic-cores around, other than dtantsur ? | 16:43 |
| JayF | You caught me | 16:45 |
| JayF | what's up? | 16:45 |
| TheJulia | I would <3 to get another review on https://review.opendev.org/c/openstack/ironic-inspector/+/824643 | 16:47 |
| JayF | I'll look depending on how simple; but I rarely voted on inspector stuff when this was my day job | 16:47 |
| TheJulia | since I need to backport it as well to fix issues we're seeing with haproxy being slightly more evil than apache but not as evil as eventlet | 16:48 |
| * TheJulia noticed the eventlet fix failed CI :( | 16:48 | |
| TheJulia | (eventlet fix as in fix against eventlet itself) | 16:48 |
| JayF | yeah, I read the chatter about this earlier | 16:49 |
| JayF | want me to land it? | 16:49 |
| TheJulia | sure | 16:49 |
| TheJulia | much appreciated | 16:49 |
| JayF | alright, code review fairy visit done | 16:49 |
| JayF | lol | 16:49 |
| TheJulia | much appreciated, thanks | 16:49 |
| * TheJulia goes back to reviewing dib patches | 16:50 | |
| iurygregory | TheJulia, I'm | 16:50 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Do not run ironic-prometheus-exporter as root https://review.opendev.org/c/openstack/bifrost/+/824735 | 16:50 |
| dtantsur | iurygregory: FYI ^^^ | 16:50 |
| iurygregory | Thanks JayF =) | 16:50 |
| iurygregory | dtantsur, ack | 16:51 |
| dtantsur | iurygregory: and I could use your opinion on https://github.com/metal3-io/ironic-image/pull/344 because httpd is such a pita... | 16:52 |
| iurygregory | dtantsur, sure! | 16:53 |
| dtantsur | I *think* I tested it well. I think. | 16:53 |
| iurygregory | do we have something that explain the scenario for reverse proxy? so I can get a better understanding =) | 16:55 |
| dtantsur | iurygregory: I would like to the TLS termination on httpd (because eventlet is pain) | 16:55 |
| dtantsur | so I'm doing the same thing that we've been doing with inspector already: | 16:55 |
| dtantsur | user -> https://httpd:6385 -> http://ironic:6388 | 16:56 |
| dtantsur | metal3 also delegates basic auth to httpd, which I'm personally not 100% fond of, but anyway | 16:56 |
| iurygregory | got it | 16:56 |
| iurygregory | let me just grab a quick lunch and I will review =D | 16:56 |
| dtantsur | btw https://review.opendev.org/c/openstack/bifrost/+/819640 does a similar thing with nginx | 16:57 |
| arne_wiebalck | bye everyone, have a good weekend o/ | 17:23 |
| opendevreview | Merged openstack/ironic master: Use driver_internal_info methods for other drivers https://review.opendev.org/c/openstack/ironic/+/818509 | 17:35 |
| opendevreview | Verification of a change to openstack/ironic bugfix/18.1 failed: Trivial: log current state when continuing cleaning https://review.opendev.org/c/openstack/ironic/+/820614 | 17:35 |
| dtantsur | have a great weekend folks! | 17:45 |
| TheJulia | o/ | 17:55 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/wallaby: Trivial: log current state when continuing cleaning https://review.opendev.org/c/openstack/ironic/+/824753 | 18:42 |
| opendevreview | Verification of a change to openstack/ironic bugfix/19.0 failed: Fix Redfish RAID deploy steps https://review.opendev.org/c/openstack/ironic/+/824425 | 18:45 |
| * TheJulia goes and takes more migraine meds | 19:10 | |
| opendevreview | Merged openstack/ironic-inspector master: Return a content-length on HTTP204 to prevent client failures https://review.opendev.org/c/openstack/ironic-inspector/+/824643 | 19:47 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector stable/xena: Return a content-length on HTTP204 to prevent client failures https://review.opendev.org/c/openstack/ironic-inspector/+/824754 | 20:00 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector stable/wallaby: Return a content-length on HTTP204 to prevent client failures https://review.opendev.org/c/openstack/ironic-inspector/+/824755 | 20:00 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector bugfix/10.9: Return a content-length on HTTP204 to prevent client failures https://review.opendev.org/c/openstack/ironic-inspector/+/824756 | 20:01 |
| opendevreview | Merged openstack/ironic bugfix/18.1: Use stable/xena upper-constraints https://review.opendev.org/c/openstack/ironic/+/824451 | 20:04 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: Remove rootwrap rule for dnsmasq systemctl https://review.opendev.org/c/openstack/ironic-inspector/+/822373 | 20:23 |
| opendevreview | Verification of a change to openstack/ironic master failed: Do not fail inspection on invalid MAC https://review.opendev.org/c/openstack/ironic/+/824523 | 21:55 |
| opendevreview | Merged openstack/ironic master: Fix validating input for redfish update_firmware https://review.opendev.org/c/openstack/ironic/+/823701 | 23:05 |
| opendevreview | Merged openstack/ironic master: Automatically configure enabled_***_interfaces https://review.opendev.org/c/openstack/ironic/+/820909 | 23:05 |
| opendevreview | Verification of a change to openstack/ironic master failed: Do not fail inspection on invalid MAC https://review.opendev.org/c/openstack/ironic/+/824523 | 23:09 |
| opendevreview | Merged openstack/ironic-inspector master: Remove rootwrap rule for dnsmasq systemctl https://review.opendev.org/c/openstack/ironic-inspector/+/822373 | 23:50 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Migrates docs from wiki https://review.opendev.org/c/openstack/ironic/+/824808 | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
