| opendevreview | Jacob Anders proposed openstack/sushy-tools master: [WIP] Translate VM name to UUID to avoid redirect issue https://review.opendev.org/c/openstack/sushy-tools/+/924297 | 05:43 |
|---|---|---|
| rpittau | good morning ironic! o/ | 06:42 |
| rpittau | TheJulia: have you seen the comment https://github.com/lextudio/pysnmp/issues/49#issuecomment-2232419149 ? | 07:18 |
| rpittau | I'm quite puzzled :/ | 07:18 |
| frickler | rpittau: https://pypi.org/project/pysnmp/#history , guess you'd need to switch to using 5.1.0 from there for now | 07:25 |
| rpittau | frickler: thanks, we've already moved back to 5.x after the initial breakage | 07:27 |
| frickler | rpittau: iiuc currently you are using pysnmp-lextudio, which is unsupported as of yesterday. pysnmp still has 4.x in u-c | 07:39 |
| rpittau | frickler: I'maware of the move, they simply removed the -lextudio suffix as they took over the pyusnmp package | 07:43 |
| rpittau | pysnmp-lextudio still exists up to 6.2.1, in u-c we reverted to 5.0.33 after the upgrade to 6.x broke us, so we're okish for now | 07:43 |
| rpittau | the problem comes with Python 3.12 as removes asyncore | 07:43 |
| rpittau | there was a thread last week from TheJulia highlighting the issue with SNMP support in ironic related to that | 07:43 |
| sylvr | Good morning #ironic ! | 08:00 |
| sylvr | I got a new error in ironic-inspector.log : https://bugs.launchpad.net/ironic/+bug/2072550 | 12:12 |
| arne_wiebalck | Good morning, Ironic! | 12:51 |
| arne_wiebalck | Our summary blog post on the bare metal SIG and OpenInfra meetup at CERN went live: https://techblog.web.cern.ch/techblog/post/openinfra-user-group-meeting-2024/ | 12:52 |
| rpittau | that's really cool, thanks arne_wiebalck :) | 12:58 |
| arne_wiebalck | all creds to luis_fdez ! | 12:59 |
| luisete | it took a while...but finally there :-) | 13:00 |
| rpittau | then thanks to luisete :) | 13:00 |
| adam-metal3 | hello Ironic, I am working on a customHardwareManager for IPA and I was wondering that how open would you (the upstream) be for a mechanism that would allow the developer of a customHardwareManager to overwrite the configDrive injection? It could be a very simple implementation (ofc I would implement it), something like a config option that would thell the "prepare_image" function to not bother running the config | 13:26 |
| adam-metal3 | drive partition creation and file injection | 13:26 |
| adam-metal3 | I think because of how the image handlig is done right now if I want to do some minor customization of the config drive handling but nothing else, I would still need to write a hardware manager and use customDeploy interface and re-implement the default hardware manager's whole image handling logic. | 13:30 |
| adam-metal3 | Instead of this my goal would be to have a tiny condition and a config variable in the upstream code e.g. "custom_config_drive_handling" if it is true then the default functions e.g. prepareImage would just not bother with the config drive, I would take care of that in a automated step in the custom hardware manager | 13:32 |
| sylvr | TheJulia : hello ! | 13:47 |
| JayF | adam-metal3: I suspect if you get to that point, you might just want to use the custom agent driver and set up all your deploy steps manually. | 14:20 |
| TheJulia | sylvr: seems like we might have broken something with the sqlalchemy migraiton | 14:22 |
| sylvr | TheJulia : is there anything I can do to help or any patch you want me to try ? | 14:23 |
| sylvr | and thank you again for your help and insight ! | 14:23 |
| sylvr | TheJulia : https://review.opendev.org/c/openstack/ironic/+/919641 are those the changes in question? | 14:29 |
| TheJulia | sylvr: no that is no the right change | 14:31 |
| sylvr | yup, saw the diff and figured it ou after | 14:31 |
| TheJulia | best guess, this is where https://review.opendev.org/c/openstack/ironic-inspector/+/860731 | 14:33 |
| TheJulia | most likely we need to check that code path around where that is raised, it is obviously not fatal and epected | 14:34 |
| sylvr | well, I'm going to try and dig in the patchset, but I'm no sure I'll understand much... | 14:36 |
| TheJulia | yeah, I suspect we might be treating the no row found as "too fatal" or suggesteive of something else | 14:38 |
| sylvr | db/api.py or db/migration.py ? | 14:42 |
| sylvr | TheJulia : do you want me to mark the bug as confirmed on ironic-inspector and the link you provided ? | 14:51 |
| sylvr | I'm leaving for the day. TheJulia : if you want me to try a patch, I'll keep my eyes on the bug report and I'll update there :) thank you again | 15:00 |
| adam-metal3 | JayF: but that is the thing I don't want to change everything, I would like to build on the defaults and just customize 1 small part of the default workflow and writing a custom deploy step hat just reimplements the existing defaults is just feels like a waste | 15:24 |
| JayF | What's your use case? | 15:31 |
| adam-metal3 | Encryption | 15:31 |
| adam-metal3 | and some partition downshifting | 15:32 |
| adam-metal3 | I mean disk encryption | 15:32 |
| JayF | I can't see how *only* customizing config drive helps you get there? | 15:34 |
| JayF | Plus I'm wondering if this (integration/support for disk encryption) is a feature, not a place for a knob | 15:34 |
| JayF | I think we're learning there can be a steep price to pay for having so many knobs to turn. | 15:34 |
| adam-metal3 | It is the whole configdrive partition creation that would be customized, it would be encrypted, and I would re-encrypt part of the stuff that I streamed to the disk also but that can be handled in the customHardwareManager. I would have proposed configDrive disk encryption as upstream feature (I am also happy to implement it, my use case is LUKS based that is quite common) abut I thought the smaller footprint | 15:40 |
| adam-metal3 | proposal is to just simply make the default ConfigDrive creation optional. | 15:40 |
| JayF | So basically, you're ending up with a LUKS encrypted root and configdrive | 15:43 |
| JayF | decrypted on boot by help of your image | 15:43 |
| JayF | I'm assuming key in the TPM or similar? | 15:43 |
| adam-metal3 | yes TPM | 15:43 |
| adam-metal3 | yep that is the end result I am aiming for | 15:44 |
| JayF | I think if we were able to support that upstream, it'd be an awesome feature and more useful than just making configdrive pluggable | 15:44 |
| JayF | but if you want to do either, the right path to start is an RFE bug. I can see positives and negatives in multiple directions and so more perspectives would help | 15:44 |
| adam-metal3 | yeah sure I will make an RFE for this then, I am working on this anyways | 15:46 |
| rpittau | good night! o/ | 16:02 |
| TheJulia | Hey folks, I'm super heads down at least this week and all of next. Any help you can render for surbhi will be greatly appreciated. surbhi, if you would take a moment to introduce yourself, that would be awesome. Folks here are super friendly. | 16:24 |
| surbhi | Hello everyone, I have provisioned the baremetal node with ubuntu22.04, Installed devstack with irnoic. I think I successfully did it. After installation when I was trying to check I was able to see openstack service list with the output yesterday. I logged in again today and whenevr I am trying to access any command I see . stack@MOC-R4PAC10U31-S3B:~/devstack$ openstack service list | 16:25 |
| surbhi | Internal Server Error (HTTP 500) | 16:25 |
| surbhi | stack@MOC-R4PAC10U31-S3B:~/devstack$ | 16:25 |
| TheJulia | That operation is entirely with keystone, I wonder if the wsgi app for keystone is running? Could apache have been restarted through log rotation and broken it?! | 16:26 |
| JayF | surbhi: TheJulia: apache restarts via log rotation have broken devstacks for cid and adamcarthur5 | 16:30 |
| JayF | in the past | 16:30 |
| surbhi | Do you know how this can be fixed? | 16:33 |
| JayF | I know this isn't a helpful answer for the place you are; but usually for me when a devstack becomes nonfunctional the fastest path to recovery is spinning a new one. | 16:34 |
| TheJulia | ++ | 16:34 |
| TheJulia | It is *not* meant for a long running deployment | 16:34 |
| JayF | I will say we've seen some VPS providers modify their image to include things like log rotation | 16:35 |
| JayF | we had endless troubles getting a devstack to last more than a day on a digitalocean droplet | 16:35 |
| JayF | we thought it was logrotation or autoupdates, but never really put a clear finger on it | 16:35 |
| cid | ++ | 16:37 |
| surbhi | I am trying to build a development environment to test our project and everything for developers so, we will have to reinstall devstack everytime will the set environment will not last for long. | 16:38 |
| cid | Facing something similar now. | 16:38 |
| cid | Though the experience of not working after installation (overnight), I think using the setup immediately after helps | 16:39 |
| surbhi | sudo systemctl status devstack@keystone.service ----------when I did this service is running but ----------Jul 17 16:22:56 MOC-R4PAC10U31-S3B devstack@keystone.service[57736]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_packet | 16:39 |
| surbhi | Jul 17 16:22:56 MOC-R4PAC10U31-S3B devstack@keystone.service[57736]: ERROR keystone packet.raise_for_error() | 16:39 |
| surbhi | Jul 17 16:22:56 MOC-R4PAC10U31-S3B devstack@keystone.service[57736]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 219, in raise_for_error | 16:39 |
| surbhi | Jul 17 16:22:56 MOC-R4PAC10U31-S3B devstack@keystone.service[57736]: ERROR keystone err.raise_mysql_exception(self._data) | 16:39 |
| surbhi | Jul 17 16:22:56 MOC- | 16:39 |
| cid | Not sure why that should be the case, but I can remember that working out. | 16:40 |
| TheJulia | is mysql running? | 16:40 |
| surbhi | stack@MOC-R4PAC10U31-S3B:~/devstack$ sudo systemctl status mysql | 16:41 |
| surbhi | ● mysql.service - MySQL Community Server | 16:41 |
| surbhi | Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) | 16:41 |
| surbhi | Active: active (running) since Wed 2024-07-17 14:12:13 UTC; 4s ago | 16:41 |
| surbhi | Process: 258768 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) | 16:41 |
| surbhi | Main PID: 258776 (mysqld) | 16:41 |
| surbhi | Status: "Server is operational" | 16:41 |
| surbhi | Tasks: 39 (limit: 154429) | 16:41 |
| surbhi | Memory: 358.8M | 16:41 |
| surbhi | CPU: 1.027s | 16:41 |
| surbhi | CGroup: /system.slice/mysql.service | 16:41 |
| surbhi | └─258776 /usr/sbin/mysqld | 16:41 |
| surbhi | Jul 17 14:12:12 MOC-R4PAC10U31-S3B systemd[1]: Starting MySQL Community Server... | 16:42 |
| surbhi | Jul 17 14:12:13 MOC-R4PAC10U31-S3B systemd[1]: Started MySQL Community Server. | 16:42 |
| JayF | please use https://paste.opendev.org/ for output in the future | 16:42 |
| surbhi | Can you look into it here - https://paste.opendev.org/show/bdbLKdhjaUuiYXM7YNbk/ | 16:45 |
| JayF | Like I said, even when this has happened to me, on a devstack I had access to, it wasn't something I found the time to get to the bottom of. I don't think I'll be able to do it here, with output in pastes either :( | 16:46 |
| cid | Services are running according to the logs. Right now, having faced this a few times, the best solution is usually a new instance. | 16:48 |
| cid | And if you decide to do that, as a last attempt, ./clean.sh and ./stach.sh | 16:50 |
| surbhi | Last time I did that. I used unstack.sh clean and then resinstall but I just need to know will it happen frequently if yes, Do I have to follow the same step everytime. I want to aware of things. | 16:52 |
| JayF | The thing that's always kept me confused is when I have a local VM, it doesn't behave this way. | 16:54 |
| JayF | But I see it happen frequently with hosted VPS-style VMs | 16:54 |
| JayF | e.g. the devstack I used with cid last week to go over runbooks is still working properly | 16:55 |
| JayF | so there's something external that's breaking the devstacks, and I've not been able to nail it down when I've looked at the ones that he had that were misbehaving | 16:55 |
| JayF | time to science and see if an apt update breaks me in the same way yours are breaking, perhaps? | 16:56 |
| surbhi | Okay. Thanks a lot for all the information. It was very helpful, I will try to see if I can figure this out if yes will let you all know. | 16:57 |
| JayF | yep, there's a libc security update, and it's restarting basically all the services including devstack, I bet this could break stuff | 16:57 |
| JayF | ...but it all still works :( : | 16:57 |
| opendevreview | cid proposed openstack/ironic master: Address TODO: set to the same value as in [pxe] after Xena https://review.opendev.org/c/openstack/ironic/+/924349 | 17:09 |
| frickler | that's really weird, my devstack VMs tend to run for weeks w/o any issue | 17:12 |
| JayF | that's why I'm saying I think it's an external issue on some hosts | 17:13 |
| JayF | only one I've seen it reliably happen on is DO droplets | 17:13 |
| surbhikanthed | frickler: I will make and share the documnets with the list of steps I followed let me know if there is anything that I am missing while setting up the env. | 17:14 |
| frickler | surbhikanthed: so are you actually running on digitalocean or where is your VM located? | 17:16 |
| surbhikanthed | I have provisioned a baremetal node with ubuntu22.04 and running over there. If you want to login you can share your ssh key and i can give you access to it. | 17:26 |
| surbhikanthed | Frickler: https://docs.google.com/document/d/1NnhAL3vhyCIqlPDVw692QFgHg_M4CRXBMwSPzru4E1I/edit?usp=sharing | 17:56 |
| JayF | TIL Our API happily lets you specify the UUID when creating many objects (including node) | 18:12 |
| mnasiadka | Good evening Ironic | 18:29 |
| mnasiadka | Any chance to have https://review.opendev.org/c/openstack/networking-generic-switch/+/907551 reviewed? We're using that in a couple of envs already, but would be nice to get it in ;-) | 18:30 |
| TheJulia | JayF: indeed! Useful for backup and restore via the API | 18:31 |
| JayF | masghar: Hey, is there any outstanding work items in Inspector migration? | 18:49 |
| JayF | Any ideas on https://review.opendev.org/c/openstack/ironic-lib/+/908119 would be helpful | 19:36 |
| JayF | tl;dr a couple of unit tests are failing because we expect strings, but bytes are returned from the library ... but only in python 3.6 | 19:36 |
| cid | I'm still seeking opinions/feedback on the best direction for the bios patch set, if any's got a minute or two. | 19:57 |
| cid | Specifically about how the configuration should look and behave. | 19:57 |
| cid | https://review.opendev.org/c/openstack/ironic/+/922243 | 19:57 |
| cid | JayF already left some feedback | 19:57 |
| cid | o/ | 19:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!