| *** dmellado_ is now known as dmellado | 00:55 | |
| vanou | good morning ironic | 01:32 |
|---|---|---|
| opendevreview | Merged openstack/ironic master: Do not disable autocommit until we fully migrate https://review.opendev.org/c/openstack/ironic/+/862476 | 02:21 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: Align iRMC driver with Ironic's default boot_mode https://review.opendev.org/c/openstack/ironic/+/862552 | 05:29 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: Align iRMC driver with Ironic's default boot_mode https://review.opendev.org/c/openstack/ironic/+/862552 | 05:46 |
| arozman | Hello Ironic! | 06:27 |
| vanou | Hi all. Recently, Fujitsu server HW team suddenly introduces multiple incompatible behavior on BMC. 1) iRMC disables IPMI by default 2) it doesn't allow http connection to iRMC on newer version of server anymore. To deal with this I plan to make multiple patches. However my concern is that are such changes approved to backport till Wallaby? Because, maybe customer uses | 06:48 |
| vanou | Wallaby version of Ironic. Of cource, I don't plan introduce new package dependency. If there are tips to make backport feasible, please tell me that. Thanks. | 06:48 |
| kubajj | Good morning Ironic! | 07:33 |
| opendevreview | Radosław Piliszek proposed openstack/tenks master: Skip more ansible-lint style checks https://review.opendev.org/c/openstack/tenks/+/862557 | 08:05 |
| opendevreview | Radosław Piliszek proposed openstack/tenks master: Follow ups to venv https://review.opendev.org/c/openstack/tenks/+/857008 | 08:05 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: Align iRMC driver with Ironic's default boot_mode https://review.opendev.org/c/openstack/ironic/+/862552 | 08:18 |
| dtantsur | JayF: mysql is an overkill for us; file vs non-file has no difference in this conversation | 09:39 |
| dtantsur | JayF: these are not sqlite limitations, these are bugs in our code. not closing transactions is a bug, even if "adult" databases kinda-sorta work around it for us. | 09:40 |
| dtantsur | vanou: oh :( my only concern would be your driver still working for people who do not update the firmware | 09:41 |
| dtantsur | (and disabling IPMI may be a reason a lot of customers will refuse to update) | 09:42 |
| dtantsur | JayF: example: firefox is an application that is an order of magnitude larger and more complex than Ironic. and really multi-thread and multi-process. if sqlite works for them, but does not work for us.. it's not sqlite's fault. | 09:44 |
| opendevreview | Jakub Jelinek proposed openstack/ironic master: WIP: Implements node inventory: database https://review.opendev.org/c/openstack/ironic/+/862569 | 09:44 |
| kubajj | Morning dtantsur, I've created the WIP change for the database I talked about last week. If you had a minute, could we have a chat about it? (I am almost sure that this is not how we want to store it.) | 09:46 |
| dtantsur | looking | 09:46 |
| dtantsur | JayF, TheJulia, what I do suspect is that oslo_db_api.retry_on_deadlock does not know how to retry sqlite exceptions | 09:47 |
| dtantsur | kubajj: I actually think this is the right direction to take. some comments inline. | 09:57 |
| kubajj | dtantsur: thanks, I'll have a look | 09:58 |
| kubajj | dtantsur: you mention that we might not need the get_..._by_id function outside of unit tests. Do you mean that we still need it for unit tests or I can remove it? | 10:31 |
| dtantsur | kubajj: do you need it for unit tests? if not, I'd probably remove it. | 10:32 |
| kubajj | Ok, I'll have a look | 10:32 |
| kubajj | dtantsur: in the comment about object file, did you mean get_by_uuid instead of get_by_node_uuid by any chance? | 10:46 |
| dtantsur | kubajj: inventories don't have a UUID, I did mean the node UUID. But I don't remember if other node-related objects have that, maybe it's done on the API level. | 10:47 |
| kubajj | dtantsur: I was looking at the node history for inspiration and that uses node id | 10:48 |
| kubajj | only deployment has get_by_node_uuid (I do not really know what it does, though) | 10:50 |
| kubajj | I will look into this in the afternoon when I hope to implement some tests | 11:00 |
| opendevreview | Jakub Jelinek proposed openstack/ironic master: WIP: Implements node inventory: database https://review.opendev.org/c/openstack/ironic/+/862569 | 11:06 |
| iurygregory | good morning Ironic | 11:36 |
| vanou | dtantsur: thanks. driver should work with older version of server too. it may detect version of server and switch driver operation based on server version. (IPMI is disabled by default configuration but user can enable it, sorry for confusing you) | 12:01 |
| TheJulia | Good morning | 12:39 |
| TheJulia | dtantsur: you are correct about oslo_db not retrying. I thought I indicated that yesterday. Conundrum is that however the interaction is with oslodb, the more we try to do cross thread things, the more it thrashes. I turned down our retry and it never succeeded in a heartbeat in 5 minutes. I turned it up to 15 and it succeeded 1-2 times a minute locally. Oslo_db or sqlalchemy also doesn't honor pooling for sqlite. Which | 12:49 |
| TheJulia | is why I'm strongly wondering if the right path for metal3 is to just use in-memory mode because even when we close out the interactions, like add literal session.close(), it still throws that error on the heartbeat thread since it doesn't try that until after the time has passed from launch. | 12:49 |
| dtantsur | TheJulia: how is in-memory related? the database is still locked if we don't close transactions. | 12:50 |
| TheJulia | This is really more a case of not bugs as much as we were/are reliant upon autocommit to make file mode work since sqlalchemy has magic ot make it kind of work. | 12:50 |
| dtantsur | we cannot just drop them and hope that mysql does the right thing eventually.. | 12:50 |
| TheJulia | in memory, the constraint on file locking IO across threads shouldn't be a thing. Shouldn't being key of course. | 12:51 |
| dtantsur | we don't have threads | 12:51 |
| TheJulia | okay, fake threads in python | 12:51 |
| dtantsur | yep, which are executed sequentially | 12:51 |
| TheJulia | in theory, but even when we close the transaction out, it still doesn't work. It could be enginefacade is also just sqlite without autocommit incompatible | 12:52 |
| dtantsur | but it's not a sqlite problem. what happens when we don't close transactions on mysql? | 12:52 |
| TheJulia | well, even if we explicitly close it out on each call we do | 12:52 |
| TheJulia | enginefacade has a commit or rollback logic | 12:52 |
| dtantsur | well, neither clearly happen | 12:53 |
| TheJulia | yup | 12:53 |
| dtantsur | which probably means a rollback | 12:53 |
| TheJulia | *or* sqlalchemy cannot support our use model | 12:53 |
| dtantsur | are we fine with e.g. touch_conductor getting rolled back? | 12:53 |
| TheJulia | huh? | 12:53 |
| dtantsur | well, the change is never committed, right? | 12:53 |
| dtantsur | so the conductor is not touched, it will eventually timeout.. boom? | 12:53 |
| TheJulia | it can't even open the db file to run the operation | 12:54 |
| TheJulia | yup, I've watched it happen locally | 12:54 |
| TheJulia | it sometimes works with the default, fwiw | 12:54 |
| TheJulia | but minutes can pass of course | 12:54 |
| dtantsur | the only explanation I have it that some transaction was not closed | 12:54 |
| TheJulia | we could try oslo_db getting retrys working, but I'm worried we're just going to thrash | 12:55 |
| TheJulia | well, it has to close the transaction and the session with the open file, the latter seems not happen | 12:55 |
| dtantsur | this transaction was changing something, and this something is not saved in reality | 12:55 |
| TheJulia | the connections via open files pile up | 12:55 |
| TheJulia | and eventually close out when the conductor goes "I'm not heartbeating" | 12:55 |
| TheJulia | and the cycle repeats | 12:55 |
| TheJulia | it maybe works 10% of the time locally on my code base before I started to force close sessions | 12:56 |
| dtantsur | TheJulia: connection are not a problem, transactions are | 12:56 |
| TheJulia | maybe 20% before then | 12:56 |
| TheJulia | dtantsur: I literally put session.flush(), session.commit(), and session.close() on all the calls before that were working | 12:57 |
| TheJulia | and no change/improvement. Honestly it feels like it has gotten worse. | 12:58 |
| dtantsur | so something is escaping. otherwise autocommit wouldn't fix the issue | 12:58 |
| TheJulia | the next step is to dig into the oslo_db code base | 12:58 |
| dtantsur | to double-check my assumption, I literally tried opening several cursors to the same file from different python processes. all worked fine. | 12:58 |
| TheJulia | *or* oslo_db's enginefacade is breaking us | 12:58 |
| dtantsur | so I cannot even make transactions race, maybe I need to open one explicitly | 12:59 |
| TheJulia | I'm leaning towards enginefacade being the culprit here, fwiw. unfortunately our code style that is used pulls a transaction for read operations | 13:01 |
| TheJulia | the only way to undo that is to rip out model_query | 13:01 |
| TheJulia | *completely* | 13:01 |
| TheJulia | and then... move away from the enginefacade session style | 13:02 |
| TheJulia | at which point we're at complete DB API rewrite territory | 13:02 |
| TheJulia | oh | 13:03 |
| TheJulia | https://github.com/openstack/oslo.db/blob/9119edc86ffa207bdf244e0d21e163c96a06ee8e/oslo_db/options.py#L17 | 13:03 |
| TheJulia | I just noticed that | 13:03 |
| dtantsur | okay, 'database is locked' happens on two parallel write transactions, I could reproduce in shell | 13:03 |
| dtantsur | IIRC sqlite_synchronous is related to whether fsync or not | 13:04 |
| dtantsur | I considered setting it to False (maybe I even did it for metal3) to bring the file closer to :memory: | 13:04 |
| TheJulia | eh, yeah, no difference really | 13:05 |
| TheJulia | it *did* successfully heartbeat the first time | 13:05 |
| TheJulia | and then proceeded to fail 10 seconds later | 13:05 |
| TheJulia | and now my conductor is going offline :) | 13:06 |
| dtantsur | btw since WAL is off by default, even reads conflict with writes | 13:07 |
| TheJulia | yeah, because all reads in the code model are transactions right now | 13:09 |
| TheJulia | added local support for retrying | 13:11 |
| * TheJulia waits | 13:11 | |
| TheJulia | no better at all, fwiw :) | 13:12 |
| dtantsur | if we lose transactions, no retries will help | 13:12 |
| dtantsur | retries are useful to add, but only to avoid occasional failures under load | 13:13 |
| TheJulia | yup | 13:13 |
| TheJulia | it hasn't actually succeeded yet | 13:14 |
| TheJulia | :( | 13:14 |
| TheJulia | yeah, its not going to succeed at even saving a single heartbeat | 13:15 |
| opendevreview | Jacob Anders proposed openstack/sushy master: [WIP] Retry BootSourceOverride request when SettingsURI is read-only https://review.opendev.org/c/openstack/sushy/+/856597 | 13:24 |
| dtantsur | TheJulia: this thing yields very useful results: https://github.com/openstack/oslo.db/blob/9119edc86ffa207bdf244e0d21e163c96a06ee8e/oslo_db/options.py#L102 | 13:26 |
| dtantsur | 2022-10-25 15:26:31.300 676470 INFO sqlalchemy.engine.Engine [-] BEGIN (implicit) | 13:27 |
| dtantsur | 2022-10-25 15:26:31.301 676470 INFO sqlalchemy.engine.Engine [-] BEGIN | 13:27 |
| dtantsur | 2022-10-25 15:26:31.302 676470 INFO sqlalchemy.engine.Engine [-] [cached since 10.46s ago] () | 13:27 |
| dtantsur | 2022-10-25 15:26:31.304 676470 INFO sqlalchemy.engine.Engine [-] UPDATE conductors SET updated_at=?, online=? WHERE conductors.hostname = ? | 13:27 |
| dtantsur | 2022-10-25 15:26:31.304 676470 INFO sqlalchemy.engine.Engine [-] [cached since 10.03s ago] ('2022-10-25 13:26:31.294757', 1, 'dtantsur-laptop') | 13:27 |
| dtantsur | 2022-10-25 15:26:31.307 676470 INFO sqlalchemy.engine.Engine [-] COMMIT | 13:27 |
| dtantsur | I'm quite suspicious about two begins, one being implicit | 13:27 |
| TheJulia | autobegin? | 13:31 |
| TheJulia | dtantsur: heh, I didn't set it that high I guess | 13:31 |
| dtantsur | when I return autocommit, I get a lot of database traffic going on. weird. | 13:34 |
| dtantsur | funnily, if I migrate touch_conductor to use the session "properly" (???), it fails in both cases | 13:35 |
| dtantsur | with autocommit it fails with sqlalchemy.exc.InvalidRequestError: No transaction is begun | 13:35 |
| dtantsur | which is super fishy | 13:35 |
| dtantsur | ah, no, it's register_conductor_hardware_interfaces that fails (probably because I'm still on your change) | 13:35 |
| TheJulia | ahh, I see what is going on now | 13:36 |
| * dtantsur -> lunch, brb | 13:36 | |
| TheJulia | yeah, it is the object handler that i theorized | 13:36 |
| TheJulia | explicit closing doesn't work since it never gets that far it looks like | 13:37 |
| TheJulia | bingo | 13:38 |
| TheJulia | I at least have a heartbeating service now, having commented out allocations from running | 13:38 |
| TheJulia | but it is query pagination | 13:40 |
| TheJulia | gah | 13:40 |
| dtantsur | TheJulia: I've instrumented the enginefacade with prints, and that's what I see on the *previous* (successful) heartbeat: https://paste.opendev.org/show/bNmDKLuYFjb5591XzdH8/ | 13:44 |
| TheJulia | https://paste.opendev.org/show/bQQqN797Ho5ok2CP9zfw/ | 13:44 |
| TheJulia | I *think* the path forward is to make a sqlalchemy 2.0 orm not entirely required paginate_query | 13:46 |
| TheJulia | or at least a "do it the 2.0 query way" | 13:46 |
| TheJulia | so we get a disconnected tuple that won't hold the db open | 13:47 |
| TheJulia | granted, we also likely need a "don't actually paginate flag" for internal calls | 13:47 |
| TheJulia | easy... enough I guess | 13:47 |
| TheJulia | okay, that is already in the logic, limitations is in the api for api clients | 14:00 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: Change boot_interface order of iRMC driver https://review.opendev.org/c/openstack/ironic/+/862616 | 14:03 |
| TheJulia | vanou: So if I understand correctly, the hardware folks have started locking out access to the BMC... except via?!? | 14:07 |
| TheJulia | dtantsur: I already wrapped paginate so just changing how we get the query seems to do the trick | 14:09 |
| JayF | dtantsur: upon overnight reflection, and after reading your conversation in here, I think the piece that I was missing is that we're running ironic in single command mode, right. So we don't have to worry about contention between actual threads just contention between green threads | 14:14 |
| TheJulia | is pypi still broken today? | 14:18 |
| * TheJulia has not looked at CI | 14:19 | |
| TheJulia | I'm converting over the paginated model_query calls on list to use query selects instead which will yield us tuple responses without the attached db connection | 14:19 |
| TheJulia | ... (which should improve db load actually...) | 14:20 |
| JayF | TheJulia: I rechecked that one before I left last night; I can check it | 14:22 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/862476 the autocommit revert piece landed overnight | 14:22 |
| JayF | dtantsur: ^ FYI in case you didn't see | 14:22 |
| JayF | TheJulia: so it at least worked for one patch after like, 4pm PST yeseterday | 14:23 |
| dtantsur | seen it, thank you! | 14:26 |
| dtantsur | JayF: correct, we only have actual threads when the API is run separately under e.g. mod_wsgi | 14:26 |
| JayF | We really have to get that running in local gate; but that's on the list for once Julia fixes the underlying bug | 14:28 |
| dtantsur | JayF: the combined process is used in bifrost (but with mysql) | 14:28 |
| JayF | Yeah; the shock for me was that we used sqlite | 14:29 |
| dtantsur | yeah, it's a metal3 thing because the database is ephemeral and because we're aiming at reducing the footprint | 14:29 |
| JayF | Don't get me wrong, I really like sqlite, but it does make the kinda work Julia is doing 2x as hard now | 14:29 |
| JayF | but at least people run this config vs when we supported postgres and nobody actually ran it lol | 14:30 |
| dtantsur | I'm still not convinced that it's the sqlite's fault rather than something sqlite merely exposes | 14:30 |
| dtantsur | lol yeah | 14:30 |
| JayF | I am not saying sqlite is to blame, per se | 14:30 |
| * dtantsur really likes postgres since his skype days | 14:30 | |
| JayF | I'm saying sqlite has more strict semantics around write locking | 14:30 |
| dtantsur | totally | 14:30 |
| JayF | which exposes "bugs" which wouldn't matter in mysql | 14:30 |
| dtantsur | may or may not matter, depending on what is actually happening | 14:31 |
| dtantsur | e.g. if we don't commit a write transaction (which I guess Julia has ruled out), it's a huge deal | 14:31 |
| dtantsur | an hanging read transaction (which is, as far as I can understand, the current hypothesis) is less of, but still some resources | 14:32 |
| * TheJulia kind of misses some of the friendliness of debugging queries in postgres | 14:32 | |
| TheJulia | well, we would *likely* just rollback, but the transaction creation overhead has been an operator complaint in the past | 14:33 |
| vanou | TheJulia: HW people started to restrict access to BMC a bit. There are 3 way to access BMC in newer server: 1)HTTPS REST API (no http anymore) 2)SNMP 3)IPMI (IPMI over LAN is disabled by default BMC configuration, but server user can change setting of BMC and enable IPMI) | 14:36 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Sqllite fix maybe? https://review.opendev.org/c/openstack/ironic/+/862506 | 14:37 |
| TheJulia | so, there is a huge issue with ^... we have to *explicitly* unique the calls if we want the overhead | 14:38 |
| TheJulia | That being said, our schema enforces it, so the check is... kind of redundant I think, but there may be cases where it makes sense, I've just not looked at the schema. | 14:38 |
| kubajj | dtantsur: TheJulia: what is the difference between id and uuid in the models? | 14:39 |
| TheJulia | vanou: that makes a lot of sense actually. Please say they are supporting DMTF Redfish :) | 14:39 |
| dtantsur | kubajj: id is a sequential number of the internal use in the database | 14:39 |
| dtantsur | kubajj: uuid is a user-visible identifier (where needed) | 14:39 |
| kubajj | Do I need uuid for inventory then? | 14:39 |
| dtantsur | kubajj: unlikely since there is a 1-1 relationship between nodes (and their UUID) and inventories | 14:41 |
| dtantsur | if we even support inventory history (which I think some people may be interested in).... but let's not go there :) | 14:41 |
| kubajj | dtantsur: What does a node have a uuid for? | 14:47 |
| dtantsur | kubajj: UUIDs are used to identify nodes, e.g. in the API | 14:47 |
| dtantsur | GET /v1/nodes/<NODE UUID> | 14:47 |
| dtantsur | (you can use names as well, but names are not required) | 14:47 |
| * dtantsur adds even more prints to enginefacade | 14:50 | |
| TheJulia | dtantsur: take a spin with my patch | 14:51 |
| TheJulia | kubajj: Id is also database assigned (generally) and the database primary key value in the tables | 14:52 |
| dtantsur | TheJulia: it take it a bit longer to fail, and probably because of a different place | 14:56 |
| kubajj | I am just trying to wrap my head around why having get_by_node_uuid would be used, but I can just include it and remove it later, right? | 14:57 |
| dtantsur | TheJulia: okay, now to trigger the failure I have to issue a curl request to create a node | 14:57 |
| vanou | TheJulia: yeah Fujitsu server has supported DMTF Redfish for years :) However there are concerns: 1)BMC doesn't catch up with latest Redfish spec. so FJ needs to check if such gap has bad effect 2)there are vendor specific Redfish path so FJ needs to check each path and start discussion around sushy-oem-* | 14:57 |
| dtantsur | TheJulia: I wonder if we do need to banish model_query from the code completely | 14:58 |
| TheJulia | dtantsur: oh joy :( | 14:58 |
| dtantsur | kubajj: something will need to handle GET /v1/nodes/<NODE UUID>/inventory | 14:58 |
| dtantsur | but maybe it's going to happen on a higher (= API) level | 14:58 |
| TheJulia | vanou: okay, on a plus side we've only had one vendor *really* need to do an oem module, and I think even then it is not required now since they were able to get the BMC engineering team to become compliant | 14:59 |
| TheJulia | at least, that was with virtual media | 14:59 |
| kubajj | dtantsur: Oh, I was focused too much on the database. With the api it makes sense. | 14:59 |
| TheJulia | dtantsur: we do need to unfortunately | 15:00 |
| dtantsur | TheJulia: btw, explicit commits do not seem needed, _session_for_write() handles them | 15:00 |
| dtantsur | (same for close() - I inserted prints to check) | 15:00 |
| TheJulia | dtantsur: I know :) | 15:00 |
| TheJulia | now :) | 15:00 |
| dtantsur | :D | 15:00 |
| TheJulia | I was just paranoid when I put them in | 15:00 |
| dtantsur | learning every day.. | 15:00 |
| TheJulia | yup | 15:01 |
| * dtantsur is curious why create_node blows up.. | 15:01 | |
| TheJulia | it... shouldn't really | 15:04 |
| TheJulia | it should be a fairly atomic operation, but maybe we're doing something else under the hood at the same time | 15:04 |
| vanou | TheJulia: I see. I'll check our server's Redfish implementation. thanks! | 15:05 |
| vanou | good night ironic o/ | 15:06 |
| TheJulia | goodnight! | 15:07 |
| * dtantsur looks at model_query and makes a large UGH | 15:08 | |
| dtantsur | TheJulia: I wonder if the minimal thing would be to replace model_query with just session.query with whatever session the caller has | 15:08 |
| dtantsur | otherwise we're definitely in the double-transaction land.. | 15:08 |
| TheJulia | yeah, don't look at model_query | 15:09 |
| TheJulia | I think the ideal thing to do is to break out the code so we're doing the right hting | 15:09 |
| TheJulia | but I think it will need to be on a case by case basis | 15:10 |
| dtantsur | TheJulia: I've replaced some model_query with session.query (added _session_for_read where needed), and I can create nodes now | 15:14 |
| ajya | TheJulia: in sushy-oem-idrac there are more functionality that is not part of standard Redfish, e.g., reboot idrac, clear job queues - things that were available in wsman. It's true that virtual media was fixed for 14G+ systems. Otherwise OEM extension will stay there for those who want additional functionality, also for RAID ops. | 15:14 |
| opendevreview | Julia Kreger proposed openstack/virtualbmc master: WIP: preserve info across boot device operations https://review.opendev.org/c/openstack/virtualbmc/+/862620 | 15:14 |
| TheJulia | ajya: today I learned! | 15:14 |
| TheJulia | thanks! | 15:14 |
| dtantsur | TheJulia: it's possible that we're going a bit overboard with sa.query even. maybe all we need is session.query, making sure the session is created at the right time and not closed until we're done with it | 15:14 |
| TheJulia | dtantsur: oooh ahhhh | 15:14 |
| TheJulia | dtantsur: yeah, the nested layer of things doesn't really help overall. By chance are you going to upload a rev of the patch with your local changes? | 15:15 |
| dtantsur | TheJulia: https://paste.opendev.org/show/817328/ | 15:15 |
| dtantsur | I can upload that | 15:15 |
| TheJulia | dtantsur: if you could, that would be great | 15:16 |
| TheJulia | I'm getting pulled in two other directions at the moment | 15:16 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: WIP: Sqlite fix maybe? https://review.opendev.org/c/openstack/ironic/+/862506 | 15:17 |
| dtantsur | TheJulia: ^^ | 15:17 |
| dtantsur | I wonder if the register_conductor changes are really needed | 15:18 |
| TheJulia | dunno, I'd almost rather keep since 2.0 is pushing away from query call usage | 15:21 |
| dtantsur | TheJulia: any reference for that? I haven't checked 2.0 yet | 15:22 |
| dtantsur | we could at least consider it a step in the right direction to avoid a lot of rewrites | 15:22 |
| TheJulia | dtantsur: the migration guide talks about how it will keep compatibility but it is basically deprecated to use the base query style that started with orm because they want users to move to the unified query usage/format | 15:23 |
| dtantsur | ack. I still think it may be beneficial to have a 2-stage migration | 15:24 |
| TheJulia | yeah, I didn't want to muck with rewiring everything for obvious reason | 15:26 |
| opendevreview | Julia Kreger proposed openstack/virtualbmc master: WIP: preserve info across boot device operations https://review.opendev.org/c/openstack/virtualbmc/+/862620 | 15:36 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: WIP: Replace more instances of model_query https://review.opendev.org/c/openstack/ironic/+/862506 | 16:01 |
| dtantsur | for the record, I don't know what internal reasons resulted in sqlalchemy 2.0, but it's a huge step back in terms of usability | 16:04 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Replace more instances of model_query https://review.opendev.org/c/openstack/ironic/+/862506 | 16:06 |
| dtantsur | I hope this is enough to get sqlite going ^^ | 16:06 |
| dtantsur | I think, we this propagates to metal3 containers, I'll create a testing patch there that tries removing __autocommit | 16:06 |
| dtantsur | s/we/when/ | 16:06 |
| dtantsur | TheJulia: I wonder if we eventually drop the ORM layer completely and switch to the core layer.. | 16:16 |
| dtantsur | since they're essentially converging, and we don't really use many specifically ORM things | 16:17 |
| opendevreview | Julia Kreger proposed openstack/sushy-tools master: WIP: preserve information on xml https://review.opendev.org/c/openstack/sushy-tools/+/862625 | 16:26 |
| kubajj | If I created a new upgrade of db with alembic, do I need to do anything with it or does ironic just start using it? (I modified the upgrade function already) | 16:27 |
| dtantsur | kubajj: the files are picked up automatically, use unit tests to confirm (you should add a test per migration - see existing ones) | 16:27 |
| kubajj | dtantsur: got it. Will do | 16:32 |
| TheJulia | dtantsur: we *do* in some of our things, but yeah. It would likely be a good (and more performent) idea eventually | 16:35 |
| TheJulia | it hides a lot of complexity around interactions, as evidenced by "register_compute" alone | 16:35 |
| opendevreview | Julia Kreger proposed openstack/sushy-tools master: WIP: preserve information on xml https://review.opendev.org/c/openstack/sushy-tools/+/862625 | 17:17 |
| kubajj | TheJulia: sorry for asking this but I've been stuck on this for more than an hour now. Have you ever seen an error similar to this? https://paste.opendev.org/show/bBn0Gw6F6XPLkuz1FZe3/ | 17:19 |
| opendevreview | Julia Kreger proposed openstack/sushy-tools master: preserve secure information on xml changes https://review.opendev.org/c/openstack/sushy-tools/+/862625 | 17:45 |
| opendevreview | Julia Kreger proposed openstack/virtualbmc master: preserve secure information on xml changes https://review.opendev.org/c/openstack/virtualbmc/+/862620 | 17:51 |
| TheJulia | kubajj: are you handing the unit test a string or a dictionary? | 17:53 |
| TheJulia | based upon the error, I would guess you've got a string with a json document inside of it | 17:53 |
| kubajj | TheJulia: I see, that might actually be the case. Thanks | 18:05 |
| JayF | Is there a core reviewer online right now, other than me an TheJulia? dtantsur? iurygregory? | 18:43 |
| opendevreview | Jay Faulkner proposed openstack/sushy-tools master: preserve secure information on xml changes https://review.opendev.org/c/openstack/sushy-tools/+/862625 | 18:50 |
| opendevreview | Jay Faulkner proposed openstack/virtualbmc master: preserve secure information on xml changes https://review.opendev.org/c/openstack/virtualbmc/+/862620 | 18:50 |
| iurygregory | JayF, i'm | 18:58 |
| JayF | iurygregory: thanks, I got ruby to help me approve the patches we need to get in | 18:58 |
| JayF | iurygregory: thank you for responding to the call though :D | 18:58 |
| * JayF puts away the Iurysignal | 18:58 | |
| iurygregory | np o/ | 18:59 |
| TheJulia | JayF: wow, reno used to be more kind | 19:04 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/xena: CI: Make swift/Vmedia job non-voting https://review.opendev.org/c/openstack/ironic-python-agent/+/861454 | 19:21 |
| * TheJulia declears "The cult of the baremetal" to be a required thing. | 19:32 | |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: Fix software raid output poisoning https://review.opendev.org/c/openstack/ironic-python-agent/+/857772 | 20:00 |
| opendevreview | David Hill proposed openstack/virtualbmc master: Add flags libvirt.VIR_DOMAIN_XML_SECURE to tree https://review.opendev.org/c/openstack/virtualbmc/+/862638 | 20:19 |
| * TheJulia sighs | 20:21 | |
| opendevreview | Merged openstack/sushy-tools master: preserve secure information on xml changes https://review.opendev.org/c/openstack/sushy-tools/+/862625 | 20:35 |
| opendevreview | Merged openstack/virtualbmc master: preserve secure information on xml changes https://review.opendev.org/c/openstack/virtualbmc/+/862620 | 20:48 |
| stevebaker[m] | JayF: the failure for https://review.opendev.org/c/openstack/ironic-lib/+/860175 it looks like the filename layout for centos-8-stream has changed https://zuul.opendev.org/t/openstack/build/06fbe60c0cf94102bf02a1ed65bacd5b/log/job-output.txt#28513 | 20:50 |
| stevebaker[m] | https://cloud.centos.org/centos/8-stream/x86_64/images/ | 20:51 |
| JayF | stevebaker[m]: if you could fix it you'd be a hero | 20:51 |
| JayF | I'll send you a grab-envelope full of stickers lol | 20:51 |
| stevebaker[m] | yup looking now | 20:51 |
| JayF | thank you \o/ | 20:55 |
| stevebaker[m] | JayF: actually I think the problem is the job is installing diskimage-builder-3.2.1, latest dib is 3.25.0. Its a branchless project and the latest should always be installed | 21:01 |
| JayF | interesting | 21:01 |
| ashinclouds[m] | Hmm it is pinned for some reason in a few places | 21:01 |
| stevebaker[m] | 3.2.1 is in requirements stable/victoria upper constraints | 21:05 |
| stevebaker[m] | both stable/train and stable/ussuri have much newer diskimage-builder in upper-constraints | 21:07 |
| JayF | I wonder if this is an IPA-B constraint being pushed down into victoria (?) | 21:10 |
| JayF | there was not anything in ussuri committed to bump the version | 21:10 |
| stevebaker[m] | diskimage-builder upper-constraints are a real roller-coaster https://paste.openstack.org/show/bfyRl4fm6JwuUqw9qVKX/ | 21:18 |
| JayF | that makes me think the previous statement isn't true? | 21:19 |
| JayF | about T/U having newer DIB than V? | 21:19 |
| stevebaker[m] | ussuri upper-constraints.txt diskimage-builder===2.35.0 | 21:21 |
| stevebaker[m] | victoria upper-constraints.txt diskimage-builder===3.2.1 | 21:21 |
| stevebaker[m] | wallaby upper-constraints.txt diskimage-builder===3.20.3 | 21:21 |
| stevebaker[m] | lol, I didn't notice major number went to 2 | 21:22 |
| JayF | yeah, I figured, these things sorta start blending together | 21:22 |
| JayF | I had to look 3-4 times to make sure I didn't do the reverse | 21:22 |
| stevebaker[m] | dib has some but minimal pip requirements, I wonder how aggressive we could be raising uppper-constraints in stable requirements. Since its branchless we should in theory be running the latest release always in all branches | 21:25 |
| stevebaker[m] | https://opendev.org/openstack/diskimage-builder/src/branch/master/requirements.txt | 21:25 |
| stevebaker[m] | I think V onwards could run the latest dib, and T, U can run the version before python2.7 was dropped | 21:31 |
| JayF | yeah it's weird to me, and IDK if I'm thinking about this right | 21:33 |
| JayF | but it's strange that DIB version is tied to stable branch at all | 21:33 |
| JayF | because it's not always meant to be co-installed | 21:33 |
| JayF | but that being said; also most people running Ussuri Ironic will be running Ussuri-adjacent dib | 21:33 |
| stevebaker[m] | sometimes yeah. Not always though | 21:36 |
| stevebaker[m] | JayF: OK, 3.4.0 changed the centos base image to the new style. This means any job using dib 3.3.1 or earlier won't be able to pull a centos-8-stream image | 21:40 |
| stevebaker[m] | (possibly other centos versions too | 21:41 |
| stevebaker[m] | dib 3.0.0 dropped py27 support | 21:41 |
| JayF | I wonder if DIB is branched | 21:41 |
| JayF | if we can push 3.3.2 with the new path or something ? | 21:41 |
| stevebaker[m] | it is deliberately not branched | 21:41 |
| JayF | I do not know the solution to this problem | 21:42 |
| JayF | if I were in your shoes, I'd probably hit the mailing list and see if there's any wisdom | 21:42 |
| JayF | we can't be the first project to run against this (I hope :X) | 21:42 |
| stevebaker[m] | We could try just running latest dib from V onwards. I think T, U would need to drop jobs that involve building centos images | 21:45 |
| stevebaker[m] | or evolve to a hard-coded image or something | 21:45 |
| JayF | Yeah; none of that sounds ideal | 21:47 |
| JayF | but we can drop the jobs for centos if that's one of the options | 21:47 |
| stevebaker[m] | I'll propose the change back to victoria and see what happens | 21:51 |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/train failed: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-python-agent/+/860190 | 21:57 |
| opendevreview | Steve Baker proposed openstack/ironic-lib stable/victoria: CI: Zuul no longer respects queue param https://review.opendev.org/c/openstack/ironic-lib/+/860175 | 22:05 |
| stevebaker[m] | lets see what a Depends-On to https://review.opendev.org/c/openstack/requirements/+/862651 does | 22:05 |
| stevebaker[m] | JayF: I think this needs to be backported to train too https://review.opendev.org/c/openstack/ironic-python-agent/+/845371 because of https://zuul.opendev.org/t/openstack/build/2f91b1f1fb8a450bb69f90e128a6737a/log/job-output.txt#460 | 22:22 |
| stevebaker[m] | (no python-dev package on jammy) | 22:22 |
| JayF | T supports py2.7, right? | 22:23 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/yoga: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862653 | 22:26 |
| stevebaker[m] | JayF: yes, and py27 is green. So either it is not running on jammy, or it doesn't need to build packages from source to run (so it doesn't use or install python-dev) | 22:29 |
| JayF | interesting | 22:29 |
| JayF | I don't understand all of this, I trust you will do the right things | 22:30 |
| JayF | and I will take a review pass on all the stuff you have up tomorrow | 22:30 |
| JayF | right now my brain is pretty fried | 22:30 |
| stevebaker[m] | lol I know what I'm doing? ;) If py27 stays green with the backport then I'll be happy but not necessarily know why | 22:31 |
| JayF | If you don't know what you are doing, nobody else will ;) | 22:32 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/xena: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862654 | 22:34 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/wallaby: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862655 | 22:35 |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: CI: Make swift/Vmedia job non-voting https://review.opendev.org/c/openstack/ironic-python-agent/+/861454 | 22:36 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/victoria: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862656 | 22:37 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/ussuri: WIP Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862658 | 22:47 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/train: WIP Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862659 | 22:50 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/yoga: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862653 | 22:54 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/xena: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862654 | 22:54 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/wallaby: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862655 | 22:55 |
| opendevreview | Steve Baker proposed openstack/ironic-python-agent stable/victoria: Drop python2 from bindep.txt https://review.opendev.org/c/openstack/ironic-python-agent/+/862656 | 22:55 |
| stevebaker[m] | Spamming done for now, lunch time | 22:56 |
| TheJulia | \o/ | 22:56 |
| opendevreview | Merged openstack/ironic-python-agent stable/ussuri: Use utf-16-le if BOM not present https://review.opendev.org/c/openstack/ironic-python-agent/+/861065 | 23:04 |
| TheJulia | is the centos8 mirror pathing impacting victoria as well? | 23:31 |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: Use utf-16-le if BOM not present https://review.opendev.org/c/openstack/ironic-python-agent/+/861197 | 23:34 |
| opendevreview | Jakub Jelinek proposed openstack/ironic master: WIP: Implements node inventory: database https://review.opendev.org/c/openstack/ironic/+/862569 | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!