| *** pmannidi is now known as pmannidi|AFK | 01:36 | |
| *** pmannidi|AFK is now known as pmannidi|Away | 02:45 | |
| *** pmannidi|Away is now known as pmannidi|AFK | 02:50 | |
| *** pmannidi|AFK is now known as pmannidi|Away | 02:50 | |
| *** pmannidi|Away is now known as pmannidi|AFK | 02:50 | |
| *** pmannidi|AFK is now known as pmannidi|Away | 02:50 | |
| *** pmannidi|Away is now known as pmannidi|AFK | 02:54 | |
| *** pmannidi|AFK is now known as pmannidi|Away | 02:54 | |
| *** pmannidi|Away is now known as pmannidi|AFK | 02:56 | |
| *** pmannidi|AFK is now known as pmannidi|Away | 02:56 | |
| *** pmannidi|Away is now known as pmannidi|AFK | 02:56 | |
| *** pmannidi|AFK is now known as pmannidi|Away | 02:56 | |
| iury|holiday | good morning janders and Ironic o/ | 05:53 |
|---|---|---|
| *** iury|holiday is now known as iurygregory | 05:53 | |
| janders | hey iurygregory o/ | 05:55 |
| iurygregory | o/ | 05:55 |
| arne_wiebalck | Good morning janders iurygregory and Ironic! | 06:44 |
| iurygregory | morning arne_wiebalck o/ | 06:44 |
| arne_wiebalck | TheJulia: thanks, much clearer! I still left some comments :) | 07:08 |
| janders | hey arne_wiebalck o/ | 07:16 |
| jssfr | hiho! I’m currently looking hardware inventorization/introspection using the ironic python agent. I note that it exports bmc_address, but not the BMC MAC address, even though that’d be readily available when queried from ipmitool. Is there any specific reason against implementing that? I might provide a patch. | 07:46 |
| vmud213 | jssfr: I think it's because we don't really use BMC MAC address | 07:56 |
| vmud213 | Most of the times the BMC provides REST API or xmlrpcs or some other mechanism to connect to it. So all that one may need to connect to the BMC is IP, port and some other essential connection properties | 07:58 |
| jssfr | vmud213, aha. My use case would be matching the introspection data with an existing hardware inventory/IPAM thing to automatedly decide what to do with the node (I should’ve mentioned that this is some "undercloud" standalone ironic use case) | 08:03 |
| vmud213 | jssfr: Sry i interpreted it wrong way. I got it now. So if you are introspecting, you also expect the BMC MAC and not just the BMC IP. Makes sense to me. | 08:06 |
| jssfr | Thanks for your feedback! | 08:24 |
| arne_wiebalck | jssfr: I think it does not hurt to add it (we are using the inventory data as well in some data centre inventory management system). I did not miss the BMC MAC yet, but I don't see a reason not to add it either :) | 08:35 |
| jssfr | I guess I’ll then have to finally figure out the OpenStack contribution process :) | 08:36 |
| cenne | ++ | 08:40 |
| dtantsur | good morning ironic! nearly half of the work week is done, right? | 08:43 |
| arne_wiebalck | jssfr: let us know in case you need help with this, the first time can be a little daunting :) | 09:06 |
| jssfr | thanks! I’m reading through the getting started docs and I guess I’ll play with the sandbox first :) | 09:08 |
| *** rpittau|afk is now known as rpittau | 09:42 | |
| rpittau | good morning ironic! o/ | 09:42 |
| opendevreview | Riccardo Pittau proposed openstack/sushy stable/wallaby: Fix Processor.sub_processors https://review.opendev.org/c/openstack/sushy/+/799762 | 10:17 |
| opendevreview | Riccardo Pittau proposed openstack/sushy stable/victoria: Fix Processor.sub_processors https://review.opendev.org/c/openstack/sushy/+/799763 | 10:18 |
| opendevreview | Merged openstack/sushy master: Add Drive.volumes https://review.opendev.org/c/openstack/sushy/+/799612 | 11:04 |
| opendevreview | Merged openstack/sushy stable/wallaby: Protect Connector against empty auth object https://review.opendev.org/c/openstack/sushy/+/799583 | 11:12 |
| opendevreview | Merged openstack/sushy stable/victoria: Protect Connector against empty auth object https://review.opendev.org/c/openstack/sushy/+/799584 | 11:37 |
| janders | see you tomorrow Ironic o/ | 12:50 |
| rpittau | bye janders :) | 12:52 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Bring boot_iso/deploy_iso handling in iLO closer to Redfish https://review.opendev.org/c/openstack/ironic/+/799696 | 12:59 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Clean up images when ejecting an ISO with Redfish https://review.opendev.org/c/openstack/ironic/+/799820 | 13:13 |
| TheJulia | good morning tzumainn | 13:22 |
| tzumainn | TheJulia, hi! | 13:22 |
| TheJulia | how goes it? | 13:23 |
| tzumainn | not too bad! | 13:24 |
| TheJulia | excellent | 13:24 |
| TheJulia | arne_wiebalck: trying to remember, are you on victoria or wallaby? | 13:52 |
| arne_wiebalck | TheJulia: Victoria (for Ironic) | 13:53 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Allow configuring enabled vendor interfaces https://review.opendev.org/c/openstack/bifrost/+/799837 | 13:57 |
| opendevreview | James Denton proposed openstack/networking-generic-switch master: Support multiple links in link_local_information dict of portgroups https://review.opendev.org/c/openstack/networking-generic-switch/+/557051 | 14:13 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Clean up images when ejecting an ISO with Redfish https://review.opendev.org/c/openstack/ironic/+/799820 | 14:31 |
| dtantsur | bfournie: JFYI ^^^ | 14:31 |
| dtantsur | ah damn, forgot a release note | 14:32 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Clean up images when ejecting an ISO with Redfish https://review.opendev.org/c/openstack/ironic/+/799820 | 14:35 |
| dtantsur | "The `libvirt` module is not importable. Check the requirements." | 14:46 |
| dtantsur | ugh | 14:46 |
| TheJulia | arne_wiebalck: ack | 14:53 |
| dtantsur | okay, so bifrost CI is broken, and I still have no clue why. Help welcome. | 14:58 |
| rpittau | centos 8 ? | 15:00 |
| dtantsur | yep | 15:00 |
| opendevreview | Merged openstack/ironic-lib master: Use the json format output of qemu-img info https://review.opendev.org/c/openstack/ironic-lib/+/799664 | 15:04 |
| JayF | I backported (wallaby) that resource provider fix in the nova driver here, https://review.opendev.org/c/openstack/nova/+/799772, if anyone wants to take a look | 15:07 |
| TheJulia | JayF: thanks, +1'ed :) | 15:08 |
| arne_wiebalck | JayF: we have seen RPs being deleted as well in the past, but did not get round to have a closer look (it is a rare, but happens). IIRC, this was indeed *during* instance deployment. I wonder if that patch would fix the issue for us as well ... I will point our Nova/Placement expert to it, thanks! | 15:18 |
| JayF | It's in nova xena (master) already | 15:19 |
| JayF | that's just the wallaby backport, which was direct and clean | 15:19 |
| dtantsur | .. and of course I cannot reproduce it locally .... | 15:21 |
| arne_wiebalck | JayF: Right. I was not aware of that patch at all, maybe my Nova/Placement colleagues are. | 15:23 |
| JayF | oh, yeah, nice | 15:24 |
| JayF | we were talking abuot it in SPUC along with your reported bug | 15:24 |
| arne_wiebalck | JayF: Our Nova is still on Stein, and the patch is tiny, so it makes sense to have a look if this is our issue. | 15:24 |
| JayF | ah, yeah go for it | 15:25 |
| JayF | arne_wiebalck: the ocata-ish version https://gist.github.com/jayofdoom/4fa315489330430ea7aeaa6e8ad62dec if that's easier to forward-port than backporting from master | 15:25 |
| arne_wiebalck | It has become much better, though. In the past we had situations where *all* was erased under certain circumstances :-S | 15:26 |
| arne_wiebalck | JayF: oh, cool, thanks! | 15:26 |
| rpittau | dtantsur: are you using python 3.6 locally ? | 15:27 |
| dtantsur | rpittau: 3.6.8, yes | 15:28 |
| dtantsur | same as in the CI, I would expect? | 15:28 |
| TheJulia | The code in nova has definitely evolved and it is less likely of a possibility with conductor groups since you have a smaller window between while your retrieving the node list | 15:28 |
| rpittau | yeah | 15:28 |
| dtantsur | rpittau: but I use "normal" CentOS, not Stream | 15:28 |
| rpittau | oh ok | 15:29 |
| arne_wiebalck | TheJulia: The main incidents were maybe before conductor groups. | 15:29 |
| arne_wiebalck | TheJulia: I think ... | 15:29 |
| TheJulia | yeah, that window being opened by the data set is really just huge in terms of where things can go sideways | 15:30 |
| arne_wiebalck | Conductor groups ftw! | 15:31 |
| TheJulia | well, zer0c00l's patch FTW | 15:32 |
| arne_wiebalck | As well, and Ironic FTW! | 15:32 |
| dtantsur | \o/ | 15:32 |
| arne_wiebalck | On a side note, we are moving our batch infra from VMs to bare metal, all via Ironic ofc. | 15:33 |
| dtantsur | arne_wiebalck: interesting, why? | 15:33 |
| arne_wiebalck | dtantsur: the main reason is Ironic | 15:33 |
| dtantsur | :D | 15:34 |
| arne_wiebalck | dtantsur: we accepted a virtualisation tax since the batch service got an API with OpenStack | 15:34 |
| arne_wiebalck | dtantsur: now that Ironic provides that API, we claim the tax back | 15:34 |
| dtantsur | got it | 15:34 |
| dtantsur | have you ever estimated how high the tax is? | 15:35 |
| arne_wiebalck | dtantsur: oh, yes: it is about 2-5% | 15:35 |
| * dtantsur collects fact for the "Why ironic" slide on numerous presentations | 15:35 | |
| dtantsur | 5% is substantial | 15:35 |
| arne_wiebalck | dtantsur: yes | 15:35 |
| arne_wiebalck | dtantsur: we were paying it to get the API | 15:36 |
| arne_wiebalck | dtantsur: now, since the same API can be used with physical nodes, the batch team can keep their machinery | 15:36 |
| arne_wiebalck | dtantsur: they have some provisioner code which creates instances, does not matter if virtual or physical :) | 15:37 |
| dtantsur | cool! | 15:38 |
| arne_wiebalck | dtantsur: I said conductor groups ftw, since we now distribute nodes over groups to use all conductors in parallel, and their provisioner can be configured to create N nodes in parallel ... they did 10, so I asked them to crank it up a bit to see where the limits are :) | 15:39 |
| arne_wiebalck | dtantsur: they moved to 50 in parallel, still all ok, now we need to wait for more/larger deliveries to come ... | 15:40 |
| dtantsur | :) | 15:40 |
| arne_wiebalck | (this is more an academic exercise, doing 10 in parallel and let it chew on the available resources is totally fine) | 15:40 |
| JayF | at onmetal, we used to do bulk builds and see how it went | 15:43 |
| JayF | we did 100 over the course of like 5 minutes, I think 2/3 succeeded even if a lot had to reschedule | 15:43 |
| JayF | and that was long before the agent driver was any good :P | 15:43 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Use selectinload for all list queries https://review.opendev.org/c/openstack/ironic/+/797337 | 15:44 |
| TheJulia | arne_wiebalck: ^^ revised | 15:44 |
| arne_wiebalck | TheJulia: +2'ed, thanks! | 15:50 |
| arne_wiebalck | JayF: 60 was a limit for us in the past, but this was with 1 controller and our previous PXE infra, so I am curious to see how far we can go now (while at the same time) I do not want to give the impression to users that doing nothing for weeks and then expect to install 1000 nodes in one go is a sensible way of working :) | 15:53 |
| TheJulia | never a sensible way of working | 15:53 |
| TheJulia | The analogy that comes to mind is when you try to delete rows from a huge database | 15:53 |
| TheJulia | You can slip in lots of small deletes and not impact service/performance that much. Things will just chug along. But if you say "I want to delete these million rows", your database is locked and your not going anywhere fast | 15:54 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [DNM] downgrade ansible https://review.opendev.org/c/openstack/bifrost/+/799858 | 15:54 |
| dtantsur | baremetal-operator uses batches of 20 IIRC | 15:56 |
| TheJulia | I actully had 1.6 million rows to delete once. I did the math on how long it would take after our monitoring system was down for 45 minutes while something like 40k rows were deleted. | 15:57 |
| dtantsur | oh omg | 15:58 |
| dtantsur | easier to create a new table and move the remaining data there | 15:58 |
| arne_wiebalck | dtantsur: exactly ... the "issue" is that new resources do not trickle in, but come in big chunks | 15:58 |
| TheJulia | I ended up putting a cron job in to slowly delete out the rows and once a week do the needful db work. | 15:59 |
| TheJulia | dtantsur: didn't have the disk space on the physical server, nor could I add any space, to make a copy of the table. | 15:59 |
| TheJulia | That was so painful | 16:00 |
| dtantsur | wow | 16:01 |
| arne_wiebalck | TheJulia: this is how we prune our (OpenStack) databases, in fact, regularly in small chunks | 16:01 |
| TheJulia | I think I did that in April, and the job finally failed to execute anymore when 0 matching rows were found in January | 16:01 |
| TheJulia | (i... completely... forgot about the cron job by then) | 16:01 |
| dtantsur | huh, we don't run online_data_migrations in bifrost? Oo | 16:02 |
| TheJulia | dtantsur: I'm fairly sure we used to | 16:02 |
| dtantsur | I'm upgrading my environment, and it complains on dbsync... | 16:02 |
| dtantsur | we do run it at some point.. I wonder if "some point" is wrong? | 16:03 |
| TheJulia | maybe? | 16:03 |
| dtantsur | It's after starting ironic, so should be correct? hmmmm... | 16:04 |
| rpittau | dtantsur: I can't reproduce it either on centos stream, wondering if it's transient ? | 16:05 |
| dtantsur | "Please run "ironic-dbsync online_data_migrations" using the previous release" | 16:05 |
| TheJulia | oh... | 16:06 |
| dtantsur | it is a bad joke. openstack doesn't support downgrades, so anyone who gets in this situation will have to edit the database.. | 16:06 |
| * dtantsur is not sure why he's never thought about it | 16:06 | |
| TheJulia | So you jumped 2 releases? | 16:06 |
| dtantsur | nope | 16:06 |
| TheJulia | then you shouldn't have needed to | 16:06 |
| dtantsur | I suspect this environment was created in wallaby | 16:06 |
| dtantsur | something feels really off | 16:06 |
| TheJulia | yeah, something does | 16:06 |
| dtantsur | (especially since I've just upgraded another environment without problems) | 16:06 |
| TheJulia | previous release makes me think you've got some really old objects in the db but that doesn't make sense | 16:07 |
| dtantsur | yeah, I'll dive into the database now | 16:07 |
| dtantsur | nodes 1.35, ports 1.10. wtf | 16:08 |
| rpittau | bye! o/ | 16:09 |
| *** rpittau is now known as rpittau|afk | 16:09 | |
| dtantsur | hmmm, it's the same on my old environment | 16:09 |
| dtantsur | wut | 16:09 |
| dtantsur | s/old/another/ | 16:09 |
| dtantsur | BIOS settings. I have 1.0 for BIOS settings. | 16:12 |
| arne_wiebalck | nodes 1.35 is Victoria, no? During the upgrade I had some controllers stay behind on Ussuri and I got "Nodes 1.34 is bad" message all over the place :-D | 16:13 |
| opendevreview | Merged openstack/ironic master: Support "swift" for ramdisk_image_download_source https://review.opendev.org/c/openstack/ironic/+/799685 | 16:14 |
| dtantsur | Node 1.35 is current | 16:14 |
| arne_wiebalck | so it has not changed since Victoria then | 16:15 |
| TheJulia | arne_wiebalck: well that is a bug too then | 16:18 |
| TheJulia | Well, common db, you can't do the data migration until your fully upgraded and your supposed to do it after upgrade, but it is redundant on a single hop upgrade if I'm remembering correctly | 16:19 |
| dtantsur | I don't understand how this logic is supposed to work any more | 16:19 |
| JayF | arne_wiebalck: https://review.opendev.org/c/openstack/ironic/+/794880/9 is before that selectinload patch and needs a core vote, too | 16:19 |
| JayF | arne_wiebalck: if you're still around and wanna look and land it | 16:19 |
| JayF | arne_wiebalck: if you do, I'll backport the whole series to wallaby today | 16:19 |
| dtantsur | how are version updated if you can run dbsync until versions are updated? | 16:20 |
| arne_wiebalck | JayF: checking ... | 16:20 |
| dtantsur | mmm, I suspect I know the problem. BIOSSetting removed 1.0 immediately after adding 1.1. there has to be an overlap | 16:21 |
| TheJulia | well, can't run... I do believe we have another bug in the migrations usage | 16:21 |
| TheJulia | oh!... | 16:21 |
| TheJulia | that seems like it could do it | 16:21 |
| TheJulia | And yeah, there has to be if memory serves | 16:21 |
| dtantsur | although I see the same was done for nodes as some point and nobody complained.. | 16:21 |
| dtantsur | I don't get it any more. but yes, I think there should be an overlap | 16:21 |
| TheJulia | rloo made sure the docs are very explicit, likely need to go back and check them | 16:22 |
| * dtantsur is reading https://docs.openstack.org/ironic/latest/contributor/rolling-upgrades.html | 16:24 | |
| opendevreview | Merged openstack/ironic master: Avoid double file removal in create_boot_iso https://review.opendev.org/c/openstack/ironic/+/799695 | 16:28 |
| dtantsur | no, I don't understand anything... | 16:29 |
| opendevreview | Merged openstack/ironic master: Add note regarding configuration drives to tuning docs https://review.opendev.org/c/openstack/ironic/+/789623 | 16:29 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic stable/wallaby: Avoid double file removal in create_boot_iso https://review.opendev.org/c/openstack/ironic/+/799774 | 16:29 |
| TheJulia | arne_wiebalck: you may want to look at https://review.opendev.org/c/openstack/ironic/+/794880/9 | 16:37 |
| JayF | TheJulia: jinx :P | 16:39 |
| JayF | I'm tryin' to get your patches merged too so I can backport them to wallaby | 16:39 |
| TheJulia | heh | 16:39 |
| TheJulia | Part of my take over the world plan | 16:40 |
| TheJulia | Our armies of metal shall be efficiently provisioned and managed | 16:40 |
| opendevreview | Derek Higgins proposed openstack/ironic stable/wallaby: Skip port create if MAC is blank https://review.opendev.org/c/openstack/ironic/+/799868 | 16:40 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Add overlap in BIOSSettings versions between 17.0 and 18.0 https://review.opendev.org/c/openstack/ironic/+/799869 | 16:43 |
| dtantsur | TheJulia, rloo, feel free to tell me it's completely misguided ^^^ because I don't understand how versions work | 16:43 |
| TheJulia | since 18 is a mid-cycle release, yes, that *is* correct | 16:45 |
| rloo | ugh, you want me to remember that stuff? :) I trust TheJulia but will take a quick look... | 16:45 |
| dtantsur | I think there should be an overlap between the cycles as well | 16:45 |
| dtantsur | rloo: yes please. I'm completely confused, maybe your memory serves you better (quite likely) | 16:46 |
| TheJulia | rloo: you if you quickly dd if=/dev/urandom of=/path/to/file/with/that/memory, it will all be okay :) | 16:46 |
| rloo | doubtful about memory. hopefully the docs i wrote will trigger something! | 16:46 |
| TheJulia | If memory serves, and I didn't look at the docs first, but the version needs to know what it can be upgraded from | 16:47 |
| TheJulia | or at least, what it is at so it can be upgraded | 16:47 |
| * TheJulia may be oversimplifying it | 16:47 | |
| rloo | BIOSSetting is new? no 1.0 entry at all... when did it get added? | 16:47 |
| * dtantsur is clueless | 16:48 | |
| dtantsur | I suspect we default to 1.0 anyway, that's what my table had | 16:48 |
| TheJulia | bios setting has been around for a long time | 16:48 |
| TheJulia | maybe 3-4 years | 16:48 |
| TheJulia | I think | 16:48 |
| dtantsur | yeah, it's not quite new | 16:51 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Bring boot_iso/deploy_iso handling in iLO closer to Redfish https://review.opendev.org/c/openstack/ironic/+/799696 | 16:52 |
| arne_wiebalck | TheJulia: LGTM! As usual, I am picking on some typos as well. Let me know if you would like to patch, or if I shall +W as is. | 16:53 |
| *** mgoddard- is now known as mgoddard | 16:54 | |
| TheJulia | arne_wiebalck: editing the patch requires the latter one to need to rebased, so you can just +W and a follow-up should be fine as long as it is not actual code | 16:54 |
| TheJulia | or if it is release note | 16:54 |
| * TheJulia looks | 16:54 | |
| rloo | ok, 3 years ago (https://opendev.org/openstack/ironic/commit/9eaff34b5ba4712bc10a521026d513d93cb019a2). we must have forgotten to add it to the mapping table. prob not worth adding it in retroactively. | 16:54 |
| TheJulia | arne_wiebalck: yeah, so... workflow it, I'll do a follow-up since it makes sense to keep clean moving forward and it is just comments/doc string stuff | 16:56 |
| NobodyCam | Good Morning Ironic folks | 16:56 |
| TheJulia | Good Morning NobodyCam | 16:56 |
| NobodyCam | hehehe | 16:56 |
| arne_wiebalck | TheJulia: done, thanks | 16:56 |
| NobodyCam | https://www.irccloud.com/pastebin/6xlaRVIR/ | 16:57 |
| arne_wiebalck | Good morning NobodyCam o/ | 16:57 |
| NobodyCam | morning arne_wiebalck | 16:57 |
| TheJulia | We're slowly taking over the world using the ironic, although I sense bifrost unhappiness? | 16:57 |
| bfournie | dtantsur: hmm, yeah I think I see, there was no BIOSSetting field in the mappings prior to https://review.opendev.org/c/openstack/ironic/+/786707/18/ironic/common/release_mappings.py as rloo noticed | 16:57 |
| NobodyCam | LOL no unhappy here. just having a giggle at the note for dnsmasq to be last | 16:58 |
| TheJulia | NobodyCam: dnsmasq + dns resolution stuffs == root of all evil | 16:59 |
| NobodyCam | :) | 16:59 |
| TheJulia | Should we ever do something silly like create our own version of http://www.eviloverlord.com/lists/overlord.html, it shall be that DNS is the root of all evil. | 16:59 |
| dtantsur | NobodyCam: yep, I keep forgetting to remove this note :) | 17:00 |
| dtantsur | also, yes. DNS, IPv6, what else? | 17:01 |
| JayF | I can't believe you all are talking about roots of evil | 17:01 |
| JayF | and nobody mentioned IPMI, or hardware in general | 17:01 |
| dtantsur | IPMI, yes | 17:02 |
| NobodyCam | I was just about to bring up tftp | 17:02 |
| * arne_wiebalck shivers | 17:02 | |
| TheJulia | This is starting to sound like we need to have a midcycle and create such a list | 17:03 |
| dtantsur | maybe one day meeting each other is a thing again.... | 17:03 |
| TheJulia | I sure hope so | 17:04 |
| dtantsur | well, eventually we'll run out of greek letters! | 17:04 |
| rloo | dtantsur: does our code support both v1 and v1.1 of BIOSSetting? | 17:04 |
| dtantsur | rloo: for some definition of "support"? | 17:05 |
| dtantsur | we have version conversion in place | 17:05 |
| rloo | dtantsur: i'm trying to remember/understand. I don't think we need an 'overlap' unless we want to continue supporting those versions. the migration is meant to convert from an older version to a newer one. | 17:05 |
| dtantsur | rloo: if there is no overlap, the migration will refuse to work | 17:05 |
| dtantsur | both dbsync and online_data_migrations verify versions before running | 17:05 |
| rloo | i think the problem is that there isn't any BIOSSetting 1.0 in the previous release mapping | 17:05 |
| dtantsur | I don't think so, what would it change? | 17:06 |
| rloo | i could be wrong though | 17:06 |
| rloo | am looking at Node versions. sometimes there is overlap, sometimes not. eg, 11.1 -> 12.0 no overlap. Trying to remember... | 17:08 |
| dtantsur | yep, that's what is confusing me | 17:10 |
| dtantsur | I don't understand how it works without an overlap... but it used to work? | 17:10 |
| rloo | that list of versions is due to this: https://review.opendev.org/c/openstack/ironic/+/408556/52/ironic/db/sqlalchemy/api.py, L1128. lots of comments but ... what does 'supported versions' mean. | 17:12 |
| rloo | is there code that migrates BIOSSetting from 1.0 to 1.1? | 17:13 |
| TheJulia | there is no data transformation, just additional fields aiui | 17:13 |
| TheJulia | so what *changes* is the object model has an additional field | 17:14 |
| dtantsur | in theory online_data_migrations have to update versions | 17:14 |
| dtantsur | and do any changes required (beyond dbsync) | 17:14 |
| TheJulia | yup | 17:14 |
| TheJulia | yup | 17:14 |
| rloo | online_data_migrations is connected to this check_versions code I think... dtantsur, you mention that the upgrade fails. what was the error? | 17:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix typos in API sanitization change notes https://review.opendev.org/c/openstack/ironic/+/799872 | 17:15 |
| rloo | (looking back at irc...) | 17:16 |
| arne_wiebalck | bye everyone o/ | 17:17 |
| dtantsur | rloo: The database is not compatible with this release of ironic (18.0.1.dev90). Please run "ironic-dbsync online_data_migrations" using the previous release | 17:19 |
| dtantsur | I also think we should provide a better recommendation than "use the previous release", which is practically impossible in many cases | 17:19 |
| dtantsur | but that's a different topic for another day | 17:19 |
| rloo | dtantsur: i think if we add BIOSSetting 1.0 to the previous releases, that will solve it. | 17:20 |
| dtantsur | rloo: how? | 17:20 |
| rloo | dtantsur: we don't want to support 1.0 in wallaby | 17:20 |
| dtantsur | then we need to remove the check on versions from dbsync | 17:20 |
| dtantsur | also, s/wallaby/xena/ you mean? | 17:20 |
| TheJulia | can I please get reviews on https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 so I can one day move the overall secure rbac model implementation effort to the proverbial done column? | 17:20 |
| rloo | 'using the previous release' means that this release cannot handle the migration cuz it had to have been doen with the previous release. | 17:21 |
| dtantsur | rloo: yep, and the previous release does nothing | 17:21 |
| dtantsur | because 1.0 is the maximum version there | 17:21 |
| rloo | i think it is cuz there is no explicit 1.0 | 17:21 |
| rloo | i saw somewhere that if it isn't specified, default is to use the latest version. | 17:21 |
| dtantsur | my table had 1.0 quite happily | 17:21 |
| opendevreview | Derek Higgins proposed openstack/ironic stable/wallaby: Skip port create if MAC is blank https://review.opendev.org/c/openstack/ironic/+/799868 | 17:21 |
| rloo | oh, so you tried adding 1.0 to 17? | 17:21 |
| dtantsur | rloo: no, it worked automatically for me | 17:22 |
| dtantsur | my initial state was BIOSSettings with 1.0 versions | 17:22 |
| dtantsur | I upgraded to latest master, and dbsync failed | 17:22 |
| dtantsur | because 1.0 is not compatible with ['1.1'] in Xena | 17:22 |
| dtantsur | I can reproduce it again and try adding 1.0 to 17 if you want to | 17:23 |
| dtantsur | hmm, interesting, it didn't fail indeed | 17:25 |
| dtantsur | now I officially have no clue how this works | 17:25 |
| dtantsur | rloo: ^^^ | 17:25 |
| rloo | i think (but haven't verified) that the situation with eg Node having more than one version in the list is due to us supporting interim versions (so eg we changed Node often w/i the older->latest releases. Cuz we support from previous named release to master or something like that. | 17:26 |
| dtantsur | yeah. it doesn't explain how the check magically works if I update an ancient release | 17:27 |
| rloo | for BIOSSetting, i think 1.1 was added in v 18.0. which is hmm.. not the first wallaby release. | 17:27 |
| rloo | RELEASE_MAPPING['victoria'] = RELEASE_MAPPING['16.0'] | 17:28 |
| rloo | RELEASE_MAPPING['wallaby'] = RELEASE_MAPPING['17.0'] | 17:28 |
| rloo | dtantsur: i think it has soemething to do with ^^ | 17:28 |
| dtantsur | yeeeep, this starts making some sense | 17:28 |
| TheJulia | well, 18.0 is xena | 17:29 |
| dtantsur | (we should have removed victoria before the first point release) | 17:29 |
| TheJulia | well, master really in xena | 17:29 |
| TheJulia | ++ | 17:29 |
| rloo | oh. | 17:29 |
| dtantsur | okay, so I feel like check_versions doesn't do what we think it does | 17:29 |
| dtantsur | it literally just checks if a version has EVER been supported | 17:29 |
| rloo | i think check_versions is used to see IF the db-migration stuff can handle the objects in the DB. if the objects have older versions than it can handle, it errors with that error you got. | 17:30 |
| dtantsur | yeah, but what's the real case? | 17:31 |
| dtantsur | we still happily support Node 1.21 from ironic 9.2 | 17:31 |
| rloo | not sure we need it now. it might have been when we/openstack was saying we'd only support migrations from previous named release to latest/master. | 17:31 |
| dtantsur | yeah, I think this was the intention | 17:32 |
| dtantsur | but that's not what the code does | 17:32 |
| rloo | but then we added stuff to do migrations from previous ones (unless we deleted that code, which we might have... can't recall now...) | 17:32 |
| TheJulia | hmmm | 17:32 |
| rloo | so i think stuff should work if we didn't delete migrations from older versions... | 17:32 |
| dtantsur | I suspect we were supposed to remove ancient versions from RELEASE_MAPPING | 17:32 |
| dtantsur | # Just after doing a named release, delete any entries associated with the | 17:32 |
| dtantsur | # oldest named release. | 17:32 |
| rloo | i thought we might have though. | 17:32 |
| dtantsur | from the code ^^^ | 17:32 |
| dtantsur | we never did that, we still have 9.2 in the mappings | 17:32 |
| dtantsur | which was Queens | 17:33 |
| TheJulia | Yeah, I think we only removed one... once. | 17:33 |
| dtantsur | okay, let me figure out which release added BIOSSettings, fix my patch and figure out the rest tomorrow | 17:33 |
| TheJulia | sounds like a plan | 17:33 |
| dtantsur | 11.0.0, Rocky | 17:34 |
| rloo | thx dtantsur! | 17:35 |
| dtantsur | thank YOU rloo for diving into that | 17:35 |
| dtantsur | we'd be still scratching heads without your help | 17:35 |
| rloo | (fingers crossed... this is why i try to document. no memory..) | 17:35 |
| rloo | that upgrade stuff is somewhat fragile in the sense of humans needing to do stuff... :-( | 17:36 |
| dtantsur | yeah, I'll think about some more automatic checks tomorrow (unless you beat me to it) | 17:37 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Add missing BIOSSettings version mapping for older releases https://review.opendev.org/c/openstack/ironic/+/799869 | 17:38 |
| dtantsur | okay, on this ^^ positive note I'm calling it a day | 17:38 |
| dtantsur | o/ | 17:38 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/wallaby: Ironic index docs/command check backport https://review.opendev.org/c/openstack/ironic/+/792873 | 18:23 |
| opendevreview | Merged openstack/ironic master: Allow node_sanitize function to be provided overrides https://review.opendev.org/c/openstack/ironic/+/794880 | 18:25 |
| JayF | TheJulia: when I backport that db-perf-1 topic, should I include the typo fixes? Leaning towards no | 18:58 |
| TheJulia | JayF: yeah, I don't think it is really needed to be backported, tbh | 18:59 |
| JayF | cool, in that case, keep eyes peeled, the last one is about to land then I'll mass-backport | 18:59 |
| JayF | hopefully before standup/lunch | 18:59 |
| opendevreview | Maksim Malchuk proposed openstack/tenks master: Fix the error 'The `libvirt` module is not importable' https://review.opendev.org/c/openstack/tenks/+/799901 | 19:07 |
| opendevreview | Merged openstack/ironic master: Use selectinload for all list queries https://review.opendev.org/c/openstack/ironic/+/797337 | 19:13 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: Only return the requested fields from the DB https://review.opendev.org/c/openstack/ironic/+/799880 | 19:16 |
| JayF | TheJulia: hmm, when backporting a chain like this, should I jsut do it one at a time, wait for them to merge? or set depends-on? | 19:17 |
| JayF | oh, this is going to be 100x easier if I do it from a cli, won't it | 19:17 |
| JayF | I'm going to do that, I thikn | 19:17 |
| JayF | anyone have insight into these failures? Am I missing some permissions or something? https://gist.github.com/jayofdoom/df50627ae77f06d0ce3e18bdb805b294 | 19:24 |
| JayF | aha, I think I get it | 19:25 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: Only return the requested fields from the DB https://review.opendev.org/c/openstack/ironic/+/799880 | 19:25 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: Set stage for objects to handle selected field lists. https://review.opendev.org/c/openstack/ironic/+/799904 | 19:25 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: API to pass fields to node object list https://review.opendev.org/c/openstack/ironic/+/799905 | 19:26 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: Allow node_sanitize function to be provided overrides https://review.opendev.org/c/openstack/ironic/+/799906 | 19:26 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/wallaby: Use selectinload for all list queries https://review.opendev.org/c/openstack/ironic/+/799907 | 19:26 |
| opendevreview | Maksim Malchuk proposed openstack/tenks master: Fix the error 'The `libvirt` module is not importable' https://review.opendev.org/c/openstack/tenks/+/799901 | 19:28 |
| JayF | ugh, I didn't topic those well and now I can't change the topic | 19:29 |
| JayF | https://etherpad.opendev.org/p/DBPerfBackports I put those, and the nova change, in here so it's easy to link to and review | 19:32 |
| TheJulia | JayF: thanks | 19:38 |
| * TheJulia reads | 19:38 | |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Remove train testing from ironic-tempest-plugin https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/799678 | 19:39 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Update min version of tox to use allowlist https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/796393 | 19:39 |
| TheJulia | JayF: U,mmmm you should be able to change the topic | 19:39 |
| JayF | TheJulia: when I try to repush with `git review -t [a better topic]` it is angry due to no changes | 19:39 |
| TheJulia | hmm | 19:40 |
| TheJulia | Interesting | 19:40 |
| JayF | TheJulia: and since it's a clean cherry-pick, I don't wanna modify it | 19:40 |
| JayF | it's not a big deal if we just land them :) | 19:40 |
| TheJulia | I wonder if that is something we lost in the most recent upgrade | 19:40 |
| TheJulia | wfm | 19:40 |
| JayF | you go review them and put +2s on them and I'll wait like a shark to attack the next core to say something in here so it gets merged | 19:40 |
| JayF | lol | 19:40 |
| TheJulia | lol | 19:43 |
| TheJulia | <3 | 19:43 |
| TheJulia | will do, need to finish current though in the document I'm writing | 19:43 |
| opendevreview | Maksim Malchuk proposed openstack/tenks master: Fix the error 'The `libvirt` module is not importable' https://review.opendev.org/c/openstack/tenks/+/799901 | 19:45 |
| TheJulia | JayF: so fwiw, it is missing the cherry picked from xyz commit... but AFAIK that was only used by one of the downstream delta tracking between branch tools | 20:25 |
| TheJulia | JayF: that got encoded in as "thou shall do this (so we don't break hpcloud branching)" a long time ago but I've never gotten a real answer on otherwise so I'm +2 on them. Someone can scream at me later if they can explain why it also needs to be human readable from just the commit message | 20:27 |
| opendevreview | Merged openstack/ironic master: Fix typos in API sanitization change notes https://review.opendev.org/c/openstack/ironic/+/799872 | 20:30 |
| stevebaker | good morning | 20:39 |
| TheJulia | o/ stevebaker | 20:42 |
| JayF | TheJulia: Probably because they were cherry-picked straight outta gerrit, not using the web ui | 20:51 |
| JayF | TheJulia: if it's an issue I can manually add (and fix thd topic) | 20:52 |
| * TheJulia shrugs | 20:52 | |
| JayF | TheJulia: given I -1'd a stable patch for that reason, I kinda feel like I should | 20:52 |
| TheJulia | git review -X does the magic fwiw | 20:52 |
| TheJulia | and if you just re-cherry-pick on to a clean branch off of stable/wallaby, then all of the cherry-pick topics should get updated | 20:52 |
| JayF | ima leave it and not review for that in the future, either | 20:53 |
| TheJulia | k | 20:53 |
| * TheJulia starts the "question policy dogma" club and adds JayF to it | 20:53 | |
| JayF | stevebaker: if you got a second, would <3 some core reviews on the backports in this etherpad: https://etherpad.opendev.org/p/DBPerfBackports (the perf stuff julia has been working) | 21:16 |
| TheJulia | JayF: I have this strange feeling I need to send you some sort of thing | 21:23 |
| JayF | WDYM? | 21:24 |
| TheJulia | https://www.amazon.com/Pieces-Accessory-Costume-Sharks-Adults/dp/B081JNWBB9 :) | 21:24 |
| JayF | I have my own cluesticks here, I'm sure my wife would volunteer to use one if you wanted | 21:24 |
| JayF | ahh | 21:24 |
| JayF | I promise you I'd never wear that :D | 21:24 |
| JayF | but the thought is nice lol | 21:24 |
| TheJulia | lol | 21:24 |
| TheJulia | :) | 21:24 |
| JayF | jroll once called me an unbalanced washing machine | 21:25 |
| JayF | so if I'm going to costume up, I think that's gotta be it lol | 21:25 |
| TheJulia | oh noes.... | 21:25 |
| jroll | lmao | 21:25 |
| TheJulia | Well, if one must wear a costume | 21:25 |
| * TheJulia pulls up the weather station data and wonders "how much am I going to roast" | 21:26 | |
| JayF | TheJulia: real story: I used to have a roommate named "Rob", who was tall, skinny, and very very quiet | 21:26 |
| JayF | TheJulia: we went one year to work as "Jay and Silent Bob" | 21:27 |
| JayF | probably #1 best costume usage ever, especially since Rob did a good job of actually being loud and obnoxious and I managed to STFU for a day | 21:27 |
| TheJulia | Heh | 21:28 |
| stevebaker | JayF: sure thing | 21:34 |
| JayF | stevebaker: thanks buddy \o/ | 21:34 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/wallaby: Ironic index docs/command check backport https://review.opendev.org/c/openstack/ironic/+/792873 | 21:35 |
| TheJulia | You know you live someplace hot when you ponder "Will 109F begin to overheat the car" | 21:36 |
| JayF | You're talking to the guy who no more than 2 weeks ago spent 115F weather inside a house with broken AC | 21:36 |
| JayF | 109F is wimpy temps /s | 21:37 |
| TheJulia | heh | 21:42 |
| TheJulia | Well, at least it is not 120F+ this week | 21:43 |
| JayF | Anyone got experience with setting up virtualbmc? AFAICT I'm doing everything right, but I'm getting errors https://gist.github.com/jayofdoom/4df151c28c8d90d36bc2609d66cee0f8 | 22:33 |
| JayF | and I have vmbcd running in debug/foreground, and there are no error messages from it | 22:33 |
| JayF | AFAICT vbmcd isn't even listening on the port after added | 22:34 |
| JayF | thank you empty IRC for rubber ducking; I needed a `vbmc start [domain]` https://github.com/openstack/ironic/blob/e3b401baf674a897883c27d2b0143283c2dd5acb/devstack/lib/ironic#L2085 | 22:35 |
| janders | good morning Ironic o/ | 22:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!