Friday, 2021-12-03

TheJuliacvstealth: so00:38
TheJuliaso Ironic will put all of the networks as single ports on to the provisioning or cleaning network. It doesn't attempt to do bonding or anything like that.00:39
TheJuliainstance vifs, can end up attaching to a portgroup which is the physical representation of what a bond would be to ironic00:39
TheJuliaat least, to the switch00:39
TheJuliaso neutron side, you don't need two interfaces in the bridge mapping for provisioning/cleaning00:40
TheJuliayou just need one00:40
TheJuliathe portgroup really doesn't apply in that context00:40
TheJuliaAnd it doesn't know about it nor have the capability to know about it when booting a deployment ramdisk00:40
TheJuliaPart of that is by design, It is hard to articulate so single interfaces separately attached is simpler00:41
TheJuliaSo neutron should spin up two or more dnsmasq instances, unless your using the same physical network for everything00:41
TheJuliawhich means your instances are being deployed to your provisioning/cleaning/deployment network, typically this is static networking switch side and pre-configured, where as the neutron network_interface on an ironic node is neutron gets additional information for port binding attachments dynamically00:42
cvstealthTheJulia: I was trying to follow best practices by not giving access to the provisioning/cleaning network to end users, so that is off of 1 interface then the service network they have access to for instances that is what I was setting up as a port group.01:48
cvstealthBoth provisioning/cleaning network and service networks are static networking, from the Neutron side it looked like for the dnsmasq to properly work in this setup it would require 2 entries for the bridge_mappings. 01:50
opendevreviewSteve Baker proposed openstack/ironic master: Refactor driver_internal_info updates to methods  https://review.opendev.org/c/openstack/ironic/+/76486301:50
opendevreviewSteve Baker proposed openstack/ironic master: Use driver_internal_info methods for driver utils  https://review.opendev.org/c/openstack/ironic/+/81850501:50
opendevreviewSteve Baker proposed openstack/ironic master: Use driver_internal_info methods for drac driver  https://review.opendev.org/c/openstack/ironic/+/81850601:50
opendevreviewSteve Baker proposed openstack/ironic master: Use driver_internal_info methods for ilo driver  https://review.opendev.org/c/openstack/ironic/+/81850701:50
opendevreviewSteve Baker proposed openstack/ironic master: Use driver_internal_info methods for redfish driver  https://review.opendev.org/c/openstack/ironic/+/81850801:50
opendevreviewSteve Baker proposed openstack/ironic master: Use driver_internal_info methods for other drivers  https://review.opendev.org/c/openstack/ironic/+/81850901:50
cvstealthIn my current mapping the systems running OVS have bridge_mapping = physnet1:br-ex and using routed networks the provisioning/cleaning network is flat and attached to br-ex. I don't have another bridge/network interface to bind the service network DHCP process to  without breaking the bond a part. Leading into asking if it was possible to have service network data in the configdrive so the 01:59
cvstealthdeployed instance could come up with a functional network stack. Currently the provisioning of instances works perfectly except there is no functional network for users to SSH into. 01:59
TheJuliacvstealth: and bonded interfaces out to the end bare metal nodes as well? Is that correct?02:35
TheJuliaThe neutron bridge mapping is really, aiui neutron internal with how it maps configured names in openstsck to the underlying physical networks on the neutron nodes, and only the neutron nodes02:37
cvstealththe bonded interaces are just on the tenant network, I don't have any bonding on the provisioning/cleaning. Each node has 3 interfaces, 1 copper for provisioning/cleaning then a dual 10/25G nic that is intended for the tenant/service network.02:42
TheJuliaOh!02:43
TheJuliaWow, so. Is this with the flat network interface in ironic or the neutron network interface?02:44
cvstealthI'm using the neutron network interface for the nodes02:44
cvstealthDuring the deploy phase neutron ports get created for both provisioning and service ports which is fine, once the deploy is done and BM reboots then the only neutron port left is the service port(s). 02:46
*** sshnaidm is now known as sshnaidm|off02:57
opendevreviewSteve Baker proposed openstack/ironic master: Ensure desired permissions on all written pxe files  https://review.opendev.org/c/openstack/ironic/+/80929503:27
opendevreviewSteve Baker proposed openstack/ironic master: Move place_loaders_for_boot to boot driver __init__  https://review.opendev.org/c/openstack/ironic/+/81972903:27
opendevreviewSteve Baker proposed openstack/ironic master: Write master grub config on startup  https://review.opendev.org/c/openstack/ironic/+/81558003:27
opendevreviewSteve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros  https://review.opendev.org/c/openstack/ironic/+/81539203:27
opendevreviewSteve Baker proposed openstack/bifrost master: Use distro conf loader_file_paths to copy ipxe/grub files  https://review.opendev.org/c/openstack/bifrost/+/81870903:27
opendevreviewSteve Baker proposed openstack/bifrost master: WIP support grub network boot  https://review.opendev.org/c/openstack/bifrost/+/80722003:27
opendevreviewSteve Baker proposed openstack/bifrost master: Don't create ironic_tftp_master_path directory  https://review.opendev.org/c/openstack/bifrost/+/82031103:27
TheJuliacvstealth: okay, we’ll, I think you can ignore the bridge mappings entirely, tbh04:10
TheJuliaIt is portgroups in ironic which are important to be defined, if I am remembering correctly04:11
cvstealthIsn't the mapping needed when setting up the flat network for the service/tenant network for the DHCP agent assignment? 04:17
TheJuliacvstealth: aiui, only the actual attachment so neutron knows where to attach virtual routers and dhcp server namespaces04:19
cvstealtheven when forcing a Neutron DCHP agent for the network I created I didn't see any configs in dnsmasq outside of the provisioning/cleaning ports 04:19
TheJuliaOh, it runs everything on command lines in namespaces04:20
TheJuliaSo no config files short of the actual leases in use at present04:20
TheJuliaWife is telling me time to go to bed :(04:21
TheJulia4:30 alarm for 5 am meeting :(04:21
cvstealthI was looking in the namespace files for the leases for the macs, all good thanks for the help and have a good night. 04:22
arne_wiebalckGood morning, Ironic!07:08
rpittaugood morning ironic! o/07:28
rpittauoh wow we don't need the py36 patches anymore, TC decided to keep that for the Yoga cycle07:30
rpittaugood news07:30
rpittauif anyone has a femtosecond please check https://review.opendev.org/c/openstack/bifrost/+/819057, just need approval, thanks :)07:35
rpittaubtw this is what I was referring to for the TC decision http://lists.openstack.org/pipermail/openstack-discuss/2021-December/026164.html07:39
jandershey rpittau o/07:53
jandersHappy Friday07:53
opendevreviewRiccardo Pittau proposed openstack/ironic-prometheus-exporter master: Use only Yoga tests  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/82032107:53
jandersgood news w/r/t py36!07:53
rpittauhey janders :)07:53
rpittauyeah, although now I have to revert some patches! :)07:54
rpittauwe have a couple of months more to work on centos stream 9 at least07:54
opendevreviewRiccardo Pittau proposed openstack/sushy master: Use only Yoga tests  https://review.opendev.org/c/openstack/sushy/+/82032207:55
opendevreviewRiccardo Pittau proposed openstack/metalsmith master: Use only Yoga tests  https://review.opendev.org/c/openstack/metalsmith/+/82032307:56
opendevreviewRiccardo Pittau proposed openstack/ironic-python-agent master: Use only Yoga tests  https://review.opendev.org/c/openstack/ironic-python-agent/+/82032407:56
opendevreviewRiccardo Pittau proposed openstack/python-ironicclient master: Use only Yoga tests  https://review.opendev.org/c/openstack/python-ironicclient/+/82032507:57
rpittauthat should be all07:57
dtantsurmorning ironic07:58
dtantsuryay re 3.6 \o/07:58
jandershey dtantsur o/07:59
rpittauhey dtantsur :)07:59
dtantsurlet the switch to 3.9 be a 2022 problem :)07:59
dtantsurrpittau: wanna update the metal3 thread?07:59
rpittauyeah, was going to do it07:59
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] Do not use RPC continue_node_deploy/clean in agent_base  https://review.opendev.org/c/openstack/ironic/+/75635408:31
mnasiadkaGood morning08:32
mnasiadkaI had a case yesterday, that erase_devices_metadata did not clean GPT partition table correctly and SW raid create_configuration cleaning step did fail due to existing partition table - is that a known bug?08:32
dtantsurmnasiadka: shouldn't be happening, except that software RAID is explicitly not removed08:34
mnasiadkadtantsur: that is logical to me (that it shouldn't be happening) - but I think agent is just running wipefs on erase_devices_metadata - and that might not be enough looking at some past bugs/ironic-python-agent code changes08:36
mnasiadkabut let me check cleaning logs to be sure it wasn't the RAID skip case08:37
dtantsuryeah08:38
dtantsurI don't think wipefs is the only thing we do08:39
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] All-in-one Ironic service with a local RPC bus  https://review.opendev.org/c/openstack/ironic/+/81962008:41
dtantsurmnasiadka: https://opendev.org/openstack/ironic-lib/src/branch/master/ironic_lib/disk_utils.py#L590-L65308:41
mnasiadkaok then, it seems one of the disk was previously an MD device, and second one had a partition table before MD was in place08:42
mnasiadkaand since target_raid_config was empty - delete_configuration step (interface "raid") did fail08:43
mnasiadkaClean step raid.delete_configuration failed on node e760f7c1-bcb9-4bee-b3b5-27a5ac8fda47. Error performing clean step delete_configuration: local variable 'raid_device' referenced before assignment08:43
mnasiadkaThat was the error.08:43
mnasiadka(when target_raid_config was empty and I ran raid:delete_configuration cleaning step)08:44
dtantsurarne_wiebalck: ^^^08:45
arne_wiebalckhmm ... interesting, I have never seen this (and we have plenty of nodes w/o target_raid_config and run this step on every delete)08:46
arne_wiebalckmnasiadka: which version is this?08:47
mnasiadkaarne_wiebalck: wallaby08:47
arne_wiebalckmnasiadka: we are still on victoria08:52
arne_wiebalckmnasiadka: there was some refactoring at some point08:52
arne_wiebalckmnasiadka: do you have the exact line already?08:53
ajyarpittau: do you know what's happening with those setup.cfg py version classifiers? 3.8, 3.9 remains? Or they need to be reverted/updated to 3.6,3.9?09:24
rpittauajya: I think the TC will take care of updating them back with py3.609:27
ajyaok, will wait09:28
rpittauthey still haven't merged the revert for the jobs yet09:28
mnasiadkaarne_wiebalck: exact line as in logs or code line? ;-)09:36
arne_wiebalckmnasiadka: code09:36
arne_wiebalckmnasiadka: there are quite some variable 'raid_device' places09:37
mnasiadkaarne_wiebalck: checking09:37
arne_wiebalckmnasiadka: ty09:37
jingvarguys09:38
jingvarTASK [bifrost-ironic-install : Download IPA kernel checksum file]09:38
jingvar<urlopen error [Errno 104]09:39
jingvarcurl https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-stable-xena.kernel.sha25609:39
jingvarThe document has moved <a href="https ....09:39
rpittaujingvar: thanks for the report, I'm checking now09:49
mnasiadkaarne_wiebalck:  get_commands_status /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py:34709:53
dtantsurmissing -L to curl?09:53
jingvarit is your playbook09:55
jingvarhow to switch to ubuntu based ipa?09:55
rpittauI'm updating the url directly so we also use the same one everywhere09:56
jingvarit is unexpected, IPA base related to Host OS09:58
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Update tarballs base url  https://review.opendev.org/c/openstack/bifrost/+/82036509:58
jingvarI'm using ubuntu focal as host os09:58
dtantsurwe don't publish ubuntu images (there are even legal reasons for that)09:59
dtantsuryou can build one yourself09:59
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Update tarballs base url  https://review.opendev.org/c/openstack/bifrost/+/82036509:59
jingvargot it10:00
arne_wiebalckmnasiadka: the line should be in the IPA (not Ironic), no? can you check the logs you have on the controller in /var/log/ironic/deploy for the node in question?10:02
mnasiadkaarne_wiebalck: sure, let me find the correct log from the past10:03
jingvarI'v failed again https://paste.opendev.org/show/811429/10:04
arne_wiebalckmnasiadka: in that dir, there should be archive files with the logs from the IPA10:04
mnasiadkaarne_wiebalck: actually no errors in the IPA logs...10:09
opendevreviewPierre Riteau proposed openstack/ironic master: Fix markup syntax  https://review.opendev.org/c/openstack/ironic/+/82036710:12
dtantsurTheJulia: I think I've finally got https://review.opendev.org/c/openstack/ironic/+/819620/ working \o/ Wanna have a look before I start finalizing it?10:20
* dtantsur jogging brb10:22
jingvarhow to pass extra vars via bifrost-cli10:31
jingvarI need a walkaround for missed ipa checksum10:33
rpittaujingvar: you can pass --extra-vars to bifrost-cli for that10:35
jingvarno help about this option10:36
jingvarFYI restart bifrost-cli under a user will fail with permissions10:39
zigoHi there. I'd like to test my own baremetal provisionning, and I was wondering if I could do it with OpenStack VMs. Would one of the people in the Ironic team able to answer my questions?10:43
zigoMy concern is booting-up VMs with PXE.10:43
zigoI know I can do my own DHCP in a VM, so that the other VMs would use it. But how can I make the slave VMs to boot on PXE ? OpenStack cannot do "-boot n" on the qemu command line, can it?10:44
zigoWhat's the way then?10:44
opendevreviewMerged openstack/ironic master: Fix markup syntax  https://review.opendev.org/c/openstack/ironic/+/82036710:52
jingvarzigo: Ironic can manage boot order and powering via driver10:54
zigojingvar: Yeah, but in a OpenStack CI env, how can you make nodes PXE boot? Is that even possible?10:55
jingvarthere is ipmi etc, never heard about Openstack Vm driver10:55
jingvarmy team did something like that but with manual powering10:56
jingvarlet me ask my team10:57
jingvar- name: "Download IPA kernel checksum file"     timeout: 300 - are you serious?11:00
zigoCurrently, I'm doing my tests in a big hardware machine with 1TB RAM, so I can do "-boot n" and then I use ipmi_sim from openipmi, which works great so far, but I was asked to try to make all of my project to run on OpenStack VMs, to make it more easy for people to contribute.11:02
jingvarI use nested VM11:02
jingvarzigo: manualy set boot via pxe in libvirts xmls11:09
zigoThat's what I'd like to avoid ... :/11:11
zigoAnyways, thanks for answering me.11:12
zigojingvar: Is it possible with OpenStack to boot on an iso image ?11:15
zigo(cause I'm booting on a live image over PXE...)11:16
jingvari think yes, I never used iso in openstack11:20
opendevreviewMerged openstack/ironic-python-agent-builder master: Replace genisoimage with xorriso  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/82012411:34
jingvarcan someone help with "/home/ubuntu/bifrost/playbooks/inventory/bifrost_inventory.py", line 22, in <module>     from oslo_config import cfg ModuleNotFoundError: No module named 'oslo_config'11:57
jingvarI must activte venv...11:58
ajyaTheJulia: left the comments for AccessError patch, up to you what you want to do with it12:01
janderssee you on Monday Ironic12:39
jandershave a great weekend everyone o/12:39
jingvarcan I pass authorized_keys via bifrrost?12:51
TheJuliaajya: replied, I think we need to treat that as a separate thing but yes, agree we should likely exit back out of basic auth12:53
TheJuliathe whole session fallback to basic was largely to support automatic cases on some bmcs where they don't actually really support session auth12:53
TheJuliaI don't think we actually do sessions, or did initially have sessions in redfish-tools too12:54
TheJuliaI think just falling back or exiting basic doesn't resolve the fact the connection data could be in a bad state. Getting the access error out to ironic at least allows us to go "oh, noes! lets start over!"13:01
TheJuliaajya: https://review.opendev.org/c/openstack/ironic/+/820077 <-- w/r/t this13:08
jingvarlast page of Bifrost doc - Add credentials to guest image -  can someone help to understand this?13:14
TheJuliajingvar: I'm training this morning13:17
*** amoralej is now known as amoralej|lunch13:39
dtantsurzigo: I think sushy-tools has an openstack driver14:05
dtantsurhttps://docs.openstack.org/sushy-tools/latest/user/dynamic-emulator.html#systems-resource-driver-openstack14:05
*** amoralej|lunch is now known as amoralej14:14
dmelladodtantsur:  TheJulia  \o/14:18
dmelladoGuess I found the first bug in bifrost14:18
dmelladoxD14:18
dmelladoon Copy the shim image to ESP it tries to get "/boot/efi/EFI/rhel/shimx64.efi"14:18
dmelladowhile that doesn't seem to exist on RHEL 8.414:18
dmellado/boot/efi/EFI/redhat/shimx64.efi14:19
dmelladoI'll submit a patch14:19
dmelladoI think it's quite obvious, but... would you like me to open a bug for this beforehand?14:19
TheJuliadmellado: you can, but it sounds super straight forward14:19
dmelladoI'll submit the patch directly then14:20
opendevreviewDaniel Mellado proposed openstack/bifrost master: Fix RedHat required defaults  https://review.opendev.org/c/openstack/bifrost/+/82038514:25
dtantsurdmellado: speaking of testing RHEL on CentOS :) good catch!14:26
dtantsurdmellado: we tend to add release notes for any user-visible changes, could you add one? https://docs.openstack.org/ironic/latest/contributor/faq.html#create-a-new-release-note14:27
dmelladodtantsur: sure, I'll amend that to the patch14:27
dtantsurdmellado: check https://docs.openstack.org/releasenotes/bifrost/unreleased.html for inspiration14:27
dtantsur(note that all notes are rendered together on that page)14:27
dmelladoI caught it just by chance then testing  on the 'bastion node'14:28
dmelladoto actually access the environment14:28
dmelladoit just 'worked fine' on my fedora xD14:28
dtantsuryeah, I think this patch is only wrong on RHEL14:28
dtantsurs/patch/path/14:28
opendevreviewDaniel Mellado proposed openstack/bifrost master: Fix RedHat required defaults  https://review.opendev.org/c/openstack/bifrost/+/82038514:36
dmelladodtantsur: there you go, your super-duper release note14:36
dtantsurThanks!14:37
dmelladoOMG, I missed git-review14:38
dmelladogithub-- xD14:38
dtantsurvery true14:38
dmelladodtantsur: TheJulia does bifrost support changing nginx default port in any way?15:00
dtantsurdmellado: you mean, from 8080 to something else?15:00
TheJuliai... I *think* it is a parameter15:01
dtantsurfile_url_port15:01
dmelladoyep, I'm just grepping 8080, just inc ase15:01
dmelladooh, found it15:01
* dmellado lazy15:01
dmelladoxD15:01
rpittaubye everyone have a great weekend! o/15:02
dmelladoenjoy rpittau 15:02
rpittauthanks!15:06
opendevreviewDmitry Tantsur proposed openstack/ironic master: All-in-one Ironic service with a local RPC bus  https://review.opendev.org/c/openstack/ironic/+/81962015:41
dtantsurTheJulia: I think this is ready ^^^15:41
dtantsurI'll try to use it in bifrost for a thorough testing15:41
TheJuliadtantsur: ack15:41
TheJuliathis training is kind of exhausting15:42
TheJuliaand getting up at 4 AM for 5 AM training is... rough15:42
dtantsurTheJulia: what are you learning? :)15:42
dtantsurugh15:42
TheJuliaagile methodologies (well, taking part in, being refreshed)15:42
dtantsursounds exhausting indeed15:43
TheJuliait all kind of blurs together for me since I've also done Lean Six Sigma in the past15:44
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Use the combined Ironic service instead of API+conductor  https://review.opendev.org/c/openstack/bifrost/+/82039015:55
dtantsurunfortunately, for rpc_transport=none I need to revive https://review.opendev.org/c/openstack/ironic/+/756354. I was trying to pretend it did not exist :)15:58
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] Do not use RPC continue_node_deploy/clean in agent_base  https://review.opendev.org/c/openstack/ironic/+/75635416:09
TheJuliadtantsur: doh16:13
dtantsurthe last attempt broke only software RAID. maybe this attempt will work? :)16:33
dtantsur(or will break everything else)16:33
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Use the combined Ironic service instead of API+conductor  https://review.opendev.org/c/openstack/bifrost/+/82039016:38
dtantsurhmm, why is ironic-inspector-tempest-partition-bios-redfish-vmedia broken?16:39
dtantsurNode 94818809-5cd2-4dbb-96ea-60f351166f5d did not appear in the baremetal introspection API after 90 seconds: Object not found16:40
dtantsurwut16:40
dtantsurFileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/syslinux/isolinux.bin'16:43
dtantsurhmm, it may be something from stevebaker[m]'s recent patches16:43
dtantsurokay, it's recommended to install isolinux before using it :)16:45
opendevreviewDmitry Tantsur proposed openstack/ironic master: Install isolinux on devstack  https://review.opendev.org/c/openstack/ironic/+/82039316:51
dtantsurstevebaker[m], TheJulia ^^16:51
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Use the combined Ironic service instead of API+conductor  https://review.opendev.org/c/openstack/bifrost/+/82039016:57
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Enable HTTP2 for Keystone when TLS is enabled  https://review.opendev.org/c/openstack/bifrost/+/82039417:01
NobodyCamGood Morning Ironic'ers...17:10
NobodyCamand OFC ...17:10
NobodyCamTGIF!17:10
dtantsurTGIF \o/17:16
dtantsuron this positive note I'm wishing everyone a great weekend :)17:16
dtantsuro/17:16
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] Document single process Ironic  https://review.opendev.org/c/openstack/ironic/+/82039617:19
NobodyCamhave a great weekend dtantsur 17:25
TheJuliaajya: hmm... I don't want to change too much but I'm getting this feeling we're going to need to do some retooling around session stuffs to fully handle the potential cases which may arise. Then again, with the ironic fix AccessError would also occur on the basic auth fail and that should raise all the way back up17:51
* TheJulia woners if we're also over-thinking this17:51
* TheJulia thinks some more on that17:51
TheJuliaand lunch17:51
*** amoralej is now known as amoralej|off18:03
opendevreviewMerged openstack/bifrost master: Fix RedHat required defaults  https://review.opendev.org/c/openstack/bifrost/+/82038519:35

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!