Tuesday, 2022-01-11

opendevreviewJulia Kreger proposed openstack/sushy master: WIP Fix session authentication issues  https://review.opendev.org/c/openstack/sushy/+/82132400:14
TheJuliaajya: I've revised  https://review.opendev.org/c/openstack/sushy/+/821324 to be more in line with what you were thinking in terms of early session creation. I'm not exactly thrilled with some of it with set_context on direct invocation, but... it was that or re-do like 23 unit tests. Maybe worth still going that route. LMK if the session creation is more in line with what you were thinking and I *think* that will short 00:16
TheJuliacircuit a number of potential failure cases with re-authing, but we'll still want to guard and fallback I guess.00:16
opendevreviewVanou Ishii proposed openstack/ironic master: [WIP] Deal with iRMC virtual media incompatibility  https://review.opendev.org/c/openstack/ironic/+/82379000:52
opendevreviewVerification of a change to openstack/ironic master failed: Use driver_internal_info methods for redfish driver  https://review.opendev.org/c/openstack/ironic/+/81850801:01
opendevreviewVerification of a change to openstack/ironic master failed: Use driver_internal_info methods for redfish driver  https://review.opendev.org/c/openstack/ironic/+/81850806:22
arne_wiebalckGood morning, Ironic!07:18
jandersgood morning arne_wiebalck o/08:28
arne_wiebalckhey janders, good morning! o/08:32
rpittaugood morning ironic! o/08:35
arne_wiebalckhey rpittau o/08:37
rpittauhey arne_wiebalck :)08:37
rpittaunew nodeset definition is working as expected, if anyone has a minute please review https://review.opendev.org/c/openstack/bifrost/+/819058 thanks!08:42
dtantsurstevebaker[m]: this is a concern on my side. I'd prefer not to bring in the whole privsep mechanism just for a convenient feature that duplicates the operator's tooling09:02
dtantsurand morning ironic09:02
stevebaker[m]Privsep only runs on startup, I've got it gracefully shutting down when copy operations have finished09:04
dtantsurstevebaker[m]: on start-up or on first usage? this makes a huge difference09:06
dtantsuri.e. if I don't use the copying feature, can I avoid it completely?09:06
dtantsur(also wow, a rare chance to meet you online!)09:06
stevebaker[m]Yes, starts on demand, ends at the end of start-up. 09:07
stevebaker[m](matrix client on my phone :)09:07
dtantsurha, nice :)09:07
dtantsurstevebaker[m]: I guess this is the part that confuses me: we're making the operator's life easier by copying files, but making it harder by requiring them to configure privsep :)09:08
dtantsurI honestly wonder why openstack doesn't just 'sudo' (and require the operators to configure sudoers)09:09
stevebaker[m]It's just one rootwrap entry, installed by deb/rpm (ok, bifrost needs it's own). Which is better than puppet/ansible/tripleo doing a bunch of distro specific copying09:15
dmelladoo/ Happy new year y'all09:17
dmelladoso... this was a laptop.. and so09:17
dtantsurstevebaker[m]: well, bifrost doesn't need the copying feature, so I'd rather keep it without privileged operations09:20
dtantsurhappy new year and hi, dmellado 09:20
dtantsurdmellado: in case you missed the gerrit notification: I've done a round of updates to the bifrost's "how to" docs09:20
dmelladojust checking my email09:21
dmelladolet me take a look09:21
stevebaker[m]The bifrost changes are up for review already, resulting in working grub network boot09:23
dtantsurstevebaker[m]: good for testing, but I'm basically -2 to adding privileged operations to bifrost's ironic, given that bifrost is literally playbooks to install ironic09:24
dtantsur(I feel less strongly about ironic itself since I recognize the case of "can I just launch /usr/bin/ironic")09:25
stevebaker[m]Ok, put your thoughts in the reviews when you get the chance. Let's discuss there09:28
stevebaker[m]Night all09:28
dtantsurgood night stevebaker[m] 09:28
dmelladodtantsur: ++ on the changes09:32
dmelladoalready merged anyway09:32
opendevreviewVerification of a change to openstack/ironic master failed: Ensure 'port' is up2date after binding:host_id  https://review.opendev.org/c/openstack/ironic/+/82381109:34
dmelladorpittau: so, when I was younger and saner, TL;DR, before xmas09:34
dmelladoyou told me about adding a user to the ipa image, mind  remembering me?09:34
rpittaudmellado: hi! happy new year! if you use DIB to create the image you can add ssh access as explained here https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html#ssh-access09:40
rpittauthe devuser element in particular allows to login from console using a local user09:41
dmelladoo/ checking ;)09:43
dmelladois there any straighforward way to put that on the bifrost install phase?09:45
dtantsurdmellado: I think there is an option for a list of elements09:45
dtantsur(documenting that would be awesome too)09:45
dmelladooh, dib_elements09:46
dmelladojust grepped it09:46
dmelladowould submit a patch for that later as well09:46
dmelladodib_elements: "ironic-python-agent-ramdisk {{ ipa_extra_dib_elements | default('') }}"09:46
dmelladoso adding ipa_extra_dib_elements  should work09:47
rpittaudocumenting a thing that I extensively use and it's super useful? yeah, might be a good idea :D09:53
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Change the TFTP directory to /var/lib/tftpboot  https://review.opendev.org/c/openstack/bifrost/+/82355210:01
dmelladoI assume I would also need create_ipa_image to be set to true10:02
dmelladoon target10:02
dtantsurle sigh: https://bugzilla.redhat.com/show_bug.cgi?id=203922410:04
rpittauoh that's why... ;/10:05
rpittaudmellado: yes you need to recreate the ipa image10:06
opendevreviewMerged openstack/ironic master: Use driver_internal_info methods for redfish driver  https://review.opendev.org/c/openstack/ironic/+/81850810:13
opendevreviewchandan kumar proposed openstack/ironic-python-agent-builder stable/wallaby: [DNM] testing dib-centos9 experimental job  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/82413610:22
opendevreviewMerged openstack/ironic master: Enable foreign keys in SQLite  https://review.opendev.org/c/openstack/ironic/+/82121910:36
arne_wiebalckafter the upgrade to wallaby, it seems pagination is broken for me: '{"code": 404, "title": "Not Found", "description": ""} (HTTP 404)' ... does that ring a bell for anyone?10:37
arne_wiebalckwith '--limit' it breaks exactly at the max_limit, so ''--limit=1000" works, "--limit 1001" breaks10:38
dtantsura regression in the wsme migration?10:54
dmelladowhy does it try to crete the ipa image with debian?10:59
dmelladowasn't that centos based?10:59
dtantsurdmellado: that's teh default for bifrost10:59
dmellado"_raw_params": "disk-image-create        -o /var/lib/httpboot/ipa            debian ironic-python-agent-ramdisk devuser\n"11:00
dtantsurI know, consistency, but bifrost's DIB code pre-dates our published images *by far*11:00
dmelladoI see, I'll modify that in the role11:00
dtantsurthere is a var for that11:00
dmelladooh, which one'11:00
dmelladoI have11:01
dmellado  dib_os_element: centos                                                                                                                                                                                                                                                                                                                                                                 11:01
dmellado                                                  11:01
dmellado  dib_os_release: 8-stream                                                                                                                                                                                                                                                                                                                                                               11:01
dmellado                                                  11:01
dmelladoon the target11:01
dmelladobut doesn't seem to be used for the ipa, maybe an specific one?11:01
dtantsurhmmm, these ones should affect it11:02
dtantsurdmellado: ah, we have ipa_dib_os_element in the playbook11:03
dtantsuryay for confusion11:03
rpittauyep, that11:03
dtantsurbut we don't have ipa_dib_os_release because consistency :)11:03
rpittaualways overrated11:04
dtantsurthis all needs cleaning up and documenting11:04
dmellado+1 to that xD11:04
rpittauI need to review my notes from the debian issue testing I got so far11:04
dmelladoI guess I may be the first one to try this let's say 'from scratch' in a while xD11:04
dtantsurit's not impossible. or maybe people just don't report their problem, it's also quite likely.11:05
rpittauI did that before holidays, but I think I lost most of the notes because of VM going foobar11:05
dtantsurthis is why we particularly appreciate people who can provide sensible reports :)11:05
rpittauthere's an issue with network interfaces not starting when debian boots11:06
rpittauI think the renaming of the interfaces interfere with that11:06
dmelladooh, I've seen that issue on renaming with 'new' kernels in a different scope11:06
dmelladoa pain11:07
rpittauso dhcp do not activate them11:07
arne_wiebalckdtantsur: pagination, seems like the server on the 2nd chunk gives 'None' instead of 'nodes' in the REQ11:10
arne_wiebalckdtantsur: the client takes that and gets a 40411:10
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Tighted permissions across the board  https://review.opendev.org/c/openstack/bifrost/+/82414411:24
dtantsurarne_wiebalck: sounds like a bug to me11:24
arne_wiebalckdtantsur: https://opendev.org/openstack/ironic/src/branch/stable/wallaby/ironic/api/controllers/v1/node.py#L2074 is missing a resource_url parameter11:25
arne_wiebalckso, it falls back to None11:25
iurygregorygood morning Ironic o/11:25
dtantsurmorning iurygregory 11:26
iurygregorydtantsur, o/11:26
arne_wiebalckdtantsur: adding "resource_url='nodes', " to the parameter list fixes it11:27
arne_wiebalckdtantsur: *hot-fixes it11:27
arne_wiebalckdtantsur: not sure this is the correct way to address this11:27
dtantsurdoesn't sound immediately wrong to me11:31
arne_wiebalckdtantsur:  the only other call to get_nodes_collections sets it before the call11:32
jandersSee you tomorrow Ironic o/11:32
arne_wiebalckdtantsur: https://opendev.org/openstack/ironic/src/branch/stable/wallaby/ironic/api/controllers/v1/node.py#L216211:33
dmelladorpittau: sigh, forgot dib_env_vars11:44
dmelladomaybe that'd be a good thing to add to the docs as well11:44
dmelladofor the dib_dev ssh key and so11:44
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Tighted permissions across the board  https://review.opendev.org/c/openstack/bifrost/+/82414412:10
dmelladodtantsur: is there a way to set http_proxy on the ipa?12:13
dtantsurdmellado: when building or when actually running12:14
dmelladoideally when building, I know using that dev_user I could just log in and set it up manually12:14
dmelladobut I'd rather avoid that12:14
dtantsurI mean, when the proxies should take effect?12:15
dmelladowhen downloading the deployment image12:15
dmelladoI found the issue there xD12:15
dtantsurdmellado: I think #5 of https://docs.openstack.org/ironic/latest/admin/drivers/ipa.html#using-proxies-for-image-download12:16
dtantsurwe may need to make it more prominent in the docs12:16
dmelladoyeah, 5 looks like the way to go12:16
dtantsurdmellado: if you could copy the parameters to https://docs.openstack.org/ironic/latest/user/deploy.html#populating-instance-information, would be really appreciated12:17
dmelladowill do as soon as I try it12:17
arne_wiebalckdtantsur: I am rolling out an updated package to confirm this works, then I'll submit a patch.12:22
arne_wiebalckdtantsur: (for the pagination issue)12:22
dmelladohmm so some misconfiguration or so it seems from my side12:28
dmelladoipa it's trying to get the image from which doesn't work 12:28
dmelladobut it gets it directly from the ip from the node12:28
dmelladodeployment_image.qcow2, I mean12:29
dmelladoJan 11 12:04:29 localhost.localdomain ironic-python-agent[1641]: 2022-01-11 12:04:29.809 1641 INFO ironic_python_agent.extensions.standby [-] Attempting to download image from
dmelladoJan 11 12:05:29 localhost.localdomain ironic-python-agent[1641]: 2022-01-11 12:05:29.871 1641 WARNING ironic_python_agent.extensions.standby [-] Unable to connect to, retrying. Error: HTTPConnectionPool(host='', port=8082): Max retries exceeded with url: /deployment_image.qcow2 (Caused by12:32
dmelladoConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7effa2de>12:32
dmelladochanging the ip to the latter one, and I can get it12:32
dmelladoprobably internal_ip12:42
dmelladoso, how can I change there12:50
dmelladoimage_source and config_drive ips12:50
dmelladoimage_source: "{{ deploy_url_protocol }}://{{ internal_ip }}:{{ file_url_port }}/{{ deploy_image_filename }}"12:51
dmelladoI set internal_ip to the proper one on the group_vars but it doesn't look to be fetched12:51
dtantsurdmellado: group_vars for what?12:52
dtantsurthis has to apply to baremetal, not target12:53
dmelladooh, baremetal, got it12:53
dtantsurwe should probably allow an explicit override to avoid changing internal_ip (which has many meanings)12:54
dtantsuractually, we have public_ip that could be used12:55
dmelladobut hey I learned on how to ssh into the ipa xD12:55
dmelladoIf you're up for a shared session on all these hitting the wall things12:56
dmelladofor the 'crossing fingers'12:56
dmelladoopendev in berlin12:56
dmelladoI'll be more than happy xD12:56
dtantsuropendev in berlin, mmmm...12:57
dtantsurassuming travelling is a common thing again12:57
dmelladoassumming that, yeah12:58
dmelladomaybe it's too much to assume12:58
dmelladogiven the current status...12:58
dtantsurI still hope :(12:58
dmelladoit'd be great, yeah...13:02
dmelladoI swear I'm growing tired of the whole covid thing....13:02
dmelladoand I had a massive mess13:02
dmelladoas they tested me as a false positive13:02
dtantsurouch :(13:31
dtantsurwell, at least it was false13:31
dmelladoyeah, but the dumb helath authorities are calling me nonstop13:31
dmelladoseems like they're not in sync with each other13:31
dmelladohmmm now configdrive it's not there?13:32
admiyodtantsur, I've been chasing the iPXE thing since we chatted last week.  I don't get what changed?  I'm assuming this is the deploy thing you were talking about https://docs.openstack.org/ironic/latest/admin/interfaces/deploy.html  but what happened to "Fetch your image from glance" that we were doing before?  If I enable iPXE in kolla, it messes up a bunch of stuff, and I really would rather not replace tftp with http right 13:36
admiyodo we want  --deploy-interface ramdisk  ?13:42
dtantsuradmiyo: Ironic works by serving the image to its ramdisk via HTTP through its HTTP server. If you don't have the HTTP server, well, the image cannot be served.13:42
dtantsurPlease don't experiment with the options you don't understand. It will just confuse you more.13:42
admiyodtantsur, I understand PXE.  I don't understand Ironic.13:43
admiyoWe had something that worked, and the world moved on.13:43
dtantsurI was referring to --deploy-interface, sorry13:43
dtantsurIf enabling the HTTP server in Kolla "messes up a bunch of stuff", please reach out to #openstack-kolla, it should not be happening13:43
dtantsurcc mgoddard 13:43
admiyoHowever, I am at the mercy of the technologies available, and that means Kolla.13:43
admiyoWell, I suspect that what has happened is that we are no longer really testing tftp based deploys that are then set up via Kolla13:44
admiyoI fully expect the Kolla folks to kick me back here.13:44
dtantsurThis team is not responsible for anything Kolla delivers. I'm trying my best, but my knowledge is very limited.13:44
admiyoYou are saygin that 100% now Ironic needs an internal http server beyond the existing API servers13:45
admiyoexisting as of my ancient install13:45
dtantsuryes, absolutely13:45
dtantsurI would also recommend against using TFTP unless you hit issues with iPXE, but that's another topic13:45
admiyothe Kolla way to set that up appears to be en enable ipxe, but that then changes the Ironic boot process13:45
admiyoi.e.  if i enable the ipxe server, the cleaning process no longer works. 13:46
admiyoWhich, to me says that a DHCP delivered option has changed or something13:46
admiyoI still get Grub, so something works, and I can wireshark it to figure out where it is getting that from13:46
dmelladodtantsur: "Deploy step deploy.write_image failed on node ef4bbe4a-3ecd-4bae-888e-52c406707206. Can't download the configdrive content for node ef4bbe4a-3ecd-4bae-888e-52c406707206 from ''. Got status code 404, response body <html>"13:47
dmelladothat configdrive shouls show up at /httpboot13:47
dmelladoIIUC, isn't it?13:47
dtantsuradmiyo: they do, indeed, modify the default boot file to use iPXE: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/ironic/defaults/main.yml#L25713:47
dtantsurdmellado: can you curl this literal thing?13:48
dmelladoI can't13:48
dmelladothat's why I'm asking13:48
dmelladoeven from the host, not just ipa13:49
opendevreviewVerification of a change to openstack/ironic master failed: Ensure 'port' is up2date after binding:host_id  https://review.opendev.org/c/openstack/ironic/+/82381113:49
dtantsurdmellado: wrong port? firewall?13:49
dmelladonope, just not there13:49
dtantsurthe file?13:50
dmelladoyeah, that's the content of my httpboot13:50
dmelladoso not sure on when that configrive is generated13:50
dmelladomaybe I'll just have to reinstall13:51
* dtantsur pings #openstack-kolla re testing13:51
dtantsurdmellado: the deployment playbooks includes a role called bifrost-configdrive-dynamic13:52
dtantsurwhich is supposed to do that13:52
dtantsuryou've changed the httpboot location, haven't you?13:52
dmelladoI have13:52
dmelladoso it's on the old location13:53
dmelladoawesome xD13:53
dtantsurdmellado: you need to pass the same parameter everywhere13:53
dmelladoso I have to change httpboot location in baremetal settings13:53
dtantsurdmellado: or just update your bifrost: my change has merged13:53
dmelladorebasing then13:55
dtantsuradmiyo: will applying https://opendev.org/openstack/kolla-ansible/commit/411668ea5a05e8182d80be63501e2f7fa1882345 fix your problem?14:06
admiyoWell, no, I think.  I enabled that explicitily, and it changes the DHCP options, like you said, so that would just hide what is breaking things on me, I think.  I can enable ipxe already, but that does not seem to skip the rest of the work I have to do....let me read further and confirm14:08
Dhuldev_Valekariurygregory:Hi Gregory, Happy New Year 2022!, While running existing ironic tempest test case "test_introspection_basic" against the dell servers we are facing assertion  failure while comparing memory properties of actual servers against the baremetal flavor values post inspection. any suggestions?14:08
dtantsuradmiyo: I mean, you can apply this patch and keep ipxe disabled14:08
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Follow up to "Run bifrost on CentOS Stream 9"  https://review.opendev.org/c/openstack/bifrost/+/82418614:08
iurygregoryDhuldev_Valekar, do you have a link for the failures?14:09
admiyoBasically, if I read this correctly, the set of changes have made sure that you cannot deploy Ironic with tftp via Kolla.14:09
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Follow up to "Run bifrost on CentOS Stream 9"  https://review.opendev.org/c/openstack/bifrost/+/82418614:09
admiyoand, if that is the case, I can deal with it.14:09
admiyo"Note that due to a limitation in Kolla Ansible, PXE and iPXE cannot be used14:10
admiyotogether in a single deployment."14:10
dtantsuradmiyo: I don't understand how you're making this conclusion. The kolla team has simply missed the fact that Ironic changed its defaults in Wallaby, so they fixed the issue in Xena+.14:10
dtantsuryes, this is correct. but it does not mean you cannot use PXE.14:10
dtantsurthe patch I linked will enable the HTTP container without changing the boot process14:11
admiyodtantsur, it means that I would have to set up my own HTTP server, as the one that Kolla provides does too much14:11
admiyoNot how I read it. 14:11
dtantsurI don't understand how you derive "PXE cannot be used" from "PXE and iPXE cannot be used together"14:11
mgoddardadmiyo: hi14:12
Dhuldev_Valekariurygregory: we are running tempest test case on local setup, I can provide log file separately, would that helpful? 14:12
mgoddardyou are right, that currently the HTTP server is tied to iPXE, which it didn't necessarily have to be, since it's required for the direct deploy interface14:12
dtantsurmgoddard: I think https://opendev.org/openstack/kolla-ansible/commit/411668ea5a05e8182d80be63501e2f7fa1882345 should be in wallaby as well because that's when we changed the default deploy interface14:13
mgoddarddtantsur: that probably makes sense14:13
admiyoHeyo.  I am OK with moving to HTTP from tftp if that is the case.  I'd rather not apply a Xena patch to Wallaby, when we went with Wallaby in the first place to avoid the chaotic frindge of the  bleeding edge!14:13
opendevreviewAija Jauntēva proposed openstack/ironic master: Fix validating input for redfish update_firmware  https://review.opendev.org/c/openstack/ironic/+/82370114:14
iurygregoryDhuldev_Valekar, would be good since the test_introspection_basic has a few tests14:14
mgoddardadmiyo: ipxe is the way14:15
mgoddarddtantsur, admiyo: we probably did not see this in kayobe because we (regrettably) provide our own defaults for ironic drivers, and iscsi stayed the default14:17
mgoddardanyhow, here's the backport: https://review.opendev.org/c/openstack/kolla-ansible/+/82415514:17
admiyoIs iSCSI still there?14:18
admiyowhich value is it that changed the default?14:20
mgoddardiscsi is still available in wallaby, just no longer the default14:20
admiyowhat value am I looking for?14:21
admiyother [iscsi] section does not even exist in the xena ironic default file.14:22
admiyo#default_deploy_interface = <None>14:22
admiyothat is from the wallaby default config.14:23
admiyo#default_deploy_interface = iscsi14:23
admiyowe have it commented out, so I should uncomment it14:23
* TheJulia wipes sleep from her eyes14:25
TheJuliaGood morning everyone14:25
jamesdentongood morning14:25
mgoddardadmiyo: if you use iscsi, you won't need the http server14:26
mgoddard(assuming you continue to use pxe)14:26
iurygregoryDhuldev_Valekar, try checking your tempest conf if the reference for the ONF.compute.flavor_ref matches the flavor you created for your node etc14:27
opendevreviewArne Wiebalck proposed openstack/ironic master: Set resource_url when getting all nodes  https://review.opendev.org/c/openstack/ironic/+/82419014:29
arne_wiebalckdtantsur: ^^ (I did not yet manage to spot the difference to victoria, so some pieces of the picture are missing)14:31
Dhuldev_Valekariurygregory: https://paste.openstack.org/raw/812032/ 14:33
arne_wiebalckdtantsur: there must be some fundamental difference since the API request times have gone down by at least a factor of 5 https://usercontent.irccloud-cdn.com/file/0V0kfRys/wallaby_pagination_speedup.png14:33
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Tighten permissions across the board  https://review.opendev.org/c/openstack/bifrost/+/82414414:34
dtantsurarne_wiebalck: well, TheJulia has done a lot of work in this area14:34
TheJuliaarne_wiebalck: is that live from your upgrade today?14:35
arne_wiebalckTheJulia: it is14:35
TheJuliaarne_wiebalck: that makes me soooo happy14:35
TheJuliaarne_wiebalck: blog post?14:35
arne_wiebalckTheJulia: see, we had 3 bands in req times before (1000 nodes, rest of groups, other calls) ... they seem to be gone :)14:36
arne_wiebalckTheJulia: this looks like a massive improvement!14:36
arne_wiebalckTheJulia: thanks for your work! :)14:37
* dtantsur dances14:37
TheJuliaarne_wiebalck: has the sporatic grouping after 11:00 and before 12:00  and again around 14:00 been identified yet?14:37
arne_wiebalckTheJulia: the upgrade was much longer than expected due to https://review.opendev.org/c/openstack/ironic/+/82419014:38
iurygregoryDhuldev_Valekar, in your setup you have nova running ? Can you share the tempest.conf?14:38
arne_wiebalckTheJulia: so, the upgrade did not finish at 11, but only at 2pm or so14:38
arne_wiebalckTheJulia: and then I restarted everything to randomise the startup times14:38
dtantsurarne_wiebalck: could you find a way to unit-test your fix?14:39
dtantsurit's so easy to regress there14:39
arne_wiebalckdtantsur: yeah, I was expecting that question :-D14:39
TheJuliaI'm surprised we didn't catch that but I don't remember how I did pagination testing anymore14:39
arne_wiebalckdtantsur: tried to sneak through :-D14:39
opendevreviewAija Jauntēva proposed openstack/ironic master: Add more sources to redfish firmware upgrade  https://review.opendev.org/c/openstack/ironic/+/82278114:39
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Add CentOS Stream 9 keystone integration job  https://review.opendev.org/c/openstack/bifrost/+/82419314:40
arne_wiebalckTheJulia: I don't understand the difference to Victoria (and why noone else sees this ... unless we are the only with Nova, a max_limit which is not 0 and on Wallaby+)14:40
opendevreviewAija Jauntēva proposed openstack/ironic master: Add known issue for iDRAC Swift firmware update  https://review.opendev.org/c/openstack/ironic/+/82278214:41
TheJuliayou I'm surprised you even found it and I've not completely wrapped my head around it14:42
arne_wiebalckTheJulia: non-working pagination breaks nova immediately14:42
TheJuliabut you are likely and early adopter being one of the larger deployments to move faster14:43
arne_wiebalckTheJulia: which is another thing which surprised me14:43
TheJuliaI wonder if it is python-ironicclient vs openstacksdk14:43
arne_wiebalckclients have not changed14:43
TheJuliamaybe we've got pagination implemented differently14:43
TheJuliadid you lookup when they swapped the node list out to openstacksdk?14:43
Dhuldev_Valekariurygregory : tempest.conf https://paste.openstack.org/raw/812033/14:43
arne_wiebalckTheJulia: no ...14:44
admiyomgoddard, I assume I am going to need all of the [iscsi] section now too?14:45
iurygregoryDhuldev_Valekar, if you check the specs for this flavor 83c4f114-ac7c-46a9-af3b-7aa28487a2f4 it will probably show 8192 so this would cause https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/test_introspection_basic.py#L27-L28 to fail14:46
dtantsuradmiyo, mgoddard, note that going back to iscsi pretty much sets you up for the same pain in the next release14:47
TheJuliaarne_wiebalck: 3 years ago14:47
arne_wiebalckTheJulia: we are little behind, but not that much :)14:48
TheJuliaarne_wiebalck: so, ussuri. I guess I focused on how python-ironicclient did it and didn't look at the sdk14:48
mgoddardadmiyo: shouldn't14:49
TheJuliabecause I likely assumed it to be the same which was bad of me14:49
Dhuldev_Valekariurygregory : Yes, in the flavor spec it is showing memory as 8192, but post introspection we are getting actual memory value which is 196608 and hence assertion is failing ( 196608 != 8192 )14:50
opendevreviewAija Jauntēva proposed openstack/ironic master: Fix prepare ramdisk for 'wait' states  https://review.opendev.org/c/openstack/ironic/+/82331114:51
iurygregorythe workaround I see would be create a flavor that matches the HW you want to use as a node and set in the tempest.conf14:52
TheJuliaarne_wiebalck: commented on the patch with an idea on how to test it14:52
arne_wiebalckTheJulia: not sure I follow ... the sdk is missing some changes to deal with your improvements as you only did them in python-ironicclient?14:52
TheJuliaarne_wiebalck: no, your change is because the link was wrong for the next iteration correct?14:53
arne_wiebalckTheJulia: correct14:53
arne_wiebalckTheJulia: the initial REQ is good, next is broken (has 'None')14:53
TheJuliaso using the links reference where I don't think python-ironicclient actually uses the link14:54
Dhuldev_Valekariurygregory : Ok, Thanks!14:54
TheJuliabecause the pattern is "last returned is the marker for next"14:54
TheJuliaat least, that is what is in my brain14:54
TheJuliaanyway, it is obvious we're lacking a test to ensure the next is returned properly14:56
TheJuliawe need to fix that14:56
admiyodtantsur, I want to get this working as is, so I have a production server I can modify, and then get a dev server set up that I can experiment with, including much cruft extraction14:56
arne_wiebalckTheJulia: oh, I see14:56
admiyoI'm thinking we go bifrost next, so I'm willing to break things14:56
arne_wiebalckTheJulia: thanks for your comment, will have a look later (meeting now)14:57
TheJuliaarne_wiebalck: ack, okay14:57
TheJuliaarne_wiebalck: may I tweet that image you shared?14:57
dmelladohmmm error by error xD14:58
dmelladoDeploy step deploy.prepare_instance_boot failed: Failed to install a bootloader when deploying node ef4bbe4a-3ecd-4bae-888e-52c406707206. Error: No partition with UUID 0x86d06aeb found on device /dev/sda14:58
dmelladoI assume that it failed to partition the disk somehow?14:58
arne_wiebalckTheJulia: we will restart nova-compute as well, just to be sure ... I would wait another couple of hours so we do not have to correct ourselves :)14:59
TheJuliaarne_wiebalck: sounds good15:00
arne_wiebalckTheJulia: but, then, yes sure; we can also add this as a comment to your performance blog posts if that fits somewhere15:00
TheJuliaI'd just revisit it, tbh15:01
dmelladorpittau: that does ring a bell? I feel like 'almost there' xD15:04
opendevreviewRiccardo Pittau proposed openstack/ironic bugfix/18.1: Explicitly pin upper sushy, reduce job count  https://review.opendev.org/c/openstack/ironic/+/82092715:05
dtantsurdmellado: what kind of the image you're using? I wonder if it's a breakage after mgoddard's patch to raise exceptions for whole disk images.15:14
mgoddarddtantsur: it could be15:15
mgoddardor it could be catching a legit error :)15:16
mgoddarde.g. using a non-UEFI image in UEFI mode15:16
dtantsurif so, the error messages leaves a lot to be desired :)15:17
dtantsurbut good point re UEFI15:17
*** dking is now known as Guest24015:17
mgoddardI used guestfish to verify the partition layout of the image15:18
dtantsuror virt-filesystems --long (which is the same thing in one CLI)15:18
*** Guest240 is now known as dking15:18
admiyo2022-01-11 07:08:30.024 8 ERROR ironic.conductor.base_manager [req-b1634a6d-44bb-4236-bb22-089a567ac315 - - - - -] Failed to register hardware types. Could not find the following interface in the 'ironic.hardware.interfaces.deploy' entrypoint: iscsi. Valid interfaces are ['direct'].: ironic.common.exception.InterfaceNotFoundInEntrypoint: Could not find the following interface in the 'ironic.hardware.interfaces.deploy' 15:28
admiyoentrypoint: iscsi. Valid interfaces are ['direct'].15:28
admiyoThat is not very friendly15:28
dtantsuradmiyo: if you have default_deploy_interface, you need to make sure it's in enabled_deploy_interfaces15:30
admiyoAh, right...15:31
admiyoenabled_deploy_interface = iscsi,direct15:31
dmelladodtantsur: I'm just using a centos one15:33
dmelladobut I noticed that the disks are actually within a raid15:33
dmelladowhich I didn't take into consideration at all15:33
dmelladocould that affect?15:33
dmelladoI was reading https://docs.openstack.org/ironic/latest/admin/raid.html15:33
dtantsurdmellado: only if you don't have drivers for that in your IPA image15:34
admiyodtantsur, any other way it could be disabled?15:34
dtantsurdmellado: try reverting https://opendev.org/openstack/ironic/commit/ca6499b62c47b6588b2382f442dbc690140453d5 and seeing if it helps15:34
dtantsuradmiyo: are you looking in the resulting ironic.conf or somewhere in kolla?15:35
dmelladodtantsur: ack, so I owuldn't need to define the raid thing15:35
opendevreviewAija Jauntēva proposed openstack/ironic master: Fix Redfish RAID for non-immediate controllers  https://review.opendev.org/c/openstack/ironic/+/82331215:35
dtantsurdmellado: only if you want to actively manager RAID15:35
dkingCan anybody think of a reason why Ironic-API can give a 200 for a GET request for a node without any problem, including getting the power state, but when given a PUT for that same node for /states/power, it returns a 404? The node and its power state are in the database.15:35
arne_wiebalckTheJulia: nova for ironic is restarted, API times still look much improved compared to before the upgrade :)15:35
dmelladodtantsur: for the time being I couldn't care less15:35
dmelladodtantsur: how can I revert a commit in bifrost? Is there a way to 'protect' the repos under /opt/stack15:36
dmelladoFWIW I assume it'll be recloned15:36
dtantsurdmellado: for the experiment of yours, just revert it in /opt/stack/ironic and do not re-install, just restart ironic-conductor15:37
dtantsurdking: sounds puzzling. sometimes 404 means a too low microversion, but likely not for /states/power15:37
dtantsuranything in the logs?15:37
admiyoI bet enabled_deploy_interface  should be enabled_deploy_interfaces15:37
* admiyo trys that15:37
dtantsuradmiyo: true! good catch15:37
opendevreviewVerification of a change to openstack/ironic master failed: Ensure 'port' is up2date after binding:host_id  https://review.opendev.org/c/openstack/ironic/+/82381115:37
dkingdtantsur: Thanks for the response. I just noticed that the DB had an error a while ago. I'm going to see if it's responding properly. However, if there were a problem, would the API be able to return information on GET?15:40
dtantsurdking: GET in our API normally works by talking to the DB directly, if that's your question15:41
dtantsuralthough a failure there should result in HTTP 500, not 40415:41
dkingThat's what I thought. And nevermind, that error was unrelated. Which I suppose I could confirm by everything else working fine.15:41
dkingYeah, that's what I would have thought.15:42
dtantsurdking: is the power state problem consistent and applies to all nodes?15:43
dkingdtantsur: I suspect so. It's been two so far. Of course, I'm not wanting to turn a bunch of servers off to test, but so far it's been consistent. I could probably find a couple more to test, though.15:46
dmelladodtantsur: same issue15:49
dkingdtantsur: Yes, so far it's consistent. I've tried several nodes now.15:49
dmelladoshould I clean the node before redeploying, or that wouldn't make a change15:49
dtantsurshouldn't change much15:49
dtantsurdmellado: ironic downloads logs from the ramdisk, I think in /var/log/ironic/deploy15:50
dtantsuryou need to check that for traces of the root cause. if you have no clue, share it with me.15:50
dtantsurdking: anything in the logs?15:50
admiyolooks like that was it...testing seems to be working.  GOtta go through the whole round trip, but if this is it, I will be very happy.  First beer for dtantsur and dmellado next face-to-face is on me15:51
dkingdtantsur: Well, not anything I can see except for the HTTP statuses. However, I just discovered that it's intermittent.15:54
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Follow up to "Run bifrost on CentOS Stream 9"  https://review.opendev.org/c/openstack/bifrost/+/82418615:54
dkingI tried a bunch of them again, and because I was doing it often enough, I noticed that sometimes, it works. Which is even more puzzling to me.15:54
dtantsurdking: if hacking the code is an option, you can catch the exception and output traceback to see where it's coming from15:56
dkingWell, even though, fortunately, this is a development instance, it's still in use, so I'm not too quick to want to do that, but I might step through the code. It seems to me that their would have to be something very off to give a 404 for a node that's in the DB.15:58
dmelladodtantsur: https://paste.openstack.org/show/812035/16:00
dmelladoI see some errors there on finding the partition 16:00
dmelladobut lack the context16:00
dmelladooh, found something16:01
dmelladoJan 11 15:46:45 localhost.localdomain ironic-python-agent[1649]: 2022-01-11 15:46:45.870 1649 DEBUG ironic_lib.utils [-] Command stdout is: "KNAME="sda" MODEL="PERC H730P Adp  " SIZE="479559942144" ROTA="1" TYPE="disk" UUID="" PARTUUID=""16:01
dmelladothat has no PARTUUID16:01
dmelladogod knows why16:01
dtantsurdmellado: "Boot mode mismatch: target boot mode is uefi, current boot mode is bios" is a comparably large issue IMO16:01
dmelladoouch! 16:01
dtantsuryou're using the IPMI driver, right? have you set legacy boot yourself?16:02
dmelladoI'm using IPMI16:02
dtantsurif that's not intended, I'd recommend you switch your machines to UEFI16:02
dmelladoand I don't recall anything about legacy anything16:02
dtantsurunfortunately, we cannot do it for you when using IPMI16:02
dmelladoI'll switch then right away16:02
dmelladoshould be on IDRAC bios, I assume16:02
dtantsurthis will likely make the issue go away, even though the bug seems real16:02
dtantsurdmellado: could you send me (maybe via email if it's too large) the whole journal for investigation?16:03
dtantsurwith some context, i.e. which parameters you used for DIB and what command you're running?16:03
dtantsur(better as an email so that I don't forget)16:03
dmelladodtantsur: sure, I'll drop you an email16:03
dmelladoparameters, commands, and all disturbing things xD16:04
dmelladothanks in advance16:04
dtantsurdmellado: once you do the initial onboarding, you can try switching the driver to redfish, it will do the boot mode management for you16:04
dmelladoin any case, just switched to UEFI16:05
dmelladogod only knows why it was set to bios...16:05
dmelladodtantsur: sent!16:14
dmelladothanks  in advance!16:14
dtantsurthank you! will likely only get to it tomorrow, I hope the switch to UEFI helps16:14
dtantsurokay, thinking about it: your image may not be UEFI compatible16:14
dtantsurdmellado: if you see ^^^, you need to add https://docs.openstack.org/diskimage-builder/latest/elements/block-device-efi/README.html to elements16:15
dtantsurTheJulia: I wonder if we should update bifrost to use ^^ by default16:16
dtantsurgiven that we use UEFI by default16:16
dmelladodtantsur: same way I added dev-user, isn't it?16:18
dmelladoipa_extra_dib_elements: devuser block-device-efi16:19
dtantsurhold on, sorry16:19
dtantsurdmellado: it's about the instance image, not IPA16:19
dtantsurare you building your instance centos image with DIB as well?16:20
dmelladooh, instance, I see16:20
dtantsur... more things to document ...16:20
dmelladolol, sorry to give you more work16:20
dmelladohow can I add elements to that?16:21
dtantsurno worries, it's a useful exercise16:21
dtantsurdmellado: just extra_dib_elements16:21
dtantsurif you could post at least a draft documentation patch, would be awesome16:21
dmelladodtantsur: I'll do16:22
dtantsurI'm afraid I'll forget what we've been talking about16:22
dmelladoI think we covered quite a few things16:22
dmelladomaybe not your easiest start deployment16:22
dmelladoanyways, gotta run, I'll leave that running and bother you some more tomorrow 16:23
dmelladothanks again!16:23
dtantsursure, see you tomorrow16:23
*** sshnaidm is now known as sshnaidm|afk16:23
bkranendonkhi folks, is there a way to customize the used nameserver in the IPA image?16:42
bkranendonkcant find info about it16:42
bkranendonkcant supply dns using dnsmasq either it seems (im using kolla)16:44
TheJuliait wouldn't be able to be a parameter on the command line for the agent to trigger and act upon as it would be too late, it would need to be in the ramdisk as one would normally customize it. Or DHCP can offer it...16:44
TheJuliawhy not?16:44
TheJuliajust not in their template?16:44
bkranendonkwell yeah I can ofc :D, but not without editing the template hehe16:45
TheJuliaI'd edit the template and toss a patch up for them16:45
TheJuliathat seems like a thing that makes sense to be able to set16:45
bkranendonkyep will do16:45
TheJuliano problem, sorry you ran into that :(16:45
TheJuliaajya: do you happen to be around?16:46
TheJulianvmd, I see you reviewed the patch16:47
arne_wiebalckbye everyone, see you tomorrow o/17:21
rpittaugood night! o/17:30
iurygregorybye rpittau o/17:31
rpittauTheJulia iurygregory one last thing for today, https://review.opendev.org/c/openstack/ironic/+/820927 is now passing CI so we can merge stuff to 18.1 again, it may need some adjustments though, please have a look when you got a chance :)18:27
TheJuliarpittau: thank you for picking that back up18:28
rpittauno problem! :)18:28
TheJulia#success https://twitter.com/ArneWiebalck/status/148093719833385369618:28
opendevstatusTheJulia: Added success to Success page (https://wiki.openstack.org/wiki/Successes)18:28
TheJuliaas much as I would like to see the wiki gone18:37
opendevreviewMerged openstack/ironic master: Ensure 'port' is up2date after binding:host_id  https://review.opendev.org/c/openstack/ironic/+/82381119:39
stevebaker[m]good morning ironic20:06
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Add CentOS Stream 9 keystone integration job  https://review.opendev.org/c/openstack/bifrost/+/82419320:23
TheJuliastevebaker[m]: https://github.com/openstack/sushy/blob/master/sushy/connector.py20:46
opendevreviewMerged openstack/ironic-python-agent-builder stable/xena: Replace genisoimage with xorriso  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/82360121:00
opendevreviewVerification of a change to openstack/ironic bugfix/18.1 failed: Explicitly pin upper sushy, reduce job count  https://review.opendev.org/c/openstack/ironic/+/82092721:10
opendevreviewJulia Kreger proposed openstack/python-ironic-inspector-client master: Disable keep-alive  https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/82424722:07
opendevreviewJulia Kreger proposed openstack/python-ironic-inspector-client master: Disable keep-alive  https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/82424722:08
TheJuliahjensas: stevebaker[m]: ^^^22:09
hjensasTheJulia: I still se the error with that patch22:20
TheJuliahjensas: can you paste the error real quick22:21
TheJuliaits working on paras's env22:21
TheJuliabut maybe different things going on22:21
TheJuliainspection is sure not working overall though22:21
hjensasUnexpected exception for ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))22:21
TheJuliahjensas: ahh yeah22:22
TheJuliahjensas: any exceptions in the service logs?22:22
TheJuliaso paras's environment is also missing br-baremetal22:24
hjensasTheJulia: no errors in ironic-inspector logs - this is a snippet showing the rule delete - https://paste.opendev.org/show/812053/22:25
TheJuliahjensas: I wonder if inspector is returning a Content-Length field22:30
hjensasTheJulia: I think it does'nt, that would show up when trying to perform the action directly using curl right?22:41
hjensas$ curl -g -i -X delete -H "User-Agent: openstacksdk/0.55.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8" -H "X-Auth-Token: $TOKEN_ID" -H "X-OpenStack-Ironic-Inspector-API-Version: 1.0"                                                                22:41
hjensasHTTP/1.1 204 NO CONTENT22:41
hjensasContent-Type: text/plain; charset=utf-822:41
hjensasX-Openstack-Ironic-Inspector-Api-Minimum-Version: 1.022:41
hjensasX-Openstack-Ironic-Inspector-Api-Maximum-Version: 1.1722:41
hjensasDate: Mon, 10 Jan 2022 19:16:18 GMT22:41
hjensasTransfer-Encoding: chunked22:41
hjensascurl: (18) transfer closed with outstanding read data remaining22:41
TheJuliait *must* reply with a content length if memory serves22:42
TheJuliaif transfer encoding gets set22:42
TheJuliaif you can get it to dump the entire post body, that migh ebe helpful22:42
TheJuliahjensas: so it is inspector22:49
TheJuliamaybe not22:49
TheJuliaflask on train22:49
TheJuliadeja vu22:50
TheJuliastarted in 10.3.022:51
TheJuliaThat is in it22:52
TheJuliabecause httpd proxies were barfing on the exhcnage22:52
stevebaker[m]TheJulia: I tried your change, it didn't help https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/824247/2/ironic_inspector_client/common/http.py#21022:55
stevebaker[m]TheJulia: you did a manual curl and even that complained?22:56
TheJuliano, I just re-ran paras's command which was failing22:59
TheJuliabut hjensas did23:00
stevebaker[m]I just did it23:00
TheJuliathe problem, ultimately is, aiui23:00
stevebaker[m]biab, out to lunch23:00
TheJuliaIf the client says "give me chunked transfers" it pasts the header on, the rfc if memory serves basically says the server has to honor that and reply with that23:01
TheJulia*but* forcing the request to be octet stream should change the request to not be a chunked transfer23:01
stevebaker[m]how about setting an explicit Content-Length: 0 for 204 responses?23:03
TheJuliaThat seems like a possibility23:03
TheJuliaI don't think 204s require it23:03
TheJuliabut why not23:03
TheJuliaif you can just hack it in on the reply, I think that would help23:03
TheJuliawell, locally where you can test/reproduce it23:03
TheJuliabtw, I put the firewall rule change up23:04
stevebaker[m]I'll try it after lunch, if you don't get there first23:04
TheJuliaI need to go start dinner in a little bit23:04
stevebaker[m]to the food!23:04
TheJuliaand call my uncle back east who has tried to call me twice23:04
hjensashttps://everything.curl.dev/http/response#chunked-transfer-encoding - It seems there should'nt be a Content-Length if it is Chunked-Transfer ?23:04
TheJuliathat is also correct23:05
TheJuliaand servers are supposted to honor it23:31
TheJuliaso I bet what is happening is we don't have an emtpy carriage return23:31
TheJuliaand actually it is new line and carriage return, based upon the standard as the delimiter23:36
TheJuliaand I think dmitry is going to consider it a braking api change23:36
hjensasTheJulia: I tried  response = flask.make_response('\n\r', code), but curl still gives me the "(18) transfer closed with outstanding read data remaining" error.23:50
TheJuliahjensas: \r\n23:56
hjensasTheJulia: still same error with \r\n23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!