Friday, 2022-03-25

opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add OpenConfig classes for LACP  https://review.opendev.org/c/openstack/networking-baremetal/+/83516103:12
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Device management driver iface  https://review.opendev.org/c/openstack/networking-baremetal/+/83517003:13
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Device management driver iface  https://review.opendev.org/c/openstack/networking-baremetal/+/83517003:52
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Device management driver iface  https://review.opendev.org/c/openstack/networking-baremetal/+/83517004:01
rpittaugood morning ironic! Happy Friday! o/05:52
arne_wiebalckGood morning rpittau and Ironic!07:07
rpittauhey arne_wiebalck :)07:07
jandersgood morning rpittau arne_wiebalck and Ironic o/07:34
jandersHappy Friday07:34
rpittauhey janders :)07:34
arne_wiebalckhey janders o/07:36
rpittauTheJulia: I've updated the ironic release in the ironic-website https://github.com/OpenStackweb/ironic-website/pull/4807:42
opendevreviewMerged openstack/ironic-tempest-plugin master: Remove get_remote_client & check_vm_connectivity methods  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/83412408:40
dtantsurmorning folks! finally Friday!08:58
rpittaugood morning dtantsur :)08:59
*** mgoddard- is now known as mgoddard09:11
dtantsurwould love some reviews on https://review.opendev.org/c/openstack/ironic/+/835098 please09:11
opendevreviewMerged openstack/ironic-tempest-plugin master: Remove create_timestamp method  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/83412709:20
rpittaummm I thought I reviewed it earlier, forgot to press SEND :P09:37
dtantsur:)09:39
opendevreviewMerged openstack/ironic-tempest-plugin master: Remove create_floating_ip & get_server_ip methods  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/83412910:54
opendevreviewMerged openstack/ironic-tempest-plugin master: Remove _get_router & _create_router methods  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/83415110:54
opendevreviewMerged openstack/ironic-tempest-plugin master: Remove _create_network & _get_server_port_id_and_ip4 methods  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/83442411:15
opendevreviewMerged openstack/ironic-tempest-plugin master: Remove setup_clients method  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/83442511:15
opendevreviewMerged openstack/ironic master: [doc] Document dynamic network burn-in  https://review.opendev.org/c/openstack/ironic/+/83507911:15
iurygregorygood morning Ironic13:00
admiyogood morning13:06
admiyoOK, debugging a PXE failure again. I can see the DHCP request via TCP dump on the controller.  I don't see the response, but I think that is due to my tcpdump filter.    The cleaning node does not see the response.  What should happ[en?  I assume conductor somehow tells the dnsmasq container that there is a new node.  How?  Is it a file that gets put somewhere in the dnsmasq containers /var/lib... _data directory or 13:22
admiyosomething else?13:22
TheJuliaadmiyo: so kind of depends on your configuration13:28
TheJulianeutron integrated?13:28
TheJuliaalso, good morning everyone13:28
TheJuliawrt ironicbaremetal.org, is anyone going to do a summary of the changes?13:29
TheJuliaplease say yes13:29
admiyoGoooood Morning TheJulia 13:30
admiyoneutron is integrated, yes13:31
admiyoTHis is a kolla deploy, with Nova13:31
admiyoso I got a VM to deploy and I can talk to it, but that is an all-in-one, so simple on the controller13:31
TheJuliaSo basically what happens is when we move the node to cleaning, we setup records for PXE booting the node via the extra dhcp options on the port13:32
TheJuliaso if you do openstack port list and Identify the port for the machine (use the mac address!)13:32
TheJuliayou should see options there13:32
TheJulianeutron handles all of that and when it fails, we remove the records13:32
admiyoThere are 5 ports for that one mac address.  Does that mean they are not getting cleaned up on a failuer?13:33
TheJuliathe dnsmasq associated with ironic is likely for introspection/discovery of hardware13:33
TheJuliaoh my13:33
TheJuliathat... shouldn't... be13:33
TheJuliaso.13:33
TheJuliaNeutron is *supposed* to have a unique constraint on mac addresses13:33
admiyoHey.  Its Me.  I break things.  This is OK13:34
admiyoI think I lied.  The macs are diffferent13:34
iurygregoryTheJulia, yes I'm working on it today =)13:35
TheJuliaadmiyo: for a moment, I was going to find the best corgi sideeye picture on the internet13:35
iurygregoryalso, good morning13:35
admiyoOK,  I am going to delete the one baremetal node I have and see what it looks like at the start13:35
TheJuliadelete the neutron port, generally deleting a node out of ironic is one of those last resort sort of things13:35
TheJuliabut it *sounds* like your physical network is not bridged through to neutron with happiness13:36
TheJuliaI'd honestly be trying to tcpdump inside neutron namespaces to verify connectivity13:36
admiyoOK...no BM nodes.  Only the controller.  No VMs.  I have 5 ports. None of them match the MAC address for the node I am about to enroll....13:36
TheJuliaokay13:36
TheJuliaokay13:36
TheJuliathen you should be okay, I think13:36
admiyo| 15d2503c-acfa-4c57-9180-39b2f68e3ff8 |      | 0c:42:a1:49:a0:f4 | ip_address='192.168.97.184', subnet_id='cb7eb95a-bbd8-45ea-bc50-a1b7d6e7bc28' | DOWN   |13:39
admiyothat is the new port13:39
admiyoIf I go to the controller, what artifact should I see that tells the dhcp server about that new thing13:40
admiyough....I love it when the network connection to the server is so slow that you see a visiable lag when I type.  And I type fast.13:40
admiyoOK, so my dhcp is (I believe) docker ps | grep dns13:41
admiyocd6e06efcdfa   eng13sys-r111.scc-lab.amperecomputing.com:5000/kolla/ubuntu-source-dnsmasq:13.0.1                     "dumb-init --single-…"   19 hours ago   Up 15 hours                               ironic_dnsmasq13:41
admiyoand so the files for that can be seen via inspect and looking at the volumes13:42
TheJuliaymm hold on13:44
TheJuliadid you create that port?13:44
TheJuliaso, neutron-dhcp-agent manages the dhcp services in a namespace13:44
TheJuliaso it gets the update internally in neutron and updates dnsmasq13:44
janderssee you next week Ironic, have a great weekend everyone o/13:46
admiyoYeah, create a port13:46
TheJuliag'night janders 13:46
rpittaubye janders o/13:47
admiyoopenstack baremetal node create   ... then openstack baremetal port create ${ironic_pxe_interface_mac} --node ${NODE_UUID}13:47
TheJuliaand then?13:47
admiyoopenstack baremetal node validate ${NODE_UUID}13:48
admiyopenstack baremetal node manage ${NODE_UUID}13:48
admiyo openstack baremetal node provide ${NODE_UUID}13:48
admiyowith a few "shows" in between to make sure things are happening13:48
TheJuliaokay13:48
admiyoNode is in clean wait13:49
TheJuliawhat network_interface are you using on your ironic node?13:49
admiyoI opened SOL console on it, saw it PXE request, time out, go to UEFI and stall13:49
TheJulia(and yes, this unfortunately kind of does matter)13:49
TheJuliaokay, that helps13:49
admiyo--deploy-interface  ?13:50
TheJulia--network-interface13:50
TheJuliathe conductor will assign the default if you don't populate it13:50
TheJuliaor what it determines is the first in the ordered list13:51
admiyoI only specify (by mac) in the port create13:51
admiyo| network_interface      | flat 13:52
TheJuliaif flat, and there is no network assigned for cleaning/deployment in the conductor configuration, you'll need to attach the a vif to the baremetal node. But, your getting further than that because your actually booting. So Ironic in this case creates a neutron port, populates the dhcp configuration, and goes from there13:52
admiyothat is what comes back from show13:52
TheJuliawell actually, I take that back you don't need to pre-attach a vif at all13:52
TheJuliaokay13:52
admiyothere should be a cleaning network13:52
TheJuliaso, in this stage, you should have a new port in neutron13:52
TheJulialook for it13:53
admiyocleaning_network = baremetal-datapl13:53
admiyohmmmmm13:53
TheJuliaso13:53
admiyo| cb7eb95a-bbd8-45ea-bc50-a1b7d6e7bc28 | baremetal-dataplane-subnet | b0ae0494-91d7-4e5d-be99-1e151c2262ac | 192.168.97.0/24 13:54
admiyoand on the controller, there is a nic on that network13:54
TheJuliaokay13:54
admiyoenP4p4s0f0   inet 192.168.116.62/24 13:55
admiyoACH!13:55
TheJuliaso do you know how to look through network namespaces?13:55
admiyo97 vs 11613:55
TheJuliawell13:55
TheJuliahold up13:55
admiyoDa Komrade!13:55
TheJuliait doesn't necessarilly have to have an address bound on the host13:55
admiyoyeah, but at a minimum it should be a correct addresss...let me sort this13:55
TheJuliawell13:55
admiyo(some bad terraform....)13:55
TheJuliathat might still be right13:56
admiyoand by bad, I mean Copy Pasta error from me13:56
TheJulialook at your namespaces, ip netns list13:56
TheJuliaand then look inside each of them for the neutron dhcp agent13:56
admiyoI would not expect that to work...its outside the netmast13:56
TheJuliaqdhcp-blah-blah13:56
admiyo#  ip netns list13:56
admiyoqrouter-f7b297ee-8980-4cc7-9027-079e60aef4fd (id: 2)13:56
admiyoqrouter-fa51bfcc-550d-4cf9-acb4-0cc6b2ae9a6a (id: 1)13:56
admiyoqdhcp-5348a828-a126-40ca-8fc9-34467d2d0e72 (id: 0)13:56
admiyoNow we are beyond my knowledge...13:57
TheJuliaip netns exec qdhcp-5348a828-a126-40ca-8fc9-34467d2d0e72 ip addr13:57
TheJulia(this is how neutron does it's magic, network namespaces)13:57
admiyoip netns exec qdhcp-5348a828-a126-40ca-8fc9-34467d2d0e72 ip addr13:58
admiyo1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 100013:58
admiyo    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:0013:58
admiyo    inet 127.0.0.1/8 scope host lo13:58
admiyo       valid_lft forever preferred_lft forever13:58
admiyo    inet6 ::1/128 scope host 13:58
admiyo       valid_lft forever preferred_lft forever13:58
admiyo29: tapf5ebcdce-db: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 100013:58
admiyo    link/ether fa:16:3e:8d:e1:3a brd ff:ff:ff:ff:ff:ff13:58
admiyo    inet 10.0.0.2/24 brd 10.0.0.255 scope global tapf5ebcdce-db13:58
admiyo       valid_lft forever preferred_lft forever13:58
admiyo    inet 169.254.169.254/32 brd 169.254.169.254 scope global tapf5ebcdce-db13:58
admiyo       valid_lft forever preferred_lft forever13:58
admiyo    inet6 fe80::a9fe:a9fe/64 scope link 13:58
admiyo       valid_lft forever preferred_lft forever13:58
admiyo    inet6 fe80::f816:3eff:fe8d:e13a/64 scope link 13:58
admiyo       valid_lft forever preferred_lft forever13:58
TheJuliaso that is likely your baremetal dataplane network right?13:58
admiyosorry for the spam13:58
admiyoso nothing on 192.168 anyway13:58
TheJuliapaste.openstack.org :)13:58
admiyoyeah, yeah...I'm a bad citizen today13:59
TheJuliaso if you do something like `ovs-vsctl show` on the host, you should see tapf5ebcdce-db in the bindings13:59
TheJuliaand to verify your getting traffic into the namespace, you *really* need to tcpdump for traffic inside the namespace13:59
TheJuliaso instead of ip addr on the ip netns exec command, use tcpdump there13:59
TheJuliaif you *don't* see the tap interface bound through to a physical interface, then you have a misconfiguration someplace. Neutron might have the wrong input in term of the physical networks and the bindings14:00
TheJuliabut if you see any traffic, then that is an awesome sign14:00
admiyo# ovs-vsctl show14:00
admiyof174cf05-6ce0-4282-a814-26020351df0d14:00
admiyo    ovs_version: "2.13.5"14:00
TheJuliahttps://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fi.pinimg.com%2Foriginals%2F95%2F40%2Fb9%2F9540b955723aaa6c65ad51a02d6b280e.jpg&f=1&nofb=114:01
TheJuliano ovs output?14:01
admiyonope14:01
TheJuliacould you be using ovn?14:02
TheJulia... but that should still show something there14:02
admiyoIts a Kolla install, defaults.  Ubuntu 21.  I did not yet specify, but I can look14:02
TheJuliayou have routers, which makes me think you have a running l3 agent14:03
TheJuliabut it sounds like neutron-ovs-agent is not running14:03
TheJuliaI'd check its logs14:03
admiyo neutron_openvswitch_agent  container is running14:04
TheJuliatime to check logs, since you don't have any ovs records14:04
admiyono stack trace in the ovslog14:05
TheJuliaerrors?14:05
TheJuliawarnings?14:05
admiyoNope14:05
admiyoblissfully unaware14:05
TheJulia"critical hit, 1000 points, your cloud is down" messages ?14:06
admiyonope14:06
TheJuliais it *actually* running in the container14:06
TheJuliaor is it having a super long nap of "no cloud for you"14:06
TheJuliaI'd try to restart the container, tbh14:07
TheJuliamaybe it will do the needful, but it sounds like it is basically not operational14:07
admiyoOK.  I want to clean up the terraform etc first and then I will.  THis gives me somefodder for my Banthas14:07
TheJuliayay, electrical inspector nor contractor is here yet14:08
opendevreviewVerification of a change to openstack/ironic master failed: Start API after conductor is started in the combined executable  https://review.opendev.org/c/openstack/ironic/+/83509814:35
TheJuliacbouchar_: o/ you around?14:38
cbouchar_TheJulia:  I sure am!14:38
TheJuliacbouchar_: do you have the exception from when you put a folder path in image_source ?14:38
cbouchar_TheJulia.  I can easily reproduce.  I'll forward via email when done.14:39
TheJuliacbouchar_: thanks!14:40
cbouchar_TheJulia: so I need to put traceback into my kickstart correct?14:41
TheJuliacbouchar_: no, not in your kickstart config14:41
TheJuliawe never even get that far when you use a path without a trailing /14:41
cbouchar_that's right.  I'm all over the place with different subjects.14:42
TheJuliano worries, sorry I've not been able to focus on this so far this week, I've been pulled in a few different directions14:45
cbouchar_TheJulia: With --instance-info image_source=http://<ip>:<port>/DVD_MNT_RHEL7/, when I deploy I see "Failed to validate deploy or power info for node <node-id>. Error: Validation of image href http://<ip>:<port>/DVD_MNT_RHEL7/ failed, reason: Got HTTP code 403 instead of 200 in response to HEAD request. (HTTP 400)"14:47
TheJuliaright... okay14:47
TheJuliathanks14:47
TheJuliacbouchar_: hey, could you remove the trailing / real quick ?14:50
cbouchar_TheJulia: ???14:51
cbouchar_trailing what?14:51
TheJuliathe trailing slash on the url14:51
TheJuliaI'm trying to make some code to gracefully handle these cases14:51
cbouchar_got you14:51
cbouchar_TheJulia: Failed to validate deploy or power info for node <node-id>. Error: Validation of image href http://<ip>:<port>/DVD_MNT_RHEL7 failed, reason: Got HTTP code 301 instead of 200 in response to HEAD request. (HTTP 400)14:54
TheJuliaawesome thanks14:54
opendevreviewVerification of a change to openstack/ironic master failed: Start API after conductor is started in the combined executable  https://review.opendev.org/c/openstack/ironic/+/83509815:09
rpittaubye everyone, have a great weekend! o/15:49
dtantsurmmm, too many RETRY_LIMITs16:26
* dtantsur reports to #opendev16:28
TheJulia:(17:09
dtantsurTeam, I'll be on PTO next week. See you on the PTG!17:22
admiyoovs-vsctl show17:38
admiyof174cf05-6ce0-4282-a814-26020351df0d17:38
admiyo    Bridge br-ex17:38
admiyo        Port enp1s0f117:38
admiyo            Interface enp1s0f117:38
admiyo        Port br-ex17:38
admiyo            Interface br-ex17:38
admiyo                type: internal17:38
admiyo    ovs_version: "2.13.5"17:38
admiyoThat is a little more sane, right?17:38
admiyobut still no tap17:39
opendevreviewJulia Kreger proposed openstack/ironic master: WIP: Make non-image deploys more sane...  https://review.opendev.org/c/openstack/ironic/+/83470917:49
TheJuliaadmiyo: much more sane17:51
TheJuliadtantsur: o/17:51
TheJuliacbouchar_: rloo: ^^^ So I've started on this after some chatting with dtantsur earlier this week. It passes unit tests locally, but I need to write a number of tests. the idea is that with it we *should* be able to allow anaconda/kickstart based usage with directory paths *or* stage2 ramdisks based on image_source. cbouchar_ since it passes the normal ironic unit tests, you might want to give it a quick patch and try, 17:52
TheJuliajust fwiw.17:52
rloothx TheJulia! zer0c00l fyi ^^17:53
cbouchar_TheJulia: So instead of using image_url, I use image_source.17:53
TheJuliacbouchar_: yes17:54
TheJuliatechnically my change also makes us tollerant of not blowing up on image redirects17:54
TheJuliawhich I guess is a good thing in limited quantity17:54
TheJuliazer0c00l: please review, I'm hoping I'm not detonating the purpose of anything, but standalone usage is a prime use case for upstream. Also, with this I *think* we can put CI in place17:59
cbouchar_TheJulia:  I'm in xena branch and this patch doesn't patch cleanly.  I could search for dependent patches but not sure what kind of hole I'll be digging into.18:09
TheJuliaoh, likely image_type stuffs18:10
TheJuliaso backporting it would be deleting some of it I guess18:10
TheJuliabut fundimentals should still be good18:11
TheJuliaI did just leave some comments on the change18:11
cbouchar_yes18:13
cbouchar_Let me keep looking18:13
cbouchar_The diff it is dependent on is somewhat big and I suspect it may also have dependencies.  I'll have to be patience for your changes to bubble down.18:21
cbouchar_git show a813c769e818:21
TheJuliayup, the image_type work18:24
TheJuliawe shoudl be able to unwind and separate that but likely only when we go to backport18:24
TheJuliaI'll go highlight what can likely go in the changeset too18:24
TheJuliacbouchar_: rloo found an issue.18:34
TheJuliarloo: replied, cbouchar_ commented on the image type stuff and noted the two lines you'll likely need to move to manager.py  on Xena18:37
* TheJulia goes and writes unit test to detect what ruby found first18:37
TheJuliaoh wait, it is already checked, it is my redirect stuffs... yeah.18:40
TheJuliacbouchar_: you around?19:01
cbouchar_yes19:02
TheJulialast week when we were chatting, I remember you did curl -X HEAD http://path/to/your/loopback/19:03
TheJuliaWhat was the error/headers it responded with?19:03
cbouchar_I don't think I have it right.  This is what I did and see.  curl -X HEAD http://<ip>:<port>/DVD_MNT_RHEL7/19:06
cbouchar_Warning: Setting custom HTTP method to HEAD with -X/--request may not work the19:06
cbouchar_Warning: way you want. Consider using -I/--head instead.19:06
TheJuliaYeah, it was -I that you used19:10
TheJuliadoh19:10
TheJuliaI could have sworn you got an error, but I just tried it with a centos-stream mirror and didn't get an error, so I'm slightly confused19:17
TheJuliathen again, it could just be individual server configuration19:18
cbouchar_Here is comes:  curl -I http://<ip>:<port>/DVD_MNT_RHEL7/19:20
cbouchar_HTTP/1.1 403 Forbidden19:20
cbouchar_Server: nginx/1.20.119:20
cbouchar_Date: Fri, 25 Mar 2022 19:19:09 GMT19:20
cbouchar_Content-Type: text/html19:20
cbouchar_Content-Length: 15319:20
cbouchar_Connection: keep-alive19:20
TheJuliaso index just not permitted I guess19:20
TheJuliaso if the path ends with / *and* 403 forbidden, then we're likely okay, path wise19:20
TheJuliaor... is that just nginx19:21
cbouchar_without trailing / I get HTTP/1.1 301 Moved Permanently19:22
TheJuliayeah19:24
* TheJulia feels sad that the home NAS does everything *but* http19:32
TheJulia:62821:22
TheJuliadoh21:22
* TheJulia calls it a day22:13
* TheJulia will resume on the make things more sane patch tomorrow or sunday22:13
admiyoip netns  exec  qdhcp-338799b4-9090-4653-bea9-16caff89de4f ip addr22:17
admiyo  only comes back with the loopback now.  Is thatprogress or regress?22:17
admiyoegress22:18
TheJuliaregression :\22:22
TheJuliasorry22:22
admiyoWhat is supposed to create things in that namespace?22:37
admiyoAm I am making life impossible for myself by doing a Kolla Nova install and Ironic at the same time?  Would I be better off with a bifrost install?  Is that better documentated?22:39
admiyoDocumentated being the word of the day22:39
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Device management driver iface  https://review.opendev.org/c/openstack/networking-baremetal/+/83517023:09
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532423:10

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!