| opendevreview | Steve Baker proposed openstack/ironic master: Write master grub config on startup https://review.opendev.org/c/openstack/ironic/+/815580 | 01:04 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros https://review.opendev.org/c/openstack/ironic/+/815392 | 01:04 |
| *** pmannidi|Interview is now known as pmannidi | 02:45 | |
| *** pmannidi is now known as pmannidi|Lunch | 03:39 | |
| *** pmannidi|Lunch is now known as pmannidi | 06:09 | |
| hgy | Hello, Ironic!\ | 06:53 |
| muellerbe | good morning ironic o/ | 07:13 |
| rpittau | good morning ironic! o/ | 07:25 |
| arne_wiebalck | Good morning, hgy muellerbe rpittau and Ironic! | 07:45 |
| rpittau | hey arne_wiebalck :) | 07:45 |
| hgy | hey muellerbe rpittau and arne_wiebalck :) | 07:46 |
| rpittau | hey hgy :) | 07:47 |
| arne_wiebalck | stevebaker[m]: this looks great and addresses my understanding of what operators want, thanks! | 07:49 |
| dmellado | dtantsur: o/ so, found some issues on the docs/files | 09:11 |
| dmellado | on the documentations in https://docs.openstack.org/bifrost/latest/install/playbooks.html | 09:11 |
| dmellado | it says: | 09:11 |
| dmellado | If you wish to build an image, based upon the settings, you will need to set create_image_via_dib to true. | 09:12 |
| dmellado | but if I go to the actual files | 09:12 |
| dmellado | on playbooks/inventory/group_vars/target | 09:12 |
| dmellado | I see | 09:12 |
| dmellado | # Setting to utilize diskimage-builder to create a bootable image. | 09:12 |
| dmellado | create_image_via_dib: "{{ not use_cirros | default(false) | bool }}" | 09:12 |
| dmellado | dib_image_type: vm | 09:12 |
| dmellado | and it's also not really clear on which file to set this up, I assume is 'target' but clarifying this would be awesome | 09:13 |
| ajya | dtantsur: what's the plan with Virtual media certificates? Asking as iDRAC hasn't implemented that yet, it's at 1.3.2. Would this somehow impact Ironic? | 09:21 |
| opendevreview | chandan kumar proposed openstack/ironic-python-agent-builder master: Replace genisoimage with xorriso https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/820124 | 09:30 |
| dtantsur | good whatever the hell it is today, ironic | 09:40 |
| dtantsur | ajya: I've laid the foundation in sushy; I don't plan on working on the ironic side until we see hardware support | 09:41 |
| dtantsur | will be optional either way | 09:41 |
| ajya | ok, thanks | 09:42 |
| dtantsur | dmellado: honestly, I get confused by group_vars in ansible, so I just use -e :D | 09:42 |
| dmellado | lol | 09:42 |
| dtantsur | I think target is correct, needs double-checking | 09:42 |
| dmellado | so overall IIUC target==localhost, so I'm ok | 09:42 |
| dtantsur | yep | 09:42 |
| dtantsur | okay, where did I put my bloody coffee? | 09:42 |
| * dtantsur goes searching | 09:43 | |
| dmellado | just not sure about dib_image_type | 09:43 |
| dmellado | I assume it should be 'baremetal'? xD | 09:43 |
| dmellado | and also the dib_os_element optiones, besides debian and 'ubuntu' | 09:43 |
| dtantsur | dmellado: so, these types are horrible legacy | 09:43 |
| dmellado | would centos just go and deploy centos 8 stream? | 09:43 |
| dtantsur | vm and baremetal are whole disk and partition images | 09:43 |
| dtantsur | I think leaving vm is actually the right thing to do | 09:43 |
| dmellado | oh, I see, kinda confusing | 09:44 |
| dtantsur | just "centos" defaults to 7 IIRC | 09:44 |
| dtantsur | you need to set release to 8-stream | 09:44 |
| dmellado | dib_os_release: 8-stream | 09:44 |
| dtantsur | dib_os_release to be precise | 09:44 |
| dtantsur | yep | 09:44 |
| dmellado | awesome, will keep digging | 09:44 |
| dmellado | thanks dtantsur | 09:44 |
| dtantsur | dmellado: if you could submit a documentation patch once it works, it would be awesome | 09:45 |
| dmellado | I'll write a commit to the docs once I'm successful | 09:45 |
| dmellado | yeah | 09:45 |
| dtantsur | right :) | 09:45 |
| dtantsur | you know, it's hard to see what is missing when you've been working on the project for 8 years | 09:45 |
| dmellado | lol, totally | 09:45 |
| dmellado | I'm basically a first-user noob on this, so cope with me xD | 09:45 |
| dmellado | (even thought I still have a t-shirt around! xD) | 09:45 |
| dtantsur | no worries. I know that bifrost docs need a lot of love | 09:45 |
| dtantsur | I have rewritten the installation docs recently, but not the other ones | 09:46 |
| dtantsur | re vm: https://docs.openstack.org/diskimage-builder/latest/elements/vm/README.html | 09:46 |
| dmellado | ok, so even if its a baremetal server | 09:47 |
| dmellado | it'll go and partition the disk | 09:47 |
| dtantsur | dmellado: not exactly. "vm" aka whole disk images come with partitioning embedded (you can tell DIB how to partition) | 09:53 |
| dtantsur | dmellado: "baremetal" are partition images come with root partition only. Ironic will create basic partitioning for you. | 09:54 |
| dtantsur | I'd stick with whole disk images, unless you have specific requirements. | 09:54 |
| dtantsur | you can ask cloud-init to grow the last partition, so the size should not be an issue. | 09:54 |
| dtantsur | dmellado: we should probably link to https://docs.openstack.org/ironic/latest/user/creating-images.html from the bifrost docs | 09:55 |
| dmellado | dtantsur: gotcha, I'll probably make some notes there | 10:04 |
| dmellado | and +1 on the linking | 10:05 |
| opendevreview | Merged openstack/sushy master: Support for creating and deleting virtual media TLS certificates https://review.opendev.org/c/openstack/sushy/+/817792 | 10:19 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: DNM try to repair RPC https://review.opendev.org/c/openstack/ironic/+/819903 | 10:25 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: DNM try to repair RPC #2 https://review.opendev.org/c/openstack/ironic/+/820141 | 10:27 |
| iurygregory | good morning Ironic o/ | 10:38 |
| arne_wiebalck | hey iurygregory o/ | 10:40 |
| dmellado | dtantsur: why is bifrost asking for auth_url if there's no keystone? xD | 10:42 |
| dmellado | TASK [bifrost-cloud-config : Execute openstack.cloud.config to collect facts] | 10:42 |
| dmellado | ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** | 10:42 |
| dmellado | jueves 02 diciembre 2021 11:41:37 +0100 (0:00:00.994) 0:00:02.052 ****** | 10:42 |
| dmellado | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url | 10:42 |
| dtantsur | ehhhmmm, some bad defaults? what are you invoking and how? | 10:47 |
| dmellado | I frist did the playbookk install, which ran fine | 10:49 |
| dmellado | ansible-playbook -vvvv -i inventory/target install.yaml | 10:50 |
| dmellado | and then I wanted to enroll the machine | 10:50 |
| dmellado | so | 10:50 |
| dmellado | bifrost-cli --debug enroll /tmp/baremetal.json | 10:50 |
| dmellado | https://paste.openstack.org/show/811390/ | 10:52 |
| dmellado | this is my baremetal.json | 10:52 |
| dtantsur | hmm, tried locally, it worked | 10:53 |
| dtantsur | dmellado: has it generated a reasonably looking ~/.config/openstack/clouds.yaml? | 10:53 |
| dmellado | oh, and I did export OS_CLOUD=bifrost | 10:53 |
| dmellado | hmm looks like it | 10:55 |
| dmellado | let me paste it around | 10:55 |
| dmellado | https://paste.openstack.org/show/811391/ | 10:55 |
| dmellado | the issue showed up when enrolling the machine | 10:56 |
| dtantsur | iiiinteresting. I've just done the same, and it worked Oo | 10:57 |
| dmellado | same pattern with the playbook | 10:59 |
| dtantsur | dmellado: if you just run `baremetal driver list`, will it work for you? | 10:59 |
| dmellado | https://paste.openstack.org/show/811392/ | 10:59 |
| dmellado | works | 11:00 |
| dmellado | $ baremetal driver list | 11:00 |
| dmellado | +---------------------+----------------+ | 11:00 |
| dmellado | | Supported driver(s) | Active host(s) | | 11:00 |
| dmellado | +---------------------+----------------+ | 11:00 |
| dmellado | | ilo | fedora | | 11:00 |
| dmellado | | ipmi | fedora | | 11:00 |
| dmellado | | manual-management | fedora | | 11:00 |
| dmellado | | redfish | fedora | | 11:00 |
| dmellado | +---------------------+----------------+ | 11:00 |
| dmellado | oh, no idrac | 11:00 |
| dmellado | could that be the issue? | 11:00 |
| dtantsur | dmellado: it's not the issue, but it will be the issue :) | 11:00 |
| dtantsur | you need to set enabled_hardware_types=ipmi,redfish,idrac (for example) | 11:01 |
| dmellado | on ironic config? | 11:01 |
| dtantsur | dmellado: when running the installation playbooks | 11:01 |
| dmellado | oh, I see | 11:02 |
| dtantsur | (you can modify ironic directly, but it's harder) | 11:02 |
| dmellado | hehl did and restarted, then boom | 11:03 |
| dmellado | anyway, need to drop off for a while, will check and bug you later, thanks in advance! | 11:03 |
| jingvar | Can someone explain how to use standalone Bifrost? I've installed it on Ubuntu, but can't run baremetal node list ( /opt/stack/bifrost/bin/baremetal is exist) | 12:28 |
| jingvar | I know about PATH, but why it was not updated during the installation | 12:30 |
| iurygregory | jingvar, are you on the virtual env created by bifrost? | 12:31 |
| iurygregory | did you set the OS_CLOUD variable? | 12:31 |
| jingvar | no | 12:33 |
| iurygregory | source /opt/stack/bifrost/bin/activate | 12:33 |
| iurygregory | export OS_CLOUD=bifrost | 12:33 |
| jingvar | yep, I just use /opt/stack/bifrost/bin/baremetal node list | 12:34 |
| jingvar | I think it is missed in the doc | 12:34 |
| jingvar | or I'm a little bit blind :) | 12:35 |
| jingvar | Where is servers.yaml foramt defention to enroll nodes? | 12:39 |
| jingvar | As I rememeber the way is enroll, inspect, provision | 12:39 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-specs master: Yoga Themes https://review.opendev.org/c/openstack/ironic-specs/+/815308 | 12:40 |
| jingvar | and I can skip inspect if fill out cpu, disk etc | 12:40 |
| dtantsur | jingvar: you can use bifrost playbooks from https://docs.openstack.org/bifrost/latest/user/howto.html | 12:43 |
| dtantsur | jingvar: or you can use pure Ironic from now on: https://docs.openstack.org/ironic/latest/install/standalone.html | 12:43 |
| iurygregory | https://docs.openstack.org/bifrost/latest/user/howto.html#enroll-hardware | 12:43 |
| iurygregory | yeah I was about to paste the link :D | 12:43 |
| jingvar | what is "ansible_ssh_host": "192.168.122.2", | 12:44 |
| jingvar | where is json schema? | 12:44 |
| dtantsur | jingvar: the node's (future) IP address. I *think* you can skip that. | 12:45 |
| dtantsur | disclaimer: I personally use pure Ironic commands | 12:45 |
| jingvar | ehhh | 12:45 |
| jingvar | "provisioning_ipv4_address": "10.0.0.9", | 12:46 |
| jingvar | "ipv4_address": "192.168.122.2", | 12:46 |
| jingvar | what is it | 12:46 |
| dtantsur | jingvar: you can potentially use a different address during provisioning. I'd not do that, just skip it. | 12:47 |
| * dtantsur feels like a lot of documentation clean up is required | 12:47 | |
| jingvar | what is a minimal way - set of variables | 12:48 |
| dtantsur | jingvar: I think provisioning_ipv4_address is what you can skip. My (auto-generated) inventory does not have it, it does seem to have everything else. | 12:49 |
| jingvar | extra unnecessary variables without explanation... | 12:50 |
| jingvar | what is minimal set of variables to provison baremetal? | 12:52 |
| dmellado | dtantsur: ok, so found the issue | 12:53 |
| dmellado | a pre-existing, old as hell /etc/openstack/clouds.yaml file | 12:53 |
| dmellado | SIGH | 12:54 |
| dmellado | xD | 12:54 |
| dtantsur | aaaahhh | 12:54 |
| jingvar | as I rememeber "uuid": "00000000-0000-0000-0000-000000000001" is unnesseray, ironic creates it automaticaly | 12:54 |
| dtantsur | jingvar: correct | 12:54 |
| dtantsur | properties can be auto-discovered | 12:54 |
| dtantsur | driver and driver_info are mandatory for sure | 12:54 |
| jingvar | and where is it | 12:54 |
| dtantsur | dmellado: I was going to suggest that once you're back :) | 12:54 |
| dtantsur | nics is mandatory (can be discovered, but not always) | 12:55 |
| jingvar | nic uses to set ipmi boot as I remebebre, but it always first nic | 12:55 |
| dtantsur | jingvar: it's the NIC you can PXE-boot from on. It may or may not be the first, depends on your setup. | 12:56 |
| jingvar | it just my experience in a virtual env | 12:56 |
| jingvar | pxe.cfg will use it mac | 12:57 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: DNM try to repair RPC https://review.opendev.org/c/openstack/ironic/+/819903 | 12:58 |
| dtantsur | dmellado: admittedly, we shouldn't try to probe all clouds. but that's how ansible-collections-openstack work. | 12:59 |
| dmellado | dtantsur: so, next issue xD | 13:01 |
| dtantsur | \o/ | 13:01 |
| dmellado | TASK [ironic-enroll-dynamic : Dynamic enrollment] | 13:02 |
| dmellado | ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** | 13:02 |
| dmellado | * | 13:02 |
| dmellado | task path: /home/dmellado/Devel/openstack/bifrost/playbooks/roles/ironic-enroll-dynamic/tasks/main.yml:22 | 13:02 |
| dmellado | jueves 02 diciembre 2021 14:01:15 +0100 (0:00:00.195) 0:00:07.110 ****** | 13:02 |
| dmellado | fatal: [cnfdf07 -> {{ groups['target'][0] if groups['target'] is defined else 'localhost' }}]: FAILED! => { | 13:02 |
| dmellado | "msg": "The task includes an option with an undefined variable. The error was: 'driver' is undefined\n\nThe error appears to be in '/home/dmellado/Devel/openstack/bifrost/playbooks/roles/ironic-enroll-dynamic/tasks/main.yml': line 22, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: | 13:02 |
| dmellado | \"Dynamic enrollment\"\n ^ here\n" | 13:02 |
| dmellado | } | 13:02 |
| dmellado | https://paste.openstack.org/show/811397/ | 13:02 |
| dmellado | ooops meant to just type the pastebin | 13:02 |
| dmellado | sorry on the flood | 13:02 |
| dtantsur | heh | 13:02 |
| dtantsur | can it be that your inventory file lacks a "driver"? | 13:02 |
| dmellado | it should have it xD | 13:04 |
| dmellado | 19 "driver": "idrac", | 13:04 |
| dmellado | 13:04 | |
| dmellado | that's the json file, oh | 13:04 |
| dmellado | and I've exported it as export BIFROST_INVENTORY_SOURCE | 13:05 |
| dtantsur | dmellado: how do you invoke enrollment? do you use the dynamic inventory? | 13:05 |
| dmellado | xport BIFROST_INVENTORY_SOURCE='/home/dmellado/Devel/openstack/bifrost/playbooks/inventory/baremetal.json' | 13:06 |
| dmellado | and then | 13:07 |
| dmellado | ansible-playbook -vvvv -i inventory/ enroll-dynamic.yaml | 13:07 |
| dmellado | but it does seem something related to the vars | 13:07 |
| dmellado | maybe it's not sourcing that well | 13:07 |
| dmellado | as when I added -e driver=idrac | 13:07 |
| dmellado | it complained about another var | 13:07 |
| dmellado | hmm | 13:07 |
| dtantsur | `-i inventory/` doesn't look correct | 13:08 |
| dmellado | oh, I'm dumb | 13:08 |
| dmellado | just read it | 13:08 |
| * dmellado facepalms | 13:08 | |
| dtantsur | we should probably remove the 2nd example or make it less prominent | 13:08 |
| dtantsur | `-i inventory/bifrost_inventory.py` is what most people will use | 13:08 |
| dmellado | there's an issue on the docs | 13:09 |
| dmellado | see the second call here | 13:09 |
| dmellado | https://docs.openstack.org/bifrost/latest/user/howto.html#enroll-hardware | 13:09 |
| dmellado | ansible-playbook -vvvv -i inventory/ enroll-dynamic.yaml | 13:09 |
| dmellado | from docs | 13:09 |
| dmellado | just noticed the issue, but it's posted there xD | 13:09 |
| dtantsur | or maybe it's supposed to take everything from the directory? I don't know. | 13:09 |
| dmellado | nah | 13:09 |
| dmellado | ansible-playbook -vvvv -i inventory/bifrost_inventory.py enroll-dynamic.yaml | 13:09 |
| dmellado | worked | 13:09 |
| dmellado | I'll be correcting these issues on the docs | 13:10 |
| dmellado | thanks in any case! | 13:10 |
| dtantsur | interestingly, using inventory/ worked for me. dunno. let's move this 2nd part to a note. | 13:10 |
| jingvar | why not just use something like that for examples https://paste.opendev.org/show/811399/ | 13:11 |
| dmellado | jingvar: +1 | 13:11 |
| dmellado | | ef4bbe4a-3ecd-4bae-888e-52c406707206 | cnfdf07 | None | None | available | False | | 13:12 |
| dmellado | awesome xD | 13:12 |
| dtantsur | this is a start :) | 13:13 |
| iurygregory | congrats =) | 13:13 |
| dtantsur | FYI `bifrost-cli enroll` should do the magical inventory handling for you | 13:13 |
| dtantsur | we probably need a similar command for deployment, but I didn't have time for it | 13:13 |
| iurygregory | bifrost-cli <3 | 13:13 |
| jingvar | where is step download a qcow and place it to right place | 13:16 |
| dtantsur | dmellado: ^^ you tried DIB-built images already? | 13:16 |
| dtantsur | jingvar: I think the default is to build an image with diskimage-builder | 13:16 |
| dtantsur | or you can download it yourself and use a file:/// URL. or put it /httpboot and use an http:// url | 13:17 |
| jingvar | whre is it in the doc? | 13:17 |
| dtantsur | dunno if it's there or not | 13:18 |
| dtantsur | as I said earlier, this part of the docs need significant expansion | 13:18 |
| opendevreview | Merged x/sushy-oem-idrac master: Add destructive field flag to export configuration https://review.opendev.org/c/x/sushy-oem-idrac/+/816815 | 13:18 |
| jingvar | why is the simple job so painfull :) | 13:18 |
| dtantsur | patches welcome. bifrost is purely a community projects with no large vendors behind it (unlike ironic itself). | 13:20 |
| iurygregory | I know there is a role you can use to create the images bifrost-create-dib-image | 13:20 |
| iurygregory | https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-create-dib-image/README.md | 13:20 |
| dtantsur | you can even use diskimage-builder CLI :) | 13:20 |
| dtantsur | this role is invoked by bifrost when you request it to build an image | 13:21 |
| jingvar | I wan't use a common cloud image | 13:21 |
| jingvar | ubuntu/centos | 13:21 |
| iurygregory | yeah, if the person already know that wants to build and image they can use during the installation =) | 13:21 |
| dtantsur | jingvar: just put it to /httpboot and use http://<IP>:8080/<filename> | 13:22 |
| dtantsur | at some point I'll reproduce it all from scratch and document the missing bits if nobody beats me to it | 13:23 |
| jingvar | As user I want to get deployed ubuntu cloud image with a network config , ssh key on baremetal with ipmi ip user/password | 13:23 |
| dtantsur | (but it's definitely not when I have a migraine since yesterday evening) | 13:23 |
| jingvar | I have to install bifrost, fillout server name, ipmi ip, ipmi user, ipmi password and enroll it | 13:25 |
| jingvar | second one, I have to download my cloudimage and place it into /http (I can skip this, uses an external provider for this kind of resources ) | 13:26 |
| jingvar | i have to place the url(qcow) into ? | 13:27 |
| dtantsur | jingvar: https://github.com/openstack/bifrost/blob/master/playbooks/inventory/baremetal.yml.example has examples, we should probably move it to the docs (I think dmellado wanted to do that/) | 13:29 |
| opendevreview | Merged x/sushy-oem-idrac master: Updating yoga tested python versions in classifier https://review.opendev.org/c/x/sushy-oem-idrac/+/819277 | 13:29 |
| jingvar | what about inspect phase | 13:32 |
| jingvar | each phase should have an correct set of fields | 13:33 |
| dtantsur | jingvar: inspection, assuming you even need it, shouldn't require anything special | 13:34 |
| dtantsur | ironic folks ( iurygregory if you have time ), my headache is not getting better, I need to step back for a bit. could you maybe help the new folks with bifrost? | 13:35 |
| dtantsur | I think we have answers to all questions somewhere, but the "where" is usually not the docs :/ | 13:36 |
| dmellado | dtantsur: take care man! | 13:36 |
| jingvar | Can we continue? | 13:37 |
| iurygregory | dtantsur, sure =) | 13:37 |
| iurygregory | dtantsur, take care! | 13:37 |
| jingvar | dtantsur: be healthy | 13:38 |
| jingvar | what is the way to configure network of baremeatl? | 13:40 |
| dmellado | dtantsur: for the record, I just set the vars for using centos8, which I think I'll add to the docs around as those weren't clear to me at all | 13:43 |
| iurygregory | jingvar, you mean to configure the dhcp? https://github.com/openstack/bifrost/blob/7271695714739af10d741a39dc4acd5e68465cb5/doc/source/user/dhcp.rst | 13:44 |
| jingvar | no, it about baremeatl propirities | 13:47 |
| iurygregory | bifrost should have the setup for your network and you wouldn't need to worry, if by proprieties you the information you can collect from the HW via introspection this will happen after you enroll the node and trigger inspection | 13:52 |
| jingvar | "/home/ubuntu/bifrost/playbooks/inventory/bifrost_inventory.py", line 22, in <module> from oslo_config import cfg ModuleNotFoundError: No module named 'oslo_config' | 13:52 |
| iurygregory | O.o | 13:53 |
| jingvar | I've added some fields into servers.yaml, delete nodes, ant it happen | 13:53 |
| iurygregory | are you running inside the venv? | 13:53 |
| jingvar | deactivate it | 13:53 |
| jingvar | ubuntu@ephemeral:~$ ./bifrost/bifrost-cli enroll ~/servers.yaml | 13:54 |
| iurygregory | if you are not in the venv I would expect that to happen... | 13:54 |
| jingvar | it not works in venv | 13:54 |
| jingvar | ubuntu@ephemeral:~$ source /opt/stack/bifrost/bin/activate | 13:55 |
| jingvar | (bifrost) ubuntu@ephemeral:~$ ./bifrost/bifrost-cli enroll ~/servers.yaml | 13:55 |
| jingvar | Traceback (most recent call last): .... | 13:55 |
| jingvar | full log https://paste.opendev.org/show/811402/ | 13:56 |
| jingvar | first time it works (withou venv) | 13:56 |
| * iurygregory looks | 13:56 | |
| jingvar | maybe it is wrong way use the doc for bifrost installation :) | 13:57 |
| jingvar | hints from your side | 13:58 |
| dtantsur | yep, can reproduce it | 13:58 |
| dtantsur | you need to run bifrost-cli from the bifrost directory | 13:58 |
| dtantsur | apparently some pbr magic prevents it from working from other directories (doh!) | 13:59 |
| jingvar | nice | 13:59 |
| jingvar | from venv or bare | 13:59 |
| dtantsur | venv | 13:59 |
| iurygregory | tks pbr -.-' | 13:59 |
| jingvar | ok, it works | 14:00 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Catch all exceptions from pbr when importing the root package https://review.opendev.org/c/openstack/bifrost/+/820173 | 14:01 |
| dtantsur | this is easy to fix ^^ | 14:01 |
| jingvar | what about ssh keys? | 14:01 |
| * dtantsur steps back again | 14:01 | |
| iurygregory | I need to rebuild my bifrost env to test things because I forgot to set ssh via password and I'm using another computer (/facepalm) | 14:01 |
| dtantsur | ssh keys are going through the configdrive. I think it's built automatically by bifrost. I think. | 14:01 |
| * dtantsur steps back for real | 14:01 | |
| jingvar | I need to place my key... | 14:01 |
| iurygregory | yeah bifrost should find the ssh key for you | 14:02 |
| iurygregory | https://github.com/openstack/bifrost/blob/7271695714739af10d741a39dc4acd5e68465cb5/playbooks/roles/bifrost-ironic-install/ssh_public_key_path.yaml#L16 | 14:02 |
| iurygregory | https://github.com/openstack/bifrost/blob/7271695714739af10d741a39dc4acd5e68465cb5/playbooks/roles/bifrost-configdrives-dynamic/README.md#role-variables you can set the variables if you want also | 14:04 |
| jingvar | https://paste.opendev.org/show/811404/ is it enough? | 14:04 |
| iurygregory | should be | 14:05 |
| jingvar | 192.168.0.10 - bifrost server ip | 14:05 |
| jingvar | dhcp range 200-220 | 14:05 |
| iurygregory | I don't think you need to set the ipv4 | 14:06 |
| jingvar | how i shuld set desired node ip | 14:06 |
| iurygregory | oh ok you want to specify, just make sure that there is available I think ( I've always let the dhcp provide the ip) | 14:10 |
| jingvar | we never do this ж) | 14:11 |
| jingvar | or it is about ipa phase | 14:11 |
| jingvar | where I have to set final IP | 14:12 |
| jingvar | and how to setup bifrost with legacy boot - by default uefi , I've read | 14:15 |
| iurygregory | oh you need to specify that in the node let me find | 14:16 |
| dtantsur | yep, or use default_boot_mode: bios when installing | 14:17 |
| iurygregory | this ^ | 14:17 |
| iurygregory | openstack baremetal node set <node-uuid> --property capabilities='boot_mode:bios' | 14:18 |
| jingvar | I talk about ironic drver types | 14:19 |
| jingvar | --uefi Boot machines in the UEFI mode by default. | 14:20 |
| jingvar | or it means if the option is enable it will use UEFI | 14:21 |
| dtantsur | TheJulia: oh, I FINALLY found the error in my patch! I change the call to client.prepare from (topic=topic) to just (topic). The first positional argument is not a topic >_< | 14:22 |
| dtantsur | ffs, it took me 3 days to debug #headbang | 14:22 |
| iurygregory | jingvar, ironic now is default to UEFI boot | 14:22 |
| jingvar | yep, enabled_boot_interfaces = ipxe,pxe,redfish-virtual-media | 14:22 |
| jingvar | for default example from the doc | 14:23 |
| iurygregory | so if you want to use legacy bios you need to have the right configs | 14:23 |
| jingvar | turn back to IP settings | 14:23 |
| jingvar | what is IP will have my node after provison? comes from? | 14:24 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] All-in-one Ironic service with a local RPC bus https://review.opendev.org/c/openstack/ironic/+/819620 | 14:24 |
| iurygregory | it will use [deploy]default_boot_mode if you don't have boot_mode set | 14:24 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Add rpc_transport == none https://review.opendev.org/c/openstack/ironic/+/820036 | 14:25 |
| jingvar | (bifrost) ubuntu@ephemeral:~/bifrost$ ansible-playbook -vvvv -i inventory/bifrost_inventory.py deploy-dynamic.yaml | 14:27 |
| jingvar | ERROR! the playbook: deploy-dynamic.yaml could not be found | 14:27 |
| dtantsur | cd playbooks | 14:27 |
| iurygregory | ^ yeah | 14:28 |
| iurygregory | you need to run from the playbooks dir or specify playbooks/deploy-dinamic.yaml I think | 14:28 |
| jingvar | [WARNING]: Could not match supplied host pattern, ignoring: baremetal | 14:29 |
| jingvar | skipping: no hosts matched | 14:29 |
| jingvar | (bifrost) ubuntu@ephemeral:~/bifrost/playbooks$ baremetal node list | 14:30 |
| jingvar | | cf3d0c9c-e877-4519-adec-aa6cb3f6d83a | ctl01.stg | None | power on | available | False | | 14:30 |
| TheJulia | dtantsur: yay! | 14:30 |
| TheJulia | jingvar: umm.. did you set a BIFROST_INVENTORY... environment variable? | 14:32 |
| * TheJulia doesn't remember what she named it percisely which is a bad sign | 14:32 | |
| jingvar | ofcourse not set | 14:32 |
| TheJulia | ahh, yeah. It has to be set to a file, or you could just set it to ironic | 14:32 |
| dtantsur | TheJulia: we could use some docs update, I guess | 14:33 |
| * dtantsur has a headache and 2 meetings ahead | 14:33 | |
| jingvar | TASK [bifrost-configdrives-dynamic : Make metadata folders - /openstack/<version>] failed | 14:33 |
| jingvar | The task includes an option with an undefined variable | 14:33 |
| * TheJulia raises an eyebrow | 14:34 | |
| jingvar | bifrost-configdrives-dynamic/tasks/main.yml | 14:34 |
| jingvar | Make metadata folders - /openstack/<version> | 14:34 |
| jingvar | The error was: 'uuid' is undefined | 14:35 |
| TheJulia | hmmm | 14:35 |
| TheJulia | something seems broken *or* it is not getting your inventory | 14:35 |
| TheJulia | if you run inventory/bifrost_inventory, what do you get? | 14:35 |
| jingvar | eee, full command line please | 14:37 |
| TheJulia | you should just be able to execute it directly | 14:37 |
| TheJulia | like ./inventory/bifrost_inventory.py | 14:37 |
| jingvar | (bifrost) ubuntu@ephemeral:~/bifrost$ sudo find / -name bifrost_inventory | 14:38 |
| jingvar | (bifrost) ubuntu@ephemeral:~/bifrost$ | 14:38 |
| TheJulia | well... there is your problem | 14:39 |
| TheJulia | hmm | 14:39 |
| jingvar | got it | 14:39 |
| jingvar | https://paste.opendev.org/show/811408/ | 14:40 |
| TheJulia | https://github.com/openstack/bifrost/blob/master/bifrost/inventory.py | 14:40 |
| TheJulia | so your inventory data must have a uuid, or since the node is already created, you can change the source to ironic and it *should* collect all of the data from ironic | 14:41 |
| jingvar | what means change source | 14:44 |
| TheJulia | well, to whatever BIFROST_INVENTORY_SOURCE is set to | 14:45 |
| TheJulia | export BIFROST_INVENTORY_SOURCE=ironic should cause it to read from ironic's data | 14:46 |
| TheJulia | which *would* have a uuid, but might not have an IP. Your file your feeding via BIFROST_INVENTORY_SOURCE needs to have the node's UUID. | 14:46 |
| TheJulia | dtantsur: So steve and I were staring at an issue one of our telecom friends filed recently yesterday and noticed this gem in sushy https://review.opendev.org/c/openstack/sushy/+/820076 | 14:47 |
| dtantsur | TheJulia: I suspect it was intentional was some fallback logic, so proceed with caution | 14:48 |
| dtantsur | (second "was" should be "for") | 14:48 |
| TheJulia | one of the parents actually expected AccessError as noted in the docs but it never seemed to be raised anywhere | 14:49 |
| TheJulia | We seemed to have gotten the client wedged in this state where is's connection data is bad and it is trying to authenticate with the session service using the old data, specifically on getting the path to the session service which if I remember correctly is *not* supposed to enforce authentication | 14:50 |
| jingvar | ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_noop | 14:50 |
| jingvar | TASK [bifrost-deploy-nodes-dynamic : Deploy to hardware - bifrost default] | 14:50 |
| TheJulia | jingvar: your going to need to give us the context of how you get the error for us to help you. I recommend using paste.openstack.org for this purpose | 14:50 |
| dtantsur | it rings bells | 14:50 |
| TheJulia | ajya: could you take a look at https://review.opendev.org/c/openstack/sushy/+/820076 | 14:50 |
| dtantsur | I remember there was an issue with network_interface being a thing both in ironic and in ansible | 14:51 |
| dtantsur | so ansible_{{ network_interface }} could misbehave | 14:51 |
| dtantsur | I thought we've fixed that, I haven't seen it in a while | 14:51 |
| * dtantsur still cannot think reliably | 14:51 | |
| jingvar | https://paste.opendev.org/show/811409/ | 14:51 |
| ajya | TheJulia: will check tomorrow | 14:51 |
| TheJulia | dtantsur: Seve and I are thinking https://review.opendev.org/c/openstack/ironic/+/820077 <-- may be a easily backportable ironic patch, but we don't *actually* see any evidence of an AttributeError getting returned in the logs :\ but logically, it should be getting AttributeErrors and bombing the sushy client when it tries to re-engage the client. | 14:52 |
| TheJulia | dtantsur: yeah, I'm likely going to loose a good chunk of today to the logs as well, I have gigabytes of unhappy ironic logs | 14:53 |
| dtantsur | I'll think about it when I can | 14:53 |
| TheJulia | so there may still be a third thing going on | 14:53 |
| dtantsur | ugh :( | 14:53 |
| iurygregory | TheJulia, +W | 14:53 |
| TheJulia | iurygregory: that was a little fast :) | 14:54 |
| TheJulia | ajya: please look at it tomorrow, and if necessary issue a revert | 14:54 |
| iurygregory | I was looking when you pasted the code =) | 14:54 |
| iurygregory | the change was quite small and after reading the story it makes sense | 14:54 |
| iurygregory | but we know that it may cause problem in other HWs... because it's our life | 14:55 |
| dmellado | TheJulia: o/ now that I've got you around I have a question re: bifrost and dib-images and so xD | 14:55 |
| dmellado | IIUC | 14:55 |
| * TheJulia hides | 14:55 | |
| dmellado | just setting this on group_vars/target | 14:55 |
| dmellado | create_image_via_dib: true | 14:55 |
| dmellado | dib_os_element: centos | 14:55 |
| dmellado | dib_os_release: 8-stream | 14:55 |
| iurygregory | +14 -1 (not many changes to review lol) | 14:55 |
| dmellado | dib_image_type: vm | 14:55 |
| dmellado | would have bifrost use centos 8 stream | 14:55 |
| dmellado | as the image, isn't it? | 14:56 |
| TheJulia | iurygregory: yeah, it just dead ends there and literally one of the logs I have compressed to like 11mb... uncompressed to 997mb and 98% of it is that debug line getting hit over... and over... and over again. | 14:56 |
| iurygregory | TheJulia, OMG | 14:56 |
| * iurygregory hopes it won't cause regressions etc but it looks ok =) | 14:57 | |
| dmellado | TheJulia: hope the question wasn't a reason for hiding xD | 14:57 |
| TheJulia | dmellado: need to get one thing off my mind with someone else | 14:57 |
| * TheJulia sends required message | 14:58 | |
| dmellado | and again, bifrost/ironic kinda newbie, but I'm not sure on how to configure networking, besides dhcp/ipv4 (how about ipv6? xD) | 14:58 |
| dtantsur | dmellado: the default is whatever the OS does. in case of CentOS it's DHCP on the first NIC. | 14:59 |
| dmellado | a-ha | 14:59 |
| dtantsur | for actual configuration there is a thing called config-drive.. which we probably also poorly documented | 14:59 |
| dtantsur | but bifrost generates it by default, I think | 14:59 |
| dmellado | awww config-drive, brings back terrible memories xD | 14:59 |
| dtantsur | haha, right | 14:59 |
| dtantsur | not many options for bare metal though, right? :) | 14:59 |
| dtantsur | I hope TheJulia remembers how to use it in bifrost | 14:59 |
| TheJulia | dtantsur: partially. I wrote a bunch of it... | 15:00 |
| * TheJulia wonders if sed s/TheJulia/TheJuliaTheElder/ is appropriate | 15:00 | |
| dmellado | I do kinda recall using config-drive to push cloud-init (looks like a lifetime ago...) | 15:00 |
| TheJulia | so anyway! | 15:00 |
| dmellado | wouldn't you have any kuryr-kubernetes questions around? I feel bad about just asking and not replying xD | 15:01 |
| dtantsur | I don't think so :) | 15:01 |
| TheJulia | dmellado: so... hmm | 15:01 |
| dtantsur | dmellado: I think you can set all this stuff per node: https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-configdrives-dynamic/tasks/main.yml#L43-L47 | 15:02 |
| TheJulia | dmellado: I never encoded hard coded support for ipv6 in bifrost, the templates were purely v4 generated for static config injection. You could absolutely do v6 if you have slaac on the attached network or a dhcp server configured for v6. If you want static v6, I think your going to need to patch the template, playbook, and default variables | 15:03 |
| dtantsur | we should dump the sacred knowledge of TheJuliaTheElder into the docs :) | 15:03 |
| TheJulia | eek | 15:03 |
| TheJulia | don't crack my head open like an egg! | 15:03 |
| TheJulia | and pour it into the docs, that woudl be weird | 15:03 |
| arne_wiebalck | :-D | 15:03 |
| dtantsur | "weird" :D | 15:03 |
| * TheJulia had *super* weird dreams last night | 15:03 | |
| dmellado | lol | 15:03 |
| dmellado | TheJulia: I swear I'll try contributing that back when my 2yo kid allows me xD | 15:04 |
| dmellado | so, then, dtantsur | 15:04 |
| TheJulia | dmellado: I totally understand | 15:04 |
| TheJulia | dmellado: my 2yo corgi child needs to go to the corgi doctor (vet) today | 15:04 |
| dmellado | IF I put that back into the baremetal.json it should be able to pick it up per node | 15:05 |
| dtantsur | I think so | 15:05 |
| dmellado | TheJulia: ouch, hope the corgi child gets better soon | 15:05 |
| dtantsur | ++ | 15:05 |
| dmellado | dtantsur: AAAAND http and http_proxy would be picket up from env | 15:05 |
| TheJulia | dmellado: yes, they will become variables if memory serves | 15:05 |
| dmellado | so, I feel I was a pain to all of you and I learnt a lot | 15:05 |
| dtantsur | it's partly our fault for not having it in the docs :) | 15:06 |
| dtantsur | no worries | 15:06 |
| jingvar | my story https://etherpad.opendev.org/p/jingvar | 15:06 |
| dmellado | I'm missing a physical openstack summit | 15:06 |
| dtantsur | dmellado: next June? | 15:06 |
| rpittau | all fingers crossed | 15:06 |
| dtantsur | which reminds me, I should submit something for it | 15:06 |
| dtantsur | https://openinfra.dev/summit/ | 15:06 |
| dmellado | I do hope | 15:07 |
| dmellado | I'd love visiting Berlin tbh | 15:07 |
| dmellado | was living there for some time and seldom have time to visit | 15:07 |
| dtantsur | and TheJulia needs to leave more stickers in brewdog :) | 15:07 |
| TheJulia | hehe | 15:07 |
| dmellado | should I submit something about 'surviving bifrost'? xD | 15:07 |
| TheJulia | we need to find a sticker artist | 15:08 |
| TheJulia | dmellado: sure! | 15:08 |
| TheJulia | dmellado: I'll take a front row seat if there is not a conflict | 15:08 |
| TheJulia | and would be happy to wear a "blame me" shirt of some sort | 15:08 |
| dmellado | LOL | 15:08 |
| dmellado | I'll get to find my t-shirt of 'it works on devstack' then | 15:09 |
| dtantsur | TheJulia: or we can have twin t-shirts "blame him ->" "<- blame her" at this point :D | 15:09 |
| dtantsur | and sit together | 15:09 |
| TheJulia | jingvar: so far... looks good | 15:09 |
| TheJulia | jingvar: if you have the inventory file with a uuid, you could just use the file as source | 15:09 |
| TheJulia | dtantsur: this sounds excellent | 15:10 |
| dmellado | that's also another question I had | 15:10 |
| dmellado | I just made up the uuid | 15:10 |
| dmellado | wieh uuid4() | 15:10 |
| TheJulia | dmellado: a lot of people do so | 15:10 |
| TheJulia | otherwise ironic does it for you! | 15:10 |
| dtantsur | dmellado: you could let ironic generate it, but making it up is perfectly fine | 15:10 |
| dmellado | not sure if you have something that would generate one on the fly | 15:10 |
| dmellado | oh, I see | 15:10 |
| TheJulia | the case with jingvar's usage is he is not doing it as one shot, but as separate playbooks, and his inventory file didn't have a uuid, where as the config drive playbook *really* needs uuid | 15:11 |
| jingvar | why I need an uid? | 15:11 |
| TheJulia | in CI I think we pull facts back before then to pull the uuid into the ansible context, which gets used on the next plays, so it works in CI then, but manually, directly with a playbook, it errors since it wants it | 15:12 |
| jingvar | at the bottom list of variables from a user | 15:12 |
| TheJulia | jingvar: It is trying to populate the metadata which includes instance related uuid information. Tools like cloud-config use this to determine if the system has changed or been re-imaged | 15:12 |
| TheJulia | and then trigger a job to run based upon the present metadata | 15:12 |
| jingvar | ok let me try with some uid | 15:13 |
| TheJulia | jingvar: oh, different error this time | 15:14 |
| TheJulia | let me add a suggestion to your etherpad | 15:14 |
| TheJulia | jingvar: try that | 15:15 |
| jingvar | unset BIFROST_INVENTORY_SOURCE | 15:17 |
| jingvar | OK | 15:17 |
| jingvar | but | 15:17 |
| jingvar | i dont see that provisioning is started | 15:18 |
| jingvar | It just env with bootable iso, I can share it | 15:20 |
| *** dking is now known as Guest7383 | 15:21 | |
| Guest7383 | I know that this might be a silly or unanswerable question, but does anybody know of issues where Inspector uses a large amount of RAM? | 15:22 |
| jingvar | it is amsing Bifrost can push uid to ironic but can't get it from ironic | 15:22 |
| dtantsur | Guest7383: I've never noticed it doing so. What is large for you? | 15:23 |
| Guest7383 | Currently, I have an Inspector instance running and it seems to be consuming well over 40G of RAM, and I'm not even sure why. This instance does get a lot of queries (once per minute for all 200 servers), but even when that stopped, it didn't change the amount of RAM used. Also, when I tried to run strace, I don't see any new allocation of RAM with incoming queries. | 15:24 |
| TheJulia | hmm, it should be able to, but the inventory source needs to be set | 15:25 |
| TheJulia | jingvar: ^^ | 15:25 |
| dtantsur | 40G WUT. this is completely crazy. | 15:25 |
| TheJulia | absurdly crazy | 15:25 |
| dtantsur | Guest7383: do you have a reproducer for it? or does it simply happened in your production? | 15:25 |
| Guest7383 | That's my thought. Fortunately, the server it's on can handle that without any noticeable problem, but I'm sure that it can't be normal. | 15:26 |
| dtantsur | it's not normal at all | 15:26 |
| dtantsur | since it's python, we expect hundreds of megabytes, but not gigabytes | 15:26 |
| TheJulia | Guest7383: oh! you know what, we likely don't delete data that is receieved | 15:26 |
| dtantsur | yeah, but we don't store it in memory... | 15:26 |
| TheJulia | so it is likely sitting in process ram and just not GC'ed | 15:26 |
| TheJulia | dtantsur: but... does it go through any object at any point in time? | 15:27 |
| dtantsur | I cannot say it's impossible, but it's support to be GC'ed | 15:27 |
| Guest7383 | Right now, I've only seen it in this one instance, and I'm not terribly sure how to cause it. We noticed it a while ago, but as there was no problem, we overlooked it. So, I'm not sure whether it's a fluke or not. This time, I wanted to gather some data before I just restart it and hope for the best. | 15:27 |
| dtantsur | s/support/supposed/ | 15:27 |
| TheJulia | dtantsur: oh, absolutely | 15:27 |
| dtantsur | it's clearly a leak somewhere. my first suspect would also be introspection data, but it's not THAT large | 15:27 |
| dtantsur | unless you have many thousands of servers | 15:28 |
| TheJulia | dtantsur: yeah | 15:28 |
| jingvar | what is wrong with my config... | 15:28 |
| Guest7383 | We have a decent amount of introspection data. We use extra-hardware. It's not absurd, but multiply that over 200 servers and that could make it noticeable. | 15:28 |
| Guest7383 | Not thousands. Just a couple hundred for this instance. | 15:29 |
| dtantsur | I don't think extra-hardware generates ~ 200 MiB of data per node | 15:29 |
| dtantsur | but it should be easy to check in swift or wherever you keep the introspection data | 15:29 |
| TheJulia | soo.. 200MiB * 200 machines..... | 15:30 |
| Guest7383 | We're using a DB. I can check, but I don't think it's huge. | 15:30 |
| TheJulia | 40,000 MiB | 15:30 |
| dtantsur | yeah, but 200MiB is a lot | 15:30 |
| TheJulia | oh, I thoguht you were saying it can | 15:31 |
| dtantsur | I would expect single-digit MiB even on a large server | 15:31 |
| TheJulia | I've seen it generate at least a couple mb | 15:31 |
| dtantsur | Guest7383: please check, just to rule it out | 15:31 |
| dtantsur | well, if it's cached introspection data, maybe take a memory dump of the process and grep for it? :) | 15:31 |
| Guest7383 | About 155M, so that was pretty close. But that's the whole thing for all servers. | 15:31 |
| Guest7383 | Oh, I just realized that I'm not using my nick here. I kept wondering who that Guest guy was. One moment. | 15:33 |
| dtantsur | as a data point: on my bifrost machine (not very active), inspector has RSS of 108972k | 15:33 |
| dtantsur | lol | 15:33 |
| dtantsur | TheJulia: it's been on my todo list for a while to conduct some sort of memory profiling for our services | 15:34 |
| TheJulia | Guest7383: lol | 15:34 |
| dtantsur | to make sure we're not leaking anything | 15:34 |
| dtantsur | I wonder if there is valgrind for python... | 15:34 |
| TheJulia | we're pretty bad about forcing unused things to be deleted, because we mentally just rely upon the thread to get GC'ed out of existance | 15:34 |
| TheJulia | but yeah | 15:34 |
| dtantsur | well, you cannot force things to be deleted in Python | 15:35 |
| jingvar | ohh, started deploy | 15:35 |
| dtantsur | deploy \o/ | 15:35 |
| dtantsur | who is up for rewriting Ironic in Rust? :) | 15:35 |
| TheJulia | jingvar: \o/ | 15:35 |
| *** Guest7383 is now known as dking_temp | 15:35 | |
| TheJulia | dtantsur: I has no braincells | 15:35 |
| TheJulia | dtantsur: well, remove it from the active object in memory which is the easiest way to keep things from getting GC'ed | 15:36 |
| TheJulia | granted if memory serves that is a copy minus the object in the processor?!? | 15:36 |
| TheJulia | minus the desire deleted object | 15:36 |
| dtantsur | yeah, if there is an active link, GC will do nothing | 15:37 |
| TheJulia | exactly | 15:37 |
| dtantsur | we don't cache too many things though? | 15:37 |
| dtantsur | db connections, hash ring | 15:37 |
| dtantsur | well, I wonder if SQLAlchemy caches anything large | 15:37 |
| TheJulia | ooooh | 15:39 |
| TheJulia | but we don't save that through sqlalchemy | 15:39 |
| dtantsur | well, dking_temp saves introspection data in the database | 15:39 |
| TheJulia | oh yeah | 15:40 |
| TheJulia | then | 15:40 |
| TheJulia | it is global | 15:40 |
| dtantsur | so, technically, we do. I still cannot imagine how 155M of data can turn into 40G of RAM usage. | 15:40 |
| TheJulia | so yeah | 15:40 |
| dtantsur | on this uncertain note, I'll go get another break | 15:41 |
| jingvar | TheJulia: but where IP for the node | 15:41 |
| TheJulia | dtantsur: I'd dig at the code right now, but I'm on the TC call | 15:41 |
| *** dking_temp is now known as dking | 15:42 | |
| dking | Well, it's good to know that I found something new, at least. | 15:43 |
| TheJulia | we need a bug of the year award | 15:44 |
| TheJulia | dking: can you file an item in storyboard please? | 15:44 |
| dking | Yeah, I think I'd be famous. | 15:44 |
| dking | That might be a good idea, but I'm not sure that I can really replicate anything or give any useful diagnostic info a this point. I was kinda hoping it was either a known issue or there would be a config setting. Since it's not, I'm wondering whether I should just restart the service to see whether it just goes away and hope it was a fluke. | 15:46 |
| jingvar | it is strange, ironic cleaned up the node (I saw console), but can't set boot from nic in deploy phase | 15:47 |
| opendevreview | Merged openstack/sushy master: Raise an AccessError with SessionService init https://review.opendev.org/c/openstack/sushy/+/820076 | 15:51 |
| dking | jingvar: Others here may have better tips, but I've had similar things happen. Does anything change in your clean steps? For instance, I've had issues when changing from Legacy boot to UEFI and also when changing the BMC driver. | 15:53 |
| dking | Another problem that we had for an unreasonable amount of time turned out to be an ARP issue because we were using dhclient in a container which couldn't inject ARP, causing PXE to always fail, but only on a certain type of server, and always on exactly the second DHCP request. | 15:55 |
| jingvar | i faced with it erale | 15:56 |
| jingvar | it ok, some time you need restart procees | 15:56 |
| jingvar | and it is virtual env ... i saw a lot strange things on it | 15:58 |
| jingvar | ovs,bridges etc | 15:58 |
| dking | Yeah, that stuff can become hard to troubleshoot sometimes. | 16:00 |
| jingvar | works, I hope I can reproduce all the steps | 16:02 |
| jingvar | I push the button | 16:04 |
| TheJulia | I... thinkI only have one more meeting after my current meeting | 16:11 |
| *** tosky_ is now known as tosky | 16:17 | |
| TheJulia | *sweet* Creating iso image failed: [Errno 2] No such file or directory: '/usr/lib/syslinux/isolinux.bin': ironic.common.exception.ImageCreationFailed: Creating iso image failed: [Errno 2] No such file or directory: '/usr/lib/syslinux/isolinux.bin' | 16:20 |
| dking | Okay, so an update. When we dump the memory, it's full of request IDs (i.e. req-2f6a44d0-7a91-4e9a-b0e0-ade4709160e2) and a bit of binary data in between. Some of those requests go back to October. | 16:24 |
| TheJulia | oh joy | 16:25 |
| TheJulia | so could.... those req's trace back to ironic asking for status data? | 16:26 |
| dking | The logs show this request for one of them: "GET /v1/introspection/6572196c-8c55-40ce-9a6e-abe9cab8d723/data HTTP/1.1" status: 200 len: 38900 time: 0.1771238 | 16:27 |
| dking | That's actually a very long time compared to what I see in the logs when things are fine. | 16:28 |
| TheJulia | dking: wsgi? | 16:29 |
| TheJulia | or self-hosting process? | 16:29 |
| dking | The logs do say "eventlet.wsgi.server", but I don't think I set up anything for wsgi directly. | 16:31 |
| TheJulia | okay | 16:32 |
| TheJulia | hmmm | 16:32 |
| TheJulia | hmm | 16:32 |
| *** priteau is now known as Guest7388 | 16:38 | |
| *** priteau_ is now known as priteau | 16:38 | |
| rpittau | well this https://review.opendev.org/c/x/ironic-staging-drivers/+/819411 is finally green..... | 16:40 |
| rpittau | but I think we somehow broke ironic-inspector grenade | 16:41 |
| TheJulia | so | 16:53 |
| TheJulia | it looks like bios vmedia stuff is borked iwth lacking isolinux | 16:54 |
| TheJulia | which seems bizzar that it just broke | 16:54 |
| TheJulia | bizar | 16:54 |
| TheJulia | still doesn't seem right... | 16:54 |
| jingvar | guys | 17:08 |
| jingvar | I wrote a scrip for begginers https://etherpad.opendev.org/p/jingvar | 17:09 |
| jingvar | but failed again | 17:09 |
| rpittau | good night! o/ | 17:13 |
| jingvar | I hope yes | 17:15 |
| jingvar | I've finished | 17:28 |
| jingvar | Thank folk | 17:28 |
| dtantsur | okay, now my patch fails all CI jobs because Ironic never reacts to heartbeats from IPA. my patch does not modify anything around IPA. | 18:22 |
| dtantsur | am I just cursed? | 18:22 |
| dtantsur | oookay, I wonder if sending RPC to itself ends up using a local call.. and doing nothing because.. reasons? | 18:25 |
| dtantsur | I'll figure out tomorrow o/ | 18:26 |
| stevebaker[m] | good morning | 19:50 |
| TheJulia | o/ stevebaker[m] | 20:47 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove redfish cache entry upon errors https://review.opendev.org/c/openstack/ironic/+/820077 | 20:49 |
| TheJulia | stevebaker[m]: ^^ | 20:52 |
| stevebaker[m] | looking | 20:58 |
| TheJulia | the actual *cache* behavior testing is... lacking there | 21:03 |
| TheJulia | in theory, the only way to validate it is to follow the base sushy objects... at least with how it is all constructed | 21:05 |
| TheJulia | dunno it is worth it | 21:05 |
| cvstealth | Sorry if this is an odd question been struggling with this for a few days, but is it possible to do routed networks and using configdrives for the network metadata? I was hoping to avoid the DHCP components in the user instances as I'm attempting to use port group, which didn't seem to play nicely together. | 21:14 |
| TheJulia | cvstealth: uhhh... technically yes | 21:15 |
| TheJulia | oh | 21:15 |
| TheJulia | no dhcp at all | 21:15 |
| TheJulia | so, that would require the instances to support that "network is connected but I don't see dhcp" addressing | 21:16 |
| TheJulia | and to connect to metadata over that | 21:16 |
| TheJulia | I don't think anything actually does that out of the box | 21:16 |
| TheJulia | oh! | 21:16 |
| TheJulia | wait | 21:16 |
| TheJulia | your not talking about a network metadata service | 21:16 |
| TheJulia | your talking about metadata from config drives | 21:16 |
| cvstealth | it looked like the configdrive was generated way early in the deployment proces and when looking at the data the network_data on it is completely empty | 21:16 |
| TheJulia | Yeah | 21:17 |
| TheJulia | we embed what we ship, but the final metadata is not available to later | 21:17 |
| TheJulia | umm.. hmm | 21:17 |
| TheJulia | well, we embed what nova ships | 21:18 |
| TheJulia | but what it ships is too early | 21:18 |
| TheJulia | I'm not sure there is a *good* way for us to regenerate that or update it | 21:18 |
| cvstealth | I've even tried to put some of the network data json on the node and didn't see that getting pushed down to the configdrive | 21:18 |
| TheJulia | I *guess* we could collect it ?maybe? but we wouldn't have the final bonds bound yet | 21:18 |
| TheJulia | :\ | 21:18 |
| TheJulia | cvstealth: from nova's side, I think it strips that | 21:19 |
| TheJulia | or ignores it | 21:19 |
| TheJulia | its not pure metadata.json contents then | 21:19 |
| cvstealth | from reading the docs it seemed like that was mostly dependent on the virtual media type of drivers and we're just using straight ipmi | 21:20 |
| TheJulia | I'm not sure there is a way to do it *and* avoid dhcp | 21:20 |
| TheJulia | nah, we write through what we get for it regardless of the hardware type | 21:21 |
| * TheJulia needs to go get lunch | 21:21 | |
| * TheJulia also needs to find glasses | 21:21 | |
| cvstealth | Thanks I'll keep poking at it, just not super clear how the service network gets wired up in Neutron for the interfaces that the deployed instance should be using. | 21:23 |
| TheJulia | Ironic does it, nova tells us what the ports *should* be | 21:26 |
| TheJulia | and then we plug them in the workflow | 21:26 |
| TheJulia | and the metadata you'd need to build a bond that way from the config drive wouldn't be assembled yet at all | 21:27 |
| TheJulia | when it is written, that is | 21:27 |
| TheJulia | At least, that is how I percieve it, I'd need to doubelcheck nova. | 21:27 |
| cvstealth | The nova side makes sense to me, it's more on the Neutron side. Something like do I need 2 interfaces in the bridge_mapping 1 for the provisioning/clean network and 1 for the service network or can it be made to work off a single bridge with Neutron putting both networks on a single dnsmasq process. | 21:30 |
| TheJulia | I need to go eat and run an errand, I can try and explain it when I get back | 21:31 |
| cvstealth | Thx for the help TheJulia, I'll certainly be around for any guidance you can provide | 21:32 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!