| stevebaker | gerrit is back baby | 01:10 |
|---|---|---|
| *** pmannidi is now known as pmannidi|brb | 01:36 | |
| *** pmannidi|brb is now known as pmannidi | 01:51 | |
| opendevreview | Merged openstack/sushy master: Fix Context for EventDestination https://review.opendev.org/c/openstack/sushy/+/801034 | 01:57 |
| -opendevstatus- NOTICE: The maintenance of the review.opendev.org Gerrit service is now complete and service has been restored. Please alert us in #opendev if you have any issues. Thank you | 03:27 | |
| *** pmannidi is now known as pmannidi|brb | 03:51 | |
| *** pmannidi|brb is now known as pmannidi | 04:57 | |
| *** mgoddard- is now known as mgoddard | 06:04 | |
| arne_wiebalck | Good morning, ironic! | 06:39 |
| jssfr | Good Morning Ironic! I have a patch for the Ironic Python Agent, messed around in the gerrit sandbox, got the company to sign the CCLA. Now the process for me to achieve "CLA" status in gerrit (so that I can push things for i-p-a) is not 100% clear. Do I agree to the Individual CLA? Do I poke someone™ to unlock me based on the CCLA signed by my employer? | 06:40 |
| iurygregory | good morning arne_wiebalck janders_ and Ironic o/ | 06:43 |
| arne_wiebalck | hey iurygregory o/ | 06:44 |
| iurygregory | jssfr, https://docs.openstack.org/contributors/common/setup-gerrit.html maybe this will help | 06:44 |
| jssfr | yep, I'm looking at that | 06:44 |
| iurygregory | in my agreements config I have ICLA OpenStack Individual Contributor License Agreement | 06:45 |
| jssfr | yep, I have that, question is if that's the right one when I'm contributing on behalf of a company | 06:45 |
| iurygregory | no idea if something would change, maybe asking in the #opendev channel =) | 06:47 |
| jssfr | thanks! | 06:47 |
| arne_wiebalck | stevebaker: thanks a lot for the bifrost video! | 06:55 |
| iurygregory | stevebaker, tks for the reviews in sushy and the vendor passthru methods o/ | 06:57 |
| iurygregory | metalsmith job in Ironic seems unhappy | 07:25 |
| opendevreview | Jonas Schäfer proposed openstack/ironic-python-agent master: Expose BMC MAC address in inventory data https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 | 07:31 |
| jssfr | I hope I did this right :) | 07:31 |
| iurygregory | well you did push a change to ironic-python-agent =D so I think things are working hehe | 07:32 |
| dtantsur | morning ironic | 08:06 |
| iurygregory | morning dtantsur | 08:43 |
| iurygregory | dtantsur, if you have some minutes https://review.opendev.org/c/openstack/releases/+/801257 sushy release with the fix for events | 09:21 |
| *** rpittau|afk is now known as rpittau | 09:31 | |
| rpittau | good morning ironic! o/ | 09:31 |
| jssfr | So the Zuul run of my changeset failed in three checks where I have no clue whether my change has caused that or not. Could someone lend me a hand to figure that out? -> https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 | 09:42 |
| rpittau | jssfr: doesn't look like your patch is the cause of the failure, I issues a recheck for the time being | 09:45 |
| jssfr | thanks! | 09:45 |
| iurygregory | morning rpittau o/ | 09:50 |
| rpittau | hey iurygregory :) | 09:51 |
| rpittau | if anyone has a minute please check https://review.opendev.org/c/openstack/bifrost/+/800461 | 10:07 |
| iurygregory | will do after lunch | 10:09 |
| iurygregory | =) | 10:09 |
| iurygregory | trying to finish a few things before going for lunch hehe | 10:09 |
| opendevreview | Merged openstack/ironic master: Add a section to redfish doc for BIOS registry fields https://review.opendev.org/c/openstack/ironic/+/793982 | 10:17 |
| opendevreview | Merged openstack/bifrost master: Remove unnecessary defaults from keystone https://review.opendev.org/c/openstack/bifrost/+/800633 | 11:12 |
| opendevreview | Verification of a change to openstack/ironic failed: Scoped RBAC Devstack Plugin support https://review.opendev.org/c/openstack/ironic/+/778957 | 11:27 |
| opendevreview | Verification of a change to openstack/ironic failed: Scoped RBAC Devstack Plugin support https://review.opendev.org/c/openstack/ironic/+/778957 | 12:29 |
| iurygregory | metalsmith seems a bit red https://zuul.opendev.org/t/openstack/builds?job_name=metalsmith-integration-glance-localboot-centos8-uefi&project=openstack%2Fironic&branch=master 14 failures / 50 runs | 12:39 |
| dtantsur | :( | 12:41 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: bifrost-cli: fix relative paths in --extra-vars https://review.opendev.org/c/openstack/bifrost/+/801298 | 12:41 |
| iurygregory | always a problem when trying to run openstack image create maybe a bump in openstackclient | 12:41 |
| iurygregory | =( | 12:41 |
| iurygregory | I'm wondering why is not happeing in all runs (maybe we are getting wrong packages...) I haven't dig much on it | 12:42 |
| iurygregory | ok other failure was 'Node did not reach manageable state in 120 seconds' (this seemsok for recheck hehe) | 12:42 |
| dtantsur | yep | 12:42 |
| rpittau | shoould we start requesting bugfix releases? The remaining patches seem all bugfixes and they could be backported if needed | 12:45 |
| iurygregory | I think the ironic bugfix we can wait a bit my vendor passthru is only missing some unit tests =) (trying to finish now and will need the bump in sushy to use 3.10) | 12:46 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/801064 | 12:46 |
| iurygregory | -1 from CI is from metalsmith job hehe | 12:46 |
| iurygregory | I'm applying the suggestion from steve and finishing the unit tests | 12:47 |
| iurygregory | but we will need https://review.opendev.org/c/openstack/requirements/+/801284 | 12:47 |
| TheJulia | good morning | 12:59 |
| rpittau | good morning TheJulia :) | 13:00 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Update qemu version https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/776507 | 13:08 |
| dtantsur | morning TheJulia | 13:08 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [DNM] Test latest ansible https://review.opendev.org/c/openstack/bifrost/+/800440 | 13:10 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [DNM] Test latest ansible https://review.opendev.org/c/openstack/bifrost/+/800440 | 13:10 |
| *** sshnaidm|afk is now known as sshnaidm | 13:10 | |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Allow configuring enabled vendor interfaces https://review.opendev.org/c/openstack/bifrost/+/799837 | 13:14 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: DNM test the upgrade job https://review.opendev.org/c/openstack/bifrost/+/800673 | 13:16 |
| iurygregory | good morning TheJulia | 13:38 |
| arne_wiebalck | TheJulia: Regarding the source of the DB load: I don't know tbh. There are two issues we probably need to disentangle: sustained high DB load and the thundering herd at startup. | 13:40 |
| TheJulia | So I guess to provide context, I *did* go look through our conductor startup, and nothing screamed like it should generate insane amounts of load starting up the conductor service alone | 13:41 |
| TheJulia | Periodics... once started could be another thing though | 13:41 |
| jssfr | Zuul rejected my change again, different error picture now: https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 should I just recheck or may there be actually something wrong with my change? | 13:42 |
| TheJulia | But they would be regularized load and maybe the need is really just spread them out a little more | 13:42 |
| dtantsur | jssfr: iurygregory mentioned earlier that metalsmith jobs are pretty much broken. you may wait it out a bit. | 13:43 |
| TheJulia | Oo | 13:43 |
| TheJulia | gateway errors? | 13:43 |
| iurygregory | nah | 13:43 |
| iurygregory | well | 13:43 |
| iurygregory | openstack image create complains and there is something about gateway I think | 13:44 |
| * iurygregory checks again | 13:44 | |
| TheJulia | yeah, I spotted that on a devstack change which failed to merge | 13:44 |
| TheJulia | and I thought it was a fluke, but.... | 13:44 |
| TheJulia | obviously not | 13:44 |
| iurygregory | yeah it gives Bad Gateway: 502 Bad Gateway | 13:44 |
| TheJulia | So why is glance blowing up then? | 13:44 |
| iurygregory | yup | 13:44 |
| TheJulia | did we not give it cookies? | 13:44 |
| jssfr | dtantsur, ah, I didn't notice. Thanks! | 13:45 |
| iurygregory | I think we forgot the cookies and *BOOM* | 13:45 |
| arne_wiebalck | TheJulia: For the sustained/background load: If I am to trust our DB monitoring, with ~8000 nodes and 20 conductor groups (so 20 nova-computes), there are 30k rows read every second, in 100 selects/sec (which is how I read the graphs, will need to check with colleagues to confirm). | 13:48 |
| TheJulia | iurygregory: looks like oslo.limits is now in place in glance and the default is rediculiously small | 13:49 |
| iurygregory | YAY?! | 13:49 |
| TheJulia | devstack@g-api.service[81425]: DEBUG oslo.limit.limit [None req-96e9e03d-e649-4144-89f5-a9bc3d96fe10 admin admin] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 1112 and delta 0] {{(pid=81425) enforce_limits /usr/local/lib/python3.8/dist-packages/oslo_limit/limit.py:268}} | 13:49 |
| * dtantsur rolls eyes | 13:49 | |
| TheJulia | so the image size limit, if memory serves goes into keystone | 13:52 |
| TheJulia | and the example docs are 1GB | 13:52 |
| TheJulia | looks like dan smith shoved limits into the devstack plugin | 13:55 |
| TheJulia | defaulting to a hardcoded value | 13:55 |
| * TheJulia goes and fixes | 13:55 | |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add vendor_passthru method for subscriptions https://review.opendev.org/c/openstack/ironic/+/801064 | 13:56 |
| iurygregory | "defaulting to a hardcoded value" that doesn't work with Ironic :D | 13:57 |
| TheJulia | https://review.opendev.org/c/openstack/devstack/+/801309 | 14:05 |
| iurygregory | awesome | 14:14 |
| opendevreview | Julia Kreger proposed openstack/metalsmith master: DNM: Test CI https://review.opendev.org/c/openstack/metalsmith/+/781549 | 14:14 |
| opendevreview | Merged openstack/sushy-tools master: Fix to handle correct path for BiosRegistry https://review.opendev.org/c/openstack/sushy-tools/+/801099 | 14:16 |
| dtantsur | I wonder if we can use limits too? | 14:24 |
| TheJulia | I think that would be a good ptg topic | 14:26 |
| TheJulia | I think we completely avoided the topic when we were strictly an admin only service, but now that the world has evolved it woudl be a good topic | 14:29 |
| dtantsur | mm, PTG, yeah | 14:30 |
| dtantsur | I remember this word :) | 14:30 |
| opendevreview | Verification of a change to openstack/metalsmith failed: Replace deprecated import of ABCs from collections https://review.opendev.org/c/openstack/metalsmith/+/801089 | 14:37 |
| TheJulia | I think someone may be using some sort of time travel technology, because the last few months seem like a complete blur | 14:42 |
| arne_wiebalck | TheJulia: I have an issue with fast-track, which may be a race (or me no seeing the issue, ofc :). | 14:43 |
| arne_wiebalck | TheJulia: https://opendev.org/openstack/ironic/src/branch/master/ironic/conductor/utils.py#L1107 returns False | 14:43 |
| arne_wiebalck | TheJulia: Logging the 2 conditions right before tell me I should not be in this if branch. | 14:44 |
| arne_wiebalck | "2021-07-19 16:13:10 INFO fast_track_able: True" | 14:44 |
| arne_wiebalck | "2021-07-19 16:13:15 INFO power state: power on" | 14:44 |
| dtantsur | are you sure it's not the next branch? | 14:45 |
| arne_wiebalck | dtantsur: yes | 14:45 |
| arne_wiebalck | dtantsur: otherwise I would not see my log statements | 14:45 |
| arne_wiebalck | dtantsur: and I would see the debug message | 14:46 |
| dtantsur | right. weird. | 14:46 |
| arne_wiebalck | dtantsur: this is the code: https://paste.opendev.org/show/807561/ | 14:46 |
| dtantsur | wow, I haven't seen anything like that | 14:47 |
| arne_wiebalck | hmm ... just realise now: there is quite a time difference between the two log statements ... | 14:49 |
| * TheJulia can look in a few minutes, typing my "what did I do last week" notes | 14:50 | |
| arne_wiebalck | getting the power state is talking to the BMC, not the DB? checking ... | 14:50 |
| arne_wiebalck | TheJulia: thanks! | 14:50 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Perform upgrade by default when installing ansible https://review.opendev.org/c/openstack/bifrost/+/800461 | 14:50 |
| arne_wiebalck | TheJulia: dtantsur: I should add that the clean step runs for 3600 seconds, so I wonder if the ramdisk heart beat timeout plays a role (300 seconds I think)? I see the node heartbeating, though. | 14:56 |
| iurygregory | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Jul 19 14:59:59 2021 UTC and is due to finish in 60 minutes. The chair is iurygregory. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| iurygregory | o/ | 15:00 |
| rpioso | \o | 15:00 |
| JayF | o/ | 15:00 |
| TheJulia | arne_wiebalck: any last_error | 15:00 |
| erbarr | o/ | 15:00 |
| TheJulia | o/ | 15:00 |
| rloo | o/ | 15:00 |
| iurygregory | Hello everyone! Welcome to our weekly meeting! | 15:00 |
| stendulker | o/ | 15:00 |
| cenne | o/ | 15:00 |
| iurygregory | You can find our agenda in the wiki | 15:00 |
| iurygregory | #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting | 15:00 |
| iurygregory | #topic Announcements / Reminders | 15:01 |
| arne_wiebalck | TheJulia: last_error is Non | 15:01 |
| opendevreview | cenne proposed openstack/ironic master: [WIP] Add api endpoint for setting boot_mode on managed node https://review.opendev.org/c/openstack/ironic/+/800084 | 15:01 |
| arne_wiebalck | None | 15:01 |
| arne_wiebalck | o/ | 15:01 |
| iurygregory | #info we will cut the new bugfix branches this week | 15:01 |
| rpittau | o/ | 15:01 |
| iurygregory | rpittau, do you want to add some info about when we will cut the release? | 15:01 |
| rpittau | ideally tomorrow :) | 15:02 |
| iurygregory | and also, do we have any more announcements or remminders? =) | 15:02 |
| TheJulia | also, release notes should be scrubbed | 15:02 |
| rpittau | yep, let's see how that goes | 15:02 |
| iurygregory | #topic Review action items from previous meeting | 15:02 |
| dtantsur | o/ | 15:03 |
| iurygregory | No action items from last meeting, skipping =) | 15:03 |
| iurygregory | I'm going a bit fast because we have quite a few topics in discussion =) | 15:03 |
| opendevreview | Merged openstack/ironic stable/wallaby: Redfish: Get only RAID controller's physical disks https://review.opendev.org/c/openstack/ironic/+/800930 | 15:03 |
| opendevreview | Merged openstack/ironic stable/wallaby: Update Redfish RAID disk_type unit test https://review.opendev.org/c/openstack/ironic/+/800931 | 15:03 |
| opendevreview | Merged openstack/ironic master: Burn-in: Add documentation https://review.opendev.org/c/openstack/ironic/+/800806 | 15:04 |
| iurygregory | #topic Review subteam status reports | 15:04 |
| opendevreview | Merged openstack/ironic-inspector master: Update requirements https://review.opendev.org/c/openstack/ironic-inspector/+/800680 | 15:04 |
| iurygregory | #link https://etherpad.opendev.org/p/IronicWhiteBoard | 15:04 |
| iurygregory | starting on L65 | 15:04 |
| JayF | I'll note I do not expect further updates for Anaconda; AFAIK there is nobody planning to tackle the CI work right now. | 15:05 |
| iurygregory | JayF, can you add the info in the etherpad? | 15:05 |
| JayF | The last update is accurate; it's got a link to the story documenting the CI hurdles. | 15:06 |
| iurygregory | oh you added in 2021-07-09 :D | 15:06 |
| iurygregory | to me the last update was the one I added hehe | 15:06 |
| TheJulia | I guess I might try to enhance the metadata scrub for happiness of ceph | 15:08 |
| iurygregory | arne_wiebalck, no updates in Node error history right? | 15:08 |
| TheJulia | iurygregory: I've not seen anything | 15:08 |
| iurygregory | yeah, I saw the patch is in merge conflict | 15:09 |
| arne_wiebalck | no sorry: TheJulia rebased the DB patch, but it needs another rebase now I see | 15:09 |
| TheJulia | ugh | 15:09 |
| TheJulia | I need to find a good code problem to sink my brain into, I'll try to again this week | 15:09 |
| iurygregory | we have updates in everything, moving on | 15:10 |
| iurygregory | #topic Deciding on priorities for the coming week | 15:10 |
| iurygregory | #link https://tinyurl.com/ironic-weekly-prio-dash | 15:10 |
| TheJulia | dtantsur: w/r/t driver info, how do you feel about some of the username insanity? | 15:10 |
| dtantsur | TheJulia: I'm totally for it, just not sure if I have time | 15:10 |
| iurygregory | Let us know if you have any patches that we should add =) (we will cut the bugfix branch by tomorrow so if there is something urgent that you would like let us know) | 15:11 |
| dtantsur | depends on how openshift staff works out (I'll know next week, I hope) | 15:11 |
| dtantsur | re patch, I only have https://review.opendev.org/c/openstack/bifrost/+/799837 | 15:11 |
| TheJulia | dtantsur: I can try and make time, its been on my mind recently | 15:11 |
| dtantsur | would be great | 15:11 |
| iurygregory | I've added https://review.opendev.org/c/openstack/ironic/+/801064 vendor passthru for subscriptions =) | 15:11 |
| dtantsur | ++ to ^^^ | 15:11 |
| iurygregory | tested in real HW Dell R640 and HPE e910 =) | 15:12 |
| TheJulia | how do we feel about https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 ? | 15:12 |
| iurygregory | bifrost ++ when testing new things :D | 15:12 |
| TheJulia | I tagged the bifrost change | 15:12 |
| iurygregory | CI seems a bit unhappy not sure if is because of the change | 15:12 |
| iurygregory | but we can add the tag and I can take a look at it later | 15:13 |
| TheJulia | yeah, true | 15:13 |
| rpittau | it's the metalsmith job | 15:13 |
| TheJulia | I was asking more from a point of "do we have any thoughts/feelings | 15:13 |
| TheJulia | " | 15:13 |
| iurygregory | oh ok I saw ipa-tempest-ironic-inspector-src | 15:13 |
| TheJulia | point of view | 15:13 |
| iurygregory | but is non-voting already =) | 15:13 |
| iurygregory | probably need a release note and a story I would say | 15:14 |
| TheJulia | Has anyone reviewed backport candidates to make sure we've actually done the backports? | 15:14 |
| rpittau | I did on some, probably not all | 15:14 |
| TheJulia | k | 15:14 |
| iurygregory | I will take a look at the backports later today | 15:15 |
| iurygregory | I've added the tag in https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 | 15:15 |
| TheJulia | k | 15:15 |
| TheJulia | iurygregory: thanks | 15:15 |
| iurygregory | jssfr, can you please add a releasenote on your patch? https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 let us know if you have any questions on how to do that =) | 15:16 |
| TheJulia | iurygregory: if there is anything that you feel should go to train, lmk | 15:16 |
| iurygregory | TheJulia, ack | 15:16 |
| iurygregory | are we done with priorities? | 15:17 |
| TheJulia | I think so | 15:17 |
| iurygregory | ok, moving on | 15:17 |
| iurygregory | #topic Discussion | 15:17 |
| iurygregory | we have a few topics from TheJulia and rpittau =) | 15:18 |
| iurygregory | TheJulia, the mic is yours =) | 15:18 |
| TheJulia | So first up is the PTG | 15:18 |
| iurygregory | PTG \o/ | 15:18 |
| TheJulia | It is coming up in... ?12-13? weeks or something like that. | 15:18 |
| * TheJulia hopes she didn't just book vacation for the same week | 15:18 | |
| TheJulia | Anyway, We as a project need to figure a few things out. | 15:18 |
| rpittau | October 18-22, 2021 | 15:19 |
| * iurygregory will look to not book vacation during the PTG week also | 15:19 | |
| TheJulia | 1) timing, how much time we need, 2) topics to be discussed, Who is going to represent Ironic with the TC and who is planning it | 15:19 |
| TheJulia | rpittau: ugh, yup, that is the week :( | 15:19 |
| * TheJulia un-does vacation booking | 15:19 | |
| rpittau | :/ | 15:19 |
| iurygregory | I can help planning | 15:19 |
| iurygregory | organize the etherpad and schedule | 15:20 |
| iurygregory | #link https://etherpad.opendev.org/p/ironic-yoga-ptg | 15:20 |
| dtantsur | same. I'll be available in October. | 15:20 |
| TheJulia | Sounds good! | 15:20 |
| rpittau | I should be there too | 15:20 |
| iurygregory | we can start adding topics we would like to discuss | 15:20 |
| TheJulia | ++ | 15:20 |
| * rpittau disapproves "yoga" btw | 15:20 | |
| iurygregory | yoda would be better :D | 15:20 |
| TheJulia | heh | 15:21 |
| dtantsur | probably trademarked :) | 15:21 |
| iurygregory | yeah =) | 15:21 |
| iurygregory | regarding time this will depend on how much topics we will have, we can't make a decision now | 15:21 |
| TheJulia | Any volunteer for representing the team with the TC ? | 15:21 |
| iurygregory | depends | 15:21 |
| dtantsur | I can do it | 15:21 |
| iurygregory | what the person needs to do hehe | 15:21 |
| dtantsur | ranting a lot | 15:22 |
| iurygregory | ok we have dtantsur \o/ | 15:22 |
| TheJulia | dtantsur: that would be much appreciated | 15:22 |
| dtantsur | leave that me, my friend :D | 15:22 |
| cenne | Yeren? | 15:22 |
| iurygregory | :D | 15:22 |
| * TheJulia can change that week to half pto now | 15:22 | |
| dtantsur | TheJulia: just remind me closer to $subj in case I forget | 15:22 |
| TheJulia | dtantsur: ack | 15:22 |
| TheJulia | so next topic, is a question of if we feel we need a virtual midcycle? | 15:22 |
| rpittau | IHMO it's been proved useful in the past, I vote for having one | 15:23 |
| dtantsur | possibly? we're in vacation season, but we can try | 15:23 |
| TheJulia | It *feels* like we've all been absurdly busy the last couple months, so I'm not sure if it would help right now | 15:23 |
| TheJulia | dtantsur: that is my concern too | 15:23 |
| TheJulia | rpittau: true | 15:23 |
| iurygregory | yeah we can try, at least having it would help to collect topics for the PTG probably | 15:23 |
| rpittau | mmm yeah, vacations | 15:24 |
| TheJulia | So maybe one for in a few weeks? | 15:24 |
| iurygregory | and maybe start discussions.. | 15:24 |
| TheJulia | iurygregory: that sounds like a good idea | 15:24 |
| dtantsur | let's have a doodle for a few slots? I'll be out on week in August and one in September | 15:24 |
| TheJulia | even if those calls are just to kind of "lets tease out some of the topics for the ptg" | 15:24 |
| iurygregory | maybe a doodle to figure out when we should do? | 15:24 |
| TheJulia | any volunteers for a doodle? | 15:24 |
| rpittau | doodle sounds good | 15:24 |
| iurygregory | I will | 15:24 |
| iurygregory | I will send to the ML | 15:24 |
| * TheJulia feels like we need #action :) | 15:24 | |
| TheJulia | Okay! So next up on topics! | 15:25 |
| iurygregory | #action iurygregory to create a doodle for the midcycle | 15:25 |
| TheJulia | I've been chatting with the Open Infra Live folks. | 15:25 |
| * iurygregory hopes this is correct ^ | 15:25 | |
| TheJulia | iurygregory: it is correct | 15:25 |
| TheJulia | Is anyone interested in particpating in an ironic overview and operator usage of ironic discusison call for OpenInfra live? | 15:26 |
| TheJulia | We've got a couple windows which *are* open we can book into, but we need to move quickly if we want to do so | 15:26 |
| TheJulia | August 5th or August 12th are presently open | 15:27 |
| dtantsur | I may be, depending on what exactly is required | 15:27 |
| dtantsur | not on the 12th though | 15:27 |
| iurygregory | I can probably help but I would be afraid to do alone the ironic overview =D | 15:27 |
| TheJulia | dtantsur: that would be ideal because I'm trying to organize an operator feedback session as well | 15:27 |
| arne_wiebalck | I am happy to join (unless it is felt there is too much CERN presence already from the last sessions :) | 15:27 |
| dtantsur | we can do it together, I have experience organizing such things (so does arne_wiebalck) | 15:27 |
| dtantsur | are these two different things? | 15:28 |
| dtantsur | operator usage and operator feedback? | 15:28 |
| TheJulia | how about this? I'll create a basic presentation slide deck, tag you guys on it, and maybe we can figure out what we want to spend 10-20 minutes kind of walking through, and then kind of try to get operators talking? | 15:28 |
| dtantsur | I suspect you won't get enough presence for 2 similar calls | 15:28 |
| dtantsur | yep | 15:28 |
| TheJulia | arne_wiebalck: I am kind of hoping for other operators at present, there has been some outreach already just waiting to hear back | 15:28 |
| TheJulia | well, the operator feeedback sesison is more openstack-wide | 15:29 |
| arne_wiebalck | TheJulia: ++ | 15:29 |
| TheJulia | not *just* ironic, and it may completely fizzle. We'll see. | 15:29 |
| dtantsur | ah | 15:29 |
| dtantsur | we had a very successful Russian ironic meetup a few months ago, but I did receive a lot of help wrt outreach | 15:29 |
| arne_wiebalck | TheJulia: the ops feedback session would also be on OpenInfra live? | 15:31 |
| TheJulia | arne_wiebalck: it would be | 15:31 |
| TheJulia | and community wide is the hope | 15:31 |
| arne_wiebalck | TheJulia: in addition or to replace the one we are trying to put together for the next SIG meeting? | 15:32 |
| TheJulia | Anyway, I wasn't planning on discussing the community wide feedback session one this morning | 15:32 |
| TheJulia | Oh, yeah, we did talk about that didn't we | 15:32 |
| TheJulia | maybe replace... maybe | 15:32 |
| TheJulia | not entirely sure right now | 15:32 |
| arne_wiebalck | we need to make we do not confuse the few operators willing to show up on either of these occasions :) | 15:33 |
| iurygregory | we can decide after the presentation and the operator feedback hehe | 15:33 |
| opendevreview | Derek Higgins proposed openstack/ironic stable/wallaby: Allow reboot to hard disk following iso ramdisk deploy. https://review.opendev.org/c/openstack/ironic/+/801343 | 15:33 |
| TheJulia | arne_wiebalck: ++ | 15:33 |
| arne_wiebalck | *make sure | 15:33 |
| TheJulia | okay, I'll create a slide deck and we'll go from there I think | 15:34 |
| TheJulia | well, the beginnings of a slide deck | 15:34 |
| iurygregory | rpittau, your topic now =) | 15:34 |
| rpittau | thanks iurygregory :) | 15:35 |
| * rpittau having some network issues | 15:35 | |
| rpittau | anyway | 15:35 |
| rpittau | maybe a crazy proposal but I'm seriosuly considering that having the lower-constraints tests back could be a good thing | 15:36 |
| rpittau | only for the current development branch | 15:36 |
| iurygregory | hummm | 15:36 |
| rpittau | this is because we usually don't update the lowest requirements | 15:36 |
| TheJulia | Why, and how will we ensure we don't leak them into stable branches forevermore? | 15:36 |
| TheJulia | This is a good point. | 15:36 |
| rpittau | well | 15:36 |
| rpittau | I came up with some interesting issues when installing packetized ironic code recently | 15:37 |
| dtantsur | I think the problem with lower-constraints where that they don't work as we expect | 15:37 |
| rpittau | depends | 15:37 |
| rpittau | if properly configured they do | 15:37 |
| TheJulia | I thought he dependency resolver changes made it impossible | 15:37 |
| iurygregory | *if properly configured they do* - this is what I'm afraid hehe | 15:38 |
| dtantsur | well, we don't have an operation "install minimum necessary packages in pip" | 15:38 |
| rpittau | I gave that a try locally and it worked ok | 15:38 |
| dtantsur | so we'll end up with lower bounds for stuff we depend on directly and with random packages that are indirect dependencies | 15:38 |
| rpittau | the alternative to automatically have such test is to have someone do an update manually from time to time | 15:39 |
| rpittau | I did that last week for example | 15:39 |
| dtantsur | We can do it 1) only for master, 2) providing some automation to update them | 15:39 |
| rpittau | the main reason is that what we have in requirements is not correct | 15:40 |
| dtantsur | well... it kind of is | 15:40 |
| rpittau | eheh now it is :) | 15:40 |
| dtantsur | that transient dependencies bring in a newer version is less of a concern | 15:40 |
| dtantsur | what we care about is the minimum version that our code is written against | 15:41 |
| dtantsur | if a dependency bump it further up - great | 15:41 |
| * TheJulia transforms to a fly on the wall with minimal opinion | 15:41 | |
| dtantsur | TheJulia: a fly next to a cat and a corgi? you're brave. | 15:41 |
| TheJulia | dtantsur: iknowrite! | 15:42 |
| iurygregory | lol | 15:42 |
| dtantsur | rpittau: I don't necessarily disagree with bringing the job back, just trying to set the expectations right | 15:43 |
| TheJulia | I'm trying to recall what the resolver breakage issue was that triggered it all to get ripped out | 15:43 |
| dtantsur | well, the resolver started taking requirements and constraints seriously | 15:43 |
| rpittau | TheJulia: basically an infinite loop cause it could not resolve proper dependencies | 15:43 |
| iurygregory | yeah ^ | 15:43 |
| TheJulia | ahh, right | 15:43 |
| dtantsur | rather than just coming up with some combination of packages that may or may not satisfy them | 15:43 |
| rpittau | yep | 15:43 |
| dtantsur | since that broke stable branches, we had no real choice other than drop the job | 15:44 |
| TheJulia | so instead of constraints, it should be treated as the lowest possible version, but then it seems like it is duplicating requirements.txt | 15:44 |
| rpittau | I think we can ignore stable branches | 15:44 |
| dtantsur | I think we have to | 15:44 |
| dtantsur | anyway, I'm curious to see how the process of updating a requirement will look like. other than that, no real objections. | 15:45 |
| TheJulia | no objections from myself as long as we don't shoot ourselves in the foot | 15:45 |
| dtantsur | wait, isn't that what software is about?? | 15:46 |
| iurygregory | same from me =) | 15:46 |
| TheJulia | dtantsur: oh my | 15:46 |
| dtantsur | :D | 15:46 |
| iurygregory | dtantsur, omg XD | 15:46 |
| TheJulia | dtantsur: no footguns as a service, right now | 15:46 |
| rpittau | I'll do a test to see if the behavior of the job has actually improved as I expect | 15:46 |
| rpittau | that's all :) | 15:46 |
| iurygregory | tks rpittau ! | 15:46 |
| iurygregory | moving to the next topic \o/ | 15:47 |
| iurygregory | #topic Baremetal SIG | 15:47 |
| iurygregory | #info Next Baremetal SIG meeting in September, skipping August due to vacations. | 15:47 |
| arne_wiebalck | The session from last week on Bifrost by TheJulia is already online. Thanks stevebaker. | 15:47 |
| TheJulia | stevebaker rocks | 15:48 |
| iurygregory | ++, great job everyone! | 15:48 |
| arne_wiebalck | I will add an entry to the website as well. | 15:48 |
| iurygregory | the presentation was amazing =) | 15:48 |
| TheJulia | I guess it flowed nicely? | 15:48 |
| iurygregory | totally =) | 15:48 |
| * TheJulia can't actually watch recordings of herself | 15:48 | |
| arne_wiebalck | So, no session in August. We need to decide on the topic for Sep (which tentatively is/was ops feedback). | 15:49 |
| iurygregory | arne_wiebalck, you will send an email to the ML saying that the next meeting will be in Sep? | 15:49 |
| arne_wiebalck | But this may move to Infralive. | 15:49 |
| iurygregory | I don't think we need to say the topic but we should probably let people know in the ML I would say =) | 15:50 |
| arne_wiebalck | iurygregory: not until we know what the topic will be, I think. | 15:50 |
| iurygregory | gotcha! | 15:50 |
| arne_wiebalck | let's see how the openinfra thing comes along | 15:50 |
| iurygregory | ack | 15:50 |
| TheJulia | arne_wiebalck: I should have that pinned down this week | 15:51 |
| arne_wiebalck | I still have some topics we could do for a 10 min presentation/discussion | 15:51 |
| TheJulia | we *should* enumerate a list of 10 minute topics | 15:51 |
| arne_wiebalck | TheJulia: great | 15:51 |
| iurygregory | I can probably do a demo with the event subscription using vendor passthru also | 15:51 |
| arne_wiebalck | TheJulia: there is no rush I thnk | 15:51 |
| arne_wiebalck | *think | 15:51 |
| TheJulia | arne_wiebalck: agreed | 15:51 |
| iurygregory | but only to create delete get subscriptions (not showing how they work sending information :D) | 15:52 |
| arne_wiebalck | iurygregory: will note this down | 15:52 |
| arne_wiebalck | in general, I think we should have some more basic topics | 15:52 |
| arne_wiebalck | to attract more people | 15:52 |
| iurygregory | it would help new comers to understand how they would implement things I would say | 15:52 |
| dtantsur | have we ever talked about metal3? is it something people want someone to talk about? | 15:52 |
| iurygregory | I can probably mention how we can update code to test etc | 15:52 |
| arne_wiebalck | dtantsur: definitely! | 15:52 |
| dtantsur | I can do that unless someone else from our team wants to? | 15:53 |
| iurygregory | I'm not metal3 expert =) | 15:53 |
| arne_wiebalck | iurygregory: yeah, a how to contribute session would be sensible as well I think | 15:53 |
| iurygregory | arne_wiebalck, ++ | 15:53 |
| arne_wiebalck | I also have a "how to debug" session in mind | 15:53 |
| TheJulia | dtantsur: I think it would be good to talk about as something integrating, helping collaborate | 15:53 |
| dtantsur | okay, sign me up | 15:54 |
| TheJulia | iurygregory: I think a newcomers episode would be awesome | 15:54 |
| dtantsur | I can do both a short version and a long one, up to you | 15:54 |
| iurygregory | seems like we will have topics for 2022 :D | 15:54 |
| TheJulia | ++ | 15:54 |
| dtantsur | not bad :) | 15:54 |
| iurygregory | yeah! | 15:55 |
| TheJulia | can we get some of these added to the baremetal sig etherpad? | 15:55 |
| arne_wiebalck | please put your ideas on the bare metal etherpad so they do not get lost! | 15:55 |
| arne_wiebalck | :-D | 15:55 |
| * arne_wiebalck types (or thinks) slower than TheJulia | 15:55 | |
| iurygregory | #link https://etherpad.opendev.org/p/bare-metal-sig | 15:55 |
| * arne_wiebalck or both | 15:55 | |
| TheJulia | iurygregory: line 160-177 | 15:56 |
| arne_wiebalck | thx everyone, that is it for the SIG, I believe | 15:56 |
| iurygregory | awesome tks arne_wiebalck ! | 15:56 |
| iurygregory | #topic RFE review | 15:56 |
| iurygregory | we don't have any items, skipping | 15:56 |
| iurygregory | #topic Open Discussion | 15:57 |
| iurygregory | I don't think we will have time to discuss something... | 15:57 |
| iurygregory | but I will give 1minute =D | 15:57 |
| TheJulia | My brain is out of spoons | 15:57 |
| TheJulia | it needs ice cream, and a chiropractor | 15:57 |
| iurygregory | ice cream ++ | 15:58 |
| iurygregory | #topic Who is going to run the next meeting? | 15:58 |
| iurygregory | Do we have any volunteers? | 15:58 |
| dtantsur | I can do it | 15:58 |
| iurygregory | tks dtantsur! | 15:58 |
| iurygregory | #endmeeting | 15:58 |
| opendevmeet | Meeting ended Mon Jul 19 15:58:47 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-07-19-14.59.html | 15:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-07-19-14.59.txt | 15:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-07-19-14.59.log.html | 15:58 |
| arne_wiebalck | thanks iurygregory ! | 15:59 |
| iurygregory | np o/ | 15:59 |
| dtantsur | oh, I see I already promised the ramdisk deploy. it can be a short topic. | 15:59 |
| iurygregory | time for some exercises, be back later o/ | 16:00 |
| * arne_wiebalck takes the slightest mention as a commitment and puts it on the etherpad :) | 16:00 | |
| opendevreview | Verification of a change to openstack/ironic-python-agent failed: Force immediate NTP time sync with chronyd at IPA startup https://review.opendev.org/c/openstack/ironic-python-agent/+/801032 | 16:10 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Test lower-constraints https://review.opendev.org/c/openstack/ironic/+/801350 | 16:15 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Test lower-constraints https://review.opendev.org/c/openstack/ironic/+/801350 | 16:15 |
| TheJulia | arne_wiebalck: that is an *excellent* practice | 16:17 |
| TheJulia | dtantsur: seems my limits change is getting a little push back :\ | 16:17 |
| cenne | TheJulia Can you help me figure what I am missing in regards to policy? I am getting a `405 Method Not Allowed` in tests. | 16:17 |
| cenne | this patch : https://review.opendev.org/c/openstack/ironic/+/800084 | 16:18 |
| TheJulia | cenne: do you have a change uploaded | 16:18 |
| TheJulia | oh | 16:18 |
| TheJulia | okay | 16:18 |
| JayF | I'll look, too, I have a bit of free time today | 16:18 |
| TheJulia | cenne: give me a few minutes to make sure nothing is on fire and I'll take a look | 16:18 |
| cenne | yes yes :) | 16:18 |
| cenne | Thanks. | 16:19 |
| TheJulia | Yes, looks like I can | 16:19 |
| dtantsur | TheJulia: le sigh. I think centos not fitting into these limits is a strong argument tbh | 16:24 |
| TheJulia | yeah | 16:32 |
| TheJulia | I'm guessing it is globally on, but tbh I didn't dig into base devstack config | 16:33 |
| TheJulia | erbarr: Are you guys re-stacking every change that is tested? | 16:47 |
| opendevreview | Verification of a change to openstack/ironic-inspector failed: Remove grenade jobs from old stable branches https://review.opendev.org/c/openstack/ironic-inspector/+/800466 | 16:47 |
| rpioso | TheJulia: I believe we are. The test executor VM that hosts DevStack and runs the test job is ephemeral. erbarr can confirm. | 16:50 |
| erbarr | yea, we checkout devstack and stack on each run, then it goes poof after the run | 16:50 |
| rpioso | erbarr: The whole VM, right? | 16:51 |
| erbarr | yea | 16:51 |
| TheJulia | erbarr: what os are you deploying to the nodes? | 16:56 |
| TheJulia | I'm asking because the change that has broken metalsmith deployments shoudl be breaking your CI if your using images over 1GB in size | 16:57 |
| erbarr | baremetal? or the test executors? | 16:57 |
| TheJulia | to baremetal | 16:57 |
| erbarr | I'm using dib on the jobs to create images based on ubuntu, is the default os version of that, last time I checked that was bionic | 16:58 |
| TheJulia | okay | 16:58 |
| TheJulia | that is why your CI still works then | 16:59 |
| erbarr | cool, should I change it to use some pre-built images? | 17:01 |
| iurygregory | bionic?! | 17:02 |
| iurygregory | your CI runs devstack in bionic? O.o | 17:02 |
| erbarr | no, devstack runs on focal except on ussuri or train, which btw, should I drop train? | 17:03 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: DNM test the upgrade job https://review.opendev.org/c/openstack/bifrost/+/800673 | 17:04 |
| erbarr | i saw this one, and that made me wonder if I should stop testing train changes: https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/799678 | 17:05 |
| TheJulia | erbarr: We still use train for osp downstream 16.x so... backport wise we're going to be super cautious and expect stuff has been tested somehow if there are any more backports to it for drivers | 17:08 |
| TheJulia | I can't in good faith accept a driver change/backport without some sort of testing or affirmation that the world won't break | 17:09 |
| erbarr | okay, I was just wondering but seems I misunderstood what that change was about, thanks :) | 17:15 |
| TheJulia | ahh, no, we remove it from the plugin because we don't anticipate we will make any tempest changes that will impact it and already getting changes into tempest with gating master against stable/train is just... a nightmare | 17:21 |
| TheJulia | cenne: so... wired. I'm not quite sure your getting an error | 17:23 |
| TheJulia | cenne: since policy enforement/checking is not really in the base api unit testing | 17:23 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Set glance limit for baremetal friendly images https://review.opendev.org/c/openstack/ironic/+/801365 | 17:32 |
| opendevreview | Julia Kreger proposed openstack/metalsmith master: Set glance limit for baremetal friendly images https://review.opendev.org/c/openstack/metalsmith/+/801367 | 17:36 |
| TheJulia | iurygregory: ^^^ should fix metalsmith | 17:36 |
| rpittau | good night! o/ | 17:38 |
| *** rpittau is now known as rpittau|afk | 17:38 | |
| TheJulia | iurygregory: fyi, the devstack change for the limit configurability has been approved, so I think you'd be safe single core approving if you want | 17:39 |
| cenne | TheJulia: Oh. Okay that is indeed weird. | 17:56 |
| * cenne wonders who's putting up a 405 | 17:58 | |
| iurygregory | TheJulia, ack will do that | 18:02 |
| iurygregory | I've added myself in the devstack change to get the notification when it merges and the metalsmith one (after the zuul +1 I will +W)) | 18:03 |
| cenne | TheJulia: Found it. Sorry. I was being silly. | 18:04 |
| cenne | I did write the function, but as it appears, I was also supposed to put it in the _custom_actions list. | 18:05 |
| JayF | that api code is tricky; I'm glad you figured it out! | 18:06 |
| cenne | Just did it. | 18:06 |
| cenne | Yes, On to the next errors :) | 18:06 |
| JayF | I always considered it a victory when I could change the error | 18:07 |
| JayF | because at least then you know you can modify the behavior in a meaningful way :D | 18:07 |
| cenne | :P | 18:08 |
| TheJulia | cenne: posted some comments to your change, looking pretty good overall :) | 18:28 |
| cenne | Thanks. ! | 18:28 |
| arne_wiebalck | TheJulia: dtantsur: the fast track glitch I reported earlier comes from Ironic not being able to get the power state ("error" != "power on"), but then inside the if the power state is "power on" ... any suggestions how to handle this? Retry maybe? Could still be faster than doing a full power cycle. | 18:47 |
| arne_wiebalck | TheJulia: dtantsur: this is quite reproducible, so I am somewhat puzzled. | 19:23 |
| iurygregory | I love how random are the results of tox -e pep8 in CI and my machine O.o https://review.opendev.org/c/openstack/ironic/+/801064 | 19:44 |
| TheJulia | arne_wiebalck: so we're trying to do an ad-hoc power state check and we're getting "error" | 20:06 |
| TheJulia | ? | 20:06 |
| arne_wiebalck | TheJulia: yes | 20:06 |
| arne_wiebalck | TheJulia: and "power on" an a retry | 20:06 |
| arne_wiebalck | TheJulia: where the retry is just 2 lines down, so *really* quickly after | 20:07 |
| arne_wiebalck | TheJulia: I can try to find out *why* it is failing | 20:07 |
| arne_wiebalck | TheJulia: not sure that leads anywhere | 20:08 |
| TheJulia | I would add a retry | 20:09 |
| TheJulia | that check shouldn't be failing | 20:09 |
| TheJulia | but not every bmc is super friendly on power state checking | 20:09 |
| timeu | 20:10 | |
| arne_wiebalck | What I observe is that it *seems* to happen only when the clean step (CPU burn-in runs for at least 1800 secs), with shorter runtime all is fine. | 20:13 |
| arne_wiebalck | I know, this sounds ... huh? | 20:13 |
| TheJulia | arne_wiebalck: I wonder if your hardware is side channeling an ethernet port for bmc access? | 20:15 |
| TheJulia | or if the bmc is actually inside the CPU management engine | 20:16 |
| arne_wiebalck | hmm ... I could expand the test to include more machines, and different types of machines | 20:17 |
| arne_wiebalck | "side channeling an ethernet port for bmc access" ... what does this mean? | 20:17 |
| arne_wiebalck | "or if the bmc is actually inside the CPU management engine" | 20:23 |
| arne_wiebalck | you mean the CPU burn-in takes out the BMC and it needs some time to recover, before it can answer requests again? | 20:24 |
| arne_wiebalck | hmm | 20:25 |
| TheJulia | arne_wiebalck: kind of yeah | 20:33 |
| TheJulia | arne_wiebalck: like, the bmc uses eth0 and the os also uses eth0 | 20:33 |
| arne_wiebalck | TheJulia: no, these are separate | 20:33 |
| arne_wiebalck | TheJulia: there is a dedicated mgmt network | 20:34 |
| arne_wiebalck | Ok, at least we know why it is in the "forbidden" if :-D | 20:34 |
| TheJulia | heh | 20:34 |
| TheJulia | okay | 20:34 |
| arne_wiebalck | I will see tomorrow if I can get more details and find out why the power get fails. | 20:35 |
| * arne_wiebalck has his last working day tomorrow before holidays so this may need to wait ... | 20:35 | |
| TheJulia | arne_wiebalck: no worries | 20:36 |
| NobodyCam | good afternoon Ironic Folks.. Happy monday | 20:38 |
| TheJulia | good morning NobodyCam | 20:40 |
| arne_wiebalck | Hey NobodyCam o/ and goodbye everyone o/ :-) | 20:41 |
| NobodyCam | hey hey TheJulia hi TheJulia and Have a great night arne_wiebalck | 20:41 |
| NobodyCam | too many TheJulia's in that last posting | 20:42 |
| TheJulia | Hi NobodyCam how are youdoing NobodyCam ;) | 20:43 |
| 2021-07-19T21:11:54 <opendevreview> cenne proposed openstack/ironic master: [WIP] Add api endpoint for setting boot_mode on managed node https://review.opendev.org/c/openstack/ironic/+/800084 | ||
| janders__ | good morning Ironic o/ | 21:43 |
| NobodyCam | Good Morning janders__ | 21:52 |
| iurygregory | morning janders__ and NobodyCam o/ | 22:11 |
| janders__ | hey NobodyCam iurygregory | 22:11 |
| opendevreview | Merged openstack/metalsmith master: Set glance limit for baremetal friendly images https://review.opendev.org/c/openstack/metalsmith/+/801367 | 22:22 |
| iurygregory | ok, CI should be happy now! | 22:32 |
| TheJulia | iurygregory: \o/ | 22:41 |
| iurygregory | but CI is drunk on my patch with random failures in the unit tests =( | 22:42 |
| iurygregory | race conditions FTW | 22:42 |
| iurygregory | \o/ | 22:42 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!