| *** pmannidi|AFK is now known as pmannidi | 00:24 | |
| stevebaker[m] | TheJulia: it may be wrong, but I did this https://review.opendev.org/c/openstack/nova/+/813729 | 02:06 |
|---|---|---|
| TheJulia | stevebaker[m]: I was thinking something similar to cut down on the generation loss that NobodyCam was talking about with the resource tracker | 03:14 |
| TheJulia | I'm *hoping* I can spend some time heads down trying to wrap my head around the startup race condition that exists, and also the on-startup issues since they are basically bolted together | 03:14 |
| stevebaker[m] | TheJulia: you mean this startup race? https://bugzilla.redhat.com/show_bug.cgi?id=2012222 | 03:18 |
| TheJulia | stevebaker[m]: no, uhh.. the resource tracker looses the compute node one where they are trying to startup all of the compute services at once | 03:19 |
| stevebaker[m] | ah | 03:19 |
| TheJulia | I've got logs on my work machine, I grok what is happening at a high level, just ahve to figure out a way to either make it more sane or retool things to not be as... built upon nova's preexisting assumptions of a compute service | 03:20 |
| stevebaker[m] | heh | 03:20 |
| TheJulia | stevebaker[m]: so. deleted verb to deleting state (which I don't think nova groks) to cleaning or error states. We likely need to consider deleting as well if nova doesn't explicitly know about it because it is visible for a fraction of a second | 03:25 |
| TheJulia | unless there are problems which I suspect all sorts of other race conditions could hit on a super busy cluster | 03:26 |
| TheJulia | in theory self.cleanup_deploy won't complete until the node has exited a locked state | 03:28 |
| TheJulia | I need to walk that code | 03:30 |
| TheJulia | not tonight | 03:30 |
| * TheJulia goes and gets a shower and crashes | 03:30 | |
| stevebaker[m] | I think a node DELETING state works by accident according to the current nova code, if it is short-lived https://opendev.org/openstack/nova/src/branch/master/nova/virt/ironic/driver.py#L1237-L1259 | 03:30 |
| arne_wiebalck | Good morning, Ironic! | 06:08 |
| iurygregory | good morning arne_wiebalck and Ironic o/ | 06:10 |
| arne_wiebalck | hey iurygregory o/ | 06:10 |
| rpittau | good morning ironic! o/ | 07:11 |
| iurygregory | morning rpittau o/ | 07:17 |
| rpittau | hey iurygregory :) | 07:18 |
| timeu | nnn | 07:36 |
| dtantsur | good morning | 08:59 |
| rpittau | hey dtantsur :) | 08:59 |
| iurygregory | morning dtantsur =) | 09:00 |
| iurygregory | dtantsur, rpittau, arne_wiebalck, TheJulia and anyone else interested in the PTG, if you have some time today to check if the new schedule is good https://paste.opendev.org/show/809919/ | 09:20 |
| dtantsur | lgtm | 09:21 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent bugfix/8.1: Software RAID: Call udev_settle before creation https://review.opendev.org/c/openstack/ironic-python-agent/+/813753 | 09:26 |
| arne_wiebalck | iurygregory: one thing I was wondering if we should have a slot on discussing what we do or how we follow up with the input from the 3 sessions with ops feedback (in addition to acknowledge the points raised :)) | 09:30 |
| iurygregory | arne_wiebalck, ++ that makes sense to me I think we can add to Thursday as first topic | 09:31 |
| jssfr | hm, weher can I subscribe for information about the baremetal SIG operators sessions? | 09:31 |
| jssfr | *where | 09:31 |
| iurygregory | if required we can also book a specific slot for this discussion | 09:32 |
| arne_wiebalck | jssfr: the sessions are announced on the openstack-discuss mailing list | 09:32 |
| arne_wiebalck | jssfr: https://etherpad.opendev.org/p/bare-metal-sig has the planning | 09:33 |
| arne_wiebalck | jssfr: and we usually tweet about these sessions as well in advance | 09:33 |
| arne_wiebalck | jssfr: if you have other suggestions where/how to announce them, let us know! | 09:33 |
| jssfr | hah, I am even subcsribed to openstack-discuss.... the signal/noise of that one isn't ideal though :/ | 09:34 |
| arne_wiebalck | iurygregory: we probably need some triaging on which of the points raised can (realistically) turned into dev objectives | 09:34 |
| jssfr | thanks for the info, I might follow the pad | 09:34 |
| arne_wiebalck | jssfr: I agree, filter for [ironic] in the subject :) | 09:34 |
| arne_wiebalck | jssfr: and/or [baremetal-sig] | 09:35 |
| arne_wiebalck | jssfr: the SIG meetings are also announced/discussed here during the weekly meetings | 09:35 |
| arne_wiebalck | here == here on IRC | 09:35 |
| jssfr | so I was kind of hoping for a low-noise announce mailing list or maybe even a caldav thing | 09:36 |
| iurygregory | arne_wiebalck, yup =) I agree, your idea would be to triage before the PTG and we would only discuss the dev objectives? | 09:37 |
| jssfr | the weeklies are also outside of my working hours, so I'm generally ignoring IRC durign that time... lots of space to improve my processes here :) | 09:38 |
| jssfr | might set up a highlight on #endmeeting or somescuh | 09:39 |
| arne_wiebalck | iurygregory: I think so, yes: there was a lot of discussion during the 3 hours, and we do not want all to read the notes during the ptg ... so we may want to distill and bring only a handful to the PTG, like the "main operator areas of interest for improvement" | 09:40 |
| arne_wiebalck | jssfr: IIRC, we tried to have split weekly meetings to include more timezones, but this never took off due to lack of interest/commitment in the end | 09:40 |
| jssfr | I imagine. | 09:41 |
| iurygregory | arne_wiebalck, yeah I totally agree, since the PTG is next week I'm wondering if we have time to triage the topics before hand | 09:41 |
| jssfr | for someone like me who'd just listen in, it's also rather pointless to have a second meeting. | 09:41 |
| iurygregory | ideas on how we can do this? =) | 09:41 |
| arne_wiebalck | iurygregory: well, the idea was to gather input from ops for the ptg; if would be a pity if, now that we have it, we are not able to include it in the discussion for the next cycle | 09:43 |
| iurygregory | arne_wiebalck, agree =) | 09:43 |
| arne_wiebalck | iurygregory: many topics are not new and they have patches/ideas/code already, e.g. console support | 09:43 |
| arne_wiebalck | iurygregory: some are not really in our hands, like integration with nova | 09:44 |
| iurygregory | yup | 09:44 |
| arne_wiebalck | iurygregory: there were some new requests (like power control fallback to the PDUs), but apart from these it is more to make the dev community aware of what operators struggle with daily | 09:45 |
| arne_wiebalck | iurygregory: and then have this given some priority, ofc :-D | 09:45 |
| rpittau | iurygregory: schedule looks good to me, unfortunately I'll have to skip almost all Friday 22 :/ | 09:45 |
| dtantsur | arne_wiebalck, jssfr, maybe we need an RSS field (on ironicbaremetal.org?) with SIG news? | 09:53 |
| arne_wiebalck | dtantsur: yes, that sounds like a good idea to me! | 09:53 |
| * arne_wiebalck has no idea how to do this ;_D | 09:54 | |
| jssfr | I never warmed up with RSS and the likes, but this would be a great opportunity to try it again :) | 09:54 |
| dtantsur | (s/field/feed/, but you got the idea) | 09:55 |
| dtantsur | usually this stuff is generated from some kind of blog or even tags in a blog | 09:56 |
| dtantsur | maybe we should just post announced on ironicbaremetal.org/blog? | 09:56 |
| cumi | Hi, everyone. My name is Cumi. I am an Outreachy applicant for the December cohort. Excited to be contributing to the Replace pecan with flask in OpenStack Ironic project. Looking forward to the experience! | 09:57 |
| dtantsur | Hi cumi, welcome! TheJulia will wake up a bit later today (she's in the USA), so if you cannot wait, please reach out by email. | 09:58 |
| rpittau | dtantsur: probably posting the announcements directly is easier to maintain | 09:58 |
| rpittau | wecleom cumi :) | 09:58 |
| rpittau | or welcome even | 09:59 |
| dtantsur | was about to ask if it's french for welcome :D | 09:59 |
| cumi | Alright. Thank you @dantsur and rpittau | 09:59 |
| cumi | * dtantsur | 09:59 |
| opendevreview | Verification of a change to openstack/sushy master failed: Add revision for drive resource https://review.opendev.org/c/openstack/sushy/+/812676 | 10:02 |
| *** eandersson8 is now known as eandersson | 10:50 | |
| opendevreview | Verification of a change to openstack/ironic-lib stable/wallaby failed: json_rpc: list_opts() should return tuple https://review.opendev.org/c/openstack/ironic-lib/+/813570 | 10:51 |
| opendevreview | Verification of a change to openstack/ironic-lib stable/wallaby failed: json_rpc: setup.cfg is missing entrypoint for json_rpc's list_opts https://review.opendev.org/c/openstack/ironic-lib/+/813571 | 10:51 |
| *** janders_ is now known as janders | 11:05 | |
| jeeva | still struggling with the networking component of ironic | 11:50 |
| jeeva | so i must create the cleaning network in openstack as a network | 11:50 |
| janders | see you tomorrow Ironic o/ | 12:12 |
| TheJulia | good morning | 13:07 |
| dtantsur | morning TheJulia | 13:11 |
| TheJulia | so very very tired | 13:13 |
| dtantsur | :( | 13:13 |
| TheJulia | Okay, cumi replied to | 13:33 |
| TheJulia | Just... wow. | 13:33 |
| iurygregory | good morning TheJulia | 13:44 |
| *** pmannidi is now known as pmannidi|AFK | 14:01 | |
| iurygregory | arne_wiebalck, TheJulia the new proposed schedule for the PTG https://paste.opendev.org/show/809976/ with the operators sessions | 14:22 |
| iurygregory | if we are ok I will update the etherpad to reflect and send an email to the openstack-discuss | 14:22 |
| arne_wiebalck | iurygregory: thanks | 14:26 |
| arne_wiebalck | iurygregory: I guess "bulk operations" is also ops inspired? | 14:26 |
| arne_wiebalck | iurygregory: if so, we could move this up (or ops feedback down) | 14:27 |
| iurygregory | arne_wiebalck, yeah that makes sense =) | 14:28 |
| opendevreview | Manuel Schönlaub proposed openstack/sushy master: Add support for NetworkAdapter resources https://review.opendev.org/c/openstack/sushy/+/813839 | 14:29 |
| dtantsur | TheJulia: just created https://storyboard.openstack.org/#!/story/2009294, it's a part of the simplified start-up effort, and on the other hand - an early stepping stone to merging inspector | 14:53 |
| opendevreview | Manuel Schönlaub proposed openstack/sushy master: Add support for NetworkDeviceFunction and NetworkPort resources. https://review.opendev.org/c/openstack/sushy/+/813850 | 14:53 |
| opendevreview | Manuel Schönlaub proposed openstack/sushy master: Add support for additional network resources. https://review.opendev.org/c/openstack/sushy/+/813850 | 14:53 |
| dtantsur | see you tomorrow folks | 15:00 |
| iurygregory | bye dtantsur | 15:14 |
| rpittau | good night! o/ | 15:48 |
| iurygregory | gn rpittau o/ | 15:50 |
| opendevreview | Manuel Schönlaub proposed openstack/sushy master: Add support for NetworkAdapter resources https://review.opendev.org/c/openstack/sushy/+/813839 | 16:38 |
| opendevreview | Manuel Schönlaub proposed openstack/sushy master: Add support for additional network resources. https://review.opendev.org/c/openstack/sushy/+/813850 | 16:59 |
| TheJulia | zer0c00l: you around? | 19:47 |
| TheJulia | I has a question w/r/t hash ring in nova compute | 19:47 |
| opendevreview | Merged openstack/sushy master: Add revision for drive resource https://review.opendev.org/c/openstack/sushy/+/812676 | 21:27 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Test nova-compute fix https://review.opendev.org/c/openstack/ironic/+/813264 | 21:39 |
| TheJulia | NobodyCam: https://review.opendev.org/c/openstack/nova/+/813897 could use eyes | 21:46 |
| Inwerejosic1 | Hi TheJulia, please I have a question regarding the gerrit account ssh connection. Is it my github ssh that I need to add? | 21:52 |
| tonyb | I'm prewtty sure the answer is "no", but is there anyway to add a specific DHCP option per node? | 22:04 |
| melwitt | TheJulia: I noticed the WIP change you uploaded to nova and wanted to ask, is there any chance this is the race you are looking to fix? https://bugs.launchpad.net/nova/+bug/1853009 | 22:31 |
| TheJulia | melwitt: that is amongst the races which should be addressed by it, I *think* | 22:36 |
| TheJulia | melwitt: I was going to try and think through that one tomorrow | 22:37 |
| TheJulia | tonyb: it is port binding info so if your a cloud admin yes you can rebind the port, with additional options, but as a user afaik no | 22:37 |
| TheJulia | Inwerejosic1: github ssh ? your ssh public key to gerrit? | 22:37 |
| melwitt | TheJulia: ok, the fix for that merged recently and I'm backporting it to train, just fyi if it affects what you're trying to do | 22:37 |
| TheJulia | it shouldn't *but* it is like there are three distinct cases that "bad things" can happen in | 22:39 |
| TheJulia | shouldn't impact it, that is | 22:39 |
| melwitt | ok | 22:40 |
| JayF | If you all figure out and solve all the races in nova-compute around the resource tracker, someone should send you dozens of cookies. | 22:41 |
| TheJulia | melwitt: downstream there is another case artom is aware of which is one of the issues where someone went and upgraded their nova computes | 22:42 |
| TheJulia | melwitt: their controllers, and to fix things they are having to do db surgery :( | 22:42 |
| TheJulia | #lastthinganyofuswant | 22:42 |
| tonyb | TheJulia: Thanks. I'll find another way | 22:44 |
| melwitt | hm ok, I'll try to find which one that is | 22:44 |
| melwitt | I hadn't heard about it yet | 22:44 |
| TheJulia | arne_wiebalck: NobodyCam: the discussion over the last couple minutes with melwitt and I may be of interest | 22:44 |
| TheJulia | melwitt: they aren't broken, just a visual inconvience I think, but also the instance.host getting tired forever seems "wrong" operationally to me | 22:44 |
| TheJulia | which is partly where my patch went to after chatting with artom | 22:45 |
| Inwerejosic1 | TheJulia, I git that resolved, I ran <ssh-keygen -t rsa -C "email@somethingmail.com", and it generated my ssh public key which I have used to complete that process | 22:45 |
| TheJulia | Inwerejosic1: intentional git pun? | 22:46 |
| NobodyCam | :) | 22:46 |
| melwitt | TheJulia: sorry what is happening to the instance.host? | 22:47 |
| TheJulia | use of puns is always appreciated! | 22:47 |
| TheJulia | melwitt: so, instance.host, aiui, is the compute node that the instance is located on, which governs the select for the population of node cache with the nova.virt.ironic driver. Operationally, it is just out of date, and I *think* the cache not properly representing reality is where the compute node stuffs starts goign sideways because the constraint is against the host on the string, btu there is a missing piece of that in which | 22:49 |
| TheJulia | I've not got wired in my head | 22:49 |
| -opendevstatus- NOTICE: Both Gerrit and Zuul services are being restarted briefly for minor updates, and should return to service momentarily; all previously running builds will be reenqueued once Zuul is fully started again | 22:50 | |
| TheJulia | ugh | 22:51 |
| Inwerejosic1 | I meant got that resolve... | 22:52 |
| melwitt | ok so it's something like they upgrade compute nodes and things rebalance but the instance.host's don't get updated to the [compute service] owner of the instance(s) that changed ownership | 22:52 |
| janders | good morning Ironic o/ | 23:00 |
| TheJulia | melwitt: and also just general bouncing of the service creates cases, from what I percieve where instances will never map to compute nodes | 23:03 |
| melwitt | noted | 23:04 |
| *** pmannidi|AFK is now known as pmannidi | 23:50 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!