Thursday, 2023-08-17

opendevreviewVerification of a change to openstack/bifrost master failed: Revert "Pin systemd to 252-15 for CentOS Stream 9"  https://review.opendev.org/c/openstack/bifrost/+/89101400:09
opendevreviewJulia Kreger proposed openstack/ironic master: DNM Enable OVN  https://review.opendev.org/c/openstack/ironic/+/88508700:18
opendevreviewJulia Kreger proposed openstack/ironic master: DNM Enable OVN  https://review.opendev.org/c/openstack/ironic/+/88508700:28
iurygregorymmalchuk, just fyi normally we try to avoid bare rechecks, since the job timed out in the patch you can just add recheck job timed_out =) 01:38
opendevreviewJulia Kreger proposed openstack/ironic master: DNM Enable OVN  https://review.opendev.org/c/openstack/ironic/+/88508703:07
opendevreviewVerification of a change to openstack/bifrost master failed: Revert "Pin systemd to 252-15 for CentOS Stream 9"  https://review.opendev.org/c/openstack/bifrost/+/89101403:14
TheJuliagoodnight04:14
opendevreviewJacob Anders proposed openstack/ironic master: Retry connecting vmedia through a DVD device if available  https://review.opendev.org/c/openstack/ironic/+/88874605:49
arne_wiebalckGood morning, Ironic!06:57
rpittaugood morning ironic! o/07:15
arne_wiebalckgood morning rpittau o/07:24
arne_wiebalckrpittau: quick question, for https://review.opendev.org/c/openstack/ironic-python-agent/+/891609 and the failure in metalsmith integration, I don't think this is related to the patch itself, but would appreciate a more experienced set of eyes :)07:26
arne_wiebalckrpittau: would you have a moment to take a brief look?07:27
mmalchukmorning Ironic o/07:29
rpittauhey arne_wiebalck :)07:30
rpittauarne_wiebalck at a glance the issue seems related to the tinycore setup, so not related to the patch itself07:30
rpittauI would recheck in a couple of hours07:30
rpittauI feel like metalsmith has been a bit abandoned, we're quite overdue for its upgrade to CS9, had a patch open but no time to finish it https://review.opendev.org/c/openstack/metalsmith/+/869374 :/07:32
arne_wiebalckrpittau: thanks!07:32
rpittaunp!07:32
opendevreviewRiccardo Pittau proposed openstack/metalsmith master: [WIP] Add centos9 based job  https://review.opendev.org/c/openstack/metalsmith/+/86937407:33
rpittauat least rebased it, let's see how it goes07:33
dtantsurrpittau: overdue is a bit an overstatement, CS8 only expires in mid-202408:02
rpittautrue,  I was comparing to the rest of our projects08:03
mmalchukrpittau tinyipa keystone integration failed everywhere. looks like not systemd issue?09:08
mmalchukon CI09:09
mmalchukfolks, please merge: https://review.opendev.org/c/openstack/bifrost/+/89078909:11
opendevreviewDmitry Tantsur proposed openstack/bifrost master: testenv: use host-model/host-passthru as a CPU model for VMs  https://review.opendev.org/c/openstack/bifrost/+/89169311:34
dtantsurzigo: hey, do you have any threads to pull to get https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1002025 fixed? We may start to discourage people from using Debian-based ramdisks otherwise..11:37
iurygregorygood morning Ironic11:55
rpittaummalchuk: everywhere no, in jammy, just rechecked the patch12:10
mmalchukrpittau ok, waiting for updates12:16
mmalchukthanks12:16
opendevreviewJacob Anders proposed openstack/ironic master: Retry connecting vmedia through a DVD device if available  https://review.opendev.org/c/openstack/ironic/+/88874612:39
fricklerdtantsur: seems there is a newer version in bookworm and bullseye-backports, can you check if that has the issue, too? https://packages.debian.org/bullseye-backports/lshw 12:51
dtantsurI guess we don't have backports enabled in IPA13:18
TheJuliagood morning13:28
TheJuliadtantsur: perhaps we need nordix's job reporting to us in a 3rd party CI mode...13:32
dtantsurI doubt they have resources for that13:32
dtantsurmorning!13:32
mmalchukmorning!13:42
mmalchukfolks, please review and merge: https://review.opendev.org/c/openstack/bifrost/+/89078913:42
TheJuliaarne_wiebalck: o/ how many conductors do you run these days?13:43
TheJuliadtantsur: posted comments on why update_node, lmk if that makes sense. I critically don't want internal tasks dying if we can help it at all13:44
TheJuliaotherwise the risk of workflow ops derailing is quite high13:44
dtantsurState changes.......13:45
TheJuliayup13:45
TheJuliawe have to record13:46
dtantsurTheJulia: I wonder if we can hack it further to only care for provisioning state changes. I'd prefer external operations (like ones from inspector) not to be retried forever.13:46
TheJuliaI finally see what you were talking about w/r/t to logic though13:46
dtantsurbut yeah, somehow protecting taskmanager stuff makes sense to me13:46
TheJuliamaybe lets try incremental towards that, add a todo if see if we can sanely do that13:46
dtantsurI'm afraid of making things worse if e.g. a call from Inspector locks a node virtually forever.. but then again, it will probably fail to release the lock, so *shrug*13:47
dtantsurlet's go for it, I guess13:47
TheJuliawell, that sort of happens today13:48
TheJuliawell, the lock was held and not released which blocked inspector13:48
TheJuliaat least on the first logs, second one at a cursory look looks similar because it started at the same basic time after power state sync13:48
arne_wiebalckTheJulia: o/13:53
arne_wiebalckTheJulia: let me check ...13:53
arne_wiebalckTheJulia: 21 CGs with 25 conductors13:54
arne_wiebalckTheJulia: for ~9000 nodes13:54
opendevreviewJulia Kreger proposed openstack/ironic master: Retool sqlite retries  https://review.opendev.org/c/openstack/ironic/+/89133314:25
opendevreviewElod Illes proposed openstack/ironic-lib master: DNM: dummy change to test gate health  https://review.opendev.org/c/openstack/ironic-lib/+/89178914:57
opendevreviewElod Illes proposed openstack/metalsmith master: DNM: dummy change to test gate health  https://review.opendev.org/c/openstack/metalsmith/+/89179014:57
opendevreviewElod Illes proposed openstack/networking-generic-switch master: DNM: dummy change to test gate health  https://review.opendev.org/c/openstack/networking-generic-switch/+/89179314:58
opendevreviewElod Illes proposed openstack/python-ironic-inspector-client master: DNM: dummy change to test gate health  https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/89182315:07
opendevreviewVerification of a change to openstack/bifrost master failed: Revert "Pin systemd to 252-15 for CentOS Stream 9"  https://review.opendev.org/c/openstack/bifrost/+/89101415:11
opendevreviewElod Illes proposed openstack/networking-baremetal master: DNM: dummy change to test gate health  https://review.opendev.org/c/openstack/networking-baremetal/+/89184215:23
dtantsurTheJulia: we have a few more instances reported at https://github.com/metal3-io/cluster-api-provider-metal3/issues/108215:38
JayFIf after we're done, someone doesn't do a summit talk with a clever title like "Unlocking SQLite in Ironic", we've failed ;) 15:45
TheJuliahttps://review.opendev.org/c/openstack/ironic/+/891335 is growing on me, but it looks like in one of the logs it all started in the window between getting a lock to begin with , and then loosing it to do the actual node_update.... 15:47
opendevreviewMerged openstack/ironic master: Add a list of children nodes unit test  https://review.opendev.org/c/openstack/ironic/+/88974615:50
opendevreviewJulia Kreger proposed openstack/ironic master: WIP: Add some service steps  https://review.opendev.org/c/openstack/ironic/+/89163015:53
TheJuliadtantsur: is there a way to look at the logs and get the hash for the ironic code ?15:54
TheJuliaso we can line that up to the state of the repo?15:54
TheJuliaor does it just build with whatever is on master branch when it runs15:54
TheJuliai guess I want to understand if https://review.opendev.org/c/openstack/ironic/+/887835 is present or not15:57
TheJuliahmm, both instances are actually rooted in an update_node request from the api16:02
rpittaugood night! o/16:11
* TheJulia has idea16:21
mmalchukTheJulia now you can push the button, all branches passed CI16:33
mmalchukhttps://review.opendev.org/q/Iab88b3a28378cc22f767204bef84128bc962721016:33
mmalchukthanks in advance16:33
mmalchukand https://review.opendev.org/c/openstack/bifrost/+/89078916:34
mmalchuktoo16:34
JayFno need to ping individuals :)16:34
JayFwe're all looking16:34
dtantsurTheJulia: ironic may log its version on start-up.. otherwise, only inspecting quay.io/metal3-io/ironic16:34
mmalchukJayF understand but TheJulia did it when you stop she16:35
JayFmmalchuk: the stable/2023.1 backport of that is on the way to the gate16:35
mmalchukJayF thanks16:35
TheJuliadtantsur: got a few minutes to chat?16:35
dtantsurTheJulia: sure16:35
TheJuliahttps://meet.google.com/ycx-qhav-kmd16:36
mmalchukJayF is here a best/correct way to ping all cores?16:36
JayFmmalchuk: if you populate the hashtag field of your patch with ironic-week-prio16:38
JayFand link it in here asking for review without pinging anyone16:38
JayFit should be a timely review, especially if it's V+1 and is a trivial backport like that was16:38
mmalchukany example?16:38
JayFI know, at least for me personally, I read almost 100% of the messages put in here, even when I've been away :) 16:38
JayFmmalchuk: for the hashtag; just click "Show More" at the top of the change, and the field will show up: https://home.jvf.cc/~jay/hashtag.png16:40
JayFmmalchuk: just hit the edit-pencil beside hashtags and enter 'ironic-week-prio'16:40
JayFthen your change shows up in our review dashboard: https://review.opendev.org/q/hashtag:%22ironic-week-prio%22+(status:open)16:40
mmalchukoh cool, no one other project uses hashtag there) lol16:41
JayFIt's very much a project-by-project thing, I think other projects use hashtags for other things16:41
JayFit's just a tag field that was created when everyone called tags hashtags ;) 16:41
mmalchukyep, but not for reviews)16:42
JayFyeah, and it's not like, you won't get reviewed without that tag16:42
JayFit's just a way of saying "this is ready to go" 16:42
JayFMy order of reviewing is generally: updating reviews on patches I already reviewed; unreviewed stuff in ironic dash; everything else16:43
mmalchukI like it16:45
JayFlol mmalchuk I think I accidentally landed zed before :/ 2023.1 going to gate now16:51
JayFdon't worry, I've already reported my rule violation to myself, and I've been punished severely 😂16:52
JayFah no, I landed a *different* 2023.1 change16:54
JayFyour hashtag paying off already :P 16:54
dtantsurTheJulia: to illustrate my thoughts, this part caches stuff, and I don't understand what and why: https://opendev.org/openstack/oslo.db/src/branch/master/oslo_db/sqlalchemy/enginefacade.py#L1012-L105517:03
TheJulialikewise :(17:05
dtantsur:(17:05
dtantsuranyway, long weekend here, talk to y'all on Monday17:05
TheJuliao/17:05
mmalchukJayF lol17:07
TheJulia... wow, I think i just speeded up our unit tests locally17:07
opendevreviewJulia Kreger proposed openstack/ironic master: Don't hold on to the API request dbapi  https://review.opendev.org/c/openstack/ironic/+/89187217:16
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: Test - Revert "Don't actually heartbeat with sqlite"  https://review.opendev.org/c/openstack/ironic/+/89187417:22
TheJulialets get metal3 to lock!17:22
JayFTheJulia: how does that recover if e.g. a connection is lost to the DB?17:25
JayFTheJulia: is the client smart enough to make the existing instance work or would we fail a request17:26
TheJuliait is supposed to reconnect17:26
JayFnice17:27
TheJuliain oslo.db's realm17:27
TheJuliabut we shall see17:27
TheJuliaI'm just surprised how much faster unit test runs are locally 17:27
TheJuliawhich means "there was way more going on there"17:27
JayFyeah, not caching connections is a very strange thing to have lived this long17:27
TheJuliawell, it also gets lazy loaded basically17:31
opendevreviewMerged openstack/bifrost stable/2023.1: Create the log file for the disk-image-create command  https://review.opendev.org/c/openstack/bifrost/+/89112419:20
mmalchukmerged... the next one can be +W19:39
JayFrefresh it my friend19:39
JayFyou can't beat me to the punch during working hours ;) 19:39
* JayF is very very empathetic to being frustrated at waiting for a thing to backport19:40
* JayF looks at https://review.opendev.org/c/openstack/oslo.messaging/+/89174219:41
mmalchukyep, I can't do that, I'm not core(19:41
JayFoh, I know you can't, but I'm just saying I try to get the stuff landed when someone cares about it19:41
JayFb/c it's no fun to wait19:41
mmalchukI'm care for all my work)19:41
mmalchukand for your too19:42
JayFWhat is your professional interest in OpenStack/Ironic?19:42
JayFaka where do you work lol19:42
mmalchukI'm Cloud Architect in the Bank19:42
JayFand have you gotten any Ironic socks+stickers yet19:42
JayFI work finance-adjacent as well 19:43
* JayF works for https://opensource.gresearch.com19:43
opendevreviewVerification of a change to openstack/bifrost stable/2023.1 failed: Remove extra symbols accidentally added  https://review.opendev.org/c/openstack/bifrost/+/89078920:21
JayFhmm, I don't see bifrost CI enough to know if that's likely a real err or not20:22
JayFgiven the change hard to see it being related20:22
JayFbut I'll leave it to someone with more confidence to recheck20:23
mmalchukits random20:25
mmalchuksame job passed on test but failed on gate20:26
JayFSo, in this case you're probably right; but we try to avoid corrolating it exactly like that20:27
JayFbceause imagine we introduce a failure that happens 25% of the time, if we just recheck because it's the same job that already passed, we eventually get in recheck hell20:28
JayFevery openstack project has been their due course of time in recheck hell (or at least recheck purgatory) where you can't merge anything until you pay off all the "random test failure" tech debt20:28
mmalchukwe need to fix CI20:29
mmalchukmake it more stable20:29
mmalchukrpittau know how20:29
JayFFor master, we're doing about as well as we have in a long time20:29
JayFbifrost was impacted by a breakage but we knew what it was and it was being fixed; that's pretty good lol20:29
JayFfor the stable/ branches, they often get less CI love keeping them stable which isn't ideal but there's only so many of us20:29
JayFI'll note that operators who run openstack in production have all the skills needed to troubleshoot failing CI jobs :P 20:30
TheJuliahmmm20:31
mmalchukcan we get the ssh access to the CI job hosts to debug?20:31
TheJuliaso seems we can't use tftp at all with ovn20:31
JayFTheJulia: why not?20:34
JayFmmalchuk: generally speaking, all the logs you need should be right there. Click on the job results, then go to Artifacts or Logs tab, and all the service logs are there20:34
JayFmmalchuk: when something is extra-breaky, we often ask the zuul admins to hold a node for us when it fails so we can ssh in; but that's a last resort sorta thing.20:35
TheJuliaIt seems like OVN's NAT is side stepping the main kernel stack20:35
TheJuliaso the connection tracker helper does nothing20:35
JayFCan we disable firewalling for CI purposes?20:36
TheJulianope, because it is natting the network20:38
JayFoh, yeah, of course20:38
JayFhmmm20:38
JayFmultinode, so we can't just yolo-yolo forward all possible ports20:38
JayFyou sure OVS isn't just operating in a different network namespace? it really completely avoids the nf stack with raw sockets? damn20:38
TheJuliastack@np0034976153:~/devstack$ sudo ip netns list20:39
TheJuliaovnmeta-825eb402-b4f1-4a60-a435-5ebb9e260f99 (id: 0)20:39
TheJuliastack@np0034976153:~/devstack$ 20:39
TheJuliatotally different concept modeling :(20:39
JayFthis is borderline infuriating :( 20:41
JayFand I've not been banging head against it like you20:41
JayFprint out a logo and burn it in effagy for stress relief, it'll help :| 20:41
TheJuliaheh20:41
JayFIt drives me bonkers when I pretty much fully understand a thing and someone just reimplements it and goes around the thing 20:42
JayFTheJulia: is there any value in running OVN on our jobs that can use vmedia? 20:42
TheJuliaeh, might work20:43
JayFif we don't currently, we probably could support vmedia in sushy-tools 20:43
TheJuliatrying to see if ther eis a "I don't want nat" option20:43
opendevreviewMerged openstack/bifrost stable/zed: Create the log file for the disk-image-create command  https://review.opendev.org/c/openstack/bifrost/+/89112521:02
mmalchuknext could be yoga change21:06
JayFgo check the timestamp, W+2'd two minutes before you got to it21:08
JayFyou're not going to win the race with me mmalchuk ;) 21:08
* JayF just found https://docs.openstack.org/ironic/latest/contributor/index.html21:09
mmalchuklol21:09
JayFwith even more links to different devstack configs21:09
JayF /o\21:10
mmalchukdevstack? meh21:10
mmalchukonly pure openstack on baremetal!21:10
* JayF looks around21:10
JayFI don't have any BMCs whatsoever in my domicile :) 21:10
mmalchukyou are working for finance... you always can bought some)21:13
JayFDon't tempt me with a good time21:14
mmalchukI will)21:14
opendevreviewJulia Kreger proposed openstack/ironic master: DNM Enable OVN  https://review.opendev.org/c/openstack/ironic/+/88508721:16
TheJuliaThat might work more... although it seems tftp transfers like to stall on the CI host I'm holding21:17
opendevreviewJulia Kreger proposed openstack/ironic master: Don't hold on to the API request dbapi  https://review.opendev.org/c/openstack/ironic/+/89187221:29
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: Test - Revert "Don't actually heartbeat with sqlite"  https://review.opendev.org/c/openstack/ironic/+/89187421:29
opendevreviewJay Faulkner proposed openstack/ironic master: WIP: Revamp Ironic contributor docs, esp. around testing env  https://review.opendev.org/c/openstack/ironic/+/89191221:42
opendevreviewJulia Kreger proposed openstack/ironic master: DNM Enable OVN  https://review.opendev.org/c/openstack/ironic/+/88508721:54
iurygregoryJayF, not sure if you already saw https://review.opendev.org/c/openstack/python-ironicclient/+/891560 =) 22:04
JayFgood stuff, ty22:04
JayFassume you've tested it well?22:04
JayFif you tell me you actually ran all the commands you can get a speedyjay review lol22:04
iurygregoryI tested the logic only lol 22:06
iurygregorydidn't have a setup to test it (sorry)22:06
JayFwhomp whomp22:06
iurygregorygoing to try next week22:06
JayFcool; yeah I looked at it and it looks OK22:07
JayFbut for me "does it work" is the ultimate question for CLI lol22:07
iurygregoryfighting dhcp to deploy a real bm with bifrost in the internal network :D (when I get this working I will be able to test)22:07
iurygregoryor maybe I will just add some support in sushy-tools and test via VM =)22:08
iurygregoryat least this would enable CI testing... if I update sushy-tools22:08
JayFI guess it's not a valid test to just run it, ensure the payload is right, and get the err22:09
JayFb/c you gotta ensure the resp payload is filled22:09
JayFiurygregory: instead of sushy-tools; maybe wire enough of it up in fake driver for testing?22:10
JayFwe should use fake driver as a fixture more often for things like this22:10
opendevreviewMerged openstack/bifrost stable/yoga: Create the log file for the disk-image-create command  https://review.opendev.org/c/openstack/bifrost/+/89112722:22
JayFmmalchuk: I think that's the end of the line; https://review.opendev.org/c/openstack/bifrost/+/891134 is not in a mergable state right now. If you need it, get it fixed up and I'll look tomorrow.22:26
iurygregoryJayF, maybe but probably metal3 will need something that can be emulated to test it22:55
iurygregorybut I think I added enough tests in the fake-firmware interface if I recall correctly 22:55
zigodtantsur: I happen to co-maintain the lshw package in Debian, this annoyed me a lot in the past as well, but https://bugs.debian.org/1002025 was fixed a long time ago, please upgrade to Bookworm ! :)23:30
zigoAlternatively, you can use the backports in one of my OpenStack unofficial debian.net repos: http://osbpo.debian.net/debian/dists/23:31
zigoI believe I have a backport for at least bullseye-victoria and up, but probably even in buster.23:31
JayFzigo: PTI for 2024.1 will include Debian oldstable still, in order to facilitate upgrades (we don't really have to respect that in IPA though)23:34
JayFzigo: sorry for using that name instead of number, I'm terrible at remembering number23:34
zigoJayF: Include in what?23:36
JayFhere, I'll link it23:36
JayFhttps://review.opendev.org/c/openstack/governance/+/89122523:36
JayFwe still have to test next release on debian 1123:36
JayFto ensure people can upgrade since we have 11 supported in this release23:36
zigoJayF: So next release, you'll test on both Deb 11 and 12 ?23:39
zigoFrom my view point, you're only 1 year late to match when I release Ironic in Debian ! :)23:40
zigoZed was the release I supported both 11 and 12, starting with Antelope, I care only about Bookworm.23:40
JayFThat is exciting and also always makes me a little nervous because our minions don't test those exact combos :D 23:41
JayFDid you ever figure out all the bits to get your full-debian-style ironic stuff working, including IPA w/a debian-style ramdisk instead of a DIB one?23:42
zigoI got something funky here, trying to fix https://bugs.debian.org/1046841 : it seems some build process of Ironic are touching my gettext stuff in the Debian folder (ie: the debconf templates and translations), generating a not-wanted diff ...23:43
zigoI can't find what's doing that ... :/23:43
zigoSorry, I really meant:23:43
zigohttps://bugs.debian.org/104650523:43
zigoJayF: Nearly, I still have the issue that Ironic doesn't like my "https://example.com/ironic-is-here-not-at-the-root-with-a-port-number/" type of endpoint ...23:45
zigoAppart from that, everyting seems working ... :/23:45
JayFCan you make sure there's a bugs.launchpad.net ironic bug for that? With a really good reproduction case?23:46
JayFand for bonus points, email me the bug at jay at gr-oss dot io or CC me on it and I'll put it on my list23:46
zigoJayF: I traced it to a generator iterating the wrong way, and then got lost in the ironic client object model ... :(23:47
zigoBut I'll try to find the time to try again.23:47
JayFyeah, just brain dump into a bug and someone may be able to follow up; especially if you are willing to draft a blogpost about it for ironicbaremetal.org :)23:47
JayFyou don't have to solve it just show the reproduction and if you have leads share them :)23:48
zigoJayF: You can have fun with this though... https://salsa.debian.org/openstack-team/debian/supply/23:48
zigo(that's my PoC ...)23:48
zigoThat's how to solve it.23:48
JayFI'm past the end of my usual workday so I don't really have the capacity to look at that closely right now23:48
JayFI'll see if I can have a look tomorrow23:48
zigos/solve/reproduce/23:48
zigoIt need instructions on how to deploy, I'm aware of that ... :/23:49
zigoI will fill the bug.23:49
JayFthanks! 23:50
zigoJayF: FYI, my "supply" thingy can setup a (nearly) working Ironic setup as a VM in any OpenStack cloud with one single command ... :P23:50
JayFyeah, that's basically what bifrost is set to do -- it's fun to see the different ways you can arrange the legos though23:51
zigoJayF: No, Bitfrost does only Ironic, I'm doing Keystone, Nova, Barbican, Neutron...23:51
zigoAll of them setup at once in a single command, deployed with puppet.23:51
zigo:)23:51
zigo(forgot glance...)23:52

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!