Tuesday, 2023-01-17

opendevreviewMerged openstack/nova master: Update Availability zone doc page  https://review.opendev.org/c/openstack/nova/+/84646300:06
bauzasgibi: re: https://bugs.launchpad.net/nova/+bug/2002951 OOM09:17
bauzasgibi: based on the example you gave, those are the tests that were run for the failing worker https://paste.opendev.org/show/bUSshY14qpkpDQ5jraEt/09:17
gibinothing really jumps out from that list09:18
bauzasme too09:18
* bauzas looks at syslog09:19
opendevreviewAaron S proposed openstack/nova master: Add further workaround features for qemu_monitor_announce_self  https://review.opendev.org/c/openstack/nova/+/86732409:29
bauzasgibi: looks like the test was downloading the image when it stacktraced09:30
bauzaswait, no09:32
bauzastimings don't match09:33
gibiI don't think OOM kill will cause a stack trace, the process will simply disappear09:34
bauzasmy bad09:34
bauzasI meant when it was killed09:34
gibialso as we discussed the point where the OOM hit might not be close to the point where the killed process used up the excessive memory09:35
bauzasI'm trying to find where the test was when the worker got killed09:35
gibifrom this we can rule out that it is on a specific provider https://paste.opendev.org/show/b1CpIgnpVmLh4YCUOIar/ I see failures on ovh, rax, inmotion09:41
bauzasgibi: TIL how to ask subunit from a CI log :09:42
bauzas(venv) [sbauza@sbauza zuul-logs.9HEwdg]$ cat testrepository.subunit | subunit-filter -s --xfail --with-tag=worker-0 | subunit-ls09:42
bauzasa grep does the same but not by the same manner :D09:42
bauzasgibi: do you have any idea why I'm seeing a tempest call 30 mins before the run is run ?09:43
bauzasbefore the *test is run ?09:43
gibiTZ difference in log?09:43
bauzasgibi: https://paste.opendev.org/show/bI0yvTNy52PzFSQUsGze/09:44
gibimaybe job-output.txt rendered after the job failed09:46
gibihm09:46
gibiI would believe the tempest_log over the job-output.txt about the time steps09:47
bauzasme too09:47
bauzasbut look, the image eventually was downloaded09:47
bauzaswe can see the log09:47
bauzaswhich means the HTTP call was done09:47
gibithe OOM hit at 22:31:13 based on syslog09:47
gibithat matches the tempest_log timestamp09:48
bauzasgood point then09:48
bauzasgibi: I briefly looked at glance logs09:48
bauzasas I said, the image was apparently fully downloaded in 7-ish secs09:48
bauzasoh wait09:51
bauzasgibi: https://paste.opendev.org/show/bLFZGO2MZTYjdRRV3DCM/09:52
bauzaslooks like we were caching the image09:55
bauzasas we got the new path, and then nothing09:55
bauzasand the timings match this time09:56
gibihm this is interesting, in all the 16 nova-ceph-multistore jobs that failed in the last 10 days the same test case got killed https://paste.opendev.org/show/bmEzF6rFgucUibd4CqTX/10:16
bauzasgibi: and I guess we'll see the same, which is we want to get the image10:20
bauzasgibi: I'm curious btw., I've seen you using a logsearch tool10:32
bauzasis that a CLI about https://opensearch.logs.openstack.org/ ?10:33
gibinope, it is https://github.com/gibizer/zuul-log-search10:33
gibimy homebrew tool for grepping zuul logs10:33
* bauzas straight git cloning10:34
gibiI took 6 recent runs and generated the list of test cases run in the killed worker10:50
gibithen I checked for intersection of the set of test cases10:50
gibiand it is only tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive10:50
gibithe one that got killed10:50
gibiso this points to that single test case as a cause10:51
bauzasgibi: I'm just grabbing another change logs for looking whether the test was also killed while downloading the image10:56
gibiack10:57
bauzasah, your tool only downloads a specific file if I use --file10:57
bauzasgibi: can I get all zuul logs from a specific change ?10:58
gibino, you need to use --file to get a log downloaded. I did it opt-in as I mostly use it to wide search and I wanted to limit the disk and bandwidth usage10:58
bauzask10:59
gibifeel free to open an issue in the repo to add such option10:59
bauzasI can workaround it for a sec10:59
opendevreviewKashyap Chamarthy proposed openstack/nova master: libvirt: At start-up skip compareCPU() with a workaround  https://review.opendev.org/c/openstack/nova/+/87079411:19
kashyapgibi: When you get a minute, can you have a quick look at the unit test?  I know I messed it up slightly but how I'm unclear :/11:20
bauzasgibi: I tried to look at alot of failing jobs and all of them are indeed failing with the same test11:36
bauzasI tried to find where in https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_volume.py#L76 we have the oomkiller 11:36
bauzasbut as you said, maybe it's killed after a few seconds11:36
sean-k-mooneybauzas: im going to add a specless bluepint the meeting adgenda and try and implement it before then. we we decided to defer it thats ok but if we agree its trivial enough i would liek to include it in A13:01
bauzassean-k-mooney: ack13:03
zigoIs there some docs somewhere explaining how to implement an OpenStack wsgi API with keystone auth?13:43
* zigo is starting a new project from scratch13:43
zigoFYI, I already got the db migration with Alembic done ...13:43
zigo(plus oslo_config setup...)13:43
zigoUser docs are sometimes lacking info, dev docs are almost inexistant ... :(13:45
bauzaszigo: you are deliberatly left with the choice you want13:55
bauzasyou just need to use keystonemiddleware lib13:55
bauzashttps://pypi.org/project/keystonemiddleware/13:55
bauzashttps://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html describes the strategies you can choose for Auth'ing13:56
bauzasa recommandation is to use paste for pipelining the WSGI middlewares13:57
zigoThanks. But there's no code example is shown in the keystonemiddleware's doc.13:58
zigoLike many stuff, I'm stuck with a "look at other project, and attempt cut/past, then see what it does" strategy...13:59
bauzashah13:59
bauzasthat13:59
zigo:)14:00
bauzasyeah, in generall the overall workflow is prescribed, like in https://docs.openstack.org/project-team-guide/index.html14:00
bauzasbut beyond this, this is the project's team responsbility to decide how to implement what they want14:00
bauzaslike, the WSGI framework they prefer14:00
bauzasor even the WSGI server they'd run with devstack14:01
bauzaszigo: but honestly, the keystonemiddleware plugin isn't that hard to use14:03
zigoI don't think that's the hardest part indeed. I just don't know where to start! :)14:03
bauzasI suppose you just way the regular 'do the auth thing' by keystonmiddleware like in https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#authentication-component14:03
bauzaszigo: we have a couple of openstack cookiecutters, if those still exist and are updated14:04
bauzasbut yeah, before incepting any code, I'd recommend to formalize your repo structure the openstack way14:04
zigoYeah, I used it. But it doesn't do:14:05
zigo- alembic migrations14:05
zigo- oslo.config14:05
zigo- api14:05
zigo...14:05
bauzas:)14:05
zigoYeah, I'm navigating through many projects to see how they are organized, and I'm trying to pick the best ones.14:05
bauzasif you're asking for a 'Project inception 101 class', I'll make you sad, it doesn't exist :)14:06
bauzasbut you can surely bug us if you want guidance14:06
bauzasI guess you know the project team guide ?14:06
zigoThanks ! :)14:06
bauzashttps://docs.openstack.org/project-team-guide/index.html14:06
*** dasm|off is now known as dasm14:07
zigoWell, I know how the community works, gerrit, release management, branches, etc.14:07
zigoI don't think I even need to read this ! :)14:07
bauzasyup, but there is a small but interesting section in that guide https://docs.openstack.org/project-team-guide/technical-guides/index.html14:07
opendevreviewBalazs Gibizer proposed openstack/nova master: Use new get_rpc_client API from oslo.messaging  https://review.opendev.org/c/openstack/nova/+/86990014:08
bauzasyou also have the API guidelines https://specs.openstack.org/openstack/api-wg/#guidelines14:08
bauzasand then you're left with reading each of the oslo libs docs14:09
bauzasassuming you want RPC 14:09
zigoThanks for all of the links.14:15
zigoI don't think I'll need RPC, but maybe along the way...14:15
bauzasartom: sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/869812 got a weak -1 because I think we need to add an upgrade section in reno14:29
bauzastl;dr: starting with 2023.1, users could request instance.example.com hostname for their instance, and it would fail14:29
bauzasbecause of dhcp_domain14:30
sean-k-mooneyit wont fail but it will be modifed as currently don14:31
sean-k-mooneybut sure lets add that14:31
artombauzas, sure, OK14:32
bauzassean-k-mooney: yeah agreed "fail" is too broad14:33
bauzassean-k-mooney: I mean their instances won't get the hostname they expect14:33
bauzasfrom the user pov14:33
kashyapgibi: I think for my unit test question in the scroll, it's probably because I accidentally removed a mock.  /me tries...14:36
gibikashyap: sorry, I haven't got back to that yet14:36
kashyapDon't worry, I don't count on instant responses :-)14:36
kashyapI know you're context-switching on several tihngs14:36
sean-k-mooneybauzas: yep exactly so im fine with calling that out in the release note14:39
bauzassean-k-mooney: i wonder if operators will scream about it14:41
bauzasof course we can't provide different input validation based on a config option 14:41
bauzasbut still, they'll have to change something probably14:42
sahidstephenfin: o/ 14:42
sahidI can see that you are involving on osprofiler14:43
sahidI have question for you :-)14:43
sean-k-mooneybauzas: we are not changing the exting behavior14:43
sean-k-mooneyso if they were not doing this before we blocked FQDNs they will get the same behavior14:44
sahidut's related to the driver Jaeger, we would like to add an option14:44
sahidhttps://github.com/openstack/osprofiler/blob/master/osprofiler/drivers/jaeger.py#L5614:44
sean-k-mooneybauzas: that is why we orginally did not mention it in the sepc14:44
sean-k-mooneywe did dicuss this option in hte past14:44
sahidbasically the point will be to have a prefix for the service_name, so we can make a difference between for example services that are running on different region14:45
sahiddoes that mae sense for you if I add an option like service_prefix14:45
sean-k-mooneybauzas: so i dont think operators will be upset that we are mainting the behvior they expect14:45
sahidfrom my understanding that one will be only useful for jaeger, so I'm considering adding a section [jaager]14:46
bauzassean-k-mooneytechnically I agree14:46
bauzassean-k-mooney: technically I agree14:46
bauzaswe never supported FQDNs14:46
sean-k-mooneyand those that used dispaly name with an fqdn hand it modifed by the config option14:47
bauzasso when passing a hostname, cloud-init was getting a FQDN based on the hostname + the default domain name from the option14:47
sean-k-mooneyso if they wanted it to not be modifed they already had to set the config option to the empty string14:47
bauzaswhich was consistent14:47
artombauzas, fixed14:47
kashyapUnrelated: Is the "nova-tox-functional-py38" job passing reliably for everyone? - it's still failing with "TypeError: getresponse() got an unexpected keyword argument 'buffering'14:47
kashyap"14:47
sean-k-mooneybauzas: when stephen added --hostname we also added the dhcp_doamin to the dns name in neutron14:48
sean-k-mooneywe did nto modify what we put in the metadtaa14:48
sean-k-mooneyand we are not going to with artoms code because he is not changing that14:48
sean-k-mooneyit was only the value in the neutron port that was changed14:48
bauzasyou know what ? I'll play the ostrich about any kind of FQDN questions14:49
bauzasonce artom uploads his change, I'll review it and I'm done14:50
sean-k-mooneybauzas: dont feel like you cant ask them14:50
sean-k-mooneyim just saying we are intentiolly not changing the behvior to not break anyone14:50
sean-k-mooneyand because we dont want cofnig dirven api behavior14:50
bauzasthe ship has sailed14:50
sean-k-mooneyso without removign the option entirly which would impact everyone14:50
sean-k-mooneywe cant really do much else14:50
bauzasbut honestly, I liked the fact that we were saying domain names was something unrelated to nova :)14:51
sean-k-mooneyyep so did i 14:51
sean-k-mooneythis is the compromsie so that we dont have to care about them again once done14:51
sean-k-mooneylike all good compromises it does not make anyone happy but we can all live with it14:52
bauzasthe ostrich theory applied to me.14:52
sean-k-mooneyi prefer magpie psychology. distract people with other shiny things that matter14:53
bauzasI need to learn the three-card monte14:55
bauzas'follow your card'14:55
kashyapbauzas: sean-k-mooney: There's a thing called "Belgian compromise", which loosely means:14:56
sean-k-mooneybauzas: added https://blueprints.launchpad.net/nova/+spec/default-ephemeral-format-unformated to meeting adjenda14:56
kashyap(quote)14:57
kashyapcomplex issues are settled by conceding something to every party concerned, through an agreement that is usually so complicated that nobody completely understands all its implications.14:57
kashyap(/quote)14:57
sean-k-mooneyhehe ya sounds like a typical eu threaty14:57
opendevreviewKashyap Chamarthy proposed openstack/nova master: libvirt: At start-up skip compareCPU() with a workaround  https://review.opendev.org/c/openstack/nova/+/87079414:58
sahidstephenfin: https://bugs.launchpad.net/osprofiler/+bug/200309214:59
artombauzas, so I should just remove the ( ) about i18n?15:24
opendevreviewAlexey Stupnikov proposed openstack/nova stable/victoria: Test aborting queued live migration  https://review.opendev.org/c/openstack/nova/+/84574815:32
opendevreviewKashyap Chamarthy proposed openstack/nova master: libvirt: At start-up allow skiping compareCPU() with a workaround  https://review.opendev.org/c/openstack/nova/+/87079415:38
bauzasartom: yup IMHO15:49
bauzasas a reminder nova meeting here in 10 mins15:50
opendevreviewArtom Lifshitz proposed openstack/nova master: Microversion 2.94: FQDN in hostname  https://review.opendev.org/c/openstack/nova/+/86981215:59
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Jan 17 16:00:19 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzasgdi, just in time16:00
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:00
bauzashi everyone16:00
dansmitho/16:00
elodilleso/16:01
gibio/16:01
bauzasokay let's start16:02
gibi(I'm a bit distracted)16:02
bauzas#topic Bugs (stuck/critical) 16:02
bauzas #info  One critical bug16:02
bauzas#info  One critical bug16:02
Ugglao/16:02
bauzas#link https://bugs.launchpad.net/nova/+bug/200295116:03
bauzasgibi: I marked this one as critical for the sake of the discussion16:03
gmanno/16:03
bauzasbut we can put it back to High16:03
bauzasin general, I tend to triage CI bugs to Critical until we agree this is not holding the gate16:03
bauzasdo we want to discuss about it now or no ?16:04
gibisure16:04
gibiI updated the bug16:04
bauzasok, so, gibi (mostly) and I looked at this one today16:05
gibiI think it is tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive test case that tirggers the OOM16:05
bauzasyeah16:05
bauzasand like I said, I tried to find wherer16:05
bauzasbut I wasn't able to see 16:06
bauzascontext : https://github.com/openstack/tempest/blob/7c8b49becef78a257e2515970a552c84982f59cd/tempest/api/compute/admin/test_volume.py#L84-L12016:06
bauzaswe try to create an image16:06
bauzasthen we create an instance16:06
bauzasand then a volume which we attach to the instance16:07
gibiI haven't had time to look into the actual tc yet16:07
sean-k-mooneyp/16:07
gibialso it would be nice to see how the python interpreter rss size grows during the test execution16:08
dansmithyeah surely seems like a benign test case16:08
sean-k-mooneywe unfortuently dont have the memtacker stuff form devstack16:08
sean-k-mooneybtu it would be nice if we coudl get that and also dmsg in the tox based tests16:08
bauzasI tried to grep the testname in n-api16:08
bauzasbut I wasn't finding it16:09
bauzasso, either we no longer use it 16:09
bauzasor we were not yet calling the nova-api 16:09
bauzaswhich means we have the kill before creating the instance16:09
bauzasbut I could be wrong16:09
bauzasanyway, folks are ok if we modify the bug to High ?16:10
bauzasbug report*16:10
gibitomorrow I will continue looking but we can also tentatively try to disable this single test to see if that removes the OOM problem16:11
gibibauzas: I'm not against having this as High16:12
bauzasok16:12
bauzasthen let's look again tomorrow and we'll see what to do16:12
bauzasthis time I'm just afraid to remove this test because we don't know why we have a OOMkill16:13
bauzasthis could arrive to another test then16:13
gibiyep, that would be my goal of disabling it temporary to see if the OOM just moves to another test case16:13
gibiand to see which test case16:13
gibito find a pattern16:13
bauzas(I also verified that nothing changed on the tempest side since 1 year for this test)16:13
dansmithgibi: you could also rename it I think and change the sort ordering16:14
bauzasyeah16:14
dansmithafaik, we run tests sorted per worker16:14
bauzasI was wondering, maybe this was a problem due to another test16:14
gibidansmith: good idea16:14
bauzasdansmith: I think we can ask stestr to modifyh the sort16:14
dansmithoh?16:14
bauzasbut I need to remember how to do it16:14
gibibauzas: on that I extracted all the test cases form the killed worker from multiple runs and the only test case overlap was this tc16:15
gibiso if other test causing the issue then it is not a single test but a set of tests16:15
gibiotherwise I would see an overlap16:15
bauzasgibi: well, yeah, but that maybe means that the previous tests were adding more memory before so that's only with this test that OOMkiller wants to kill 16:15
bauzasas you see, this is a very simple test16:16
gibithat is my point above, if a single test adds the extra memory usage then that woudl show up as an overlap between runs16:16
gibibut it doesn't16:16
bauzasgibi: that's why I'll try to see how to ask stestr to modify the sort16:16
gibiyeah, moving this tc to the end can help to see if there is a set of tests that trigger this behavior16:17
gibianyhow I think we can move on16:18
bauzascool16:19
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 27 new untriaged bugs (+0 since the last meeting)16:19
bauzasI triaged a few bugs todaty16:19
bauzas#link https://etherpad.opendev.org/p/nova-bug-triage-2023011016:19
bauzasnothing to report here by now16:20
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:20
bauzasgibi: wants to get the bug baton this week ?16:20
gibibauzas: sure I can16:21
bauzasthanks alot16:21
bauzas#info bug baton is being passed to gibi16:21
bauzas#topic Gate status 16:21
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:21
bauzaswe already discussed about the main one, wanting to discuss other CI bugs ?16:22
gibijust a sort summary16:22
bauzaslooks not16:22
bauzasah 16:22
bauzaswe're listening to you16:22
gibiI see failures in our functional tests16:22
gibione is about missing db tables so it is probably interference between test cases16:23
gibiwe saw that before 16:23
gibifixed it but not we had a non 100% fix16:23
bauzas:/16:23
gibiand there is a failure with db cursor need a reset16:24
gibiit might be related to the above16:24
gibinot sure yet16:24
bauzaslovely16:24
gibithese two I wanted to mention16:24
gibibut there are other open bugs that appear in the gate time to time16:25
bauzasflipping strest worker runs would help to trigger the races16:25
gibiso it is fairly hard to land things overall16:25
bauzasI could try to reproduce those functests locally16:25
bauzasthis would exhaust my laptop, but worth trying16:25
bauzasgibi: let's then discuss this tomorrow as well16:26
gibisure16:26
bauzasI mean, I have my power mgmt series to work on, but if we can't land things, nothing will merge either way.16:26
sean-k-mooneythe gate is not totally blocked16:27
sean-k-mooneybut its flaky enough that its hard16:27
bauzasyeah, but rechecking is not a great option16:27
gibiyepp16:27
sean-k-mooneyya its not16:27
bauzasagreed, I'm not sending the signal our gate is busted16:27
bauzasbut we know this is hard16:27
sean-k-mooneyone thing i have noticed is the py3.10 functional job seams more stable then py3816:27
bauzasand let me go to the next topic and you'll understand why16:27
sean-k-mooneyfor the db issues16:28
sean-k-mooneybut that could be just the ones i happend to look at16:28
bauzasok16:28
clarkbsean-k-mooney: 3.10 introduced a much more deterministic thread scheduler. Also its quite a bit quicker in some projects which helps generally16:28
bauzasah, gdk16:28
bauzaswe probably have tests not correctly cleaning up data16:28
bauzasso we need to bisect them16:28
sean-k-mooneyya so im wondifing if we are blocked we might want to make the 3.8 one non voting while we try to fix this16:29
sean-k-mooneybut there are other issues so i dont think that will help much16:29
sean-k-mooneyjust somethign to keep in mind16:29
bauzassean-k-mooney: before going that road, lemme try to bisect the faulty tests16:29
dansmithI've seen it both waysm16:29
sean-k-mooneyyep16:29
dansmith3.10 passing with 3.8 failing and the other way16:29
dansmithso I don't think disabling one gets us much16:29
sean-k-mooneyok then its jsut flaky16:29
bauzaslovely16:29
bauzasmoving on 16:29
bauzaswe have some agenda today16:29
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:30
bauzasthat's fun16:30
bauzasdespite https://review.opendev.org/c/openstack/tempest/+/866049 was merged, we still have the centos9-fips job timeouting16:30
bauzasso I looked at the job def16:30
bauzasand looks to me it no longer depends on the job I added extra timeout :)16:30
bauzasso basically the patch that took 2 months to get landed is basically useless for our pipeline 16:31
bauzasfunny, as I said16:31
gmannI think we had progress on running fips testing on ubuntu but need to check if we have job ready. that can replace c9-fips jobs16:31
bauzasso I'll just add the extra timeout on our local job definition16:31
opendevreviewDan Smith proposed openstack/nova master: WIP: Detect host renames and abort startup  https://review.opendev.org/c/openstack/nova/+/86392016:32
bauzasgmann: that's good to hear16:32
gmannnot merged yet #link https://review.opendev.org/c/openstack/project-config/+/86711216:32
bauzasgmann: we could put fips in check pipeline then16:32
gmannyeah that is plan once we have ubuntu based job 16:32
bauzasgmann: as a reminder, given centos9s, fips is on periodic pipeline16:32
gmannyeah16:33
bauzasanyway, this time it should be quickier16:33
bauzasI'll just update our .zuul.yaml 16:33
bauzasoh wait16:33
bauzashttps://zuul.openstack.org/job/tempest-integrated-compute-centos-9-stream is actually defined in tempest16:34
bauzasso I don't get why we don't benefit from the extra timeout16:34
gmannyeah, we will prepare the tempest job and then add in project side gate16:34
sean-k-mooneythe job definition is yes16:34
bauzasanyway, I don't want us to spill too much time about it16:35
gmannnot sure on timeout. c9s has been flasky for fips always16:35
bauzaslet's move on16:35
gmannyes16:35
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:35
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:35
bauzas#topic Release Planning 16:35
bauzas#link https://releases.openstack.org/antelope/schedule.html16:35
bauzas#info Antelope-3 is in 4 weeks16:35
bauzastick tack16:35
bauzas#info 17 Accepted blueprints for 2023.1 Antelope 16:35
bauzaswhich is the same amount than yoga16:36
bauzasthis is a large number given our team16:36
bauzasgiven this, I'll create an etherpad for tracking each of them16:36
sean-k-mooneythere are 3 i expect to complete this week possibely more16:36
sean-k-mooneydependign on review bandwith16:36
bauzassean-k-mooney: me too, but that still requires us some effort16:36
sean-k-mooneyi am a little worried for soem of them but hopeful we will land the majoriy of them16:37
bauzasI mean, I know me, I'll need to put my review energy on the right way and an etherpad will help me to direct my energy productively16:37
sean-k-mooneyi dobth it will be too much over half16:37
bauzas#link https://blueprints.launchpad.net/nova/antelope16:37
bauzasyou can find the list of those blueprints there ^16:38
bauzas#topic Review priorities 16:38
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:38
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:38
bauzasnothing to mention here16:39
bauzas#topic Stable Branches 16:39
bauzaselodilles: floor is yours16:39
elodilles#info stable branches don't seem to be blocked, but patches mostly need rechecks16:39
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:39
elodillesand last but not least: Xena will transition to Extended Maintenance after the release of 2023.1 Antelope16:39
elodillesso to prepare for that:16:40
elodilles#info release patches were generated for *stable/xena* : https://review.opendev.org/q/topic:xena-stable+reviewer:sbauza%2540redhat.com16:40
sean-k-mooneythe release team proposed doign a release of several repos for xena. do we want ot wait for the tox pin to be merged16:40
elodillessean-k-mooney: which one do you mean?16:40
sean-k-mooneythe ones you were linking16:40
elodilles(and that was all from me about stable branches)16:40
gmanntox pin is merged for stable branches. it is done at central place in openstck-zuul-jobs repo16:40
bauzasthat's fun, stable branches are more stable than master :)16:41
sean-k-mooneyso we dont have the pin to tox<4 on xena yet16:41
sean-k-mooneygmann: oh ok16:41
sean-k-mooneyi tought we needed to do it in the tox.ini too16:41
sean-k-mooneyso that it worked if you run tox loclaly16:41
gmannlet me check if osc-placement and python client is merged or not16:41
elodillesno, the workaround was merged last week, as gmann says16:41
sean-k-mooneywill that work outside ci16:41
sean-k-mooneyim not sure hwo you can fix it centrally unless we did it in upper-constraits?16:42
gmannyeah tox one is merged but this placement functional test this is not yet #link https://review.opendev.org/q/I4e3e5732411639054baaa9211a29e2e2c8210ac016:42
gmannbauzas: sean-k-mooney elodilles ^^16:42
elodillesoh, i missed that somehow16:42
elodilleswill review ASAP16:42
bauzasack16:42
elodillessorry for that16:42
bauzastab open16:42
bauzasI'll do my homework after the meeting16:43
sean-k-mooneyso my question still is not really answered16:43
elodilles(the stable ones o:))16:43
gmannthanks 16:43
sean-k-mooneywhere is tox pinned in https://github.com/openstack/nova/blob/stable/xena/tox.ini16:43
elodillessean-k-mooney: in that case we can wait until the xena one merges :)16:43
bauzassean-k-mooney: how to cap tox under 3 ?16:43
gmannsean-k-mooney: only for CI. you mean to pin it in tox.ini itself ?16:43
bauzasunder 4, I mean16:44
sean-k-mooneyyes so that developers can also run tox locally to test backports16:44
gmannsean-k-mooney: if we want to fix it for local run to make sure we do not run it with tox4 then yes we need to pin in tox.ini also but that can be done if we really need16:44
sean-k-mooneyi was asking should we do that before doing the final release for extended mainance16:44
elodilleshmmm. good question.16:45
bauzasthat sounds doable to me16:45
gmannfor local run I think both way ok either make sure we have tox<4 in our env or pin it in tox.ini16:45
sean-k-mooneyi replciated the pin in ci downstream16:46
gmannwe did for python-novaclient https://review.opendev.org/c/openstack/python-novaclient/+/869598/2/tox.ini#416:46
gmann#link https://review.opendev.org/c/openstack/python-novaclient/+/869598/2/tox.ini#416:46
sean-k-mooneyyes16:47
sean-k-mooneyso do we want to do it for all the other nova delivberable16:47
elodillesthen i'm OK to do the same and release after that merged16:47
sean-k-mooneyif so we should do it before the em tansition16:47
elodillesyes, I'm OK with that, I don't see now any reason not to do it before the transition16:48
elodilles(the generated xena release patches don't have deadlines, but best not to postpone them for weeks)16:49
bauzasok, sounds an agreement, we just need an owner16:50
sean-k-mooneyi can do it for os-vif maybe some of the others16:50
bauzasack16:51
sean-k-mooneyits really just one line and ensuring it works loocally16:51
bauzasI know16:51
elodillessean-k-mooney: ping me if i forgot the reviews o:)16:51
bauzasanyway I guess we're done with this topic and we have a specless blueprint ask in a sec16:51
bauzasso, moving on16:51
bauzas#topic Open discussion 16:52
bauzas(sean-k-mooney) https://blueprints.launchpad.net/nova/+spec/default-ephemeral-format-unformated16:52
sean-k-mooney ya so tl;dr is currently we use libguestfs in two places in nova16:53
sean-k-mooneyfile injection which is deprecated for a long time16:53
sean-k-mooneyand formating the filesystem of the addtional ephmeral disks16:53
bauzastrue16:53
sean-k-mooneyi would like to have a way to allwo tthe ephmeral disk to be unformated16:53
sean-k-mooneymaking libguestfs optional16:53
sean-k-mooneyto the proposal is either add unformated to https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_ephemeral_format16:54
sean-k-mooneyor sligly cleaner add a bool opt to trun off the formating16:54
bauzaswhat does the default value which is None ?16:54
sean-k-mooneyand i want to kwno if there is a prefernce and if we think this could be a spec or specless16:54
dansmitheither is okay with me, I guess format=unformatted seems better to me because it's just another option for an existing knob16:55
sean-k-mooneyi need to check the default of none but i belive it makes it os dependednt16:55
bauzassean-k-mooney: I see None as the default value, what's then the behaviour ?16:55
bauzasok16:55
sean-k-mooneyi need to dig into this a little more16:55
sean-k-mooneybut basically i wanted ot know if peopel think this is ok to do this cycle16:56
sean-k-mooneyor shoudl we discuss in the ptg and do it next cycle16:56
bauzasI think this is a very small feature16:56
bauzasself-containede16:56
dansmithyeah no need for lots of discussion, IMHO16:56
bauzasparticularly if we go with adding a new value16:56
sean-k-mooneyok so 1 i need to document what none does. 2 determin if it can disable the formating today alredy16:56
bauzastrue16:57
sean-k-mooneyand 3 if not add unformated as an option to expcitly do that16:57
bauzassounds a simple plan to me16:57
sean-k-mooneyso at a minium ill add a docs change to say what none does16:57
sean-k-mooneyand we can then evaluate in the gerrit review if we need unformated16:57
gibisounds good to me16:58
bauzasanyone objecting about this smallish effort for this cycle ?16:58
sean-k-mooneyif this ends up not being small i will punt to next cycle16:58
bauzasI don't expect any behavioural change16:58
bauzasso I'm fine with approving it as a specless blueprint based on such assumption16:59
bauzasand you're free to close this one as deferred if we consider this is only a doc patch16:59
sean-k-mooneycorrect the default would be what we have today and the unformated behavior woudl be opt in16:59
bauzasany objections ?16:59
dansmithno objection from me16:59
bauzascool16:59
bauzas#agreed https://blueprints.launchpad.net/nova/+spec/default-ephemeral-format-unformated accepted as specless blueprint for the 2023.1 cycle17:00
bauzasthat's it for me17:00
bauzasnothing else on the agenda17:00
bauzasthanks all17:00
bauzas#endmeeting17:00
opendevmeetMeeting ended Tue Jan 17 17:00:26 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-01-17-16.00.html17:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-01-17-16.00.txt17:00
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-01-17-16.00.log.html17:00
elodillesthanks o/17:01
gibio/17:01
bauzassean-k-mooney: https://github.com/openstack/nova/blob/9e2ca01988b8889738eba3c9af336ad82d214e1b/nova/virt/libvirt/utils.py#L22617:03
sean-k-mooney so none is virt driver dpendent and for libvirt its ext417:05
sean-k-mooneyif the glance image does not have an OS17:06
sean-k-mooney*os_type set17:06
sean-k-mooneyalso why are we importing the constant form the privsep module  nova.privsep.fs.FS_FORMAT_EXT417:06
sean-k-mooneythat just feel lazy17:06
sean-k-mooneybauzas: actully thats in create_ploop_image17:07
bauzascorrect17:07
sean-k-mooneyso that is only used for openvz17:07
sean-k-mooneythat not what we do for qemu/kvm17:07
bauzasindeed17:08
bauzashttps://github.com/openstack/nova/blob/b8a5961161da4a33c4d9c80e3025d9ff6eaf5326/nova/privsep/fs.py#L299-L30217:08
sean-k-mooneyhttps://github.com/openstack/nova/blob/9e2ca01988b8889738eba3c9af336ad82d214e1b/nova/privsep/fs.py#L257-L25917:08
bauzasyup17:08
sean-k-mooneyso for qemu/kvm we default to vfat17:09
bauzasanyway, the behaviour of None seems consistent17:09
sean-k-mooneyits virt driver and virt_type dependent17:09
sean-k-mooneyso ya a new option is what we want17:09
sean-k-mooneywell value 17:09
bauzasthis is just saying "let the virt driver decide for me or the os type"17:09
sean-k-mooneyof unformatted17:09
sean-k-mooneyyep17:09
sean-k-mooneyok will ill update the docs text to call that out17:10
bauzascorrect, we need an extra explicit value 17:10
bauzasif we just want an unformatted partition 17:10
sean-k-mooneyi breifly looked at this this morning but didnt have time to fully get to the bottom of it17:10
sean-k-mooneynot even a partion a blank file17:10
sean-k-mooneyso if we set unformatted we will jsut get the empty disk17:11
sean-k-mooneyand its up to the user to partion and format it as they see fit17:11
sean-k-mooneyjust like a blank cinder volume17:11
bauzasI see17:11
opendevreviewMerged openstack/osc-placement master: Use pypi released version of placement in functional tests  https://review.opendev.org/c/openstack/osc-placement/+/86975517:50
sean-k-mooneysigh... https://review.opendev.org/c/openstack/nova/+/869900 will never merge17:53
sean-k-mooneybauzas: summerised usecase here https://blueprints.launchpad.net/nova/+spec/default-ephemeral-format-unformated18:14
sean-k-mooneyin the whiteboard18:14
bauzassean-k-mooney: all good thanks18:14
dansmithyeah looking at the latest ceph-multistore oom, one of tempest's workers is using >900MiB of ram, where the others are <1018:43
dansmithso something is clearly going haywire18:43
dansmithgmann: you're aware of this right?18:43
dansmithI assume if you knew of any recent tempest changes that could be responsible, you'd have spoken up by now :)18:43
gmanndansmith: no, I cannot recall any relevant change happened in tempest18:57
dansmithyeah I looked too, and nothing much lately18:57
gmannand no stestr version change recently 18:59
dansmithI'm trying to stack for the first time this year so I can repro and I'm getting this failure to install pbr.build when it tries to install os-testr19:01
dansmithModuleNotFoundError: No module named 'pbr.build'19:01
dansmithis this known?19:01
dansmithgmann: ^19:01
gmannno, i did not see this before19:03
sean-k-mooneystephenfin: fixed a pbr issues recently not sure if its releaed19:04
sean-k-mooneythis https://review.opendev.org/q/topic:pep-517 but that looks more tox 4 related19:05
dansmithblargh19:06
dansmithI can install os-testr without constraints, but it fails like this otherwise19:06
sean-k-mooneywait os-testr19:07
gmannyeah I think those were tox4 related 19:07
sean-k-mooneywhat is using that19:07
dansmithsean-k-mooney: +./stack.sh:main:803                       pip_install -U os-testr19:07
sean-k-mooneywe shoud not be using os-testr anywhere anymore19:07
sean-k-mooneyeverthing should be using stestr19:07
gmannthat is installed successfully in gate I think that is using constraint?19:07
gmannhttps://zuul.opendev.org/t/openstack/build/0fc9dc8ecbe748498c941c6f21cbf057/log/job-output.txt#436719:07
dansmithyeah I dunno why I can't install it19:08
sean-k-mooneyits in uc ya https://opendev.org/openstack/requirements/src/branch/master/openstack_requirements/tests/files/upper-constraints.txt#L35919:08
dansmithunless it's a mirror sync thing?19:08
sean-k-mooneyhttps://opendev.org/openstack/devstack/src/branch/master/stack.sh#L80319:08
sean-k-mooneycomment ou tthat line19:09
gmannnot this week but last week i stack successfully 19:09
clarkbnote contraints don't pick what is installed. Only what version to install if something is to be installed19:09
dansmithsean-k-mooney: heh, yeah, I could but..19:09
sean-k-mooneyclarkb: yep i know just pointing out that it would be constrained if it was installed for other projects19:10
dansmithI purged pbr and os-testr locally, which might have gotten past it19:10
sean-k-mooneypip_install -U os-testr19:10
sean-k-mooneyso that is unconstiaed in devstack19:11
sean-k-mooneyunless pip_install in devstack add uc by default19:11
* sean-k-mooney checks19:11
gmannhttps://zuul.opendev.org/t/openstack/build/0fc9dc8ecbe748498c941c6f21cbf057/log/job-output.txt#436719:12
dansmithsean-k-mooney: it does19:12
dansmithI could install it myself without uc19:12
dansmithbut just purging those two packages locally seems to have worked19:12
dansmithso maybe some not-so-correct version deps19:13
sean-k-mooneywhat os are you using out of interest19:13
gmannmaybe19:13
dansmithsean-k-mooney: focal19:13
sean-k-mooneyyou said you had not stacked this year yet is it 20.0419:13
sean-k-mooneyok that should also work19:14
sean-k-mooneywell for the next 4-6 weeks19:14
gmannfocal should all work for latest master/constraints19:14
sean-k-mooneyyep19:15
sean-k-mooneyi just ment that after RC1 and we reopen master for bobcat19:15
sean-k-mooneythen it wont be in the testing runtime19:15
sean-k-mooneynor wil python 3.819:15
sean-k-mooneyso it will proably be fine but it could start breaking 19:15
opendevreviewSofia Enriquez proposed openstack/nova master: Check NFS protocol  https://review.opendev.org/c/openstack/nova/+/85403020:02
opendevreviewDan Smith proposed openstack/nova master: WIP: Detect host renames and abort startup  https://review.opendev.org/c/openstack/nova/+/86392020:05

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!