Tuesday, 2021-08-03

songwenpingbauzas, sean-k-mooney[m]: thanks for last week's response.01:03
bauzasgibi: morning07:29
bauzaslooks like the nova-lvm job is on the weeds07:29
*** rpittau|afk is now known as rpittau07:30
gibibauzas: o/ I have to be off for 2 hours 07:32
gibiit seems it is in the weeds for daays07:33
bauzasno worries, I'll try to look at the problme07:33
bauzasyup07:33
bauzasa snapshot not found or something like this07:33
gibilast success was  2021-07-16T08:36:3907:33
bauzasyep http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22nova-lvm%5C%22%20build_status%3A%5C%22FAILURE%5C%2207:34
bauzas(if logstash is still working correctly)07:34
gibibut then how we managed to merge things?!07:35
gibiwas this job turned off?07:35
gibior is it selectively runnig?07:36
gibiit doesn't have a too special irrelevant-files in .zuul.yaml07:37
gibiso I don't get it07:37
gibiI have a bunch of patches that nova-lvm isn't run on07:37
bauzasexcellent question07:38
gibilike https://review.opendev.org/c/openstack/nova/+/800086/1307:38
gibipingig lyarwood :)07:38
bauzaszuul is pretty https://zuul.openstack.org/builds?job_name=nova-lvm07:38
bauzasnone of the job builds worked since 2 weeks07:38
gibiand I'm dropping off now. be back later07:38
bauzasgibi: no worries and thanks07:38
yongliheHI, everyone07:40
bauzasok, the nova-lvm job is triggered on libvirt changes hence the non-failure on other changes and why we merged other stuff07:42
bauzashttps://zuul.openstack.org/job/nova-lvm07:43
bauzasnow, looks like all runs fail with the same tempest tests with the same exception being a snapshot not found07:43
bauzashuzzaaaaah \o/ https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-n-cpu.txt?severity=4#12685-1268907:46
bauzasNotImplementedError: direct_snapshot() is not implemented07:46
bauzaslooks like we regressed07:47
bauzashmmmm07:51
lyarwoodMorning07:51
lyarwoodyeah I've not had time to look at nova-lvm failures thanks to a few downstream fires07:52
lyarwoodthey should be out now so I'll look this morning07:52
bauzasok, I have better understanding07:52
lyarwoodbauzas: that isn't the failure07:52
bauzascorrect07:52
bauzaswe catch the notimplemented exception07:52
bauzasit's failing later07:52
bauzaswhen calling glance I'd say07:52
bauzashttps://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-n-cpu.txt?severity=4#1277207:53
lyarwoodyeah the g-api 502 is something I think I've seen before when we run out of space during the upload07:53
bauzasyup07:53
bauzaswe want to upload the snapshot and we fail, right?07:53
* bauzas goes looking into glance logs07:54
lyarwoodwe are in the process of uploading the snapshot and something fails07:54
lyarwoodit might even be a timeout in httpd07:54
lyarwoodgiven the size of the images we are uploading for these tests07:54
lyarwoodbrb07:55
bauzaslyarwood: https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-g-api.txt?severity=3#355507:56
bauzasthis error matches the req-id of the n-cpu issue07:56
bauzaswell, can't figure out more why the temporary image fails08:11
bauzasabhishekk: around ?08:12
bauzasabhishekk: we have a nova job failure (nova-lvm) which fails constantly because of glance API returning 502 https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/controller/logs/screen-g-api.txt?severity=3#355508:12
bauzason an image upload08:12
* bauzas will write an email to openstack-discuss08:13
* kashyap waves08:21
abhishekkbauzas, looking08:34
bauzasabhishekk: thanks, wrote an email about this08:35
bauzas(to summarize)08:35
abhishekkyes, just went through it08:35
abhishekkbauzas, are you sure failure is because of this warning ?08:36
bauzasabhishekk: I'm nearly sure the test is failing because glance is returning a 50208:37
abhishekkok, will have a look and get back to you08:37
bauzasabhishekk: but I relate the glance g-api warning with the same req-id of the failing glance image upload08:38
bauzascorrelation isn't consequence, I know, but that's a starting point08:38
abhishekkack08:38
lyarwoodbauzas: the test is 100% failing because of the 502 it just isn't clear why g-api or httpd is returning it08:39
lyarwoodsorry got my head stuck downstream again08:39
bauzasyup, that's what I said to abhishekk08:39
bauzaswe have an image upload failure08:39
bauzasbut we don't know why08:40
bauzaslyarwood: no worries about downstream rush, I feel your pain, maybe just one thought : as this is failing, should we make the job non-voting until we figure out the cause ?08:40
lyarwoodYup happy to if you can raise a bug08:41
bauzasgood call, I forgot this :facepalm:08:41
abhishekkI can see HTTP 413 in g-api log08:43
bauzaslyarwood: abhishekk: gibi: https://bugs.launchpad.net/nova/+bug/193876508:46
abhishekkack08:48
bauzasI need to disappear for 10 mins08:53
abhishekkI think this is causing problem, DEBUG oslo.limit.limit [None req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-ListImageFiltersTestJSON-1499681503 tempest-ListImageFiltersTestJSON-1499681503-project] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 1024 and delta 0] {{(pid=108370) enforce_limits /usr/local/lib/python3.8/dist-packages/oslo_limit/limit.py:268}}09:00
abhishekkAug 01 03:51:33.042224 ubuntu-focal-ovh-bhs1-0025715861 devstack@g-api.service[108370]: [pid: 108370|app: 0|req: 188/333] 127.0.0.1 () {46 vars in 1190 bytes} [Sun Aug  1 03:51:32 2021] PUT /v2/images/4d204807-d5e5-4736-821f-fae99a0c066e/file => generated 448 bytes in 743 msecs (HTTP/1.1 413) 4 headers in 187 bytes (1 switches on core 0)09:00
lyarwoodsounds about right, these images are going to be pretty big09:01
lyarwoodhttps://opendev.org/openstack/devstack/commit/5a642450d6ac94ff1ea2bea3e7ce3887ca79dcc409:02
lyarwoodthere we go09:02
* lyarwood increases this for nova-lvm09:02
abhishekkI think this will fix the problem09:03
opendevreviewLee Yarwood proposed openstack/nova master: zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm  https://review.opendev.org/c/openstack/nova/+/80332209:11
opendevreviewLee Yarwood proposed openstack/nova master: DNM - Trigger nova-lvm to test I824655387a10ac9b813c1b7b2399e25ed95f7fc3  https://review.opendev.org/c/openstack/nova/+/80332309:11
lyarwoodbauzas: / abhishekk ; ^ there we go09:11
* lyarwood wasn't sure if the first change would actually trigger nova-lvm thus the second09:12
abhishekklyarwood, ack09:12
bauzaslyarwood: and we return a 502 if we go over the limit ? :O09:16
lyarwoodbauzas: 413 but somehow that's translated into a 503 by the time n-cpu sees it09:18
lyarwoodbauzas: that could be a separate bug report against devstack tbh09:18
bauzasyup, agreed09:18
lyarwood502 even sorry09:18
bauzaseither way, excellent catch09:19
bauzasI didn't know about glance limiting now the uploads09:19
opendevreviewAlexey Stupnikov proposed openstack/nova master: Clean up unreal UUID values in tests  https://review.opendev.org/c/openstack/nova/+/80322609:19
abhishekkI see 6 413 in glance logs, that sums up total failure cases in the job09:34
abhishekkjob is passing now09:59
lyarwoodabhishekk: awesome thanks for your help with that :)10:04
gibilyarwood, bauzas thanks for fixing that up. 10:34
lyarwoodgrenade jobs blocked by https://bugs.launchpad.net/grenade/+bug/1938763 FWIW10:56
gibilyarwood: thanks I saw the error in the job run but I did not know that it is filed as a bug. I only see one failure in grenade by this so far10:58
lyarwoodnp https://zuul.opendev.org/t/openstack/builds?job_name=grenade&job_name=nova-grenade-multinode&job_name=grenade-multinode&branch=master&result=failure shows it a little better10:59
gibilyarwood: yepp that see a lot better that it is hit grenade not just nova grenade11:00
gibithanks11:00
gibi(i filtered the query for the nova grenade job only)11:01
opendevreviewSylvain Bauza proposed openstack/nova master: Provide and use other RCs for mdevs if needed  https://review.opendev.org/c/openstack/nova/+/80323311:06
opendevreviewSylvain Bauza proposed openstack/nova master: Expose the mdev class  https://review.opendev.org/c/openstack/nova/+/80174311:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Support move ops with extended resource request  https://review.opendev.org/c/openstack/nova/+/80008712:08
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] refactor interface attach with qos  https://review.opendev.org/c/openstack/nova/+/80008812:08
opendevreviewBalazs Gibizer proposed openstack/nova master: Support interaface attach / detach with new resource request format  https://review.opendev.org/c/openstack/nova/+/80008912:08
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place  https://review.opendev.org/c/openstack/nova/+/79362112:11
opendevreviewBalazs Gibizer proposed openstack/nova master: WIP support extended res req in heal port allocation  https://review.opendev.org/c/openstack/nova/+/80206012:12
bauzasgibi: stephenfin: sean-k-mooney: others, I'm eventually done with the generic-mdevs series12:21
bauzasgibi: stephenfin: sean-k-mooney: given I'll be on PTO for 3 weeks after Friday, if you want to review the series, I'd love to see your comments before I'm done12:22
bauzashttps://review.opendev.org/q/topic:%22bp%252Fgeneric-mdevs%22+(status:open%20OR%20status:merged)12:22
sean-k-mooneybauzas: almost finished getting through the 450+ emails i got when i was off for 2 days12:23
sean-k-mooneybut sure ill try an take a look today 12:23
opendevreviewSylvain Bauza proposed openstack/nova master: Expose the mdev class  https://review.opendev.org/c/openstack/nova/+/80174312:28
* bauzas goes to look at melwitt's change12:28
gibibauzas: ack, I will try to do an initial read today12:38
opendevreviewAlexey Stupnikov proposed openstack/nova master: Clean up unreal UUID values in tests  https://review.opendev.org/c/openstack/nova/+/80322613:20
opendevreviewElod Illes proposed openstack/nova stable/train: DNM: devstack-gate test  https://review.opendev.org/c/openstack/nova/+/80335114:10
gibibauzas: finished reading your mdev series, left comments in the reviews14:14
bauzasmelwitt: reviewed https://review.opendev.org/c/openstack/placement/+/669170 and had a concern about the default name which is lowercase14:25
bauzasgibi: thanks, appreciated, looking14:25
bauzas(good timing fwiw)14:25
bauzasgibi: oh the renaming thing14:26
bauzasgibi: I should have been explicit14:26
bauzasgiven I'll on PTO in 4 days, I wanted to only modify the needed names14:27
bauzasand for the internal methods names, I'll be adding a follow-up after the last change14:27
gibibauzas: followup for the internal renaming works for me14:32
bauzaslooking at your other comments14:33
gibibauzas: still the externally visible things should be renamed along the way. E.g. the exception texts 14:33
gibiInvalidLibvirtGPUConfig14:34
gibiI guess the messag from that ends up in the logs14:34
bauzasyou're maybe right14:35
bauzasgibi: I don't think it would be an upgrade modification if we provide different logs, right?14:35
gibinot an upgrade impact no14:36
opendevreviewBalazs Gibizer proposed openstack/nova master: Support move ops with extended resource request  https://review.opendev.org/c/openstack/nova/+/80008714:48
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] refactor interface attach with qos  https://review.opendev.org/c/openstack/nova/+/80008814:48
opendevreviewBalazs Gibizer proposed openstack/nova master: Support interaface attach / detach with new resource request format  https://review.opendev.org/c/openstack/nova/+/80008914:48
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place  https://review.opendev.org/c/openstack/nova/+/79362114:49
opendevreviewBalazs Gibizer proposed openstack/nova master: WIP support extended res req in heal port allocation  https://review.opendev.org/c/openstack/nova/+/80206014:49
opendevreviewAlexey Stupnikov proposed openstack/nova master: Fix incorrect reservation IDs in unit tests  https://review.opendev.org/c/openstack/nova/+/80336315:33
gibinova meeting starts in 10 minutes here in the channell15:50
sean-k-mooneygibi: were you not ment to be on PTO this week?15:50
sean-k-mooneyor is that next week15:50
sean-k-mooneyoh it was last week15:50
gibiyepp that was last week15:56
gibiand that will be next week as well (which I should mention on the meeting too)15:56
bauzasme too15:59
bauzasfor the next 3 weeks15:59
gibi#startmeeting nova16:00
opendevmeetMeeting started Tue Aug  3 16:00:19 2021 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
gibio/16:00
dansmitho/16:00
bauzashumpf, I was about to upload :)16:00
elodilleso/16:00
bauzas\o16:00
gibi#topic Bugs (stuck/critical)16:01
gibiwe have one critical bugg16:01
gibi#link https://bugs.launchpad.net/nova/+bug/1938765 and the fix is #link https://review.opendev.org/c/openstack/nova/+/80332216:01
gibiwhere is the bug bot?16:02
gmanno/16:02
gibianyhow so the critical bug is: nova-lvm job constantly fails on Glance image upload16:02
bauzasbut thanks to lyarwood we have a fix16:02
gibiwe have a fix proposed but also the grenade jobs are broken16:02
gmanngrenade fix is discussed in QA meeting today and it is approved #link https://review.opendev.org/c/openstack/grenade/+/80331716:02
gmannit is in gate now16:02
gibigmann: thanks16:03
bauzasthanks ditto16:03
gibiwe have a stable untriaged bug backlog 16:03
gibi#link 11 new untriaged bugs (+0 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New16:03
gibiany specific bug we need to talk about?16:03
gibi#topic Gate status 16:04
gibias we touched already we have a blocked gate16:04
gibidue to grenade16:04
dansmithquestion about the lvm jobm,16:04
gibiand the nova-lvm job16:04
gibidansmith: sure16:04
dansmiththat clearly wasn't always hard fail because we merged the quota thing weeks ago16:04
dansmithwas it just right on the edge of running out of space or what?16:05
gibidansmith: nova-lvm is triggered only on libvirt changes16:05
gibias I learned this morning16:05
dansmithahh, okay makes sense16:05
gibihttps://github.com/openstack/nova/blob/master/.zuul.yaml#L15216:05
bauzasyup16:06
bauzassee my email thread16:06
bauzasit only touches the libvirt ones16:06
bauzasas I guess the lvm imagebackend is only supported by the libvirt driver, right?16:07
dansmithyeah, I didn't get that from your email16:07
gibiI guess so too16:07
dansmithbut I see that's what you meant16:07
bauzasbut, yeah we could maybe regress due to some other modification and don't see it16:07
gibiso besides the two gate blockers we have a list of bugs causing some trouble on the gate #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure16:08
gibibut looking at that list I don't see any new one (besides the nova-lvm bug)16:08
gibiplacement periodic jobs are green16:09
gibi#link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly16:09
gibiany gate failure we need to talk about today?16:09
gibi#topic Release Planning 16:10
gibiWe past M2 and spec freeze. M3 is in 4 weeks.16:10
gibiWe have 21 approved an open blueprints and we have 4 weeks to finish them. Please focus review effort on bps in Needs Code Review state.16:10
gibiNext deadline is non-client library freeze at 16th of August16:10
gibithese are the features we are tracking for Xena #link https://launchpad.net/nova/+milestone/xena-3 16:11
gibiis there any bp that needs special attention?16:11
gibi(besides the generic "I need code review" requests)16:12
gibi#topic PTG Planning 16:13
gibiPTG timeslots has been booked see #link http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023787.html16:13
gibiThe PTG etherpad is ready to be filled with topics: #link https://etherpad.opendev.org/p/nova-yoga-ptg16:13
gibiIf you see a need for a specific cross project section then please let me know16:13
*** rpittau is now known as rpittau|afk16:13
gibiany question about the upcoming PTG?16:13
gibi#topic Stable Branches 16:15
gibiwallaby, victoria, ussuri gates look OK16:15
gibitrain, stein, rocky (and probably queens, pike?) gates fail: grenade + nova-live-migration jobs ("rsync: change_dir "/home/zuul/src/*/openstack/neutron-lbaas" failed: No such file or directory")16:15
gibipossible fix: https://review.opendev.org/c/openstack/devstack-gate/+/80334616:15
gibiEOM( elodilles 16:15
gibiany question about stable?16:16
gibi#topic Sub/related team Highlights 16:17
gibiLibvirt (bauzas) 16:17
bauzaswell, I think we discussed about the blocking job...16:17
bauzasfor the moment, hold your breath if you wanna touch libvirt16:17
gibijepp16:18
gibi#topic Open discussion16:18
gibiI have two small topic16:18
bauzasfor other stuff, I don't know, we're close to M-3 so maybe it should be a right time about thinking about the libvirt release versions16:18
bauzasthat's it16:18
gibiohh sory16:18
gibisorry16:18
bauzasnah16:18
gibido we need to bump min version?16:18
bauzasnothing urgent but I'll be off the grid for the next 3 weeks16:19
bauzasgibi: good question, I don't know 16:19
bauzasthat's why i asked for it16:19
bauzasbut meh, let's move on16:20
gibilast time we bumped in wallaby https://review.opendev.org/c/openstack/nova/+/75470016:20
gibianyhow if somebody feels that we need a libvirt min version bump then it is the good time to propose it16:21
gibigoing back to 16:22
gibi#topic Open discussion16:22
gibi1) this week Thrusday we will have the Asia friendly meeting slot16:22
gibiAug 5 2021 8:00 UTC #openstack-nova (http://www.timeanddate.com/worldclock/fixedtime.html?iso=2021-08-05T08:00:00)16:23
gibiI will chair it as usual16:23
gibi2) I will be on PTO next week so I need somebody to chair the next week's weekly meeting16:23
gibior I will cancel itt16:23
gibiit16:23
bauzascan't run it16:24
bauzasas I'll be on PTO for the next 3 weeks16:24
gibithen I will cancel it16:26
gibino worries16:26
bauzasheh16:26
gibiany other topic for today?16:26
stephenfinyeah, cancelling seems best given the amount of people that won't be around (I'm thinking I'll be out next week too)16:26
stephenfinno other topics from me16:27
gibiwe should just close nova next week :D16:27
gibiso if no other topic then that was it16:28
gibithanks for joining today16:28
gibi#endmeeting16:28
opendevmeetMeeting ended Tue Aug  3 16:28:39 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:28
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-03-16.00.html16:28
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-03-16.00.txt16:28
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-03-16.00.log.html16:28
bauzasgibi: thanks16:28
gmannthanks gibi 16:28
gibibauzas: thanks for running it last week16:29
bauzasgibi: well, we had more people than last time I ran, so this was good :D16:29
gibi:)16:30
opendevreviewSylvain Bauza proposed openstack/nova master: Rename vgpu options to mdev  https://review.opendev.org/c/openstack/nova/+/80160716:50
opendevreviewSylvain Bauza proposed openstack/nova master: Provide the mdev class for every PCI device  https://review.opendev.org/c/openstack/nova/+/80291816:50
opendevreviewSylvain Bauza proposed openstack/nova master: Provide and use other RCs for mdevs if needed  https://review.opendev.org/c/openstack/nova/+/80323316:50
opendevreviewSylvain Bauza proposed openstack/nova master: Expose the mdev class  https://review.opendev.org/c/openstack/nova/+/80174316:50
opendevreviewSylvain Bauza proposed openstack/nova master: Change the admin-visible logs for mdev support  https://review.opendev.org/c/openstack/nova/+/80337816:50
opendevreviewSylvain Bauza proposed openstack/nova master: WIP: Cleanup GPU vs. mdev wording  https://review.opendev.org/c/openstack/nova/+/80337916:50
bauzasgibi: the last change is for you ^ ;)16:50
gibithank <316:51
* bauzas bails out16:51
gibis16:51
opendevreviewBalazs Gibizer proposed openstack/nova master: Support interaface attach / detach with new resource request format  https://review.opendev.org/c/openstack/nova/+/80008917:54
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place  https://review.opendev.org/c/openstack/nova/+/79362117:56
opendevreviewBalazs Gibizer proposed openstack/nova master: WIP support extended res req in heal port allocation  https://review.opendev.org/c/openstack/nova/+/80206017:56
opendevreviewMerged openstack/nova stable/victoria: Move 'check-cherry-picks' test to gate, n-v check  https://review.opendev.org/c/openstack/nova/+/79704020:03
opendevreviewMerged openstack/nova stable/ussuri: Move 'check-cherry-picks' test to gate, n-v check  https://review.opendev.org/c/openstack/nova/+/79705022:24

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!