Tuesday, 2022-11-08

opendevreviewDan Smith proposed openstack/nova master: Test ceph-multistore with a real image  https://review.opendev.org/c/openstack/nova/+/86086400:56
opendevreviewMerged openstack/nova stable/yoga: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86187101:34
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: add regression test case for bug 1978983  https://review.opendev.org/c/openstack/nova/+/86260305:58
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: For evacuation, ignore if task_state is not None  https://review.opendev.org/c/openstack/nova/+/86260405:58
ygk_12345hi all08:59
ygk_12345I am trying to use pdb trace in the nova-api service in OSA setup.  but when I start it manually , it is not dropping into  the pdb shell. can anyone help me here ?09:01
bauzasygk_12345: you need to run a WSGI server that can support pdb09:01
bauzasygk_12345: example with httpd mod_wsgi https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#python-interactive-debugger09:04
bauzassimple way with uwsgi https://stackoverflow.com/questions/18427948/how-to-debug-python-application-under-uwsgi09:06
ygk_12345bauzas: doesn't OSA nova-api support pdb ?09:14
bauzasygk_12345: I suppose nova-api wsgi app runs on a wsgi server (which I guess is httpd) which runs on daemon mode09:15
bauzasif it would be uwsgi, that would still be a daemon09:16
bauzasif you wanna use pdb, you need to run the wsgi server on demand and not as a daemon thru systemd or whatever09:16
bauzasor rather use a remote pdb 09:17
bauzasbut then you would need to make sure that you run on a single nova-api worker 09:18
ygk_12345bauzas: I am using this manual command "/openstack/venvs/uwsgi-20.2.1-python3/bin/uwsgi --ini /etc/uwsgi/nova-api-os-compute.ini "09:19
ygk_12345bauzas: I have inserted importpdb;pdb.set_trace()  in the file "/openstack/venvs/nova-20.2.1/lib/python3.6/site-packages/nova/objects/instance.py . but I see no pdb prompt09:20
bauzasygk_12345: you should probably add --honour-stdin and --workers 1 to the uwsgi command09:23
ygk_12345bauzas: let me try that09:23
bauzasygk_12345: and then, add a breakpoint in some easy rest endpoint call, like in nova.openstack.compute.api.servers on the get_instances() method09:24
bauzasoh wait, you're not getting a BdBQuit error ?09:25
bauzasthat means that you haven't walked yet to where you added the breakpoint 09:25
ygk_12345bauzas: still no luck09:26
ygk_12345bauzas: i am running the command nova list09:26
bauzasygk_12345: yup, because as i said just above, you haven't went to that line of code09:26
bauzass/went/gone09:26
ygk_12345get_all  functions is executed from /openstack/venvs/nova-20.2.1/lib/python3.6/site-packages/nova/objects/instance.py  right ?09:27
bauzasygk_12345: oh, you're not looking at the nova-api logs ?09:27
ygk_12345bauzas: no, I am tracing the code09:27
bauzasygk_12345: just to clarify, you won't get a prompt on your client call09:28
ygk_12345bauzas: yes 09:28
bauzasthe interactive debugging session will run on the nova-api process that executes the server-side code for listing the instances09:28
bauzasyou can trigger it thru a client call of course09:29
bauzasygk_12345: to answer your question, no, a nova list doesn't lookup the instance records09:31
bauzasahah, I see your problem09:31
bauzassince Newton, we don't expose the Instance record when you nova list09:32
bauzaswe rather expose an InstanceMapping record, eventually a BuildRequest object09:32
bauzasyou should rather try a nova show 09:32
ygk_12345can anyone help me with this ?09:36
opendevreviewRajesh Tailor proposed openstack/os-vif stable/zed: Move mtu update request into ovsdb transaction  https://review.opendev.org/c/openstack/os-vif/+/86399311:31
opendevreviewRajesh Tailor proposed openstack/os-vif stable/yoga: Move mtu update request into ovsdb transaction  https://review.opendev.org/c/openstack/os-vif/+/86399411:32
opendevreviewRajesh Tailor proposed openstack/os-vif stable/xena: Move mtu update request into ovsdb transaction  https://review.opendev.org/c/openstack/os-vif/+/86399511:34
admin1hi all .. my nova-api logs is filled with  ERROR oslo_service.periodic_task oslo_ db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query'  .. but it reconnects  .. and this is happening every 1-2 mins ..   other apis are fine12:15
admin1am I hitting some kind of nova limits or connection/cache timeouts 12:15
admin1mysql  connection limit is not reached12:15
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86400613:57
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86400713:57
ygk_12345bauzas: now I am able to get the pdb prompt in the uwsgi process, but whatever pdb commands I type are not executing. also the cursor is coming out of pdb prompt by one line and waiting there14:01
*** dasm|off is now known as dasm14:02
opendevreviewDan Smith proposed openstack/nova master: Test ceph-multistore with a real image  https://review.opendev.org/c/openstack/nova/+/86086414:36
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86400614:47
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86400714:47
opendevreviewDan Smith proposed openstack/nova master: Test ceph-multistore with a real image  https://review.opendev.org/c/openstack/nova/+/86086414:54
Kirill_Hi, do we have a meeting today?15:01
elodillesKirill_: if i'm not mistaken we will have @ 16:00 UTC15:03
Kirill_got it, thanks15:03
opendevreviewsean mooney proposed openstack/nova master: Support multiple config file with mod_wsgi  https://review.opendev.org/c/openstack/nova/+/86401415:16
bauzasreminder : nova meeting in 34 min15:26
opendevreviewAmit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86380615:42
han-guangyuHello, My name is Han Guangyu.15:43
han-guangyuCurrently, I work in a startup operating system company. Some time ago, the work I participated in : building rpm package for openstack, deploying some small private clouds using openstack, dealing with usage problems encountered by customers.15:43
han-guangyuI have some basic open source community participation experience. Such as, I have been involved the work of OpenStack SIG of openEuler community(https://www.openeuler.org/en/ A OS community). The basic community process and the use of tools such as git and gerrit are ok for me15:43
han-guangyuNow, My job allowed me to transition into an openstack developer. So, I hope and have time to get involved in the openstck community.15:43
han-guangyuBut because there is no accumulation, I now mainly rely on self-study. At present, I have initially debugged the process of creating an instance, and I have an understanding of the whole, but I have not gone into the details. Earlier I assigned myself a low-hanging-frout bug, and I wanted to finish it (Sorry delay to work it). 15:43
han-guangyuI have an understanding of the basic syntax of Python, but I have only written some simple scripts and have not actually participated in the development of large-scale python projects. So, Nova is complicated, for me.15:43
han-guangyuJust an introduction to myself. And I would like to ask, for someone without much python experience, do you have any suggestions for self-learning nova source code?15:44
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Nov  8 16:00:04 2022 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzashello folks, welcome in the nova meeting16:00
gibio/16:00
Kirill_0/16:00
elodilleso/16:00
* gibi is distracted16:00
gmanno/16:00
bauzask, let's start16:01
auniyalO/16:01
bauzas #topic Bugs (stuck/critical)16:01
bauzas#topic Bugs (stuck/critical) 16:01
bauzas#info No Critical bug16:01
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 5 new untriaged bugs (-3 since the last meeting)16:01
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:01
bauzasthanks gibi and others for having looked at the bugs16:01
bauzasany bug to discuss ?16:02
bauzaslooks not16:03
Ugglao/16:03
han-guangyuo/16:03
bauzasmelwitt: could you be using the bug baton next week ?16:03
gibibauzas: yeah I did some triage last week16:04
gibinot much this week thouh16:04
gibithough16:04
bauzascool16:04
bauzasok, I'll ask again to melwitt to know if she can help for bug triage this week 16:05
bauzasmoving on16:05
bauzas#topic Gate status 16:05
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:05
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:05
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:05
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:05
bauzasas you can see, no issues for the CI periodic jobs (except for fips)16:06
bauzasI need to modify the job to have more 30 mins 16:06
bauzas*deadline16:06
bauzasany concern ?16:07
bauzaslooks not16:07
bauzasmoving on again16:07
bauzas#topic Release Planning 16:07
bauzas#link https://releases.openstack.org/antelope/schedule.html16:08
bauzas#info Antelope-1 is planned in 1 week16:08
bauzassl16:08
bauzasso16:08
bauzaswhen we should have a specs review day ?16:08
bauzasnext tuesday works ?16:08
sean-k-mooneyproably yes16:09
sean-k-mooneyill be around and hopefully have more time to review next week16:09
bauzasok, anyone having an issue ?16:10
bauzasfor next tuesday 16:10
gibiworks for me16:10
bauzasok16:11
bauzas#agreed We will have have a spec review day on Nov 15th, please prepare your specs before this day and make sure you can be around on this day if you have reviews16:12
bauzas#action bauzas to write an openstack-discuss email for this16:12
bauzas#topic Review priorities 16:12
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:13
bauzasas you can see, now we have more review-prio usage from contributors :)16:14
bauzas<316:14
bauzasso, please, look at those when you review16:14
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:14
bauzasany question here ?16:14
sean-k-mooneyits not qiute what i was suggestign for the usage but ok16:15
sean-k-mooney+1 was commit to review +2 was asking for other cores to also review16:15
sean-k-mooneybut either woks16:15
bauzask16:16
bauzasat least contributors can +1 if they want core reviews for some change they reviewed (not for their own changes of course)16:17
bauzasmoving on ?16:18
bauzas#topic Stable Branches 16:18
bauzaselodilles : your time16:18
elodilles#info wallaby transitioned to Extended Maintenance ( https://review.opendev.org/c/openstack/releases/+/862298 )16:18
elodillesand we have released a final wallaby release16:19
elodilles#info stable branches state should be OK (except stein and older)16:19
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:19
elodillesand last but not least:16:19
elodillesas said 2 weeks ago: stein, rocky and queens EOL'ing patch was proposed: https://review.opendev.org/862520 + mail was sent to ML https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030980.html16:20
elodillesas I see no mail replies arrived on the thread16:20
elodillesso I propose to continue with the EOL'ing16:20
bauzasyup, agreed16:20
bauzasI just +1d the change16:20
elodillesbauzas: thanks!16:21
bauzaswe agreed on it during the PTG16:21
bauzasand given none has concerns by the email, that means we can EOL 16:21
bauzaselodilles: I guess you can now remove the -W16:21
elodillesremoved it now16:22
bauzasthbanks16:22
bauzasanything else to discuss about the stable branches ?16:23
elodillesnothing from me16:23
bauzasok, last topic then16:24
bauzas#topic Open discussion 16:24
bauzaswe have an item today16:25
bauzasKirill_ : vnc console for baremetal. new spec: https://review.opendev.org/c/openstack/nova-specs/+/86377316:25
bauzasI guess you want reviews ? 16:25
Kirill_hi! yes. also i've emplemented it)16:26
bauzasKirill_: OK, then no worries, we'll look at your spec at least on next Tuesday16:27
sean-k-mooneyso you used to be able to do this in the past16:27
bauzasbut if you can discuss about the design, sure, let us knwo16:27
sean-k-mooneywith the ipmi driver and vncinabox or something like that16:27
bauzasif you *want16:27
sean-k-mooneyshellinabox16:28
sean-k-mooneythat was the ironic optional component16:28
sean-k-mooneyhttps://docs.openstack.org/ironic/latest/admin/console.html16:28
sean-k-mooneyif you use that you can enabel the console to work in horizong today16:29
Kirill_i work on idrac servers+adding ilo console16:29
Kirill_but the main gaul is16:29
Kirill_new graphical console interface16:29
Kirill_i also emplemented it. but will create mr after approve of this spec16:30
Kirill_in my spec i added all links. also link to new graphical console interface approved spec16:31
sean-k-mooneyok i would liek to see some context in the spec why the existing horizon integration which used to work with nova provisiond server no longer works16:31
sean-k-mooneyok the reaons im asking this i si used ot have ironic deployed with the geniric ipmi driver and shellinabox at home a few years ago and it worked fine in horizon16:32
sean-k-mooneyuseing a difffent ironic driver ideally shoudl not requrie changed to nova16:32
sean-k-mooneyso if we add this new feature i would hope it will work for all ironic drivers16:32
Kirill_for vnc 16:32
Kirill_we need to create a handshake16:32
Kirill_for this handshake we need a password from ironic16:33
sean-k-mooneyoh the old solution used the serial console proxy16:33
sean-k-mooneynot the novnc proxy16:33
sean-k-mooneythat the delta16:33
sean-k-mooneyok makes sense16:33
Kirill_we need a graphical console not serial16:33
sean-k-mooneyack ya just was trying to figure out why this is fiddernt then https://docs.openstack.org/ironic/latest/admin/console.html16:34
sean-k-mooneyits the grapical console not serial16:34
Kirill_it is not the same console as vnc16:35
bauzassean-k-mooney: Kirill_:  could you maybe discuss this in the spec then ?16:36
bauzasor after the meeting ?16:36
Kirill_++16:36
bauzasI'm about to end the meeting16:36
bauzasso you can continue to discuss in the IRC channel after16:36
bauzasok, anyone else having an item to discuss ?16:37
Kirill_just to sum up. on 15th November we will review all specs (and mine too)?)))16:37
bauzasyes16:38
Kirill_got it, thanks16:38
bauzasok, then let's close the meeting16:39
bauzasthanks all16:39
bauzas#endmeeting16:39
opendevmeetMeeting ended Tue Nov  8 16:39:45 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:39
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2022/nova.2022-11-08-16.00.html16:39
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-11-08-16.00.txt16:39
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2022/nova.2022-11-08-16.00.log.html16:39
elodillesthanks bauzas o/16:40
opendevreviewDan Smith proposed openstack/nova master: Test ceph-multistore with a real image  https://review.opendev.org/c/openstack/nova/+/86086417:12
opendevreviewAmit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86380619:05
opendevreviewAmit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86405519:05
opendevreviewAmit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86380620:08
opendevreviewAmit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86405520:08
*** dasm is now known as dasm|off22:28
clarkbHello nova. I've been testing server rescue behavior recently and run into two different interesting behaviors. The first is when rescuing a non bfv instance if the rootfs label configured in grub and fstab is the same between the rescue image and the instance being recsued you can boot into the rescue image kernel but have the rescued instance / mounted23:00
clarkbThis is problematic because if the problem is say in systemd init steps you'd still be broken in the rescued setup. Additionally it may not always be the case that the kernel and your rescued filesystem are compatbile enough to boot23:00
clarkbThat said I'm not sure if nova can do anything to deal with this problem direclty. I suspect one of the best options is for clouds to have a purpose built rescue image that isn't likely to collide in this way23:01
clarkbThe other issue is when doing rescue on a bfv instance the api accepts the request (as long as yo uset the api version high enough) but then the instance promptly goes into an error state with Driver Error: Cannot access storage file and no such file or directory with a path.23:02
clarkbIt almost looks like nova / libvirt are looking for a disk file rather than looking for the volume via whatever the volume provider mechansim is23:02
clarkbI don't have enough access on the cloud side to debug this further so I'm not sure if this is a nova issue or a cloud configuration issue etc.23:03
clarkbIt does make me wonder if we've got any rescue testing to ensure this generally works? Also, should we try to write more docs on how to ensure rescues work?23:03
clarkbI'm coming at this as an end user wishing this functioned bette rand wondering what I/we can do to get there. But I lack a lot of background and knowledge on the inner working here :)23:04
clarkbseparately, I do wonder if it makes more sense for bfv rescue to be a process more like "stop instnace, detach volume from instance, attach volume to another instance, make changes, reattach to original instance and start original instance" but this fails because you can't detach a root device23:05
opendevreviewmelanie witt proposed openstack/nova master: DNM testing images_type = raw with resize enabled  https://review.opendev.org/c/openstack/nova/+/86241623:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!