*** hongbin has quit IRC | 00:04 | |
*** hongbin has joined #openstack-nova | 00:05 | |
*** gyee has quit IRC | 00:21 | |
*** rambo_li has joined #openstack-nova | 00:27 | |
*** songwenping_ has joined #openstack-nova | 00:28 | |
*** mlavalle has quit IRC | 00:30 | |
*** hamalq has quit IRC | 00:36 | |
*** rambo_li has quit IRC | 00:41 | |
*** hongbin has quit IRC | 00:41 | |
*** spatel has joined #openstack-nova | 00:56 | |
*** zhanglong has joined #openstack-nova | 01:01 | |
*** hongbin has joined #openstack-nova | 01:06 | |
*** rcernin has quit IRC | 01:11 | |
*** rcernin has joined #openstack-nova | 01:11 | |
*** zhanglong has quit IRC | 01:13 | |
*** zhanglong has joined #openstack-nova | 01:14 | |
*** nweinber has joined #openstack-nova | 01:15 | |
*** Liang__ has joined #openstack-nova | 01:23 | |
*** avolkov has quit IRC | 01:25 | |
*** eandersson has quit IRC | 01:29 | |
*** eandersson has joined #openstack-nova | 01:29 | |
*** songwenping__ has joined #openstack-nova | 01:38 | |
*** nweinber has quit IRC | 01:39 | |
*** songwenping_ has quit IRC | 01:41 | |
*** rambo_li has joined #openstack-nova | 01:50 | |
*** songwenping_ has joined #openstack-nova | 01:53 | |
*** songwenping__ has quit IRC | 01:56 | |
*** spatel has quit IRC | 02:09 | |
*** spatel has joined #openstack-nova | 02:14 | |
*** spatel has quit IRC | 02:16 | |
*** openstackgerrit has joined #openstack-nova | 02:22 | |
openstackgerrit | renminmin proposed openstack/nova master: Fix instance booted with snapshot image and bdms https://review.opendev.org/745821 | 02:22 |
---|---|---|
*** lbragstad has quit IRC | 02:32 | |
*** songwenping_ has quit IRC | 02:42 | |
*** songwenping_ has joined #openstack-nova | 02:43 | |
openstackgerrit | Merged openstack/nova master: Detach is broken for multi-attached fs-based volumes https://review.opendev.org/741712 | 02:50 |
*** Liang__ has quit IRC | 03:01 | |
*** xinranwang__ has joined #openstack-nova | 03:02 | |
*** rcernin has quit IRC | 03:06 | |
*** Liang__ has joined #openstack-nova | 03:14 | |
*** psachin has joined #openstack-nova | 03:31 | |
*** ayoung has quit IRC | 03:48 | |
*** ayoung has joined #openstack-nova | 03:48 | |
*** rcernin has joined #openstack-nova | 03:52 | |
*** ayoung has quit IRC | 03:53 | |
*** ayoung has joined #openstack-nova | 03:56 | |
*** lseki has quit IRC | 04:01 | |
*** Liang__ has quit IRC | 04:02 | |
*** eharney has quit IRC | 04:05 | |
*** Liang__ has joined #openstack-nova | 04:05 | |
*** zhanglong has quit IRC | 04:06 | |
*** mkrai has joined #openstack-nova | 04:23 | |
*** eharney has joined #openstack-nova | 04:24 | |
*** CeeMac has quit IRC | 04:26 | |
*** hongbin has quit IRC | 04:26 | |
*** vishalmanchanda has joined #openstack-nova | 04:33 | |
*** evrardjp has joined #openstack-nova | 04:33 | |
*** Liang__ has quit IRC | 04:34 | |
*** ratailor has joined #openstack-nova | 04:34 | |
openstackgerrit | Harshavardhan Metla proposed openstack/nova master: Raising a failure in case of 403 https://review.opendev.org/735068 | 04:53 |
*** sridharg has joined #openstack-nova | 05:10 | |
*** JamesBenson has quit IRC | 05:25 | |
*** JamesBenson has joined #openstack-nova | 05:27 | |
*** JamesBenson has quit IRC | 05:31 | |
*** jsuchome has joined #openstack-nova | 05:45 | |
*** songwenping__ has joined #openstack-nova | 06:01 | |
*** zhanglong has joined #openstack-nova | 06:02 | |
*** CeeMac has joined #openstack-nova | 06:04 | |
*** songwenping_ has quit IRC | 06:04 | |
*** brinzhang has quit IRC | 06:06 | |
*** zhanglong has quit IRC | 06:11 | |
*** rambo_li has quit IRC | 06:12 | |
*** zhanglong has joined #openstack-nova | 06:21 | |
*** songwenping_ has joined #openstack-nova | 06:28 | |
*** xinranwang__ has quit IRC | 06:31 | |
*** songwenping__ has quit IRC | 06:31 | |
*** markvoelker has joined #openstack-nova | 06:45 | |
*** slaweq has joined #openstack-nova | 06:46 | |
*** Liang__ has joined #openstack-nova | 06:46 | |
*** zhanglong has quit IRC | 06:48 | |
*** markvoelker has quit IRC | 06:49 | |
*** JamesBenson has joined #openstack-nova | 06:50 | |
*** zhanglong has joined #openstack-nova | 06:51 | |
*** JamesBenson has quit IRC | 06:55 | |
*** maciejjozefczyk has joined #openstack-nova | 06:57 | |
*** bbowen has quit IRC | 06:59 | |
*** mvorwerk has joined #openstack-nova | 07:21 | |
*** mkrai has quit IRC | 07:21 | |
*** tesseract has joined #openstack-nova | 07:22 | |
*** Liang__ has quit IRC | 07:23 | |
*** Liang__ has joined #openstack-nova | 07:24 | |
*** mkrai has joined #openstack-nova | 07:26 | |
*** markvoelker has joined #openstack-nova | 07:35 | |
*** markvoelker has quit IRC | 07:39 | |
*** tosky has joined #openstack-nova | 07:42 | |
*** zhanglong has quit IRC | 07:46 | |
*** markvoelker has joined #openstack-nova | 07:50 | |
*** markvoelker has quit IRC | 07:51 | |
*** zhanglong has joined #openstack-nova | 07:54 | |
*** nightmare_unreal has joined #openstack-nova | 08:05 | |
openstackgerrit | Alexandre Arents proposed openstack/nova master: Rebase qcow2 images when unshelving an instance https://review.opendev.org/696084 | 08:07 |
*** maciejjozefczyk_ has joined #openstack-nova | 08:14 | |
*** maciejjozefczyk has quit IRC | 08:15 | |
*** maciejjozefczyk has joined #openstack-nova | 08:15 | |
*** mkrai has quit IRC | 08:17 | |
*** maciejjozefczyk_ has quit IRC | 08:19 | |
openstackgerrit | Jie Li proposed openstack/nova master: fix revert resize failed https://review.opendev.org/746020 | 08:27 |
*** belmoreira has joined #openstack-nova | 08:30 | |
*** brinzhang has joined #openstack-nova | 08:43 | |
*** mkrai has joined #openstack-nova | 08:48 | |
*** JamesBenson has joined #openstack-nova | 08:51 | |
*** JamesBenson has quit IRC | 08:56 | |
*** links has joined #openstack-nova | 09:03 | |
*** rcernin has quit IRC | 09:05 | |
*** avolkov has joined #openstack-nova | 09:14 | |
*** Liang__ has quit IRC | 09:24 | |
*** JamesBenson has joined #openstack-nova | 09:26 | |
*** JamesBenson has quit IRC | 09:30 | |
*** jangutter has quit IRC | 09:47 | |
*** jangutter has joined #openstack-nova | 09:47 | |
*** rcernin has joined #openstack-nova | 09:57 | |
*** brinzhang_ has joined #openstack-nova | 09:58 | |
*** songwenping__ has joined #openstack-nova | 09:58 | |
*** brinzhang has quit IRC | 10:01 | |
*** songwenping_ has quit IRC | 10:02 | |
*** songwenping__ has quit IRC | 10:02 | |
*** songwenping__ has joined #openstack-nova | 10:02 | |
*** bbowen has joined #openstack-nova | 10:07 | |
*** mvorwerk has quit IRC | 10:17 | |
*** mkrai has quit IRC | 10:28 | |
*** mkrai has joined #openstack-nova | 10:31 | |
*** mvorwerk has joined #openstack-nova | 10:32 | |
*** brinzhang0 has joined #openstack-nova | 10:33 | |
*** brinzhang_ has quit IRC | 10:36 | |
*** brinzhang has joined #openstack-nova | 10:37 | |
*** brinzhang0 has quit IRC | 10:39 | |
*** nightmare_unreal has quit IRC | 10:40 | |
*** rcernin has quit IRC | 10:40 | |
*** rcernin has joined #openstack-nova | 10:47 | |
*** songwenping_ has joined #openstack-nova | 10:48 | |
*** songwenping__ has quit IRC | 10:51 | |
*** rcernin has quit IRC | 11:00 | |
*** jangutter has quit IRC | 11:01 | |
*** songwenping_ has quit IRC | 11:29 | |
*** songwenping_ has joined #openstack-nova | 11:29 | |
*** brinzhang_ has joined #openstack-nova | 11:32 | |
*** brinzhang has quit IRC | 11:36 | |
*** mkrai has quit IRC | 11:43 | |
*** raildo has joined #openstack-nova | 11:49 | |
*** mvorwerk has quit IRC | 11:56 | |
*** songwenping__ has joined #openstack-nova | 11:58 | |
*** songwenping_ has quit IRC | 12:01 | |
*** mvorwerk has joined #openstack-nova | 12:09 | |
*** zhanglong has quit IRC | 12:10 | |
*** brinzhang0 has joined #openstack-nova | 12:13 | |
*** brinzhang_ has quit IRC | 12:16 | |
*** brinzhang0 has quit IRC | 12:26 | |
*** songwenping_ has joined #openstack-nova | 12:28 | |
*** mvorwerk has quit IRC | 12:30 | |
*** songwenping__ has quit IRC | 12:31 | |
*** ayoung has quit IRC | 12:33 | |
openstackgerrit | Merged openstack/nova master: [Trivial] Remove wrong format_message() conversion https://review.opendev.org/744280 | 12:35 |
*** songwenping__ has joined #openstack-nova | 12:43 | |
*** ayoung has joined #openstack-nova | 12:45 | |
*** nweinber has joined #openstack-nova | 12:46 | |
*** JamesBenson has joined #openstack-nova | 12:46 | |
*** songwenping_ has quit IRC | 12:47 | |
*** mvorwerk has joined #openstack-nova | 12:47 | |
*** JamesBenson has quit IRC | 12:51 | |
openstackgerrit | Mark Goddard proposed openstack/nova master: docs: fix aggregate weight multiplier property names https://review.opendev.org/746128 | 12:53 |
*** JamesBenson has joined #openstack-nova | 13:01 | |
*** dklyle has quit IRC | 13:07 | |
*** ociuhandu has quit IRC | 13:09 | |
*** mvorwerk has quit IRC | 13:10 | |
*** lseki has joined #openstack-nova | 13:12 | |
*** lbragstad has joined #openstack-nova | 13:13 | |
*** ratailor has quit IRC | 13:14 | |
*** mkrai has joined #openstack-nova | 13:15 | |
*** songwenping_ has joined #openstack-nova | 13:23 | |
*** ociuhandu has joined #openstack-nova | 13:24 | |
*** songwenping__ has quit IRC | 13:26 | |
*** mvorwerk has joined #openstack-nova | 13:27 | |
*** ociuhandu has quit IRC | 13:29 | |
*** tkajinam has quit IRC | 13:32 | |
*** mvorwerk has quit IRC | 13:42 | |
*** dave-mccowan has quit IRC | 13:59 | |
*** dave-mccowan has joined #openstack-nova | 14:00 | |
*** ociuhandu has joined #openstack-nova | 14:02 | |
*** ociuhandu_ has joined #openstack-nova | 14:19 | |
*** dave-mccowan has quit IRC | 14:21 | |
*** ociuhandu has quit IRC | 14:22 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in baremetal nodes policies https://review.opendev.org/741716 | 14:28 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in baremetal nodes policy https://review.opendev.org/741717 | 14:28 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566 | 14:31 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568 | 14:31 |
*** dklyle has joined #openstack-nova | 14:33 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add scope and new default roles in extensions policies https://review.opendev.org/743046 | 14:34 |
artom | Is logstash... working? | 14:36 |
artom | I'm trying to understand if "Multiple possible networks found, use a Network ID to be more specific" in the nova-next job is a recurring thing | 14:36 |
*** songwenping__ has joined #openstack-nova | 14:43 | |
*** songwenping_ has quit IRC | 14:46 | |
*** mkrai has quit IRC | 14:52 | |
*** mkrai_ has joined #openstack-nova | 14:52 | |
*** sridharg has quit IRC | 14:52 | |
*** k_mouza has joined #openstack-nova | 14:54 | |
sean-k-mooney | artom: its a few days delayed | 14:55 |
sean-k-mooney | i think 3-5 days | 14:56 |
*** ayoung has quit IRC | 14:56 | |
sean-k-mooney | artom: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Multiple%20possible%20networks%20found,%20use%20a%20Network%5C%22 | 14:57 |
sean-k-mooney | set it to 30days and you get hits | 14:57 |
artom | Ah yes, indeed. I feel like we've hit this before, for a different job | 14:58 |
artom | gmann ^^ sound familiar? | 14:58 |
sean-k-mooney | we have hit this many times | 14:58 |
sean-k-mooney | its a tempest bug | 14:58 |
sean-k-mooney | the tests are not wrtten correctly | 14:58 |
artom | I kinda figured that already :P | 14:59 |
artom | I was looking for a previous fix that I could plagiarize to avoid having to think about this ;) | 14:59 |
artom | Or to straight up "delegate" to someone else :P | 14:59 |
sean-k-mooney | the first failure was in this requiremnt update to neutron libs https://review.opendev.org/#/c/745237 | 15:01 |
sean-k-mooney | that may or may not be related to the cause | 15:02 |
*** mlavalle has joined #openstack-nova | 15:10 | |
gmann | artom: sean-k-mooney yeah, we fixed this issue while we see error. | 15:12 |
artom | gmann, do you remember what the fix was? Do we need to redo it somewhere else? | 15:15 |
*** JamesBen_ has joined #openstack-nova | 15:16 | |
gmann | by default, tempest tests does not pass the dedicated network while creating server so we go test by test and request for unique network to be created for test | 15:16 |
gmann | artom: yeah | 15:16 |
*** JamesBenson has quit IRC | 15:17 | |
gmann | artom: https://review.opendev.org/#/c/721767/ | 15:17 |
*** songwenping_ has joined #openstack-nova | 15:17 | |
gmann | which test failing? | 15:17 |
artom | gmann, looks like we're hitting it in tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume | 15:18 |
artom | And something else, but I can't figure out the test from the kibana info | 15:18 |
gmann | artom: ah that is scenario test which always has the unique network, let me confirm | 15:19 |
sean-k-mooney | there are a number of test failing | 15:20 |
*** songwenping__ has quit IRC | 15:20 | |
sean-k-mooney | tempest-AttachInterfacesTestJSON tempest-test_creds-1679784040 | 15:22 |
gmann | for API tests, root cause is same and we can fix it in Tempest side | 15:22 |
sean-k-mooney | actully i need to filter out passing jobs | 15:23 |
sean-k-mooney | apparently we have some tests that try to create that error | 15:23 |
artom | sean-k-mooney, ah, it would explain why https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1a8/744280/6/check/nova-next/1a8a26e/controller/logs/screen-n-api.txt is from a passing job | 15:24 |
artom | I guess start by fixing test_server_sequence_suspend_resume then? | 15:24 |
sean-k-mooney | oh so 212 hits instead of ~500 | 15:25 |
sean-k-mooney | and some of those have failure that are not related to this | 15:26 |
sean-k-mooney | this will be annoying to fine a query for | 15:26 |
gmann | test_server_sequence_suspend_resume does not explicit create the network and test try to get one from neutron - https://github.com/openstack/tempest/blob/master/tempest/scenario/manager.py#L257 | 15:26 |
gmann | artom: the fix i mentioned will not help for test_server_sequence_suspend_resume , we need to do either create the network explicitly or add some common logic in scenario manager also like we do in API test | 15:27 |
lyarwood | I'm supposed to be running the meeting at the top of the hour but I've ended up home alone with my 10 week old who is wide awake and causing chaos, would anyone mind stepping in and running it this week? | 15:28 |
gmann | this query -http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22HTTP%20exception%20thrown:%20Multiple%20possible%20networks%20found,%20use%20a%20Network%20ID%20to%20be%20more%20specific.%5C%22%20AND%20NOT%20message:%5C%22AttachInterfacesTestJSON%5C%22%20AND%20tags:%5C%22screen-n-api.txt%5C%22&from=7d | 15:29 |
*** k_mouza has quit IRC | 15:30 | |
*** vishalmanchanda has quit IRC | 15:32 | |
*** admin0 has joined #openstack-nova | 15:32 | |
admin0 | when i run nova-manage --debug db sync .. it says ERROR: Could not access cell0. .. i check the username and password and the user (nova_api) can acess nova_cell0 database .. what could be wrong ? | 15:32 |
admin0 | or does nova_api need access to nova database also ? | 15:34 |
sean-k-mooney | this is the query i came up with http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Multiple%20possible%20networks%20found,%20use%20a%20Network%5C%22%20AND%20NOT%20message:%5C%22AttachInterfacesTestJSON%5C%22%20AND%20NOT%20message:%5C%22tempest-test_creds%5C%22%20%20AND%20NOT%20build_status:%5C%22SUCCESS%5C%22&from=30d | 15:34 |
*** swp20 has joined #openstack-nova | 15:35 | |
*** songwenping_ has quit IRC | 15:36 | |
sean-k-mooney | admin0: nova manage gets the db creds from the nova.conf | 15:36 |
admin0 | i have validated it and it is correct | 15:36 |
admin0 | is there a way to debug what actually it is using | 15:37 |
gmann | yeah, AttachInterfacesTestJSON is most in failing | 15:37 |
admin0 | --debug does not seem to produce that much of a log | 15:37 |
sean-k-mooney | gmann: right but none of those builds failed | 15:37 |
lyarwood | AttributeError: module 'libvirt' has no attribute 'VIR_ERR_DEVICE_MISSING' - whops, my bad. | 15:38 |
sean-k-mooney | well wehn the failed that was nto the issue | 15:38 |
gmann | yeah | 15:38 |
admin0 | sean-k-mooney, i did a grep mysql /etc/nova.conf .. then installed mysql-client .. and then when i do mysql -h (hostname as in the conf) -u username -p ( copy paste of the password) .. it works just fine i can login and select the database | 15:39 |
gmann | artom: sean-k-mooney do you have link where test_server_sequence_suspend_resume failing ? | 15:39 |
admin0 | i did a create table and drop table ( the same test table) to validate that the login is rw | 15:40 |
sean-k-mooney | admin0: odd im not certin what the issue is off the top of my head | 15:41 |
admin0 | is there a way to get more verbose/debug steps in nova-manage db sync ? | 15:41 |
*** belmoreira has quit IRC | 15:42 | |
*** sangeet has joined #openstack-nova | 15:42 | |
admin0 | nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0 worked fine .. only in db sync its giving issues | 15:44 |
*** priteau has joined #openstack-nova | 15:47 | |
artom | gmann, https://zuul.opendev.org/t/openstack/build/0c87381c639e4d629fa2bd48ea2da4f2 | 15:48 |
gmann | artom: thanks, i can try to build logic of creating unique network in scenario test too but may be during weekend or tomorrow. | 15:49 |
gmann | failing scenario tests on this should be less-frequent than API tests or rare | 15:50 |
*** mgoddard has quit IRC | 15:54 | |
lyarwood | I'm supposed to be running the meeting at the top of the hour but I've ended up home alone with my 10 week old who is wide awake and causing chaos, would anyone mind stepping in and running it this week? | 15:55 |
lyarwood | ^ can anyone take this? | 15:55 |
lyarwood | or we skip this week | 15:56 |
gmann | lyarwood: sorry, i will be in internal meeting at the same time, even would not be able to join too | 15:58 |
lyarwood | or mum is home, I can run through the agenda quickly | 15:59 |
lyarwood | #openstack-meeting-3 for a very quick meeting if people have anything | 16:00 |
*** mgoddard has joined #openstack-nova | 16:01 | |
*** jsuchome has quit IRC | 16:10 | |
sean-k-mooney | very efficent of you | 16:10 |
sean-k-mooney | lyarwood: ^ | 16:11 |
sean-k-mooney | you should run all meetings :) | 16:11 |
lyarwood | haha I didn't prep and just rushed through everything, 10/10. | 16:11 |
lyarwood | I'll sort the agenda out ahead of time next week while gibi_pto_24th is still out | 16:11 |
lyarwood | apologies for the mess today | 16:12 |
tosky | I just wanted thanks for the Zuul v3 effort and please keep going but you were faster :'( | 16:14 |
*** mriedem has joined #openstack-nova | 16:14 | |
*** songwenping_ has joined #openstack-nova | 16:18 | |
lyarwood | tosky: hehe sorry and thanks, will keep pushing until M3, have the evacuate stuff worked out locally, just need to test and push. | 16:21 |
*** swp20 has quit IRC | 16:21 | |
*** mkrai_ has quit IRC | 16:22 | |
*** psachin has quit IRC | 16:25 | |
*** tesseract has quit IRC | 16:30 | |
*** songwenping_ has quit IRC | 16:31 | |
*** songwenping_ has joined #openstack-nova | 16:31 | |
*** gyee has joined #openstack-nova | 16:40 | |
*** ociuhandu_ has quit IRC | 16:43 | |
*** admin0 has left #openstack-nova | 16:43 | |
*** tosky has quit IRC | 16:43 | |
*** ociuhandu has joined #openstack-nova | 16:44 | |
*** songwenping__ has joined #openstack-nova | 16:48 | |
*** ociuhandu has quit IRC | 16:50 | |
*** songwenping_ has quit IRC | 16:51 | |
*** jangutter has joined #openstack-nova | 16:54 | |
*** ociuhandu has joined #openstack-nova | 16:57 | |
*** ociuhandu has quit IRC | 17:04 | |
sangeet | I need help. I am doing "nova-manege db sync" and it faile with error "ERROR: Could not access cell0.". "nova-manage cell_v2 map_cell0" "nova-manage api_db sync" were run successfully before. I have ssl enabled in my deployment. The issue seems that _url_cfg.connection in _TransactionFactory object is somehow not corect for nova_cell0. Hence the DB is rejection the connection. Please see http://paste.openstack.org/show/796820/. | 17:17 |
sangeet | But connection in [cell0_database] in nova.conf is set correctly. Where is _TransactionFactory object reading the connection value form? | 17:17 |
sean-k-mooney | sangeet: your the second person to ask abou this in 2 hours :) | 17:19 |
sangeet | Interesting | 17:19 |
sangeet | sean-k-mooney was it answered? Sorry I can not scroll up for previous chats. | 17:20 |
sean-k-mooney | am no | 17:20 |
sean-k-mooney | same bhavior "nova-manage --debug api_db sync and nova-manage --debug cell_v2 map_cell0" bot worked by sync failed | 17:20 |
sangeet | :-( | 17:21 |
sean-k-mooney | i think if you want to run sync on cell0 | 17:21 |
sean-k-mooney | you need to create a second config adn set teh database connection to point to cell0 | 17:22 |
sean-k-mooney | at least i think you had to do that in the past e.g. nova-manage did not read cell0_database | 17:22 |
sean-k-mooney | but i could be miss remembering | 17:22 |
sangeet | That is waht I have set .. please look at http://paste.openstack.org/show/796820/. | 17:23 |
sean-k-mooney | let me compare to my devstack | 17:23 |
sean-k-mooney | so have | 17:24 |
sean-k-mooney | [database] | 17:24 |
sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | 17:24 |
sean-k-mooney | [api_database] | 17:24 |
sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_api?charset=utf8 | 17:24 |
sean-k-mooney | so my db is point to cell0 in the conotler config at least | 17:25 |
sean-k-mooney | my cell1 contoller then point to cell1 | 17:26 |
sean-k-mooney | [database] | 17:26 |
sean-k-mooney | connection = mysql+pymysql://root:password@127.0.0.1/nova_cell1?charset=utf8 | 17:26 |
sangeet | sean-k-mooney - Please not .. if I do a new openstack deployment with ssl it is working fine. Its only when I am doing an update from no ssl to ssl, I see the issue. Somehow url_cfg.connection in _TransactionFactory object is taking old connection | 17:26 |
sangeet | for nova_cell0 only | 17:27 |
sean-k-mooney | nova_cell0 is special | 17:27 |
sean-k-mooney | its not a real cell | 17:27 |
sean-k-mooney | it has a special meanin in the code | 17:27 |
sean-k-mooney | i dont know what [cell0_database] is | 17:28 |
sangeet | nova.conf has correct values as well ... so just can't figure out how the connection is still old link | 17:28 |
sean-k-mooney | i have never seen the "[cell0_database]" config section before | 17:29 |
sean-k-mooney | and its not in our config docs | 17:29 |
*** links has quit IRC | 17:30 | |
sean-k-mooney | sangeet: did you deploy with openstack helm? | 17:33 |
sangeet | yes | 17:33 |
sean-k-mooney | then its a helm bug i think | 17:33 |
sean-k-mooney | https://opendev.org/openstack/openstack-helm/src/branch/master/nova/values.yaml#L1724 | 17:34 |
sean-k-mooney | it is the only thing that creats that config section | 17:34 |
sean-k-mooney | i cant see anything in any other tool that does and i dont see any code that uses it in nova | 17:34 |
sean-k-mooney | there are also not docs related to it | 17:34 |
sean-k-mooney | based on http://codesearch.openstack.org/?q=cell0_database&i=nope&files=&repos= | 17:34 |
sangeet | Its ok ... we can ignore [cell0_database]. Even [database] has correct value of connection. | 17:36 |
*** priteau has quit IRC | 17:36 | |
sean-k-mooney | it does but cell0_database might be breaking nova manage somehow | 17:37 |
*** priteau has joined #openstack-nova | 17:45 | |
sean-k-mooney | sangeet: can you try adding --local_cell | 17:45 |
sean-k-mooney | i think i know why its failing | 17:47 |
sean-k-mooney | as i said that config option is not valid. no code in nova exits to use it | 17:48 |
sean-k-mooney | wehn you dont pass --loca_cell we lookup the cell0 from the db | 17:48 |
sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L188-L189 | 17:48 |
sean-k-mooney | so unless you have updated the cell0 database connection string in the cell maping table it will use the old value | 17:49 |
sean-k-mooney | sangeet: if you pass --local_cell it will skip that lookup although it wont update the schem of cell0 | 17:50 |
sean-k-mooney | you will have to do that manually | 17:50 |
sangeet | It is automated but let me try manually | 17:50 |
sean-k-mooney | but ya in anycae openstack helm has a bug | 17:50 |
sangeet | you are saying run nova-manabe db sync --local_cell? | 17:50 |
sean-k-mooney | yes | 17:51 |
sangeet | Let me try that real quick | 17:51 |
sean-k-mooney | openstack helm is not generting correct local.confs and as a result the automated db sync will fail if you are going form non ssl to ssl | 17:52 |
sean-k-mooney | since it will use the non ssl connection string from the db | 17:52 |
sangeet | awesome ... that worked. Can you please explain what --local_cell opetion does | 17:52 |
*** priteau has quit IRC | 17:53 | |
sean-k-mooney | it tells nova manage not to lookup the db connection string form nova_api.cell_mappings db table | 17:53 |
sangeet | Interestinfg | 17:53 |
sean-k-mooney | you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell | 17:54 |
sean-k-mooney | sangeet: alternitvily you can just update teh connection string in the cell mappign table in the nova_api db | 17:54 |
sean-k-mooney | in my devstack deployment its just mysql+pymysql://root:password@127.0.0.1/nova_cell0?charset=utf8 | 17:54 |
sean-k-mooney | but you would need to add the ssl parmater to your version | 17:54 |
sangeet | Let me see what I have in the my database | 17:55 |
sean-k-mooney | mysql -e "select database_connection from nova_api.cell_mappings where name='cell0';" | 17:56 |
sean-k-mooney | by the way you should not really paste the result publicy | 17:56 |
sean-k-mooney | since it has the password in it... | 17:56 |
sean-k-mooney | anyway it does not have the ssl cert info right | 17:57 |
sangeet | You are correct .. it is without ssl | 17:57 |
*** songwenping_ has joined #openstack-nova | 17:57 | |
sean-k-mooney | yep so you could just update it and then it would work | 17:57 |
sangeet | going back to useing --local-cell, do you see any issue with it? Also when you say "you need to run nova-manage db sync on one contoler on each cell wehn you use --local-cell" | 17:59 |
sean-k-mooney | by should work i mean nova manage shoudl work without --local-cell and as a result the automated update should work too | 17:59 |
sangeet | does it mean I have to run on each node | 17:59 |
sean-k-mooney | no so each cell has its own db right | 17:59 |
sangeet | yes | 18:00 |
sean-k-mooney | so each cell db needs to be updated | 18:00 |
sean-k-mooney | wehn you use --local-cell it only update well the locall cell db | 18:00 |
sean-k-mooney | not all deployment have db access between cells | 18:00 |
sangeet | will "nova-manage db sync --local-cell" not update all cells? | 18:00 |
*** songwenping__ has quit IRC | 18:01 | |
sean-k-mooney | correct it only updates the cell pointed to in the database section | 18:01 |
sangeet | but "nova-manage db sync " does all? | 18:01 |
sean-k-mooney | actully no | 18:01 |
sean-k-mooney | it does the local cell and cell0 | 18:01 |
sean-k-mooney | you need to run nova-manage in each cell anyway | 18:02 |
sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L185-L186 | 18:02 |
sangeet | So it mean I run the command with our without --local-cell, it will update same cells? | 18:03 |
sean-k-mooney | no the default behavor is to update the cell in [database]/connection and cell0 | 18:04 |
sean-k-mooney | so if you run the command on the host with the conducto for cell1 it will update cell1 and cell0 | 18:04 |
sean-k-mooney | ran on the host with the conductor for cell2 it will update cell2 and cell0 | 18:05 |
sean-k-mooney | if you add --local-cell it will skip updating cell0 | 18:05 |
sean-k-mooney | unless [database]/connection point to cell0 | 18:05 |
sean-k-mooney | basically the nova.conf used by the nova api should have [database]/connection pointing at cell0, each conductors nova.conf should have [database]/connection pointing at the cell db | 18:06 |
sean-k-mooney | and the compute services should not have the database details defiend at all | 18:07 |
sean-k-mooney | openstack helm shoudl then be automatinn runing the db sync once for each cell db using the correct cofigs | 18:08 |
sean-k-mooney | and the simplet way to do that is to delegate runing nova-magange db sync to the first conductor host in each cell | 18:08 |
sean-k-mooney | you can safly run it on each conductor host | 18:09 |
sean-k-mooney | as it is not destructive its just wasteful to do it | 18:09 |
openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/746210 | 18:09 |
openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/746210 | 18:12 |
*** songwenping__ has joined #openstack-nova | 18:13 | |
*** songwenping_ has quit IRC | 18:16 | |
*** songwenping_ has joined #openstack-nova | 18:28 | |
*** songwenping__ has quit IRC | 18:32 | |
*** songwenping__ has joined #openstack-nova | 18:33 | |
*** songwenping_ has quit IRC | 18:36 | |
*** songwenping_ has joined #openstack-nova | 18:42 | |
*** songwenping_ has quit IRC | 18:43 | |
*** songwenping_ has joined #openstack-nova | 18:43 | |
*** songwenping__ has quit IRC | 18:45 | |
*** songwenping_ has quit IRC | 18:47 | |
*** maciejjozefczyk has quit IRC | 18:47 | |
*** songwenping_ has joined #openstack-nova | 18:48 | |
*** openstackgerrit has quit IRC | 19:02 | |
sangeet | sean-k-mooney .. thanks a lot for your help | 19:21 |
sean-k-mooney | sangeet: no worries hopefully if admin0 comes back and sees this it might help them too | 19:39 |
*** nweinber has quit IRC | 20:09 | |
*** tosky has joined #openstack-nova | 20:10 | |
*** dave-mccowan has joined #openstack-nova | 20:23 | |
*** suryasingh has quit IRC | 20:24 | |
*** slaweq has quit IRC | 20:37 | |
*** slaweq has joined #openstack-nova | 20:43 | |
*** slaweq has quit IRC | 20:48 | |
mnaser | does nova run ceph jobs on every commit? | 20:56 |
*** raildo has quit IRC | 20:59 | |
dansmith | yes | 21:03 |
eandersson | btw I have a PR open for cloud-init for the OpenStack Datasource to increase the retry limit when identifying the metadata service. The current defaults only allows a single failure before giving up. https://github.com/canonical/cloud-init/pull/501 | 21:29 |
eandersson | If anyone has any thoughts feel free to drop a comment. | 21:30 |
*** songwenping__ has joined #openstack-nova | 21:33 | |
*** songwenping_ has quit IRC | 21:37 | |
*** jmlowe has quit IRC | 21:42 | |
*** jmlowe has joined #openstack-nova | 21:43 | |
*** openstackgerrit has joined #openstack-nova | 21:45 | |
openstackgerrit | Merged openstack/nova master: Fix FIP policy for admin_or_owner https://review.opendev.org/741811 | 21:45 |
*** prometheanfire has joined #openstack-nova | 21:47 | |
prometheanfire | can we get more cores on https://review.opendev.org/744992 (oneliner, with comment) | 21:47 |
melwitt | prometheanfire: done | 21:58 |
*** mgoddard has quit IRC | 22:00 | |
*** rcernin has joined #openstack-nova | 22:00 | |
prometheanfire | thanks | 22:01 |
*** songwenping_ has joined #openstack-nova | 22:03 | |
*** swp20 has joined #openstack-nova | 22:05 | |
*** mgoddard has joined #openstack-nova | 22:05 | |
*** songwenping__ has quit IRC | 22:06 | |
*** songwenping_ has quit IRC | 22:07 | |
*** mriedem has left #openstack-nova | 22:16 | |
openstackgerrit | Merged openstack/nova master: Add new default roles in multinic policies https://review.opendev.org/742330 | 22:21 |
openstackgerrit | Merged openstack/nova master: Correct the check_str and pass actual target in FIP pools policy https://review.opendev.org/741797 | 22:21 |
openstackgerrit | Merged openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763 | 22:22 |
*** prometheanfire has left #openstack-nova | 22:24 | |
*** songwenping_ has joined #openstack-nova | 22:33 | |
*** songwenping__ has joined #openstack-nova | 22:36 | |
*** swp20 has quit IRC | 22:36 | |
*** songwenping_ has quit IRC | 22:40 | |
*** tkajinam has joined #openstack-nova | 22:58 | |
*** ociuhandu has joined #openstack-nova | 23:00 | |
*** ociuhandu has quit IRC | 23:05 | |
*** mlavalle has quit IRC | 23:08 | |
*** tosky has quit IRC | 23:09 | |
*** ircuser-1 has quit IRC | 23:29 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!