Tuesday, 2017-09-26

*** acormier has joined #openstack-nova00:00
*** baoli has joined #openstack-nova00:01
*** jaypipes has quit IRC00:02
mriedemsuperconductor foils me again00:04
mriedemnova-manage db archive_deleted_rows in devstack uses nova.conf by default, which is cell0 in devstack00:04
*** gbarros has quit IRC00:04
*** acormier has quit IRC00:05
*** gbarros has joined #openstack-nova00:05
*** baoli has quit IRC00:06
mriedemwe could make nova-manage db archive_deleted_rows use the api database to get the cells...00:08
*** itlinux has quit IRC00:08
*** yangyape_ has quit IRC00:09
*** yangyapeng has joined #openstack-nova00:10
*** acormier has joined #openstack-nova00:11
openstackLaunchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Medium,Triaged]00:12
*** yangyapeng has quit IRC00:14
*** acormier has quit IRC00:16
*** yamamoto has quit IRC00:16
*** acormier has joined #openstack-nova00:17
*** Swami has quit IRC00:17
*** yamamoto has joined #openstack-nova00:21
*** acormier has quit IRC00:22
openstackgerritMatt Riedemann proposed openstack/nova master: For map_instances argument destination is not defined  https://review.openstack.org/50223600:24
openstackgerritMatt Riedemann proposed openstack/nova master: Fix --max-count handling for nova-manage cell_v2 map_instances  https://review.openstack.org/50223600:26
*** crushil_ has joined #openstack-nova00:28
*** Shunli has joined #openstack-nova00:31
*** mikal_ has quit IRC00:32
*** mikal has joined #openstack-nova00:34
*** kenperkins has quit IRC00:36
*** crushil_ has quit IRC00:38
*** zhurong has joined #openstack-nova00:42
*** litao__ has joined #openstack-nova00:53
*** acormier has joined #openstack-nova00:54
*** Apoorva_ has joined #openstack-nova00:57
*** acormier has quit IRC00:58
*** Apoorva has quit IRC01:00
*** ijw has quit IRC01:02
*** Apoorva_ has quit IRC01:02
*** ijw has joined #openstack-nova01:02
*** acormier has joined #openstack-nova01:06
*** ijw has quit IRC01:07
*** acormier has quit IRC01:07
*** acormier has joined #openstack-nova01:08
*** yangyapeng has joined #openstack-nova01:12
*** phuongnh has joined #openstack-nova01:14
*** phuongnh has quit IRC01:20
openstackgerritjichenjc proposed openstack/nova master: propagate OSError to MigrationPreCheckError  https://review.openstack.org/46977901:20
*** moshele has quit IRC01:20
*** gbarros_ has joined #openstack-nova01:25
*** gbarros has quit IRC01:28
openstackgerritjichenjc proposed openstack/nova master: check query param for server groups function  https://review.openstack.org/50034701:31
*** esberglu has quit IRC01:33
*** yufei has joined #openstack-nova01:33
*** hongbin has joined #openstack-nova01:36
*** mingyu_ has joined #openstack-nova01:38
*** mingyu has quit IRC01:39
*** yangyapeng has quit IRC01:40
*** yangyapeng has joined #openstack-nova01:40
openstackgerritjichenjc proposed openstack/nova master: fix race condition of instance host  https://review.openstack.org/49445801:44
*** yufei has quit IRC01:47
*** yufei has joined #openstack-nova01:47
*** yamamoto has quit IRC01:50
*** yamamoto has joined #openstack-nova01:50
*** crushil has quit IRC01:50
*** yangyapeng has quit IRC01:50
*** yangyapeng has joined #openstack-nova01:51
*** yamamoto_ has joined #openstack-nova01:51
*** gcb has joined #openstack-nova01:52
*** yufei has quit IRC01:54
*** yufei has joined #openstack-nova01:54
*** yamamoto has quit IRC01:55
*** yufei has left #openstack-nova01:55
*** yufei has joined #openstack-nova01:55
*** yangyape_ has joined #openstack-nova01:56
*** yangyapeng has quit IRC01:56
yufei@alex_xu, could you please help take a look at this patch when you are free? a small patch which remove re-auth logic for service-users when nova call ironic.  https://review.openstack.org/#/c/502382/01:57
*** hieulq has joined #openstack-nova02:05
*** yangyape_ has quit IRC02:12
*** yangyapeng has joined #openstack-nova02:13
*** yangyapeng has quit IRC02:15
*** hieulq has quit IRC02:15
*** yangyapeng has joined #openstack-nova02:15
*** mingyu_ has quit IRC02:17
*** yangyapeng has quit IRC02:20
openstackgerritwanghao proposed openstack/nova master: Set min_disk in the image meta based on the root volume's size  https://review.openstack.org/40739702:25
*** gbarros_ has quit IRC02:26
*** OctopusZhang has joined #openstack-nova02:26
*** yufei has quit IRC02:30
*** OctopusZhang is now known as yufei02:30
*** Tom has joined #openstack-nova02:32
*** yingjun has joined #openstack-nova02:39
*** mingyu has joined #openstack-nova02:39
*** erlon has quit IRC02:46
*** hieulq has joined #openstack-nova02:48
*** thorst has quit IRC02:51
*** thorst has joined #openstack-nova02:51
*** hongbin has quit IRC02:51
*** hongbin has joined #openstack-nova02:52
*** bhagyashris has joined #openstack-nova02:54
*** Dinesh_Bhor has joined #openstack-nova02:54
*** pooja_jadhav has joined #openstack-nova02:54
*** neha_alhat has joined #openstack-nova02:54
*** neha_alhat has quit IRC02:55
*** thorst has quit IRC02:56
*** bhagyashris has quit IRC02:56
*** neha_alhat has joined #openstack-nova02:56
*** bhagyashris has joined #openstack-nova02:56
*** ps_jadhav has joined #openstack-nova02:57
*** Dinesh_Bhor has quit IRC02:57
*** jichen has joined #openstack-nova02:57
*** Dinesh_Bhor has joined #openstack-nova02:57
*** mikal has quit IRC02:59
*** pooja_jadhav has quit IRC02:59
*** yangyapeng has joined #openstack-nova03:01
*** mikal has joined #openstack-nova03:02
*** yangyape_ has joined #openstack-nova03:03
openstackgerritSteven Webster proposed openstack/nova master: Update nova network info when doing rebuild for evacuate operation  https://review.openstack.org/38285303:05
openstackgerritSteven Webster proposed openstack/nova master: Race condition between audit and migrate/resize revert  https://review.openstack.org/40099503:06
*** yangyapeng has quit IRC03:07
*** dave-mcc_ has quit IRC03:09
*** crushil has joined #openstack-nova03:14
*** crushil has quit IRC03:17
*** itlinux has joined #openstack-nova03:19
*** ps_jadhav has quit IRC03:20
*** bhagyashris has quit IRC03:20
*** itlinux has quit IRC03:22
*** neha_alhat has quit IRC03:22
*** itlinux has joined #openstack-nova03:24
*** Dinesh_Bhor has quit IRC03:24
gmannalex_xu: do you remember why limit and marker are single_param in this - https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/hypervisors.py#L2203:26
*** pooja_jadhav has joined #openstack-nova03:27
*** bhagyashris has joined #openstack-nova03:27
*** Dinesh_Bhor has joined #openstack-nova03:28
*** neha_alhat has joined #openstack-nova03:28
gmannmriedem: alex_xu: because it was added in microversion ? because we agreed on keep accepting the multi param for 'limit' and other pagination query param and let controller code to fetch the one(which is second one as default due to dict)03:29
*** acormier has quit IRC03:29
*** acormier has joined #openstack-nova03:30
*** pooja_jadhav has quit IRC03:30
*** neha_alhat has quit IRC03:30
*** Apoorva has joined #openstack-nova03:30
*** bhagyashris has quit IRC03:34
*** acormier has quit IRC03:34
*** Dinesh_Bhor has quit IRC03:34
*** vladikr has quit IRC03:35
*** vladikr has joined #openstack-nova03:35
*** itlinux has quit IRC03:35
*** gouthamr has quit IRC03:35
*** Tom has quit IRC03:37
*** itlinux has joined #openstack-nova03:38
*** acormier has joined #openstack-nova03:39
gmannmriedem: alex_xu similar for additionalProperties, it is False. i think those are correct as those query are introduced with microversion. I will keep 2.33 schema also same way to avoid any inconsistency between 2.33 and 2.5303:39
*** hongbin has quit IRC03:40
*** vladikr has quit IRC03:45
*** neha_alhat has joined #openstack-nova03:45
*** vladikr has joined #openstack-nova03:45
*** Dinesh_Bhor has joined #openstack-nova03:46
*** bhagyashris has joined #openstack-nova03:47
*** yufei has quit IRC03:47
*** pooja_jadhav has joined #openstack-nova03:49
*** avolkov has joined #openstack-nova03:51
*** thorst has joined #openstack-nova03:53
*** vladikr has quit IRC03:53
*** vladikr has joined #openstack-nova03:53
*** Tom has joined #openstack-nova03:55
*** wlfightup has joined #openstack-nova03:56
wlfightupif resize same host, the schuduler don't filter delta, only filter new flavor.  i think that it is don't right03:57
*** udesale has joined #openstack-nova03:59
*** acormier has quit IRC03:59
*** Tom has quit IRC04:00
*** itlinux has quit IRC04:00
wlfightupanyone there ?04:11
*** mtreinish has quit IRC04:24
*** vladikr has quit IRC04:25
*** cfriesen has quit IRC04:25
*** cfriesen_ has joined #openstack-nova04:25
*** psachin has joined #openstack-nova04:26
openstackgerritGhanshyam Mann proposed openstack/nova master: Implement query param schema for GET hypervisor(2.33)  https://review.openstack.org/50734404:32
gmannmriedem: alex_xu this one for 2.33-   https://review.openstack.org/50734404:33
*** mtreinish has joined #openstack-nova04:34
*** Apoorva has quit IRC04:36
*** Apoorva has joined #openstack-nova04:37
*** bhagyashri_s has joined #openstack-nova04:37
*** crushil has joined #openstack-nova04:38
*** lajoskatona has joined #openstack-nova04:38
*** bhagyashris has quit IRC04:40
*** Apoorva has quit IRC04:41
*** trungnv has joined #openstack-nova04:42
*** Apoorva has joined #openstack-nova04:45
*** gcb has quit IRC04:48
*** claudiub has joined #openstack-nova04:49
*** ratailor has joined #openstack-nova04:54
alex_xugmann: we should keep backward-compatible in 2.33?05:02
*** Apoorva has quit IRC05:02
*** Apoorva has joined #openstack-nova05:05
*** Apoorva has quit IRC05:05
*** sree has joined #openstack-nova05:06
*** Apoorva has joined #openstack-nova05:08
*** sree has quit IRC05:09
*** yangyape_ has quit IRC05:09
*** sree has joined #openstack-nova05:09
*** yangyapeng has joined #openstack-nova05:09
*** avolkov has quit IRC05:11
*** avolkov has joined #openstack-nova05:11
*** mriedem has quit IRC05:12
*** adreznec has quit IRC05:13
*** bnemec has quit IRC05:14
*** yangyapeng has quit IRC05:14
gmannalex_xu: but there was no query param before 2.33 so making those as single value and no additional property should be fine?05:15
gmannalex_xu: only issue was, we did not restrict and documented about that while doing 2.3305:15
gmannalex_xu: which can break people if someone using it with multi param and additional params  >2.33 (<2.53 as it restrict those with schema)05:16
*** Apoorva has quit IRC05:18
gmannalex_xu: i think i agree to keep those same as it is behaving currently.05:19
alex_xugmann: yea, keep the same behaving currently05:23
*** gcb has joined #openstack-nova05:24
*** crushil_ has joined #openstack-nova05:27
*** Eran_Kuris has joined #openstack-nova05:28
*** crushil has quit IRC05:29
*** felipemonteiro has joined #openstack-nova05:35
*** yamamoto_ has quit IRC05:37
*** yamamoto has joined #openstack-nova05:37
*** yufei has joined #openstack-nova05:43
*** Tom has joined #openstack-nova05:48
*** trinaths has joined #openstack-nova05:54
*** sahid has joined #openstack-nova05:56
*** armax has joined #openstack-nova05:59
*** felipemonteiro has quit IRC06:00
*** adreznec has joined #openstack-nova06:01
*** cfriesen_ has quit IRC06:02
wlfightupcan care me?06:02
*** thorst has quit IRC06:03
*** Oku_OS-away is now known as Oku_OS06:06
*** mdnadeem has joined #openstack-nova06:06
*** trinaths1 has joined #openstack-nova06:06
*** trinaths2 has joined #openstack-nova06:07
*** trinaths has quit IRC06:07
*** trinaths1 has quit IRC06:11
*** jaosorior has quit IRC06:12
*** pcaruana has joined #openstack-nova06:20
*** andreas_s has joined #openstack-nova06:21
*** zhurong has quit IRC06:22
*** chyka has quit IRC06:24
*** jaosorior has joined #openstack-nova06:35
*** moshele has joined #openstack-nova06:36
*** Oku_OS is now known as Oku_OS-away06:40
*** crushil_ has quit IRC06:45
openstackgerritMerged openstack/nova master: Live Migration sequence diagram  https://review.openstack.org/50637006:45
*** rcernin has joined #openstack-nova06:47
*** thorst has joined #openstack-nova06:59
*** Oku_OS-away is now known as Oku_OS06:59
*** sree has quit IRC07:02
*** sree has joined #openstack-nova07:02
*** karthiks has quit IRC07:03
*** karthiks has joined #openstack-nova07:03
*** Atom1234 has joined #openstack-nova07:05
*** sree has quit IRC07:07
*** belmoreira has joined #openstack-nova07:11
*** dtantsur|afk is now known as dtantsur07:11
*** Tom has quit IRC07:13
*** tesseract has joined #openstack-nova07:14
*** Tom has joined #openstack-nova07:15
*** Tom has quit IRC07:15
*** Tom has joined #openstack-nova07:16
*** belmoreira has quit IRC07:19
openstackgerritAlex Xu proposed openstack/nova-specs master: Add trait support in the allocation candidates API  https://review.openstack.org/49771307:19
*** wlfightup has quit IRC07:20
*** wlfightup has joined #openstack-nova07:21
*** ratailor_ has joined #openstack-nova07:24
*** belmoreira has joined #openstack-nova07:25
*** jangutter has quit IRC07:25
ratailoranybody expert in openstackclient, I want to know why the default version is taken as minimum api microversion 2.1, whereas in novaclient it is maximum the client can support. Is there any design consensus about it.07:26
*** ragiman has joined #openstack-nova07:28
*** armax has quit IRC07:29
*** tssurya has joined #openstack-nova07:29
*** belmoreira has quit IRC07:29
*** ratailor_ has quit IRC07:29
ratailoramotoki, ^^07:31
*** phuongnh has joined #openstack-nova07:39
bauzasgood morning Novaz07:40
*** sree has joined #openstack-nova07:41
gibibauzas: good morning07:42
*** ratailor is now known as ratailor|Lunch07:43
* bauzas has a lovely talk to prepare for telling about Pike and the PTG07:44
*** belmoreira has joined #openstack-nova07:44
bauzas3 days, 20 slides07:45
bauzasI mean, 3 days left, 20 slides to write07:45
bauzasI can do it :)07:45
gmannbauzas: all the best :)07:45
*** sree has quit IRC07:46
gibiI feel your pain. I created a one slider for a local meetup.07:48
openstackgerritjichenjc proposed openstack/nova-specs master: Adds spec for instance live resize  https://review.openstack.org/14121907:50
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix test_get_volume_config method  https://review.openstack.org/48946707:50
*** xinliang has quit IRC07:55
bauzasgibi: that's the problem when you say to your friends 'sure, I can talk about that'.07:56
bauzasthen, you should know you're doomed for the whole week07:56
*** tiendc has joined #openstack-nova07:59
gibibauzas: I decided to do it a bit blindly and only created a single slide. I will talk about what I will remember. Bit more stress during the talk lot less stress during the week before the talk :)07:59
*** belmoreira has quit IRC07:59
*** alexchadin has joined #openstack-nova08:00
*** sree has joined #openstack-nova08:02
openstackgerritjichenjc proposed openstack/nova-specs master: Adds spec for instance live resize  https://review.openstack.org/14121908:02
*** Tom__ has joined #openstack-nova08:02
bauzasgibi: probably a good idea08:02
bauzasgibi: just some pointers and job done08:02
bauzasmore interactive08:03
bauzasgibi: you saved my day, exactly like you saved our bugs08:03
bauzasyou saved *from* our bugs :)08:04
bauzasunrelated, haven't seen yet the new zuul08:04
*** aloga has quit IRC08:04
*** aloga has joined #openstack-nova08:04
*** Tom has quit IRC08:06
*** markvoelker has quit IRC08:06
gibiat least the status page looks the same for zuul but I guess that means nothing08:08
*** alexchadin has quit IRC08:08
*** xinliang has joined #openstack-nova08:08
gibibauzas: do you have the link for melwitt's interview about what-happened-in-Pike-in-nova?08:09
gibibauzas: I can even play that video on the meetup :)08:10
bauzasgibi: I can find it08:11
bauzasI wouldn't like to be talking alot of Nova08:11
gibibauzas: here it is https://www.youtube.com/watch?v=d7vFqM3ZEdI08:11
gibibauzas: ohh you have to cover the whole OpenStack? that scary08:12
*** Atom1234 has quit IRC08:15
*** zhurong has joined #openstack-nova08:20
*** OctopusZhang has joined #openstack-nova08:20
*** yufei has quit IRC08:23
*** OctopusZhang is now known as yufei08:23
*** mvk has quit IRC08:26
*** belmoreira has joined #openstack-nova08:27
*** Shunli has quit IRC08:27
*** Shunli has joined #openstack-nova08:28
openstackgerritShan Guo proposed openstack/nova master: Remove SCREEN_LOGDIR from devstack install setting  https://review.openstack.org/50742508:28
*** belmoreira has quit IRC08:30
*** ralonsoh has joined #openstack-nova08:31
*** yamamoto has quit IRC08:31
*** alexchadin has joined #openstack-nova08:31
openstackgerritGhanshyam Mann proposed openstack/nova master: Implement query param schema for GET hypervisor(2.33)  https://review.openstack.org/50734408:33
*** yamamoto has joined #openstack-nova08:34
*** TuanLA has joined #openstack-nova08:35
*** ratailor|Lunch is now known as ratailor08:37
*** yamamoto has quit IRC08:39
openstackgerritMerged openstack/nova master: Add functional for live migrate delete  https://review.openstack.org/49958308:41
*** wlfightup has quit IRC08:42
*** wlfightup has joined #openstack-nova08:42
*** udesale has quit IRC08:45
*** yamamoto has joined #openstack-nova08:47
*** yamamoto has quit IRC08:47
openstackgerritGhanshyam Mann proposed openstack/nova master: Implement query param schema for GET hypervisor(2.33)  https://review.openstack.org/50734408:50
*** udesale has joined #openstack-nova08:51
*** mvk has joined #openstack-nova08:54
*** cdent has joined #openstack-nova08:55
*** OctopusZhang has joined #openstack-nova08:55
*** jpena|off is now known as jpena08:57
*** yufei has quit IRC08:58
*** OctopusZhang is now known as yufei08:58
*** belmoreira has joined #openstack-nova08:59
*** sree has quit IRC09:00
*** sree has joined #openstack-nova09:00
*** derekh has joined #openstack-nova09:00
*** sree has quit IRC09:05
*** moshele has quit IRC09:05
*** purplerbot has joined #openstack-nova09:06
*** moshele has joined #openstack-nova09:07
openstackgerritTakashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration  https://review.openstack.org/33473209:07
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Microversion 2.54 - Enable cold migration with target host  https://review.openstack.org/40670709:08
*** yamamoto has joined #openstack-nova09:10
*** yangyapeng has joined #openstack-nova09:10
*** udesale has quit IRC09:11
*** phuongnh has quit IRC09:11
*** udesale has joined #openstack-nova09:11
*** yamamoto has quit IRC09:12
*** yangyapeng has quit IRC09:12
*** belmoreira has quit IRC09:13
*** yangyapeng has joined #openstack-nova09:13
*** yamamoto has joined #openstack-nova09:14
*** mvk has quit IRC09:15
*** yamamoto has quit IRC09:16
*** yamamoto has joined #openstack-nova09:16
*** dtantsur is now known as dtantsur|lunch09:21
*** mvk has joined #openstack-nova09:28
*** yangyapeng has quit IRC09:30
*** wlfightup has quit IRC09:31
*** yamamoto has quit IRC09:31
*** Shunli has quit IRC09:31
*** yangyapeng has joined #openstack-nova09:31
*** wlfightup has joined #openstack-nova09:33
*** yangyapeng has quit IRC09:34
*** yangyapeng has joined #openstack-nova09:35
*** yufei has quit IRC09:35
*** yamamoto has joined #openstack-nova09:35
*** gszasz has joined #openstack-nova09:36
*** liuyulong has quit IRC09:36
*** yangyapeng has quit IRC09:39
*** yassine_ has quit IRC09:51
*** zhurong has quit IRC09:52
openstackgerritLei Zhang proposed openstack/nova-specs master: Request traits in Nova  https://review.openstack.org/46879709:53
*** takashin has left #openstack-nova09:54
*** jichen has quit IRC09:56
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova-specs master: Enable SR-IOV NIC offload feature discovery  https://review.openstack.org/50489509:57
*** sdague has joined #openstack-nova10:00
*** yingjun has quit IRC10:02
*** markvoelker has joined #openstack-nova10:07
*** ijw has joined #openstack-nova10:08
*** trinaths2 has quit IRC10:11
*** wlfightup has quit IRC10:11
*** sree has joined #openstack-nova10:11
*** wlfightup has joined #openstack-nova10:12
*** ijw has quit IRC10:13
*** armax has joined #openstack-nova10:15
*** sree has quit IRC10:16
*** sshwarts has joined #openstack-nova10:21
*** yangyapeng has joined #openstack-nova10:22
*** trinaths has joined #openstack-nova10:23
*** trinaths has quit IRC10:26
*** yangyapeng has quit IRC10:26
*** trinaths has joined #openstack-nova10:28
openstackgerritYikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated.  https://review.openstack.org/50747310:34
openstackgerritYikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated.  https://review.openstack.org/50747310:36
*** zhurong has joined #openstack-nova10:36
*** udesale has quit IRC10:37
*** markvoelker has quit IRC10:41
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova master: Change 'InstancePCIRequest' spec field  https://review.openstack.org/44925710:45
*** yufei has joined #openstack-nova10:47
*** grim-lock has joined #openstack-nova10:53
*** udesale has joined #openstack-nova10:53
openstackgerritGhanshyam Mann proposed openstack/nova master: Implement query param schema for simple_tenant_usage  https://review.openstack.org/50748010:54
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova master: Read Neutron port 'binding_profile' during boot  https://review.openstack.org/50748110:54
*** grim-lock has quit IRC10:54
*** alexchadin has quit IRC10:58
*** alexchadin has joined #openstack-nova10:59
*** Tom__ has quit IRC11:01
*** Tom_ has joined #openstack-nova11:01
openstackgerritBalazs Gibizer proposed openstack/nova master: cover migration cases with functional tests  https://review.openstack.org/49386511:02
*** cdent has quit IRC11:03
openstackgerritBalazs Gibizer proposed openstack/nova master: use context mgr in instance.delete  https://review.openstack.org/44376411:04
openstackgerritBalazs Gibizer proposed openstack/nova master: Transform missing delete notifications  https://review.openstack.org/41029711:04
openstackgerritBalazs Gibizer proposed openstack/nova master: Send soft_delete from context manager  https://review.openstack.org/47645911:04
*** sulo has left #openstack-nova11:04
*** Tom_ has quit IRC11:06
*** vladikr has joined #openstack-nova11:06
*** nicolasbock has joined #openstack-nova11:07
gibibauzas: could you make a quick look at https://review.openstack.org/#/c/504930/ ? it is a single line of test case improvement that removes an test instability.11:09
*** vladikr has quit IRC11:10
*** vladikr has joined #openstack-nova11:11
*** armax has quit IRC11:12
*** moshele has quit IRC11:12
*** liverpooler has quit IRC11:13
openstackgerritZhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware  https://review.openstack.org/50748611:14
*** alexchadin has quit IRC11:14
*** yassine has joined #openstack-nova11:15
*** vladikr has quit IRC11:18
*** ralonsoh_ has joined #openstack-nova11:21
openstackgerritMatthew Booth proposed openstack/nova master: libvirt: Don't VIR_MIGRATE_NON_SHARED_INC without migrate_disks  https://review.openstack.org/50720211:21
openstackgerritMatthew Booth proposed openstack/nova master: python3: Specify unicode explicitly calling etree.tostring()  https://review.openstack.org/50748811:21
mdboothstephenfin: You were looking at ^^^11:22
mdboothThere was a python3 problem in the first version of that patch. Turns out it's a general problem with etree.tostring() which other tests were ignoring by mocking out :/11:22
mdboothHence the prior patch to fix all etree.tostring() calls11:23
*** ralonsoh has quit IRC11:24
*** smatzek has joined #openstack-nova11:26
*** TuanLA has quit IRC11:29
*** alexchadin has joined #openstack-nova11:30
*** trinaths has quit IRC11:30
*** trinaths has joined #openstack-nova11:31
*** thorst has quit IRC11:31
mdboothgibi: Before I rebase it, any chance you could promote your +1 here to a +2: https://review.openstack.org/#/c/479802/8 ?11:33
mdboothThen I'll rebase in a bit after it merges11:33
*** zhurong has quit IRC11:35
openstackgerritMatthew Booth proposed openstack/nova master: Ensure errors_out_migration errors out migration  https://review.openstack.org/47980211:37
openstackgerritMatthew Booth proposed openstack/nova master: Use Migration object in ComputeManagerMigrationTestCase  https://review.openstack.org/50212611:37
openstackgerritMatthew Booth proposed openstack/nova master: Automatically revert resize which fails on destination  https://review.openstack.org/46252111:37
mdboothActually meh it was a clean rebase anyway. Should keep the existing +2.11:37
*** yangyapeng has joined #openstack-nova11:38
*** zhurong has joined #openstack-nova11:39
*** markvoelker has joined #openstack-nova11:39
*** acormier has joined #openstack-nova11:43
*** acormier has quit IRC11:47
*** tylerderosagrund has joined #openstack-nova11:49
*** vladikr has joined #openstack-nova11:55
openstackgerritLajos Katona proposed openstack/nova master: Extend ServerMovingTests with custom resources  https://review.openstack.org/49739911:56
*** moshele has joined #openstack-nova11:59
*** gabor_antal has joined #openstack-nova12:00
*** trinaths1 has joined #openstack-nova12:02
*** trinaths has quit IRC12:02
*** moshele has quit IRC12:03
*** jpena is now known as jpena|lunch12:03
*** tiendc has quit IRC12:04
*** trinaths has joined #openstack-nova12:04
*** moshele has joined #openstack-nova12:05
*** wlfightup has quit IRC12:05
*** trinaths1 has quit IRC12:07
*** dave-mcc_ has joined #openstack-nova12:09
*** moshele has quit IRC12:11
*** MVenesio has joined #openstack-nova12:11
*** thorst has joined #openstack-nova12:11
*** markvoelker has quit IRC12:12
openstackgerritAlex Xu proposed openstack/nova-specs master: Add trait support in the allocation candidates API  https://review.openstack.org/49771312:13
bauzasgibi: sorry was outside, but back now, +W'd12:14
*** edmondsw has joined #openstack-nova12:14
*** dtantsur|lunch is now known as dtantsur12:15
gibibauzas: thanks12:16
*** litao__ has quit IRC12:16
gibimdbooth: I'm on a meeting but opened the review so I will check it12:16
efriedratailor Did you get your questions answered about the osc microversion?12:17
*** manasm has joined #openstack-nova12:17
ratailorefried, not yet. could you help ?12:17
efriedratailor Your observation is correct, and by design.  As for the reasoning behind it... that's complicated.12:18
efriedratailor But if you want a specific microversion in osc, you can ask for it with an env var, CLI opt, or (I think) conf var.12:18
bauzasefried: if the microversion is not asking for a new attribute :p12:19
efriedYeah, there's that :)12:19
mdboothefried: OSC will happily send invalid requests because it doesn't understand those microversions, though.12:19
manasmbauzas: running into an issue with resize on an upgraded env from Ocata to Pike, the resize fails at following call in scheduler_utils.setup_instance_group method -12:20
mdboothAt that point, the user might as well just open telnet and type the http themselves :)12:20
efriedratailor A lot of it has to do with the fact, as bauzas and mdbooth are pointing out, that osc has not maintained functional parity with microversions.12:20
manasmrequest_spec.instance_group.hosts = list(group_info.hosts)12:20
*** moshele has joined #openstack-nova12:20
*** sree has joined #openstack-nova12:20
manasmbauzas: this is because the instance_group is set to None in the request_spec.12:20
manasmIs there a known issue around that?12:21
bauzasmanasm: do you have an open bug for that?12:21
* mdbooth really wishes CI would short-circuit when it fails.12:21
manasmnot yet12:21
ratailorefried, To make it as stable as possible. right ?12:21
ratailorefried, and backward compatible.12:21
efriedratailor The reasoning may not be quite that high-minded :)12:21
ratailorefried, cool12:21
mdboothThis is a data corruptor with a simple fix: https://review.openstack.org/#/c/507202/12:22
bauzasmanasm: AFAIR, we create a new InstanceGroup object when looking at the primitive dicts12:22
ratailorefried, Thanks!12:22
efriedratailor If you want more history/explanation on the osc side, find dtroyer.  I don't remember offhand which channel is the accepted one for talking about the CLI, but I know he's in #openstack-dev12:22
mdboothAffects libvirt live migration with attached volumes12:22
efriedratailor Possibly #openstack-sdks12:23
ratailorefried, sure, Thanks!12:23
*** gbarros has joined #openstack-nova12:24
efriedratailor yw, good luck.12:24
ratailorefried, actually I was searching him only, but didn't found in core openstack channels.12:24
ratailorefried, Thanks for your help!12:24
*** tylerderosagrund has quit IRC12:24
efriedIt may be a tad early for him to be on just yet, but he's at least got bouncers in those two channels.12:24
manasmbauzas: yes, we have the new InstanceGroup object but the we are trying to assign it to request_spec.instance_group which is None.12:25
bauzasefried: ratailor: FWIW, I already provided a point for the Sydney Forum about OSC vs. our CLI in https://etherpad.openstack.org/p/SYD-nova-brainstorming L3512:25
bauzasI'd love to hear operators about what they use and how12:26
ratailorbauzas, ++112:27
*** acormier has joined #openstack-nova12:27
bauzasmanasm: you said when you resize?12:28
bauzasmanasm: so we're getting the ReqSpec from the DB first12:28
bauzasif there was no group yet for that instance, then the ReqSpec.instance_group field could be None12:28
manasmyes, btw we get the object from db using - objects.InstanceGroup.get_by_instance_uuid(context, instance_uuid), and if it is None then we don't hit the issue.12:28
*** udesale has quit IRC12:29
*** armax has joined #openstack-nova12:30
*** yufei has quit IRC12:31
*** acormier has quit IRC12:31
bauzasmanasm: wait, I need to look at the code for resize12:32
bauzasdo you have a stacktrace to share?12:32
*** markvoelker has joined #openstack-nova12:32
*** pchavva has joined #openstack-nova12:34
*** hemna__ has joined #openstack-nova12:34
*** manasm has quit IRC12:36
*** trinaths has quit IRC12:36
*** armax has quit IRC12:37
*** yamamoto has quit IRC12:38
*** trinaths has joined #openstack-nova12:38
kashyapmdbooth: FWIW, looks good.  Compact summary, and a nice fix: https://review.openstack.org/#/c/507202/212:38
mdboothIt breaks some tempest tests, though :/12:39
mdboothProbably means I didn't think of something.12:40
mdboothIt does look obvious, though.12:40
stephenfinmdbooth: Reviewed both. Tidy job12:40
* mdbooth hasn't investigated the tempest failures yet, though.12:40
kashyapmdbooth: Wonder if you want to link to to the virDomainMigrateFlags page: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainMigrateFlags12:40
kashyapWhich talks about WTF actually the flag VIR_MIGRATE_NON_SHARED_INC is12:40
kashyapIf you think it's not needed, disregard me12:41
kashyapMaybe not required.  Those who want to know about it, a simple look up will fetch it12:42
kashyapBTW, the "tempest-dsvm-full-xenial" succeeded12:44
*** liverpooler has joined #openstack-nova12:45
*** zhurong has quit IRC12:46
*** jaypipes has joined #openstack-nova12:49
*** lyan has joined #openstack-nova12:49
*** artom has joined #openstack-nova12:50
*** artom has quit IRC12:51
kashyapAh, it's the "gate-grenade-dsvm-neutron-ubuntu-xenial" that failed.12:51
*** artom has joined #openstack-nova12:51
mdboothkashyap: Yep, in the block migration tests :)12:52
*** artom has quit IRC12:52
mdboothI suspect the failure there is probably related ;)12:52
*** bnemec has joined #openstack-nova12:52
mdboothkashyap: If you have time to investigate those failures, btw, I have to do something else right now. Otherwise I'll get to them later.12:52
*** artom has joined #openstack-nova12:52
kashyapmdbooth: I'm looking at the log - http://logs.openstack.org/02/507202/2/check/gate-tempest-dsvm-py35-ubuntu-xenial/8485b63/console.html12:52
kashyapI'll note on the review if I learn something new from the log12:53
openstackgerritYikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated.  https://review.openstack.org/50747312:53
* kashyap above meant: 'gate-grenade-dsvm-neutron-multinode-live-migration-nv'12:54
*** gbarros has quit IRC12:54
kashyapSo the right log is - http://logs.openstack.org/02/507202/2/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/b210147/console.html12:54
*** yamamoto has joined #openstack-nova12:55
openstackgerritkonstantin proposed openstack/nova master: switch from filesystem to disk for parallels containers  https://review.openstack.org/50668712:55
openstackgerritkonstantin proposed openstack/nova master: don't add device address if there is no any units  https://review.openstack.org/50668612:55
*** bhagyashri_s has quit IRC12:55
*** pooja_jadhav has quit IRC12:56
*** neha_alhat has quit IRC12:56
openstackgerritYikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated.  https://review.openstack.org/50747312:57
*** erlon has joined #openstack-nova12:57
*** esberglu has joined #openstack-nova12:57
*** sree has quit IRC12:58
*** sree has joined #openstack-nova12:58
*** sree has quit IRC12:59
*** sree has joined #openstack-nova12:59
openstackgerritEric Fried proposed openstack/nova master: nova.utils.get_ksa_adapter()  https://review.openstack.org/48813712:59
*** Dinesh_Bhor has quit IRC12:59
*** sree has quit IRC13:00
openstackgerritEric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests  https://review.openstack.org/49224713:00
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/ocata: Updated from global requirements  https://review.openstack.org/49025613:00
*** sree has joined #openstack-nova13:00
*** jpena|lunch is now known as jpena13:01
*** alexchadin has quit IRC13:01
openstackgerritEric Fried proposed openstack/nova master: Get auth from context for glance endpoint  https://review.openstack.org/49005713:02
openstackgerritEric Fried proposed openstack/nova master: Get auth from context for glance endpoint  https://review.openstack.org/49005713:02
openstackgerritSean Dague proposed openstack/nova master: Break out BasicTestCase  https://review.openstack.org/50725313:06
openstackgerritSean Dague proposed openstack/nova master: Don't use mock.patch.stopall  https://review.openstack.org/50752713:06
openstackgerritSean Dague proposed openstack/nova master: Remove REQUIRES_LOCKING as nothing needs process locking in the tests  https://review.openstack.org/50752813:06
openstackgerritSean Dague proposed openstack/nova master: WIP: demonstrate no use of external locking  https://review.openstack.org/50752913:06
*** pooja_jadhav has joined #openstack-nova13:06
*** yamamoto has quit IRC13:06
*** bhagyashris has joined #openstack-nova13:07
*** neha_alhat has joined #openstack-nova13:07
*** lucasxu has joined #openstack-nova13:07
openstackgerritEric Fried proposed openstack/nova master: Don't fix protocol-less glance api_servers anymore  https://review.openstack.org/50531713:08
*** Dinesh_Bhor has joined #openstack-nova13:08
*** ratailor has quit IRC13:10
*** yingjun has joined #openstack-nova13:11
*** mriedem has joined #openstack-nova13:11
openstackgerritYikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated.  https://review.openstack.org/50747313:12
sdaguegibi: so, interesting fact from this morning, I'm pretty convinced we don't need any of the REQUIRES_LOCKING code13:16
*** gbarros has joined #openstack-nova13:16
sdaguegibi: also, if you are able to take a look at the qemu 2.10 support patch, that would be cool - https://review.openstack.org/#/c/505673/13:20
sdagueor bauzas13:20
*** shaner has quit IRC13:21
*** baoli has joined #openstack-nova13:23
*** smatzek has quit IRC13:27
*** udesale has joined #openstack-nova13:27
*** cleong has joined #openstack-nova13:29
*** jangutter has joined #openstack-nova13:32
*** felipemonteiro has joined #openstack-nova13:32
*** avolkov has quit IRC13:33
*** felipemonteiro_ has joined #openstack-nova13:34
*** manasm has joined #openstack-nova13:35
mdboothstephenfin: Eurgh. So it seems that in python2.7 libvirt binding's migrateToURI3() barfs if destination_xml is unicode.13:36
*** felipemonteiro has quit IRC13:38
mdboothstephenfin:  think I'm going to leave that etree.tostring() patch always producing unicode, but I'm going to add a 2/3 hack next to the migrateToURI3 which handles the difference, because IMHO the libvirt bindings should accept a unicode string there, and it's just saner.13:38
kashyapMatt, hmm, the assertThat() mismatch seems to be related to ID:13:39
kashyap  2017-09-26 12:26:17.199293 | 2017-09-26 12:26:17.198 |     reference = u'ubuntu-xenial-2-node-rax-ord-11137317-923993'13:39
kashyap  2017-09-26 12:26:17.200739 | 2017-09-26 12:26:17.200 |     actual    = u'ubuntu-xenial-2-node-rax-ord-11137317'13:39
mdboothkashyap: Yes, that's just asserting that the migration happened13:39
mdboothi.e. assert id == the destination13:39
*** moshele has quit IRC13:39
mdboothI hate python313:40
mdboothAnd python213:40
mdboothBut the 2->3 split especially13:40
kashyapAh, I see.  (about the assert ID)13:40
*** armax has joined #openstack-nova13:41
*** burt has joined #openstack-nova13:44
*** yingjun has quit IRC13:45
*** avolkov has joined #openstack-nova13:46
openstackgerritEd Leafe proposed openstack/nova master: Add Selection objects  https://review.openstack.org/49923913:49
sdaguemriedem: I was diving through thinking more about https://review.openstack.org/#/c/507239/ last night, why do you think that external locking is required there? Because it should blow up if not provided but is needed13:51
mriedemsdague: i thought that also locked those tests to run serially13:52
mriedemmaybe i should be using https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/fixture/lockutils.py#L2213:52
mriedem^ is actually what i started using13:52
sdaguemriedem: serial to what?13:52
openstackgerritkonstantin proposed openstack/nova master: switch from filesystem to disk for parallels containers  https://review.openstack.org/50668713:52
openstackgerritkonstantin proposed openstack/nova master: don't add device address if there is no any units  https://review.openstack.org/50668613:52
*** mnestratov has joined #openstack-nova13:53
*** cfriesen_ has joined #openstack-nova13:53
mriedemsdague: to other tests13:53
mriedemlike, things that would be mocking those nova.rpc globals13:53
*** tidwellr has joined #openstack-nova13:53
sdagueall the tests in a worker run serially13:53
*** eantyshev has joined #openstack-nova13:53
sdagueand the tests between workers are in completely different processes13:53
sdaguewith no shared state13:54
eantyshevmikal: Hello! Regarding your review https://review.openstack.org/#/c/49232513:54
*** trinaths has left #openstack-nova13:54
sdagueThe reason we had REQUIRES_LOCKING at all was because oslo required a directory name or it exploded13:54
sdaguebut we feed those all temp directories anyway, they never cross lock between workers13:55
*** belmoreira has joined #openstack-nova13:55
eantyshevit fails on parallels virt_type, and I'd like to update it for you, don't you mind?13:56
sdaguebut I actually don't think that default behavior holds any more, and we can probably fully delete that variable anyway, as it definitely confuses people as to what it does13:56
sdaguemriedem: https://review.openstack.org/#/c/507253/ - I was experimenting this morning13:56
*** acormier has joined #openstack-nova13:57
*** eharney has joined #openstack-nova13:58
*** armax has quit IRC13:58
mriedemavolkov: you might like to take a crack at this https://bugs.launchpad.net/nova/+bug/171946013:59
openstackLaunchpad bug 1719460 in OpenStack Compute (nova) "(perf) Unnecessarily joining instance.services when listing instances regardless of microversion" [Medium,Triaged]13:59
mriedemshould be pretty simple13:59
*** awaugama has joined #openstack-nova14:01
*** shaner has joined #openstack-nova14:01
*** smatzek has joined #openstack-nova14:01
*** belmoreira has quit IRC14:01
*** artom has quit IRC14:02
*** crushil has joined #openstack-nova14:02
*** artom has joined #openstack-nova14:02
dansmithmriedem: any outcome from your testing yesterday?14:03
* gibi reads back14:03
mriedemdansmith: i've got the clean slate, just getting setup to start the 2nd scenario with the 500 ACTIVE and 500 ERROR instances14:03
mriedemfor the cell0 and cell1 listing14:03
*** rmart04 has joined #openstack-nova14:03
mriedemgoing to need to do that flavor thing because otherwise it's a 60 second rpc timeout per call to select_destinations14:04
*** belmoreira has joined #openstack-nova14:04
*** rmart04 has quit IRC14:04
gibimriedem, sdague: this also means that the problems we see with the rpc tests in bug 1685333 is not beacuase of the lack of locking14:06
openstackbug 1685333 in OpenStack Compute (nova) "Fatal Python error: Cannot recover from stack overflow. - in py35 unit test job" [High,Confirmed] https://launchpad.net/bugs/168533314:06
sdaguemriedem: is there a reset on rpc variables that is needed that's not happening?14:06
mriedemsdague: the TestRPC class does a reset per test method14:07
*** yamamoto has joined #openstack-nova14:07
*** armax has joined #openstack-nova14:07
sdaguegibi: I don't see how it could be. It might be a deadlock14:07
mriedemdansmith: also came across this last night https://bugs.launchpad.net/nova/+bug/171948714:07
openstackLaunchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu)14:07
sdaguethe biggest issue though is it doesn't have the timeout bits in place, so it's hard to see what's going on14:07
sdagueI think if we trigger the timeout we get a stack trace14:08
gibisdague: that would be nice14:08
dansmithmriedem: meh14:08
*** rmart04 has joined #openstack-nova14:08
*** Tom__ has joined #openstack-nova14:08
mriedemit's wishlist, sure14:08
gibisdague, mriedem: we are at the start of the cycle so I'm brave enough to try to remove the whole locking code and see what happens14:09
mriedemdansmith: oh yeah, also came across this last night https://review.openstack.org/#/c/502236/14:09
*** rmart04 has quit IRC14:09
dansmithack yeah14:11
mriedemgibi: you can be brave locally to start :)14:11
sdaguegibi: yeh, well we should at least get the test_rpc under timeout control, regardless of the rest of it14:11
*** Tom__ has quit IRC14:13
gibimriedem: I can definitly do that14:13
avolkovmriedem: ack14:13
openstackgerritMatt Riedemann proposed openstack/nova master: Make TestRPC inherit from the base nova TestCase  https://review.openstack.org/50723914:13
mriedem^ removes the lock thing14:14
mriedemso we'll have the timeout fixture14:14
*** yamamoto has quit IRC14:14
gibisdague: agree. I found one more place where we use the testtools.TestCase directly. I left a comment in your review14:15
sdaguegibi: in the fixtures?14:15
gibisdague: here https://github.com/openstack/nova/blob/62c4535a85f7d37f1c9da1e8a747f25ec63dc785/nova/tests/unit/api/openstack/test_requestlog.py#L3814:16
sdagueah, cool, good catch14:16
mriedemi thought ^ was intentional14:16
mriedemfor the placement split or something14:16
gibisdague: I think fixtures are OK to derive from testtools.TestCase as we use fixtures like mixins14:17
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Fix --max-count handling for nova-manage cell_v2 map_instances  https://review.openstack.org/50755214:17
sdaguegibi: yeh, some of the more advanced ones should see the timeout14:17
*** baoli has quit IRC14:17
sdaguebut I think that's follow on14:17
sdaguemriedem: it's a good question14:17
*** baoli has joined #openstack-nova14:17
gibisdague, mriedem: at least this request_log test should be also under timeout control14:18
sdaguegibi: so, I'd actually rather handle nova/tests/unit/api/openstack/test_requestlog.py as follow on, because those actually do most of the fixture setup (except the timeout one) manually14:18
sdagueso it's going to be a bit more extensive change there14:18
sdagueI do agree that we should get that under timeout control14:18
sdaguebut test_rpc is failing a lot now14:19
gibisdague: I'm OK with that approach. Then I'm +2 on your patch introducing BasicTestCase14:19
openstackgerritMatt Riedemann proposed openstack/nova stable/ocata: Fix --max-count handling for nova-manage cell_v2 map_instances  https://review.openstack.org/50755614:19
openstackgerritMatt Riedemann proposed openstack/nova stable/newton: Fix --max-count handling for nova-manage cell_v2 map_instances  https://review.openstack.org/50755714:20
*** rmart04 has joined #openstack-nova14:20
*** rmart04 has quit IRC14:20
manasmbauzas: here is the exception I saw with the resize -2017-09-22 07:49:16.377 13573 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 567, in setup_instance_group14:20
manasm2017-09-22 07:49:16.377 13573 ERROR nova.api.openstack.extensions     request_spec.instance_group.hosts = list(group_info.hosts)14:20
manasm2017-09-22 07:49:16.377 13573 ERROR nova.api.openstack.extensions14:20
manasm2017-09-22 07:49:16.377 13573 ERROR nova.api.openstack.extensions AttributeError: 'NoneType' object has no attribute 'hosts'14:21
manasm2017-09-22 07:49:16.377 13573 ERROR nova.api.openstack.extensions14:21
jaypipesmriedem, dansmith, gibi, sdague, bauzas: any of you noticed weird glitches in the new Gerrit web UI where the screen blinks and flashes when you open up long in-page comments?14:21
*** eantyshev has left #openstack-nova14:21
gibiat least not yet14:21
jaypipesit's a good thing I don't have Tourettes.14:21
jaypipesor epilepsy I gues14:21
*** felipemonteiro_ has quit IRC14:22
jaypipesefried: around? want to chat about "trait inheritance"...14:22
efriedjaypipes I thought you'd never ask :*14:23
efriedjaypipes I have also experienced the gerrit UI glitchiness.14:23
jaypipesefried: oh, good (or bad...) at least I'm not the only one14:23
efriedSo yeah, trait inheritance...14:23
efriedDid you see my long-winded comment with example based on (or at least attributed to) your response to my response etc. etc.?14:24
jaypipesefried: yeah, so it's absolutely correct that whatever is constructing the provider tree will need to attach traits at the appropriate provider leel14:24
efriedYuh.  And the spec (ultimately the docs) will need to dictate what level(s) is/are "appropriate".14:24
*** sree has quit IRC14:24
efriedBecause the code is gonna hafta do some work to percolate 'em around, if that's supported.14:25
jaypipesefried: no, there's no percolating around...14:25
sdaguejaypipes: url?14:26
efriedjaypipes sdague Talking about this 'un: https://review.openstack.org/#/c/497713/6/specs/queens/approved/add-trait-support-in-allocation-candidates.rst@4214:26
jaypipessdague: are you talking about the gerrit thing or the nested providers thing? :)14:27
efried(oh, sdague unless you were... yeah...)14:27
sdaguejaypipes: gerrit thing14:27
jaypipessdague: mostly seen it happen on specs with long (>8 replies) inline comment "threads"14:27
jaypipessdague: next time it happens I'll ping you a link14:27
efriedFor me, the gerrit thing is intermittent, happens when I'm expanding comments on a long page with lots of comments14:27
sdaguegerrit sends back a lot of ajax calls to get all those bits14:28
efriedBut not reproducible, cause I pop up to the review and back down and do the same thing and it doesn't happen the second time.14:28
sdagueif it's gone slow, or your connection is weird, it might take a while for them to pile in and render14:28
efriedI don't think it's ajax.  Seems like client-side js focus() calls.14:28
*** belmoreira has quit IRC14:28
jaypipessdague: nah, it's more like a loop in the UI that happens.14:28
jaypipessdague: ya, what efried said :)14:28
sdaguejaypipes: well, web console in chrome might help explain things14:29
jaypipeslike it can't decide which comment to align to the top of the screen canvas14:29
*** zhouyaguo has quit IRC14:29
jaypipessdague: when it happens again I'll ping ya14:29
efriedI noticed focus bugs before the upgrade too, usually when composing a comment on a long page, it would jump around (shoving my comment box off the visible screen)14:29
jaypipesefried: yeah, that's happened for a long time14:29
sdaguenote, we also inject a lot of our own custom client side js to do the CI rollup, so it's entirely possible that is related to the issue14:30
*** lucasxu has quit IRC14:30
sdagueregardless seeing if you can get an inspect console on the issue would be handy14:31
*** Eran_Kuris has quit IRC14:32
jaypipessdague: will do14:32
jaypipesefried: you want me to response here or on the review?14:32
efriedjaypipes Okay, so in the example in the comment I linked above: does that work as stated?14:32
*** tetsuro has joined #openstack-nova14:33
efriedjaypipes Wouldn't mind discussing a bit, then putting a summary in the review.  Just to make sure we're talking about the same thing, save time.14:33
jaypipesefried: your statement there, right after "How about" is spot on.14:33
jaypipesefried: this is excellent:14:34
jaypipes"With nested resource providers, traits defined on a parent RP are assumed to belong to all its child (descendant) RPs. However, traits defined on a child RP do not apply to the parent (ancestor) RPs. There is no implied sharing of traits within aggregates."14:34
jaypipesefried: even more explicit would be pointing out that aggregates don't actually have *any* traits associated to themselves at all (there's no aggregate_metadata table like there is in Nova)14:34
jaypipesefried: only resource providers have traits associated with them.14:34
efriedRightright, point being that RP1 doesn't inherit any traits from RP2 just because they're in the same aggregate.14:35
jaypipesefried: correct. it's worth spelling that out. aggregates are only grouping mechanisms, nothing more.14:35
efriedjaypipes Okay, cool.  So traits are inherited in NRPs, downwards but not upwards.  And the example below that sentence would work as described.  I guess the implementation details aren't important, but I'm a bit curious how it would work if you didn't actually internally copy the traits from the parent to its children.14:36
*** coreywright has quit IRC14:37
jaypipesefried: don't worry about the implementation details of the queries at this point.14:37
efriedjaypipes Roger that.  So okay, it sounds like we're in agreement.  Thanks for the talk.14:38
jaypipesefried: just typing up in the review... gimme a few14:39
*** ratailor has joined #openstack-nova14:43
jaypipesalex_xu: not sure if you're still up, but if you can make the few requested changes on https://review.openstack.org/#/c/497713/ from me and efried, I'm ready to +2 that one.14:43
ratailorsdague, you around ?14:43
openstackgerritMatthew Booth proposed openstack/nova master: Make etree.tostring() emit unicode everywhere  https://review.openstack.org/50748814:44
openstackgerritMatthew Booth proposed openstack/nova master: libvirt: Don't VIR_MIGRATE_NON_SHARED_INC without migrate_disks  https://review.openstack.org/50720214:44
ratailorsdague could you have a look at https://review.openstack.org/#/c/504885/14:44
*** lbragstad has quit IRC14:44
*** cdent has joined #openstack-nova14:45
openstackgerritBalazs Gibizer proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase  https://review.openstack.org/49953914:46
openstackgerritBalazs Gibizer proposed openstack/nova master: factor out compute service start in ServerMovingTest  https://review.openstack.org/50303714:46
openstackgerritBalazs Gibizer proposed openstack/nova master: Test resource allocation during soft delete  https://review.openstack.org/49515914:46
*** moshele has joined #openstack-nova14:46
*** mdnadeem has quit IRC14:46
openstackgerritMoshe Levi proposed openstack/nova master: Don't overwrite binding-profile  https://review.openstack.org/50561314:46
*** hongbin has joined #openstack-nova14:47
gibisdague: I'm looking at https://review.openstack.org/#/c/505673 (qemu 2.10 support). Do we have a way to unit test the change you made in the libvirt/driver.py ?14:48
*** ratailor has quit IRC14:50
*** coreywright has joined #openstack-nova14:51
*** ratailor has joined #openstack-nova14:51
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova master: Change 'InstancePCIRequest' spec field  https://review.openstack.org/44925714:52
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova master: Read Neutron port 'binding_profile' during boot  https://review.openstack.org/50748114:52
*** ralonsoh_ is now known as ralonsoh14:52
*** lbragstad has joined #openstack-nova14:54
gibiif somebody want some easy patches to review then I'd like to suggest a test improvement series starts https://review.openstack.org/#/c/499539 and mriedem already +2 on it.14:54
bauzasjaypipes: I did noticed14:55
*** eharney has quit IRC14:56
bauzasjaypipes: Firefox FTW14:56
*** sshwarts has quit IRC14:56
jaypipesbauzas: I'm on firefox...14:56
bauzaswhen I say FTW, it's sarcastic14:56
bauzasso yeah, I'm hitted too14:56
bauzasI suspected a French regulation cause, but looks like it's not :p14:57
*** gyee has joined #openstack-nova14:57
mriedemgibi: good point, there is no unit test coverage of the libvirt driver change or the images change to use the new flag14:58
*** gyee has quit IRC14:58
mriedemgibi: -1 that sonofa14:58
*** gyee has joined #openstack-nova14:58
gibimriedem: I think the image part is covered by https://review.openstack.org/#/c/505673/5/nova/tests/unit/virt/libvirt/test_utils.py@18114:59
bauzasmriedem: not sure I love assigning a new module attribute for the QEMU version, but I guess it's probably the only easy way15:00
*** manasm has quit IRC15:00
bauzasmriedem: I'm thinking of cases where you don't reload your module and you're doomed15:00
*** Oku_OS is now known as Oku_OS-away15:00
mriedemoh i see15:00
bauzasbut I guess upgrading qemu would be part of a nova minor upgrade as well, so that would require restarting n-cpu as well15:01
mriedembauzas: jamespage went an alternate route where the version was passed down through the many plumbing layers, and it was quickly unmanageable15:01
mriedemanother alternative is handling the error, parsing the output and retrying with the flag, which also sucks15:01
bauzasmriedem: yeah, I was thinking the image module wasn't written like that, but unfortunately it is15:01
*** lajoskatona has left #openstack-nova15:01
bauzasit's just an helper module15:02
bauzasso no way to pass a flag but just adding a global var which sucks in my mind but which sucks less than the other possibilities you envisaged15:02
*** moshele has quit IRC15:05
*** slaweq_ has joined #openstack-nova15:10
mriedemweird, seeing this in the logs when creating 500 servers using the fake driver15:10
mriedemSep 26 15:09:38 devstack nova-compute[30351]: DEBUG nova.compute.resource_tracker [None req-f0c0e899-18e6-47f9-b13b-34829646d07e demo demo] Instance 62604071-77c2-46e1-9f57-d7192edc3f82 has been deleted (perhaps locally). Deleting allocations that remained for this instance against this compute host: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 512, u'DISK_GB': 1}}. {{(pid=30351) _remove_deleted_instances_allocations /opt/stack/15:10
mriedemSep 26 15:09:38 devstack nova-compute[30351]: INFO nova.scheduler.client.report [None req-f0c0e899-18e6-47f9-b13b-34829646d07e demo demo] Deleted allocation for instance 62604071-77c2-46e1-9f57-d7192edc3f8215:10
*** eharney has joined #openstack-nova15:11
mriedemdoesn't really matter for what i'm testing, but it's odd15:11
*** lucasxu has joined #openstack-nova15:13
*** penick has joined #openstack-nova15:15
*** gbarros has quit IRC15:15
*** felipemonteiro has joined #openstack-nova15:15
mriedemstephenfin: what's your actual -1 on this now? https://review.openstack.org/#/c/457532/15:15
mriedemi'd like to see this get in so i can use it in devstack15:15
mriedemdtroyer told us to make the command names "openstack resource provider <action>"15:15
mriedemto be consistent with everything else in osc15:15
mriedembeing new to osc, i'm going to follow his recommendations15:15
*** rnoriega_ is now known as rnoriega15:16
stephenfinmriedem: There isn't one. Removed now15:16
mriedemand yeah, it's a bit large, but it lays the base crud ops for RPs15:16
mriedemplus the common framework stuff15:16
*** felipemonteiro_ has joined #openstack-nova15:16
*** andreas_s has quit IRC15:18
*** brault has quit IRC15:19
*** felipemonteiro has quit IRC15:20
*** brault has joined #openstack-nova15:20
*** chyka has joined #openstack-nova15:21
*** felipemonteiro_ has quit IRC15:21
cdentjaypipes: one of the other things I did almost immediately after you asked was limiting aggregate checks: https://review.openstack.org/#/c/489633/15:21
*** tonyb has quit IRC15:22
sdaguegibi: it's tested in images.py15:23
sdaguemriedem / gibi: https://review.openstack.org/#/c/505673/5/nova/tests/unit/virt/libvirt/test_utils.py15:23
jaypipescdent: cool, will check shortly.15:23
mriedemsdague: i think he meant there is no unit test for the driver code setting the variable in https://review.openstack.org/#/c/505673/5/nova/tests/unit/virt/libvirt/test_driver.py15:23
gibisdague: that test sets the images.QEMU_VERSION  directly, but you have some code in the driver.py that sets images.QEMU_VERSION15:23
*** tonyb has joined #openstack-nova15:24
mriedemlike test_next_min_qemu_version_ok15:24
openstackgerritChris Dent proposed openstack/nova master: [placement] gabbi tests for shared custom resource class  https://review.openstack.org/48520915:24
mriedemgibi: you can -1 and i can add the test later15:25
sdaguemriedem: he already did15:25
mriedemdansmith: you said os reboot and restack would cleanup devstack?15:27
dansmithmriedem: reboot and stack15:27
*** gbarros has joined #openstack-nova15:28
*** lyan has quit IRC15:32
openstackgerritEvgeny Antyshev proposed openstack/nova master: Add ploop procedures to privsep.libvirt  https://review.openstack.org/50756915:32
*** sree has joined #openstack-nova15:32
*** psachin has quit IRC15:32
*** lyan has joined #openstack-nova15:32
*** gbarros has quit IRC15:34
*** lyan has quit IRC15:34
*** lyan has joined #openstack-nova15:34
*** mingyu has quit IRC15:34
*** gbarros has joined #openstack-nova15:36
*** mingyu has joined #openstack-nova15:36
*** sree has quit IRC15:37
*** gbarros has quit IRC15:37
*** jdillaman has joined #openstack-nova15:38
*** armax has quit IRC15:38
*** gbarros has joined #openstack-nova15:40
openstackgerritSean Dague proposed openstack/nova master: Support qemu >= 2.10  https://review.openstack.org/50567315:40
sdaguegibi / mriedem unit test added15:40
gibisdague: looking15:41
*** felipemonteiro has joined #openstack-nova15:42
gibisdague: thanks, looks good, +215:45
*** kristian__ has joined #openstack-nova15:45
sdaguemriedem: ok, you next, reminding you of the "be faster and merge things" principle from the PTG :)15:45
dansmithsdague: he's currently working on being fast and merge things on my instance list patch15:47
dansmithsdague: but I'll look at yours so he can continue to be fast and merge on my patch15:47
sdagueit's pretty small and should be straight forward15:47
*** kristian__ has quit IRC15:48
* sdague reruns tests locally to figure out how the REQUIRED_LOCKING https://review.openstack.org/#/c/507528/ passed locally before 15:48
*** kristian__ has joined #openstack-nova15:48
mriedemsdague: i shouldn't approve my own change15:50
sdaguemriedem: it's barely your change any more15:51
*** manasm has joined #openstack-nova15:52
*** tssurya has quit IRC15:53
*** sbezverk has quit IRC16:00
*** jistr is now known as jistr|mtg16:01
gibiI've just realized that I have to migrate the notification burndown chart from openshift 2 to 3 until end of September. This will be joyful16:02
mdboothstephenfin: Are the changes to https://review.openstack.org/#/c/507488/ evil?16:02
*** jdillaman has quit IRC16:02
mdboothThinking specifically of the 'if six.py2:' in libvirt/guest.py16:03
mdboothI still think it's saner to have unicode everywhere, and convert to something else at the point of use16:03
dansmithcdent: are you working your way up that migration uuid series? if so, I'll hold off pushing that fix you just identified until you have a chance to nit out on anything else16:04
stephenfinmdbooth: No, I'd probably do the same thing16:04
dansmithmeaning, you can identify more nit-ish things and I can fix them, vs. just pushing for nits later16:04
* stephenfin awaits 2020 and the death of Python 2.7 eagerly16:04
cdentdansmith: yup (on phone at the moment though)16:04
dansmithcdent: okay, np16:05
*** rcernin has quit IRC16:05
*** ratailor has quit IRC16:06
stephenfinmdbooth: though I would like to see one of the tests slightly reworked, if that makes sense to you?16:06
mdboothstephenfin: Sure thing. I'll take a look in the morning.16:07
openstackgerritMerged openstack/nova master: Remove SCREEN_LOGDIR from devstack install setting  https://review.openstack.org/50742516:07
*** udesale has quit IRC16:08
*** Apoorva has joined #openstack-nova16:09
*** mvk has quit IRC16:11
*** moshele has joined #openstack-nova16:15
*** armax has joined #openstack-nova16:15
*** yassine has quit IRC16:16
*** jistr|mtg is now known as jistr16:18
*** markmc has quit IRC16:20
*** armax has quit IRC16:20
bauzasedleafe: jaypipes: cdent: I thought we would be talking of how we would lead reschedules in https://review.openstack.org/#/c/498830/16:20
*** markmc has joined #openstack-nova16:21
bauzasedleafe: jaypipes: cdent: I see a couple of comments in that spec review, but have you settled down on discussing reschedules as being out of scope for that spec?16:21
dansmithcdent: ah I see you have +1s on most of the rest of the set anyway, so I'll just push16:21
cdentdansmith: i’m in the midst of the last one now16:22
dansmithcdent: ah okay then I'll wait16:22
cdentso give me a couple of minutes (phone call was shorter than expected)16:22
dansmithyeah not trying to rush you16:22
openstackgerritEric Berglund proposed openstack/nova master: PowerVM Driver: config drive  https://review.openstack.org/40940416:23
dansmithI just want to push that review button and get that squirt of dopamine you know :P16:23
edleafebauzas: reschedules will be a different spec16:24
bauzasedleafe: so the spec is litterally just for mentioning which object the scheduler will return to conductor ?16:24
rybridgesHey guys. I am using the ocata release and am wondering if there is any way to print the user data associated with an instance that i own from the cli with the openstack client16:25
bauzasedleafe: looks uber too much16:25
bauzasI mean, super heavy16:25
edleafebauzas: since this will be sent over RPC, we needed agreement on it so that we don't find ourselves changing it later16:26
bauzasI'd be up concentrating our minds on how we plan to pass that object16:26
bauzasedleafe: we did a couple of RPC changes that didn't require a spec fortunately16:26
bauzasbut I leave the mic to mriedem16:26
edleafebauzas: the idea is to get it close to correct before we make the change16:26
dansmithbauzas: specs are cheap16:26
dansmithif edleafe wants separate specs, I don't think there's a problem16:27
edleafebauzas: and given the amount of discussion on the Selection object spec, I'd say it was a good thing to do16:27
*** yufei has joined #openstack-nova16:27
dansmithwe should focus on getting the work done and not the process16:27
bauzasdansmith: well, I'd rather then look at code, but okay :)16:27
bauzasyeah that16:27
edleafebauzas: the work is being done in parallel16:27
*** kristian__ has quit IRC16:27
*** r-daneel has joined #openstack-nova16:27
bauzasedleafe: well, okay16:28
*** kristian__ has joined #openstack-nova16:28
bauzasedleafe: the thing is, if you want a spec, fine with me, but then precise the scope16:28
*** yufei has left #openstack-nova16:28
bauzassince it was a work item, I was expecting more16:28
dansmithmriedem: so I was just looking at this for evac and live migration.. this method _moves_ allocations to the destination, not copies AFAICT: https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L222-L22416:29
dansmithmriedem: is that right?16:29
bauzasif the spec isn't targeting to mention how reschedules would be done, fair enough but just make sure you clearly scope that16:29
cdentdansmith: dobne16:29
dansmithcdent: yes the last one is less done16:30
* cdent nods16:30
mriedemdansmith: copies16:32
*** kristian__ has quit IRC16:32
dansmithmriedem: oh does claim_resources() do the doubling thing?16:33
mriedemit takes the allocations for the instance on the source node, and makes those same allocations for the instance on the dest node16:33
dansmithwhich will erase the source allocation16:33
mriedemit's basically what the scheduler would do,16:33
dansmithbecause... only one consumer16:33
mriedemoh it calls claim_resources,16:33
mriedemso yeah it doubles16:33
mriedemthis is the thing where force=True16:34
mriedemso we don't call the scheduler to double the allocs16:34
mriedemand i said i wanted to move back into the scheduler, but we'd need a skip_filters flag in select_destinations16:34
dansmithokay I didn't think claim_resources was the doubling one, but maybe so, I'll dig a bit16:34
*** jdillaman has joined #openstack-nova16:34
mriedemclaim_resources calls the double stuff method16:34
dansmithcdent: can you look at my comment on the DRY thing and see if you buy what I'm sellin' ?16:35
cdentdansmith: I will buy that with an entire whole dollar, if you comment the plan16:38
dansmithcdent: you saw the "when we have an atomic operation we should remove this" right?16:40
*** jangutter has quit IRC16:40
cdentyes, but (unless I missed it) there’s no “this dupe with that other thing but we don’t care cuz”16:42
dansmithI will add more words16:42
cdentI’ll still buy it for a dollar even if you don’t16:42
*** dtantsur is now known as dtantsur|afk16:42
openstackgerritDan Smith proposed openstack/nova master: Make allocation cleanup honor new by-migration rules  https://review.openstack.org/49894816:45
openstackgerritDan Smith proposed openstack/nova master: Pre-create migration object  https://review.openstack.org/49895016:45
openstackgerritDan Smith proposed openstack/nova master: Revert allocations by migration uuid  https://review.openstack.org/49894916:45
openstackgerritDan Smith proposed openstack/nova master: Refactor resource tracker to account for migration allocations  https://review.openstack.org/50641916:45
openstackgerritDan Smith proposed openstack/nova master: Make migration uuid hold allocations for migrating instances  https://review.openstack.org/50642016:45
*** yufei has joined #openstack-nova16:50
*** yufei has quit IRC16:51
openstackgerritSean Dague proposed openstack/nova master: Move ploop commands to privsep.  https://review.openstack.org/49232516:51
*** claudiub has quit IRC16:52
openstackgerritSean Dague proposed openstack/nova master: Move ploop commands to privsep.  https://review.openstack.org/49232516:54
*** derekh has quit IRC16:54
*** trinaths has joined #openstack-nova16:56
*** trinaths has left #openstack-nova16:56
*** trinaths1 has joined #openstack-nova16:57
*** trinaths1 has left #openstack-nova16:57
mriedemnotifications meeting in openstack-meeting-4 in 2 minutes16:58
*** cdent has quit IRC16:58
gibi... and now it is started17:00
*** baoli has quit IRC17:01
*** slaweq_ has quit IRC17:02
*** baoli has joined #openstack-nova17:02
*** abalutoiu has joined #openstack-nova17:07
*** rcernin has joined #openstack-nova17:11
*** manasm has quit IRC17:16
*** Swami has joined #openstack-nova17:23
openstackgerritMoshe Levi proposed openstack/nova master: Don't overwrite binding-profile  https://review.openstack.org/50561317:27
*** crushil has quit IRC17:27
*** crushil has joined #openstack-nova17:27
*** kristian__ has joined #openstack-nova17:29
*** moshele has quit IRC17:32
openstackgerritEric Berglund proposed openstack/nova master: PowerVM Driver: config drive  https://review.openstack.org/40940417:32
*** jpena is now known as jpena|off17:32
*** kristian__ has quit IRC17:34
*** abalutoiu has quit IRC17:36
*** gszasz has quit IRC17:36
*** tesseract has quit IRC17:37
*** ragiman has quit IRC17:38
mriedemdansmith: aha, i think i'm hitting issues in devstack where placement isn't getting cleaned up for instances that get 'local' deleted in the api17:38
mriedemnot totally sure yet, but failing to burst 500 new instances, hitting NoValidHost17:38
mriedemand i assume it's placement b/c it's not the scheduler filters17:38
*** gbarros has quit IRC17:38
dansmithmriedem: and why do you have locally-deleted instances for this test?17:39
melwittfor local deletes, allocations aren't cleaned up till the compute host heals it17:39
dansmithright, what melwitt said17:39
mriedemmysql> select count(id) from consumers;17:39
mriedem| count(id) |17:39
mriedem|      2002 |17:39
mriedem1 row in set (0.01 sec)17:39
*** cdent has joined #openstack-nova17:39
mriedemmelwitt: there is no compute for these17:40
mriedemthey failed during scheduling17:40
mriedemalthough yeah why would placement have allocations for these...17:40
mriedemstack@devstack:~$ nova list | grep -c ERROR17:40
melwittoh, hm17:40
mriedemso i've got 1000 instances in ERROR state, and 2002 consumers in the api db17:40
*** artom has quit IRC17:41
melwittallocations are written at claim time?17:41
mriedemfrom the scheduler yeah17:41
*** kristian__ has joined #openstack-nova17:41
melwittso that would explain the ones you do have. but I guess your point is why are there more allocation consumers than non error instances17:42
*** kristia__ has joined #openstack-nova17:42
mriedemthat's because i've deleted 1000 over time17:42
mriedemi was hitting messaging timeouts between conductor and the scheduler earlier today, so had 500 in error which i needed to be active, so deleted all of those, restarted conductor and scheduler, and was able to create a single instance17:43
mriedemso tried with 500 more again17:43
mriedemand hit novalidhost on all of those17:43
openstackgerritMerged openstack/nova master: cleanup test-requirements  https://review.openstack.org/50706317:44
*** kristian__ has quit IRC17:45
*** gabor_antal has quit IRC17:46
*** jmlowe has quit IRC17:46
*** lbragstad has quit IRC17:47
*** gabor_antal has joined #openstack-nova17:48
*** moshele has joined #openstack-nova17:48
*** ralonsoh has quit IRC17:51
*** lbragstad has joined #openstack-nova17:51
*** ijw has joined #openstack-nova17:53
*** kristia__ has quit IRC17:54
*** ijw has quit IRC17:55
openstackgerritDan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation  https://review.openstack.org/50763817:55
dansmithjaypipes: cdent: ^ quick stab at the live migrate version of this17:55
dansmithit's probably rough at this point, but worth a look I think17:56
*** vvargaszte has joined #openstack-nova17:58
*** ijw has joined #openstack-nova17:58
*** ijw has quit IRC17:58
*** ijw has joined #openstack-nova17:58
*** baoli has quit IRC18:00
*** baoli has joined #openstack-nova18:01
*** Apoorva_ has joined #openstack-nova18:02
*** vvargaszte has quit IRC18:04
*** Apoorva has quit IRC18:06
*** kristian__ has joined #openstack-nova18:08
*** Apoorva_ has quit IRC18:08
*** kristia__ has joined #openstack-nova18:08
*** Apoorva has joined #openstack-nova18:08
*** moshele has quit IRC18:09
*** kristi___ has joined #openstack-nova18:11
*** lucasxu has quit IRC18:11
*** dave-mcc_ is now known as dave-mccowan18:12
*** kristian__ has quit IRC18:12
*** kristia__ has quit IRC18:14
*** moshele has joined #openstack-nova18:17
*** jmlowe has joined #openstack-nova18:18
*** moshele has quit IRC18:20
*** gbarros has joined #openstack-nova18:21
*** xyang1 has joined #openstack-nova18:24
-openstackstatus- NOTICE: The infra team is continuing work to bring Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information.18:24
*** slaweq_ has joined #openstack-nova18:27
*** xyang1 has quit IRC18:27
*** moshele has joined #openstack-nova18:31
cdentdansmith: haven’t had a chance to give it a proper look, but saw a weird when skimming the live migrate thing18:32
*** techstepper has joined #openstack-nova18:32
*** Apoorva has quit IRC18:33
dansmithit's returning True-ish which is what I wanted for the functional tests18:33
dansmithso.. working as designed? :)18:33
*** Apoorva has joined #openstack-nova18:33
openstackgerritMerged openstack/nova master: Set the Pike release version for scheduler RPC  https://review.openstack.org/50724518:34
cdentgo python!18:34
mriedemwtf, so i can't create multiple instances, i get novalidhost, but i can create one at a time18:38
melwittare you using multi-create?18:38
mriedemwasn't a problem yesterday18:38
mriedembut i had a bit of a cleaner env yesterday18:38
*** gabor_antal has quit IRC18:39
*** gabor_antal has joined #openstack-nova18:39
melwittmulti-create will reject you if any one of min_count can't be accommodated. so one at a time would work if you're in that situation, if some/most of them fit18:39
mriedemyesterday i created 100, then like 200, then 500 more or something18:40
mriedemeventually got to 100018:40
mriedemi can just restack this env, but it makes me worry that we aren't properly cleaning up allocations somewhere18:41
melwittdid you say yesterday you don't have computes, or something like that? I just wonder what happens with FakeDriver, if it somehow doesn't call the healing allocations code18:43
mriedemwe don't heal since pike18:43
mriedemif you don't have computes < pike, we don't heal18:43
mriedemthis is just a single compute, single node devstack18:44
mriedemwith the fake driver and noop quota18:44
melwittoh wait, sorry I was thinking of local delete18:44
melwittI was trying to think if with FakeDriver, does the code that deletes allocations when an instance is deleted, run18:44
melwittor if that even matters18:45
*** vvargaszte has joined #openstack-nova18:45
mriedemthe compute manager cleans up allocations when an instance is deleted18:45
dansmithmriedem: we heal for deletes18:45
dansmithmriedem: did you archive them after local delete before you started up?18:45
mriedemi've been archiving yeah18:46
melwittI'm not sure whether he had local deletes18:46
dansmithso that's why18:46
dansmithI thought he did18:46
melwittI thought he did too but I'm getting a little confused18:46
*** slaweq_ has quit IRC18:46
mriedemyesterday i didn't have any instances in ERROR state, so they were all in the cell18:46
mriedemi deleted all of those and then archived cell0 and cell118:47
*** moshele has quit IRC18:47
dansmithdeleted them locally?18:47
mriedemtoday i've been trying to get 500 ERROR during scheduling, and 500 ACTIVE18:47
mriedembased on the flavor i use18:47
mriedemdansmith: deleted via the api18:47
*** slaweq_ has joined #openstack-nova18:47
dansmithmriedem: with compute down or no?18:47
mriedemremember me complaining about how long that was taking yesterday?18:47
mriedemtook 2+ hours to delete 1000 ACTIVE instances18:47
dansmithI do, but I didn't remember all your details18:48
mriedemyeah i basically trying to get back to clean state before starting today18:48
mriedemso was archiving the db's last night18:48
mriedembtw, bauzas pointed this out before, but we log this way too many times18:49
mriedemSep 26 18:44:37 devstack nova-compute[30351]: DEBUG nova.compute.resource_tracker [None req-992d494e-d328-4204-bcfe-80d926cf0a65 demo demo] We're on a Pike compute host in a deployment with all Pike compute hosts. Skipping auto-correction of allocations. {{(pid=30351) _update_usage_from_instance /opt/stack/nova/nova/compute/resource_tracker.py:1071}}18:49
dansmithmriedem: unrelated, see this: http://status.openstack.org/openstack-health/#/test/nova.tests.functional.test_servers.ServersTestV219.test_description_errors?duration=P3M18:52
*** slaweq_ has quit IRC18:52
dansmithmriedem: I think this test is occasionally taking up to 240s locally when it should be about 8s18:52
*** lucasxu has joined #openstack-nova18:53
dansmithand I think it's because it creates a server that it never cleans up and then abruptly exits where we take down conductor before the compute service finishes waiting on a call or something18:53
dansmithso I have a patch to just make it clean up the server and I _think_ it's working18:53
mriedemthe one weird spike in august is, weird18:53
openstackLaunchpad bug 1719714 in OpenStack Compute (nova) "Excessive logging of "We're on a Pike compute host in a deployment with all Pike compute hosts."" [Medium,Confirmed]18:53
dansmithmriedem: it would have just been ordering reasons18:54
dansmithmriedem: note the rising tail at present too18:54
*** mnestratov has quit IRC18:57
*** baoli has quit IRC18:57
*** baoli has joined #openstack-nova18:59
*** ianw|pto is now known as ianw19:01
*** egonzalez has joined #openstack-nova19:02
*** ijw has quit IRC19:03
*** mvk has joined #openstack-nova19:06
*** tbachman has quit IRC19:08
*** jaosorior has quit IRC19:09
*** moshele has joined #openstack-nova19:10
*** sahid has quit IRC19:11
*** elod has quit IRC19:12
mriedemalright i'm just going to restack19:18
mriedemnuts to this19:18
*** tbachman has joined #openstack-nova19:19
*** sree has joined #openstack-nova19:25
mriedemdansmith: jaypipes: bauzas: https://review.openstack.org/#/c/498947/619:28
mriedemthat test_servers thing is wrong19:28
*** ijw has joined #openstack-nova19:28
*** vvargaszte has quit IRC19:28
openstackgerritMatthew Treinish proposed openstack/nova master: Add slowest command to tox.ini  https://review.openstack.org/50765719:29
mtreinishdansmith: ^^^19:29
mriedemthere are 2 tests for failures during evacaute on the dest19:29
mriedem1. test_evacuate_claim_on_dest_fails - that is testing when the claim fails with ComputeResourcesUnavailable19:29
mriedem2. test_evacuate_rebuild_on_dest_fails - that is testing when the claim is successful but the driver.rebuild method raises some exception19:29
dansmithmtreinish: sweet19:29
jaypipesmriedem: sorry, I disagree with you.19:30
mriedemi wrote those tests19:30
*** sree has quit IRC19:30
mriedemso please explain how i'm wrong that they are now made redundant in that change19:30
jaypipesmriedem: that test raising TestingException was not useful. Because TestingException isn't what is ever raised by any code.19:30
mriedemit's simulating the virt driver raising the error during rebuild19:30
mriedemAFTER the successful claim19:31
mriedemit could be ProcessExecutionError19:31
mriedemfrom driver.spawn()19:31
mriedemif you like19:31
*** slaweq_ has joined #openstack-nova19:31
mriedemthese 2 tests are testing very specific failures19:31
dansmithmriedem: right but we don't run the claim teardown code in that case19:31
mriedemdansmith: correct, which is why we run the allocation cleanup manually19:32
mriedemand that's what that is testing19:32
*** vladikr has quit IRC19:32
mriedemthe test you changed isn't meant to test drop_move_claim19:32
mriedemthe docstring explains that19:32
jaypipesmriedem: if the point of the test (as is in that docstring) is to ensure allocations are cleaned up after a failed rebuild, then the test should raise the exception that would be raised *after* a claim has been made for the new resources.19:32
dansmithjaypipes: he's saying another one does that19:32
mriedemjaypipes: you realize the virt drivers can raise any kinds of crazy shit right?19:33
dansmithmriedem: so in this case you want the test to validate that the allocations _don't_ get cleaned up is that right?19:33
*** ijw has quit IRC19:33
jaypipesmriedem: Matt, I'm trying to be civil.19:33
*** rcernin has quit IRC19:34
dansmiththis is what it's testing: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2800-L282719:35
dansmiththe except exception case of that19:35
mriedemso ^ is testing that drop_move_claim removes the allocation when the claim was successful but the virt driver raised some exception19:35
*** kristi___ has quit IRC19:35
jaypipesmriedem: OK, I see that now.19:35
*** ijw has joined #openstack-nova19:35
mriedemhttps://review.openstack.org/#/c/499874/ added the other test19:36
mriedemthat was a recreate test for a bug19:36
mriedemfixed in https://review.openstack.org/#/c/499878/19:36
dansmithmriedem: we get it19:37
dansmithmriedem: can you answer my question above about what you want it to do?19:37
mriedemthe test should go back to whatever it was testing19:39
mriedemwhich is the case that the claim passes, but the virt driver raises19:39
mriedemso we'd remove the allocation via drop_move_claim before19:39
dansmithright, but you assert some behavior that happens inside drop_move_claim19:40
dansmithwhich no longer happens19:40
*** ijw has quit IRC19:40
mriedemthen that drop_move_claim behavior has to be replayed elsewhere i guess19:40
jaypipeswhat if the it's a same-host rebuild? :(19:40
mriedemthere is no claim for a same host rebuild19:40
mriedemso you wouldn't hit ComputeResourcesUnavailable19:40
dansmithexactly, but you could hit other exceptions19:41
jaypipesmriedem: but you *would* hit the TestingException "crazy shit"19:41
jaypipesmriedem: and you're asserting that we'd delete the allocation against the instance in that case, right?19:41
mriedemthat doesn't have anything to do with dropping an allocation though19:41
jaypipesmriedem: oh, sorry, you're asserting that the *update_available_resource()* call would clean up allocations for a failed build?19:42
dansmiththe test _is_ asserting that the dest host's allocation was cleaned up by drop_move_claim19:42
mriedemwe don't ever want to remove allocations for a *rebuild*19:42
mriedemthe tests are specifically for evacuate19:42
mriedemwhere the scheduler creates allocations on the dest host19:43
mriedemwe fail the evacuate on the dest host, so we need to remove those allocatoins created by the scheduler19:43
*** armax has joined #openstack-nova19:43
dansmiththere's a specific reason why I made this change,19:44
dansmithand I talked it through with jaypipes which is why I made this19:44
mriedemi'm sorry for being grouchy about this,19:44
mriedembut i've spent the better part of the last 6 weeks fixing these allocation bugs,19:44
dansmithso I'll have to go re-load all my context on this before I can really think about it19:44
mriedemso being told i don't understand the test pisses me off19:44
jaypipesmriedem: understood. and you're saying that you want the drop_move_claim() to remove those resources when ComputeResourcesUnvailable is raised but you want update_available_resource() to delete the allocations when a virt driver exception is raised?19:44
*** Apoorva has quit IRC19:44
dansmithjaypipes: no19:44
*** vladikr has joined #openstack-nova19:45
dansmithmriedem: without offending your test sensibilities, you see this right? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L281219:45
dansmiththat's what *should* have been deleting the target's allocation19:45
dansmithonly in the unavailable case19:46
dansmithbut in reality, we were always doing it for the other cases as well19:46
dansmithhow? because we always ran drop_move_claim19:47
dansmithreset the test to where it was and run it with the oddball exception and we'll assert that the dest host claim is zero, but it no longer is after this change19:48
dansmiththis: https://pastebin.com/fMUmmgMC19:48
mriedemi don't know if we're talking about the same thing,19:49
mriedemi wrote https://review.openstack.org/#/c/499877/ to show that we didn't need to manually remove allocations on the dest when the driver failed19:49
mriedembecause of drop_move_claim19:49
mriedemthe other test was to show that we needed to add https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L281219:49
mriedemwhen the claim itself fails19:49
mriedemraising ComputeResourcesUnavailable19:49
dansmithright, but that doesn't make sense right?19:49
dansmithif we fail for some driver reason, we're now "on" that dest host and should be able to run a same-host rebuild on it19:50
dansmithwhich won't re-claim for us19:50
mriedemno, we're not on that host19:50
mriedemif driver.spawn fails, we're not on that host19:50
mriedemthe instance is only on the dest host if we get here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L283619:50
mriedemwhich doesn't happen if the driver fails19:50
*** tbachman has quit IRC19:50
dansmithwhy is this here? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L281219:50
mriedemsee https://review.openstack.org/#/c/499878/19:51
mriedemplus the comment above it19:51
*** awaugama has quit IRC19:52
mriedemso i don't know what else is going on in this patch, i didn't get that far, i saw the commit message and change to the test and wanted to bring that up since it's approved19:54
dansmithmriedem: okay yeah I really thought that the rt claim would set host and node19:54
dansmithmriedem: you better -2 that or something so it doesn't merge19:55
mriedemi don't think that works19:55
dansmithmriedem: -2 will make it not merge I think, it just won't kick it19:55
openstackgerritMatt Riedemann proposed openstack/nova master: Move allocation manipulation out of drop_move_claim()  https://review.openstack.org/49894719:55
openstackgerritDan Smith proposed openstack/nova master: BUMPMove allocation manipulation out of drop_move_claim()  https://review.openstack.org/49894719:55
mriedemi know a new commit will do it19:55
mikalI have a cold, hold me19:55
mriedemgdi mikal19:56
mriedemyou've stepped into the wrong room at the wrong time19:56
mriedemway out of line donny19:56
*** jmlowe has quit IRC19:56
jaypipesyou're outta your element, mikal19:56
mikaljaypipes: that's always been true though19:56
mriedemmikal, btw, given your plentiful rackspaceness, do you know if rax ever used this thing http://lists.openstack.org/pipermail/openstack-operators/2017-September/014267.html ?19:57
mikalmriedem: my name isn't donny?19:57
mriedemgdi mikal19:57
mriedemi'm walter in this scenario19:57
*** lucasxu has quit IRC19:57
mriedemfilled with rage19:57
mikalmriedem: so, Rackspace is definitely running code newer than kilo, so if they haven't noticed they don't need it?19:57
mikalmriedem: private cloud didn't use it, public might but johnthetubaguy would know more about that19:58
mikalmriedem: ahhh, ok, I shall study before next time19:58
mriedemmikal: anyone that was relying on it and has newer than kilo, and didn't notice, then yeah i guess we don't need it19:58
mriedemthat's the assertion in the ML thread and commit to remove it anyway19:59
*** baoli has quit IRC20:00
melwittdoes anything ever set 'group_members' in filter_properties? I'm not finding anything https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L20520:03
*** baoli has joined #openstack-nova20:04
sdaguemikal: I retool your ploop patch with the fixes from the virtuozo folks20:04
melwittthis line is making reschedule fail with "'NoneType' object is not iterable" after a late-affinity-check failure. I don't see how reschedule after late check ever could work20:04
*** pcaruana has quit IRC20:05
* mtreinish laughs everytime he sees ploop20:06
mriedemmelwitt: it could be a case of group_members being set on the request spec initially, and then it's transformed into the primitive filter_properties stuff which doesn't include the group_members?20:08
mriedemi've seen some wonky stuff with how the request spec transforms to/from the legacy filter props20:08
mikalsdague: ta, looking at it now20:09
mriedemmelwitt: see _to_legacy_group_info ?20:09
mriedemit sets group_updated=True but doesn't include group_members20:09
melwittmriedem: I don't see that RequestSpec has any group_members in it. I grepped for "group_members" in nova and found nothing that ever sets it20:09
*** dave-mcc_ has joined #openstack-nova20:09
melwittyeah, I see that. that's the only thing that looks like it could be related20:10
mikalsdague: looks like there is a rebase error there though? The console pty stuff is now in that patch.20:10
mtreinishdansmith: http://logs.openstack.org/57/507657/1/check/gate-nova-python27-ubuntu-xenial/55aebe1/console.html#_2017-09-26_19_42_08_17271920:10
mikalsdague: I shall rectify20:10
melwittmriedem: late affinity check reschedule just looks totally broken unless I'm blind. which may very well be the case20:10
mriedemmelwitt: this added it https://review.openstack.org/#/c/148277/20:11
*** dave-mccowan has quit IRC20:12
melwittlooks like that line in scheduler/utils that sets it is gone now20:12
openstackgerritEd Leafe proposed openstack/nova master: Add alternate hosts  https://review.openstack.org/48621520:12
openstackgerritEd Leafe proposed openstack/nova master: Add Selection objects  https://review.openstack.org/49923920:12
openstackgerritEd Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver  https://review.openstack.org/49585420:12
melwitttrying to find what removed it20:13
mriedemmelwitt: https://review.openstack.org/#/c/469037/20:13
mriedempike ^20:13
*** egonzalez has quit IRC20:13
melwittokay, so the conductor logic is still relying on stuff being in filter_properties via RequestSpec.from_primitives20:14
mriedemlooks like it, left some comments in https://review.openstack.org/#/c/469037/6/nova/objects/request_spec.py20:15
mriedemlikely need a functional regression test to show the failure20:15
mriedemthen fix on top and backport both to pike20:15
mriedemmelwitt: do you have a bug for this?20:16
melwittmriedem: no, I can open one. wanted to sanity check with yall first20:17
mriedemsomeone was saying they hit this exact same thing this morning to bauzas20:17
*** Apoorva has joined #openstack-nova20:18
melwitthah, what a coinky dink20:18
mriedemplease keep that sailor talk for at home20:19
melwittaye aye sir!20:20
melwittoh, that's pirate. nvm20:20
openstackgerritDan Smith proposed openstack/nova master: Move allocation manipulation out of drop_move_claim()  https://review.openstack.org/49894720:20
openstackgerritDan Smith proposed openstack/nova master: Make allocation cleanup honor new by-migration rules  https://review.openstack.org/49894820:20
openstackgerritDan Smith proposed openstack/nova master: Pre-create migration object  https://review.openstack.org/49895020:20
openstackgerritDan Smith proposed openstack/nova master: Revert allocations by migration uuid  https://review.openstack.org/49894920:20
openstackgerritDan Smith proposed openstack/nova master: Refactor resource tracker to account for migration allocations  https://review.openstack.org/50641920:20
openstackgerritDan Smith proposed openstack/nova master: Make migration uuid hold allocations for migrating instances  https://review.openstack.org/50642020:20
openstackgerritDan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation  https://review.openstack.org/50763820:20
mriedemdansmith: what's the word20:21
melwittbird bird bird bababir bird's the word20:22
*** dave-mccowan has joined #openstack-nova20:23
*** liverpooler has quit IRC20:23
*** belmoreira has joined #openstack-nova20:25
*** Tom_ has joined #openstack-nova20:25
*** dave-mcc_ has quit IRC20:25
*** moshele has quit IRC20:25
*** smatzek has quit IRC20:29
*** Tom_ has quit IRC20:29
melwittmriedem: https://bugs.launchpad.net/nova/+bug/171973020:30
openstackLaunchpad bug 1719730 in OpenStack Compute (nova) "Reschedule after the late affinity check fails with "'NoneType' object is not iterable"" [Undecided,New]20:30
*** jmlowe has joined #openstack-nova20:30
* efried thought "Grease" is the word20:31
*** ltomasbo has quit IRC20:31
*** jpena|off has quit IRC20:32
efriedmriedem sdague https://review.openstack.org/#/c/488137/ should be ready again20:32
*** cleong has quit IRC20:33
*** ltomasbo has joined #openstack-nova20:35
*** baoli_ has joined #openstack-nova20:35
*** jpena|off has joined #openstack-nova20:35
*** ijw has joined #openstack-nova20:36
*** felipemonteiro has quit IRC20:36
*** baoli has quit IRC20:36
*** sahid has joined #openstack-nova20:37
*** techstepper has quit IRC20:39
*** penick has quit IRC20:39
*** crushil has quit IRC20:42
*** ijw has quit IRC20:43
*** penick has joined #openstack-nova20:44
*** pchavva has quit IRC20:46
*** yikun_jiang has quit IRC20:47
*** yikun_jiang has joined #openstack-nova20:47
*** armax has quit IRC20:50
*** armax has joined #openstack-nova20:50
*** armax has quit IRC20:50
*** armax has joined #openstack-nova20:51
*** armax has quit IRC20:51
*** armax has joined #openstack-nova20:52
*** armax has quit IRC20:52
*** armax has joined #openstack-nova20:52
*** armax has quit IRC20:53
*** itlinux has joined #openstack-nova20:53
*** ijw has joined #openstack-nova20:54
*** kfarr has joined #openstack-nova20:56
*** kfarr has quit IRC20:57
*** hemna__ has quit IRC21:00
*** mnestratov has joined #openstack-nova21:00
*** sahid has quit IRC21:03
*** smatzek has joined #openstack-nova21:04
*** jmlowe has quit IRC21:07
openstackgerritMatt Riedemann proposed openstack/nova master: Add recreate test for live migrate rollback not cleaning up dest allocs  https://review.openstack.org/50767721:07
mriedemdansmith: ^ thus begins another round of these21:07
*** baoli_ has quit IRC21:07
*** jmlowe has joined #openstack-nova21:09
openstackgerritEric Berglund proposed openstack/nova master: PowerVM Driver: config drive  https://review.openstack.org/40940421:11
*** belmoreira has quit IRC21:12
*** armax has joined #openstack-nova21:12
*** MVenesio has quit IRC21:14
*** slaweq_ has quit IRC21:14
*** MVenesio has joined #openstack-nova21:15
*** thorst has quit IRC21:16
*** thorst has joined #openstack-nova21:16
*** jmlowe has quit IRC21:18
*** MVenesio has quit IRC21:19
*** belmoreira has joined #openstack-nova21:19
*** vladikr has quit IRC21:20
*** thorst has quit IRC21:21
*** belmoreira has quit IRC21:22
*** tidwellr has quit IRC21:27
*** tbachman has joined #openstack-nova21:28
*** pino has joined #openstack-nova21:30
*** eharney has quit IRC21:30
pinoHi Folks, I'm just getting started on a project that would provide an alternative to using key-pairs for instances: ssh certificates. This requires injecting into the instance (before startup) a host certificate, a user CA public key, and authorized principals file(s); then modifying sshd_config to use them. What's the right way to hook into the co21:36
pinompute instance lifecycle?21:36
pinoI'm just experimenting, but I'm wondering if this would be best built as part of Nova itself, or separately hook into the lifecycle.21:37
*** mnestratov has quit IRC21:38
jaypipespino: definitely not part of Nova itself, no. apart from writing files to a config drive, Nova doesn't mess with the VM.21:38
openstackgerritMatt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback  https://review.openstack.org/50768721:39
pinojaypipes: ok, fair enough... but shouldn't support for ssh certificates be modelled similar to keypair support?21:39
jaypipespino: honestly, I'm not sure what the diff is between a key pair, with the private part of the pair downloaded to the user and the public part laid down on the VM config drive, and the SSH certificates thing you're describing.21:41
pinoAnd in terms of doing it outside of Nova, do you agree Nova notifications are not the right mechanism? The injection of various files, plus modification of the sshd_config must be done before first boot. Any advice about where, and how to do the hook?21:41
jaypipespino: I'm not an expert in ssh stuff, apologies.21:41
pinojaypipes: I probably gave too much detail. I'm just looking for some hints about how I can hook into the startup workflow and block it until I've configured the VMs SSH the way I want it.21:43
*** r-daneel has quit IRC21:43
*** armax has quit IRC21:43
jaypipespino: I think cloud-init is more what you are looking for?21:44
*** claudiub has joined #openstack-nova21:44
mriedempino: https://docs.openstack.org/nova/latest/user/vendordata.html21:45
mriedemsetup an external rest service that provides metadata to the guest when it's created21:45
mriedemexample https://github.com/openstack/novajoin21:46
*** edmondsw has quit IRC21:46
penickpino: I use SSH CA in my environment, maybe I can help?21:47
pinomriedem: I saw that but wasn't sure it was the right approach. I'll take a closer look, thanks for the example.21:47
penickI think I see what you're trying to do, and I think what you're probably going to want is to build a small webservice to create and sign SSH certificates, then tie that in with the nova vendordata stuff to get injected into the instance on boot21:48
mriedemit's a wild penick21:48
mriedemi wonder what the keyword is here21:48
penickI identify as feral21:48
pinojaypipes: I'm looking at cloud-init too... but I want my setup script to run without the user's help (they shouldn't have to do any setup).21:48
*** ijw has quit IRC21:49
pinopenick: that makes perfect sense.21:49
*** edmondsw_ has joined #openstack-nova21:49
pinoOk, so I have a few topics/approaches to study. Thanks!21:50
*** ijw has joined #openstack-nova21:50
penicknp :)21:51
*** edmondsw_ has quit IRC21:54
*** yamahata has joined #openstack-nova21:54
*** cdent has quit IRC21:55
*** yassine has joined #openstack-nova21:58
*** esberglu has quit IRC21:58
*** penick has quit IRC22:01
*** pino has quit IRC22:03
openstackgerritEric Fried proposed openstack/nova master: Use ksa adapter for keystone conf & requests  https://review.openstack.org/50769322:10
*** esberglu has joined #openstack-nova22:12
*** sdague has quit IRC22:13
*** burt has quit IRC22:15
*** jaypipes has quit IRC22:16
*** esberglu has quit IRC22:16
*** yassine has quit IRC22:17
*** acormier has quit IRC22:20
*** jmlowe has joined #openstack-nova22:24
*** Sukhdev has joined #openstack-nova22:24
rybridgesHey guys, I have a question. I am trying to inject some default user data into every instance while it is provisioning at this location -> https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L1011  (I am adding an internal patch for this)  In my patch, I create the user data, merge it with any existing user data on the instance, and then try to write the new user data to the22:26
rybridgesdatabase. I am stuck getting it to write into the database. instance.save(22:26
*** smatzek has quit IRC22:27
rybridgesinstance.save() is throwing stack traces. so I am trying to use the update_instance() method defined in the API class defined here https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L262222:27
rybridgesbut that does not seem to actually be saving the user data in the instance for some reason22:27
rybridgesit calls build_req.save() in the update_instance() method22:27
*** lyan has quit IRC22:29
openstackgerritMatt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback  https://review.openstack.org/50768722:32
melwittrybridges: doing it that way is a bad idea IMHO. if you're looking to have default data injected into every instance, you should look into the vendordata stuff that was linked earlier22:32
*** dave-mccowan has quit IRC22:36
rybridgesso its not default perse, it will actually change based on some parameters. i just figured it would be easier for people to understand my problem if i said default22:36
*** itlinux has quit IRC22:37
rybridgesi dont like the vendordata stuff because it requires us to write an external webservice which complicates our deployment22:37
rybridgesi would rather just make a small patch which hits an entry point and injects the user data into the instance22:37
rybridgesit is much simpler and easier to debug/work with22:37
rybridgesi feel like it should not be this difficult..22:38
rybridgesto just save the instance and get the user data written to the db22:38
mriedemyou know what's going to complicate your deployment?22:39
mriedemconstantly rebasing your fork, and when we change the internals that it depends on22:39
*** lbragstad has quit IRC22:40
rybridgeswe plan on adding a vendordata service eventually22:40
rybridgesalso, the rebase is basically nothing22:41
rybridgesmy patch is 3 lines22:41
melwitt+1. I've been there before (patching nova) and would not recommend it22:41
rybridgesbecause i use an entry point that just passes locals to another function defined in a separate package22:41
*** moshele has joined #openstack-nova22:41
rybridgeswhat i am trying to do is very simple22:42
rybridgesi just want to get a poc working right now22:42
mriedemdansmith: i joked about this, but somone made it a reality http://forumtopics.openstack.org/cfp/details/622:42
mriedemnew devstack setup, still can't create 500 vms at once, they all go to NoValidHost, so maybe i didn't try to create this many at once yesterday - i wonder if i'm making placement bomb out, and the scheduler is rolling everything back22:45
rybridgesis user data immutable now? similar to how provision_updated_at is immutable in ironic22:47
openstackgerritDan Smith proposed openstack/nova master: Move allocation manipulation out of drop_move_claim()  https://review.openstack.org/49894722:48
openstackgerritDan Smith proposed openstack/nova master: Make allocation cleanup honor new by-migration rules  https://review.openstack.org/49894822:48
openstackgerritDan Smith proposed openstack/nova master: Pre-create migration object  https://review.openstack.org/49895022:48
openstackgerritDan Smith proposed openstack/nova master: Revert allocations by migration uuid  https://review.openstack.org/49894922:48
openstackgerritDan Smith proposed openstack/nova master: Refactor resource tracker to account for migration allocations  https://review.openstack.org/50641922:48
openstackgerritDan Smith proposed openstack/nova master: Make migration uuid hold allocations for migrating instances  https://review.openstack.org/50642022:48
openstackgerritDan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation  https://review.openstack.org/50763822:48
melwittmriedem: did the scheduler logs offer any clues?22:50
mriedemi think those have wrapped by now22:51
mriedemi can create in chunks of 100 just fine22:51
*** yassine has joined #openstack-nova22:51
melwittokay. was just curious22:52
mriedemnow i've got 100 ACTIVE instances, with 100 consumers in the api db and 300 allocations,22:52
mriedemwhich makes sense b/c 1 cpu, 1 ram, 1 disk allocation per instance22:52
mriedem500 in the nova_cell0 db22:52
mriedemubuntu@devstack:~$ sudo journalctl -a -u devstack@n-sch.service | grep Unable22:53
mriedemSep 26 22:28:37 devstack nova-scheduler[2951]: WARNING nova.scheduler.client.report [None req-af92d5f2-4c99-4231-966e-939e1da04239 demo admin] Unable to submit allocation for instance 5f9f4f7d-8a2f-4fb8-b30a-024ed2e8e49d (409 {"errors": [{"status": 409, "request_id": "req-cda80554-6083-45b0-87bf-9e9c9924213f", "detail": "There was a conflict when trying to complete your request.\n\n Inventory changed while attempting to alloc22:53
mriedem Another thread concurrently updated the data. Please retry your update  ", "title": "Conflict"}]})22:53
mriedemSep 26 22:28:37 devstack nova-scheduler[2951]: DEBUG nova.scheduler.filter_scheduler [None req-af92d5f2-4c99-4231-966e-939e1da04239 demo admin] Unable to successfully claim against any host. {{(pid=2951) _schedule /opt/stack/nova/nova/scheduler/filter_scheduler.py:221}}22:53
mriedemand then that removes all allocations for all instances22:54
mriedemdansmith: melwitt: ^22:54
mriedemso yeah that's my failure here22:54
melwittum, so is that a new scheduling race condition that has to be resolved with reschedules? to replace the old claim race?22:55
mriedemit does retry http://paste.openstack.org/show/621996/22:55
mriedemwe do a retry in the scheduler22:55
melwittyeah, but I thought after "claims in the scheduler" we don't have concurrent request race problems that get kicked out to be retried22:56
mriedemi don't know why that's logged 6 times22:56
dansmithI do22:56
dansmithbecause you're scheduling so many things to one compute, and it only retries a certain number of times22:56
mriedemdo tell :)22:56
mriedemi'm not surprised it's hitting a conflict22:56
mriedembut why is that logged 3 times?22:57
dansmithmelwitt: we still have concurrent updates that we have to retry22:57
mriedem6 i mean22:57
mriedembecause ^ we retry 3 times22:57
mriedemyou know, hitting a conflict that we have to retry once in 500 instances with a single compute, is pretty good22:57
mriedemalthough i'm not sure which of the 500 this is that falied22:57
melwittso this is like the claim race except worse in that the things should have succeeded but can't. maybe it won't happen in real life because by the time that retry limit would be hit, the compute host would already be rejecting claims22:57
dansmithwell, actually.. are you doing one boot there or is there anything else going on?22:58
mriedemsingle boot, --min-count 50022:58
mriedemonly thing that could be changing inventory is the RT?22:58
*** claudiub has quit IRC22:58
melwittoh, okay so all or none. so nvm what I said22:58
dansmithso we should be only making one call to scheduler I guess22:58
mriedemhmm, so before claims in the scheduler, if you multi-create, wouldn't we only fail some of these after claims in the compute and reschedules?23:00
mriedemso chances are you'd have some/most active, but others in error after reschedules?23:00
mriedemnow it's all or none23:00
dansmithall or none for the claim process23:00
dansmithyou can still fail for other reasons23:00
*** chyka has quit IRC23:01
mriedemfilters i suppose yeah23:01
dansmithbut yeah, people really hate this behavior that we tell them they can boot 500 things and later fail 20% of them because we suck23:01
dansmiththis is much better23:01
dansmithso I guess my thinking about why these are changing was because of concurrent requests, but if you're just doing one boot, I'm not sure23:02
melwittyeah, that's what I was trying to think in real life the retries would have a much better chance of succeeding bc not so many concentrated on one host, right23:02
dansmithmaybe the compute is tickling the inventory in some way, such that during 500 of them it gets changed23:02
melwittbecause the claim would detect the host full and then it would move on to another host23:03
dansmithmelwitt: yeah, I'm pretty confident it's related to the having of one compute and 500 instances23:03
melwittthe thing I'm wondering is if the host list order is consistent, could this happen in real life because they'll all try the first host in the list first. but as things are claimed, that host will fill up and then no longer be considered23:04
*** smatzek has joined #openstack-nova23:04
mriedemmelwitt: i'm wondering the same23:04
*** thorst has joined #openstack-nova23:04
mriedemif we're not shuffling the hosts a bit23:04
melwittso won't get bombarded that badly?23:04
dansmithmriedem: I want to know what the instance uuid is for each of those lines23:05
*** yamamoto has joined #openstack-nova23:05
dansmithmriedem: like maybe it retried once for a few instances, then three times for one and bailed the whole process23:06
dansmithbecause any one of them gets to three retries and it should fail and stop the num_instances loop I think23:06
mriedemwe don't have the instance uuid in that message23:07
dansmith...that's what I'm asking for yeah23:07
mriedemok, adding23:07
mriedemthat explains the 6 log messages23:07
*** thorst has quit IRC23:08
dansmithwhile you're re-testing, I'd like to suggest that we not hold up the instance list stuff on this if it takes too much longer23:08
dansmithwe can revert it if there's a real performance regression pretty easily, and we're delaying soak-time for any non-perf-related bugs we might be able to resolve just from our own infra workload23:09
mriedemwell, i wasted most of my day on this 500 novalidhost thing,23:09
mriedemnow i know i can create in 100 chunks23:09
dansmithand we know we're not regressing the performance of the infra jobs that have run against this23:09
mriedemso i'm at the point that i've got 500 in cell0 and 500 in cell1, and going to get numbers on those23:09
openstackgerritMatt Riedemann proposed openstack/nova master: Log instance uuid when retrying claims in the scheduler  https://review.openstack.org/50770523:12
openstackgerritMoshe Levi proposed openstack/nova master: Don't overwrite binding-profile  https://review.openstack.org/50561323:16
*** hongbin has quit IRC23:22
openstackgerritEd Leafe proposed openstack/nova-specs master: Return Selection Objects  https://review.openstack.org/49883023:24
dansmithI'm actually not sure why we'd be hitting concurrent updates during an allocation event,23:25
dansmithgiven that we're not passing rp_generation23:25
*** sree has joined #openstack-nova23:26
dansmitheither the allocation fits or doesn't23:26
melwitthm, yeah23:27
dansmithit must be that placement doesn't hide transaction commits from us23:27
dansmithbased on the commit that added it23:27
*** chyka has joined #openstack-nova23:28
dansmithwhich would just be "tons of churn on one provider" as the reason23:28
melwittwhat does that mean? placement not hiding transaction commits23:29
*** sree has quit IRC23:30
dansmithan allocation will increment the generation of the provider on the placement side23:31
dansmithand a transaction will abort if we try to do it at the same time as something else23:31
melwittah, right. that's what I was thinking of23:31
dansmithplacement should really retry those for us server-side I would think, for an allocation type request23:31
melwittI couldn't remember where the RP generation was related there23:31
*** chyka has quit IRC23:32
dansmithdoesn't matter, but I would think it would be better, especially given the advice for that error is "try exactly the same thing again"23:32
melwittyeah, I was thinking the same23:32
mriedemthis was the error from the server23:32
mriedemThere was a conflict when trying to complete your request.\n\n Inventory changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update23:32
mriedemi'm not sure why inventory would change23:32
mriedemthat's static in this case23:32
*** smatzek has quit IRC23:32
dansmithany allocation will change the generation23:33
mriedemthe update_available_resource periodic will post inventory, but only if it changes23:33
dansmithso any two allocations can conflict23:33
melwittI'm +1 on the idea of retrying server-side23:33
*** takashin has joined #openstack-nova23:34
melwittI'm a little worried how many conflicts can we get in real life by ppl trying to create 500 servers and the scheduler is doing a "pack" pattern23:34
melwittas far as how to tune how many retries23:35
melwittto allow23:35
dansmiththis is also a highly synthetic scenario with a "virt driver" that doesn't get looked at much.. it could be doing something to re-stab inventory for no reason or something23:35
mriedemfor each instance, we go through the filters23:36
*** acormier has joined #openstack-nova23:36
mriedemand doesn't the scheduler have some kind of tracking on the HostState objects themselves for chosen hosts?23:36
edleafemriedem: that error message is poorly worded. It should be something like "available inventory has changed"23:36
melwittoh, the "inventory changed" yeah, I don't know anything about that. so that means it wasn't an allocation writing conflict?23:36
dansmithmelwitt: it's al related23:36
dansmithedleafe: that's not really accurate, AFACT, since allocations will cause the generation to increase23:37
dansmithedleafe: and thus it could be nothing changed with inventory to cause that23:37
edleafedansmith: uh, that's why I added "available".23:37
edleafedansmith: IOW, some inventory has been allocated23:37
mriedemthis is basically the inventory reporting for the fake driver https://github.com/openstack/nova/blob/master/nova/virt/fake.py#L11123:37
edleafeand changed the generation23:37
dansmithedleafe: okay, I wouldn't word it that way for clarity, but okay :)23:38
edleafeI wouldn't word it that way either, but I was guessing the author's intent23:38
edleafe*cough* cdent *cough*23:39
dansmithmriedem: that's the info that we use to generate it, yes23:39
melwittyeah, if it can happen without inventory (total possible capacity) changing, then that error message is confusing to me23:39
edleafemelwitt: yeah, it's worded very poorly23:40
dansmithmriedem: I'd look to see if the compute is hitting placement /inventory ever after the first go, and maybe check the nothing-changed short-circuit to make sure we're never going through it23:40
*** acormier has quit IRC23:40
mriedemthe inventory nothing changed?23:40
melwittI dunno, I think I know just enough for it to be confusing. for an end user, it might not be confusing23:40
dansmithmriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L571-L57223:40
*** NightKhaos has quit IRC23:40
mriedemmelwitt: end user won't see it, they'll see NoValidHost on 500 instances23:40
mriedemthe operator will see it23:41
melwittgood point23:41
dansmithmriedem: or just look at placement logs to see if inventory is hit any time after compute startup23:41
*** smatzek has joined #openstack-nova23:41
*** smatzek_ has joined #openstack-nova23:43
*** Swami has quit IRC23:46
mriedemgdi, how do i regex search with grep23:46
mriedemsudo journalctl -a -u devstack@placement-api.service | grep '.*PUT.*\/inventories.*'23:47
*** smatzek has quit IRC23:47
*** NightKhaos has joined #openstack-nova23:47
dansmithPUT.*invent should be all you need23:48
*** penick has joined #openstack-nova23:48
mriedemdoesn't work23:49
mriedemah, well,23:51
mriedemPUT.*alloc works23:51
mriedemso it probably just wrapped23:51
mriedemand it's not updating inventory, as it shouldn't23:51
mriedemgot my 1000 instances now, so will do the test stuff once i'm done with dinner23:52
dansmithso I'd also check to make sure compute isn't doing the ocata healing during boot or something like that23:53
*** penick has quit IRC23:53
*** chyka has joined #openstack-nova23:54
*** penick has joined #openstack-nova23:55
takashinSpec cores, could you review https://review.openstack.org/#/c/489029/ ? It got one +2.23:57
*** vladikr has joined #openstack-nova23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!