Friday, 2018-08-31

« Thursday, 2018-08-30 Index Saturday, 2018-09-01 »
*** brinzhang has joined #openstack-nova00:04
*** mtreinish has quit IRC00:07
*** mtreinish has joined #openstack-nova00:08
*** lbragstad has quit IRC00:12
*** lbragstad has joined #openstack-nova00:14
*** harlowja has quit IRC00:34
*** Sundar has quit IRC00:41
*** erlon has quit IRC00:47
*** hongbin has joined #openstack-nova01:01
*** mriedem_lawnboy is now known as mriedem01:04
*** mriedem has quit IRC01:10
*** imacdonn has quit IRC01:19
*** imacdonn has joined #openstack-nova01:19
*** Dinesh_Bhor has joined #openstack-nova01:33
*** erlon has joined #openstack-nova01:38
*** bhagyashris has joined #openstack-nova01:46
*** munimeha1 has quit IRC01:49
*** erlon has quit IRC01:49
*** lei-zh has joined #openstack-nova01:52
*** dpawlik has joined #openstack-nova02:18
*** dpawlik has quit IRC02:22
*** lei-zh has quit IRC02:31
*** lei-zh1 has joined #openstack-nova02:31
*** psachin has joined #openstack-nova02:55
*** bhagyashris has quit IRC03:04
*** moshele has joined #openstack-nova03:11
*** moshele has quit IRC03:11
openstackgerritLeopard Ma proposed openstack/nova master: Replace assertRaisesRegexp with assertRaisesRegex  https://review.openstack.org/59841503:15
openstackgerrithuanhongda proposed openstack/nova master: Fix instance delete stuck in deleting task_state  https://review.openstack.org/59808403:21
*** lei-zh1 has quit IRC03:32
*** nicolasbock has quit IRC03:33
openstackgerritBrin Zhang proposed openstack/nova-specs master: Resource retrieving: add changes-before filter  https://review.openstack.org/59197603:44
*** udesale has joined #openstack-nova03:59
*** jaypipes has quit IRC04:04
*** jaypipes has joined #openstack-nova04:04
*** dave-mccowan has quit IRC04:05
*** lei-zh1 has joined #openstack-nova04:06
*** lei-zh1 has quit IRC04:11
*** hamzy has joined #openstack-nova04:15
*** Bhujay has joined #openstack-nova04:18
*** links has joined #openstack-nova04:29
*** tbachman has quit IRC04:43
*** tbachman has joined #openstack-nova04:43
*** jchhatbar has joined #openstack-nova04:49
*** jchhatbar has quit IRC05:03
*** janki has joined #openstack-nova05:09
*** hongbin has quit IRC05:12
*** janki has quit IRC05:23
*** bhagyashris has joined #openstack-nova05:24
*** lei-zh1 has joined #openstack-nova05:46
*** Luzi has joined #openstack-nova05:51
*** psachin has quit IRC06:00
*** ivve has quit IRC06:02
*** ivve has joined #openstack-nova06:14
*** dpawlik has joined #openstack-nova06:18
*** holser_ has joined #openstack-nova06:20
*** Bhujay has quit IRC06:20
*** dpawlik has quit IRC06:22
*** Dinesh_Bhor has quit IRC06:23
openstackgerritLeopard Ma proposed openstack/nova master: Replace assertRaisesRegexp with assertRaisesRegex  https://review.openstack.org/59841506:25
*** janki has joined #openstack-nova06:27
*** holser_ has quit IRC06:29
*** Dinesh_Bhor has joined #openstack-nova06:32
*** ccamacho has joined #openstack-nova06:43
*** ccamacho has quit IRC06:44
*** alexchadin has joined #openstack-nova06:46
*** ccamacho has joined #openstack-nova06:52
*** ccamacho has quit IRC06:57
*** ccamacho has joined #openstack-nova06:58
*** lei-zh1 has quit IRC06:58
*** lei-zh1 has joined #openstack-nova06:59
*** rcernin has quit IRC07:00
*** luksky has joined #openstack-nova07:04
*** tssurya has joined #openstack-nova07:12
*** sahid has joined #openstack-nova07:18
*** pcaruana has joined #openstack-nova07:19
*** kosamara has quit IRC07:19
*** bhagyashris has quit IRC07:24
*** janki has quit IRC07:25
*** bjolo has joined #openstack-nova07:26
*** dpawlik has joined #openstack-nova07:28
*** holser_ has joined #openstack-nova07:29
*** holser_ has quit IRC07:30
*** holser_ has joined #openstack-nova07:30
*** jpena|off is now known as jpena07:31
*** janki has joined #openstack-nova07:32
*** dpawlik has quit IRC07:32
*** dpawlik has joined #openstack-nova07:38
*** Bhujay has joined #openstack-nova07:41
*** dpawlik has quit IRC07:46
*** dpawlik_ has joined #openstack-nova07:46
*** jchhatbar has joined #openstack-nova07:55
openstackgerritChen proposed openstack/nova master: Fix SOFT_DELETE vm_state string  https://review.openstack.org/59848207:56
*** janki has quit IRC07:57
*** alexchadin has quit IRC07:57
*** luksky has quit IRC07:58
*** kosamara has joined #openstack-nova08:06
*** alexchadin has joined #openstack-nova08:07
*** Bhujay has quit IRC08:17
*** luksky has joined #openstack-nova08:28
*** hshiina has joined #openstack-nova08:35
openstackgerritBrin Zhang proposed openstack/nova-specs master: Resource retrieving: add changes-before filter  https://review.openstack.org/59197608:45
openstackgerritBrin Zhang proposed openstack/nova-specs master: Resource retrieving: add changes-before filter  https://review.openstack.org/59197608:48
*** ttsiouts has joined #openstack-nova08:49
*** alexchadin has quit IRC09:01
*** dpawlik_ has quit IRC09:04
openstackgerritBalazs Gibizer proposed openstack/nova master: Consumer gen support for delete instance allocations  https://review.openstack.org/59159709:06
*** dtantsur|afk is now known as dtantsur09:07
*** Sigyn has quit IRC09:07
*** Sigyn has joined #openstack-nova09:08
*** dpawlik has joined #openstack-nova09:11
*** alexchadin has joined #openstack-nova09:15
*** lei-zh1 has quit IRC09:21
*** moshele has joined #openstack-nova09:27
*** gmann has quit IRC09:27
*** hamzy_ has joined #openstack-nova09:37
*** hamzy has quit IRC09:37
*** stakeda has quit IRC09:41
*** gmann has joined #openstack-nova09:43
openstackgerritChen proposed openstack/nova master: Fix filter server list with SOFT_DELETED status  https://review.openstack.org/59744309:45
*** brinzhang has quit IRC09:45
*** trungnv has quit IRC09:52
*** priteau has joined #openstack-nova09:53
*** bzhao__ has joined #openstack-nova09:53
*** alexchadin has quit IRC09:53
*** dpawlik has quit IRC09:56
*** dpawlik_ has joined #openstack-nova09:56
-openstackstatus- NOTICE: Jobs using devstack-gate (legacy devstack jobs) have been failing due to an ara update. We use now a newer ansible version, it's safe to recheck if you see "ImportError: No module named manager" in the logs.09:57
*** tbachman_ has joined #openstack-nova10:06
*** tbachman has quit IRC10:07
*** tbachman_ is now known as tbachman10:07
*** alexchadin has joined #openstack-nova10:07
*** moshele has quit IRC10:09
*** ttsiouts has quit IRC10:09
openstackgerritMerged openstack/nova master: Delete instance_group_member records from API DB during archive  https://review.openstack.org/58894310:13
*** hoangcx has quit IRC10:16
*** Dinesh_Bhor has quit IRC10:17
*** tbachman_ has joined #openstack-nova10:18
*** tbachman has quit IRC10:19
*** tbachman_ is now known as tbachman10:19
*** cdent has joined #openstack-nova10:21
*** Bhujay has joined #openstack-nova10:21
*** Bhujay has quit IRC10:26
dr_gogeta86hi10:31
dr_gogeta86there is any known tecnique to migrate VMs with high memory pressure10:31
dr_gogeta86i can't evaquate a node for some machine who uses a lot of ram10:32
*** nicolasbock has joined #openstack-nova10:33
*** dave-mccowan has joined #openstack-nova10:35
*** tbachman_ has joined #openstack-nova10:41
*** tbachman_ has quit IRC10:42
*** tbachman has quit IRC10:43
*** Dinesh_Bhor has joined #openstack-nova10:45
*** stephenfin is now known as finucannot10:46
*** erlon has joined #openstack-nova10:49
gibidr_gogeta86: I suggest to read the spec https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/auto-live-migration-completion.html10:52
*** gibi is now known as giblet10:53
*** alexchadin has quit IRC10:53
dr_gogeta86tnx10:54
*** alexchadin has joined #openstack-nova10:57
gibletdr_gogeta86: and here some final documentation for the features proposed in that spec https://docs.openstack.org/nova/latest/admin/configuring-migrations.html#advanced-configuration-for-kvm-and-qemu11:10
*** ttsiouts has joined #openstack-nova11:12
*** udesale has quit IRC11:12
*** alexchadin has quit IRC11:28
*** jpena is now known as jpena|lunch11:28
openstackgerritRadoslav Gerganov proposed openstack/nova master: Do not dump all instances in the scheduler  https://review.openstack.org/59861111:38
*** alexchadin has joined #openstack-nova11:39
*** alexchadin has quit IRC11:39
*** alexchadin has joined #openstack-nova11:40
*** alexchadin has quit IRC11:40
*** alexchadin has joined #openstack-nova11:58
*** tonyb has quit IRC12:02
*** tbachman has joined #openstack-nova12:03
*** tbachman has quit IRC12:07
openstackgerritsahid proposed openstack/nova master: hardware: fix memory check usage for small/large pages  https://review.openstack.org/53216812:13
sahidfinucannot: ^12:14
*** tbachman has joined #openstack-nova12:15
*** brinzhang has joined #openstack-nova12:18
*** tonyb has joined #openstack-nova12:19
*** jpena|lunch is now known as jpena12:23
*** jaypipes is now known as leakypipes12:44
finucannotsahid: ack12:49
*** cdent has quit IRC12:55
*** tssurya has quit IRC12:59
*** Hazelesque has quit IRC13:03
*** alexchadin has quit IRC13:09
*** Hazelesque has joined #openstack-nova13:10
*** eharney has joined #openstack-nova13:23
*** ttsiouts has quit IRC13:24
sahidfinucannot, leakypipes thanks13:26
leakypipessahid: de rien, mon ami13:29
*** ttsiouts has joined #openstack-nova13:30
*** bnemec is now known as beekneemech13:31
sahid:-)13:31
*** alexchadin has joined #openstack-nova13:34
*** alexchadin has quit IRC13:34
*** alexchadin has joined #openstack-nova13:35
*** alexchadin has quit IRC13:35
*** alexchadin has joined #openstack-nova13:35
*** alexchadin has quit IRC13:36
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Delete instance_group_member records from API DB during archive  https://review.openstack.org/59895313:36
*** alexchadin has joined #openstack-nova13:36
*** alexchadin has quit IRC13:36
*** alexchadin has joined #openstack-nova13:37
*** lbragstad is now known as elbragstad13:37
*** poojajadhav has joined #openstack-nova13:38
*** efried is now known as fried_rice13:38
poojajadhavfried_rice: Hi13:39
fried_ricepoojajadhav: Greetings.13:39
poojajadhavfried_rice: Can u help in simple tenant usage API13:39
*** luksky has quit IRC13:40
fried_ricepoojajadhav: whoah. I'm certainly not the best resource for that, but I can see what I can do (read: who I can find who knows something about it).13:40
poojajadhavfried_rice: Actually, I have started looking into the bug  : https://bugs.launchpad.net/nova/+bug/171557013:41
openstackLaunchpad bug 1715570 in OpenStack Compute (nova) "simple tenant usage api calculating disk usages incorrectly" [Undecided,In progress] - Assigned to Bhagyashri Shewale (bhagyashri-shewale)13:41
poojajadhavfried_rice: To resolve the issue of the usages of the (disk, memory_mb, vcpu) I am trying to get usages from placement usage api and then will use that infor for simple tenantr usage api. But facing issue  when getting palcement usages.13:43
poojajadhavfried_rice: Getting errot like  : [dict(resource_class_id=item[0], usage=item[1])for item in query.all()] *** ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova_cell0.consumers' doesn't exist") [SQL: u'SELECT allocations.resource_class_id AS allocations_resource_class_id, coalesce(sum(allocations.used), %(coalesce_2)s) AS coalesce_1 \nFROM allocations INNER JOIN consumers ON allocations.consumer_id = cons13:43
fried_riceleakypipes: ^ ?13:44
fried_ricepoojajadhav: Wait, how are you trying to retrieve the usages?13:45
fried_ricevia the placement API or by talking directly to "the database"?13:45
*** mriedem has joined #openstack-nova13:45
openstackgerritBalazs Gibizer proposed openstack/nova master: Consumer gen support for put allocations  https://review.openstack.org/59164713:45
poojajadhavfried_rice: same like done in placement https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/handlers/usage.py#L104-L10913:46
leakypipesfried_rice: simple tenant usage unfortunately is a pile of legacy and doesn't query placement at all.13:47
*** ttsiouts has quit IRC13:47
leakypipesfried_rice: it's all looking at instances, compute_nodes table etc13:47
leakypipesin the nova cell DBs.13:47
mdboothpip's giving me an ssl error (CERTIFICATE_VERIFY_FAILED) trying to download https://files.pythonhosted.org/packages/2a/fd/2a8b894ee3451704cf8525a6a94b87d5ba24747b7bbd3d2f7059189ad79f/stestr-2.1.1.tar.gz on F28. Chrome thinks it's fine. Anybody seen anything similar?13:47
*** alex_xu has quit IRC13:47
fried_ricepoojajadhav: You definitely shouldn't be attempting to access the placement database directly from anywhere other than within the placement API code itself. If you're trying to get placement information from nova, you would need to do it by talking to the placement API service through the REST interface, via the SchedulerReportClient.13:48
poojajadhavfried_rice, leakypipes: If the instance is BFV then also its shwoing the flavor,root_gb as per https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/simple_tenant_usage.py#L165-L16813:49
fried_ricepoojajadhav: I thought we had some patches already proposed that were looking at fixing this...13:50
dansmithleakypipes: despite the legacyness, placement doesn't provide the data it would need anyway13:50
mriedemthe is_bfv fix doesn't extend to the simple tenant usage API right?/13:51
mriedemsince the API doesn't look at the request spec13:52
poojajadhavfried_rice : But in the simple tenant usage api, we can get flavors details and falvor is exists in nova_api only.13:52
*** alex_xu has joined #openstack-nova13:53
dansmithmriedem: unrelated, but .. haven't we already had a discussion about allowing delete-on-termination to be provided in attach volume?13:53
dansmithI found a very old blueprint and code from like 2013, but couldn't find anything more recent13:53
dansmithand I thought we had (and you were opposed)13:54
mriedemdansmith: someone was asking for adding delete_on_termination for ports during server create to match bdms, but not sure about adding that flag during attach13:54
mdboothdansmith: It does seem like a kinda weird thing to do, tbh. Also, how would that work with multi-attach?13:54
mriedemi agree it would be weird13:54
dansmithmdbooth: mriedem: yeah, I think it violates the "nova created it so nova deleted it" rule13:55
mriedemdelete_on_termination, to me, makes the most sense when nova creates the resoruce13:55
dansmithand could be quite surprising13:55
poojajadhavfried_rice : Can u please locate me to those patches13:55
fried_ricepoojajadhav: I'm probably thinking of something different.13:55
dansmithmriedem: anyway i was sure you had already thrown down the hammer on this previously so I was just looking for a shortcut13:55
leakypipesdansmith: ack13:55
mriedempoojajadhav: https://github.com/openstack/nova/commit/03c596a9f4324e572bc04d4bbad09a6d3d47366c#diff-cbbdc4d7c140314a7e0b2d97ebcd1f9c ?13:56
mriedemdansmith: b/c today is friday red hat bug squash day and you have a customer that wants this ability?13:56
dansmithmriedem: I told you never to call me here13:56
mriedemand you can say "ol man riderman doesn't like it"13:56
dansmithmriedem: no, I was hoping for a link to an ML thread where "ol man riderman said no"13:57
mriedemfor that you'd have to dig :)13:57
dansmithwell, I did13:57
*** giblet is now known as giblet_off13:57
mriedemi wouldn't be surprised if it came up13:57
leakypipesI think we'd need hansmoleman for that.13:57
*** mlavalle has joined #openstack-nova13:58
mriedemhttps://review.openstack.org/#/c/89777/13:58
poojajadhavmriedem: I have checked on current master, in the usages for BFV instance it shows local_gb=1.13:58
mriedemthere are -1s in that spec from not me13:58
dansmithmriedem: just procedural13:58
dansmithI already found that and the conversation on the ML about it13:59
mriedempoojajadhav: yeah, because the simple tenant usage API doesn't take into account whether or not an instance is volume-backed13:59
mriedemas i said above13:59
dansmithnobody seemed to actually discuss it13:59
mriedemphil day was -1 onn it13:59
mriedemfor the reasons we just said13:59
mriedem"I think the option for delete on termination when booting an instance is a hang over from the original boot from volume work where Nova would create a new (in effect ephemeral) volume from an image in Glance.  I'm not sure that use case really exist anymore - but even if it does it doesn't seem like it should be extended to more general cases."14:00
mriedemdoesn't exist anymore....oh phil14:00
mriedemheh jay was +1 on it14:00
mriedem2014 jay loved bfv!14:00
poojajadhavmriedem: but should we consider instance is volume-backed or not for simple tenant usage api?14:01
mriedempoojajadhav: yes we likely should14:01
melwitt.14:01
mriedempoojajadhav: you could start by reporting a bug, related to bug 1469179 and mention the issue14:01
openstackbug 1469179 in OpenStack Compute (nova) "instance.root_gb should be 0 for volume-backed instances" [Medium,Fix released] https://launchpad.net/bugs/1469179 - Assigned to Dan Smith (danms)14:01
leakypipesmriedem: I must have mistaken it to mean delete bfv.14:01
mriedemleakypipes: heh14:01
poojajadhavmriedem: bug is already reported https://bugs.launchpad.net/nova/+bug/171557014:02
openstackLaunchpad bug 1715570 in OpenStack Compute (nova) "simple tenant usage api calculating disk usages incorrectly" [Undecided,In progress] - Assigned to Bhagyashri Shewale (bhagyashri-shewale)14:02
*** gbarros has joined #openstack-nova14:03
mriedempoojajadhav: ok14:06
mriedemi'm not really sure how we could efficiently calculate volume-backed for all servers in the simple tenant usage API call - since it's not a flag on the instance itself14:06
mriedemunless we started reporting that into the instance system_metadata or something14:07
mriedemdansmith: any thoughts on that?14:07
*** Luzi has quit IRC14:07
*** ttsiouts has joined #openstack-nova14:08
dansmithmriedem: yeah, we'd need a new element of data I think to be able to do it at all efficiently14:10
dansmithsysmeta would not be that thing though14:10
poojajadhavmriedem: I was thought like currently placement usage api giving correct usages so we can get them and use it for the simple tenant usage api14:10
dansmithpoojajadhav: that's not what simple tenant usage wants or needs14:10
dansmithit needs historical data14:10
poojajadhavdansmith: hmm14:10
mriedemsimple tenant usage is asking for an aggregate usage of resources for a given tenant over a specific time period14:11
mriedemplacement doesn't expose consumer type (instance or volume or container) nor created_at time14:12
dansmithnor keep anything after deletion14:12
dansmithi.e. no historical archive14:12
mriedemright14:12
mriedemwhen a record is deleted in placement, it's gone14:12
mriedemit's not put in a box in the basement like in nova14:12
* mriedem eagerly awaits the placement specs for adding all of this into placement now...14:13
leakypipesmriedem: gonna have to wait until the glacier unfreezes I'm afraid. /me goes back to reviewing nova-specs14:14
*** mriedem has quit IRC14:14
*** mriedem has joined #openstack-nova14:15
poojajadhavmriedem: I was trying to fix it.. but which can be then efficient way now :(14:16
mriedemwell, system_metadata isn't great14:19
mriedemdansmith: thinking out loud, how crazy would it be to have a joinedload field in the db api when we get an instance that adds a boolean attribute to the sqla instance object that queries the bdms table for that instance to see if it's volume-backed, i.e. (1) bdm exists for the instance (2) boot_index=0, (3) destination_type='volume' ?14:20
mriedemessentially nova.compute.utils.is_volume_backed_instance but in the db api14:21
dansmithare those all columns?14:22
dansmitheven if they are, I dunno if you can easily synthesize a field on an ORM object or not14:23
*** links has quit IRC14:24
mriedemwe do that in other places, like what we had for instance_group.policies14:24
mriedemdefinitely not something i'm going to spend a lot of time thinking about today14:25
mriedemi'll throw it in the bug as an idea14:25
*** cdent has joined #openstack-nova14:26
*** hamzy_ is now known as hamzy14:26
dansmithwe synthesize something with a subquery?14:27
mriedemwell i was thinking stuff like htis14:29
mriedem*this14:29
mriedem_policies = orm.relationship(InstanceGroupPolicy,14:29
mriedem            primaryjoin='InstanceGroup.id == InstanceGroupPolicy.group_id')14:29
mriedemor this from the services table:14:30
mriedeminstance = orm.relationship(14:30
mriedem        "Instance",14:30
mriedem        backref='services',14:30
mriedem        primaryjoin='and_(Service.host == Instance.host,'14:30
mriedem                    'Service.binary == "nova-compute",'14:30
mriedem                    'Instance.deleted == 0)',14:30
mriedem        foreign_keys=host,14:30
mriedem    )14:30
mriedemthat's how we get the instance.services column right?14:30
mriedemcalling it a column isn't really appropriate i suppose14:30
mriedemmodel attribute14:30
dansmiththat's a join so we get the right thing,14:33
dansmiththat just gets us a list of matching records, not a boolean calculated,14:33
*** zzzeek has quit IRC14:33
dansmithbut we could turn it into a boolean on the nova object if there's a bdm in that list I guess14:34
*** zzzeek has joined #openstack-nova14:34
dansmithdo we keep the bdms after instance deletion? I think we use the flavor data for that usage info, so we'd need the bdms to still be there on deleted instances for that to work properly14:34
mriedemthe bdms are soft deleted just like the instance14:35
dansmithokay14:35
*** moshele has joined #openstack-nova14:38
openstackgerritMerged openstack/nova master: api-ref: add a warning about calling swap volume directly  https://review.openstack.org/59649214:38
*** zzzeek has quit IRC14:40
*** alexchadin has quit IRC14:41
*** hshiina has quit IRC14:42
*** zzzeek has joined #openstack-nova14:42
*** jpena is now known as jpena|off14:42
*** jpena|off is now known as jpena14:43
*** jaosorior has quit IRC14:43
*** zzzeek has quit IRC14:46
*** zzzeek has joined #openstack-nova14:47
mriedembauzas: see my todo/question in https://review.openstack.org/#/c/598365/1/nova/compute/resource_tracker.py - i'm assuming this is something operators may do in case they need to 'reset' allocation ratios on computes in the db back to the default14:48
mriedemi.e. i set a config value for the allocation ratio, then want to get back to defaults, so remove my custom config and restart nova-compute and expect the compute to report the default allocation ratio14:48
bauzasmriedem: ack, I need to go out, but I'll look at it14:50
bauzasmriedem: mmm, about your question14:51
bauzasmriedem: I think we can say something in a reno note that in case they want to use the default values, they should pass 16.0 for example14:52
bauzassome upgrade section14:52
bauzassetting it back to 0.0 looks insane to me14:52
bauzasor another way to fix the problem would be to use the object method there14:53
mriedemif we leave it and doc it anywhere, it should be in the option help,14:53
mriedemrelease notes aren't read and are lost in the sands of time once you FFU14:53
bauzasright, I meant in both a relnote and the help14:53
mriedemneed to wrap my head around this a bit myeslf14:53
*** poojajadhav has quit IRC14:53
mriedem*myself14:53
bauzasor I need to think about just modifying the conf default value14:54
bauzaswith your change too14:54
mriedemmy change needs to be backported so we wouldn't do that14:54
mriedemin the same change14:54
bauzasI mean, both fixing the RT *and* changing the default values14:54
bauzasI need to go out but I'll think about it14:55
dansmithman zuul is backed up real bad14:57
* dansmith offers it some fiber14:57
mriedemdansmith: yeah https://bugs.launchpad.net/openstack-gate/+bug/179015214:58
openstackLaunchpad bug 1790152 in OpenStack-Gate "jobs failing with ansible "ImportError: No module named manager" since 8/30" [Undecided,Fix released]14:58
mriedemalready fixed but broke everything last night14:58
dansmithyeah, but are we still on that backlog?15:00
dansmithat 2am this morning they said "go ahead and recheck" -- maybe it's just the influx of everyone doing that/15:00
*** tbachman has quit IRC15:02
mriedemi assume so yes - everyone woke up and started rechecking the world15:02
*** brinzhang has quit IRC15:02
openstackgerritMerged openstack/nova master: Remove old check_attach version check in API  https://review.openstack.org/58834815:02
mriedem^ feels good to have that gone15:04
*** ccamacho has quit IRC15:12
openstackgerritMatt Riedemann proposed openstack/nova master: Don't persist zero allocation ratios in ResourceTracker  https://review.openstack.org/59836515:14
*** tbachman has joined #openstack-nova15:17
cdentsean-k-mooney: in case you're following along at home my car failed its mot rather spectacularly15:22
*** moshele has quit IRC15:24
sean-k-mooneycdent: i was actully wondering about that15:25
*** sahid has quit IRC15:26
*** macza has joined #openstack-nova15:26
cdenti'll find out the details in a few minutes when I go to pick it up. the parts it needs may be impossible to get.15:26
sean-k-mooneycdent: fixable or not viable15:26
*** pcaruana has quit IRC15:26
cdentwell, fixable with a) much money, b) access to parts. Both require some effort to resolve.15:27
*** jchhatbar has quit IRC15:27
sean-k-mooneyya what is it by the way. you mentioned it was 21 years old but not the make or modle15:27
*** jlvillal is now known as jlvilla-viva15:30
*** tbachman has quit IRC15:33
*** ttsiouts has quit IRC15:38
*** ttsiouts has joined #openstack-nova15:39
*** tbachman has joined #openstack-nova15:39
*** erlon has quit IRC15:41
*** tbachman_ has joined #openstack-nova15:42
*** ttsiouts has quit IRC15:43
*** dpawlik_ has quit IRC15:44
*** tbachman has quit IRC15:44
*** tbachman_ is now known as tbachman15:44
*** munimeha1 has joined #openstack-nova15:56
*** gbarros has quit IRC15:58
*** r-daneel has joined #openstack-nova16:01
*** jpena is now known as jpena|off16:03
*** openstackgerrit has quit IRC16:06
*** holser_ has quit IRC16:07
*** gbarros has joined #openstack-nova16:21
*** mriedem is now known as mriedem_away16:21
*** fried_rice is now known as fried_rolls16:27
sean-k-mooneymelwitt: finucannot  since sahid was fine https://review.openstack.org/#/q/topic:bug/1759420+(status:open+OR+status:merged) and melwitt mentioned to not squash unless i respin are they good to go?16:34
melwittsean-k-mooney: probably. I'd ask mriedem_away if he thinks squashing is important but he's away for atm16:35
finucannotsean-k-mooney: Sure. I've +2d the bottom one but, in hindsight, I might leave both to melwitt seeing as she got there first (assuming that's alright with her)16:35
melwittfinucannot: I'm fine with you taking them, I can +1 them anyway. I want to ask mriedem_away about squashing since it's related to the backport16:37
finucannotmelwitt: Yeah, it can wait until next week, in that case16:37
melwittwe routinely backport 2-change deals for functional regression tests, but this is a unit regression test, just want to make sure with him since I'm not sure it matters16:38
sean-k-mooneysure just comment or ping me if you want it squashed. its like 30 seconds to do but the gate is busy at the moment so did not want to push a change for no reason16:38
melwittok, I'll let you know16:38
*** r-daneel has quit IRC16:38
*** r-daneel has joined #openstack-nova16:38
*** gbarros has quit IRC16:47
*** Swami has joined #openstack-nova16:48
*** mdbooth has quit IRC16:56
*** hoonetorg has joined #openstack-nova17:07
*** cfriesen has joined #openstack-nova17:23
*** cfriesen has quit IRC17:27
*** cfriesen has joined #openstack-nova17:28
*** cfriesen has quit IRC17:31
*** cfriesen has joined #openstack-nova17:31
hamzyhey y'all, my ethernet connections in instances installed via TripleO doesn't work.  I seem to see all the bridges and connections setup, but some final qemu hardware connection is broken or missing http://paste.openstack.org/show/729241/17:43
sean-k-mooneyhamzy: that all seams fine at first glance17:44
sean-k-mooneyhamzy: how did you determihn that the connectivity did not work17:44
hamzysean-k-mooney, on lines 78+ I can't get the IP via DHCP, or when I set it manually, ping the router17:45
sean-k-mooneyhamzy: is dhcp enable on the neutron subnet17:48
cfriesenwho's a good person to talk to about performance issues?17:48
cfriesenI mean process startup times17:48
sean-k-mooneycfriesen: as in makeing them better or debuging a problem17:49
sean-k-mooneycfriesen: if your debugging a proablem i proably cant help but if you have a proposal on how to make it faster then im happy to be a sound board to bounce ideas off17:50
cfriesensean-k-mooney: in nova/compute/flavors.py we call "from nova.api.validation import parameter_types".  This appears to be really expensive (~6 seconds in a recent test) due to the regex stuff, which makes a controller node startup fairly expensive17:50
hamzysean-k-mooney, yes I believe that it is... all ports on 'openstack port list' are active17:50
*** mriedem_away is now known as mriedem17:50
sean-k-mooneycfriesen: ocuch 6 seconds for a singel function call for paramater validation is nuts17:51
sean-k-mooneyhamzy: yes they will be active regardless of if dhcp is enable on the subnet17:52
cfriesensean-k-mooney: that was in vbox, but it's still multiple seconds on bare metal.  internally we moved the import down into the actual code so it's only done if we actually do the flavor creation17:52
sean-k-mooneythey also will have ips assigned17:52
sean-k-mooneycfriesen: wait the import is executing code? is it building the regex on import or something if so that is terible17:53
hamzysean-k-mooney, http://paste.openstack.org/show/729253/17:53
hamzyis the DHCP agent only run on the controller?17:54
sean-k-mooneycfriesen: sigh... https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L144-L492 its all at file scope we do this on every import17:54
cfriesensean-k-mooney: precisely17:55
sean-k-mooneycfriesen: can you open a bug for that. i would have to check what this actully used for but personally i think we should lazy initallies the values or replace them with caching factoryFunctions17:56
cfriesensean-k-mooney: sure17:56
sean-k-mooneythe simplest thing to do is proably make them all properties but not sure that is leagal at file/module scope17:57
sean-k-mooneyhamzy: yes it should only be running on the contoller/networking nodes17:57
hamzysean-k-mooney, I would think that "sudo ip addr add ..." and ping .254 (router) should work as a test for network connectivity, right?17:58
sean-k-mooneyhamzy: can you do an "openstack subnet list" and then "openstack subnet show" for the subnet the vm is connected too17:58
leakypipescfriesen, sean-k-mooney: I'm actually only seeing a single re.compile() in that module...17:58
leakypipescfriesen, sean-k-mooney: It would seem like it would be more efficient to be re.compile()'ing all the regexes that are in module scope in that module.17:59
*** dtantsur is now known as dtantsur|afk17:59
sean-k-mooneyleakypipes: actully i had assumed the ValidationRegex object was compiling them but i see now its not18:00
leakypipessean-k-mooney: no it is not...18:01
leakypipessean-k-mooney: or at least, it isn't AFAICT18:01
sean-k-mooneyya looking over the file level stuff its not really doing anything expecive18:02
cfriesentry running python and then doing "from nova.api.validation import parameter_types"18:02
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/api/validation/__init__.py however might be18:02
sean-k-mooneyalso not really be i hate when people hide stuff in __init__.py files18:03
cfriesenjust retested and it's 3 secs on pretty beefy hardware18:04
sean-k-mooneycfriesen: and if you move the import to the fuction where its used what happens?18:05
cfriesensean-k-mooney: then you don't hit the delay at process startup, only when you actually run the function18:05
sean-k-mooneycfriesen: yes but what is the delta in start up18:06
sean-k-mooneyif its 3 second to 2.9 then do we care18:06
cfriesenno, it's 3 secs just for "from nova.api.validation import parameter_types"18:06
sean-k-mooneyoh that is different18:06
leakypipessean-k-mooney: well, there's no state in the dunderinit file at least... just functions.18:07
sean-k-mooneycfriesen: can you temporally comment out the file level suff just to see if it changes18:07
* leakypipes goes to benchmark what cfriesen is seeing18:07
*** luksky has joined #openstack-nova18:08
leakypipesholy fucksticks, Batman.18:09
leakypipes[jaypipes@uberbox nova]$ source .tox/py27/bin/activate18:09
leakypipes(py27) [jaypipes@uberbox nova]$ time python -m nova.api.validation.parameter_types18:09
leakypipesreal0m5.653s18:09
leakypipesuser0m3.331s18:09
leakypipessys0m0.201s18:09
sean-k-mooneycfriesen: the only thin that looks even remotely expensive is maybe _build_regex_range and even then the most expecive thing i see there is re.escape18:09
sean-k-mooneyi dont really like this https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L128-L14218:10
sean-k-mooneydef _get_all_chars():18:11
sean-k-mooney    for i in range(0xFFFF):18:11
sean-k-mooney        yield six.unichr(i)18:11
sean-k-mooneyso that is got to loop 65535 times18:11
sean-k-mooney*going too18:12
sean-k-mooneyand we call the function 17 times18:12
sean-k-mooneyso that 1.1 million callse to re.escape every time we load that module18:14
sean-k-mooneyleakypipes: cfriesen shall i fix this or do one of ye wnat to do it18:14
*** openstackgerrit has joined #openstack-nova18:15
openstackgerritMatt Riedemann proposed openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API  https://review.openstack.org/46967518:15
cfriesensean-k-mooney: go for it. :)18:15
sean-k-mooneycfriesen: good find by the way that is nuts18:16
*** moshele has joined #openstack-nova18:16
cfriesensean-k-mooney: https://bugs.launchpad.net/nova/+bug/179019518:17
openstackLaunchpad bug 1790195 in OpenStack Compute (nova) "performance problems starting up nova process due to regex code" [Undecided,New]18:17
leakypipescfriesen: ++18:18
*** priteau has quit IRC18:18
*** tbachman has quit IRC18:19
cfriesensean-k-mooney: I think that's likely the culprit.  Changing it to "for i in range(0x1):" made "python -m nova.api.validation.parameter_types"  take 1.5 sec instead of 7.18:21
mriedemhmm, so https://review.openstack.org/#/c/598366/ passed with https://review.openstack.org/#/c/598365/ but i'm not sure if that tells us much18:21
sean-k-mooneycfriesen: ill fix this two ways. first ill cache te result of _build_regex_range wit a decorator and second ill probably convert the file level vars to propertys so they are lazy evaulted18:21
cfriesensean-k-mooney: sounds good18:22
*** tbachman has joined #openstack-nova18:23
openstackgerritAdam Harwell proposed openstack/nova stable/pike: Add apply_cells to nova-manage  https://review.openstack.org/59905018:48
openstackgerritMerged openstack/nova master: Report client: update_from_provider_tree w/reshape  https://review.openstack.org/58504918:51
openstackgerritMerged openstack/nova master: Compute: Handle reshaped provider trees  https://review.openstack.org/57623618:51
openstackgerritMerged openstack/nova master: Do test_reshape with an actual startup  https://review.openstack.org/59721818:52
openstackgerritMerged openstack/nova master: Fix reshaper report client functonal test nits  https://review.openstack.org/59833018:52
cfriesenany chance of a second core looking at https://review.openstack.org/#/c/588657/ ?   It's a fairly straightforward change related to image properties, claims, and evacuate.18:55
*** tbachman has quit IRC18:58
*** openstackstatus has quit IRC18:58
mriedemthat is an excellent patch18:58
leakypipescfriesen: ack, I'll review shortly.19:01
cfriesenleakypipes: much appreciated19:07
leakypipescfriesen: you KNOW how much I love the instance migration/evacuate/shelve/funkychicken code paths.19:08
cfriesenjust the thing for a friday19:08
sean-k-mooneycfriesen: so the decorator alone went from 0m3.535s to 1.128s19:11
cfriesencool19:11
*** tbachman has joined #openstack-nova19:18
*** fried_rolls is now known as fried_rice19:23
leakypipescfriesen: +W19:33
cfriesenleakypipes: sweet, thx19:34
*** openstackstatus has joined #openstack-nova19:38
*** ChanServ sets mode: +v openstackstatus19:38
mriedemfried_rice: you wanted to know when i reported this https://bugs.launchpad.net/nova/+bug/179020419:43
openstackLaunchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [Medium,Triaged]19:43
mriedemcfriesen: ^ fyi19:43
dansmithmriedem: shall we? https://review.openstack.org/#/c/59835319:44
mriedemdansmith: see my comment on the test?19:44
dansmithoh, sure19:44
mriedemiow, if i remove the fix the test would still pass i think19:44
fried_ricemriedem: ack, thx19:44
dansmithmriedem: yep probably19:44
dansmithmriedem: the way the gate is that makes this probably tuesday before it's in19:45
dansmithI dunno what the fail rate is19:45
mriedemmy gut says the pass rate is not great19:45
dansmithfail rate?19:46
mriedembut this has also been this way since queens i think when i changed devstack to use [upgrade_levels]compute=auto so what's a few more days19:46
mriedemi don't actually know what the pass/fail rate is right now19:47
mriedemheh http://grafana.openstack.org/d/QBHIN5Smk/tempest-failure-rate?orgId=119:47
mriedemthese dashboards are all using old job names19:48
dansmithokay i thought this was spiking in the gate based on the discussion19:49
mriedemnot it's definitely not our worst failure19:50
dansmithack19:50
mriedemhttp://status.openstack.org/elastic-recheck/#178948419:50
openstackgerritDan Smith proposed openstack/nova master: Move conductor wait_until_ready() delay before manager init  https://review.openstack.org/59835319:50
mriedemit's not nothing either19:50
*** openstackstatus has quit IRC19:55
*** eharney has quit IRC19:57
*** openstackstatus has joined #openstack-nova19:58
*** ChanServ sets mode: +v openstackstatus19:58
*** cdent has quit IRC20:02
*** itlinux has joined #openstack-nova20:03
mriedemdansmith: +220:05
*** openstackstatus has quit IRC20:11
*** openstackstatus has joined #openstack-nova20:12
*** ChanServ sets mode: +v openstackstatus20:12
mriedemfried_rice: more details https://bugs.launchpad.net/nova/+bug/1790204/comments/1 - i expect reschedules would make any solution for that bug even messier20:13
openstackLaunchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [Medium,Triaged]20:13
fried_ricemriedem: ack. I'm subscribed.20:13
*** nicolasbock has quit IRC20:16
*** itlinux has quit IRC20:16
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Fix image-defined numa claims during evacuate  https://review.openstack.org/59906220:17
mriedemwe've got an easy gate fix here https://review.openstack.org/59835320:17
mriedemwould be good to get that in given the gate borkages of late20:17
fried_ricemriedem, dansmith: +A20:19
mriedemthanks20:20
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Move conductor wait_until_ready() delay before manager init  https://review.openstack.org/59906420:22
*** holser_ has joined #openstack-nova20:23
*** itlinux has joined #openstack-nova20:28
*** niceplace has joined #openstack-nova20:33
*** med_ has joined #openstack-nova20:33
*** openstackstatus has quit IRC20:36
*** openstackstatus has joined #openstack-nova20:38
*** ChanServ sets mode: +v openstackstatus20:38
*** itlinux has quit IRC20:44
*** harlowja has joined #openstack-nova20:46
*** imacdonn has quit IRC20:48
*** tbachman has quit IRC20:51
openstackgerritEric Fried proposed openstack/nova master: Other host allocs may appear in gafpt during evac  https://review.openstack.org/59730120:51
*** tbachman has joined #openstack-nova20:52
openstackgerritEric Fried proposed openstack/nova master: Other host allocs may appear in gafpt during evac  https://review.openstack.org/59730120:55
openstackgerritsean mooney proposed openstack/nova master: add caching to _build_regex_range  https://review.openstack.org/59907120:55
*** holser_ has quit IRC20:58
*** luksky has quit IRC21:00
*** imacdonn has joined #openstack-nova21:01
*** munimeha1 has quit IRC21:07
mriedemthat wily old shelve https://bugs.launchpad.net/nova/+bug/179022121:10
openstackLaunchpad bug 1790221 in OpenStack Compute (nova) "shelved offloaded server still shows old AZ while shelved" [Low,Triaged]21:10
cfriesenmriedem: why is that bad?21:12
cfriesenthe unshelve api doesn't say anything about selecting a new AZ21:12
mriedemcfriesen: a shelved instance isn't on a host,21:12
mriedemnote that the server details don't show a host or node when it's shelved21:12
cfriesenmriedem: sure, but the AZ was selected at boot time as part of the boot criteria21:13
mriedemif it's not on a host, it's clearly not in an AZ21:13
mriedemnope21:13
mriedemi didn't create the server with an az21:13
mriedemso when i unshelve it, it can go into any othre az21:13
mriedemrelated https://review.openstack.org/#/c/567701/21:14
cfriesenI had thought that once we pick an AZ at boot it might have implications on which cinder backends are available, so you couldn't leave it afterwards21:15
mriedemoh fuck no21:15
mriedemrelated https://review.openstack.org/#/c/469675/21:15
cfriesencool, learned something new21:15
mriedemwe only give a shit about volume az if cinder.cross_az_attach=False, which it's not by default21:15
mriedemif it is, well, good luck friend21:16
mriedemb/c nova sucks at handling that21:16
mriedemhttps://review.openstack.org/#/c/467674/ will show you exactly why21:16
mriedemif you remove that nova fix as a dependency, you can't create a server with a pre-existing volume at all21:17
mriedembecause 'nova' != None21:17
mriedemKABLAMMO21:17
mriedemgranted, my patch for that cross_az_attach fix is fugly21:18
*** med_ has quit IRC21:36
*** moshele has quit IRC21:37
*** med_ has joined #openstack-nova21:43
* melwitt will bbl21:54
*** elbragstad has quit IRC22:02
*** itlinux has joined #openstack-nova22:11
openstackgerritMatt Riedemann proposed openstack/nova master: Null out instance.availability_zone on shelve offload  https://review.openstack.org/59908722:20
*** harlowja has quit IRC22:24
*** mriedem has quit IRC22:32
*** itlinux has quit IRC22:34
*** markvoelker has joined #openstack-nova22:40
*** erlon has joined #openstack-nova23:04
*** macza has quit IRC23:07
*** Swami has quit IRC23:08
*** tbachman has joined #openstack-nova23:18
*** cfriesen has quit IRC23:36
*** r-daneel has quit IRC23:36
*** erlon has quit IRC23:56
« Thursday, 2018-08-30 Index Saturday, 2018-09-01 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!