| *** cfriesen_ has joined #openstack-nova | 00:00 | |
| *** itlinux has quit IRC | 00:02 | |
| *** mtanino has quit IRC | 00:07 | |
| *** moshele has joined #openstack-nova | 00:07 | |
| *** jaypipes has quit IRC | 00:09 | |
| openstackgerrit | Moshe Levi proposed openstack/nova master: hardware offload support for openvswitch https://review.openstack.org/398265 | 00:11 |
|---|---|---|
| tonyb | mikal, melwitt: the etherpad I was thinking of was https://etherpad.openstack.org/p/nova-low-hanging-fruit which has been migrated to the wiki | 00:13 |
| *** kbaegis has quit IRC | 00:13 | |
| tonyb | So I guess start a new one? | 00:13 |
| tonyb | I know low-hanging-fruit isn't exactly what we were thinking of but I'm easily confused | 00:13 |
| *** esberglu has joined #openstack-nova | 00:14 | |
| mikal | tonyb: I have a tweak to the localfs patch I'd like to upload before you do any rebasing please | 00:15 |
| *** gmann has quit IRC | 00:15 | |
| *** gmann has joined #openstack-nova | 00:15 | |
| *** kylek3h has joined #openstack-nova | 00:15 | |
| mikal | tonyb: I might upload that now unless you shout no in the next ten seconds | 00:15 |
| tonyb | mikal: Sure | 00:16 |
| tonyb | mikal: I was kinda using the 'I' in the sense mikal or tony sense | 00:16 |
| *** kylek3h has quit IRC | 00:16 | |
| mikal | LOL | 00:16 |
| mikal | We are one entity now? | 00:16 |
| mikal | That must be very embarassing for you | 00:16 |
| tonyb | mikal: as long as you don't touch the last_bytes review go nuts ;P | 00:16 |
| moshele | mriedem: hi | 00:16 |
| tonyb | mikal: only here where privsep is concerend | 00:16 |
| mikal | Oh sigh. I need to rebase. Please hold. | 00:17 |
| tonyb | mikal: It's more of that 'tonyb took my keyboard and wont give it back pair programming' | 00:17 |
| mikal | tonyb: true that. tonyb doesn't know how to share. | 00:17 |
| tonyb | mikal: that's what my teachers said a school | 00:17 |
| mikal | Heh | 00:17 |
| tonyb | mikal: you might want to hold off until last_bytes merged unless that also needs a fix | 00:18 |
| mikal | That didn't | 00:18 |
| mikal | But it seems to hate merging | 00:18 |
| * mikal checks zuul | 00:18 | |
| tonyb | the py35 sdvm job seems to fail lots but perhaps it just hates us | 00:19 |
| mikal | Hmmm, 472228 is in the check queue not the gate queue? | 00:19 |
| tonyb | mikal: we had to recheck | 00:19 |
| mikal | Looks like its going to pass there though | 00:19 |
| *** jamesdenton has quit IRC | 00:19 | |
| tonyb | which does both queues | 00:19 |
| mikal | Does it go to the gate queue automagically? | 00:19 |
| mikal | Oh, so it runs all the tests twice? | 00:19 |
| mikal | Thus maximizing the changes of a flakey test? | 00:19 |
| tonyb | if it still has +W it will just move to gate when it gets a +1 from jenkins | 00:19 |
| tonyb | mikal: Yeah but if anything has merged since the check was run then the results could now be wrong so it has to do bith | 00:20 |
| tonyb | *both* | 00:20 |
| *** jamesdenton has joined #openstack-nova | 00:21 | |
| mikal | So why not just re-run only the gate job? | 00:21 |
| *** kbaegis has joined #openstack-nova | 00:21 | |
| *** raunak has left #openstack-nova | 00:22 | |
| *** randomhack has joined #openstack-nova | 00:22 | |
| tonyb | 'cause gate and check have different configs and you could effectivle bypass a job that will fail but not catch it | 00:22 |
| mikal | I hate everything | 00:24 |
| *** kbaegis has quit IRC | 00:28 | |
| *** randomhack has quit IRC | 00:31 | |
| *** sree_ has joined #openstack-nova | 00:32 | |
| *** randomhack has joined #openstack-nova | 00:32 | |
| *** sree_ is now known as Guest41234 | 00:32 | |
| *** psachin has joined #openstack-nova | 00:36 | |
| *** Guest41234 has quit IRC | 00:37 | |
| *** randomhack has quit IRC | 00:37 | |
| *** hshiina has joined #openstack-nova | 00:37 | |
| *** yingjun has joined #openstack-nova | 00:41 | |
| *** kiennt has joined #openstack-nova | 00:42 | |
| openstackgerrit | yuanyue proposed openstack/nova master: Add a periodic task to destroy ReqSpecs of deleted instances https://review.openstack.org/484694 | 00:47 |
| *** acormier has quit IRC | 00:48 | |
| *** acormier has joined #openstack-nova | 00:49 | |
| *** moshele has quit IRC | 00:49 | |
| *** kbaegis has joined #openstack-nova | 00:51 | |
| *** dixiaoli has joined #openstack-nova | 00:51 | |
| *** kylek3h has joined #openstack-nova | 00:52 | |
| *** acormier has quit IRC | 00:53 | |
| *** kylek3h has quit IRC | 00:53 | |
| *** moshele has joined #openstack-nova | 00:53 | |
| *** itlinux has joined #openstack-nova | 00:53 | |
| *** edmondsw has joined #openstack-nova | 00:54 | |
| *** itlinux has quit IRC | 00:58 | |
| *** edmondsw has quit IRC | 00:58 | |
| *** vladikr has joined #openstack-nova | 00:58 | |
| *** phuongnh has joined #openstack-nova | 01:02 | |
| *** Sukhdev has quit IRC | 01:06 | |
| *** moshele has quit IRC | 01:06 | |
| *** imacdonn_ has quit IRC | 01:08 | |
| mikal | tonyb: check passed | 01:08 |
| rm_work | anyone know what the var for the nova-cpu config file is in devstack | 01:08 |
| *** rajathagasthya has quit IRC | 01:08 | |
| rm_work | like | 01:08 |
| rm_work | https://github.com/problemv/devstack_deploy/blob/master/local.conf#L31-L34 | 01:08 |
| rm_work | I have that but it needs to be in nova-cpu conf too i think? | 01:08 |
| *** jamielennox has quit IRC | 01:08 | |
| rm_work | config file split recently? | 01:08 |
| tonyb | rm_work: That should work. What are you seeing as a problme | 01:10 |
| tonyb | mikal: \o/ | 01:10 |
| rm_work | tonyb: it doesn't ... use it? | 01:10 |
| rm_work | apparently there is another conf now | 01:10 |
| rm_work | https://www.irccloud.com/pastebin/8fg3XGaC/ | 01:10 |
| rm_work | adding it to nova-cpu.conf seems to work | 01:11 |
| *** iceyao has joined #openstack-nova | 01:11 | |
| rm_work | but i don't know how to automate it | 01:11 |
| rm_work | $NOVA_CPU_CONF maybe? | 01:11 |
| *** mdnadeem has joined #openstack-nova | 01:11 | |
| tonyb | rm_work: where are you getting devstack from? | 01:13 |
| *** jamielennox has joined #openstack-nova | 01:15 | |
| tonyb | rm_work: http://paste.openstack.org/show/616640 | 01:15 |
| tonyb | rm_work: I don't see anything in devstack that is splitting things like that | 01:15 |
| *** iceyao has quit IRC | 01:15 | |
| rm_work | so you don't have a nova-cpu.conf? | 01:16 |
| *** iceyao has joined #openstack-nova | 01:16 | |
| *** armax has joined #openstack-nova | 01:16 | |
| tonyb | rm_work: nope: http://logs.openstack.org/28/472228/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/91e93eb/logs/etc/nova/ | 01:17 |
| rm_work | http://logs.openstack.org/28/472228/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/91e93eb/logs/etc/nova/nova-cpu.conf.txt.gz | 01:17 |
| tonyb | rm_work: Oh wait it is there | 01:17 |
| rm_work | with the libvirt section | 01:17 |
| rm_work | as of ... recently maybe? | 01:18 |
| *** harlowja has quit IRC | 01:19 | |
| tonyb | rm_work: You're right NOVA_CPU_CONF http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n54 | 01:21 |
| rm_work | cool :) good guess | 01:21 |
| mriedem | rm_work: tonyb: merged yesterday | 01:22 |
| mriedem | check the dev list about multi-tier conductor | 01:22 |
| tonyb | mriedem: Oh I'm not that behind then | 01:22 |
| rm_work | yeah, octavia folks are just *bleeding edge* :P | 01:23 |
| rm_work | we always run into this stuff first | 01:23 |
| tonyb | rm_work: my first grep didn't checkout master (it only downlaoded it) :( | 01:23 |
| tonyb | mriedem: oh the conductor fleet stuff | 01:24 |
| mriedem | yeah | 01:25 |
| mriedem | a change merged today that allows you to disable that behavior if you need to | 01:25 |
| mriedem | https://review.openstack.org/#/c/487485/ | 01:26 |
| *** cdelatte has quit IRC | 01:27 | |
| mriedem | although, even with that set to singleconductor, i'm seeing the superconductor running http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/ | 01:29 |
| mriedem | dansmith: ^ maybe why the ironic job is still failing? | 01:29 |
| *** kylek3h has joined #openstack-nova | 01:29 | |
| dansmith | mriedem: in single conductor, the super conductor is the only one that gets hit | 01:29 |
| *** kylek3h has quit IRC | 01:29 | |
| mriedem | but we still start the cell conductor? | 01:30 |
| mriedem | i'm seeing a request go through the scheduler, it picks a host, but then i don't see the request going through n-cpu | 01:30 |
| dansmith | yeah, it was a hack for grenade remember | 01:30 |
| *** yamamoto_ has quit IRC | 01:31 | |
| mriedem | req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 | 01:32 |
| mriedem | http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/screen-n-super-cond.txt.gz#_Jul_26_21_32_58_412591 | 01:32 |
| mriedem | super conductor just seems to die there | 01:32 |
| mriedem | and i never see req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 show up in n-cpu logs | 01:32 |
| dansmith | is that req a boot request? | 01:33 |
| dansmith | you'd never see it hit cpu if it got novalidhost right? | 01:33 |
| mriedem | it didn't get novalidhost | 01:33 |
| mriedem | i see n-sch pick a host | 01:33 |
| mriedem | and yeah that's the boot request | 01:34 |
| mriedem | super-cond, sch and cpu are all using nova.conf | 01:34 |
| mriedem | which points at transport_url = rabbit://stackrabbit:secretrabbit@15.184.66.253:5672/ | 01:34 |
| mriedem | db connection is set to nova_cell1 | 01:34 |
| dansmith | yeah, which is a pretty normal setup | 01:34 |
| *** gongysh has joined #openstack-nova | 01:35 | |
| dansmith | is it normal to have cpu deleting tons of orphan nodes? | 01:35 |
| *** yamamoto has joined #openstack-nova | 01:35 | |
| dansmith | and this is single node so no chance we're missing logs from another conductor I guess | 01:36 |
| mriedem | lots of ComputeHostNotFound in the n-cpu logs | 01:36 |
| mriedem | this is the host/node that's picked in the scheduler | 01:37 |
| mriedem | Selected host: (ubuntu-xenial-infracloud-vanilla-10103211, eda7dc99-9e17-479c-b234-9d87956e9c56) ram: 384MB disk: 10240MB io_ops: 0 instances: 0 | 01:37 |
| dansmith | so you think conductor got the selected host and cast to compute but it disappeared? | 01:37 |
| mriedem | or conductor didn't get it.. | 01:37 |
| dansmith | unfortunately, we don't get to see the _actual_ trnsport_url loaded in compute | 01:38 |
| dansmith | that req showed up in conductor, no? | 01:38 |
| *** zhurong has joined #openstack-nova | 01:38 | |
| dansmith | yeah, it clearly got to the conductor | 01:38 |
| mriedem | yeah i see block_device_mapping in conductor | 01:39 |
| mriedem | which comes after we get the scheduler response | 01:39 |
| mriedem | http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/screen-n-super-cond.txt.gz#_Jul_26_21_32_58_377797 | 01:39 |
| mriedem | i never see "Starting instance" in the n-cpu logs | 01:40 |
| dansmith | well, they're all using the same config, and the config is good, I don't really see how it's breaking between conductor and compute | 01:40 |
| mriedem | so it looks like the request never gets to build_and_run_instance in n-cpu | 01:40 |
| mriedem | me neither | 01:40 |
| dansmith | yeah | 01:40 |
| dansmith | but I dunno how to explain that | 01:40 |
| dansmith | the hostname and node id seem to match up between compute and what scheduler selected | 01:43 |
| *** kristian__ has joined #openstack-nova | 01:43 | |
| dansmith | if it picked a host that didn't exist we would cast to the wrong topic and it'd get dropped, but.. | 01:43 |
| dansmith | they match | 01:43 |
| *** esberglu has quit IRC | 01:43 | |
| dansmith | no rabbit dump at the end so we can't see if there are different topics or messages sitting in queues | 01:45 |
| *** gongysh has quit IRC | 01:47 | |
| *** raunak has joined #openstack-nova | 01:47 | |
| *** vladikr has quit IRC | 01:47 | |
| *** kristian__ has quit IRC | 01:47 | |
| mriedem | ok so scheduler picks the host here | 01:48 |
| mriedem | Jul 26 21:32:57.959669 ubuntu-xenial-infracloud-vanilla-10103211 nova-scheduler[12180]: DEBUG nova.scheduler.filter_scheduler [None req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 tempest-BaremetalBasicOps-1511666062 tempest-BaremetalBasicOps-1511666062] Selected host: (ubuntu-xenial-infracloud-vanilla-10103211, eda7dc99-9e17-479c-b234-9d87956e9c56) ram: 384MB disk: 10240MB io_ops: 0 instances: 0 {{(pid=12180) _schedule /opt/stack/n | 01:48 |
| mriedem | ova/nova/scheduler/filter_scheduler.py:157}} | 01:48 |
| *** saphi has joined #openstack-nova | 01:48 | |
| mriedem | super-conductor creates the bdms in cell1 here | 01:48 |
| mriedem | Jul 26 21:32:58.377797 ubuntu-xenial-infracloud-vanilla-10103211 nova-conductor[13817]: DEBUG nova.conductor.manager [None req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 tempest-BaremetalBasicOps-1511666062 tempest-BaremetalBasicOps-1511666062] [instance: 6e534573-5533-41d6-b9d7-28bfd7dbfe2d] block_device_mapping [BlockDeviceMapping(attachment_id=<?>,boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,delete | 01:48 |
| mriedem | >,deleted_at=<?>,destination_type='local',device_name=None,device_type='disk',disk_bus=None,guest_format=None,id=<?>,image_id='2120252b-74dd-46b0-8649-ef7bcbdd0f50',instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=<?>,volume_id=None,volume_size=None)] {{(pid=14569) _create_block_device_mapping /opt/stack/new/nova/nova/conductor/manager.py:829}} | 01:48 |
| *** thorst has joined #openstack-nova | 01:49 | |
| mriedem | 8 seconds before that in n-cpu we're deleting orphan compute nodes | 01:49 |
| mriedem | http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/screen-n-cpu.txt.gz#_Jul_26_21_32_50_782647 | 01:49 |
| *** tetsuro has joined #openstack-nova | 01:49 | |
| dansmith | but, the node doesn't affect the host/topic it goes to | 01:49 |
| *** thorst has quit IRC | 01:49 | |
| *** acormier has joined #openstack-nova | 01:50 | |
| mriedem | yeah | 01:50 |
| mriedem | there is a gmr at the end of n-cpu | 01:50 |
| dansmith | transport_url is obscured there too | 01:51 |
| dansmith | also, | 01:51 |
| dansmith | scheduler wouldn't have selected it if it wasn't getting written to the right db, | 01:51 |
| dansmith | but I guess it could be going to the wrong conductor | 01:52 |
| dansmith | so we could try setting the other conductor to the same mq vhost in this scenario, or not start it and see if n-cpu is timing out | 01:52 |
| dansmith | although I don't really see how we could be on the wrong vhost, since we write the config early and don't restart cpu at all | 01:53 |
| *** acormier has quit IRC | 01:54 | |
| mriedem | i was wondering if something was going to the cell1 conductor, but there is nothing in it's logs | 01:56 |
| mriedem | after it dumps the config anyway | 01:56 |
| dansmith | yeah, seems unlikely, but it also has a barebones config, so maybe it's on warning only logs or something? | 01:57 |
| mriedem | it's logging debug messages | 01:58 |
| dansmith | yeah I guess so | 01:59 |
| dansmith | well I dunno, I'm dry on ideas | 01:59 |
| mriedem | there is a rabbit log | 01:59 |
| mriedem | http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/rabbitmq/rabbit@ubuntu-xenial-infracloud-vanilla-10103211.txt.gz | 01:59 |
| dansmith | yeah, but no queue dump right? | 01:59 |
| mriedem | no | 01:59 |
| mriedem | last thing we'd see in super-cond is Jul 26 21:32:58.377797 | 01:59 |
| mriedem | and that's where the rabbit log ends | 01:59 |
| mriedem | know how to dump the queue in the rabbitmq logs? | 02:01 |
| dansmith | yeah, | 02:02 |
| dansmith | rabbitmqctl report | 02:02 |
| dansmith | I think gives you a ton of data | 02:02 |
| dansmith | yup, that's all the status dumps in one command | 02:03 |
| mriedem | hmm, but do we just dump that at the very end when collecting logs? | 02:03 |
| *** Apoorva has quit IRC | 02:03 | |
| mriedem | i know where logs are collected in devstack-gate | 02:03 |
| *** yingjun has quit IRC | 02:03 | |
| *** markvoelker has quit IRC | 02:03 | |
| dansmith | yeah, that will tell us if something is still in a queue | 02:03 |
| mriedem | ok i could tinker with that | 02:04 |
| mriedem | and fix jay's placement 'handle moves' patch | 02:04 |
| dansmith | so the channels section of that dump will have some stuff like consumer_count and messages_unacknowledged | 02:05 |
| dansmith | which would be the things that are sitting from a cast unretrieved | 02:05 |
| dansmith | and then you should see the compute-$hostname topics for anything listening | 02:05 |
| *** yushb has joined #openstack-nova | 02:05 | |
| *** markvoelker has joined #openstack-nova | 02:06 | |
| dansmith | not sure that will dump all vhosts | 02:06 |
| dansmith | report doesn't take a vhost, so I imagine it will dump all of them | 02:06 |
| openstackgerrit | Merged openstack/nova master: deprecate ``wsgi_log_format`` config variable https://review.openstack.org/486623 | 02:07 |
| dansmith | yeah looks like it | 02:07 |
| dansmith | mriedem: I'm going to retire for the evening but will pick things back up in the morning unless you fix it before then | 02:08 |
| *** mtanino_ has joined #openstack-nova | 02:09 | |
| mriedem | adios | 02:14 |
| *** jamielennox has quit IRC | 02:17 | |
| melwitt | tonyb: excellent memory. I had forgotten about the low hanging fruit etherpad | 02:21 |
| tonyb | melwitt: :D | 02:22 |
| *** kylek3h has joined #openstack-nova | 02:23 | |
| mriedem | https://review.openstack.org/487665 | 02:23 |
| *** jamielennox has joined #openstack-nova | 02:24 | |
| *** tetsuro_ has joined #openstack-nova | 02:29 | |
| *** dillaman has joined #openstack-nova | 02:29 | |
| *** kylek3h_ has joined #openstack-nova | 02:30 | |
| *** Shunli has joined #openstack-nova | 02:30 | |
| *** yamahata has quit IRC | 02:30 | |
| *** kylek3h has quit IRC | 02:33 | |
| *** esberglu has joined #openstack-nova | 02:38 | |
| *** baoli has quit IRC | 02:38 | |
| *** coreywright has quit IRC | 02:40 | |
| *** esberglu has quit IRC | 02:42 | |
| *** edmondsw has joined #openstack-nova | 02:42 | |
| *** kylek3h_ has quit IRC | 02:45 | |
| *** edmondsw has quit IRC | 02:46 | |
| *** sree_ has joined #openstack-nova | 02:47 | |
| *** sree_ is now known as Guest81593 | 02:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | 02:48 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | 02:48 |
| mriedem | ^ fixes the random hashseed failures in the unit tests for the bottom change | 02:48 |
| *** Guest81593 has quit IRC | 02:51 | |
| *** randomhack has joined #openstack-nova | 02:54 | |
| *** Shunli has quit IRC | 02:55 | |
| *** moshele has joined #openstack-nova | 02:55 | |
| *** Shunli has joined #openstack-nova | 02:55 | |
| *** coreywright has joined #openstack-nova | 02:58 | |
| *** randomhack has quit IRC | 02:58 | |
| *** Shunli has quit IRC | 03:00 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208 | 03:03 |
| openstackgerrit | Merged openstack/nova master: Move the last_bytes util method to libvirt https://review.openstack.org/472228 | 03:05 |
| mikal | tonyb: the last_bytes refactor change just passed the gate. | 03:06 |
| tonyb | huzzah! | 03:08 |
| mikal | Gonna upload my smallish change in a minute | 03:08 |
| mikal | And then I'll take a look at that pyhon3.2 thing | 03:08 |
| tonyb | mikal: okay, I'll check it out when it arrives | 03:08 |
| *** dixiaoli has quit IRC | 03:09 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | 03:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | 03:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | 03:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: rootwrap.d cleanup mislabeled files https://review.openstack.org/486831 | 03:09 |
| mikal | tonyb: my only change there is a relno, so I wouldn't spend a lot of time looking at it | 03:09 |
| *** zerda3 has joined #openstack-nova | 03:10 | |
| *** chyka has joined #openstack-nova | 03:10 | |
| *** zhurong has quit IRC | 03:13 | |
| openstackgerrit | Merged openstack/nova master: add a retry on DBDeadlock to _set_allocations() https://review.openstack.org/487483 | 03:16 |
| *** chyka has quit IRC | 03:19 | |
| *** thorst has joined #openstack-nova | 03:22 | |
| *** thorst has quit IRC | 03:30 | |
| *** harlowja has joined #openstack-nova | 03:31 | |
| *** udesale has joined #openstack-nova | 03:36 | |
| *** trungnv has quit IRC | 03:40 | |
| *** psachin has quit IRC | 03:41 | |
| *** moshele has quit IRC | 03:43 | |
| *** trungnv has joined #openstack-nova | 03:48 | |
| *** tuanluong has joined #openstack-nova | 03:51 | |
| *** acormier has joined #openstack-nova | 03:52 | |
| *** mtanino_ has quit IRC | 03:53 | |
| *** acormier has quit IRC | 03:56 | |
| *** psachin has joined #openstack-nova | 03:58 | |
| *** mriedem has quit IRC | 04:02 | |
| *** mamandle has joined #openstack-nova | 04:07 | |
| *** nicolasbock has joined #openstack-nova | 04:07 | |
| *** _pewp_ has quit IRC | 04:18 | |
| *** yushb has quit IRC | 04:18 | |
| *** iceyao has quit IRC | 04:19 | |
| *** _pewp_ has joined #openstack-nova | 04:20 | |
| *** iceyao has joined #openstack-nova | 04:22 | |
| openstackgerrit | yuanyue proposed openstack/nova master: Add a periodic task to destroy ReqSpecs of deleted instances https://review.openstack.org/484694 | 04:22 |
| *** hoonetorg has joined #openstack-nova | 04:26 | |
| *** iceyao has quit IRC | 04:27 | |
| *** itlinux has joined #openstack-nova | 04:29 | |
| *** gongysh has joined #openstack-nova | 04:31 | |
| *** links has joined #openstack-nova | 04:33 | |
| *** moshele has joined #openstack-nova | 04:36 | |
| *** yushb has joined #openstack-nova | 04:38 | |
| *** gouthamr has quit IRC | 04:38 | |
| *** zhurong has joined #openstack-nova | 04:39 | |
| *** iceyao has joined #openstack-nova | 04:39 | |
| *** tetsuro has quit IRC | 04:41 | |
| *** sree_ has joined #openstack-nova | 04:42 | |
| *** sree_ is now known as Guest24586 | 04:42 | |
| *** Guest24586 has quit IRC | 04:42 | |
| *** iceyao has quit IRC | 04:44 | |
| *** iceyao has joined #openstack-nova | 04:47 | |
| *** prateek has joined #openstack-nova | 04:52 | |
| *** harlowja has quit IRC | 04:52 | |
| *** kbaegis has quit IRC | 04:53 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955 | 04:55 |
| *** mamandle has quit IRC | 04:57 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964 | 04:58 |
| *** yushb has quit IRC | 04:58 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042 | 04:58 |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.54 - Enable cold migration with target host https://review.openstack.org/406707 | 04:58 |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.54 - List/Show all server migration types https://review.openstack.org/430839 | 04:59 |
| *** kristian__ has joined #openstack-nova | 05:00 | |
| *** kristian__ has quit IRC | 05:00 | |
| *** kristian__ has joined #openstack-nova | 05:00 | |
| *** yamahata has joined #openstack-nova | 05:01 | |
| *** sridharg has joined #openstack-nova | 05:02 | |
| *** kristian__ has quit IRC | 05:04 | |
| *** gongysh has quit IRC | 05:05 | |
| *** gongysh has joined #openstack-nova | 05:10 | |
| *** gongysh has quit IRC | 05:13 | |
| *** acormier has joined #openstack-nova | 05:16 | |
| *** acormier has quit IRC | 05:21 | |
| *** trinaths has joined #openstack-nova | 05:25 | |
| *** trinaths has left #openstack-nova | 05:25 | |
| *** thorst has joined #openstack-nova | 05:26 | |
| *** sree_ has joined #openstack-nova | 05:28 | |
| *** sree_ is now known as Guest40093 | 05:28 | |
| *** trinaths has joined #openstack-nova | 05:29 | |
| *** armax has quit IRC | 05:30 | |
| *** armax has joined #openstack-nova | 05:31 | |
| *** armax has quit IRC | 05:31 | |
| *** armax has joined #openstack-nova | 05:32 | |
| *** armax has quit IRC | 05:32 | |
| *** armax has joined #openstack-nova | 05:33 | |
| *** armax has quit IRC | 05:33 | |
| *** thorst has quit IRC | 05:33 | |
| *** jdwidari has joined #openstack-nova | 05:36 | |
| *** jdwidari has quit IRC | 05:36 | |
| *** ekuris has joined #openstack-nova | 05:37 | |
| *** vishwana_ has quit IRC | 05:40 | |
| *** vishwanathj has joined #openstack-nova | 05:41 | |
| *** amuresan has joined #openstack-nova | 05:43 | |
| *** mamandle has joined #openstack-nova | 05:44 | |
| *** kiennt has quit IRC | 05:44 | |
| *** hongbin has joined #openstack-nova | 05:47 | |
| *** udesale__ has joined #openstack-nova | 05:47 | |
| *** kristian__ has joined #openstack-nova | 05:47 | |
| *** yufei has joined #openstack-nova | 05:49 | |
| *** udesale has quit IRC | 05:50 | |
| *** yufei has quit IRC | 05:50 | |
| *** trinaths has quit IRC | 05:51 | |
| *** kristian__ has quit IRC | 05:52 | |
| *** litao__ has joined #openstack-nova | 05:53 | |
| *** priteau has joined #openstack-nova | 05:55 | |
| *** trinaths has joined #openstack-nova | 05:56 | |
| *** ratailor has joined #openstack-nova | 05:58 | |
| *** yushb has joined #openstack-nova | 06:00 | |
| *** thorst has joined #openstack-nova | 06:01 | |
| *** kiennt has joined #openstack-nova | 06:02 | |
| *** kristian__ has joined #openstack-nova | 06:04 | |
| *** jjung has joined #openstack-nova | 06:05 | |
| *** thorst has quit IRC | 06:05 | |
| *** Guest40093 has quit IRC | 06:09 | |
| *** yingjun has joined #openstack-nova | 06:10 | |
| openstackgerrit | NidhiMittalHada proposed openstack/python-novaclient master: Help text for "--matching" is not clear. https://review.openstack.org/487708 | 06:11 |
| *** udesale has joined #openstack-nova | 06:12 | |
| *** udesale__ has quit IRC | 06:14 | |
| *** esberglu has joined #openstack-nova | 06:16 | |
| *** kiennt has quit IRC | 06:16 | |
| *** kiennt_ has joined #openstack-nova | 06:17 | |
| *** kiennt has joined #openstack-nova | 06:17 | |
| *** kiennt has quit IRC | 06:17 | |
| *** edmondsw has joined #openstack-nova | 06:18 | |
| *** esberglu has quit IRC | 06:21 | |
| *** edmondsw has quit IRC | 06:23 | |
| *** zerda3 has quit IRC | 06:23 | |
| openstackgerrit | jichenjc proposed openstack/nova master: add cli to support matrix https://review.openstack.org/481498 | 06:29 |
| openstackgerrit | jichenjc proposed openstack/nova master: Enhance support matrix document https://review.openstack.org/482013 | 06:30 |
| openstackgerrit | jichenjc proposed openstack/nova master: Enhance support matrix document https://review.openstack.org/482020 | 06:31 |
| *** sree_ has joined #openstack-nova | 06:31 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Add quiesce and unquiesce in support matrix https://review.openstack.org/482390 | 06:31 |
| *** sree_ is now known as Guest19469 | 06:31 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Add description for tag support in attach https://review.openstack.org/481479 | 06:33 |
| openstackgerrit | jichenjc proposed openstack/nova master: Add device tag support info in support matrix https://review.openstack.org/481478 | 06:33 |
| *** zhurong has quit IRC | 06:33 | |
| openstackgerrit | jichenjc proposed openstack/nova master: WIP:add cert when send keystone request https://review.openstack.org/485121 | 06:40 |
| *** raunak has quit IRC | 06:44 | |
| openstackgerrit | jichenjc proposed openstack/nova master: refactory some code on get_password https://review.openstack.org/449458 | 06:48 |
| *** damien_r has joined #openstack-nova | 06:48 | |
| *** tetsuro_ has quit IRC | 06:49 | |
| *** priteau has quit IRC | 06:51 | |
| *** pilgrimstack has joined #openstack-nova | 06:51 | |
| *** markus_z has joined #openstack-nova | 06:51 | |
| *** trinaths has quit IRC | 06:51 | |
| *** trinaths1 has joined #openstack-nova | 06:51 | |
| openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | 06:52 |
| *** andreas_s has joined #openstack-nova | 06:53 | |
| *** randomhack has joined #openstack-nova | 06:54 | |
| *** randomhack has quit IRC | 06:58 | |
| *** sshwarts has joined #openstack-nova | 07:02 | |
| *** hongbin has quit IRC | 07:04 | |
| *** pilgrimstack has quit IRC | 07:05 | |
| *** Shunli has joined #openstack-nova | 07:08 | |
| *** Shunli has quit IRC | 07:09 | |
| *** Shunli has joined #openstack-nova | 07:09 | |
| *** ioggstream has joined #openstack-nova | 07:11 | |
| *** Shunli has quit IRC | 07:11 | |
| *** Shunli has joined #openstack-nova | 07:11 | |
| *** Shunli has quit IRC | 07:13 | |
| *** Shunli has joined #openstack-nova | 07:13 | |
| openstackgerrit | Merged openstack/python-novaclient master: Removed extra word 'method' from the NOTE https://review.openstack.org/480988 | 07:13 |
| *** rcernin has joined #openstack-nova | 07:13 | |
| *** Shunli has quit IRC | 07:15 | |
| *** Shunli has joined #openstack-nova | 07:15 | |
| *** Shunli has quit IRC | 07:17 | |
| *** Shunli has joined #openstack-nova | 07:17 | |
| *** int32bit has joined #openstack-nova | 07:18 | |
| *** Shunli has quit IRC | 07:20 | |
| *** Shunli has joined #openstack-nova | 07:21 | |
| *** Shunli has quit IRC | 07:22 | |
| *** alexchadin has joined #openstack-nova | 07:26 | |
| *** acormier has joined #openstack-nova | 07:31 | |
| *** cfriesen_ has quit IRC | 07:33 | |
| *** obre has quit IRC | 07:36 | |
| *** obre has joined #openstack-nova | 07:36 | |
| *** acormier has quit IRC | 07:37 | |
| gibi | bauzas: hi! good morning! There is a bugfix that only need a second +2 https://review.openstack.org/#/c/475276/ Could you check it if your time allows? | 07:40 |
| *** Shunli has joined #openstack-nova | 07:43 | |
| *** Shunli has quit IRC | 07:44 | |
| *** Shunli has joined #openstack-nova | 07:45 | |
| *** ralonsoh has joined #openstack-nova | 07:46 | |
| *** roeyc has joined #openstack-nova | 07:49 | |
| *** ralonsoh has quit IRC | 07:52 | |
| *** ralonsoh has joined #openstack-nova | 07:52 | |
| *** jaosorior has joined #openstack-nova | 07:52 | |
| *** yassine has joined #openstack-nova | 08:00 | |
| *** thorst has joined #openstack-nova | 08:02 | |
| *** esberglu has joined #openstack-nova | 08:04 | |
| bauzas | gibi: roger, roger. | 08:06 |
| *** edmondsw has joined #openstack-nova | 08:06 | |
| *** thorst has quit IRC | 08:06 | |
| *** testuro has joined #openstack-nova | 08:07 | |
| *** esberglu has quit IRC | 08:08 | |
| *** edmondsw has quit IRC | 08:11 | |
| gibi | bauzas: thanks | 08:15 |
| *** lucas-afk is now known as lucasagomes | 08:18 | |
| *** claudiub has joined #openstack-nova | 08:21 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send soft_delete from context manager https://review.openstack.org/476459 | 08:25 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: use context mgr in instance.delete https://review.openstack.org/443764 | 08:25 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297 | 08:26 |
| *** efoley has joined #openstack-nova | 08:28 | |
| *** derekh has joined #openstack-nova | 08:31 | |
| *** dpawlik has quit IRC | 08:32 | |
| *** danpawlik has joined #openstack-nova | 08:34 | |
| *** pilgrimstack has joined #openstack-nova | 08:34 | |
| *** yushb has quit IRC | 08:34 | |
| *** yushb has joined #openstack-nova | 08:35 | |
| *** gongysh has joined #openstack-nova | 08:36 | |
| openstackgerrit | jichenjc proposed openstack/nova master: refactory some code on get_password https://review.openstack.org/449458 | 08:43 |
| *** dtantsur|afk is now known as dtantsur | 08:44 | |
| mamandle | alex_xu: bauzas: sfinucan: mriedem: please take a look at https://review.openstack.org/#/c/483911/4, comments addressed. | 08:44 |
| bauzas | mamandle: given it's a bugfix and since today is pike-3 milestone day (so FeatureFreeze), I'll prioritize blueprints | 08:51 |
| bauzas | mamandle: that said, just ping me next week and I'll look at it, for sure :) | 08:51 |
| mamandle | bauzas: sure, thanks :) | 08:51 |
| *** priteau has joined #openstack-nova | 08:55 | |
| *** priteau has quit IRC | 08:59 | |
| *** priteau has joined #openstack-nova | 08:59 | |
| *** yamahata has quit IRC | 09:01 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: stabilize test_create_delete_server functional test https://review.openstack.org/487772 | 09:03 |
| *** mvk has quit IRC | 09:04 | |
| *** rmart04 has joined #openstack-nova | 09:10 | |
| *** sambetts|afk is now known as sambetts | 09:14 | |
| *** udesale has quit IRC | 09:14 | |
| *** kiennt_ is now known as kiennt | 09:15 | |
| vdrok | good morning folks! | 09:15 |
| *** chyka has joined #openstack-nova | 09:16 | |
| *** chyka has quit IRC | 09:20 | |
| *** kristian__ has quit IRC | 09:22 | |
| *** kristian__ has joined #openstack-nova | 09:23 | |
| takashin | bauzas: Please review https://review.openstack.org/#/c/408955/. Your comments were addressed. | 09:24 |
| bauzas | takashin: yup, I saw your update | 09:27 |
| openstackgerrit | jichenjc proposed openstack/nova master: add cert when send keystone request https://review.openstack.org/485121 | 09:28 |
| bauzas | takashin: I knew you already provided your spec since 2 cycles | 09:28 |
| bauzas | takashin: unfortunately, I didn't have time to review it until last week | 09:28 |
| bauzas | so it's my bad | 09:28 |
| *** bhagyashris has quit IRC | 09:29 | |
| bauzas | takashin: but now we are very close to the FF, would you accept that we could fast-merging that spec for Queens and ask me to be my main priority review for Q ? | 09:29 |
| bauzas | takashin: I really think you worked fine, and adding a new 'forced_destination' seems good with me | 09:30 |
| takashin | bauzas: I want to get it merged by Pike release. | 09:30 |
| bauzas | tbh, if we would even add it for the beginning of Queens, I could add another spec for using it for the nova boot call | 09:30 |
| *** yingjun has quit IRC | 09:30 | |
| bauzas | takashin: the problem is that we have 3 changes, one internal, one for the REST API and the last for the python novaclient | 09:31 |
| bauzas | and we only have like 6 hours for that | 09:31 |
| bauzas | I totally understand you would be sad if we would push it to Queens, but the main problem we had was about not a lot of reviews | 09:31 |
| bauzas | given we'll open Queens in two weeks, I do wonder if we could just push it for 2 weeks | 09:32 |
| *** Shunli has quit IRC | 09:32 | |
| bauzas | the main problem is about having the feature not in Pike | 09:32 |
| *** int32bit has quit IRC | 09:42 | |
| *** mvk has joined #openstack-nova | 09:44 | |
| *** dtantsur is now known as dtantsur|bbl | 09:51 | |
| *** esberglu has joined #openstack-nova | 09:52 | |
| *** saphi has quit IRC | 09:52 | |
| *** jjung has quit IRC | 09:53 | |
| *** edmondsw has joined #openstack-nova | 09:54 | |
| *** masber has quit IRC | 09:55 | |
| *** sdague has joined #openstack-nova | 09:55 | |
| *** esberglu has quit IRC | 09:57 | |
| *** alexchadin has quit IRC | 09:57 | |
| *** edmondsw has quit IRC | 09:58 | |
| *** yassine has quit IRC | 09:59 | |
| *** yamamoto has quit IRC | 10:00 | |
| sdague | vdrok: how is the ironic patch going? | 10:02 |
| *** thorst has joined #openstack-nova | 10:03 | |
| vdrok | sdague: I have not yet had a chance to look, will do in ~half an hour | 10:03 |
| alex_xu | bauzas: why we don't have API for delete the aggregates in the placement? I remember there are some discussion, but I forget the detail... | 10:03 |
| *** iceyao has quit IRC | 10:04 | |
| sdague | johnthetubaguy: https://review.openstack.org/#/c/357726 - this patch is ready to go if we reno it | 10:06 |
| sdague | can you rev for that? | 10:06 |
| *** thorst has quit IRC | 10:07 | |
| *** alexchadin has joined #openstack-nova | 10:11 | |
| *** udesale has joined #openstack-nova | 10:17 | |
| *** phuongnh has quit IRC | 10:23 | |
| *** thorst has joined #openstack-nova | 10:24 | |
| *** yushb has quit IRC | 10:28 | |
| *** thorst has quit IRC | 10:28 | |
| kiennt | sdague: Hi, i am working on Heat gate job for rolling upgrade testing. I already setup multinode grenade gate job but it doesn't have any tests for testing mixed version. | 10:30 |
| kiennt | I find out that Nova already had rolling-upgrade tag, so I want to ask about the way Nova do the tests | 10:31 |
| *** yamamoto has joined #openstack-nova | 10:31 | |
| kiennt | thank you | 10:31 |
| *** iceyao has joined #openstack-nova | 10:31 | |
| openstackgerrit | Sean Dague proposed openstack/nova master: Assume neutron port_binding extensions enabled https://review.openstack.org/357726 | 10:32 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Use admin context to create neutron ports https://review.openstack.org/357540 | 10:32 |
| sdague | kiennt: it's all about what makes sense for heat and it's collection of services. How would you expect heat would be upgraded across a set of nodes during upgrade? | 10:33 |
| sdague | johnthetubaguy - ok, reno added for - https://review.openstack.org/#/c/357726/ | 10:34 |
| *** cdent has joined #openstack-nova | 10:34 | |
| *** iceyao has quit IRC | 10:35 | |
| *** jjung has joined #openstack-nova | 10:35 | |
| andreaf | sdague: around? is there a flag in d-g or grenade that decides which host is upgraded in a multinode setup | 10:35 |
| *** acormier has joined #openstack-nova | 10:35 | |
| sdague | andreaf: no, it's baked in logic | 10:36 |
| sdague | the primary node is what is upgraded | 10:36 |
| sdague | subnodes are not | 10:36 |
| andreaf | sdague: ok that was my impression looking at devstack-vm-gate.sh, thanks | 10:37 |
| andreaf | kiennt: ^^^ | 10:37 |
| *** lucasagomes has quit IRC | 10:37 | |
| andreaf | sdague: so everything is setup on old first and then primary is upgraded to new right? | 10:37 |
| vdrok | sdague: so as mriedem said, in that job I see that the conductor called into scheduler which selected a host http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/screen-n-sch.txt.gz#_Jul_27_04_27_52_998156, then conductor proceeded (as I see | 10:38 |
| vdrok | http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/screen-n-super-cond.txt.gz#_Jul_27_04_27_53_252817, which happens after select_destinations), but in the compute log there is no build_and_run_instance, don't see the 'Starting instnace...' | 10:38 |
| kiennt | sdague, andreaf: So, after upgrade, if I run some tests (ex.tempest), it will run in services (2 versions). That will test these mixed services could talk to each other or not. Is this right? | 10:39 |
| andreaf | kiennt: yeah that's the idea | 10:39 |
| *** acormier has quit IRC | 10:40 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | 10:40 |
| kiennt | andreaf, sdague: Thank you, guys. This is very important information with me. I have stucked at this for a long time, because i'm completely new with all these things. | 10:42 |
| kiennt | Seem like I only need some tempest tests after upgrade now because I already have multinode gate. | 10:42 |
| *** Shunli has joined #openstack-nova | 10:42 | |
| openstackgerrit | Artem Tiumentcev proposed openstack/python-novaclient master: Add 'local_gb' disk quota support https://review.openstack.org/484391 | 10:43 |
| openstackgerrit | Chris Dent proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | 10:43 |
| *** Shunli has quit IRC | 10:44 | |
| *** gszasz has joined #openstack-nova | 10:44 | |
| *** iceyao has joined #openstack-nova | 10:45 | |
| *** thorst has joined #openstack-nova | 10:47 | |
| *** abhishekk has joined #openstack-nova | 10:47 | |
| sdague | andreaf: yes, primary node is installed, subnode is installed. Tests are run. Then primary node is upgraded. | 10:48 |
| andreaf | sdague: oh I see, so we don't run any test after upgrade? on check resources? | 10:49 |
| andreaf | s/on/only | 10:49 |
| sdague | andreaf: sorry, no we run tests again at the end | 10:50 |
| *** iceyao has quit IRC | 10:50 | |
| andreaf | sdague: ok thanks | 10:50 |
| sdague | so, because this is an intergration between devstack-gate and grenade, I don't know that it's ever gotten documented very well | 10:50 |
| sdague | I'd be happy to write up a webpage with what it is, if you figure out where you'd like it to live :) | 10:50 |
| sdague | do we currently have overarching qa docs outside of tempest / grenade / devstack ones? | 10:51 |
| *** thorst has quit IRC | 10:51 | |
| vdrok | sdague: just guessing, in the failing job I see transport_url with cell1 at http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova_cell1.conf.txt.gz but without cell1 at | 10:54 |
| vdrok | http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova.conf.txt.gz | 10:54 |
| sdague | ok, we'll need dansmith I think. Exactly where the mismatches are here are beyond me. | 10:56 |
| *** smatzek has joined #openstack-nova | 10:57 | |
| vdrok | same here :( | 10:59 |
| *** ioggstream has quit IRC | 11:01 | |
| *** yassine has joined #openstack-nova | 11:06 | |
| *** chyka has joined #openstack-nova | 11:07 | |
| *** iceyao has joined #openstack-nova | 11:10 | |
| sdague | vdrok: actually, I might have figured it out | 11:10 |
| sdague | we were missing one last conditional | 11:11 |
| sdague | https://review.openstack.org/487809 | 11:11 |
| *** chyka has quit IRC | 11:11 | |
| sdague | I changed the ironic patch to depend on that one | 11:12 |
| *** testuro has quit IRC | 11:14 | |
| *** iceyao has quit IRC | 11:15 | |
| *** lucasagomes has joined #openstack-nova | 11:16 | |
| *** kazsh has quit IRC | 11:18 | |
| *** kazsh has joined #openstack-nova | 11:19 | |
| takashin | oomichi: Are you around? | 11:20 |
| *** alexchadin has quit IRC | 11:20 | |
| *** Ritesh has joined #openstack-nova | 11:21 | |
| vdrok | sdague: cool, thank you! | 11:23 |
| sdague | vdrok: fingers crossed | 11:23 |
| *** amuresan has quit IRC | 11:23 | |
| Ritesh | Hi, | 11:24 |
| Ritesh | Can anyone please review this patch | 11:24 |
| Ritesh | review pending since long time | 11:24 |
| Ritesh | https://review.openstack.org/#/c/457886/ | 11:24 |
| *** vishwanathj has quit IRC | 11:26 | |
| *** vishwanathj has joined #openstack-nova | 11:26 | |
| openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | 11:32 |
| *** chyka has joined #openstack-nova | 11:33 | |
| *** chyka has quit IRC | 11:37 | |
| *** iceyao has joined #openstack-nova | 11:37 | |
| *** esberglu has joined #openstack-nova | 11:40 | |
| *** jjung has quit IRC | 11:42 | |
| *** iceyao has quit IRC | 11:42 | |
| *** kylek3h has joined #openstack-nova | 11:42 | |
| *** kylek3h has quit IRC | 11:42 | |
| *** edmondsw has joined #openstack-nova | 11:42 | |
| *** jamesden_ has joined #openstack-nova | 11:44 | |
| *** esberglu has quit IRC | 11:44 | |
| vdrok | sdague: ouch http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cpu.txt.gz#_Jul_27_11_32_29_018076 | 11:45 |
| *** jamesdenton has quit IRC | 11:45 | |
| sdague | vdrok: ok, the conductor didn't get the right config file - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cond.txt.gz#_Jul_27_11_29_55_156482 | 11:47 |
| *** iceyao has joined #openstack-nova | 11:47 | |
| *** tuanluong has quit IRC | 11:47 | |
| sdague | that's solvable | 11:47 |
| *** edmondsw has quit IRC | 11:47 | |
| sdague | vdrok: ok, lets try again | 11:50 |
| vdrok | yup :) | 11:51 |
| *** litao__ has quit IRC | 11:51 | |
| openstackgerrit | Yaroslav proposed openstack/nova master: Added Overbooking DB quota driver https://review.openstack.org/484359 | 11:54 |
| *** thorst has joined #openstack-nova | 11:54 | |
| *** mamandle has quit IRC | 11:55 | |
| *** baoli has joined #openstack-nova | 11:56 | |
| *** kylek3h has joined #openstack-nova | 12:02 | |
| sean-k-mooney | o/ would people object to adding a numa aware varient of reserved_host_memory_mb in queens. we have been seeing OOM errors on our ci systems pereodically for some time even when the system has several GB of free ram. | 12:03 |
| sean-k-mooney | turns out that the OOM killer kicks in when any numa node is out of memory and hte kernel need to alloacte memory there not when the system is out of memory globally | 12:04 |
| *** jjung has joined #openstack-nova | 12:07 | |
| *** Ritesh has quit IRC | 12:09 | |
| *** catintheroof has joined #openstack-nova | 12:11 | |
| *** _nick is now known as yankcrime | 12:11 | |
| *** kylek3h has quit IRC | 12:11 | |
| *** kiennt has quit IRC | 12:11 | |
| *** pilgrimstack has quit IRC | 12:12 | |
| *** p01nt3r75 has joined #openstack-nova | 12:12 | |
| *** pilgrimstack has joined #openstack-nova | 12:13 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | 12:14 |
| *** yamamoto has quit IRC | 12:14 | |
| *** pilgrimstack has left #openstack-nova | 12:16 | |
| *** kbaegis has joined #openstack-nova | 12:18 | |
| *** yushb has joined #openstack-nova | 12:20 | |
| *** abhishekk has quit IRC | 12:20 | |
| *** mamandle has joined #openstack-nova | 12:21 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | 12:21 |
| *** bhagyashris has joined #openstack-nova | 12:22 | |
| *** acormier has joined #openstack-nova | 12:23 | |
| *** mdnadeem has quit IRC | 12:23 | |
| catintheroof | Hi, does anyone has a good guide on how to configure live migration on ocata ? doing the same that worked as of mitaka, doesnt anymore | 12:24 |
| *** gongysh has quit IRC | 12:24 | |
| *** psachin has quit IRC | 12:26 | |
| *** liverpooler has joined #openstack-nova | 12:26 | |
| *** mriedem has joined #openstack-nova | 12:27 | |
| mriedem | oh hi everybody | 12:27 |
| vdrok | sdague: gah, same thing, compute does not get the build instance request http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz | 12:28 |
| vdrok | morning mriedem | 12:28 |
| openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | 12:29 |
| mriedem | vdrok: did my attempts at dumping the rabbitmqctl report ever work? | 12:29 |
| vdrok | mriedem: seems like that file is not there http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/rabbitmq/ | 12:30 |
| mriedem | yeah...i was getting permission denied errors | 12:30 |
| mriedem | http://logs.openstack.org/64/487664/2/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/c20a804/logs/devstack-gate-cleanup-host.txt | 12:31 |
| mriedem | 2017-07-27 03:38:03.562 | /home/jenkins/workspace/gate-swift-dsvm-functional-ubuntu-xenial-nv/devstack-gate/functions.sh: line 669: /var/log/rabbitmq/cleanup-host-report.txt: Permission denied | 12:31 |
| mriedem | i was thinking maybe because i couldn't write to /var/log/rabbitmq, so i moved it but then it's not even getting called | 12:31 |
| mriedem | https://review.openstack.org/#/c/487664/3/functions.sh | 12:32 |
| mriedem | maybe because we don't have anything in /opt/stack/logs/rabbitmq | 12:32 |
| *** karthiks has quit IRC | 12:32 | |
| mriedem | heh, this is wrong anyway | 12:33 |
| mriedem | if [ -f $BASE/logs/rabbitmq/ ]; then | 12:33 |
| mriedem | -f should be -d | 12:33 |
| *** edmondsw has joined #openstack-nova | 12:33 | |
| vdrok | hrm :) | 12:34 |
| *** Guest19469 has quit IRC | 12:35 | |
| sdague | vdrok: well conductor seems like it's working | 12:35 |
| vdrok | is there some kind of diagram for dummies about who calls what in cellsv2 setup? | 12:36 |
| mriedem | vdrok: https://review.openstack.org/#/c/487183 is a start | 12:36 |
| mriedem | vdrok: with singleconductor being set, it should be the same as before the multi-conductor thing | 12:37 |
| vdrok | sdague: yup, that is solved :) | 12:37 |
| vdrok | mriedem: thanks! | 12:37 |
| mriedem | dansmith and i were looking at it last night, and n-cpu, n-sch and n-super-cond are all using the same nova.conf with the same transport_url | 12:37 |
| sdague | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz#_Jul_27_12_08_16_707963 - what's up here? | 12:38 |
| mriedem | i traced the build request from super-conductor to scheduler back to super-conductor to the point that super-conductor should rpc cast to nova-compute | 12:38 |
| sdague | That looks like ironic API failed | 12:38 |
| mriedem | but the request never got to n-cpu | 12:38 |
| mriedem | sdague: that's latent | 12:38 |
| sdague | mriedem: oh... would be nice to fix that :P | 12:38 |
| mriedem | sdague: vdrok: opened a bug for that yesterday https://bugs.launchpad.net/ironic/+bug/1706772 | 12:38 |
| openstack | Launchpad bug 1706772 in Ironic "InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on startup with Ironic driver" [Undecided,Confirmed] | 12:38 |
| mriedem | sdague: sure, but it's some keystone thing in ironic-api | 12:38 |
| mriedem | endpoint discovery blows up | 12:39 |
| sdague | mriedem: ok | 12:39 |
| mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-ir-api.txt.gz#_Jul_27_12_08_16_701856 | 12:39 |
| mriedem | so we ended last night knowing that conductor and scheduler are working but when super conductor should cast to n-cpu, everything stops there | 12:39 |
| *** dgonzalez_ has joined #openstack-nova | 12:40 | |
| *** seife_ has joined #openstack-nova | 12:40 | |
| *** iceyao has quit IRC | 12:40 | |
| mriedem | super-conductor logs stop, rabbitmq logs stop | 12:40 |
| mriedem | and the request isn't going to like the cell1 conductor, there is nothing in it's logs | 12:40 |
| mriedem | so i've got the d-g patch to try and dump the rabbitmqctl report when cleaning up the host and collecting logs | 12:40 |
| mriedem | but so far i haven't had a run with that work yet | 12:41 |
| mriedem | https://review.openstack.org/#/c/487664/ | 12:41 |
| *** acormier has quit IRC | 12:41 | |
| *** seife_ has quit IRC | 12:41 | |
| *** dgonzalez_ has quit IRC | 12:41 | |
| *** acormier has joined #openstack-nova | 12:42 | |
| vdrok | mriedem: sdague also uploaded this one https://review.openstack.org/487809 | 12:43 |
| sdague | mriedem: the fix was definitely wrong last night, it was still spawning 2 conductors | 12:44 |
| *** acormier has quit IRC | 12:46 | |
| vdrok | sdague: mriedem: regarding that bug, it seems like it's the same thing again, http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/apache/error.txt.gz, apache was being restarted at that moment | 12:47 |
| *** pchavva has joined #openstack-nova | 12:47 | |
| vdrok | we did work around it already a couple of times | 12:47 |
| sdague | vdrok: why apache getting restarted? | 12:48 |
| vdrok | sdague: previously it was because of some configuration changes, not sure why now, need to look into logs | 12:49 |
| sdague | anyway... the main issue | 12:49 |
| sdague | mriedem: so with this nova.conf - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova.conf.txt.gz | 12:50 |
| mriedem | sdague: yeah the 2 conductors was the culprit, but dan and i couldn't see or think of anything about why that would be causing issues | 12:50 |
| sdague | how do we know where cell0 db is? | 12:50 |
| mriedem | sdague: conductor gets the cell mappings from the api db, | 12:50 |
| mriedem | and the cell mapping has the cell0 mq and db urls in it | 12:50 |
| *** sree has joined #openstack-nova | 12:50 | |
| mriedem | cdent: replied some and asked a question about another test in here https://review.openstack.org/#/c/487589/ | 12:51 |
| cdent | yeah, just reading/responding to that. thanks | 12:51 |
| *** links has quit IRC | 12:52 | |
| mriedem | cdent: the *best* way to tell if an instance is being migrated would be to check if it has a migration_context attribute in it | 12:52 |
| sdague | mriedem: ok, so which config file should the single conductor be running off of? | 12:52 |
| mriedem | sdague: nova.conf | 12:52 |
| sdague | which gives it a transport url of - transport_url = rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/ | 12:53 |
| sdague | 12:53 | |
| mriedem | cdent: i share concerns about relying on the existing allocations to know if we're doing a migration or not | 12:54 |
| mriedem | because of (1) timing and (2) weird operatoins like soft delete and shelve | 12:54 |
| mriedem | and the wonkiness that is the RT | 12:54 |
| mriedem | sdague: so in the singleconductor case, i think the only nova config that matters is nova.conf http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/ | 12:55 |
| *** sree has quit IRC | 12:55 | |
| mriedem | nova-cpu.conf and nova_cell1.conf aren't used | 12:55 |
| mriedem | nova-cpu.conf wouldn't even work b/c it doesn't have any information in it about which compute driver to use, or how to talk to placement/cinder/neutron/etc | 12:55 |
| *** ratailor has quit IRC | 12:56 | |
| mriedem | the cell1 mapping is created here: | 12:56 |
| mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/devstacklog.txt.gz#_2017-07-27_12_04_56_099 | 12:56 |
| mriedem | nova-manage --config-file /etc/nova/nova.conf --config-file /etc/nova/nova_cell1.conf cell_v2 create_cell --name cell1 | 12:56 |
| *** ratailor has joined #openstack-nova | 12:56 | |
| mriedem | note that is using nova_cell1.conf rather than nova.conf | 12:56 |
| sdague | mriedem: sure, where would the message queue get set | 12:56 |
| sdague | maybe that's the missing piece, dumping the cell mappings | 12:57 |
| mriedem | well we know the mq and db for cell1, it's taken from http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova_cell1.conf.txt.gz | 12:57 |
| sdague | right | 12:57 |
| sdague | but that's not used anywhere | 12:57 |
| *** ratailor has quit IRC | 12:57 | |
| mriedem | it's used when creating the cell1 mapping | 12:58 |
| sdague | so that's stating that the cell1 mq is going to be on a vhost | 12:58 |
| mriedem | nova-manage --config-file /etc/nova/nova.conf --config-file /etc/nova/nova_cell1.conf cell_v2 create_cell --name cell1 | 12:58 |
| sdague | but nova-compute is started not listening to that vhost | 12:58 |
| mriedem | right nova-compute is listening on rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/ | 12:59 |
| mriedem | the cell1 mapping is sending to rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/nova_cell1 | 12:59 |
| sdague | but conductor isn't sending messages there, right? | 12:59 |
| mriedem | because https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1042 | 13:00 |
| sdague | https://github.com/openstack-dev/devstack/blob/9596fdddccd04c26aa5adb923b9bd8e64c6593ec/lib/nova#L709-L712 | 13:00 |
| mriedem | conductor does an mq switch when it casts to compute | 13:00 |
| sdague | mriedem: ok, so are you agreeing or disagreeing with me that nova-cond and nova-compute aren't talking on the same mq :) | 13:01 |
| mriedem | can i phone a friend? | 13:01 |
| mriedem | i think i'm agreeing, | 13:01 |
| mriedem | and that would explain why we're not seeing the message get to n-cpu | 13:01 |
| *** gcb has quit IRC | 13:02 | |
| sdague | yeh | 13:02 |
| sdague | is there a nova-manage command to dump the cell mappings? | 13:02 |
| sdague | I think that's kind of critical to see the mismatch | 13:02 |
| *** lucasxu has joined #openstack-nova | 13:03 | |
| mriedem | yes, | 13:03 |
| mriedem | nova-manage cell_v2 list_cells | 13:03 |
| *** gcb has joined #openstack-nova | 13:04 | |
| mriedem | with --verbose | 13:04 |
| mriedem | --verbose dumps the db and mq urls | 13:04 |
| *** mamandle has quit IRC | 13:04 | |
| mriedem | fwiw, this is a run before the fleetify patch where we create cell1 | 13:05 |
| mriedem | http://logs.openstack.org/23/485823/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/206b79e/logs/devstacklog.txt.gz#_2017-07-21_01_52_17_794 | 13:05 |
| mriedem | nova-manage cell_v2 create_cell --transport-url rabbit://stackrabbit:secretrabbit@10.0.1.31:5672/ --name cell1 | 13:05 |
| mriedem | there is a single nova.conf | 13:05 |
| mriedem | and it's using the same transport_url http://logs.openstack.org/23/485823/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/206b79e/logs/etc/nova/nova.conf.txt.gz | 13:06 |
| mriedem | so yeah, i think that's the problem, the cell1 mapping is using the nova_cell1 mq and nova-compute is using the main mq | 13:06 |
| sdague | ok, fix proposed | 13:08 |
| sdague | including the dump of the cell mapping | 13:08 |
| mriedem | i left a comment in ps2, | 13:10 |
| mriedem | but it looks like you addressed it in ps4 | 13:10 |
| mriedem | with https://review.openstack.org/#/c/487809/4/lib/nova@592 | 13:11 |
| *** lyan has joined #openstack-nova | 13:11 | |
| dansmith | mriedem: oh snap.. just woke up, but excellent call | 13:14 |
| *** yamamoto has joined #openstack-nova | 13:14 | |
| *** cristicalin has joined #openstack-nova | 13:15 | |
| mriedem | sdague connected the dots for me, | 13:15 |
| mriedem | plus sleep helps | 13:15 |
| dansmith | I should have thought of that | 13:15 |
| mriedem | hard to think of anything at the end of a day like yesterday | 13:15 |
| sdague | mriedem: oh, sorry, I wasn't even looking at comments, I was just reading logs trying to understand | 13:16 |
| mriedem | cdent: regarding functional testing, i was thinking the same yesterday, but didn't have time, | 13:17 |
| vdrok | morning dansmith , missed all the fun :) | 13:17 |
| dansmith | vdrok: me too, but.. good morning vdrok :P | 13:18 |
| mriedem | but was thinking it could be relatively simple to write a functional test that starts 2 compute services, creates a server, checks allocations are just on the source host, does a resize to the 2nd compute, checks allocations are retained for both nodes, then does a resize confirm (and another test that does a resize revert), and then validates allocations after that is done | 13:18 |
| cdent | relatively | 13:19 |
| mriedem | cdent: but me working on something like that probably can't happen until tomorrow at this rate | 13:19 |
| * cdent nods | 13:19 | |
| mriedem | i actually have *gasp* family commitments tonight | 13:19 |
| * cdent gasps | 13:19 | |
| cdent | I’m sicker than sick at the moment, but bored enough to still be hanging out | 13:20 |
| sean-k-mooney | mriedem: hi i know your pretty busy with the ff today but any chance of inlcuding the 1 patch for mellanox's ovs offload and the 2 required for our nic feature based scheudling blueprint in pike or will they be pushed to queens? | 13:21 |
| *** yamamoto has quit IRC | 13:21 | |
| kashyap | mdbooth: When you get a moment, hope I addressed all of your remarks here: "libvirt: Post-migration, set cache value for Cinder volume(s)" -- https://review.openstack.org/#/c/485752/ | 13:22 |
| gibi | mriedem: do we already have functional test that boot VMs which really uses the placement service. If there is such then I can try to put together the above suggested resize test | 13:22 |
| mriedem | sean-k-mooney: i think the ovs offload from moshele is probably doable - i think i'd like a bp in nova for that though, just something simple to mirrow the neutron RFE | 13:22 |
| mriedem | since nova doesn't do RFE bugs for blueprints | 13:22 |
| mriedem | gibi: we do | 13:22 |
| mriedem | gibi: the PlacementFixture is used in the _IntegratedHelpersMixin | 13:23 |
| mriedem | which several functional tests use | 13:23 |
| gibi | mriedem: cool, then I will put something together | 13:23 |
| mriedem | there are other functional regression tests which don't use that mixin but still use the placement fixture to avoid warnings from the compute services in the logs | 13:23 |
| gibi | mriedem: I anyhow wanted to test VM moving with placement | 13:23 |
| *** esberglu has joined #openstack-nova | 13:23 | |
| mriedem | gibi: cool, that would be something that builds on https://review.openstack.org/#/c/483566/ | 13:23 |
| mriedem | the other tricky thing though is the test has to use the filter scheduler, | 13:24 |
| sean-k-mooney | mriedem: ok good to know. i saw that netronomes patchs seem to have merged so im sure jangutter is happy with that it would be nice to finish netronomes version too. im sure it will make moshele equally happy | 13:24 |
| mriedem | lots of our functional tests just use the chance scheduler | 13:24 |
| *** esberglu has quit IRC | 13:24 | |
| mriedem | gibi: so you'd have to use the filter scheduler with a minimal set of filters enabled | 13:24 |
| *** esberglu has joined #openstack-nova | 13:24 | |
| mriedem | like RetryFilter and ComputeFilter i think | 13:24 |
| gibi | mriedem: I'm aware of that based on the server group functional tests | 13:24 |
| sdague | dansmith / mriedem is there a reason to not make --verbose the default behavior for list_cells | 13:24 |
| *** alexchadin has joined #openstack-nova | 13:25 | |
| sean-k-mooney | s/netronome/mellanox/ | 13:25 |
| dansmith | sdague: makes it harder to be used by scripts | 13:25 |
| mriedem | gibi: this is one that i worked on https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py | 13:25 |
| mriedem | host most of the things yo'ud need i think | 13:25 |
| sdague | I'm trying to understand the usage for just cellid | 13:25 |
| jangutter | sean-k-mooney, mriedem: yep, I'm pretty happy with moshele's patch -> it's cross-platform enough for us to use as well. | 13:25 |
| dansmith | mriedem: sdague: is someone fixing the cell mapping? | 13:25 |
| mriedem | 2 nodes | 13:25 |
| mriedem | filter scheduler + placement | 13:25 |
| sdague | dansmith: in what way? | 13:25 |
| mriedem | you'd just need to remove the part about it making one compute fail | 13:25 |
| gibi | mriedem: cool, thanks for the pointer | 13:25 |
| sdague | dansmith: doesn't my ps4 do that? | 13:26 |
| dansmith | sdague: I haven't looked | 13:26 |
| mriedem | dansmith: https://review.openstack.org/#/c/487809/4 | 13:26 |
| mriedem | sdague: we don't use --verbose by default for list_cells since it dumps your transport and db mqs | 13:26 |
| mriedem | s/mqs/urls/ | 13:26 |
| mriedem | which have creds in them | 13:26 |
| sdague | mriedem: ok, so run it through secret filter | 13:26 |
| *** iceyao has joined #openstack-nova | 13:27 | |
| mriedem | do we have a secret filter? | 13:27 |
| sdague | I guess, it's not really very discoverable that to get the good stuff you need --verbose | 13:27 |
| *** yingjun_1 has joined #openstack-nova | 13:27 | |
| mriedem | --dump-your-secrets? | 13:27 |
| mriedem | --hand-over-your-cash? | 13:27 |
| sdague | https://github.com/openstack/oslo.utils/blob/f69aa3c3866f1d1800184b6c905c77988278cd18/oslo_utils/strutils.py#L248 | 13:28 |
| *** eharney has joined #openstack-nova | 13:28 | |
| mriedem | sdague: that's not going to do what you want | 13:28 |
| dansmith | and anyone scripting with this already will be very confused if they stop getting a clean list of uuids | 13:28 |
| mriedem | sdague: that's based on known keys https://github.com/openstack/oslo.utils/blob/f69aa3c3866f1d1800184b6c905c77988278cd18/oslo_utils/strutils.py#L57 | 13:28 |
| mriedem | the url doesn't have keys in it | 13:28 |
| sdague | mriedem: sure, I'm just saying it's pretty solvable | 13:28 |
| mriedem | sure, we could parse the url and mask the user/password part | 13:29 |
| sdague | dansmith: ok, well as someone not steeped in it, it's not obvious how you get the full mapping | 13:29 |
| mriedem | and rebuild the url before displaying | 13:29 |
| mriedem | so let's throw that as a debug/improvement idea on the big etherpad | 13:30 |
| *** hshiina has quit IRC | 13:30 | |
| sdague | http://paste.openstack.org/show/616726/ | 13:30 |
| sdague | that's kind of the brute path I'd expect an operator to go through trying to figure it out | 13:30 |
| bhagyashris | cdent, gibi, jaypipes: Hi I am trying to test the shared resource provider thing on the current master but facing some issue to create the instance I have followed the steps mentioned in [1]: http://paste.openstack.org/show/616724/ and got the error at n-cond and n-cpu side as mentioned in [2]: http://paste.openstack.org/show/616725/ | 13:31 |
| *** gouthamr has joined #openstack-nova | 13:31 | |
| dansmith | sdague: making help work seems like an obvious improvement we should make | 13:31 |
| mriedem | the nova-manage help system is a latent sucky problem right? | 13:31 |
| cdent | bhagyashris: looking | 13:32 |
| mriedem | maybe cliff or whatever stephenfin wanted to convert it to helps there | 13:32 |
| mriedem | so -h is implied if you don't specify args | 13:32 |
| bhagyashris | cdent, gibi, jaypipes: without using the nfs i am able to create the instance successfully. | 13:32 |
| *** cleong has joined #openstack-nova | 13:32 | |
| bhagyashris | cdent: ok thank you :) | 13:32 |
| sdague | so I actually don't understand the argument about the extra columns being a problem | 13:33 |
| sdague | because if you are parsing those tables | 13:33 |
| sdague | you are already doing awk positional | 13:33 |
| sdague | or something equiv | 13:33 |
| dansmith | sdague: yeah, this is actually not the thing I was thinking of | 13:33 |
| * dansmith is on a 6am call | 13:33 | |
| sdague | dansmith: ok, once you are off, we can chat then :) | 13:33 |
| dansmith | sdague: so not opposed to it being verbose all the time, but sounds like obscuring the url is something mriedem thinks is important | 13:34 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add support for Windows network commands https://review.openstack.org/487405 | 13:34 |
| *** alexchadin has quit IRC | 13:35 | |
| gibi | bhagyashris: that is something new for me. It seems to be specific to your shared disk case | 13:35 |
| sdague | dansmith: I'm totally fine with password masking by default | 13:35 |
| sdague | I think that's very reasonable | 13:35 |
| bhagyashris | gibi: yes | 13:35 |
| cdent | bhagyashris: are you able to tell if the conductor started the process of telling the compute to build the instance? If so, then the placement side of things is working properly, and whatever has gone wrong there with the lock may be the issue. From what you’ve pasted it is hard to tell what’s going on | 13:36 |
| mriedem | god it's awesome how the docs migration has broken all cached urls https://docs.openstack.org/nova/latest/cells.html | 13:37 |
| cdent | quite | 13:37 |
| mriedem | search for "docs nova-manage list_cells" in google and ^ is the first hit | 13:37 |
| mriedem | and it's a 404 | 13:37 |
| *** mamandle has joined #openstack-nova | 13:37 | |
| cdent | I would have thought there would be redirects | 13:38 |
| moshele | mriedem: ok I will create bp | 13:38 |
| *** jdillaman has quit IRC | 13:38 | |
| mriedem | cdent: me too | 13:38 |
| mriedem | but maybe we need to do those explicitly now | 13:39 |
| mriedem | i'm complaining in #openstack-doc | 13:39 |
| *** efried_zzz is now known as efried | 13:40 | |
| *** tbachman has joined #openstack-nova | 13:41 | |
| sdague | hmmm no jay yet? He had wanted this merged, but it needed a reno - https://review.openstack.org/#/c/357726/ | 13:41 |
| *** yingjun_1 has quit IRC | 13:42 | |
| mriedem | how about we hold that for queens | 13:42 |
| dims | mriedem : it's under /user, no? | 13:42 |
| bhagyashris | cdent: ohk, Just wanted to confirm one thing. the steps i have mentioned in the [1]: http://paste.openstack.org/show/616724/ for nfs shared resource provider thing to test is correct? | 13:42 |
| cdent | bhagyashris: let me read through that again, to be sure | 13:43 |
| *** yingjun_1 has joined #openstack-nova | 13:43 | |
| mriedem | dims: yes, but i know that, someone else won't | 13:43 |
| mriedem | AJaeger is going to rebuild the site map | 13:43 |
| mriedem | to see if that fixes it | 13:43 |
| bhagyashris | cdent: ohk thank you. because without nfs configurations i am able to create the instance successfully. | 13:44 |
| bhagyashris | cdent: I have gone through all the blueprints related to shared resource provider thing all are implemented. Based upon that I have started to test this. | 13:46 |
| cdent | bhagyashris: the steps you’ve listed look mostly correct. the part that I have no insight into is step 2. Do you know that that part is working? | 13:46 |
| mriedem | YES http://logs.openstack.org/64/487664/4/check/gate-neutron-dsvm-functional-ubuntu-xenial-nv/bbc569f/logs/rabbitmq/cleanup-host-report.txt.txt.gz | 13:47 |
| mriedem | dansmith: ^ | 13:47 |
| mriedem | got the rabbitmq report | 13:47 |
| bhagyashris | cednt: yeaj its working in that I have just created nfs sever on one host and mounted that nfs on compute node A and B | 13:48 |
| dansmith | mriedem: but it's empty | 13:48 |
| mriedem | that's not from a run with the ironic patch though, but got it working | 13:48 |
| mriedem | right, i'm running that through with the ironic stack now | 13:48 |
| dansmith | mriedem: okay | 13:48 |
| dansmith | mriedem: but we're waiting on his PS4 with hopeful eyes right? | 13:49 |
| gibi | bhagyashris: the steps in your paste also looks good to me except step 7 | 13:49 |
| *** roeyc has quit IRC | 13:49 | |
| mriedem | dansmith: yes | 13:49 |
| gibi | bhagyashris: I think there you have to put the compute resource_providers to the same aggregate | 13:49 |
| mriedem | having the rabbitmq dump in ci runs is useful anyway | 13:49 |
| dansmith | mriedem: totes | 13:49 |
| *** mtanino_ has joined #openstack-nova | 13:49 | |
| *** jaypipes has joined #openstack-nova | 13:50 | |
| mriedem | nova super meeting in 10 minutes | 13:50 |
| mriedem | i'll let jaypipes do the supernovas unite call | 13:50 |
| bhagyashris | gibi: yeah that i Have done for the compute resource provider too | 13:50 |
| gibi | bhagyashris: then it is looks good for me | 13:51 |
| bhagyashris | cdent: I have given reference of step 2 here: http://paste.openstack.org/show/616729/ | 13:52 |
| bhagyashris | gibi: ohk | 13:52 |
| mriedem | vdrok: dansmith: sdague: the ironic patch is looking good now | 13:52 |
| mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/214efb9/ | 13:52 |
| dansmith | ttfl | 13:52 |
| jaypipes | supernovas unite. | 13:53 |
| mriedem | cell1 MQ is back on the same as the others http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/214efb9/logs/etc/nova/nova_cell1.conf.txt.gz | 13:53 |
| jangutter | I guess you guys have seen https://what-if.xkcd.com/73/ | 13:54 |
| *** jjung has quit IRC | 13:54 | |
| jaypipes | jangutter: heh, no, I hadn't. nice though :) | 13:54 |
| cdent | bhagyashris: based on the error message you are getting I would guess your nfs server is not set up in a way to allow file locks over nfs. which is not something we can really diagnose and resolve here | 13:55 |
| jangutter | "The physicist who mentioned this problem to me told me his rule of thumb for estimating supernova-related numbers: However big you think supernovae are, they're bigger than that." | 13:55 |
| *** udesale has quit IRC | 13:55 | |
| *** awaugama has joined #openstack-nova | 13:55 | |
| *** kristia__ has joined #openstack-nova | 13:55 | |
| dansmith | jangutter: well, I'll have you know I've been watching what I eat and exercising every day, so I'm hoping to not be any bigger than I expect | 13:56 |
| *** cristicalin has quit IRC | 13:56 | |
| *** baoli has quit IRC | 13:57 | |
| bhagyashris | cdent: ohk. troubleshooting... | 13:57 |
| openstackgerrit | Chris Dent proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | 13:57 |
| *** kristian__ has quit IRC | 13:57 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | 13:58 |
| cdent | jaypipes: did some test fixes on the first of those ^ | 13:58 |
| openstackgerrit | Sean Dague proposed openstack/nova master: WIP: always show urls in list_cells https://review.openstack.org/487860 | 13:58 |
| sdague | mriedem / dansmith ^^^ is the no tests version of changing list_cells behavior | 13:58 |
| jaypipes | cdent: yeah, saw you working on that, thank you Chris! | 13:59 |
| sdague | if people are generally good with that, I'll fill in tests | 13:59 |
| jaypipes | cdent: honestly, I kinda knocked out that patch in 20 minutes after a conversation with dansmith on IRC | 13:59 |
| cdent | fair | 13:59 |
| sdague | mriedem: nice! would be good to get ironic back up and running | 13:59 |
| bhagyashris | cdent, gibi: thank you. :) Trying to solve this | 13:59 |
| sdague | ah... jaypipes returned | 13:59 |
| mriedem | sdague: there is a sqla db url parser thing we use elsewhere in _map_cell0 | 13:59 |
| openstackgerrit | Moshe Levi proposed openstack/nova master: hardware offload support for openvswitch https://review.openstack.org/398265 | 13:59 |
| mriedem | might be better to use that, but would have to look later | 14:00 |
| mriedem | meeting time | 14:00 |
| sdague | jaypipes: you can wanted this merged in may - https://review.openstack.org/#/c/357726/ | 14:00 |
| sdague | I fixed up my one issue with it | 14:00 |
| mriedem | sdague: jaypipes: as i said, can we hold that for queens? | 14:01 |
| mamandle | sfinucan: mriedem: updated patch is out for review for https://review.openstack.org/#/c/483911/, can you please take a look if possible? Thanks! | 14:01 |
| sdague | mriedem: if you want, is there a reason for that? | 14:01 |
| mriedem | risk | 14:01 |
| mriedem | vs need | 14:01 |
| sdague | ok, I guess the comments in there were that it was pretty uncontroversial | 14:02 |
| *** baoli has joined #openstack-nova | 14:02 | |
| jaypipes | mriedem: I'd like to get it in Pike. I just don't see what risk is being abated by holding that patch until Queens. | 14:02 |
| mriedem | can we talk about it after the meeting? | 14:03 |
| jaypipes | mriedem: the only risk it introduces is for deployments who disable port binding extension, and there's been no evidence of any of those. | 14:03 |
| jaypipes | mriedem: sure thing. | 14:03 |
| jaypipes | sorry | 14:03 |
| *** mvk has quit IRC | 14:06 | |
| *** prateek has quit IRC | 14:06 | |
| *** acormier has joined #openstack-nova | 14:06 | |
| *** saphi has joined #openstack-nova | 14:10 | |
| *** yingjun_1 has quit IRC | 14:12 | |
| *** kristia__ has quit IRC | 14:12 | |
| *** ekuris has quit IRC | 14:14 | |
| *** sree has joined #openstack-nova | 14:14 | |
| *** saphi has quit IRC | 14:14 | |
| *** jjung has joined #openstack-nova | 14:14 | |
| *** trinaths1 has left #openstack-nova | 14:16 | |
| openstackgerrit | Moshe Levi proposed openstack/nova master: hardware offload support for openvswitch https://review.openstack.org/398265 | 14:16 |
| *** saphi has joined #openstack-nova | 14:16 | |
| *** sree has quit IRC | 14:18 | |
| *** jamesden_ has quit IRC | 14:23 | |
| *** burt has joined #openstack-nova | 14:24 | |
| *** jamesdenton has joined #openstack-nova | 14:25 | |
| openstackgerrit | Moshe Levi proposed openstack/nova master: hardware offload support for openvswitch https://review.openstack.org/398265 | 14:30 |
| *** lucasxu has quit IRC | 14:30 | |
| *** kristian__ has joined #openstack-nova | 14:31 | |
| *** moshele has quit IRC | 14:32 | |
| *** lucasxu has joined #openstack-nova | 14:32 | |
| *** vladikr has joined #openstack-nova | 14:33 | |
| *** brad[] has quit IRC | 14:33 | |
| *** marst has joined #openstack-nova | 14:33 | |
| *** brad[] has joined #openstack-nova | 14:34 | |
| *** dtantsur|bbl is now known as dtantsur | 14:39 | |
| *** armax has joined #openstack-nova | 14:39 | |
| *** jamesdenton has quit IRC | 14:39 | |
| *** jamesdenton has joined #openstack-nova | 14:40 | |
| *** kaisers_ has joined #openstack-nova | 14:41 | |
| *** jjung has quit IRC | 14:41 | |
| *** jjung has joined #openstack-nova | 14:43 | |
| *** saphi_ has joined #openstack-nova | 14:43 | |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted certificates to InstanceExtras https://review.openstack.org/457711 | 14:44 |
| *** saphi has quit IRC | 14:44 | |
| mriedem | vdrok: sdague: dansmith: ironic patch failed multinode and grenade https://review.openstack.org/#/c/487458/ | 14:44 |
| mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial/6cf92e0/logs/subnode-2/devstacklog.txt.gz#_2017-07-27_14_10_41_647 | 14:45 |
| mriedem | ah | 14:45 |
| mriedem | sdague: you can't run list_cells from the subnode | 14:46 |
| mriedem | https://review.openstack.org/#/c/487809/4/lib/nova@953 | 14:47 |
| vdrok | mriedem: yup, saw that. seems like I need to add the same variable to grenade settings | 14:47 |
| mriedem | so, running list_cells can only happen on the primary, which is done before the subnodes, so list_cells is pretty useless there | 14:47 |
| mriedem | what we'd really want is to call back into the primary from devstack-gate to run list_cells at the very end | 14:47 |
| mriedem | but i'd make that a separate effort from what's being fixed in https://review.openstack.org/#/c/487809/ | 14:48 |
| mriedem | sdague: ^ agree? | 14:48 |
| sdague | mriedem: why is list_cells useless there? | 14:49 |
| *** sree has joined #openstack-nova | 14:49 | |
| sdague | the cells are all configured on the primary, right? | 14:49 |
| mriedem | yeah, good point | 14:49 |
| sdague | I do understand the issue around not running on subnode | 14:50 |
| mriedem | ok so just conditional on n-api and we're good there | 14:50 |
| * mriedem looks at the grenade failures | 14:50 | |
| sdague | though, it seemed to work fine | 14:50 |
| sdague | the grenade failure for ironic is because they are using the old param to start_compute | 14:51 |
| sdague | that got broken with our introduction of CELLSV2_SETUP | 14:51 |
| mriedem | yeah vdrok just pushed a change for that | 14:52 |
| mriedem | sdague: ok so you want to update the devstack patch or i can | 14:52 |
| sdague | mriedem: so... this did pass on the subnodes | 14:52 |
| vdrok | I'll remove the nomulticell flag that we did introduce in a later patch | 14:52 |
| mriedem | huh? | 14:52 |
| mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial/6cf92e0/logs/subnode-2/devstacklog.txt.gz#_2017-07-27_14_10_41_647 | 14:52 |
| mriedem | sdague: ^ is the stack on the subnode | 14:52 |
| *** vishwanathj has quit IRC | 14:53 | |
| *** Swami has joined #openstack-nova | 14:53 | |
| sdague | oh, sorry, it was an nv job | 14:53 |
| *** vishwanathj has joined #openstack-nova | 14:54 | |
| *** kristian__ has quit IRC | 14:54 | |
| sdague | mriedem: so, wrap it in n-api enabled? | 14:54 |
| mriedem | sdague: i just updated it | 14:55 |
| *** mamandle has quit IRC | 14:55 | |
| sdague | mriedem: ++ | 14:55 |
| *** sridharg has quit IRC | 14:56 | |
| sdague | mriedem / dansmith - https://review.openstack.org/#/c/487246/ my proposed solution on the nova-compute wait for ready | 14:58 |
| sdague | it seems to have worked on the 3 node job correctly | 14:58 |
| *** lucasxu has quit IRC | 14:59 | |
| *** raunak has joined #openstack-nova | 15:01 | |
| *** raunak has quit IRC | 15:01 | |
| *** lucasxu has joined #openstack-nova | 15:02 | |
| mriedem | oh that's devstack, i thought that was nova | 15:02 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | 15:02 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | 15:02 |
| sdague | mriedem: right, it's a devstack change | 15:03 |
| *** kristian__ has joined #openstack-nova | 15:03 | |
| *** ys__ has joined #openstack-nova | 15:03 | |
| sdague | but trying to put it in devstack directly instead of more orchestration in d-g that gets messy | 15:03 |
| *** rcernin has quit IRC | 15:03 | |
| mriedem | jaypipes: sdague: went through https://review.openstack.org/#/c/357726/ - i guess if you want to put it in then ok | 15:03 |
| mriedem | we should fix the extension name in the reno | 15:03 |
| jaypipes | k | 15:04 |
| mriedem | it's one of those issues that we're not going to hear about for 18 months | 15:04 |
| *** kristian__ has quit IRC | 15:04 | |
| mriedem | when pike is the oldest stable branch | 15:04 |
| *** links has joined #openstack-nova | 15:04 | |
| *** kristian__ has joined #openstack-nova | 15:04 | |
| *** takashin has left #openstack-nova | 15:05 | |
| dansmith | sdague: that won't reliably wait for the third node, right? | 15:05 |
| ys__ | Hi, All. After I changed cpu_allocation_ratio in controller, Should I restart all nova services or just nova scheduler? | 15:05 |
| *** kristia__ has joined #openstack-nova | 15:06 | |
| *** baoli has quit IRC | 15:06 | |
| dansmith | ys__: see topic please | 15:06 |
| mriedem | ys__: that's used in both the scheduler and the compute service | 15:06 |
| *** yingjun has joined #openstack-nova | 15:07 | |
| *** links has quit IRC | 15:07 | |
| *** annegentle has joined #openstack-nova | 15:07 | |
| mriedem | actually the option is only used in the compute service | 15:07 |
| mriedem | to update the compute node recored, | 15:07 |
| mriedem | *record | 15:07 |
| mriedem | which is used by the scheduler | 15:07 |
| *** kristia__ has quit IRC | 15:07 | |
| sdague | dansmith: yes, it will | 15:09 |
| *** baoli has joined #openstack-nova | 15:09 | |
| *** kristian__ has quit IRC | 15:09 | |
| dansmith | sdague: how? | 15:09 |
| sdague | each node is waiting for it's own hostname to show up | 15:09 |
| sdague | stack.sh doesn't complete until it has | 15:09 |
| dansmith | and something else waits for stack.sh on all the nodes before we run tempest/ | 15:10 |
| sdague | yes, stack.sh executions are linear | 15:10 |
| dansmith | okay | 15:10 |
| sdague | otherwise tempest would run before services were setup | 15:10 |
| *** tbachman has quit IRC | 15:12 | |
| *** jamesdenton has quit IRC | 15:13 | |
| mriedem | dansmith: devstack-gate is waiting for the subnode stacks to be done | 15:13 |
| *** jjung has quit IRC | 15:13 | |
| mriedem | before calling discover_hosts | 15:13 |
| mriedem | so yeah this looks ok | 15:14 |
| dansmith | ack | 15:14 |
| mriedem | http://logs.openstack.org/46/487246/2/experimental/gate-tempest-dsvm-neutron-dvr-ha-multinode-full-ubuntu-xenial-nv/6cd2a5b/logs/devstack-gate-discover-hosts.txt.gz | 15:14 |
| mriedem | ^ is the 3 node job on that change | 15:14 |
| dansmith | I hadn't scrolled right enough to see it was querying its own record, so assumed this was waiting for _a_ compute | 15:14 |
| mriedem | sdague: dansmith: btw, would like to get rid of those ugly ass DEBUG outputs for oslo.concurrency from nova-manage https://review.openstack.org/#/c/487179/ | 15:15 |
| mriedem | ^ is probably backportable if you want me to open a bug | 15:15 |
| *** jamesdenton has joined #openstack-nova | 15:15 | |
| *** annegentle has quit IRC | 15:16 | |
| *** annegentle has joined #openstack-nova | 15:17 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add oslo_concurrency=INFO to default log levels for nova-manage https://review.openstack.org/487179 | 15:18 |
| *** jamesden_ has joined #openstack-nova | 15:20 | |
| *** jamesdenton has quit IRC | 15:20 | |
| *** annegentle has quit IRC | 15:21 | |
| s-dean | Hi, I got past the Cells issue yesterday and managed to list hypervisors and display nodes and services, then i cocked up on install neutron, and now im back to square one. cant list hypervisors | 15:21 |
| s-dean | <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1039 | 15:22 |
| *** imacdonn has joined #openstack-nova | 15:22 | |
| mriedem | installing neutron shouldn't do anything with nova | 15:22 |
| s-dean | i swear all services are connected to rabbit so why cant I list hypervisors and services | 15:22 |
| s-dean | MessagingTimeout: Timed out waiting for a reply to message ID e65f06c471cc4e75858f936cf4dff041 | 15:22 |
| s-dean | is there a database that i can check to see the transport URL | 15:23 |
| s-dean | i had to start a fresh again | 15:23 |
| *** sree has quit IRC | 15:23 | |
| mriedem | s-dean: nova-manage cell_v2 list_cells --verbose | 15:23 |
| mriedem | will dump the db and mq urls for the cell mappings | 15:23 |
| mriedem | which should just be cell0 and cell1 | 15:24 |
| s-dean | ive ran that command | 15:24 |
| s-dean | transport is fine | 15:24 |
| s-dean | to my eye | 15:24 |
| s-dean | hell conductor can even see my compute node | 15:24 |
| mriedem | and the cell1 transport url is the same as the transport url in nova.conf? | 15:24 |
| s-dean | yes | 15:24 |
| s-dean | =INFO REPORT==== 27-Jul-2017::16:17:08 === | 15:24 |
| s-dean | Connection <0.1768.0> (10.30.0.2:32936 -> 10.30.0.2:5672) has a client-provided name: nova-api:4929:dec90db5-2cda-44ea-8ce7-e007fe5280a2 | 15:24 |
| mriedem | are the computes in the nova_api.host_mappings table? | 15:24 |
| s-dean | not yet | 15:25 |
| s-dean | tried to list the hypervisor | 15:25 |
| mriedem | that's probably your problem | 15:25 |
| mriedem | run: nova-manage cell_v2 discover_hosts | 15:25 |
| mriedem | --verbose option on that too | 15:25 |
| s-dean | Found 1 computes in cell: f2852cfb-ae7f-47d0-bef2-d00069bb57ca | 15:26 |
| s-dean | and again ERROR 500 on nova compute api | 15:26 |
| s-dean | just times out | 15:26 |
| s-dean | i can see it connecting to rabbit | 15:27 |
| s-dean | ive had this issue at least 4 times now | 15:28 |
| s-dean | only yesterday, dont know what i did different but it was happy and let me list hypervisors and services | 15:28 |
| *** vishwanathj has quit IRC | 15:29 | |
| *** vishwanathj has joined #openstack-nova | 15:29 | |
| *** andreas_s has quit IRC | 15:29 | |
| *** annegentle has joined #openstack-nova | 15:30 | |
| s-dean | this is the rabbit mq driver | 15:30 |
| *** coreywright has quit IRC | 15:30 | |
| s-dean | oslo_messaging _drivers amqpdriver.py | 15:30 |
| *** yassine has quit IRC | 15:31 | |
| *** psachin has joined #openstack-nova | 15:32 | |
| *** vishwana_ has joined #openstack-nova | 15:33 | |
| mriedem | does the cell_mapping in the host_mappings table entries match cell1 in the cell_mappings table? | 15:33 |
| *** vishwanathj has quit IRC | 15:33 | |
| mriedem | i.e. are the computes mapped to cell1 properly? | 15:33 |
| *** cfriesen_ has joined #openstack-nova | 15:34 | |
| sdague | mriedem: was just talking with dhellman, the sitemap fix isn't going to fix the 404s | 15:34 |
| sdague | we're going to need to build a redirect list up ourselves | 15:34 |
| *** baoli has quit IRC | 15:34 | |
| *** vishwana_ has quit IRC | 15:35 | |
| *** baoli has joined #openstack-nova | 15:35 | |
| *** vishwanathj has joined #openstack-nova | 15:35 | |
| *** gyee has joined #openstack-nova | 15:35 | |
| *** rajathagasthya has joined #openstack-nova | 15:36 | |
| s-dean | in that table, this is what i have | 2017-07-27 14:35:05 | NULL | 1 | 2 | compute01 | | 15:36 |
| s-dean | | 2017-07-27 14:35:05 | NULL | 1 | 2 | compute01 | | 15:36 |
| mriedem | sdague: ok :( | 15:36 |
| *** claudiub has quit IRC | 15:37 | |
| *** zz9pzza has quit IRC | 15:37 | |
| mriedem | what are the columns for that? | 15:37 |
| s-dean | created at updated at id cell_id host | 15:38 |
| mriedem | ok so 1 entry | 15:38 |
| s-dean | yeah | 15:38 |
| mriedem | and the id for cell1 in the cell_mappings table is 2? | 15:38 |
| *** annegentle has quit IRC | 15:38 | |
| s-dean | yes | 15:38 |
| s-dean | corresponds to cell_mappings table | 15:39 |
| s-dean | cell1 = id 2 | 15:39 |
| mriedem | and this is ocata code or mater (pike)? | 15:39 |
| mriedem | *master | 15:39 |
| s-dean | ocata | 15:39 |
| mriedem | and nova hypervisor-list fails yes? | 15:40 |
| mriedem | does nova service-list work? | 15:40 |
| s-dean | checking | 15:41 |
| s-dean | no | 15:41 |
| s-dean | just hangs | 15:41 |
| s-dean | let me turn tls off and recreate the endpoint with http | 15:42 |
| *** gszasz has quit IRC | 15:42 | |
| *** edmondsw has quit IRC | 15:42 | |
| *** coreywright has joined #openstack-nova | 15:43 | |
| openstackgerrit | Merged openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208 | 15:43 |
| s-dean | sdague replied to one of my bugs yesterday saying that i had a rabbitmq misconfiguration, and then out of the blue it worked but now im back to square one. | 15:44 |
| *** moshele has joined #openstack-nova | 15:44 | |
| *** sambetts is now known as sambetts|afk | 15:45 | |
| *** hshiina has joined #openstack-nova | 15:46 | |
| *** yushb has quit IRC | 15:48 | |
| *** hongbin has joined #openstack-nova | 15:48 | |
| s-dean | where does os-hypervisors | 15:48 |
| s-dean | live in the DB | 15:48 |
| s-dean | ? | 15:48 |
| *** baoli has quit IRC | 15:49 | |
| *** rmart04 has quit IRC | 15:50 | |
| *** edmondsw has joined #openstack-nova | 15:50 | |
| *** rmart04 has joined #openstack-nova | 15:50 | |
| *** p01nt3r75 has quit IRC | 15:52 | |
| bauzas | s-dean: compute_nodes | 15:52 |
| *** Swami has quit IRC | 15:54 | |
| mriedem | sdague: bauzas: dansmith: should we get this in before we release novaclient for pike? https://review.openstack.org/#/c/487513/ | 15:54 |
| mriedem | we could also backport since it's latent | 15:54 |
| *** saphi_ has quit IRC | 15:54 | |
| bauzas | mriedem: dunno, your thoughts ? | 15:54 |
| bauzas | honestly it's for the CLI | 15:55 |
| bauzas | so a backport looks good to me | 15:55 |
| s-dean | ok well that table has data inside it, and references my compute node | 15:55 |
| *** efoley has quit IRC | 15:56 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | 15:57 |
| *** baoli has joined #openstack-nova | 15:57 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: doc: Remove Makefile https://review.openstack.org/480096 | 15:58 |
| *** moshele has quit IRC | 15:58 | |
| *** phuongnh has joined #openstack-nova | 16:00 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle hash ring rebalancing in ironic flavor migration https://review.openstack.org/487925 | 16:01 |
| s-dean | does setting cell type in the nova.conf screw things up ? | 16:02 |
| *** moshele has joined #openstack-nova | 16:02 | |
| s-dean | is that option only for V1 | 16:02 |
| mriedem | s-dean: cell type is only for cells v1 | 16:02 |
| s-dean | right ok | 16:02 |
| ildikov | mriedem: I know it's code freeze week, but can you come to the Cinder-Nova meeting? | 16:02 |
| s-dean | thats my issue | 16:03 |
| mriedem | s-dean: you shouldn't touch anything in the [cells] group in nova.conf | 16:03 |
| s-dean | ok, man have cell disabled makes it sound like you should set it to enabled | 16:04 |
| s-dean | but that activates cells v1 right | 16:04 |
| mriedem | the naming is unfortunate and confusing yes | 16:04 |
| s-dean | omg | 16:04 |
| s-dean | alright | 16:04 |
| mriedem | [cells]/enable=True is not what you want | 16:04 |
| s-dean | hahha | 16:04 |
| s-dean | ok | 16:04 |
| s-dean | cheers for your help ill let you get back to your development | 16:05 |
| s-dean | and go and reinstall again | 16:05 |
| s-dean | for the 10th time | 16:05 |
| mriedem | s-dean: note https://github.com/openstack/nova/blob/master/nova/conf/cells.py#L24-L27 | 16:05 |
| mriedem | the config options and their help is also here https://docs.openstack.org/nova/ocata/sample_config.html | 16:06 |
| *** lucasxu has quit IRC | 16:06 | |
| s-dean | that link is for the Dev docs | 16:06 |
| s-dean | right | 16:06 |
| mriedem | jaypipes: have you checked the CI results on the claims in scheduler change yet? | 16:08 |
| mriedem | how are we looking? | 16:08 |
| mriedem | i'm nervous there is no chatter in here about that patch | 16:08 |
| *** rajathag_ has joined #openstack-nova | 16:08 | |
| mriedem | vdrok: grenade failed http://logs.openstack.org/58/487458/5/check/gate-grenade-dsvm-ironic-ubuntu-xenial/f1fd08b/logs/grenade.sh.txt.gz#_2017-07-27_15_59_36_717 | 16:09 |
| *** rajathagasthya has quit IRC | 16:09 | |
| vdrok | yup, just commented on the patch. apart from the definition in upgrade/settings I also see this http://logs.openstack.org/58/487458/5/check/gate-grenade-dsvm-ironic-ubuntu-xenial/f1fd08b/logs/grenade.sh.txt.gz#_2017-07-27_15_58_24_805. guess I have to define it once again in the upgrade.sh? | 16:10 |
| *** rmart04 has quit IRC | 16:10 | |
| *** saphi_ has joined #openstack-nova | 16:13 | |
| openstackgerrit | Doug Hellmann proposed openstack/nova master: WIP: add a redirect for the old cells landing page https://review.openstack.org/487932 | 16:13 |
| *** roeyc has joined #openstack-nova | 16:15 | |
| *** roeyc has quit IRC | 16:16 | |
| *** moshele has quit IRC | 16:19 | |
| *** sshwarts has quit IRC | 16:23 | |
| *** dtp has joined #openstack-nova | 16:28 | |
| *** ralonsoh has quit IRC | 16:28 | |
| *** lyan has quit IRC | 16:29 | |
| *** kaisers_ has quit IRC | 16:29 | |
| *** psachin has quit IRC | 16:32 | |
| *** phuongnh has quit IRC | 16:32 | |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | 16:36 |
| jaypipes | mriedem: sorry, was at lunch | 16:41 |
| jaypipes | mriedem: checking now. | 16:41 |
| jaypipes | mriedem: bottom patch has an unrelated failure: http://logs.openstack.org/89/487589/6/check/gate-nova-python35/ce8b77c/testr_results.html.gz | 16:42 |
| jaypipes | mriedem: top patch (the claims one) has no failures. | 16:42 |
| mriedem | yeah i've seen that weird rpc timeout one | 16:42 |
| *** iceyao has quit IRC | 16:45 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: add FAQ entry for cells v1 config options https://review.openstack.org/487938 | 16:49 |
| mriedem | s-dean: ^ | 16:49 |
| *** hshiina has quit IRC | 16:50 | |
| dansmith | mriedem: I was thinking of expanding the warning on my new cells doc to explain not only it not applying to cellsv1, but also that anything about the nova-cells service, the [cells] section of the config, etc | 16:50 |
| dansmith | since "cellsv1" is not a neat box everyone understands | 16:50 |
| mriedem | yeah there is a warning here https://docs.openstack.org/ocata/config-reference/compute/cells.html | 16:52 |
| mriedem | but doesn't mention the config options, | 16:52 |
| mriedem | or nova-cells servie | 16:52 |
| mriedem | which are part of the confusion | 16:52 |
| *** chyka has joined #openstack-nova | 16:52 | |
| dansmith | aye | 16:53 |
| sdague | vdrok: ok https://review.openstack.org/#/c/487809/ is +A | 16:53 |
| sdague | the ironic patch is looking pretty good | 16:54 |
| vdrok | sdague: thanks! | 16:54 |
| sdague | vdrok: you should start lining up +2s on that one | 16:54 |
| vdrok | will do as soon as I have a pass :) | 16:55 |
| *** lyan has joined #openstack-nova | 16:58 | |
| cfriesen_ | mriedem: thanks for fixing those nits on the unshelve. I'm a couple hours behind you. :) | 16:58 |
| mriedem | np, thanks for fixing | 16:59 |
| *** Apoorva has joined #openstack-nova | 16:59 | |
| *** baoli has quit IRC | 16:59 | |
| *** rmart04 has joined #openstack-nova | 17:01 | |
| *** rmart04 has quit IRC | 17:01 | |
| openstackgerrit | Merged openstack/nova master: Add 'updated_at' field to InstancePayload in notifications https://review.openstack.org/475276 | 17:02 |
| *** derekh has quit IRC | 17:03 | |
| mriedem | this fixes another one of our random functional test failures https://review.openstack.org/#/c/487772/ | 17:03 |
| *** zz9pzza has joined #openstack-nova | 17:04 | |
| *** baoli has joined #openstack-nova | 17:06 | |
| sdague | mriedem nice | 17:07 |
| *** zz9pzza has quit IRC | 17:08 | |
| *** harlowja has joined #openstack-nova | 17:09 | |
| *** baoli has quit IRC | 17:10 | |
| *** baoli has joined #openstack-nova | 17:11 | |
| *** iceyao has joined #openstack-nova | 17:11 | |
| openstackgerrit | Merged openstack/python-novaclient master: Add 'Forced down' column in serivce-list https://review.openstack.org/487263 | 17:11 |
| *** tbachman has joined #openstack-nova | 17:11 | |
| *** mvk has joined #openstack-nova | 17:13 | |
| *** iceyao has quit IRC | 17:15 | |
| *** rcernin has joined #openstack-nova | 17:16 | |
| openstackgerrit | Merged openstack/python-novaclient master: Be clear about hypevisors.search used in a few CLIs https://review.openstack.org/487513 | 17:17 |
| *** chyka has quit IRC | 17:18 | |
| *** chyka has joined #openstack-nova | 17:18 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle hash ring rebalancing in ironic flavor migration https://review.openstack.org/487954 | 17:19 |
| openstackgerrit | Doug Hellmann proposed openstack/nova master: WIP: add a redirect for the old cells landing page https://review.openstack.org/487932 | 17:19 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: hardware offload support for openvswitch https://review.openstack.org/398265 | 17:19 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle ironicclient failures in Ironic driver https://review.openstack.org/487925 | 17:19 |
| mriedem | jaypipes: +2 on moshe's hw offload thingamajig https://review.openstack.org/#/c/398265/ | 17:20 |
| *** dtantsur is now known as dtantsur|afk | 17:24 | |
| *** Apoorva_ has joined #openstack-nova | 17:26 | |
| *** Apoorva has quit IRC | 17:29 | |
| *** jamesden_ has quit IRC | 17:29 | |
| *** jamesdenton has joined #openstack-nova | 17:31 | |
| *** lucasagomes is now known as lucas-afk | 17:31 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test resize with placement api https://review.openstack.org/487958 | 17:33 |
| gibi | mriedem: here is my first stab at the resize tests with placement api ^^ | 17:33 |
| gibi | mriedem: it is pretty unstable at the current form and ther are some asserts commented out that are failing right now | 17:34 |
| mriedem | gibi: cool, thanks | 17:34 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle hash ring rebalancing in ironic flavor migration https://review.openstack.org/487954 | 17:34 |
| gibi | mriedem: can be a real bug or can be some race in the test | 17:34 |
| edleafe | dansmith: ^^ updated to address your comments | 17:34 |
| gibi | mriedem: I have to leave for today but check back tomorrow morning | 17:34 |
| *** chyka has quit IRC | 17:35 | |
| dansmith | edleafe: cool, lets get ironic eyes on it and then I think we're good | 17:35 |
| openstackgerrit | Sean Dague proposed openstack/nova master: always show urls in list_cells https://review.openstack.org/487860 | 17:35 |
| cdent | gibi++ | 17:35 |
| openstackgerrit | Merged openstack/python-novaclient master: Help text for "--matching" is not clear. https://review.openstack.org/487708 | 17:36 |
| sdague | gibi: thanks for solving that test race | 17:36 |
| *** chyka has joined #openstack-nova | 17:37 | |
| sdague | mriedem: the sqla_url functions don't really provide the same reassembly that you get from urlparse | 17:37 |
| sdague | which is why it was not used in https://review.openstack.org/#/c/487860/ | 17:37 |
| *** Apoorva_ has quit IRC | 17:39 | |
| *** yamahata has joined #openstack-nova | 17:40 | |
| *** ijw has joined #openstack-nova | 17:41 | |
| *** Apoorva has joined #openstack-nova | 17:41 | |
| *** penick has joined #openstack-nova | 17:45 | |
| mriedem | "Alex Xu 2:33 AM oh..damn it." | 17:49 |
| mriedem | someone is going to have to have a talk with alex_xu | 17:49 |
| mriedem | bauzas: so your -1 on this is just stale at this point https://review.openstack.org/#/c/483566/ | 17:52 |
| mriedem | bauzas: can you remove the -1 and re-review? if you're ok, but don't want to +W until i go through it too that's fine | 17:52 |
| jaypipes | mriedem: yeah, I'm on it. | 17:53 |
| mriedem | you know, this low sodium vegetarian lentil soup that laura buys could really use some pork | 17:53 |
| jaypipes | omg lmao | 17:54 |
| openstackgerrit | Doug Hellmann proposed openstack/nova master: WIP: add a redirect for the old cells landing page https://review.openstack.org/487932 | 17:54 |
| bauzas | mriedem: sure , just at dinner atm so will do in the next hour | 17:55 |
| *** kbaegis has quit IRC | 17:55 | |
| *** kbaegis has joined #openstack-nova | 17:55 | |
| *** yingjun has quit IRC | 17:56 | |
| *** saphi_ has quit IRC | 17:56 | |
| smcginnis | mriedem: Some of the vegetarian pork like they tried to give dims? | 17:58 |
| dims | LOL | 17:58 |
| dansmith | just don't give him quinoa whatever you do | 17:58 |
| *** zz9pzza has joined #openstack-nova | 17:58 | |
| dims | amen! | 17:58 |
| dansmith | pork = funny joke | 17:58 |
| dansmith | quinoa = unfunny disaster | 17:58 |
| sdague | right, all the quinoa goes to dansmith | 17:58 |
| mriedem | like a gremlin eating after midnight | 17:59 |
| dansmith | sdague: joke's on you homeboy, I like quinoa | 17:59 |
| sdague | I am confused by dansmith liking something that's not a hamburger | 18:00 |
| dansmith | http://www.bonappetit.com/recipe/stellar-quinoa-burger | 18:01 |
| *** rajathag_ has quit IRC | 18:01 | |
| sdague | dansmith: that looks pretty good, will need to add that to my recipe stash | 18:02 |
| dansmith | first hit on google.. looks gross though because it has sweet potato in it | 18:02 |
| sdague | heh | 18:03 |
| sdague | oh, so you don't actually eat that :) | 18:03 |
| *** zz9pzza has quit IRC | 18:03 | |
| dansmith | sweet potato is gross | 18:03 |
| dansmith | regular potato == perfect | 18:03 |
| vdrok | thanks for the help with multicell, all jobs green :) | 18:05 |
| dansmith | woot | 18:08 |
| mriedem | final novaclient release is up https://review.openstack.org/487966 | 18:09 |
| mriedem | onto the claims in the scheduler patch, | 18:10 |
| mriedem | i see the move accounting happening on that change in the multinode patch | 18:11 |
| mriedem | http://logs.openstack.org/66/483566/20/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/374e3c3/logs/screen-n-sch.txt.gz#_Jul_27_14_36_56_138843 | 18:11 |
| mriedem | New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'209a32d3-f240-4bcc-9d9d-8ae371b97d42'}, 'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}, {u'resource_provider': {u'uuid': u'fdba3ea4-883a-4dc4-a2d6-49d723f9559e'}, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}]} | 18:11 |
| *** moshele has joined #openstack-nova | 18:11 | |
| mriedem | this one is a resize New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'209a32d3-f240-4bcc-9d9d-8ae371b97d42'}, 'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}, {u'resource_provider': {u'uuid': u'fdba3ea4-883a-4dc4-a2d6-49d723f9559e'}, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}]} | 18:11 |
| mriedem | oops | 18:11 |
| mriedem | http://logs.openstack.org/66/483566/20/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/374e3c3/logs/screen-n-sch.txt.gz#_Jul_27_14_48_57_853377 | 18:12 |
| mriedem | New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'fdba3ea4-883a-4dc4-a2d6-49d723f9559e'}, 'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}, {u'resource_provider': {u'uuid': u'209a32d3-f240-4bcc-9d9d-8ae371b97d42'}, u'resources': {u'VCPU': 1, u'MEMORY_MB': 128}}]} | 18:12 |
| mriedem | memory bumps up | 18:12 |
| *** corey_ has joined #openstack-nova | 18:12 | |
| mriedem | so that all seems cool | 18:12 |
| *** corey_ is now known as Guest25498 | 18:12 | |
| *** cleong has quit IRC | 18:13 | |
| mriedem | i don't expect anything to be busted with soft delete, since with soft delete we do'nt delete the instance until it's reclaimed | 18:13 |
| mriedem | so the allocations shouldn't change until that happens | 18:13 |
| cdent | gibi’s test suggeests that cleanups are not happening | 18:13 |
| cdent | i’m experimenting with them now to see if I can see anything wrong/weird | 18:13 |
| *** moshele has quit IRC | 18:13 | |
| mriedem | i do see the source node cleaning up allocations during live migration | 18:15 |
| mriedem | http://logs.openstack.org/66/483566/20/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/374e3c3/logs/screen-n-cpu.txt.gz#_Jul_27_14_35_26_077233 | 18:15 |
| mriedem | jaypipes: dansmith: do we need to worry about this? https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1068 | 18:15 |
| mriedem | wiping out all of the allocations for an instance because it's no longer on the source node | 18:16 |
| dansmith | hmm, I thought not because of the claim at the end on the destination, but let me look | 18:16 |
| dansmith | mriedem: yeah, we should check the allocations before we delete to see if we own any of them I think, or delete the ones that pertain to us | 18:17 |
| dansmith | instead of just nuking them all | 18:17 |
| dansmith | good call | 18:17 |
| *** rajathagasthya has joined #openstack-nova | 18:19 | |
| mriedem | it also seems that https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1068 could simply updated to be: | 18:20 |
| mriedem | if not instance.host or instance.host != CONF.host: | 18:20 |
| mriedem | continue | 18:20 |
| jaypipes | err, not sure about that... | 18:21 |
| mriedem | otherwise yeah, pass the cn uuid to https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L999 and we could make sure to only remove allocations for the source node + instance | 18:21 |
| mriedem | note that if we have to change anything here in the compute, it wouldn't be there for ocata computes | 18:22 |
| jaypipes | mriedem: an allocation is an all-or-none thing, though. | 18:22 |
| mriedem | jaypipes: i don't know what that means | 18:22 |
| mriedem | we're doubling up allocations here https://review.openstack.org/#/c/487589/6/nova/scheduler/client/report.py | 18:22 |
| mriedem | to maintain the source node allocations | 18:22 |
| mriedem | but https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1068 will clearly blast those away | 18:23 |
| jaypipes | mriedem: You can't delete "part of an allocation". | 18:23 |
| mriedem | why not? we amended part of an allocation | 18:23 |
| mriedem | here https://review.openstack.org/#/c/487589/6/nova/scheduler/client/report.py | 18:23 |
| jaypipes | no, we replaced it. | 18:23 |
| mriedem | so we patched something in, we can't patch something out? | 18:23 |
| jaypipes | PUT /allocations overwrites. | 18:23 |
| mriedem | yes i know | 18:23 |
| jaypipes | mriedem: hold up. | 18:23 |
| mriedem | i'm saying, | 18:23 |
| mriedem | we have to do the same thing for https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1068 | 18:24 |
| mriedem | to remove the allocations for the instance on the source node | 18:24 |
| mriedem | but leave the allocations for the instance on the dest node | 18:24 |
| mriedem | is it hangout time? | 18:24 |
| dansmith | we either have to not delete, | 18:25 |
| jaypipes | mriedem: the dest host will end up writing the allocation entirely (only including the allocated resources on the dest host) when the move operation completes successfully. | 18:25 |
| dansmith | or put the allocation with our part removed | 18:25 |
| dansmith | jaypipes: right but the source will then delete it without checking it | 18:25 |
| dansmith | and I think those two things probably race with each other | 18:25 |
| jaypipes | mriedem: so I think what we need to do is just ensure _update_usage_from_instances() does not call that _remove_deleted_instances_allocations() for instances currently in a move operation | 18:25 |
| jaypipes | dansmith: understood. we just need to ensure we don't call that delete for any instances in a move operation | 18:26 |
| dansmith | jaypipes: or instances that are just finishing a move operation that it thinks have been deleted | 18:26 |
| dansmith | jaypipes: I don't think you can know that it's in a move operation if you're late to the party | 18:26 |
| mriedem | dansmith: the instance would have a migration_context? | 18:26 |
| bauzas | mriedem: back there | 18:26 |
| jaypipes | guys, won't https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1060 skip instances that are currently being mnoved? | 18:27 |
| *** lyan has quit IRC | 18:27 | |
| openstackgerrit | Merged openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | 18:27 |
| * bauzas scrolling | 18:27 | |
| dansmith | mriedem: not if you'reon the late end of the race | 18:27 |
| dansmith | currently being moved is not the problem | 18:27 |
| dansmith | "just moved a half second ago" is the problem right? | 18:27 |
| jaypipes | but instance.host will be not None. | 18:27 |
| jaypipes | the only time instance.host is None is when the instance is deleted. | 18:27 |
| *** lyan has joined #openstack-nova | 18:28 | |
| *** jmlowe has quit IRC | 18:28 | |
| dansmith | jaypipes: actually not | 18:28 |
| dansmith | jaypipes: that's reversed.. None means "not yet booted".. it's still $host after you delete | 18:28 |
| dansmith | it's skipping not yet booted instances | 18:28 |
| dansmith | if the hostname doesn't match you'll continue on to delete it because it's not on our host right? | 18:29 |
| jaypipes | ah, shit. | 18:29 |
| dansmith | since this logic was all based on just accounting for the local host, I don't think it quite works | 18:29 |
| dansmith | plus, | 18:29 |
| dansmith | there is a race between fetchign the instance and its host flipping to the other side | 18:29 |
| *** annegentle has joined #openstack-nova | 18:29 | |
| mriedem | right the not instance.host is for scheduling instances the first time | 18:30 |
| mriedem | per the comment | 18:30 |
| mriedem | "# Allocations related to instances being scheduled should not be # deleted if we already wrote the allocation previously." | 18:30 |
| dansmith | yeah | 18:30 |
| dansmith | this was the "leafe race" | 18:30 |
| mriedem | "already wrote the allocation previously" == in the scheduler | 18:30 |
| mriedem | right | 18:30 |
| jaypipes | but allocations_to_delete won't contain migrating instances. Because migrating instances are in RT.tracked_instances, no? | 18:30 |
| dansmith | jaypipes: not at the instance after we complete the migration right? | 18:31 |
| * jaypipes thinks | 18:31 | |
| mriedem | that i don't know | 18:31 |
| mriedem | b/c it's f'ed | 18:31 |
| dansmith | jaypipes: either way, this is not a big deal: | 18:31 |
| dansmith | just pull the allocation like we do, and if we're not in it, we don't touch it | 18:32 |
| dansmith | and maybe something else about if there are two in there, then don't delete either, although not sure if that's important or not | 18:32 |
| jaypipes | dansmith: sure, that's a fairly safe thing. | 18:32 |
| *** acormier has quit IRC | 18:33 | |
| jaypipes | ok, guys, lemme update the move patch. | 18:33 |
| mriedem | just looking at where this starts from, the periodic is in here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L667 | 18:33 |
| mriedem | jaypipes: just do it on top of that one since it's in the gate | 18:33 |
| mriedem | we get the instances for the host and node (source) | 18:33 |
| mriedem | clear out all tracked instances https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1030 | 18:34 |
| dansmith | jaypipes: mriedem: ++ for not interrupting the one in the gate | 18:34 |
| mriedem | then we go here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1044 | 18:34 |
| mriedem | which is where i get lost | 18:34 |
| *** acormier has joined #openstack-nova | 18:35 | |
| bauzas | mriedem: what's the concern? | 18:35 |
| mriedem | i think it just gets added here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L995 | 18:35 |
| mriedem | because we clear self.tracked_instances before calling that method | 18:36 |
| jaypipes | yeah | 18:36 |
| mriedem | so is_new_instance = uuid not in self.tracked_instances will be True | 18:36 |
| mriedem | and then self.tracked_instances[uuid] = obj_base.obj_to_primitive(instance) | 18:36 |
| bauzas | mriedem: you want to know whether migrating instances are in tracked_instances ? | 18:36 |
| dansmith | mriedem: even still, we pull the allocation, we should never just blindly delete it without looking to make sure it's what we expect, which is kinda the point of the generation stuff, but in reverse here | 18:36 |
| jaypipes | bauzas: they aren't. they are. depends on when in the update_available_resource() method you're looking at ;) | 18:36 |
| *** ijw_ has joined #openstack-nova | 18:36 | |
| bauzas | mriedem: if that's the question, I don't think so | 18:36 |
| bauzas | mriedem: because we lookup at all the existing instances | 18:37 |
| jaypipes | dansmith: ack | 18:37 |
| mriedem | dansmith: yeah, that seems simplest too | 18:37 |
| dansmith | anybody else looking forward to most of this code going away? :) | 18:37 |
| mriedem | o/ | 18:37 |
| bauzas | so since we update the instance.host once the migration is done, the target RT doesn't see it | 18:37 |
| mriedem | so here is another question, which no one is going to like | 18:37 |
| mriedem | this is going to be a change in how the compute is behaving | 18:37 |
| mriedem | and ocata computes won't have this | 18:37 |
| mriedem | so, | 18:37 |
| jaypipes | dansmith: hmm... | 18:38 |
| mriedem | do we (1) make claims in the scheduler dependent on pike computes, or (2) throw to the wind and rely on the dest periodic self-heal fixing the allocations? | 18:38 |
| jaypipes | dansmith: so we're calling get_allocations_for_resource_)provider() and passing in the source compute node UUID. | 18:38 |
| jaypipes | dansmith: so we're guaranteed that the only allocations returned are instances that are "on the source host" according to placement. | 18:38 |
| dansmith | mriedem: well the healing doesn't happen all the time, as he pointed out yesterday, only when instances are added/removed, right? | 18:39 |
| dansmith | mriedem: so I'm not sure we'll actually heal over stuff that ocata computes don't do :/ | 18:39 |
| dansmith | jaypipes: delete_allocation_for_instance() operates only on an instance uuid | 18:39 |
| dansmith | jaypipes: if we call that after the flip happened for whatever reason we'll do the wrong thing | 18:40 |
| dansmith | jaypipes: even if we started ten minutes ago and got blocked for a while or something | 18:40 |
| dansmith | jaypipes: really we should use generation on delete to make sure we don't delete something we're not intending to :/ | 18:40 |
| jaypipes | dansmith: right, but didn't you want me to check to see if the allocation had the source compute node UUID in it and if not, don't call delete allocation? | 18:40 |
| *** ijw has quit IRC | 18:40 | |
| dansmith | jaypipes: yes, but delete allocation is only fetching via instance uuid | 18:41 |
| dansmith | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L999-L1001 | 18:41 |
| *** ijw_ has quit IRC | 18:41 | |
| jaypipes | dansmith: I understand that, but get_allocations_for_resource_provider() will always only return allocations that "have the source compute node UUID in them" | 18:41 |
| jaypipes | dansmith: so that logic isn't going to filter anything out... | 18:41 |
| mriedem | hangout? | 18:41 |
| dansmith | jaypipes: yeah dude I get that, but we query that way, then time passes, then we delete | 18:42 |
| jaypipes | mriedem: sure | 18:42 |
| dansmith | https://hangouts.google.com/call/5pmzfm5wpfckpptt4l5hxjw5cyu | 18:43 |
| bauzas | can I join ? | 18:45 |
| bauzas | :) | 18:45 |
| bauzas | need more context | 18:45 |
| mriedem | everyone can join | 18:45 |
| smcginnis | Not if you're in China. | 18:46 |
| smcginnis | :) | 18:46 |
| mriedem | nothing stopping them | 18:47 |
| mriedem | climb that mountain | 18:47 |
| smcginnis | :D | 18:47 |
| *** jjung has joined #openstack-nova | 18:48 | |
| *** armax has quit IRC | 18:49 | |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L667 | 18:52 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1030 | 18:52 |
| mriedem | if instance.vm_state not in vm_states.ALLOW_RESOURCE_REMOVAL: | 18:52 |
| *** zz9pzza has joined #openstack-nova | 18:52 | |
| dansmith | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1007 | 18:52 |
| mriedem | on unshelve we set the host/node on the instance here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4423 | 18:56 |
| *** tbachman has quit IRC | 18:56 | |
| mriedem | and change the vm_state here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4440 | 18:57 |
| *** ijw has joined #openstack-nova | 18:57 | |
| *** jmlowe has joined #openstack-nova | 18:57 | |
| *** zz9pzza has quit IRC | 18:58 | |
| openstackgerrit | Merged openstack/nova master: stabilize test_create_delete_server functional test https://review.openstack.org/487772 | 19:01 |
| *** ijw has quit IRC | 19:01 | |
| *** Apoorva_ has joined #openstack-nova | 19:09 | |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L797 | 19:11 |
| *** Apoorva has quit IRC | 19:12 | |
| *** xyang1 has joined #openstack-nova | 19:12 | |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1012 | 19:13 |
| mriedem | instance_claim updating allocations https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L224 | 19:14 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/488034 | 19:16 |
| *** damien_r has quit IRC | 19:17 | |
| *** damien_r1 has joined #openstack-nova | 19:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/488086 | 19:19 |
| *** Apoorva_ has quit IRC | 19:19 | |
| *** Apoorva has joined #openstack-nova | 19:19 | |
| *** damien_r has joined #openstack-nova | 19:20 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | 19:21 |
| *** damien_r1 has quit IRC | 19:22 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: nova.utils.get_endpoint_data() https://review.openstack.org/488137 | 19:22 |
| efried | mriedem jaypipes mordred Having started poking at the cinderclient construction, I think ^this^ may be a better alternative to get_service_url | 19:23 |
| mriedem | efried: the house is on fire | 19:23 |
| dansmith | mriedem: jaypipes https://etherpad.openstack.org/p/y9sUcb6XW6 | 19:24 |
| mriedem | efried: have sdague check out the service catalog stuff | 19:24 |
| mriedem | he knows more about that than i do | 19:24 |
| efried | mriedem Roger wilco. | 19:24 |
| *** eharney_ has joined #openstack-nova | 19:24 | |
| *** eharney has quit IRC | 19:25 | |
| *** eharney_ is now known as eharney | 19:25 | |
| *** jamesdenton has quit IRC | 19:26 | |
| *** jamesdenton has joined #openstack-nova | 19:26 | |
| *** acormier has quit IRC | 19:28 | |
| mordred | efried: yes - that's a great approach | 19:29 |
| efried | mordred Cool, thanks for looking. | 19:29 |
| *** tbachman has joined #openstack-nova | 19:29 | |
| *** acormier has joined #openstack-nova | 19:30 | |
| *** eharney has quit IRC | 19:30 | |
| mriedem | https://review.openstack.org/#/c/244489/ | 19:31 |
| *** vishwanathj has quit IRC | 19:34 | |
| *** vishwanathj has joined #openstack-nova | 19:35 | |
| cfriesen_ | jaypipes: did you ever get anywhere with the issue we discussed at the end of June around duplicate scsi device numbers when using virtio-scsi? | 19:37 |
| jaypipes | cfriesen_: nope :( | 19:37 |
| mriedem | cfriesen_: the house is on fire | 19:37 |
| cfriesen_ | jaypipes: I think bug 1702999 is related, as is the "cannot attach new volume to an instance" thread on the openstack-operators list | 19:37 |
| openstack | bug 1702999 in OpenStack Compute (nova) "Can't attach volume if instance boot from volume and virtio-scsi is enabled in the image" [Undecided,Incomplete] https://launchpad.net/bugs/1702999 | 19:37 |
| jaypipes | oh wait, yeah I think we did have a patch for that... | 19:38 |
| jaypipes | cfriesen_: gimme a while... on call | 19:38 |
| mriedem | cfriesen_: this? https://review.openstack.org/#/q/topic:bug/1686116 | 19:38 |
| *** awaugama has quit IRC | 19:42 | |
| cfriesen_ | mriedem: looks like it might help. in the case I looked at it would boot (using sda) but trying to attach volumes would fail. | 19:42 |
| cfriesen_ | might be the case that 1702999 is already fixed | 19:42 |
| *** pwiltsey has joined #openstack-nova | 19:45 | |
| *** zz9pzza has joined #openstack-nova | 19:46 | |
| *** nicolasbock has quit IRC | 19:48 | |
| cdent | jaypipes: if you end up with something that has lose ends by the time you go to bed, feel free to let me know the state of things and I can poke in my morning | 19:49 |
| jaypipes | cdent: thx Chris, will do. | 19:49 |
| *** zz9pzza has quit IRC | 19:51 | |
| *** pchavva has quit IRC | 19:56 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | 19:57 |
| openstackgerrit | Doug Hellmann proposed openstack/nova master: add a redirect for the old cells landing page https://review.openstack.org/487932 | 19:59 |
| *** lyan has quit IRC | 20:02 | |
| *** lyan has joined #openstack-nova | 20:03 | |
| *** liverpooler has quit IRC | 20:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/488034 | 20:10 |
| jaypipes | dansmith: fuuug... so confirm_resize() doesn't run on the destination host. It runs on the source host. :( | 20:12 |
| mriedem | yeah it doesn't call back into rt | 20:12 |
| mriedem | _prep_resize is on dest host right? | 20:12 |
| mriedem | confirm just cleans up shit on the source | 20:12 |
| mriedem | *i think* | 20:12 |
| *** eharney has joined #openstack-nova | 20:13 | |
| jaypipes | mriedem: yeah. and that's not the stage of the move operation that we want to have the destination host call PUT /allocations :( | 20:13 |
| mriedem | and i think revert on the source doesn't do anything either, since it's already got resources claimed | 20:13 |
| mriedem | so there is nothing to unclaim | 20:13 |
| bauzas | folks, I will have to bail out, but I'll look at the IRC channel tomorrow morning | 20:14 |
| mriedem | i'm just about to push a change to add some logging and crap in the scheduler.reportclient.delete_allocations_for_instance to sanity check the allocations before we blow them away, to at least see if we're hitting weird stuff in there during migrate tests | 20:14 |
| mriedem | bauzas: o/ | 20:14 |
| bauzas | fer sur, if you need my review, lemme know | 20:14 |
| dansmith | jaypipes: ugh | 20:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | 20:15 |
| *** smatzek has quit IRC | 20:15 | |
| sdague | efried: link me | 20:15 |
| efried | sdague https://review.openstack.org/488137 bam | 20:16 |
| efried | sdague TIA. | 20:16 |
| sdague | mriedem: https://review.openstack.org/#/c/487860/ - nova-manage list_cells enhancement | 20:16 |
| sdague | with working tests | 20:16 |
| jaypipes | dansmith, mriedem: so this means really the only thing we can do during confirm_resize() (since it's on the source host) is recalculate the allocation (which will be the doubled-up thing) on the source host RT and remove all entries in the allocation set that refer to the source compute host UUID | 20:16 |
| sdague | I will keep bugging you about it :) | 20:16 |
| dansmith | jaypipes: yeah | 20:18 |
| dansmith | jaypipes: I was thinking something different, but that's smarter :) | 20:18 |
| *** pwiltsey has quit IRC | 20:19 | |
| sdague | efried: I'm surprised this is 'glance' and not 'image' - https://review.openstack.org/#/c/488137/1/nova/image/glance.py@127 | 20:19 |
| *** pwiltsey has joined #openstack-nova | 20:19 | |
| *** cfriesen_ has quit IRC | 20:20 | |
| *** vladikr has quit IRC | 20:20 | |
| efried | sdague It's the conf group name, which needs to correspond to the project name. | 20:20 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check delete_allocation_for_instance https://review.openstack.org/488187 | 20:20 |
| mriedem | dansmith: jaypipes: cdent: ^ | 20:20 |
| mriedem | just for testing at this point | 20:20 |
| efried | sdague Which we then look up in service-types-authority to get the service_type, which is indeed `image` | 20:20 |
| *** xyang1 has quit IRC | 20:21 | |
| sdague | efried: ok, I was surprised we couldn't just call it image to start with, but if that's how it is, that's fine | 20:21 |
| jaypipes | mriedem: coo. | 20:21 |
| sdague | efried: do we have a test job running this with api_servers not set in devstack? | 20:21 |
| mriedem | jaypipes: "is recalculate the allocation (which will be the doubled-up thing) on the source host RT and remove all entries in the allocation set that refer to the source compute host UUID" is i thought what dansmith and i were talking about earlier, | 20:22 |
| mriedem | which is similar to what my patch is checkingfor | 20:22 |
| efried | sdague Yeah, now that you're saying it, I admit it feels a tad weird. But the point is that nova.utils.get_endpoint_data needs to be able to use that param to find the appropriate conf to load, as well as to find the service_type if it's not specified in the conf. | 20:22 |
| dansmith | mriedem: well, I was assuming we could do it on the destination host | 20:22 |
| dansmith | mriedem: but it doesn't really matter, so yes it's pretty much what we were saying | 20:23 |
| mriedem | ok, i wasn't - i was thinking this was purely source host | 20:23 |
| sdague | efried: yeh, it would be nice in the future if we could just specify "image" as well | 20:23 |
| dansmith | well, you're just smarter than us | 20:23 |
| mriedem | heh | 20:23 |
| mriedem | i wouldn't go that far | 20:23 |
| jaypipes | I would | 20:23 |
| efried | sdague You can specify it in the conf: [glance] service_type = image | 20:24 |
| jaypipes | in any case, I'm on it. | 20:24 |
| dansmith | hey | 20:24 |
| efried | sdague I think you're saying you want to specify the service type directly to nova.utils.get_endpoint_data | 20:24 |
| mriedem | dansmith: was there more to that hey or just that your feelings were hurt? | 20:25 |
| mriedem | dansmith: is this similar to what you were thinking? https://review.openstack.org/#/c/488187/1/nova/scheduler/client/report.py@1085 | 20:25 |
| efried | sdague That would get confusing if the operator did in fact specify [glance] service_type = <whatever> | 20:25 |
| dansmith | mriedem: like, it's okay for me to say you're smarter than me, but not okay for jaypipes to say it | 20:25 |
| mriedem | oh i get it | 20:25 |
| jaypipes | everyone's smarter than me | 20:26 |
| mriedem | feel free to compliment me on my ability to dig up useless pop trivia | 20:26 |
| mriedem | but not my smarts in general | 20:26 |
| mriedem | laura had to explain her work schedule to me for this weekend at least 4 times | 20:26 |
| dansmith | heh | 20:26 |
| sdague | efried: ah | 20:27 |
| *** jmlowe has quit IRC | 20:27 | |
| sdague | efried: I didn't realize people were allowed to override these | 20:27 |
| sdague | efried: what's the use case there? | 20:27 |
| dansmith | mriedem: yes, that's similar to what I was thinking | 20:27 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/488034 | 20:27 |
| sdague | efried: anyway, on this patch, I think it looks overall good, I want to see a devstack run with api_servers not set to see it working | 20:28 |
| sdague | after which I'll +2 | 20:28 |
| sdague | and I'll leave the rest of my questions for mordred | 20:28 |
| sdague | and you at later dates | 20:28 |
| efried | sdague Well, the overall use case is to consolidate/centralize/consistentify (look it up) the way we get clients. | 20:28 |
| efried | sdague So for glance it might be a no-brainer that the service type should always be 'image'. | 20:29 |
| melwitt | mriedem: what type of things will be allowed after today feature freeze? quota cleanups (like test coverage, removing unused stuff, changing the name of recheck_quota config option) or just bug fixes? | 20:29 |
| efried | sdague But we want to be able to do it more or less the same way for e.g. cinder, which (egads) is nowhere near as simple. | 20:29 |
| * smcginnis likes the word consistentify | 20:29 | |
| sdague | efried: yeh, the cinder edge case definitely is a thing. | 20:30 |
| *** eharney has quit IRC | 20:30 | |
| efried | sdague If you want a leetle preview of what that *might* look like: https://review.openstack.org/#/c/487621/1/nova/volume/cinder.py | 20:30 |
| mriedem | melwitt: test coverage is obviously ok, and removing dead code | 20:31 |
| melwitt | k | 20:31 |
| mriedem | jaypipes: let me know before you push anything since i'd like to see a run with https://review.openstack.org/#/c/488187/ | 20:32 |
| *** Guest25498 has quit IRC | 20:32 | |
| *** ijw_ has joined #openstack-nova | 20:32 | |
| mriedem | sdague: i think the list_cells thing will have to be a next week thing | 20:32 |
| mriedem | brain is fried on placement stuff today | 20:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | 20:33 |
| *** ijw_ has quit IRC | 20:33 | |
| sdague | mriedem: that's fine, as long as we can still move it in | 20:33 |
| *** ijw has joined #openstack-nova | 20:33 | |
| *** ijw has quit IRC | 20:35 | |
| jaypipes | mriedem: will do. | 20:39 |
| *** ijw has joined #openstack-nova | 20:40 | |
| mriedem | should have results in ~15 minutes for the live migration job | 20:40 |
| *** zz9pzza has joined #openstack-nova | 20:41 | |
| *** xyang1 has joined #openstack-nova | 20:41 | |
| *** cfriesen_ has joined #openstack-nova | 20:43 | |
| *** zz9pzza has quit IRC | 20:45 | |
| mriedem | sdague: looks like https://review.openstack.org/#/c/487246/ broke the powervm in tree CI | 20:50 |
| mriedem | efried: thorst: ^ | 20:50 |
| mriedem | i'm seeing that here http://184.172.12.213/87/488187/1/check/nova-in-tree-pvm/9c61778/logs/ | 20:51 |
| mriedem | openstack --os-cloud devstack-admin --os-region RegionOne compute service list --host powervm-ci-powervm-devstacked-28324 --service nova-compute -c ID -f value | 20:51 |
| efried | mriedem looking | 20:51 |
| mriedem | shit and it killed ironic | 20:52 |
| mriedem | http://logs.openstack.org/46/487246/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/bec38d6/logs/devstacklog.txt.gz#_2017-07-26_11_55_57_117 | 20:52 |
| sdague | mriedem: ah... gah | 20:52 |
| *** priteau has quit IRC | 20:53 | |
| sdague | right, all the compute for multiple hosts drivers | 20:53 |
| sdague | mriedem: ok, hitting revert | 20:54 |
| mriedem | ack | 20:54 |
| mriedem | the one time i actually look at in-tree powervm CI results :) | 20:54 |
| sdague | heh | 20:54 |
| sdague | you get credit for finding it first | 20:55 |
| mriedem | yay | 20:55 |
| mriedem | i have notified the ironicers | 20:55 |
| sdague | well the revert is submitted and approved | 20:55 |
| mriedem | yeah i see it | 20:55 |
| sdague | so... actually, why isn't the host set that way on powervm and ironic? | 20:56 |
| *** gbarros has joined #openstack-nova | 20:56 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle hash ring rebalancing in ironic flavor migration https://review.openstack.org/487954 | 20:56 |
| sdague | efried: you have a powervm setup somewhere that you can query? | 20:56 |
| efried | esberglu needs to be involved here. | 20:57 |
| *** pwiltsey_ has joined #openstack-nova | 20:57 | |
| *** jjung has quit IRC | 20:57 | |
| *** pwiltsey has quit IRC | 20:58 | |
| edleafe | dansmith: ^^ incorporated rloo's suggestions | 20:59 |
| *** pwiltsey_ has quit IRC | 20:59 | |
| efried | That powervm failure *might* be unrelated. We shouldn't be trying to connect to localhost. | 20:59 |
| efried | sdague Did that change *when* the compute service gets started? | 21:00 |
| sdague | mriedem: I'm actually not sure why hostname wouldn't match in the db | 21:00 |
| sdague | efried: ?? | 21:00 |
| mriedem | efried: no | 21:00 |
| efried | sdague Yeah, I wouldn't have thought so. | 21:00 |
| mriedem | efried: it's polling for the compute node to show up | 21:00 |
| mriedem | by the hostname | 21:00 |
| *** pwiltsey has joined #openstack-nova | 21:00 | |
| efried | So the net is, we're looking into it. | 21:00 |
| sdague | efried: I'm ok with a revert atm because it broke ironic, and we had enough breaks on them this week | 21:01 |
| sdague | but I am curious why those don't seem to line up | 21:01 |
| *** vishwanathj has quit IRC | 21:01 | |
| thorst | I think the main thing for powervm is it shouldn't be taking that long to start up...so that's what we're looking into :-/ | 21:01 |
| *** vishwanathj has joined #openstack-nova | 21:02 | |
| *** dfisher has joined #openstack-nova | 21:02 | |
| mriedem | oh right i forgot it takes 10 minutes for the powervm node to register | 21:03 |
| mriedem | in init_host | 21:03 |
| tonyb | mikal, sdague, melwitt: I don't knwo if this email was wider distrubuted but you know how we moved last_bytes recently .. it seems it was used by nova-lxd | 21:03 |
| melwitt | I think I saw that email | 21:04 |
| sdague | tonyb: they are out of tree, kind of don't care | 21:04 |
| tonyb | mikal, sdague, melwitt: having the (out of tree) nova-lxd driver call into the libvirt code isn't cool :( so shoudl I revert it? | 21:04 |
| tonyb | sdague: Well that was my initial response | 21:04 |
| *** pwiltsey has quit IRC | 21:04 | |
| edmondsw | mriedem I started a change to get the powervm driver up faster but put it aside when we closed things down for pike | 21:04 |
| tonyb | sdague: especially as when we get to queens we move it again | 21:05 |
| sdague | tonyb: no, show up and interact in the community if you are using internal functions in the rest of the tree | 21:05 |
| melwitt | tonyb: didn't they say they're already copy-pasting it somewhere? | 21:05 |
| mriedem | edmondsw: i don't see how nova has control over how fast the backend node comes up | 21:05 |
| tonyb | melwitt: not in the email I have but there may be more | 21:05 |
| mriedem | edmondsw: you were just working on auto-enable the service i thought | 21:05 |
| melwitt | okay, lemme check. maybe I misunderstood it | 21:05 |
| edmondsw | mriedem https://review.openstack.org/#/c/471773/ | 21:05 |
| edmondsw | should complete init_host much faster | 21:07 |
| sdague | tonyb: I wasn't on any such email, but my patience is low for out of tree driver that's not working in the community | 21:07 |
| melwitt | tonyb: okay, I just saw it was the last comment in the review https://review.openstack.org/#/c/472228/ and he's saying that they've now copy-pasted it, not that they had been until now. so I misread it | 21:08 |
| tonyb | okay so we more or less want to say, "Sorry. this is part of the move to privsep so until xxx merged you'll just need to work around it in your driver' | 21:08 |
| tonyb | melwitt: Thanks. | 21:09 |
| tonyb | I'll contact them after breakfast. | 21:09 |
| melwitt | yeah, if it's going to be available again in privsep after the work is done, that seems not so bad to me | 21:09 |
| tonyb | I just wanted to make check we're on the same page | 21:09 |
| tonyb | cool | 21:09 |
| *** gbarros has quit IRC | 21:11 | |
| *** gbarros has joined #openstack-nova | 21:12 | |
| *** dtp has quit IRC | 21:12 | |
| *** cdent has quit IRC | 21:12 | |
| *** yamamoto_ has joined #openstack-nova | 21:13 | |
| sdague | tonyb: yes, also, work upstream :P | 21:13 |
| *** baoli has quit IRC | 21:18 | |
| *** vishwanathj has quit IRC | 21:19 | |
| *** vishwanathj has joined #openstack-nova | 21:20 | |
| *** yamamoto_ has quit IRC | 21:20 | |
| efried | sdague mriedem Okay, we've figured out how that change affected us. We need to do some stuff to the systemctl service file and restart compute. As of now, we're doing that after devstack finishes, not caring that compute wasn't coming up all the way during the stack. With this change, it started to matter that compute wasn't starting. | 21:23 |
| efried | So we're going to figure out how to tweak the service file before stacking starts. Should make compute come right up during the stack process. | 21:23 |
| mriedem | live migration job is done on my sanity check change http://logs.openstack.org/87/488187/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/482b6f4/ | 21:24 |
| *** thorst has quit IRC | 21:25 | |
| *** thorst has joined #openstack-nova | 21:26 | |
| mriedem | oh i've got a bug in there | 21:27 |
| *** vishwanathj has quit IRC | 21:27 | |
| *** thorst_ has joined #openstack-nova | 21:27 | |
| *** vishwanathj has joined #openstack-nova | 21:28 | |
| *** vishwanathj has quit IRC | 21:28 | |
| *** vishwanathj has joined #openstack-nova | 21:29 | |
| *** mgiles has quit IRC | 21:29 | |
| *** thorst has quit IRC | 21:30 | |
| *** esberglu has quit IRC | 21:31 | |
| *** thorst_ has quit IRC | 21:32 | |
| *** vishwanathj has quit IRC | 21:32 | |
| *** vishwanathj has joined #openstack-nova | 21:33 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check delete_allocation_for_instance https://review.openstack.org/488187 | 21:33 |
| *** zz9pzza has joined #openstack-nova | 21:35 | |
| dansmith | jaypipes: https://bugs.launchpad.net/nova/+bug/1707071 | 21:35 |
| openstack | Launchpad bug 1707071 in OpenStack Compute (nova) "Compute nodes will fight over allocations during migration" [Undecided,New] | 21:35 |
| dansmith | dude how awesome is that bug number? | 21:35 |
| *** penick has quit IRC | 21:35 | |
| dansmith | palindromic and all primes | 21:36 |
| *** gongysh has joined #openstack-nova | 21:36 | |
| *** gongysh has quit IRC | 21:36 | |
| dansmith | well, I guess zero isn't a prime.. damn | 21:37 |
| *** zz9pzza has quit IRC | 21:39 | |
| mriedem | hmm, so on rebuild to the same host | 21:40 |
| *** jamesdenton has quit IRC | 21:40 | |
| mriedem | conductor calls select_destinations | 21:40 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L770 | 21:41 |
| mriedem | will we double allocate that host then? | 21:41 |
| *** jamesdenton has joined #openstack-nova | 21:42 | |
| mriedem | oh nvm | 21:43 |
| mriedem | that's if not host | 21:43 |
| *** Apoorva has quit IRC | 21:43 | |
| *** tbachman has quit IRC | 21:43 | |
| *** Apoorva has joined #openstack-nova | 21:44 | |
| mriedem | whew | 21:44 |
| *** esberglu has joined #openstack-nova | 21:45 | |
| *** thorst has joined #openstack-nova | 21:45 | |
| smcginnis | dansmith: You'll have to try to catch 1737371 | 21:47 |
| *** esberglu has quit IRC | 21:49 | |
| *** thorst has quit IRC | 21:50 | |
| *** esberglu has joined #openstack-nova | 21:51 | |
| dansmith | smcginnis: yeah | 21:52 |
| *** armax has joined #openstack-nova | 21:53 | |
| melwitt | time to buy a lotto ticket | 21:56 |
| *** dtp has joined #openstack-nova | 22:02 | |
| *** yamamoto has joined #openstack-nova | 22:04 | |
| *** pwiltsey has joined #openstack-nova | 22:05 | |
| *** mtanino_ has quit IRC | 22:06 | |
| *** edmondsw has quit IRC | 22:06 | |
| *** tbachman has joined #openstack-nova | 22:07 | |
| *** edmondsw has joined #openstack-nova | 22:07 | |
| mriedem | jaypipes: dansmith: ok, https://review.openstack.org/#/c/483566/ | 22:07 |
| *** lbragstad has quit IRC | 22:08 | |
| mriedem | there is one thing in there that worries me | 22:08 |
| mriedem | https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@171 | 22:08 |
| dansmith | hrm | 22:09 |
| mriedem | oh L86 | 22:10 |
| mriedem | if len(selected_hosts) < num_instances: | 22:10 |
| dansmith | what happens if we return less than enough hosts to whatever calls us? | 22:10 |
| *** gouthamr has quit IRC | 22:10 | |
| dansmith | heh yeah that | 22:11 |
| *** gouthamr_ has joined #openstack-nova | 22:11 | |
| dansmith | and min/max_instances is only a quota check, so no problem there | 22:11 |
| *** edmondsw has quit IRC | 22:12 | |
| mriedem | ok +2 | 22:12 |
| mriedem | i tried my hardest | 22:12 |
| mriedem | to find fault | 22:12 |
| *** thorst has joined #openstack-nova | 22:13 | |
| dansmith | ack | 22:14 |
| melwitt | mriedem: docstring doesn't match params here https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@252 if you wanted something :P | 22:14 |
| mriedem | there might still be something to https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@208 | 22:14 |
| *** thorst has quit IRC | 22:14 | |
| mriedem | where if we know allocation requests are constantly failing for a host, we should stop trying it | 22:14 |
| mriedem | omg | 22:15 |
| mriedem | -10 | 22:15 |
| melwitt | hehe | 22:15 |
| mriedem | you know i think i was looking for where cn_uuid was used in there too | 22:15 |
| mriedem | b/c of the docstring | 22:15 |
| *** armax has quit IRC | 22:17 | |
| *** armax has joined #openstack-nova | 22:18 | |
| mriedem | pike-3 tag https://review.openstack.org/#/c/488218/ | 22:18 |
| *** cfriesen_ has quit IRC | 22:19 | |
| mriedem | jaypipes: thanks for hanging in there | 22:21 |
| mriedem | ooo just in time as laura rolls back up to the house with the kid | 22:21 |
| mriedem | time to get my county fair food eating clothes on | 22:21 |
| dansmith | heh | 22:21 |
| mriedem | over sized drawers, bib, etc | 22:21 |
| mriedem | sdague: the compute host wait thing also blew up the cellsv1 job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-cells-ubuntu-xenial/8ac25f1/logs/devstacklog.txt.gz | 22:24 |
| *** rcernin has quit IRC | 22:26 | |
| *** catintheroof has quit IRC | 22:27 | |
| mriedem | dansmith: jaypipes: here is the sanity check in action on the live migration job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/107a810/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_27_22_21_14_766229 | 22:28 |
| mriedem | i didn't see any cases where the source node deleting the allocation isn't in the list of current allocations | 22:28 |
| mriedem | but it's definitely stomping over the 'double up' allocatoins | 22:28 |
| mriedem | Removing allocations for instance which are currently against more than one compute node resource provider. Current allocations: {u'13b1e5e0-66ef-4533-9a07-b1a3220d6b00': {u'generation': 8, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}, u'7aa9619d-db83-4da9-b822-f4d66e7143f8': {u'generation': 6, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}} | 22:29 |
| *** zz9pzza has joined #openstack-nova | 22:29 | |
| dansmith | okay I think that's just getting lucky, | 22:33 |
| dansmith | as I think it could either win or lose that race, but cool | 22:33 |
| *** zz9pzza has quit IRC | 22:34 | |
| *** gouthamr_ has quit IRC | 22:37 | |
| mriedem | oh no you didn't | 22:37 |
| mriedem | https://www.youtube.com/watch?v=mBluR6cLxJ8 | 22:37 |
| *** openstack has joined #openstack-nova | 22:44 | |
| *** med_ has quit IRC | 22:44 | |
| melwitt | dunno | 22:44 |
| *** med_ has joined #openstack-nova | 22:44 | |
| *** baoli has joined #openstack-nova | 22:44 | |
| mikal | I'm going to be a bad man and reply to him cc'ing openstack-dev I suppose | 22:44 |
| mikal | I don't want to exclude the nova team from the conversation | 22:45 |
| *** med_ is now known as Guest13936 | 22:45 | |
| *** oomichi has quit IRC | 22:45 | |
| *** mriedem is now known as mriedem_afk | 22:46 | |
| *** lbragstad has joined #openstack-nova | 22:47 | |
| *** oomichi has joined #openstack-nova | 22:47 | |
| *** rdo has joined #openstack-nova | 22:47 | |
| *** baoli has quit IRC | 22:49 | |
| *** gbarros has joined #openstack-nova | 22:51 | |
| *** xinliang has quit IRC | 22:51 | |
| *** kbaegis has quit IRC | 22:53 | |
| *** gouthamr has joined #openstack-nova | 22:53 | |
| *** annegentle has quit IRC | 22:53 | |
| *** tbachman has quit IRC | 22:53 | |
| *** yassine has joined #openstack-nova | 22:56 | |
| *** tbachman has joined #openstack-nova | 22:57 | |
| *** Apoorva_ has joined #openstack-nova | 22:59 | |
| *** xyang1 has quit IRC | 23:00 | |
| *** Apoorva has quit IRC | 23:02 | |
| *** Apoorva_ has quit IRC | 23:02 | |
| *** hongbin has quit IRC | 23:03 | |
| *** Apoorva has joined #openstack-nova | 23:03 | |
| *** itlinux has joined #openstack-nova | 23:03 | |
| *** xinliang has joined #openstack-nova | 23:04 | |
| *** penick has joined #openstack-nova | 23:05 | |
| *** lbragstad has quit IRC | 23:12 | |
| *** gbarros has quit IRC | 23:12 | |
| *** lyan has quit IRC | 23:15 | |
| *** thorst has joined #openstack-nova | 23:15 | |
| *** gbarros has joined #openstack-nova | 23:19 | |
| *** ijw has quit IRC | 23:19 | |
| *** thorst has quit IRC | 23:21 | |
| *** sdague has quit IRC | 23:21 | |
| *** zz9pzza has joined #openstack-nova | 23:23 | |
| *** zz9pzza has quit IRC | 23:27 | |
| *** takashin has joined #openstack-nova | 23:29 | |
| *** dfisher has quit IRC | 23:33 | |
| *** jaypipes has quit IRC | 23:35 | |
| tonyb | mikal: Yeah the only email was the private one but you've seen the conversation here and on the review | 23:35 |
| tonyb | mikal: I don't see you reply did you send it already? | 23:36 |
| *** edmondsw has joined #openstack-nova | 23:37 | |
| *** Apoorva_ has joined #openstack-nova | 23:40 | |
| *** edmondsw has quit IRC | 23:41 | |
| mikal | tonyb: I cc'ed you and sent it to -dev | 23:43 |
| mikal | Subject: Re: [Nova] Broken nova-lxd | 23:43 |
| *** Apoorva has quit IRC | 23:43 | |
| mikal | http://lists.openstack.org/pipermail/openstack-dev/2017-July/120259.html | 23:44 |
| tonyb | mikal: okay I don't see it yet probably stuck in a mail queue somewhere | 23:44 |
| *** thorst has joined #openstack-nova | 23:46 | |
| *** baoli has joined #openstack-nova | 23:50 | |
| openstackgerrit | Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | 23:50 |
| *** catintheroof has joined #openstack-nova | 23:52 | |
| *** thorst has quit IRC | 23:55 | |
| *** thorst has joined #openstack-nova | 23:56 | |
| *** rajathagasthya has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!