| *** salv-orlando has joined #openstack-nova | 00:07 | |
| *** gmann has quit IRC | 00:07 | |
| *** gmann has joined #openstack-nova | 00:07 | |
| *** salv-orlando has quit IRC | 00:12 | |
| *** felipemonteiro__ has quit IRC | 00:12 | |
| *** edmondsw has joined #openstack-nova | 00:45 | |
| *** zhurong has joined #openstack-nova | 00:45 | |
| *** tuanluong has joined #openstack-nova | 00:48 | |
| *** timello has joined #openstack-nova | 00:48 | |
| *** slaweq has joined #openstack-nova | 00:51 | |
| *** Shunli has joined #openstack-nova | 00:54 | |
| *** slaweq has quit IRC | 00:55 | |
| *** edmondsw has quit IRC | 00:59 | |
| *** phuongnh has joined #openstack-nova | 01:00 | |
| *** edmondsw has joined #openstack-nova | 01:00 | |
| *** mriedem has joined #openstack-nova | 01:02 | |
| mriedem | o/ | 01:02 |
|---|---|---|
| mriedem | if there is one thing you should get at the tea house, it's coffee | 01:02 |
| *** edmondsw has quit IRC | 01:05 | |
| *** artom has joined #openstack-nova | 01:06 | |
| *** salv-orlando has joined #openstack-nova | 01:08 | |
| *** liujiong has joined #openstack-nova | 01:11 | |
| *** salv-orlando has quit IRC | 01:13 | |
| *** xinliang_ has quit IRC | 01:17 | |
| *** liusheng has quit IRC | 01:23 | |
| *** xinliang_ has joined #openstack-nova | 01:30 | |
| *** gcb has joined #openstack-nova | 01:32 | |
| *** hongbin has joined #openstack-nova | 01:33 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Deprecate binary argument in nova service enable/disable/force-down CLIs https://review.openstack.org/477212 | 01:45 |
| mriedem | Vek: getting some weird output with ^ | 01:53 |
| mriedem | i think it doesn't work because binary is not an option, it's an argument | 01:58 |
| *** tonygunk has quit IRC | 02:01 | |
| *** zhurong has quit IRC | 02:01 | |
| *** s1061123 has joined #openstack-nova | 02:05 | |
| *** tonygunk has joined #openstack-nova | 02:08 | |
| *** salv-orlando has joined #openstack-nova | 02:09 | |
| *** salv-orlando has quit IRC | 02:14 | |
| *** gouthamr has joined #openstack-nova | 02:16 | |
| mriedem | melwitt: novaclient functional test runs are hitting some very weird over quota failures; i added a patch to novaclient to dump the nova list when the tests fail with overquota on instances, and nothing is listed http://logs.openstack.org/17/477217/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/0aaa523/console.html#_2017-06-25_15_10_36_826560 | 02:21 |
| mriedem | it's also weird because we're not using the counting quotas stuff for instances yet | 02:21 |
| mriedem | i wonder if we have something broken in the delete path with cells, like we're not decrementing quota on delete | 02:22 |
| *** takashin has quit IRC | 02:22 | |
| *** liusheng has joined #openstack-nova | 02:24 | |
| *** tbachman has joined #openstack-nova | 02:25 | |
| *** fried_rice is now known as efried | 02:27 | |
| *** sbezverk has joined #openstack-nova | 02:34 | |
| *** tbachman has quit IRC | 02:35 | |
| *** zhurong has joined #openstack-nova | 02:40 | |
| *** hshiina has joined #openstack-nova | 02:51 | |
| *** huanxie has joined #openstack-nova | 02:53 | |
| *** slaweq has joined #openstack-nova | 02:53 | |
| *** gbarros has joined #openstack-nova | 02:56 | |
| *** zhurong has quit IRC | 02:56 | |
| *** slaweq has quit IRC | 02:58 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug over quota failures in novaclient functional https://review.openstack.org/477315 | 03:03 |
| *** armax has quit IRC | 03:04 | |
| *** armax has joined #openstack-nova | 03:04 | |
| *** zhurong has joined #openstack-nova | 03:07 | |
| *** gcb has quit IRC | 03:14 | |
| *** armax has quit IRC | 03:14 | |
| *** litao__ has joined #openstack-nova | 03:15 | |
| *** gcb has joined #openstack-nova | 03:15 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: DNM: debug overquota failures https://review.openstack.org/477323 | 03:17 |
| *** wangqwsh has joined #openstack-nova | 03:19 | |
| *** gouthamr has quit IRC | 03:22 | |
| *** zhurong has quit IRC | 03:23 | |
| *** vladikr_ has joined #openstack-nova | 03:28 | |
| *** zhurong has joined #openstack-nova | 03:35 | |
| *** artom_ has joined #openstack-nova | 03:38 | |
| *** artom has quit IRC | 03:42 | |
| openstackgerrit | Liu Qing proposed openstack/nova master: Initial empty repository https://review.openstack.org/477352 | 03:42 |
| *** hferenc has quit IRC | 03:42 | |
| openstackgerrit | Liu Qing proposed openstack/nova master: Add refresh connection info in rollback live migration https://review.openstack.org/477353 | 03:42 |
| *** gbarros has quit IRC | 03:42 | |
| *** vladikr_ has quit IRC | 03:46 | |
| *** gbarros has joined #openstack-nova | 03:48 | |
| *** kornicameister has joined #openstack-nova | 03:54 | |
| *** yamamoto has joined #openstack-nova | 03:54 | |
| *** lpetrut has joined #openstack-nova | 03:56 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Amend the "list-show-all-server-migration-types" spec https://review.openstack.org/460847 | 03:58 |
| *** takashin has joined #openstack-nova | 03:59 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732 | 03:59 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Specify keymap on server boot https://review.openstack.org/448830 | 03:59 |
| *** yamahata has joined #openstack-nova | 04:00 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Verify parameters in os-migrations.inc https://review.openstack.org/474879 | 04:00 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add missing parameters in limits.inc https://review.openstack.org/475640 | 04:00 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add 'updated_at' field to InstancePayload in notifications https://review.openstack.org/475276 | 04:00 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Use PCIAddressField in oslo.versionedobjects https://review.openstack.org/466201 | 04:00 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add X-Openstack-Request-Id description https://review.openstack.org/474847 | 04:00 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Avoid forcing translation on logging calls https://review.openstack.org/413876 | 04:01 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Fix missing parameters in API Versions https://review.openstack.org/475226 | 04:01 |
| *** amotoki_away is now known as amotoki | 04:02 | |
| *** gbarros has quit IRC | 04:07 | |
| *** diga has joined #openstack-nova | 04:08 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Fix parameters in server-security-groups https://review.openstack.org/476434 | 04:09 |
| *** salv-orlando has joined #openstack-nova | 04:10 | |
| *** hongbin has quit IRC | 04:11 | |
| *** kornicameister has quit IRC | 04:11 | |
| *** cNilesh has joined #openstack-nova | 04:11 | |
| *** salv-orlando has quit IRC | 04:15 | |
| *** adisky__ has joined #openstack-nova | 04:23 | |
| *** cNilesh has quit IRC | 04:24 | |
| *** cNilesh has joined #openstack-nova | 04:24 | |
| *** cNilesh has quit IRC | 04:24 | |
| *** cNilesh has joined #openstack-nova | 04:24 | |
| *** cNilesh has quit IRC | 04:27 | |
| *** cNilesh has joined #openstack-nova | 04:27 | |
| *** edmondsw has joined #openstack-nova | 04:28 | |
| *** edmondsw has quit IRC | 04:32 | |
| *** lpetrut has quit IRC | 04:38 | |
| *** links has joined #openstack-nova | 04:40 | |
| *** kornicameister has joined #openstack-nova | 04:41 | |
| *** slaweq has joined #openstack-nova | 04:55 | |
| *** lpetrut has joined #openstack-nova | 04:55 | |
| *** slaweq has quit IRC | 04:59 | |
| *** artom_ has quit IRC | 05:00 | |
| *** salv-orlando has joined #openstack-nova | 05:04 | |
| *** yangyapeng has joined #openstack-nova | 05:06 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Stop using mox stubs in test_console_auth_tokens.py https://review.openstack.org/477374 | 05:20 |
| *** udesale has joined #openstack-nova | 05:24 | |
| *** udesale has quit IRC | 05:24 | |
| *** yangyapeng has quit IRC | 05:35 | |
| *** MasterOfBugs has joined #openstack-nova | 05:40 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug over quota failures in novaclient functional https://review.openstack.org/477315 | 05:42 |
| *** psachin has joined #openstack-nova | 05:44 | |
| *** yangyapeng has joined #openstack-nova | 05:47 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Stop using mox stubs in test_keypairs.py https://review.openstack.org/477384 | 05:51 |
| *** winston-d_ has joined #openstack-nova | 05:53 | |
| mriedem | alex_xu: my guess is that we are reading and writing quotas to different databases | 05:53 |
| alex_xu | mriedem: ok | 05:53 |
| mriedem | but i'm not sure what changed recently around 6/24 that would cause that except for maybe the devstack patch to fleetify conductor | 05:53 |
| mriedem | but if it was that change, i'm not sure why it would just be breaking novaclient functional job and not everything else | 05:53 |
| alex_xu | ok, got it | 05:54 |
| *** rcernin has joined #openstack-nova | 05:59 | |
| *** moshele has joined #openstack-nova | 06:03 | |
| *** yangyape_ has joined #openstack-nova | 06:08 | |
| *** yangyapeng has quit IRC | 06:10 | |
| *** artom_ has joined #openstack-nova | 06:11 | |
| *** Oku_OS-away is now known as Oku_OS | 06:12 | |
| *** MasterOfBugs has quit IRC | 06:13 | |
| *** MasterOfBugs has joined #openstack-nova | 06:13 | |
| *** yangyape_ has quit IRC | 06:14 | |
| *** edmondsw has joined #openstack-nova | 06:16 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug over quota failures in novaclient functional https://review.openstack.org/477315 | 06:16 |
| *** artom_ has quit IRC | 06:17 | |
| *** yangyapeng has joined #openstack-nova | 06:17 | |
| *** yangyapeng has quit IRC | 06:18 | |
| *** salv-orlando has quit IRC | 06:19 | |
| *** salv-orlando has joined #openstack-nova | 06:19 | |
| *** edmondsw has quit IRC | 06:20 | |
| *** tesseract has joined #openstack-nova | 06:24 | |
| *** salv-orlando has quit IRC | 06:24 | |
| *** andreas_s has joined #openstack-nova | 06:27 | |
| *** salv-orlando has joined #openstack-nova | 06:28 | |
| *** edand has joined #openstack-nova | 06:33 | |
| *** yangyapeng has joined #openstack-nova | 06:34 | |
| *** yangyape_ has joined #openstack-nova | 06:35 | |
| *** salv-orl_ has joined #openstack-nova | 06:37 | |
| *** markus_z has joined #openstack-nova | 06:38 | |
| *** yangyapeng has quit IRC | 06:39 | |
| *** salv-orlando has quit IRC | 06:39 | |
| *** edand has quit IRC | 06:39 | |
| *** yamahata has quit IRC | 06:43 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add 'updated_at' field to InstancePayload in notifications https://review.openstack.org/475276 | 06:44 |
| *** pcaruana has joined #openstack-nova | 06:45 | |
| *** jpena|off is now known as jpena | 06:50 | |
| *** lpetrut has quit IRC | 06:53 | |
| *** slaweq has joined #openstack-nova | 06:56 | |
| *** slaweq has quit IRC | 07:00 | |
| *** yangyape_ has quit IRC | 07:04 | |
| *** lpetrut has joined #openstack-nova | 07:09 | |
| *** artom_ has joined #openstack-nova | 07:09 | |
| *** hferenc has joined #openstack-nova | 07:10 | |
| openstackgerrit | sahid proposed openstack/nova master: libvirt: update the logic to configure volume with scsi controller https://review.openstack.org/459741 | 07:11 |
| openstackgerrit | sahid proposed openstack/nova master: libvirt: update logic to configure device for scsi controller https://review.openstack.org/459740 | 07:11 |
| *** damien_r has joined #openstack-nova | 07:15 | |
| *** Shunli has quit IRC | 07:16 | |
| *** Shunli has joined #openstack-nova | 07:16 | |
| *** artom_ has quit IRC | 07:22 | |
| *** lpetrut has quit IRC | 07:42 | |
| *** ralonsoh has joined #openstack-nova | 07:44 | |
| *** lyan has joined #openstack-nova | 07:51 | |
| mriedem | alex_xu: i've reverted the devstack change https://review.openstack.org/477436 and will run a novaclient change that depends on that one to see if it's what caused the breakage | 07:54 |
| mriedem | i think this is probably not failing for the normal dsvm jobs because those are multi-tenant, and the novaclient functional tests are single tenant | 07:54 |
| alex_xu | mriedem: ok, I have nothing found... | 07:54 |
| *** sdatko has quit IRC | 07:54 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: DNM: debug overquota failures https://review.openstack.org/477323 | 07:55 |
| *** yangyapeng has joined #openstack-nova | 07:55 | |
| alex_xu | mriedem: before overquota, there is instance deletion, the log show the quota is -1...http://logs.openstack.org/17/477217/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/0aaa523/logs/screen-n-api.txt.gz#_Jun_25_15_10_32_557593 | 07:57 |
| *** pcaruana has quit IRC | 07:58 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_availability_zone.py https://review.openstack.org/477439 | 07:59 |
| *** rcernin has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-nova | 08:00 | |
| mriedem | the -1 is to decrement usage | 08:00 |
| mriedem | resource: instances, tracked usage: -1, actual usage: 1 | 08:01 |
| mriedem | yeah so there is 1 instance and it should decrement usage on delete, so you'd get back to 0 instances used | 08:01 |
| *** takashin has left #openstack-nova | 08:03 | |
| *** edmondsw has joined #openstack-nova | 08:04 | |
| *** edmondsw has quit IRC | 08:08 | |
| *** alexey_weyl has joined #openstack-nova | 08:11 | |
| alexey_weyl | Hi, | 08:11 |
| *** rcernin has joined #openstack-nova | 08:12 | |
| alexey_weyl | We are having some problems with nova in the Vitrage project | 08:12 |
| alexey_weyl | We don't get the notification on the oslo bus about the compute.instance.create.end | 08:12 |
| *** pcaruana has joined #openstack-nova | 08:12 | |
| alexey_weyl | and some other notifications as well. | 08:12 |
| alexey_weyl | are you familiar with such a problem? | 08:12 |
| alexey_weyl | This has happened sometime between the 23rd till now | 08:13 |
| *** yangyapeng has quit IRC | 08:18 | |
| *** junbo has quit IRC | 08:21 | |
| gibi | alexey_weyl: have you already opened a bug? | 08:21 |
| gibi | alexey_weyl: I'm not aware of an open bug regarding this | 08:21 |
| gibi | alexey_weyl: do you use legacy or versioned notification? | 08:22 |
| alexey_weyl | gibi: I am not sure about the notification type that we use | 08:22 |
| alexey_weyl | gibi: was there something changed that might have caused that | 08:22 |
| alexey_weyl | we have looked at the changes since the 23rd and havent seen something there | 08:23 |
| *** junbo has joined #openstack-nova | 08:25 | |
| gibi | alexey_weyl: does vitrage listen to notifications topic or versioned_notifications topic? | 08:26 |
| gibi | alexey_weyl: anyhow I will fire up a devstack to see if I can reproduce the problem | 08:26 |
| gibi | alexey_weyl: do you have a list of notifications that is missing? or is it just compute.instance.create.end? | 08:27 |
| alexey_weyl | gibi: notifications topic | 08:27 |
| gibi | alexey_weyl: OK, then this is the legacy notificatitons | 08:27 |
| alexey_weyl | at the moment the most important one is the 'compute.instance.create.end' | 08:27 |
| gibi | alexey_weyl: OK | 08:27 |
| alexey_weyl | gibi: thanks. waiting for you update | 08:28 |
| openstackgerrit | Alex Xu proposed openstack/python-novaclient master: [DNM] Wait for server deletion and list servers when over quota in tests https://review.openstack.org/477446 | 08:28 |
| gibi | alexey_weyl: I see the notification with nova version bc4c91e7be8e4c52506a8f970078ecee6655ee82 but that is a bit old, so I updating now to master | 08:35 |
| alexey_weyl | gibi: we are talking about changes that was were made sometime between the 23rd and the 25th of June. | 08:36 |
| gibi | alexey_weyl: OK, thanks for the info | 08:37 |
| *** yamamoto has quit IRC | 08:38 | |
| gibi | alexey_weyl: I also saw the notification with the latest nova master 3863eca0ac5a9437354288de673c831d5dc20f2e | 08:39 |
| *** bauwser is now known as bauzas | 08:39 | |
| *** yamamoto has joined #openstack-nova | 08:39 | |
| alexey_weyl | gibi: wow thats weird | 08:42 |
| *** derekh has joined #openstack-nova | 08:42 | |
| gibi | alexey_weyl: I use basic devstack nova conf | 08:42 |
| alexey_weyl | gibi: can I send you a link to our tempests and see why this is happening | 08:42 |
| gibi | gibi: sure | 08:42 |
| gibi | alexey_weyl: sure | 08:42 |
| *** zz9pzza has quit IRC | 08:42 | |
| alexey_weyl | gibi: http://logs.openstack.org/95/477395/2/check/gate-vitrage-dsvm-api-ubuntu-xenial/a16cd7c/ | 08:43 |
| *** zz9pzza has joined #openstack-nova | 08:43 | |
| mriedem | alexey_weyl: gibi: maybe https://review.openstack.org/#/c/436094/ | 08:43 |
| *** psachin has quit IRC | 08:43 | |
| mriedem | that merged on the 24th | 08:43 |
| mriedem | changes configuration for nova in devstack | 08:43 |
| mriedem | so maybe something changed with the config being used | 08:43 |
| alexey_weyl | gibi: the creation of the vms started at 07:34:55 | 08:43 |
| mriedem | there are now 3 nova config files | 08:44 |
| *** yamamoto has quit IRC | 08:45 | |
| *** avolkov has quit IRC | 08:45 | |
| mriedem | http://logs.openstack.org/95/477395/2/check/gate-vitrage-dsvm-api-ubuntu-xenial/a16cd7c/logs/etc/nova/ | 08:45 |
| *** avolkov has joined #openstack-nova | 08:45 | |
| mriedem | notification_topics = notifications,vitrage_notifications | 08:46 |
| gibi | mriedem: thank, then I have to update my devstack I guess as well | 08:46 |
| *** diga has quit IRC | 08:46 | |
| gibi | mriedem, alexey_weyl: If I can reproduce the problem then I can test if a revert helps or not | 08:46 |
| mriedem | there are already 2 reverts up for that devstack change :) | 08:46 |
| mriedem | i think it's causing some issues with novaclient functional tests too | 08:47 |
| *** MasterOfBugs has quit IRC | 08:47 | |
| gibi | mriedem, alexey_weyl: btw the instance.create comes from nova-compute service | 08:47 |
| mriedem | https://review.openstack.org/#/c/477436/ | 08:47 |
| gibi | anyhow I'm updating my devstack now and doing unstack/stack | 08:47 |
| mriedem | nova-compute is using this config file now: http://logs.openstack.org/95/477395/2/check/gate-vitrage-dsvm-api-ubuntu-xenial/a16cd7c/logs/etc/nova/nova-cpu.conf.txt.gz | 08:47 |
| *** zz9pzza has quit IRC | 08:47 | |
| gibi | alexey_weyl: could you try the vitrage tempest test with the above devstack patch reverted in the meantime? | 08:48 |
| *** dtantsur|afk is now known as dtantsur | 08:48 | |
| alexey_weyl | gibi: how can I do that? | 08:50 |
| gibi | alexey_weyl: can you run the tempest test locally? if yes then pull down the devstack revert then stack with that then run the tempest | 08:50 |
| *** crushil has quit IRC | 08:50 | |
| mriedem | alexey_weyl: add this to your change's commit message: | 08:50 |
| mriedem | Depends-On: I541abe7311e5721d79e300fcb2a0b8ffdcd3c015 | 08:50 |
| mriedem | zuul will pick up on the depends-on | 08:51 |
| mriedem | and pull in the devstack revert | 08:51 |
| gibi | mriedem: the notification part of that conf of nova-compute looks good to me | 08:51 |
| *** Shunli has quit IRC | 08:51 | |
| *** crushil has joined #openstack-nova | 08:51 | |
| *** Shunli has joined #openstack-nova | 08:51 | |
| gibi | mriedem: clever :) | 08:52 |
| *** zz9pzza has joined #openstack-nova | 08:52 | |
| *** yangyapeng has joined #openstack-nova | 08:52 | |
| alexey_weyl | gibi: mriedim: sorry, bu my devstack is very old, and I don't know why but runnig tempests locally doesn't work on it lately as well | 08:53 |
| bauzas | is it me becoming crazy or I can see some mriedem on the floor ? | 08:53 |
| alexey_weyl | I can try but it can take me a couple of hours atleast | 08:54 |
| gibi | alexey_weyl: what mriedem suggest only needs a patch proposed in gerrit with a depends-on to the devstack revert and the jenkins will run the test for you | 08:54 |
| *** yangyape_ has joined #openstack-nova | 08:54 | |
| mriedem | bauzas: on the floor? | 08:54 |
| gibi | bauzas, mriedem: I like the idea to have mriedem in this timezone :) | 08:54 |
| mriedem | i am in all timezones at all times now | 08:55 |
| bauzas | mriedem: some kind of dancing analogy | 08:55 |
| mriedem | oh | 08:55 |
| mriedem | yeah i'm in china this week | 08:55 |
| mriedem | been waiting for you lazy bums to wake up in the EU | 08:55 |
| gibi | lol | 08:55 |
| bauzas | oh man, I should wake up way earlier then now that the boss is watching me | 08:55 |
| mriedem | alex_xu: the novaclient change passed with the devstack revert | 08:56 |
| mriedem | so it's definitely something with that change that broke quotas | 08:56 |
| bauzas | mriedem: yet again impacted by some French mandatory holiday stuff, so will be afk Tues-Wed *again* | 08:56 |
| alexey_weyl | gibi: mriedem: trying what you suggested :) | 08:56 |
| *** slaweq has joined #openstack-nova | 08:57 | |
| *** yangyapeng has quit IRC | 08:57 | |
| *** lpetrut has joined #openstack-nova | 08:58 | |
| gibi | alexey_weyl: I pushed the patch https://review.openstack.org/477455 | 08:58 |
| gibi | alexey_weyl: now we have to just wait for the tempest result | 08:58 |
| alexey_weyl | gibi: thanks, I have pushed it as well :) | 08:59 |
| gibi | then I revert mine and let you continue | 08:59 |
| gibi | i mean abandond | 08:59 |
| alexey_weyl | gibi: you shouldn't have :) | 09:00 |
| gibi | alexey_weyl: you mean we can have two results instead of one? | 09:00 |
| alexey_weyl | gibi: yes :) | 09:01 |
| gibi | alexey_weyl: OK, why not :) | 09:01 |
| alex_xu | mrhillsman: or quotas doesn't work as in some expected case... | 09:01 |
| alex_xu | mriedem: or quotas doesn't work as in some expected case... | 09:01 |
| mriedem | alex_xu: it could be there is a bug yes | 09:02 |
| mriedem | i don't quite know what that devstack change totally does, it's not trivial, so we'll probably need dan's help when he's up | 09:02 |
| *** slaweq has quit IRC | 09:02 | |
| alex_xu | mrhillsman: sorry..tab to auto-compelete too early | 09:02 |
| *** khappone has quit IRC | 09:03 | |
| *** yamamoto has joined #openstack-nova | 09:03 | |
| mriedem | i know the single-node jobs went from 1 nova.conf to 3 | 09:03 |
| mriedem | so i suspect some issue there | 09:03 |
| alex_xu | mriedem: and a cell conductor separate than super conductor | 09:04 |
| mriedem | alex_xu: in the single node job there is only 1 conductor i think | 09:05 |
| *** khappone has joined #openstack-nova | 09:05 | |
| mriedem | oh nvm | 09:05 |
| mriedem | there are 2 | 09:05 |
| alex_xu | mriedem: but I can see two log files | 09:05 |
| alex_xu | yea | 09:05 |
| mriedem | yeah so super conductor is using /etc/nova/nova.conf | 09:06 |
| mriedem | and the cell1 conductor is using nova-cellv1.conf | 09:06 |
| mriedem | *nova-cell1.conf | 09:06 |
| mriedem | i don't know which conductor n-cpu is talking to | 09:07 |
| *** avolkov has quit IRC | 09:07 | |
| *** avolkov has joined #openstack-nova | 09:07 | |
| mriedem | probably the cell1 conductor | 09:07 |
| alex_xu | mriedem: yea, http://logs.openstack.org/17/477217/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/0aaa523/logs/screen-n-cond-cell1.txt.gz | 09:07 |
| alex_xu | cell1 conductor's log is all about quota commit | 09:07 |
| *** diga has joined #openstack-nova | 09:08 | |
| *** avolkov has quit IRC | 09:09 | |
| * alex_xu didn't found something wrong | 09:09 | |
| *** avolkov has joined #openstack-nova | 09:12 | |
| gibi | alexey_weyl, mriedem: with the suspected devstack change I was able to reproduce the problem with the notifications | 09:15 |
| gibi | alexey_weyl, mriedem: the only notification that is received from a nova boot is the scheduler.select_destinations | 09:15 |
| bauzas | gibi: we use different RPC topics, right? | 09:16 |
| bauzas | between notifications and other RPC services ? | 09:17 |
| bauzas | or are we sharing the same configs? | 09:17 |
| gibi | bauzas: we are using different topics | 09:17 |
| gibi | bauzas: notifications are on topic 'notifications' and 'versioned_notifications' | 09:17 |
| bauzas | a topic is a thing | 09:17 |
| alexey_weyl | gibi: I saw also that compute.instance.update was sent | 09:18 |
| gibi | interestingly notification from nova-api works (I tried an aggregate create) | 09:18 |
| bauzas | gibi: I suspect notification emitted from a cell don't use the same RPC backend | 09:19 |
| bauzas | than notifications emitted from the API | 09:19 |
| bauzas | because https://review.openstack.org/#/c/436094/35/lib/nova@636 | 09:19 |
| *** wangqwsh has quit IRC | 09:20 | |
| gibi | I have to look into the different configs now | 09:21 |
| gibi | OK my previous comment about fault reproduction wasn't correct | 09:22 |
| *** yangyapeng has joined #openstack-nova | 09:22 | |
| gibi | my nova-cpu.conf was not set properly for notifications | 09:22 |
| gibi | now I receive notifications from nova-compute as well | 09:22 |
| gibi | in my devstack with the suspected commit | 09:23 |
| bauzas | the suspected commit being ? | 09:23 |
| *** yangyape_ has quit IRC | 09:23 | |
| gibi | bauzas: https://review.openstack.org/#/c/436094 | 09:23 |
| bauzas | okay, so we're on the same page | 09:24 |
| gibi | bauzas: we are trying to figure out what breaks notificaiton in vitrage tempest | 09:24 |
| *** klipka has joined #openstack-nova | 09:24 | |
| bauzas | I saw | 09:24 |
| gibi | bauzas: we suspected the above devstack commit | 09:24 |
| gibi | bauzas: but I still cannot reproduce the problem | 09:25 |
| mriedem | i know it's breaking quotas in the novaclient functional job | 09:25 |
| mriedem | which is single-tenant | 09:25 |
| bauzas | gibi: mriedem: I'm confused about which RPC backend compute and cell conductor use for notifications | 09:27 |
| *** crushil has quit IRC | 09:28 | |
| bauzas | gibi: mriedem: IIUC, those services are using what we give to them, just using separate topics for notifcations, correct? | 09:28 |
| *** crushil has joined #openstack-nova | 09:28 | |
| *** Shunli has quit IRC | 09:28 | |
| bauzas | gibi: mriedem: so, notifications emitted from a cell (either a local conductor or a compute service) are using the "cell" RPC backend but with a distinct MQ topic | 09:29 |
| bauzas | now, my question is, are we sure that vitrage is correctly subscribing to those extra backends too ? | 09:29 |
| gibi | bauzas: as far as I understand notifications use the same transport_url but different topic | 09:29 |
| bauzas | gibi: but the transport URL could be different if you're intra-cell, that's the whole point of cells v2 | 09:30 |
| gibi | bauzas: so if vitrage doesn't subscribe to /nova_cell1 then they will not get the notifications | 09:30 |
| bauzas | that's my point | 09:30 |
| gibi | bauzas: make sense | 09:30 |
| bauzas | or, it would require some relaying | 09:31 |
| mriedem | compute and cell conductor are using the same transport url | 09:31 |
| mriedem | http://logs.openstack.org/17/477217/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/0aaa523/logs/etc/nova/nova-cpu.conf.txt.gz | 09:31 |
| mriedem | http://logs.openstack.org/17/477217/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/0aaa523/logs/etc/nova/nova_cell1.conf.txt.gz | 09:31 |
| bauzas | mriedem: of course | 09:31 |
| mriedem | transport_url = rabbit://stackrabbit:secretrabbit@10.0.1.148:5672/nova_cell1 | 09:31 |
| bauzas | mriedem: but not the same transport URL as the API services right? | 09:31 |
| mriedem | correct | 09:31 |
| bauzas | see my point ? | 09:31 |
| mriedem | api is using transport_url = rabbit://stackrabbit:secretrabbit@10.0.1.148:5672/ | 09:31 |
| gibi | this is the vitrage config from the failed tempest http://logs.openstack.org/95/477395/2/check/gate-vitrage-dsvm-api-ubuntu-xenial/a16cd7c/logs/etc/vitrage/vitrage.conf.txt.gz | 09:32 |
| bauzas | so we emit notifications on a very different URL | 09:32 |
| bauzas | CFQD. | 09:32 |
| bauzas | oops, French word | 09:32 |
| bauzas | https://english.stackexchange.com/questions/128047/is-there-an-equivalent-for-french-cqfd | 09:33 |
| gibi | bauzas, mriedem, alexey_weyl: so vitrage needs to listen to the cell MQ as well | 09:33 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Use ``assert_has_calls`` to check function calls https://review.openstack.org/477469 | 09:33 |
| bauzas | gibi: that's the short-term fix | 09:33 |
| bauzas | gibi: but the fact is that I think we probably never thought whether we should emit notifications per-cell | 09:34 |
| gibi | bauzas: yeah, we have to agree on the expected behavior in case of cells | 09:34 |
| *** sambetts|afk is now known as sambetts | 09:35 | |
| mriedem | my guess the quotas issue is similar, in that on server create we commit quota usage changes in one db but on delete we decrement from another db | 09:39 |
| gibi | mriedem, bauzas: while in case of quota it make sense to relay the request from the cell up but in case of notification it might be overkill | 09:43 |
| gibi | tempest just proved that the devstack revert could solve the vitrage failure https://review.openstack.org/#/c/477455/ | 09:44 |
| *** jamesden_ has joined #openstack-nova | 09:46 | |
| *** rcernin has quit IRC | 09:46 | |
| *** jamesdenton has quit IRC | 09:47 | |
| *** aarefiev_afk is now known as aarefiev | 09:47 | |
| *** mvk has joined #openstack-nova | 09:47 | |
| *** edmondsw has joined #openstack-nova | 09:52 | |
| alexey_weyl | gibi: what should I do then? | 09:55 |
| *** edmondsw has quit IRC | 09:56 | |
| bauzas | alexey_weyl: so, we need to discuss internally about cells | 09:56 |
| *** salv-orl_ has quit IRC | 09:56 | |
| gibi | alexey_weyl: I think the devstack patch will be reverted by https://review.openstack.org/#/c/477436/ and that will fix vitrage job temporarly | 09:57 |
| bauzas | alexey_weyl: but the problem is that we now have multiple MQs | 09:57 |
| alexey_weyl | MQs? | 09:57 |
| bauzas | message queues | 09:57 |
| alexey_weyl | I see | 09:57 |
| *** salv-orlando has joined #openstack-nova | 09:57 | |
| *** Prashant has joined #openstack-nova | 09:57 | |
| bauzas | ie. one per cell + something we say a "api MQ" | 09:57 |
| bauzas | gibi: FWIW, do you know how notifications work with cells v1 ? | 09:58 |
| gibi | bauzas: I have no idea | 09:58 |
| *** slaweq has joined #openstack-nova | 09:59 | |
| gibi | bauzas: what is the MQ transport_url for the compute in cell v1? | 09:59 |
| bauzas | gibi: what mriedem said | 09:59 |
| bauzas | (11:31:22) mriedem: transport_url = rabbit://stackrabbit:secretrabbit@10.0.1.148:5672/nova_cell1 | 09:59 |
| bauzas | oh oops | 09:59 |
| bauzas | sorry | 09:59 |
| bauzas | I thought you said "cell 1" | 09:59 |
| gibi | np | 09:59 |
| bauzas | for cells v1, we have multiple MQs too | 09:59 |
| Prashant | Hi Team..I have setup running stable/ocata..After scaling around 2000+ instance, I am not able to launch single instance.. All I could see is that scheduler filter is returning 0 hosts. The only log on nova-compute I see is | 10:00 |
| Prashant | 2017-06-26 09:17:38.591 11246 DEBUG nova.scheduler.client.report [req-5c4276f0-68ae-454c-a8f0-310f7de9b4de - - - - -] [instance: c08adb40-38c9-4bdb-b60b-1d4a58d9fdfd] Sending allocation for instance [{' resource_provider': {'uuid': '5c471cf8-7e64-41b4-9c7d-a49071690dc8'}, 'resources': {'MEMORY_MB': 2048, 'VCPU': 1, 'DISK_GB': 20}}] _allocate_for_instance /usr/lib/python2.7/dist-packages/nova/scheduler/ client/report.py:837 | 10:00 |
| Prashant | Any idea how to resolve this issue? | 10:00 |
| Prashant | nova hypervisor-list shows all computes are up and running | 10:00 |
| Prashant | and I do have resource to launch more instances | 10:00 |
| *** mkucia has joined #openstack-nova | 10:00 | |
| gibi | bauzas: transport_url = rabbit://stackrabbit:secretrabbit@10.0.1.16:5672/child_cell | 10:01 |
| gibi | bauzas: this is in cell v1 nova-cells.conf | 10:01 |
| bauzas | for example with devstack | 10:01 |
| bauzas | yup | 10:01 |
| Prashant | "2017-06-26 09:17:38.615 11246 WARNING nova.scheduler.client.report [req-5c4276f0-68ae-454c-a8f0-310f7de9b4de - - - - -] Unable to submit allocation for instance c08adb40-38c9-4bdb-b60b-1d4a58d9fdfd (409 409 Conflict There was a conflict when trying to complete your request. Unable to allocate inventory: Unable to create allocation for 'DISK_GB' on resource provider '5c471cf8-7e64-41b4-9c7d-a49071690dc8'. The requested a | 10:01 |
| bauzas | we have child cells and parent cell | 10:01 |
| gibi | bauzas: now if nova-compute service uses that conf then nova-computes sends the notifications there | 10:01 |
| *** salv-orlando has quit IRC | 10:01 | |
| bauzas | gibi: exactly | 10:02 |
| bauzas | so, possibly the same reason | 10:02 |
| bauzas | we should actually be discussing with operators | 10:02 |
| bauzas | mriedem: wanting to open a thread or me ? | 10:02 |
| *** rcernin has joined #openstack-nova | 10:02 | |
| *** slaweq has quit IRC | 10:03 | |
| mriedem | bauzas: on the ML? my account is through google which is blocked here | 10:03 |
| mriedem | for cellsv1, the child cell uses this config http://logs.openstack.org/15/477315/3/check/gate-tempest-dsvm-cells-ubuntu-xenial/fc4b3a1/logs/etc/nova/nova-cells.conf.txt.gz | 10:03 |
| mriedem | so it uses a separate mq transport_url = rabbit://stackrabbit:secretrabbit@213.32.79.197:5672/child_cell | 10:03 |
| gibi | and in devstack nova-compute uses nova-cells.conf | 10:03 |
| gibi | run_process n-cpu '/usr/local/bin/nova-compute --config-file /etc/nova/nova-cells.conf | 10:03 |
| bauzas | mriedem: yup, about if it's okay to ask operators about mutliple MQs | 10:04 |
| gibi | so we have the same situation with nova cells v1 | 10:04 |
| mriedem | well, that's kind of the whole idea, separate mq/db per cell | 10:04 |
| mriedem | i believe there is a way to configure for dedicated mq with oslo messaging | 10:05 |
| *** kornicameister has quit IRC | 10:05 | |
| bauzas | mriedem: the problem is that we use the same MQ between the services and the notifcations | 10:05 |
| *** liujiong has quit IRC | 10:05 | |
| mriedem | so people can put notification traffic on a dedicated mq | 10:05 |
| *** artom has joined #openstack-nova | 10:06 | |
| bauzas | mriedem: orly? | 10:07 |
| bauzas | mriedem: if so, we could possibly modify https://review.openstack.org/#/c/436094/35/lib/nova to use the same notification MQ for all cells | 10:07 |
| mriedem | yes rlrossit added that in mitaka or newton | 10:07 |
| mriedem | right | 10:07 |
| mriedem | i thought godaddy was looking at doing dedicated mq too | 10:07 |
| mriedem | and dtp was working on that | 10:07 |
| bauzas | so it's not needing operators discussion then | 10:08 |
| bauzas | just something internal | 10:08 |
| bauzas | about devstack | 10:08 |
| mriedem | i don't know how to set it up though | 10:08 |
| mriedem | https://docs.openstack.org/developer/oslo.messaging/opts.html ? | 10:08 |
| bauzas | mriedem: we could tho possibly add some relnote | 10:08 |
| bauzas | mriedem: saying that if you multi-cell, you need to setup your notifications MQs to not use the internal transport URL | 10:09 |
| gibi | https://docs.openstack.org/developer/oslo.messaging/opts.html#oslo_messaging_notifications.transport_url | 10:09 |
| *** yangyapeng has quit IRC | 10:09 | |
| gibi | there is transport_url in the oslo_messaging_notifications section | 10:09 |
| *** purplerbot has quit IRC | 10:09 | |
| bauzas | gibi: mriedem: perfect | 10:09 |
| mriedem | ah | 10:09 |
| gibi | I guess that is the way to override the default transport_url | 10:09 |
| *** purplerbot has joined #openstack-nova | 10:09 | |
| bauzas | I can shime a patch | 10:09 |
| mriedem | ok so devstack would use a global transport url for all notifications for nova | 10:10 |
| bauzas | the fact is that we're not fully reverted | 10:10 |
| bauzas | mriedem: that's my proposal indeed | 10:10 |
| mriedem | the quotas thing is still busted | 10:10 |
| bauzas | mriedem: what's the problem with quotas ? | 10:10 |
| * bauzas hates June | 10:10 | |
| bauzas | well, /me hates paperwork rather | 10:11 |
| *** lyan has quit IRC | 10:11 | |
| mriedem | https://bugs.launchpad.net/python-novaclient/+bug/1700364 | 10:12 |
| openstack | Launchpad bug 1700364 in python-novaclient "Functional tests are failing with over-quota errors on number of instances" [High,In progress] - Assigned to Matt Riedemann (mriedem) | 10:12 |
| mriedem | bauzas: ^ | 10:12 |
| bauzas | yeah, just saw the gerrit topic | 10:12 |
| mriedem | the devstack change breaks the novaclient functional job | 10:12 |
| mriedem | which is single tenant | 10:12 |
| mriedem | because i think we're using different databases for quota during create and delete | 10:12 |
| mriedem | it wouldn't show up in the normal dsvm gate jobs b/c those are multi-tenant | 10:13 |
| bauzas | I don't see the relationship between the fact that jobs are single-tenant and the fact that we possibly have multiple DBs for quotas | 10:14 |
| bauzas | because of how the job is setup ? | 10:14 |
| bauzas | then fix the job no ? | 10:14 |
| bauzas | mriedem: ^ | 10:14 |
| mriedem | no, that would paper over the issue | 10:15 |
| mriedem | it's actually good that this job caught it | 10:15 |
| bauzas | so, I'm missing something crucial | 10:15 |
| mriedem | http://logs.openstack.org/17/477217/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/0aaa523/console.html#_2017-06-25_15_11_09_706222 | 10:15 |
| mriedem | so i pushed this change to novaclient to debug | 10:16 |
| bauzas | if the job was a canary bird, that's cool, but how can we lead to have different DB layouts depending whether you're multi-tenant ? | 10:16 |
| mriedem | we try to create a server and get an overquota error for instances, | 10:16 |
| mriedem | so i do a nova list | 10:16 |
| mriedem | and nothing comes back | 10:16 |
| mriedem | bauzas: it's not, it's just that multi-tenant jobs we only ever use 1 instance for quota | 10:16 |
| mriedem | whereas with this single tenant job, it's using the same project for all quota usage | 10:16 |
| bauzas | mriedem: FWIW, opening a bug for notifications with multi-cell | 10:17 |
| mriedem | and it's not decrementing on delete properly, | 10:17 |
| mriedem | so we eventually go over quota | 10:17 |
| mriedem | yeah we needed a separate bug for that for sure | 10:17 |
| mriedem | if vitrage didn't already report one | 10:17 |
| *** yamamoto has quit IRC | 10:17 | |
| bauzas | they didn't AFAIK | 10:17 |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Add Cinder v3 detach to swap_volume https://review.openstack.org/456971 | 10:18 |
| mriedem | so here is one thing i think, | 10:21 |
| *** kornicameister has joined #openstack-nova | 10:21 | |
| mriedem | on delete, we get the instance and target the context to the cell that the instance is in http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2427 | 10:21 |
| *** rmart04 has joined #openstack-nova | 10:21 | |
| mriedem | so when we create the quota usage decrement reservation in the api, it's using that context targeted to the cell | 10:21 |
| mriedem | but when we create the server, we don't have a cell context | 10:21 |
| mriedem | so it goes to cell0 | 10:21 |
| mriedem | so i think on create we're doing quota stuff in cell0, and on delete we doing stuff in cell1 | 10:22 |
| bauzas | mriedem: https://bugs.launchpad.net/nova/+bug/1700496 | 10:23 |
| openstack | Launchpad bug 1700496 in OpenStack Compute (nova) "Notifications are emitted per-cell instead of globally" [Undecided,New] | 10:23 |
| bauzas | mriedem: I was thinking of that | 10:23 |
| bauzas | since we target cells | 10:23 |
| mriedem | i think the answer for quotas with cells v2 has been quotas in the api and counting quotas, | 10:23 |
| mriedem | which isn't in yet | 10:23 |
| mriedem | so the fleetify change is probably dependent on that, | 10:23 |
| mriedem | or we need to fix how we do quotas pre-fleetify | 10:24 |
| bauzas | we at least need to make sure that we increment/decrement on the same target | 10:24 |
| mriedem | correct | 10:24 |
| bauzas | I unfortunately need to go afk | 10:24 |
| bauzas | China is what TZ ? | 10:24 |
| mriedem | which i think means we need to temporarily untarget the context when creating the reservation in the api on delete | 10:24 |
| bauzas | mriedem: ^ | 10:25 |
| mriedem | UTC+08:00 | 10:25 |
| bauzas | okay, 6 hours in front of mine | 10:25 |
| bauzas | becoming very late | 10:25 |
| bauzas | mriedem: if you need to disappear, I can catchup with dan | 10:26 |
| * bauzas goes awol | 10:26 | |
| mriedem | i will disappear in a bit | 10:27 |
| mriedem | going to push a wip patch to nova quick | 10:27 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add missing microversion documentation https://review.openstack.org/477477 | 10:27 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: Make API history doc more consistent https://review.openstack.org/477478 | 10:27 |
| sfinucan | mriedem, bauzas: I _think_ the first of those ^^^ was missing for some of melwitt's recent patches | 10:28 |
| sfinucan | If you could take a look at some point | 10:28 |
| mriedem | sfinucan: just did | 10:29 |
| mriedem | comment inline | 10:29 |
| *** lyan has joined #openstack-nova | 10:30 | |
| *** jaosorior has joined #openstack-nova | 10:30 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: Make API history doc more consistent https://review.openstack.org/477478 | 10:30 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add missing microversion documentation https://review.openstack.org/477477 | 10:30 |
| sfinucan | mriedem: done | 10:31 |
| *** lpetrut_ has joined #openstack-nova | 10:32 | |
| mriedem | +2 mofo | 10:32 |
| sfinucan | ta | 10:32 |
| *** artom has quit IRC | 10:34 | |
| *** lpetrut has quit IRC | 10:34 | |
| *** alexey_weyl has quit IRC | 10:36 | |
| openstackgerrit | Pallavi proposed openstack/python-novaclient master: Fixed wrap argument to accept only non-negative values https://review.openstack.org/477483 | 10:37 |
| *** Prashant has quit IRC | 10:37 | |
| *** phuongnh has quit IRC | 10:39 | |
| *** mriedem has quit IRC | 10:41 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807 | 10:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: allow filter providers in tree https://review.openstack.org/377215 | 10:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: set/check if inventory change in tree https://review.openstack.org/470575 | 10:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: integrate ProviderTree to report client https://review.openstack.org/415921 | 10:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: adds ProviderTree for nested resources https://review.openstack.org/415920 | 10:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693 | 10:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: add nested resource providers https://review.openstack.org/377138 | 10:44 |
| *** gszasz has joined #openstack-nova | 10:45 | |
| *** dixiaoli has joined #openstack-nova | 10:46 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: WIP: Automatically revert resize which fails on destination https://review.openstack.org/462521 | 10:49 |
| *** ericyoung has joined #openstack-nova | 10:52 | |
| *** pcaruana has quit IRC | 10:53 | |
| *** gszasz has quit IRC | 10:53 | |
| openstackgerrit | Chason Chan proposed openstack/nova master: import installation guide pages from openstack-manuals https://review.openstack.org/477488 | 10:57 |
| *** yamamoto has joined #openstack-nova | 10:59 | |
| *** slaweq has joined #openstack-nova | 10:59 | |
| *** dixiaoli has quit IRC | 11:01 | |
| *** slaweq has quit IRC | 11:04 | |
| *** gszasz has joined #openstack-nova | 11:05 | |
| *** pcaruana has joined #openstack-nova | 11:05 | |
| *** artom has joined #openstack-nova | 11:10 | |
| *** artom has quit IRC | 11:10 | |
| *** artom has joined #openstack-nova | 11:11 | |
| *** sdague has joined #openstack-nova | 11:12 | |
| *** gcb has quit IRC | 11:15 | |
| *** lpetrut_ has quit IRC | 11:21 | |
| *** ericyoung has quit IRC | 11:22 | |
| *** lpetrut has joined #openstack-nova | 11:24 | |
| *** yangyapeng has joined #openstack-nova | 11:24 | |
| *** smatzek has joined #openstack-nova | 11:25 | |
| *** ericyoung has joined #openstack-nova | 11:25 | |
| *** d9k_ has joined #openstack-nova | 11:26 | |
| *** jaypipes has joined #openstack-nova | 11:26 | |
| *** d9k has quit IRC | 11:28 | |
| openstackgerrit | Chason Chan proposed openstack/nova master: import the admin guide content from openstack-manuals https://review.openstack.org/477497 | 11:28 |
| *** yangyapeng has quit IRC | 11:28 | |
| *** slagle_ is now known as slagle | 11:30 | |
| *** alexchadin has joined #openstack-nova | 11:31 | |
| *** yangyapeng has joined #openstack-nova | 11:34 | |
| *** hshiina has quit IRC | 11:34 | |
| *** yamamoto has quit IRC | 11:37 | |
| *** edmondsw has joined #openstack-nova | 11:40 | |
| *** edmondsw has quit IRC | 11:44 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | 11:45 |
| *** VAhl has quit IRC | 11:55 | |
| *** diga has quit IRC | 11:56 | |
| *** moshele has quit IRC | 11:58 | |
| *** moshele has joined #openstack-nova | 11:58 | |
| *** slaweq has joined #openstack-nova | 12:00 | |
| *** edmondsw has joined #openstack-nova | 12:01 | |
| *** rcernin has quit IRC | 12:02 | |
| *** litao__ has quit IRC | 12:02 | |
| *** edmondsw has quit IRC | 12:03 | |
| *** rcernin has joined #openstack-nova | 12:04 | |
| *** slaweq has quit IRC | 12:04 | |
| *** moshele has quit IRC | 12:06 | |
| *** cleong has joined #openstack-nova | 12:09 | |
| *** moshele has joined #openstack-nova | 12:10 | |
| *** moshele has quit IRC | 12:11 | |
| *** moshele has joined #openstack-nova | 12:11 | |
| *** vladikr_ has joined #openstack-nova | 12:12 | |
| *** yamamoto has joined #openstack-nova | 12:13 | |
| *** jpena is now known as jpena|lunch | 12:14 | |
| *** liverpooler has joined #openstack-nova | 12:15 | |
| *** hoonetorg has quit IRC | 12:16 | |
| *** slaweq has joined #openstack-nova | 12:17 | |
| *** slaweq has quit IRC | 12:17 | |
| *** gbarros has joined #openstack-nova | 12:20 | |
| *** rcernin has quit IRC | 12:21 | |
| *** rcernin has joined #openstack-nova | 12:21 | |
| openstackgerrit | Merged openstack/nova master: Add missing microversion documentation https://review.openstack.org/477477 | 12:27 |
| *** hoonetorg has joined #openstack-nova | 12:28 | |
| *** salv-orlando has joined #openstack-nova | 12:29 | |
| jaypipes | jianghuaw: see comment on https://review.openstack.org/#/c/463649/6. | 12:29 |
| jaypipes | jianghuaw: left you three alternatives. :) give them a thought and let me know what you prefer! | 12:29 |
| *** ralonsoh has quit IRC | 12:30 | |
| *** baoli has joined #openstack-nova | 12:31 | |
| *** sridharg has joined #openstack-nova | 12:33 | |
| *** salv-orlando has quit IRC | 12:33 | |
| *** rfolco has joined #openstack-nova | 12:34 | |
| *** Drankis has joined #openstack-nova | 12:35 | |
| *** baoli has quit IRC | 12:35 | |
| *** moshele has quit IRC | 12:36 | |
| *** mdrabe has joined #openstack-nova | 12:41 | |
| *** jamesdenton has joined #openstack-nova | 12:42 | |
| *** jamesden_ has quit IRC | 12:43 | |
| *** gbarros has quit IRC | 12:43 | |
| *** edmondsw has joined #openstack-nova | 12:43 | |
| *** lpetrut has quit IRC | 12:44 | |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for allocations https://review.openstack.org/470933 | 12:46 |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for RP allocations https://review.openstack.org/450550 | 12:46 |
| *** ygl has joined #openstack-nova | 12:46 | |
| *** moshele has joined #openstack-nova | 12:48 | |
| *** mvk has quit IRC | 12:50 | |
| *** tuanluong has quit IRC | 12:50 | |
| *** artom has quit IRC | 12:52 | |
| *** artom has joined #openstack-nova | 12:52 | |
| *** sridharg has quit IRC | 12:55 | |
| *** edmondsw_ has joined #openstack-nova | 12:56 | |
| *** edmondsw_ has quit IRC | 12:59 | |
| *** coreywright has quit IRC | 13:03 | |
| *** jdillaman has joined #openstack-nova | 13:03 | |
| *** gbarros has joined #openstack-nova | 13:06 | |
| *** lpetrut has joined #openstack-nova | 13:06 | |
| *** acormier has joined #openstack-nova | 13:08 | |
| *** cNilesh has quit IRC | 13:11 | |
| *** cNilesh has joined #openstack-nova | 13:11 | |
| *** lucasxu has joined #openstack-nova | 13:13 | |
| *** gbarros has quit IRC | 13:14 | |
| *** gouthamr has joined #openstack-nova | 13:17 | |
| *** coreywright has joined #openstack-nova | 13:17 | |
| *** slaweq has joined #openstack-nova | 13:18 | |
| *** jheroux has joined #openstack-nova | 13:19 | |
| *** mvk has joined #openstack-nova | 13:20 | |
| *** ralonsoh has joined #openstack-nova | 13:21 | |
| *** slaweq has quit IRC | 13:22 | |
| *** jpena|lunch is now known as jpena | 13:24 | |
| *** MVenesio has joined #openstack-nova | 13:25 | |
| *** gouthamr has quit IRC | 13:26 | |
| *** esberglu has joined #openstack-nova | 13:26 | |
| ildikov | sfinucan: hi :) | 13:27 |
| *** cNilesh has quit IRC | 13:27 | |
| *** pchavva has joined #openstack-nova | 13:28 | |
| *** felipemonteiro has joined #openstack-nova | 13:28 | |
| ildikov | sfinucan: IIRC you reviewed some Cinder attach/detach related patches not that long ago | 13:28 |
| ildikov | sfinucan: and I was hoping you could take a look at these two patches: https://review.openstack.org/#/c/456971/ https://review.openstack.org/#/c/463987/ | 13:29 |
| *** salv-orlando has joined #openstack-nova | 13:30 | |
| *** felipemonteiro_ has joined #openstack-nova | 13:30 | |
| *** edmondsw_ has joined #openstack-nova | 13:31 | |
| *** thorst has joined #openstack-nova | 13:31 | |
| *** edmondsw_ has quit IRC | 13:32 | |
| *** eharney has joined #openstack-nova | 13:33 | |
| *** nolwenn has joined #openstack-nova | 13:33 | |
| *** cNilesh has joined #openstack-nova | 13:33 | |
| *** nolwenn has quit IRC | 13:33 | |
| sfinucan | ildikov: I'm working my way through nested-resource-provider patches atm, but will take a look at those after :) | 13:34 |
| *** salv-orlando has quit IRC | 13:34 | |
| *** felipemonteiro has quit IRC | 13:34 | |
| ildikov | sfinucan: thanks much! :) | 13:34 |
| *** felipemonteiro_ has quit IRC | 13:37 | |
| *** kylek3h has joined #openstack-nova | 13:37 | |
| *** cNilesh has quit IRC | 13:38 | |
| *** felipemonteiro has joined #openstack-nova | 13:38 | |
| *** diga has joined #openstack-nova | 13:38 | |
| *** gbarros has joined #openstack-nova | 13:39 | |
| *** winston-d_ has quit IRC | 13:39 | |
| *** ygl has quit IRC | 13:39 | |
| *** ijw has joined #openstack-nova | 13:41 | |
| *** gcb has joined #openstack-nova | 13:42 | |
| *** xyang1 has joined #openstack-nova | 13:43 | |
| *** gbarros has quit IRC | 13:45 | |
| *** ckopper has quit IRC | 13:46 | |
| jgriffith | anybody by chance familiar with what is needed for a multi-node devstack to address this: | 13:47 |
| jgriffith | No host-to-cell mapping found for selected host os-2. Setup is incomplete.: HostMappingNotFound: Host 'os-2' is not mapped to any cell | 13:47 |
| jgriffith | as soon as I add a second compute node I'm toast. The multi-node gate jobs appear to be running, so I can only assume there's an extra step they perform in their setup somewhere? | 13:48 |
| *** lyan has quit IRC | 13:48 | |
| *** acormier has quit IRC | 13:49 | |
| *** acormier has joined #openstack-nova | 13:49 | |
| superdan | jgriffith: yeah, you need to run 'nova-manage cell_v2 discover_hosts --verbose' | 13:50 |
| superdan | jgriffith: the multi-host gate job has an extra hook it runs after setup, under tools/ in devstack | 13:50 |
| *** superdan is now known as dansmith | 13:50 | |
| jgriffith | well that was easy :) | 13:50 |
| jgriffith | thanks sulo | 13:51 |
| jgriffith | dansmith | 13:51 |
| jgriffith | or sudo.. whichever you prefer | 13:51 |
| dansmith | np | 13:51 |
| dansmith | heh | 13:51 |
| jgriffith | oh damn.. spoke too soon... let me see why it failed | 13:51 |
| dansmith | jgriffith: did that spit out "creating a mapping for .. " ? | 13:52 |
| jgriffith | yes | 13:52 |
| dansmith | cool | 13:52 |
| bauzas | dansmith: not sure you got the whole story about https://review.openstack.org/#/c/477436/ | 13:53 |
| edleafe | Scheduler subteam meetings in 7 minutes in #openstack-meeting-alt | 13:53 |
| bauzas | dansmith: given our lovely PTL is on China this week, let me try to summary this morning discussion | 13:53 |
| dansmith | bauzas: ah, nope | 13:53 |
| bauzas | dansmith: so we have 2 problems with the mergez | 13:53 |
| bauzas | oh heh, jeez | 13:53 |
| bauzas | merguez | 13:53 |
| bauzas | 1/ is https://bugs.launchpad.net/python-novaclient/+bug/1700364 | 13:54 |
| openstack | Launchpad bug 1700364 in python-novaclient "Functional tests are failing with over-quota errors on number of instances" [High,In progress] - Assigned to Matt Riedemann (mriedem) | 13:54 |
| bauzas | 2/ is https://bugs.launchpad.net/nova/+bug/1700496 | 13:54 |
| openstack | Launchpad bug 1700496 in OpenStack Compute (nova) "Notifications are emitted per-cell instead of globally" [Undecided,Confirmed] | 13:54 |
| bauzas | #2 is easily fixable | 13:54 |
| bauzas | with your change | 13:54 |
| bauzas | by just using the API MQ | 13:54 |
| bauzas | for the notifications | 13:54 |
| dansmith | not surprised about the quota issue, yeah | 13:54 |
| *** smatzek has quit IRC | 13:54 | |
| bauzas | #1 needs some look | 13:54 |
| bauzas | I tried to look at the issue | 13:55 |
| dansmith | we just need to get the counting quotas stuff in | 13:55 |
| dansmith | sdague: see that revert? | 13:55 |
| bauzas | there is a concurrent one AFAIK | 13:55 |
| bauzas | dansmith: I wonder if the root case for #1 could just be fixed if we would be sure that we call the same cell target between create and delete | 13:56 |
| dansmith | bauzas: yeah it's probably another quota leak bug, but ... counting quotas will fix it all anyway | 13:57 |
| bauzas | dansmith: which change should I review then ? | 13:57 |
| dansmith | https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/cells-count-resources-to-check-quota-in-api | 13:57 |
| bauzas | dansmith: sorry, last week was very difficult to follow, and this week too again :( | 13:57 |
| *** links has quit IRC | 13:58 | |
| bauzas | dansmith: like I said to matt, I'll be *again* on PTO tomorrow and Wed | 13:58 |
| dansmith | jeez | 13:58 |
| bauzas | yup, catched up by the HRs... | 13:58 |
| bauzas | dansmith: so the problem is that we have a long series with https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/cells-count-resources-to-check-quota-in-api | 13:59 |
| *** esberglu has quit IRC | 13:59 | |
| bauzas | dansmith: would we really need to wait for the above to be all merged before adding again the devstack modification ? | 14:00 |
| *** esberglu has joined #openstack-nova | 14:00 | |
| *** mriedem has joined #openstack-nova | 14:00 | |
| dansmith | bauzas: that is about 10% of what the series was, | 14:00 |
| dansmith | and since we're probably failing on instance quota, we need at least that one to merge I would think | 14:01 |
| bauzas | okay | 14:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Untarget context when creating quota reservation on delete https://review.openstack.org/477548 | 14:02 |
| mriedem | is it all figured out already? | 14:02 |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: DNM: debug overquota failures https://review.openstack.org/477323 | 14:04 |
| *** esberglu has quit IRC | 14:04 | |
| dansmith | mriedem: is what figured out? | 14:04 |
| dansmith | mriedem: I assume we should just revert the devstack patch, get the counting quotas stuff in, and then take another go at it, right? | 14:04 |
| *** edmondsw_ has joined #openstack-nova | 14:05 | |
| *** salv-orlando has joined #openstack-nova | 14:05 | |
| sdague | dansmith: I did | 14:05 |
| mriedem | dansmith: ok that's what i was thinking earlier today | 14:06 |
| mriedem | since this is all dependent on quotas in the api | 14:06 |
| dansmith | yeah | 14:06 |
| dansmith | makes more sense to just get those in than try to fix whatever is wrong currently I think | 14:06 |
| bauzas | okay, so I have homework | 14:06 |
| mriedem | so the revert is approved? | 14:07 |
| dansmith | the only other option would be a hack to set quota driver to noop on affected jobs | 14:07 |
| dansmith | but there's still the notification issue | 14:07 |
| *** gbarros has joined #openstack-nova | 14:07 | |
| bauzas | mriedem: could we maybe depend-on the python functional test change on the top of the quota series so we're fully sure it fixes the problem | 14:07 |
| bauzas | ? | 14:07 |
| bauzas | dansmith: the notification issue is easy to fix | 14:07 |
| mriedem | bauzas: we could try that yeah | 14:08 |
| *** ckopper has joined #openstack-nova | 14:08 | |
| dansmith | I haven't even looked at the notification thing, but I assume it's just a matter of them listening to the cell bus right? | 14:08 |
| bauzas | dansmith: I can just prep a patch, but it would honestly rely on your own change | 14:08 |
| mriedem | dansmith: we can set a global transport_url for notifications | 14:08 |
| dansmith | oh I see | 14:08 |
| dansmith | right, I forgot they have their own config for that | 14:08 |
| bauzas | dansmith: it's just about setting a global transport url for all notifications in devstack | 14:08 |
| mriedem | so inter-service rpc is same as before, notifications get their own notification queue | 14:08 |
| *** lpetrut has quit IRC | 14:08 | |
| bauzas | and we could arbitrarly use the API transport URL for it | 14:08 |
| dansmith | bauzas: I'm not sure what "my own change" you're talking about | 14:08 |
| *** edmondsw_ has quit IRC | 14:09 | |
| mriedem | so there are 2 reverts i know of https://review.openstack.org/#/q/f63aa021cf996b500b6a570c0114c490bb57461a,n,z | 14:09 |
| mriedem | are there more? | 14:09 |
| *** felipemonteiro_ has joined #openstack-nova | 14:09 | |
| mriedem | let me push a novaclient patch that depends on the tip of the quotas series too | 14:09 |
| sdague | oh, I thought the revert already landed | 14:09 |
| sdague | sending in now | 14:10 |
| bauzas | dansmith: I'd just amend https://review.openstack.org/#/c/436094/ by setting the notification transport URL to be the API one in https://review.openstack.org/#/c/436094/35/lib/nova@637-ish | 14:10 |
| dansmith | mriedem: I don't think we need to revert it twice :) | 14:10 |
| dansmith | bauzas: oh I see, yeah, I'll re-propose that and throw that in | 14:10 |
| sdague | dansmith: revert with vengence! | 14:10 |
| dansmith | sdague: apparently | 14:10 |
| mriedem | dansmith: i just hadn't seen the other before i reverted for testing | 14:10 |
| sdague | mriedem: I just approved yours | 14:11 |
| dansmith | ack | 14:11 |
| bauzas | okay, looks like we have a plan | 14:12 |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: DNM testing counting quotas https://review.openstack.org/477551 | 14:12 |
| mriedem | ^ is the counting quotas test | 14:12 |
| bauzas | I actually love it when a plan comes together | 14:12 |
| *** felipemonteiro has quit IRC | 14:12 | |
| *** smatzek has joined #openstack-nova | 14:14 | |
| *** tidwellr has joined #openstack-nova | 14:16 | |
| *** nolwenn has joined #openstack-nova | 14:18 | |
| *** cNilesh has joined #openstack-nova | 14:18 | |
| *** slaweq has joined #openstack-nova | 14:19 | |
| *** gbarros has quit IRC | 14:19 | |
| dansmith | mriedem: sdague: is it kosher to re-use the change-id for the re-patch? | 14:21 |
| dansmith | so that it's easy to find the original? | 14:21 |
| mriedem | i don't think you can do that on the same branch | 14:22 |
| mriedem | for the same repo | 14:22 |
| mriedem | use the same topic branch? | 14:22 |
| dansmith | oh okay | 14:22 |
| jianghuaw | jaypipes, thanks for the advice on the resolution traits. Yeah, the third one(w640xH480) looks perfect to me:-) Just as you said, it explicitly signifies width and height. It also makes the resolutions be organized in a consistent way like others modules. | 14:22 |
| *** alexchadin has quit IRC | 14:22 | |
| dansmith | mriedem: bauzas what's the actual conf variable needed for the notification mq thing? | 14:22 |
| *** burt has joined #openstack-nova | 14:23 | |
| *** slaweq has quit IRC | 14:23 | |
| bauzas | dansmith: see the bug report | 14:23 |
| jaypipes | jianghuaw: cool! push up a revision and I'm good to +2 that :) | 14:23 |
| bauzas | dansmith: but I'll find you, sec | 14:23 |
| bauzas | dansmith: https://docs.openstack.org/developer/oslo.messaging/opts.html#oslo_messaging_notifications.transport_url | 14:23 |
| dansmith | oh I didn't see a bug for the notifications one | 14:23 |
| jianghuaw | jaypipes, the reworked patch is at here: https://review.openstack.org/#/c/463649/ | 14:23 |
| jianghuaw | thanks. | 14:23 |
| bauzas | dansmith: https://bugs.launchpad.net/nova/+bug/1700496 | 14:23 |
| openstack | Launchpad bug 1700496 in OpenStack Compute (nova) "Notifications are emitted per-cell instead of globally" [Undecided,Confirmed] | 14:23 |
| jaypipes | awesome, thanks jianghuaw :) | 14:24 |
| *** hongbin has joined #openstack-nova | 14:24 | |
| dansmith | bauzas: yeah, I know transport_url, but under what group? | 14:24 |
| dansmith | [oslo_messaging_notifications] ? | 14:24 |
| bauzas | dansmith: that's an oslo.messaging opt | 14:24 |
| dansmith | I realize that | 14:25 |
| bauzas | dansmith: which is [DEFAULT] AFAICS | 14:25 |
| jianghuaw | jaypipes, thanks for the help:-) | 14:25 |
| bauzas | hmmpfff | 14:25 |
| bauzas | I realize you | 14:25 |
| dansmith | bauzas: no, transport_url under default is used for regular rpc | 14:25 |
| bauzas | transport_url is very meaningless | 14:25 |
| jaypipes | jianghuaw: glad we could come to an alternative that we both liked :) | 14:25 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: move resources_from_request_spec() to utils https://review.openstack.org/476629 | 14:26 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/476631 | 14:26 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Add uuid to scheduler's HostState object https://review.openstack.org/476630 | 14:26 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: return 400 Bad Request when empty string resources https://review.openstack.org/477019 | 14:26 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: support GET /allocation_candidates https://review.openstack.org/475448 | 14:26 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: WIP - placement: claim resources in scheduler https://review.openstack.org/476632 | 14:26 |
| dansmith | gibi: around? | 14:26 |
| jianghuaw | jaypipes, :-) | 14:26 |
| bauzas | dansmith: I should actually try to read the coee | 14:26 |
| jaypipes | edleafe, bauzas, dansmith: ^^ rebased placement-allocation-requests. | 14:26 |
| bauzas | jaypipes: excellent. | 14:26 |
| bauzas | everything I can do today is the best to me | 14:26 |
| bauzas | dansmith: https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/notifier.py#L41 | 14:28 |
| *** tbachman has joined #openstack-nova | 14:28 | |
| mriedem | alex_xu: my bag is now officially in shanghai airport \o/ | 14:28 |
| bauzas | dansmith: and https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/opts.py#L51 | 14:28 |
| dansmith | so what I said above | 14:28 |
| bauzas | dansmith: so, yeah the group seems to be 'oslo_messaging_notifications' | 14:29 |
| *** MikeG451 has quit IRC | 14:29 | |
| bauzas | dansmith: yup | 14:29 |
| mriedem | yes conf.oslo_messaging_notifications.transport_url | 14:29 |
| *** ijw has quit IRC | 14:29 | |
| *** syjulian has quit IRC | 14:30 | |
| jaypipes | jianghuaw: +2 from me on both GPU traits patches. | 14:30 |
| *** ijw has joined #openstack-nova | 14:30 | |
| bauzas | mriedem: dansmith: note that by default, Nova will emit notifications within each cell MQ | 14:30 |
| bauzas | we only fix that in devstack | 14:30 |
| jianghuaw | jaypipes, thanks very much:-) | 14:30 |
| *** yangyape_ has joined #openstack-nova | 14:30 | |
| bauzas | so I wonder if we should communicate that somehow | 14:30 |
| bauzas | and with which manner? | 14:31 |
| dansmith | only if you use multiple cells | 14:31 |
| bauzas | of course | 14:31 |
| dansmith | and only if you take the default, | 14:31 |
| bauzas | only if you specify distinct transport URLs even | 14:31 |
| dansmith | which is pretty obvious if you're setting it up and you never give some services anything but a cell transport | 14:31 |
| *** tbachman has quit IRC | 14:31 | |
| *** gbarros has joined #openstack-nova | 14:31 | |
| bauzas | sure, but I do wonder if some operators deploying a second cell wouldn't be becoming crazy if they see a drop in notifications | 14:32 |
| *** markmcclain has quit IRC | 14:32 | |
| *** huanxie has quit IRC | 14:32 | |
| dansmith | I doubt they would become crazy | 14:32 |
| *** ijw has quit IRC | 14:32 | |
| alex_xu | mriedem: congrats :) | 14:32 |
| mriedem | there are going to be a number of things we'll need to have in the docs/release notes for multi-cell, | 14:32 |
| bauzas | just because they forgot to specify either to emit to the same transport URL, or to modify their clients to subscribe to the new MQs | 14:32 |
| mriedem | there was something i was just saying to dansmith the other day we'd have to put in the release notes for multi-cell, | 14:32 |
| *** esberglu has joined #openstack-nova | 14:32 | |
| mriedem | but now it escapes me what that was | 14:32 |
| *** ijw has joined #openstack-nova | 14:32 | |
| *** yangyapeng has quit IRC | 14:33 | |
| bauzas | dansmith: well, operators who bill on notifications would surely become crazy if they forgot to bill for some usage :) | 14:33 |
| *** markmcclain has joined #openstack-nova | 14:33 | |
| dansmith | mriedem: this seems like a doc thing to me, not a reno thing.. renos for this kind of thing just makes it harder to read and digest that enormous list of things | 14:33 |
| mriedem | dansmith: agree | 14:33 |
| bauzas | dansmith: agree | 14:33 |
| mriedem | so we should start a "multi-cell considerations" section of the cells page probably | 14:33 |
| mriedem | notifications are 1, if only i could remember what the other thing was from last week | 14:34 |
| bauzas | but that's one of the many gotchas an operator can encouter if they misconfigure their cloud | 14:34 |
| dansmith | bauzas: you're also supposing that they _want_ them to be unified | 14:34 |
| bauzas | dansmith: that's correct | 14:34 |
| dansmith | I would kinda hope people with multiple cells are sharding the billing bit too | 14:34 |
| bauzas | dansmith: that's an operator concern | 14:34 |
| bauzas | dansmith: but I just want them to know that if they want to start a new cell, they have extra things to decide :) | 14:35 |
| bauzas | where notifications are emitted is one thing | 14:35 |
| bauzas | I leave mriedem descibe the other thing | 14:35 |
| gibi | dansmith: I'm here. Agreed with the notification specific transport_url stuff in the morning | 14:35 |
| *** READ10 has joined #openstack-nova | 14:35 | |
| dansmith | gibi: I was looking for confirmation on the actual conf value to use, but I think we've got it | 14:35 |
| bauzas | that's not because devstack sends all notifications on the same transport URL that operators all need to do the same thing of course | 14:36 |
| gibi | dansmith: yeah, what you got seems OK to me | 14:36 |
| dansmith | gibi: this is the re-do of that patch, which that value set if you want to test: https://review.openstack.org/#/c/477556/ | 14:36 |
| gibi | dansmith: checking... | 14:36 |
| gibi | dansmith: so does $(get_transport_url) means the API MQ in https://review.openstack.org/#/c/477556/1/lib/nova@585 ? | 14:38 |
| dansmith | gibi: yeah, with no vhost it should return the base one, which is what we're using at the top | 14:39 |
| gibi | dansmith: then it is OK for me. Also it will be good for vitrage tempest based on my knowledge | 14:40 |
| dansmith | gibi: do you want to change your vitrage canary patch to point to this for testing? | 14:40 |
| gibi | dansmith: let's do that | 14:40 |
| *** yamahata has joined #openstack-nova | 14:41 | |
| gibi | dansmith: done https://review.openstack.org/#/c/477455/ | 14:42 |
| dansmith | cool | 14:43 |
| *** moshele has quit IRC | 14:43 | |
| *** chyka has joined #openstack-nova | 14:43 | |
| *** cfriesen_ has joined #openstack-nova | 14:47 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807 | 14:54 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693 | 14:54 |
| *** gbarros has quit IRC | 14:57 | |
| *** felipemonteiro_ has quit IRC | 14:57 | |
| *** gbarros has joined #openstack-nova | 14:58 | |
| *** mriedem has quit IRC | 14:58 | |
| mordred | efried: heya - heads up the keystoneauth patch series has everything you need now except for one thing | 15:00 |
| mordred | efried: https://review.openstack.org/#/q/topic:version-discovery | 15:00 |
| efried | mordred What's the one thing? | 15:00 |
| efried | (before I go try to winnow it out) | 15:00 |
| mordred | efried: I still need to add service-type alias support - but interface lists are there, as are all of the version bits - and it properly returns microversion info as needed | 15:00 |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for traits https://review.openstack.org/474186 | 15:01 |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for allocations https://review.openstack.org/470933 | 15:01 |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for RP usages https://review.openstack.org/450105 | 15:01 |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for RP allocations https://review.openstack.org/450550 | 15:01 |
| efried | mordred Roger wilco. Thanks! | 15:01 |
| mordred | efried: I'll get the service-type alias stuff done today or tomorrow | 15:01 |
| efried | nice | 15:01 |
| *** Drankis has quit IRC | 15:01 | |
| *** rcernin has quit IRC | 15:04 | |
| *** mriedem has joined #openstack-nova | 15:04 | |
| *** tuanla_fujitsu has joined #openstack-nova | 15:05 | |
| *** tuanla__ has joined #openstack-nova | 15:05 | |
| avolkov | jaypipes: Hey Jay | 15:06 |
| mriedem | looks like the novaclient functional test is passing in the change that depends on the counting quotas series | 15:06 |
| avolkov | quick question about placement-api-ref POST for inventories | 15:06 |
| mriedem | dansmith: so we should probably make that a dependency for the devstack fleetify change too | 15:06 |
| avolkov | is it kind of deprecated? should we make api reference for it? | 15:06 |
| dansmith | mriedem: which? the functional change or the top of the quotas set? | 15:07 |
| mriedem | i was just dependent on the top of quotas set | 15:07 |
| * dansmith doesn't know what the former is | 15:07 | |
| dansmith | is it an actual change or a canary? | 15:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807 | 15:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693 | 15:07 |
| mriedem | https://review.openstack.org/#/c/477551/ | 15:07 |
| mriedem | ^ was the test | 15:07 |
| dansmith | gotcha | 15:07 |
| mriedem | i'm saying make the devstack re-fleetify depend on https://review.openstack.org/#/c/410946/ also | 15:07 |
| dansmith | mriedem: I want to see the logs from the current fleetify run but I will push up a dependency when it's done | 15:08 |
| mriedem | ok | 15:08 |
| mriedem | with that i'm signing off | 15:08 |
| *** mriedem has quit IRC | 15:08 | |
| dansmith | o/ | 15:08 |
| jaypipes | avolkov: on call for another 40 mnutes.... | 15:10 |
| *** rmart04 has quit IRC | 15:11 | |
| sfinucan | jaypipes: I've reviewed the bp/nested-resource-providers series. 50-50 split between +2 and "needs more info/other comments addressed" | 15:15 |
| sfinucan | if you take a look at the latter, ping me and I'll re-review them | 15:15 |
| *** ijw has quit IRC | 15:16 | |
| *** ijw has joined #openstack-nova | 15:16 | |
| *** marst has joined #openstack-nova | 15:17 | |
| jaypipes | sfinucan: no worries, and thank you for the reviews! | 15:17 |
| sfinucan | No problemo. Onto artom's patches | 15:17 |
| *** links has joined #openstack-nova | 15:17 | |
| *** slaweq has joined #openstack-nova | 15:19 | |
| *** aarefiev is now known as aarefiev_afk | 15:20 | |
| *** yamamoto has quit IRC | 15:21 | |
| artom | sfinucan, on hey | 15:21 |
| artom | Thanks :) | 15:21 |
| *** yamahata has quit IRC | 15:21 | |
| *** zz9pzza has quit IRC | 15:24 | |
| *** slaweq has quit IRC | 15:24 | |
| *** zz9pzza has joined #openstack-nova | 15:24 | |
| *** yamamoto has joined #openstack-nova | 15:24 | |
| *** links has quit IRC | 15:25 | |
| *** felipemonteiro has joined #openstack-nova | 15:25 | |
| *** felipemonteiro_ has joined #openstack-nova | 15:27 | |
| *** zz9pzza has quit IRC | 15:28 | |
| *** yamamoto has quit IRC | 15:29 | |
| *** felipemonteiro has quit IRC | 15:30 | |
| *** lyan has joined #openstack-nova | 15:31 | |
| *** cNilesh has quit IRC | 15:32 | |
| gibi | dansmith: the vitrage tempest still fails but it is only the quota problem not the notification problem http://logs.openstack.org/55/477455/2/check/gate-vitrage-dsvm-api-ubuntu-xenial/60b8e32/console.html | 15:32 |
| gibi | dansmith: so I think your revorked devstack patch is OK from notification perspective | 15:33 |
| *** ijw has quit IRC | 15:34 | |
| *** andreas_s has quit IRC | 15:35 | |
| *** Oku_OS is now known as Oku_OS-away | 15:37 | |
| *** gcb has quit IRC | 15:41 | |
| bauzas | gibi: you could depend-on both | 15:42 |
| bauzas | that would be ideal for testing | 15:42 |
| *** mkucia has quit IRC | 15:46 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add abstract OVSDB API https://review.openstack.org/476612 | 15:48 |
| dansmith | gibi: thanks | 15:48 |
| *** damien_r has quit IRC | 15:48 | |
| *** gbarros has quit IRC | 15:50 | |
| *** markus_z has quit IRC | 15:50 | |
| openstackgerrit | Chris Friesen proposed openstack/python-novaclient master: 2.47: Show flavor info in server details https://review.openstack.org/435141 | 15:50 |
| ericyoung | mdbooth: I think ftersin was looking for you last week, to review https://review.openstack.org/#/c/407440/ | 15:52 |
| *** nic1 has joined #openstack-nova | 15:54 | |
| *** zz9pzza has joined #openstack-nova | 15:57 | |
| *** sridharg has joined #openstack-nova | 15:58 | |
| cfriesen_ | mriedem: andreykurilin: new version of novaclient 2.47 patch is up. ^ Just as an FYI, I'm off for two weeks starting next Monday. | 15:58 |
| *** annegentle has joined #openstack-nova | 15:58 | |
| *** gszasz has quit IRC | 16:03 | |
| *** mvk has quit IRC | 16:03 | |
| *** pcaruana has quit IRC | 16:03 | |
| *** zz9pzza has quit IRC | 16:03 | |
| *** lpetrut has joined #openstack-nova | 16:04 | |
| cfriesen_ | incidentally, we tripped over something interesting. In a block live migration we don't have any logs for the initial disk copy. If the instance is dirtying disk blocks faster than the migration bandwidth it'll simply time out with no obvious reason. | 16:05 |
| *** amotoki is now known as amotoki_away | 16:05 | |
| cfriesen_ | the existing logs in nova-compute don't show the disk migration data | 16:05 |
| *** ijw has joined #openstack-nova | 16:06 | |
| *** rajathagasthya has joined #openstack-nova | 16:06 | |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP/POC Implement new attach Cinder flow https://review.openstack.org/330285 | 16:07 |
| *** sridharg has quit IRC | 16:07 | |
| *** ijw has quit IRC | 16:08 | |
| *** lucasxu has quit IRC | 16:08 | |
| *** Guest41957 has left #openstack-nova | 16:08 | |
| *** Apoorva has joined #openstack-nova | 16:10 | |
| *** gyee has joined #openstack-nova | 16:11 | |
| *** gbarros has joined #openstack-nova | 16:12 | |
| *** Redcavalier has joined #openstack-nova | 16:12 | |
| *** blaisebool has joined #openstack-nova | 16:14 | |
| *** lyan has quit IRC | 16:14 | |
| *** pcaruana has joined #openstack-nova | 16:14 | |
| *** moshele has joined #openstack-nova | 16:14 | |
| *** gszasz has joined #openstack-nova | 16:15 | |
| *** mvk has joined #openstack-nova | 16:16 | |
| Redcavalier | Hi, there is a bug that is causing me quite a lot of problem right now and I was wondering if I could have somebody take a look at it. Bug is 1699228 in Nova. | 16:16 |
| *** slaweq has joined #openstack-nova | 16:20 | |
| *** diga has quit IRC | 16:21 | |
| *** mvk has quit IRC | 16:21 | |
| *** jpena is now known as jpena|off | 16:24 | |
| *** slaweq has quit IRC | 16:25 | |
| *** yamamoto has joined #openstack-nova | 16:27 | |
| sfinucan | artom: I got one done, but I'll work through the rest tomorrow | 16:28 |
| artom | sfinucan, no worries | 16:29 |
| sfinucan | Also, ildikov, I looked at those patches but it's code I'm not that familiar with, so it'll take me a while to ramp up on it. Just FYI | 16:29 |
| artom | sfinucan, thanks again for your commitment to that series | 16:29 |
| *** yamamoto has quit IRC | 16:32 | |
| cfriesen_ | Redcavalier: can you use virtio-blk for now? | 16:32 |
| ildikov | sfinucan: sure, it's ok, thanks for looking into it | 16:32 |
| ildikov | sfinucan: feel free to ping me if you have any questions | 16:32 |
| ildikov | sfinucan: I'm not an expert either, but started to have some ideas about that code part :) | 16:33 |
| Redcavalier | cfriesen_, if I use virtio-blk, I can't use iscsi discard, which is causing me quite a lot of problem regarding SAN space usage. That's the main reason I'm switching to virtio-scsi. | 16:33 |
| *** slaweq has joined #openstack-nova | 16:35 | |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP/POC Implement new attach Cinder flow https://review.openstack.org/330285 | 16:36 |
| Redcavalier | cfriesen_, virtio-blk would be perfect, if it didn't force me to end up with 2TB block devices that actually have only 50 GB of actual data on them, with no way to trim the garbage. | 16:37 |
| *** slaweq has quit IRC | 16:37 | |
| *** slaweq has joined #openstack-nova | 16:39 | |
| jaypipes | cfriesen_: yet more BFV woes. :( | 16:39 |
| cfriesen_ | Redcavalier: seems like a valid issue to me. There were patches proposed to add discard support to virtio-blk to linux, but it looks like they didn't get merged. | 16:41 |
| jaypipes | Redcavalier: can you do a normal non-boot-from-volume instance and just attach a volume to it? | 16:42 |
| jaypipes | cfriesen_: what does "'__com.redhat_drive_add'" have to do with anything? | 16:42 |
| jaypipes | cfriesen_: is that a Ceph-specific thing? | 16:43 |
| * jaypipes trying to understand how this is a bug in Nova. | 16:43 | |
| Redcavalier | jaypipes, I don'T have ephemeral space for that | 16:43 |
| jaypipes | Redcavalier: ? a tiny boot image and then just attach a normal volume to it. | 16:43 |
| ftersin | mriedem: Hi. Since mdbooth cannot work on ScaleIO review for some reason, could someone else review that? | 16:44 |
| cfriesen_ | jaypipes: is it possible this is a bug in RHEL libvirt? or is nova passing an invalid argument to libvirt | 16:45 |
| jaypipes | cfriesen_: I don't see how this is a bug in Nova. | 16:45 |
| Redcavalier | jaypipes, I could try, I haven't built our setup with that in mind. Also, regarding the bug itself, it seems to be an issue with the way libvirt interprets the data it receives from Nova. | 16:45 |
| *** ircuser-1 has joined #openstack-nova | 16:46 | |
| *** slaweq has quit IRC | 16:46 | |
| jaypipes | Redcavalier: I'm looking into it now... | 16:46 |
| *** dtp has joined #openstack-nova | 16:46 | |
| Redcavalier | jaypipes, mohamed naser from vexxhost seems to think that it has something to do with the first block device being sda and the additional block device being identified as vda. | 16:47 |
| jaypipes | Redcavalier: I don't think so. It's complaining about the SCSI device identifier (drive-scsi0-0-0-0) not the device name. | 16:48 |
| Redcavalier | jaypipes, alright, thx for having a look, it's been causing me headaches for a while. | 16:49 |
| jaypipes | Redcavalier: I'm still looking :) | 16:49 |
| jaypipes | Redcavalier: this is Newton, yes? | 16:51 |
| cfriesen_ | jaypipes: I wonder if nova is supposed to be specifying bus/target/unit to libvirt when using virtio-scsi | 16:51 |
| Redcavalier | jaypipes, yes, but it was reproduced in mitaka too. | 16:52 |
| jaypipes | k | 16:52 |
| *** gbarros has quit IRC | 16:52 | |
| jaypipes | cfriesen_: Nova is specifying those things. Are you asking whether Nova should *not* be? | 16:52 |
| cfriesen_ | jaypipes: in that case are we specifying duplicated values? I'm assuming the zeros in "drive-scsi0-0-0-0" in the bug are controller/bus/target/unit. | 16:54 |
| *** tobasco has quit IRC | 16:54 | |
| jaypipes | cfriesen_: could very well be... looking at the code now :) | 16:55 |
| *** annegentle has quit IRC | 16:55 | |
| jaypipes | cfriesen_: I'm just trying no to throw up too much in my mouth while reading this code. | 16:56 |
| cfriesen_ | jaypipes: always a good plan | 16:56 |
| *** jcromer has quit IRC | 16:56 | |
| jaypipes | cfriesen_: so the libvirt driver is "constructing" the device name by doing this: | 16:57 |
| jaypipes | disk_dev = mountpoint.rpartition("/")[2] | 16:57 |
| jaypipes | cfriesen_: and then passing that straight into the volume connector. | 16:57 |
| *** gszasz has quit IRC | 16:58 | |
| *** derekh has quit IRC | 16:59 | |
| jaypipes | cfriesen_: hold up... no, that's not right. | 16:59 |
| cfriesen_ | jaypipes: there's that call to blockinfo.get_info_from_bdm() | 17:00 |
| jaypipes | cfriesen_: yup, that's precisely what I'm looking at now. | 17:00 |
| *** slaweq has joined #openstack-nova | 17:01 | |
| jaypipes | ffs. nova/block_device.py != nova/virt/block_device.py | 17:01 |
| cfriesen_ | jaypipes: I'm not seeing anything obvious setting controller/bus/target/unit of LibvirtConfigGuestDeviceAddressDrive | 17:01 |
| *** pcaruana has quit IRC | 17:03 | |
| *** slaweq has quit IRC | 17:05 | |
| jaypipes | cfriesen_: doesn't help that there's no logging at all in any of this crap code. :( | 17:09 |
| openstackgerrit | Vladyslav Drok proposed openstack/nova master: neutron: re-raise 401 for a non-admin client https://review.openstack.org/422696 | 17:10 |
| openstackgerrit | Spencer Yu proposed openstack/python-novaclient master: Nova client should retry with Retry-After value https://review.openstack.org/447766 | 17:10 |
| jaypipes | cfriesen_: I was going to ask Redcavalier to enable DEBUG logs until I realized there's virtually no logging at all in any of this code :( | 17:11 |
| jaypipes | there IS one debug message that might be useful, though... in guest.py:attach_device. | 17:12 |
| jaypipes | Redcavalier: any way you can enable DEBUG logging and re-run your scenario? | 17:13 |
| Redcavalier | jaypipes, we (openstack canada guys and I) did some troubleshooting on the code to have a clearer picture. Let me see if there's somethings I can provide you. | 17:13 |
| jaypipes | Redcavalier: I'm curious to see the results of this debug line: https://github.com/openstack/nova/blob/stable/newton/nova/virt/libvirt/guest.py#L295 | 17:13 |
| cfriesen_ | jaypipes: Redcavalier: it'd also be good to have the original instance XML from the boot | 17:13 |
| jaypipes | cfriesen_: agreed. | 17:13 |
| *** Swami has joined #openstack-nova | 17:14 | |
| Redcavalier | XML: https://pastebin.com/9au1jCaY | 17:15 |
| Redcavalier | cfriesen_, jaypipes https://pastebin.com/9au1jCaY | 17:16 |
| Redcavalier | That XML is if you attach while the vm is shutdown and then try to boot it. | 17:17 |
| *** moshele has quit IRC | 17:18 | |
| cfriesen_ | Redcavalier: jaypipes: the controller/bus/target/unit values are identical, so I'm not surprised libvirt doesn't like it | 17:18 |
| cfriesen_ | Redcavalier: jaypipes: also, one being vda and the other sda seems sketchy | 17:19 |
| cfriesen_ | but I'm not sure that would actually impact anything in the guest | 17:19 |
| *** lucasxu has joined #openstack-nova | 17:24 | |
| cfriesen_ | jaypipes: looks like this was the original patch to add virtio-scsi support: https://review.openstack.org/#/c/70263/6/nova/virt/libvirt/driver.py | 17:25 |
| *** MVenesio has quit IRC | 17:27 | |
| *** MVenesio has joined #openstack-nova | 17:27 | |
| jaypipes | cfriesen_: still going through the XML .. | 17:28 |
| cfriesen_ | jaypipes: I wonder if it'd be okay without the "address" being specified, since the libvirt XML docs imply the address is optional. | 17:30 |
| jaypipes | cfriesen_: I'm more curious why there are two block devices, one vda and one sda. | 17:32 |
| cfriesen_ | jaypipes: he booted with one and then attached the other | 17:33 |
| jaypipes | I don't think Redcavalier did that. | 17:33 |
| cfriesen_ | "That XML is if you attach while the vm is shutdown and then try to boot it." | 17:34 |
| jaypipes | the bug report says this is a" boot from new volume created in cinder with a selected image on it." | 17:35 |
| cfriesen_ | "Once your VM is booted up, attaching a drive will make the following error appear in the nova-compute log" | 17:35 |
| Redcavalier | jaypipes, cfriesen_ sda is the root block device, vda is the new one attached to it | 17:35 |
| jaypipes | ah, gotcha.; | 17:35 |
| jaypipes | sorry, misunderstood. | 17:35 |
| jaypipes | I wonder why it's filling in the address XML element with 0's | 17:36 |
| *** ZZelle has joined #openstack-nova | 17:39 | |
| Redcavalier | jaypipes, regarding debug line 295, I'm having trouble identifying the exact resulting line in my debug logs. Do you want me to copy all the logs I have? | 17:39 |
| jaypipes | Redcavalier: not yet, no. will tell you if we need that. still going off the pasted XML you supplied earlier. | 17:39 |
| Redcavalier | jaypipes, alright, I'll go eat, bbiab. | 17:41 |
| *** ralonsoh has quit IRC | 17:41 | |
| cfriesen_ | jaypipes: related somehow to LibvirtDriver._prepare_device_bus() maybe? | 17:43 |
| *** jdillaman has quit IRC | 17:45 | |
| *** gjayavelu has joined #openstack-nova | 17:46 | |
| *** felipemonteiro_ has quit IRC | 17:49 | |
| *** felipemonteiro has joined #openstack-nova | 17:49 | |
| jaypipes | cfriesen_: I'm not seeing anywhere at all in the libvirt/* modules where controller, target, unit or bus attributes of the LibvirtConfigGuestrDeviceAddressDrive object are ever being set. | 17:49 |
| jaypipes | cfriesen_: https://github.com/openstack/nova/blob/stable/newton/nova/virt/libvirt/config.py#L1175 | 17:49 |
| jaypipes | cfriesen_: I've grep'd the source code and nothing is ever setting that. | 17:49 |
| *** felipemonteiro_ has joined #openstack-nova | 17:50 | |
| * jaypipes wonders if they default to 0 in the format_dom() methods? | 17:50 | |
| jaypipes | or perhaps libvirt converts None values to 0? | 17:50 |
| cfriesen_ | jaypipes: yeah, I hit the same thing. it looks like the scsi address bits were added by https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html but it's not clear to me how the scsi address was supposed to be set. | 17:53 |
| *** felipemonteiro has quit IRC | 17:53 | |
| jaypipes | lyarwood, artom: either one of you around? | 17:55 |
| jaypipes | lyarwood, artom: see cfriesen_'s question above. we're wondering about how https://github.com/openstack/nova/blob/stable/newton/nova/virt/libvirt/config.py#L1175 attributes ever get set. | 17:55 |
| *** mvk has joined #openstack-nova | 17:58 | |
| cfriesen_ | jaypipes: the format_dom() routine was originally added by Sahid, it looks like. | 17:58 |
| cfriesen_ | jaypipes: and quite recently | 17:59 |
| *** jdillaman has joined #openstack-nova | 17:59 | |
| *** Apoorva_ has joined #openstack-nova | 18:01 | |
| *** rmart04 has joined #openstack-nova | 18:03 | |
| *** Apoorva has quit IRC | 18:04 | |
| *** dtantsur is now known as dtantsur|afk | 18:07 | |
| *** salv-orlando has quit IRC | 18:07 | |
| *** salv-orlando has joined #openstack-nova | 18:08 | |
| *** lpetrut has quit IRC | 18:09 | |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP/POC Implement new attach Cinder flow https://review.openstack.org/330285 | 18:11 |
| *** salv-orlando has quit IRC | 18:12 | |
| *** acormier_ has joined #openstack-nova | 18:14 | |
| *** acormier_ has quit IRC | 18:16 | |
| *** slaweq has joined #openstack-nova | 18:16 | |
| *** acormier has quit IRC | 18:18 | |
| *** acormier has joined #openstack-nova | 18:18 | |
| *** felipemonteiro_ has quit IRC | 18:23 | |
| *** tuanla__ has quit IRC | 18:23 | |
| *** tuanla_fujitsu has quit IRC | 18:23 | |
| *** acormier has quit IRC | 18:24 | |
| *** slaweq has quit IRC | 18:24 | |
| *** acormier has joined #openstack-nova | 18:24 | |
| openstackgerrit | Merged openstack/nova master: Transform instance.live_migration_rollback notification https://review.openstack.org/402124 | 18:26 |
| *** felipemonteiro has joined #openstack-nova | 18:28 | |
| *** rmart04 has quit IRC | 18:28 | |
| artom | jaypipes, I'm around | 18:29 |
| *** felipemonteiro_ has joined #openstack-nova | 18:29 | |
| jaypipes | artom: see the discussion above... cfriesen_ and I are trying to figure out where the address attributes are being set for Drive devices. | 18:30 |
| jaypipes | artom: wondered if you knew. :) | 18:30 |
| artom | jaypipes, not off the top of my head | 18:30 |
| * artom tries to dig a bit | 18:30 | |
| artom | jaypipes, although that particular class you linked is only used for device metadata AFAIK | 18:31 |
| jaypipes | artom: unfortunately, it seems to be confusing QEMU. https://bugs.launchpad.net/nova/+bug/1699228 | 18:32 |
| openstack | Launchpad bug 1699228 in OpenStack Compute (nova) "Additional block devices cannot be attached when libvirt driver is virtio-scsi" [Undecided,New] | 18:32 |
| jaypipes | artom: this is the XML that is causing issues on attach_volume() https://pastebin.com/9au1jCaY | 18:32 |
| jaypipes | Redcavalier is the submitter of the bug. cfriesen_ and I have been trying to get to the bottom of it... | 18:32 |
| jaypipes | with little luck :) | 18:32 |
| *** felipemonteiro has quit IRC | 18:33 | |
| artom | jaypipes, so the problem is that lines 74 and 67, got example, are identical? | 18:33 |
| artom | *for | 18:33 |
| artom | Hrmm | 18:33 |
| cfriesen_ | artom: seems so, yes | 18:34 |
| jaypipes | artom: yep :) | 18:34 |
| jaypipes | but neither cfriesen_ or I can see where those elements are actually being defined. | 18:34 |
| *** cfriesen_ is now known as cfriesen | 18:34 | |
| jaypipes | welcome back. | 18:34 |
| artom | My intuition is that LibvirtConfigGuestDeviceAddressDrive, which was intended to be read-only from the XML and then exposed in the device role tagging metadata, somehow started participating in actually writing the XML | 18:34 |
| jaypipes | :P | 18:34 |
| cfriesen | lol | 18:34 |
| artom | And all those 0 are just the default values for the fields | 18:35 |
| jaypipes | artom: yeah, we figured that about the default 0 values. But still, how would that section be written to the XML config file? | 18:35 |
| cfriesen | artom: except the default values are None, at least in LibvirtConfigGuestDeviceAddressDrive | 18:35 |
| *** markmcclain has quit IRC | 18:35 | |
| cfriesen | artom: and the format_dom() routine is actually very new | 18:36 |
| cfriesen | artom: the problem was apparently reproduced on Mitaka | 18:36 |
| *** annegentle has joined #openstack-nova | 18:38 | |
| *** markmcclain has joined #openstack-nova | 18:38 | |
| artom | cfriesen, that class didn't exist in Mitaka | 18:38 |
| artom | So if it was really reproduced in Mitaka, we're looking in the wrong place | 18:38 |
| *** gjayavelu has quit IRC | 18:40 | |
| bswartz | hey nova guys -- I'm trying to debug n-cpu and I'm facing a problem | 18:40 |
| bswartz | I have devstack setup, and when I start n-cpu from systemd, it functionals normally | 18:40 |
| bswartz | when I stop the systemd service and directly invoke /usr/local/bin/nova-compute, I get this error however: "ERROR nova.virt.libvirt.host [-] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" | 18:40 |
| bswartz | what do I need to do differently? | 18:41 |
| clarkb | bswartz: you need to start the service as a user that can read/write to that "file" | 18:41 |
| clarkb | bswartz: the systemd unit does this for you | 18:41 |
| bswartz | clarkb: systemd runs the binary as "bswartz" | 18:41 |
| bswartz | which is me (obviously) | 18:41 |
| clarkb | bswartz: I think it may set the group as libvirt or whatever is the proper group | 18:41 |
| bswartz | there's nothing like that mentioned in the systemd unit file | 18:42 |
| bswartz | I'll try adding myself to the appropriate group and trying again | 18:42 |
| clarkb | bswartz: lib/nova: run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $compute_cell_conf" $LIBVIRT_GROUP | 18:42 |
| clarkb | bswartz: you sure LIBVIRT_GROUP isn't ending up in the unit ? | 18:43 |
| cfriesen | artom: has anyone ever tested having multiple virtio-scsi volumes attached to the same instance? maybe it's a day-one bug? | 18:43 |
| bswartz | clarkb: ah crap I missed a line -- it was right there in plain sight | 18:44 |
| bswartz | Group = libvirt | 18:44 |
| *** jmlowe has joined #openstack-nova | 18:44 | |
| bswartz | thank you! | 18:44 |
| clarkb | iniset -sudo $unitfile "Service" "Group" "$group" | 18:44 |
| clarkb | ya that | 18:44 |
| artom | cfriesen, no idea :/ | 18:44 |
| artom | lyarwood or mdbooth might know | 18:44 |
| *** salv-orlando has joined #openstack-nova | 18:49 | |
| artom | cfriesen, jaypipes https://bugs.launchpad.net/nova/+bug/1528877 seems to be related judging by the error message | 18:49 |
| openstack | Launchpad bug 1486204 in OpenStack Compute (nova) "duplicate for #1528877 libvirt: user specified volume device names are not ignored" [Medium,In progress] - Assigned to Maxim Nestratov (mnestratov) | 18:49 |
| artom | Anyways, it's almost 22:00 here, sorry I couldn't of more help, I'm off | 18:51 |
| jaypipes | no worries | 18:51 |
| Redcavalier | cfriesen, jaypipes, artom I had seen 1528877 before making the bug report, but I was under the feeling that 1528877 seem to be reporting more than one bug. | 18:52 |
| *** gjayavelu has joined #openstack-nova | 18:52 | |
| Redcavalier | it's also considered a duplicate of 1486204 which appears to be completely unrelated | 18:52 |
| artom | Redcavalier, entirely possible, but your error is present the trace | 18:52 |
| cfriesen | artom: jaypipes: I wonder if the mismatched device names are breaking some auto-numbering thing in libvirt | 18:52 |
| jaypipes | not sure :( | 18:53 |
| Redcavalier | artom, true. Just wanted to state why I made a new bug report | 18:53 |
| *** artom has quit IRC | 18:57 | |
| *** adisky__ has quit IRC | 18:58 | |
| *** lpetrut has joined #openstack-nova | 19:03 | |
| cfriesen | jaypipes: artom: Redcavalier: I just booted with the specified image properties, and the original XML as emitted by nova doesn't have an "address" element at all. | 19:03 |
| cfriesen | http://paste.openstack.org/show/613748/ | 19:04 |
| jaypipes | cfriesen: that's with virtio-scsi? | 19:04 |
| cfriesen | jaypipes: yes...and it's using the "vda" device name rather than "sda" | 19:05 |
| jaypipes | hmm | 19:05 |
| Redcavalier | that's strage | 19:05 |
| jaypipes | this is newton? | 19:05 |
| Redcavalier | what metadata properties did you set? | 19:05 |
| cfriesen | jaypipes: yes. (image properties are hw_scsi_model=virtio-scsi and hw_disk_bus=scsi) | 19:05 |
| Redcavalier | same as me | 19:06 |
| jaypipes | I'm at the end of my knowledge on this :( I think we need to bring in lyarwood or mdbooth... | 19:07 |
| Redcavalier | cfriesen, are you able to reproduce the bug if you add the second drive though? | 19:07 |
| cfriesen | jaypipes: if I remove the image properties I get this in the xml: <target bus="virtio" dev="vda"/> | 19:08 |
| jaypipes | cfriesen: for the root disk, yes? | 19:09 |
| cfriesen | yes | 19:09 |
| cfriesen | I suspect that the fact we're using vda for the initial device is breaking autonumbering in libvirt. | 19:09 |
| jaypipes | cfriesen: and Redcavalier is saying that when he attach_volume()'s the second disk, it gets added with a device name of "sda" and the same address attributes (target/controller/unit/bus) | 19:09 |
| cfriesen | is there a way to specify virtio-scsi vs virtio-blk when attaching a volume? | 19:10 |
| *** smatzek has quit IRC | 19:11 | |
| jaypipes | no idea :( | 19:11 |
| *** slaweq has joined #openstack-nova | 19:11 | |
| Redcavalier | not that I know of. I believe it attaches to whatever controller there is, hence why we specify hw_disk_bus=scsi. Not an expert though, but that's what I understood from everything I read. | 19:12 |
| jaypipes | sounds about right to me. | 19:13 |
| *** sdague has quit IRC | 19:17 | |
| *** acormier has quit IRC | 19:18 | |
| *** acormier has joined #openstack-nova | 19:18 | |
| cfriesen | jaypipes: I tried overriding the "disk_prefix" config option in nova.conf, but it had no effect. it almost seems like we're not properly calling get_dev_prefix_for_disk_bus | 19:20 |
| *** acormier_ has joined #openstack-nova | 19:21 | |
| *** acormier has quit IRC | 19:22 | |
| *** acormier_ has quit IRC | 19:24 | |
| *** huanxie has joined #openstack-nova | 19:24 | |
| *** acormier has joined #openstack-nova | 19:24 | |
| jmlowe | Is anybody having trouble with detaching volumes in ocata? | 19:29 |
| *** salv-orlando has quit IRC | 19:29 | |
| jmlowe | nova-compute throws errors trying to decode the json of the block device mapping connection string | 19:30 |
| *** salv-orlando has joined #openstack-nova | 19:30 | |
| *** ChanServ has quit IRC | 19:30 | |
| avolkov | jaypipes: returning to my question, is it better to document POST inventories API request as is? | 19:33 |
| avolkov | or there is some plan to refactor/change that request? | 19:34 |
| *** smatzek has joined #openstack-nova | 19:34 | |
| *** salv-orlando has quit IRC | 19:35 | |
| *** ChanServ has joined #openstack-nova | 19:35 | |
| *** card.freenode.net sets mode: +o ChanServ | 19:35 | |
| *** smatzek has quit IRC | 19:35 | |
| *** smatzek has joined #openstack-nova | 19:36 | |
| *** gjayavelu has quit IRC | 19:37 | |
| *** sdague has joined #openstack-nova | 19:37 | |
| *** nic1 is now known as nic | 19:39 | |
| dansmith | melwitt: I think this was just a transplant from your original patch right? https://review.openstack.org/#/c/446244/28 | 19:43 |
| jaypipes | avolkov: link for me please? :) | 19:43 |
| dansmith | melwitt: it's simple enough that even though my name is on it I think I can just +W it | 19:43 |
| *** slaweq has quit IRC | 19:45 | |
| *** slaweq has joined #openstack-nova | 19:45 | |
| *** yamahata has joined #openstack-nova | 19:47 | |
| *** Apoorva_ has quit IRC | 19:47 | |
| *** Apoorva has joined #openstack-nova | 19:48 | |
| *** tobias_ has joined #openstack-nova | 19:48 | |
| tobias_ | is there anyone here who can PM me that's really good at rsync and virtualization-cases? | 19:49 |
| tobias_ | also NFS. | 19:50 |
| *** tobias_ is now known as tobias_fiberdata | 19:51 | |
| avolkov | jaypipes: sure, https://review.openstack.org/#/c/451855/ | 19:51 |
| avolkov | see comments from cdent | 19:52 |
| jaypipes | avolkov: thx. looking now. | 19:52 |
| avolkov | it's the last piece of placement-api-ref | 19:52 |
| jaypipes | tobias_fiberdata: probably better to email openstack@ mailing list. | 19:52 |
| tobias_fiberdata | jaypipes, great tip :) | 19:53 |
| tobias_fiberdata | thanks | 19:53 |
| jaypipes | avolkov: those are abandoned.. | 19:53 |
| *** huanxie has quit IRC | 19:54 | |
| avolkov | jaypipes: cdent wrote: "We should consider not documenting this method at all, list it as deprecated", are you agree?" | 19:55 |
| *** xyang1 has quit IRC | 19:55 | |
| dansmith | melwitt: also I'm happy to address my own feedback on the utility patch if you're busy | 19:57 |
| *** xyang1 has joined #openstack-nova | 19:59 | |
| melwitt | dansmith: yeah, most of the patches are transplants but better safe than sorry I guess. that one is simple though | 19:59 |
| melwitt | looking at the utility patch now | 20:00 |
| *** marst has quit IRC | 20:01 | |
| *** marst has joined #openstack-nova | 20:01 | |
| *** edand has joined #openstack-nova | 20:02 | |
| *** rajathagasthya has quit IRC | 20:03 | |
| *** liverpooler has quit IRC | 20:04 | |
| *** edand has quit IRC | 20:06 | |
| *** edand has joined #openstack-nova | 20:06 | |
| jaypipes | avolkov: ah! sorry, I wasn't following you. Yes, I agree 100% with that. | 20:10 |
| *** ericyoung has quit IRC | 20:10 | |
| *** yamahata has quit IRC | 20:18 | |
| *** tobias_ has joined #openstack-nova | 20:19 | |
| *** yassine has quit IRC | 20:20 | |
| *** tesseract has quit IRC | 20:21 | |
| *** moshele has joined #openstack-nova | 20:23 | |
| *** tobias_fiberdata has quit IRC | 20:23 | |
| moshele | sean-k-mooney: hi can you review again https://review.openstack.org/#/c/460278/ ? | 20:24 |
| sean-k-mooney | moshele: sure ill take a look now | 20:25 |
| moshele | moshele: thanks | 20:25 |
| *** yassine has joined #openstack-nova | 20:26 | |
| *** gbarros has joined #openstack-nova | 20:27 | |
| *** salv-orlando has joined #openstack-nova | 20:28 | |
| *** rajathagasthya has joined #openstack-nova | 20:30 | |
| *** rajathagasthya has quit IRC | 20:30 | |
| *** rajathagasthya has joined #openstack-nova | 20:31 | |
| *** annegentle has quit IRC | 20:33 | |
| *** felipemonteiro has joined #openstack-nova | 20:33 | |
| *** felipemonteiro_ has quit IRC | 20:34 | |
| *** burt has quit IRC | 20:36 | |
| *** pewp has quit IRC | 20:36 | |
| *** pewp has joined #openstack-nova | 20:36 | |
| *** pchavva has quit IRC | 20:37 | |
| *** felipemonteiro_ has joined #openstack-nova | 20:38 | |
| sean-k-mooney | moshele: looks good to me. i have 1 or 2 question in line regarding multi vendor supprot in the future but it should not block the patch | 20:41 |
| *** slaweq has quit IRC | 20:41 | |
| *** felipemonteiro has quit IRC | 20:41 | |
| *** slaweq has joined #openstack-nova | 20:42 | |
| *** jcromer has joined #openstack-nova | 20:43 | |
| *** jcromer has left #openstack-nova | 20:44 | |
| *** rajathag_ has joined #openstack-nova | 20:44 | |
| *** rajathagasthya has quit IRC | 20:45 | |
| *** slaweq has quit IRC | 20:46 | |
| openstackgerrit | Merged openstack/os-vif master: hardware offload support for openvswitch https://review.openstack.org/460278 | 20:46 |
| sean-k-mooney | moshele: ^ the gate is quick today | 20:48 |
| *** jheroux has quit IRC | 20:49 | |
| *** gbarros has quit IRC | 20:50 | |
| moshele | moshele: :) | 20:55 |
| moshele | sean-k-mooney: :) | 20:55 |
| *** moshele has quit IRC | 20:57 | |
| sean-k-mooney | moshele: if you get a chance can you reply to the comments i left inline and ill check them in the morning, im just heading home for the day so wont be on irc for much longer | 20:57 |
| *** cleong has quit IRC | 20:58 | |
| *** annegentle has joined #openstack-nova | 21:02 | |
| *** annegentle has quit IRC | 21:03 | |
| *** gjayavelu has joined #openstack-nova | 21:04 | |
| *** felipemonteiro_ has quit IRC | 21:08 | |
| *** slaweq has joined #openstack-nova | 21:08 | |
| *** kevinbenton has quit IRC | 21:12 | |
| *** gbarros has joined #openstack-nova | 21:13 | |
| *** kevinbenton has joined #openstack-nova | 21:15 | |
| *** edmondsw has quit IRC | 21:17 | |
| *** dixiaoli has joined #openstack-nova | 21:17 | |
| *** smatzek has quit IRC | 21:18 | |
| *** edmondsw has joined #openstack-nova | 21:18 | |
| *** annegentle has joined #openstack-nova | 21:18 | |
| *** acormier has quit IRC | 21:19 | |
| *** acormier has joined #openstack-nova | 21:20 | |
| *** jmlowe has quit IRC | 21:20 | |
| *** dixiaoli has quit IRC | 21:21 | |
| *** gbarros has quit IRC | 21:23 | |
| *** tobias_ has quit IRC | 21:23 | |
| *** edmondsw has quit IRC | 21:23 | |
| *** edmondsw has joined #openstack-nova | 21:24 | |
| *** acormier has quit IRC | 21:24 | |
| *** edmondsw has quit IRC | 21:28 | |
| *** edand_ has joined #openstack-nova | 21:28 | |
| *** edmondsw has joined #openstack-nova | 21:31 | |
| *** edand has quit IRC | 21:31 | |
| *** esberglu has quit IRC | 21:32 | |
| *** esberglu has joined #openstack-nova | 21:32 | |
| *** edmondsw has quit IRC | 21:35 | |
| openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.resize.error notifications https://review.openstack.org/417882 | 21:37 |
| *** esberglu has quit IRC | 21:37 | |
| *** tidwellr has quit IRC | 21:41 | |
| *** lucasxu has quit IRC | 21:41 | |
| *** xyang1 has quit IRC | 21:41 | |
| *** lpetrut has quit IRC | 21:43 | |
| *** thorst has quit IRC | 21:44 | |
| *** thorst has joined #openstack-nova | 21:46 | |
| *** edmondsw has joined #openstack-nova | 21:50 | |
| *** thorst has quit IRC | 21:50 | |
| *** Redcavalier has quit IRC | 21:52 | |
| *** tonygunk has quit IRC | 21:54 | |
| *** edmondsw has quit IRC | 21:55 | |
| *** esberglu has joined #openstack-nova | 21:57 | |
| *** esberglu has quit IRC | 21:58 | |
| *** esberglu has joined #openstack-nova | 21:58 | |
| *** kylek3h has quit IRC | 22:01 | |
| openstackgerrit | Steven Webster proposed openstack/nova master: Race condition between audit and migrate/resize revert https://review.openstack.org/400995 | 22:02 |
| *** kylek3h has joined #openstack-nova | 22:02 | |
| *** acormier has joined #openstack-nova | 22:02 | |
| *** acormier has quit IRC | 22:04 | |
| *** thorst has joined #openstack-nova | 22:05 | |
| *** acormier has joined #openstack-nova | 22:05 | |
| *** eharney has quit IRC | 22:06 | |
| *** kylek3h has quit IRC | 22:06 | |
| *** thorst has quit IRC | 22:09 | |
| *** yamahata has joined #openstack-nova | 22:17 | |
| *** edmondsw has joined #openstack-nova | 22:18 | |
| *** mrhillsman is now known as codebauss | 22:22 | |
| *** ericyoung has joined #openstack-nova | 22:22 | |
| *** codebauss is now known as mrhillsman | 22:22 | |
| *** edmondsw has quit IRC | 22:23 | |
| *** ckopper has quit IRC | 22:25 | |
| *** yamahata has quit IRC | 22:31 | |
| *** slaweq has quit IRC | 22:31 | |
| *** marst has quit IRC | 22:32 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Add get_count_by_vm_state() to InstanceList object https://review.openstack.org/446244 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add scatter gather utilities for cells https://review.openstack.org/475957 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Remove 'reserved' count from used limits https://review.openstack.org/446242 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Remove useless quota_usage_refresh from nova-manage https://review.openstack.org/446243 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Count instances to check quota https://review.openstack.org/416521 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Make key_pairs use check_deltas for quota https://review.openstack.org/477699 | 22:33 |
| openstackgerrit | melanie witt proposed openstack/nova master: Make security_group_rules use check_deltas for quota https://review.openstack.org/477700 | 22:33 |
| *** marst has joined #openstack-nova | 22:35 | |
| *** rajathag_ has quit IRC | 22:40 | |
| *** chyka_ has joined #openstack-nova | 22:44 | |
| *** ericyoung has quit IRC | 22:44 | |
| *** chyka has quit IRC | 22:46 | |
| *** annegentle has quit IRC | 22:47 | |
| *** ckopper has joined #openstack-nova | 22:47 | |
| *** MikeG451 has joined #openstack-nova | 22:50 | |
| *** jmlowe has joined #openstack-nova | 22:51 | |
| *** syjulian has joined #openstack-nova | 22:54 | |
| *** acormier has quit IRC | 22:55 | |
| *** acormier has joined #openstack-nova | 22:55 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.48: Standardization of VM diagnostics https://review.openstack.org/473228 | 22:55 |
| *** sdague has quit IRC | 22:56 | |
| *** acormier has quit IRC | 23:00 | |
| *** annegentle has joined #openstack-nova | 23:08 | |
| *** marst has quit IRC | 23:13 | |
| *** acormier has joined #openstack-nova | 23:15 | |
| *** jmlowe has quit IRC | 23:17 | |
| *** hongbin has quit IRC | 23:20 | |
| *** rajathagasthya has joined #openstack-nova | 23:22 | |
| *** takashin has joined #openstack-nova | 23:28 | |
| *** jmlowe has joined #openstack-nova | 23:30 | |
| *** Swami has quit IRC | 23:41 | |
| *** jmlowe has quit IRC | 23:42 | |
| *** rajathag_ has joined #openstack-nova | 23:42 | |
| *** jmlowe has joined #openstack-nova | 23:44 | |
| *** marst has joined #openstack-nova | 23:44 | |
| *** rajathagasthya has quit IRC | 23:45 | |
| *** diga has joined #openstack-nova | 23:46 | |
| *** acormier has quit IRC | 23:53 | |
| *** acormier has joined #openstack-nova | 23:53 | |
| *** acormier has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!