Thursday, 2014-07-03

jerryztchaypo: just tried to create an undercloud stack but the template seems to be in wrong format. the log looked like this: http://paste.openstack.org/show/85360/00:02
jerryztchaypo: and later i changed     Default: default to Default: ‘’ , it went through00:03
*** yfaykya has quit IRC00:03
tchaypois this the templates I just pushed up for review?00:04
jerryzhttps://github.com/openstack/tripleo-heat-templates/blob/9e33760e06116fd44d9766b4b5828a29e69ec736/undercloud-source.yaml#L10400:05
jerryztchaypo: i used the one from trunk00:05
*** yfaykya has joined #tripleo00:05
tchaypothat would explain why the error message wasn't making sense to me in that context :)00:05
tchaypowhere was the line that you changed?00:06
jerryz10400:06
jerryz105 acctually00:06
tchaypohence the link you just pasted00:06
jerryztchaypo: yes, i just the default to '', and it went through00:07
jerryzchange00:07
tchaypothat's awesome. git blame says that hasn't changed since 2013-10-0100:07
*** saurabhs has quit IRC00:08
tchaypohowever, i see that it's talking about an "EC2 KeyPair to enable SSH access"00:08
tchaypoI vaguely remember StevenK making some changes to something sshish recently00:08
jerryztchaypo: i will get back to you in a few minutes. i will be in a short meeting00:09
tchayposure00:09
*** saurabhs has joined #tripleo00:09
*** yfaykya has quit IRC00:09
tchaypoI'm about to jump on a train and head towards lifeless soon too00:10
*** yfaykya has joined #tripleo00:11
tchayponope, StevenK's change that I was thinking of is https://review.openstack.org/#/c/101413/1/scripts/devtest_testenv.sh but i don't see how that could be at all relevant00:15
tchaypojerryz: when you get a chance, could you paste more of the logs and maybe open a bug?00:15
*** yfaykya has quit IRC00:16
tchaypoIt sounds like you have a good workaround for now but I'm confused about what could have changed to break this - I'm guessing it would be good to figure that out00:16
*** rcarrillocruz has quit IRC00:16
*** yfaykya has joined #tripleo00:17
*** chuckC has quit IRC00:19
*** rcarrillocruz has joined #tripleo00:21
*** yfaykya has quit IRC00:22
*** yfaykya has joined #tripleo00:23
adam_glifeless, FYI tracking upgrade related work @  https://etherpad.openstack.org/p/tripleo-upgrades00:26
*** yfaykya has quit IRC00:28
*** yfaykya has joined #tripleo00:29
*** yfaykya has quit IRC00:34
*** yfaykya has joined #tripleo00:36
greghaynesadam_g: have you started on a patch to special case one-node mysql clusters?00:37
greghaynesIf not, I can hash that out pretty quick00:37
*** saurabhs has left #tripleo00:38
adam_ggreghaynes, not yet. testing a libvirt change atm00:40
*** yfaykya has quit IRC00:40
*** yfaykya has joined #tripleo00:42
*** yfaykya has quit IRC00:46
*** yfaykya has joined #tripleo00:48
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: WIP: Check git HEAD for os-{apply,cloud}-config  https://review.openstack.org/10439600:49
openstackgerritA change was merged to openstack/diskimage-builder: Add RHEL7 to Red Hat family in pkg-map  https://review.openstack.org/10293600:50
*** yfaykya has quit IRC00:52
*** yfaykya has joined #tripleo00:54
*** yfaykya has quit IRC00:58
*** ccrouch has joined #tripleo00:59
*** yfaykya has joined #tripleo01:00
*** yfaykya has quit IRC01:05
*** yfaykya has joined #tripleo01:06
*** chuckC has joined #tripleo01:07
*** yfaykya has quit IRC01:11
*** yfaykya has joined #tripleo01:12
*** eghobo has quit IRC01:13
*** nosnos has joined #tripleo01:13
*** yfaykya has quit IRC01:17
*** yfaykya has joined #tripleo01:18
*** nosnos has quit IRC01:20
openstackgerritA change was merged to openstack/diskimage-builder: Set LC_ALL=C into dib-run-parts env  https://review.openstack.org/10380501:22
*** yfaykya has quit IRC01:23
*** weshay has quit IRC01:23
*** yfaykya has joined #tripleo01:25
*** morazi has quit IRC01:27
jerryztchaypo: turned out that the error was because i didn't run user-config to create the default keypair01:28
tchaypojerryz: hence no keypair named default?01:28
tchaypothat makes sense :)01:29
*** yfaykya has quit IRC01:29
*** xuhaiwei has joined #tripleo01:29
*** yfaykya has joined #tripleo01:31
xuhaiweiI am building devtest environment with ironic, but got this error ERROR: The Parameter (PowerManager) was not defined in template.  I checked undercloud-vm-ironic-deploy.yaml, there is no PowerManager parameter indeed01:32
xuhaiweiIs this a bug?01:32
openstackgerritAdam Gandelman proposed a change to openstack/tripleo-image-elements: Move libvirt's qemu configuration dir to state fs  https://review.openstack.org/10440701:34
*** yfaykya has quit IRC01:35
*** yfaykya has joined #tripleo01:37
*** marun is now known as marun_afk01:39
*** yfaykya has quit IRC01:41
openstackgerritIan Wienand proposed a change to openstack/diskimage-builder: Don't match editor backup files in environment  https://review.openstack.org/10440901:42
*** yfaykya has joined #tripleo01:43
*** howleyt has quit IRC01:45
*** howleyt has joined #tripleo01:45
*** yfaykya has quit IRC01:47
tchaypoxuhaiwei: I beleive I ran into this a few weeks ago01:49
xuhaiweitchaypo: you mean there is no problem?01:49
tchaypofrom memory the cause was that something had generated some environment file before I switched to using ironic, and the fix was to delete that file so it would be regenerated01:49
*** yfaykya has joined #tripleo01:49
tchaypoxuhaiwei: if I remembered which file needs to be deleted I could be more helpful01:49
xuhaiweitchaypo: but the template without PowerManager parameter is not strange?01:51
tchaypoi think the file is undercloud-env.json01:52
tchaypoor overcloud-env.json01:52
xuhaiweiI think it's undercloud-json because this error happens when building undercloud01:52
tchaypoxuhaiwei: grep for PowerManager in http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2014-06-17.log for a similar converation01:53
*** yfaykya has quit IRC01:54
tchaypoah, and http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2014-06-24.log has the bit where I ran across it01:54
xuhaiweithank you01:55
*** yfaykya has joined #tripleo01:55
*** yfaykya has quit IRC02:00
*** yfaykya has joined #tripleo02:01
*** blamar has joined #tripleo02:01
openstackgerritJulia Kreger proposed a change to openstack/tripleo-image-elements: Prevent multiple MySQL masters  https://review.openstack.org/10441402:04
openstackgerritJulia Kreger proposed a change to openstack/tripleo-image-elements: Prevent multiple MySQL masters  https://review.openstack.org/10441402:05
*** yfaykya has quit IRC02:06
TheJuliagreghaynes: ^^^02:06
*** yfaykya has joined #tripleo02:07
*** Matt1 has quit IRC02:09
xuhaiweitchaypo: cloud you please explain why PowerManager is not needed in the template?02:11
*** yfaykya has quit IRC02:12
*** yfaykya has joined #tripleo02:13
*** yfaykya has quit IRC02:18
*** yfaykya has joined #tripleo02:20
*** yfaykya has quit IRC02:24
*** yfaykya has joined #tripleo02:26
*** killer_prince has quit IRC02:26
*** yfaykya has quit IRC02:30
*** yfaykya has joined #tripleo02:32
*** yfaykya has quit IRC02:37
*** ramishra has joined #tripleo02:37
*** yfaykya has joined #tripleo02:38
*** yfaykya has quit IRC02:43
*** yfaykya has joined #tripleo02:44
*** yfaykya has quit IRC02:49
*** yfaykya has joined #tripleo02:50
*** yfaykya has quit IRC02:55
*** yfaykya has joined #tripleo02:56
tchaypoxuhaiwei: nope, sorry, I don't know. I haven't had to look at the difference between the power managers.02:58
tchaypoI only know that it bit me once and I had to remove the env file to get it working again02:59
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133458103:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133504203:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611003:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133593203:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611503:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133501603:00
uvirtbotLaunchpad bug 1334581 in tripleo "os-collect-config.conf need support to Heat userdata" [Undecided,In progress]03:00
uvirtbotLaunchpad bug 1335042 in tripleo "diskimage-builder: extlinux installation failure" [Undecided,New]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133364903:00
uvirtbotLaunchpad bug 1336110 in tripleo "mysql does not start on boot post-upgrade" [Undecided,New]03:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133245803:00
uvirtbotLaunchpad bug 1335932 in diskimage-builder "/bin/dib-run-parts fails for spanish locale" [Undecided,Fix committed]03:00
uvirtbotLaunchpad bug 1336115 in tripleo "instances no longer running post-upgrade" [Undecided,In progress]03:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133252103:00
uvirtbotLaunchpad bug 1335016 in tripleo "diskimage-builder fails for Debian" [Undecided,New]03:00
uvirtbotLaunchpad bug 1333649 in tripleo "/etc/hosts entries written by hosts element doesn't survive reboot" [Undecided,In progress]03:00
*** untriaged-bot has quit IRC03:00
uvirtbotLaunchpad bug 1332458 in diskimage-builder "Not able to build armhf or arm64 images on amd64 arch systems." [Undecided,In progress]03:00
uvirtbotLaunchpad bug 1332521 in diskimage-builder "failed to remove `/tmp/image.A2aupYJb/mnt/tmp/in_target.d': Device or resource busy" [Undecided,New]03:00
*** eghobo has joined #tripleo03:00
*** yfaykya has quit IRC03:01
*** yfaykya has joined #tripleo03:02
openstackgerritJames Polley proposed a change to openstack/tripleo-image-elements: Support new virtual_interfaces definition for VIPs  https://review.openstack.org/10436803:03
greghaynesTheJulia: Reviewed03:04
TheJuliaJust saw, agreed, making change now03:05
openstackgerritJames Polley proposed a change to openstack/tripleo-image-elements: Support new virtual_interfaces definition for VIPs  https://review.openstack.org/10436803:05
lifelessgreghaynes: next HA?03:05
greghayneslifeless: I think all thats left are the cert gen ones03:06
greghaynesfor MVP03:06
greghayneswhich is kinda spec-blocked03:06
greghaynesspecifically, https://review.openstack.org/#/c/95101/ and https://review.openstack.org/#/c/94609/03:07
*** yfaykya has quit IRC03:07
greghayneshttp://logs.openstack.org/95/103995/2/check-tripleo/check-tripleo-overcloud-f20/7849310/console.html#_2014-07-03_00_59_26_52303:08
greghaynesthats odd03:08
greghaynesdefinitely not caused by that patch03:08
*** yfaykya has joined #tripleo03:09
*** killer_prince has joined #tripleo03:10
*** killer_prince is now known as lazy_prince03:10
greghayneslifeless: Your input on the top comment for https://review.openstack.org/#/c/103692/5/specs/juno/ssl_pki.rst would be helpful03:11
greghaynessince its related to a comment of yours on previous patch03:12
*** yfaykya has quit IRC03:13
*** yfaykya has joined #tripleo03:15
tchaypogreghaynes: do you have time to take a look at https://review.openstack.org/#/c/103656/ ?03:19
*** yfaykya has quit IRC03:20
greghaynespsh, you dont like a giant list of context-less links!?03:20
tchaypoI'm not smart enough to figure out what they're for03:21
*** yfaykya has joined #tripleo03:21
greghaynesI love how we have to recheck no bug on docs-only patches03:21
openstackgerritIan Wienand proposed a change to openstack/diskimage-builder: Don't match editor backup files in environment  https://review.openstack.org/10440903:22
tchaypofrom memory, in this case it was because when it was setting up the doc test environment it failed to install pbr because the connection to pypi timed out03:23
openstackgerritA change was merged to openstack/tripleo-incubator: Tidy/prettify/segregate the master toctree  https://review.openstack.org/10365603:24
greghaynestchaypo: ^03:24
tchaypowoo03:25
tchaypoand lo, http://docs.openstack.org/developer/tripleo-incubator/ did reflect the changes03:26
*** yfaykya has quit IRC03:26
tchaypoFor my next trick03:26
*** yfaykya has joined #tripleo03:27
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Tweak heading levels  https://review.openstack.org/10443403:30
*** yfaykya has quit IRC03:31
*** yfaykya has joined #tripleo03:33
*** ramishra has quit IRC03:33
*** yfaykya has quit IRC03:38
*** yfaykya has joined #tripleo03:39
*** ramishra has joined #tripleo03:40
openstackgerritIan Wienand proposed a change to openstack/diskimage-builder: Don't match editor backup files in environment  https://review.openstack.org/10440903:40
*** yfaykya has quit IRC03:44
*** yfaykya has joined #tripleo03:45
greghaynesugh, we need to fix haproxy logging03:46
greghaynesdebugging some of these fails isnt exactly... possible... otherwise03:46
TheJuliaheh03:47
*** yfaykya has quit IRC03:50
greghaynesA lot of these spurious test fails I keep seeing are network reset or could not connect to host/etc from openstack clients, I wonder if were reporting stack_complete before we actually bind to our haproxy vip...03:50
lifelessgreghaynes: +103:50
*** yfaykya has joined #tripleo03:51
openstackgerritJulia Kreger proposed a change to openstack/tripleo-image-elements: Prevent multiple MySQL masters  https://review.openstack.org/10441403:52
*** nosnos has joined #tripleo03:52
*** ramishra has quit IRC03:52
lifelessgreghaynes: I see no comments on rev 503:53
*** ramishra has joined #tripleo03:53
lifelessgreghaynes: but that may be gertty failing to sync03:53
greghaynesI see comments on rev 5 :)03:54
openstackgerritIan Wienand proposed a change to openstack/diskimage-builder: Don't match editor backup files in environment  https://review.openstack.org/10440903:54
TheJuliaGoodnight03:56
greghaynesgn03:56
*** yfaykya has quit IRC03:56
*** yfaykya has joined #tripleo03:58
*** yfaykya has quit IRC04:02
*** yfaykya has joined #tripleo04:04
*** eghobo has quit IRC04:05
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Switch to only one top-level heading per doc  https://review.openstack.org/10443404:08
*** yfaykya has quit IRC04:08
*** yfaykya has joined #tripleo04:10
*** yfaykya has quit IRC04:15
*** yfaykya has joined #tripleo04:16
*** ajc_ has joined #tripleo04:17
*** rushiagr has joined #tripleo04:20
*** yfaykya has quit IRC04:21
*** yfaykya has joined #tripleo04:22
*** rushiagr_ has joined #tripleo04:26
*** yfaykya has quit IRC04:27
*** blamar has quit IRC04:28
*** rushiagr has quit IRC04:28
*** yfaykya has joined #tripleo04:28
*** yfaykya has quit IRC04:33
*** yfaykya has joined #tripleo04:34
*** yfaykya has quit IRC04:39
*** yfaykya has joined #tripleo04:40
*** pcrews has quit IRC04:42
*** yfaykya has quit IRC04:45
*** rakesh_hs has joined #tripleo04:46
*** yfaykya has joined #tripleo04:46
*** blamar has joined #tripleo04:50
*** ramishra_ has joined #tripleo04:50
*** yfaykya has quit IRC04:51
*** ramishr__ has joined #tripleo04:51
*** yfaykya has joined #tripleo04:53
*** ramishra has quit IRC04:53
*** ramishra_ has quit IRC04:55
*** amitpp has joined #tripleo04:55
*** yfaykya has quit IRC04:57
*** yfaykya has joined #tripleo04:59
lifelessgreghaynes: ok back focused here04:59
*** nati_ueno has joined #tripleo05:02
openstackgerritIan Wienand proposed a change to openstack/diskimage-builder: Remove fixfiles from rpm-distro finalize  https://review.openstack.org/10345105:03
openstackgerritIan Wienand proposed a change to openstack/diskimage-builder: [WIP] Initial centos7 support  https://review.openstack.org/10444905:03
*** yfaykya has quit IRC05:03
lifelessgreghaynes: commented on https://review.openstack.org/#/c/103692/05:04
openstackgerritazher ullah khan proposed a change to openstack/diskimage-builder: Optimizing directory creation.  https://review.openstack.org/10429205:04
greghayneslifeless: tyty05:04
*** blamar has quit IRC05:05
*** yfaykya has joined #tripleo05:05
*** rameshg87_afk is now known as rameshg8705:05
*** rameshg87 has quit IRC05:05
*** rameshg87 has joined #tripleo05:07
*** ramishr__ has quit IRC05:08
*** ramishra has joined #tripleo05:08
*** yfaykya has quit IRC05:10
*** pcrews has joined #tripleo05:11
*** yfaykya has joined #tripleo05:11
openstackgerritlokesh s proposed a change to openstack/diskimage-builder: Solves chroot error while building cross compiled images  https://review.openstack.org/10348305:12
*** loki184 has joined #tripleo05:14
*** yfaykya has quit IRC05:15
*** yfaykya has joined #tripleo05:17
rameshg87lifeless: hello05:19
lifelessrameshg87: hi?05:20
rameshg87lifeless: just needed your thoughts on one of the possible solutions for the comment you gave on https://review.openstack.org/#/c/97736/2/specs/juno/deploy-ironic-bm.rst05:21
rameshg87lifeless: i have added the comment in the ironic design spec since it has been moved.05:21
*** yfaykya has quit IRC05:22
rameshg87lifeless: but i think we can overcome this problem of low memory by creating enough swap space at the end of the disk(after creating other partitions) and adding it as swap for deploy05:22
rameshg87lifeless: we can delete the partition for deploy-swap once the deploy is complete05:22
rameshg87lifeless: how does that sound ?05:22
lifelessrameshg87: https://etherpad.openstack.org/p/glance-multicast-image-transfer05:23
lifelessrameshg87: we can't use other disks - they may be in use.05:23
lifelessrameshg87: with precious data.05:23
*** yfaykya has joined #tripleo05:23
rameshg87lifeless: can't we just create an extra partition on the same disk ?05:23
rameshg87lifeless: same disk on which we are deploying ..05:24
lifelessrameshg87: no - because rebuild --preserve-ephemeral05:24
lifelessrameshg87: also it would just result in IO thrashing05:24
rameshg87lifeless: i agree on the io thrashing05:26
rameshg87lifeless: but regarding rebuild with --preserve-ephemeral05:27
rameshg87lifeless: the partitions currently are created in 0-ephemeral,1-swap,2-root in that order05:27
lifelesssure, but you aren't allowed to overwrite ephemeral :)05:28
*** pcrews has quit IRC05:28
lifelessand swap may not exist05:28
*** yfaykya has quit IRC05:28
rameshg87lifeless: can't we just add a new partition after root "3-swap-deploy"05:28
*** tzumainn has quit IRC05:29
*** dshulyak_ has joined #tripleo05:29
rameshg87lifeless: i guess adding a partition at the end of the disk doesn't affect the 0-ephermeral partition05:29
rameshg87lifeless: am i correct ?05:29
*** marun_afk has quit IRC05:29
lifelessrameshg87: the root partition will have been grown in that case, so there's no space to add a partition.05:29
*** yfaykya has joined #tripleo05:29
rameshg87lifeless: yes, in the worst case we will need an extra space of "as much as the image size" for the disk05:31
*** robsparker has quit IRC05:31
rameshg87lifeless: but can't we just document that as a limitation for low-memory systems that extra disk space might be required05:31
lifelesswell at the moment with the pxe deploy driver we just work05:31
lifelessseeems like a poor tradeoff to change that to 'doesn't work'05:32
lifelessI mean, you're writing a new deploy driver to make things better, right?05:32
rameshg87lifeless: hmm, yes :-)05:33
*** yfaykya has quit IRC05:34
rameshg87lifeless: so do you feel we should just fetch the uncompressed image directly from glance in such a case05:35
*** yfaykya has joined #tripleo05:35
lifelessrameshg87: that would be an option yes05:36
lifelessrameshg87: and or do the multicast stuff - I'd love to see that happen05:36
lifelessrameshg87: I'd be super happy to provide guidance05:36
rameshg87lifeless: just a question, multicast is also expected to stream uncompressed data, right ?05:37
lifelessrameshg87: yes05:38
rameshg87lifeless: doesn't torrent scale better than multicase image transfer05:38
rameshg87lifeless: torrents have an advanctage that they will pull the data from other bm nodes too05:39
rameshg87lifeless: so that the load will be distributed05:39
rameshg87lifeless: isn't it so ?05:39
greghayneslifeless: https://review.openstack.org/#/c/104414/ not 'required' for HA but would be nice before we make that default05:39
*** yfaykya has quit IRC05:40
lifelessrameshg87: torrents send the data N times and receive it N times, and do random IO05:41
lifelessrameshg87: multicast sends the data once and recieves it N times, and does linear IO.05:41
lifelessrameshg87: -> multicast is way more efficient than torrents.05:41
*** yfaykya has joined #tripleo05:42
rameshg87lifeless: hmm, okay ..05:42
*** rdopieralski has joined #tripleo05:43
greghaynesThere was some software that used to do this for thin clients (multicast deployments)...05:43
* greghaynes tries to remember name05:43
*** Penick has joined #tripleo05:45
lifelessgreghaynes: there are links to such in the glance etherpad above05:46
*** robsparker has joined #tripleo05:46
*** yfaykya has quit IRC05:46
greghaynesah, flamethrower05:46
greghayneswoo for dovs05:46
greghayneser, docs05:47
*** yfaykya has joined #tripleo05:48
lifelessfrisbee might be the go, but it looks like it needs some love05:48
greghaynesI suspect were going to get a lot of push back from operators not wanting to allow multicast :/05:51
lifelessthats what pluggable is for05:52
lifelessno reason we can't do torrent as well05:52
lifelessthough honestly, if we're pushing 2GB of data around, C is win05:52
*** yfaykya has quit IRC05:52
*** yfaykya has joined #tripleo05:54
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Allow single node mysql clusters to restart  https://review.openstack.org/10445505:56
*** rcarrill` has joined #tripleo05:56
*** rcarrillocruz has quit IRC05:57
*** yfaykya has quit IRC05:58
*** ohadlevy_ is now known as ohadlevy05:58
*** ohadlevy has joined #tripleo05:58
*** yfaykya has joined #tripleo06:00
*** yfaykya has quit IRC06:04
*** yfaykya has joined #tripleo06:06
*** jtomasek has joined #tripleo06:09
*** yfaykya has quit IRC06:11
*** mrunge has joined #tripleo06:12
*** yfaykya has joined #tripleo06:12
loki184StevenK : regarding this patch https://review.openstack.org/#/c/103483/ can we name it as cross_arch_binaries06:13
*** yfaykya has quit IRC06:17
*** yfaykya has joined #tripleo06:18
*** jeremydei has joined #tripleo06:19
*** nati_ueno has quit IRC06:20
*** yfaykya has quit IRC06:23
*** yfaykya has joined #tripleo06:25
*** lazy_prince is now known as killer_prince06:28
*** yfaykya has quit IRC06:29
*** yfaykya has joined #tripleo06:31
*** jcoufal has joined #tripleo06:31
*** yfaykya has quit IRC06:35
*** yfaykya has joined #tripleo06:37
*** killer_prince is now known as lazy_prince06:39
tchaypoStevenK: http://logs.openstack.org/72/103072/1/check-tripleo/check-tripleo-overcloud-f20/d18b889/console.html.gz#_2014-06-27_10_41_04_02506:41
*** yfaykya has quit IRC06:42
*** rcarrillocruz has joined #tripleo06:42
*** yfaykya has joined #tripleo06:43
*** rcarrill` has quit IRC06:44
*** dshulyak_ has quit IRC06:44
openstackgerritlokesh s proposed a change to openstack/diskimage-builder: Solves chroot error while building cross arch images  https://review.openstack.org/10348306:45
*** yfaykya has quit IRC06:48
*** yfaykya has joined #tripleo06:49
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-image-elements: Make use of ip route replace in ensure-bridge  https://review.openstack.org/10446106:50
*** jprovazn has joined #tripleo06:53
*** yfaykya has quit IRC06:54
*** yfaykya has joined #tripleo06:55
openstackgerritA change was merged to openstack/tripleo-image-elements: Add nova url to neutron.conf  https://review.openstack.org/10356806:57
*** xuhaiwei has quit IRC07:00
*** yfaykya has quit IRC07:00
*** yfaykya has joined #tripleo07:01
*** yfaykya has quit IRC07:06
*** yfaykya has joined #tripleo07:07
*** yfaykya has quit IRC07:12
*** yfaykya has joined #tripleo07:13
ekarlsois there a bug with the haproxy element ?07:14
ekarlsoit says in the readme that a haproxy.services element can define haproxy.services and it will override the global services but it doesn't seem to work ?07:14
*** yfaykya has quit IRC07:18
*** yfaykya has joined #tripleo07:20
*** nati_ueno has joined #tripleo07:20
*** Penick has quit IRC07:23
*** lsmola__ has joined #tripleo07:24
*** yfaykya has quit IRC07:24
*** akrivoka has joined #tripleo07:25
*** yfaykya has joined #tripleo07:26
*** ifarkas has joined #tripleo07:26
*** markmc has joined #tripleo07:27
*** yfaykya has quit IRC07:30
*** yfaykya has joined #tripleo07:32
*** nati_ueno has quit IRC07:33
ekarlsoseems like we hadn't updateed our heat stuff for the latest haproxy :07:33
*** rushiagr_ has quit IRC07:35
*** lucasagomes has joined #tripleo07:36
*** yfaykya has quit IRC07:37
*** yfaykya has joined #tripleo07:38
*** yfaykya has quit IRC07:42
*** yfaykya has joined #tripleo07:44
openstackgerritNicholas Randon proposed a change to openstack/tripleo-image-elements: Move masquerade host_ip to config.json  https://review.openstack.org/10128007:44
*** yfaykya has quit IRC07:49
*** yfaykya has joined #tripleo07:50
*** akrivoka has quit IRC07:51
*** yfaykya has quit IRC07:55
*** gfidente has joined #tripleo07:55
*** akrivoka has joined #tripleo07:56
*** yfaykya has joined #tripleo07:56
*** yfaykya has quit IRC08:01
*** yfaykya has joined #tripleo08:02
*** nosnos has quit IRC08:05
*** nosnos has joined #tripleo08:05
*** derekh_ has joined #tripleo08:05
*** yfaykya has quit IRC08:07
derekh_Is it possible to add consistent timestamps to the lines in upstart log files ?08:08
*** yfaykya has joined #tripleo08:09
*** jtomasek_ has joined #tripleo08:11
*** martyntaylor has joined #tripleo08:12
*** yfaykya has quit IRC08:14
derekh_specificly I want to add timestamps to the lines in the os-collect-config upstart logs http://logs.openstack.org/78/90278/2/experimental-tripleo/check-tripleo-overcloud-precise/519b0b6/logs/overcloud-controller0-xv57imtwlsmg_logs/os-collect-config.txt.gz08:14
*** athomas has joined #tripleo08:14
*** yfaykya has joined #tripleo08:15
*** rushiagr_ has joined #tripleo08:16
*** yfaykya has quit IRC08:19
*** pblaho has joined #tripleo08:20
*** yfaykya has joined #tripleo08:21
*** jp_at_hp has joined #tripleo08:22
*** jtomasek_ has quit IRC08:22
*** yfaykya has quit IRC08:25
openstackgerritDerek Higgins proposed a change to openstack/tripleo-specs: Tripleo CI improvements  https://review.openstack.org/9502608:26
*** regebro has joined #tripleo08:27
*** yfaykya has joined #tripleo08:27
*** yfaykya has quit IRC08:31
*** loki184 has quit IRC08:32
*** yfaykya has joined #tripleo08:33
*** yfaykya has quit IRC08:38
*** yfaykya has joined #tripleo08:39
*** jcoufal has quit IRC08:41
*** yfaykya has quit IRC08:44
*** jcoufal has joined #tripleo08:44
*** yfaykya has joined #tripleo08:45
*** yfaykya has quit IRC08:50
*** yfaykya has joined #tripleo08:51
*** pelix has joined #tripleo08:55
*** yfaykya has quit IRC08:56
*** yfaykya has joined #tripleo08:58
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133458109:00
uvirtbotLaunchpad bug 1334581 in tripleo "os-collect-config.conf need support to Heat userdata" [Undecided,In progress]09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133504209:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611009:00
uvirtbotLaunchpad bug 1335042 in tripleo "diskimage-builder: extlinux installation failure" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133593209:00
uvirtbotLaunchpad bug 1336110 in tripleo "mysql does not start on boot post-upgrade" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611509:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133501609:00
uvirtbotLaunchpad bug 1335932 in diskimage-builder "/bin/dib-run-parts fails for spanish locale" [Undecided,Fix committed]09:00
uvirtbotLaunchpad bug 1336115 in tripleo "instances no longer running post-upgrade" [Undecided,In progress]09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133364909:00
uvirtbotLaunchpad bug 1335016 in tripleo "diskimage-builder fails for Debian" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133245809:00
uvirtbotLaunchpad bug 1333649 in tripleo "/etc/hosts entries written by hosts element doesn't survive reboot" [Undecided,In progress]09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133252109:00
uvirtbotLaunchpad bug 1332458 in diskimage-builder "Not able to build armhf or arm64 images on amd64 arch systems." [Undecided,In progress]09:00
*** untriaged-bot has quit IRC09:00
uvirtbotLaunchpad bug 1332521 in diskimage-builder "failed to remove `/tmp/image.A2aupYJb/mnt/tmp/in_target.d': Device or resource busy" [Undecided,New]09:00
*** yfaykya has quit IRC09:02
*** loki184 has joined #tripleo09:03
*** yfaykya has joined #tripleo09:04
*** loki184 has quit IRC09:04
openstackgerritLorcan Browne proposed a change to openstack/tripleo-image-elements: Split swift-storage element into separate swift service elements  https://review.openstack.org/10307209:05
*** yfaykya has quit IRC09:08
*** yfaykya has joined #tripleo09:10
derekh_hmm, we seem to be getting a lot of bug 1337230 in the last 2 days09:13
uvirtbotLaunchpad bug 1337230 in tripleo "Loosing connection to keystone during CI" [High,Triaged] https://launchpad.net/bugs/133723009:13
* derekh_ starts digging through logs09:14
*** yfaykya has quit IRC09:14
*** yfaykya has joined #tripleo09:16
*** loki184 has joined #tripleo09:17
*** rcarrill` has joined #tripleo09:17
*** rcarrillocruz has quit IRC09:19
*** yfaykya has quit IRC09:21
*** loki184 has quit IRC09:22
*** yfaykya has joined #tripleo09:22
*** yfaykya has quit IRC09:27
*** yfaykya has joined #tripleo09:28
*** yfaykya has quit IRC09:33
*** rcarrillocruz has joined #tripleo09:33
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-image-elements: ensure cinder-volume is capable of image to volume conversions  https://review.openstack.org/10449409:34
*** yfaykya has joined #tripleo09:34
*** rcarrill` has quit IRC09:35
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-image-elements: ensure cinder-volume is capable of image to volume conversions  https://review.openstack.org/10449409:36
openstackgerritLorcan Browne proposed a change to openstack/tripleo-image-elements: Split swift-storage element into separate swift service elements  https://review.openstack.org/10307209:36
*** rakesh_hs has quit IRC09:38
*** rakesh_hs has joined #tripleo09:39
*** yfaykya has quit IRC09:39
*** lazy_prince is now known as killer_prince09:40
*** yfaykya has joined #tripleo09:40
*** yfaykya has quit IRC09:45
*** yfaykya has joined #tripleo09:47
*** ramishra has quit IRC09:47
*** killer_prince is now known as lazy_prince09:48
*** yfaykya has quit IRC09:51
*** yfaykya has joined #tripleo09:53
*** martyntaylor has quit IRC09:55
*** ramishra has joined #tripleo09:55
*** yfaykya has quit IRC09:57
*** yfaykya has joined #tripleo09:59
openstackgerritMatthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Auto-collect local checks  https://review.openstack.org/9132410:03
*** yfaykya has quit IRC10:03
*** yfaykya has joined #tripleo10:05
*** yfaykya has quit IRC10:10
*** yfaykya has joined #tripleo10:11
proffalkenare we still using launchpad for blueprints or have we migrated to storyboard?10:12
*** rakesh_hs has quit IRC10:14
openstackgerritLorcan Browne proposed a change to openstack/tripleo-image-elements: Add swift-ring element  https://review.openstack.org/10450610:14
*** rakesh_hs has joined #tripleo10:15
*** yfaykya has quit IRC10:16
*** yfaykya has joined #tripleo10:17
*** pblaho has quit IRC10:21
*** yfaykya has quit IRC10:22
*** yfaykya has joined #tripleo10:23
*** marun has joined #tripleo10:28
*** yfaykya has quit IRC10:28
gfidenteping devananda10:29
*** yfaykya has joined #tripleo10:29
openstackgerritDerek Higgins proposed a change to openstack/tripleo-heat-templates: Test Revert "Compute all hosts matrix once"  https://review.openstack.org/10451010:31
*** yfaykya has quit IRC10:34
*** yfaykya has joined #tripleo10:36
*** yfaykya has quit IRC10:41
*** yfaykya has joined #tripleo10:42
*** yfaykya has quit IRC10:46
*** yfaykya has joined #tripleo10:48
*** yfaykya has quit IRC10:52
*** yfaykya has joined #tripleo10:54
*** lazy_prince has quit IRC10:55
*** lazy_prince has joined #tripleo10:56
*** ramishra has quit IRC10:57
*** rcarrillocruz has quit IRC10:57
*** jprovazn has quit IRC10:57
*** jpeeler has quit IRC10:57
*** ramishra has joined #tripleo10:57
*** yfaykya has quit IRC10:59
*** derekh_ has quit IRC10:59
*** rcarrillocruz has joined #tripleo10:59
*** derekh_ has joined #tripleo11:00
*** jprovazn has joined #tripleo11:00
*** yfaykya has joined #tripleo11:00
derekh_jprovazn: see https://review.openstack.org/#/c/104510/ just testing it at the moment, I tried to just revert the commit I wanted but their touch a lot of the same code11:01
derekh_jprovazn: if you want to try a smaller revert then we can try that instead11:02
jprovaznderekh_, ah, interesting. I'll try locally later today11:02
derekh_jprovazn: ok11:03
jprovaznderekh_, but ACK for removing both - trying to keep the dependent patch w/o the all-hosts patch would be very painful11:04
*** yfaykya has quit IRC11:05
*** yfaykya has joined #tripleo11:06
*** yfaykya has quit IRC11:11
openstackgerritA change was merged to openstack/tripleo-incubator: Remove obsolete neutron docs.  https://review.openstack.org/10399511:12
*** rcarrill` has joined #tripleo11:12
*** yfaykya has joined #tripleo11:12
*** rcarrillocruz has quit IRC11:14
*** yfaykya has quit IRC11:17
*** yfaykya has joined #tripleo11:18
*** yfaykya has quit IRC11:23
*** yfaykya has joined #tripleo11:24
openstackgerritMatthew Macdonald-Wallace proposed a change to openstack/tripleo-specs: Re-assert System State  https://review.openstack.org/10452411:25
*** tcammann_ has quit IRC11:26
janghey all. quick poll: is a fresh pull of upstream buildable at the mo? Thought I'd check here if anyone's seeing known problems (neutron-dhcp-agent TRACEs in the seed, no undercloud) before I dive into a colleague's non-working setup11:26
*** yfaykya has quit IRC11:29
*** rcarrillocruz has joined #tripleo11:29
*** jcoufal has quit IRC11:30
*** jcoufal has joined #tripleo11:30
*** yfaykya has joined #tripleo11:31
*** rcarrill` has quit IRC11:32
*** yfaykya has quit IRC11:35
*** yfaykya has joined #tripleo11:37
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of long JQ expression  https://review.openstack.org/9968111:39
*** yfaykya has quit IRC11:41
*** dshulyak_ has joined #tripleo11:41
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of long JQ expression  https://review.openstack.org/9968111:42
*** yfaykya has joined #tripleo11:43
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Wait for os-collect-config to complete on the seed  https://review.openstack.org/9651511:45
*** dshulyak_ has quit IRC11:45
*** yfaykya has quit IRC11:48
*** yfaykya has joined #tripleo11:49
openstackgerritJan Provaznik proposed a change to openstack/os-cloud-config: Generate keystone PKI into JSON file  https://review.openstack.org/10452811:51
*** mkerrin has quit IRC11:51
lazy_princelifeless: there..?11:52
*** jtomasek_ has joined #tripleo11:53
*** jtomasek has quit IRC11:53
*** yfaykya has quit IRC11:54
*** nosnos has quit IRC11:55
*** yfaykya has joined #tripleo11:55
*** dprince has joined #tripleo11:55
openstackgerritJan Provaznik proposed a change to openstack/tripleo-incubator: Generate overcloud keystone keys/certs  https://review.openstack.org/9510111:58
*** nosnos_ has joined #tripleo11:58
*** mkerrin has joined #tripleo11:59
*** yfaykya has quit IRC11:59
*** jcoufal has quit IRC12:01
*** yfaykya has joined #tripleo12:01
*** tzumainn has joined #tripleo12:01
openstackgerritDerek Higgins proposed a change to openstack/tripleo-heat-templates: Revert "Compute all hosts matrix once"  https://review.openstack.org/10451012:02
*** ramishra has quit IRC12:03
*** ramishra has joined #tripleo12:04
*** jcoufal has joined #tripleo12:04
openstackgerritJan Provaznik proposed a change to openstack/diskimage-builder: Remove package mapping for mariadb-rdo-galera-server  https://review.openstack.org/10280812:04
*** yfaykya has quit IRC12:06
*** yfaykya has joined #tripleo12:07
*** ramishra has quit IRC12:08
*** yfaykya has quit IRC12:12
*** pblaho has joined #tripleo12:13
openstackgerritJan Provaznik proposed a change to openstack/tripleo-incubator: Use mariadb by default on fedora  https://review.openstack.org/10281812:13
*** yfaykya has joined #tripleo12:13
*** ramishra has joined #tripleo12:15
proffalkenIs there a specific channel for Ironic or do questions go in here?12:16
lazy_princeproffalken: openstack-ironic12:16
*** yamahata has joined #tripleo12:16
proffalkenlazy_prince: ta, I'll update the wiki page12:16
openstackgerritA change was merged to openstack/tripleo-image-elements: Syntax fix the keepalived docs.  https://review.openstack.org/9979812:17
*** yfaykya has quit IRC12:18
*** yfaykya has joined #tripleo12:20
openstackgerritA change was merged to openstack/tripleo-image-elements: MySQL: Enable passthrough configuration  https://review.openstack.org/10029112:22
*** jcoufal has quit IRC12:24
*** jcoufal has joined #tripleo12:24
*** yfaykya has quit IRC12:24
*** dtantsur has quit IRC12:24
*** dtantsur has joined #tripleo12:24
*** yfaykya has joined #tripleo12:26
*** weshay has joined #tripleo12:26
*** yfaykya has quit IRC12:30
*** ajc_ has quit IRC12:30
*** yfaykya has joined #tripleo12:32
*** athomas_ has joined #tripleo12:34
*** athomas has quit IRC12:34
*** yfaykya has quit IRC12:36
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-image-elements: add DEBUG logging for the django dashboard  https://review.openstack.org/10453612:37
*** zigo has quit IRC12:37
*** zigo has joined #tripleo12:37
*** yfaykya has joined #tripleo12:38
*** zigo has quit IRC12:41
*** zigo has joined #tripleo12:42
*** yfaykya has quit IRC12:43
*** yfaykya has joined #tripleo12:44
*** zigo has quit IRC12:45
*** rlandy has joined #tripleo12:48
*** yfaykya has quit IRC12:49
janglifeless (and anyone who cares) - the eventlet fixes for paramiko et al are slated to land in 0.1612:49
*** yfaykya has joined #tripleo12:50
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Wait for os-collect-config to complete on the seed  https://review.openstack.org/9651512:50
*** yfaykya has quit IRC12:55
*** yfaykya has joined #tripleo12:56
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-image-elements: add INFO level logging for the django dashboard  https://review.openstack.org/10453612:57
*** martyntaylor has joined #tripleo12:58
*** pbelanyi_ is now known as pbelanyi12:59
*** ramishra has quit IRC12:59
*** yfaykya has quit IRC13:01
*** rcarrillocruz has quit IRC13:01
*** rcarrillocruz has joined #tripleo13:02
*** martyntaylor has left #tripleo13:02
*** yfaykya has joined #tripleo13:02
*** yfaykya has quit IRC13:07
*** yfaykya has joined #tripleo13:09
*** yfaykya has quit IRC13:13
gfidentetchaypo, was reviewing this: https://review.openstack.org/#/c/103039 but I'm not sure how that would change the current status where I already have a dedicated ip for haproxy, so need some guidance to understand the but goal13:13
gfidente*understand the goal13:14
*** morazi has joined #tripleo13:14
*** yfaykya has joined #tripleo13:15
gfidenteare you thinking about the ctrlplane being the 'internal' endpoint and the additional public VIP (ideally not on br-ex) being the 'public' endpoint?13:15
gfidentetchaypo, ^^13:15
*** amitpp has quit IRC13:17
*** pcrews has joined #tripleo13:18
*** yfaykya has quit IRC13:19
*** yfaykya has joined #tripleo13:21
*** julim has joined #tripleo13:21
*** yfaykya has quit IRC13:25
*** yfaykya has joined #tripleo13:27
*** yfaykya has quit IRC13:32
*** moorryan has quit IRC13:32
*** mrunge has quit IRC13:33
*** yfaykya has joined #tripleo13:33
openstackgerritPetr Blaho proposed a change to openstack/tuskar: Updates db module from oslo-incubator  https://review.openstack.org/10454813:34
*** jdob has joined #tripleo13:36
*** nosnos_ has quit IRC13:37
*** yfaykya has quit IRC13:38
*** yfaykya has joined #tripleo13:39
*** rakesh_hs has quit IRC13:42
*** yfaykya has quit IRC13:44
derekh_I believe we have a regression causing a lot (not all) of overcloud failures, can I get some eyes on this revert to try and help the situation https://review.openstack.org/#/c/104510/13:45
*** yfaykya has joined #tripleo13:45
derekh_it actually reverts 2 patches cause one was right on top of the other and I aint familliar enough with the heat templates at the moment to sort out the various conflicts, if somebody else want to try that then fire ahead but we should do something soon13:47
derekh_as the current failure rate if way too high13:47
derekh_SpamapS: your probably not here yet but ^13:49
*** nosnos has joined #tripleo13:49
*** yfaykya has quit IRC13:50
* jprovazn tries reproduce locally13:51
*** yfaykya has joined #tripleo13:51
*** rushiagr_ has quit IRC13:55
*** yfaykya has quit IRC13:56
*** yfaykya has joined #tripleo13:57
*** zigo has joined #tripleo13:58
*** nosnos has quit IRC13:59
*** yfaykya has quit IRC14:02
*** dkehnx has quit IRC14:02
*** yfaykya has joined #tripleo14:04
*** jang has quit IRC14:05
*** yfaykya has quit IRC14:08
*** yfaykya has joined #tripleo14:10
*** ramishra has joined #tripleo14:14
*** yfaykya has quit IRC14:15
*** yfaykya has joined #tripleo14:16
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of JQ for novaBM and ironic  https://review.openstack.org/10455914:17
*** ramishra has quit IRC14:18
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of long JQ expression  https://review.openstack.org/9968114:20
*** moorryan has joined #tripleo14:20
*** yfaykya has quit IRC14:20
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of JQ for novaBM and ironic  https://review.openstack.org/10455914:21
*** andreaf has joined #tripleo14:22
*** yfaykya has joined #tripleo14:22
*** dkehn_ has joined #tripleo14:22
tzumainnshadower, ping14:22
*** dkehn_ is now known as dkehnx14:23
shadowertzumainn: pong14:23
*** ramishra has joined #tripleo14:24
tzumainnshadower, heya, I have a couple of questions I was hoping you might know the answer to, have a moment?14:24
shadowertzumainn: yep!14:24
tzumainnshadower, okay, so the first - now that tuskar-ui is going to treat flavor/image the same as any other parameter to the role heat template - will those parameters have consistent names?14:25
shadowertzumainn: I think they should14:26
shadowertzumainn: you mean like "controller_image" vs. "swift_image"14:26
tzumainnshadower, yep, exactly - optimally, there's a way to derive them from the role14:27
*** yfaykya has quit IRC14:27
tzumainnalthough, hm, that implies that the role name is limited14:27
shadowertzumainn: so, I think t-h-t should have it consistent. But the question is what is tuskar going to do with custom non-tht roles14:27
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Wait for os-collect-config to complete on the seed  https://review.openstack.org/9651514:28
tzumainnshadower, in other words, are we going to enforce the use of certain parameter names?14:28
tzumainnjdob, any thought as to ^ ?14:28
*** yfaykya has joined #tripleo14:28
tzumainnI think it's reasonable. . . I think. . . but maybe it's not?14:28
shadowertzumainn: I can't speak for the entire tripleo core team but I'd argue for it14:28
jdobmy thinking was that you'd use just "image" in all cases14:29
jdobfiguring the templates can be stand alone14:29
shadowerjdob: I think he means in the master template14:29
jdobwhen tuskar combines them with environments and a master template file, it'll prefix them14:29
jdobwith a similar namespace as used in the resource_registry14:29
jdobso, for example:14:29
jdobTuskar::Foo : foo.yaml14:29
jdoband in the master template14:29
shadowerjdob: so tuskar won't be using the tht master template at all? Just the roles?14:29
jdobFoo-image14:29
tzumainnjdob, shadower, so the base role template, standalone, might have something like 'image', 'flavor', and then tuskar would prefix them when combining them?14:29
jdobFoo-flavor14:29
jdoband same for the outputs14:29
jdobFoo-ip14:29
jdobshadower: right, generating its own master template14:30
shadowerright fair14:30
shadowertzumainn: so there you go ^14:30
jdobthis is good, this is exactly what I wanted to run past you guys this morning14:30
jdobi just wanted to clean up the output from the code I had first14:30
tzumainnshadower, okay - auxillary question - will the template have flavor and image parameters, or flavor_id/image_id?14:30
jdobmore generally, we should define those conventions14:31
jdobso that custom roles can fit into the same logic14:31
tzumainnjdob, yep, that makes a ton of sense14:31
shadowertzumainn: so given where the conversation went, define "the template" -- a role template or the master one?14:31
jdobhe means role one14:31
shadoweragreed with having conventions14:31
shadowerright14:31
shadowerI'd say "flavour" and "image" ;-)14:31
jdobah shit14:32
shadowerbut yea, I'd go away with the "_id" suffix personally14:32
jdobone of our most important words and its one of those fuzzy spelling ones :)14:32
shadowerhaha14:32
tzumainnokay, so I'm passing in and retrieving the actual nova flavor?  not an id?14:32
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Wait for os-collect-config to complete on the seed  https://review.openstack.org/9651514:32
jdobi once read an article on how to write bad code and it talked about interchanging spellings like that within the same file14:32
shadowertzumainn: Heat accepts both the name and id14:32
shadowertzumainn: so it's just the matter of what we call it14:33
*** yfaykya has quit IRC14:33
tzumainnshadower, ah, got it14:33
shadowercould be image_id_or_name of course14:33
jdobimage_thing14:33
tzumainnshadower, okay, completely different question, not sure you're the one to ask, but. . . when heat provisions a server as a resource, is it possible to have it populate various metadata fields on the server?14:34
shadowertzumainn: which metadata fields?14:34
shadowerin ironic?14:34
shadoweror is that a generic nova tihng?14:34
*** yfaykya has joined #tripleo14:34
tzumainnI was thinking the nova Server or Instance or whatever they call it now14:34
shadowertzumainn: still not sure I follow though. Can you give me an example?14:35
tzumainnyep, so. . .14:35
tzumainnshadower, okay, so in the case where tuskar-ui is looking at a bunch of nodes, and wants to know what kind of role has been provisioned on each of them14:37
tzumainnwe would want to go from node -> instance -> role; but the mapping to role is actually done at the heat resource level, using the resource_type attribute (I think)14:38
*** yfaykya has quit IRC14:39
openstackgerritJulia Kreger proposed a change to openstack/tripleo-image-elements: Add detection of mysql restart failure  https://review.openstack.org/10456914:39
*** dkehn_ has joined #tripleo14:39
shadowertzumainn: oh. Well so remember that Heat doesn't know a thing about roles. That's a Tuskar concept. Heat just sees resources and their types. The fact that in our case we'll create a custom type for each role is just how we happen to structure things14:39
tzumainnshadower, right, that's fine14:40
tzumainnI'm wondering if there's a way to pass that resource_type info into the nova server instance14:40
tzumainnand trying to find an example of nova server metadata, which I can't for some reason14:40
*** yfaykya has joined #tripleo14:40
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of JQ for novaBM and ironic  https://review.openstack.org/10455914:41
*** dkehn_ has quit IRC14:42
*** dkehn has quit IRC14:42
*** dkehn_ has joined #tripleo14:42
shadowertzumainn: yeah I'm not sure about nova. iirc ironic will (maybe does already?) have support for node-level metadata14:42
*** ramishra has quit IRC14:43
*** dkehnx has quit IRC14:43
shadowertzumainn: either way, I don't think Heat supports setting that kind of metadata now14:43
*** rcarrill` has joined #tripleo14:43
*** dkehn_ is now known as dkehnx14:43
*** dkehn has joined #tripleo14:43
tzumainnshadower, ah, okay - thanks, that's what I wanted to know!14:44
shadowertzumainn: I suppose you could write the Heat template so it returns all the resource -> role mappings as the output14:44
*** yfaykya has quit IRC14:45
tzumainnshadower, oh, I'm not worried about that14:45
shadowerokay14:45
*** rcarrillocruz has quit IRC14:45
tzumainnbut starting from node/server, without any particular stack in mind, I'm wondering how I can get the heat resource_type14:45
tzumainnmaybe that's the better question - I can go from resource > server > node pretty easily, is there any way of doing the opposite, esp if I don't know what the stack is?14:46
*** yfaykya has joined #tripleo14:47
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of JQ for novaBM and ironic  https://review.openstack.org/10455914:47
shadowertzumainn: right now, I don't think so. You'd have to get that mapping and store it somewhere. Or convince Heat to store it somewhere on the node14:48
tzumainnshadower, yeah, the latter approach was what made me wonder if it was possible to store some heat-related metadata in the node or server14:49
tzumainnbut it appears not?14:49
shadoweryeah14:49
tzumainnokay, cool - thanks!14:49
derekh_jprovazn: occurrences of this happening for the last 7 days http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiKlVuYWJsZSB0byBlc3RhYmxpc2ggY29ubmVjdGlvbipcIiBBTkQgYnVpbGRfbmFtZToqdHJpcGxlbyoiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDQzNzgyOTcwNjZ914:50
*** jprovazn has left #tripleo14:50
*** jprovazn has joined #tripleo14:51
*** yfaykya has quit IRC14:51
*** ddieterly has joined #tripleo14:52
*** yfaykya has joined #tripleo14:53
*** ramishra has joined #tripleo14:53
openstackgerritAna Krivokapic proposed a change to openstack/tuskar-ui: Ceilometer graphs for nodes  https://review.openstack.org/10418814:53
jprovaznderekh_, thanks14:53
*** rdopieralski has quit IRC14:54
*** yfaykya has quit IRC14:57
*** yfaykya has joined #tripleo14:59
*** lazy_prince is now known as killer_prince14:59
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133458115:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133504215:00
uvirtbotLaunchpad bug 1334581 in tripleo "os-collect-config.conf need support to Heat userdata" [Undecided,In progress]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133510515:00
uvirtbotLaunchpad bug 1335042 in tripleo "diskimage-builder: extlinux installation failure" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611015:00
uvirtbotLaunchpad bug 1335105 in tuskar-ui "Devtest init script: sourcing tripleorc fails" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133593215:00
uvirtbotLaunchpad bug 1336110 in tripleo "mysql does not start on boot post-upgrade" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611515:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133501615:00
uvirtbotLaunchpad bug 1335932 in diskimage-builder "/bin/dib-run-parts fails for spanish locale" [Undecided,Fix committed]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133364915:00
uvirtbotLaunchpad bug 1336115 in tripleo "instances no longer running post-upgrade" [Undecided,In progress]15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133245815:00
uvirtbotLaunchpad bug 1335016 in tripleo "diskimage-builder fails for Debian" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133252115:00
uvirtbotLaunchpad bug 1333649 in tripleo "/etc/hosts entries written by hosts element doesn't survive reboot" [Undecided,In progress]15:00
uvirtbotLaunchpad bug 1332458 in diskimage-builder "Not able to build armhf or arm64 images on amd64 arch systems." [Undecided,In progress]15:00
*** untriaged-bot has quit IRC15:00
uvirtbotLaunchpad bug 1332521 in diskimage-builder "failed to remove `/tmp/image.A2aupYJb/mnt/tmp/in_target.d': Device or resource busy" [Undecided,New]15:00
*** rcarrillocruz has joined #tripleo15:02
*** yfaykya has quit IRC15:03
*** rcarrill` has quit IRC15:04
*** yfaykya has joined #tripleo15:05
openstackgerritAlexis Lee proposed a change to openstack/tripleo-specs: Control mechanism for os-apply-config  https://review.openstack.org/9235615:05
*** ramishra has quit IRC15:08
*** yfaykya has quit IRC15:09
devanandagfidente: pong15:11
*** yfaykya has joined #tripleo15:11
gfidentedevananda, wanted to ask you to comment https://review.openstack.org/#/c/96184/15:11
gfidenteit seems to me the less intrusive way to sort that in nova.bm15:12
gfidenteuntil we can drop it from tripleo15:12
gfidentein favour of ironic15:12
*** pbelanyi has quit IRC15:13
*** yfaykya has quit IRC15:16
*** pblaho has quit IRC15:17
*** yfaykya has joined #tripleo15:17
devanandagfidente: just reviewed the bug -- yes, that looks reasonable to me15:19
gfidentedevananda, thanks!15:19
*** yfaykya has quit IRC15:21
*** ifarkas has quit IRC15:22
*** jpeeler has joined #tripleo15:23
*** yfaykya has joined #tripleo15:23
*** lynxman has quit IRC15:24
*** ifarkas has joined #tripleo15:24
*** lynxman has joined #tripleo15:25
*** yfaykya has quit IRC15:28
openstackgerritA change was merged to openstack/tripleo-incubator: Fix Ceilometer undercloud endpoint registration when UI not specified  https://review.openstack.org/10159615:28
*** yfaykya has joined #tripleo15:29
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-image-elements: ensure cinder-volume is capable of image to volume conversions  https://review.openstack.org/10449415:31
*** jcoufal has quit IRC15:31
SpamapSlifeless: yes, yaml null would be       a-key: null    <-- doc['a-key'] == None15:33
*** yfaykya has quit IRC15:34
SpamapSderekh_: that patch you're reverting fixes a huge scale problem. :(15:35
*** yfaykya has joined #tripleo15:36
SpamapSso I think we need to fix the signal handling to only send a signal after the all hosts matrix is done, since the change triggers an occ run.15:36
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Add new package-installs element  https://review.openstack.org/10458615:37
*** yfaykya has quit IRC15:40
derekh_SpamapS: Its currently causing a lot of failures in CI, I fear the solution you suggest will take time to get right, so I'm thinking revert and fix is a better solution15:40
derekh_SpamapS: infact would it be better to not start o-r-c at all until the hosts matrix is done15:41
*** yfaykya has joined #tripleo15:42
SpamapSderekh_: I'm not saying we can't land the revert. I'm saying :(15:42
derekh_SpamapS: gotchya15:42
SpamapSderekh_: no, orc running before 100% of all values are available is necessary as we do things half-way that enable further steps unfortunately.15:43
derekh_SpamapS: ahh ok15:43
SpamapSderekh_: but we should hold back the stack reaching COMPLETE.15:43
SpamapSderekh_: which isn't actually that hard at all.15:44
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Add rdo-icehouse element  https://review.openstack.org/10459215:44
SpamapSderekh_: but I'm fine w/ landing revert.. fixing this up.. and then re-landing compute all matrix once15:44
SpamapSderekh_: the "hard" part is that we need to change our signal handling so we can have multiple signals, not just one.15:44
openstackgerritJon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-image-elements: Add new sosreport element  https://review.openstack.org/10459315:44
SpamapSalso the hard part is "zomg SpamapS has too many plates spinning"15:45
derekh_SpamapS: ok, sounds good to me15:45
derekh_SpamapS: on the easy part15:45
derekh_:-)15:45
derekh_SpamapS: so can heat currently accept more then one signal type ?15:46
SpamapSderekh_: type is not the right term.15:46
derekh_value ?15:46
*** yfaykya has quit IRC15:47
SpamapSderekh_: I don't know what you're getting at, but there's no such thing as a "signal type" that I am aware of, so I am confused.15:47
*** yfaykya has joined #tripleo15:48
derekh_SpamapS: you said we need to change our signal handling so we can have multiple signals, essentially I'm wondering if that requires a heat change ?15:49
SpamapSderekh_: definitely not!15:49
openstackgerritNicholas Randon proposed a change to openstack/tripleo-image-elements: Move masquerade host_ip to config.json  https://review.openstack.org/10128015:49
SpamapSderekh_: we just need to take the code in the o-r-c element's o-r-c script that sends signals, and turn it into a tool that will easily send signals.. and let each element that expects a signal, send a signal.15:50
derekh_SpamapS: ok, so if you have a lot of stuff on your plate I can take a look at doing it (If ya want)15:50
SpamapSderekh_: sure, we might need to chat about how this should work. I don't want there to be a signal for everything either.. ;)15:51
derekh_SpamapS: on another note, do you know if its possible to add timestamps to lines in upstart log files ?15:51
SpamapSsince we're already crushing Heat with o-c-c polling ;)15:51
SpamapSderekh_: it is not easy. Things that use upstart log files should be adding their own timestamps.15:52
derekh_SpamapS: ok, if you want to jot down roughly what your thinking, I'll take a look and can get back to you with questions15:52
*** yfaykya has quit IRC15:52
SpamapSderekh_: otherwise, | logger15:52
SpamapSand let syslog do the dirty work15:52
derekh_SpamapS: ok, so where is is coming from, I recently added os-collect-config logs to be indexed by logstash15:53
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Wait for os-collect-config to complete on the seed  https://review.openstack.org/9651515:53
derekh_SpamapS: but to make a long story short, its pretty much expecting lines to contain timestamps15:53
SpamapSderekh_: so there are two avenues. One is to add multi-signal support, and add one more signal for the all-hosts matrix. The other shorter avenue is to change the templates so that the all hosts matrix deployments all provide 'completion-signal'.15:53
*** rcarrill` has joined #tripleo15:53
*** eghobo has joined #tripleo15:54
*** yfaykya has joined #tripleo15:54
SpamapSderekh_: ah, so there's an expectation of uniformity there?15:54
derekh_SpamapS: what I have done is broke on both distros, I know how to sort out F20 but can see a way to fix ubuntu easily15:54
derekh_SpamapS: yup15:54
SpamapSderekh_: we could have o-c-c add them. Would it matter that some items would have double timestamps?15:55
derekh_SpamapS: I don't think so, we currently have double times stamps in the F20 logs15:55
*** rcarrillocruz has quit IRC15:55
SpamapSderekh_: we could also be a little smart and add timestamps to lines that don't have them only.15:56
openstackgerritJon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-incubator: Add the select-cloud script  https://review.openstack.org/8280715:56
*** nati_ueno has joined #tripleo15:56
SpamapSderekh_: _OR_ .. just punt to syslog15:56
SpamapSbut that carries that "syslog drops things because it gets busy" caveat15:57
derekh_SpamapS: hmm, if we punt to syslog we wont have per service logs anymore will we ?  suppose we could seperate them again like we do for systemd15:57
SpamapSderekh_: the way I see it, we've kind of abused the upstart console log. It let us be lazy and not have a solution. But arguably syslog is a more scalable production solution as boxes can be configured easily to ship those logs elsewhere or one of the other bazillion things rsyslog can do.15:58
*** yfaykya has quit IRC15:58
*** eghobo has quit IRC15:59
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Add rdo-release element  https://review.openstack.org/10459815:59
*** yfaykya has joined #tripleo16:00
derekh_SpamapS: ya, I like the idea of getting away from the upstart logs, syslog will make things a bit more consistent between distros16:00
SpamapSderekh_: anyway, another option is to have the services log directly to files.16:02
SpamapSderekh_: but that doesn't really solve the o-c-c problem. hrm16:02
SpamapSderekh_: I got no good answers.16:03
derekh_SpamapS: ok, I'll try the syslog thing and see where that takes me16:04
derekh_SpamapS: then I'll take a look at the signals problem16:04
*** eghobo has joined #tripleo16:04
SpamapSderekh_: re the all hosts matrix problem. Looking closer at the templates, I think we can solve this with a really tiny change.16:04
SpamapSderekh_: before that revert lands.. let me push something up..16:05
*** yfaykya has quit IRC16:05
derekh_SpamapS: ok, go for it16:05
*** rcarrill` has quit IRC16:06
*** yfaykya has joined #tripleo16:06
*** ramishra has joined #tripleo16:09
SpamapSderekh_: taking longer to type commit message than do the patch.. doh16:09
vinshHey folks.. what does the "utf8" on "mysql://neutron:unset@localhost/ovs_neutron?charset=utf8" do?16:09
vinshits been dropped in: https://review.openstack.org/#/c/101080/216:10
vinshjust questions internally on what that string even does..16:10
*** rcarrillocruz has joined #tripleo16:11
openstackgerritClint 'SpamapS' Byrum proposed a change to openstack/tripleo-heat-templates: Make stack complete depend on all nodes  https://review.openstack.org/10460216:11
*** yfaykya has quit IRC16:11
SpamapSvinsh: It is just an overstatement of what should be the default anyway. Some distros still default to latin1. AFAIK, all modern distros use a configuration which defaults to utf8.16:12
vinshACK. TY.16:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-specs: Configurable /mnt/state  https://review.openstack.org/9487616:12
*** yfaykya has joined #tripleo16:12
*** akrivoka has quit IRC16:13
*** ramishra has quit IRC16:13
SpamapSvinsh: I don't see where utf8 is dropped in that patch though.16:14
*** eguz has joined #tripleo16:15
vinshAH, I mean that.. where its referenced in other templates/files.. it includes that on the end16:15
vinshso, like in the ml2 conf file I think16:16
vinshsome other template.. like overcloud-compute16:16
vinshNo biggie, we just didn't get it16:16
*** yfaykya has quit IRC16:17
*** nati_ueno has quit IRC16:17
*** nati_uen_ has joined #tripleo16:17
*** eguz has quit IRC16:17
*** yfaykya has joined #tripleo16:18
*** eghobo has quit IRC16:19
openstackgerritA change was merged to openstack/tuskar-ui: Fixed docstrings to match api reorganization  https://review.openstack.org/10357916:20
*** lucasagomes_ has joined #tripleo16:21
*** lucasagomes_ has quit IRC16:22
*** lucasagomes has quit IRC16:22
*** yfaykya has quit IRC16:23
openstackgerritJames Slagle proposed a change to openstack/tripleo-specs: Configurable /mnt/state  https://review.openstack.org/9487616:23
*** dshulyak_ has joined #tripleo16:24
*** yfaykya has joined #tripleo16:25
openstackgerritAlexis Lee proposed a change to openstack/tripleo-specs: Control mechanism for os-apply-config  https://review.openstack.org/9235616:25
*** ifarkas has quit IRC16:27
*** athomas_ has quit IRC16:27
*** yfaykya has quit IRC16:29
*** rcarrill` has joined #tripleo16:30
*** lsmola__ has quit IRC16:30
*** yfaykya has joined #tripleo16:31
*** markmc has quit IRC16:31
*** rcarrillocruz has quit IRC16:32
*** regebro has quit IRC16:34
*** yfaykya has quit IRC16:35
*** yfaykya has joined #tripleo16:37
*** amitpp has joined #tripleo16:39
*** yfaykya has quit IRC16:42
*** yfaykya has joined #tripleo16:43
*** yfaykya has quit IRC16:47
*** dshulyak_ has quit IRC16:47
*** yfaykya has joined #tripleo16:49
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Add global exclusions to dib-lint  https://review.openstack.org/10425516:51
*** yfaykya has quit IRC16:54
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: Log tripleo services to syslog  https://review.openstack.org/10461916:54
*** rcarrill` has quit IRC16:54
*** yfaykya has joined #tripleo16:55
*** derekh_ has quit IRC16:58
*** yfaykya has quit IRC17:00
*** rcarrillocruz has joined #tripleo17:00
*** yfaykya has joined #tripleo17:01
*** yfaykya has quit IRC17:06
*** yfaykya has joined #tripleo17:07
*** ramishra has joined #tripleo17:10
*** rushiagr_ has joined #tripleo17:10
*** yfaykya has quit IRC17:12
*** yfaykya has joined #tripleo17:14
*** ramishra has quit IRC17:14
*** bnemec is now known as bnemec_appt17:15
*** yfaykya has quit IRC17:18
*** yfaykya has joined #tripleo17:20
*** blamar has joined #tripleo17:20
*** jp_at_hp has quit IRC17:20
openstackgerritClint 'SpamapS' Byrum proposed a change to openstack/os-collect-config: Make heat the default collection method  https://review.openstack.org/9452717:22
openstackgerritClint 'SpamapS' Byrum proposed a change to openstack/os-collect-config: Cache auth_ref from keystoneclient  https://review.openstack.org/9452617:22
*** yfaykya has quit IRC17:24
openstackgerrittherese-mchale proposed a change to openstack/tripleo-heat-templates: Allow the database password to be configurable  https://review.openstack.org/10462017:25
*** yfaykya has joined #tripleo17:26
*** yfaykya has quit IRC17:30
*** nati_uen_ has quit IRC17:31
*** nati_ueno has joined #tripleo17:31
*** yfaykya has joined #tripleo17:32
*** rcarrill` has joined #tripleo17:36
*** rcarrillocruz has quit IRC17:36
*** yfaykya has quit IRC17:36
*** yfaykya has joined #tripleo17:38
*** blamar has quit IRC17:39
*** rushiagr_ has quit IRC17:42
*** yfaykya has quit IRC17:43
*** yfaykya has joined #tripleo17:44
*** amitpp has quit IRC17:45
*** yfaykya has quit IRC17:49
*** yfaykya has joined #tripleo17:50
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: ensure-bridge: bring up bridges first on Debian  https://review.openstack.org/10462517:50
*** dkehn_ has joined #tripleo17:53
openstackgerrittherese-mchale proposed a change to openstack/tripleo-incubator: Setup MySQL database password  https://review.openstack.org/10462817:53
*** dkehn has quit IRC17:55
*** yfaykya has quit IRC17:55
dprincemkerrin: I'm curious to see what you think of https://review.openstack.org/10462517:55
*** dkehnx has quit IRC17:56
*** yfaykya has joined #tripleo17:56
*** dkehn has joined #tripleo17:57
*** marun is now known as marun_afk17:57
*** yfaykya has quit IRC18:01
*** yfaykya has joined #tripleo18:02
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Update RHEL7 element since release  https://review.openstack.org/10429618:03
*** yfaykya has quit IRC18:07
*** yfaykya has joined #tripleo18:09
openstackgerritJames Slagle proposed a change to openstack/tripleo-specs: Add a local datasource collector to occ  https://review.openstack.org/10096518:13
*** yfaykya has quit IRC18:14
openstackgerritJames Slagle proposed a change to openstack/tripleo-specs: Add a local datasource collector to occ  https://review.openstack.org/10096518:14
*** yfaykya has joined #tripleo18:15
*** yfaykya has quit IRC18:20
*** yfaykya has joined #tripleo18:21
*** yfaykya has quit IRC18:26
*** yfaykya has joined #tripleo18:27
*** yfaykya has quit IRC18:31
openstackgerritEd Bak proposed a change to openstack/tripleo-incubator: Provide ability to move the gateway away from the seed  https://review.openstack.org/10464118:32
*** morazi has quit IRC18:32
*** nati_ueno has quit IRC18:32
*** yfaykya has joined #tripleo18:33
greghaynesslagle: up for a re-review on https://review.openstack.org/#/c/103692/5/specs/juno/ssl_pki.rst ?18:36
*** ddieterly has quit IRC18:36
*** yfaykya has quit IRC18:38
*** yfaykya has joined #tripleo18:39
*** ramishra has joined #tripleo18:39
*** eghobo has joined #tripleo18:39
SpamapShrm18:40
SpamapShttps://jenkins03.openstack.org/job/check-tripleo-overcloud-f20/530/console .. no signal coming in18:40
* SpamapS starts local test18:40
*** yfaykya has quit IRC18:44
*** ramishra has quit IRC18:44
*** yfaykya has joined #tripleo18:45
slaglegreghaynes: sure, will do18:46
greghaynestyty18:47
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Allow single node mysql clusters to restart  https://review.openstack.org/10445518:48
*** yfaykya has quit IRC18:50
*** yfaykya has joined #tripleo18:51
*** yfaykya has quit IRC18:56
*** yfaykya has joined #tripleo18:58
Nggreghaynes: reviewed 104455, looks like it's missing something19:01
greghaynesNg: hah19:02
greghaynesminor detail19:02
*** pelix has quit IRC19:02
*** yfaykya has quit IRC19:02
*** ccrouch has quit IRC19:03
*** yfaykya has joined #tripleo19:04
*** dshulyak_ has joined #tripleo19:04
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Allow single node mysql clusters to restart  https://review.openstack.org/10445519:06
greghaynesNg: If youall have a setup to test that it would be win19:06
Nggreghaynes: we do!19:06
greghaynesw00t19:07
Ngadam_g: GheRivero ^^ should fix one of the upgrade bugs :)19:07
*** yfaykya has quit IRC19:09
*** yfaykya has joined #tripleo19:10
*** dshulyak_ has quit IRC19:12
openstackgerritEd Bak proposed a change to openstack/tripleo-incubator: Provide ability to move the gateway away from the seed  https://review.openstack.org/10464119:13
openstackgerritEd Bak proposed a change to openstack/tripleo-incubator: Provide ability to move the gateway away from the seed  https://review.openstack.org/10464119:15
adam_gNg, cool. our internal pypi mirrors are blocked again. i have new gigabbytes of ram that arrived i should be able to test soon at home19:15
*** yfaykya has quit IRC19:15
Ngnice19:15
greghaynesSpamapS: Is it possible that elements in our hosts: list come in piecemeal?19:16
*** yfaykya has joined #tripleo19:16
*** jang has joined #tripleo19:17
* greghaynes thinks this may be possbile19:19
*** yfaykya has quit IRC19:20
*** yfaykya has joined #tripleo19:22
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Allow single node mysql clusters to restart  https://review.openstack.org/10445519:23
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Code update for new Tuskar API  https://review.openstack.org/10466019:24
greghaynesSpamapS: hrm, your change succeeded for me :/19:24
greghaynesthe one that timed out in jenkins19:24
*** jang1 has joined #tripleo19:26
*** robsparker has quit IRC19:26
*** yfaykya has quit IRC19:27
*** yfaykya has joined #tripleo19:28
greghayneshuh, http://logs.openstack.org/02/104602/1/check-tripleo/check-tripleo-overcloud-f20/a682ea4/console.html doesnt grab any of the controller0 logs19:32
*** yfaykya has quit IRC19:33
*** yfaykya has joined #tripleo19:34
*** robsparker has joined #tripleo19:37
*** yfaykya has quit IRC19:39
*** ramishra has joined #tripleo19:40
*** yfaykya has joined #tripleo19:40
*** ramishra has quit IRC19:44
*** yfaykya has quit IRC19:45
*** yfaykya has joined #tripleo19:47
*** pblaho has joined #tripleo19:49
GheRivero@Ng @greghaynes Sure! I was already testing it.19:50
*** yfaykya has quit IRC19:51
*** blamar has joined #tripleo19:51
*** Penick has joined #tripleo19:51
*** yfaykya has joined #tripleo19:53
*** morazi has joined #tripleo19:55
tchaypoThanks gfidente , I'll tweak the message today19:55
*** Penick has quit IRC19:57
*** yfaykya has quit IRC19:57
*** yfaykya has joined #tripleo19:59
*** blamar has quit IRC20:03
*** yfaykya has quit IRC20:04
*** dkehn__ has joined #tripleo20:04
*** yfaykya has joined #tripleo20:05
*** dprince has quit IRC20:07
*** dkehn_ has quit IRC20:07
*** pblaho has quit IRC20:08
*** pblaho has joined #tripleo20:08
*** yfaykya has quit IRC20:09
*** yfaykya has joined #tripleo20:11
*** rcarrillocruz has joined #tripleo20:12
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Remove map-packages from RHEL element  https://review.openstack.org/10466920:13
*** rcarrill` has quit IRC20:14
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Adds RHEL common element  https://review.openstack.org/10467020:15
*** yfaykya has quit IRC20:16
*** yfaykya has joined #tripleo20:17
*** dkehn__ is now known as dkehnx20:19
*** pblaho has quit IRC20:21
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Update RHEL7 element since release  https://review.openstack.org/10429620:22
*** yfaykya has quit IRC20:22
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Update RHEL7 element since release  https://review.openstack.org/10429620:23
*** yfaykya has joined #tripleo20:23
*** ccrouch has joined #tripleo20:25
*** jprovazn has quit IRC20:25
*** bnemec_appt is now known as bnemec20:27
*** yfaykya has quit IRC20:28
*** yfaykya has joined #tripleo20:29
*** marun_afk is now known as marun20:32
*** yfaykya has quit IRC20:34
*** yfaykya has joined #tripleo20:36
openstackgerritJames Slagle proposed a change to openstack/tripleo-incubator: Update write-tripleorc  https://review.openstack.org/9728420:36
openstackgerritJames Slagle proposed a change to openstack/tripleo-incubator: Add vars to devtest_variables.sh  https://review.openstack.org/9728320:36
*** yfaykya has quit IRC20:40
*** ramishra has joined #tripleo20:41
*** yfaykya has joined #tripleo20:42
*** yfaykya has quit IRC20:46
*** ramishra has quit IRC20:46
*** yfaykya has joined #tripleo20:48
*** yfaykya has quit IRC20:52
*** yfaykya has joined #tripleo20:54
*** jdob has quit IRC20:55
*** yfaykya has quit IRC20:58
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133458121:00
uvirtbotLaunchpad bug 1334581 in tripleo "os-collect-config.conf need support to Heat userdata" [Undecided,In progress]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133504221:00
uvirtbotLaunchpad bug 1335042 in tripleo "diskimage-builder: extlinux installation failure" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133510521:00
uvirtbotLaunchpad bug 1335105 in tuskar-ui "Devtest init script: sourcing tripleorc fails" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611021:00
uvirtbotLaunchpad bug 1336110 in tripleo "mysql does not start on boot post-upgrade" [High,New]21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133593221:00
uvirtbotLaunchpad bug 1335932 in diskimage-builder "/bin/dib-run-parts fails for spanish locale" [Undecided,Fix committed]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133611521:00
uvirtbotLaunchpad bug 1336115 in tripleo "instances no longer running post-upgrade" [Undecided,In progress]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133747521:00
uvirtbotLaunchpad bug 1337475 in tripleo "devtest can't move the gateway to another ip address other than the seed" [Undecided,In progress]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133501621:00
uvirtbotLaunchpad bug 1335016 in tripleo "diskimage-builder fails for Debian" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133364921:00
*** yfaykya has joined #tripleo21:00
uvirtbotLaunchpad bug 1333649 in tripleo "/etc/hosts entries written by hosts element doesn't survive reboot" [Undecided,In progress]21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133245821:00
uvirtbotLaunchpad bug 1332458 in diskimage-builder "Not able to build armhf or arm64 images on amd64 arch systems." [Undecided,In progress]21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/133252121:00
uvirtbotLaunchpad bug 1332521 in diskimage-builder "failed to remove `/tmp/image.A2aupYJb/mnt/tmp/in_target.d': Device or resource busy" [Undecided,New]21:00
*** untriaged-bot has quit IRC21:00
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Add rdo-icehouse element  https://review.openstack.org/10459221:01
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Add new package-installs element  https://review.openstack.org/10458621:01
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Update fedora-rdo-icehouse element  https://review.openstack.org/10468621:01
*** yfaykya has quit IRC21:04
*** yfaykya has joined #tripleo21:06
*** yfaykya has quit IRC21:11
*** yfaykya has joined #tripleo21:12
*** yfaykya has quit IRC21:17
*** yfaykya_ has joined #tripleo21:19
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Add rhel7-rdo-icehouse element  https://review.openstack.org/10468921:19
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Add rhel7-rdo-icehouse element  https://review.openstack.org/10468921:23
*** jang has quit IRC21:24
*** yfaykya_ has quit IRC21:24
*** yfaykya has joined #tripleo21:26
*** rcarrill` has joined #tripleo21:28
SpamapS2014-07-03 21:24:27.371 11841 TRACE nova.api.openstack ConnectionFailed: Connection to neutron failed: Maximum attempts reached21:28
SpamapSwhats weird about that is that neutron was running, no errors reported, 4 minutes before21:30
*** rcarrillocruz has quit IRC21:30
openstackgerritRichard Su proposed a change to openstack/tripleo-heat-templates: Make swift server ports configurable  https://review.openstack.org/10469221:30
greghaynesSpamapS: I had a suspicion that we are reporting stack_complete before haproxy is actually routing to the backends21:30
SpamapSgreghaynes: OH!21:30
SpamapSgreghaynes: does 'service haproxy restart' return before haproxy is fully up?21:31
greghaynespossibly because keepalived hasnt bound to the vip yet21:31
*** yfaykya has quit IRC21:31
SpamapSgreghaynes: I don't have an haproxy.. hm21:31
greghaynesah21:31
SpamapSI wonder if my image is too old21:31
* SpamapS rebuilds overcloud images21:31
greghaynesSpamapS: did you see my comments on your jenkins run?21:31
greghayneslooked like the controller just didnt boot at all21:31
*** yfaykya has joined #tripleo21:32
greghaynesugh, but its hanging again21:32
greghaynesfor some reason we arent tarring it up or grabbing logs from the controller after that fail21:33
SpamapSgreghaynes: well if it didn't boot...21:33
greghaynesyea, but it happened again, so must be related to that patch21:34
greghayneswhen I run your patch locally everything works for me21:34
*** yfaykya has quit IRC21:37
SpamapSgreghaynes: unfortunately without the console output of that failed node we really can't know whats up. :-P21:38
greghaynesyea :(21:38
*** yfaykya has joined #tripleo21:38
SpamapSgreghaynes: heat event-list or resource-list might also be interesting21:39
openstackgerritJonathan Brownell proposed a change to openstack/diskimage-builder: Increase source-repositories support for tarballs  https://review.openstack.org/10469421:41
*** yfaykya has quit IRC21:43
*** yfaykya has joined #tripleo21:44
SpamapSgreghaynes: yeah so it doesn't make any sense at all why that patch would make a box not even be SSH'able.. ?!21:46
* SpamapS will ponder whilst seeking caffeine21:46
greghaynesyrp21:46
*** yfaykya has quit IRC21:49
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: Log tripleo services to syslog  https://review.openstack.org/10461921:49
*** yfaykya has joined #tripleo21:51
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: Ability to change default swift ports  https://review.openstack.org/10268721:54
*** yfaykya has quit IRC21:55
lifelessSpamapS: logging via upstart doesn't timestamp? we should probably change our strategy :)21:55
*** ccrouch has quit IRC21:56
*** yfaykya has joined #tripleo21:57
jerryzi have a problem registering a baremetal through dhcp, the bm was not able to get dhcp offer. could anybody tell me which dnsmasq serves dhcp in the first place? system dnsmasq or neutron one?21:57
*** yfaykya has quit IRC22:01
*** yfaykya has joined #tripleo22:03
*** ccrouch has joined #tripleo22:07
jerryzlifelesss: ^^ could you help?22:07
lifelessjerryz: neutron22:08
jerryzlifeless: should i kill the addition dnsmasq?22:08
*** yfaykya has quit IRC22:08
lifelessjerryz: which addition dnsmasq?22:09
jerryzlifeless: whenever i restarted seed vm, a dnsmasq instance would start but not the neutron-dhcp-agent one22:09
*** yfaykya has joined #tripleo22:10
lifelessjerryz: can you be more precise about the one that starts22:10
lifelesse.g. its command line22:11
jerryzlifeless: http://paste.openstack.org/show/85458/22:11
lifelessjerryz: I have one of those running on my seed and its not a problem22:12
*** julim has quit IRC22:12
lifelessjerryz: check with tcpdump to make sure your seed is receiving the DHCP request.22:12
jerryzlifeless: then, it comes to the second issue,  the neutron one does not start after reboot, i have to do "service neutron-dhcp-agent restart"  to let it boot up dnsmasq22:13
jerryzthen i found mysql didn't auto start whenever i rebooted22:14
jerryzlifeless: so i started mysql server and check if there are services that failed to start due to mysql failure.22:14
*** yfaykya has quit IRC22:15
lifelessjerryz: something is wrong with your seed then, since that should be all automatic.22:15
jerryzlifeless: even i started them all, dnsmasq didn't get the mac address added to the host list, when a bm asks for dhcp offer, dnsmasq couldn't provide one.22:15
lifelessjerryz: have you registered the node with Ironic, and given ironic the right MAC? e.g. do you have a valid nodes.json ?22:16
*** jtomasek_ has quit IRC22:16
*** yfaykya has joined #tripleo22:16
jerryzlifeless: i used nova baremetal-node-create22:16
lifelessjerryz: could you file a bug (if there isn't one) about restart being a problem22:16
lifelessjerryz: use Ironic :), baremetal-* is deprecated22:16
lifelessjerryz: you'll need to build a new seed if yours was build with nova-bm22:17
tchaypomorning jerryz22:17
tchaypog1222:17
jerryztchaypo: morning22:17
tchaypooh, that's why nothing is working22:17
tchaypothe thing that remaps my capslock to esc is borked22:17
jerryzlifeless: i saw tripleo CI is still using baremetal? i might be wrong?22:19
*** yfaykya has quit IRC22:21
jerryzlifeless: check-tripleo-overcloud-precise's console log suggested USE_IRONIC=022:21
*** yfaykya has joined #tripleo22:22
tchaypoit's deprecated but ironic isn't fully ready yet, so we test both22:24
tchaypotripleo-ironic-seed-precise does what the name suggests :)22:25
*** yfaykya has quit IRC22:27
*** yfaykya has joined #tripleo22:28
*** tzumainn has quit IRC22:29
*** tzumainn has joined #tripleo22:29
*** eghobo has quit IRC22:29
jerryztchaypo: so ironic is preferred way or not?22:30
lifelesstzumainn: ironic is ready, we still support both.22:31
lifelessjerryz: Ironic is preferred.22:31
*** yfaykya has quit IRC22:33
*** yfaykya has joined #tripleo22:35
Nglifeless: I could use some help getting the tooling bootstrapped for hp2, I've been picking around the other baremetal setups we have, but I'm lost :/22:35
lifelessNg: ok; made the shared directory ?22:36
jerryzlifeless: i will try to build a seed with ironic and see if any luck this time. thanks22:36
Nglifeless: yep, got that and all the code in place, went through a delightful wiki->Word->Excel->csv->machine-information-tab.txt process and made nodes.json from it22:36
*** andreaf has quit IRC22:37
lifelessNg: cool; got a refs file and a . thing to source to apply all that ?22:37
Nglifeless: newp22:38
*** yfaykya has quit IRC22:39
lifelessso if you look in hp122:39
lifelessin shared22:39
lifelessthis bit is a little awful and I blame adam_g :)22:40
lifelesslook at seed.hp1.tripleo.org.rc22:40
adam_g?22:40
lifelessadam_g: the source-to-trigger-refs-generation idiom22:40
lifelessadam_g: with tripleo-end-to-end22:40
*** yfaykya has joined #tripleo22:41
*** yamahata has quit IRC22:41
adam_glifeless, i wasn't aware that was being used for anything outside the 306/405 rack22:43
greghayneshaha, I have never heard of a temporary script leaking out like that22:44
lifelessadam_g: yes you were, cause I gave you pull requests when I started using it on hp1 ;)22:45
lifelessadam_g: and then I put a spec up based on it, asa  draft for you to iterate on22:45
*** rlandy has quit IRC22:45
lifelessadam_g: review 9665922:45
lifelessadam_g: :)22:45
adam_glifeless, sorry, that never equated on my end to 'you've signed up to support some tooling central to hp1'22:46
*** yfaykya has quit IRC22:46
*** yfaykya has joined #tripleo22:47
lifelessadam_g: huh, no not expecting you to support it22:47
lifelessadam_g: by blame, I was slighty teasing, and also explaining the genesis of the code to Ng22:47
lifelessadam_g: or rather, I should say that we're all in this together, so I have no expectation of it being up to you and you alone to support it; we all need a solution to the multiple-outstanding-patches problem22:48
tchaypolifeless: re your comment on https://review.openstack.org/#/c/100078/6/toci_devtest.sh,cm - if the new behaviour is triggered without needing to pass any extra flags, wouldn't it be checked by the existing overcloud test?22:49
lifelesstchaypo: I didn't say no flags, I said that we may not need to specify that parameter. I may also be wrong.22:50
tchaypoare you talking about just one of the parameters, or all four of the publc-network params?22:51
*** yfaykya has quit IRC22:52
tchaypomaybe passing them as flags like this is the wrong way though - perhaps we should be reading it from $TE_DATAFILE22:52
*** yfaykya has joined #tripleo22:53
tchaypoTo be clear, I'm not saying I think that there's anything wrong with having the new behaviour triggered without needing flags - just that if it *does* happen automatically it seems like the new test is redundant as the new behaviour will be exercised by the existing overcloud check22:53
openstackgerritJonathan Brownell proposed a change to openstack/diskimage-builder: Increase source-repositories support for tarballs  https://review.openstack.org/10469422:54
lifelessI'm saying that specific parameter22:55
lifelessnot all of them22:55
lifelessand yes, I suspect  one of the json fiels (.e.g. the bm-networks on in my patchset) is whats needed22:55
*** yfaykya has quit IRC22:57
Nglifeless: ok, that's done and run22:58
*** yfaykya has joined #tripleo22:59
lifelessNg: we setup brbm before, so you should be able to just run devtest_setup.sh and devtest_seed.sh at this point22:59
tchaypolifeless: I'm not sure which specific parameter you mean - the comment just shows up under line 112 which has 4 parameters23:00
tchaypobut in any case I'm going to remove them all23:01
lifelesstchaypo: so have a look at my vlan patchsets23:01
lifelesstchaypo: thats triggering the code in question23:01
*** yfaykya has quit IRC23:03
openstackgerritJames Polley proposed a change to openstack-infra/tripleo-ci: Add a vlan check  https://review.openstack.org/10007823:04
*** yfaykya has joined #tripleo23:05
lifelessdefintely something odd in connection mgmt in gertty/requests... its leaking up to 10.23:08
lifelesshowever using a session object does seem to have helped substantially23:09
*** yfaykya has quit IRC23:10
*** michchap has quit IRC23:11
*** michchap has joined #tripleo23:11
*** yfaykya has joined #tripleo23:11
*** nijaba has quit IRC23:12
*** nijaba has joined #tripleo23:13
*** nijaba has quit IRC23:13
*** nijaba has joined #tripleo23:13
*** yfaykya has quit IRC23:16
*** yfaykya has joined #tripleo23:17
jerryzlifeless: somehow, in my seed vm, mysql was not in auto start service list23:18
jerryzlifeless: how the element make sure it is in autostart?23:18
*** yfaykya has quit IRC23:22
lifelessjerryz: it shouldn't matter; can I ask - are you rebooting your seed vm? If so, that might be broken right now, so don't reboot it :)23:22
lifelessjerryz: rebuild / redeploy, but not reboot23:22
jerryzlifeless: yes, i rebooted it23:22
jerryzlifeless: i build it first and ship it somewhere else to start it23:23
lifelessjerryz: thats not rebooting - it should be fine23:23
*** yfaykya has joined #tripleo23:24
jerryzlifeless: my reboot sequence is : virsh destroy and virsh start23:24
lifelessjerryz: where is the copy in that?23:24
*** ccrouch has quit IRC23:25
jerryzlifeless: the machine where i built seed.qcow2 was not the one i ran virsh start23:25
lifelessjerryz: I get that23:25
lifelessbut I'm saying23:25
lifelessif you do destroy, copy fresh image, start23:26
lifelessits the first boot for the image23:26
lifelessif you do copy, destry, start, destroy start, then the image does get a reboot and we might expect issues since we don't test that23:26
jerryzlifeless: oh, i see. i didn't expect that.23:27
lifelessobviously we *should* support reboots23:28
*** yfaykya has quit IRC23:28
lifelessbut I would like to get you past any initial friction by staying on the tested path23:28
*** yfaykya has joined #tripleo23:30
jerryzlifeless: ok. you know last time when i built the seed without build-only, the seed vm  started extremely slowly on the nested kvm, which is why i moved it to a baremetal and start, which happened to cause the reboot.23:32
SpamapSlifeless: no it doesn't timestamp. It functions as a _very_ simple ring buffer and they didn't want to do newline-scanning.23:32
*** yfaykya has quit IRC23:34
*** yfaykya has joined #tripleo23:36
SpamapS[11336.078364] EXT4-fs error (device sda3): ext4_journal_check_start:56: Detected aborted journal23:40
SpamapS[11336.072370] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT23:40
greghaynesSpamapS: on the test run?23:40
SpamapS[11336.072333] ata3.00: device reported invalid CHS sector 023:40
SpamapSgreghaynes: locally on my undercloud23:41
*** yfaykya has quit IRC23:41
SpamapS[11326.623483] ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen23:41
SpamapS[11326.625323] ata3.00: failed command: READ FPDMA QUEUED23:41
SpamapSsomething went horribly wrong :-P23:41
SpamapSwhy ata?23:41
SpamapSshouldn't we be doing virtio?23:41
*** yfaykya has joined #tripleo23:42
*** ramishra has joined #tripleo23:43
*** yfaykya has quit IRC23:47
*** ramishra has quit IRC23:48
*** yfaykya has joined #tripleo23:48
*** eghobo has joined #tripleo23:52
*** yfaykya has quit IRC23:53
mordredSpamapS: ata for life23:54
*** yfaykya has joined #tripleo23:54
*** derekh_ has joined #tripleo23:57
derekh_hi people who are still awake, looks like the overcloud job is busted , so I think we now have an other problem in addition to https://bugs.launchpad.net/tripleo/+bug/133723023:59
uvirtbotLaunchpad bug 1337230 in tripleo "Loosing connection to keystone during CI" [High,In progress]23:59
*** yfaykya has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!