| ianw | oh, i see what's going on, a lot of confusion between "unmount" and "cleanup" | 00:02 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 00:58 |
| *** yolanda_ has joined #openstack-dib | 01:05 | |
| *** yolanda has quit IRC | 01:08 | |
| *** yolanda has joined #openstack-dib | 01:09 | |
| *** yolanda_ has quit IRC | 01:11 | |
| *** yolanda_ has joined #openstack-dib | 01:13 | |
| *** yolanda has quit IRC | 01:16 | |
| *** yolanda__ has joined #openstack-dib | 01:19 | |
| *** yolanda_ has quit IRC | 01:22 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 01:26 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 02:09 |
| *** brault has joined #openstack-dib | 04:52 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Call kpartx remove in umount, not cleanup https://review.openstack.org/578657 | 05:10 |
| *** openstackgerrit has quit IRC | 05:34 | |
| ianw | abaindur / clarkb / johnsom : ^ this may be of interest, if still having problems | 05:49 |
| ianw | yolanda__: on consideration, i also feel like that's probably related to your issue in https://review.openstack.org/576876 | 05:51 |
| yolanda__ | ianw, not sure... because that acts on one partition. But the problem we were hitting was that we were removing the device used by several partitions, while others were still using it | 05:54 |
| yolanda__ | i see those patches as complementary | 05:54 |
| yolanda__ | if we remove the device on the first umount anyway, the other partitions may be still using it, right? | 05:55 |
| ianw | yolanda__: well we should remove the partitions, then the device? | 05:56 |
| yolanda__ | yep, but all the partitions. Will the order be ok with your patch if there are several? | 05:57 |
| yolanda__ | i found that we were traversing the tree, arriving to the first partition, then detaching the device, without even looking if there were others | 05:57 |
| yolanda__ | ianw, maybe we can add that unit testing to yours and ensure everything is removed cleanly | 05:58 |
| yolanda__ | but anyway, i think that the code to ensure that we clean up things on last partition and not first, should be there, even at one phase or another... | 05:59 |
| *** yolanda__ is now known as yolanda | 05:59 | |
| ianw | yolanda: i'm not sure why it matters where the "kpartx -d" gets called? | 06:00 |
| yolanda | ianw, because if kpartx -d still contains volumes, from other partitions, it cannot be detached | 06:00 |
| yolanda | it still has things in use, so it refuses to detach | 06:02 |
| ianw | yolanda: can you give the example config where it happens? | 06:02 |
| yolanda | ianw, https://review.openstack.org/#/c/576876/4/diskimage_builder/block_device/tests/config/lvm_tree_multiple_partitions.yaml | 06:04 |
| yolanda | if you try to build like that without my patch, it won't create the image, will fail when calling kpartx, because it calls it first on BSP | 06:04 |
| yolanda | and the volumes are on root, so it's busy | 06:04 |
| *** andreas_s has joined #openstack-dib | 06:08 | |
| ianw | yolanda: so the LVM hasn't been cleaned up yet, but we're calling the "kpartx -d"? | 06:11 |
| yolanda | yep, that's because we don't even arrived to traverse the tree at that point | 06:12 |
| yolanda | because as we first hit BSP partition, the first thing that we do is call kpartx -d there, without even visiting ESP or root | 06:12 |
| yolanda | that's the issue | 06:12 |
| ianw | hrrm, so in like https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/block_device/level1/lvm.py#n47 | 06:13 |
| yolanda | ianw, but the order matters. See, to make it work, i had to reverse the order like https://review.openstack.org/#/c/576495/1/elements/overcloud-secure-uefi/block-device-default.yaml | 06:14 |
| yolanda | if i put root at the end, it was the first partition that had the volumes, then it removes it ok, and calls kpartx there, but the other 2 partitions don't have volumes so it does not complain | 06:14 |
| yolanda | that's why we need to detach only when we are at the last partition attached to that block device | 06:14 |
| yolanda | and nothing else is hanging from there | 06:15 |
| ianw | yeah ... ok | 06:16 |
| yolanda | i guess we never arrived to test that use case | 06:16 |
| ianw | it's all because the "Partitioning" object isn't a real object that's in the graph | 06:18 |
| yolanda | ianw, another solution could be traverse tree by level on tree, and don't arrive to that kpartx untl all lower levels have been cleared, but this sounds as a more complex patch | 06:25 |
| yolanda | also lvm is at same level of partitioning, so if we wanted that approach, partitioning should be on an upper level | 06:25 |
| *** trungnv has quit IRC | 06:39 | |
| *** trungnv has joined #openstack-dib | 06:53 | |
| *** openstackgerrit has joined #openstack-dib | 07:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: block-device lvm: fix umount phase https://review.openstack.org/503958 | 07:17 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 07:17 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Call kpartx remove in umount, not cleanup https://review.openstack.org/578657 | 07:17 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM testing for xenial-build-succeeds https://review.openstack.org/578708 | 07:17 |
| ianw | yolanda: i dropped a comment. lmn what you think ... | 07:22 |
| yolanda | ianw, ok let me rebase that | 07:27 |
| ianw | yolanda: i'm like 99% sure it's that dangling BSP partition in https://imgur.com/a/weI07WP that has exposed this | 07:28 |
| yolanda | ianw, not sure... i'd say that also just with ESP it will happen, depending on the order | 07:29 |
| yolanda | if we reach ESP first, we'd try to do kpartx there, while root volumes are in use | 07:29 |
| ianw | yolanda: but, again i think, if you follow ESP down, it goes into the mkfs and the mount, and then it all gets into a chain from there | 07:30 |
| ianw | so when working backwards, all that will unroll before you'll get to cleaning up ESP | 07:30 |
| ianw | (i think) :) | 07:30 |
| yolanda | ianw, i guess we can just try | 07:31 |
| *** andreas_s has quit IRC | 07:31 | |
| yolanda | with uefi, this could be a common schema now, not just a corner case as before | 07:32 |
| *** andreas_s has joined #openstack-dib | 07:32 | |
| ianw | yes | 07:34 |
| *** yolanda_ has joined #openstack-dib | 07:35 | |
| *** yolanda is now known as Guest63448 | 07:37 | |
| *** yolanda_ is now known as yolanda | 07:37 | |
| yolanda | anyway, i want to test that patch chain on my overcloud secure images | 07:37 |
| *** pavel_V has joined #openstack-dib | 07:38 | |
| *** Guest63448 has quit IRC | 07:38 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 07:52 |
| *** andreas_s has quit IRC | 07:59 | |
| *** andreas_s_ has joined #openstack-dib | 08:03 | |
| *** brault has quit IRC | 08:03 | |
| *** andreas_s_ has quit IRC | 08:03 | |
| *** brault has joined #openstack-dib | 08:04 | |
| *** pmannidi has quit IRC | 08:13 | |
| *** andreas_s has joined #openstack-dib | 08:54 | |
| *** andreas_s has quit IRC | 08:54 | |
| *** andreas_s has joined #openstack-dib | 08:54 | |
| *** andreas_s has quit IRC | 08:55 | |
| *** andreas_s has joined #openstack-dib | 08:59 | |
| *** andreas_s has quit IRC | 09:01 | |
| *** andreas_s has joined #openstack-dib | 09:01 | |
| *** andreas_s has quit IRC | 09:02 | |
| *** andreas__ has joined #openstack-dib | 09:06 | |
| *** andreas__ has quit IRC | 09:06 | |
| *** andreas_s_ has joined #openstack-dib | 09:07 | |
| *** andreas_s_ has quit IRC | 09:22 | |
| *** andreas_s has joined #openstack-dib | 09:27 | |
| *** andreas_s has quit IRC | 09:29 | |
| *** andreas_s has joined #openstack-dib | 09:29 | |
| *** trungnv has quit IRC | 10:00 | |
| *** andreas_s has quit IRC | 10:12 | |
| *** andreas_s has joined #openstack-dib | 10:16 | |
| *** andreas_s has quit IRC | 11:04 | |
| *** andreas_s has joined #openstack-dib | 11:05 | |
| *** andreas_s has quit IRC | 11:10 | |
| *** andreas_s has joined #openstack-dib | 11:11 | |
| *** andreas_s has quit IRC | 11:19 | |
| *** andreas_s has joined #openstack-dib | 11:20 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 11:29 |
| openstackgerrit | Ankit Kumar proposed openstack/diskimage-builder master: Adding new dib element https://review.openstack.org/578773 | 11:50 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add new modprobe element https://review.openstack.org/577170 | 11:56 |
| *** vmlinuz has joined #openstack-dib | 12:02 | |
| *** vmlinuz has joined #openstack-dib | 12:02 | |
| *** andreas_s has quit IRC | 12:03 | |
| *** andreas_s has joined #openstack-dib | 12:04 | |
| *** andreas_s has quit IRC | 12:08 | |
| *** andreas_s has joined #openstack-dib | 12:08 | |
| *** vmlinuz has quit IRC | 12:16 | |
| *** mjturek has joined #openstack-dib | 13:00 | |
| *** chhagarw has joined #openstack-dib | 13:04 | |
| *** chhavi__ has joined #openstack-dib | 13:04 | |
| *** andreas_s has quit IRC | 13:06 | |
| *** andreas_s has joined #openstack-dib | 13:07 | |
| *** andreas_s has quit IRC | 13:07 | |
| *** andreas_s has joined #openstack-dib | 13:07 | |
| *** andreas_s has quit IRC | 13:08 | |
| *** andreas_s_ has joined #openstack-dib | 13:13 | |
| *** andreas_s_ has quit IRC | 13:17 | |
| *** chhagarw has quit IRC | 13:18 | |
| *** andreas_s has joined #openstack-dib | 13:27 | |
| *** andreas_s has quit IRC | 13:40 | |
| *** andreas_s has joined #openstack-dib | 13:49 | |
| *** andreas_s has quit IRC | 13:58 | |
| *** andreas_s has joined #openstack-dib | 13:59 | |
| *** andreas_s has quit IRC | 14:08 | |
| *** andreas_s has joined #openstack-dib | 14:10 | |
| *** andreas_s has quit IRC | 14:10 | |
| *** andreas_s has joined #openstack-dib | 14:11 | |
| *** andreas_s has quit IRC | 14:17 | |
| *** andreas_s has joined #openstack-dib | 14:20 | |
| *** andreas_s has quit IRC | 14:22 | |
| *** andreas_s has joined #openstack-dib | 14:23 | |
| *** vmlinuz has joined #openstack-dib | 14:30 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix bootloader for efi on rhel systems https://review.openstack.org/576041 | 15:02 |
| *** vmlinuz has quit IRC | 15:16 | |
| *** vmlinuz has joined #openstack-dib | 15:28 | |
| *** vmlinuz has quit IRC | 15:28 | |
| *** vmlinuz has joined #openstack-dib | 15:28 | |
| *** andreas_s has quit IRC | 15:42 | |
| *** raopajay has quit IRC | 16:26 | |
| *** mjturek has quit IRC | 17:57 | |
| *** mjturek has joined #openstack-dib | 18:10 | |
| *** mjturek has quit IRC | 18:15 | |
| *** mjturek has joined #openstack-dib | 18:16 | |
| *** yolanda_ has joined #openstack-dib | 18:25 | |
| *** yolanda has quit IRC | 18:27 | |
| *** ajun has joined #openstack-dib | 18:46 | |
| ajun | hi clarkb and ianw. I was able to build the image from yesterday on an old Ubuntu 14 VM. However, when I tried also on a newer Centos 7.5 VM, it failed, but almost immediately with a different error (debootstrap not found). I also have the -x logs from yesterday with the failure on my Centos 7.2 machine | 18:48 |
| *** ajun is now known as abaindur | 18:48 | |
| clarkb | aj debootstrap is a debian tool that dib uses to build debian and ubuntu images from scratch. I believe there is a packge for it on centos (might be in epel?) | 18:48 |
| abaindur | ianw: Here are the verbose/debug logs run with -x, on my Centos 7.2 machine: https://ufile.io/xh9tc | 18:49 |
| abaindur | clarkb: What is strange is that when I was running it on my Centos 7.2 machine, I see NO reference to debootstrap at all. | 18:49 |
| abaindur | I see no such logs even loading the 08-debootstrap | 18:50 |
| abaindur | dib-run-parts Thu Jun 28 07:18:39 UTC 2018 Running /tmp/dib_build.4ANCPf8l/hooks/root.d/08-debootstrap | 18:50 |
| abaindur | and I don't have any debootstrap module installed on my centos 7.2 machine from yesterday | 18:50 |
| abaindur | Here is log (with -x also) run on centos 7.5 machine: https://ufile.io/bmzte | 18:54 |
| *** chhavi__ has quit IRC | 18:58 | |
| *** abaindur has quit IRC | 19:23 | |
| *** abaindur has joined #openstack-dib | 19:35 | |
| *** abaindur has quit IRC | 20:02 | |
| *** abaindur has joined #openstack-dib | 20:03 | |
| *** mjturek has quit IRC | 20:12 | |
| *** abaindur has quit IRC | 20:21 | |
| *** abaindur has joined #openstack-dib | 20:21 | |
| *** mjturek has joined #openstack-dib | 20:30 | |
| *** mjturek has quit IRC | 21:25 | |
| ianw | abaindur: well, good start that we now see "exec_sudo: device-mapper: remove ioctl on loop2p1 failed: Device or resource busy" | 21:28 |
| ianw | also weird : exec_sudo: fstrim: /tmp/dib_build.hHR0rXLq/mnt/: the discard operation is not supported | 21:30 |
| abaindur | im filing a bug but do you have any recommendations on where to properly host the file? | 21:45 |
| abaindur | That ufile.io only hosts for 30 days | 21:45 |
| abaindur | github's gist has a character limit | 21:45 |
| *** mjturek has joined #openstack-dib | 21:58 | |
| *** mjturek has quit IRC | 22:01 | |
| ianw | abaindur: you can attach files to the bug in launchpad | 22:09 |
| ianw | abaindur: can you paste the output of "mount" to paste.openstack.org? | 22:09 |
| ianw | i'm wondering what's hanging around | 22:09 |
| abaindur | theres a lot of temporary mounts. i rebooted and they went away, but same error | 22:18 |
| abaindur | hang on | 22:18 |
| abaindur | ianw: https://gist.github.com/xagent003/45c58f0636762b17aa0217ba06e2bb83 | 22:19 |
| ianw | hrm | 22:23 |
| ianw | /run/media/root/cloudimg-rootfs | 22:24 |
| ianw | that's weird. are you in an gnome type session? | 22:24 |
| ianw | i wonder if some sort of auto-mounter is finding our disks and mounting them behind our back | 22:24 |
| abaindur | Here is the mounts now after things cleaned up: | 22:29 |
| abaindur | https://gist.github.com/xagent003/ef3afd2f273851b5a407f210a06ffdd0 | 22:29 |
| abaindur | I do not see any cloudimg-rootfs | 22:29 |
| ianw | that looks about right, but i'm a little worried that something is detecting the disks we make during build and mounting them, much as what happens when you plug in a usb stick | 22:31 |
| abaindur | i'll have to ask someone else. that is beyond my knowledge :) this is a vm on vmware vcenter | 22:31 |
| abaindur | i'm not sure what vmtoolsd is doing | 22:32 |
| abaindur | i know there are some vmware managed pkgs on this VM | 22:32 |
| ianw | is it just a headless server, or are you using X windows with it? | 22:33 |
| abaindur | i do have Mate installed on it. i use it for VNC | 22:34 |
| ianw | hmm, i strongly suspect that something in there is mounting the disks we create | 22:35 |
| ianw | https://access.redhat.com/solutions/20107 describes how to disable it | 22:36 |
| ianw | for the default gnome at least. | 22:36 |
| abaindur | ahh hmm thanks | 22:45 |
| *** pmannidi has joined #openstack-dib | 22:54 | |
| *** yolanda__ has joined #openstack-dib | 23:07 | |
| *** abaindur has quit IRC | 23:08 | |
| *** yolanda_ has quit IRC | 23:11 | |
| *** vmlinuz has quit IRC | 23:14 | |
| *** yolanda__ has quit IRC | 23:18 | |
| *** abaindur has joined #openstack-dib | 23:22 | |
| *** dhill_ has quit IRC | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!