Saturday, 2019-02-09

*** salmankhan has quit IRC00:00
*** random_yanek has quit IRC00:06
*** slaweq has joined #openstack00:11
*** Roedy has quit IRC00:20
*** random_yanek has joined #openstack00:22
*** mikebrow has joined #openstack00:22
*** mikebrow_ has quit IRC00:24
*** slaweq has quit IRC00:24
*** forgotmynick has joined #openstack00:35
*** albertom has quit IRC00:38
*** visibilityspots has joined #openstack00:56
*** zbr has quit IRC01:00
*** markvoelker has joined #openstack01:10
*** slaweq has joined #openstack01:16
*** gyee has quit IRC01:16
*** slaweq has quit IRC01:24
*** cyb-r has quit IRC01:25
*** visibilityspots has quit IRC01:27
*** gary_perkins has quit IRC01:28
*** gary_perkins has joined #openstack01:29
*** brokencycle has quit IRC01:32
*** daMaestro has joined #openstack01:38
*** visibilityspots has joined #openstack01:39
*** markvoelker has quit IRC01:42
*** visibilityspots has quit IRC01:43
*** visibilityspots has joined #openstack01:54
*** eblip has quit IRC02:10
*** eblip has joined #openstack02:14
*** gnufied has quit IRC02:15
*** slaweq has joined #openstack02:15
*** slaweq has quit IRC02:25
*** ileixe has quit IRC02:26
*** ileixe has joined #openstack02:27
*** bobh has joined #openstack02:27
*** bobh has quit IRC02:29
*** maharg101 has quit IRC02:33
*** markvoelker has joined #openstack02:40
*** ileixe has quit IRC02:48
*** ileixe has joined #openstack02:48
*** slaweq has joined #openstack03:11
*** markvoelker has quit IRC03:13
*** slaweq has quit IRC03:24
*** ntiy has joined #openstack03:24
*** forgotmynick has quit IRC03:25
*** visibilityspots has quit IRC03:28
*** bobh has joined #openstack03:31
*** ArchiFleKs has quit IRC03:54
*** ArchiFleKs has joined #openstack04:00
*** bobh has quit IRC04:04
*** jtomasek has quit IRC04:09
*** markvoelker has joined #openstack04:10
*** slaweq has joined #openstack04:11
*** cyb-r has joined #openstack04:17
*** slaweq has quit IRC04:24
*** markvoelker has quit IRC04:43
*** slaweq has joined #openstack05:12
*** walshh__ has quit IRC05:13
*** slaweq has quit IRC05:24
*** markvoelker has joined #openstack05:40
*** slaweq has joined #openstack06:11
*** markvoelker has quit IRC06:13
*** ntiy has quit IRC06:20
*** visibilityspots has joined #openstack06:21
*** slaweq has quit IRC06:24
*** sauvin has joined #openstack06:45
*** b7219264_ has quit IRC06:51
*** Son_Goku has joined #openstack06:52
*** sauvin has quit IRC06:52
*** b7219264 has joined #openstack06:59
*** markvoelker has joined #openstack07:10
*** lpetrut has joined #openstack07:11
*** slaweq has joined #openstack07:16
*** slaweq has quit IRC07:24
*** daMaestro has quit IRC07:25
*** visibilityspots has left #openstack07:27
*** markvoelker has quit IRC07:43
*** kjackal has joined #openstack07:43
*** daniel2 has quit IRC07:48
*** sauvin has joined #openstack08:11
*** Roedy has joined #openstack08:12
*** lpetrut has quit IRC08:13
*** sauvin has quit IRC08:13
*** slaweq has joined #openstack08:13
*** sauvin has joined #openstack08:14
*** jtomasek has joined #openstack08:18
*** slaweq has quit IRC08:22
*** lpetrut has joined #openstack08:30
*** Emine has joined #openstack08:31
*** kjackal has quit IRC08:35
*** _fragatina has joined #openstack08:37
*** markvoelker has joined #openstack08:39
*** daniel2 has joined #openstack08:39
*** zbr|ssbarnea has joined #openstack08:40
*** zbr|ssbarnea has quit IRC08:42
*** _fragatina has quit IRC08:42
*** _fragatina has joined #openstack08:43
*** jbadiapa has quit IRC08:45
*** e0ne has joined #openstack08:47
*** _fragatina has quit IRC08:48
*** _fragatina has joined #openstack08:48
*** _fragatina has quit IRC08:52
*** _fragatina has joined #openstack08:52
*** zbr|ssbarnea has joined #openstack08:53
*** daniel2 has quit IRC08:55
*** _fragatina has quit IRC08:58
*** e0ne has quit IRC09:06
*** Warped has quit IRC09:09
*** slaweq has joined #openstack09:12
*** markvoelker has quit IRC09:13
*** slaweq has quit IRC09:25
*** lpetrut has quit IRC09:27
*** shibboleth has joined #openstack09:33
*** suuuper has joined #openstack09:35
*** kjackal has joined #openstack09:42
*** slaweq has joined #openstack09:49
*** _fragatina has joined #openstack10:00
*** e0ne has joined #openstack10:09
*** markvoelker has joined #openstack10:11
*** e0ne has quit IRC10:11
*** daniel2 has joined #openstack10:18
*** sshnaidm|off has quit IRC10:18
*** sshnaidm|off has joined #openstack10:21
*** _fragatina has quit IRC10:24
*** Warped has joined #openstack10:28
*** kjackal has quit IRC10:28
*** jonaspaulo has joined #openstack10:40
*** Emine has quit IRC10:50
*** _fragatina has joined #openstack10:52
*** markvoelker has quit IRC10:53
*** _fragatina has quit IRC11:23
*** _fragatina has joined #openstack11:23
*** _fragatina has quit IRC11:28
*** matthieuv42 has joined #openstack11:31
*** matthieuv42 has quit IRC11:36
*** supamatt has quit IRC11:42
*** markvoelker has joined #openstack11:50
*** kjackal has joined #openstack11:56
*** _fragatina has joined #openstack12:00
*** shibboleth has quit IRC12:01
*** supamatt has joined #openstack12:02
*** e0ne has joined #openstack12:09
*** _fragatina has quit IRC12:16
*** e0ne has quit IRC12:17
*** kjackal has quit IRC12:17
*** kjackal has joined #openstack12:18
*** slaweq has quit IRC12:19
*** noogie has quit IRC12:20
*** noogie has joined #openstack12:20
*** tolisbar1 has joined #openstack12:22
*** markvoelker has quit IRC12:23
*** tolisbar has quit IRC12:23
*** tolisbar1 is now known as tolisbar12:23
*** slaweq has joined #openstack12:28
*** Emine has joined #openstack12:34
*** lpetrut has joined #openstack12:35
*** _fragatina has joined #openstack12:43
*** e0ne has joined #openstack12:48
*** lpetrut has quit IRC12:54
*** e0ne has quit IRC12:55
*** _fragatina has quit IRC13:04
*** ArchiFleKs has quit IRC13:09
*** kjackal has quit IRC13:10
*** kjackal has joined #openstack13:11
*** gmoro has joined #openstack13:14
*** brokencycle has joined #openstack13:15
*** tolisbar has quit IRC13:17
*** markvoelker has joined #openstack13:20
*** _fragatina has joined #openstack13:23
*** Emine has quit IRC13:23
*** ArchiFleKs has joined #openstack13:24
*** riposte has quit IRC13:25
*** negronjl has quit IRC13:26
*** riposte has joined #openstack13:27
*** negronjl has joined #openstack13:29
*** _fragatina has quit IRC13:38
*** _fragatina has joined #openstack13:39
*** bringha has joined #openstack13:46
*** markvoelker has quit IRC14:00
*** jonaspaulo has quit IRC14:02
*** kjackal has quit IRC14:06
*** jacknemrod has joined #openstack14:10
*** jonaspaulo has joined #openstack14:14
*** bobh has joined #openstack14:15
*** bobh has quit IRC14:15
*** brokencycle has quit IRC14:24
*** jtomasek has quit IRC14:34
*** markvoelker has joined #openstack14:58
*** _fragatina has quit IRC15:06
*** bringha has quit IRC15:14
*** bringha has joined #openstack15:18
*** bringha has quit IRC15:29
*** markvoelker has quit IRC15:30
*** crimastergogo has joined #openstack15:33
*** jtomasek has joined #openstack15:39
*** crimastergogo has quit IRC15:47
*** crimastergogo has joined #openstack15:47
*** needssleep is now known as TheJulia15:58
*** jtomasek has quit IRC15:59
*** bringha has joined #openstack16:04
*** kjackal has joined #openstack16:04
*** salmankhan has joined #openstack16:13
*** Son_Goku has quit IRC16:14
*** bringha has quit IRC16:15
*** Son_Goku has joined #openstack16:20
*** kjackal has quit IRC16:23
*** markvoelker has joined #openstack16:28
*** forgotmynick has joined #openstack16:28
*** takamatsu_ has quit IRC16:36
*** jacknemrod has quit IRC16:49
*** markvoelker has quit IRC17:00
*** jroll has quit IRC17:11
*** gmoro has quit IRC17:28
*** BH23 has joined #openstack17:33
*** jroll has joined #openstack17:33
*** kjackal has joined #openstack17:48
*** mulbc has quit IRC17:54
*** ChrisNBlum has joined #openstack17:54
*** markvoelker has joined #openstack18:02
*** crimastergogo has quit IRC18:13
*** pv2b has joined #openstack18:22
*** BH23 has quit IRC18:27
pv2bHello. I'm setting up DevStack on my PC inside of a VirtualBox VM for learning and exploration purposes. I'm struggling with understanding DevStack sets up networking, but I'm not really finding any comprehensive documentation regarding what different config variables exist. All I'm finding is various "cookbooks" using config options that I'm not seeing documented anywhere. Is there a specific document I should be looking at?18:31
pv2bI mean ideally I'd want something that's close-ish to how something might work in production, where I might bridge out an interface on my local LAN, and carve out a set of IP addresses that can be used as "floating" ip addresses (mimicking what would otherwise be real public IPs). I imagine VM traffic might also egress through those floating IPs?18:33
*** awalende has joined #openstack18:39
*** bgmccollum has joined #openstack18:57
*** markvoelker has quit IRC19:02
*** shibboleth has joined #openstack19:07
*** e0ne has joined #openstack19:13
*** Son_Goku has quit IRC19:15
*** suuuper has quit IRC19:17
*** jonaspaulo has quit IRC19:26
*** salmankhan has quit IRC19:29
*** shibboleth has quit IRC19:42
factorpv2b:From what I understand DevStack sets up this initial network state. So you can work on an older or newer stack. Not worry about the initial setup.  Tripleo I think setup this initial state as well. Packstack does not setup this initial setup,so I had to learn how to do it manually. Working on some documentation on this. But still rough draft.  .19:46
factorpv2b, Also for learning details of Openstack networking I had found the latest version on the Openstack site. "Learning OpenStack Networking - Third Edition" from Packt  https://play.google.com/store/books/details/James_Denton_Learning_OpenStack_Networking?id=TMhsDwAAQBAJ&hl=en19:48
factorI have not bought it yet. But looking to when I get a chance.19:48
pv2bI'll take a look at that book link. Thanks. I'm not sure what you mean with what you wrote. Are you saying it's probably better for me to try to configure OpenStacks network properly post-deployment rather than worrying about initial deployment?19:49
factorpv2b, Depends what setup you will end up with. Packstack(Others maybe also) has to be setup manually(Maybe because neutron is in flux, not all of it has been moved to the openstack CLI or GUI )19:54
*** shibboleth has joined #openstack19:54
factorI could not for the life of me setup a hybrid cloud with the GUI. Had to use CLI and a mix of openstack and neutron commands.19:55
factorBut to reiterate this very well maybe because neutron has not finished it migration to the openstack cli.19:56
factorits^19:56
pv2bAll right. I was hoping that there was a simple way to get this set up that I was missing. Well, I guess I've got to learn this stuff at some point anyway :)19:57
factor:) just giving you a heads up.19:57
factorpv2b, the neutron stuff is being deprecated , but is still needed for some core applications.19:58
*** e0ne has quit IRC19:58
pv2bWhen you say "neutron stuff" I assume you're talking about some management tools rather than neutron itself?19:59
factorneutron is the netowrk module of openstack.19:59
factornetwork^19:59
pv2bYeah, I just wanted to make sure you're not telling me *neutron* is going away, because that would be super surprising to me :)19:59
*** markvoelker has joined #openstack19:59
factorAll of the module API's are being moved to the single openstack interface.19:59
factorCorrect20:00
factorOnly its interface.20:00
*** brokencycle has joined #openstack20:01
factorpv2b, So when learning the neutron interface, be sure to gear up to learn a newer method in near future.20:02
pv2bThat's good advice, thanks :-)20:02
factornp20:03
pv2b(Especially since I'm the kind of person who resists change, I still type "ifconfig" and not "ip a")20:03
factorsame20:03
factorroute2 package.20:03
factorI love the Openstack control of ip tables. giving you the whole scope via the GUI as well.20:05
pv2bYou mean the security groups UI in Horizon?20:06
factorYes. Doing all of that manually, I could do it, but I would get lost. Openstack really lets me wrap my head around it.20:07
factorTis puts to rest if you left any holes open or not.20:08
*** shibboleth has quit IRC20:17
*** renich has quit IRC20:20
*** ArchiFleKs has quit IRC20:21
*** jonaspaulo has joined #openstack20:31
*** markvoelker has quit IRC20:32
*** ArchiFleKs has joined #openstack20:36
*** Son_Goku has joined #openstack21:06
*** awalende has quit IRC21:09
*** awalende has joined #openstack21:10
*** awalende has quit IRC21:14
*** h3xagon has joined #openstack21:25
h3xagonI'm having a hell of a problem that we've been troubleshooting on and off for weeks now and it's come to the point of rebuilding for the 3rd time.  Short of it is that we have a 2 node OS with Compute + Cinder (lvm) on one system for back end storage. We've added the lvm filters in the conf file as directed. We clean up all the instances, volumes. Check pvs/lvs etc on the underlying system and everything shows clean. We spin up a couple21:28
h3xagonof systems and all of a sudden out of nowhere we have an ubuntu-vg pop up that was created WEEKS ago but has since been removed and isn't on the system. Anyone have any suggestions on what to check?21:28
h3xagonOh and we're on queens21:30
*** markvoelker has joined #openstack21:32
larsksRunning 'pvs' after the ubuntu-vg shows up should tell you which blog device(s) it is coming from.21:36
larsks*block21:36
h3xagonIt gives me WARNING: Device for PV <UUID> not found or rejected by filter.21:36
h3xagonboth on vgs and pvs.21:37
h3xagonthe output where it lists the vg is...21:37
h3xagonunknown device    ubuntu-vg    lvm2 a-m   19.00g   15.00g21:37
h3xagonThere "shouldn't" be a block device associated since the vg shouldn't even exist but we can't figure out where the heck it's getting the information from that rebuilds this defunct vg.21:38
larsksHmm.  We might be able to track it down using 'dmsetup'.21:39
h3xagonAh let me check that.21:39
larsksIf you run "dmsetup ls", do you see the ubuntu-vg listed?21:39
h3xagonI do... as follows.21:41
h3xagonubuntu--vg-ubuntu--lv   (252:10)21:41
larsksOkay.  So then "dmsetup table /dev/dm-10'21:41
h3xagoncomplete... returns...21:42
h3xagon0 8388608 linear 8:35 204821:42
larsksOkay, so that 8:35 is pointing at the actual block device.  "ls -l /dev/sd*" and look for "8, 25".21:43
*** slaweq has quit IRC21:43
h3xagonmatches /dev/sdc321:44
larsksOkay.  That's where it's coming from, then.  What is /dev/sdc?21:45
h3xagonHrm... well it wasn't there prior to me creating a new instance on what we were hoping was a clean system. What I did was (using horizon) 1) create an empty volume 2) launch an instance with an ubuntu ISO 3) attach the empty volume 4) install ubuntu to the volume 5) turn off the instance and delete it 6) create a new instance with the bootable volume that was just installed.21:47
h3xagonThat's when I checked vgs/lvs and saw the ubuntu-vg show back up. but that's not the name of my vg at all... and this vg is showing up "outside" of cinder-volumes. I do see the VG that should be associated with my instance inside cinder-volumes.21:48
*** renich has joined #openstack21:48
h3xagonand my volume name for this is utility-prd-vol-0121:49
larsksI may not be following completely, but we seem to have identified the device that is causing the problem, right?  We know it's /dev/sdc3.  What is /dev/sdc? How is it used by your system?21:49
h3xagonNope, it magically showed up when I created the instance inside horizon21:49
h3xagonNo idea where it came from, it's not a physical device in the system at all. We have an underlying Raid5 with 1 volume group for the OS/swap and then another volume group (cinder-volumes) for OS.21:50
larsksThat's very odd.  But...what *is* it? Is it an actual hard drive? Or something else?  What about "udevadm info /dev/sdc"? That would tell you think like manufacturrer/model/etc.21:50
h3xagonYeah that's what we've been saying hehe... we've been working on this for months tried to just push past this problem and it resulted in hanging vgs/lvs commands this past week which scraped a month of progress.21:51
h3xagonlet me do that quick.21:51
h3xagonWhat we've been seeing is whenever we spin an instance it creates /dev/sd* incremented beyond what's there currently.21:52
h3xagonSo if I spin a new instance I would expect to see /dev/sdd* on the filesystem.21:52
larsksWhere is your cinder volume storage coming from?21:52
larsksLocal disk, remote san, ...?21:52
h3xagonlocal disk.21:53
larsksSo, I'm not sure why your system is seeing those; I would have expected the lvm filters to prevent that.  A quick solution would just be to sanitize the storage being used by cinder.21:55
h3xagonhttps://pastebin.com/HDxnjTMB21:55
h3xagonbasically that setup21:55
h3xagonI'm good with sanitization. =)21:55
*** renich has quit IRC21:56
h3xagonMore or less that's what we've tried to do... 1) remove all instances from OS 2) remove all volumes from OS 3) use lv/pv/dmsetup commands to remove any residual volumes other than the OS and cinder-volumes 4) check /dev/ for block devices that shouldn't be there and remove them 5) reboot and try again.21:57
h3xagonAre we missing something in that process (I hope).21:57
*** kjackal has quit IRC21:57
larsksYou're going to need to zero out the underlying blocks, because what is probably happening is that there is stale data on disk that only becomse visible when cinder happens to carve out a volume starting at the right place. Because it's in the middle of the device, pvs/lvs/etc aren't normally going to see it.21:58
*** kjackal has joined #openstack21:58
larsksThere are some docs here on volume wiping which might also help: https://docs.openstack.org/security-guide/block-storage/volume_wiping.html21:59
larsks(That is how you configure cinder to automatically wipe volumes when they are deleted)21:59
larsksWhat do your existing lvm filters look like?22:00
larsksActually, have to run.  I am suspicious that your lvm filters aren't correctly configured, because your host shouldn't be able to see lv devices on cinder volumes.22:02
h3xagonso /dev/sda1 = BIOS Boot, sda2 = /boot, sda3 = Ubuntu OS [LVM], sda4 = cinder-volumes[LVM]  ---- lvm.conf filter is "filter = [ "a/dev/sda3/", "a/dev/sda4", "r/.*/" ]22:02
h3xagonI really appreciate the help this has been great having another set of eyes on this. Do you mind if I PM where we end up just in case you can spot something sideways in what we're doing when you're back? if not I understand and thank you for the help either way.22:03
*** slaweq has joined #openstack22:12
*** slaweq has quit IRC22:22
*** markvoelker has quit IRC22:23
*** kjackal has quit IRC22:44
*** shibboleth has joined #openstack22:58
*** cyberworm54 has quit IRC23:00
*** cyberworm54 has joined #openstack23:08
*** bobh has joined #openstack23:09
*** bobh has quit IRC23:09
h3xagonI "might" have found some indication of what's going on.... so this problem happens with linux instances using LVM. It does "not" happen with Windows systems, and I suspect it doesn't happen with non LVM instances (checking this now). So to your point larsks it looks like it might be filter related or something.... it's almost like a circular problem LVM base... picking up the LVM disks at the instance level and it shouldn't be so it's23:11
h3xagoncausing issues with not only the instances but also weirdness at the compute/storage level since the OS tools are picking up the instance disks as well. Ugh...23:11
*** slaweq has joined #openstack23:11
*** markvoelker has joined #openstack23:20
*** shibboleth has quit IRC23:22
*** fresta_ has joined #openstack23:24
*** fresta has quit IRC23:24
*** slaweq has quit IRC23:24
pv2bh3xagon: Okay, I'm by no means an expert, but your filters look kinda bogus, i'm surprised it's picking up sda3 and sda4 like that23:32
pv2bshouldn't it be something more along the lines of filter = [ "a|^/dev/sda3/$|", "a|^/dev/sda4$|", "r/.*/" ]  ?23:33
pv2bthis is a shot in the dark and a guess here, but... based on these docs: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/lvm_filters23:33
pv2bI would guess that the "a/dev/sda3" would actually be a syntax error23:33
h3xagonThat could very well be the issue.23:34
pv2bbecause it's like the first character after the / is the quote character and... oh wait23:34
pv2b"a/dev/sda3" if it's ignoring the stuff after the second / you're just matching any device23:34
*** wwriverrat has joined #openstack23:35
h3xagonHrm...23:35
pv2bI mean I've never evern configured an lvm filter, it just looks... off... based on how it's supposed to be, i mean it looks like an "a" followed by like a regex...23:36
*** bobh has joined #openstack23:37
h3xagonYeah definitely looking at it.23:37
pv2bhttps://github.com/lvmteam/lvm2/blob/54668feaabead997cdf099c970999af667bcf274/lib/filters/filter-regex.c#L8023:40
pv2bYou'd expect an error message "Invalid seperator at the end of regex.".... somewhere.... if this is the case23:40
*** markvoelker has quit IRC23:52
pv2bYeah, perusing the code a bit, if there's a parse error in any of the regexes, the whole regex filter thing errors out, and if that happens, all the filters just get removed. Also if you're not actually seeing an error it's because you might be running an older version of lvm2, looks like around 2 years ago the log level was changed from INFO to ERROR (so if it's not showing INFO messages, that'd explain the silent failure)23:53
pv2bhttps://github.com/lvmteam/lvm2/blob/420af27f088a3808816ba3ab5b47dfa400fef308/lib/commands/toolcontext.c#L106723:53
*** bobh has quit IRC23:56
h3xagonpv2b ty for the help, I will definitely dig into what you provided. I ended up getting a call for updates on this and then the wifey is hollering at me for not eating. >=) So having to jet for a couple min to scarf food.23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!