Sunday, 2016-08-14

*** britthouser has joined #openstack-kolla00:04
*** Pavo has joined #openstack-kolla00:04
*** sdake has quit IRC00:08
Jeffrey4l_i am in, too.00:38
*** jtriley has joined #openstack-kolla00:45
*** jtriley has quit IRC00:50
Pavoanyone here? having an issue with kolla-build01:23
Pavohttp://pastebin.com/9Gb4EM4Y01:24
Pavomy kolla-build.conf looks like this01:24
Pavohttp://pastebin.com/qHTT3SYK01:24
Pavoany suggestions?01:25
mark-caseyPavo: I usually just pass command line options to kolla-build, but a bit of searching suggests you need "[DEFAULT]" as the first line01:30
Pavooh ok ty01:30
mark-caseyPavo: welcome... hope it works01:32
Pavoyes that worked01:32
Pavoty01:32
mark-caseycool, welcome01:32
Pavonow I am having an issue with adding [Service]01:45
PavoMountFlags=shared01:45
Pavoto /etc/systemd/system/docker.service.d/kolla.conf01:45
Pavonot seeing docker.service.d/01:45
*** sdake has joined #openstack-kolla01:47
Pavolol now ERROR:kolla.cmd.build:base:Tag mitaka not found in repository docker.io/library/centos01:48
mark-caseyPavo: base OS is centos and you're building centos containers, right?01:50
Pavoyes01:50
Pavofollowing this guide01:50
Pavohttp://egonzalez.org/openstack-kolla-deployment-from-rdo-packages/01:51
mark-caseyyou may have to create docker.service.d01:51
Pavoyeah figured that one out lol01:53
mark-caseyIMO that guide is less than ideal because pushing to dockerhub will probably take a long time. are you deploying kolla multinode or all in one?01:54
Pavo2 node deployment on VMs01:56
Pavojust trying it out01:56
mark-caseysure. are you running kolla-build and kolla-ansible and such inside one of these VMs on on another machine?01:57
mark-casey*or on01:57
Pavoseems like its trying to look for tag Mitaka in docker.io/library/centos when its trying to pull the images01:59
*** Jeffrey4l_ has quit IRC02:00
*** daneyon_ has joined #openstack-kolla02:08
PavoI see all the images here https://hub.docker.com/u/kolla/02:09
*** daneyon_ has quit IRC02:12
mark-caseyPavo: when you ran 'kolla-build -n pavo --push'  you would actually find those images at https://hub.docker.com/u/pavo. The guide is having you make your own copy. That is why it said run 'docker login' and why it said the build/push would take 3-9 hours.02:13
mark-caseyprobably easier to run a local registry02:14
*** rhallisey has quit IRC02:17
*** sdake has quit IRC02:17
mark-caseyPavo: did you run kolla-build inside one of the VMs or doing that on your physical machine/another machine?02:19
mark-caseyalso, you'll find plenty of help here if you do want to deploy to a few VMs, but there is also an all-in-one Vagrant environment for trying Kolla out. I just don't use it much so I didn't recommend it first thing.02:22
Pavoyeah I made a local registry and used kolla-build --registry 192.168.1.40:4000 --push02:27
Pavostill saying the same thing though02:28
mark-caseyoh, awesome. did you add the --insecure-registry flag to the docker systemd dropin?02:29
mark-caseyin addition to MountFlags?02:29
Pavoyeah02:29
Pavook its working now02:30
Pavojust took the tag = mitaka out02:30
mark-caseyyeah. sorry was going through the list coming around to that :D02:30
Pavoguess it will use mitaka since its the latest build02:30
PavoI tell ya though, I am loving this openstack container idea though02:31
mark-caseynot an expert on that part but it follows the version of kolla you check out02:31
mark-caseyyea, me too02:31
Pavoor I gues I should say openstack containized02:31
mark-caseyso if you check out stable/mitaka of kolla you deploy that version, if you check out (or pip install) master you build/deploy images that track master of the other projects02:32
PavoI have been deploying openstack using packstack myself for about a year now02:33
Pavobut figured I would see how this goes02:33
Pavoand just starting in docker02:33
Pavoso its a BIG step for me02:33
mark-caseyit's not perfect but I like this better than fuel which I was using before. The image model is better IMO and cleaning up and re-deploying is really easy because almost all changes on nodes were containerized02:34
*** klint has joined #openstack-kolla02:35
Pavooh I started out with fuel02:35
Pavof that noise02:35
mark-caseyindeed :D02:35
Pavohad so........... many issues with fuel02:35
Pavoand with the new version the nics flip flop around sometimes which is a pain to TS02:35
mark-caseyyeah. having not used docker a ton myself in the past kolla had a decent learning curve as well. but community support is pretty good02:37
Pavook hit another issue02:37
PavoERROR:kolla.image.build.kolla-toolbox:Get https://192.168.1.40:4000/v1/_ping: http: server gave HTTP response to HTTPS client02:37
Pavowhy is it trying v1 of registry when I am running v202:38
Pavoanyway to fix that?02:38
mark-caseycan you ps -ef | grep docker on the VMs?02:39
Pavohttp://pastebin.com/J1qtHFay02:39
Pavof0686b06c57b        registry:2          "/entrypoint.sh /etc/"   50 minutes ago      Up 48 minutes       0.0.0.0:4000->5000/tcp   registry02:40
mark-caseyI believe the dockerd line is missing --insecure-registry ...02:42
Pavooh so you mean I need kolla-build --insecure-registry 192.168.1.40:4000 --push instead?02:43
mark-caseyno. that part was right. what should happen is on the VMs when you add the drop in inside /etc/systemd/system/docker.service.d  you include that flag on the ExecStart. Then systemctl daemon-reload and systemctl restart dockerd02:44
Pavooh ok02:44
Pavook how do I do that02:46
PavoI only have kolla.conf in that dir02:46
mark-caseyyeah was trying to find it in docs or in some of my stuff02:47
*** jtriley has joined #openstack-kolla02:47
Pavofound it02:48
Pavounder /etc/sysconfig/docker02:48
Pavobut I have that02:48
Pavowhoops was missing an I02:49
Pavolol02:49
*** williamc_ has quit IRC02:52
mark-caseyoh ok. the best practice may have changed. you can do it where you did mountflags by setting execstart blank and then putting the original command with the flag as the following setting. Shouldn't matter as long as it shows in ps -ef | docker on your VMs02:53
Pavowell I guess I could have used the registry I am running on ym mac02:54
Pavolol02:54
mark-caseyperhaps. IIRC any non-localhost registry must be HTTPS and authenticated unless it is listed as that flag02:55
mark-caseyof which you can list multiple02:55
Pavowell I changed it to localhost:4000 and its working it seems02:56
mark-caseyis .40 the mac?02:57
Pavonah a VM02:57
PavoI will re-run it again and push it to my mac registry so if I blow these VMs away I will already have them in my other regsitry02:58
mark-caseysounds good. you'll know quickly if it's happy. that error about v1 registry (it was more mad about HTTP!=HTTPS) is the symptom of it not liking the registry security03:01
Pavoyeah but if I have it in that file I should be good though right03:02
*** haplo37- has quit IRC03:03
*** jtriley has quit IRC03:04
mark-caseyyep03:05
Pavowonder if I could build a centos repo container, so I can host my own repos03:07
*** haplo37_ has joined #openstack-kolla03:09
*** haplo37_ has quit IRC03:09
mark-caseyprobably. I've never done it on centos. I usually try to find fast public mirrors in a nearby city03:09
*** haplo37_ has joined #openstack-kolla03:09
mark-caseythough I'm on ubntu03:09
Pavoyeah but if I wanted a completely offline system it would ideal03:10
mark-caseyright03:10
Pavoshould be able to just make a apache docker container and map the volume for /var/www/html to a filesystem share with all the repos03:10
PavoI would think03:11
*** dwalsh has joined #openstack-kolla03:33
*** dwalsh has quit IRC03:38
*** jtriley has joined #openstack-kolla03:56
*** daneyon_ has joined #openstack-kolla03:56
*** daneyon_ has quit IRC04:00
*** jtriley has quit IRC04:01
*** sdake has joined #openstack-kolla04:14
mark-caseysdake: ping04:22
sdakesup mark-casey04:22
mark-caseyon container portability... if you're on centos and ubuntu build fails, would you expect that bindmounting docker socket into an ubuntu container and trying again would help?04:23
mark-caseyi.e. is it the kernel or the os that is the problem04:23
sdakekernel syscall interface, rtnetlink04:24
sdakebindimoutning the docker socket wont fix those things04:24
mark-caseyyea, was afraid of that :)04:25
sdakemark-casey got a project underway: https://drive.google.com/open?id=0B8q6xDPETSkHZk01M1Z4YlN4d1E04:26
sdakei've moved far beyond that image - thank god04:26
mark-caseyheh, sweet!04:26
mark-caseyI remember you mentioning the... bah... head/pickup thing :D   is it using a needle?04:27
sdakenah its the tonearm04:29
sdakethe thing that holds the cartridge04:29
*** rstarmer has joined #openstack-kolla04:29
sdakethe cartridge holds the stylus (needle)04:29
mark-caseyright. makes sense04:29
sdakethis is a a closeup of the bearing assembly: https://drive.google.com/open?id=0B8q6xDPETSkHX0gyN18zMUhBOW804:31
mark-caseylooks very well engineered04:35
Pavois this going to be an issue? http://pastebin.com/CRUYk2PD04:43
Pavowhen I try to build those images it says binary not yet available for centos'04:44
rstarmeranyone seen an error in an all-in-one deployment TASK [common : Creating log volume], I'm getting a TypeError... http://pastebin.com/PuiU4vZy04:48
*** sdake has quit IRC04:49
mark-caseyPavo: I don't think so, not unless you need those services now; I don't think they'll be used for most "compute kit" or "compute kit+ceph" type setups04:51
Pavooh ok04:52
Pavogetting 2 fails on deploy04:52
Pavohttp://pastebin.com/0E0FN93Y04:52
Pavoand04:52
Pavohttp://pastebin.com/uHMn4tif04:52
mark-caseyrstarmer: retry with kolla-ansible -vvvv ... ? maybe more details there.04:55
mark-caseyPavo: it can't reach the registry. For example if you'd moved the images to your mac registry and hadn't updated globals.yml04:56
mark-caseywell, no. it said it got a 50004:56
mark-caseyoh, yea. ok. no route to host04:57
Pavoyeah I seen that now05:00
Pavotryikng to remove those images and getting (cannot be forced) - image has dependent child images05:00
mark-caseyI had that earlier tonight and there was one named <none>05:00
mark-caseywhich was the child I needed to remove05:00
rstarmermark-casey: I've not tried to debug the kolla_docker module, but that seems to be where it's complaining... http://pastebin.com/91DXugyW05:01
mark-caseyrstarmer: https://bugs.launchpad.net/kolla/+bug/158727905:06
openstackLaunchpad bug 1587279 in kolla "kolla-ansible deploy fail" [Critical,Confirmed]05:06
rstarmermark-casey:     # TODO(SamYaple): Replace with required_if when Ansible 2.0 lands05:08
rstarmer    if (module.params.get('action') != 'pull_image'05:08
rstarmerI guess that means I just need to make sure I have 1.9.4 installed... which gets me to my next error, but I'll rebuild and force 1.9.4.05:09
rstarmerare there "clean this up to nothing" instructions somewhere?05:09
mark-caseythere are clean-containers and clean-images scripts that need to run on the target nodes even though they're usually on the deployer node05:10
mark-caseyI do it like this: ansible -i multinode -m script -a '/kolla/tools/cleanup-containers' all05:10
mark-caseyansible -i multinode -m script -a '/kolla/tools/cleanup-images' all05:10
mark-caseythe thing in -a being the path on deploy host05:11
rstarmerthx05:11
*** fragatina has joined #openstack-kolla05:13
*** fragatina has quit IRC05:17
*** Pavo has quit IRC05:19
*** unicell has quit IRC05:19
*** unicell has joined #openstack-kolla05:19
*** sdake has joined #openstack-kolla05:31
*** zhubingbing has joined #openstack-kolla05:42
*** daneyon_ has joined #openstack-kolla05:44
*** daneyon_ has quit IRC05:48
*** sdake has quit IRC05:50
zhubingbinghello05:51
*** sdake has joined #openstack-kolla06:03
*** sdake has quit IRC06:07
*** senk_ has joined #openstack-kolla06:13
rstarmerany idea why, if I didn't set up a local registry, 'kolla-ansible deploy' is trying to pull a kolla-toolbox container from docker.io, and even a different version than in my globals.yml?06:14
rstarmernevermind, my mistake, I apparently _did_ set the version in globals...06:21
rstarmerall that trouble because of ansible 2.0... wow :(06:43
mark-caseyyeah. it was a pretty substantial refactor from 1.9.x06:44
mark-caseyhowever the porting has already been done in kolla master06:45
mark-caseyto use ansible 206:45
mark-caseylater all06:49
*** mark-casey has quit IRC06:50
*** unicell has quit IRC06:51
*** zhurong has joined #openstack-kolla07:11
*** zhurong has quit IRC07:24
*** bootsha has joined #openstack-kolla07:28
*** daneyon_ has joined #openstack-kolla07:32
*** daneyon_ has quit IRC07:37
*** bootsha has quit IRC08:18
zhubingbingShaun Smekel08:27
zhubingbinghi08:27
*** Jeffrey4l has joined #openstack-kolla08:30
*** Jeffrey4l has quit IRC08:38
*** bootsha has joined #openstack-kolla08:41
*** bootsha has quit IRC08:43
*** bootsha has joined #openstack-kolla08:45
*** zhubingbing_ has joined #openstack-kolla08:52
*** zhubingbing has quit IRC08:54
*** bootsha has quit IRC09:00
zhubingbing_Anybody here? who can help me?09:11
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the memcache location value in horizon local settings  https://review.openstack.org/35523209:17
*** Jeffrey4l has joined #openstack-kolla09:23
*** hieulq_ has joined #openstack-kolla09:28
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838809:32
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935109:41
*** hieulq_ has quit IRC09:43
*** hieulq_ has joined #openstack-kolla09:44
*** bootsha has joined #openstack-kolla09:50
*** opennode has joined #openstack-kolla10:09
*** duonghq has joined #openstack-kolla10:13
*** bootsha has quit IRC10:16
*** zhubingbing_ has quit IRC10:21
*** bootsha has joined #openstack-kolla10:27
*** senk__ has joined #openstack-kolla10:37
*** senk_ has quit IRC10:37
*** hieulq_ has quit IRC10:38
*** hieulq_ has joined #openstack-kolla10:48
*** vbel has quit IRC11:00
*** vbel has joined #openstack-kolla11:00
*** bootsha has quit IRC11:06
*** kbaegis has joined #openstack-kolla11:16
kbaegisHi all11:16
*** zhubingbing has joined #openstack-kolla11:25
*** zhurong has joined #openstack-kolla11:33
*** hieulq_ has quit IRC11:43
duonghqhello there11:46
*** bootsha has joined #openstack-kolla11:56
kbaegisAnyone know what the docker_namespace globals variable is used for?11:58
kbaegisI have a registry setup, but I can't find them with a curl11:58
kbaegis"curl https://crucible.lab:4000/v2/_catalog    {"repositories":["ubuntu"]}"11:58
kbaegisOh, my fault11:59
kbaegisnvm11:59
*** zhubingbing has quit IRC11:59
*** klint has quit IRC12:40
kbaegisHey, I can set the "openstack_release" to a blank string, right?12:42
kbaegiskolla-build didn't tag my images by the looks of it12:43
kbaegis"The requested image does not exist: crucible.lab:4000/kolla/centos-source-heka:"12:43
kbaegiswhereas: "curl https://crucible.lab:4000/v2/_catalog yields "kolla/centos-binary-heka"12:44
kbaegisOh12:44
*** kbaegis has quit IRC12:54
*** kbaegis has joined #openstack-kolla12:54
kbaegisEvidently forgot to reconfigure kolla-build.conf :)12:55
kbaegisAt least I got to configure a full 32 threads this time around :)12:55
duonghqkbaegis: So, I made that?12:56
*** zhubingbing has joined #openstack-kolla12:56
duonghq*you12:56
zhubingbinghi12:56
zhubingbingduonghq12:57
duonghqnice to see you, zhubingbing12:57
zhubingbingHow do you spend your weekends12:57
*** kbaegis has quit IRC12:58
duonghqthis weekend is really busy for me, hope that it is easier in Sep12:59
duonghqand you?12:59
*** kbaegis has joined #openstack-kolla13:02
duonghqkbaegis: is your Kolla ok right now?13:02
zhubingbingme too,so busy13:05
kbaegisduonghq Yeah, it's chugging along :)  Forgot to run it inside screen though13:06
duonghqnice13:07
kbaegisI'm the crazy guy trying to deploy kolla on gentoo.  HAProxy was broken earlier, so I'm trying the compile-from-source approach to see if the centos compiler chain doesn't miss something the packaged builds do13:07
*** tfukushima has joined #openstack-kolla13:07
kbaegisBeen a great learning opportunity for some of dockers edge-cases, configs, etc13:08
duonghqkbaegis: which branch do you use? master?13:09
kbaegisYeah.  Tried rolling back to mitaka, but that didn't work either13:10
kbaegiscontainer was stuck in an ugly loop13:10
duonghqhmm, you pushed this to new land, very interesting13:12
kbaegis:) I'll keep you guys updated13:12
kbaegisProvided it remains interesting13:12
duonghqHope that you write something like blog entry after everything is up and run.13:13
kbaegisI'll try to do that. :) I only have 3 machines in my lab: 2 d525 atoms, and my new 16 core xeon box13:15
kbaegisThe next step is to see if I can run the controllers in 2-4 gb of ram w/ swap13:15
kbaegisPurely as standby nodes13:15
duonghqall 3 machines run gentoo, huh?13:16
kbaegisYes :D13:18
duonghqlook forward to your result, I used to be Gentoo user :)13:20
*** bootsha has quit IRC13:23
*** bootsha has joined #openstack-kolla13:23
*** chrichip has quit IRC13:25
*** chrichip has joined #openstack-kolla13:25
*** dwalsh has joined #openstack-kolla13:25
kbaegisNot anymore?13:27
duonghqYeah, now I use Arch and Ubuntu13:27
kbaegisIt's a PITA if you don't have your own build server13:27
kbaegisIf you do, it's just like running canonical :)13:27
duonghqbut I really like emerge's slot of Gentoo13:28
kbaegisYes13:28
kbaegisThose are handy. My personal favorite though are the unlicensed binaries- ffmpeg and all13:28
kbaegisYou can compile them, but you can't redistribute13:28
kbaegisLots of codecs, etc13:29
duonghqyes13:30
duonghqsomeday I'll comeback to Gentoo13:33
*** bootsha has quit IRC13:33
kbaegisYeah, it's a cool community. It really makes you understand how stuff works. Gentoo: because linux isn't hard enough ;)13:34
duonghqagree13:36
duonghqtuning compiling process is really fun13:37
duonghq(compile arguments indeed)13:37
kbaegisI'm going to be setting up some heat templates and mistral flows for an autoscaling package compiler.  I think gentoo really needs something like that.  Step 1, though, is getting kolla up13:39
duonghqDo you think it's quite overkill?13:40
duonghq(IMO)13:40
kbaegisYeah, absolutely. :)13:40
kbaegisBut it's good fun and practice13:40
duonghqabsolutely13:41
*** duonghq has left #openstack-kolla13:46
*** duonghq has quit IRC13:46
openstackgerritMerged openstack/kolla: Add --user-domain when adding heat role admin  https://review.openstack.org/35347113:57
openstackgerritMerged openstack/kolla: Add upgrade tasks for Ceilometer  https://review.openstack.org/35100714:08
openstackgerritMerged openstack/kolla: Minor Readability Adjustments to Operating Kolla  https://review.openstack.org/34709914:08
kbaegisSoo.  Here's my kolla-build.conf: http://pastebin.com/016hg8ti14:13
kbaegiskolla build completes/pushes successfully. The images show up appropriately under $(docker images) but when I do curl https://crucible.lab:4000/v2/_catalog, they're nowhere to be fou d14:14
kbaegis*found14:14
openstackgerritMerged openstack/kolla: Customizations for Tempest  https://review.openstack.org/35439714:14
*** dwalsh has quit IRC14:17
kbaegisIOError: [Errno 2] No such file or directory: '/var/lib/kolla/config_files/config.json' in haproxy14:19
kbaegisweird. It looks like it's not copying the directories in14:20
kbaegisDoes that sound right?14:20
*** Pavo has joined #openstack-kolla14:21
kbaegisHuh.  looking at the dockerfile, this should work.  Somehow it's not executing the && operator14:25
kbaegisOr it's reading the \\n ineffectively14:25
kbaegisThe group gets created, but the mkdir -p doesn't14:30
kbaegisweird14:30
kbaegiswhen you run kolla-build, is that reading from the docker files under /usr/share/kolla/docker/?14:47
kbaegisI'm trying to figure out why the keepalived image is hanging "Restarting (1)"14:54
*** dave-mccowan has joined #openstack-kolla14:55
*** zhurong has quit IRC14:56
kbaegisSeems like it's crashing on INFO:__main__:Validating config file14:56
*** dave-mcc_ has joined #openstack-kolla14:58
*** dave-mccowan has quit IRC15:01
kbaegisis there an option in globals to disable the VIP?15:05
*** dave-mccowan has joined #openstack-kolla15:07
*** dave-mcc_ has quit IRC15:09
kbaegiswell, centos-source-keepalived is the ONLY thing I can't get working15:12
kbaegishaproxy is actually working15:13
Pavook I have on the deployer node a running registry but when I do a kolla-build --registry 192.168.1.50:4000 --push, getting an error during build that that host is trying to use v1 and is not responding, I have INSECURE_REGISTRY="--insecure-registry 192.168.1.50:4000" in /etc/sysconfig/docker15:21
Pavoany suggestions15:21
*** kbaegis has quit IRC15:32
*** kbaegis has joined #openstack-kolla15:32
kbaegisHey Pavo. Only way I got it working was by setting up a secure registry: https://docs.docker.com/registry/deploying/#get-a-certificate15:33
Pavook ty I'll take a look at that15:34
kbaegisThat seems to work fine.  Otherwise you'll get errors like "https sent to http endpoint" or something15:34
kbaegisMy actual command: docker run -d -p 192.168.254.3:4000:5000 --restart=always --name registry -v $(pwd)/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:215:35
Pavohow would that work for Kolla when it tries to use 4000 instead of 500015:35
kbaegisSee above.  It's the same15:35
Pavooh ok I see now15:35
Pavothat should be in Kolla documentation15:35
kbaegisYou may need DNS for a FQDN15:35
kbaegisWell, there's some complexity :)15:35
kbaegisI run BIND locally, but not everyone has that15:36
Pavook so how do I get the certs?15:39
kbaegismkdir certs&&openssl req -newkey rsa:8192 -nodes -sha512 -keyout certs/domain.key -x509 -days 3650 -out certs/domain.crt15:39
kbaegisfrom ~/15:39
Pavoah ok yeah forgot about oppenssl15:40
kbaegisYour OS has a specific way to then update the certs15:40
kbaegisI use gentoo, so mine is different15:40
Pavoyeah I use centos15:41
kbaegisca-update or something15:41
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the ubuntu xenial gate  https://review.openstack.org/35514916:23
openstackgerritJeffrey Zhang proposed openstack/kolla: Get packages from cdn rather than github  https://review.openstack.org/32906016:26
*** zhubingbing has quit IRC16:30
*** daneyon_ has joined #openstack-kolla16:34
*** daneyon_ has quit IRC16:38
kbaegisSo when I execute a kolla container with 'bash', I can't find a /var/lib/kolla directory17:10
kbaegisis this supposed to be in an attached volume, copied, in or created?17:11
kbaegisAll I see is that keepalived container is perpetually restarting.  I'm trying to fix it, but the logs are useless17:11
kbaegisthere's a bunch of junk under /etc/keepalived/keepalived.conf17:13
kbaegisI can't even tell if that's used though17:14
kbaegisIf anyone could explain this to me, it would be greatly appreciated17:17
kbaegisdone with a -v looks like17:21
*** tfukushima has quit IRC17:23
kbaegisHmm17:27
kbaegisWell, finally got it fully deployed.  HAProxy isn't assigning the VIP appropriately. Workaround: sudo ip addr add <VIP CIDR> dev <IF>17:48
kbaegisGoing to report it on the repo as a bug17:48
*** opennode has quit IRC18:01
*** senk__ has quit IRC18:04
*** dave-mccowan has quit IRC18:18
*** daneyon_ has joined #openstack-kolla18:22
*** tfukushima has joined #openstack-kolla18:24
*** daneyon_ has quit IRC18:27
*** tfukushima has quit IRC18:28
kbaegisneutron-server returning the following: http://pastebin.com/7KzJSMpD18:46
kbaegisbuild log: INFO:kolla.image.build.openstack-base:Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again18:47
*** dave-mccowan has joined #openstack-kolla18:55
*** dave-mcc_ has joined #openstack-kolla18:57
*** opennode has joined #openstack-kolla18:59
*** dave-mccowan has quit IRC19:00
*** kbaegis has quit IRC19:24
*** tfukushima has joined #openstack-kolla19:25
*** tfukushima has quit IRC19:30
*** kbaegis has joined #openstack-kolla19:49
Pavokbaegis ok I tried the solution you told me about the registry being TLS and still not working on deploy19:49
Pavogetting this error19:49
Pavohttp://pastebin.com/R6rX7pwA19:50
Pavoand I used this docker command19:50
Pavodocker run -d -p 192.168.1.50:4000:5000 --restart=always --name registry -v `pwd`/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:219:50
kbaegis"192.168.1.50:4000:"?19:55
kbaegisDid you have a spare : at the end?19:55
Pavoyeah I copied your command you posted19:55
kbaegisYou should be able to curl 192.168.1.50:4000/v2/_ping and get a 40419:55
kbaegisTry that19:55
Pavocurl: (7) Failed connect to 192.168.1.50:4000; Connection refused19:56
Pavobae090cb7b99        registry:2          "/entrypoint.sh /etc/"   11 minutes ago      Restarting (1) 4 minutes ago   192.168.1.50:4000->5000/tcp   registry19:56
kbaegisdocker log bae090cb7b9919:56
*** opennode has quit IRC19:57
kbaegisI'd also check your routing19:57
kbaegisIf your bridge has a default route out from the container19:57
Pavodocker: 'log' is not a docker command.19:57
kbaegislogs?19:57
kbaegisplural I think19:57
kbaegistry docker run -it ubuntu:latest bash and do a route print19:58
Pavohmmm seems like its not seeing my certs19:58
kbaegisOh19:59
kbaegisWell did you put them in ./certs/?19:59
Pavotime="2016-08-14T19:59:16Z" level=fatal msg="open /certs/domain.crt: no such file or directory"19:59
Pavothey are in root dir19:59
kbaegisFrom whereever you ran19:59
kbaegisdocker run -d -p 192.168.1.50:4000:5000 --restart=always --name registry -v `pwd`/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:2?19:59
kbaegisYou put them in absolute?19:59
kbaegispwd grabs your current path20:00
Pavoah20:00
kbaegisdocker run -d -p 192.168.1.50:4000:5000 --restart=always --name registry -v /certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:2?20:00
Pavostill not seeing them20:01
Pavogrrrr20:01
kbaegisHmm20:02
kbaegispermissions?20:02
Pavohttp://pastebin.com/EAtbK1Ai20:02
Pavomaybe20:02
kbaegisregenerate the self-signed cert20:02
Pavook20:02
kbaegiscd /&&mkdir certs&&openssl req -newkey rsa:8192 -nodes -sha512 -keyout certs/domain.key -x509 -days 3650 -out certs/domain.crt20:03
Pavook done20:05
kbaegisrerun the container :)20:06
Pavook think its working now20:06
Pavobut when I do curl 192.168.1.50:4000/v2/_ping nothing is returning20:07
PavoERROR:kolla.image.build.mariadb:Get https://192.168.1.50:4000/v1/_ping: x509: cannot validate certificate for 192.168.1.50 because it doesn't contain any IP SANs20:08
*** daneyon_ has joined #openstack-kolla20:10
*** daneyon_ has quit IRC20:15
*** kbaegis has quit IRC20:21
*** kbaegis has joined #openstack-kolla20:22
*** Pavo has quit IRC20:24
*** tfukushima has joined #openstack-kolla20:27
*** tfukushima has quit IRC20:31
*** Jeffrey4l_ has joined #openstack-kolla20:34
*** Jeffrey4l has quit IRC20:35
*** dave-mcc_ has quit IRC20:50
kbaegisHey Pavo, still around?21:01
kbaegisOkay, next problem21:12
kbaegisPath /run/netns is mounted on /run but it is not a shared mount21:12
kbaegisneutron-dhcp-agent21:12
kbaegiswell, tried chmod 777 /run/netns21:17
kbaegisLet's see if that absurdity fixes it21:18
kbaegisidempotent indeed21:18
*** tfukushima has joined #openstack-kolla21:28
*** tfukushima has quit IRC21:32
kbaegishttp://pastebin.com/vN1VRgCX21:33
kbaegisIs that really correct?  Keep getting "linux mounts: Path /run/netns is mounted on /run but it is not a shared mount"21:33
rstarmerhas anoyone run into a collision between haproxy and mariadb on an all-in-one install running from master?21:40
kbaegisYeah, I had issues21:42
kbaegisI'm running gentoo, though, which is the unsupportable red-headed stepchild21:42
rstarmerkbaegis: how did you resolve it? I think it's because I'm trying to deploy All-In-One but haproxy/keepalived are trying to run as well.21:59
sbezverkrstarmer: you can diasable keepalive in all-in-one21:59
kbaegisHow?  I was trying to figure that one out?22:00
kbaegisSo for the keepalived/haproxy22:00
kbaegisIt only looks for the VIP, which you can assign manually22:00
kbaegisMy workaround: sudo ip addr add <CIDR addr> dev <IF>)22:01
sbezverkin globals.yml: enable_keepalive: "no"22:01
kbaegistyvm22:03
kbaegisCouldn't find that anywhere22:03
rstarmerah, I see what happened, my setup ansible didn't fill in the VIP addr... fixing (I think...)22:03
kbaegisovs is still broken for me, and I can't figure out the shared /run/netns mount for dhcp agent22:04
sbezverkhave you changed mount=shared for docker?22:04
sbezverkmake sure this22:05
sbezverkmount --make-shared /run22:05
sbezverkcommand is persistent22:05
kbaegisOh, nice22:07
kbaegis'/run/netns is not mountpoint or bad option'22:08
sbezverkkbaegis: correction, the actual command is enable_haproxy: "no"22:08
kbaegissbezverk tyvm22:08
kbaegisSo if /run is part of my root fs, how do I use mount to remount it?22:09
sbezverkin globals, it will disable both haproxy and keepalived since yyou do not really need then with aio installation22:09
rstarmerthanks, that I can add to by build ansible...22:09
sbezverkyou need to run mount command exactl as I pasted22:10
sbezverkon the host (target) where you run kolla22:10
kbaegisDidn't return erros, tyvm :)22:10
kbaegisLet's see if this works22:10
kbaegisDisabled my manual vip assignment22:11
kbaegisdisabled ovs22:11
kbaegismay actually deploy throughout now :)22:11
kbaegis*crosses fingers*22:16
rstarmerkbaegis: likewise... I think I've just re-set things back to a single interfaces (with bridge/veth pair for my neutron external env.)22:26
rstarmerhope to have a running system here shortly22:26
kbaegisday 3 for me22:26
kbaegishorizon is up22:26
kbaegis:D22:26
kbaegis105 containers22:27
kbaegisNice22:27
rstarmerneutron keeps failing for me22:27
kbaegisWhat's the error you're getting?22:28
kbaegisI had to recompile my kernel with netfilter options22:28
rstarmerfatal: [localhost -> localhost]: FAILED! => {"changed": true, "failed": true, "msg": "Container exited with non-zero return code"}22:28
kbaegisand docker logs?22:28
rstarmerthere's no container to get logs from22:28
kbaegisdocker ps -a22:28
kbaegisThere should be one that's just "created" or "restarting"22:29
kbaegisYAY!!! First Kolla build22:29
kbaegis:)22:29
kbaegisAlbeit with 4 critical workarounds over 3 days22:29
rstarmerImportError: No module named fixtures22:29
kbaegisYeah, I had that issue too22:29
kbaegisI think master is broken22:29
kbaegisHere's my fix22:29
kbaegisneutron_plugin_agent: "linuxbridge"22:30
kbaegisI also went from centos-source to centos-binary, and rolled back to release 2.0.122:30
rstarmerI've got the agent for linux bridge already22:30
kbaegisI'm going to go check on my beautifully provisioned horizon setup and I'll be back22:30
rstarmerwill go try 2.0.122:30
kbaegisDid you set release tag to 2.0.1?22:31
rstarmerthanks,22:31
kbaegissure22:31
rstarmerno, I'll do that now22:31
kbaegishope that helps :)22:31
rstarmerme too!22:31
kbaegisI need to dig through the git logs next and send a email/bill for whoever wrote the regression22:31
rstarmerha! :D22:32
kbaegisActually, totally my fault for syncing from master22:32
kbaegisBut still :)22:32
kbaegisGreat learning opportunity22:32
kbaegisHorizon is listening on port 80, right?22:35
rstarmerwell, I tried building stable/mitaka with ansible 2.0... also a bad idea :P22:35
rstarmerhorizon should be on 8022:35
rstarmershould redirect to /dashboard/22:35
kbaegisbeautiful22:35
kbaegisHave to set up a new subdomain, but whatever :)22:40
rstarmerfor my next trick (once I get this process stabilized), I'm going to try to do a multi-machine config, will see what fun that creates.  Anyone know how to tell neutron to use a smaller MTU with this model?  Is there somewhere that I can pass the MTU parameter to the container?22:40
kbaegisForgot, horizon doesn't implement an easy relative path that I'm aware of22:40
kbaegisI think openvswitch is the appropriate place for that.  Why change MTU/MSS?22:40
rstarmerlooking at my env. it looks like the've dropped the /dashboard path anyway22:40
*** Pavo has joined #openstack-kolla22:41
*** dwalsh has joined #openstack-kolla22:41
rstarmerGood question.  I was thinking I'd have to tell neutron to use a smaller MTU, but if I do tenant networks over VXLAN, I think that's already baked in.  My "external" network is unfortunately only a bridge on the network node(s) anyway.22:42
*** dwalsh has quit IRC22:49
kbaegisHmm22:50
kbaegisSo I'm having issues with horizon.  I don't have access to that subnet (by design).  Anyone experienced with hiding horizon behind apache?22:50
kbaegisI need apache to work as a proxy22:50
kbaegiskolla is completely deployed :)22:51
rstarmerhave you tried an ssh tunnel just to verify?22:51
rstarmerssh user@host -L 8080:localhost:8022:51
rstarmer^ sort of thing?22:51
kbaegisYeah, I can curl it22:51
kbaegisIt's not routable, nor is it supposed to be22:51
kbaegismod_rewrite will be a last resort22:52
kbaegisBut should work22:52
rstarmerI thought it was just mod_proxy that you could configure22:52
rstarmerin a container...22:52
rstarmer:D22:53
rstarmersame error with 2.0.1 images22:53
rstarmerImportError: No module named fixtures22:53
kbaegisWeird.  I'm not getting that anymore after disabling22:54
kbaegisHmm22:54
kbaegisI can't remember the specific workaround, but I definitely remember running into that one22:54
kbaegisI thought I just reverted22:55
kbaegisHere's my functional config22:57
kbaegishttp://pastebin.com/raqnSdmB22:57
kbaegisIsn't mod_proxy supposed to alter the return addresses as well?23:01
rstarmerhmm, it's been a long time since I actually configured mod_proxy...23:01
rstarmerthanks for the config though, I'm going to reset to stable/mitaka ansible 1.9.4, etc. and will re-try with 2.0.123:02
Pavoman getting this private registry to work is a pain in the ass23:08
kbaegisYes it is23:08
kbaegisSorry, you dc'ed before I could help23:08
Pavono problem23:08
kbaegisSo there's a way to update your os'es certs23:09
Pavothe tls way is giving me nothing but 509 errors23:09
kbaegisupdate-ca-certificates or somesuch23:09
kbaegisYeah, if you can't curl https://<IP>:<PORT> without the -k flag, it won't work23:09
kbaegishttps://docs.docker.com/v1.7/registry/authentication/23:10
Pavothis is what I don't understand23:11
Pavofor the CN myregistrydomain.com23:11
Pavowhere to set that?23:11
kbaegis"cp auth/domain.crt /etc/pki/ca-trust/source/anchors/myregistrydomain.com.crt23:11
kbaegisupdate-ca-trust"23:11
kbaegisOh23:11
PavoI just used my IP23:11
kbaegisThat's the resolvable23:11
kbaegisIP won't work23:11
kbaegisYou can use hostname and /etc/hosts file23:11
kbaegisOr you can set up DNS23:11
Pavowell I have everything in hosts file23:11
kbaegisbecause the certificate encodes the fqdn or hostname in the cn field23:11
kbaegisBasically, it checks that "bob.com" belongs to bob23:12
Pavoso I should use something like deployer.localhost?23:12
kbaegissure23:12
kbaegisThat'll work23:12
Pavook I'll try that after this push gets done23:12
kbaegisregistry.local23:12
kbaegisSure23:12
kbaegisYou have to write that into cn when you gen your cert23:12
Pavook hang on23:13
kbaegisIt'll ask for common name, fqdn or somethign23:13
Pavoso like this23:14
Pavohttp://pastebin.com/BSg8iMks23:14
Pavoand my /etc/hosts file looks like this23:15
Pavohttp://pastebin.com/4hHF64Pa23:15
Pavolol ok now getting new error23:20
Pavo0cc151c648af23:20
Pavooops23:20
PavoERROR:kolla.image.build.elasticsearch:Get https://deploy.local:4000/v1/_ping: x509: certificate signed by unknown authority23:20
Pavowhy does it keep trying to use v1 of registry23:21
PavoI think thats the main issue23:21
rstarmerI got that working.https://github.com/kumulustech/kolla-multinode/blob/master/deploy.yml#L73-L12423:25
kbaegisPavo, I think it's just that it's an "unknown authority", i.e. self-signed cert23:26
kbaegisPlease read this: https://docs.docker.com/v1.7/registry/authentication/23:26
rstarmerhttps://github.com/kumulustech/kolla-multinode/blob/master/deploy.yml#L149-L15223:26
kbaegisthe command you need is: cp auth/domain.crt /etc/pki/ca-trust/source/anchors/myregistrydomain.com.crt ; update-ca-trust23:26
rstarmerbasically, if you set the cert up right, and tell docker to use it, you don't have to deal with the insecure registry.23:26
rstarmerthat seems to work for me as I can build my images and push them23:27
*** jtriley has joined #openstack-kolla23:27
Pavokbaegis so I would use cp certs/domain.crt /etc/pki/ca-trust/source/anchors/domain.crt ; update-ca-trust23:28
kbaegisYeah23:29
kbaegisOr23:29
kbaegisJust the crt23:29
kbaegisYes23:29
Pavook I did cp /certs/domain.crt /etc/pki/ca-trust/source/anchors/deploy.local.crt ; update-ca-trust23:29
kbaegisSweet23:29
Pavosame error23:29
kbaegisnow do a curl https://<ip>:<port>/v2/_ping23:30
kbaegisReally?23:30
kbaegisHmm23:30
kbaegisAnd it's centos?23:30
kbaegisthe directions I gave you were actually for RHEL23:30
kbaegisI'll be annoyed if they differ23:30
rstarmertry curl https://ip:port/v2/_containers23:30
rstarmerI don't think there's a _ping for v223:30
kbaegisYou're right23:30
kbaegiswhat he shouldn't get either way is an ssl rejection23:30
kbaegisw/o -k23:30
rstarmertrue23:30
rstarmerdid you restart docker daemon after fiddling with cets?23:31
rstarmercerts23:31
Pavo[root@deploy ~]# curl https://deploy.local:4000/v2/containers23:31
Pavo404 page not found23:31
rstarmer_containers23:32
Pavono but I will right nos23:32
Pavonow23:32
rstarmer_ is important23:32
rstarmerBut even the 404 is better, you are not getting a cert error23:32
rstarmerbut dockerd needs to pick that up as well23:32
Pavo[root@deploy ~]# curl https://deploy.local:4000/v2/_containers23:32
Pavo404 page not found23:32
Pavoe5dfca0c25c9        registry:2          "/entrypoint.sh /etc/"   9 minutes ago       Up 27 seconds       0.0.0.0:4000->5000/tcp   registry23:33
rstarmerI used to get '{[]}' as  a response23:34
rstarmeranything in the log?23:34
rstarmerdoes it complain about certs/TLS issues?23:35
Pavotime="2016-08-14T23:34:48Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.6.3 instance.id=04674853-2f7e-4852-bd13-e93ac029d8cf version=v2.5.023:36
Pavo192.168.1.50 - - [14/Aug/2016:23:35:04 +0000] "GET /v2/_containers HTTP/1.1" 404 19 "" "curl/7.29.0"23:36
Pavocurl still giving me 40423:36
Pavoguess I'm gonna start all over23:37
Pavogood thing I took a snapshot before I started lol23:37
rstarmerdid you try pushing something to the repo:23:38
rstarmerPull (or build) some image from the hub23:38
rstarmerdocker pull ubuntu23:38
rstarmerTag the image so that it points to your registry23:38
rstarmerdocker tag ubuntu localhost:4000/myfirstimage23:38
rstarmerPush it23:38
rstarmerdocker push localhost:4000/myfirstimage23:38
rstarmerthat's paraphrased from the docker instructions23:39
rstarmerchange the localhost:4000 to https://repo:4000/23:39
rstarmersee if it lets you push/pull23:39
rstarmerthough it really seems like it should have just been happy23:39
Pavono I guess I should have tried that before reverting lol23:39
PavoI'll try that if I have an issue again23:39
rstarmeralso, you can also launch the repo with a compose file, I felt that was more repeatable than the comand line (where I tended to forget something)...23:40
kbaegisits _catalog23:40
kbaegisNot _containers23:40
rstarmerand then there's that...23:41
rstarmermy compose file (which I called a template, even though there's nothing templated): https://github.com/kumulustech/kolla-multinode/blob/master/templates/registry_yml.j223:41
rstarmernote the cert directory mapping, which may be different for you23:41
rstarmerand I set a "secret" just so I wouldn't get the complaint about not having set one....23:42
Pavonice templates23:45
Pavohave you tried using pci passthrough or sr-iov with kolla yet?23:46
rstarmerhaven't, though there is no reason that it _shouldn't_ work.  Maybe my @Cisco friends will give me access to a system again where I could try something like that :D23:47
*** zhurong has joined #openstack-kolla23:51
*** jtriley has quit IRC23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!