Monday, 2015-09-28

*** achanda has quit IRC00:12
*** Kennan_Vacation2 has quit IRC00:25
*** Kennan_Vacation has joined #kolla00:30
*** Kennan_Vacation is now known as Kennan00:34
*** achanda has joined #kolla01:13
*** dims_ has quit IRC01:28
*** diogogmt has quit IRC01:28
*** diogogmt has joined #kolla01:29
*** Kennan2 has joined #kolla01:33
*** harmw_ has joined #kolla01:36
*** Kennan has quit IRC01:38
*** harmw has quit IRC01:38
*** erkules_ has joined #kolla01:39
*** dimsum__ has joined #kolla01:40
*** erkules has quit IRC01:42
*** dwalsh has joined #kolla01:57
*** CBR09 has joined #kolla02:12
*** CBR09 has quit IRC02:16
*** banix has joined #kolla02:27
*** klint has joined #kolla02:42
*** dimsum__ has quit IRC02:48
*** dimsum__ has joined #kolla03:09
*** dims_ has joined #kolla03:10
*** dwalsh has quit IRC03:10
*** dimsum__ has quit IRC03:14
*** unicell has quit IRC03:19
*** unicell has joined #kolla03:19
*** diogogmt has quit IRC03:24
*** dimsum__ has joined #kolla03:25
*** dimsum__ has quit IRC03:27
*** dims_ has quit IRC03:28
*** achanda has quit IRC03:32
*** achanda has joined #kolla03:33
*** banix has quit IRC03:41
*** achanda has quit IRC03:41
*** alisonh has quit IRC03:54
*** diga has joined #kolla03:56
*** alisonh has joined #kolla03:59
*** dimsum__ has joined #kolla04:27
*** dimsum__ has quit IRC04:34
*** achanda has joined #kolla05:12
openstackgerritSam Yaple proposed openstack/kolla: Fixes inconsistency in ansible inventories  https://review.openstack.org/22830605:38
*** tfukushima has joined #kolla05:42
openstackgerritSam Yaple proposed openstack/kolla: Unify commands in documentation  https://review.openstack.org/22830805:49
coolsvapSamYaple, there is no official date for rc's, we can do it today or at max tomorrow i suppose, its upto project how many rc's they need06:03
coolsvaphttps://wiki.openstack.org/wiki/Liberty_Release_Schedule06:03
SamYaplecoolsvap: ah. thanks for the info06:03
SamYaplethats good. i have quite a few patches that need to merge since kolla is b0rked pretty bad right now06:03
coolsvapSamYaple, but the liberty release should be by oct 1506:04
SamYapleyes06:04
*** diga has quit IRC06:04
SamYaplecoolsvap: if you would'nt mind hitting the review queue pretty hard, that would be appreciated06:04
coolsvapSamYaple, yes06:05
openstackgerritMerged openstack/kolla: Upgrade packages when installing openstack-clients  https://review.openstack.org/22616706:20
*** diga has joined #kolla06:29
*** dimsum__ has joined #kolla06:31
Kennan2Hi SamYaple:06:33
*** Kennan2 is now known as Kennan06:33
SamYaplehello Kennan206:33
SamYaplehow are you today?06:33
KennanI found msg: Destination directory /etc/kolla/nova-libvirt does not exist06:33
Kennanwhen I tried ansible deploy06:33
SamYapleKennan: https://review.openstack.org/#/c/228246/06:33
SamYaplethere are actually a few bugs introduced by my sloppy removal of config-external and others trying to rush for the sept 25th deadline. I have put up corrected patches for all issues i have seen06:35
*** dimsum__ has quit IRC06:36
KennanOK SamYaple: let me try that patch in my env06:37
SamYapleKennan: due to related changes openvswitch is also broken.06:40
SamYaplei have a patchseries up there to fix all that06:41
KennanSamYaple: do you mean this /opt/kolla/config-external.sh: No such file or directory ?06:43
SamYapleKennan: correct06:44
SamYaplesome of the services (openvswitch) are trying to source tha tfile and it doesnt exist06:45
SamYapleit never di anything, but the fact that it doesnt exist is bombing it out06:45
SamYaplecoolsvap: fyi all of the above problems in master are fix in this commit (and the following patchsets) https://review.openstack.org/#/c/228244/06:45
SamYaplei would call them top priority06:45
coolsvapSamYaple, yes, i am looking at those06:46
SamYapleI have tested the whole stream, but not neccessarily the individual patches06:47
SamYaplefor example, i break ceph between two of those patches, but nothing else breaks06:47
coolsvapSamYaple, alright thanks for the info06:48
*** itsuugo has joined #kolla06:56
*** achanda has quit IRC06:56
*** inc0 has joined #kolla06:59
*** achanda has joined #kolla06:59
*** achanda has quit IRC07:02
inc0howdy folks07:03
SamYapleinc0! wonderful07:03
SamYapleneutron-server b0rked07:03
SamYaplelogging related07:03
coolsvapKennan, though its not a Kolla patchset, but would you mind opening a tracker bug associated with the check-requirements job change set?07:03
inc0oh? how so?07:03
SamYapleinc0: ill fetch you the error. its wierd. i think its a neutron-server bug07:03
SamYaplebut its broke for sure07:03
Kennancoolsvap: do you mean I open a bug in kolla ?07:04
coolsvapKennan, yes07:05
Kennanok I would open a bug, and add comment in bugs about this link(code change)07:05
coolsvapa (tracker) bug07:05
coolsvap thanks!07:05
Kennanhi coolsvap:07:10
KennanI did not find bug type during bug creation07:10
Kennanhttps://bugs.launchpad.net/kolla/+bug/150032907:10
openstackLaunchpad bug 1500329 in kolla "[Tracker]Kolla needs related check requirements job and add it in requirements family" [Undecided,New]07:10
KennanI just create this bug as usual bug07:11
coolsvapKennan, its not a bug type as such07:11
coolsvapits tracker in the sense of its purpose, the changes you are making are not lost and are tracked07:11
Kennanso is it ok for this bug I created ?07:12
coolsvapyes07:12
Kennanok. thx07:13
*** tfukushima has quit IRC07:14
*** tfukushima has joined #kolla07:14
*** itsuugo has quit IRC07:16
*** itsuugo has joined #kolla07:17
*** mbound has joined #kolla07:20
*** akwasnie has joined #kolla07:30
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Swift logging to rsyslog  https://review.openstack.org/22788107:36
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog  https://review.openstack.org/22790007:36
inc0SamYaple, do we want logfile per every swift service?07:37
SamYapleyes, like everything else07:37
inc0duh, I don't like swift already07:38
SamYapleyup07:38
SamYapleinc0: btw, quite a few bug fixes/patches in the queue to fix the broken-in-several-ways master that is kolla right now07:38
*** CBR09 has joined #kolla07:39
mlilHi again! Since last time I asked for help I have now managed to get further, but now I've stopped on another error message that I'm not sure what to do about:07:39
mlilfailed: [controller1.opnfv] => {"failed": true, "parsed": false}07:39
mlilTraceback (most recent call last):07:39
mlil  File "/home/opnfv/.ansible/tmp/ansible-tmp-1443198399.34-42584921028986/docker", line 3185, in <module>07:39
mlil    main()07:39
mlil  File "/home/opnfv/.ansible/tmp/ansible-tmp-1443198399.34-42584921028986/docker", line 1482, in main07:39
mlil    docker_api_version = dict(required=False, default=DEFAULT_DOCKER_API_VERSION, type='str'),07:39
mlilNameError: global name 'DEFAULT_DOCKER_API_VERSION' is not defined07:39
mlilI guess I have missed some configuration, the question is which configuration?07:40
inc0SamYaple, which ones are most pressing07:41
inc0?07:41
CBR09Did you install docker-py>=1.2.0?07:44
*** jmccarthy has quit IRC07:46
CBR09mlil: you can try install docker-py and run again07:46
*** jmccarthy has joined #kolla07:47
mlilHmm, Ubuntu doesn't know what "docker-py", but I know I installed "docker-engine". I'll do some googling to see if the packet just needs another name...07:47
CBR09mlil: try pip install docker-py07:50
mlilWell, I did find python-docker package, I'll see if that works. If not, I'll try pip.07:51
SamYaplemlil: what version of ansible are you rinnung?08:05
SamYaplemlil: any packaged versoin of docker-py will likely be too old08:05
SamYapleinc0: https://review.openstack.org/#/c/228244/ starting with that patchstream and up are most pressing08:06
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Swift logging to rsyslog  https://review.openstack.org/22788108:06
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog  https://review.openstack.org/22790008:06
mlilWell, the python-docker itself is of version 0.5.3-2. So perhaps you're right SamYaple, it's too old.08:07
inc0SamYaple, care to review plz? Murano configs are in swift patchset, but I won't split it now I guess..08:07
mlilSamYaple: ansible is of version 1.9.308:07
SamYaplemlil: that is a good version, my guess is just too old docker-py08:08
SamYaplelike CBR09 said08:08
SamYapleinc gotta dock you for the murano in that patchset08:08
SamYapleinc0: ^08:08
SamYapleyou end up double adding them08:08
inc0no, look at murano08:09
inc0changes in config is not there any more08:09
inc0that's how git works08:09
inc0so when these 2 merges, it will be exactly the same effect08:09
SamYapleyou are wrong08:09
SamYaplehttps://review.openstack.org/#/c/227900/3/ansible/roles/common/templates/rsyslog.conf.j208:10
inc0hmm, strange then locally it looks just fine08:10
inc0well whatever, let me get this right08:10
SamYapleinc0: other comment in swift patch08:10
SamYapleplease review08:10
SamYapleyou didnt respond to the query before08:10
*** CBR09 has quit IRC08:11
inc0ahh must've missed that08:11
SamYapleso about log_name and log_level08:11
*** athomas has joined #kolla08:11
*** gfidente has joined #kolla08:12
SamYapleall the other patches have just use_syslog and syslog_facility08:12
inc0log name is required08:12
SamYaplethats going to be a problem08:12
inc0yeah, that's swift, that's different08:12
SamYapleare you sure they dont import oslo.logging08:12
inc0swift doesn't follow convention08:12
inc0pretty sure08:12
SamYaplehmmm then youll have to do a bit more templating in the swift conf08:14
SamYapleto setup the file name per service08:14
inc0it's not that bad08:15
SamYapleactually inc0 you should be able to use a variable for that08:17
inc0that's what I'm doing08:17
inc0service name08:17
inc0just let me test it out08:17
inc0(forgot to include that to patchset)08:17
SamYaplelog_facility = {{ service_name }}08:18
SamYapleyea thatll work08:18
inc0log_name, but yeah08:18
inc0that was my idea08:18
inc0hmm...haproxy is busted08:19
SamYaplelots of things are08:19
SamYaplesee review stream above08:19
inc0damn.08:20
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Swift logging to rsyslog  https://review.openstack.org/22788108:33
*** dimsum__ has joined #kolla08:33
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog  https://review.openstack.org/22790008:33
inc0guys, I'll leave log_level for now as I can't really test without it currently08:34
*** erkules_ is now known as erkules08:37
*** erkules has quit IRC08:37
*** erkules has joined #kolla08:37
*** dimsum__ has quit IRC08:38
SamYaplebut you tested murano right inc0?08:38
SamYaple;)08:38
inc02 days ago, yes08:39
SamYaplelol look at it again08:39
inc0damn, forgot to ctrl+s08:39
SamYaple:D08:40
inc0shame on me08:40
SamYapleshame shame shame08:40
SamYaplepbourke_: you around bud?08:40
pbourke_SamYaple: yeah, hi08:40
pbourke_hows it going08:40
SamYaplehey man. we gots some major breaking bugs in master right now08:41
pbourke_:/08:41
SamYapleill take the blame, but this patchset fixes it all08:41
SamYaplehttps://review.openstack.org/#/c/228244/08:41
SamYapleit corrects 4-5 bugs and then there are more corrections that arent as urgent in the queue08:41
SamYapleif you get some time we could use reviews across all of kolla today08:42
pbourke_sure thing08:42
SamYaplethanks man08:42
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog  https://review.openstack.org/22790008:43
SamYaplecome on inc0!08:43
inc0damn..08:44
SamYaplehaha one of those days08:44
inc0it's called Monday08:45
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog  https://review.openstack.org/22790008:45
SamYaplenote to all: https://review.openstack.org/#/c/227784/ that is a patch for project-config to fix DocImpact for Kolla, external +1s for pressure are welcome!08:46
pbourke_SamYaple: https://review.openstack.org/#/c/228244/1/ansible/roles/neutron/templates/openvswitch-db-server.json.j2 - should this be using rsyslog?08:51
SamYaplepbourke_: yea but thats not what that patch is for08:51
SamYapleits just a copy from the config-external code08:51
SamYaplerather start.sh08:51
SamYaplehey wait a minute arent you the one that complains when i combine unrelated patches?08:52
inc0no, that was me08:52
SamYaplei think pbourke_ has too...08:53
pbourke_i do :)08:53
SamYapleeither way, there are still a few services that need to be rsyslog'd pbourke_08:53
SamYapleopenvswitch and haproxy among them08:53
pbourke_makes sense if it was existing, that's fine08:53
inc0yeah, one thing at the time plz08:53
openstackgerritMerged openstack/kolla: Replace config-external - openvswitch  https://review.openstack.org/22824408:55
*** exploreshaifali has joined #kolla08:59
SamYaplehey pbourke_ when you file/close bugs would you mind setting the milestone and assigning it to yourself? it helps tracking and statistics09:00
SamYapleotherwise me and sdake are going back through and fixing it09:00
pbourke_ok09:01
*** inc0 has quit IRC09:03
SamYaple74 bugs fixed for rc1 release :)09:04
SamYapleprobably more if we merge in queue stuff09:04
*** inc0 has joined #kolla09:05
openstackgerritMerged openstack/kolla: Swift logging to rsyslog  https://review.openstack.org/22788109:08
*** achanda has joined #kolla09:10
pbourke_SamYaple: for removing set_configs - it seems not every component is using the new config mechanism yet. so is it correct to remove set_configs for these?09:10
pbourke_SamYaple: e.g. ceph https://review.openstack.org/#/c/228247/1/docker/ceph/ceph-osd/start.sh09:11
SamYaplepbourke_: yes but two patches up i fix that09:11
SamYaplethats the only one09:11
pbourke_ah I keep forgetting to check the dependencies09:11
SamYapleno it depends on that. i did the ordering wrong09:11
SamYaplebasically everythings is broken right now, only ceph will be broken once the set_configs removal merges09:12
SamYaplethen the following patches fix ceph09:12
*** inc0 has quit IRC09:14
*** inc0_ has joined #kolla09:14
openstackgerritMerged openstack/kolla: Replace config-external - rsyslog  https://review.openstack.org/22824509:14
mlilSamYaple: I now have non-working pip, regardless if I reinstall it or not, can't even get version text from it... Seems like I have to re-install my nodes. Just 5 machines to go so I should be finished any day now... I guess this was caused by changing my mind over how to install stuff. We'll see how the re-install goes. But thanks for the help so far!09:15
*** sdake has joined #kolla09:15
SamYaplemlil: if this is ubuntu you will need to relog from your user09:15
SamYaplewhen pip upgrading it will change pip install location and bash hashing wont keep up09:16
SamYapleeasiest fix is to relog09:16
SamYaplepbourke_: can you update/close this bug if it is not valid? https://bugs.launchpad.net/kolla/+bug/149845609:18
openstackLaunchpad bug 1498456 in kolla "The default vagrant provider should be virtualbox" [Undecided,New]09:18
mlilSamYaple: Oh, so I don't need to re-install then? Yepp, just needed to log out and in again. Interesting... Thanks for saving me the effort of re-installation! ;-)09:19
SamYapleyea the dist installed pip is in /usr/bin/pip, but when you `pip install -U pip` it gets installed in /usr/local/bin/pip09:19
SamYaplebash hashes things in such a way that it won't pick up on the new pip09:20
SamYaplecouple of ways to fix it, but relogging is easiest09:20
SamYaplevagrant people, someone please grab this bug: https://bugs.launchpad.net/kolla/+bug/149692009:22
openstackLaunchpad bug 1496920 in kolla "vagrant VM can not create virbr0" [Undecided,New]09:22
SamYapleKennan: are yo uaround?09:25
KennanHi SamYaple:09:25
SamYaplehi! docs question09:25
SamYaplehttps://review.openstack.org/#/c/228308/09:25
SamYaplecoolsvap has asked me to add a NOTE/warning09:25
SamYapleis there special blocks for that?09:25
SamYaplelike "NOTE: deployer is responsible to running with the appropriate privleges, some commands require root elevation"09:26
Kennanyes, add something like that, some commands require run with sudo09:27
SamYapleKennan: right but is there a special command block to get it to appear as a special warning/note section at docs.openstack.org09:29
* SamYaple doesn't know docs09:29
pbourke_tbh if you feel we have to add a note like that, the commands should stay as is09:30
KennanSamYaple: not know any special commands.09:30
pbourke_but its not a big deal09:31
SamYaplepbourke_: i dont think we need it09:32
SamYaplebut i dont care really09:32
pbourke_same09:32
inc0_yay all of openstack services are merged09:34
inc0_rsyslog loggin I mean09:35
openstackgerritMerged openstack/kolla: Create nova-libvirt directory before copying files  https://review.openstack.org/22824609:35
*** dimsum__ has joined #kolla09:36
SamYaplethanks Kennan!09:37
openstackgerritMerged openstack/kolla: Fixes inconsistency in ansible inventories  https://review.openstack.org/22830609:42
openstackgerritMerged openstack/kolla: Remove all fail sections for Ubuntu  https://review.openstack.org/21822509:42
*** dimsum__ has quit IRC09:45
*** dimsum__ has joined #kolla09:51
openstackgerritMerged openstack/kolla: Allow different defaults for different providers  https://review.openstack.org/22714709:51
inc0_guys, can you build ubuntu source openstack-base?09:51
SamYapleinc0_: yea why09:52
inc0_Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main binutils amd64 2.24-5ubuntu1309:52
inc0_seems like packages are busted somehow09:52
SamYapledid you build with --no-cache09:52
openstackgerritZhao Lei proposed openstack/kolla: Remove quotes from subshell call in bash script  https://review.openstack.org/22673409:52
inc0_trying09:53
SamYapleim going a full rebuild for you too inc0_09:54
inc0_it never hurts09:55
*** achanda has quit IRC09:55
sdakemorning09:55
inc0_hi sdake, feeling better?09:56
sdakestill alive09:56
SamYaplemorning sdake09:56
SamYaplewere working through them bugs and reviews09:56
SamYaplequite the team09:56
SamYapleinc0_: openstack-base builds for me no problem09:59
SamYaplemy guess is out-of-date apt cache pulling old package09:59
inc0_probably10:00
inc0_still building for me10:00
inc0_but we have network based on ToP - TCP over pidgeons10:02
SamYapleah good ole reliable RFC 114910:04
inc0_exactly10:05
*** dwalsh has joined #kolla10:07
openstackgerritSteven Dake proposed openstack/kolla: Implement a install_type and install_metatype  https://review.openstack.org/22484110:07
openstackgerritSteven Dake proposed openstack/kolla: Significant refactor of base image  https://review.openstack.org/22289310:07
openstackgerritMerged openstack/kolla: Bind novnc to single interface  https://review.openstack.org/22823810:10
openstackgerritSteven Dake proposed openstack/kolla: eat doesn't need to specify python-oslo-reports  https://review.openstack.org/22796710:11
*** exploreshaifali has quit IRC10:11
openstackgerritSteven Dake proposed openstack/kolla: Heat doesn't need to specify python-oslo-reports  https://review.openstack.org/22796710:12
* sdake wtb functional gating10:13
inc0_would be nice, yeah10:14
openstackgerritSteven Dake proposed openstack/kolla: Implement a install_type and install_metatype  https://review.openstack.org/22484110:17
openstackgerritSteven Dake proposed openstack/kolla: Heat doesn't need to specify python-oslo-reports  https://review.openstack.org/22796710:17
openstackgerritSteven Dake proposed openstack/kolla: Significant refactor of base image  https://review.openstack.org/22289310:17
sdakewhat is going on with the centos source gate?10:20
sdakeSamYaple didy ou fix the setuptools thing mandre was working on?10:22
SamYaplesdake: mandre did. i believe it merged10:22
inc0_sdake, it's undeterministic10:22
inc0_it breaks from time to time10:22
sdakeinc0 no it was breaking all the time10:22
openstackgerritMerged openstack/kolla: Unify commands in documentation  https://review.openstack.org/22830810:22
inc0_https://review.openstack.org/#/c/227900/ this is 2hrs ago and it works10:23
SamYaplesdake there is _something_ in the centos source base that is installing python things. find it and kill it10:23
SamYaplealso, someone needs to move the base files for centos to openstack-base, or I am going to and i can't gaurantee non-breakage10:23
inc0_I don't like non-breakage just before release...10:24
sdakewhat is pending work for rc1 while we have this discusison10:24
SamYaplethat is pending work10:25
SamYaplehttps://blueprints.launchpad.net/kolla/+spec/openstack-common-container10:25
sdakei get that, what else is pending work10:26
inc0_logging container if we include non-openstack services10:26
inc0_all of openstack is done and merged10:26
openstackgerritMerged openstack/kolla: Remove quotes from subshell call in bash script  https://review.openstack.org/22673410:26
sdakei've been out of action for about 3 weeks with back to back flu or reallybad colds10:27
sdakeso if someone could bring me up to date on what is left for rc1 it would be helpful :)10:27
sdakesam sent me an email yesterday about some breakage in teh code base , is all that fixed up?10:27
SamYaplesdake: some patches left to merge10:28
SamYaplehttps://review.openstack.org/#/c/228247/10:28
SamYaplethat one is the worst offender10:28
sdakei'll review that now10:28
sdakeanything not in the queue that needs attention?10:28
sdakebesides the common centos container?10:28
inc0_ironic is almost merged, but ansible part is still in review10:30
inc0_ceph was merged10:30
SamYapleyea that ansible stuff needs to merge so i can fix config-external on it10:30
inc0_yup, I've noticed it's still old-style10:30
sdakesamyaple this is -1'ed https://review.openstack.org/#/c/228248/10:32
*** pbourke_ has quit IRC10:33
SamYaplesdake: you did?10:33
SamYaplepbourke did10:33
sdakeno i did not10:33
SamYaplesorry i injected letters in my brain :)10:33
*** pbourke_ has joined #kolla10:34
openstackgerritMerged openstack/kolla: remove set_configs  https://review.openstack.org/22824710:34
*** diga has quit IRC10:35
openstackgerritSam Yaple proposed openstack/kolla: Allow conf files to be optional  https://review.openstack.org/22824810:36
openstackgerritSam Yaple proposed openstack/kolla: Convert ceph to json  https://review.openstack.org/22824910:36
sdakepbourke_ can you review https://review.openstack.org/#/c/224841/ and its children please10:38
sdakesamyaple i'll take care o the common refactor for centos10:38
openstackgerritMerged openstack/kolla: Stop haproxy from listening on ports it shouldnt  https://review.openstack.org/22823310:38
SamYaplethank you sdake10:38
SamYaplelet me know if you have questions, but you should be able to clean up the base alot like that10:39
sdakeinc0_ can yo utell me more of what is needed for logging container for non-openstack services?10:39
SamYaplebasically just repo enablement at that point10:39
sdakeright10:39
sdakethat is what i want the base image to be10:39
SamYapleinc0_: sdake: i have a patch for openvswitch and haproxy for syslog10:39
SamYapleim testing10:39
openstackgerritMerged openstack/kolla: bind haproxy to single interface  https://review.openstack.org/22823710:40
inc0_ovs, mariadb, rabbitmq, haproxy10:40
inc0_libvirt10:40
sdakesounds like 2 of 4 are being sorted out10:40
sdake2 of 510:40
SamYaplealso, the logging is... wierd. it doesnt currently log tracebacks into the log inc0_10:40
SamYapleit only logs those to stderr like always10:41
inc0_SamYaple, not these which stops service from working10:41
SamYaple(so docker logs neutroN_server)10:41
inc0_if you can't even run service, it won't use rsyslog10:41
sdakedoes deployment work atm?10:41
SamYaplesdake: neutron_server is borked due to logging10:41
SamYaplei think its a neutron-server bug though10:41
SamYapleinc0_: no all services dont log tracebacks to log10:42
sdakeso can we get it working10:42
SamYapleits not configured correctly10:42
sdakeneutron server taht is10:42
inc0_ok I'll look into it, just tell me reproduction steps plz10:42
SamYapleyea i think its a neutron bug though. i dug into it a little bit10:42
sdakeif we don't have deployment working i can't test a refactor of base10:42
inc0_or even better - file a but;)10:42
SamYapleinc0_: create an exception, any kind10:42
sdakefile a but?10:42
SamYaplethats how he likes it dont judge10:42
inc0_I've seen traces in log10:42
sdake;=)10:43
inc0_but I'll confirm that10:43
SamYapleinc0_: ok ill double check too10:43
sdakeSamYaple can you fix this neutron server thing so i can get deployment working with a bsae refactor?10:44
*** tfukushima has quit IRC10:44
SamYaplesdake: I couldn't fix it, no. i worked on it for about 30 minutes before other things10:45
SamYapleyou can augment the file ot turn of logging10:45
sdakewhat happens exactly?10:46
sdakeneutron sserver crashes or something?10:46
SamYaplerun it youll see the stack trace, but like i said it appears to be a bug10:46
sdakerc1 has been snapped of neutron10:48
sdakei wonder if they are doing an rc210:48
openstackgerritMerged openstack/kolla: Implement a install_type and install_metatype  https://review.openstack.org/22484110:49
sdakepbourke_ can you ack https://review.openstack.org/#/c/227967/ please10:50
sdakesamyaple on https://review.openstack.org/#/c/228248/2/docker/base/set_configs.py10:53
SamYapleyea10:53
sdakeif validate_source returns false, nothign happens in the set_configs code?10:53
SamYapleit returns to L162 and continues in that for loop to the next file10:54
SamYaplebasically it just skips the copy/set perms step10:54
openstackgerritMerged openstack/kolla: Significant refactor of base image  https://review.openstack.org/22289310:54
sdakewouldn't we prefer exiting over continuing on in a failed situation?10:54
sdakewhat is the "optional" thing for exactly?10:54
SamYaplesdake: look at the next patch10:54
SamYapleor the commit msg for that patch10:54
sdakeok will do10:54
sdakeSamYaple does ceph not have configurable ports?10:57
SamYapleyes and no, its not as simple as that to the point im not ok with including them just yet10:58
sdakeok so outstanding work is10:59
sdake1) neutron backtrace10:59
sdake2) syslog is missing 5 services for logging10:59
sdake3) openstack base needs a refactor for centos10:59
SamYaplea few non-ansible openstack services still have config-external11:00
SamYaplebut thats not breaking, just work needed to remove them11:00
SamYaple(work ive started)11:00
SamYaplethe only thing breaking is 111:00
sdakewell i'm not branching on a doa histroy ;)11:00
SamYapleworst case we can turn off logging for neutron-server and just put it on the todo list11:01
sdakesounds gtm11:01
SamYaplewould like to attempt a fix though11:01
*** inc0_ has quit IRC11:04
*** inc0 has joined #kolla11:05
sdakei think the plan then is to branch around 10am PST on tuesday11:06
sdake(in 24 hours)11:07
*** inc0 has quit IRC11:07
sdakeonce we have done a fix for neutron-server11:07
sdakewe cn backport other changes that don't make it in time11:07
*** inc0 has joined #kolla11:07
sdakeas part of rc211:07
SamYaplecool11:07
openstackgerritMerged openstack/kolla: Heat doesn't need to specify python-oslo-reports  https://review.openstack.org/22796711:07
sdakeno luke, I am your Father11:10
sdakeserach your feelings you know it to be true!11:10
openstackgerritSam Yaple proposed openstack/kolla: Rename ovs to openvswitch  https://review.openstack.org/22840411:13
SamYapleim really hoping to get that merged because if we dont it will cause lots of confusion later to change and i dont want confusion but i dont want it to stay like it is11:14
SamYapleit breaks my regex all the time (doesnt' build openvswitch)11:14
*** inc0 has quit IRC11:14
* SamYaple cross his fingers on the last minute change....11:14
*** inc0_ has joined #kolla11:14
*** tfukushima has joined #kolla11:18
sdakedoes anyone want to take this over11:20
sdakehttps://review.openstack.org/#/c/216112/11:20
sdakei think tim decided he wasn't going to fix it :)11:21
SamYaplesdake: I could never reproduce that11:22
openstackgerritMerged openstack/kolla: Allow conf files to be optional  https://review.openstack.org/22824811:22
sdakesamyaple one way is to killall -9 httpd11:22
sdakevia docker exec11:23
SamYaplewill attemp11:24
SamYaplet11:24
sdakei dont quite get why we are using pid files anyway11:24
sdakepid file is to protect against a process running twice11:24
sdakeour ssytem could never let that situation occur11:25
SamYapleyea for httpd/apache we should remove that in the config11:25
SamYapledoes keystone even bind out run?11:25
openstackgerritMerged openstack/kolla: Improve cleanup script  https://review.openstack.org/21770611:25
sdakei am not sure SamYaple11:25
openstackgerritMerged openstack/kolla: Convert ceph to json  https://review.openstack.org/22824911:26
SamYaplesdake: my 10gb and 750 ssds are coming in tomorrow11:26
sdakenice11:26
sdakethe 750 ssd rocks11:26
SamYapleyea i got one for each of my servers11:26
sdakei ordered a 8 core 2670 2nd cpu for my system11:27
SamYaplegoing ot split it between ceph cache tier and docker builds11:27
sdakei have ram to core inbalance11:27
SamYapleyea i run dual Intel(R) Xeon(R) CPU E5-2687W v211:27
sdakethat the 16 hyperthread cores will fix11:27
sdakeis that an 8 core?11:27
SamYapleyup11:27
*** inc0_ has quit IRC11:28
SamYaple4ghz11:28
sdakenice11:28
sdakethose are super epxensive as I recall11:28
SamYapleits lightning11:28
sdakeI went for cheap 2.6ghz11:28
SamYaplei do all my video encoding on it11:28
SamYapleyea its worth it. i crush me some numbers for sure11:28
sdakexeon smokes i7 anyway regardless of ghz11:28
SamYaplei think it has all those nice extra instruction sets for things11:28
SamYaplebetter silicon too11:29
SamYaplenot as much variance11:29
sdakethe main thing is xeon has a 20 or 25mb cache11:29
*** akwasnie has quit IRC11:29
sdakethat is a huge differentiating factor11:29
SamYaplejust L3 though11:29
SamYaplethat breaks cross processor anyway11:29
sdakestill huge ;)11:29
SamYapleso you have to do your affinity correctly11:29
sdakenuma should sort that out iirc11:30
SamYapleopenstack doesnt default to respecting numa IIRC11:30
sdakeif there are onfig options that i probably something we will want to reoslve11:30
SamYaplewe would need to verify it first, but yea safe defaults are cool11:31
sdakeinc0 rebase https://review.openstack.org/#/c/227900/ plz11:32
SamYapleoh boy. look at that review queue11:32
SamYaplehttps://review.openstack.org/#/q/status:open+project:openstack/kolla,n,z11:32
SamYaplesdake: i went through most of the bugs and blueprints earlier11:35
SamYaplewe can still solve some for rc111:35
SamYaplebut none are major i believe11:35
SamYaplehttps://bugs.launchpad.net/kolla/+bug/148574211:36
openstackLaunchpad bug 1485742 in kolla "global search and replace /opt/kolla with /var/lib/kolla" [Critical,Confirmed] - Assigned to Sam Yaple (s8m)11:36
SamYaplenow that set_config is all merged, I can do this bug11:36
*** gfidente has quit IRC11:36
SamYapleyou want this before rc1 sdake?11:36
sdakeyes11:36
sdakeor before liberty11:37
SamYaplek should be quick, then I can test11:37
sdakeso that is a #4 item of things to do11:37
sdakeprobably #2 on priority list11:37
sdakei am pushing master atm11:38
sdakeonce it wraps up i'll do a deploy11:38
SamYaplepsuhing where?11:38
sdaketo my registry11:38
SamYaplecool11:38
SamYaplesdake: https://bugs.launchpad.net/kolla/+bug/148570411:39
openstackLaunchpad bug 1485704 in kolla "remove cinder_volume_driver config option from globals.yml" [Medium,Triaged] - Assigned to Ryan Hallisey (rthall14)11:39
SamYapleneed your reply11:39
sdakedid the docker cats ever sort out the push operation for v2 registry?11:39
SamYaplein v1.9 i believe there are some improvements11:40
SamYaplebut the big improvement got reverted xD11:40
SamYapleinc0 sdake neutron-server stacktrace http://paste.fedoraproject.org/272182/44047714/11:41
SamYapleseems busted to me11:41
SamYaplesdake: neutron_agents also busted, just masked by supervisord11:43
SamYapleneutron syslog logging may be busted as a whole11:44
SamYaplehmmmm inc0 I am getting no logs from any services now11:45
sdakeinc0 is not in irc11:45
SamYapleahh you remove the privleged: True from the container! for shame!11:45
SamYaplehold on patch incoming11:46
*** itsuugo has quit IRC11:47
*** shardy_ has joined #kolla12:00
*** shardy has quit IRC12:00
*** itsuugo has joined #kolla12:03
sdakeSamYaple looks to me that a possible issue is neutron either supports logging to a file/directory OR syslog12:04
sdakebut not both at th esame time12:04
sdakeand we are likely specifying both with the dist setup12:04
SamYaplewell make the change and test, i havent figured it out and i got pulled away12:05
sdakethe facility is thus not being setup12:05
*** shardy_ has quit IRC12:05
sdakei dont hav ea solution just reading code12:05
*** shardy has joined #kolla12:05
*** mbound has quit IRC12:12
sdakeSamYaple what does this mean12:14
sdakeTASK: [haproxy | Ensuring latest haproxy config is used] **********************12:14
sdakefailed: [minime-02] => {"changed": false, "cmd": ["docker", "exec", "haproxy", "/opt/kolla/ensure_latest_config.sh"], "delta": "0:00:00.013635", "end": "2015-09-28 12:14:40.150424", "rc": 255, "start": "2015-09-28 12:14:40.136789", "stdout_lines": [], "warnings": []}12:14
sdakeFATAL: all hosts have already failed -- aborting12:14
sdakehaproxy is in a restart loop12:17
sdake[ALERT] 270/121622 (7) : parsing [/etc/haproxy/haproxy.cfg:29] : 'bind' : invalid address: '192.1.68.1.148' in '192.1.68.1.148:3306'12:17
sdakeoh i see the problem12:18
sdakenm12:18
*** tfukushi_ has joined #kolla12:32
*** tfukushima has quit IRC12:34
*** gfidente has joined #kolla12:34
*** gfidente has joined #kolla12:34
sdakesamyaple what does this line do? : https://github.com/openstack/oslo.log/blob/master/oslo_log/log.py#L29312:36
SamYapleit will call syslog.f() i believe12:38
SamYaplei have no idea why None is there12:38
SamYapleso it should call syslog.LOG_KERN()12:38
SamYaple" If the named attribute does not exist, default is returned if provided, otherwise AttributeError is raised."12:39
SamYapleso if LOG_KERN() doesn't exist, it returns None12:39
SamYaplewell i guess it doesnt have to be a function12:39
SamYaplesyslog.LOG_KERN12:39
SamYaplesdake: so all that is doing is checking what log facilties are available after the import https://docs.python.org/2/library/syslog.html12:41
SamYapleand if syslog wasn't imported successfully it wil lsay no log faciliets are vailable12:41
SamYapleso i bet syslog isn't being imported successfully12:42
sdakeit seems like syslog isn't imported successfully12:42
SamYapleim going rpdb it12:42
sdakebecause in the type error, it prints out no log facilities12:42
SamYapleyes i think we have both come to this conclusion now12:42
sdakeit just has an empty list12:42
sdakelooking at oslo.log, I dont see why syslog wouldn't import successuflly12:43
sdakehttps://github.com/openstack/oslo.log/blob/master/oslo_log/log.py#L3912:44
SamYapleit imports sucessfully for me12:44
SamYapleyea thats what i was saying earlier12:44
SamYapleif its none, then all things will be none12:45
SamYaplecause of the getattr12:45
dimsum__not sure if this is related...https://review.openstack.org/#/c/227688/12:45
*** dimsum__ is now known as dims12:45
SamYapleyea... thats right12:45
SamYaplei wonder how often the source binaries update12:45
SamYapleat tarballs.openstack12:46
SamYapleoh it doesn't matter since thats oslo.log12:46
SamYapledims: thanks thats exactly it12:46
SamYaplesdake i suggest a patch12:46
SamYapleotherwise we have to wait for a new version of oslo.log to snap and requirements to be updated :D12:47
* dims nods12:47
*** inc0 has joined #kolla12:48
*** inc0_ has joined #kolla12:51
*** itsuugo has quit IRC12:51
*** dwalsh_ has joined #kolla12:51
*** inc0 has quit IRC12:54
*** klint has quit IRC12:59
sdakesamyaple you suggest a patc hfor what12:59
SamYaplesdake: https://review.openstack.org/#/c/227688/13:00
SamYaplelike a .patch13:00
sdakewhy does this only happen in the neutron container?13:01
SamYapleim not seeing the other containers log either, but....13:01
sdakeso install oslo.log in openstack common container and patch it there?13:04
openstackgerritSam Yaple proposed openstack/kolla: Remove config-external for rsyslog  https://review.openstack.org/22844813:08
SamYaplesdake: that fixes the privelged thing for rsyslog13:08
SamYaplesdake: no because the other services seem fine...13:10
SamYaplegoing to check pip versions13:10
sdakeVersion: 1.11.1.dev213:10
SamYapleoslo.log==1.11.013:10
SamYaplethats keystone13:11
SamYapleoslo.log==1.11.013:11
SamYapleneutron13:11
SamYaplewierd13:11
sdakekeystone = Version: 1.11.1.dev213:11
sdakewith pip show13:11
SamYapleno thats the delorean stuff13:12
SamYaplebut either way we both have the issue13:12
inc0_I'm back13:12
SamYapleinc0_: https://review.openstack.org/22844813:12
SamYaplethat fixes the /dev/log creation13:12
SamYapleyou need privelged to create teh device node13:12
SamYaplemy container was borked13:12
inc0_SamYaple, but you don't create /dev/log, you create /var/lib/kolla/dev13:13
inc0_do you have correct perms on /var/lib/kolla?13:13
SamYapleinc0_: im jsut telling you what the logs say on a clean vanilla run13:14
SamYaplereruns work fine13:14
inc0_probably that was the case13:14
inc0_I'm not sure if privileged is required if you make /var/lib/kolla chmod 77713:14
SamYaplewhich we arenlt going to do of course13:14
inc0_(I did and it works for me)13:14
inc0_yeah I guess13:15
inc0_one way or another13:15
*** dims_ has joined #kolla13:15
inc0_privileged is cleaner I guess13:15
SamYaplewell be priv dropping in mitaka anyway13:15
SamYaplesome containers will still need privleged, but they wont be running as root internally13:15
SamYapleso itll be fine13:15
inc0_yeah, let's sort this out later then13:16
SamYapleinc0_: https://review.openstack.org/#/c/227900/ cliky the rebasey13:16
SamYapleor rebase. idk if web rebase will work13:17
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog  https://review.openstack.org/22790013:17
*** dims has quit IRC13:17
inc0_cliky the +2 SamYaple13:17
SamYaplealready +1'd13:17
inc0_thanks mate13:17
SamYapleanytime13:18
SamYaplesdake: https://bugs.launchpad.net/kolla/+bug/148967613:19
openstackLaunchpad bug 1489676 in kolla "Leftover /var/run files prevent Keystone start" [High,Triaged]13:19
SamYapleI cant reproduce that with ubuntu even with your suggestions13:19
sdakeok13:20
SamYaplepbourke_: you around for a +2? all tested I promise 13:11 < SamYaple> thats keystone13:21
SamYaplehttps://review.openstack.org/#/c/228404/13:21
*** mbound has joined #kolla13:22
*** mbound has quit IRC13:22
*** mbound has joined #kolla13:23
*** tfukushi_ has quit IRC13:26
SamYaplesdake: just applying the patch to neutron_server fixes it13:26
SamYapleno other containers have the issue wierd13:26
sdakewe should probably apply globally i suspect13:27
*** tfukushima has joined #kolla13:27
SamYapleim cool with that ill work it up13:27
sdakecool I was working on it but if you want to tackle it wfm :)13:27
sdakei'll be happy to test centos bin and sourc ewhen your done13:28
SamYapleoh you can do it no problem13:28
SamYapleor i will no problem13:28
SamYapletell me what to do13:28
sdakego ahead and do it13:28
*** tfukushima has quit IRC13:28
sdakeand i'll test centos rdo and source13:28
sdakejus tto make sur eit works everywhere13:29
openstackgerritMerged openstack/kolla: Rename ovs to openvswitch  https://review.openstack.org/22840413:29
sdakeas long as you can do it shortly (by today)13:29
*** tfukushima has joined #kolla13:31
*** rhallisey has joined #kolla13:33
rhalliseymorning13:33
sdakehey ryan13:34
rhalliseyreview q looks nice13:38
SamYaplerhallisey: we merrged like 30 patches this morning13:38
rhalliseynicce13:38
rhalliseyI was getting my first cavity filled13:39
rhalliseymy 23 year streak of no dental work ended today13:39
*** jtriley has joined #kolla13:40
*** sdake_ has joined #kolla13:44
*** sdake has quit IRC13:47
*** achanda has joined #kolla13:49
*** sdake_ has quit IRC13:50
*** clayton has quit IRC13:51
*** achanda has quit IRC13:51
openstackgerritMerged openstack/kolla: Murano logging to rsyslog  https://review.openstack.org/22790013:54
*** sdake has joined #kolla13:57
*** clayton has joined #kolla13:58
*** itsuugo has joined #kolla13:59
*** itsuugo has quit IRC14:01
SamYaplesdake: ive got tied up i cant submit that to tonight for that patch14:01
sdakeok i'll do it then14:01
sdakethanks14:02
jpeelerwhen does rc-1 get cut? i just noticed ironic never made it in14:02
sdakejpeeler i am branching liberty and tagging rc1 tuesday 10am PST14:02
sdakerecommend gettinga rebase, so we can merge your patch14:02
jpeeleryeah i'll look at it now14:03
SamYaplejpeeler: that ironic stuff won't work because of changes to config_external, but merge it anyway14:09
SamYapleill submit the final json stuff for it14:09
rhalliseywell it should still work14:10
rhalliseyit just will need to be changed14:10
SamYaplerhallisey: no it won't work14:10
SamYaplerhallisey: https://review.openstack.org/#/c/228247/14:10
SamYaplebut ive been waiting for his ironic patch to merge to finish that out14:11
SamYapleplus i have to add pxe support for ubuntu14:11
rhalliseyoh14:11
*** absubram has quit IRC14:16
inc0_sdake, I don't think I'll make it with libvirt logging till tomorrow14:20
inc0_or ovs14:20
*** achanda has joined #kolla14:22
SamYapleinc0_: ive got ovs and haproxy14:25
SamYaplei can look at the others tonight14:25
SamYapleill have a few hours to do it14:25
SamYaplebtw guys, did you notice how clean our start.sh got? and uniform?14:26
inc0_well I'll look at it tomorrow14:26
inc0_but I'm a bit afraid to rush it14:26
inc0_if we break something, it will be severe14:26
SamYapleI am going to probe the waters on removing a specific start.sh per container but pushing in a generic start.sh in the base container and sourcing a specific (if needed) script later14:27
SamYapleso no more required start.sh per image14:27
jpeelernice14:27
SamYapleno more duplicate code anywhere!14:27
SamYaplewithout symlinks!14:27
inc0_SamYaple, sounds like good improvement for Mitaka:P14:27
SamYapleyea but im going to try to jam it in liberty anyway lol14:28
openstackgerritJeff Peeler proposed openstack/kolla: Add Ansible support for Ironic  https://review.openstack.org/21974714:28
SamYaplejpeeler: im pretty sure your containers for irnoic are broken, but they will still build and deploy (just crash)14:29
SamYaplethat is ok ill fix in patch tonight, lets just get it merged. or you can fix now14:29
jpeelerWell... needs to be updated to use json config and what else?14:30
SamYaplejpeeler: thats it i believe, and the pxe for ubuntu14:31
SamYaplebut id be ok with that missing rc114:31
jpeeleri can probably do the config file changes now14:32
SamYapleok then the containers should launch if you do14:32
*** dims has joined #kolla14:35
SamYaplesdake: https://review.openstack.org/#/c/228448/ you can mark that off your list14:35
*** dims_ has quit IRC14:35
inc0_SamYaple, all of CI builds failed there:S14:39
inc0_also failed to build rsyslog14:40
*** achanda has quit IRC14:40
sdakesamyaple INFO:__main__:base: ---> 991b8bb114a814:42
sdakeINFO:__main__:base:Step 14 : RUN patch -p1 -d /usr/lib/python2.7/site-packages < /0001-Fix-unintended-assignment-of-syslog.patch14:42
sdakeINFO:__main__:base: ---> Running in 031bcdad12ca14:42
sdakeINFO:__main__:base:patching file oslo_log/log.py14:42
sdakeINFO:__main__:base:Reversed (or previously applied) patch detected!  Assume -R? [n]14:42
sdakeINFO:__main__:base:Apply anyway? [n]14:42
sdakeINFO:__main__:base:Skipping patch.14:42
sdakeINFO:__main__:base:1 out of 1 hunk ignored -- saving rejects to file oslo_log/log.py.rej14:42
sdakeINFO:__main__:base:Removing intermediate container 031bcdad12ca14:42
sdakeso it looks like delorean has the patch applied already14:42
*** diga has joined #kolla14:47
*** inc0_ has quit IRC14:53
*** absubram has joined #kolla14:59
*** banix_ has joined #kolla15:05
*** jtriley has quit IRC15:07
sdakeSamYaple ping?15:16
*** jasonamyers has joined #kolla15:22
*** jtriley has joined #kolla15:23
*** banix_ is now known as banix15:24
*** absubram has quit IRC15:25
*** inc0 has joined #kolla15:27
SamYaplesdake: pong15:38
SamYapleabout to sleep though15:38
sdakehey sam15:38
sdakeso not sure why patch is saying the patch is already applied15:39
sdakebut it is clearly not15:39
sdakesome wierdness going on15:39
sdakebut i'll sort it out, sleep well :)15:39
SamYapleok feel free to submit a review and ill update it tonight if its still broke15:39
SamYaplethen test the agents cause they were busted too15:39
sdakeya feel free to hack on my change if you dont like it15:39
sdakegoal is to get it in fast15:39
SamYaplebut ill work on it tonight to get it all in tiptop shape15:40
sdakei'll try to test deploy of ubuntu when i get it in working order15:41
*** mat425 has joined #kolla15:41
SamYaplethanks sdake15:42
sdakepriority #1 for me atm15:42
*** inc0 has quit IRC15:45
mat425Hi all guys, i'm trying kolla, but i'm instantly sucked with an issue :(. For dev purpose i tried kolla using Vagrant, so i basically followed the steps on github. but once vagrant up command arrives on "Configuring and enabling network interfaces" it fails, the error now is "No guest IP was given to the Vagrant core NFS helper. This is an internal error that should be reported as a bug." but before i got a generic one15:46
*** dims has quit IRC15:46
*** dims has joined #kolla15:46
sdakemat425 the person your looking for is mandre, but he is asleep atm I think15:46
sdakeyou ahve to catch him about 5 hours earlier15:46
SamYaplepbourke_ may be able to help15:47
*** inc0 has joined #kolla15:47
sdakemat425 i'd highly recommend just doing an AIO deploy on vm or bare metal to get started15:47
sdakerather then use vagrant15:47
sdakeour vagrant dev env is a little bit pre-alpha quality ;)15:47
sdakeit just went through a major refactor to add libvirt support15:47
sdakeyour running on linux, correct?15:47
SamYaplei would say its solid beta15:47
sdakeok solid beta15:47
mat425ah okok got it15:47
sdakestill not rockin quality like the rest of kolla15:48
mat425yes running linux, ubuntu 14.0415:48
sdakewhich is deployable now15:48
sdakemat425 i haven't tried out the new vagrant work which added the libvirt port15:48
sdakeso I don't know how to help you15:48
mat425i had to install nfs and libvirt even though i've not really got why15:48
pbourke_mat425: you'd be better using virtualbox imo15:48
sdakepbourke_ could possibly help too, but he is probably gone (2 hours earlier yous hould be able to catch him)15:48
sdakemat425 are you on vbox?15:48
mat425vagrant with virtualbox yes15:49
sdakeahh, well pbourke is your man then :) he knows that part of the system well15:49
pbourke_mat425: change this line to "virtualbox" https://github.com/openstack/kolla/blob/master/vagrant/Vagrantfile#L715:49
mat425i was trying vagrant just for a fast deployment, i have to deploy it on a cluster later on15:49
sdakeya vagrant is good for eval15:50
pbourke_and if you want multinode change this line to true https://github.com/openstack/kolla/blob/master/vagrant/Vagrantfile#L2515:50
sdakejust not sure what state its in currently because I haven't personally tested it15:50
pbourke_then tell tell everyone how I was right in that virtualbox should be the defualt ;)15:50
sdakepbourke fwiw I can't get vbox to run on fedora15:50
sdakebut i dont mind changing the default back to vbox15:51
pbourke_ah I dont want to go down that road again15:51
sdakewas that issue never settled?15:51
pbourke_we never put it to a vote15:51
sdakesorry, I've been mia for a bit, been down with back to back flu/cold15:51
sdakedid we need to vote on it?15:51
sdakei'll kick that off if there is a logjam on the issue15:51
mat425ok good, i'm gonna try it right now, but anyway it's ok for me to install it on a clean new virtualbox15:51
sdakei'd rather not havea vote on every little detail but this is a big detail15:52
mat425(still with vagrant)15:52
pbourke_yeah I think its divided enough that a vote would help. its not a big deal either way but when I see newcomers getting stuck...15:52
mat425by the way thanks15:52
pbourke_its like one less thing for them to trip up on15:52
sdakepbourke is it not documented how to set vbox vs libvirt?15:53
mat425actually it was really clear how to do it15:54
sdakepbourke beyond the deafult setting, was there any other issue that needed community agreement?15:54
sdakeor gaps in the implementation choices?15:54
pbourke_sdake: not that Im aware of15:54
sdake(so I can craft the email)15:55
sdakeso its just the "what hsould the default setting be"?15:55
pbourke_yes15:55
sdakeroger i'll send an email out today15:55
pbourke_we're not going to fall out over it but would be good to put it to bed once and for all I think15:55
pbourke_thx15:55
sdakeagreed, we use votes to solve logjams about policy15:56
sdakeif tis a technical issue, we should just be able to fix that with patches15:56
sdakebut this sounds like a policy issue (defaults are policy issues)15:56
sdakemat425 note if your lauching your vms inside a vm, you need to add an augmentation blob15:57
sdakeits in the documentation15:57
mat425another question, do you think that, once deployed, adding extra modules is still that easy? (i need also to install Monasca and Openstack-Neat)15:57
sdake(launching vms ifrom nova inside a vm)15:57
sdakeso for monasca, we would probably want to add a container set for that and ansible playbook stuff for it15:58
sdakei am not familiar with openstack-neat15:58
sdakebut we dont' typically containerize stuff that isn't under big tent governance15:58
sdakealthough if someone submits code, we will merge it15:58
sdakeas for doing the work, its "easy" for someone experienced inthe process and hard for someone not :)15:58
pbourke_to answer the question though, I dont think there should be anything stopping you adding these once deployed15:59
sdakeright15:59
pbourke_even if outside containers15:59
sdakeit would probably take a cor ereviever about 8-16 hours to add a new service15:59
sdakewow, 301 plays15:59
sdakekolla getting busy :)15:59
mat425ok thank you15:59
sdakeand we are totally open to accepting these features into master16:00
sdake(if you do the work for example)16:00
sdakeour objective is to deploy the big tent in containers16:00
sdakemat425 we really want to make sure you have a good experience in your deployment, so feel free to hit up the devs with your questions16:02
sdakemat425 may I ask - you mentioned youw ere goin to do a deployment later, how many nodes?16:02
mat425thanks! i've already asked 2 questions here and the answers were fast and complete, i totally gonna hit this chat :)16:04
mat425i'm working on my thesis (MSc), at uni we have a 4 machine cluster16:05
mat425a little one16:05
sdakecool16:05
sdakewell kolla will defiantely work for that envrionment16:05
sdakeover 100 nodes and i'm not clear how well things would work out of the box16:05
mat425Maybe (with a big M) i can test my work on a bigger cluster at hp16:05
sdake(which is why I asked)16:05
mat425ok16:05
mat425i had a look also to the multinode set up (still in vagrant), don't really get why a cluster of 5 nodes needs 3 controller16:06
mat425:\16:06
pbourke_mat425: I've been meaning to change it to 216:09
*** blahRus has joined #kolla16:09
pbourke_mat425: 2 is min for HA16:09
mat425what is HA?16:10
*** inc0 has quit IRC16:10
pbourke_high availability16:12
mat425ah yeah sure16:12
mat425o16:12
mat425ok16:12
*** cloudnau_ has joined #kolla16:16
*** unicell has quit IRC16:17
*** inc0 has joined #kolla16:35
*** Kennan2 has joined #kolla16:36
*** Kennan has quit IRC16:36
sdake3 controllers are needed pbourke for ha16:37
sdake2 wont do it16:37
sdakepbourke_ ^^16:37
pbourke_oh16:38
pbourke_is that because of galera?16:38
sdakei think so16:38
openstackgerritJeff Peeler proposed openstack/kolla: Add Ansible support for Ironic  https://review.openstack.org/21974716:38
sdakei could be mistaken16:38
*** achanda has joined #kolla16:40
pbourke_ok good to know16:43
sdakesam would know for certain16:43
sdakeeveryone tells me 3 nodes16:43
sdakeso I just paroot :)16:43
sdakeparrot16:43
*** tfukushima has quit IRC16:43
inc0yeah it's for galera16:43
inc0either 3 or 5...odd number anyway16:43
inc0sdake, you of all people should know mr. corosync;)16:44
sdakecorosync doesn't car eabout quorum16:44
sdakeEVS doesn't have any notion of quorum :)16:44
sdakeEVS is designed to split brain16:44
sdakehow proper distributed systems should be made imo16:44
sdakebut nobody can work with it because it requires more brain power to iplement systems on top of it :)16:45
inc0same goes for riak as far as I know16:45
inc0it behaves very well in terms of HA, but it's so twisted noone knows how to use it16:46
*** sdake_ has joined #kolla16:46
*** sdake has quit IRC16:49
*** achanda has quit IRC16:54
*** unicell has joined #kolla16:58
*** inc0 has quit IRC17:06
*** mbound has quit IRC17:07
*** rhallisey has quit IRC17:34
*** athomas has quit IRC17:48
*** achanda has joined #kolla17:54
*** loth has joined #kolla17:56
*** diga has quit IRC17:58
*** dolpher has joined #kolla17:59
*** achanda has quit IRC17:59
*** athomas has joined #kolla18:00
*** mbound has joined #kolla18:07
*** mbound has quit IRC18:13
*** mbound has joined #kolla18:18
openstackgerritJeff Peeler proposed openstack/kolla: Add Ansible support for Ironic  https://review.openstack.org/21974718:23
*** sdake has joined #kolla18:28
*** sdake_ has quit IRC18:31
sdakejpeeler is rhallisey about?18:33
sdakeI have an emergency with the neutron agents being busted18:33
sdakethat is a result of the json config changes18:33
*** akwasnie has joined #kolla18:40
*** jasonamyers has quit IRC18:47
*** achanda has joined #kolla18:56
*** achanda has quit IRC19:00
jpeelerhe appears to be gone19:01
jpeelerah he just came back19:01
*** rhallisey has joined #kolla19:01
jpeelerrhallisey: sdake was looking for you 30 minutes ago. neutron agents json config related19:02
sdakerhallisey this is blocking the release of rc1: https://bugs.launchpad.net/kolla/+bug/150056619:02
openstackLaunchpad bug 1500566 in kolla "neutron_agents is broken with set_configs JSON" [Critical,Confirmed] - Assigned to Sam Yaple (s8m)19:02
sdakecan you fix it please19:02
sdakei want to tag and branch asap19:02
sdake(10am tomorrow)19:02
*** banix has quit IRC19:03
rhalliseywhat specifically is broekn19:03
rhalliseyoh I see the paste one second19:03
rhalliseyoh I didn't write a patch for this one19:04
rhalliseysince there are like 4 services in there19:04
rhalliseyI'll take a look though19:04
rhalliseysam must've patched it19:05
rhalliseyI didn't think the new config stuff would work because there are lik 4 services19:05
*** athomas has quit IRC19:06
rhalliseyoh actually19:08
sdakejpeeler really nice job on ironic19:08
rhalliseyI think I know..19:08
sdakereally hard technical work - hope it was rewrading :)19:08
sdakewell its busted19:08
sdakeit needs to be fixed ;)19:08
jpeelersdake: thanks, but it's sort of limited as you know and a bit rushed19:09
sdakeyup19:09
jpeelerhopefully can get it fully good next release cycle19:09
sdakeit is limited19:09
sdakebut we will make it rockstar status in the next release :)19:09
sdakeit is till really nicely done19:09
sdakegiven the limitations of ironic19:09
*** itsuugo has joined #kolla19:11
sdakejpeeler isn't pxe setup for ubuntu same as for rpm distros?19:12
rhalliseysdake, we're going to break apart neutron_agents in the future right?19:12
jpeelersdake: actually i think it's slightly more complicated. the files need some copying as the packages are different (i think)19:13
sdakeit doesn't matter what we do in the future, we are releasing rc1 based upon a fat container19:13
sdakejpeeler ok19:13
sdakerhallisey please review https://review.openstack.org/#/c/219747/19:14
sdakeneeds to merge19:14
sdakeunless there is some critical problem19:14
sdakejpeeler please file a tech debt bug and assign to rc1 for the pxe + ubuntu stuff19:14
sdakeif you dont' want sam busting your implementation, i'd suggest adding the json config stuff in after this patch hits the repo jpeeler19:15
jpeeleralready did :)19:15
*** dolpher has quit IRC19:16
rhalliseysdake, the way the json stuff works will require not having a fat container.  But I can just add back in the config-external bits just for neutron agents.  Unless I figure out a way around it.19:16
sdakewrong answer19:16
sdakeboth of those are wrong answers :)19:17
sdakemake it work with json19:17
sdakejust refactor set_config19:17
sdakeconfig-external is gone19:17
sdakesam removed he infrastructure ot support it19:17
*** marun has joined #kolla19:17
sdakeironic is probablybroken at this moment as well19:17
rhalliseyit would be an easy fix.  Just put it into start.s19:17
rhalliseystart.sh19:17
marunI'm working on deploying just neutron with kolla19:18
rhalliseyI would be changing set_configs for something that will go away in the future19:18
jpeelersdake: i updated the ironic patch, it should be completely on par as before19:18
marunI'm seeeing a hard vm crash when ovs containers are started.19:18
sdakejpeeler do you use the new json config method?19:18
jpeelersdake: yes19:18
rhalliseyI don't want to refactor the python script to account for something that we are fixing later19:18
marunHas anyone else seen this?19:18
sdakemarun have never seen that, but atm neutron_agents is completely borked19:19
sdakei really wish people woudl *TEST* their changes19:19
sdakeok rhallisey19:19
sdakewhat do you propose we do?19:19
sdakenot release rc1?19:19
rhalliseyno your misunderstanding.  I skipped this piece because I knew it wouldn't work19:20
sdakecan you explain why it can't work?19:20
rhalliseylet me just write a patch for it19:20
sdakeit could be made to work19:20
rhalliseybecause we have 4 services and the python script looks for 1 config,json19:20
sdakeintel 750 PCIE SSD = lifesaver19:20
sdakecan't you just hae the 1 config json for al four services anyway?19:21
rhalliseyya I could, but there would also be 4 commands in there19:21
sdakeand ahve it copy over a directoyr structure or something?19:21
sdakei see, so the problem is the commands not hte copy files19:21
rhalliseyI can do the copy19:21
rhalliseythen hard code the commands in start.sh19:22
sdakeok lets do that19:22
rhalliseyyes axactly19:22
sdakecool winnign :)19:22
rhalliseyya no problem.  I was aware of this issue when going through19:22
sdakethanks for helping me throug hthat19:22
sdakesorry if i seem shsort, feeling sickly19:22
rhalliseyno worries19:22
sdakeya I think sam wasn't when he completely removed config-external infrastructure support in kolla-common.sh19:22
rhalliseyI forgot to tell sam this when he ripped out config-external19:22
rhalliseyI was going to do that part but sam kinda just jumped in19:23
sdakeya he does that :)19:23
sdakethat is why he gets paid the big bucks19:23
sdakeor something :)19:23
sdakeremoving config-external is good19:23
sdakea hardcode in this case is the right answer19:24
sdakejpeeler can you write a tech debt bug for the pxe + ubuntu support19:24
jpeelerhttps://bugs.launchpad.net/kolla/+bug/150057719:24
openstackLaunchpad bug 1500577 in kolla "ironic needs discoverd service for source and PXE service for deb binary" [Undecided,New] - Assigned to Sam Yaple (s8m)19:24
sdakerhallisey can you ack that ironic patch19:24
sdakemarun so, once rhallisey fixes this problem, I'd suggest trying his change19:25
marunsdake: fix for neutron agents?19:25
rhalliseyI'll review in a moment19:25
sdakebut another possiblity is your neutron network and management network are not set properly in your globals.yml19:25
sdakemarun ack19:25
rhalliseylet me just push this.  Should take a second19:25
marunsdake: I'm not sure it's a networking issue.  my vm literally dies19:25
sdakeyou mean the qemu process exits?19:26
marunsdake: vmware fusion on os x19:26
marunsdake: but same thing, yeah19:26
marunsdake: although....19:26
marunsdake: maybe it's just losing connectivity :/19:27
sdakeso your running kolla within a vm and deploy a vm ?19:27
sdakemarun right that is my suspicion, your networks are not setup properly19:27
sdakethe neutron network has to not be your management network19:27
sdakeif it is, it will lose connectivity to the vm19:27
*** banix has joined #kolla19:28
marunsdake: looks like vm actually dies, actually19:28
sdakehrm19:29
sdakedid you override libvirt to use qemu instead of kvm for hypervisoring?19:29
marunsdake: not deploying kvm19:29
*** erkules_ has joined #kolla19:29
sdakeyour booting with nova yes?19:29
marunsdake: No19:29
sdakeok, confused :)19:30
marunsdake: my intention is to be able to deploy neutron with kolla, and just neutron.19:30
marunsdake: the first part is getting things deploying19:30
sdakei heard he part about your only using ovs19:30
sdakequick Q< why do you want to do that?19:30
marunsdake: the second part is to get a fake hypervisor working, think container running docker-in-docker instead of kvm19:30
sdakejust wondering on the use case19:30
*** erkules has quit IRC19:31
marunsdake: neutron development shouldn't require any other openstack service19:31
marunsdake: it adds unnecessary complexity19:31
sdakeis that maru newby?19:31
marunyes19:31
sdakehey o/19:31
marunsdake: I'm helping with openshift on openstack integration, and having to work with a full openstack deployment isn't really necessary for most of the work19:31
sdakeok, well i'm not totally sure about how to make your use cae work, but i'd start with getting a full stack operational in your fusion hypervisor19:32
sdakeocne that works, you can trim the fat19:32
sdakedoes that make sense?19:32
sdakei dont know if anyone has ever deployed on fusion before19:32
sdakemost people on osx use vbox19:32
sdakeor all :)19:32
marunsdake: Maybe I should switch to vbox19:33
marunsdake: since I'm not doing nested virt19:33
sdakeworth a go19:33
sdakeeven fi you do nested virt, you can run qemu for the virt19:33
marunsdake: fusion on osx is the only way to have accelerated nested virt.  vbox doesn't support19:33
sdakei'd really recommend tryign to get a full stack going19:33
sdakeoh wasn't aware of that19:33
marunsdake: but as you say, qemu still works19:33
sdakethat way you know you have something working in th kolla frame19:33
sdakeand then once that works, you can try to trim nova19:34
sdakeyou should be asble to do so19:34
sdaketurn off all services19:34
marunsdake: I'm deploying with --tags rabbitmq,mariadb,neutron-only19:34
marunsdake: where neutron-only doesn't pull in compute19:34
sdakeya that wont work19:34
sdakeyou ned haproxy and keepalived19:34
marunsdake: really?19:34
sdakereally19:34
sdakeyou can make it work but it requires some special magic19:34
sdakeagain, your tyring to optimize too many steps at once19:35
marunsdake: trying to keep things simple :)19:35
sdakebig, smaller, smaller, smaller etc19:35
*** gfidente has quit IRC19:35
sdakeyour goal is a simple system but you are taking complex steps to achcieve it :)19:35
marunsdake: why are haproxy and keepalived required?19:35
sdaketakesimple steps to achieve it19:35
sdakeeverything goes through haproxy19:35
sdakekeepalived assigns the ip for haproxy19:35
marunsdake: ok19:36
sdakeyou can make kolla work witthout it19:36
sdakebut I don't have a howto on the subject19:36
sdakeagain, baby steps :)19:36
marunfair enough19:36
marunregarding deployment, is it possible to deploy to the same host (devstack-style) or do I need a separate host to target?19:37
marunI can see how network munging on the deployment host could get messsy19:37
openstackgerritSteven Dake proposed openstack/kolla: Make oslo.log work properly with syslog  https://review.openstack.org/22859519:38
sdakesame host19:38
sdakeor different host19:38
sdakeboth work19:38
sdakei do both in my environment19:38
sdakewhen i do same host, its AIO testing19:38
sdakewhen I do different hosts, its 3 bare metal HA setup19:38
marunok, good to know19:39
marunand thanks for the pointer re: intel 75019:39
sdakeya, rockin hardware19:40
marunI've been meaning to rebuild a dev host, but it's only real advantage over mac with pcie is 32gb ram.  having significantly faster io is a good excuse to go back to developing off my laptop19:40
sdake$38519:40
sdakei have 128gb ram19:40
sdakeand 3 32gb ram bare metal machines as well19:41
marunthat's some nice iron19:41
sdake128gb ram 32 cores (16 real, 16 hyperthreaded)19:41
sdakehp z82019:41
sdakeitspretty nice gear :)19:41
sdakethe other machcines are minidell 710s I htink19:41
marunI'm not really working on openstack anymore, though, and more modest hardware works nicely for container workloads :)19:42
sdakemarun what are you working on now then?19:44
sdakejpeeler rhallisey this is a new patcch but the openstack bot didn't show it19:44
sdakehttps://review.openstack.org/#/c/228595/19:44
marunsdake: project atomic/openshift19:45
sdakedontt report to pmyers anymore?19:45
marunnope19:45
sdakeinteresitng19:45
marunnot that I've managed to entirely escape, openstack integration is a hot topic19:46
*** dolpher has joined #kolla19:47
*** achanda has joined #kolla19:56
*** achanda has quit IRC20:01
sdakejpeeler I think your ironic patch breaks the gates https://review.openstack.org/#/c/219747/20:04
sdakeya I bet :)20:04
jpeelersdake: it's not actually broken, it's just reporting that source isn't implemented,20:06
sdakejpeeler i -1 that ironic patch - it is definately an ironic patch that break it20:06
sdakecool so override the tet patch so it doesn't return failure pls :)20:06
sdaketest patch20:06
sdakeI want people to trust the gate20:07
jpeeleri got -1 reviews for not putting them in there20:07
sdakeand we can't do that if it is broken20:07
sdakewe just  fixed it htis morning20:07
sdakewhat I mean is thetest_build.py or whatver its called20:07
sdakehas a blacklist or whitelist or whatever20:07
sdakeadd it there20:07
jpeelerah20:07
jpeelerok, though i just realized that script doesn't support differences between types20:08
sdakeyup20:08
sdakeI hvae a bug for that in the tracker20:08
sdakea whitelist for source and whitelist for binary20:08
sdakefeel free to fix that too - its in rc2 ;)20:08
sdakebut if not, just whitelist it entirely I think it shouldbe ok20:08
*** dolpher has quit IRC20:08
sdakeit works for murano and gnocchi i think20:08
jpeeleryeah, i'm sure it'll make it green20:09
sdakeas long as it passes20:09
sdakeI want the gate to be trusted :)20:09
sdakeit was trusted for a long time and someone broke the gate20:09
sdakeand its taken like 2 weeks to fix it20:09
sdakeI don't even know what patch broke it20:09
sdake11 blueprints and 87 bugs, not bad output for only 3 weeks ;)20:10
openstackgerritJeff Peeler proposed openstack/kolla: Add Ansible support for Ironic  https://review.openstack.org/21974720:10
sdakerhallisey is that patch coming along?20:10
sdakeif you don't finish, please git review it so same can take a look20:10
sdakerhallisey ^^20:11
sdakesame/sam20:11
sdakemarun who is your manager now?20:11
marunsdake: jack rieden20:13
sdakeheard he is a ogod guy20:13
sdakejpeeler I don't understand your request on the oslo.log patch20:13
sdakejpeeler yese patch should be short lived20:13
jpeelerjust wanted to make it clear to not just remove the patch, but to remove the patch packages too20:14
sdakethere is a big TODO that the stuff needs to be removed20:14
sdakewhich patch packages?20:14
sdakeyou mean 0001-Fix-*patch?20:14
sdakedo you mean the patch files ?20:14
jpeelerthe yum install patch, and apt-get install patch20:14
sdakeoh right20:14
sdakethanks20:14
openstackgerritSteven Dake proposed openstack/kolla: Make oslo.log work properly with syslog  https://review.openstack.org/22859520:15
sdaketry now :)20:15
sdakerhallisey ^^ please review that change20:16
*** shardy has quit IRC20:17
*** sdake_ has joined #kolla20:25
openstackgerritRyan Hallisey proposed openstack/kolla: Neutron Agents json support  https://review.openstack.org/22861120:25
rhalliseysdake, ^ try that20:25
rhalliseyyou should be able to build that 10x faster then me20:25
*** sdake has quit IRC20:28
openstackgerritRyan Hallisey proposed openstack/kolla: Neutron Agents json support  https://review.openstack.org/22861120:29
rhalliseywas missing a comma20:29
rhalliseysdake_, oh by the way, I know why cinder doesn't work20:30
sdake_and taht is what?20:30
rhalliseyremember that cinder fix I did that added tgt back in?  For some reason tgt is no longer in cinder20:30
rhalliseyno idea how20:31
*** jtriley has quit IRC20:34
sdake_rhallisey sincey ou fixed the bug please assign it to yourself thnaks :)20:34
rhalliseyI did20:35
rhalliseycan yo help me confirm20:35
rhalliseyyour machine is like 100 times faster than mine20:35
sdake_wasn't the neutron agents the entire reason we did the json copy thing?20:37
sdake_you mean you didn't test it?20:37
rhalliseyno not neutron-agents20:38
* sdake_ groans20:38
sdake_how do you know it works?20:38
*** sdake_ is now known as sdake20:38
rhalliseyit was for neutron-server and ovs-agent20:38
*** dolpher has joined #kolla20:39
rhalliseysdake, the container shouldn't exit20:39
*** achanda has joined #kolla20:39
rhalliseythe logs should show the configs being copied20:39
rhalliseysdake, I'm testing it. It just is going to take an hour to build this20:40
rhalliseyyour fancy computer takes like 30s20:40
*** sdake has quit IRC20:42
*** dolpher has quit IRC20:43
*** sdake has joined #kolla20:43
sdakerhallisey please ack https://review.openstack.org/#/c/228595/20:43
sdakerhallisey please review https://review.openstack.org/#/c/219747/20:43
sdakewe are almost through the home stretch of rc1 :)20:43
rhalliseynice! Looking at ironic..20:44
openstackgerritSteven Dake proposed openstack/kolla: Remove config-external for rsyslog  https://review.openstack.org/22844820:46
sdakemerge that oslo.log patch first20:47
sdakeits easy peasy to review20:47
sdakerhallisey ^^20:47
openstackgerritMerged openstack/kolla: Make oslo.log work properly with syslog  https://review.openstack.org/22859520:50
sdakerhallisey ta20:50
sdakerhallisey btw the reason I have smoking fast hardware is so I can test *my* changes faster, not be the single point of testing for everyone20:53
rhalliseysdake, ya I understand.  I figured you had an env ready to go to test it20:54
rhalliseyall you need to do it build and run20:54
rhalliseybut I don't care I'll do it20:54
sdakesame as you right? :)20:54
sdakei've got it running in a moment20:54
sdakebut if its busted i'm not debugging it :)20:54
sdakedoesn't work20:54
rhalliseyI know it's the same :P.  Mines just going to take longer so if you want to wait idc20:54
rhalliseywhat do the logs get?20:55
sdakehang tight20:55
rhalliseykk20:55
rhalliseythanks20:55
sdakebtw, all developers should be testing their changes20:55
sdakei am getting alot of private messages from other devs annoyed that people don't test their changes20:55
sdakeit pisses everyone off whne shit gets broken constantlly20:55
sdakepart of the problem is we have no automated functionalgate20:55
sdakewhich we need to solve20:55
sdakebut part of the commit process is testing the change20:56
sdakeI realize its onerous to test changes because the build and deploy takes 45 minutes20:56
sdakebut its onerous on other developers to bisect a busted commit20:56
rhalliseyif I don't test it I'll just -1 it20:56
rhalliseyagreeed20:56
rhalliseydidn't*20:57
sdakerhallisey http://ur1.ca/nvi3n -> http://paste.fedoraproject.org/272382/7388214420:58
sdakedon't submit changes without commit unless its end of day20:58
sdakeplease20:58
sdakewithout tewswting i mean20:58
rhalliseycan I -1 if I didn't test though?20:59
sdakehow is that helpful unless its end of day and your doing a handoff?21:00
rhalliseyjust to receive comments21:00
sdakei see, well we aren't after comments here, we are after a working solution :)21:00
sdakei want to pull the trigger on this release tomorrow morning21:00
rhalliseyin this case it was so you could help me test :)21:01
sdakeyes lazy lazy21:01
rhalliseybut I'm built now so I should be good21:01
sdakeit frustrates other developers21:01
sdakeincluding me :)21:01
rhalliseyI blame my hardware21:01
sdakei tell ya, that intel 750 drastically ahs improved my well being21:01
rhalliseysam said he will help me though21:01
sdakeya21:02
sdakebut if youcan get it done plase do21:02
sdakei'd like to be able to test for sevearl hours tonight21:02
rhalliseyshouldn't be a problem21:02
sdakevarious setups21:02
sdakewithout having to wake up at 4am to do the testing21:02
rhalliseyonce I have it cached it goes a little quicker21:02
sdakeafter sam has fixed theproblem21:02
sdakeyup21:02
sdakebut ya, its busted21:03
sdakei only printed out dhcp output21:03
sdakethe othe services also seem to fail for various reasons21:03
sdake(neutron-agents)21:03
rhalliseyI think I know how..21:05
rhalliseyor why rather21:05
rhalliseythey keep running over eachothers config21:05
rhalliseyneed to run set_configs.py in supervisord21:05
rhalliseyor in the first service run21:05
*** akwasnie has quit IRC21:09
rhalliseystuck on haproxy..21:10
rhalliseycrap I need to re build everything21:14
rhalliseythis is going to take a while..21:14
*** daneyon_ has joined #kolla21:16
*** daneyon has quit IRC21:20
* rhallisey builds for next 2 hours21:22
*** daneyon has joined #kolla21:23
sdakebuilding docker images is the best part of my job21:24
sdakebuildbuild build - 30 seconds of debug21:24
sdakebuidl build build for 15 inutes21:24
sdake30 seconds of debug21:24
sdakerinse repeat21:24
rhalliseyif you want to test I can push :)21:25
*** sdake_ has joined #kolla21:25
*** daneyon_ has quit IRC21:26
*** sdake__ has joined #kolla21:27
*** sdake has quit IRC21:28
*** itsuugo has quit IRC21:29
*** sdake_ has quit IRC21:30
*** dwalsh has quit IRC21:31
*** dwalsh_ has quit IRC21:32
sdake__rhallisey I know what the issue is21:37
sdake__set_configs is running 4 copies at once21:38
sdake__it is not designed to run parallel21:38
rhalliseyya that's what I said21:38
sdake__it is racing21:38
rhalliseyI think you were dc'd21:38
sdake__howdo we make it run just one copy?21:38
rhalliseyI have the fix, but I have to rebuild all my images21:38
sdake__paste teh diff21:38
sdake__i'll test it21:38
rhalliseyI changed the priority of dhcp21:39
sdake__how would that help21:39
rhalliseyand only ran set_configs.py in dhcp21:39
sdake__the others would race21:39
sdake__oh i see21:39
sdake__does that mean dhcp runs unitl completion?21:39
sdake__paste a diff, I'll build and test quickly21:39
sdake__or pusha  new review21:39
sdake__push a new review, that willbe better21:39
rhalliseyhttp://fpaste.org/272391/14434763/21:40
rhalliseyok21:40
rhalliseyI'm not sure if dhcp will run until completion21:40
sdake__well then that wont work21:40
sdake__(if it doesn't run until completion)21:41
*** sdake__ is now known as sdake21:41
rhalliseyI have another strategy, but we won't use json stuff.  The config-external stuff was never removed from neutron-agents21:41
rhalliseythis fix would be like a 4 line change21:41
sdakestrategy means something entirely different to you then it does to me :)21:41
rhalliseyjust don't source kolla-common21:42
sdakeyes but the other agents would start up21:42
sdakewithout their files being copied21:42
rhalliseythey will still be copied21:42
rhalliseycofig-external will be called21:42
sdakegroan21:42
rhalliseylook at what I removed in the patch21:42
sdakeyes I dont think that will work21:43
sdakelooks completely racy to me21:43
rhalliseyneutron-agents had it's own built in config-extneral21:43
rhalliseyya but this use to work21:43
rhalliseythe only change mad for neutron-agents is that kolla-common is overwriting CMD21:43
rhalliseya fix would be to remove sourcing it21:44
rhalliseyand they would work again like they did before21:44
rhalliseywith config-external21:44
sdakewell the question is, did they work correclty with json copying?21:45
sdakethey were in separate directories21:45
sdakethe problem now is they are all in the same directory21:45
rhalliseyI can just change it back21:45
sdakeputting them all in one directory is what creates the race condition21:46
sdakewhy not just have 4 copies and 4 json files21:46
rhalliseythat would require changing the python bits21:47
rhalliseyfor a 'fat' container that isn't going to be around forever21:47
rhalliseylet me push a patch to show you what I mean21:48
sdakewell i'm stumped i don't see a tidy solution21:49
sdakeother then making set_config.py handle fat containers21:49
openstackgerritRyan Hallisey proposed openstack/kolla: Alternate solution to get Neutron Agents work with JSON refactor  https://review.openstack.org/22863821:50
rhalliseysdake ^ that will prevent $CMD from being run over when running kolla-common21:51
rhalliseysince neutron-agents is fully self contained.  It never used the base config-external21:52
sdakebut it uses config-external.sh right?21:52
sdakewhich is not designed to read json blobs21:52
rhalliseyya a local config-external.sh21:52
rhalliseyeach of those services has a local copy21:52
rhalliseylook in docker/neutron/neutron-agents/neutron-dhcp-agent21:53
rhalliseyyou'll see what I mean21:53
rhalliseysince it never runs set_configs.py those errors you were seeing should go away and CMD should run as normal21:54
sdakeyes but there is no json parsing of the file list?21:55
rhalliseyand it will copy configs using the local cofig-external21:55
rhalliseyso there's no json here21:55
sdakeits got the hard coded file copy21:55
rhalliseycorrect21:55
sdakesomethign just seems wrong about that21:55
sdakethe idea is to get rid of config-external.sh21:56
sdakenot have a special case of it21:56
rhalliseywell it's how we use to do it.  Besides that, because of the fat container, command will never get full use anyway21:56
rhalliseyit will get hard coded in the start script21:56
rhalliseyso I can implement fat container support into the python bits, but the problem is doing that without having to change everything21:58
rhalliseyI intentionally left the config alone for neutron-agents because it was a fat container and a self contained unit with config-external21:58
sdakei suspect we will be stuck with fat containers forever because of neutron and docker upstream business prioirities21:58
rhalliseyoh really..21:59
sdakethe feature is a 3 character change to get us to having non-fat container support21:59
sdakebut it breaks docker machine21:59
sdakebadly breaks21:59
sdakefrom waht I can tell there is no way to make them work together the way we need21:59
sdakebasically, docker thinks its more important for machine to work then to have the feature we need to make thin containers work for neutron22:00
sdakewhich makes sense from docker's perspetive22:00
sdakei am deploying your test patch now22:00
sdaketo see if it works22:00
rhalliseykk22:00
sdakei dont like how we have two interfaces into the container22:00
sdakei'd prefer to change the python bits22:01
sdakeand have one interface22:01
rhalliseyok22:01
*** mat425 has quit IRC22:01
sdakeeven if that means the interface changes22:01
sdakethe time to chang ethe interface is now22:01
sdakebefore we pull the trigger on 1.022:01
sdakeboy i wish yo uwould have brought this up earlier22:02
sdakerather then telling me on release day :(22:02
rhalliseyya sorry.  I didn't know how this would've affected us22:02
sdakeyes, that is called risk, which means communication is necessary :)22:03
rhalliseyI thought we were going to thin neutron at some point22:03
sdakewe will if we ever can22:03
sdakebut that doesn't mean we carry two interfaces into the container runtime22:03
rhalliseyya, but it sounds like almost never22:03
sdakethat is the fundamental problem with that thinking22:03
sdakethat we carry two interfaces with our 1.0 api release22:03
sdakewe need one interface22:03
sdakethe one interface is the json communication mechanism22:04
sdakei think a more complex solution which would work is to use a file lock22:05
rhalliseyugh this gets tricky..22:07
rhalliseywithout know what the json files are named22:07
sdakehttps://docs.python.org/2/library/fcntl.html#fcntl.lockf22:07
rhalliseymaybe if config.json isn't present I can just look for *.json22:07
sdakethat is racy22:07
rhalliseyya it is22:08
rhalliseycrap22:08
sdakefcntl.ockf would work perfectly22:08
sdakeit is designed for this exact problem22:08
rhalliseyok then... I could have an env variable with the name22:08
sdakethat is adding shit to the api22:08
rhalliseyk looking22:08
rhalliseycould it still be racy if we only lock in python though22:10
sdakeas long as there is no deletion22:12
sdakethere is currently deletion22:12
sdakei dont understand why you can't put them in separate directories as was with the original work22:12
rhalliseywill this lock prevent 2 python scripts from running at the same time or lock the python script in until ocmpletion22:12
sdakeright22:12
sdakebut if you delete a file after its been loaded by another service22:13
sdakethat would probablyreak something22:13
sdakei wonder if there is a coutning lock in fcntl22:14
* sdake reads man page22:14
rhalliseyso I could put them in separate dirs, but sam wanted them all in one dir22:14
rhalliseythough I don't know if that would help22:14
rhalliseyunless you talking about individual json files22:14
sdakeseparte dirs would solve the deletion problem22:14
sdakewe can't delete22:14
sdakei dont understand why we delete presently22:14
sdakedo you knwo why?22:15
rhalliseysam added that.  That would do a replace of an existing config22:15
rhalliseyactually why would you need that..22:15
rhalliseywouldn't you just start a new container if changing your json ?>>22:15
rhalliseyI think you would have to22:16
rhalliseyand why can't we just copy over instead of delete then copy22:16
* rhallisey looks22:16
*** dims has quit IRC22:17
rhalliseyI suppose we can remove the deletion.  There shouldn't ever be anything to delete22:18
sdakewhy is the deletion there?22:21
rhalliseyI'm not sure.  I can't really think of a use case22:22
sdakeya copy over should be fine22:22
sdakeif we have copy over we can use fcntl to lock22:22
sdakeits a little bit more complicated then not using fcntl,but posix locks have been around since 90s22:23
sdakeor 80s maybe22:23
sdakei really wish I understood why sam has that delete in there22:23
rhalliseyhow about instead of copy over just copy once22:23
rhalliseyif it exists just leave it alone22:23
sdakethat doens't work for the copy_always mode22:24
rhalliseyok true22:24
sdakecopy always could be problematic as well22:25
sdakewe could partially copy a file while a different service has loaded it22:25
sdakeafter the lock was released22:25
sdakewhich would result is really hard to solve partial data bugs22:25
sdakewhat we need is to implement a counting semaphore with fcntl22:26
sdakeI think that wouldsolve the problem nicely22:26
rhalliseyyikes.  Are we sure supervisord won't hep us here at all22:26
sdakeno idea22:26
sdakeit may have somethig helpful22:27
sdakeRT FM22:27
rhalliseyI feel like it would22:27
sdakecheck out the manual22:27
sdakei would assume it has dependencies22:28
sdakefi we can just run the copy operation one time22:28
sdakethat would solve the problem22:28
sdakecopy runs (set_config) as one service22:29
sdakeother services depend on it22:29
sdakeand dont do set_config22:29
sdakeor source kolla_common.sh or whatever the current magic is22:29
rhalliseyhttps://github.com/Supervisor/supervisor/issues/12222:29
rhalliseyso we could run one of the services by hand first22:31
rhalliseyor22:31
rhalliseylet's just run the python script22:32
rhalliseybefore supervisod in the dockerfile22:32
sdakeyou can't run two scripts like that back to back22:32
sdakethere is only one entrypoint22:32
sdakeyou could make a scirpt which calls one then the other however22:32
sdakewhich is pretty terribad :)22:32
rhalliseyRUN python set_configs.py :P why not22:33
sdakeRUN runs at build time not runtime22:33
rhalliseydon't use it as an entrypoint22:33
rhalliseyerr22:33
rhalliseythat's true22:33
rhalliseyya haha22:33
sdakebtw, i tested your patch it does not work22:33
sdakethe second one22:33
rhalliseyrun supervisordd as a script22:33
rhalliseyoh really :(22:34
sdakesee if there are dependencies in supervisord22:34
sdakeall the config files end up in neutron-agent22:34
sdakeand I think they need to be in other dirs as well but not sure22:34
sdakeif supervisord has dependencies we can justuse that22:34
rhalliseystartsecs=522:34
sdakenot really liking that much22:34
sdakei really hate sleep hacks22:35
rhalliseyI don't either22:35
sdakenoob programming 10122:35
sdakedependencies22:35
sdakesupervisord must have dependencies22:35
rhallisey6 days ago22:35
rhallisey+1 ?? This is from May, 2012, I think that supervisor will never have dependencies.22:35
rhalliseySomeone could suggest alternatives with dependencies management?22:35
rhallisey:(22:35
rhalliseyi'll keep looking22:35
rhalliseylooks like it's sleep or a script22:35
sdakehttp://stackoverflow.com/questions/17443692/python-supervisord-program-dependency22:36
sdakethe last example looks good22:37
rhalliseyhehe nice22:38
rhalliseyok.  I'll add that into the first patch22:38
sdakeok i think the shell script on entry with starting set_config followed by supervisrod is the best way to go22:38
sdakeso basically a meta start.sh22:39
sdakethat will always work correctly22:39
sdakeor we can use the last example and wait for set_configs to exit22:41
rhalliseyok so this will wait for dhcp to start or just run set_configs22:41
sdaketwo operations, run set config, then run supervisord22:41
sdakei'm not sure if the process reaping will work correctly with that or not22:41
sdakebut our processing reaping in kolla is fubared anyway22:42
sdakefkn docker22:42
sdakelet me look at the supervisord code a bit sec22:42
sdakewhat does startsec do?22:44
sdakedely for # of seconds before starting?22:44
rhalliseyya22:47
*** banix has quit IRC22:50
openstackgerritSteven Dake proposed openstack/kolla: Neutron Agents json support  https://review.openstack.org/22861122:52
sdakehave a look at that22:52
sdakewe dont need a delay in there either22:52
sdakejsut slows things down with no gain22:52
rhalliseyyou beat me to it22:53
rhalliseywas about to push22:53
rhalliseymine has a loop, but I don't see why it's needed22:53
rhalliseyI just make sure $CMD exists22:53
sdakegetting ready to deploy22:54
rhalliseykk22:54
rhalliseythat should work22:54
rhalliseyassuming the json stuff I added is correct there22:54
rhalliseyI think it is22:54
sdakerhallisey remove dyour workflwo vote23:00
rhalliseyit's only -1 on the alternate solution23:01
rhalliseyrefresh23:01
rhalliseysdake, did that work for you?23:03
sdakestill deploying23:03
sdakemy 750 doesnt'speed up deploy times23:03
sdakesince i deploy to a different machine23:03
sdakeand the different machine is runningwith overlayfs23:03
sdakewhich is already much faster then btrfs23:03
sdakeERROR:__main__:The source to copy does not exist: /opt/kolla/config_files/fwaas_driver.ini23:06
rhalliseyls /etc/kolla/neutron-agents23:07
rhalliseyis fwaas_driver.ini in there?23:07
sdakeno, a directory called "neutron-agents" is in that directory ;)23:08
rhalliseyoh23:08
rhalliseywhat's in the next neutron-agents :)23:08
rhalliseydo you find all the config files23:08
sdakechill young padowan23:08
sdakenothign is in opt/kolla from what i can tell23:10
sdake /opt/kolla that is23:10
sdakethe j2 copies fwaas_driverini directly to /etc/neutron23:12
sdakerather then /opt/kolla/whatever23:12
sdakewhat directory should it be coming from23:12
rhallisey/opt/kolla/config_files23:12
rhalliseyI'm confused, I mounted /etc/kolla/neutron-agents to /opt/kolla/config_files23:13
rhalliseycan you rebuild --no-cache23:13
sdakelook at the neutronagents dockerfie.j2 file23:13
sdakewhy would i rebuild --no-cache23:13
sdakethe j2 file is wrong23:14
sdakereublidng --no-cache owuld take 5-10 minutes - no reason for that23:14
rhalliseyoh I see it23:15
rhalliseywell23:15
openstackgerritSteven Dake proposed openstack/kolla: Neutron Agents json support  https://review.openstack.org/22861123:15
sdakedeploying now23:15
rhalliseynice23:16
rhalliseyI was wondering where that was coming from.  I thought you had bad docker caching or something23:17
rhalliseythat issue drove me crazy for horizon23:17
sdakewhich issue23:17
sdakei have neverseen teh cache misbhehae really23:17
sdakebut when I do something that i athink has cache consequences is I generally run with --no-cache ;)23:17
rhalliseyI've had a few23:18
sdakedid yo u finish taht review for jeff23:19
rhalliseyit would drive me crazy.  I'd build for 30min then run and get the same errors23:19
rhalliseyI haven't23:19
rhalliseyI will tonight23:19
sdakeok23:19
rhalliseyI need to eat dinner and stuff23:19
sdakeright23:19
sdake4:2023:20
sdakeyou know whta that means23:20
bmaceyou will be having the munchies soon?23:20
sdakehaha I wish23:21
sdakeobama take my kids away23:21
rhalliseylol23:21
sdake9 minutes 42 seconds to deploy single node23:25
sdakedid not work23:25
sdaketrying small change23:27
sdakeand also running with --no-cache23:27
sdakei dont get why perl modules re installed int he bae image23:29
sdakeperl is not part of the system :(23:30
sdakefawk openstack-base is breaking on oslo.log23:31
*** loth1 has joined #kolla23:32
*** loth has quit IRC23:33
rhalliseysdake, hows it look23:50
rhalliseygetting past neutron?23:50
sdakenah got jammed up on this patch not merging correclty23:50
sdakesorting it out now23:50
openstackgerritSteven Dake proposed openstack/kolla: Neutron Agents json support  https://review.openstack.org/22861123:56
openstackgerritSteven Dake proposed openstack/kolla: Fix oslo_log  https://review.openstack.org/22866923:56
*** achanda has quit IRC23:58
sdakebuilding should hae reulsts in t-20 minutes23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!