Monday, 2019-10-07

*** goldyfruit_ has joined #openstack-kolla00:02
*** BjoernT has joined #openstack-kolla02:20
*** BjoernT_ has joined #openstack-kolla02:29
*** BjoernT has quit IRC02:31
*** mikal has quit IRC02:54
*** BjoernT_ has quit IRC03:26
*** BjoernT has joined #openstack-kolla03:27
*** BjoernT has quit IRC03:28
*** BjoernT_ has joined #openstack-kolla03:29
*** BjoernT_ has quit IRC03:34
*** BjoernT has joined #openstack-kolla03:34
*** BjoernT has quit IRC04:10
*** BjoernT has joined #openstack-kolla04:11
*** igordc has joined #openstack-kolla04:19
*** BjoernT has quit IRC04:26
*** igordc has quit IRC05:22
*** Luzi has joined #openstack-kolla05:40
*** igordc has joined #openstack-kolla05:51
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Revert "Debian: publish images"  https://review.opendev.org/68696306:08
hrwmorning06:08
*** gdwornicki has quit IRC06:12
*** jbadiapa has joined #openstack-kolla06:20
yoctozeptomorning06:20
*** pcaruana has joined #openstack-kolla06:22
*** cah_link has joined #openstack-kolla06:23
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Revert "Debian: publish images"  https://review.opendev.org/68696306:24
*** radeks has joined #openstack-kolla06:25
yoctozeptohrw: so no publishing06:26
hrwyoctozepto: k-a job builds own ones06:27
hrwyoctozepto: and after dicsussion with infra I decided that it would be probably better to not publish06:27
yoctozeptohrw: ok06:29
yoctozeptohrw: did longer timeouts help anyhow?06:29
hrwhttp://zuul.openstack.org/stream/dd2b85054ef24bb897195e5ab945eacd?logfile=console.log will show06:31
yoctozeptohrw: http://zuul.openstack.org/builds?job_name=kolla-publish-debian-source06:32
yoctozeptoit helped06:32
yoctozeptoso need to scrap stale images06:32
yoctozeptonot just revert06:33
hrwjust like I thought - once we pushed whole set next builds are short. 1.5h06:34
hrwno. it not reused any06:35
hrw1:10 for pushing06:35
yoctozeptono, it does only reuse the layers created during that particular run06:37
yoctozeptoso not earlier06:37
yoctozeptothere is no efficiency in that06:37
hrwI just want to sleep a bit more06:41
CosmicSound:)06:42
yoctozeptohrw: what are you doing during the night?06:43
hrwyoctozepto: I dislike waking up at 6:30 and leaving home in 5 minutes06:44
yoctozeptoI see06:45
yoctozeptoI thought you were working from home though06:45
hrwI do06:46
hrwthere are days when I have to wake up early and do some 'firefighting'06:49
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157306:52
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637606:52
yoctozeptoI see06:52
mnasiadkamorning06:58
mnasiadkaI was woken today at 5:50...06:58
*** rpittau|afk is now known as rpittau07:30
*** radeks_ has joined #openstack-kolla07:45
*** radeks has quit IRC07:46
*** ileixe has joined #openstack-kolla07:47
ileixeHi guys07:47
ileixeI wonder how others deal with custom code using kolla + kolla-ansible07:48
yoctozeptohi ileixe07:48
ileixeIs there any usable way to get it?07:48
yoctozeptohow custom07:48
ileixehi yoctozepto07:48
ileixeFor exmpale07:48
ileixewe has some rebased commits on upstream branch07:48
ileixeAnd also some custom PIP packages07:49
ileixeI wonder how integrate it using kolla07:49
yoctozeptoileixe: for the first use kolla-build.conf to define location of sources07:50
yoctozeptofor the second you probably want to override some template07:50
*** igordc has quit IRC07:50
yoctozeptosee https://docs.openstack.org/kolla/latest/admin/image-building.html07:50
ileixeThanks for quick reply :)07:52
ileixeIs there any way to extend sources at runtime?07:52
ileixeI think we'd happy with original images if we can07:52
yoctozeptoileixe: extra volumes maybe?07:56
yoctozeptosee https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#mount-additional-docker-volumes-in-containers07:57
ileixeYep, basically i thought we can use dev-mode for that. I wonder kolla-ansible / kolla has any builtin way.07:58
ileixeextend_start.sh looks promising, but I'm not sure.07:58
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637607:59
hrwileixe: also: send those 'some rebased commits on upstream branch' for review maybe?07:59
ileixehrw :) Yes that's the true answer08:00
hrwkeeping out-of-tree code out-of-tree is bad for you08:00
hrwdev-mode is implemented for subset of images iirc08:00
ileixeYep I 100% understand, but you know it's really hard to implement everything in upstream08:01
yoctozeptomgoddard, mnasiadka: looks like ansible handles ipv6 addresses properly starting with 2.808:01
yoctozeptobump? :D08:01
hrwileixe: been there, done that...08:02
mnasiadkayoctozepto: 2.9 will be out in a week - so I guess we could bump it in U, unless you think you want to bump now08:02
mnasiadkayoctozepto: other option is to bump in the ipv6 CI job and state Ipv6 requires 2.8 in the release note08:02
yoctozeptomnasiadka: well, we can always tell ppl in docs that ipv6 requires ansible 2.808:03
yoctozeptoso agreed wow08:03
yoctozepto<pieseł image>08:03
mnasiadkayeah, ceph-ansible will require 2.8 as well, but I'm not gonna push it towards T ;)08:03
mnasiadkaso if we have two features requiring 2.8 - it might make sense to convince mgoddard to bump to 2.8 in U :)08:03
yoctozeptoyeah, ipv6 is another story, but I think it's ok to require it via the docs for ipv6 only08:03
yoctozeptoyeah, U is definetely for 2.8 or later08:04
hrwU should be 2.8+ anyway08:04
yoctozeptothat's Ussuri you know08:04
mnasiadkaand 2.9 is around the corner, so to be safe we should go for 2.8+ as the default08:04
hrwwill remember in middle of cycle08:04
hrwyoctozepto: your ipv6 is Train or Ussuri material?08:07
hrwas we are at feature freeze08:07
yoctozeptohrw: train08:08
yoctozeptoit's huge unfortunately08:08
hrwthat's why asked08:09
*** k_mouza has joined #openstack-kolla08:12
*** pcaruana has quit IRC08:13
*** k_mouza has quit IRC08:17
*** k_mouza has joined #openstack-kolla08:18
*** pcaruana has joined #openstack-kolla08:18
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix swift-proxy-server memcached configuration  https://review.opendev.org/68697508:24
*** skramaja has joined #openstack-kolla08:29
*** k_mouza has quit IRC08:33
*** k_mouza_ has joined #openstack-kolla08:33
mgoddardmorning koalas08:40
hrwelo mgoddard08:40
mgoddardyoctozepto: I could potentially be convinced for 2.8 in U08:41
mgoddardthe thing is though, you have to try quite hard to get an old version, so it should work in most cases08:41
mgoddardand +1 for documenting these things08:41
mgoddardand a precheck if it doesn't fail before that point :)08:41
*** k_mouza_ has quit IRC09:02
*** k_mouza has joined #openstack-kolla09:02
*** csmart has joined #openstack-kolla09:30
stingrayzamorning folks o/09:30
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157309:32
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Use any_errors_fatal in pre.yml and run.yml  https://review.opendev.org/68692109:33
*** gdwornicki has joined #openstack-kolla09:42
mgoddardmorning stingrayza09:46
*** k_mouza has quit IRC09:52
yoctozeptomgoddard: our local ironman, could you look at: https://4eff4bb69c321960be39-770d619687de1bce0976465c40e4e9ca.ssl.cf2.rackcdn.com/685952/2/check/kolla-ansible-ubuntu-source-ironic/93a6452/primary/logs/kolla/all-ERROR.txt.gz - this is ipv4 job (without ipv6 patches)09:58
*** gdwornicki has quit IRC09:58
yoctozeptothe dnsmasq error seems random09:58
yoctozeptothere are jobs without the error09:58
yoctozeptobut the nova-compute-ironic is constant09:58
yoctozeptolet me know how problematic these are09:59
mgoddardyoctozepto: ah yeah09:59
mgoddardI have an ironicclient patch for it09:59
*** ondrejduchon has joined #openstack-kolla09:59
mgoddardhttps://review.opendev.org/67594009:59
yoctozepto(thinking of raising bug report like for cinder and masakari)09:59
mgoddardand a workaround in k-a: https://review.opendev.org/67589309:59
mgoddardjust needs a review :)09:59
*** tonythomas has joined #openstack-kolla10:00
yoctozeptoyou are so reliable, bugs are fixed before reporting :-)10:01
yoctozeptowhat about nova one?10:01
mgoddardnot sure about the nova one10:02
yoctozeptothe nova errors appear each time10:03
yoctozeptoso that's a plus of them ;p10:03
mgoddardglad those error files are proving useful :)10:04
yoctozeptomgoddard: great for extra sanity checks :-)10:04
yoctozeptook, reporting nova case10:05
mgoddardtonythomas: those errors go away after an instance is provisioned on the node10:06
mgoddardsorry tonythomas, wrong nick :)10:06
mgoddardyoctozepto: ^10:07
yoctozeptomgoddard: yeah, so are we doing something too early or is nova-compute-ironic doing it too early? :-)10:08
yoctozeptoshould be warnings then10:08
mgoddardI'm not sure, but ERROR is probably too strong10:09
yoctozeptowell, reported10:11
yoctozeptowill see when the time comes10:11
yoctozeptolet it be noted10:11
yoctozeptothrowing https://bugs.launchpad.net/kolla-ansible/+bug/1847050 at goldyfruit_10:15
openstackLaunchpad bug 1847050 in kolla-ansible "CI: masakari-instancemonitor reports an error" [Medium,Triaged]10:15
*** k_mouza has joined #openstack-kolla10:17
hrwyoctozepto, mgoddard: I do not think that debian-source images we pushed are any danger. we already had 3y old ones there and so far no one complained10:18
mgoddardhrw: if someone in infra is willing to clean them up that would be good, but otherwise not much we can do10:19
andreinHello everyone10:20
andreinIf I were to try to add support for novajoin in kolla-ansible, where would I start?10:20
mgoddardhello andrein10:21
andreinhi mgoddard!10:21
mgoddardwe're currently in feature freeze, so it will have to wait until the Ussuri cycle. You can still start work on it though10:21
andreinI understand :)10:22
mgoddardfirst thing would be to add a blueprint on launchpad, outlining the problem and a brief decription of your proposed solution10:22
mgoddardif you want to discuss it, bring it up at an IRC meeting or just ad-hoc in here10:22
mgoddardand when you're ready, propose a patch with 'Implements: blueprint bp/<your blueprint name>' in the commit10:23
mgoddard(not the first line)10:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Docs: improve Nova documentation  https://review.opendev.org/68699510:24
andreinI was hoping I could start a discussion here before creating the blueprint. Is there anyone around that has any experience setting up novajoin?10:26
*** gfidente has joined #openstack-kolla10:27
yoctozeptoandrein: not me, maybe mgoddard, mnasiadka or someone from hrw's side10:34
yoctozeptoglad you came to do something about novajoin, we have these images but they were on the path to deprecation due to no usage in downstreams10:35
andreinyoctozepto: I'll do my best to rescue them from the image shelter :)10:36
yoctozeptoah no, tripleo seems to have used it at least at some point10:36
hrwwhat nojavoin is btw?10:36
yoctozeptohrw: turns out some redhat stuff for IPA10:37
yoctozeptoat least tripleo uses it that way though10:37
hrw" Nova metadata dynamic vendordata server for enrolling IPA instances " - bla bla bla something bla bla bla10:37
yoctozeptoa dynamic vendordata plugin for the OpenStack nova metadata service to manage host instantiation in an IPA server10:37
yoctozeptotwo sources, same truth10:38
yoctozeptoipa blah10:38
yoctozeptoprobably joining ipa domain then ;p10:38
yoctozeptoandrein: are we right?10:38
andreinfrom what I've read, it supposed to automatically join VMs to an IPA domain10:38
andrein:)10:38
yoctozeptoah, so we all read the same stuff :-)10:39
andreinyep10:39
yoctozeptogood10:39
hrwI care about IPA mostly in pubs10:39
yoctozeptoxD10:40
andreinXD10:40
andreinso, the first step in the novajoin docs is https://opendev.org/x/novajoin/src/branch/master/scripts/novajoin-ipa-setup10:40
yoctozeptojust don't tell me k-a is going to deploy IPA please :D10:41
andreinit looks like that script gets a kerberos ticket and sets up the CAcerts10:41
yoctozeptoah, good, I was worried by the filename10:41
andreinyoctozepto: I'm pretty sure novajoin depends on an already existing IPA instace :)10:42
yoctozeptowho knows, k-a can deploy opendaylight, though I would also assume ppl wanting that would deploy it themselves10:42
yoctozeptowell, that "can" is obviously theoretical10:43
yoctozeptoit has not been tested10:43
andreinhow would I integrate running that script with kolla-ansible? add a new kolla-ansible command (like bifrost-deploy?)10:43
yoctozeptoI think you have to answer that question yourself10:44
yoctozeptowrite down some notes10:44
yoctozeptotry things10:44
yoctozeptocreate the blueprint10:44
yoctozepto:D10:44
andreinbreak things in the process10:44
yoctozeptoindeed10:44
andreinok, I know the drill10:44
andreingotta set up a proper development server, because I have a feeling I'll be breaking stuff a lot10:44
*** skramaja has quit IRC10:48
yoctozeptoandrein: and improper development server would be production server in read only mode? :D10:49
andreinhow did you know? :))10:53
andreinwhat's read only mode anyway?10:53
yoctozeptoandrein: ;-)10:57
yoctozeptohrw: is BaptisteGer deploying ipv6 on centos, ubuntu or debian?10:57
hrwyoctozepto: he is on channel - ask him ;d10:58
hrwBaptisteGer: ^^10:58
yoctozeptohrw: but he did not say hello today and you did :-)10:58
yoctozeptoso asking you first10:59
yoctozeptoyou should know ;p10:59
BaptisteGeryoctozepto: on ubuntu source11:01
BaptisteGerHello everybody :)11:01
BaptisteGeryoctozepto: I've been trying to catch up with your patches, but they get updated quickly :)11:02
hrwBaptisteGer: that's why it is better to update once per day and keep an eye on changes ;d11:03
BaptisteGerI have issues with datbase_address being urls instead of ips in 'login_host: {{ database_address }}" blocks in bootstrap (and other calls in mariadb tasks)11:03
*** pcaruana has quit IRC11:11
*** kplant has joined #openstack-kolla11:16
yoctozeptoBaptisteGer: should have been fixed, I mean, it works for me (TM) and in CI (R) (C)11:18
yoctozeptoasking because centos is b0rken11:18
yoctozeptomaybe will try to retest it when centos 8 is up11:19
yoctozeptobut 7 is a no-no11:19
yoctozeptoissues ranging from haproxy through mariadb and rabbitmq11:19
yoctozeptoBaptisteGer: thanks for confirming you are using a supported platform :-)11:20
yoctozeptoalso, if you hit issues with the latest patch - take note of its rev no and post comments on review with any problems11:21
yoctozeptoubuntu containers on centos 7 host work fine too so it's centos 7 images that have lame old software11:21
BaptisteGeryoctozepto: comment is done with 'Reply' ?11:22
hrwBaptisteGer: yes.11:23
BaptisteGeryoctozepto: Started out with debian and was told ubuntu source was better so.. :)11:23
hrwBaptisteGer: you can also leave comment at any line of review11:23
BaptisteGerhrw: cheers11:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Docs: improve Nova documentation  https://review.opendev.org/68699511:23
openstackgerritMerged openstack/kolla stable/queens: Install rsync in swift-base containers on Debian/Ubuntu  https://review.opendev.org/68610911:24
openstackgerritMerged openstack/kolla stable/stein: Install rsync in swift-base containers on Debian/Ubuntu  https://review.opendev.org/68610611:24
openstackgerritMerged openstack/kolla stable/rocky: Install rsync in swift-base containers on Debian/Ubuntu  https://review.opendev.org/68610811:24
*** pcaruana has joined #openstack-kolla11:42
yoctozeptowe still suffer from random "wsrep has not prepared the node blah"12:08
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637612:17
mnasiadkayoctozepto: maybe we need to add a check, if it's prepared? :D :D :D12:19
yoctozeptomnasiadka: but we have those, I think12:19
yoctozeptolooks like it could be due to flaky network connections12:19
yoctozeptotwo jobs at the same time had mariadb issues in rackspace12:20
mnasiadkamaybe we need to fine tune some timeout values?12:21
*** goldyfruit_ has quit IRC12:21
mnasiadkabut the usual timeouts are like 15 seconds, so it should be enough...12:22
mnasiadkayoctozepto: I think it would make sense to do query WSREP_READY - wait for it to be ready, if it's not after some reasonable time - just fail everything12:29
BaptisteGeryoctozepto: All database_address instances are fine, your patch has everything ! :) Posted small comments on the gerrit12:39
hrwBaptisteGer: " at the moment in group_vars, kolla_internal_vip_address and kolla_internal_fqdn are the same " - shouldn't first be IP and second be FQDN?12:42
hrwignore. 'by default the same'12:44
BaptisteGerhrw: I was told to go for DRY so :) Will ignore !12:46
openstackgerritMark Goddard proposed openstack/kayobe master: Fix validation of locally built IPA when ironic is disabled  https://review.opendev.org/68701712:52
*** dave-mccowan has joined #openstack-kolla13:04
*** BjoernT has joined #openstack-kolla13:18
primeministerphrw: morning13:31
*** goldyfruit_ has joined #openstack-kolla13:35
*** factor has quit IRC13:48
*** vmixor has joined #openstack-kolla14:11
*** spiette has quit IRC14:16
*** spiette has joined #openstack-kolla14:16
*** csmart has quit IRC14:20
*** hjensas has quit IRC14:34
*** hjensas has joined #openstack-kolla14:41
*** k_mouza_ has joined #openstack-kolla14:42
*** k_mouza has quit IRC14:45
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Added support for custom logging.conf  https://review.opendev.org/67981215:11
*** k_mouza_ has quit IRC15:18
*** k_mouza has joined #openstack-kolla15:19
*** Luzi has quit IRC15:25
* hrw out15:31
*** factor has joined #openstack-kolla15:35
*** rpittau is now known as rpittau|adk16:02
*** rpittau|adk is now known as rpittau|afk16:02
*** goldyfruit___ has joined #openstack-kolla16:09
*** goldyfruit_ has quit IRC16:11
*** ondrejduchon has quit IRC16:18
*** csmart has joined #openstack-kolla16:30
*** billp has joined #openstack-kolla16:40
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157316:43
*** k_mouza_ has joined #openstack-kolla16:46
*** k_mouza has quit IRC16:49
*** k_mouza_ has quit IRC16:51
*** mgoddard has quit IRC16:57
*** mgoddard has joined #openstack-kolla17:02
*** pcaruana has quit IRC17:21
*** k_mouza has joined #openstack-kolla17:26
*** k_mouza has quit IRC17:30
openstackgerritDincer Celik proposed openstack/kolla stable/stein: build: enable Octavia for Ubuntu/binary  https://review.opendev.org/68712518:00
*** pcaruana has joined #openstack-kolla18:00
*** goldyfruit___ has quit IRC18:03
*** goldyfruit has joined #openstack-kolla18:06
*** primeministerp has quit IRC18:10
*** bline has quit IRC18:15
*** bline has joined #openstack-kolla18:18
*** jonaspaulo has joined #openstack-kolla18:26
openstackgerritMerged openstack/kolla-ansible master: Fix swift-proxy-server memcached configuration  https://review.opendev.org/68697518:37
*** cah_link has quit IRC18:47
*** primeministerp has joined #openstack-kolla18:47
*** spartakos has joined #openstack-kolla19:16
*** pcaruana has quit IRC19:32
*** pbing19 has joined #openstack-kolla19:44
pbing19when  am trying to deploy mariadb using kolla ansible facing issue19:45
pbing19 Could not open mysql.plugin table. Some plugins may be not loaded19:45
pbing19am i doing something wrong here..19:46
*** kplant has quit IRC19:48
*** spartakos has quit IRC19:50
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Remove /etc/hosts entries pointing hostname to localhost  https://review.opendev.org/68523319:51
*** spartakos has joined #openstack-kolla19:59
*** spartakos has quit IRC20:02
*** igordc has joined #openstack-kolla20:13
primeministerphey now anyone know why I'd be getting this:  fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", "delta": "0:00:00.484588", "end": "2019-10-07 16:22:07.563633", "msg": "non-zero return code", "rc": 137, "start": "2019-10-07 16:22:07.079045", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}20:22
primeministerpduring an kolla-ansible -i all-in-one deploy20:22
primeministerpon aarch6420:22
*** tonyb has joined #openstack-kolla20:53
*** goldyfruit_ has joined #openstack-kolla21:05
*** goldyfruit has quit IRC21:08
openstackgerritViktor Michalek proposed openstack/kolla master: Custom CA certificates installation mechanism  https://review.opendev.org/68612121:09
*** goldyfruit_ has quit IRC21:22
openstackgerritViktor Michalek proposed openstack/kolla master: Custom CA certificates installation mechanism  https://review.opendev.org/68612121:28
openstackgerritViktor Michalek proposed openstack/kolla master: Custom CA certificates installation mechanism  https://review.opendev.org/68612121:28
*** BjoernT has quit IRC21:35
*** vmixor has quit IRC22:01
*** pbing19 has quit IRC22:04
*** jonaspaulo has quit IRC22:08
*** goldyfruit_ has joined #openstack-kolla22:10
*** radeks_ has quit IRC22:31
*** BjoernT has joined #openstack-kolla22:37
*** BjoernT has quit IRC22:42
*** goldyfruit_ has quit IRC23:20
*** goldyfruit_ has joined #openstack-kolla23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!