| *** eck` is now known as eck`gone | 00:04 | |
| *** dxiri has quit IRC | 00:22 | |
| *** bugzy_ has joined #tripleo | 00:22 | |
| *** bugzy has quit IRC | 00:26 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 00:29 |
|---|---|---|
| EmilienM | sshnaidm|rover: what's url for sova again? | 00:29 |
| sshnaidm|rover | EmilienM, http://cistatus.tripleo.org/ | 00:30 |
| EmilienM | sshnaidm|rover: you never sleep? | 00:30 |
| EmilienM | anyway, thanks | 00:30 |
| sshnaidm|rover | EmilienM, and you? :D | 00:30 |
| EmilienM | lol | 00:31 |
| sshnaidm|rover | EmilienM, well, this overcloud ara makes me crazy, no idea why it doesn't work.. | 00:31 |
| sshnaidm|rover | EmilienM, do we have undercloud containerized and config download in all master jobs? | 00:32 |
| EmilienM | sshnaidm|rover: I've noticed that it doesn't collect all the playbooks right? | 00:32 |
| EmilienM | sshnaidm|rover: no, containerized undercloud is only on fs001 and 035 now, due to timeout issues | 00:32 |
| openstackgerrit | Merged openstack/paunch master: Implement group_add https://review.openstack.org/578620 | 00:33 |
| sshnaidm|rover | EmilienM, it collects now only those (is it from container update?) http://logs.openstack.org/62/578462/7/check/tripleo-ci-centos-7-containers-multinode/67f8589/logs/ara_oooq_oc/ | 00:34 |
| EmilienM | sshnaidm|rover: yeah that's weird | 00:34 |
| EmilienM | sshnaidm|rover: and the tripleo-modify-image | 00:35 |
| EmilienM | which means the callback isn't configured well enough | 00:35 |
| sshnaidm|rover | ansible.cfg seems correct http://logs.openstack.org/62/578462/7/check/tripleo-ci-centos-7-containers-multinode/67f8589/logs/undercloud/var/lib/mistral/070b7a05-f674-4997-bf68-3ccd8a8ce1d6/ansible.cfg.gz | 00:36 |
| EmilienM | sshnaidm|rover: btw I added proxy stuff to my patch | 00:36 |
| EmilienM | yeah it looks good indeed | 00:37 |
| EmilienM | have you tried to deploy manually | 00:37 |
| EmilienM | and run ansible manually | 00:37 |
| EmilienM | and see how it goes? | 00:37 |
| EmilienM | brb | 00:38 |
| sshnaidm|rover | will try tomorrow, previous time it worked well locally | 00:39 |
| sshnaidm|rover | EmilienM, so for docker proxy it's enough to set docker_registry_mirror | 00:39 |
| *** dbecker has joined #tripleo | 00:40 | |
| sshnaidm|rover | let's see how much time now it takes, w/o proxy it was 30-40 mins | 00:40 |
| sshnaidm|rover | it it's the same then proxy doesn't work as I suspected.. | 00:41 |
| *** psahoo has joined #tripleo | 00:43 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Run only CI playbooks in run.yml zuul playbook https://review.openstack.org/565740 | 00:51 |
| *** dmacpher has joined #tripleo | 00:53 | |
| sshnaidm|rover | EmilienM, you can find it useful: http://38.145.34.131:3000/d/2kHMNHvik/exploration?orgId=1&var-influxdb_filter=passed%7C%3D%7CFalse | 00:54 |
| sshnaidm|rover | EmilienM, it's all failed jobs for example, you can play with filters above.. | 00:54 |
| *** dbecker has quit IRC | 00:54 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Run only CI playbooks in run.yml zuul playbook https://review.openstack.org/565740 | 00:56 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 00:56 | |
| EmilienM | sshnaidm|afk: ack thanks, and good night | 00:57 |
| EmilienM | for the record, things start to merge again it seems, I've restored a dozen of patches to see if that actually works better | 00:59 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 01:07 |
| *** ehengtan has joined #tripleo | 01:09 | |
| *** mrsoul_ has joined #tripleo | 01:12 | |
| *** mrsoul` has joined #tripleo | 01:13 | |
| *** edmondsw has joined #tripleo | 01:13 | |
| *** mschuppert has quit IRC | 01:15 | |
| *** mrsoul has quit IRC | 01:16 | |
| *** edmondsw has quit IRC | 01:18 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Deploy container-multinode (fs010) with a containerized undercloud"" https://review.openstack.org/560264 | 01:30 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Switch scenario00{1,2,3,4,7} to containerized undercloud"" https://review.openstack.org/575330 | 01:30 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Switch the undercloud to be containerized by default https://review.openstack.org/576218 | 01:31 |
| *** fzdarsky__ has joined #tripleo | 01:31 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs003: deploy with instack-undercloud https://review.openstack.org/576225 | 01:31 |
| ehengtan | Can anyone help with my openstack network configurations ? :( | 01:32 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch 3nodes and scenario000-upgrades to containerized undercloud https://review.openstack.org/579755 | 01:33 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Switch the undercloud to be containerized by default https://review.openstack.org/576218 | 01:33 |
| *** fzdarsky_ has quit IRC | 01:34 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Make tripleo-ci-centos-7-undercloud-oooq non-voting" https://review.openstack.org/576258 | 01:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: standalone run a post config to generate a clouds.yaml https://review.openstack.org/575597 | 01:41 |
| *** dxiri has joined #tripleo | 01:43 | |
| ehengtan | Can my pxe_drac host placed outside of my provisioning network? | 01:45 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 01:47 |
| openstackgerrit | Pan Xingyu proposed openstack/os-net-config master: Optimize the link address in docs https://review.openstack.org/580245 | 02:05 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 02:14 |
| *** bkopilov has quit IRC | 02:14 | |
| *** ehengtan has quit IRC | 02:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 02:27 |
| *** psachin` has joined #tripleo | 02:29 | |
| *** ykarel has joined #tripleo | 02:35 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 02:36 |
| *** dxiri has quit IRC | 02:40 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 02:44 |
| *** skramaja has joined #tripleo | 02:55 | |
| *** khyr0n has joined #tripleo | 02:59 | |
| *** edmondsw has joined #tripleo | 03:02 | |
| *** agopi has quit IRC | 03:02 | |
| *** edmondsw has quit IRC | 03:06 | |
| *** dhill_ has quit IRC | 03:08 | |
| *** ykarel has quit IRC | 03:29 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates stable/pike: Ensure node is rebooted before enabling DPDK https://review.openstack.org/580262 | 03:31 |
| *** saneax has quit IRC | 03:34 | |
| *** ramishra has joined #tripleo | 03:39 | |
| *** udesale has joined #tripleo | 03:44 | |
| *** psahoo has quit IRC | 03:46 | |
| *** dciabrin has quit IRC | 03:47 | |
| *** dciabrin has joined #tripleo | 03:47 | |
| *** ykarel has joined #tripleo | 03:47 | |
| *** bkopilov has joined #tripleo | 03:48 | |
| *** shreshtha has joined #tripleo | 03:48 | |
| *** shreshtha has quit IRC | 03:49 | |
| *** shreshtha has joined #tripleo | 03:49 | |
| *** dxiri has joined #tripleo | 03:52 | |
| *** gvrangan has joined #tripleo | 03:54 | |
| *** numans has quit IRC | 03:56 | |
| *** dxiri has quit IRC | 03:59 | |
| *** ykarel_ has joined #tripleo | 04:04 | |
| *** ykarel has quit IRC | 04:04 | |
| *** ykarel_ has quit IRC | 04:12 | |
| *** gvrangan has quit IRC | 04:20 | |
| *** khyr0n has quit IRC | 04:32 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Disable pid file usage in the swift_rsync container https://review.openstack.org/577403 | 04:36 |
| openstackgerrit | Merged openstack/tripleo-ui stable/queens: Prepare 8.3.2 (queens) https://review.openstack.org/580207 | 04:36 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Prepare 8.3.4 (queens) https://review.openstack.org/580206 | 04:36 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: Prepare 7.4.14 (pike) https://review.openstack.org/580203 | 04:36 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/ocata: Prepare 6.5.14 (ocata) https://review.openstack.org/580202 | 04:36 |
| *** mdnadeem has joined #tripleo | 04:38 | |
| *** ykarel_ has joined #tripleo | 04:38 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 04:39 |
| *** zzzeek has quit IRC | 04:40 | |
| *** zzzeek has joined #tripleo | 04:43 | |
| EmilienM | sshnaidm|afk: new timeout, we need a new bug report probably: http://logs.openstack.org/39/577339/1/gate/tripleo-ci-centos-7-nonha-multinode-oooq/e97fdad/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-05_04_18_26 | 04:44 |
| EmilienM | sshnaidm|afk: a timeout on zaqar/mistral probably | 04:44 |
| EmilienM | chkumar|ruck: ^ | 04:45 |
| EmilienM | I'm out | 04:45 |
| *** holser_ has joined #tripleo | 04:45 | |
| *** mburned is now known as mburned_out | 04:48 | |
| Tengu | have a good night mburned_out ;) | 04:51 |
| *** edmondsw has joined #tripleo | 04:51 | |
| *** edmondsw has quit IRC | 04:55 | |
| tonyb | Anyone aroudn that can help me understand puppet deps in terms of changing when a task happens in a module? | 04:57 |
| *** cshastri has joined #tripleo | 04:59 | |
| *** janki has joined #tripleo | 05:01 | |
| *** mdnadeem has quit IRC | 05:02 | |
| *** karthiks has quit IRC | 05:03 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 05:03 |
| *** karthiks has joined #tripleo | 05:03 | |
| EmilienM | @all please read topic when you start your day, thanks | 05:05 |
| Tengu | EmilienM: go to bed ;). | 05:07 |
| Tengu | tonyb: care to give some more info regarding your question? | 05:07 |
| *** ratailor has joined #tripleo | 05:09 | |
| tonyb | Tengu: Sure let me find some links for you | 05:10 |
| *** zzzeek has quit IRC | 05:10 | |
| *** zzzeek has joined #tripleo | 05:11 | |
| tonyb | Tengu: so the issue I'm seeing is that http://git.openstack.org/cgit/openstack/puppet-ironic/tree/manifests/inspector.pp#n305 is failing because the pxelinuc.cfg directory doesn;t exist. It's created http://git.openstack.org/cgit/openstack/puppet-ironic/tree/manifests/pxe.pp#n81 so I'm assuming that the that I need to re-arrange the anchors so that the directory is created befoer the file is | 05:14 |
| tonyb | Tengu: but I'm not really sure how to do that. | 05:14 |
| *** ccamacho has quit IRC | 05:14 | |
| Tengu | 2s, checking the code. | 05:14 |
| tonyb | Tengu: thanks | 05:14 |
| Tengu | tonyb: you might want to add a require => File["${tftp_root_real}/pxelinux.cfg"] for the file { "${tftp_root_real}/pxelinux.cfg/default": resource. | 05:15 |
| Tengu | tonyb: or, even, require => Class['ironic::pxe'] | 05:16 |
| Tengu | with the latter it will wait for the ironic::pxe class to be complete before creating the "default" file in the pxelinux.cfg directory. | 05:16 |
| tonyb | Tengu: Thanks I'll give that a try. | 05:17 |
| Tengu | np. you might ping me for puppet issues, I have some knowledge :) | 05:17 |
| tonyb | Tengu: Good to know thanks! | 05:17 |
| *** waleedm has joined #tripleo | 05:17 | |
| *** numans has joined #tripleo | 05:22 | |
| *** bogdando has joined #tripleo | 05:23 | |
| tonyb | Tengu: If I want to add require => Class['ironic::pxe'] as you described is it in addition to the existing Anchor? | 05:23 |
| *** mdnadeem has joined #tripleo | 05:23 | |
| *** jbadiapa has quit IRC | 05:24 | |
| *** psachin` has quit IRC | 05:27 | |
| *** gvrangan has joined #tripleo | 05:27 | |
| ramishra | Hi guys, any idea why I'm getting this error running a reproducer script? ansible error is cryptic to say the least. http://paste.openstack.org/show/725069/ | 05:29 |
| *** psachin` has joined #tripleo | 05:29 | |
| *** yolanda has joined #tripleo | 05:30 | |
| Tengu | tonyb: yep | 05:31 |
| *** quiquell|off is now known as quiquell | 05:31 | |
| tonyb | Tengu: awesome. You can tell I'm new to puppet ;P | 05:31 |
| Tengu | tonyb: we all started one day ;) | 05:32 |
| Tengu | tonyb: so the final will be something like require => [Anchor[....], Class[...]] | 05:33 |
| *** dxiri has joined #tripleo | 05:33 | |
| tonyb | Tengu: It's true :) | 05:33 |
| Tengu | tonyb: hmm you might need to update the unit tests as well. 2s, checking. | 05:34 |
| Tengu | tonyb: https://github.com/openstack/puppet-ironic/blob/master/spec/classes/ironic_inspector_spec.rb#L167 | 05:36 |
| tonyb | Tengu: Thanks. I'm still not able to run the tests locally so I do the slow and wasteful push to gerrit and debug there. | 05:39 |
| Tengu | tonyb: what's your issue with running the tests? | 05:40 |
| chkumar|ruck | EmilienM: I have seen this timedout yesterday in one of the job | 05:40 |
| chkumar|ruck | EmilienM: Do we need bug for each timedout? | 05:41 |
| tonyb | Tengu: I'd have to look again. There was some issue with me being on Fedora28, but even when I created a Centos7 VM they failed somehow. | 05:42 |
| tonyb | Tengu: I'll try again and see where they go wrong. | 05:42 |
| Tengu | tonyb: ok - I'll run a test ony ma fedora 28 | 05:42 |
| ykarel_ | ramishra, looks like a bug, is it reproducable? i mean u see multiple times? | 05:42 |
| *** ykarel_ is now known as ykarel | 05:43 | |
| tonyb | Tengu: It's entirely possible that I just don't know what I'm doing and it's my issue not anything complex | 05:43 |
| Tengu | hmm. that was weird. "on my fedora" is more like what I wanted to write. | 05:43 |
| Tengu | tonyb: lemme see ;). | 05:43 |
| ramishra | ykarel: all the time with master multinode scripts:) | 05:43 |
| tonyb | Tengu: IIRC I was tryign to follow: https://docs.openstack.org/puppet-openstack-guide/latest/contributor/testing.html | 05:43 |
| Tengu | tonyb: well, there's an issue with ruby version not meeting one of the lib used by rspec version. | 05:44 |
| Tengu | that's the first issue. Will fire up a centos7 container. | 05:44 |
| ykarel | ramishra, hmm i haven't seen this with multinodes i generally use, can you link me to the script | 05:44 |
| ramishra | ykarel: http://logs.openstack.org/20/577620/5/check/tripleo-ci-centos-7-containers-multinode/5a33abc/logs/reproducer-quickstart.sh | 05:45 |
| ykarel | ramishra, trying, i generally try similar but not faced issue, trying now to see | 05:47 |
| ykarel | chkumar|ruck, i think if timeout is for different reason, different bugs should be there, as diagnosis and fix for those might be different | 05:48 |
| ramishra | ykarel: yeah, the error is too cryptic to tell me where to look for, though I like ansible;) | 05:48 |
| ykarel | ramishra, it didn't failed for me, it passed ahead | 05:50 |
| ykarel | TASK [multinodes : Wait for provisioned hosts to become reachable] ******************************************************************************* | 05:50 |
| ykarel | changed: [localhost] => (item=subnode-0) | 05:50 |
| ykarel | changed: [localhost] => (item=subnode-1) | 05:50 |
| *** holser_ has quit IRC | 05:50 | |
| chkumar|ruck | ykarel: it is the same issue | 05:50 |
| ykarel | chkumar|ruck, there is already bug for it? | 05:51 |
| ykarel | ramishra, what u get for:- openstack stack output show multinode_stack subnode_ip_pairs -c output_value -f value | 05:51 |
| ramishra | Looks like it's creating only one node for me, Is my command to run the reproducer wrong or something? | 05:53 |
| *** ksambor has joined #tripleo | 05:53 | |
| *** moshele has joined #tripleo | 05:53 | |
| chkumar|ruck | ykarel: nope creating it | 05:53 |
| chkumar|ruck | ykarel: https://bugs.launchpad.net/tripleo/+bug/1780183 | 05:53 |
| openstack | Launchpad bug 1780183 in tripleo "Overcloud failed to Deploy in tripleo-ci-centos-7-nonha-multinode-oooq" [Undecided,New] | 05:53 |
| ramishra | ykarel: http://paste.openstack.org/show/725070/ | 05:53 |
| chkumar|ruck | gates are in pretty bad shape | 05:54 |
| ykarel | ramishra, how you are running, passing any custom param? i just ran that script and output:- http://paste.openstack.org/show/725071/ | 05:55 |
| ykarel | chkumar|ruck, ack | 05:55 |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config master: WIP: Testing requirements https://review.openstack.org/580290 | 05:56 |
| ramishra | ykarel: nope, vanilla run for me too | 05:58 |
| *** alee has joined #tripleo | 05:58 | |
| ramishra | let me try again | 05:58 |
| ykarel | ramishra, ack | 05:59 |
| *** gvrangan has quit IRC | 06:00 | |
| *** fzdarsky__ has quit IRC | 06:01 | |
| ramishra | ykarel: I get two nodes http://paste.openstack.org/show/725072/, Is my ansible version have something to do with it? | 06:02 |
| ramishra | I used a different reproducer script earlier | 06:02 |
| ykarel | ramishra, i think somehow subnode_count was 1 for you, but don't think ansible can do that | 06:04 |
| *** nyechiel has joined #tripleo | 06:05 | |
| ramishra | ykarel: nope, it's 2, check the last paste, As I mentioned I was trying all kinds of things and tried single node earlier | 06:06 |
| *** dxiri has quit IRC | 06:06 | |
| ykarel | ramishra, which reproducer script u were using, which failed | 06:06 |
| ramishra | ykarel: the one shared with you and all others I've tried till now | 06:07 |
| ykarel | ramishra, how it worked this time? sorry i missed | 06:08 |
| ramishra | ykarel: nothing worked for me yet, still the same error single node, multinode script whatever I use:) | 06:09 |
| *** dparkes has joined #tripleo | 06:10 | |
| ykarel | ramishra, ok, with http://paste.openstack.org/show/725072/ also u are seeing the issue, right? i confused with:- Looks like it's creating only one node for me | 06:11 |
| ykarel | if yes, than ansible has something to do with it | 06:12 |
| *** gvrangan has joined #tripleo | 06:13 | |
| bogdando | o/ PTAL https://review.openstack.org/#/q/topic:shared_ansible_cfg+(status:open+OR+status:merged) | 06:14 |
| ramishra | ykarel: yep, that statement of mine was confusing, that was a different script I used and the hence the output has one node, trying with different version of ansible, I'm on f28 so may be something to do with it. | 06:14 |
| chkumar|ruck | ykarel: it is a known issue https://bugs.launchpad.net/tripleo/+bug/1780186 | 06:15 |
| openstack | Launchpad bug 1780186 in tripleo "build-test-packages : Fetch local rdoinfo copy failing on pike FS-16, 17, 18 on Pike 24 hour periodic jobs" [Undecided,New] | 06:15 |
| bogdando | and queens backports for containers defaults https://review.openstack.org/#/c/567863/ https://review.openstack.org/#/c/567810/ | 06:15 |
| chkumar|ruck | coming on periodic jobs | 06:15 |
| chkumar|ruck | 3 pike jobs failing | 06:15 |
| chkumar|ruck | ? | 06:15 |
| *** mrsoul_ is now known as mschuppert | 06:15 | |
| ykarel | ramishra, but ansible should be installed in venv | 06:15 |
| *** bogdando changes topic to "Welcome to Rocky. CI status: PLAID - High amount of timeouts in the gate, do not recheck/+W now | https://docs.openstack.org/tripleo-docs/latest" | 06:15 | |
| ramishra | ykarel: yep, I unsinatlled all ansible and installed 2.6 and it worked now, thanks for the help | 06:15 |
| ykarel | ramishra, Ok Good | 06:16 |
| ykarel | chkumar|ruck, seen similar issue in past, | 06:18 |
| chkumar|ruck | ykarel: do we want to wait for next run? or retrigger the periodic job? | 06:18 |
| ykarel | chkumar|ruck, can wait as pike was promoted yesterday | 06:19 |
| *** ccamacho has joined #tripleo | 06:19 | |
| *** saneax has joined #tripleo | 06:26 | |
| *** dxiri has joined #tripleo | 06:27 | |
| *** gvrangan has quit IRC | 06:28 | |
| *** gvrangan has joined #tripleo | 06:29 | |
| *** sanjayu_ has joined #tripleo | 06:30 | |
| *** hamdyk has joined #tripleo | 06:31 | |
| *** dxiri has quit IRC | 06:31 | |
| *** lvdombrkr has joined #tripleo | 06:32 | |
| *** saneax has quit IRC | 06:33 | |
| *** quiquell is now known as quiquell|bbl | 06:35 | |
| *** avivgt has joined #tripleo | 06:40 | |
| *** dbecker has joined #tripleo | 06:43 | |
| *** jbadiapa has joined #tripleo | 06:43 | |
| *** gvrangan has quit IRC | 06:43 | |
| *** moshele has quit IRC | 06:44 | |
| *** gvrangan has joined #tripleo | 06:44 | |
| *** mrunge has quit IRC | 06:45 | |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Add ability to filter container images to modify https://review.openstack.org/579918 | 06:46 |
| *** mrunge has joined #tripleo | 06:46 | |
| *** psahoo has joined #tripleo | 06:47 | |
| *** matbu has joined #tripleo | 06:49 | |
| *** jfrancoa has joined #tripleo | 06:51 | |
| *** alee has quit IRC | 06:52 | |
| *** alee has joined #tripleo | 06:52 | |
| *** dbecker has quit IRC | 07:05 | |
| *** jpena|off is now known as jpena | 07:05 | |
| *** ffiore has joined #tripleo | 07:06 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Update scenario003 to deploy separate messaging backends https://review.openstack.org/566483 | 07:07 |
| *** ffiore has quit IRC | 07:09 | |
| *** gkadam has joined #tripleo | 07:11 | |
| *** holser_ has joined #tripleo | 07:12 | |
| openstackgerrit | Merged openstack/tripleo-common master: Workflow for configuring BIOS settings to ironic nodes https://review.openstack.org/576449 | 07:13 |
| *** ffiore has joined #tripleo | 07:14 | |
| *** sshnaidm|afk has quit IRC | 07:14 | |
| *** nyechiel has quit IRC | 07:14 | |
| ykarel | bogdando, patches are being merged ^^, some more comming looks like timeout issue seen when more jobs are running | 07:15 |
| chkumar|ruck | ykarel: Does pike periodic jobs does not took the pike promoted hash? | 07:16 |
| chkumar|ruck | ykarel: https://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/77a32c9/tempest.html.gz | 07:16 |
| *** quiquell|bbl is now known as quiquell | 07:16 | |
| chkumar|ruck | is running as we skipped it na? | 07:16 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: WIP Remove restart_flag leftovers for bundles https://review.openstack.org/580305 | 07:16 |
| ykarel | chkumar|ruck, we have not skipped job, we just removed it from promotion criteria, the issue is expected as https://review.openstack.org/#/c/579426/ is not merged yet | 07:18 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add support for NodeDataLookup for Ceph deployment https://review.openstack.org/577956 | 07:19 |
| *** nyechiel has joined #tripleo | 07:20 | |
| *** sshnaidm|afk has joined #tripleo | 07:21 | |
| *** peereb has joined #tripleo | 07:22 | |
| chkumar|ruck | ykarel: bogdando https://review.openstack.org/#/c/568602/ on this one scenario001 failed just now | 07:25 |
| chkumar|ruck | logs will be available soon | 07:25 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates master: NUMA aware vswitches https://review.openstack.org/578742 | 07:25 |
| ykarel | chkumar|ruck, ack. So need to look more what's the difference, but somehow failures are gone down | 07:26 |
| chkumar|ruck | ykarel: sshnaidm|afk in early morning, scenario 0,1,3 were passing, 2 was failing | 07:27 |
| chkumar|ruck | now it is just opposite | 07:27 |
| *** amoralej|off is now known as amoralej | 07:29 | |
| bogdando | so the last hope to break the CI jobs walltime is https://review.openstack.org/#/c/580037/. | 07:29 |
| bogdando | We have no ways to improve the actual jobs run time yet. | 07:29 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Allow VIP resource to have customized ops https://review.openstack.org/554672 | 07:29 |
| *** tosky has joined #tripleo | 07:32 | |
| chkumar|ruck | bogdando: ykarel is there to find out a way at particular job why it is timing out? | 07:33 |
| *** ratailor_ has joined #tripleo | 07:33 | |
| *** jpich has joined #tripleo | 07:34 | |
| bogdando | chkumar|ruck: http://logs.openstack.org/02/568602/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/8cd49eb/ara-report/ shows profiling details for each task | 07:34 |
| bogdando | that may help somewhat | 07:34 |
| *** ratailor has quit IRC | 07:35 | |
| bogdando | except that it hides all run time under a single /opt/stack/new/tripleo-ci/toci_gate_test.sh :D | 07:35 |
| bogdando | also, the job console log contains Runtime: for each task | 07:36 |
| bogdando | not for each ._. | 07:38 |
| bogdando | so prolly then timestmps associated with task names give the needed profiling info | 07:38 |
| chkumar|ruck | bogdando: when the job timed out at a particular task it does not show the time stamp | 07:39 |
| bogdando | right, but it shows the longest ones, and they are sitting in the middle | 07:39 |
| *** gvrangan has quit IRC | 07:40 | |
| *** suuuper has joined #tripleo | 07:43 | |
| *** jaganathan has joined #tripleo | 07:44 | |
| bogdando | chkumar|ruck: curl http://logs.openstack.org/02/568602/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/8cd49eb/job-output.txt.gz | grep -E '\*\*$|\-\-\s\S+s$' | 07:45 |
| bogdando | shows the most important infos I guess | 07:45 |
| bogdando | not for the latest one http://logs.openstack.org/02/568602/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/8cd49eb/job-output.txt.gz#_2018-07-05_02_33_49_019818 though | 07:47 |
| bogdando | jistr: hi, any ideas how to break down execution steps of that ^^ ? | 07:47 |
| *** florianf has joined #tripleo | 07:47 | |
| chkumar|ruck | bogdando: look nice, yup, | 07:48 |
| bogdando | chkumar|ruck: so for that particular task we have https://pastebin.com/8RvEhcR2 , plus that last one aborted after a 1h of exec | 07:52 |
| bogdando | not that something went awry, but just hit by walltime | 07:52 |
| bogdando | note the time for Prepare for the containerized deployment is only a few minutes. Thanks to stevebaker and EmilienM work with containers workflows optimizations! :) | 07:53 |
| bogdando | I think that time shows how much it took to only feetch the missing containers, and omitting the those we had for undercloud install already | 07:54 |
| *** rcernin has quit IRC | 07:54 | |
| * bogdando dreams what if we could reuse the intalled undercloud for all multi-node jobs in the running pipeline | 07:55 | |
| chkumar|ruck | bogdando: it will save time | 07:57 |
| * bogdando wonders why those 2 undercloud-setup took sooooo loooong | 07:57 | |
| bogdando | so much of the packages to install for containerized deploy, really? | 07:57 |
| bogdando | and for VXlan configs, why so long? | 07:58 |
| chkumar|ruck | bogdando: https://review.rdoproject.org/etherpad/p/chkumar-ruck-rover-sprint16-notes | 07:58 |
| chkumar|ruck | check line no: 18 to 24 | 07:58 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-heat-templates master: Add scenario010 for testing Octavia https://review.openstack.org/518331 | 07:58 |
| bogdando | chkumar|ruck: yeah, hope that helps | 07:59 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Make the errors around fencing and IHA clearer https://review.openstack.org/580315 | 07:59 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates master: NUMA aware vswitches https://review.openstack.org/578742 | 08:03 |
| *** moshele has joined #tripleo | 08:08 | |
| *** ykarel is now known as ykarel|lunch | 08:08 | |
| *** mrunge has quit IRC | 08:12 | |
| bogdando | chkumar|ruck: fixed the grep regex, so now it shows Prepare for the containerized deployment - 2965.22s | 08:14 |
| bogdando | which is the way too long, and points out to the unexpected cointainers images preparation/updates et al steps | 08:15 |
| *** sshnaidm|afk is now known as sshnaidm|rover | 08:16 | |
| *** Goneri has joined #tripleo | 08:17 | |
| *** akrivoka has joined #tripleo | 08:17 | |
| *** mrunge has joined #tripleo | 08:24 | |
| *** edmondsw has joined #tripleo | 08:28 | |
| *** avivgt has quit IRC | 08:28 | |
| *** edmondsw has quit IRC | 08:32 | |
| *** yamahata has quit IRC | 08:32 | |
| *** fzdarsky__ has joined #tripleo | 08:36 | |
| *** derekh has joined #tripleo | 08:36 | |
| *** alee has quit IRC | 08:37 | |
| *** alee has joined #tripleo | 08:37 | |
| *** gvrangan has joined #tripleo | 08:38 | |
| *** mburned_out is now known as mburned | 08:45 | |
| bogdando | EmilienM, stevebaker: https://bugs.launchpad.net/tripleo/+bug/1780216 | 09:00 |
| openstack | Launchpad bug 1780216 in tripleo "No shared with containerized UC images skipped when preparing for overcloud deployment " [High,New] | 09:00 |
| *** ratailor_ has quit IRC | 09:02 | |
| bogdando | not sure where now to get the undercloud preparation logs | 09:03 |
| bogdando | with the recent swithces, there is not more undercloud prep containers log | 09:03 |
| bogdando | never-mind its no more containerized UC there | 09:06 |
| *** ykarel|lunch is now known as ykarel | 09:08 | |
| * bogdando sigh | 09:16 | |
| *** avivgt has joined #tripleo | 09:18 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Add the ability to scaleup the openshift stack https://review.openstack.org/578285 | 09:20 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: HA support for OpenShift https://review.openstack.org/579609 | 09:20 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Replace raw_get with dict access syntax https://review.openstack.org/579843 | 09:20 |
| *** cmyster has quit IRC | 09:30 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Adds reproducer check+exit+warning dependencies - virtualenv+others https://review.openstack.org/578081 | 09:32 |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** cmyster has joined #tripleo | 09:39 | |
| *** cmyster has joined #tripleo | 09:39 | |
| *** Goneri has quit IRC | 09:40 | |
| *** Goneri has joined #tripleo | 09:42 | |
| *** cmyster has quit IRC | 09:45 | |
| *** cmyster has joined #tripleo | 09:46 | |
| *** cmyster has joined #tripleo | 09:46 | |
| numans | bandini, Hi, regarding your patch - https://review.openstack.org/#/c/580127/ | 09:49 |
| *** spsurya_ has quit IRC | 09:49 | |
| numans | i think if you want you can remove "--user=root" even from manifests/profile/pacemaker/ovn-dbs-bundle.pp | 09:49 |
| numans | bandini, right now, ovn-northd docker image is not configured with any user. so even if we remove "--user=root" from ovn-dbs-bundle.pp it would still run as root. | 09:50 |
| numans | So, I will update the ovn-northd's Dockerfile.j2 in kolla and add "USER openvswitch". | 09:51 |
| *** alee has quit IRC | 09:53 | |
| *** alee has joined #tripleo | 09:53 | |
| *** cmyster has quit IRC | 09:54 | |
| *** Goneri has quit IRC | 09:54 | |
| *** Goneri has joined #tripleo | 09:58 | |
| bandini | numans: ack. but if you change the kolla user then we will need to think about what happens on minor update (i.e. any file will be owned by root:root and won't be accessible if you move to a new user) | 10:06 |
| bandini | let's make sure we have those steps somewhere before we remove it? | 10:07 |
| *** jbadiapa has quit IRC | 10:15 | |
| *** bogdando has quit IRC | 10:16 | |
| sshnaidm|rover | mwhahaha, EmilienM slagle jpich please approve chkumar|ruck (Chandan Kumar) as a member in launchpad tripleo team: https://launchpad.net/~tripleo | 10:16 |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config master: WIP: Testing requirements https://review.openstack.org/580290 | 10:16 |
| jpich | sshnaidm|rover: Huh, didn't realise I could do this. Should be done now? | 10:17 |
| sshnaidm|rover | chkumar|ruck, can you check if you can set priority etc to bugs? ^^ | 10:18 |
| sshnaidm|rover | jpich, you're one of admins :) | 10:18 |
| jpich | Not sure how that happened! :) | 10:18 |
| sshnaidm|rover | jpich, one of chosen! | 10:18 |
| jpich | \o/ | 10:18 |
| *** gvrangan has quit IRC | 10:22 | |
| *** zshi has quit IRC | 10:23 | |
| *** zshi has joined #tripleo | 10:23 | |
| sshnaidm|rover | chkumar|ruck, if bug happens in the same branch, better to mention it in the headline: https://bugs.launchpad.net/tripleo/+bug/1780183 | 10:24 |
| openstack | Launchpad bug 1780183 in tripleo "Ocata: Overcloud failed because of "WebSocket timed out" in CI" [High,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:24 |
| *** bkopilov has quit IRC | 10:28 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Replace deprecated include module with include/import_tasks module https://review.openstack.org/579844 | 10:32 |
| *** mdnadeem has quit IRC | 10:34 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Run cinder's db sync only on bootstrap node https://review.openstack.org/579732 | 10:39 |
| numans | bandini, sure. | 10:44 |
| *** gvrangan has joined #tripleo | 10:44 | |
| *** bogdando has joined #tripleo | 10:45 | |
| *** mdnadeem has joined #tripleo | 10:46 | |
| chkumar|ruck | sshnaidm|rover: sure | 10:46 |
| chkumar|ruck | sshnaidm|rover: I am added to the tripleo bug group, thanks :-) | 10:47 |
| chkumar|ruck | sshnaidm|rover: upstream tripleo gate queue looks green now | 10:49 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: OVN: Remove environment files which deploy OVN dbs in non HA https://review.openstack.org/579796 | 10:54 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Handle blank lines followed by comment in nic conversion script https://review.openstack.org/578956 | 10:54 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fluentd: disable log path docker transformation https://review.openstack.org/580193 | 10:54 |
| openstackgerrit | Merged openstack/tripleo-common master: Workflows for manipulating deployment status https://review.openstack.org/578518 | 10:54 |
| openstackgerrit | Merged openstack/tripleo-common master: Force lowercase hostnames https://review.openstack.org/576534 | 10:54 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Handle roles w/o deprecated_nic_config_name during template cleanup https://review.openstack.org/576946 | 10:54 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use correct release file for periodic jobs https://review.openstack.org/578793 | 10:54 |
| openstackgerrit | Merged openstack/tripleo-common master: ensure unique ironic node ID with UCS driver https://review.openstack.org/528726 | 10:54 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Do not grant caps if pool name is empty https://review.openstack.org/579600 | 10:54 |
| mwhahaha | :o | 10:54 |
| *** leanderthal has joined #tripleo | 10:57 | |
| *** cmyster has joined #tripleo | 11:00 | |
| *** cmyster has joined #tripleo | 11:00 | |
| *** bfournie has joined #tripleo | 11:01 | |
| *** radeks has joined #tripleo | 11:01 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Double the docker puppet process counts https://review.openstack.org/580342 | 11:03 |
| *** amoralej is now known as amoralej|lunch | 11:04 | |
| *** jpena is now known as jpena|lunch | 11:05 | |
| *** cmyster has quit IRC | 11:05 | |
| *** radeks has quit IRC | 11:05 | |
| *** radeks has joined #tripleo | 11:06 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Reproducer don't fail if workspace path doesn't exist yet https://review.openstack.org/579587 | 11:09 |
| *** gvrangan has quit IRC | 11:12 | |
| *** pradk has joined #tripleo | 11:12 | |
| openstackgerrit | yatin proposed openstack-infra/tripleo-ci master: WIP - Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 11:15 |
| ykarel | EmilienM, just trying, if ^^ can help, | 11:15 |
| *** colonwq has joined #tripleo | 11:17 | |
| bandini | numans: awesome, thanks ;) | 11:18 |
| *** suuuper has quit IRC | 11:19 | |
| *** psahoo has quit IRC | 11:22 | |
| bogdando | \o/ | 11:23 |
| *** gvrangan has joined #tripleo | 11:28 | |
| *** edmondsw has joined #tripleo | 11:29 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Allow custom host kernel params for libguestfs https://review.openstack.org/576772 | 11:31 |
| *** sshnaidm|rover is now known as sshnaidm|rov|lnc | 11:33 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Delete not-used services-docker files https://review.openstack.org/567810 | 11:35 |
| *** dprince has joined #tripleo | 11:39 | |
| *** radeks has quit IRC | 11:39 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Delete not-used services-docker files https://review.openstack.org/567810 | 11:39 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Add the ability to scaleup the openshift stack https://review.openstack.org/578285 | 11:42 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: HA support for OpenShift https://review.openstack.org/579609 | 11:42 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Replace raw_get with dict access syntax https://review.openstack.org/579843 | 11:43 |
| *** lifeless has quit IRC | 11:45 | |
| openstackgerrit | Martin Schuppert proposed openstack/puppet-tripleo master: New profile to configure libvirt-guests on compute host https://review.openstack.org/580351 | 11:47 |
| *** radeks has joined #tripleo | 11:48 | |
| *** abishop has joined #tripleo | 11:48 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Add NovaResumeGuestsStateOnHostBoot and NovaResumeGuestsShutdownTimeout https://review.openstack.org/577414 | 11:49 |
| *** suuuper has joined #tripleo | 11:49 | |
| *** dbecker has joined #tripleo | 11:49 | |
| *** suuuper has quit IRC | 11:50 | |
| bogdando | PTAL https://review.openstack.org/#/q/topic:shared_ansible_cfg+(status:open+OR+status:merged) | 11:50 |
| *** morazi has joined #tripleo | 11:52 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Use the ara-report mechanism from logs.o.o https://review.openstack.org/580090 | 11:53 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Use the ara-report mechanism from logs.o.o https://review.openstack.org/580089 | 11:53 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Use the ara-report mechanism from logs.o.o https://review.openstack.org/580088 | 11:54 |
| *** lblanchard has joined #tripleo | 11:55 | |
| *** bkopilov has joined #tripleo | 11:57 | |
| *** lvdombrkr89 has joined #tripleo | 11:58 | |
| *** mcornea has joined #tripleo | 11:58 | |
| *** skramaja_ has joined #tripleo | 11:58 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Use the ara-report mechanism from logs.o.o https://review.openstack.org/580088 | 11:58 |
| *** dhill_ has joined #tripleo | 11:58 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common stable/queens: Force lowercase hostnames https://review.openstack.org/580352 | 11:58 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Use the ara-report mechanism from logs.o.o https://review.openstack.org/580089 | 11:58 |
| *** skramaja has quit IRC | 11:59 | |
| *** lvdombrkr has quit IRC | 11:59 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: [WIP] quick'n'dirty way to at least protect and correct rights on files https://review.openstack.org/580354 | 12:00 |
| Tengu | bogdando EmilienM -^^ (kind of revival) | 12:03 |
| *** tzumainn has joined #tripleo | 12:04 | |
| *** edmondsw has quit IRC | 12:07 | |
| *** leanderthal has quit IRC | 12:08 | |
| *** avivgt has quit IRC | 12:10 | |
| *** panda is now known as panda|lunch | 12:11 | |
| *** thrash|g0ne is now known as thrash | 12:12 | |
| openstackgerrit | Alan Bishop proposed openstack/puppet-tripleo stable/pike: Run cinder's db sync only on bootstrap node https://review.openstack.org/580360 | 12:13 |
| *** edmondsw has joined #tripleo | 12:13 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix stackviz installation https://review.openstack.org/580361 | 12:15 |
| *** sshnaidm|rov|lnc is now known as sshnaidm|rover | 12:15 | |
| *** edmondsw_ has joined #tripleo | 12:16 | |
| *** edmondsw has quit IRC | 12:19 | |
| *** leanderthal has joined #tripleo | 12:19 | |
| *** jbadiapa has joined #tripleo | 12:20 | |
| *** trown|outtypewww is now known as trown | 12:20 | |
| *** rlandy has joined #tripleo | 12:23 | |
| *** quiquell is now known as quiquell|lunch | 12:27 | |
| *** raildo has joined #tripleo | 12:32 | |
| *** pradk has quit IRC | 12:32 | |
| *** leanderthal has quit IRC | 12:35 | |
| *** janki has quit IRC | 12:36 | |
| *** radeks has quit IRC | 12:37 | |
| *** gvrangan has quit IRC | 12:43 | |
| *** cshastri has quit IRC | 12:46 | |
| *** radeks has joined #tripleo | 12:47 | |
| *** leanderthal has joined #tripleo | 12:48 | |
| *** quiquell|lunch is now known as quiquell | 12:52 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: [WIP] quick'n'dirty way to at least protect and correct rights on files https://review.openstack.org/580354 | 12:53 |
| *** artom has joined #tripleo | 12:57 | |
| *** eck`gone is now known as eck` | 12:57 | |
| openstackgerrit | Bob Fournier proposed openstack/tripleo-heat-templates stable/queens: Handle blank lines followed by comment in nic conversion script https://review.openstack.org/580371 | 12:58 |
| *** panda|lunch is now known as panda | 12:59 | |
| mwhahaha | cschwede, abishop: who is dbckz, they came asking about some manila stuff in puppet-openstack but left before i could point them at the puppet-tripleo bits | 13:01 |
| cschwede | mwhahaha: maybe https://github.com/dbckz ? but that's just a guess | 13:02 |
| abishop | mwhahaha: that nic does not sound like anyone that I know | 13:02 |
| mwhahaha | oh well, if they propose a patch i'll point them over there | 13:03 |
| mwhahaha | we're missing netapp_enabled_share_protocols support | 13:03 |
| mwhahaha | in puppet-tripleo/puppet-manila | 13:03 |
| mwhahaha | but we define it in THT | 13:03 |
| *** agopi has joined #tripleo | 13:04 | |
| abishop | mwhahaha: btw, gouthamr is my peer on manila (he's US west coast) | 13:08 |
| tbarron | mwhahaha: is dbckz raising a bug or posting a review? | 13:08 |
| mwhahaha | tbarron: no idea, they just came in to #puppet-openstack to ask about netapp_enabled_share_protocols | 13:08 |
| tbarron | mwhahaha: ack | 13:08 |
| mwhahaha | and left before i could tell them about puppet-tripleo :D | 13:09 |
| EmilienM | sshnaidm|rover: sure, what's the launchpad id? | 13:09 |
| sshnaidm|rover | EmilienM, jpich already done it, thanks | 13:10 |
| *** leanderthal has quit IRC | 13:11 | |
| *** mjturek has joined #tripleo | 13:11 | |
| *** florianf has quit IRC | 13:12 | |
| EmilienM | ykarel: I'm not sure why your change yet | 13:12 |
| ykarel | EmilienM, have u checked the failure logs? | 13:13 |
| ykarel | after the change, it moved forward | 13:13 |
| EmilienM | mwhahaha: yesterday I purged the gate, and restored some patches | 13:14 |
| EmilienM | I'm about to restore other patches now, the gate doesn't seem crazy busy | 13:14 |
| mwhahaha | k | 13:14 |
| mwhahaha | i restored a few after the merge fest earlier | 13:14 |
| EmilienM | ok | 13:15 |
| Tengu | EmilienM: heya! after all, your patch regarding rights on undercloud files is back online, but aiming the current code version :). I've already added you as a reviewer. Just waiting for zuul to output its result before removing my wk-1 | 13:15 |
| *** pradk has joined #tripleo | 13:22 | |
| *** mjturek has quit IRC | 13:24 | |
| EmilienM | Tengu: ok I'll have a look shortly | 13:24 |
| ykarel | EmilienM, sorry if i made the mess in the patch | 13:26 |
| ykarel | i did to move it forward a little | 13:26 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Correct unit file name check for heat_api_cfn validation. https://review.openstack.org/580378 | 13:27 |
| *** amoralej|lunch is now known as amoralej | 13:28 | |
| EmilienM | ykarel: the reason why I've stopped docker service is because there is a service validation in the playbook that fails before the handler didn't run yet | 13:28 |
| *** jpena|lunch is now known as jpena | 13:28 | |
| EmilienM | ykarel: so I prefer to prepare containers / stop / deploy the undercloud that will start docker again | 13:29 |
| EmilienM | (with new config) | 13:29 |
| ykarel | EmilienM, yup but ansible handler started those in the pre playbook again | 13:29 |
| EmilienM | ok | 13:29 |
| ykarel | so need to move those task to different playbook, make sense? | 13:30 |
| *** ansmith has joined #tripleo | 13:32 | |
| openstackgerrit | Alan Bishop proposed openstack/puppet-tripleo master: Avoid hard-coded settings in Manila HA containers https://review.openstack.org/545060 | 13:32 |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-heat-templates master: Unify the Manila HA and non-HA docker configurations https://review.openstack.org/545061 | 13:33 |
| EmilienM | ykarel: probably | 13:37 |
| EmilienM | ykarel: the thing that worries me is I haven't seen (yet) improvements on the deployment time | 13:38 |
| *** mdnadeem has quit IRC | 13:38 | |
| EmilienM | sshnaidm|rover: how are we on the rate of failures in the gate? | 13:38 |
| sshnaidm|rover | EmilienM, not bad, there are 4 in last day | 13:39 |
| EmilienM | yeah | 13:39 |
| sshnaidm|rover | EmilienM, zuul queue is ok | 13:39 |
| EmilienM | sounds like things get better | 13:39 |
| *** pchavva has joined #tripleo | 13:39 | |
| EmilienM | so stevebaker's patch really helped (again) | 13:39 |
| *** lblanchard has quit IRC | 13:40 | |
| *** ykarel is now known as ykarel|away | 13:40 | |
| EmilienM | mwhahaha: I've restored all the rest, let's see how things merge | 13:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Deploy container-multinode (fs010) with a containerized undercloud"" https://review.openstack.org/560264 | 13:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Switch scenario00{1,2,3,4,7} to containerized undercloud"" https://review.openstack.org/575330 | 13:42 |
| *** skramaja_ has quit IRC | 13:43 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs003: deploy with instack-undercloud https://review.openstack.org/576225 | 13:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch 3nodes and scenario000-upgrades to containerized undercloud https://review.openstack.org/579755 | 13:43 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Make tripleo-ci-centos-7-undercloud-oooq non-voting" https://review.openstack.org/576258 | 13:44 |
| *** ykarel|away has quit IRC | 13:45 | |
| *** florianf has joined #tripleo | 13:46 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Use the ara-report mechanism from logs.o.o https://review.openstack.org/580090 | 13:47 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix ssl warnings on tempest containerized https://review.openstack.org/580384 | 13:48 |
| EmilienM | mwhahaha: scenario001, 2h36 on master | 13:48 |
| EmilienM | but sometimes 2h54 (just saw it) | 13:48 |
| EmilienM | I still believe if we can populate the docker registry in 'pre' tasks it should save us 15 min | 13:49 |
| *** EmilienM changes topic to "Welcome to Rocky. CI status: Orange - please report timeouts or gate issues - recheck/+w OK | https://docs.openstack.org/tripleo-docs/latest" | 13:50 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Make sure we apply qemu config changes https://review.openstack.org/579807 | 13:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Set live_migration_inbound_addr for ssh transport https://review.openstack.org/579544 | 13:50 |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: [WIP] quick'n'dirty way to at least protect and correct rights on files https://review.openstack.org/580354 | 13:50 |
| *** leanderthal has joined #tripleo | 13:51 | |
| verdurin | Any idea why ceph-ansible would be looking for /dev/vdb on baremetal Ceph OSD nodes? | 13:52 |
| *** quiquell is now known as quiquell|mtg | 13:53 | |
| *** nyechiel_ has joined #tripleo | 13:57 | |
| *** rmascena has joined #tripleo | 13:57 | |
| *** agopi has quit IRC | 13:58 | |
| *** raildo has quit IRC | 13:59 | |
| *** hamdykhader has joined #tripleo | 14:00 | |
| *** nyechiel has quit IRC | 14:01 | |
| openstackgerrit | Martin Schuppert proposed openstack/puppet-tripleo stable/queens: Allow configuration of NFS backend for Nova https://review.openstack.org/580386 | 14:01 |
| *** leanderthal has quit IRC | 14:02 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: Allow configuration of NFS backend for Nova https://review.openstack.org/580388 | 14:03 |
| *** hamdyk has quit IRC | 14:04 | |
| *** waleedm has quit IRC | 14:04 | |
| *** hamdyk has joined #tripleo | 14:05 | |
| *** hamdykhader has quit IRC | 14:05 | |
| *** yrabl has joined #tripleo | 14:06 | |
| *** shreshtha has quit IRC | 14:06 | |
| *** ykarel has joined #tripleo | 14:08 | |
| *** moshele has quit IRC | 14:08 | |
| ykarel | EmilienM, i have not seen the difference in deployment time, so can't say | 14:10 |
| ykarel | btw which patch improved the situation | 14:10 |
| *** rmascena is now known as raildo | 14:11 | |
| *** rmascena has joined #tripleo | 14:13 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: Enable secure TUNNELLED mode for NFS https://review.openstack.org/580392 | 14:16 |
| EmilienM | ykarel: http://logs.openstack.org/37/580037/31/check/tripleo-ci-centos-7-undercloud-containers/a4b0be4/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-05_12_16_51 | 14:16 |
| EmilienM | ykarel: in a meeting now, but after i'll check why docker registry isn't available | 14:16 |
| *** raildo has quit IRC | 14:16 | |
| ykarel | EmilienM, ack | 14:16 |
| ykarel | it's available but something to do with ssl | 14:17 |
| *** toure|gone is now known as toure | 14:17 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Execute Nova DB tasks iff the node have MySQL installed. https://review.openstack.org/580393 | 14:19 |
| *** rmascena is now known as raildo | 14:24 | |
| chkumar|ruck | ykarel: http://logs.openstack.org/88/579888/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/250539f/job-output.txt.gz#_2018-07-05_13_56_33_432745 | 14:25 |
| *** agopi has joined #tripleo | 14:25 | |
| chkumar|ruck | something new | 14:25 |
| ykarel | chkumar|ruck, it's job timeout | 14:26 |
| *** agopi_ has joined #tripleo | 14:27 | |
| chkumar|ruck | ykarel: | 14:27 |
| chkumar|ruck | yup | 14:27 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Execute Nova DB tasks iff the node have MySQL installed. https://review.openstack.org/580393 | 14:28 |
| *** hamdyk has quit IRC | 14:28 | |
| *** agopi has quit IRC | 14:30 | |
| *** itlinux has quit IRC | 14:40 | |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-validations master: Refactor service client getters https://review.openstack.org/578127 | 14:43 |
| *** ccamacho1 has joined #tripleo | 14:44 | |
| *** ccamacho1 has quit IRC | 14:44 | |
| *** ccamacho1 has joined #tripleo | 14:45 | |
| *** ccamacho has quit IRC | 14:45 | |
| *** dprince has quit IRC | 14:45 | |
| *** dparkes has quit IRC | 14:46 | |
| *** ansmith has quit IRC | 14:48 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Execute Nova DB tasks iff the node have MySQL installed. https://review.openstack.org/580393 | 14:52 |
| *** tcw1 has joined #tripleo | 14:53 | |
| *** tcw has quit IRC | 14:57 | |
| *** tcw1 is now known as tcw | 14:57 | |
| *** nyechiel has joined #tripleo | 15:00 | |
| *** leanderthal has joined #tripleo | 15:01 | |
| karthiks | mwhahaha, EmilienM requirements-check was failing for https://review.openstack.org/#/c/541620/ . Can you suggest any pointers for this issue ? | 15:02 |
| mwhahaha | karthiks: the jsconschema line is out of sync | 15:03 |
| mwhahaha | karthiks: so it needs to get updated (probably in a different patch) | 15:03 |
| *** ykarel is now known as ykarel|away | 15:03 | |
| *** nyechiel_ has quit IRC | 15:04 | |
| karthiks | mwhahaha, similar message is thrown for netaddr as well | 15:05 |
| mwhahaha | yea since we dont' run the requirements job all the time it appears that we're out of sync | 15:05 |
| bogdando | mwhahaha, EmilienM: let's merge https://review.openstack.org/#/c/567810/ please | 15:06 |
| *** janki has joined #tripleo | 15:06 | |
| bogdando | marios: ^^ | 15:07 |
| EmilienM | bogdando: looking now | 15:07 |
| EmilienM | bogdando: but I already see problem with ovn rebase | 15:07 |
| bogdando | EmilienM: oops, which one | 15:08 |
| EmilienM | bogdando: nevermind | 15:09 |
| EmilienM | bogdando: it's good, let me finish the review but I think we can land it | 15:09 |
| karthiks | mwhahaha, skramaja tried https://review.openstack.org/#/c/580290/. It seems to have gone past the test-requirement | 15:09 |
| EmilienM | bogdando: I'll let mwhahaha do a last review but +2 from me | 15:10 |
| mwhahaha | seems like os-net-config hasn't been keeping up with requirements | 15:10 |
| bogdando | EmilienM: thank you | 15:12 |
| karthiks | mwhahaha, any guidelines available for fixing it ? | 15:12 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 15:13 |
| mwhahaha | karthiks: no idea, maybe #openstack-requirements can help. i usually just wing it | 15:13 |
| karthiks | mwhahaha, Thanks | 15:13 |
| Tengu | hmmm. apparently, there's a small issue with the "deployment_user" not being set everytime we need it, EmilienM. weird. have to find out why it's sometimes not in the "self". | 15:14 |
| *** rpioso|afk is now known as rpioso | 15:20 | |
| *** peereb has quit IRC | 15:22 | |
| openstackgerrit | wu.chunyang proposed openstack/paunch master: Add release note link in README https://review.openstack.org/580406 | 15:23 |
| *** itlinux has joined #tripleo | 15:25 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix stackviz installation https://review.openstack.org/580361 | 15:25 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Run only CI playbooks in run.yml zuul playbook https://review.openstack.org/565740 | 15:25 |
| *** sanjayu_ has quit IRC | 15:25 | |
| *** bnemec has quit IRC | 15:25 | |
| *** quiquell|mtg is now known as quiquell|off | 15:26 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch 3nodes and scenario000-upgrades to containerized undercloud https://review.openstack.org/579755 | 15:29 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Support ARA statistics in InfluxDB for longest tasks https://review.openstack.org/580238 | 15:32 |
| sshnaidm|rover | weshay, https://review.openstack.org/#/c/580238/ | 15:32 |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-common master: Include microversion in OS_VOLUME_API_VERSION https://review.openstack.org/580410 | 15:33 |
| *** dparkes has joined #tripleo | 15:33 | |
| sshnaidm|rover | weshay, example: http://logs.openstack.org/38/580238/1/check/tripleo-ci-centos-7-undercloud-containers/045f812/logs/influxdb_data | 15:33 |
| sshnaidm|rover | weshay, will create grafana for it later, but this is the idea actually | 15:33 |
| sshnaidm|rover | weshay, is it ok or something else needed? | 15:34 |
| *** radeks has quit IRC | 15:35 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Removing test from skip list https://review.openstack.org/580423 | 15:39 |
| pradk | can i request reviews on https://review.openstack.org/#/c/576220/ | 15:40 |
| *** agopi_ is now known as agopi | 15:45 | |
| weshay | sshnaidm|rover, k.. that is nice | 15:47 |
| weshay | sshnaidm|rover, any progress on the overcloud ara? | 15:47 |
| *** dparkes has quit IRC | 15:47 | |
| sshnaidm|rover | weshay, nope :( will try to talk with David is he'll be online | 15:48 |
| EmilienM | sshnaidm|rover: he's online but not on this channel anymore | 15:48 |
| sshnaidm|rover | EmilienM, oh, ok | 15:49 |
| sshnaidm|rover | EmilienM, actually I'm not sure he can help me because it seems like not problem with ara, but with how we execute ansible in deployment.. | 15:49 |
| weshay | sshnaidm|rover, https://review.openstack.org/#/c/580256/ | 15:51 |
| *** dxiri has joined #tripleo | 15:51 | |
| bogdando | mwhahaha: wanted to ask you for the recent systemd unit patch, are you sure that systemd containers stopper unit needs to filter by --managed-by paunch, or tripleo-.*? just not sure, wanted to double-check its OK | 15:52 |
| bogdando | for the ref http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/common/deploy-steps-tasks.yaml#n215 | 15:52 |
| sshnaidm|rover | weshay, nice.. how is it related though? | 15:53 |
| weshay | just fyi | 15:53 |
| *** dprince has joined #tripleo | 15:54 | |
| *** radeks has joined #tripleo | 15:55 | |
| bogdando | mwhahaha: I mean https://bugzilla.redhat.com/show_bug.cgi?id=1597797 | 15:55 |
| openstack | bugzilla.redhat.com bug 1597797 in python-paunch "containers are not gracefully stopped on shutdown/reboot" [High,Modified] - Assigned to aschultz | 15:55 |
| sshnaidm|rover | EmilienM, do you know where are those commands configured? http://logs.openstack.org/62/578462/7/check/tripleo-ci-centos-7-containers-multinode/67f8589/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-07-03_15_42_26 | 15:56 |
| *** bnemec has joined #tripleo | 15:56 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 16:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO https://review.openstack.org/564285 | 16:00 |
| *** bogdando has quit IRC | 16:00 | |
| *** leanderthal has quit IRC | 16:02 | |
| *** udesale has quit IRC | 16:04 | |
| *** nyechiel_ has joined #tripleo | 16:08 | |
| *** nyechiel has quit IRC | 16:08 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: [DNM] test new jobs configuration https://review.openstack.org/579653 | 16:08 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Migrating 2-nodes jobs to the new base configuration https://review.openstack.org/578456 | 16:09 |
| *** yolanda_ has joined #tripleo | 16:13 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: [WIP] quick'n'dirty way to at least protect and correct rights on files https://review.openstack.org/580354 | 16:13 |
| Tengu | \o/ I think it's good now. I'll check back tomorrow | 16:13 |
| *** lvdombrkr89 has quit IRC | 16:14 | |
| EmilienM | sshnaidm|rover: looking | 16:15 |
| *** ksambor has quit IRC | 16:16 | |
| EmilienM | sshnaidm|rover: https://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/image/image_uploader.py#n199 | 16:16 |
| sshnaidm|rover | EmilienM, thanks | 16:16 |
| *** yolanda has quit IRC | 16:17 | |
| *** moshele has joined #tripleo | 16:17 | |
| *** jpich has quit IRC | 16:18 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: collect ara OC https://review.openstack.org/578462 | 16:19 |
| *** jfrancoa has quit IRC | 16:20 | |
| *** radeks has quit IRC | 16:22 | |
| *** pradk has quit IRC | 16:23 | |
| weshay | EmilienM, mwhahaha didn't realize we had the unbound dns logged fyi. as example http://logs.openstack.org/40/565740/20/check/tripleo-ci-centos-7-undercloud-containers/58347c4/logs/dns_cache.txt.gz | 16:24 |
| EmilienM | weshay: yeah | 16:24 |
| *** pradk has joined #tripleo | 16:25 | |
| mwhahaha | weshay: yea we turned that on last time we had these problems | 16:27 |
| hjensas | A couple simple reviews, in case anyone have time: https://review.openstack.org/580230 , https://review.openstack.org/580223 (thanks!) | 16:29 |
| *** agopi is now known as agopi|lunch | 16:30 | |
| *** radeks has joined #tripleo | 16:30 | |
| *** yrabl has quit IRC | 16:31 | |
| *** janki has quit IRC | 16:31 | |
| EmilienM | weshay: I'll continue here | 16:47 |
| EmilienM | weshay, mwhahaha : today I want to figure out if my patch ( https://review.openstack.org/#/c/580037/ ) actually helps | 16:47 |
| *** panda is now known as panda|off | 16:47 | |
| EmilienM | I've seen shorter numbers in the way containers are pulled when deploying the undercloud but still unsure about undercloud (working on it now) | 16:47 |
| EmilienM | I also think an ultimate optimization would be to run a mirror per region, and pull directly from mirror | 16:48 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/queens: Default environment/services/* to docker https://review.openstack.org/567863 | 16:48 |
| mwhahaha | EmilienM: so it's not necessarily going to help with shorter numbers, but it moves the time out of the run time | 16:48 |
| EmilienM | also the "yum update" done in the containers takes a lot of time IMHO, we also want to investigate that | 16:48 |
| EmilienM | I'm not sure all containers need yum update, but well | 16:48 |
| EmilienM | mwhahaha: right | 16:49 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-validations master: Refactor service client getters https://review.openstack.org/578127 | 16:51 |
| *** bnemec has quit IRC | 16:51 | |
| *** psachin` has quit IRC | 16:52 | |
| *** yolanda__ has joined #tripleo | 16:53 | |
| *** ykarel|away has quit IRC | 16:54 | |
| openstackgerrit | Flavio Percoco proposed openstack/puppet-tripleo master: Bind openshift-master to the public network https://review.openstack.org/580451 | 16:56 |
| *** rpioso is now known as rpioso|eat | 16:56 | |
| *** yolanda_ has quit IRC | 16:56 | |
| EmilienM | mwhahaha, weshay : http://paste.openstack.org/show/725131/ some comparaison | 16:57 |
| *** thrash is now known as thrash|biab | 16:58 | |
| weshay | EmilienM, nice.. 6 seconds ---> .1 second | 16:59 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: HA support for OpenShift https://review.openstack.org/579609 | 16:59 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Replace raw_get with dict access syntax https://review.openstack.org/579843 | 16:59 |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-heat-templates master: Unify the Manila HA and non-HA docker configurations https://review.openstack.org/545061 | 16:59 |
| tbarron | abishop: ty ^^ (will leave the downstream one as it is) | 17:01 |
| abishop | tbarron: yup! | 17:02 |
| *** ffiore has quit IRC | 17:02 | |
| *** derekh has quit IRC | 17:02 | |
| *** amoralej is now known as amoralej|off | 17:03 | |
| *** ccamacho1 has quit IRC | 17:03 | |
| *** bogdando has joined #tripleo | 17:05 | |
| EmilienM | weshay: I'm not sure again, I'll let you know asap | 17:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Do not disable ipv6 on loopback interface for epmd https://review.openstack.org/580120 | 17:06 |
| *** trown is now known as trown|lunch | 17:07 | |
| *** cmyster has joined #tripleo | 17:08 | |
| *** cmyster has joined #tripleo | 17:08 | |
| *** agopi|lunch is now known as agopi | 17:08 | |
| EmilienM | meantime, things seem to merge | 17:10 |
| EmilienM | mwhahaha, weshay : https://review.openstack.org/#/c/560264/ takes 2h 18m 16s | 17:10 |
| EmilienM | I think it's good to go | 17:10 |
| EmilienM | but I think it's far from timeout anyway | 17:12 |
| mwhahaha | that wasn't an issue with time, i thought it was an issue with patches not being included | 17:12 |
| mwhahaha | is that fixed? | 17:12 |
| *** mjturek has joined #tripleo | 17:12 | |
| *** yolanda_ has joined #tripleo | 17:12 | |
| EmilienM | mwhahaha: what do you mean? | 17:12 |
| *** moshele has quit IRC | 17:13 | |
| EmilienM | mwhahaha: it's a featureset patch, it should be tested live | 17:13 |
| mwhahaha | https://review.openstack.org/#/c/559740/ | 17:13 |
| EmilienM | right but I think we're updating containers now | 17:13 |
| EmilienM | let me confirm | 17:13 |
| mwhahaha | yea my point is we reverted because of that, not because of job length | 17:13 |
| *** ramishra has quit IRC | 17:13 | |
| EmilienM | right | 17:14 |
| *** yolanda__ has quit IRC | 17:15 | |
| EmilienM | mwhahaha: http://logs.openstack.org/64/560264/40/check/tripleo-ci-centos-7-containers-multinode/9f28970/logs/undercloud/var/log/journal.txt.gz#_Jul_05_14_49_41 | 17:16 |
| EmilienM | undercloud containers are now updated | 17:16 |
| mwhahaha | k | 17:18 |
| *** bnemec has joined #tripleo | 17:18 | |
| weshay | EmilienM, check out limestone container prep https://review.rdoproject.org/grafana/dashboard/db/tripleo-ci?orgId=1&var-pipeline=All&var-branch=All&var-cloud=limestone-regionone&var-type=All&var-jobtype=All&from=now-30d&to=now | 17:18 |
| weshay | since 6/27 | 17:19 |
| EmilienM | weshay: let me see what we've landed | 17:19 |
| weshay | EmilienM, vexhost anyone https://review.rdoproject.org/grafana/dashboard/db/tripleo-ci?orgId=1&from=now-30d&to=now&var-pipeline=All&var-branch=All&var-cloud=vexxhost-ca-ymq-1&var-type=All&var-jobtype=All | 17:19 |
| *** khyr0n has joined #tripleo | 17:20 | |
| *** rlandy is now known as rlandy|brb | 17:21 | |
| EmilienM | weshay: crazy when you compare from early june | 17:22 |
| EmilienM | but it's normal | 17:22 |
| EmilienM | we "update" containers in our CI | 17:22 |
| EmilienM | and it takes about that time | 17:22 |
| EmilienM | weshay: if you ok (or not), I'll let you press the final review on https://review.openstack.org/#/c/560264/ | 17:23 |
| *** jpena is now known as jpena|off | 17:24 | |
| *** dtantsur is now known as dtantsur|afk | 17:26 | |
| *** yamahata has joined #tripleo | 17:27 | |
| *** ansmith has joined #tripleo | 17:28 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix ssl warnings on tempest containerized https://review.openstack.org/580384 | 17:28 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Disable ssl certificate validation in tempest https://review.openstack.org/578052 | 17:28 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Add --stack to update, upgrade and ffwd-upgrade 'run' CLI. https://review.openstack.org/568273 | 17:28 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Update config docs https://review.openstack.org/579178 | 17:28 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Fix net_config_override https://review.openstack.org/580084 | 17:28 |
| EmilienM | weshay: did you rename to "Containers"? | 17:33 |
| EmilienM | in https://review.rdoproject.org/grafana/dashboard/db/tripleo-ci?orgId=1&var-pipeline=All&var-branch=All&var-cloud=limestone-regionone&var-type=All&var-jobtype=All&from=now-30d&to=now | 17:33 |
| EmilienM | if yes, make it "Container prepare" plese | 17:33 |
| weshay | ah k | 17:34 |
| weshay | EmilienM, which networking config do we need for multinode https://review.openstack.org/#/c/560264/40 | 17:34 |
| EmilienM | weshay: anyway the numbers make total sense | 17:34 |
| weshay | EmilienM, simple-bridge or net-config-undercloud/ | 17:34 |
| weshay | > | 17:34 |
| weshay | ? | 17:34 |
| weshay | geez | 17:34 |
| *** thrash|biab is now known as thrash | 17:35 | |
| *** bogdando has quit IRC | 17:35 | |
| EmilienM | weshay: net-config-simple-bridge for now | 17:36 |
| weshay | EmilienM, YOU SURE? | 17:36 |
| weshay | EmilienM, because it doesn't work w/ ovb | 17:36 |
| weshay | on master and queens | 17:36 |
| EmilienM | weshay: this is fs010 | 17:36 |
| EmilienM | multinode-container | 17:36 |
| weshay | yes | 17:37 |
| weshay | just relating my experience w/ fs002 and this change | 17:37 |
| * weshay looks at the job results carefully | 17:37 | |
| *** rlandy|brb is now known as rlandy | 17:40 | |
| *** nyechiel_ has quit IRC | 17:42 | |
| *** sshnaidm|rover has quit IRC | 17:43 | |
| *** bkopilov has quit IRC | 17:43 | |
| *** holser_ has quit IRC | 17:43 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Configure SELinux in the undercloud https://review.openstack.org/578857 | 17:44 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Octavia amphora image handling updates https://review.openstack.org/566557 | 17:44 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add networking-ansible ML2 plugin support https://review.openstack.org/577620 | 17:44 |
| *** nyechiel_ has joined #tripleo | 17:51 | |
| *** sshnaidm|rover has joined #tripleo | 17:52 | |
| *** bkopilov has joined #tripleo | 17:52 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Add mechanism for cookie-based sessions in endpoints. https://review.openstack.org/579660 | 17:55 |
| *** moshele has joined #tripleo | 17:56 | |
| *** trown|lunch is now known as trown | 17:59 | |
| *** dparkes has joined #tripleo | 18:01 | |
| EmilienM | weshay: tell us more about 8082 port and the thing you talked with apevec | 18:01 |
| weshay | EmilienM, well if we don't have access to the mirror logs, then we would have to setup our own mirror | 18:02 |
| *** moshele has quit IRC | 18:02 | |
| weshay | run some tests against it and see what's up | 18:02 |
| weshay | afaict | 18:02 |
| *** akrivoka has quit IRC | 18:04 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates stable/queens: Do not disable ipv6 on loopback interface for epmd https://review.openstack.org/580460 | 18:04 |
| *** marrusl has joined #tripleo | 18:04 | |
| *** alee has quit IRC | 18:10 | |
| *** tcw has quit IRC | 18:10 | |
| *** alee has joined #tripleo | 18:10 | |
| *** tcw has joined #tripleo | 18:12 | |
| *** yolanda__ has joined #tripleo | 18:13 | |
| *** florianf has quit IRC | 18:14 | |
| *** yolanda_ has quit IRC | 18:15 | |
| *** ccamacho has joined #tripleo | 18:23 | |
| *** mjturek has quit IRC | 18:26 | |
| openstackgerrit | Bob Fournier proposed openstack/tripleo-heat-templates stable/queens: Fix incorrect Redis VIP declaration in fixed-ip-vips-v6.yaml https://review.openstack.org/580464 | 18:29 |
| EmilienM | weshay: FYI https://review.openstack.org/#/c/575520/ paul said it woul dhelp | 18:30 |
| *** pabelanger has joined #tripleo | 18:30 | |
| EmilienM | pabelanger told me https://review.openstack.org/#/c/575520/ would help | 18:30 |
| pabelanger | o. | 18:30 |
| pabelanger | yes | 18:30 |
| EmilienM | but also that we would need to make smaller containers on the long term | 18:30 |
| pabelanger | there is a lot of cache misses on tripleo containers | 18:30 |
| pabelanger | right now it is 50GB cache | 18:30 |
| EmilienM | tbh I'm not aware of any effort to reduce the containers | 18:31 |
| EmilienM | dprince: ^ thoughts on this one? | 18:31 |
| pabelanger | long term, focus on making containers smaller, it really will speed up job runtimes here | 18:31 |
| pabelanger | last I looked, container usage in tripleo was very large | 18:31 |
| pabelanger | size wise | 18:31 |
| EmilienM | right I even think we should change our container format in the long term I'm not sure we're doing right with the kolla containers | 18:32 |
| dprince | EmilienM: does that size cover stable/maint branch containers too? | 18:32 |
| EmilienM | dprince: good question | 18:32 |
| weshay | rlandy, http://logs.openstack.org/90/580090/9/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/3d4ce8e/zuul-info/inventory.yaml | 18:33 |
| pabelanger | which size, of the cache? | 18:34 |
| *** rpioso|eat is now known as rpioso | 18:35 | |
| *** yolanda_ has joined #tripleo | 18:35 | |
| *** fzdarsky__ has quit IRC | 18:37 | |
| *** yolanda__ has quit IRC | 18:39 | |
| *** dxiri has quit IRC | 18:39 | |
| *** yolanda__ has joined #tripleo | 18:39 | |
| *** yolanda_ has quit IRC | 18:42 | |
| weshay | https://review.openstack.org/#/c/519596/ | 18:44 |
| weshay | https://review.openstack.org/#/c/546474/ | 18:44 |
| rlandy | weshay: https://review.openstack.org/#/c/565740/ | 18:45 |
| *** cmyster has quit IRC | 18:46 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 18:46 | |
| *** cmyster has joined #tripleo | 18:47 | |
| *** cmyster has joined #tripleo | 18:47 | |
| pabelanger | openstackgerrit | Merged openstack-infra/system-config master: Increase apache mirror cache to 70GB https://review.openstack.org/575520 | 18:51 |
| pabelanger | will be live in 15mins | 18:51 |
| *** fzdarsky__ has joined #tripleo | 18:51 | |
| pabelanger | EmilienM: weshay: ^ | 18:51 |
| EmilienM | let's see how that help | 18:52 |
| pabelanger | it will, but doubt it will get 100% cache hits | 18:54 |
| *** ccamacho has quit IRC | 18:54 | |
| *** dprince has quit IRC | 19:00 | |
| *** kbyrne has joined #tripleo | 19:05 | |
| *** dxiri has joined #tripleo | 19:07 | |
| *** moshele has joined #tripleo | 19:14 | |
| *** dprince has joined #tripleo | 19:26 | |
| *** radeks has quit IRC | 19:27 | |
| *** gkadam has quit IRC | 19:32 | |
| *** lifeless has joined #tripleo | 19:39 | |
| *** kbyrne has quit IRC | 19:42 | |
| *** kbyrne has joined #tripleo | 19:45 | |
| *** moshele has quit IRC | 19:46 | |
| *** alee has quit IRC | 19:46 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: make fs21 like fs20 w/o skip list https://review.openstack.org/580480 | 19:47 |
| *** moshele has joined #tripleo | 19:51 | |
| *** sanjay__u has joined #tripleo | 19:56 | |
| *** Goneri has quit IRC | 20:03 | |
| *** pradk has quit IRC | 20:05 | |
| *** ccamacho has joined #tripleo | 20:19 | |
| *** moshele has quit IRC | 20:21 | |
| *** pradk has joined #tripleo | 20:21 | |
| *** gouthamr has quit IRC | 20:27 | |
| *** stevebaker has quit IRC | 20:28 | |
| *** dmellado has quit IRC | 20:28 | |
| *** ffiore has joined #tripleo | 20:35 | |
| *** morazi has quit IRC | 20:37 | |
| *** raildo has quit IRC | 20:40 | |
| *** morazi has joined #tripleo | 20:42 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use the ara-report mechanism from logs.o.o https://review.openstack.org/580090 | 20:47 |
| openstackgerrit | Rafael Folco proposed openstack-infra/tripleo-ci master: Create a new parent layer for multinode jobs https://review.openstack.org/580500 | 20:47 |
| mschuppert | hi, are there again gate issues? https://review.openstack.org/#/c/579813/2 seems not to enter the queue but passed ci checks two times. not sure if it helps if I trigger it again a recheck. | 20:47 |
| EmilienM | weshay: http://tripleo.org/cistatus.html is down for me | 20:48 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Conditionally Enable Nuage VRS https://review.openstack.org/577339 | 20:49 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: DNM test prep containers https://review.openstack.org/580505 | 20:54 |
| *** pchavva has quit IRC | 20:55 | |
| EmilienM | gate looks ok'ish, a bit busy but nothing crazy | 20:58 |
| EmilienM | I don't see much resets | 20:58 |
| pabelanger | EmilienM: we'll start to invalidate the cache again in 1 minute (ever hour) | 20:59 |
| pabelanger | but so far new size is helping misses | 21:00 |
| EmilienM | ok | 21:00 |
| EmilienM | brb 15 min | 21:01 |
| mschuppert | ok thx, I'll run another recheck then. will wait till tomorrow morning my time. | 21:01 |
| *** dprince has quit IRC | 21:04 | |
| *** pradk- has joined #tripleo | 21:07 | |
| *** pradk has quit IRC | 21:07 | |
| *** pkilambi__ has joined #tripleo | 21:08 | |
| *** pradk- has quit IRC | 21:12 | |
| *** abishop has quit IRC | 21:14 | |
| *** pkilambi__ has quit IRC | 21:21 | |
| *** itlinux has quit IRC | 21:24 | |
| *** dparkes has quit IRC | 21:31 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 21:32 |
| *** gouthamr has joined #tripleo | 21:35 | |
| *** toure is now known as toure|gone | 21:35 | |
| *** holser_ has joined #tripleo | 21:36 | |
| tonyb | Tengu: If you're around can you explain what I did wrong in https://review.openstack.org/#/c/580306/1/manifests/inspector.pp@311 ? | 21:37 |
| *** morazi has quit IRC | 21:37 | |
| EmilienM | tonyb: I thought we EOLed newton in tripleo but I still see the branches | 21:39 |
| EmilienM | what's missing? | 21:39 |
| tonyb | EmilienM: It's all on me nothign is missing as such I haven't haven't done the ground work. I'll do it today and get them EOL'd next week | 21:39 |
| EmilienM | tonyb: no worries, I was just curious | 21:40 |
| EmilienM | let me know if there is anything I can do to help | 21:41 |
| tonyb | EmilienM: Will do | 21:41 |
| tonyb | EmilienM: if you had 30 seconds could you look at https://review.openstack.org/#/c/580306/1/manifests/inspector.pp@311 and tell me what the correct syntax is | 21:46 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Don't trigger repo injection if nothing was built https://review.openstack.org/576417 | 21:49 |
| *** stevebaker has joined #tripleo | 21:52 | |
| *** rcernin has joined #tripleo | 22:00 | |
| *** holser_ has quit IRC | 22:01 | |
| *** mcornea has quit IRC | 22:02 | |
| *** holser_ has joined #tripleo | 22:06 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix stackviz installation https://review.openstack.org/580361 | 22:14 |
| EmilienM | tonyb: of course, looking now | 22:24 |
| EmilienM | tonyb: I pushed over your patch, hope you don't mind | 22:26 |
| tonyb | EmilienM: No problem at all. Quite the opposite I'm happy for others to correct my work like that. | 22:29 |
| EmilienM | stupid puppet syntax but I think we needed a proper array for the require | 22:30 |
| tonyb | EmilienM: I tried to find an example but it's hard when you aren't sure what to grep for ;P | 22:31 |
| tonyb | stevebaker: Is there an easy way to use mofify image manually, I don't seem to be able to get it to work and waitign while it doesn't work on 50+ containers is a little time consuming I'd like to work on just one if possible to debug the yum stuff | 22:34 |
| EmilienM | pabelanger: http://logs.openstack.org/37/580037/33/check/tripleo-ci-centos-7-undercloud-containers/a0b93ad/job-output.txt.gz#_2018-07-05_22_02_29_590852 | 22:37 |
| pabelanger | EmilienM: you need to add it to required-projects | 22:38 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 22:38 |
| EmilienM | pabelanger: ah ok | 22:38 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 22:40 |
| *** holser_ has quit IRC | 22:40 | |
| *** edmondsw_ has quit IRC | 22:40 | |
| *** edmondsw has joined #tripleo | 22:41 | |
| *** ffiore has quit IRC | 22:41 | |
| pabelanger | http://logs.openstack.org/82/579382/5/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/f293b9f/job-output.txt.gz#_2018-07-05_21_25_07_849365 | 22:43 |
| pabelanger | that's the change to zuul-jobs to fix ^ | 22:43 |
| pabelanger | also, for some reason 30 minutes wasn't enough to collect logs for a jobs: http://logs.openstack.org/96/579896/2/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/4484401/job-output.txt.gz#_2018-07-05_21_14_46_790452 | 22:44 |
| pabelanger | perhaps we should see about reducing the amount of IO the log collection process does | 22:44 |
| *** edmondsw has quit IRC | 22:45 | |
| stevebaker | tonyb: you can test in isolation by having a modified containers-prepare-parameter.yaml which has an includes of only the images you want to modify, then run that with "openstack tripleo container image prepare". If you give me your containers-prepare-parameter.yaml and say what images you want to modify then I'll can come up with something | 22:45 |
| *** noslzzp has joined #tripleo | 22:46 | |
| tonyb | stevebaker: And I can run that several times without issue? in that it'll just recreate the updated image each time? | 22:46 |
| stevebaker | tonyb: no, if an image exists with the target name it will skip the rebuild. You need to change the modify_append_tag value each time you want to force a rebuild | 22:47 |
| tonyb | stevebaker: Cool | 22:48 |
| stevebaker | tonyb: or delete the image from the registry | 22:48 |
| tonyb | stevebaker: Okay I'll try either of those. | 22:48 |
| tonyb | stevebaker: my current theory is that yum-priorities are forcing the already installed version to be selected rather than the one from my repo | 22:52 |
| stevebaker | tonyb: do you have this fix? https://review.openstack.org/#/c/579803/ | 22:53 |
| tonyb | stevebaker: let me check | 22:53 |
| stevebaker | tonyb: also if the package update works on the undercloud host, it should work in the container | 22:54 |
| tonyb | stevebaker: give a few I'm just rebuilding my wouldbe director node | 22:54 |
| tonyb | stevebaker: Yeah that was my thinking (and it does work on the host) | 22:55 |
| stevebaker | unless the containers are newer I guess | 22:56 |
| tonyb | stevebaker: Actually I *think* they're older by 2 days | 22:57 |
| tonyb | stevebaker: I'm going to switch the host to the same repos just to be sure | 22:57 |
| tonyb | stevebaker: for soem reason I couldn't find them last time I looked but they're right where I'd expect them to be. | 22:58 |
| *** rcernin has quit IRC | 22:58 | |
| stevebaker | tonyb: btw I'll be writing a dedicated modify-role to support this sort of development, I'd like to collect your use cases after this is all settled | 22:59 |
| tonyb | stevebaker: For sure. | 23:00 |
| tonyb | stevebaker: It may be that I'm doing soemthign simple in a complex way ;P | 23:00 |
| *** rcernin has joined #tripleo | 23:01 | |
| *** tosky has quit IRC | 23:01 | |
| stevebaker | tonyb: doesn't alt-arch mean doing that constantly? ;) | 23:01 |
| tonyb | stevebaker: LOL | 23:04 |
| EmilienM | stevebaker: if you have time, could you please take a look at https://review.openstack.org/#/c/580037/ and tell me what you think? | 23:04 |
| openstackgerrit | Ben Nemec proposed openstack/puppet-tripleo master: Make Designate pools.yaml configurable https://review.openstack.org/580524 | 23:05 |
| EmilienM | stevebaker: specially the namespace I'm pushing; I'm not sure it's actually efficient as I've seen pulls again later when undercloud is deployed | 23:06 |
| EmilienM | need to drop for ~1h30, bbl | 23:06 |
| *** yolanda_ has joined #tripleo | 23:07 | |
| stevebaker | EmilienM: if this runs on the undercloud, I'm not seeing how it saves any time. Its doing the same things but earlier and all-at-once instead of spread throughout the undercloud/overcloud deployment | 23:07 |
| EmilienM | stevebaker: right my goal is to doing it earlier | 23:08 |
| EmilienM | yes it runs on the undercloud | 23:08 |
| *** yolanda__ has quit IRC | 23:08 | |
| EmilienM | stevebaker: the whole idea is to do it earlier in pre, not in run | 23:08 |
| EmilienM | to reduce chances to hit timeouts in run | 23:09 |
| *** mrunge has quit IRC | 23:09 | |
| *** mrunge has joined #tripleo | 23:09 | |
| EmilienM | stevebaker: but I feel like the namespace diff makes it to it's pulled again later. If you have thoughts on the patches, please let me know in the review | 23:09 |
| *** yolanda__ has joined #tripleo | 23:11 | |
| stevebaker | EmilienM: do pre and run both draw from the same job length max time though? Maybe that is where I'm misunderstanding | 23:11 |
| *** yolanda_ has quit IRC | 23:13 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Enable configuration of Designate's pools.yaml https://review.openstack.org/580526 | 23:13 |
| *** fzdarsky__ has quit IRC | 23:14 | |
| *** mrunge_ has joined #tripleo | 23:18 | |
| *** mrunge has quit IRC | 23:19 | |
| *** tzumainn has quit IRC | 23:19 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 23:21 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: WIP: test w/ updated registry location https://review.openstack.org/580528 | 23:24 |
| *** rlandy has quit IRC | 23:25 | |
| *** mrunge_ has quit IRC | 23:25 | |
| *** mrunge has joined #tripleo | 23:26 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: add job variables via zuul https://review.openstack.org/580529 | 23:28 |
| tonyb | stevebaker: for the record, yes I do have that bugfix | 23:33 |
| stevebaker | tonyb: ok when you get to the bottom of it I'd like to make sure CI isn't affect by this kind of thing too - I suspect not though | 23:33 |
| tonyb | stevebaker: Yup, I suspect it's all down to me ;P | 23:35 |
| *** rpioso is now known as rpioso|afk | 23:36 | |
| tonyb | stevebaker: I wonder if it's worth addign soemthing like: mkdir -p rpmlists; for container in $(docker ps --format "{{.Names}}") ; do docker exec -t $container rpm -qa > rpmlists/$container ; done | 23:38 |
| tonyb | somewhere in the log collection phase to be abel to verify that the containers did get the code expected | 23:39 |
| tonyb | mind you it's probbaly rarely helpful | 23:39 |
| stevebaker | tonyb: yeah that is something which quickstart should do | 23:39 |
| *** yolanda_ has joined #tripleo | 23:40 | |
| *** yolanda__ has quit IRC | 23:42 | |
| *** agopi is now known as agopi|off | 23:51 | |
| tonyb | stevebaker: Ah I think I've found it. I was using a local directory to hold my repo file '/home/stack/repos.d' which has been copied to /etc as repos.d | 23:52 |
| tonyb | stevebaker: so if I rename my dir it shoudl work | 23:52 |
| tonyb | stevebaker: I kinda feel like that needs to be called out in the docs | 23:53 |
| stevebaker | tonyb: yes, or fixed in the role | 23:54 |
| tonyb | stevebaker: I'll look into both. The docs are easier but less correct | 23:55 |
| tonyb | stevebaker: | 23:55 |
| *** agopi|off has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!