| *** honza has joined #tripleo | 00:00 | |
| *** Slower has joined #tripleo | 00:00 | |
| *** rickflare has joined #tripleo | 00:00 | |
| *** adarazs has joined #tripleo | 00:00 | |
| *** faceman has joined #tripleo | 00:00 | |
| *** pliu has joined #tripleo | 00:00 | |
| *** beagles has joined #tripleo | 00:00 | |
| *** mburned_out has joined #tripleo | 00:00 | |
| *** jidar has joined #tripleo | 00:00 | |
| *** soc_off_ has joined #tripleo | 00:00 | |
| *** pabelanger has joined #tripleo | 00:00 | |
| *** cinerama` has joined #tripleo | 00:00 | |
| *** bugzy has joined #tripleo | 00:00 | |
| *** jaosorior has joined #tripleo | 00:00 | |
| *** funzo has joined #tripleo | 00:00 | |
| *** bandini has joined #tripleo | 00:00 | |
| *** saneax has joined #tripleo | 00:00 | |
| *** arxcruz has joined #tripleo | 00:00 | |
| *** numans has joined #tripleo | 00:00 | |
| *** jschlueter has joined #tripleo | 00:00 | |
| *** myoung has joined #tripleo | 00:00 | |
| *** yolanda has joined #tripleo | 00:00 | |
| *** saneax_-_ has joined #tripleo | 00:00 | |
| *** zshi has joined #tripleo | 00:00 | |
| *** zzzeek has joined #tripleo | 00:00 | |
| *** openstackgerrit has joined #tripleo | 00:00 | |
| *** dmsimard|off has joined #tripleo | 00:00 | |
| *** rmcadams has joined #tripleo | 00:00 | |
| *** owalsh has joined #tripleo | 00:00 | |
| *** jkilpatr has joined #tripleo | 00:00 | |
| *** cmyster has joined #tripleo | 00:00 | |
| *** tzumainn has joined #tripleo | 00:00 | |
| *** therve has joined #tripleo | 00:00 | |
| *** afazekas has joined #tripleo | 00:00 | |
| *** bfournie has joined #tripleo | 00:00 | |
| *** karthiks has joined #tripleo | 00:00 | |
| *** itlinux has joined #tripleo | 00:00 | |
| *** dhill_ has joined #tripleo | 00:00 | |
| *** d0ugal has joined #tripleo | 00:00 | |
| *** mmethot_ has joined #tripleo | 00:00 | |
| *** morazi has joined #tripleo | 00:00 | |
| *** sshnaidm|off has joined #tripleo | 00:00 | |
| *** dtrainor has joined #tripleo | 00:00 | |
| *** yamahata has joined #tripleo | 00:00 | |
| *** dsneddon has joined #tripleo | 00:00 | |
| *** rbowen has joined #tripleo | 00:00 | |
| *** chlong_ has joined #tripleo | 00:00 | |
| *** chem has joined #tripleo | 00:00 | |
| *** jcoufal has joined #tripleo | 00:00 | |
| *** dsavineau has joined #tripleo | 00:00 | |
| *** slagle has joined #tripleo | 00:00 | |
| *** rbrady has joined #tripleo | 00:00 | |
| *** amoralej|off has joined #tripleo | 00:00 | |
| *** dalvarez has joined #tripleo | 00:00 | |
| *** flaper87 has joined #tripleo | 00:00 | |
| *** rwsu has joined #tripleo | 00:00 | |
| *** bkopilov has joined #tripleo | 00:00 | |
| *** stee_3_ has joined #tripleo | 00:00 | |
| *** artom has joined #tripleo | 00:00 | |
| *** panda has joined #tripleo | 00:00 | |
| *** jrist has joined #tripleo | 00:00 | |
| *** mandre has joined #tripleo | 00:00 | |
| *** ipsecguy_ has joined #tripleo | 00:00 | |
| *** spredzy has joined #tripleo | 00:00 | |
| *** social has joined #tripleo | 00:00 | |
| *** dr_gogeta86 has joined #tripleo | 00:00 | |
| *** SlickNik has joined #tripleo | 00:00 | |
| *** lvdombrkr has joined #tripleo | 00:00 | |
| *** kbyrne has joined #tripleo | 00:00 | |
| *** dobson has joined #tripleo | 00:00 | |
| *** vkhanna has joined #tripleo | 00:00 | |
| *** apetrich has joined #tripleo | 00:00 | |
| *** bswartz has joined #tripleo | 00:00 | |
| *** oanson has joined #tripleo | 00:00 | |
| *** mwhahaha has joined #tripleo | 00:00 | |
| *** hamzy has joined #tripleo | 00:00 | |
| *** jistr|off has joined #tripleo | 00:00 | |
| *** lifeless_ has joined #tripleo | 00:00 | |
| *** portdirect has joined #tripleo | 00:00 | |
| *** hewbrocca_afk has joined #tripleo | 00:00 | |
| *** markmc has joined #tripleo | 00:00 | |
| *** mjblack has joined #tripleo | 00:00 | |
| *** sdoran has joined #tripleo | 00:00 | |
| *** gregwork has joined #tripleo | 00:00 | |
| *** lhinds has joined #tripleo | 00:00 | |
| *** ianw has joined #tripleo | 00:00 | |
| *** leseb has joined #tripleo | 00:00 | |
| *** michapma has joined #tripleo | 00:00 | |
| *** ggillies has joined #tripleo | 00:00 | |
| *** tvignaud has joined #tripleo | 00:00 | |
| *** colonwq has joined #tripleo | 00:00 | |
| *** stevebaker has joined #tripleo | 00:00 | |
| *** rodrigods has joined #tripleo | 00:00 | |
| *** tepper.freenode.net sets mode: +o mwhahaha | 00:00 | |
| *** vpickard has joined #tripleo | 00:00 | |
| *** remix_tj has joined #tripleo | 00:00 | |
| *** weshay has joined #tripleo | 00:00 | |
| *** sdake has joined #tripleo | 00:00 | |
| *** isq_ has joined #tripleo | 00:00 | |
| *** mrunge has joined #tripleo | 00:00 | |
| *** japestinho has joined #tripleo | 00:00 | |
| *** jwb has joined #tripleo | 00:00 | |
| *** bcafarel has joined #tripleo | 00:00 | |
| *** Tyrantelf has joined #tripleo | 00:00 | |
| *** number80 has joined #tripleo | 00:00 | |
| *** fpan has joined #tripleo | 00:00 | |
| *** toure has joined #tripleo | 00:00 | |
| *** lucas-afk has joined #tripleo | 00:00 | |
| *** sai has joined #tripleo | 00:00 | |
| *** tristanC has joined #tripleo | 00:00 | |
| *** shadower has joined #tripleo | 00:00 | |
| *** Lokesh_Jain__ has joined #tripleo | 00:00 | |
| *** zaneb has joined #tripleo | 00:00 | |
| *** andreaf has joined #tripleo | 00:00 | |
| *** sseago has joined #tripleo | 00:00 | |
| *** greghaynes has joined #tripleo | 00:00 | |
| *** ansiwen has joined #tripleo | 00:00 | |
| *** assassin has joined #tripleo | 00:00 | |
| *** tdasilva has joined #tripleo | 00:00 | |
| *** patrickeast has joined #tripleo | 00:00 | |
| *** mgkwill has joined #tripleo | 00:00 | |
| *** tbarron has joined #tripleo | 00:00 | |
| *** dtantsur|afk has joined #tripleo | 00:00 | |
| *** vkmc has joined #tripleo | 00:00 | |
| *** melwitt has joined #tripleo | 00:00 | |
| *** hrybacki has joined #tripleo | 00:00 | |
| *** v1k0d3n has joined #tripleo | 00:00 | |
| *** tonyb has joined #tripleo | 00:00 | |
| *** zoli has joined #tripleo | 00:00 | |
| *** gchamoul has joined #tripleo | 00:00 | |
| *** lyarwood has joined #tripleo | 00:00 | |
| *** mhenkel has joined #tripleo | 00:00 | |
| *** timothyb89 has joined #tripleo | 00:00 | |
| *** thrash|g0ne has joined #tripleo | 00:00 | |
| *** hexo_ has joined #tripleo | 00:00 | |
| *** larsks has joined #tripleo | 00:00 | |
| *** mgagne has joined #tripleo | 00:00 | |
| *** StevenK has joined #tripleo | 00:00 | |
| *** EmilienM has joined #tripleo | 00:00 | |
| *** migi has joined #tripleo | 00:00 | |
| *** akrzos has joined #tripleo | 00:00 | |
| *** tepper.freenode.net sets mode: +o EmilienM | 00:00 | |
| *** fungi has joined #tripleo | 00:00 | |
| *** rajinir has joined #tripleo | 00:00 | |
| *** leifmadsen has joined #tripleo | 00:00 | |
| *** zul has joined #tripleo | 00:00 | |
| *** percevalbot` has joined #tripleo | 00:00 | |
| *** fabbione has joined #tripleo | 00:00 | |
| *** Hazelesque has joined #tripleo | 00:00 | |
| *** CaptTofu has joined #tripleo | 00:00 | |
| *** NobodyCam has joined #tripleo | 00:00 | |
| *** alanmeadows has joined #tripleo | 00:00 | |
| *** Ng has joined #tripleo | 00:00 | |
| *** tepper.freenode.net sets mode: +v Ng | 00:00 | |
| *** eck` has joined #tripleo | 00:00 | |
| *** radez has joined #tripleo | 00:00 | |
| *** ChanServ has joined #tripleo | 00:00 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 00:00 | |
| *** rook has joined #tripleo | 00:00 | |
| *** rook has quit IRC | 00:00 | |
| *** rook has joined #tripleo | 00:00 | |
| *** cmyster is now known as Guest61123 | 00:03 | |
| *** rook is now known as Guest52189 | 00:03 | |
| *** paramite has joined #tripleo | 00:04 | |
| *** trown has joined #tripleo | 00:04 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| *** catintheroof has joined #tripleo | 00:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add netbase to ensure /etc/protocols is placed for debian https://review.openstack.org/490656 | 00:37 |
| *** ecerquei has joined #tripleo | 00:44 | |
| *** itlinux has quit IRC | 00:47 | |
| *** limao has joined #tripleo | 00:47 | |
| *** catintheroof has quit IRC | 00:48 | |
| *** mmethot_ has quit IRC | 00:52 | |
| *** mmethot_ has joined #tripleo | 00:52 | |
| *** itlinux has joined #tripleo | 00:53 | |
| *** cshastri has joined #tripleo | 00:55 | |
| *** dixiaoli has joined #tripleo | 00:58 | |
| *** ecerquei_ has joined #tripleo | 00:58 | |
| *** ecerquei has quit IRC | 01:01 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| *** zzzeek has quit IRC | 01:14 | |
| *** zzzeek has joined #tripleo | 01:15 | |
| *** bfournie has quit IRC | 01:19 | |
| *** eck` is now known as eck`gone | 01:21 | |
| *** rwsu has quit IRC | 01:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 01:28 |
| *** dhill_ has quit IRC | 01:38 | |
| *** limao has quit IRC | 01:39 | |
| *** limao has joined #tripleo | 01:39 | |
| *** tzumainn has quit IRC | 01:45 | |
| *** zshi_laptop has joined #tripleo | 01:45 | |
| weshay | EmilienM, FYI.. https://review.openstack.org/#/c/493715/ I still have to update toci | 01:48 |
| *** yamahata has quit IRC | 02:00 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 02:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| *** chlong_ has quit IRC | 02:12 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Set the AFS mirrors in the upstream env config for oooq https://review.openstack.org/493726 | 02:14 |
| *** limao has quit IRC | 02:15 | |
| *** ecerquei_ has quit IRC | 02:16 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493728 | 02:17 |
| *** limao has joined #tripleo | 02:19 | |
| *** kbyrne has quit IRC | 02:20 | |
| EmilienM | weshay: ok thx | 02:33 |
| weshay | EmilienM, I have a typo | 02:33 |
| weshay | fixing | 02:33 |
| EmilienM | stevebaker: I'm back online, so where are we? | 02:34 |
| EmilienM | stevebaker: nevermind, I saw weshay's comments | 02:36 |
| stevebaker | EmilienM: yep, just found those | 02:36 |
| EmilienM | weshay: thx for the patches, lgtm | 02:41 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493728 | 02:42 |
| EmilienM | weshay: http://logs.openstack.org/28/493728/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/2d85677/console.html#_2017-08-15_02_26_51_269071 | 02:42 |
| EmilienM | ah you fixed it | 02:42 |
| weshay | :) | 02:42 |
| weshay | told you I had a typo.. NOW GO EAT and stuff | 02:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario004 to run Tempest https://review.openstack.org/491113 | 02:46 |
| *** itlinux has quit IRC | 02:54 | |
| *** itlinux has joined #tripleo | 03:00 | |
| EmilienM | weshay: https://review.openstack.org/#/c/491102/ ready for review | 03:01 |
| *** jcoufal has quit IRC | 03:05 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 03:10 |
| *** ramishra has joined #tripleo | 03:15 | |
| michapma | EmilienM: I think neutron-server is running, but maybe ODL isn't, so I need to write patches to grab the ODL logs in quickstart so I can see what's going on. | 03:20 |
| *** ramishra has quit IRC | 03:23 | |
| *** daidv has joined #tripleo | 03:28 | |
| EmilienM | michapma: that would be required to run ODL in tripleo gate. | 03:31 |
| EmilienM | michapma: thx for helping here | 03:31 |
| *** gkadam has joined #tripleo | 03:31 | |
| *** links has joined #tripleo | 03:34 | |
| michapma | EmilienM: sorry I haven't already done it - in the middle of moving house, it's been a bit disruptive. | 03:34 |
| EmilienM | michapma: I bet! good luck | 03:36 |
| *** psachin has joined #tripleo | 03:37 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/461000 | 03:38 |
| EmilienM | stevebaker: would you have time to help on enabling pingtest in multinode container jobs? | 03:39 |
| EmilienM | stevebaker: I've been working on it for some time but need help | 03:39 |
| weshay | my patch did not work | 03:40 |
| EmilienM | weshay: damn :( | 03:40 |
| weshay | it was skipped | 03:40 |
| *** gbarros has joined #tripleo | 03:41 | |
| weshay | EmilienM, oh wait.. wrong job :) | 03:42 |
| * weshay looks again | 03:42 | |
| weshay | http://logs.openstack.org/28/493728/2/check/gate-tripleo-ci-centos-7-containers-multinode/d6ef8fa/logs/undercloud/etc/docker/daemon.json.txt.gz | 03:43 |
| *** psahoo has joined #tripleo | 03:44 | |
| weshay | FAILED! => {"changed": false, "failed": true, "msg": "Unable to restart service docker: Failed to restart docker.service: Method call timed out\nSee system logs and 'systemctl status docker.service' for details.\n"} | 03:44 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493728 | 03:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/container: run Barbican non-containerized https://review.openstack.org/493734 | 03:46 |
| stevebaker | EmilienM: sure, can you point me at changes? | 03:48 |
| EmilienM | stevebaker: I think scenario002 and 003 are good. 001 and 004 with ceph, not so good | 03:48 |
| EmilienM | I don't understand why glance can't upload an image when glance is containerized and rbd is used as a backend (ceph not containerized yet in CI) | 03:49 |
| EmilienM | that thing ^ needs to be figured out | 03:49 |
| EmilienM | logs: http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-12_13_29_17 | 03:49 |
| stevebaker | EmilienM: ok, I'll take a look | 03:51 |
| EmilienM | thanks | 03:51 |
| EmilienM | i'm filling a bug | 03:51 |
| EmilienM | stevebaker: https://bugs.launchpad.net/tripleo/+bug/1710773 | 03:54 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 03:54 |
| EmilienM | stevebaker: if you want more details | 03:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on scenarios container jobs https://review.openstack.org/490129 | 03:55 |
| EmilienM | weshay: ^ scenario002 + 003 ok with pingtest when containerized - 001 / 004 WIP, see bug report | 03:55 |
| stevebaker | EmilienM: ok, thanks | 03:55 |
| EmilienM | weshay: once this works, we'll switch to tempest, but taking baby steps here | 03:55 |
| *** mmethot_ has quit IRC | 04:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario002 to run Tempest https://review.openstack.org/491102 | 04:02 |
| * EmilienM out | 04:03 | |
| *** rickflare has quit IRC | 04:04 | |
| *** d0ugal has quit IRC | 04:06 | |
| *** gbarros has quit IRC | 04:07 | |
| *** gbarros has joined #tripleo | 04:10 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 04:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 04:10 |
| *** gbarros has quit IRC | 04:16 | |
| *** gbarros has joined #tripleo | 04:17 | |
| *** rcernin has joined #tripleo | 04:20 | |
| *** jaganathan has joined #tripleo | 04:20 | |
| *** d0ugal has joined #tripleo | 04:20 | |
| *** limao has quit IRC | 04:33 | |
| *** rcernin has quit IRC | 04:52 | |
| *** paramite has quit IRC | 04:58 | |
| *** limao has joined #tripleo | 05:03 | |
| *** karthiks has quit IRC | 05:04 | |
| *** karthiks has joined #tripleo | 05:05 | |
| *** gbarros has quit IRC | 05:05 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 05:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 05:10 |
| *** zshi_laptop has quit IRC | 05:11 | |
| *** yprokule has joined #tripleo | 05:32 | |
| *** mdnadeem has joined #tripleo | 05:44 | |
| *** nyechiel has joined #tripleo | 05:47 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 05:48 |
| *** marios has joined #tripleo | 05:48 | |
| *** leifmadsen has quit IRC | 05:57 | |
| *** leifmadsen has joined #tripleo | 05:58 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 06:00 |
| *** ramishra has joined #tripleo | 06:00 | |
| *** brault has joined #tripleo | 06:00 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 06:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 06:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 06:10 |
| *** agurenko has joined #tripleo | 06:10 | |
| *** ramishra has quit IRC | 06:11 | |
| *** ramishra has joined #tripleo | 06:14 | |
| *** iranzo has joined #tripleo | 06:14 | |
| *** jtomasek has joined #tripleo | 06:14 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo stable/ocata: Always start httpd at the same time https://review.openstack.org/493754 | 06:17 |
| jaosorior | bandini, marios: If you have time could you check this out https://review.openstack.org/#/c/492878/ ? | 06:27 |
| bandini | jaosorior: lgtm | 06:31 |
| marios | jaosorior: ack ... left a comment, not sure about the duplicate declarations? | 06:33 |
| jaosorior | marios: oh, it's not a duplicate declaration. It's just the funky docker bind-mount syntax: <origin file name>:<destination file name>:'ro' | 06:35 |
| jaosorior | marios: in this case, the origin and destination are the same path | 06:35 |
| marios | jaosorior: ah ok thanks :) i saw you had it like that in two places so wasn't sure , revoting then | 06:36 |
| jaosorior | marios: yeah, it's not very intuitive :/ | 06:38 |
| jaosorior | marios, bandini: Would sure use a review here too https://review.openstack.org/#/c/489593/ :D | 06:38 |
| *** nyechiel has quit IRC | 06:39 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Containarise Barbican API https://review.openstack.org/481451 | 06:40 |
| marios | jaosorior: ack | 06:45 |
| *** mrch has joined #tripleo | 06:50 | |
| openstackgerrit | Michael Henkel proposed openstack/os-net-config master: This patch adds initial support for the Contrai vRouter interface https://review.openstack.org/492492 | 06:50 |
| *** jlinkes has joined #tripleo | 06:50 | |
| jaosorior | marios: replied | 06:54 |
| *** jprovazn has joined #tripleo | 06:56 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Containarise Barbican API https://review.openstack.org/481451 | 06:58 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Use rabbitmq ipv6 flag https://review.openstack.org/475457 | 06:59 |
| marios | jaosorior: thanks revoted :) | 06:59 |
| openstackgerrit | Merged openstack/tripleo-common master: Add GUI logging workflows https://review.openstack.org/469196 | 06:59 |
| jaosorior | marios: damn, now that I think about it. We really need to document all the stuff that is autogenerated by t-h-t :/ | 07:00 |
| openstackgerrit | Merged openstack/tripleo-common master: Remove support for py34 https://review.openstack.org/471574 | 07:00 |
| jaosorior | it's getting really developer-unfriendly | 07:00 |
| *** paramite has joined #tripleo | 07:01 | |
| bandini | yeah | 07:01 |
| marios | jaosorior: yeah, but its not new i mean we've had that kind of stuff for ages (like setting some hiera for all services in a yaql query etc) | 07:01 |
| marios | jaosorior: i just hadn't come across this one before. | 07:01 |
| *** rcernin has joined #tripleo | 07:01 | |
| jaosorior | marios: right. But it's not something you should "know", we should have a reference with all the "magic" hiera keys | 07:02 |
| marios | jaosorior: also, not sure how/where we'd document it its tricky. we could literally have a document we update for all hiera values and outputs from the stack or something | 07:02 |
| marios | jaosorior: yeah | 07:02 |
| jaosorior | at least the autogenerated ones | 07:02 |
| jaosorior | it's the least we can do as a platform | 07:02 |
| *** pcaruana has joined #tripleo | 07:06 | |
| *** dsariel has joined #tripleo | 07:08 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 07:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 07:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 07:10 |
| *** shardy has joined #tripleo | 07:13 | |
| *** florianf has joined #tripleo | 07:17 | |
| *** agurenko has quit IRC | 07:17 | |
| honza | jtomasek: florianf: would you mind prioritizing reviewing this patch today? https://review.openstack.org/#/c/473933/ | 07:18 |
| florianf | honza: sure | 07:18 |
| honza | florianf: thanks! | 07:18 |
| *** brault has quit IRC | 07:19 | |
| *** remix_tj has quit IRC | 07:21 | |
| apetrich | so honza for the https://review.openstack.org/#/c/469608/ we have a totally different issue | 07:23 |
| honza | apetrich: should i be worried? | 07:23 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert network templates to be rendered via j2 https://review.openstack.org/492218 | 07:24 |
| apetrich | honza, don't know yet, I'm still looking at it, | 07:25 |
| *** shardy has quit IRC | 07:25 | |
| *** agurenko has joined #tripleo | 07:25 | |
| *** tesseract has joined #tripleo | 07:27 | |
| *** brault has joined #tripleo | 07:31 | |
| *** iranzo has quit IRC | 07:31 | |
| jtomasek | honza: sure | 07:33 |
| *** jpich has joined #tripleo | 07:35 | |
| *** egonzalez has joined #tripleo | 07:40 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 07:41 |
| *** mcornea has joined #tripleo | 07:44 | |
| *** brault has quit IRC | 07:46 | |
| *** nyechiel has joined #tripleo | 07:56 | |
| *** aufi has joined #tripleo | 07:59 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 08:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 08:10 |
| *** shardy has joined #tripleo | 08:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add node list to get_edges() https://review.openstack.org/493781 | 08:13 |
| *** athomas has joined #tripleo | 08:13 | |
| *** oidgar has joined #tripleo | 08:16 | |
| *** dixiaoli_ has joined #tripleo | 08:18 | |
| *** tdasilva has quit IRC | 08:18 | |
| *** dixiaoli has quit IRC | 08:18 | |
| *** ccamacho has joined #tripleo | 08:20 | |
| *** ramishra has quit IRC | 08:22 | |
| *** snecklifter has joined #tripleo | 08:24 | |
| jtomasek | honza: I am getting this as a logsUrl: http://paste.openstack.org/show/618376/ | 08:26 |
| *** lucas-afk is now known as lucasagomes | 08:26 | |
| honza | jtomasek: :( | 08:26 |
| honza | jtomasek: I'll investigate | 08:26 |
| *** milan has joined #tripleo | 08:26 | |
| jtomasek | honza: I don't have the logging container which causes the problem, but I think the message should have an error state I guess | 08:27 |
| honza | exactly | 08:27 |
| honza | jtomasek: we should also be running the queue draining workflow when you request to download logs but it might be too late to get that in | 08:28 |
| jtomasek | honza: yeah, because otherwise it will provide only old logs, right? | 08:28 |
| honza | jtomasek: or none :) | 08:28 |
| honza | jtomasek: it's a small patch, maybe it'll land :) | 08:29 |
| jtomasek | honza: I think thats a bug we need to fix, so it should not be a problem to land it | 08:29 |
| honza | cool, i'm on it | 08:29 |
| *** ipsecguy_ has quit IRC | 08:36 | |
| *** dsariel has quit IRC | 08:36 | |
| *** ipsecguy has joined #tripleo | 08:36 | |
| *** iranzo has joined #tripleo | 08:40 | |
| honza | jtomasek: interesting, I actually get a proper error when requesting logs when the container doesn't exist yet | 08:45 |
| *** nyechiel has quit IRC | 08:45 | |
| *** brault has joined #tripleo | 08:46 | |
| *** limao has quit IRC | 08:50 | |
| shardy | jaosorior: Hey, good morning, pls could you revisit https://review.openstack.org/#/c/486260/ when you have a moment? | 08:51 |
| shardy | passed CI inc experimental jobs so I think it's good to land now? | 08:51 |
| *** jaganathan has quit IRC | 08:51 | |
| *** limao has joined #tripleo | 08:52 | |
| *** brault has quit IRC | 08:54 | |
| *** brault has joined #tripleo | 08:54 | |
| *** lifeless_ is now known as lifeless | 08:56 | |
| jaosorior | shardy: will do | 08:56 |
| shardy | jaosorior: thanks! | 08:57 |
| jaosorior | shardy: can you check this one out https://review.openstack.org/#/c/491832/ ? | 08:58 |
| shardy | jaosorior: ack looking | 08:58 |
| jpich | honza: Hi! Do you know what's the story with https://review.openstack.org/#/c/473542/ ? Looks like it got kinda wedged? I'm not sure of the current status of things, if it's fine to recheck | 08:59 |
| *** brault has quit IRC | 08:59 | |
| honza | jpich: I think it should be fine to recheck. | 09:00 |
| *** tosky has joined #tripleo | 09:00 | |
| honza | jpich: not sure why jenkins hasn't merged it yet :( | 09:00 |
| shardy | I tried re-approving it, looks like it somehow got dropped from the gate queue | 09:01 |
| jpich | honza: There probably was a reboot of some sort midway and it got lost | 09:01 |
| honza | +1 | 09:01 |
| jpich | Thanks! | 09:01 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Configure the set of tests to execute in scenario001 https://review.openstack.org/490376 | 09:06 |
| *** akrivoka has joined #tripleo | 09:08 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 09:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 09:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 09:10 |
| *** salmankhan has joined #tripleo | 09:13 | |
| *** dougbtv__ has quit IRC | 09:15 | |
| *** limao has quit IRC | 09:23 | |
| *** shardy has quit IRC | 09:24 | |
| *** shardy has joined #tripleo | 09:26 | |
| lvdombrkr | hello guys, after openstack overcloud node import instackenv.json i have error : ('Connection aborted.', BadStatusLine("''",)) | 09:26 |
| lvdombrkr | who have any ideas? | 09:26 |
| lvdombrkr | ? | 09:33 |
| *** dsneddon has quit IRC | 09:33 | |
| *** assassin has quit IRC | 09:33 | |
| *** iranzo has quit IRC | 09:36 | |
| *** cylopez has joined #tripleo | 09:36 | |
| *** cshastri has quit IRC | 09:45 | |
| *** iranzo has joined #tripleo | 09:46 | |
| *** dixiaoli_ has quit IRC | 09:49 | |
| lvdombrkr | who have any ideas? | 09:54 |
| *** shardy has quit IRC | 09:54 | |
| *** shardy has joined #tripleo | 09:54 | |
| *** nyechiel has joined #tripleo | 09:57 | |
| jtomasek | honza: publish_ui_logs_to_swift workflow creates the container? | 10:01 |
| honza | jtomasek: yes | 10:01 |
| honza | jtomasek: that workflow will be called when requesting logs once this patch i'm working now lands | 10:01 |
| jtomasek | honza: I may not even have the zaqar logging enabled etc. but in any case, once you create the patch ^ it should get fixed | 10:02 |
| *** ramishra has joined #tripleo | 10:06 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-common master: Publish logs before exporting them https://review.openstack.org/493819 | 10:07 |
| honza | jtomasek: ^^^ | 10:08 |
| * jtomasek pulls it | 10:08 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 10:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 10:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 10:10 |
| *** dtantsur|afk is now known as dtantsur | 10:15 | |
| lvdombrkr | hello guys, after openstack overcloud node import instackenv.json i have error : ('Connection aborted.', BadStatusLine("''",)) | 10:15 |
| *** psachin_ has joined #tripleo | 10:18 | |
| *** psachin has quit IRC | 10:18 | |
| *** janki has joined #tripleo | 10:20 | |
| jtomasek | honza: that worked, one more test... | 10:20 |
| jtomasek | honza: are there any wireframes for logs download functionality in UI? | 10:21 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP - docker: Stop all ceiloemeter services during compute upgrade https://review.openstack.org/493825 | 10:23 |
| janki | jaosorior, hey | 10:23 |
| *** psachin_ has quit IRC | 10:26 | |
| shardy | jaosorior: Hey https://review.openstack.org/#/c/492218/ passed CI when you have a moment to revisit | 10:31 |
| jtomasek | honza: the UI logs are not much fun to read though, I think it would be much better if we did not log any actions and application state, just the errors and stack traces | 10:31 |
| shardy | gate queue looks pretty long again today tho :( | 10:32 |
| jtomasek | honza: also I just enabled zaqar logging, refreshed the browser, downloaded logs and ended up with the log already log-rotated once | 10:32 |
| *** jkilpatr has quit IRC | 10:34 | |
| lvdombrkr | hello guys, after openstack overcloud node import instackenv.json i have error : ('Connection aborted.', BadStatusLine("''",)) | 10:37 |
| *** hewbrocca_afk is now known as hewbrocca | 10:38 | |
| jprovazn | hi, anyone else hit an issue with uploading cirros image in CI - http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/be30874/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-08-15_08_58_47 ? | 10:40 |
| *** psachin_ has joined #tripleo | 10:40 | |
| jprovazn | based on CI status tese 001 jobs are usually green, but I hit this one twice recently (and I think it's not related to my patch) | 10:41 |
| *** assassin has joined #tripleo | 10:43 | |
| *** assassin has joined #tripleo | 10:43 | |
| *** jkilpatr has joined #tripleo | 10:52 | |
| *** brault has joined #tripleo | 10:56 | |
| *** dobson has quit IRC | 10:56 | |
| *** brault has quit IRC | 11:00 | |
| d0ugal | honza: https://review.openstack.org/#/c/469608/6 | 11:08 |
| d0ugal | honza: I just commented, found the reason for error when deleting. | 11:09 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 11:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 11:10 |
| apetrich | honza, also I will add a change to delete the cron_trigger to your patch | 11:10 |
| lvdombrkr | hello guys, after openstack overcloud node import instackenv.json i have error : ('Connection aborted.', BadStatusLine("''",)) | 11:10 |
| d0ugal | lvdombrkr: I feel like that might be related to Zaqar (or just the websocket connection failing for another reason) | 11:15 |
| d0ugal | lvdombrkr: the operation may have still completed - you just might not have gotten the messages confirming it. | 11:15 |
| *** iranzo has quit IRC | 11:16 | |
| lvdombrkr | d0ugal: thank you but unfortinetly task not complited | 11:19 |
| *** dobson has joined #tripleo | 11:20 | |
| openstackgerrit | Adriano Petrich proposed openstack/instack-undercloud master: Add an hourly cron trigger for tripleo-ui logging https://review.openstack.org/469608 | 11:20 |
| jprovazn | one containerization-related question: when trying https://review.openstack.org/#/c/482680/ all goes fine except the container fails to start with error "Device or resource busy: '/etc/hostname'" - http://paste.openstack.org/show/618394/ | 11:20 |
| jprovazn | it might be similar to this one https://bugs.launchpad.net/tripleo/+bug/1709689 | 11:21 |
| openstack | Launchpad bug 1709689 in tripleo "kolla rabbitmq container setup should not try to delete /etc/hosts" [Medium,Triaged] - Assigned to Jiří Stránský (jistr) | 11:21 |
| lvdombrkr | d0ugal: i checked Zaqar its up and running without issues | 11:23 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 11:23 |
| d0ugal | lvdombrkr: Maybe check mistral for errors then. | 11:23 |
| *** tosky has quit IRC | 11:30 | |
| lvdombrkr | d0ugal: its can be also server-server connection issue? | 11:30 |
| *** morazi has quit IRC | 11:32 | |
| jaosorior | janki: hey | 11:33 |
| *** ansmith has joined #tripleo | 11:35 | |
| janki | jaosorior, thanks for looking into the patch :). I saw you deleted OpenDayLight related images. I believe they are needed | 11:39 |
| jaosorior | janki: where did I delete them? | 11:41 |
| jaosorior | janki: oh, maybe it was a rebase issue | 11:41 |
| *** tosky has joined #tripleo | 11:41 | |
| *** lucasagomes is now known as lucas-hungry | 11:41 | |
| lvdombrkr | d0ugal: i think i found issue, there was point to point connection between servers and one of the servers not supported MDI | 11:41 |
| janki | jaosorior, https://review.openstack.org/#/c/481451/13..15/environments/docker-centos-tripleoupstream.yaml | 11:41 |
| janki | jaosorior, however, if you compare with Base, those images are deleted. So I guess the patch won't cause any issue | 11:42 |
| jaosorior | janki: I think those were deleted in another patch. | 11:43 |
| jaosorior | not sure where though | 11:43 |
| *** thrash|g0ne is now known as thrash | 11:43 | |
| janki | jaosorior, ack. thanks :) | 11:44 |
| *** pkovar has joined #tripleo | 11:45 | |
| jaosorior | janki: actually, it was me that deleted those references | 11:47 |
| jaosorior | janki: I merely ran the command that's referenced in the comment of that file | 11:48 |
| jaosorior | janki: so it might be that those images are missing from the tripleo-common list | 11:48 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: docker: Stop all ceilometer services during compute upgrade https://review.openstack.org/493825 | 11:48 |
| janki | jaosorior, those are needed. ODL is getting containarised this release (i work in that dfg so i know) | 11:48 |
| janki | jaosorior, let me check | 11:48 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Also write an upgrade_tasks_playbook https://review.openstack.org/490848 | 11:48 |
| *** dougbtv has joined #tripleo | 11:48 | |
| *** dobson has quit IRC | 11:50 | |
| *** dobson has joined #tripleo | 11:50 | |
| *** kambiz has quit IRC | 11:51 | |
| * honza reads | 11:52 | |
| honza | jtomasek: yes, it seems like the logging generates ridiculous amounts of data | 11:53 |
| *** kambiz has joined #tripleo | 11:54 | |
| honza | d0ugal: apetrich: nice, thanks for looking into it | 11:56 |
| janki | jaosorior, it was deleted in this commit https://github.com/openstack/tripleo-heat-templates/commit/f24d5d4c0237d2703cf2744aa6db65865401e94e i am not sure why | 11:56 |
| *** eck`gone is now known as eck` | 11:57 | |
| *** abishop has joined #tripleo | 11:57 | |
| *** dprince has joined #tripleo | 12:00 | |
| jaosorior | janki: right, like I mentioned, I merely ran the command listed in the file | 12:01 |
| janki | jaosorior, thanks again for looking into it :) | 12:02 |
| jaosorior | janki: this https://github.com/openstack/tripleo-heat-templates/blob/master/environments/docker-centos-tripleoupstream.yaml#L3 | 12:02 |
| *** bfournie has joined #tripleo | 12:03 | |
| jaosorior | janki: yep, they're missing from tripleo-common | 12:04 |
| janki | jaosorior, I will send out a patch in some time | 12:04 |
| jaosorior | ok | 12:04 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Fix OVB pike job https://review.openstack.org/493848 | 12:05 |
| jaosorior | janki: by the way, the barbican-api patch is failing cause of a puppet issue. Haven't figured out why yet. | 12:05 |
| janki | jaosorior, how did you figure that out. I mean which log files to check, how to know which file to check | 12:05 |
| janki | jaosorior, all I see is "previous command failed" error | 12:06 |
| jaosorior | janki: ok, lets dig into it. so lets go to the logs of the job that failed (scenario002 that deploys containers) | 12:07 |
| jaosorior | janki: http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/ | 12:07 |
| jaosorior | first we go to console.html http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/console.html | 12:07 |
| jaosorior | if we go near the bottom of the file, we'll see in which ansible step it failed | 12:07 |
| jaosorior | here http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/console.html#_2017-08-15_08_28_00_491920 | 12:08 |
| jaosorior | which means that the overcloud deployment failed | 12:08 |
| *** bfournie has quit IRC | 12:08 | |
| jaosorior | so alright, lets go to the undercloud logs, which, in the home directory, we can find a bunch of logs, one of them being the deployment logs http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/logs/undercloud/home/jenkins/ | 12:08 |
| jaosorior | the deployment logs are called overcloud_deploy.log.txt.gz | 12:08 |
| jaosorior | so lets check that one out | 12:08 |
| jaosorior | http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 12:09 |
| jaosorior | going near the bottom of that file we can see that it failed in the puppet steps http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-08-15_08_27_12 | 12:09 |
| jaosorior | unfortunately it doesn't show the full log (maybe it's a bug? ) | 12:09 |
| jaosorior | but we can actually see the full log in the post-ci logs | 12:09 |
| jaosorior | and we can find those in /var/log/ in the undercloud | 12:10 |
| jaosorior | which is this dir http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/logs/undercloud/var/log/ | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 12:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 12:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 12:10 |
| jaosorior | and the file is postci.txt.gz | 12:10 |
| jaosorior | this one | 12:10 |
| jaosorior | http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/logs/undercloud/var/log/ | 12:10 |
| jaosorior | I mean, this http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/logs/undercloud/var/log/postci.txt.gz | 12:10 |
| jaosorior | finally | 12:10 |
| jaosorior | in the end of postci.txt we can see the puppet error | 12:10 |
| jaosorior | "Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: File[/etc/my.cnf.d] is already declared in file /etc/puppet/modules/tripleo/manifests/profile/base/database/mysql/client.pp:89; cannot redeclare at /etc/puppet/modules/mysql/manifests/server/config.pp:44 at | 12:11 |
| jaosorior | /etc/puppet/modules/mysql/manifests/server/config.pp:44:7 at /etc/puppet/modules/barbican/manifests/db/mysql.pp:60 on node centos-7-2-node-rax-ord-10441918-806258.localdomain", | 12:11 |
| jaosorior | "2017-08-15 08:26:53,208 INFO: 10299 -- Finished processing puppet configs", | 12:11 |
| jaosorior | seems to be a conflict with the barbican puppet manifest and the mysql client manifest | 12:11 |
| jaosorior | but I'm not entirely sure why we have a conflict and other services don't have this issue. | 12:12 |
| jaosorior | since most services seem to include the mysql client in the step_config | 12:12 |
| janki | jaosorior, ahhh! Now I know. thanks a ton | 12:13 |
| jaosorior | janki: no biggie | 12:13 |
| janki | jaosorior, also in this file http://logs.openstack.org/51/481451/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/a9f14c3/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 12:13 |
| janki | there is an error saying no nodes found. serach for "{u'errors': [u'Not enough baremetal nodes" | 12:14 |
| *** pchavva has joined #tripleo | 12:14 | |
| janki | is this relevant? | 12:15 |
| janki | but I guess, it then updates the plan and redeploys -> Removing the current plan files | 12:15 |
| janki | 2017-08-15 08:12:19 | Uploading new plan files | 12:15 |
| jaosorior | janki: that's strange, that shouldn't have happened. | 12:15 |
| jaosorior | janki: if it got to the puppet bits, it means that it did actually get a node | 12:15 |
| janki | jaosorior, so whats the next step now? removing mysqlcleint.yaml will fail as well | 12:16 |
| jaosorior | that's correct | 12:17 |
| jaosorior | I'm not sure | 12:17 |
| jaosorior | we gotta figure out what do other services do | 12:17 |
| jaosorior | and why barbican fails and other services don't | 12:17 |
| *** rhallisey has joined #tripleo | 12:18 | |
| jtomasek | ping dprince | 12:19 |
| dprince | jtomasek: hi | 12:19 |
| jtomasek | dprince: hi, I've been looking into containerized deployment through GUI | 12:20 |
| jtomasek | dprince: and I stumbled across a problem of populating docker images | 12:21 |
| *** oidgar has quit IRC | 12:21 | |
| *** sshnaidm|off is now known as sshnaidm | 12:21 | |
| *** rlandy has joined #tripleo | 12:21 | |
| jtomasek | dprince: http://tripleo.org/install/containers_deployment/overcloud.html#preparing-the-environment | 12:21 |
| jtomasek | dprince: this functionality is not available through any mistral workflow/action, so I am trying to figure out how to resolve it | 12:21 |
| dprince | jtomasek: can the UI bundle an extra environment file which specififies all the containers. This could then get setup via the default plan | 12:23 |
| jtomasek | dprince: yes, I could add docker_registry.yaml into capabilities map and let user enable it through UI, which would populate the relevant parameters | 12:25 |
| jaosorior | shardy, bandini: Could you check this out https://review.openstack.org/#/c/492963/2 ? | 12:25 |
| jtomasek | dprince: I wonder if we can consider docker registry preparation as an undercloud installation step or rather it is a part of a deployment itself | 12:25 |
| *** sbrzozow has joined #tripleo | 12:26 | |
| *** lucas-hungry is now known as lucasagomes | 12:26 | |
| dprince | jtomasek: yes, exactly. Considering this an undercloud prep step could help solve it | 12:28 |
| *** gkadam has quit IRC | 12:28 | |
| janki | jaosorior, will try to do it some time this week | 12:28 |
| jtomasek | dprince: can we add docker_registry.yaml into default plan (tht) with some reasonable defaults? | 12:28 |
| *** morazi has joined #tripleo | 12:29 | |
| shardy | jaosorior: ack, lgtm | 12:30 |
| *** psahoo has quit IRC | 12:30 | |
| *** ykarel has joined #tripleo | 12:32 | |
| dprince | jtomasek: let me see if I can find a place to inject it | 12:32 |
| jtomasek | dprince: thanks | 12:33 |
| *** ykarel_ has joined #tripleo | 12:33 | |
| jaosorior | bandini: is anything in upstream CI already deploying the HA container bits? | 12:33 |
| *** morazi has quit IRC | 12:33 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test pike job https://review.openstack.org/493857 | 12:34 |
| jtomasek | dprince: one more question, for pike is there a plan to merge the docker.yaml with overcloud-resource-registry-puppet.yaml -> make the containerized deployment by default? | 12:34 |
| dprince | jtomasek: not for upstream TripleO. | 12:34 |
| jtomasek | dprince: ok | 12:34 |
| dprince | jtomasek: but for RHOSP yes | 12:35 |
| *** morazi has joined #tripleo | 12:35 | |
| *** sbrzozow has quit IRC | 12:35 | |
| *** bfournie has joined #tripleo | 12:36 | |
| *** oidgar has joined #tripleo | 12:36 | |
| *** ykarel has quit IRC | 12:36 | |
| *** mmethot_ has joined #tripleo | 12:38 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: [WIP] Switch to scheduling based on resource classes https://review.openstack.org/490851 | 12:42 |
| sshnaidm | mandre, ping wrt https://review.openstack.org/#/c/493726 | 12:43 |
| *** jmelvin has joined #tripleo | 12:44 | |
| *** mmethot_ has quit IRC | 12:45 | |
| *** adarazs is now known as adarazs_afk | 12:45 | |
| *** jlabarre has joined #tripleo | 12:47 | |
| *** ecerquei_ has joined #tripleo | 12:47 | |
| *** sbrzozow has joined #tripleo | 12:47 | |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Add param to configure snat mechanism https://review.openstack.org/493861 | 12:48 |
| jaosorior | mandre: around? | 12:51 |
| *** dsariel has joined #tripleo | 12:52 | |
| *** catintheroof has joined #tripleo | 12:53 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Don't log app state in Zaqar by default https://review.openstack.org/493863 | 12:56 |
| honza | jtomasek: ^ this patch adds a new config option where you can toggle the app state in logs (default off) | 12:56 |
| jtomasek | honza: ack, cool! | 12:57 |
| weshay | sshnaidm, mandre so do we need a fix for lp 1710533 still? https://review.openstack.org/#/q/topic:bug/1710533 | 12:59 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] https://launchpad.net/bugs/1710533 - Assigned to wes hayutin (weshayutin) | 12:59 |
| *** fultonj has joined #tripleo | 13:00 | |
| *** tzumainn has joined #tripleo | 13:00 | |
| *** jcoufal has joined #tripleo | 13:00 | |
| *** janki has quit IRC | 13:01 | |
| sshnaidm | weshay, mandre In my patch https://review.openstack.org/#/c/491923/ I tried to fix it, but it didn't change the docker.io to local registry. I think it's wrong parameter, wanted to make clear this with mandre. | 13:01 |
| *** ecerquei_ has quit IRC | 13:02 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates stable/ocata: Fix rpms being installed via DeployArtifactURLs https://review.openstack.org/493866 | 13:02 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates stable/newton: Fix rpms being installed via DeployArtifactURLs https://review.openstack.org/493867 | 13:02 |
| *** aufi has quit IRC | 13:02 | |
| shardy | agurenko: ^^ FYI proposed backports of the RPM fix we discussed | 13:03 |
| weshay | sshnaidm, k.. using the undercloud is the best approach but we'll have to confirm the undercloud is doing the right thing | 13:03 |
| *** ecerquei has joined #tripleo | 13:03 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart-extras master: Fix set_overcloud_workers=False being ignored for >mitaka https://review.openstack.org/493586 | 13:07 |
| *** morazi has quit IRC | 13:09 | |
| *** morazi has joined #tripleo | 13:10 | |
| shardy | bandini: Hey when you get time pls can you revisit https://review.openstack.org/#/c/492218/, I had to rebase after it was approved | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 13:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 13:10 |
| slagle | anyone else notice how we no longer show events lower than AllNodesDeploySteps during the overcloud deploy? | 13:15 |
| mwhahaha | mandre: https://review.openstack.org/#/c/493726/ is your -1 still valid? | 13:16 |
| *** liverpooler has joined #tripleo | 13:18 | |
| *** aufi has joined #tripleo | 13:18 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 13:22 |
| *** jlabarre has quit IRC | 13:26 | |
| *** atoth has joined #tripleo | 13:27 | |
| *** jlabarre has joined #tripleo | 13:28 | |
| *** adarazs_afk is now known as adarazs | 13:29 | |
| *** Guest52189 is now known as rook | 13:29 | |
| shardy | slagle: Hi, what do you mean by lower than? | 13:29 |
| shardy | slagle: the event output changed a bit since we now apply a common ansible playbook for each step, but it looks OK to me locally | 13:30 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Use undercloud docker proxy https://review.openstack.org/491923 | 13:30 |
| fultonj | can anyone advise me on the state of this review? https://review.openstack.org/#/c/492303 it looks like it's ready to merge but i want to confirm it's not stuck for some reason | 13:31 |
| *** lblanchard has joined #tripleo | 13:31 | |
| shardy | fultonj: I think the gate queue is just really long atm unfortunately | 13:32 |
| shardy | check the zuul link, there's stuff in there that's been gating for 12+ hours :( | 13:32 |
| fultonj | shardy: ok, thanks. np, just wanted to make sure it wasn't waiting for someone to do something. | 13:32 |
| slagle | shardy: just wasn't seeing events from nested stacks lower than that | 13:32 |
| *** psachin_ has quit IRC | 13:32 | |
| slagle | shardy: but i checked a CI job log and I see them there, so dunno. | 13:33 |
| slagle | guess it's just me | 13:33 |
| *** gbarros has joined #tripleo | 13:34 | |
| shardy | slagle: ah, hmm, not sure either, seems OK to me but my undercloud isn't absolutely up to date | 13:36 |
| *** chlong_ has joined #tripleo | 13:38 | |
| *** aufi_ has joined #tripleo | 13:39 | |
| *** oidgar has quit IRC | 13:41 | |
| *** aufi has quit IRC | 13:42 | |
| *** paramite has quit IRC | 13:47 | |
| *** ykarel_ has quit IRC | 13:47 | |
| *** psachin_ has joined #tripleo | 13:48 | |
| mwhahaha | we need to land the docker proxy patches to fixup the gate | 13:48 |
| mwhahaha | otherwise we're going to end up with another 24+ hour queue | 13:49 |
| *** jprovazn has quit IRC | 13:52 | |
| EmilienM | hello | 13:52 |
| pabelanger | Yes, because the gate just reset again on 404 docker.io: http://logs.openstack.org/70/492970/3/gate/gate-tripleo-ci-centos-7-containers-multinode/ea159ce/logs/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz | 13:52 |
| EmilienM | mandre: ^ please review again https://review.openstack.org/#/c/493726 | 13:52 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Adds PostUpgradeConfigStepsDeployment to drive post config ansible https://review.openstack.org/493878 | 13:53 |
| *** psachin_ has quit IRC | 13:53 | |
| EmilienM | weekly meeting, here, in ~7 min, please add items on https://etherpad.openstack.org/p/tripleo-meeting-items | 13:53 |
| *** mrch has quit IRC | 13:54 | |
| EmilienM | mwhahaha: not sure yet why https://review.openstack.org/#/c/493728/ failed in container job | 13:54 |
| EmilienM | weshay: ^ if you up | 13:54 |
| mwhahaha | EmilienM: i don't know if tripleo-ci gets zuul cloned? | 13:54 |
| honza | akrivoka: yes, all of the logging patches reference 'blueprint websocket-logging' https://blueprints.launchpad.net/tripleo/+spec/websocket-logging | 13:55 |
| akrivoka | honza: awesome | 13:55 |
| *** ramishra has quit IRC | 13:55 | |
| EmilienM | mwhahaha: http://logs.openstack.org/28/493728/3/check/gate-tripleo-ci-centos-7-containers-multinode/d49033b/console.html#_2017-08-15_04_36_45_916088 | 13:55 |
| mwhahaha | EmilienM: it failed on restarting docker, odd | 13:55 |
| EmilienM | yeah, probably related to weshay's patch | 13:56 |
| mwhahaha | of course the logs seem to be useless | 13:56 |
| *** cdearborn has joined #tripleo | 13:56 | |
| EmilienM | http://logs.openstack.org/28/493728/3/check/gate-tripleo-ci-centos-7-containers-multinode/d49033b/logs/undercloud/var/log/messages.txt.gz#_Aug_15_04_36_45 | 13:56 |
| EmilienM | mwhahaha: ^ | 13:56 |
| mwhahaha | hmm bad char | 13:57 |
| slagle | ccamacho: fyi, for https://review.openstack.org/#/c/493518/, i don't think that bug is upgrade specific, as I'm seeing the same thing on a new deploy | 13:57 |
| weshay | mwhahaha, http://logs.openstack.org/28/493728/3/check/gate-tripleo-ci-centos-7-containers-multinode/d49033b/logs/undercloud/var/log/messages.txt.gz#_Aug_15_04_36_45 | 13:57 |
| weshay | ha | 13:57 |
| *** dparkes has joined #tripleo | 13:58 | |
| slagle | ccamacho: the issue seems to be that the swift ring is under /var/lib/config-data/swift, but we bind mount /var/lib/config-data/puppet-generated/swift into the container(s) | 13:58 |
| pabelanger | weshay: EmilienM mwhahaha: http://logs.openstack.org/28/493728/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq/5e2cc19/logs/undercloud/etc/docker/daemon.json.txt.gz | 13:58 |
| pabelanger | you need double quotes around URL | 13:58 |
| weshay | pabelanger, k | 13:58 |
| EmilienM | in https://review.openstack.org/#/c/493728/3/roles/overcloud-prep-containers/templates/docker_daemon.json.j2 | 13:58 |
| weshay | thanks | 13:58 |
| EmilienM | weshay: you fix it? | 13:58 |
| EmilienM | ok thx! | 13:58 |
| weshay | ya | 13:59 |
| EmilienM | all right folks, it's meeting time | 13:59 |
| *** ramishra has joined #tripleo | 13:59 | |
| *** pkovar has quit IRC | 14:00 | |
| EmilienM | #startmeeting tripleo | 14:00 |
| openstack | Meeting started Tue Aug 15 14:00:22 2017 UTC and is due to finish in 60 minutes. The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| EmilienM | #topic agenda | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:00 | |
| EmilienM | * review past action items | 14:00 |
| EmilienM | * one off agenda items | 14:00 |
| EmilienM | * bugs | 14:00 |
| EmilienM | * Projects releases or stable backports | 14:00 |
| EmilienM | * CI | 14:00 |
| EmilienM | * Specs | 14:00 |
| EmilienM | * open discussion | 14:00 |
| EmilienM | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:00 |
| EmilienM | Hi everyone! who is around today? | 14:00 |
| fultonj | o/ | 14:00 |
| shardy | o/ | 14:00 |
| abishop | o/ | 14:00 |
| owalsh | o/ | 14:01 |
| jpich | o/ | 14:01 |
| marios | o/ | 14:01 |
| slagle | hi | 14:01 |
| mwhahaha | hi2u | 14:01 |
| EmilienM | #topic review past action items | 14:01 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:01 | |
| adarazs | o/ | 14:01 |
| jtomasek | o/ | 14:01 |
| myoung | o/ | 14:01 |
| EmilienM | EmilienM to switch master to run new upgrade jobs and not old ones anymore (done) | 14:01 |
| trown | o/ | 14:01 |
| EmilienM | jaosorior and abishop to talk together about plans for queens relating to barbican backends (prepare ptg session if needed + discuss about migration tool) (postponed) - not sure about the status | 14:02 |
| abishop | to clarify, my involvement is making sure existing deployments using legacy encryption key manager work | 14:02 |
| abishop | and that includes future migration from legacy key manager to barbican | 14:02 |
| abishop | cinder guy (eharney) is hoping key manager migration can be accomplished within cinder | 14:02 |
| abishop | so, no immediate OOO action required (i.e. for Denver PTG) | 14:02 |
| abishop | I'll continue to monitor, and will re-raise issue if OOO changes are needed | 14:02 |
| EmilienM | ok good to know | 14:02 |
| EmilienM | abishop: thanks | 14:02 |
| EmilienM | gfidente to send an ML note about moving ceph rgw from scenario004 to 001 | 14:03 |
| jrist | o/ | 14:03 |
| EmilienM | not sure Guilio is around, we can postpone this topic unless someone has thoughts | 14:03 |
| jaosorior | EmilienM: so yeah, action was taken :D | 14:03 |
| florianf | o/ | 14:03 |
| beagles | o/ | 14:03 |
| sshnaidm | o\ | 14:03 |
| jrist | EmilienM: I don't think he is around but maybe he'll see this later | 14:03 |
| EmilienM | #topic one off agenda items | 14:03 |
| *** kbyrne has joined #tripleo | 14:03 | |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:03 | |
| EmilienM | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:03 |
| EmilienM | sshnaidm: floor is yours | 14:04 |
| sshnaidm | yeah | 14:04 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Download logs interface https://review.openstack.org/473933 | 14:04 |
| sshnaidm | so clarkb, fungi suggest we will manage whitelist for /etc configurations to collect in logs server | 14:04 |
| *** ramishra has quit IRC | 14:04 | |
| sshnaidm | "strongly recommend" I would say | 14:05 |
| openstackgerrit | Andy Smith proposed openstack/tripleo-heat-templates master: WIP OpenStack containerized qpid-dispatch-router service https://review.openstack.org/479049 | 14:05 |
| adarazs | will that really save much? I think we're already filtering out the bigger items | 14:05 |
| sshnaidm | I did some calculations | 14:05 |
| EmilienM | yes, for months, I think | 14:05 |
| sshnaidm | In general multinode job /etc folders take about 8MB, within 33MB of all logs. From 8MB we actually need about 5.5MB and don't need 2.5MB ( it's about 7% of logs). | 14:05 |
| *** pkovar has joined #tripleo | 14:05 | |
| sshnaidm | So we can save 7% of place, I'm not really sure it worth the work... | 14:05 |
| *** ramishra has joined #tripleo | 14:06 | |
| sshnaidm | Therefore I'd like to bring it to discussion here | 14:06 |
| adarazs | doesn't sound like it. but we can definitely exclude some more files if it's straightforward. | 14:06 |
| *** jlinkes_ has joined #tripleo | 14:06 | |
| mwhahaha | 7% is a lot if you think about the number of jobs we actually run | 14:06 |
| mwhahaha | that's not trivial | 14:06 |
| EmilienM | on the other hand, infra provides us free resources and gently ask to help saving them | 14:06 |
| *** jlinkes has quit IRC | 14:06 | |
| *** links has quit IRC | 14:06 | |
| sshnaidm | adarazs, there was a arguments that with new release of centos it could be included big files and will break logs server like it was in centos 7.3 with java | 14:06 |
| pabelanger | Ya, at our scale, 1% is worth it | 14:06 |
| mwhahaha | i don't know if a whitelist is the best way to do it, but we do need to be better about excluding more | 14:07 |
| mwhahaha | we collect alot too much | 14:07 |
| *** strigazi is now known as strigazi_off | 14:07 | |
| pabelanger | keep in mind, we are also re-writing devstack-gate for zuulv3 in ansible, so I'm pretty sure we're likely write a generic role for jobs to use to collect, whitelisted this | 14:07 |
| EmilienM | if a whitelist is too much work, then improve the exclude list | 14:07 |
| sshnaidm | so we have 2 options right now : whitelist and bigger exclude list | 14:07 |
| *** strigazi_off is now known as strigazi_OFF | 14:08 | |
| sshnaidm | I'm against whitelist because it will require a maintenance | 14:08 |
| * adarazs is just a bit weary of a maintained whitelist and the constant "why we don't collect X" requests :/ | 14:08 | |
| sshnaidm | all new services we need to add to it | 14:08 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Adds PostUpgradeConfigStepsDeployment to drive post config ansible https://review.openstack.org/493878 | 14:08 |
| sshnaidm | manually | 14:08 |
| adarazs | sshnaidm: yep, exactly. | 14:08 |
| mwhahaha | for context, here is an example of a fully loaded etc dir that we log http://logs.openstack.org/28/493728/3/check/gate-tripleo-ci-centos-7-containers-multinode/d49033b/logs/undercloud/etc/ | 14:09 |
| adarazs | more aggressive exclusions I'm okay with. | 14:09 |
| mwhahaha | do we really need the skel, udev, rc* dirs, etc? | 14:09 |
| sshnaidm | mwhahaha, no, but it take about 7% | 14:09 |
| *** agurenko has quit IRC | 14:09 | |
| sshnaidm | So I'd suggest to start from big exclude list and to see if it satisfies | 14:09 |
| sshnaidm | wdyt? | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 14:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 14:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 14:10 |
| mwhahaha | yes i think we need to at least do that | 14:10 |
| pabelanger | I'm not sure I agree with whiltelist is more work, we do that for devstack-gate / logstash.o.o today. Its not like we are inundated with requests everyday to add more things. Sure, it will take a bit to build up the whitelist, just like it will take a while to exclude things | 14:10 |
| EmilienM | if we can keep reducing the size of logs, let's try that | 14:10 |
| sshnaidm | pabelanger, ? | 14:10 |
| pabelanger | and, whitelisting is much nicer to logs.o.o | 14:10 |
| sshnaidm | pabelanger, it's much more work than big exclude list | 14:10 |
| pabelanger | why is it much more work? | 14:11 |
| adarazs | the thing is we're not like a usual project in openstack, we don't have a well defined set of config files to collect, or rather it's a very big set that's constantly changing. | 14:11 |
| EmilienM | we're talking about /etc only right now right? | 14:11 |
| sshnaidm | pabelanger, because it requires a manual maintanance | 14:11 |
| sshnaidm | EmilienM, yes | 14:11 |
| *** gbarros has quit IRC | 14:11 | |
| pabelanger | sshnaidm: look at d-g, and how we handle /etc today | 14:11 |
| pabelanger | you would do the same | 14:11 |
| sshnaidm | pabelanger, we are different | 14:11 |
| EmilienM | I think a whitelist for /etc isn't too bad - we know what services we deploy (or plan to deploy) | 14:12 |
| EmilienM | but I might miss something | 14:12 |
| sshnaidm | EmilienM, the only difference is manual or automatic maintenance | 14:12 |
| mwhahaha | so i think this is pointing out the use of CI for debugging | 14:12 |
| mwhahaha | which if you need something, you should spin up an env locally | 14:12 |
| *** gbarros has joined #tripleo | 14:13 | |
| mwhahaha | and add it to the whitelist later | 14:13 |
| mwhahaha | either way we're capturing too much | 14:13 |
| mwhahaha | and it's been asked to be fixed for a while | 14:13 |
| mwhahaha | so to start we can do a bigger exclude list | 14:13 |
| EmilienM | see how it works | 14:14 |
| mwhahaha | but whitelist probably makes sense longer term | 14:14 |
| pabelanger | yes | 14:14 |
| mwhahaha | if we can't get it down with an exclude list we must switch to a white list | 14:14 |
| mwhahaha | manual work or not | 14:14 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493728 | 14:14 |
| EmilienM | mwhahaha: +1 | 14:14 |
| sshnaidm | mwhahaha, if we want whitelist, no need to make exclude list then.. | 14:14 |
| mwhahaha | so can we get a larger exclude list for next week? | 14:14 |
| *** iranzo has joined #tripleo | 14:14 | |
| adarazs | I'm fine with an exclude list, just not with an explicit whitelist. | 14:14 |
| sshnaidm | mwhahaha, let's choose one way | 14:15 |
| mwhahaha | it's about making incremental progress, right now we're not doign anything but arguing | 14:15 |
| mwhahaha | infra asked for a whitelist | 14:15 |
| mwhahaha | if we don't want to do that, then PoC an exclude list and lets go | 14:15 |
| mwhahaha | but progress needs to be made like now | 14:15 |
| mwhahaha | this has been a topic for far too long | 14:15 |
| adarazs | mwhahaha: as far as I understand the topic is infra wanting explicit whitelist and sshnaidm doesn't think it's good. | 14:15 |
| sshnaidm | ok, I'll prepare both and let's see who wins | 14:16 |
| adarazs | :) | 14:16 |
| * sshnaidm done | 14:16 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: [WIP] Switch to scheduling based on resource classes https://review.openstack.org/490851 | 14:16 |
| mwhahaha | k, can you have something by next week maybe? | 14:16 |
| EmilienM | I don't think we need to spend time on both now, we probably have other things to do as well | 14:17 |
| sshnaidm | mwhahaha, even today | 14:17 |
| * EmilienM thinks sshnaidm is a machine | 14:17 | |
| * sshnaidm not sure | 14:17 | |
| mwhahaha | #actions sshnaidm to prepare log exclusion/whitelist patches for review | 14:17 |
| akrivoka | honza: dumb question, what's the difference between registering and enrolling nodes? | 14:17 |
| mwhahaha | moving on :D | 14:18 |
| weshay | sshnaidm, make the patch specific to the upstream env | 14:18 |
| EmilienM | anything else for off items? | 14:18 |
| honza | akrivoka: none, afaik | 14:18 |
| EmilienM | #topic bugs | 14:18 |
| *** openstack changes topic to "bugs (Meeting topic: tripleo)" | 14:18 | |
| *** agopi|away has joined #tripleo | 14:18 | |
| EmilienM | #link https://launchpad.net/tripleo/+milestone/pike-rc1 | 14:18 |
| EmilienM | beside CI issues that we're already working on, do we have outstanding bugs that we need to get fixed in Pike RC1? | 14:19 |
| *** ykarel_ has joined #tripleo | 14:19 | |
| akrivoka | honza: is there any reason to introduce new terminology (enroll) when we have existing (register)? (https://review.openstack.org/#/c/488526/) | 14:19 |
| EmilienM | if I don't hear anything from anyone, I'll propose TripleO Pike RC1 by Friday morning. | 14:20 |
| *** pradk has joined #tripleo | 14:20 | |
| marios | EmilienM: there are some upgrades related things, i am looking at 2 personally https://bugs.launchpad.net/tripleo/+bug/1706951 and https://bugs.launchpad.net/tripleo/+bug/1708115 not sure we'll get everyting but we'll try | 14:20 |
| openstack | Launchpad bug 1706951 in tripleo "Ocata to Pike upgrade fails when cinder-volume runs on host because cinder-manage db sync runs when galera is unavailable" [Critical,In progress] - Assigned to Marios Andreou (marios-b) | 14:20 |
| openstack | Launchpad bug 1708115 in tripleo "Ensure non-controller are usable after upgrade and before converge." [Critical,Triaged] | 14:20 |
| slagle | EmilienM: i'm investigating a swift issue currently | 14:20 |
| honza | akrivoka: it's an ironic thing, i guess https://github.com/openstack/tripleo-common/blob/master/workbooks/baremetal.yaml#L1034 | 14:20 |
| shardy | EmilienM: due to the gate issues, I suspect some of the FFE things will slip into an RC2 | 14:20 |
| shardy | but sounds good | 14:20 |
| EmilienM | marios: ok, upgrade patches are backportable in any case, don't worry | 14:20 |
| slagle | EmilienM: https://bugs.launchpad.net/tripleo/+bug/1710606. but it's not limmited to upgrades afaict | 14:20 |
| openstack | Launchpad bug 1710606 in tripleo "O -> P - Upgrade: swift_object_expirer, swift_container_replicator, swift_object_replicator, swift_rsync, swift_account_replicator, swift_proxy containers are restarting after upgrade" [Critical,In progress] - Assigned to Carlos Camacho (ccamacho) | 14:20 |
| *** iranzo has quit IRC | 14:20 | |
| slagle | but it's already aligned against rc1 | 14:21 |
| EmilienM | slagle: oh this one :( ok | 14:21 |
| honza | akrivoka: i was reusing the terminology from tripleo-common | 14:21 |
| marios | EmilienM: ack thanks | 14:21 |
| EmilienM | shardy: yes I think RC2 will happen | 14:21 |
| slagle | EmilienM: yes. on a new deploy, i'm seeing the same thing | 14:21 |
| honza | akrivoka: but i'm open to changing that! | 14:21 |
| EmilienM | shardy: do we automatically move all FFEs to RC2? or just some of them? | 14:21 |
| EmilienM | I'll look at the remaining ffes end of this week | 14:22 |
| EmilienM | slagle: it's weird we don't hit that in the CI, or do we? | 14:23 |
| shardy | EmilienM: maybe we should review the status and decide if any should be deferred to queens? | 14:23 |
| slagle | EmilienM: i don't know. do we have verification of swift in the overcloud? | 14:23 |
| shardy | same with bugs, we probably need to start reducing the number of things we're tracking? | 14:23 |
| EmilienM | shardy: yeah, probably... | 14:24 |
| dtantsur | sorry for appearing out of blue, but I'm solving an ironic-related upgrade complication. just want you to be aware of it. | 14:24 |
| shardy | EmilienM: but given the gate issues we probably should be flexible if patches are posted | 14:24 |
| EmilienM | slagle: yes, I guess, with the pingtest, it uploads an image to glance with swift backend | 14:24 |
| dtantsur | this is https://bugs.launchpad.net/tripleo/+bug/1708653 | 14:24 |
| openstack | Launchpad bug 1708653 in tripleo "Need to set resource_class on Ironic nodes after upgrade to Pike" [High,In progress] - Assigned to Dmitry Tantsur (divius) | 14:24 |
| florianf | This is a regression that should probably get merged in rc1: https://review.openstack.org/#/c/482979/ | 14:24 |
| florianf | (tripleo-validations) | 14:24 |
| EmilienM | florianf: no bug report? | 14:24 |
| EmilienM | but ok | 14:25 |
| florianf | EmilienM: Let me create one | 14:25 |
| lvdombrkr | hello guys, as i understood by default 1 compute and 1 control node will be deployed, how i deploy compute and controller node all in in one? | 14:25 |
| EmilienM | ok moving on | 14:25 |
| EmilienM | #topic projects releases or stable backports | 14:25 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:25 | |
| shardy | EmilienM: also we need a bug to track the remaining pieces that enable minor updates with containers | 14:25 |
| EmilienM | shardy: I haven't seen a blueprint for that :( | 14:25 |
| shardy | there's a couple of update related bugs targetted to rc1 already, so I'll re-title one | 14:25 |
| EmilienM | it's part of the Container support blueprint, I guess | 14:25 |
| shardy | EmilienM: well it's a bug, minor updates without downtime | 14:26 |
| EmilienM | ok | 14:26 |
| * shardy thinks there's one for that already, just not specific to containers | 14:26 | |
| EmilienM | shardy: no problem for this one | 14:26 |
| EmilienM | so we'll see how it goes but | 14:26 |
| *** pkovar has quit IRC | 14:26 | |
| *** jlinkes_ has quit IRC | 14:26 | |
| EmilienM | #action EmilienM to prepare tripleo pike rc1 by friday if things go right | 14:26 |
| jaosorior | stable/ocata upgrade jobs seem to be timing out a lot :/ | 14:27 |
| mwhahaha | it's been that way for months now | 14:27 |
| EmilienM | if things don't go right, we'll probably defer to next week | 14:27 |
| EmilienM | jaosorior: yes it's not new | 14:27 |
| mwhahaha | stable/ocata is effectively blocked on the upgrade jobs | 14:27 |
| jaosorior | ah | 14:27 |
| jaosorior | well crap | 14:27 |
| EmilienM | they used to work ~ fine | 14:28 |
| EmilienM | but indeed since ~2 months (I think) they timeout a lot | 14:28 |
| EmilienM | #topic CI | 14:28 |
| *** openstack changes topic to "CI (Meeting topic: tripleo)" | 14:28 | |
| florianf | jaosorior, marios: Thanks! ;-) | 14:28 |
| EmilienM | the last time I checked was upgrade tasks taking time and it makes the job timeouting on some infra clouds | 14:28 |
| marios | np :) | 14:28 |
| *** agopi|away has quit IRC | 14:29 | |
| EmilienM | I posted https://bugs.launchpad.net/tripleo/+bug/1702955 | 14:29 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout when running in RAX cloud" [Critical,Triaged] | 14:29 |
| shardy | it'd be good to figure out which upgrade_tasks, chances are it's stuck downloading the new packages? | 14:29 |
| shardy | that's where most time goes, particularly without a local mirror | 14:29 |
| mwhahaha | well it should be using the local mirror now | 14:30 |
| EmilienM | I added an alert on the bug and hopefully get some attention | 14:30 |
| shardy | yeah I just wonder if that's working as expected | 14:30 |
| mwhahaha | but it just requires someone go look into it in depth | 14:30 |
| EmilienM | shardy: the local mirror works fine, afik but I can double check | 14:30 |
| EmilienM | I'll look at it if no one has time | 14:30 |
| EmilienM | do we have anything about CI? | 14:31 |
| sshnaidm | I didn't see upgrades jobs ever passing.. | 14:31 |
| EmilienM | sshnaidm: on stable/ocata, they do pass | 14:31 |
| EmilienM | weshay: did you do CI squad meeting last week? | 14:31 |
| mwhahaha | we need to merge the docker proxy today | 14:31 |
| mwhahaha | if possible | 14:31 |
| sshnaidm | EmilienM, yep, only there | 14:31 |
| weshay | yes.. I need to send the notes | 14:32 |
| mwhahaha | otherwise tomorrow we're going to end up with a 24hour+ gate | 14:32 |
| jaosorior | mwhahaha: what's the docker proxy review? | 14:32 |
| weshay | for that and the rdo mtg | 14:32 |
| EmilienM | #action CI / URGENT: review https://review.openstack.org/#/c/493728 and https://review.openstack.org/#/c/493726/ | 14:32 |
| mwhahaha | -^ | 14:32 |
| EmilienM | I think mandre isn't around but his -1 can be ignored | 14:32 |
| lvdombrkr | hello guys, as i understood by default 1 compute and 1 control node will be deployed, how i can deploy compute and controller node all in one? | 14:32 |
| sshnaidm | mwhahaha, did you see my patch? I wounder if it will be enough https://review.openstack.org/#/c/491923/ | 14:33 |
| weshay | does the undercloud configure a proxy for docker? | 14:33 |
| EmilienM | lvdombrkr: hey, we're in weekly meeting, and we're almost done | 14:33 |
| *** jlinkes has joined #tripleo | 14:33 | |
| lvdombrkr | EmilienM: sorry guys )) | 14:33 |
| mwhahaha | sshnaidm: possibly, so we need to figure out between weshay's patches and yours | 14:34 |
| EmilienM | sshnaidm: are you sure you can get NODEPOOL_DOCKER_REGISTRY_PROXY without sourcing the env on the image? | 14:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix Heat condition for RHEL registration yum update https://review.openstack.org/492632 | 14:34 |
| weshay | sshnaidm, patch worked as well http://logs.openstack.org/23/491923/2/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/891461e/logs/undercloud/etc/docker/daemon.json.txt.gz | 14:34 |
| *** mdnadeem has quit IRC | 14:34 | |
| sshnaidm | EmilienM, not sure I understand, which image..? | 14:35 |
| *** chem has quit IRC | 14:35 | |
| sshnaidm | ok, let's talk after mtg maybe | 14:35 |
| EmilienM | yeah | 14:35 |
| EmilienM | #topic specs | 14:35 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:35 | |
| EmilienM | do we have anything specs related this week? | 14:35 |
| *** ykarel_ has quit IRC | 14:35 | |
| EmilienM | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:35 |
| *** limao has joined #tripleo | 14:36 | |
| EmilienM | #topic open discussion | 14:36 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:36 | |
| EmilienM | quick reminder about the PTG, next month | 14:36 |
| EmilienM | #link https://etherpad.openstack.org/p/tripleo-ptg-queens | 14:36 |
| EmilienM | feel free to propose topics | 14:36 |
| EmilienM | we'll work on the agenda in the following weeks | 14:37 |
| EmilienM | anyone has anything before we close the meeting and go back to normal work? | 14:37 |
| EmilienM | thanks folks | 14:37 |
| EmilienM | #endmeeting | 14:37 |
| *** openstack changes topic to "CI Status: GREENish | "you can't land a Cessna with workarounds, so don't try in Quickstart" - TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC here." | 14:37 | |
| openstack | Meeting ended Tue Aug 15 14:37:34 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:37 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-08-15-14.00.html | 14:37 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-08-15-14.00.txt | 14:37 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-08-15-14.00.log.html | 14:37 |
| *** pkovar has joined #tripleo | 14:38 | |
| slagle | EmilienM: so we don't test swift in the overcloud in any containers job that i can tell | 14:38 |
| EmilienM | slagle: damn | 14:38 |
| slagle | i guess glance is using the file backend in containers-multinode | 14:38 |
| slagle | however, we don't capture the config so i can't really tell | 14:39 |
| slagle | EmilienM: i mean, i don't see any swift logs here: http://logs.openstack.org/48/490848/5/check/gate-tripleo-ci-centos-7-containers-multinode/684f38f/logs/subnode-2/var/log/containers/ | 14:41 |
| slagle | so...? | 14:41 |
| slagle | i guess that means it's not used | 14:41 |
| EmilienM | slagle: yeah and I just checked scenario002/container we don't deploy swift in containerz | 14:41 |
| EmilienM | so no tests at all :( | 14:41 |
| slagle | well, now actually, i see some swift process running | 14:42 |
| slagle | but no logs? | 14:42 |
| slagle | maybe it's failing, so no logs get generated | 14:42 |
| slagle | i bet glance is falling back to the http backend | 14:43 |
| ccamacho | hey slagle, sorry man today is public holiday in Spain, one thing, the swift ring files issue it also affects upgrades as in a non containerized deployment the ring files are located by default in /etc/swift/, so we should copy them to | 14:43 |
| ccamacho | slagle if they will live here /var/lib/config-data/swift or /var/lib/config-data/puppet-generated/swift/ we need to migrate them from /etc/swift | 14:44 |
| openstackgerrit | Adriano Petrich proposed openstack/instack-undercloud master: Add an hourly cron trigger for tripleo-ui logging https://review.openstack.org/469608 | 14:46 |
| *** gbarros has quit IRC | 14:49 | |
| *** agopi|away has joined #tripleo | 14:50 | |
| *** gbarros has joined #tripleo | 14:50 | |
| *** dmarlin has joined #tripleo | 14:54 | |
| *** jprovazn has joined #tripleo | 14:54 | |
| *** agopi|away has quit IRC | 14:56 | |
| *** agopi|away has joined #tripleo | 14:56 | |
| *** oidgar has joined #tripleo | 14:57 | |
| *** brault has joined #tripleo | 14:57 | |
| *** limao has quit IRC | 14:59 | |
| pabelanger | weshay: left comment on 493728 | 14:59 |
| *** dparkes has quit IRC | 14:59 | |
| *** rcernin has quit IRC | 15:01 | |
| *** brault has quit IRC | 15:02 | |
| openstackgerrit | Feng Pan proposed openstack/tripleo-heat-templates master: Add NeutronOverlayIPVersion parameter to neutron-plugins-ml2 service https://review.openstack.org/466162 | 15:07 |
| trozet | jaosorior: hi, why was opendaylight removed in the nova-api patch? https://review.openstack.org/#/c/475366/15/environments/docker-centos-tripleoupstream.yaml | 15:09 |
| *** yprokule has quit IRC | 15:09 | |
| jaosorior | trozet: I merely ran the command mentioned in the patch | 15:09 |
| jaosorior | * in the file | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 15:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 15:10 |
| jaosorior | trozet: if the OpenDaylight images were missing from tripleo-common (which they are) the command would remove them. | 15:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 15:10 |
| trozet | jaosorior: this is intentional | 15:10 |
| jaosorior | trozet: so, seems to me like it has to be fixed in tripleo-common, and then re-added | 15:10 |
| trozet | jaosorior: they are intentionally missing from tripleo-common | 15:10 |
| trozet | jaosorior: so they dont get pulled by default to the undercloud. A user has to manually pull them | 15:11 |
| jaosorior | trozet: oh, well, a comment would be nice :) and sorry for removing them. I actually didn't notice when I did the patch. | 15:11 |
| jaosorior | bnemec: commented here https://review.openstack.org/#/c/426348/ | 15:12 |
| *** florianf has quit IRC | 15:12 | |
| trozet | jaosorior: what script did you run? | 15:13 |
| jaosorior | trozet: can you do a commit to add them back and add a comment about them? I'll +2 it immediately | 15:13 |
| jaosorior | trozet: the one in the comment in that file./ | 15:13 |
| jaosorior | trozet: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/docker-centos-tripleoupstream.yaml#L3 | 15:13 |
| trozet | jaosorior: oh | 15:13 |
| jaosorior | trozet: the file has a comment on top saying it's autogenerated. | 15:14 |
| trozet | jaosorior: i guess it didnt used to be when i added it | 15:14 |
| jaosorior | trozet: hence I blindly did the command and thought it was all good. | 15:14 |
| trozet | jaosorior: perhaps it is ok | 15:15 |
| trozet | jaosorior: i include the images here as well https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services-docker/neutron-opendaylight.yaml#L15 | 15:15 |
| jaosorior | trozet: I actually like that more. Adding them in a separate environment. | 15:15 |
| dtrainor | I want to say I saw a bug somewhere that might explain the following message, but I can't find it. Look familiar to anyone? haproxy[1442]: proxy ironic has no server available! | 15:16 |
| bnemec | jaosorior: Yeah, so it turns out that is a lot more complicated than I had hoped. | 15:16 |
| trozet | jaosorior: yeah so it is fine i think | 15:16 |
| bnemec | Unless you move the keystone admin endpoint to the external network. | 15:16 |
| jaosorior | bnemec: yep, that's what I mentioned in the patch. | 15:16 |
| jaosorior | bnemec: maybe that's the way to go :/ | 15:17 |
| jaosorior | hrybacki: what do you think? | 15:17 |
| hrybacki | jaosorior: ijn mtg atm | 15:17 |
| *** hrybacki is now known as hrybacki|mtg | 15:17 | |
| jaosorior | bnemec: asking around | 15:18 |
| bnemec | jaosorior: I don't think this is going to get fixed for pike in any case, which was part of why I revisited it. | 15:18 |
| bnemec | We can document how to ssl it if you move it to external, or say that you need ssl-everywhere to do it. | 15:18 |
| *** florianf has joined #tripleo | 15:22 | |
| *** stendulker has joined #tripleo | 15:23 | |
| *** pcaruana has quit IRC | 15:24 | |
| *** gkadam has joined #tripleo | 15:25 | |
| *** akrzos is now known as akrzos-lunch | 15:26 | |
| *** agurenko has joined #tripleo | 15:28 | |
| *** mdnadeem has joined #tripleo | 15:29 | |
| *** links has joined #tripleo | 15:31 | |
| *** Lokesh_Jain__ has quit IRC | 15:34 | |
| *** mdnadeem has quit IRC | 15:39 | |
| *** jlinkes has quit IRC | 15:40 | |
| *** tesseract has quit IRC | 15:44 | |
| *** agurenko has quit IRC | 15:46 | |
| *** hrybacki|mtg is now known as hrybacki | 15:49 | |
| hrybacki | jaosorior: reading up now | 15:49 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Adding a Missing configuration to support QoS in ODL Closes-Bug: 1708131 https://review.openstack.org/492027 | 15:50 |
| openstack | bug 1708131 in tripleo "AttributeError: 'NoneType' object has no attribute 'get_policies'" [High,Invalid] https://launchpad.net/bugs/1708131 - Assigned to Itzik Brown (itzikb1) | 15:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 15:50 |
| *** oidgar has quit IRC | 15:52 | |
| *** marios has quit IRC | 15:52 | |
| *** agurenko has joined #tripleo | 15:52 | |
| jaosorior | shardy: if someone would configure a service to listen on the external network. What name would be used of the network? 'external'? | 15:54 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Adds SSH Banner text into sshd_config https://review.openstack.org/492152 | 15:54 |
| jaosorior | shardy: say, in OSP10 | 15:54 |
| shardy | jaosorior: yup | 15:55 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/network/service_net_map.j2.yaml#L62 | 15:55 |
| shardy | jaosorior: you'd pass a ServiceNetMap with the service you want to override the default mapping for | 15:56 |
| shardy | and set it to e.g external instead of internal_api | 15:56 |
| shardy | or whatever the default is in ServiceNetMapDefaults | 15:56 |
| jaosorior | shardy: thanks! | 15:57 |
| EmilienM | weshay: can you upload a new patch set with https://review.openstack.org/#/c/493728/4/roles/overcloud-prep-containers/templates/docker_daemon.json.j2 please? | 15:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493728 | 15:59 |
| EmilienM | weshay: I did it | 15:59 |
| weshay | EmilienM, I did | 15:59 |
| weshay | oh crud | 16:00 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Add -n/--networks-data option https://review.openstack.org/493933 | 16:00 |
| weshay | thanks | 16:00 |
| EmilienM | weshay: no prob | 16:00 |
| EmilienM | can a core approve https://review.openstack.org/#/c/493726/ please | 16:00 |
| weshay | don't understand though.. | to_json should have added the quotes to the string | 16:00 |
| weshay | the variable was being read just fine | 16:00 |
| EmilienM | weshay: to_json adds quotes? | 16:01 |
| weshay | ya | 16:01 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Switch to scheduling based on resource classes https://review.openstack.org/490851 | 16:01 |
| *** florianf has quit IRC | 16:01 | |
| weshay | adding them on the outside of the braces doesn't do anything I think | 16:01 |
| EmilienM | pabelanger: ^ | 16:01 |
| weshay | EmilienM, tested it locally and | to_json did the trick | 16:01 |
| *** links has quit IRC | 16:01 | |
| pabelanger | k, I wasn't sure of that | 16:02 |
| EmilienM | weshay: ok, so I can revert my last change | 16:02 |
| shardy | EmilienM: done | 16:02 |
| pabelanger | so, should be fine then | 16:02 |
| weshay | EmilienM, your change is fine | 16:02 |
| weshay | don't think it will change it | 16:02 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493728 | 16:03 |
| *** artom has quit IRC | 16:03 | |
| EmilienM | slagle: tl;dr I'm working on enabling pingtest for container multinode jobs (disabled now) and I haven't tested swift yet - if you feel like it's critical, I would raise alert on https://bugs.launchpad.net/tripleo/+bug/1710606 so we get CI escalation and more visibility from the swift team | 16:04 |
| openstack | Launchpad bug 1710606 in tripleo "O -> P - Upgrade: swift_object_expirer, swift_container_replicator, swift_object_replicator, swift_rsync, swift_account_replicator, swift_proxy containers are restarting after upgrade" [Critical,In progress] - Assigned to Carlos Camacho (ccamacho) | 16:04 |
| *** yamahata has joined #tripleo | 16:05 | |
| *** stendulker has quit IRC | 16:05 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Remove extra keystone admin haproxy listen and allow TLS https://review.openstack.org/493937 | 16:06 |
| jaosorior | bnemec: ^^ | 16:06 |
| *** oidgar has joined #tripleo | 16:08 | |
| slagle | EmilienM: it's not blocking ci, so not alert worthy i guess | 16:09 |
| *** jpich has quit IRC | 16:09 | |
| *** lucasagomes is now known as lucas-afk | 16:10 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 16:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 16:10 |
| *** egonzalez has quit IRC | 16:10 | |
| *** pkovar has quit IRC | 16:11 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix parsing of DockerCephDaemonImage parameter https://review.openstack.org/491759 | 16:12 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Remove extra keystone admin haproxy listen and allow TLS https://review.openstack.org/493937 | 16:12 |
| *** itlinux has quit IRC | 16:13 | |
| *** aufi_ has quit IRC | 16:15 | |
| *** pkovar has joined #tripleo | 16:16 | |
| *** florianf has joined #tripleo | 16:16 | |
| EmilienM | shardy: trying to debug upgrade jobs again | 16:18 |
| EmilienM | from ocata to pike http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container-upgrades-nv/15136b5/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-08-15_04_30_48 | 16:18 |
| EmilienM | we merged https://review.rdoproject.org/r/#/c/8561/ to have the patch from jistr in the package | 16:18 |
| *** thrash is now known as thrash|biab | 16:19 | |
| jaosorior | EmilienM: as far as I got the issue was that the upgrade was failing because it couldn't fetch the images for the containers. | 16:19 |
| EmilienM | Could not fetch contents for file:///tmp/tripleoclient-jxoda5/tripleo-heat-templates/docker/services/congress.yaml | 16:20 |
| EmilienM | the file doesn't exist in Ocata | 16:20 |
| jaosorior | EmilienM: ok, that's a new one | 16:20 |
| EmilienM | jaosorior: nope, we have this problem since ~1 month | 16:20 |
| shardy | EmilienM: Ok looking, sounds like we're mising up the paths in the deploy command (again) | 16:20 |
| jaosorior | EmilienM: it has been failing for very different reasons throughout this month | 16:21 |
| EmilienM | jaosorior: https://review.openstack.org/#/c/489874/ | 16:21 |
| shardy | Yeah there have been a few different issues I think | 16:21 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-common master: Publish logs before exporting them https://review.openstack.org/493819 | 16:21 |
| jaosorior | jistr fixed some, and more happened afterwards | 16:21 |
| EmilienM | the one with congress file was always here | 16:21 |
| EmilienM | but https://review.openstack.org/#/c/489874/ never merged | 16:21 |
| EmilienM | I'm going to revert the package change | 16:22 |
| EmilienM | and try again to get the upstream patch merged, even if we have to disable voting on the upgrade jobs | 16:22 |
| EmilienM | shardy: the path to THT ? | 16:23 |
| EmilienM | shardy: it sounds like it takes the arguments like it would deploy from master, but THT is deployed from stable/ocata | 16:26 |
| EmilienM | I'm investigating, maybe a quickstart thing | 16:26 |
| *** mcornea has quit IRC | 16:27 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: Set resource_class=baremetal for newly enrolled nodes https://review.openstack.org/493943 | 16:28 |
| EmilienM | trying to debug this script: http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container-upgrades-nv/15136b5/logs/undercloud/home/jenkins/overcloud-deploy.sh | 16:34 |
| EmilienM | shardy: I'm confused, we had upgrades working a few weeks ago iirc, isn't? | 16:35 |
| EmilienM | I remember jistr made it | 16:35 |
| *** pkovar has quit IRC | 16:36 | |
| *** rlandy is now known as rlandy|brb | 16:36 | |
| shardy | EmilienM: yeah, I think the patch you reference is related, and yes at one point jistr got the container upgrades job working, but not the scenarios AFAIK | 16:36 |
| EmilienM | shardy: I'll debug that today | 16:37 |
| shardy | Yeah maybe we should start with that, then move to the containers | 16:37 |
| EmilienM | jaosorior: any idea why https://review.openstack.org/#/c/493734/ fails? | 16:37 |
| shardy | sorry scenarios I mean | 16:37 |
| *** rlandy|brb is now known as rlandy | 16:37 | |
| jaosorior | EmilienM: on it | 16:38 |
| EmilienM | jaosorior: thx | 16:38 |
| EmilienM | shardy: it worked 7 days ago: http://logs.openstack.org/90/487390/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/77646bb/console.html.gz#_2017-08-01_12_54_54_235576 | 16:39 |
| EmilienM | doing diff now | 16:40 |
| weshay | EmilienM, fyi.. the solution that sshnaidm put up to add AFS mirrors for containers seems like a better patch imho | 16:40 |
| jaosorior | EmilienM: wow... that's strange "Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Systemd start for mariadb failed!", | 16:40 |
| shardy | logs.openstack.org/78/493878/2/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/7f6436d/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-08-15_15_28_59 | 16:40 |
| *** milan has quit IRC | 16:40 | |
| weshay | https://review.openstack.org/#/c/491923/ vs. https://review.openstack.org/#/c/493728/ | 16:41 |
| shardy | it looks like we've got some validations which prevent the upgrade running | 16:41 |
| * shardy looks for patches | 16:41 | |
| jaosorior | EmilienM: something tries to start mariadb. | 16:41 |
| sshnaidm | shardy, I saw you +w the containers patch | 16:42 |
| EmilienM | weshay: indeed http://logs.openstack.org/23/491923/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf4752a/logs/undercloud/etc/docker/daemon.json.txt.gz | 16:42 |
| weshay | it's also passing all the gates | 16:43 |
| EmilienM | sshnaidm, shardy: yeah I think we can approve https://review.openstack.org/#/c/491923 which is much simpler | 16:44 |
| *** trown is now known as trown|lunch | 16:44 | |
| shardy | sshnaidm, EmilienM: ack happy to go with that if you prefer | 16:44 |
| EmilienM | yeah, easier | 16:44 |
| * shardy just wants to see the gate queue improve ;) | 16:45 | |
| sshnaidm | yeah, we have this parameter already, just need to use it | 16:45 |
| EmilienM | ok | 16:45 |
| EmilienM | sshnaidm: thx | 16:45 |
| shardy | yes thanks sshnaidm | 16:46 |
| jaosorior | EmilienM: know what the issue is. Will push a fix. | 16:49 |
| *** dtantsur is now known as dtantsur|afk | 16:49 | |
| EmilienM | jaosorior: thanks! | 16:50 |
| *** ramishra has quit IRC | 16:51 | |
| *** psahoo has joined #tripleo | 16:52 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Move barbican's database creation to mysql profile https://review.openstack.org/493953 | 16:53 |
| jaosorior | EmilienM: ^^ | 16:53 |
| *** ramishra has joined #tripleo | 16:54 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/container: run Barbican non-containerized https://review.openstack.org/493734 | 16:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/container: run Barbican non-containerized https://review.openstack.org/493734 | 16:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on scenarios container jobs https://review.openstack.org/490129 | 16:54 |
| EmilienM | jaosorior: you're welcome :P | 16:54 |
| jaosorior | EmilienM: lol, that was fast. | 16:54 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Containarise Barbican API https://review.openstack.org/481451 | 16:55 |
| *** jlabarre has quit IRC | 16:56 | |
| EmilienM | jaosorior: can you rebase on top of https://review.openstack.org/#/c/493734/ maybe? we'll probably iterate on this one | 16:56 |
| EmilienM | but not required if you think we can containerize it this week | 16:57 |
| *** brault has joined #tripleo | 16:58 | |
| *** jlabarre has joined #tripleo | 16:59 | |
| *** psahoo has quit IRC | 16:59 | |
| EmilienM | shardy: sounds like right now it's failing on undercloud upgrade, maybe not too bad http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/console.html#_2017-08-15_05_02_57_498701 | 17:01 |
| EmilienM | I'll keep looking | 17:01 |
| EmilienM | err, overcloud upgrade | 17:01 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Containarise Barbican API https://review.openstack.org/481451 | 17:01 |
| jaosorior | EmilienM: done | 17:01 |
| EmilienM | probably in the upgrade tasks | 17:01 |
| EmilienM | jaosorior: merci | 17:01 |
| EmilienM | i'll baby sit the patches | 17:01 |
| *** brault has quit IRC | 17:02 | |
| *** itlinux has joined #tripleo | 17:03 | |
| *** agurenko has quit IRC | 17:04 | |
| EmilienM | shardy: sounds like puppet run returns 2 instead of 0 at step0 | 17:05 |
| EmilienM | at step1 sorry | 17:05 |
| EmilienM | http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/logs/subnode-2/var/log/messages.txt.gz#_Aug_15_04_41_56 | 17:05 |
| EmilienM | but I can't find the Error yet | 17:05 |
| *** catintheroof has quit IRC | 17:05 | |
| *** akrzos-lunch is now known as akrzos | 17:05 | |
| *** catintheroof has joined #tripleo | 17:06 | |
| shardy | /usr/bin/docker-current: Error: image tripleoupstream/centos-binary-neutron-server:latest not found.", | 17:06 |
| shardy | looks like docker-puppet failed to pull a bunch of images | 17:06 |
| *** oidgar has quit IRC | 17:06 | |
| EmilienM | Aug 15 05:26:23 centos-7-2-node-rax-dfw-10439456-805750 os-collect-config: "(outputs.stderr|default('')).split('\n')|union(outputs.stdout_lines|default([]))": [ | 17:07 |
| EmilienM | http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/logs/subnode-2/var/log/messages.txt.gz#_Aug_15_05_26_23 | 17:07 |
| EmilienM | Aug 15 05:26:23 centos-7-2-node-rax-dfw-10439456-805750 os-collect-config: TASK [Run docker-puppet tasks (generate config)] ******************************* | 17:07 |
| EmilienM | Aug 15 05:26:23 centos-7-2-node-rax-dfw-10439456-805750 os-collect-config: ok: [localhost] | 17:07 |
| EmilienM | Aug 15 05:26:23 centos-7-2-node-rax-dfw-10439456-805750 os-collect-config: TASK [debug] ******************************************************************* | 17:07 |
| EmilienM | Aug 15 05:26:23 centos-7-2-node-rax-dfw-10439456-805750 os-collect-config: fatal: [localhost]: FAILED! => { | 17:07 |
| EmilienM | Aug 15 05:26:23 centos-7-2-node-rax-dfw-10439456-805750 os-collect-config: "(outputs.stderr|default('')).split('\n')|union(outputs.stdout_lines|default([]))": [ | 17:07 |
| EmilienM | is this one critical shardy ? | 17:07 |
| *** rbowen has quit IRC | 17:08 | |
| EmilienM | oh yeah and all these errors to pull containers during the upgrade | 17:08 |
| *** jlabarre has quit IRC | 17:08 | |
| *** rbowen has joined #tripleo | 17:08 | |
| EmilienM | dprince: ^ can you help please? | 17:08 |
| *** jkilpatr has quit IRC | 17:08 | |
| shardy | EmilienM: I think it's the same problem, it couldn't pull some of the images | 17:09 |
| shardy | so maybe the registry cache patch will help | 17:09 |
| EmilienM | Unable to find image '192.168.24.1:8787/tripleoupstream/centos-binary-neutron-server:latest' | 17:10 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 17:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| EmilienM | shardy: it's not a problem with docker.io this time right? | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 17:10 |
| *** itlinux has quit IRC | 17:10 | |
| *** catintheroof has quit IRC | 17:10 | |
| *** jlabarre has joined #tripleo | 17:10 | |
| shardy | EmilienM: No it seems to be failing pulling from the registry on the undercloud | 17:11 |
| shardy | so we may either need to not use that registry (pull directly from the cache) or ensure it's populated before the upgrade | 17:11 |
| shardy | it's weird that not all images fail tho | 17:11 |
| *** thrash|biab is now known as thrash | 17:11 | |
| EmilienM | shardy: what is the real workflow in production? | 17:14 |
| EmilienM | pull from the undercloud cache iiuc, right? | 17:14 |
| shardy | EmilienM: well you might already have a local registry | 17:14 |
| shardy | but yeah you don't want $many nodes all downloading the same images, so either undercloud or some other local registry would be reccommended I think | 17:15 |
| shardy | personally I don't use the undercloud registry, it runs on the baremetal host | 17:15 |
| shardy | http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/logs/undercloud/var/log/messages.txt.gz#_Aug_15_05_26_23 | 17:16 |
| shardy | that shows a bunch of 404s for the requests from the undercloud registry | 17:17 |
| * shardy looks to see if any actually worked | 17:17 | |
| EmilienM | did we make a change on that recently? | 17:17 |
| shardy | well the workflow to prepare the environment and upload the images changed a bit in tripleoclient quite recently | 17:18 |
| * shardy looks at quickstart logs | 17:18 | |
| shardy | http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/logs/undercloud/home/jenkins/overcloud-prep-containers.sh | 17:19 |
| shardy | I don't see any logfile for that | 17:19 |
| *** hewbrocca is now known as hewbrocca_afk | 17:19 | |
| EmilienM | we could add logfile and depends-on my THT patch that test upgrades | 17:19 |
| shardy | Hmm, yeah the yaml file was generated | 17:20 |
| EmilienM | shardy: I'm a bit worried about --tag latest as well | 17:20 |
| EmilienM | it's fine for now, since we haven't open Queens | 17:20 |
| *** jcoufal has quit IRC | 17:20 | |
| EmilienM | but when Queens is open, we need to have a parameter for the tag (and use release name maybe?) | 17:21 |
| shardy | yeah we'll need some way to specify per-branch latest | 17:21 |
| *** jcoufal has joined #tripleo | 17:21 | |
| EmilienM | shardy: so the error now is only during upgrades, not during classic deployments | 17:22 |
| EmilienM | so we might miss a step during the upgrade process | 17:22 |
| EmilienM | do we setup registry on the undercloud? | 17:22 |
| EmilienM | probably yes | 17:22 |
| dprince | EmilienM: reading scrollback. help with docker pull timeouts I gather? | 17:23 |
| EmilienM | http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-08-15_04_08_45 | 17:23 |
| shardy | EmilienM: yeah we can see it running in the logs I linked | 17:23 |
| shardy | it just doesn't seem to have the right images uploaded | 17:23 |
| EmilienM | dprince: yes, we're debugging the upgrade jobs from ocata to pike | 17:23 |
| *** tosky has quit IRC | 17:23 | |
| dprince | shardy: for the upgrade job. Are we uploading containers twice then? | 17:24 |
| dprince | shardy: I suppose not. Baremetal -> containers | 17:24 |
| shardy | I was expecting to see a overcloud-prep-containers.sh.log.gz so it'd be good to figure out why that isn't written | 17:24 |
| shardy | dprince: yeah should be once I think, we deploy w/baremetal then upgrade | 17:24 |
| shardy | http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/logs/undercloud/home/jenkins/ | 17:24 |
| EmilienM | maybe the ansible task didn't run | 17:25 |
| shardy | dprince: that seems to show the right prep-containers script for quickstart, and it seems to have generated overcloud_containers.yaml.txt.gz | 17:25 |
| EmilienM | http://logs.openstack.org/00/461000/29/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/bbd7241/console.html#_2017-08-15_04_30_43_398538 | 17:25 |
| EmilienM | it was skipped | 17:25 |
| EmilienM | because we deployed a baremetal first | 17:25 |
| EmilienM | and then upgrade | 17:25 |
| shardy | aha | 17:25 |
| EmilienM | :) | 17:25 |
| EmilienM | ok let's see how to enable that | 17:26 |
| EmilienM | I'm pretty sure containerized_overcloud is set to False | 17:27 |
| dprince | okay, so perhaps conflicting quickstart options or something | 17:27 |
| EmilienM | yea | 17:27 |
| EmilienM | containerized_overcloud is False because indeed we need a baremetal for ocata | 17:27 |
| EmilienM | but containerized_overcloud needs to change to True for the upgrade | 17:27 |
| EmilienM | ok it all makes sense now | 17:28 |
| *** nyechiel has quit IRC | 17:29 | |
| shardy | Yeah or we can run the prep containers task just before doing the upgrade | 17:29 |
| EmilienM | I don't think we can set containerized_overcloud to True for the whole workflow anyway | 17:30 |
| EmilienM | it will try to deploy ocata in containers, i guess | 17:30 |
| shardy | Yeah | 17:30 |
| EmilienM | though we have containerized_overcloud_upgrade | 17:30 |
| EmilienM | we could re-use it | 17:30 |
| shardy | Yeah that sounds right, although jistr|off is the expert on how this was wired in | 17:31 |
| pabelanger | EmilienM: weshay: I can see docker client hitting /registry-1.docker reverse proxy cache on mirrors, but something is still wrong | 17:31 |
| weshay | ugh | 17:32 |
| pabelanger | are you doing docker pull docker.io/tripleoupstream/centos-binary-horizon:latest or docker pull tripleoupstream/centos-binary-horizon:latest | 17:32 |
| pabelanger | because something is adding v2 into the URL, and I think it is the client | 17:32 |
| pabelanger | 66.187.229.232 - - [15/Aug/2017:17:30:39 +0000] "GET /v2/tripleoupstream/centos-binary-octavia-base/manifests/latest HTTP/1.1" 403 443 "-" "docker/1.12.6 go/go1.7.4 kernel/3.10.0-514.26.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/2.4.2)" | 17:33 |
| pabelanger | which 403 | 17:33 |
| EmilienM | pabelanger: I think we're a tool from kolla, isn't dprince ? | 17:33 |
| pabelanger | Hmm... | 17:35 |
| EmilienM | shardy: could we override containerized_overcloud to True after some tasks? | 17:35 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Enable TLS configuration for containerized HAProxy https://review.openstack.org/491599 | 17:35 |
| dprince | EmilienM: we have some code in python-tripleoclient and tripleo-common that pulls images into the local registry on the undercloud | 17:36 |
| dprince | EmilienM: is that what you were asking? | 17:36 |
| shardy | EmilienM: maybe, but won't it be too late as quickstart only tries to run the task once? | 17:36 |
| * shardy looks at quickstart | 17:36 | |
| EmilienM | dprince: see pabelanger's question when you have time, I'm not able to investigate now, already debugging failing upgrades | 17:37 |
| shardy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2#L15 | 17:39 |
| shardy | EmilienM: it looks like we laready do the container image prepare if containerized_overcloud_upgrade_pull_images is defined | 17:39 |
| EmilienM | it's true in roles/overcloud-upgrade/defaults/main.yml | 17:39 |
| EmilienM | maybe it's just not defined | 17:40 |
| shardy | The two prepare commands there aren't the same as in overcloud-prep-containers.sh, and that one also has image upload | 17:41 |
| weshay | not sure I'm following | 17:41 |
| weshay | re: <EmilienM> shardy: could we override containerized_overcloud to True after some tasks? | 17:42 |
| pabelanger | okay, so you are not using the docker client | 17:42 |
| shardy | so perhaps they need to be aligned, stevebaker should be around soon and can probably confirm exactly how it should look | 17:42 |
| pabelanger | you are using python bindings for docker | 17:42 |
| weshay | EmilienM, if you want some tasks to be called and not others we can do that | 17:42 |
| shardy | weshay: I think we're good, containerized_overcloud_upgrade_pull_images seems to do what is needed, but the code it runs may not be correct | 17:42 |
| shardy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2#L15 | 17:42 |
| shardy | on upgrade a bunch of images are missing, so something around those prepare commands isn't working | 17:43 |
| shardy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2 | 17:43 |
| shardy | that one looks different, so perhaps we need to sync them, or find a way to share common code | 17:44 |
| * shardy really wishes we could just do task includes instead of the templated bash :( | 17:44 | |
| *** trown|lunch is now known as trown | 17:44 | |
| *** jkilpatr has joined #tripleo | 17:44 | |
| weshay | shardy, ya.. we can stuff them into one jinja template and break them out w/ variables | 17:44 |
| shardy | yeah we could do a j2 include I guess | 17:44 |
| weshay | that would help keep things in sync | 17:44 |
| EmilienM | yes +1 for j2 | 17:45 |
| shardy | https://github.com/openstack/tripleo-quickstart-extras/commit/90e703768326622eab1cf8dfa80daddccc3f88c8 | 17:46 |
| shardy | I think maybe this is the problem | 17:46 |
| shardy | it added prepare/upload for the script | 17:46 |
| *** florianf has quit IRC | 17:46 | |
| shardy | but removed the upload from the major-upgrade script template | 17:46 |
| EmilienM | when things will be better we'll run upgrade job in oooq gate | 17:47 |
| *** tosky has joined #tripleo | 17:47 | |
| EmilienM | shardy: the "openstack overcloud container image upload" you mean? | 17:48 |
| EmilienM | in roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2 | 17:48 |
| shardy | Yeah we do it in one script but not the other | 17:49 |
| shardy | which may or may not be related - locally I always do prepare then upload | 17:49 |
| shardy | stevebaker can hopefully confirm exactly what's needed but it looks inconsistent to me | 17:49 |
| weshay | it's usually pretty ugly when you have two roles share the same template and may not be possible in the long run as upgrades are moving to their own repo | 17:49 |
| weshay | so that may not be the best idea | 17:50 |
| shardy | could major-upgrade-overcloud-containers.sh run overcloud-prep-containers.sh? | 17:50 |
| * shardy tries to not rant about shell scripts again ;) | 17:51 | |
| shardy | I need to drop for a while, feel free to drop me a mail if this work needs to continue tomorrow :) | 17:52 |
| *** shardy is now known as shardy_afk | 17:53 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-quickstart-extras master: docker: Switch to trunk.registry.rdoproject.org https://review.openstack.org/493964 | 17:53 |
| dprince | weshay, pabelanger, EmilienM: trying that too ^^^ | 17:54 |
| weshay | shardy_afk, I'll put a review up so you can see what it could look like | 17:54 |
| EmilienM | dprince: I think we want to use regional AFS mirrors (that can be mirrors from RDO if we want) | 17:54 |
| pabelanger | yes, we want our regional mirrors | 17:55 |
| pabelanger | the issue is, kolla_build doesn't use docker client (go), it has python bindings | 17:55 |
| dprince | EmilienM: that should happend transparrently based on how dockers daemon.json is configured | 17:55 |
| pabelanger | which, I have no deal if this is going to work or not | 17:55 |
| EmilienM | I'm not sure https://review.openstack.org/493964 is what we need now | 17:55 |
| dprince | EmilienM: it is an idea, I thought I would try it | 17:55 |
| EmilienM | yeah it's good to try it | 17:56 |
| EmilienM | but the idea is to use mirrors | 17:56 |
| dprince | EmilienM: understood, but the infra mirrors should be transparrent regardless of what registry I use fwiw | 17:56 |
| pabelanger | no, we only proxy cache docker.io | 17:57 |
| pabelanger | they are not transparent proxies | 17:57 |
| pabelanger | okay, using docker-py 2.0.0 I have it working | 17:57 |
| weshay | I wish I understood that last part better regarding the infra mirrors and the rdo proxy.. when folks have time, not now.. maybe we can get a little more detail | 17:57 |
| dprince | pabelanger: the code we use for image upload is here http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/image/image_uploader.py#n23 | 17:57 |
| pabelanger | I cannot test with latest release, my server is too old | 17:57 |
| weshay | sorry.. rdo registry, not proxy | 17:57 |
| dprince | pabelanger: but you could cache RDO if that is what we wanted too right? | 17:58 |
| dprince | pabelanger: I mean, it is configurable | 17:58 |
| pabelanger | dprince: Right, I haven't even asked how you build the stuff on docker.io today. So, if a registery is needed, we have been talking about building a private one for docker.openstack.org (for example) | 17:59 |
| pabelanger | okay, so docker 2.1.0 python bindings I am able to make this work | 18:00 |
| pabelanger | anything higher then that is an issue | 18:01 |
| pabelanger | http://paste.openstack.org/show/618425/ | 18:02 |
| pabelanger | so, need to check which version of client you are installing | 18:02 |
| dprince | pabelanger: python2-docker-2.4.2-1.2.el7.noarch is that too high then? | 18:02 |
| *** brault has joined #tripleo | 18:03 | |
| pabelanger | dprince: somebody will need to do a test for me | 18:04 |
| pabelanger | I get: docker.errors.APIError: 400 Client Error: Bad Request ("client is newer than server (client API version: 1.26, server API version: 1.24)") | 18:04 |
| *** brault has quit IRC | 18:04 | |
| pabelanger | but I am running an older daemon on fedora-25 | 18:04 |
| *** artom has joined #tripleo | 18:09 | |
| *** artom_ has joined #tripleo | 18:10 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 18:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] | 18:10 |
| dprince | pabelanger: what test do you need? | 18:10 |
| dprince | pabelanger: also, I think stevebaker will be online soonish and may be the most helpful resource we have on this | 18:10 |
| pabelanger | the better step would be to apply /etc/docker/daemon.json setting, then run openstack overcloud container image prepare --images-file /home/jenkins/overcloud_containers.yaml --namespace tripleoupstream --tag latest --pull-source docker.io manually with wireshare running to capture the pcap. Because on mirror side, something is injecting /v2/ before tripleo images and I do not know why | 18:10 |
| pabelanger | because something is different from my local docker client testing | 18:11 |
| dprince | pabelanger: the docker bindings might just be trying a /v2/ registry URL optimistically. And if it fails they revert back to v1 | 18:12 |
| dprince | pabelanger: I see that in a ping_registry function anyway in the bindings | 18:13 |
| pabelanger | Actually, I just looked again. I do see v2 in my attempts | 18:13 |
| *** artom__ has joined #tripleo | 18:13 | |
| *** artom has quit IRC | 18:13 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: upload containers to undercloud in upgrade scenario https://review.openstack.org/493972 | 18:13 |
| weshay | EmilienM, dprince for upgrades ^ | 18:13 |
| dprince | pabelanger: yeah, that makes sense | 18:14 |
| pabelanger | 23.233.28.9 - - [15/Aug/2017:18:13:54 +0000] "GET /registry-1.docker/v2/tripleoupstream/centos-binary-octavia-api/manifests/sha256:7ca5b5d0137d206472613e6f4c86c282816342c0970fc32df85ecd951601b58c HTTP/1.1" 200 6630 "-" "docker/1.12.6 go/go1.7.6 kernel/4.11.10-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/2.1.0)" | 18:14 |
| *** artom_ has quit IRC | 18:14 | |
| pabelanger | so, locally for me it worked | 18:14 |
| pabelanger | why did your job get 403 then | 18:14 |
| pabelanger | Oh | 18:14 |
| *** artom has joined #tripleo | 18:15 | |
| EmilienM | weshay: yeah I'm reviewing it | 18:15 |
| pabelanger | 198.72.124.82 - - [15/Aug/2017:17:02:28 +0000] "GET /v2/tripleoupstream/centos-binary-sahara-api/manifests/latest HTTP/1.1" 403 441 "-" "docker/1.12.6 go/go1.7.4 kernel/3.10.0-514.26.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/2.4.2)" | 18:15 |
| pabelanger | it striped /registry-1.docker for some reason | 18:15 |
| pabelanger | difference is 2.1.0 vs 2.4.2 | 18:16 |
| pabelanger | so, somebody need to run my pastebin using 2.4.2 client | 18:16 |
| *** salmankhan has quit IRC | 18:16 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs https://review.openstack.org/493973 | 18:16 |
| sshnaidm | mwhahaha, EmilienM pabelanger ^^ | 18:16 |
| *** artom__ has quit IRC | 18:17 | |
| EmilienM | sshnaidm: sounds good, let's see how this works | 18:17 |
| dprince | pabelanger: what is in your docker deamon.json? | 18:18 |
| dprince | pabelanger: i've got docker 2.4.2 on Centos and can run it now if you wish | 18:19 |
| pabelanger | dprince: http://paste.openstack.org/show/618427/ | 18:19 |
| pabelanger | dprince: ya, if you use that mirror, I can watch apache | 18:19 |
| pabelanger | ready on this side | 18:20 |
| dprince | pabelanger: docker.errors.APIError: 400 Client Error: Bad Request ("client is newer than server (client API version: 1.26, server API version: 1.24)") | 18:20 |
| pabelanger | yup | 18:20 |
| pabelanger | that is what I get | 18:20 |
| pabelanger | maybe it needs to be using api.build() | 18:20 |
| pabelanger | I think that is what kolla_build did | 18:20 |
| pabelanger | and haven't tested that just yet | 18:20 |
| *** itlinux has joined #tripleo | 18:21 | |
| itlinux | good morning all! | 18:21 |
| itlinux | and afternoon and night :) | 18:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/461000 | 18:22 |
| EmilienM | weshay: testing your patch ^ | 18:23 |
| weshay | EmilienM, that patch looks like a WIP man.. didn't you listen to rlandy | 18:24 |
| weshay | how do you expect to get anything done if you are afraid to merge.. | 18:24 |
| weshay | whooosy | 18:24 |
| *** tdasilva has joined #tripleo | 18:25 | |
| EmilienM | weshay: I created https://bugs.launchpad.net/tripleo/+bug/1710938 | 18:26 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,Triaged] | 18:26 |
| *** artom_ has joined #tripleo | 18:28 | |
| dprince | pabelanger: try creating your client like this | 18:29 |
| dprince | pabelanger: client = docker.DockerClient(base_url='unix://var/run/docker.sock', version="1.24") | 18:29 |
| pabelanger | dprince: I see that attempt | 18:29 |
| dprince | pabelanger: well it is hanging now for me | 18:29 |
| pabelanger | it's fetching images, I didn't add logging to that | 18:30 |
| pabelanger | 216.252.204.85 - - [15/Aug/2017:18:29:01 +0000] "GET | 18:30 |
| pabelanger | /cloudfront/registry-v2/docker/registry/v2/blobs/sha256/e6/e6e5bfbc38e5499ed4d1492dd0c2ef27423dd0797c165eea7c2d165c6dcc217b/data?Expires=1502822941&Signature=DNwQJWSiYBS838hlyw3apNENhafrv4kHAyRVkzEVaJ7UkLY54-TXz11ztpKMGDyFSq28Vi~qMTjD3hYHtC0mbB43XzwvYGPlOVVj~92wElFDKJSXpnrB4VkdtnfO3w3kKHB0rceU7hFECZJaIyw9-I7o1bTcDyIskkq96zeU7H0_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q HTTP/1.1" 200 59324560 | 18:30 |
| pabelanger | "http://mirror.ord.rax.openstack.org:8081/registry-1.docker/v2/tripleoupstream/centos-binary-octavia-api/blobs/sha256:e6e5bfbc38e5499ed4d1492dd0c2ef27423dd0797c165eea7c2d165c6dcc217b" "docker/1.12.6 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/2.4.2)" | 18:30 |
| dprince | pabelanger: so maybe if I downgrade our client version it'll fix this? | 18:30 |
| pabelanger | dprince: maybe, I'm doing to try the single build command too. It is possible that is causing the issue | 18:31 |
| pabelanger | see, I thought you were first downloading images | 18:31 |
| pabelanger | but that isn't the case | 18:31 |
| *** artom has quit IRC | 18:31 | |
| dprince | pabelanger: I can do this I think | 18:31 |
| pabelanger | k | 18:31 |
| dprince | pabelanger: fwiw, it happens regardless of whether I use your proxy mirror or now | 18:31 |
| pabelanger | I need to grab a coffee, will keep streaming | 18:31 |
| dprince | or not | 18:31 |
| pabelanger | k | 18:32 |
| dprince | pabelanger, EmilienM is there a bug I should reference for this? | 18:32 |
| dprince | pabelanger: so our code uses "version='auto'" which also works fine for me | 18:34 |
| *** ecerquei has quit IRC | 18:34 | |
| pabelanger | ya, that works for me also | 18:35 |
| dprince | pabelanger: so where does that leave us then? | 18:35 |
| pabelanger | dprince: test using openstack overcloud container image prepare | 18:36 |
| pabelanger | because, I didn't see that work in gate | 18:36 |
| *** ecerquei has joined #tripleo | 18:36 | |
| EmilienM | dprince: https://bugs.launchpad.net/tripleo/+bug/1710533 | 18:41 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:41 |
| pabelanger | dprince: I see action on apache | 18:45 |
| dprince | pabelanger: I'm running it :) | 18:45 |
| pabelanger | okay, so, it seems to be working | 18:46 |
| dprince | pabelanger: yes, http://paste.openstack.org/show/618431/ | 18:46 |
| dprince | weshay: your patch wires this in I think here right? https://review.openstack.org/#/c/493972/1/roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2 | 18:46 |
| pabelanger | Hmm, okay so lets recheck weshay patch then | 18:46 |
| dprince | weshay: if so, then perhaps this is all that is needed to resolve the issues | 18:46 |
| pabelanger | dprince: https://review.openstack.org/493728/ was the patch from weshay | 18:47 |
| pabelanger | I just recheck to hit another mirror | 18:48 |
| pabelanger | but this is good news if dprince has it working locally | 18:48 |
| pabelanger | dprince: thanks for running the commands | 18:49 |
| dprince | pabelanger: np. let me know if you need more troubleshooting | 18:49 |
| *** artom_ is now known as artom | 18:50 | |
| *** dsariel has quit IRC | 18:54 | |
| *** artom has quit IRC | 18:54 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml https://review.openstack.org/493984 | 18:55 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml https://review.openstack.org/493984 | 18:56 |
| slagle | EmilienM: ok, i think i may have figured out the problem with swift | 18:57 |
| slagle | both swift-storage and swift-ringbuilder have puppet_config tasks that use a config_valume name "swift" | 18:58 |
| slagle | so one overwrites the other when docker-puppet.py runs | 18:58 |
| slagle | since we use rsync with --delete-after to save the generated files | 18:58 |
| slagle | dprince: can you check my logic there? ^ | 18:58 |
| dprince | slagle: oh, the rsync stuff was added to docker-puppet.py later. So perhaps this case wasn't caught with the switch | 19:00 |
| dprince | slagle: you could modify the ringbuilder to use a separate volume so long as it still gets the config files it needs | 19:00 |
| slagle | yea i'm not sure how to fix | 19:01 |
| slagle | since the ring files have to end up in /etc/swift somehow | 19:02 |
| dprince | slagle: is there a bug for this? | 19:02 |
| slagle | you can't really rsync with --delete-after | 19:02 |
| slagle | dprince: there will be. just started looking into it today :) | 19:02 |
| *** jprovazn has quit IRC | 19:02 | |
| atoth | hey all, I keep running out of disk space on my undercloud (I've been doing lots of overcloud test deploys). I'm wondering outside of the logs, what files should I be cleaning off the undercloud to fight the bloat? | 19:02 |
| dprince | atoth: rm -Rf /var/lib/docker | 19:03 |
| *** tosky has quit IRC | 19:03 | |
| *** jkilpatr has quit IRC | 19:03 | |
| *** rbowen has quit IRC | 19:03 | |
| *** pradk has quit IRC | 19:03 | |
| *** chlong_ has quit IRC | 19:03 | |
| *** abishop has quit IRC | 19:03 | |
| *** leifmadsen has quit IRC | 19:03 | |
| *** d0ugal has quit IRC | 19:03 | |
| *** tosky has joined #tripleo | 19:03 | |
| atoth | dprince, thanks, I'll give that a go | 19:03 |
| *** abishop has joined #tripleo | 19:03 | |
| *** chlong_ has joined #tripleo | 19:03 | |
| *** jkilpatr has joined #tripleo | 19:03 | |
| *** rbowen has joined #tripleo | 19:03 | |
| *** leifmadsen has joined #tripleo | 19:04 | |
| *** d0ugal has joined #tripleo | 19:04 | |
| atoth | dprince, actually, very little in var lib docker for me, but there is quite a large usage in /var/lib/ironic should those image directories be deleted too? | 19:06 |
| *** kbyrne has quit IRC | 19:07 | |
| dprince | atoth: sure, there is some caching there which I think Ironic mostly regenerates | 19:07 |
| dprince | atoth: restart docker after deleting /var/lib/docker BTW | 19:07 |
| atoth | dprince, cool, thanks for the info, off to find the rest of the bloat :-) | 19:08 |
| dprince | slagle: one idea, would be to make the ringbuilder also creates its own config files | 19:08 |
| *** kbyrne has joined #tripleo | 19:08 | |
| dprince | slagle: if you add in the stuff from 'puppet_tags' to that resource, and then give it a unique volume things might be happy again | 19:09 |
| dprince | slagle: initially, perhaps even try to make 'config_volume' for ringbuilder unique | 19:09 |
| dprince | slagle: http://paste.openstack.org/show/618436/ | 19:10 |
| *** pradk has joined #tripleo | 19:10 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 19:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 19:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,Triaged] | 19:10 |
| *** dsneddon has joined #tripleo | 19:10 | |
| *** dsneddon has quit IRC | 19:10 | |
| EmilienM | stevebaker: when you up https://bugs.launchpad.net/tripleo/+bug/1710938 | 19:11 |
| trozet | EmilienM: i see the latest odl one failed again...looking in the logs | 19:11 |
| slagle | dprince: right, but then i need to get what ends up under puppet-generated/swift-ringbuilder mounted at /etc/swift in the actual swift containers | 19:11 |
| EmilienM | slagle, mwhahaha : can you please review https://review.openstack.org/#/c/493953/ and https://review.openstack.org/#/c/493734/ ? cc jaosorior : your stuff worked, thanks | 19:12 |
| slagle | and they already have puppet-generated/swift mounted there | 19:12 |
| dprince | EmilienM: we think Wes's patch is a fix for that. weshay should you take this bug and link in your patch so we don't get duplicate things going on here | 19:12 |
| EmilienM | trozet: ok | 19:12 |
| weshay | dprince, aye | 19:12 |
| EmilienM | dprince: ok, thank you | 19:12 |
| EmilienM | dprince: which one? https://review.openstack.org/#/c/493972 or the mirror thing? | 19:13 |
| dprince | EmilienM: the AFS mirror patch that wes did | 19:13 |
| EmilienM | ah ok | 19:13 |
| EmilienM | dprince: but we replaced it by https://review.openstack.org/#/c/491923/ I though | 19:14 |
| dprince | EmilienM: see the backlog but we tested the 'openstack container image upload' and it was working with the infra mirrors. That means it isn't wired in correctly in quickstart somehow | 19:14 |
| EmilienM | thought$ | 19:14 |
| dprince | EmilienM: so I think Wes is on it. Reviewing and getting his patches correct seems to be the way forward here | 19:14 |
| trozet | EmilienM: /home/jenkins/overcloud-validate.sh 2>&1 failed, Overcloud pingtest, FAIL | 19:15 |
| dprince | slagle: we could copy them in manually | 19:15 |
| EmilienM | weshay, dprince: can we link the patches please? | 19:15 |
| dprince | slagle: with an init container | 19:15 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: upload containers to undercloud in upgrade scenario https://review.openstack.org/493972 | 19:15 |
| dprince | EmilienM: that was my request!!! | 19:15 |
| EmilienM | dprince: sorry, I was confused.. My bad, i'm context switching and bad at it today :( | 19:15 |
| weshay | Changed in tripleo: | 19:15 |
| weshay | assignee:nobody → wes hayutin (weshayutin) | 19:15 |
| weshay | status:Triaged → In Progress | 19:15 |
| EmilienM | trozet: you now need to look subnode-2 logs (neutron, nova, etc) | 19:16 |
| dprince | slagle: like we do with the 'mkdir && chown' tasks for container log files | 19:16 |
| trozet | EmilienM: so that is a ping test between 2 nova instances? | 19:16 |
| EmilienM | weshay: can you remove the WIP maybe? so we merge it if it works and if container folks it's the right process to do | 19:16 |
| dprince | slagle: or, you could modify docker-puppet.py to make the rsync-delete optional | 19:16 |
| trozet | EmilienM: or is that a ping test to a FIP of an instance? | 19:16 |
| EmilienM | trozet: yes, Overcloud pingtest is the ping test | 19:17 |
| dprince | slagle: either of those seem reasonable to you? | 19:17 |
| EmilienM | to a FIP of the instance | 19:17 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: upload containers to undercloud in upgrade scenario https://review.openstack.org/493972 | 19:17 |
| EmilienM | weshay: thx | 19:17 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/461000 | 19:17 |
| EmilienM | weshay: testing again ^ | 19:18 |
| weshay | time to make the donuts | 19:20 |
| slagle | dprince: yea. the optional delete probably. i think there might be another issue as well with the rsync command in that it uses --newer-than to filter out anything changed since the container started | 19:21 |
| slagle | ecxept that since download the rings from the undercloud, those are always going to have older timestamps | 19:21 |
| trozet | EmilienM: looks like another sync failure, evenw ith the right plugin: https://gist.githubusercontent.com/trozet/161259d55c48c51a256e5cbbf68801b5/raw/74656bc02a85af8296e8cfba03833b1f8deda2cf/sync_failure_qos | 19:23 |
| trozet | EmilienM: i dont think that is the reason for hte ping failure though | 19:23 |
| EmilienM | trozet: I have zero knowledge in ODL plugin deployment | 19:23 |
| trozet | EmilienM: i'm just saying at least we found a bug with the CI :) | 19:23 |
| EmilienM | trozet: I knew we would find some ;-) | 19:24 |
| trozet | EmilienM: to debug the ping failure, I need to have the ovs-ofctl -O openflow13 dump-flows br-int output from each node | 19:24 |
| trozet | EmilienM: and /opt/opendaylight/data/log/karaf.log | 19:24 |
| EmilienM | trozet: see what sshnaidm is doing with log collection and tell him what you need | 19:25 |
| EmilienM | dprince: have you tried zaqar on the containerized overcloud already? | 19:26 |
| dprince | EmilienM: just undercloud | 19:27 |
| jrist | EmilienM: fyi on our https://review.openstack.org/#/q/topic:bp/websocket-logging patches | 19:27 |
| dprince | EmilienM: it works there great | 19:27 |
| jrist | EmilienM: most are + or +A but there are a few still coming | 19:27 |
| EmilienM | dprince: yeah, but not on overcloud, http://logs.openstack.org/29/490129/6/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/e218abd/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-15_19_10_07 | 19:27 |
| jrist | https://review.openstack.org/#/c/493819/ could use reviews | 19:27 |
| *** bfournie has quit IRC | 19:27 | |
| dprince | 966509 | 19:28 |
| EmilienM | dprince: nothing useful in logs: http://logs.openstack.org/29/490129/6/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/e218abd/logs/subnode-2/var/log/containers/zaqar/zaqar-server.log.txt.gz | 19:28 |
| *** ecerquei_ has joined #tripleo | 19:28 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 19:28 |
| EmilienM | jrist: thx for the update | 19:28 |
| *** salmankhan has joined #tripleo | 19:29 | |
| trozet | EmilienM: do you know if you collect the nova console log for the instance somewhere? | 19:30 |
| *** salmankhan has quit IRC | 19:30 | |
| *** ecerquei has quit IRC | 19:30 | |
| EmilienM | trozet: I'm not sure of that, but I think we don't | 19:31 |
| trozet | EmilienM: doh...it is right there in the validate log: http://logs.openstack.org/05/486905/19/experimental/gate-tripleo-ci-centos-7-scenario008-multinode-oooq-nv/f9c3664/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-15_17_43_00 | 19:32 |
| trozet | EmilienM: yeah didnt get an IP from dhcp this is good info | 19:32 |
| *** bfournie has joined #tripleo | 19:33 | |
| *** salmankhan has joined #tripleo | 19:34 | |
| *** bfournie has quit IRC | 19:36 | |
| EmilienM | trozet: neutron dhcp agent probably? | 19:38 |
| EmilienM | trozet: or metadata agent | 19:38 |
| trozet | EmilienM: no it is ODL, QA has seen this as well where first instance does not get DHCP ip | 19:39 |
| trozet | EmilienM: ODL not configuring the switches correctly | 19:39 |
| *** pchavva has quit IRC | 19:40 | |
| tdasilva | slagle, dprince: sorry to jump-in, trying to follow on the swift convo and just wanted to make sure you are aware of this patch: https://review.openstack.org/#/c/493518 | 19:41 |
| EmilienM | trozet: I guess you see now why I wanted ODL part of tripleo gate | 19:43 |
| EmilienM | it just doesn't work now | 19:43 |
| EmilienM | trozet: it would be great to have logs from ODL | 19:44 |
| trozet | EmilienM: I wanted it too :) | 19:44 |
| trozet | EmilienM: yeah. michapma was asking me which logs we want earlier today. I think he may be adding them | 19:44 |
| trozet | EmilienM: will add comments to the gerrit in 1 min | 19:45 |
| *** salmankhan has quit IRC | 19:45 | |
| EmilienM | thrash: who can look at https://bugs.launchpad.net/tripleo/+bug/1710959 ? | 19:48 |
| openstack | Launchpad bug 1710959 in tripleo "zaqar doesn't work well when containerized" [High,Triaged] | 19:48 |
| dprince | EmilienM: not sure on the Zaqar failure. Who is working on that one? | 19:48 |
| dprince | EmilienM: there should be two running containers. One for zaqar-server and another for the websocket though | 19:49 |
| EmilienM | dprince: nobody is working on that one | 19:49 |
| EmilienM | pabelanger, weshay: so do we still need/want https://review.openstack.org/#/c/493728 ? I thought https://review.openstack.org/#/c/491923/ was enough | 19:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/multinode: do not run containerized Zaqar https://review.openstack.org/494005 | 19:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on scenarios container jobs https://review.openstack.org/490129 | 19:50 |
| * EmilienM afk lunch | 19:50 | |
| slagle | tdasilva: yea, was talking briefly with carlos about that earlier | 19:51 |
| weshay | pabelanger, ya.. I think this is the better way https://review.openstack.org/#/c/491923/ | 19:51 |
| slagle | tdasilva: i don't think the problem is limited to upgrades | 19:51 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add upgrade task to move swift ring files to be reachable by docker https://review.openstack.org/493518 | 19:51 |
| pabelanger | weshay: why is it better? I don't know much how these systems work | 19:52 |
| weshay | pabelanger, less code, same result | 19:52 |
| weshay | pabelanger, it also uses the undercloud config to do it | 19:52 |
| pabelanger | weshay: which jobs would have used that code? | 19:52 |
| weshay | so less in ci | 19:52 |
| tdasilva | slagle: ah, interesting...i've only seen bugs related to upgrades so far. does it happen on a fresh install consistently? | 19:52 |
| pabelanger | http://logs.openstack.org/23/491923/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf4752a/logs/undercloud/etc/docker/daemon.json.txt.gz | 19:52 |
| weshay | pabelanger, anything that deploys an undercloud | 19:52 |
| pabelanger | okay, so we set it up | 19:52 |
| pabelanger | let me check apache | 19:53 |
| slagle | tdasilva: i don't know about consistently. i've installed it once, it failed once | 19:53 |
| tdasilva | slagle: got it | 19:53 |
| weshay | pabelanger, tastes great, and it's less filling | 19:53 |
| tdasilva | slagle: just saw this: https://bugs.launchpad.net/tripleo/+bug/1710952 | 19:53 |
| openstack | Launchpad bug 1710952 in tripleo "missing swift rings causes swift containers stuck in docker restart loop" [Critical,New] | 19:53 |
| slagle | yea, that's the one i filed | 19:54 |
| slagle | i'm going to try a patch, but i'm not sure it's right | 19:54 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement https://review.openstack.org/442716 | 19:54 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name https://review.openstack.org/442755 | 19:54 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Add docker templates to configure Ironic inspector https://review.openstack.org/457822 | 19:54 |
| pabelanger | weshay: EmilienM: http://paste.openstack.org/show/618441/ something is still not configured correctly | 19:56 |
| *** sbrzozow has quit IRC | 19:56 | |
| pabelanger | weshay: EmilienM: it does hit the mirror | 19:56 |
| pabelanger | fails | 19:56 |
| pabelanger | then likely goes straight to docker.io | 19:56 |
| pabelanger | we need to add debug: true into daemon.json and see what is happening | 19:56 |
| pabelanger | because dprince tested this locally and it worked | 19:56 |
| weshay | pabelanger, hrm... for that you may want to use my patch | 19:57 |
| *** rlandy is now known as rlandy|brb | 19:58 | |
| pabelanger | http://logs.openstack.org/37/491437/4/gate/gate-tripleo-ci-centos-7-containers-multinode/a89431a/logs/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz just reset the gate | 20:00 |
| mwhahaha | meh | 20:00 |
| pabelanger | what is happening there | 20:01 |
| weshay | the undercloud only has the one option for the daemon | 20:01 |
| weshay | # An optional docker 'registry-mirror' that will beconfigured in | 20:01 |
| weshay | # /etc/docker/daemon.json. (string value) | 20:01 |
| weshay | #docker_registry_mirror = | 20:01 |
| pabelanger | Completed upload for docker image tripleoupstream/centos-binary-collectd:latest | 20:01 |
| pabelanger | where is that uploading too? | 20:01 |
| mwhahaha | the undercloud | 20:01 |
| pabelanger | what does: openstack overcloud container image prepare do? | 20:02 |
| pabelanger | that fetches images right? | 20:02 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder https://review.openstack.org/494008 | 20:02 |
| pabelanger | so why does openstack overcloud container image upload hit docker.io? | 20:02 |
| mwhahaha | that's the source | 20:02 |
| mwhahaha | destination -> undercloud | 20:02 |
| slagle | tdasilva: i dunno, maybe this will work: https://review.openstack.org/494008 | 20:02 |
| pabelanger | okay, so 2 different downloads? | 20:02 |
| mwhahaha | should just be one unless we're not properly using the undercloud | 20:03 |
| mwhahaha | if you then don't properly configure the undercloud as the source for the deploy it would download them again | 20:03 |
| *** marrusl has quit IRC | 20:04 | |
| *** ecerquei_ has quit IRC | 20:05 | |
| *** salmankhan has joined #tripleo | 20:06 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Move barbican's database creation to mysql profile https://review.openstack.org/493953 | 20:06 |
| *** bfournie has joined #tripleo | 20:06 | |
| *** bfournie has quit IRC | 20:07 | |
| pabelanger | mwhahaha: weshay: EmilienM: where is the debug log for your python-tripleoclient ? | 20:07 |
| mwhahaha | pabelanger: for which action? | 20:07 |
| pabelanger | both commands in http://logs.openstack.org/37/491437/4/gate/gate-tripleo-ci-centos-7-containers-multinode/a89431a/logs/undercloud/home/jenkins/overcloud-prep-images.sh.txt.gz | 20:08 |
| pabelanger | sorry | 20:08 |
| pabelanger | http://logs.openstack.org/37/491437/4/gate/gate-tripleo-ci-centos-7-containers-multinode/a89431a/logs/undercloud/home/jenkins/overcloud-prep-containers.sh.txt.gz | 20:08 |
| mwhahaha | pabelanger: i don't think those have a debug log | 20:08 |
| mwhahaha | so it would be stdout or not at all | 20:09 |
| *** bfournie has joined #tripleo | 20:09 | |
| pabelanger | k | 20:09 |
| *** agopi|away has quit IRC | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 20:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 20:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 20:10 |
| pabelanger | next question, http://logs.openstack.org/37/491437/4/gate/gate-tripleo-ci-centos-7-containers-multinode/a89431a/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz | 20:10 |
| pabelanger | why do we have duplicate images? | 20:10 |
| pabelanger | because I am starting to think, the containers are getting downloaded twice between the 2 commands | 20:11 |
| weshay | pabelanger, are you sure the source is from the same ip? | 20:11 |
| pabelanger | or is docker doing debuppig there? | 20:11 |
| *** akrivoka has quit IRC | 20:12 | |
| pabelanger | weshay: I'm not sure. I'm just a little surprised to see 2 entries for containers | 20:12 |
| pabelanger | and not sure if that is expected or not | 20:12 |
| fultonj | where can i find the output of tripleo.sh in the CI logs genereated by scenario001-multinode-oooq-container ? | 20:13 |
| *** agopi|away has joined #tripleo | 20:13 | |
| pabelanger | weshay: right now I am trying to understand why openstack overcloud container image upload needs to talk with docker.io | 20:14 |
| weshay | you are referring to | 20:14 |
| fultonj | specifically, i want to see if this line [1] was run in [2] | 20:14 |
| weshay | 192.168.24.1:8787/tripleoupstream/centos-binary-ceilometer-compute latest 6d6acbbf1501 2 weeks ago 760.7 MB | 20:14 |
| weshay | docker.io/tripleoupstream/centos-binary-ceilometer-compute latest 6d6acbbf1501 2 weeks ago 760.7 MB | 20:14 |
| fultonj | [1] https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L533 | 20:14 |
| fultonj | [2] http://logs.openstack.org/88/479288/32/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/2dd0e52/logs/ | 20:14 |
| pabelanger | weshay: ya | 20:15 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Create tripleo-admin user on deployed servers https://review.openstack.org/490470 | 20:15 |
| pabelanger | weshay: I don't understand why there are 2 entries | 20:16 |
| weshay | I don't think that is indicating it's downloading twice.. | 20:16 |
| EmilienM | back | 20:16 |
| pabelanger | okay, looking at code, both openstack overcloud container image prepare and openstack overcloud container image upload pull from docker.io. I am guessing the 2nd one is smart enought to use cached version | 20:18 |
| pabelanger | however, why do you need to pull again, why not just push the image from your local disk? | 20:18 |
| pabelanger | that would avoid hitting docker.io again, right? | 20:19 |
| pabelanger | http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/image/image_uploader.py#n108 | 20:19 |
| pabelanger | is what I am looking at | 20:19 |
| *** rlandy|brb is now known as rlandy | 20:21 | |
| EmilienM | jaosorior: https://bugs.launchpad.net/tripleo/+bug/1710807 | 20:22 |
| openstack | Launchpad bug 1710807 in tripleo "FreeIPA enroll can't work" [Medium,Triaged] | 20:22 |
| EmilienM | pabelanger: I would ping stevebaker on that question he has done some work here | 20:23 |
| *** akrivoka has joined #tripleo | 20:25 | |
| *** agopi|away is now known as agopi | 20:25 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 20:26 |
| pabelanger | EmilienM: stevebaker: Sure. So I can confirm, running docker pull on cached images, does hit docker.io again, because it wants to have the latest images. But, in this case, do you actually want to to that? Or just use the iamges that were downloaded from the command before? Because you could just push directly to your local registry and avoid a 2nd hit to docker.io | 20:27 |
| pabelanger | I mean, once we have docker using our reverse proxy cache, it should be fine. but it will save you some time to checksum all your containers again | 20:28 |
| EmilienM | pabelanger: I asked to dprince if he can reply, I don't have the answer tbh | 20:29 |
| EmilienM | pabelanger: I would promote the idea of having one single hit to docker.io | 20:30 |
| fultonj | EmilienM: where can i find the output of tripleo.sh in the CI logs genereated by scenario001-multinode-oooq-container ? | 20:31 |
| EmilienM | fultonj: it's using quickstart | 20:31 |
| pabelanger | EmilienM: well, there is another way to do this too. Set up your local registry as a proxy cache by default, having it cache docker.io requests. Then you don't actually need to push things into the local registry, it caches them itself. However, as stevebaker points out, in the docs: https://docs.docker.com/registry/recipes/mirror/ they say it doesn't support a private registry | 20:31 |
| pabelanger | however, I think it should be tested | 20:32 |
| EmilienM | fultonj: not sure what output you're looking for | 20:32 |
| EmilienM | fultonj: but undercloud/home/jenkins/* contains a lot of useful things | 20:32 |
| fultonj | EmilienM: the latest failure in https://review.openstack.org/#/c/479288 is from the ceph-ansible package not being on the undercloud | 20:32 |
| *** agopi has quit IRC | 20:32 | |
| fultonj | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L533 | 20:33 |
| EmilienM | indeed, it's not: http://logs.openstack.org/88/479288/32/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/2dd0e52/logs/rpm-qa.txt.gz | 20:33 |
| fultonj | ^ has put it there in the past | 20:33 |
| EmilienM | fultonj: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L533 isn't executed in quickstart | 20:33 |
| pabelanger | EmilienM: stevebaker: dprince: mwhahaha: weshay: So, to help move this along, could somebody work with me tonight or tomorrow to re-run the gate-tripleo-ci-centos-7-containers-multinode on some other place then gate? We can use the patches that exist now, but really need to see why the URL it is access is not correct. | 20:34 |
| pabelanger | in the mean time, I am going to place an autohold in nodepool for gate-tripleo-ci-centos-7-containers-multinode so I can look the next time it fails | 20:34 |
| fultonj | EmilienM: ok, so quickstart needs a similar change for scenario001-multinode-oooq-container | 20:34 |
| EmilienM | fultonj: most probably. | 20:34 |
| mwhahaha | pabelanger: if you just want to debug the image prepare command i can work with you on that | 20:36 |
| fultonj | EmilienM: i have memories of passing the same job because tripleo.sh took care of it | 20:36 |
| fultonj | was it recently changed or am i thinking of a diff job? | 20:36 |
| stevebaker | morning | 20:37 |
| weshay | ah.. I want an autohold and a pony | 20:37 |
| pabelanger | mwhahaha: I think dprince did that eariler with me? That seemed to work properly. But ya, lets try it again | 20:37 |
| mwhahaha | pabelanger: let me get my env setup | 20:37 |
| mwhahaha | pabelanger: actually i have an old env that should work | 20:39 |
| mwhahaha | pabelanger: do you just want me to turn debug on with a specific proxy? | 20:39 |
| pabelanger | mwhahaha: ya, lets use infracloud-vanilla | 20:40 |
| mwhahaha | pabelanger: pastebin? | 20:40 |
| pabelanger | mirror.regionone.infracloud-vanilla.openstack.org | 20:40 |
| pabelanger | 1 sec | 20:40 |
| fultonj | When was scenario001-multinode-oooq-container changed to use oooq ? | 20:41 |
| mwhahaha | sec | 20:41 |
| pabelanger | mwhahaha: http://paste.openstack.org/show/618448/ | 20:41 |
| mwhahaha | clearing out my local repo | 20:42 |
| *** morazi has quit IRC | 20:42 | |
| pabelanger | docker pull hello-world | 20:43 |
| pabelanger | should confirm hitting the cache | 20:43 |
| pabelanger | I just downloaded it | 20:43 |
| mwhahaha | pabelanger: just pulled it | 20:44 |
| pabelanger | I didn't see request | 20:44 |
| pabelanger | did you restart docker? | 20:44 |
| mwhahaha | nope, oops | 20:44 |
| pabelanger | k | 20:45 |
| pabelanger | seen that | 20:45 |
| mwhahaha | pabelanger: how about now | 20:45 |
| mwhahaha | k | 20:45 |
| pabelanger | okay, now lets have you try the openstack client commands | 20:45 |
| mwhahaha | pabelanger: running image upload | 20:46 |
| pabelanger | Ya, see that too | 20:46 |
| mwhahaha | pabelanger: ok so the 2nd one might be from the deploy | 20:46 |
| mwhahaha | pabelanger: I'll let you know when this finishes and then we can watch teh deploy | 20:46 |
| pabelanger | no, because I would see this attempt in apache logs from the jobs | 20:46 |
| pabelanger | and I don't | 20:46 |
| pabelanger | it uses the wrong URL | 20:47 |
| pabelanger | which makes me think, maybe something in job is not getting configured properly? | 20:47 |
| mwhahaha | pabelanger: maybe we're not restarting docker | 20:47 |
| *** matbu has quit IRC | 20:47 | |
| mwhahaha | but i thought you were seeing the pulls duplicated | 20:47 |
| pabelanger | 1 sec. let me get pastebine | 20:48 |
| pabelanger | http://paste.openstack.org/show/618441/ is all I see | 20:48 |
| pabelanger | first request is right | 20:48 |
| pabelanger | but the next, is wrong | 20:48 |
| pabelanger | GET /v2/tripleoupstream/centos-binary-aodh-api/manifests/latest is incorrect | 20:48 |
| mwhahaha | pabelanger: is that from my run? | 20:49 |
| pabelanger | mwhahaha: no, that is from gate job | 20:49 |
| pabelanger | GET /registry-1.docker/v2/tripleoupstream/centos-binary-aodh-api/manifests/latest | 20:49 |
| pabelanger | is what you hit | 20:49 |
| mwhahaha | pabelanger: which job logs was that from | 20:50 |
| pabelanger | gate is stripping /registry-1.docker | 20:50 |
| pabelanger | checking | 20:50 |
| mwhahaha | brb gotta get kid from bus then we can return to docker shenanigans | 20:50 |
| pabelanger | mwhahaha: http://logs.openstack.org/23/491923/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf4752a/console.html was the job run | 20:51 |
| *** matbu has joined #tripleo | 20:54 | |
| *** trown is now known as trown|outtypewww | 20:54 | |
| *** rcernin has joined #tripleo | 20:54 | |
| *** jcoufal has quit IRC | 20:56 | |
| pabelanger | mwhahaha: we should add debug: true to docker daemon also because this is how I know it is setup properly: http://paste.openstack.org/show/618449/ | 20:56 |
| pabelanger | Actually | 20:56 |
| pabelanger | 1 sec | 20:56 |
| *** catintheroof has joined #tripleo | 20:57 | |
| *** salmankhan has quit IRC | 20:57 | |
| pabelanger | okay, that is right. I am not logged in to docker.io | 20:57 |
| stevebaker | pabelanger: morning. If needed I can explain what prepare and upload do | 21:03 |
| pabelanger | stevebaker: sure, it would be helpful | 21:03 |
| *** artom has joined #tripleo | 21:03 | |
| *** lblanchard has quit IRC | 21:05 | |
| stevebaker | pabelanger: the only thing prepare does is generate yaml files from a template + arguments. When specifying --image-file it will prepare the yaml file that is passed to the upload command | 21:06 |
| stevebaker | pabelanger: the upload command will go through that file and do a pull and push for each image entry, which pulls from docker.io and pushes the undercloud registry | 21:07 |
| stevebaker | pabelanger: which results in the images being stored on the undercloud docker service *and* the undercloud docker registry | 21:08 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 21:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 21:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 21:10 |
| pabelanger | stevebaker: re: prepare, does that not use kolla_build? | 21:10 |
| stevebaker | pabelanger: prepare has nothing to do with kolla_build. It just prepares yaml files for what you actually want to do (one of build, upload or deploy an overcloud) | 21:12 |
| stevebaker | pabelanger: https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/overcloud.html#preparing-the-environment | 21:12 |
| *** brault has joined #tripleo | 21:12 | |
| pabelanger | stevebaker: okay, I see now thank you. I was looking at http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/tripleoclient/v1/container_image.py#n153 | 21:13 |
| mwhahaha | pabelanger: back btw | 21:13 |
| pabelanger | stevebaker: you just use kolla_builder to generate templates | 21:13 |
| pabelanger | mwhahaha: are you able to run openstack overcloud container image prepare? | 21:14 |
| stevebaker | pabelanger: ah yeah, kolla_builder is the yaml template loader | 21:14 |
| *** dprince has quit IRC | 21:14 | |
| mwhahaha | pabelanger: yea i already ran that (it doesn't pull the web) | 21:14 |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila Share for HA https://review.openstack.org/482680 | 21:14 |
| mwhahaha | pabelanger: i ran that before i started the upload | 21:14 |
| pabelanger | mwhahaha: right that now makes sense, are you then able to run the overcloud container image upload command? | 21:15 |
| mwhahaha | pabelanger: that's what i was running. I just stopped it | 21:15 |
| mwhahaha | pabelanger: let me run it again | 21:15 |
| pabelanger | k | 21:15 |
| pabelanger | 1 sec | 21:15 |
| pabelanger | a single image should be fine | 21:15 |
| pabelanger | ready on apache logs | 21:15 |
| pabelanger | stevebaker: thanks, that explains more | 21:16 |
| *** jcoufal has joined #tripleo | 21:16 | |
| mwhahaha | pabelanger: ok so what i'll do is i'll take the yaml and slim it down to a single image | 21:16 |
| mwhahaha | sec | 21:16 |
| pabelanger | ya, that is fine | 21:16 |
| *** agopi has joined #tripleo | 21:16 | |
| mwhahaha | k here comes an aodh-api pull | 21:16 |
| *** brault has quit IRC | 21:16 | |
| mwhahaha | running | 21:17 |
| pabelanger | ya, that works as expected | 21:17 |
| pabelanger | I see you hitting proper URL | 21:17 |
| pabelanger | mwhahaha: if you look at journald, you should see docker making an attempt to mirror | 21:17 |
| pabelanger | http://paste.openstack.org/show/618450/ | 21:18 |
| pabelanger | your working attempt | 21:18 |
| pabelanger | compare with failures above I linked | 21:18 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Don't use deprecated/removed cli args in net-iso docs https://review.openstack.org/494019 | 21:18 |
| mwhahaha | pabelanger: yea sec | 21:18 |
| *** liverpooler has quit IRC | 21:18 | |
| mwhahaha | pabelanger: http://paste.openstack.org/show/618451/ | 21:19 |
| pabelanger | OMG | 21:20 |
| mwhahaha | pabelanger: http://paste.openstack.org/show/618452/ | 21:20 |
| pabelanger | I know why | 21:20 |
| pabelanger | just reproduced it | 21:20 |
| mwhahaha | trailing slash? | 21:20 |
| * mwhahaha guesses | 21:20 | |
| pabelanger | yes | 21:20 |
| mwhahaha | death to trailing slashes | 21:21 |
| pabelanger | /faceplam | 21:21 |
| mwhahaha | or lack there of | 21:21 |
| pabelanger | ya | 21:21 |
| pabelanger | let me fix in openstack-infra | 21:21 |
| *** akrivoka has quit IRC | 21:23 | |
| pabelanger | remote: https://review.openstack.org/494021 NODEPOOL_DOCKER_REGISTRY_PROXY needs trailing slash for docker | 21:23 |
| pabelanger | EmilienM: weshay: stevebaker: mwhahaha: ^ that is our fix | 21:24 |
| pabelanger | however, we should make your playbooks smarter and ensure trailing / exists | 21:24 |
| *** jcoufal_ has joined #tripleo | 21:25 | |
| mwhahaha | the playbook doesn't seem to be the right place for this validation | 21:25 |
| pabelanger | agree, I'll defer to you where the validation should happen | 21:26 |
| mwhahaha | trailing slashes is one of those things that always pops up tho | 21:26 |
| *** abishop has quit IRC | 21:28 | |
| *** jcoufal has quit IRC | 21:28 | |
| *** jcoufal_ has quit IRC | 21:29 | |
| *** salmankhan has joined #tripleo | 21:31 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Exclude list for logs collection https://review.openstack.org/494022 | 21:34 |
| pabelanger | we would have see it sooner, if docker didn't silently fail back to docker.io | 21:34 |
| mwhahaha | i think it's a bug in the mirror url construction in docker | 21:35 |
| mwhahaha | but i'm taking a look | 21:35 |
| *** salmankhan has quit IRC | 21:38 | |
| mwhahaha | pabelanger: it's the reverse proxy's fault. we're using apache right? | 21:42 |
| pabelanger | mwhahaha: ya | 21:43 |
| pabelanger | mwhahaha: I think it is client side, because it does it /registry-1.docker/v2/ first, but gets 401 | 21:45 |
| mwhahaha | pabelanger: no they both get a 401 | 21:45 |
| mwhahaha | pabelanger: at least in my ngrep, let me double check | 21:45 |
| pabelanger | it then try with GET /v2/library/hello-world/manifests | 21:45 |
| mwhahaha | pabelanger: http://paste.openstack.org/show/618453/ | 21:46 |
| mwhahaha | first 4 blocks were w/o the trailing slash | 21:46 |
| mwhahaha | kinda hard to read | 21:46 |
| pabelanger | ya | 21:46 |
| pabelanger | that is the same I get | 21:46 |
| pabelanger | look at first GET | 21:46 |
| pabelanger | it is correct | 21:46 |
| pabelanger | add back trailing slash and go again | 21:47 |
| pabelanger | first GET will be same | 21:47 |
| mwhahaha | yea that's what it is | 21:47 |
| pabelanger | so, likely bug some place in docker | 21:48 |
| mwhahaha | yea | 21:48 |
| pabelanger | landing patch now, once merged, I'll pull trigger on new images | 21:48 |
| *** agopi has quit IRC | 21:48 | |
| pabelanger | then I'll promote 491923 in gate | 21:49 |
| *** ccamacho has quit IRC | 21:49 | |
| mwhahaha | pabelanger: interesting that the registry v2 test code actually seems to point to http://host/registry.v1/ as being an invalid mirror https://github.com/moby/moby/blob/a30ef99e8dd2c3e7a54b6410a5709f61db59c07f/registry/config_test.go#L127-L169 | 21:51 |
| pabelanger | heh, wonder the logic on that | 21:53 |
| pabelanger | maybe not to hardcode versioning | 21:53 |
| mwhahaha | yea seems weird | 21:53 |
| pabelanger | started new centos-7 DIB, should take 30mins to build | 21:54 |
| pabelanger | then another hour or so to upload | 21:55 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 22:00 |
| *** dsneddon_ has joined #tripleo | 22:00 | |
| stevebaker | pabelanger, mwhahaha: by the way, once things settle with the image proxy, it would be nice if the required docker daemon.json changes could happen via the puppet-tripleo docker setup, then we can easily do the same on the overcloud | 22:01 |
| mwhahaha | it's already supported | 22:02 |
| mwhahaha | tripleo::profile::base::docker::registry_mirror | 22:02 |
| * mwhahaha points to the fact that this should be part of the instack-undercloud setup | 22:02 | |
| mwhahaha | i think that's actually what sshnaidm's patches does | 22:03 |
| mwhahaha | https://github.com/openstack/instack-undercloud/commit/19470b58ec749da31c83406ca0c005522e8d96d5 | 22:03 |
| mwhahaha | the undercloud.conf supports it | 22:03 |
| mwhahaha | https://review.openstack.org/#/c/454880/ | 22:04 |
| mwhahaha | stevebaker: you even approved it :D | 22:04 |
| stevebaker | mwhahaha: April? A lifetime ago | 22:05 |
| mwhahaha | i know right? | 22:05 |
| mwhahaha | yea sshnaidm's patch sets undercloud_docker_registry_mirror which populates the undercloud.conf | 22:06 |
| mwhahaha | so we're good | 22:06 |
| stevebaker | mwhahaha: sweet, thanks | 22:06 |
| *** jlabarre has quit IRC | 22:07 | |
| *** brault has joined #tripleo | 22:07 | |
| *** chlong_ has quit IRC | 22:08 | |
| EmilienM | stevebaker: so I confirm https://review.openstack.org/#/c/493972 is the right fix | 22:09 |
| EmilienM | stevebaker: the upgrade job now goes further | 22:10 |
| EmilienM | but timeouts :( | 22:10 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 22:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 22:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 22:10 |
| EmilienM | http://logs.openstack.org/00/461000/31/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/de2864e/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-08-15_21_59_08 | 22:10 |
| mwhahaha | upload just takes so long | 22:10 |
| EmilienM | yes | 22:10 |
| EmilienM | more than 20 min | 22:10 |
| stevebaker | if only we had a ... caching proxy | 22:11 |
| mwhahaha | it's basically the images+all rpms even if you aren't deploying them | 22:11 |
| mwhahaha | it's more than that though | 22:11 |
| mwhahaha | we're over collecting than what we used to | 22:11 |
| mwhahaha | because we're essentially downloading everything rather than just teh services that were deployed | 22:11 |
| mwhahaha | traditional upgrades would only update only the packages deployed + new images | 22:12 |
| *** brault has quit IRC | 22:12 | |
| stevebaker | mwhahaha: It is possible to exclude images in the prepare call to end up with a more targeted image list | 22:12 |
| mwhahaha | it seems like we need the inverse, calculate what we need and then do the image pull | 22:13 |
| stevebaker | EmilienM: since the upload completes despite the timeout, I think that can be approved | 22:13 |
| mwhahaha | for the deployment | 22:13 |
| EmilienM | stevebaker: I did, but I'm not really satisfied with what we have now | 22:14 |
| EmilienM | stevebaker: can't we tell which containers we want from the THT services that we use? | 22:19 |
| stevebaker | mwhahaha, EmilienM: I started going down that path https://review.openstack.org/#/c/448328/ | 22:19 |
| stevebaker | mwhahaha, EmilienM: It would be worth resurrecting that at some point | 22:19 |
| mwhahaha | yea | 22:19 |
| EmilienM | stevebaker: please restore | 22:19 |
| EmilienM | stevebaker: the experience I had with upgrades today was terrible | 22:19 |
| stevebaker | EmilienM: just due to the time of the image downloads? | 22:23 |
| EmilienM | stevebaker: yes. Do you have a bug report already for that? | 22:23 |
| EmilienM | stevebaker: right now, we still can't test upgrades | 22:23 |
| *** limao has joined #tripleo | 22:24 | |
| stevebaker | EmilienM: won't downloads be significantly faster once the proxy change lands? | 22:24 |
| EmilienM | stevebaker: we still need to push on the local registry | 22:25 |
| EmilienM | it sounds like it takes a bunch of time, no? | 22:25 |
| EmilienM | see my logs ^ | 22:25 |
| mwhahaha | it might help if we're using consistent upgrade images at least in CI | 22:26 |
| mwhahaha | but it's still going to be a lot of extra transit | 22:26 |
| stevebaker | EmilienM: it does one pull then one push, I believe most of that time is the pull from docker.io | 22:26 |
| EmilienM | stevebaker: anyway, can we restore your work? I find it good to have | 22:28 |
| stevebaker | EmilienM: oh, definitely. I'm just assuming it is too late for pike | 22:28 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-docs master: add how to check gate with tripleo to the contrib doc https://review.openstack.org/487598 | 22:31 |
| EmilienM | stevebaker: do you think it's a lot of work? | 22:31 |
| EmilienM | stevebaker: I think we could backport it, to me it looks really important | 22:31 |
| EmilienM | for the upgrades... | 22:31 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/461000 | 22:33 |
| EmilienM | let's try with the proxy ^ | 22:33 |
| stevebaker | EmilienM: not really, its a manualish process to discover the service -> images mapping, then a modification to the prepare command to parse the supplied heat env file (environments/docker.yaml) to know what services will be deployed | 22:33 |
| stevebaker | EmilienM: I'll resurrect and raise a bug to track | 22:34 |
| EmilienM | thx | 22:35 |
| pabelanger | centos-7 DIBs uploading now | 22:44 |
| *** itlinux has quit IRC | 22:44 | |
| EmilienM | pabelanger: ok thx | 22:44 |
| EmilienM | mwhahaha: on upgrade jobs, from newton to ocata, AllNodesPostUpgradeSteps 1391.0 | 22:44 |
| EmilienM | what the heck takes this time | 22:44 |
| mwhahaha | no idea | 22:44 |
| EmilienM | so AllNodesPostUpgradeSteps is all upgrade steps | 22:44 |
| EmilienM | ControllerUpgrade_Step1 183.0 | 22:44 |
| EmilienM | and ControllerUpgrade_Step3 116.0 | 22:44 |
| mwhahaha | EmilienM: which logs are you looking at? | 22:44 |
| mwhahaha | 1391 is about ~23 mins | 22:44 |
| stevebaker | EmilienM: I'll let you set the importance and milestone https://bugs.launchpad.net/tripleo/+bug/1710992 | 22:44 |
| openstack | Launchpad bug 1710992 in tripleo "All container images are uploaded/specified, even for services not deployed" [Critical,Triaged] - Assigned to Steve Baker (steve-stevebaker) | 22:44 |
| mwhahaha | yum's not terribly fast, neither is the puppet catalog calculations | 22:44 |
| EmilienM | stevebaker: done | 22:44 |
| *** dsavineau has quit IRC | 22:44 | |
| stevebaker | EmilienM: thanks | 22:44 |
| EmilienM | stevebaker: I'm out for a couple of hours from now but I'm back in the evening, probably all my evening and try to make progress on these upgrade things | 22:44 |
| EmilienM | stevebaker: imho 1710992 is super high prio | 22:44 |
| EmilienM | I don't see how we can test upgrades upstream otherwise | 22:44 |
| EmilienM | and testing upgrades is our highest prio now | 22:44 |
| EmilienM | anyway, bbl | 22:44 |
| stevebaker | EmilienM: ok, should I stop working on https://bugs.launchpad.net/tripleo/+bug/1691403 ? | 22:44 |
| openstack | Launchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [High,In progress] - Assigned to Steve Baker (steve-stevebaker) | 22:44 |
| EmilienM | stevebaker: unless you have other priorities, we should focus on shipping RC1 and upgrades are not working now | 22:45 |
| EmilienM | but if oyu have to work on neutron, go ahead | 22:45 |
| EmilienM | I'm afk for real now | 22:45 |
| stevebaker | EmilienM: ok, talk to you later | 22:46 |
| *** dsavineau has joined #tripleo | 22:47 | |
| pabelanger | only infracloud-chocolate, infracloud-vanilla, rax-iad and rax-ord left | 22:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Make network-isolation-v6 environment rendered for all roles https://review.openstack.org/474486 | 23:08 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 23:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 23:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 23:10 |
| *** catintheroof has quit IRC | 23:11 | |
| *** rcernin has quit IRC | 23:12 | |
| *** tosky has quit IRC | 23:22 | |
| *** brault has joined #tripleo | 23:24 | |
| *** itlinux has joined #tripleo | 23:26 | |
| pabelanger | EmilienM: sshnaidm: mwhahaha: So, just seen: https://review.openstack.org/430688 Basically, we should never do this. Aside of potentially running an insecure version of openssh on nodes now, we should be doing this in openstack-infra so it applies to all projects. | 23:28 |
| pabelanger | 2017-08-15 23:18:22.410876 | openssh-6.6.1p1-35.el7_3.x86_64 is a duplicate with openssh-6.6.1p1-33.2.el7_3.x86_64 is actually what is happening now | 23:28 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Revert "Installs openssh packages that try to workaround sshd start problem" https://review.openstack.org/494039 | 23:28 |
| *** brault has quit IRC | 23:28 | |
| mwhahaha | hmm seem that it's conflicting | 23:29 |
| mwhahaha | i think it might have already been reverted | 23:29 |
| mwhahaha | we had issues because of 7.4 stuff | 23:29 |
| pabelanger | no, I still see it getting installed | 23:29 |
| mwhahaha | orly | 23:29 |
| mwhahaha | k let me fix that up | 23:29 |
| mwhahaha | i remember that patch but i don't recall why | 23:30 |
| mwhahaha | but it was supposed to be temporarry | 23:30 |
| pabelanger | Ya, been 6 months now | 23:30 |
| *** thrash is now known as thrash|g0ne | 23:33 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Revert "Installs openssh packages that try to workaround sshd start problem" https://review.openstack.org/494039 | 23:34 |
| * mwhahaha dusts off his look of disapproval ಠ_ಠ | 23:34 | |
| mwhahaha | we should be tracking this tech debt with bugs | 23:34 |
| * mwhahaha thinks it's time to propose these tech debt items have bugs | 23:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci master: Revert "Installs openssh packages that try to workaround sshd start problem" https://review.openstack.org/494040 | 23:35 |
| mwhahaha | heh we've tried to revert that 3 times now | 23:35 |
| mwhahaha | https://review.openstack.org/#/q/6e8e27488da31b3b282fe1ce5e07939b3fa11b2f,n,z | 23:35 |
| pabelanger | Wow | 23:35 |
| pabelanger | I'll abandon mine | 23:35 |
| mwhahaha | was just released 14 days ago | 23:36 |
| pabelanger | I think I remember something about sshd failing to restart, but don't remember us doing anything to fix it | 23:38 |
| pabelanger | in openstack-infra | 23:38 |
| mwhahaha | i think it was only in the tripleo case | 23:38 |
| mwhahaha | which is why it was this way, either way it should have been tracked better | 23:38 |
| mwhahaha | cause no one would have remembered to revert that | 23:39 |
| pabelanger | Ya, there is a few ways we could have done it. But please next time, atleast keep openstack-infra in the loop. | 23:40 |
| pabelanger | currently waiting for a job to hit rax-ord | 23:40 |
| pabelanger | confirmed that new variable was setup properly | 23:40 |
| *** dmarlin has left #tripleo | 23:40 | |
| *** limao has quit IRC | 23:40 | |
| pabelanger | installing undercloud now | 23:40 |
| *** rhallisey has quit IRC | 23:41 | |
| *** rlandy has quit IRC | 23:42 | |
| *** gbarros has quit IRC | 23:46 | |
| pabelanger | should be merging 5 things this time around | 23:47 |
| mwhahaha | pabelanger: do you have a reference to where you saw that ssh problem in the logs? | 23:50 |
| pabelanger | mwhahaha: I think telnet://23.253.175.9:19885 | 23:50 |
| mwhahaha | no not that one, i'll go poke at some job logs | 23:51 |
| mwhahaha | it wasn't in the oooq jobs | 23:51 |
| mwhahaha | might have been multinode | 23:51 |
| pabelanger | telnet://23.253.92.76:19885 then | 23:52 |
| mwhahaha | yea that installs it but no warnings | 23:52 |
| mwhahaha | even better | 23:52 |
| mwhahaha | so no one would have remembered to ever revert it | 23:52 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Set file mode permission of Ceph keyrings https://review.openstack.org/492303 | 23:53 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Internal TLS support for mongodb container https://review.openstack.org/492878 | 23:53 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Updating tempestmail template mail https://review.openstack.org/487795 | 23:53 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Do not run clustercheck on the host after O->P upgrade https://review.openstack.org/487889 | 23:53 |
| *** brault has joined #tripleo | 23:55 | |
| *** dsneddon_ is now known as dsneddon | 23:57 | |
| dsneddon | Does anyone know why this patch wouldn't be merged yet? https://review.openstack.org/#/c/492218 | 23:57 |
| mwhahaha | dat gate backup | 23:57 |
| pabelanger | was 24 hours | 23:57 |
| dsneddon | mwhahaha, Yeah, that's what I thought, just wanted to make sure. | 23:57 |
| mwhahaha | http://status.openstack.org/zuul/ | 23:57 |
| pabelanger | down to 14h | 23:57 |
| mwhahaha | feel free to follow along (and cry) | 23:58 |
| dsneddon | Well, if it doesn't merge before I leave on PTO, I can have someone else help merge the follow-up patch. Thanks. | 23:58 |
| pabelanger | the good news is, I haven't see any new issues with gate failures today | 23:58 |
| pabelanger | docker.io reverse proxy cache will help alot | 23:59 |
| pabelanger | likey evern speed up container jobs | 23:59 |
| dsneddon | pabelanger, Is this a gate failure? https://review.openstack.org/#/c/486260/ | 23:59 |
| mwhahaha | yup | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!