| mwhahaha | we're working on the fix for that | 00:00 |
|---|---|---|
| *** brault has quit IRC | 00:00 | |
| pabelanger | dsneddon: ya, look at elastic recheck comment | 00:00 |
| pabelanger | that will point you to bug | 00:00 |
| dsneddon | I see, Docker failed to download. Thanks. | 00:00 |
| pabelanger | https://review.openstack.org/491923 and https://review.openstack.org/494021 are the fixes | 00:01 |
| pabelanger | mwhahaha: http://logs.openstack.org/32/491832/2/gate/gate-puppet-tripleo-puppet-syntax-4-centos-7/8d16a46/console.html#_2017-08-15_23_56_06_019235 | 00:03 |
| pabelanger | jobs also using new rubygems reverse proxy cache | 00:03 |
| mwhahaha | yay \o/ | 00:03 |
| pabelanger | won't be much more things to cache | 00:04 |
| pabelanger | BLAM | 00:04 |
| pabelanger | something just hit docker reverse proxy cache | 00:04 |
| pabelanger | 23.253.111.138 | 00:04 |
| pabelanger | now 23.253.90.54 | 00:08 |
| *** snecklifter has quit IRC | 00:09 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 00:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 00:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 00:10 |
| *** snecklifter has joined #tripleo | 00:14 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 00:15 |
| fultonj | mwhahaha: may i get your advice on an issue i am trying to solve? | 00:15 |
| mwhahaha | fultonj: i've got a few minutes, what's up? | 00:16 |
| fultonj | mwhahaha: we have two cent repos in scenario001-multinode-oooq-container | 00:16 |
| fultonj | http://logs.openstack.org/88/479288/32/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/2dd0e52/logs/undercloud/etc/yum.repos.d/ | 00:16 |
| fultonj | one is enabled and one is disabled | 00:16 |
| mwhahaha | so we just went through the process of disabling all repos except the ones defined in oooq | 00:17 |
| fultonj | as a result of this a job to install ceph-ansible fails with... | 00:17 |
| fultonj | Repository centos-ceph-jewel is listed more than once in the configuration | 00:17 |
| fultonj | yes, i saw that | 00:17 |
| mwhahaha | the disabled ones are the ones coming from the image | 00:17 |
| mwhahaha | the enabled ones are the ones in oooq | 00:17 |
| fultonj | sounds reasonable | 00:17 |
| fultonj | but they conflict from having two configs so i can't use either one | 00:18 |
| mwhahaha | what do you mean | 00:18 |
| fultonj | if your /etc/yum.repos.d matches what we set up | 00:18 |
| fultonj | http://logs.openstack.org/88/479288/32/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/2dd0e52/logs/undercloud/etc/yum.repos.d/ | 00:19 |
| fultonj | e.g. if you start a centos box and populate it as ^ | 00:19 |
| mwhahaha | so i think the problem is they are named the same | 00:19 |
| mwhahaha | we should renme centos-ceph-jewel | 00:19 |
| fultonj | then you cannot install ceph-ansible | 00:19 |
| mwhahaha | or match cases | 00:19 |
| mwhahaha | where does ceph-ansible come from? | 00:19 |
| mwhahaha | also what's the error | 00:19 |
| fultonj | my comments in https://review.openstack.org/#/c/479288/32 have the details | 00:20 |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 00:20 |
| fultonj | mwhahaha: can you see my last comment? in that review? | 00:20 |
| fultonj | i now want to add a fix to get this job working | 00:21 |
| fultonj | but wanted your advice on an approach | 00:21 |
| pabelanger | I noticed that too, repo setup is not using case sensative for repo file, so some get duplicated | 00:21 |
| mwhahaha | fix the names in oooq | 00:21 |
| pabelanger | but 1 is enabled other disabled | 00:21 |
| mwhahaha | to match the rpm provide ones | 00:21 |
| mwhahaha | stupid cases | 00:21 |
| pabelanger | +1 | 00:21 |
| EmilienM | pabelanger: back online, so you said docker proxy works now? | 00:22 |
| pabelanger | EmilienM: yes, I see traffic | 00:23 |
| pabelanger | just waiting for some jobs to pass | 00:23 |
| *** catintheroof has joined #tripleo | 00:23 | |
| EmilienM | ok good | 00:23 |
| mwhahaha | fultonj: https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/consistent-master.yml#L38-L79 | 00:23 |
| mwhahaha | fultonj: make the file names match the rpm versions | 00:23 |
| EmilienM | pabelanger: from https://review.openstack.org/#/c/491923/ ? | 00:23 |
| pabelanger | then might promote https://review.openstack.org/491923 | 00:23 |
| fultonj | So, CentOS-Ceph-Jewel.repo is on the image and whatever sets up centos-ceph-jewel.repo should have made its changes to CentOS-Ceph-Jewel.repo ? | 00:23 |
| pabelanger | EmilienM: https://review.openstack.org/494021 is the actual fix | 00:23 |
| mwhahaha | fultonj: if we had done that we wouldn't have neede dto disable anything so yea | 00:24 |
| pabelanger | needed to spin new images | 00:24 |
| EmilienM | pabelanger: right but we need https://review.openstack.org/#/c/491923/ as well for tripleo ci, no? | 00:24 |
| pabelanger | EmilienM: yes, that is in gate pipeline now | 00:24 |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 00:24 |
| EmilienM | in the check I think | 00:24 |
| fultonj | mwhahaha: ok, thanks. I'll set up that fix. | 00:25 |
| pabelanger | EmilienM: and gate... hmm | 00:25 |
| pabelanger | did somebody recheck | 00:25 |
| pabelanger | will keep and eye on it | 00:26 |
| pabelanger | EmilienM: Oh, I know why. I promoted 461000,32, to kick off new jobs. zuul pulled it back in | 00:29 |
| mwhahaha | fultonj: alternatively change the names of the repos to include quickstart in the front | 00:29 |
| pabelanger | EmilienM: trying to warm up the reverse proxy cache for docker.io | 00:29 |
| mwhahaha | fultonj: i think the problem is the way they are getting read since they are duplicated in the name that when yum reads all the files it's merging them together in an odd fashion | 00:29 |
| EmilienM | pabelanger: cool | 00:30 |
| pabelanger | EmilienM: so, fingers crossed this is the last large thing needed for reverse proxies | 00:30 |
| EmilienM | pabelanger: I'll be online and do recheck in case | 00:31 |
| EmilienM | stevebaker as well in case, maybe we can point the patches to recheck if they fail | 00:31 |
| stevebaker | pabelanger, EmilienM: yep, let me know if there is anything I should nurse through the gate | 00:33 |
| EmilienM | stevebaker: https://review.openstack.org/#/c/491923/ | 00:33 |
| pabelanger | k, mirror.ord.rax.o.o is primed | 00:33 |
| pabelanger | took 25mins to cache | 00:33 |
| pabelanger | waiting for next job to hit to see how long it takes ansible to run | 00:33 |
| stevebaker | EmilienM: I'll watch it | 00:33 |
| fultonj | mwhahaha: would we be relying on a side-effect for alphabetical sorting of the repo files? e.g. did foo.repo disable the bar repos but quickstart-foo.repo enabled it only because the enable statement would be evaluated later? | 00:34 |
| pabelanger | 3 more merges happening | 00:34 |
| mwhahaha | fultonj: no, i mean not for file name but also for repo name | 00:34 |
| mwhahaha | fultonj: the problem is that we have [centos-ceph-jewel] created in two different files | 00:34 |
| mwhahaha | fultonj: so if we rename it to [quickstart-centos-ceph-jewel] and it i tin quickstart-centos-ceph-jewel.repo we don't have this problem anymore | 00:35 |
| fultonj | mwhahaha: i see, let me try that then in a prposed oooq patch to change the RDO repos in https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/consistent-master.yml#L38-L79 | 00:36 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-specs master: Propose tech-debt policy https://review.openstack.org/494044 | 00:36 |
| fultonj | i'll open a bug too | 00:36 |
| fultonj | thanks | 00:37 |
| mwhahaha | fultonj: imho this is a difficiency in our repo management in ansible | 00:37 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Consolidate deployment in major-upgrade-composable-steps https://review.openstack.org/493499 | 00:39 |
| openstackgerrit | Merged openstack/tripleo-ui master: Upgrade webpack and webpack-dev-server https://review.openstack.org/475866 | 00:39 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Replace enabled languages with excluded languages in UI https://review.openstack.org/473542 | 00:39 |
| fultonj | mwhahaha: I am curious what you mean by that. I get that quickstart enables some respos w/ ansible and what this name conflict is, but is there some general guideline, or lack of, in quickstart or CI? | 00:40 |
| fultonj | for repos | 00:41 |
| mwhahaha | fultonj: we're just using ansible to create files not manage the whole yum configuration | 00:41 |
| fultonj | mwhahaha: i get it | 00:41 |
| mwhahaha | fultonj: unlike in puppet where you can purge your entire yum definition and just declare only have these, we're having to do it in a very ackward way in ansible | 00:41 |
| mwhahaha | so the disable all, add new is because of the file management | 00:42 |
| mwhahaha | it's not nice :/ but it's the only way without rm -rf /etc/yum.repo.d/* which is not nice for operators | 00:42 |
| *** limao has joined #tripleo | 00:43 | |
| fultonj | so it's an issue w/ ansible's yum_module | 00:43 |
| fultonj | or is it that we should stop using shell commands and only use the yum module | 00:43 |
| mwhahaha | probably the later | 00:44 |
| mwhahaha | latter | 00:44 |
| mwhahaha | because we don't use the yum module | 00:44 |
| mwhahaha | either way the fix right now would be stop duplicating the repo name, long term it would probably be better to fully use ansible rather than shell scripts for system related configs | 00:45 |
| fultonj | yeah, makes sense. | 00:45 |
| mwhahaha | for tripleo stuff it might make sense for shell scripts | 00:46 |
| mwhahaha | but yum configs, not so much | 00:46 |
| *** cshastri has joined #tripleo | 00:48 | |
| pabelanger | wow, 7 incoming merges | 00:51 |
| pabelanger | like we are doing things right | 00:51 |
| *** stendulker has joined #tripleo | 00:52 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: enable container periodic job https://review.openstack.org/475743 | 00:56 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Use images created by periodic pike job in pike configs https://review.openstack.org/492200 | 00:57 |
| openstackgerrit | Merged openstack/tripleo-ui master: MistralApiService, SwiftApiService error handling https://review.openstack.org/484382 | 00:57 |
| *** tzumainn has quit IRC | 00:57 | |
| openstackgerrit | Merged openstack/tripleo-ui master: IronicApiService error handling https://review.openstack.org/486976 | 00:57 |
| openstackgerrit | Merged openstack/tripleo-ui master: IronicInspectorApiService error handling https://review.openstack.org/486977 | 00:57 |
| openstackgerrit | Merged openstack/tripleo-ui master: HeatApiService error handling https://review.openstack.org/486978 | 00:57 |
| openstackgerrit | Merged openstack/tripleo-ui master: KeystoneApiService error handling https://review.openstack.org/486979 | 00:57 |
| pabelanger | nice | 00:57 |
| EmilienM | yeah | 00:57 |
| EmilienM | though these jobs don't run a lot of container jobs | 00:57 |
| EmilienM | sorry | 00:58 |
| EmilienM | these patches* | 00:58 |
| *** artom has quit IRC | 00:58 | |
| pabelanger | Ya, still a few more until 491923 | 00:58 |
| pabelanger | 90mins out if things keep passing | 00:58 |
| *** dixiaoli has joined #tripleo | 01:01 | |
| *** dixiaoli_ has joined #tripleo | 01:02 | |
| *** dixiaoli has quit IRC | 01:02 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 01:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 01:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 01:10 |
| *** number80 has quit IRC | 01:10 | |
| *** gkadam has quit IRC | 01:28 | |
| *** yamahata has quit IRC | 01:35 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Convert network templates to be rendered via j2 https://review.openstack.org/492218 | 01:38 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Bind mount tripleo.cnf in transient bootstrap containers https://review.openstack.org/492963 | 01:38 |
| *** catintheroof has quit IRC | 01:47 | |
| *** gbarros has joined #tripleo | 01:48 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: overcloud_containers.yaml.j2 map images to services https://review.openstack.org/448328 | 02:01 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: upload containers to undercloud in upgrade scenario https://review.openstack.org/493972 | 02:03 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Make env variables for multinode into ansible vars https://review.openstack.org/483930 | 02:04 |
| EmilienM | trown|outtypewww: rebased ^ | 02:04 |
| *** stendulker has quit IRC | 02:07 | |
| *** trozet has quit IRC | 02:08 | |
| *** cdearborn has quit IRC | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 02:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| *** trozet has joined #tripleo | 02:10 | |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 02:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| *** bkopilov has quit IRC | 02:11 | |
| *** trozet has quit IRC | 02:17 | |
| *** trozet has joined #tripleo | 02:20 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-quickstart master: Rename centos repositories created by quickstart https://review.openstack.org/494056 | 02:30 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 02:32 |
| *** yamahata has joined #tripleo | 02:34 | |
| *** trozet has quit IRC | 02:38 | |
| *** ramishra has quit IRC | 02:44 | |
| *** ramishra has joined #tripleo | 02:46 | |
| *** jmelvin has quit IRC | 02:46 | |
| *** catintheroof has joined #tripleo | 02:50 | |
| itlinux | has anyone seen this one http://paste.openstack.org/show/618483/ ?? TY | 02:53 |
| *** catintheroof has quit IRC | 03:03 | |
| *** myoung|remote has joined #tripleo | 03:09 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 03:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 03:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| *** myoung|remote has quit IRC | 03:10 | |
| *** eck` is now known as eck`gone | 03:19 | |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart-extras master: Display output from machine provisioning script https://review.openstack.org/493023 | 03:28 |
| *** stendulker has joined #tripleo | 03:28 | |
| *** bkopilov has joined #tripleo | 03:30 | |
| *** psahoo has joined #tripleo | 03:39 | |
| *** udesale has joined #tripleo | 03:44 | |
| *** links has joined #tripleo | 03:47 | |
| EmilienM | stevebaker: https://review.openstack.org/#/c/448328/ reviewed | 03:48 |
| *** ykarel_ has joined #tripleo | 03:49 | |
| EmilienM | michapma: I was rebasing https://review.openstack.org/#/c/486905/ on top of a THT patch that was supposed to help | 03:50 |
| EmilienM | michapma: not sure why you rebased on master | 03:50 |
| EmilienM | mwhahaha: I rebased it on https://review.openstack.org/#/c/492027/ before | 03:50 |
| stevebaker | EmilienM: thanks. that change is mostly from a script which parses THT, so I'll dig in to see if it is doing the right thing | 03:50 |
| EmilienM | stevebaker: oh, ok | 03:51 |
| EmilienM | I'm out | 03:52 |
| *** shreshtha has joined #tripleo | 03:53 | |
| *** ramishra has quit IRC | 03:55 | |
| *** ramishra has joined #tripleo | 03:57 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml https://review.openstack.org/493984 | 04:00 |
| *** gbarros has quit IRC | 04:06 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 04:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 04:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| *** dtrainor has quit IRC | 04:11 | |
| *** dtrainor has joined #tripleo | 04:11 | |
| *** brault has joined #tripleo | 04:16 | |
| *** brault has quit IRC | 04:21 | |
| *** limao has quit IRC | 04:22 | |
| *** nyechiel has joined #tripleo | 04:44 | |
| *** brault has joined #tripleo | 04:46 | |
| *** brault has quit IRC | 04:50 | |
| *** limao has joined #tripleo | 05:01 | |
| *** janki has joined #tripleo | 05:03 | |
| *** pdeore has joined #tripleo | 05:06 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Remove extra keystone admin haproxy listen and allow TLS https://review.openstack.org/493937 | 05:07 |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 05:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 05:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| *** anshul has joined #tripleo | 05:19 | |
| *** rbrady has quit IRC | 05:22 | |
| *** rbrady has joined #tripleo | 05:22 | |
| *** rbrady has joined #tripleo | 05:22 | |
| *** pdeore is now known as pdeore|afk | 05:23 | |
| *** yprokule has joined #tripleo | 05:24 | |
| *** mdnadeem has joined #tripleo | 05:27 | |
| *** udesale__ has joined #tripleo | 05:35 | |
| *** udesale has quit IRC | 05:35 | |
| *** mdnadeem has quit IRC | 05:37 | |
| *** agurenko has joined #tripleo | 05:41 | |
| *** brault has joined #tripleo | 05:44 | |
| *** pdeore|afk is now known as pdeore | 05:44 | |
| *** pcaruana has joined #tripleo | 05:45 | |
| *** masco has joined #tripleo | 05:46 | |
| *** brault has quit IRC | 05:48 | |
| *** mdnadeem has joined #tripleo | 05:49 | |
| *** pgadiya has joined #tripleo | 05:50 | |
| *** iranzo has joined #tripleo | 05:56 | |
| *** marios has joined #tripleo | 06:00 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 06:00 |
| *** dpawar has joined #tripleo | 06:00 | |
| *** iranzo1 has joined #tripleo | 06:01 | |
| *** iranzo has quit IRC | 06:01 | |
| *** dsariel has joined #tripleo | 06:02 | |
| *** iranzo1 has quit IRC | 06:03 | |
| *** iranzo has joined #tripleo | 06:03 | |
| *** iranzo has joined #tripleo | 06:03 | |
| *** jbadiapa has joined #tripleo | 06:04 | |
| *** brault has joined #tripleo | 06:05 | |
| *** iranzo has quit IRC | 06:09 | |
| *** brault has quit IRC | 06:10 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 06:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 06:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| *** skramaja has joined #tripleo | 06:12 | |
| *** iranzo has joined #tripleo | 06:15 | |
| *** iranzo has quit IRC | 06:15 | |
| *** iranzo has joined #tripleo | 06:15 | |
| *** jfrancoa has joined #tripleo | 06:20 | |
| *** udesale has joined #tripleo | 06:23 | |
| *** udesale__ has quit IRC | 06:23 | |
| *** dciabrin has joined #tripleo | 06:25 | |
| *** iranzo has quit IRC | 06:28 | |
| *** iranzo has joined #tripleo | 06:28 | |
| *** iranzo has joined #tripleo | 06:28 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Certmonger: Make postsave command configurable https://review.openstack.org/494085 | 06:29 |
| *** limao has quit IRC | 06:29 | |
| *** limao has joined #tripleo | 06:29 | |
| *** bogdando has joined #tripleo | 06:31 | |
| openstackgerrit | Itzik Brown proposed openstack/tripleo-heat-templates master: Adding a Missing configuration to support QoS in ODL Closes-Bug: 1708131 https://review.openstack.org/492027 | 06:31 |
| openstack | bug 1708131 in tripleo "AttributeError: 'NoneType' object has no attribute 'get_policies'" [High,In progress] https://launchpad.net/bugs/1708131 - Assigned to Emilien Macchi (emilienm) | 06:31 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Certmonger: Make postsave command configurable https://review.openstack.org/494085 | 06:33 |
| *** udesale has quit IRC | 06:35 | |
| skramaja | d0ugal: could you take a look at https://review.openstack.org/#/c/491753/? | 06:36 |
| *** udesale has joined #tripleo | 06:36 | |
| *** florianf has joined #tripleo | 06:37 | |
| *** cylopez has joined #tripleo | 06:39 | |
| *** aufi has joined #tripleo | 06:40 | |
| *** paramite has joined #tripleo | 06:41 | |
| *** jaganathan has joined #tripleo | 06:42 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Use undercloud docker proxy https://review.openstack.org/491923 | 06:42 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Don't unregister on system/resource delete https://review.openstack.org/492970 | 06:43 |
| *** stendulker has quit IRC | 06:44 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Update updatePlan action https://review.openstack.org/489196 | 06:46 |
| *** pgadiya has quit IRC | 06:54 | |
| *** jprovazn has joined #tripleo | 06:55 | |
| *** rcernin has joined #tripleo | 06:57 | |
| honza | I'm running into nokogiri build issues in the puppet syntax job, is that a known thing? http://logs.openstack.org/08/469608/8/check/gate-instack-undercloud-puppet-syntax-4-centos-7/a3ccc18/console.html#_2017-08-16_04_24_27_928151 | 07:05 |
| *** pgadiya has joined #tripleo | 07:07 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 07:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 07:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| *** shardy_afk is now known as shardy | 07:10 | |
| *** ebarrera has joined #tripleo | 07:13 | |
| honza | jtomasek: do we have a bug for the CORS issue with PATCH in ironic? which services are affected? is there a repro case? | 07:13 |
| apetrich | honza, https://review.openstack.org/#/c/469608/ is passing and ready for reviews :) | 07:14 |
| apetrich | a lot to merge to get that in though | 07:14 |
| honza | apetrich: passing? what about that weird nokogiri issue? | 07:15 |
| * apetrich needs coffee.. | 07:15 | |
| apetrich | honza, sorry | 07:15 |
| honza | apetrich: no, no, maybe that's fine | 07:15 |
| *** rcernin has quit IRC | 07:16 | |
| apetrich | honza, oh that's the coverage that has been acting up | 07:16 |
| *** number80 has joined #tripleo | 07:17 | |
| shardy | Hi all, can I get a second reviewer for https://review.openstack.org/#/c/490839 please, thanks! | 07:18 |
| *** ccamacho has joined #tripleo | 07:18 | |
| apetrich | honza, sorry I was looking into the wrong review. I WILL GET MORE COFFEE. bbiam | 07:18 |
| *** jlinkes has joined #tripleo | 07:18 | |
| honza | apetrich: i linked directly to the error above | 07:19 |
| *** jpena|off is now known as jpena | 07:19 | |
| shardy | coffee++ :) | 07:19 |
| *** rcernin has joined #tripleo | 07:19 | |
| honza | if that helps with the uncaffeinated state | 07:19 |
| * apetrich curses libxml for the first time this month | 07:20 | |
| *** brault has joined #tripleo | 07:25 | |
| bogdando | folks, do we need this https://review.openstack.org/#/c/492178/ ? I can use it from my fork as well | 07:26 |
| bogdando | I hoped it helps to tshoot failed CI jobs' logs | 07:27 |
| bogdando | please try it, if you do :) | 07:27 |
| *** jpich has joined #tripleo | 07:28 | |
| jtomasek | honza: I don't see it in launchpad, so it has not been filed yet, I can do so | 07:29 |
| bogdando | please review the logrotate for contnainers' logs topic https://review.openstack.org/#/q/topic:bug/1700912 | 07:29 |
| honza | jtomasek: please and thank you :) | 07:29 |
| shardy | bogdando: I added derekh to the getthelogs one, as AFAICS he's the only person who's ever touched that script before | 07:29 |
| apetrich | back | 07:30 |
| bogdando | shardy: hm, I hoped more ppl are used to grep through the CI jobs logs | 07:30 |
| bogdando | but thanks shardy! :) | 07:30 |
| shardy | bogdando: maybe, but it'd be good to get derek to take a look anyway I think | 07:30 |
| shardy | I'll try to review later, but tbh I don't use that script locally | 07:30 |
| bogdando | shardy: how do you normally inspect logs of failed jobs? | 07:31 |
| bogdando | I need more feedback from folks, the script may help with that task | 07:31 |
| shardy | bogdando: just using the web ui and knowledge of which files to look at | 07:31 |
| bogdando | yeah, right. That works :) | 07:31 |
| shardy | I can see that grep would be useful sometimes, but I've only occasionally needed to do that | 07:32 |
| *** shardy is now known as shardy_afk | 07:32 | |
| bogdando | shardy: it really starts to be needed, then debugging distributed (HA) issues ;) | 07:32 |
| bogdando | and sorted in time | 07:32 |
| bogdando | so that's the only reason I started to download logs then apply filters like I track here https://github.com/bogdando/fuel-log-parse | 07:33 |
| bogdando | otherwise, it's hard to get a full view of pacemaker cluster distributed events | 07:33 |
| bogdando | but for non HA jobs, it's ok to just look into web ui, agree | 07:34 |
| bogdando | s/then/when debugging* | 07:34 |
| d0ugal | skramaja: sure | 07:37 |
| jtomasek | honza: https://bugs.launchpad.net/tripleo/+bug/1711061 | 07:37 |
| openstack | Launchpad bug 1711061 in tripleo "ironic cors don't allow PATCH method" [High,Triaged] | 07:37 |
| skramaja | thanks d0ugal | 07:37 |
| honza | jtomasek: excellent bug report, thanks | 07:37 |
| *** oidgar has joined #tripleo | 07:40 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 07:40 |
| *** brault has quit IRC | 07:40 | |
| *** egonzalez has joined #tripleo | 07:41 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Add logrotate-crond configuration https://review.openstack.org/490048 | 07:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make swift's endpoint type configurable for gnocchi storage https://review.openstack.org/494103 | 07:47 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Persist containerized services httpd logs https://review.openstack.org/489955 | 07:48 |
| *** brault has joined #tripleo | 07:49 | |
| bogdando | this one https://review.openstack.org/#/c/489955 was ready for merge, please take care of it once it passes CI | 07:54 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 07:55 |
| oidgar | hi everybody, I'm still struggling to deploy overcloud with brand new undercloud server | 07:59 |
| oidgar | when I'm running "openstack overcloud deploy --templates" it fails on "overcloud.AllNodesDeploySteps.ComputeDeployment_Step4.0" | 07:59 |
| oidgar | I'm getting in addition "[overcloud]: CREATE_FAILED Create timed out" | 08:00 |
| jaosorior | oidgar: what's the output of openstack stack failures list --long overcloud ? | 08:00 |
| oidgar | jaosorior: nothing | 08:00 |
| oidgar | stdout and stderr are empty | 08:00 |
| jaosorior | uhm... so it did fail on the timeout | 08:00 |
| jaosorior | oidgar: I guess the only way to check the controller nodes' logs and check what's taking so long | 08:01 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat] https://review.openstack.org/491399 | 08:01 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 08:01 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral] https://review.openstack.org/493471 | 08:01 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer] https://review.openstack.org/490392 | 08:01 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron] https://review.openstack.org/489957 | 08:01 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi] https://review.openstack.org/491433 | 08:01 |
| *** dparkes has joined #tripleo | 08:01 | |
| oidgar | jaosorior: which logs do you recommend to check? | 08:01 |
| oidgar | jaosorior: and I think the problem is with the compute node | 08:01 |
| oidgar | jaosorior: also I get "Permission denied" when trying to connect with heat-admin user... but I changed the ssh key before deploying overcloud so the question is if this can affect | 08:04 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Add support for openstack inventory provider https://review.openstack.org/463701 | 08:04 |
| jaosorior | oidgar: I'm honestly not sure if that would be a problem. Maybe? | 08:05 |
| *** lucas-afk is now known as lucasagomes | 08:06 | |
| jaosorior | Hey guys, could someone check this puppet-tripleo change out https://review.openstack.org/#/c/493937/ ? | 08:06 |
| oidgar | jaosorior: I guess I'll try to reinstall again (in the 100th time this week) the undercloud vm to make sure it's sterile... | 08:06 |
| jaosorior | Also, any reviews for this series is appreciated https://review.openstack.org/#/q/topic:bp/tls-via-certmonger-containers+status:open | 08:06 |
| marios | jfrancoa: o/ hey did you drop api accidentally here https://review.openstack.org/#/c/491433/7 | 08:07 |
| michapma | EmilienM: I didn't realise you'd rebased on something in flight - I only meant to add the depends-on to get the ODL log | 08:07 |
| oidgar | jaosorior: thanks for the help anyway, let's see if it work after this small change | 08:08 |
| jfrancoa | marios: o/ I did understand from your comment that it was not necessary (remove this?) | 08:08 |
| marios | jfrancoa: oh i meant the highlighted bit https://review.openstack.org/#/c/491433/6/docker/services/gnocchi-api.yaml ("post upgrade") | 08:09 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 08:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 08:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 08:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| *** derekh has joined #tripleo | 08:10 | |
| marios | jfrancoa: sorry for that, i do think it still makes sense to have a check it *is* under httpd before stopping there. Its just that the 'post upgrade' part was confusing (I think it was copied from the puppet/services/ one possibly?) since the 'upgrade to httpd gnochi api ' happened last cycle | 08:10 |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Add param to configure snat mechanism https://review.openstack.org/493861 | 08:11 |
| jfrancoa | marios: ups... :-S yes, I took it from the cinder-api.yaml service. I want to wait for CI results anyway, because this was the only commit from bug/1704389 which got a -1 and I was wondering if that step could be the problem | 08:12 |
| jfrancoa | marios: I will add the step back once I check the CI jobs | 08:12 |
| marios | jfrancoa: thanks, added comment at https://review.openstack.org/#/c/491433/6 as well | 08:13 |
| jfrancoa | marios: good, thanks! | 08:14 |
| *** jlinkes has quit IRC | 08:14 | |
| marios | jfrancoa: added another note at https://review.openstack.org/#/c/491433/6/docker/services/gnocchi-api.yaml :) when you get a chance thanks | 08:17 |
| *** shardy_afk is now known as shardy | 08:17 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Containarise Barbican API https://review.openstack.org/481451 | 08:20 |
| *** mcornea has joined #tripleo | 08:21 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-heat-templates master: Adding a Missing configuration to support QoS in ODL Closes-Bug: 1708131 https://review.openstack.org/492027 | 08:21 |
| openstack | bug 1708131 in tripleo "AttributeError: 'NoneType' object has no attribute 'get_policies'" [High,In progress] https://launchpad.net/bugs/1708131 - Assigned to Itzik Brown (itzikb1) | 08:21 |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 08:21 |
| *** gfidente has joined #tripleo | 08:23 | |
| *** gfidente has quit IRC | 08:23 | |
| *** gfidente has joined #tripleo | 08:23 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 08:24 |
| *** achadha has joined #tripleo | 08:24 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Don't log app state in Zaqar https://review.openstack.org/493863 | 08:24 |
| *** ramishra has quit IRC | 08:24 | |
| *** jlinkes has joined #tripleo | 08:25 | |
| *** amoralej|off is now known as amoralej | 08:25 | |
| achadha | any changes with pacemaker/corosync in 7.4? I see deployments fail coz corosync-cluster doesn't form? Started seeing this only after moving to 7.4 (everything worked fine in 7.3) | 08:25 |
| *** ramishra has joined #tripleo | 08:27 | |
| *** chem has joined #tripleo | 08:27 | |
| openstackgerrit | Itzik Brown proposed openstack/tripleo-heat-templates master: Adding a Missing configuration to support QoS in ODL https://review.openstack.org/492027 | 08:29 |
| openstackgerrit | Merged openstack/tripleo-validations master: Fix `when` statement in deployment image validation https://review.openstack.org/482979 | 08:34 |
| *** dixiaoli has joined #tripleo | 08:34 | |
| *** dixiaoli has quit IRC | 08:36 | |
| *** dixiaoli_ has quit IRC | 08:38 | |
| *** ykarel_ is now known as ykarel|lunch | 08:48 | |
| *** jbadiapa_ has joined #tripleo | 08:50 | |
| *** jbadiapa has quit IRC | 08:51 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/494127 | 08:53 |
| *** salmankhan has joined #tripleo | 08:57 | |
| *** hewbrocca_afk is now known as hewbrocca | 08:58 | |
| *** psahoo has quit IRC | 08:59 | |
| *** stendulker has joined #tripleo | 09:05 | |
| *** psahoo has joined #tripleo | 09:05 | |
| *** lucasagomes is now known as lucas-brb | 09:08 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Fix plan export URL https://review.openstack.org/488164 | 09:09 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: App config and I18n selectors refactor https://review.openstack.org/494141 | 09:09 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 09:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 09:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| *** milan has joined #tripleo | 09:13 | |
| *** dixiaoli has joined #tripleo | 09:14 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Allow users to specify the Introspection batch size https://review.openstack.org/451228 | 09:15 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Allow users to specify the Introspection concurrency https://review.openstack.org/451228 | 09:18 |
| social | marios: re updates and --update-plan-only vs update_noop.yaml | 09:18 |
| marios | social: o/ | 09:21 |
| *** limao has quit IRC | 09:22 | |
| jaosorior | bandini: got a chance to check this out https://review.openstack.org/#/c/493561/ ? | 09:24 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Switch to overcloudrc.v3 for running tempest https://review.openstack.org/493030 | 09:25 |
| social | marios: so our issue is that update-plan-only didn't update any parameters and the update_tasks didn't get created in heat so it's kinda useless? | 09:25 |
| marios | social: in a call right now (join us?) | 09:25 |
| social | marios: on call with lyan | 09:26 |
| *** tosky has joined #tripleo | 09:26 | |
| social | *lee | 09:26 |
| marios | social: ack | 09:31 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml https://review.openstack.org/493984 | 09:34 |
| *** achadha has quit IRC | 09:35 | |
| *** deadnull has joined #tripleo | 09:36 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat] https://review.openstack.org/491399 | 09:39 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 09:39 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral] https://review.openstack.org/493471 | 09:39 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi] https://review.openstack.org/491433 | 09:39 |
| *** deadnull has quit IRC | 09:41 | |
| jfrancoa | marios: does this make sense? https://review.openstack.org/#/c/491399/8/docker/services/heat-api.yaml | 09:41 |
| openstackgerrit | Merged openstack/diskimage-builder master: Increase timeout for removal https://review.openstack.org/493026 | 09:45 |
| *** dixiaoli has quit IRC | 09:45 | |
| *** ykarel|lunch is now known as ykarel | 09:46 | |
| *** cshastri has quit IRC | 09:47 | |
| *** dixiaoli has joined #tripleo | 09:48 | |
| sshnaidm | jaosorior, hi | 09:49 |
| sshnaidm | jaosorior, I prepare white/exclude lists for tripleo jobs logs and I'd like to ensure I include all required logs from IPA job | 09:50 |
| sshnaidm | jaosorior, sorry, not logs - /etc configs | 09:50 |
| sshnaidm | jaosorior, I've added /etc/ipa, /etc/sssd. Something else..? Do you need /etc/gss /etc/gssproxy, /etc/gnupg ? | 09:51 |
| *** akrivoka has joined #tripleo | 09:51 | |
| jaosorior | sshnaidm: not really | 09:55 |
| jaosorior | but /etc/ipa and /etc/sssd are good :) | 09:55 |
| sshnaidm | jaosorior, and all /etc/krb5.* ? | 09:55 |
| jaosorior | sshnaidm: oh, that would be good too. Not the keytab file though | 09:56 |
| sshnaidm | jaosorior, ok | 09:57 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test undercloud containers https://review.openstack.org/494151 | 10:01 |
| *** snecklifter is now known as snecklifter|away | 10:01 | |
| *** dixiaoli has quit IRC | 10:03 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: Set resource_class=baremetal for newly enrolled nodes https://review.openstack.org/493943 | 10:05 |
| marios | jfrancoa: yes it lgtm but agree on checking running | 10:06 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Switch to scheduling based on resource classes https://review.openstack.org/490851 | 10:06 |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| dtantsur | bnemec: the ordering should be good now ^^^ | 10:07 |
| skramaja | dsneddon: i think the templates for dpdk shoul be separate given the fact that there should not be mixing of ovs and ovs_user bridges. i have left a comment on https://review.openstack.org/#/c/474533/, ptal | 10:07 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 10:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 10:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| *** dsariel has quit IRC | 10:10 | |
| *** dtrainor has quit IRC | 10:11 | |
| *** cylopez has quit IRC | 10:16 | |
| chem | marios: could we land https://review.openstack.org/#/c/491799/ ? | 10:16 |
| *** lucas-brb is now known as lucasagomes | 10:18 | |
| marios | chem: looking | 10:20 |
| *** yamahata has quit IRC | 10:21 | |
| chem | marios: thanks (and this one for the road as well https://review.openstack.org/#/c/493825/ ...) | 10:22 |
| marios | chem: done, i guess it needs to go to stable/ocata too (from the bug). i was initially confused thought it was clone for bug https://bugs.launchpad.net/tripleo/+bug/1706951 but it is newton.. ocata in this case | 10:23 |
| openstack | Launchpad bug 1706951 in tripleo "Ocata to Pike upgrade fails when cinder-volume runs on host because cinder-manage db sync runs when galera is unavailable" [Critical,In progress] - Assigned to Marios Andreou (marios-b) | 10:23 |
| *** snecklifter|away is now known as snecklifter | 10:24 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs https://review.openstack.org/493973 | 10:25 |
| chem | marios: yeap | 10:25 |
| marios | chem: k voted on the second one too, just wondering is there a case where we don't have polling but have compute? | 10:27 |
| *** chandankumar is now known as chkumar|travel | 10:27 | |
| chem | marios: legit question | 10:28 |
| chem | lyarwood: hi, regarding https://review.openstack.org/#/c/493825/ are we sure that polling is always there or do we need to check ? | 10:29 |
| *** udesale has quit IRC | 10:29 | |
| *** jbadiapa_ has quit IRC | 10:30 | |
| *** jbadiapa_ has joined #tripleo | 10:31 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Exclude list for logs collection https://review.openstack.org/494022 | 10:32 |
| *** atoth has quit IRC | 10:33 | |
| lyarwood | chem: looking | 10:34 |
| gfidente | flaper87 EmilienM I am unable to reproduce the image upload issue in my env, that is, it works for me | 10:35 |
| lyarwood | chem: I think it's the other way around, polling should always be there now and compute might not be iirc | 10:35 |
| lyarwood | chem: I'll add a comment with links | 10:36 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs https://review.openstack.org/493973 | 10:37 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Remove unused docker-centos-tripleoupstream.yaml https://review.openstack.org/494158 | 10:39 |
| *** florianf has quit IRC | 10:42 | |
| chem | lyarwood: thanks :) | 10:44 |
| lyarwood | chem: done, so yeah we need to check both really | 10:44 |
| lyarwood | chem: we moved to polling in Pike, so anyone moving from BM to containers (a useful test case) would hit this | 10:45 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Stop hardcoding host's config volume path https://review.openstack.org/494163 | 10:47 |
| chem | lyarwood: oki, cool, and thanks marios, he's the one who spotted it :) | 10:48 |
| lyarwood | marios++ | 10:48 |
| marios | thanks chem lyarwood | 10:48 |
| *** stendulker has quit IRC | 10:49 | |
| *** dparkes has quit IRC | 10:49 | |
| *** bkopilov has quit IRC | 10:50 | |
| gfidente | flaper87 you around? | 10:52 |
| gfidente | flaper87 I think I need your skills here | 10:52 |
| gfidente | :D | 10:52 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Adds PostUpgradeConfigStepsDeployment to drive post config ansible https://review.openstack.org/493878 | 10:53 |
| marios | shardy: had a go at making it a variable instead of overriding tags when you get another chance thanks ^^ | 10:53 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: docker: Stop all active ceilometer services during compute upgrade https://review.openstack.org/493825 | 10:53 |
| *** jkilpatr has quit IRC | 10:54 | |
| shardy | marios: hey, thanks - I wasn't blocking on it btw, just wanted to have the discussion | 10:54 |
| shardy | marios: seems like the tags are proving a headache for running outside heat, so I was thinking we'd move away from them in favor of vars | 10:55 |
| shardy | will check it out | 10:55 |
| marios | shardy: i kinda like it this way and it wasn't too hard to change so I thought i'd post it for discussion. | 10:55 |
| shardy | marios: ack, thanks, will review | 10:55 |
| marios | shardy: thank you | 10:55 |
| shardy | I still kinda wonder if we should just sed all the things and convert t-h-t to variables instead of tags | 10:55 |
| marios | shardy: it is a bit risk though i mean given everything else going on | 10:55 |
| shardy | marios: yeah, I guess so | 10:55 |
| marios | shardy: minor update, update non controller, then a list of blocking bugs like the cinder db sync that this change is addressing | 10:56 |
| shardy | maybe we can find a way to transition to vars in early queens tho | 10:56 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: docker: Stop all active ceilometer services during compute upgrade https://review.openstack.org/493825 | 10:56 |
| *** florianf has joined #tripleo | 10:57 | |
| *** jpena is now known as jpena|lunch | 11:05 | |
| trown|outtypewww | EmilienM: that rebase is incorrect... you undid part of my change... can we please stop merging conflicting patches ahead of the 3nodes patches with no tripleo-ci core +2's on them? | 11:07 |
| chem | lyarwood: maybe we should add the check only on step2, unless I miss something ? | 11:08 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 11:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| *** lucasagomes is now known as lucas-hungry | 11:10 | |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 11:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: docker: Stop all active ceilometer services during compute upgrade https://review.openstack.org/493825 | 11:10 |
| lyarwood | chem: ack yeaaaah, not sure why I left it as common, thanks | 11:10 |
| gfidente | jprovazn is it possible the problem you found in https://review.openstack.org/482496 is https://bugs.launchpad.net/tripleo/+bug/1710773 ? | 11:17 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org) | 11:17 |
| * jprovazn looks | 11:18 | |
| jprovazn | gfidente: thanks for pointing this out, I'm trying to reproduce this since this morning | 11:20 |
| gfidente | jprovazn yeah except that I was unable to | 11:20 |
| gfidente | that config works for me | 11:20 |
| jprovazn | gfidente: well, my first thought is that it's not directly caused by containerized glance | 11:20 |
| jprovazn | gfidente: because I'm hitting this in CI on scenario001-multinode-oooq too | 11:20 |
| *** dsariel has joined #tripleo | 11:20 | |
| gfidente | that is using ceph too | 11:20 |
| gfidente | see the bug title | 11:21 |
| jprovazn | gfidente: but not containerized, right? | 11:21 |
| gfidente | ah non-containerized right | 11:21 |
| jprovazn | gfidente: yea, but what I did actualyl find | 11:21 |
| gfidente | I think it says writes are currently waiting for active PGs | 11:22 |
| jprovazn | gfidente: if I omitted pooldefaultsize to set it to 1 (and left 3), then stack deployed but ceph cluster was unhappy and tempest was failing with this issue - uploading iamge to glance | 11:22 |
| gfidente | right yes | 11:22 |
| gfidente | was about to post just that change to try it | 11:22 |
| gfidente | checking the scenario envs :D | 11:22 |
| jprovazn | gfidente: cool | 11:23 |
| jprovazn | gfidente: I was looking at them, but it seems that defaultsize is 1 for 001 | 11:23 |
| gfidente | PGs are degraded because of size if it's not 1 | 11:23 |
| jprovazn | gfidente: but it might be some other env file overwrites the ci/environments/scenario001-multinode.yaml | 11:23 |
| jprovazn | gfidente: I failed to find any logs from overcloud, I guess we don't copy them, do we? | 11:24 |
| gfidente | right though not override the same param I hope | 11:24 |
| gfidente | I see it set in scenario004 as well | 11:26 |
| *** jbadiapa_ has quit IRC | 11:27 | |
| gfidente | jprovazn http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/subnode-2/etc/ceph/ceph.conf.txt.gz pool_default_size is indeed wrong | 11:28 |
| *** jkilpatr has joined #tripleo | 11:29 | |
| *** atoth has joined #tripleo | 11:29 | |
| jprovazn | gfidente: I wonder where does it sneak in? | 11:30 |
| gfidente | yeah | 11:30 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: App config and I18n selectors refactor https://review.openstack.org/494141 | 11:30 |
| *** brault has quit IRC | 11:31 | |
| jprovazn | gfidente: hm, though in my patch where I hit glance error too, I can see that pool_size is set to 1 - http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/3753922/logs/subnode-2/ | 11:35 |
| gfidente | is it the same issue then? :D | 11:35 |
| gfidente | jprovazn not the same it seems http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/3753922/logs/subnode-2/var/log/glance/api.log.txt.gz#_2017-08-15_12_24_39_536 | 11:36 |
| gfidente | jprovazn oh indeed we set it to 1 only in the non-containers | 11:38 |
| jprovazn | gfidente: that would explain it | 11:38 |
| openstackgerrit | Marios Andreou proposed openstack/instack-undercloud master: Adds execution of post-upgrade validations on undercloud upgrade https://review.openstack.org/474578 | 11:39 |
| gfidente | jprovazn not the issue you're seeing though | 11:41 |
| jprovazn | gfidente: ack | 11:41 |
| *** pgadiya has quit IRC | 11:41 | |
| *** jbadiapa_ has joined #tripleo | 11:42 | |
| *** shreshtha has quit IRC | 11:43 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Set default OSD pool size to 1 in scenario 001/004 containers https://review.openstack.org/494176 | 11:43 |
| *** rhallisey has joined #tripleo | 11:43 | |
| gfidente | jprovazn so that one started now but it's weird, looks as if something is happening too early, like the problem you had with the key ownership | 11:43 |
| gfidente | what caused that? | 11:43 |
| *** pkovar has joined #tripleo | 11:43 | |
| jprovazn | gfidente: you mean what caused http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/3753922/logs/subnode-2/var/log/glance/api.log.txt.gz#_2017-08-15_12_24_39_536 ? | 11:44 |
| *** ansmith has quit IRC | 11:53 | |
| *** cylopez has joined #tripleo | 11:55 | |
| *** brault has joined #tripleo | 11:56 | |
| *** artom has joined #tripleo | 11:56 | |
| *** thrash|g0ne is now known as thrash | 11:58 | |
| *** lucas-hungry is now known as lucasagomes | 11:59 | |
| gfidente | jprovazn yeah we didn't change when the pools are created, maybe the ceph logs have more details about that too | 12:00 |
| EmilienM | hello | 12:00 |
| *** morazi has joined #tripleo | 12:00 | |
| shardy | 'morning EmilienM | 12:00 |
| jprovazn | gfidente: I didn't find anything suspect in ceph logs so far, I'll see if I can reproduce with the same 001 env and w/o including manila service (it passes locally with it) | 12:01 |
| *** trown|outtypewww is now known as trown | 12:04 | |
| *** pdeore has quit IRC | 12:05 | |
| *** dtrainor has joined #tripleo | 12:05 | |
| *** rlandy has joined #tripleo | 12:05 | |
| *** abishop has joined #tripleo | 12:06 | |
| *** jcoufal has joined #tripleo | 12:10 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 12:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 12:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| gfidente | EmilienM I think we have a fix for the scenario jobs | 12:11 |
| EmilienM | yeah I saw, cool | 12:11 |
| gfidente | regarding the upgrade CI instead, I think it currently does not test ceph | 12:11 |
| EmilienM | finger crossed | 12:11 |
| EmilienM | thanks btw! | 12:11 |
| gfidente | I would like to add that too instead | 12:11 |
| gfidente | wdyt? | 12:11 |
| fultonj | gfidente: i reproduced the problem, with glance getting the 503, set the new pool size and now I don't get the 503, so i think your fix is good. | 12:12 |
| gfidente | fultonj hey morning there | 12:12 |
| EmilienM | gfidente: any help in upgrades is very welcome so I think: yes please | 12:12 |
| fultonj | Also, https://review.openstack.org/#/c/479288 Convert scenario001-multinode-containers job to ceph-ansible is now green | 12:13 |
| fultonj | hey gfidente | 12:13 |
| gfidente | fultonj right that was great | 12:13 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras master: Make env variables for multinode into ansible vars https://review.openstack.org/483930 | 12:13 |
| gfidente | I am not sure what is happening in oooq with the repos | 12:13 |
| gfidente | trown can probably help reviewing that | 12:13 |
| EmilienM | gfidente: see comment https://review.openstack.org/#/c/479288 | 12:14 |
| fultonj | gfidente: the repos problem is explained in https://bugs.launchpad.net/tripleo/+bug/1711006 | 12:14 |
| openstack | Launchpad bug 1711006 in tripleo "tripleo-quickstart creating redundant repositories" [High,In progress] - Assigned to John Fulton (jfulton-org) | 12:14 |
| fultonj | i dropped in a proposed fix. i then made the ci proposal depend on it | 12:14 |
| fultonj | basically, we had redundant repos conflicting which prevented ceph-ansible from getting installed | 12:14 |
| *** dprince has joined #tripleo | 12:15 | |
| gfidente | fultonj yeah I understand the issue I think, not sure if case is something we should fix in oooq | 12:15 |
| fultonj | mwhahaha suggested remaming them which seemed to solve the issue | 12:15 |
| gfidente | fultonj seems very messy to me | 12:15 |
| gfidente | but I might not know the reasons why | 12:16 |
| rcernin | shardy: we found yesterday the issue why the heat was timing out, there was 3,7s clock offset | 12:16 |
| fultonj | gfidente: it was working and then a change was introduced, to support mirrors, which broke it | 12:16 |
| EmilienM | fultonj, gfidente : another option would be to remove them from the nodepool image | 12:16 |
| EmilienM | but we already disable them, so it's clean as well | 12:16 |
| gfidente | fultonj have a link to change which added support for mirrors? | 12:17 |
| fultonj | gfidente: yes | 12:17 |
| fultonj | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/consistent-master.yml#L38-L79 | 12:17 |
| EmilienM | fultonj: your patch lgtm | 12:17 |
| fultonj | sec | 12:17 |
| fultonj | thanks | 12:17 |
| EmilienM | I checked, all repos use mirrors like before | 12:17 |
| shardy | rcernin: aha, and that broke the RPC? | 12:18 |
| shardy | rcernin: was the ntp service not enabled on the nodes? | 12:18 |
| fultonj | https://github.com/openstack/tripleo-quickstart/commit/84e0168e49fce2f3d79559866ebea8f7050eef51 | 12:18 |
| fultonj | gfidente: that's the link you asked for ^ | 12:18 |
| rcernin | shardy: it was not. there is a bug with chrony and ntpd currently https://bugzilla.redhat.com/show_bug.cgi?id=1413754 | 12:19 |
| openstack | bugzilla.redhat.com bug 1413754 in rhosp-director-images "Chrony and ntpd are present in overcloud-full images" [Medium,Closed: errata] - Assigned to emacchi | 12:19 |
| shardy | rcernin: ah, I see, well good to hear you tracked down the problem :) | 12:19 |
| rcernin | shardy: we did, but now trying to understand why that 3,7s actually matters so much. | 12:19 |
| fultonj | alex suggested quickstart, in the long run, use the yum module and not shell commands to manage repos so it's less messy | 12:19 |
| shardy | rcernin: do you have tls enabled? | 12:20 |
| *** dpawar has quit IRC | 12:21 | |
| *** dpawar has joined #tripleo | 12:22 | |
| rcernin | shardy: no | 12:22 |
| *** sbrzozow has joined #tripleo | 12:23 | |
| gfidente | EmilienM fultonj yeah but I don't understand the attempt to reuse same names | 12:23 |
| gfidente | in lowercase | 12:23 |
| gfidente | though now I understand better :D | 12:23 |
| gfidente | thanks :D | 12:23 |
| numans | EmilienM, can you please see if this fine with you - https://review.openstack.org/#/c/468860/. Once this scenario007 container job is merged in project-config - https://review.openstack.org/#/c/490622/, I will update the other corresponding patches. | 12:23 |
| EmilienM | numans: sure, will look asap | 12:23 |
| numans | EmilienM, thanks. | 12:24 |
| EmilienM | numans: are you sure you want the job voting? | 12:24 |
| *** dpawar has quit IRC | 12:26 | |
| numans | EmilienM, since scenario007 baremetal job is voting, I submitted the patch same way | 12:26 |
| EmilienM | gfidente: any comment on https://review.openstack.org/#/c/479288 ? | 12:26 |
| EmilienM | numans: it was non voting first | 12:27 |
| EmilienM | numans: and we made it voting afterward | 12:27 |
| numans | EmilienM, oh ok. i missed that part. i will respin another patch then. thanks | 12:27 |
| *** bkopilov has joined #tripleo | 12:28 | |
| EmilienM | numans: we documented the bits btw: http://specs.openstack.org/openstack/tripleo-specs/specs/policy/adding-ci-jobs.html | 12:28 |
| numans | EmilienM, thanks. | 12:29 |
| *** jpena|lunch is now known as jpena | 12:29 | |
| EmilienM | sshnaidm: https://status-tripleoci.rhcloud.com/ is down | 12:30 |
| EmilienM | weshay: ^ did you guys changed something? | 12:31 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml https://review.openstack.org/493984 | 12:31 |
| sshnaidm | EmilienM, http://cistatus.tripleo.org/ | 12:31 |
| sshnaidm | EmilienM, there were redirect, will check what happenes | 12:32 |
| EmilienM | sshnaidm: or just change the html | 12:32 |
| EmilienM | sshnaidm: wow, new layout? | 12:32 |
| sshnaidm | EmilienM, yeah, a few improvements | 12:33 |
| *** skramaja has quit IRC | 12:33 | |
| abishop | shardy: https://review.openstack.org/480587 has one +2 but needs another. could you take a look? | 12:33 |
| shardy | sshnaidm: nice work on the cistatus :) | 12:34 |
| shardy | abishop: ack will do | 12:34 |
| weshay | EmilienM, http://cistatus.tripleo.org/ | 12:34 |
| shardy | abishop: the blueprint link in the comment seems to be broken, can you pls link it, and does it have a FFE? | 12:35 |
| sshnaidm | shardy, EmilienM if you're interested, take a look: https://docs.google.com/document/d/15CwUUDi4Oca95IuY9s2gme03Z8COvJqmvp7uKdEKeU8/edit# | 12:35 |
| shardy | abishop: I know this work has been ongoing for a while, but I can't see it targetted to rc1? | 12:35 |
| abishop | shardy: hmm, I'll need to check on that | 12:36 |
| shardy | https://launchpad.net/tripleo/+milestone/pike-rc1 | 12:36 |
| EmilienM | sshnaidm: yeah, nice doc | 12:36 |
| honza | apetrich: are you working on the nokogiri/ruby/libxml issue? should i? | 12:37 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder https://review.openstack.org/494008 | 12:38 |
| *** psahoo has quit IRC | 12:38 | |
| apetrich | honza, sorry I got distracted with other reviews. If you have the cycles, yes. Also I don't have much idea on how to go about that | 12:39 |
| honza | apetrich: I think I have even less :) | 12:39 |
| lvdombrkr | hello guys, as i understood by default 1 compute and 1 control node will be deployed, how can i deploy compute and controller node all in one? | 12:39 |
| shardy | lvdombrkr: you can deploy with an environment file similar to this: | 12:40 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/multinode.yaml#L17..L62 | 12:40 |
| shardy | https://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html | 12:40 |
| ccamacho | shardy quick question have you seen this error lately ? "/usr/bin/docker-current: Error while pulling image: Get https://index.docker.io/v1/repositories/tripleoupstream/centos-binary-iscsid/images: dial tcp: lookup index.docker.io on 192.168.24.5:53: server misbehaving.", | 12:40 |
| shardy | lvdombrkr: that has some background, and shows how to disable the compute node | 12:40 |
| shardy | lvdombrkr: you can then build up ControllerServices based on what services you want | 12:41 |
| apetrich | honza, no worries I can give it a go. Do you know if that was working before? | 12:41 |
| shardy | lvdombrkr: note there's no dependency mechanisim yet, so you need to know which services depend on which others | 12:41 |
| shardy | or figure that out while testing ;) | 12:41 |
| honza | apetrich: This is the first time I have seen that error, here or anywhere. AFAIK that job was passing earlier on that patch. | 12:41 |
| apetrich | honza, oh that's true | 12:42 |
| honza | apetrich: If the CI pipeline wasn't so backed up right now, I'd just give it a cheeky recheck... | 12:42 |
| shardy | lvdombrkr: there is also https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/composable_services.html but I notice that has a mistake, it should just say ControllerServices not CustomControllerServices, and you can specify an environment file instead of a custom roles_data.yaml | 12:43 |
| shardy | I'll push an update to fix that later | 12:43 |
| shardy | ccamacho: No but if it's in CI there's been a bunch of issues related to caching the docker images I think | 12:44 |
| ccamacho | shardy mmm nope, thats from a fresh master install with containers :( | 12:44 |
| *** psahoo has joined #tripleo | 12:44 | |
| shardy | ccamacho: ah, sorry I'm working on composable networking this week, so not tested latest w/containers | 12:45 |
| shardy | I'd probably try pushing everything to a local registry and using that instead | 12:45 |
| ccamacho | shardy thanks! Ill do that | 12:45 |
| shardy | ccamacho: FWIW I don't use the undercloud registry, I have a local box configured like this: | 12:46 |
| shardy | http://paste.openstack.org/show/618523/ | 12:46 |
| shardy | ccamacho: that works as a pull-through cache so you don't actually need to push anything | 12:46 |
| EmilienM | shardy: just saw your comment on https://bugs.launchpad.net/tripleo/+bug/1710938 | 12:47 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:47 |
| EmilienM | shardy: do you suggest that CI runs a different workflow that customers would run? | 12:47 |
| ccamacho | shardy ack thanks! | 12:47 |
| shardy | EmilienM: I think it is a workflow customers would run if they have a local registry of their own | 12:47 |
| *** jlabarre has joined #tripleo | 12:47 | |
| shardy | EmilienM: so I'm just saying not all container jobs necessarily have to use the undercloud registry | 12:47 |
| shardy | particularly if it saves time and fixes the upgrades job? | 12:48 |
| shardy | e.g do we waste time pulling to the undercloud, then again from undercloud to subnode? | 12:48 |
| EmilienM | shardy: yes I would support the idea | 12:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat] https://review.openstack.org/491399 | 12:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 12:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral] https://review.openstack.org/493471 | 12:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi] https://review.openstack.org/491433 | 12:48 |
| *** lucasagomes is now known as lucas-brb | 12:48 | |
| *** catintheroof has joined #tripleo | 12:49 | |
| shardy | EmilienM: I'm not quite clear how the infra caching works, but regardless it seems like we're doing a double download which could be avoided for single-node CI tests | 12:49 |
| EmilienM | I agree | 12:49 |
| *** jaganathan has quit IRC | 12:49 | |
| shardy | EmilienM: did you manage to chat with stevebaker? I wondered if that was what he had in mind | 12:50 |
| shardy | then you'd just do the prepare CLI and not the upload | 12:50 |
| EmilienM | shardy: what we had in mind is to download/upload containers that are used only and not everything | 12:50 |
| EmilienM | he has a patch to start this work, a sec | 12:50 |
| *** atoth has quit IRC | 12:50 | |
| EmilienM | shardy: https://review.openstack.org/#/c/448328/ | 12:50 |
| shardy | EmilienM: yeah, so I'm suggesting we just do the prepare step to generate the environment | 12:51 |
| shardy | which will only contain the images we need, and point to the infra proxy | 12:51 |
| EmilienM | shardy: can we do that already now? I guess yes | 12:51 |
| shardy | EmilienM: yes, you just switch the IP/port to point somewhere else instead of the undercloud when doing the prepare | 12:52 |
| shardy | I already do that locally as does dprince I think | 12:52 |
| *** jmelvin has joined #tripleo | 12:52 | |
| lvdombrkr | shardy: what will happen if i have only one node in instackenv.json and i will run just default: openstack overcloud deploy --templates | 12:53 |
| lvdombrkr | ? | 12:54 |
| shardy | lvdombrkr: it will fail because by default we deploy two nodes | 12:54 |
| *** tzumainn has joined #tripleo | 12:55 | |
| dprince | shardy: yes, it works great | 12:55 |
| shardy | EmilienM: I would guess we just change local_docker_registry_host to docker_registry_host here: | 12:55 |
| shardy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L43 | 12:55 |
| EmilienM | shardy: to the regional AFS mirror I guess? | 12:55 |
| shardy | EmilienM: I guess it'd be like this: | 12:56 |
| shardy | https://github.com/openstack/tripleo-quickstart/commit/8a78e84f173743a5f2042f86854c8b0302dcd4f5 | 12:56 |
| shardy | sshnaidm: ^^ can we just point the overcloud nodes at undercloud_docker_registry_mirror, e.g is it a docker-registry instance? | 12:57 |
| EmilienM | shardy: we might need to change https://review.openstack.org/#/c/493972/4/roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2 as well, not sure we need to do the upload then | 12:57 |
| shardy | EmilienM: sorry, yeah maybe to start with we only change it there | 12:57 |
| shardy | and see if we can get the upgrade job working | 12:57 |
| jtomasek | honza: are you working on rebasing https://review.openstack.org/#/c/473933/ ? | 12:58 |
| shardy | or add a boolean to allow pull-from-proxy or not | 12:58 |
| jtomasek | honza: I can do it if you want | 12:58 |
| honza | jtomasek: rebasing? why? | 12:58 |
| honza | jtomasek: I was hoping it would merge as is. | 12:59 |
| EmilienM | shardy: ok I'll do that, thx | 12:59 |
| jtomasek | honza: the actions error handling is broken there as the error handling patches merged. It is a simple update | 12:59 |
| lvdombrkr | shardy: to be shore that there is no discomunication,my goal is to in one node combaine controller and compute | 12:59 |
| honza | jtomasek: I see. I'll do that right now. | 13:00 |
| jtomasek | honza: ok, cool | 13:00 |
| *** oidgar_ has joined #tripleo | 13:00 | |
| shardy | lvdombrkr: yup, if you look at the links I provided, you can see you need an environment file which modifies the default ControllerServices (to add the compute services to the Controller node) | 13:01 |
| shardy | lvdombrkr: and ComputeCount: 0 to disable deploying any compute node | 13:01 |
| shardy | lvdombrkr: you can customize the ControllerServices list to suit your requirements, including adding the compute bits to the one "controller" node | 13:01 |
| *** trozet has joined #tripleo | 13:02 | |
| EmilienM | shardy: we alraedy have containerized_overcloud_upgrade_pull_images boolean | 13:02 |
| shardy | lvdombrkr: the multinode environment I linked does exactly what you describe every commit in our CI, but there's some additional config in there you probably don't need | 13:02 |
| EmilienM | I think we could just re-use that | 13:02 |
| *** oidgar has quit IRC | 13:02 | |
| EmilienM | shardy: do we need to execute these bits in your idea? https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2#L16-L29 | 13:03 |
| trozet | EmilienM: good morning. With that odl scenario do you know if it is using a br-ex bridge? | 13:06 |
| *** dsariel has quit IRC | 13:06 | |
| shardy | EmilienM: I think we'd need only one prepare, which points at undercloud_docker_registry_mirror instead of local_docker_registry_host | 13:06 |
| shardy | be good to get sshnaidm to check that first tho | 13:06 |
| *** liverpooler has joined #tripleo | 13:06 | |
| *** atoth has joined #tripleo | 13:07 | |
| sshnaidm | shardy, sorry, can you elaborate? | 13:07 |
| shardy | EmilienM: this prepare workflow has been changing a bit lately so may need to check locally - I've not tested it for a few days | 13:07 |
| shardy | sshnaidm: we're figuring out how to skip pulling images to the undercloud docker registry | 13:07 |
| shardy | sshnaidm: and instead, generate an environment file with overcloud images prepare which points directly to the infra proxy | 13:08 |
| shardy | e.g the subnode pulls directly from the undercloud docker proxy from NODEPOOL_DOCKER_REGISTRY_PROXY | 13:08 |
| *** shreshtha has joined #tripleo | 13:08 | |
| sshnaidm | shardy, I see | 13:08 |
| shardy | sshnaidm: the idea is it saves some time, for a single node test we don't really need to proxy the images twice | 13:08 |
| shardy | and we need to find some time to fix the upgrade job | 13:09 |
| *** brault has quit IRC | 13:09 | |
| sshnaidm | shardy, so to skip pull-and-push step? | 13:09 |
| shardy | sshnaidm: yup | 13:09 |
| EmilienM | yes | 13:09 |
| *** brault has joined #tripleo | 13:09 | |
| EmilienM | I was looking at setting containerized_overcloud_upgrade_pull_images to False but it won't work | 13:09 |
| shardy | EmilienM: are you set up to test this locally or would you like me to try and etherpad the steps? | 13:09 |
| EmilienM | we need to write the DockerInsecureRegistryAddress anyway | 13:09 |
| EmilienM | shardy: I don't have the setup right now | 13:10 |
| shardy | EmilienM: Ok I'll try it now, sec | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 13:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| *** oidgar_ is now known as oidgar | 13:10 | |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 13:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Switch to scheduling based on resource classes https://review.openstack.org/490851 | 13:10 |
| gfidente | EmilienM you might I think I know where the ceph logs are :D | 13:11 |
| gfidente | EmilienM I think in the containers only right now :( | 13:11 |
| sshnaidm | shardy, I'll try to make this step conditional then, and using afs mirror instead of local_docker_registry_host ? | 13:11 |
| EmilienM | gfidente: we need to fix that | 13:11 |
| gfidente | EmilienM yeah I see what you mean | 13:11 |
| *** ansmith has joined #tripleo | 13:12 | |
| *** psahoo has quit IRC | 13:12 | |
| gfidente | EmilienM I can probably mount /var/log/ceph in the containers | 13:12 |
| gfidente | and that should do it | 13:13 |
| gfidente | or put it in /containers/ ? | 13:13 |
| EmilienM | gfidente: like we had before /var/log/ceph probably | 13:14 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Download logs interface https://review.openstack.org/473933 | 13:15 |
| honza | jtomasek: ^^^ this new error handling is *awesome* | 13:15 |
| lvdombrkr | shardy: thank you now its getting clear. but there is no ready tampletes which i can use without any corrections ? cause there is greate chance to failed if creating tamplete wihout big bacground | 13:15 |
| shardy | gfidente: the logs for most other containers not managed by pacemaker are under /var/log/containers/foo | 13:16 |
| gfidente | shardy ack | 13:16 |
| *** cdearborn_ has joined #tripleo | 13:16 | |
| EmilienM | oh yes right | 13:16 |
| shardy | lvdombrkr: you could cut/paste the multinode one as a starting point, sec let me do it for you | 13:16 |
| abishop | shardy: not sure what's going on, because blueprint is here: https://blueprints.launchpad.net/tripleo/+spec/tripleo-derive-parameters | 13:16 |
| gfidente | EmilienM shardy fine thanks I'll update it quickly | 13:17 |
| lvdombrkr | shardy: i will be grateful | 13:17 |
| EmilienM | gfidente: thanks | 13:17 |
| abishop | shardy: it was marked implemented in m3, and we're just polishing and cleaning up TODOs | 13:17 |
| EmilienM | gfidente: next step, I'll enable pingtest and see how it works | 13:17 |
| shardy | lvdombrkr: something like http://paste.openstack.org/show/618524/ | 13:17 |
| gfidente | EmilienM pingtest should work with both puppet-ceph and ceph-ansible | 13:17 |
| gfidente | net the known issue with glance | 13:18 |
| *** chem has quit IRC | 13:18 | |
| shardy | lvdombrkr: you can customize ControllerServices depending on your requirements, we don't have a fully documented all in one example though unfortunately | 13:18 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-specs master: Propose tech-debt policy https://review.openstack.org/494044 | 13:19 |
| EmilienM | gfidente: I won't believe you until I see it in CI ;-) | 13:19 |
| shardy | abishop: Ok no problem, is this the final patch, or do we need a bug to track the remaining patches? | 13:19 |
| pabelanger | morning | 13:19 |
| gfidente | EmilienM true and also, the logs change, should I put it into the scenario001 conversion submission | 13:19 |
| gfidente | or better to split it into its own given scope? | 13:19 |
| *** pradk has quit IRC | 13:19 | |
| EmilienM | gfidente: wherever you like | 13:19 |
| *** chem has joined #tripleo | 13:20 | |
| EmilienM | gfidente: at this stage :D | 13:20 |
| pabelanger | EmilienM: mwhahaha: weshay: This is the next things we need to solve: http://logs.openstack.org/08/487408/9/gate/gate-tripleo-ci-centos-7-scenario004-multinode-oooq/7744844/logs/delorean_logs/b1/df/b1df493d43ffc59b67f9ea0e0bf4c0bca4279fe0_dev/rpmbuild.log.txt.gz | 13:20 |
| gfidente | better to use less submissions probably | 13:20 |
| pabelanger | gate just reset because DLRN failed to download from buildlogs.centos.org | 13:20 |
| abishop | shardy: I think there are a couple more small items, for example https://review.openstack.org/489239 | 13:20 |
| pabelanger | anybody know the status of having DLRM / mock use our AFS mirrors? | 13:20 |
| lvdombrkr | shardy: greate, thank you!!! | 13:20 |
| abishop | shardy: would it be better to file lp bugs for each item like this? | 13:21 |
| shardy | abishop: or we should perhaps reopen the blueprint if it's still in progress? | 13:21 |
| EmilienM | pabelanger: you better ask jpena or amoralej on this one, tbh | 13:21 |
| EmilienM | gfidente: yes I support that | 13:21 |
| shardy | abishop: if the remaining bits can be considered bugs in the implementation landed then sure, we can raise some bugs | 13:22 |
| weshay | pabelanger, ah.. to add the mirror variables to the rpm build? | 13:22 |
| pabelanger | weshay: ya, mock.cfg files need to be updated | 13:23 |
| abishop | shardy: I think lp bugs are better fit, so I'll make sure lp bugs are opened and resubmit the reviews | 13:23 |
| shardy | abishop: Ok thanks - sounds like we can't get these gating atm anyway, let me know when you've pushed and we can fast-approve when the gate is better | 13:23 |
| shardy | at least the bugs will help us track the patches for the rc releases | 13:23 |
| abishop | shardy: will do, thx | 13:23 |
| weshay | adarazs, fyi ^ see pabelanger comments here | 13:25 |
| weshay | pabelanger, that is probably a hot issue we need to fix right now? | 13:25 |
| *** ramishra has quit IRC | 13:25 | |
| weshay | pabelanger, adarazs I'll open a bug | 13:26 |
| weshay | ugh | 13:26 |
| *** dsariel has joined #tripleo | 13:26 | |
| pabelanger | weshay: ya, sooner the fix, the less resets we get in the gate, and merge code faster | 13:27 |
| weshay | aye | 13:27 |
| adarazs | weshay: thanks, please create a bug for it. are we talking about messing with mock.cfg when quickstart gates build packages? | 13:27 |
| *** ramishra has joined #tripleo | 13:28 | |
| *** janki has quit IRC | 13:28 | |
| jtomasek | honza: :) | 13:28 |
| *** ecerquei_ has joined #tripleo | 13:28 | |
| sshnaidm | who can check what failed with zaqar subscription here? http://logs.openstack.org/50/480950/32/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/f602d77/logs/undercloud/var/log/zaqar/zaqar-server.log.txt.gz#_2017-08-16_11_09_04_004 | 13:28 |
| *** janki has joined #tripleo | 13:28 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/container: run Barbican non-containerized https://review.openstack.org/493734 | 13:28 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/multinode: do not run containerized Zaqar https://review.openstack.org/494005 | 13:29 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on scenarios container jobs https://review.openstack.org/490129 | 13:29 |
| EmilienM | gfidente, fultonj ^ testing pingtest on container jobs with ceph NOT containerized yet | 13:29 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 13:30 |
| EmilienM | gfidente: i'll test containerized ceph (with pingtest) once we have logs | 13:30 |
| gfidente | EmilienM non-containerized needs the glance fix | 13:30 |
| *** oidgar_ has joined #tripleo | 13:31 | |
| EmilienM | gfidente: which one? | 13:31 |
| *** chlong_ has joined #tripleo | 13:31 | |
| gfidente | EmilienM https://review.openstack.org/494176 | 13:31 |
| EmilienM | gfidente: correct, I added it in Depends-On :D | 13:31 |
| EmilienM | so I'll tell you how it works | 13:31 |
| EmilienM | with pingtest | 13:31 |
| gfidente | great thanks | 13:32 |
| gfidente | ++ | 13:32 |
| *** ansmith has quit IRC | 13:32 | |
| derekh | EmilienM can you take another look at this if you have a chance https://review.openstack.org/#/c/485261/ | 13:32 |
| *** oidgar has quit IRC | 13:33 | |
| EmilienM | derekh: where the creation of a server is tested? | 13:34 |
| EmilienM | derekh: this patch regress scenario004 in the sense it's not testing server creation anymore | 13:34 |
| EmilienM | http://logs.openstack.org/61/485261/5/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq/7dabdfc/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-08_14_37_12 | 13:34 |
| EmilienM | scenario004 is a ceph + rgw use case (used by glance) - so we don't know if that use case works anymore | 13:34 |
| derekh | EmilienM: there isn't one, yet, I've asked in my email if this was ok for now? and then I'd add one | 13:35 |
| *** jlabarre has quit IRC | 13:35 | |
| EmilienM | derekh: I wouldn't not do it *afterward* - it would be a regression in scenario004 testing in the meantime | 13:35 |
| derekh | EmilienM: ok so I need to add that first | 13:36 |
| *** ykarel is now known as ykarel|afk | 13:36 | |
| *** lblanchard has joined #tripleo | 13:38 | |
| *** jlabarre has joined #tripleo | 13:38 | |
| *** lblanchard has quit IRC | 13:39 | |
| openstackgerrit | Numan Siddique proposed openstack/puppet-tripleo master: Add OVN DBs bundle support for pacemaker HA https://review.openstack.org/484373 | 13:39 |
| *** ykarel|afk has quit IRC | 13:40 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: Support OVN DBs docker HA using pacemaker bundle https://review.openstack.org/484376 | 13:41 |
| pabelanger | weshay: adarazs bug 1710678 | 13:41 |
| openstack | bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,Triaged] https://launchpad.net/bugs/1710678 - Assigned to Alfredo Moralejo (amoralej) | 13:41 |
| amoralej | pabelanger, EmilienM, implementing ^ is not trivial, i need some time for it, we have a couple of ideas to do it without breaking DLRN but i need to do some tests | 13:43 |
| pabelanger | amoralej: do you have your thoughts posted some place? Maybe we can have a few people itterate on this | 13:44 |
| amoralej | pabelanger, not yet, i will post in the LP | 13:44 |
| dalvarez | beagles, ping :) | 13:46 |
| *** gbarros has joined #tripleo | 13:46 | |
| beagles | dalvarez, pong! | 13:46 |
| dalvarez | ops 1sc :) | 13:47 |
| openstackgerrit | Adriano Petrich proposed openstack/instack-undercloud master: Add an hourly cron trigger for tripleo-ui logging https://review.openstack.org/469608 | 13:47 |
| pabelanger | amoralej: k, for now, we could symlink /etc/mock/default.cfg to a new openstack-infra spedific config and see what happens. | 13:49 |
| *** lblanchard has joined #tripleo | 13:49 | |
| mwhahaha | pabelanger: EmilienM, looks like we didn't quite get all the mirror.centos.org references, http://logs.openstack.org/39/494039/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/2adf7d9/console.html#_2017-08-15_23_48_44_501109 | 13:50 |
| pabelanger | mwhahaha: EmilienM: Yes ^ is because tripleo-ci is running yum commands before mirrors are properly setup | 13:51 |
| EmilienM | mwhahaha: probably in tripleo-ci repo | 13:51 |
| pabelanger | so, that needs to be fixed too | 13:51 |
| *** lucas-brb is now known as lucasagomes | 13:53 | |
| *** dparkes has joined #tripleo | 13:55 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: container/upgrades: unconditionally create containers-default-parameters.yaml https://review.openstack.org/494214 | 13:55 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: DNM - fs011: disable container pull/push https://review.openstack.org/494215 | 13:55 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 13:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 13:57 |
| EmilienM | sshnaidm: ^ trying something | 13:58 |
| EmilienM | pabelanger, mwhahaha : ok I'm looking into that now | 13:58 |
| *** chlong_ has quit IRC | 13:59 | |
| *** ansmith has joined #tripleo | 14:00 | |
| *** udesale has joined #tripleo | 14:01 | |
| *** jprovazn has quit IRC | 14:01 | |
| *** agurenko has quit IRC | 14:03 | |
| *** masco has quit IRC | 14:03 | |
| jaosorior | bnemec: could you check this out https://review.openstack.org/#/c/493937/ | 14:06 |
| *** ebarrera has quit IRC | 14:07 | |
| *** ebarrera has joined #tripleo | 14:07 | |
| *** nyechiel has quit IRC | 14:09 | |
| *** oidgar_ is now known as oidgar | 14:09 | |
| *** dtantsur is now known as dtantsur|bbl | 14:10 | |
| *** dsariel has quit IRC | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 14:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 14:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| *** ykarel|afk has joined #tripleo | 14:11 | |
| *** ecerquei_ has quit IRC | 14:12 | |
| *** atoth has quit IRC | 14:12 | |
| *** ecerquei has joined #tripleo | 14:12 | |
| EmilienM | sshnaidm: I'm trying to see where we should deploy qemu-img instead of here http://logs.openstack.org/39/494039/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/2adf7d9/console.html#_2017-08-15_23_48_43_966405 | 14:19 |
| EmilienM | sshnaidm: so we could take benefit from the regional mirrors | 14:19 |
| EmilienM | sshnaidm: it sounds like we need qemu-img in the multinode preparation, do you know a place in quickstart where we could do it *after* having the repos in place? or is it too early? | 14:20 |
| *** janki has quit IRC | 14:22 | |
| *** atoth has joined #tripleo | 14:25 | |
| *** mdnadeem has quit IRC | 14:26 | |
| amoralej | pabelanger, there is no mock.cfg, dlrn creates one on the fly | 14:26 |
| amoralej | that's the main problem | 14:26 |
| sshnaidm | EmilienM, sorry, to do what? | 14:26 |
| sshnaidm | EmilienM, iirc it's needed to create a fake image, right? | 14:26 |
| EmilienM | sshnaidm: yeah, but we want to download qemu-img from the mirror not from centos.org | 14:27 |
| EmilienM | sshnaidm: trying to find the best place to put this code | 14:27 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Add --set argument to prepare command https://review.openstack.org/489083 | 14:27 |
| sshnaidm | EmilienM, I see, checking.. | 14:27 |
| pabelanger | EmilienM: weshay: mwhahaha: I think we need to add debug: true to /etc/docker/daemon.json, wanted to see why it is possible to get 404 client error: not found | 14:29 |
| weshay | pabelanger, k.. not sure if that is possible via the undercloud variable, maybe one could hack that in.. or we could restore my patches for debug purposes | 14:30 |
| weshay | I can do the latter | 14:30 |
| weshay | if that would help | 14:30 |
| *** jmelvin has quit IRC | 14:30 | |
| pabelanger | weshay: ya, I think collecting debug logs from docker is the next step | 14:30 |
| weshay | k.. will need a few minutes.. mtgs mtgs mtgs | 14:31 |
| weshay | mtg-stack | 14:31 |
| *** links has quit IRC | 14:32 | |
| *** jmelvin has joined #tripleo | 14:32 | |
| *** dsariel has joined #tripleo | 14:36 | |
| lvdombrkr | shardy: i get error after openstack overcloud deploy --templates one.yaml | 14:37 |
| lvdombrkr | Configuration has 1 errors, fix them before proceeding. | 14:37 |
| lvdombrkr | Not enough nodes - available: 1, requested: 2 | 14:37 |
| openstackgerrit | Honza Pokorny proposed openstack/instack-undercloud master: Add PATCH to list of allowed methods for Ironic https://review.openstack.org/494228 | 14:37 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-specs master: Propose tech-debt policy https://review.openstack.org/494044 | 14:41 |
| *** pchavva has joined #tripleo | 14:42 | |
| sshnaidm | EmilienM, I'm not sure in which stage we need this fake 1G image for multinode. If it's just before overcloud install, then I'll move this step to oooq roles and we're good. If it's required to bootstrap nodes, then we need to look how to hack it with tripleo.sh. The bootstrapping is done still with tripleo.sh --bootstrap-nodes :( | 14:42 |
| sshnaidm | slagle, ^^ | 14:42 |
| shardy | lvdombrkr: try penstack overcloud deploy --templates -e one.yaml | 14:43 |
| shardy | assuming one.yaml has the environment file with parameter_defaults in it | 14:43 |
| shardy | openstack overcloud deploy ... | 14:43 |
| gfidente | trown can you check https://review.openstack.org/#/c/494056/ ? has enough votes but I think you want to be aware of that? | 14:44 |
| slagle | sshnaidm: what are you asking? | 14:44 |
| *** pradk has joined #tripleo | 14:44 | |
| sshnaidm | slagle, when we actually need the fake image in multinode job? | 14:45 |
| slagle | sshnaidm: if you don't want it done with tripleo.sh, then change it | 14:45 |
| *** gkadam has joined #tripleo | 14:45 | |
| sshnaidm | slagle, when the fake image is started to being used | 14:45 |
| slagle | sshnaidm: i dont know if oooq uses that or not | 14:45 |
| trown | gfidente: thanks | 14:45 |
| trown | gfidente: looks good and importantly does not conflict with anything :P | 14:46 |
| sshnaidm | slagle, not related to oooq, I'm asking for multinode deployment generally | 14:46 |
| gfidente | trown cool I wasn't sure about the naming choice | 14:46 |
| lvdombrkr | shardy: same error | 14:46 |
| trown | gfidente: ya seems fine to me | 14:46 |
| gfidente | fultonj thanks ^^ | 14:47 |
| fultonj | cool | 14:47 |
| pabelanger | Hmm 2 failures in mirror.sto2.citycloud.o.o for docker.io | 14:47 |
| lvdombrkr | lvdombrkr: i just copy/paste list of services what you give me to one.yaml | 14:47 |
| *** agurenko has joined #tripleo | 14:48 | |
| sshnaidm | EmilienM, either I can put it here: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L1436 if repo-setup of tirpleo.sh uses mirrors | 14:48 |
| EmilienM | sshnaidm: yes we could do that | 14:48 |
| pradk | jaosorior, Hi, can i abandon this https://review.openstack.org/#/c/486709/ | 14:49 |
| pradk | jaosorior, seems like things are working fine without this now? | 14:49 |
| slagle | sshnaidm: that's what i was trying to answer since multinode is generally done with oooq. i dont know if oooq needs it or not | 14:49 |
| slagle | sshnaidm: i believe there are still a few multinode jobs not using oooq. you could remove it and see if those pass | 14:50 |
| sshnaidm | slagle, ok, will try | 14:50 |
| sshnaidm | EmilienM, ^^ | 14:50 |
| EmilienM | slagle: afik the only job that doesn't use oooq now is 3nodes but it's being fixed by trown | 14:51 |
| EmilienM | slagle: oh and the upgrade job on stable/ocata | 14:51 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test removing fake image from multinode https://review.openstack.org/494233 | 14:53 |
| *** vkhanna1 has joined #tripleo | 14:53 | |
| *** jprovazn has joined #tripleo | 14:54 | |
| *** snecklifter has quit IRC | 14:55 | |
| EmilienM | sshnaidm: can you test a patch in THT stable/ocata as well please? the upgrade job doesn't use quickstart | 14:56 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DONT REVIEW: test without fake image https://review.openstack.org/494235 | 14:56 |
| sshnaidm | EmilienM, ok, after master | 14:56 |
| *** vkhanna1 has quit IRC | 14:57 | |
| lvdombrkr | shardy: any ideas? | 14:57 |
| *** sri_ has joined #tripleo | 14:57 | |
| lvdombrkr | shardy: i belive that im not first with that setup | 14:57 |
| lvdombrkr | ))) | 14:57 |
| *** snecklifter has joined #tripleo | 14:58 | |
| sshnaidm | EmilienM, undercloud-containers doesn't use ooq too | 14:58 |
| EmilienM | indeed | 14:58 |
| *** dtantsur|bbl is now known as dtantsur | 14:58 | |
| EmilienM | pabelanger: I could add debug option with weshay after our mtg | 14:59 |
| pabelanger | thanks | 14:59 |
| *** ykarel|afk is now known as ykarel | 14:59 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 15:01 |
| adarazs | amoralej, EmilienM: I see this is assigned to Alfredo: https://bugs.launchpad.net/tripleo/+bug/1710678 -- did you start working on it? | 15:01 |
| *** _milan_ has joined #tripleo | 15:01 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,Triaged] - Assigned to Alfredo Moralejo (amoralej) | 15:01 |
| d0ugal | thrash: https://review.openstack.org/#/c/462916/27..28/workbooks/baremetal.yaml | 15:01 |
| *** ecerquei has quit IRC | 15:01 | |
| d0ugal | thrash: does that look good? | 15:01 |
| *** ecerquei_ has joined #tripleo | 15:01 | |
| thrash | d0ugal: the message can now just use $.introspection_attempt.. No need to add 1 | 15:02 |
| thrash | right? | 15:02 |
| *** dsariel has quit IRC | 15:02 | |
| thrash | or... Does it start at 0? | 15:03 |
| d0ugal | thrash: it starts at 0, I think that might be the real bug. | 15:03 |
| d0ugal | I forget why I done that, and my comment doesn't help :) | 15:03 |
| mhenkel | slagle: Is there a patch for https://bugzilla.redhat.com/show_bug.cgi?id=1412352 ? | 15:04 |
| openstack | bugzilla.redhat.com bug 1412352 in diskimage-builder "rhel-registration script broken with satellite 6.2.5+" [Medium,Assigned] - Assigned to jslagle | 15:04 |
| thrash | d0ugal: hehe... solve it as you like.. I think leaving it as '+ 1' is probably the easiest, if not the most readable | 15:04 |
| *** oidgar has quit IRC | 15:04 | |
| *** milan has quit IRC | 15:04 | |
| thrash | d0ugal: if you start it at 1, you'll need to make sure your inequalities are correct | 15:04 |
| *** sshnaidm is now known as sshnaidm|afk | 15:05 | |
| slagle | mhenkel: no | 15:05 |
| mhenkel | slagle: any plans? | 15:06 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 15:07 |
| *** rcernin has quit IRC | 15:07 | |
| weshay | amoralej, adarazs please sync up re: https://bugs.launchpad.net/tripleo/+bug/1710678 | 15:08 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,Triaged] - Assigned to Alfredo Moralejo (amoralej) | 15:08 |
| *** aufi has quit IRC | 15:08 | |
| adarazs | weshay: I just asked him if he's working on it a minute ago | 15:08 |
| adarazs | no answer yet though | 15:08 |
| slagle | mhenkel: not specifically. it's not a wide spread issue apparently. so it must be something environmental, e.g. not a high priority at the moment | 15:08 |
| lvdombrkr | guys, anybode bofore deplot overcloud with compute and controller in one node? | 15:09 |
| mhenkel | slagle: as noted in the bug, the issue is introduced by satellite >6.1 (I tested on 6.2). So, all customers using that version or beyond will face the issue. | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 15:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 15:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| *** ebarrera has quit IRC | 15:10 | |
| *** cylopez has quit IRC | 15:12 | |
| openstackgerrit | Vineet Paul proposed openstack/tripleo-heat-templates master: Drop extraconfig for nova-nuage https://review.openstack.org/494239 | 15:13 |
| dalvarez | beagles, in order to create the ini file for the metadata agent, i copied the puppet-neutron/lib/puppet/provider/neutron_metadata_agent_config to puppet-neutron/lib/puppet/provider/ovn_metadata_agent_config | 15:14 |
| dalvarez | and also added this line to puppet-neutron/manifests/deps.pp: Anchor['neutron::config::begin'] -> Ovn_metadata_agent_config<||> ~> Anchor['neutron::config::end'] | 15:14 |
| dalvarez | Does it make sense? | 15:14 |
| amoralej | adarazs, i'll ping you after mtg | 15:14 |
| adarazs | amoralej: great. no idea how to modify the centos mirror it uses for now :) | 15:15 |
| adarazs | amoralej: I mean apart from replacing stuff in rdo.yml before the build. | 15:16 |
| lvdombrkr | guys, anybode bofore deplot overcloud with compute and controller in one node? | 15:16 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Adds PostUpgradeConfigStepsDeployment to drive post config ansible https://review.openstack.org/493878 | 15:17 |
| adarazs | lvdombrkr: I'm not sure it's possible for now with tripleo. though I remember people trying to get that working. and maybe there are some experimental versions somewhere. | 15:18 |
| beagles | dalvarez, on call at the moment, I'll ping you when done | 15:18 |
| dalvarez | beagles, sure no worries, thanks a lot! | 15:18 |
| akrivoka | jtomasek: the new node wrapping workflow needs to include register, introspect and provide - correct? | 15:19 |
| lvdombrkr | adarazs: my goal is olso just test env. | 15:19 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Add support for installing Ceph MDS via ceph-ansible https://review.openstack.org/479426 | 15:20 |
| akrivoka | jtomasek: what about the manage nodes operation? it is not referenced in the user docs (https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/basic_deployment_ui.html#manage-nodes) - when would this operation be used in the workflow? does it need to get included in the wrapping workflow? | 15:20 |
| shardy | adarazs: I think lvdombrkr has an undercloud and wants to deploy a single node overcloud, but perhaps I'm mistaken | 15:21 |
| adarazs | oh, okay :) | 15:21 |
| adarazs | I misunderstood. | 15:21 |
| slagle | mhenkel: if not using rhel registration during the actual image build, then i dont think katello-ca-consumer will get installed until the actual overcloud deployment | 15:22 |
| slagle | mhenkel: at that point, the hostname ought to be set by cloud-init | 15:22 |
| slagle | mhenkel: so, are we talking about the case where you're building rhel images with dib and registering the image build? | 15:22 |
| lvdombrkr | shardy: you catch my idea | 15:23 |
| lvdombrkr | shardy: so any soulutions? | 15:23 |
| shardy | lvdombrkr: I already linked everything needed to do it, but I'll try to create some more step by step instructions later | 15:24 |
| shardy | the solution is to set the ComputeCount to 0 and specify the ControllerServices needed | 15:24 |
| shardy | as I said earlier | 15:24 |
| *** gkadam is now known as gkadam-afk | 15:24 | |
| shardy | sorry if I didn't explain it clearly enough, probably we need some better docs | 15:24 |
| *** udesale has quit IRC | 15:25 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Download logs interface https://review.openstack.org/473933 | 15:25 |
| lvdombrkr | shardy: i excatlly do that, i creata file one.yaml and copy a list which you prepare for me | 15:26 |
| lvdombrkr | thats all as i understood? | 15:26 |
| shardy | lvdombrkr: yes, I'll have to test it later if it didn't work for you | 15:27 |
| *** shardy is now known as shardy_afk | 15:27 | |
| lvdombrkr | shardy_afk: but you tested such a setup before? | 15:28 |
| *** yprokule has quit IRC | 15:29 | |
| trozet | michapma: hi | 15:29 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Render VIPs dynamically based on network_data.yaml https://review.openstack.org/486260 | 15:30 |
| *** gkadam-afk has quit IRC | 15:31 | |
| *** gkadam has joined #tripleo | 15:32 | |
| *** gbarros_ has joined #tripleo | 15:32 | |
| *** links has joined #tripleo | 15:32 | |
| *** gbarros has quit IRC | 15:33 | |
| akrivoka | jtomasek: seems like manage is used to put nodes back from available to manageable... so this operation does not need be a part of register -> introspect -> provide wrapping workflow, IIUC | 15:33 |
| *** mdnadeem has joined #tripleo | 15:37 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Add params needed for the ceph-ansible switch to containers playbook https://review.openstack.org/494246 | 15:41 |
| *** brault has quit IRC | 15:41 | |
| *** yolanda has quit IRC | 15:44 | |
| *** yolanda has joined #tripleo | 15:44 | |
| *** ccamacho has quit IRC | 15:44 | |
| *** brault has joined #tripleo | 15:45 | |
| *** ccamacho has joined #tripleo | 15:46 | |
| *** yamahata has joined #tripleo | 15:47 | |
| michapma | trozet: hey | 15:47 |
| trozet | michapma: hey are you adding those logging items to the scenario somehow? | 15:48 |
| michapma | trozet: I thought I did, https://review.openstack.org/#/c/494043/ | 15:48 |
| michapma | trozet: but I don't think it worked. | 15:48 |
| trozet | michapma: oh i didnt see this patch | 15:49 |
| *** links has quit IRC | 15:49 | |
| michapma | trozet: I only put it up today | 15:49 |
| *** jfrancoa has quit IRC | 15:50 | |
| trozet | michapma: we also need all those ovs outputs, is that also possible? | 15:50 |
| *** gbarros_ has quit IRC | 15:50 | |
| michapma | trozet: there is another part of oooq that I think runs some diagnostics so we could maybe add it there | 15:51 |
| trozet | michapma: ok cool | 15:51 |
| *** marios has quit IRC | 15:51 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Switch to scheduling based on resource classes https://review.openstack.org/490851 | 15:54 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Remove printing the introspection status https://review.openstack.org/494249 | 15:56 |
| michapma | trozet: ah, yep it might belong in here: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/collect.yml | 15:56 |
| michapma | trown: would that be a suitable location for some ODL diagnostics? only run when ODL is active of course. | 15:56 |
| *** pcaruana has quit IRC | 15:57 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Allow users to specify the Introspection concurrency https://review.openstack.org/451228 | 15:59 |
| trown | michapma: ya | 16:00 |
| michapma | trown: I might make an odl-collect and include that when the package is there so it doesn't get cluttered | 16:00 |
| trown | michapma: ya that makes sense | 16:01 |
| michapma | trown: I can see us wanting a bunch of these for different plugins and whatnot | 16:01 |
| trown | michapma: ya the collect-logs role is even used by some other teams without using quickstart | 16:01 |
| *** gbarros has joined #tripleo | 16:02 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Let users customise the node timeout for introspection https://review.openstack.org/494254 | 16:05 |
| *** ykarel is now known as ykarel|afk | 16:07 | |
| *** bogdando has quit IRC | 16:09 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 16:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 16:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| *** ramishra has quit IRC | 16:13 | |
| *** trown is now known as trown|lunch | 16:13 | |
| *** achadha has joined #tripleo | 16:14 | |
| *** anshul has quit IRC | 16:14 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs https://review.openstack.org/494043 | 16:16 |
| *** jlinkes has quit IRC | 16:18 | |
| michapma | trozet: I updated the patch, so the diags should be in an odl subdir in here for each run: http://logs.openstack.org/05/486905/21/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/df4e8e5/logs/subnode-2/var/log/extra/ | 16:19 |
| *** mcornea has quit IRC | 16:19 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 16:21 |
| *** jmelvin has quit IRC | 16:21 | |
| *** itlinux has quit IRC | 16:22 | |
| owalsh | lhinds: you still here? | 16:23 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: Use CENTOS_MIRROR for DLRN builds if available https://review.openstack.org/494262 | 16:25 |
| *** stendulker has joined #tripleo | 16:26 | |
| honza | apetrich: Green across the board, well done :) | 16:28 |
| *** nyechiel has joined #tripleo | 16:29 | |
| *** ccamacho has quit IRC | 16:32 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/newton: SSHD Service extensions https://review.openstack.org/494263 | 16:32 |
| *** florianf has quit IRC | 16:32 | |
| *** snecklifter has quit IRC | 16:35 | |
| *** yamahata has quit IRC | 16:36 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/newton: SSHD Service extensions https://review.openstack.org/494263 | 16:40 |
| *** iranzo has quit IRC | 16:43 | |
| *** _milan_ has quit IRC | 16:46 | |
| *** rcernin has joined #tripleo | 16:49 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on containers-multinode-upgrade https://review.openstack.org/494268 | 16:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 16:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 16:50 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Update SSL docs to refer to the generated environments https://review.openstack.org/494269 | 16:50 |
| EmilienM | weshay: with pingtest now, let's see | 16:50 |
| owalsh | EmilienM: Any idea why we use --validation-errors-nonfatal in the gate? https://review.openstack.org/492152 got through with the wrong heat template version | 16:51 |
| weshay | all eyes on zuul :) | 16:51 |
| *** jpich has quit IRC | 16:51 | |
| bnemec | owalsh: It's used because otherwise the Ironic node count validation will fail the deployment. | 16:52 |
| weshay | EmilienM, if I update https://review.openstack.org/#/c/494268/ will that affect the current test? | 16:53 |
| bnemec | owalsh: That wouldn't have been caught by the pre-deploy validations though, would it? | 16:53 |
| *** egonzalez has quit IRC | 16:53 | |
| EmilienM | weshay: what do you want to update? | 16:53 |
| weshay | EmilienM, the doc | 16:53 |
| owalsh | bnemec: yea, did when I deployed newton on my local env | 16:54 |
| EmilienM | weshay: I'll have to make recheck | 16:54 |
| *** lucasagomes is now known as lucas-afk | 16:54 | |
| weshay | meh.. I'll do it in another patch | 16:54 |
| EmilienM | weshay: ah, let me -1 it myself | 16:54 |
| weshay | EmilienM, oh jebus.. I'll just put a patch on top of it | 16:55 |
| EmilienM | :) | 16:55 |
| weshay | EmilienM, you're an idiot | 16:55 |
| bnemec | owalsh: Hmm. Maybe we should add that to the pep8 check: https://github.com/openstack/tripleo-heat-templates/blob/master/tools/yaml-validate.py#L431 | 16:55 |
| EmilienM | weshay: HR ! | 16:55 |
| weshay | ya.. go cry to HR.. THIS IS FREENODE | 16:56 |
| EmilienM | I'll call HR | 16:56 |
| mwhahaha | status confirmed | 16:56 |
| owalsh | bnemec: ahh... no... resource is set to None so it wouldn't have been included in the pre-deploy validation | 16:56 |
| owalsh | bnemec: but my cherry-pick in https://review.openstack.org/494263 enables the service by default so I found it in my local env | 16:57 |
| EmilienM | owalsh: I have no idea but it's sad :) | 16:57 |
| *** rhallisey has quit IRC | 16:57 | |
| EmilienM | owalsh: probably because we want to ignore some problems we have to fix | 16:57 |
| bnemec | owalsh: Yeah, that makes sense. | 16:57 |
| bnemec | EmilienM: <bnemec> owalsh: It's used because otherwise the Ironic node count validation will fail the deployment. | 16:57 |
| EmilienM | gfidente, fultonj : I got pingtest working with ceph-ansible | 16:58 |
| *** dprince has quit IRC | 16:58 | |
| fultonj | EmilienM: that's great! | 16:58 |
| EmilienM | gfidente, fultonj: err, but ceph isn't containerized yet nevermind | 16:58 |
| gfidente | so with puppet-ceph | 16:58 |
| owalsh | bnemec: sounds like something for pep8 alright | 16:58 |
| EmilienM | gfidente, fultonj : yes | 16:58 |
| EmilienM | gfidente: https://review.openstack.org/#/c/494176/ helped | 16:58 |
| gfidente | still good, when migrating to ceph-ansible it'll have to stay green | 16:58 |
| fultonj | yeah | 16:59 |
| gfidente | right so I think we can merge pingtest regardless | 16:59 |
| EmilienM | scenario004 is still red though, let me show you logs | 16:59 |
| *** jmelvin has joined #tripleo | 16:59 | |
| EmilienM | http://logs.openstack.org/29/490129/8/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container/fb47733/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-16_15_37_21 | 16:59 |
| EmilienM | so everything is containerized except ceph ^ | 16:59 |
| *** strigazi_OFF is now known as strigazi | 17:00 | |
| *** rhallisey has joined #tripleo | 17:02 | |
| openstackgerrit | Vineet Paul proposed openstack/tripleo-heat-templates master: Drop extraconfig for nova-nuage https://review.openstack.org/494239 | 17:02 |
| *** rhallisey has joined #tripleo | 17:02 | |
| *** shardy_afk is now known as shardy | 17:03 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: update fs doc, pingtest for fs011 https://review.openstack.org/494274 | 17:04 |
| weshay | EmilienM, ^ | 17:04 |
| adarazs | amoralej, weshay: I had my first jab at https://bugs.launchpad.net/tripleo/+bug/1710678 -- but I think I didn't find the right place to override the CentOS mirrors. let me know if you know where to do it. | 17:05 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,In progress] - Assigned to Attila Darazs (adarazs) | 17:05 |
| weshay | k | 17:06 |
| adarazs | I commented on the bug. and made a change that didn't seem to work on my local test. | 17:06 |
| adarazs | gotta log off for today. | 17:06 |
| *** derekh has quit IRC | 17:06 | |
| amoralej | adarazs, that will not help | 17:06 |
| adarazs | amoralej: I noticed. :) | 17:06 |
| amoralej | adarazs, let me explain how dlrn works in the LP | 17:06 |
| adarazs | amoralej: thanks, please do. I'll pick it up tomorrow | 17:07 |
| adarazs | or you can take over, if you have free cycles. :) | 17:07 |
| *** gbarros has quit IRC | 17:07 | |
| *** salmankhan has quit IRC | 17:09 | |
| EmilienM | weshay: thx | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 17:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 17:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| *** karthiks has quit IRC | 17:10 | |
| *** hewbrocca is now known as hewbrocca_afk | 17:10 | |
| gfidente | EmilienM fultonj the ceph logs can probably be capture from within the ceph containers | 17:11 |
| gfidente | I see there are commands in tripleo-ci/scripts/get_docker_logs.sh | 17:11 |
| gfidente | where we might add what we need | 17:11 |
| *** salmankhan has joined #tripleo | 17:11 | |
| gfidente | but I am going AFK for a while, will be back later, if you manage to get anything out add on review so I +2 :D | 17:12 |
| fultonj | ok, thanks gfidente i will look at that | 17:12 |
| gfidente | cool, tty | 17:12 |
| *** gfidente is now known as gfidente|afk | 17:12 | |
| *** ansmith has quit IRC | 17:13 | |
| *** yamahata has joined #tripleo | 17:15 | |
| *** salmankhan has quit IRC | 17:15 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder https://review.openstack.org/494008 | 17:16 |
| *** dtantsur is now known as dtantsur|afk | 17:19 | |
| jtomasek | akrivoka: sorry, I am back now | 17:19 |
| jtomasek | akrivoka: correct, the wrapping workflow shoudl do register > introspect > provide | 17:19 |
| jtomasek | akrivoka: also it would be nice if we could optionally decide which workflows will be run (make introspect and provide optional) so we can let user choose via checkbox | 17:19 |
| *** pkovar has quit IRC | 17:19 | |
| *** ykarel|afk has quit IRC | 17:21 | |
| amoralej | adarazs, i just commented in https://bugs.launchpad.net/tripleo/+bug/1710678 let me know if it's clear enough | 17:21 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,In progress] - Assigned to Attila Darazs (adarazs) | 17:21 |
| *** dougbtv_ has joined #tripleo | 17:21 | |
| amoralej | tricky part is generate custom.cfg | 17:21 |
| *** ansmith has joined #tripleo | 17:22 | |
| *** gbarros has joined #tripleo | 17:23 | |
| *** shardy has quit IRC | 17:23 | |
| *** dougbtv has quit IRC | 17:24 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Document relation between self-signed certs and parameters https://review.openstack.org/494278 | 17:25 |
| *** gkadam has quit IRC | 17:25 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on 2 scenarios container jobs https://review.openstack.org/490129 | 17:26 |
| *** dprince has joined #tripleo | 17:26 | |
| EmilienM | weshay: ^ ok this patch should work (until we fix the rest), so we can have pingtest on 2 container jobs from now | 17:26 |
| EmilienM | weshay: little progress :) | 17:27 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: README: Fix CI coverage layout https://review.openstack.org/494279 | 17:28 |
| weshay | EmilienM++ | 17:29 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs https://review.openstack.org/493973 | 17:33 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Exclude list for logs collection https://review.openstack.org/494022 | 17:35 |
| *** dsneddon is now known as dsneddon_pto | 17:36 | |
| *** achadha has quit IRC | 17:36 | |
| *** rlandy is now known as rlandy|brb | 17:37 | |
| *** achadha has joined #tripleo | 17:38 | |
| *** itlinux has joined #tripleo | 17:39 | |
| *** agurenko has quit IRC | 17:40 | |
| *** jtomasek is now known as jtomasek|afk | 17:41 | |
| *** tosky has quit IRC | 17:41 | |
| weshay | need reviews https://review.openstack.org/#/c/487598/ | 17:41 |
| weshay | hrm.. | 17:42 |
| weshay | : [Errno 256] No more mirrors to try.", "2017-08-16 17:14:36 | http://mirror.ord.rax.openstack.org:8080/buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: [Errno -1] Error importing repomd.xml for delorean-pike-testing: Damaged repomd.xml file"], "warnings": []} | 17:43 |
| *** achadha has quit IRC | 17:43 | |
| weshay | pabelanger, ^ | 17:43 |
| weshay | http://logs.openstack.org/00/461000/36/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/b33e3ab/console.html#_2017-08-16_17_14_36_350085 | 17:43 |
| pabelanger | weshay: we are going to land https://review.openstack.org/494265/ | 17:44 |
| pabelanger | is this might be related | 17:44 |
| pabelanger | weshay: tl;dr repdata from buildlogs.cdn.centos.org is likely out of date | 17:44 |
| weshay | k | 17:45 |
| weshay | EmilienM, fyi.. ^ your pingtest jobs hit a bump, unrelated to your work | 17:48 |
| EmilienM | ok | 17:48 |
| EmilienM | you wanna make me cry right? | 17:48 |
| *** stendulker has quit IRC | 17:50 | |
| *** ebarrera has joined #tripleo | 17:51 | |
| *** anshul has joined #tripleo | 17:51 | |
| *** trown|lunch is now known as trown | 17:52 | |
| *** milan has joined #tripleo | 17:53 | |
| *** achadha has joined #tripleo | 17:54 | |
| *** gbarros has quit IRC | 17:55 | |
| *** achadha has quit IRC | 17:55 | |
| *** strigazi is now known as strigazi_OFF | 17:55 | |
| *** achadha has joined #tripleo | 17:55 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs010: enable tempest https://review.openstack.org/494284 | 17:56 |
| EmilienM | weshay: early try of running tempest again on container-multinode | 17:56 |
| EmilienM | weshay: just to see what would block us | 17:57 |
| weshay | Slower, ^ | 17:58 |
| *** achadha_ has joined #tripleo | 17:58 | |
| weshay | getting tempest running against the container jobs upstream is something that is a priority atm | 17:58 |
| *** achadha_ has quit IRC | 17:59 | |
| *** achadha has quit IRC | 17:59 | |
| Slower | weshay: ok cool. I've worked with tempest before too | 18:01 |
| weshay | sshnaidm|afk, are you on? | 18:01 |
| sshnaidm|afk | weshay, yep | 18:01 |
| weshay | sshnaidm|afk, do you recall the issues we had w/ using ovb + net-iso + containers? | 18:01 |
| *** rlandy|brb is now known as rlandy | 18:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: move creating fake image to oooq extras https://review.openstack.org/494235 | 18:02 |
| *** achadha has joined #tripleo | 18:02 | |
| weshay | I'm at a loss here, and I was going to try running the upstream ovb playbook w/ the ./devmode.sh path | 18:02 |
| EmilienM | arxcruz: hey, you still around? | 18:02 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: test creating fake image in oooq extras https://review.openstack.org/494233 | 18:02 |
| sshnaidm|afk | weshay, network isolation didn't work with containers | 18:03 |
| weshay | sshnaidm|afk, do you have time to chat about that | 18:03 |
| EmilienM | arxcruz: I think I found out why scenario001 doesn't run Telemetry autoscaling scenario test, it's beacuse "panko" isn't enabled in tempest.conf. Can we fix that? | 18:03 |
| weshay | sshnaidm|afk, you tried multinic right? | 18:03 |
| sshnaidm|afk | weshay, I think both singlenic, but singlenic anyway wouldn't work with upstream ci | 18:03 |
| sshnaidm|afk | weshay, so mostly multinic, yes | 18:03 |
| weshay | sshnaidm|afk, right... but single-nic should work w/ ovb in devmode | 18:04 |
| weshay | maybe I'll go that route again | 18:04 |
| weshay | sshnaidm|afk, Slower ah.. so the upstream container jobs also have net-iso turned off | 18:05 |
| weshay | https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset016.yml#L5 | 18:05 |
| weshay | rlandy, ^ | 18:05 |
| *** nyechiel has quit IRC | 18:05 | |
| weshay | so the only confirmed network-isolation + containers is single-nic-vlan | 18:06 |
| weshay | or none | 18:06 |
| weshay | ok.. I think that's our answer.. and something we need to figure out | 18:06 |
| sshnaidm|afk | weshay, maybe it's worth to try multinic again.. | 18:06 |
| weshay | EmilienM, dprince, Slower are there any notes on using network isolation w/ containerized deployments? | 18:07 |
| Slower | ok | 18:07 |
| rlandy | weshay: I have one successful deploy with net iso | 18:07 |
| weshay | rlandy, on ovb? | 18:07 |
| weshay | which net-iso? | 18:07 |
| Slower | weshay: not that I know of | 18:07 |
| weshay | this may be a problem gentleman | 18:07 |
| *** tosky has joined #tripleo | 18:07 | |
| weshay | men | 18:07 |
| dprince | weshay: should be the same notes as BM actually | 18:07 |
| weshay | and ladies | 18:07 |
| weshay | ok | 18:07 |
| Slower | dprince: yeah | 18:08 |
| dprince | weshay: we use --net=host which mostly covers us | 18:08 |
| Slower | it shouldn't be any different really | 18:08 |
| weshay | rlandy, what worked for you.. libvirt + single-nic is known good.. we know that much | 18:08 |
| dprince | weshay: anything in particular you are hitting issues w/ | 18:08 |
| weshay | dprince, I'll put some information together.. and send it to you guys | 18:09 |
| weshay | dprince, Slower just in full disclosure afaict.. all the container jobs upstream have net-iso turned off.. we have one job in rdo that runs w/ single-nic | 18:09 |
| *** pcaruana has joined #tripleo | 18:09 | |
| trozet | michapma: thanks | 18:10 |
| weshay | rlandy, ? | 18:10 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 18:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 18:10 |
| rlandy | weshay: this was net iso - no pacemaker - https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-devmode-master-ovb-rdocloud-public-bond-168/undercloud/home/stack/. Successful deploy but validate failed | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| Slower | hmm, so I rebooted my compute node and now I can't even ssh into it | 18:10 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 18:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 18:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| sshnaidm|afk | weshay, https://bugs.launchpad.net/tripleo/+bug/1689556 | 18:10 |
| openstack | Launchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [High,In progress] | 18:10 |
| weshay | rlandy, ok.. that is great news | 18:11 |
| weshay | rlandy, looks like the overcloud had no dns | 18:11 |
| weshay | 2017-08-15 23:12:30 | curl: (6) Could not resolve host: download.cirros-cloud.net; Name or service not known | 18:11 |
| weshay | rlandy, can you share that config so that Slower and I can poke at it | 18:11 |
| rlandy | weshay: my runs since then are failing before I get to deploy - | 18:11 |
| weshay | doh | 18:11 |
| rlandy | on prep containerized deployment | 18:12 |
| *** rhallisey has quit IRC | 18:13 | |
| Slower | bleh | 18:13 |
| weshay | rlandy, k.. point me at the config | 18:13 |
| rlandy | that job was my last shot at running through the devmode workflow | 18:14 |
| rlandy | $WORKSPACE/tripleo-quickstart/devmode.sh --ovb --no-gate --delete-all-stacks -CD | 18:14 |
| weshay | oh | 18:14 |
| rlandy | since jkilpatr said all he did was set containerized_overcloud to be true, I tried that again | 18:15 |
| rlandy | the featuresets use deploy_composable_scenario: true | 18:16 |
| jkilpatr | well it does work, just did it today, validations don't but the deploy runs all the way through | 18:16 |
| *** nyechiel has joined #tripleo | 18:16 | |
| rlandy | are we supposed to be running with deploy_composable_scenario? | 18:16 |
| rlandy | weshay: ^^? | 18:16 |
| Slower | rlandy: this is with the default-to-containers patch right? | 18:16 |
| rlandy | Slower: yes | 18:16 |
| Slower | ok | 18:17 |
| *** rhallisey has joined #tripleo | 18:17 | |
| Slower | I ran it yesterday and got some strange error | 18:17 |
| rlandy | Slower: I did two extra things ... | 18:17 |
| Slower | no attribute 'rc' or something | 18:17 |
| rlandy | -e undercloud_flavor=m1.xlarge -e enable_pacemaker=false | 18:18 |
| Slower | I posted it in the patch actually | 18:18 |
| Slower | rlandy: ok cool | 18:18 |
| * rlandy checks patch | 18:18 | |
| Slower | rlandy: I just started it again 10 mins ago | 18:18 |
| rlandy | Slower: the -CD avoids the message on containers | 18:20 |
| rlandy | but sshnaidm|afk also objected to that | 18:20 |
| rlandy | we wanted to warn people that they were getting a containerized overcloud | 18:21 |
| *** mcornea has joined #tripleo | 18:24 | |
| weshay | rlandy, this is in a patch? <rlandy> -e undercloud_flavor=m1.xlarge -e enable_pacemaker=false | 18:24 |
| rlandy | weshay: no - just testing with it atm | 18:25 |
| weshay | rlandy, k.. good idea | 18:25 |
| rlandy | and you mentioned that pacemaker should make no diff | 18:25 |
| rlandy | weshay: we need to rethink the prompts for deploying with containers | 18:25 |
| weshay | ya.. it certainly won't deploy containerized ha | 18:25 |
| rlandy | initially we agreed to prompt the user to agree to a containerized deployment | 18:26 |
| weshay | you.. I think we can make it default if release = pike or master | 18:26 |
| rlandy | both Slower and sshnaidm|afk are objecting to that | 18:26 |
| rlandy | it is | 18:26 |
| rlandy | but we still stop and tell people | 18:26 |
| weshay | yes | 18:27 |
| rlandy | weshay: so - is there agreement now that we no longer stop and tell users | 18:27 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario003 to run Tempest https://review.openstack.org/494290 | 18:27 |
| rlandy | you can avoid the questions with -CD or -ND options | 18:28 |
| *** mcornea has quit IRC | 18:31 | |
| *** rhallisey has quit IRC | 18:34 | |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Disables QoS with OpenDaylight deployments https://review.openstack.org/492027 | 18:34 |
| pabelanger | weshay: okay, seeing Error importing repomd.xml for delorean-pike-testing: Damaged repomd.xml file in more then 1 place now. Mirrors should be updated shortly, but is the repo file actually valid? | 18:34 |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates master: Add support for deploying RGW with ceph-ansible https://review.openstack.org/480804 | 18:34 |
| *** salmankhan has joined #tripleo | 18:34 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Stricter heat template version validation https://review.openstack.org/494292 | 18:37 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario007 to run Tempest https://review.openstack.org/494293 | 18:38 |
| EmilienM | weshay: so, scenario001, 002 and 004 have some blockers that you already know. 003 and 007 are under testing. That's the progress done on "running tempest instead of pingtest" | 18:39 |
| pabelanger | amoralej: is the repo file for delorean-pike-testing corrupt? | 18:41 |
| pabelanger | amoralej: or just badly out of sync because of 494265 | 18:42 |
| amoralej | pabelanger, it's just out of sync | 18:43 |
| amoralej | only jobs requiring last changes in the repo are failing | 18:43 |
| *** gbarros has joined #tripleo | 18:43 | |
| arxcruz | EmilienM: working on that, deploying a local machine to test | 18:44 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario002 to run Tempest https://review.openstack.org/491102 | 18:44 |
| EmilienM | arxcruz: ok thx | 18:44 |
| pabelanger | amoralej: okay, thanks. mirrors should be updating in 15-30mins | 18:44 |
| amoralej | nice, thx for your help | 18:45 |
| rlandy | Slower: for Pike and later, the default is containerized overcloud and containerized undercloud? | 18:46 |
| rlandy | ie: what is the expected undercloud status wrt containers | 18:46 |
| *** amoralej is now known as amoralej|off | 18:47 | |
| *** nyechiel has quit IRC | 18:47 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario004 to run Tempest https://review.openstack.org/491113 | 18:47 |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila Share for HA https://review.openstack.org/482680 | 18:48 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 18:49 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Document relation between self-signed certs and parameters https://review.openstack.org/494278 | 18:52 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Update SSL docs to refer to the generated environments https://review.openstack.org/494269 | 18:52 |
| *** anshul has quit IRC | 18:55 | |
| *** _milan_ has joined #tripleo | 18:55 | |
| Slower | rlandy: there's no plan to containerize undercloud atm | 18:57 |
| *** milan has quit IRC | 18:57 | |
| *** _milan_ has quit IRC | 18:57 | |
| Slower | rlandy: several of us devs would like to just to simplify things | 18:57 |
| Slower | rlandy: I mean to really switch to it | 18:58 |
| Slower | right now it's just a developmen tool | 18:58 |
| rlandy | Slower: thanks - will make the options explicit to containerized_overcloud for now | 18:58 |
| Slower | rlandy: perfect | 18:58 |
| Slower | rlandy: mostly we just don't have the time this cycle to deal with containerizing undercloud | 18:59 |
| Slower | so don't worry about testing it | 18:59 |
| Slower | I think there may be one job that checks it or something, not sure | 18:59 |
| *** _milan_ has joined #tripleo | 18:59 | |
| Slower | larsks: was it you that had the super nifty paste cmd? | 18:59 |
| *** mdnadeem has quit IRC | 19:00 | |
| larsks | Slower: possibly! curl -T <anything> -s chunk.io | 19:00 |
| larsks | Or, for stdin, <anything> | curl -T- -s chunk.io | 19:00 |
| Slower | larsks: that's it, thanks! | 19:00 |
| openstackgerrit | Merged openstack/tripleo-docs master: add how to check gate with tripleo to the contrib doc https://review.openstack.org/487598 | 19:01 |
| *** gbarros has quit IRC | 19:07 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 19:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 19:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 19:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| weshay | weshay | 19:10 |
| weshay | heh | 19:10 |
| trown | trown | 19:10 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 19:11 |
| EmilienM | weshay: thoughts on https://review.openstack.org/#/c/494039/ ? | 19:12 |
| pabelanger | EmilienM: yes please | 19:15 |
| EmilienM | pabelanger: I already voted :D | 19:15 |
| trown | EmilienM: we know the ssh fix is in the latest package? | 19:18 |
| trown | EmilienM: aiui the original issue was intermittent, so a single CI run doesnt really show us whether it is still present | 19:18 |
| mwhahaha | the bz was closed out 2 weeks ago and current version upstream is 6.6.1p1-35 | 19:19 |
| mwhahaha | meaning we're installing an older version | 19:20 |
| pabelanger | trown: right, if these is still an issue however, we should work on it in openstack-infra, so all centos-7 jobs benefit. I recomment revert and setup elastic-recheck query if we need to track it | 19:24 |
| trown | k, into the queue it goes | 19:25 |
| trown | when it comes out... nobody knows | 19:25 |
| trown | I will see myself out | 19:25 |
| mwhahaha | you don't get to leave | 19:25 |
| trown | hehe | 19:25 |
| EmilienM | lol | 19:25 |
| *** pchavva has quit IRC | 19:26 | |
| *** trown is now known as trown|brb | 19:29 | |
| *** rhallisey has joined #tripleo | 19:30 | |
| *** salmankhan has quit IRC | 19:35 | |
| *** trown|brb is now known as trown | 19:39 | |
| trown | 110230 | 19:39 |
| weshay | Slower, https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/build-test-packages | 19:41 |
| pabelanger | weshay: amoralej|off: mirrors have been updated to first hit buildlogs.centos.org now | 19:41 |
| *** jprovazn has quit IRC | 19:43 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add NeutronOverlayIPVersion parameter to neutron-plugins-ml2 service https://review.openstack.org/466162 | 19:46 |
| *** rcernin has quit IRC | 19:55 | |
| *** homeski has joined #tripleo | 19:56 | |
| *** jpena is now known as jpena|off | 19:58 | |
| *** sbrzozow has quit IRC | 19:59 | |
| *** _milan_ has quit IRC | 20:01 | |
| *** milan has joined #tripleo | 20:01 | |
| *** homeski has quit IRC | 20:02 | |
| *** homeski has joined #tripleo | 20:03 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 20:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 20:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 20:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| *** trozet has quit IRC | 20:11 | |
| *** akrivoka has quit IRC | 20:11 | |
| *** liverpooler has quit IRC | 20:12 | |
| openstackgerrit | Merged openstack/tripleo-common master: Provide a list of unused user env parameters in plan https://review.openstack.org/491753 | 20:17 |
| *** dprince has quit IRC | 20:18 | |
| *** homeski_ has joined #tripleo | 20:18 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Fix rpms being installed via DeployArtifactURLs https://review.openstack.org/493867 | 20:20 |
| *** ansmith has quit IRC | 20:20 | |
| *** homeski has quit IRC | 20:20 | |
| *** gbarros has joined #tripleo | 20:21 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: Update Pike storage documentation for ceph-ansible https://review.openstack.org/487155 | 20:23 |
| *** akrivoka has joined #tripleo | 20:23 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Configure the set of tests to execute in scenario001 https://review.openstack.org/490376 | 20:25 |
| *** trozet has joined #tripleo | 20:26 | |
| *** trown is now known as trown|outtypewww | 20:28 | |
| *** achadha has quit IRC | 20:29 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario007 to run Tempest https://review.openstack.org/494293 | 20:29 |
| *** achadha has joined #tripleo | 20:30 | |
| *** achadha has quit IRC | 20:31 | |
| EmilienM | pabelanger, mwhahaha: do you remember if we mirror cirros image somewhere? (I'm not joking) | 20:31 |
| EmilienM | http://logs.openstack.org/84/494284/1/check/gate-tripleo-ci-centos-7-containers-multinode/b45c603/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-08-16_19_30_43 | 20:31 |
| mwhahaha | yea | 20:31 |
| mwhahaha | it's local on the img | 20:31 |
| *** achadha has joined #tripleo | 20:32 | |
| mwhahaha | in a folder | 20:32 |
| mwhahaha | EmilienM: https://review.openstack.org/#/c/475479/\ | 20:32 |
| EmilienM | ok thxw | 20:33 |
| *** achadha_ has joined #tripleo | 20:34 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs010: enable tempest https://review.openstack.org/494284 | 20:35 |
| EmilienM | mwhahaha: mhh, I'm wondering about the failure though | 20:36 |
| *** jcoufal_ has joined #tripleo | 20:36 | |
| EmilienM | mwhahaha: sounds like the image is uploaded in config_tempest | 20:36 |
| EmilienM | not in oooq | 20:36 |
| mwhahaha | EmilienM: right we probably need to be configuring the source in tempest | 20:37 |
| mwhahaha | previously oooq did it for pingtest | 20:37 |
| *** homeski has joined #tripleo | 20:37 | |
| *** achadha has quit IRC | 20:38 | |
| EmilienM | mwhahaha: indeed, it's not http://logs.openstack.org/84/494284/1/check/gate-tripleo-ci-centos-7-containers-multinode/b45c603/logs/undercloud/home/jenkins/tempest/etc/tempest.conf.txt.gz | 20:38 |
| *** achadha_ has quit IRC | 20:39 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on containers-multinode-upgrade https://review.openstack.org/494268 | 20:39 |
| *** jcoufal has quit IRC | 20:39 | |
| *** homeski_ has quit IRC | 20:40 | |
| *** trozet has quit IRC | 20:40 | |
| *** pcaruana has quit IRC | 20:41 | |
| *** jlabarre has quit IRC | 20:47 | |
| weshay | rlandy, I think moving to xlarge will make a difference in the container deployment | 20:47 |
| *** jlabarre has joined #tripleo | 20:47 | |
| weshay | 16gb memory is the requirement | 20:48 |
| rlandy | weshay: ok - will add that to another review | 20:48 |
| weshay | rlandy, is your job still running | 20:48 |
| rlandy | weshay: it just failed deploy | 20:48 |
| weshay | ugh | 20:48 |
| * weshay takes a look | 20:48 | |
| rlandy | contacting Slower to either debug or move on to script testing | 20:49 |
| *** brault has quit IRC | 20:49 | |
| rlandy | Slower: ^^ got a failed deployment this time | 20:49 |
| rlandy | https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-devmode-master-ovb-rdocloud-public-bond-172/undercloud/home/stack/ | 20:50 |
| rlandy | which we can either debug or remove to work on the script | 20:50 |
| weshay | rlandy, Slower "status_reason": "Deployment cancelled.", | 20:50 |
| *** ecerquei_ has quit IRC | 20:50 | |
| rlandy | out of time? | 20:51 |
| rlandy | 2017-08-16 20:18:29 | 2017-08-16 20:17:56Z [overcloud]: CREATE_FAILED Create timed out | 20:51 |
| rlandy | could be that step 3 stalled | 20:52 |
| *** jcoufal_ has quit IRC | 20:52 | |
| rlandy | weshay: can rerun overcloud deploy with a longer timeout | 20:52 |
| *** dciabrin has quit IRC | 20:52 | |
| rlandy | worth the try? | 20:53 |
| weshay | sure.. | 20:53 |
| *** trozet has joined #tripleo | 20:53 | |
| *** ebarrera has quit IRC | 20:53 | |
| rlandy | ok - trying deploy again | 20:54 |
| *** atoth has quit IRC | 20:54 | |
| *** rhallisey has quit IRC | 20:54 | |
| radez | slagle: I'm going to jump to #triple-o my question is not so ironic related I odn't think | 20:57 |
| radez | the two messages were: | 20:57 |
| radez | (heat-config) [ERROR] Skipping group os-apply-config with no hook script None | 20:57 |
| radez | os-collect-config[2023]: No hosts in Heat, nothing written. | 20:57 |
| radez | do either of those indicate bad things? my hosts are not running puppet when they come up and I'm trying to figure out why | 20:58 |
| *** achadha has joined #tripleo | 20:59 | |
| *** _milan_ has joined #tripleo | 21:03 | |
| *** milan has quit IRC | 21:03 | |
| *** dciabrin has joined #tripleo | 21:05 | |
| Slower | rlandy: I've seen that too.. the cancelled thing | 21:06 |
| rlandy | Slower: I am trying the deploy again with a longer timeout | 21:06 |
| rlandy | did you see the canceled deploy only on ovb for elsewhere? | 21:06 |
| Slower | only on ovb | 21:07 |
| Slower | maybe a resource limit thing? | 21:07 |
| rlandy | default timeout of 90 mins | 21:07 |
| rlandy | but it should not take that long | 21:08 |
| Slower | I'm getting a failure in fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "\"/home/stack\"/validate-overcloud-ipmi-connection.sh" | 21:08 |
| *** shreshtha_ has joined #tripleo | 21:10 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 21:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 21:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 21:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| pabelanger | weshay: where did we land on the daemon.json debug:true again? We need your patches? | 21:10 |
| rlandy | Slower: I don't see the ipmi connection failure | 21:11 |
| weshay | ah geez, ya.. let me do that now | 21:11 |
| weshay | before it slips my mind again, sorry | 21:11 |
| pabelanger | sure, np. | 21:11 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add ServiceData to hidden params https://review.openstack.org/493244 | 21:11 |
| sshnaidm|afk | pabelanger, just fyi http://paste.openstack.org/show/618576/ | 21:12 |
| *** lblanchard has quit IRC | 21:12 | |
| *** shreshtha has quit IRC | 21:12 | |
| sshnaidm|afk | pabelanger, can it be fixed? happened in two jobs | 21:12 |
| pabelanger | sshnaidm|afk: yes, that should be fixed https://review.openstack.org/494265/ | 21:13 |
| pabelanger | sshnaidm|afk: if it happens again, please ping me | 21:13 |
| pabelanger | issue is repodata on buildlogs.cdn.centos.org was stale | 21:13 |
| sshnaidm|afk | pabelanger, greate | 21:13 |
| pabelanger | now we get it directly from buildlogs.centos.org | 21:13 |
| openstackgerrit | Merged openstack/tripleo-ui master: Don't log app state in Zaqar https://review.openstack.org/493863 | 21:17 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493728 | 21:17 |
| weshay | pabelanger, ok.. check my patches here.. https://review.openstack.org/#/q/topic:bug/1710533 | 21:18 |
| weshay | https://review.openstack.org/#/c/493728/7/roles/overcloud-prep-containers/templates/docker_daemon.json.j2 | 21:18 |
| *** abishop has quit IRC | 21:19 | |
| pabelanger | thanks I'll look shortly | 21:20 |
| *** catintheroof has quit IRC | 21:21 | |
| *** itlinux has quit IRC | 21:21 | |
| *** eck`gone is now known as eck` | 21:26 | |
| openstackgerrit | Merged openstack/tripleo-ui master: Fix plan export URL https://review.openstack.org/488164 | 21:27 |
| slagle | radez: it has to do with you having some Heat deployments that are of type "group: os-apply-config", yet in your images, you do not have that hook script installed (it's deprecated) | 21:32 |
| slagle | radez: sounds like you may be mixing old templates with newer images | 21:32 |
| *** jlabarre has quit IRC | 21:33 | |
| *** nyechiel has joined #tripleo | 21:35 | |
| *** cylopez has joined #tripleo | 21:36 | |
| *** cylopez has quit IRC | 21:36 | |
| *** achadha has quit IRC | 21:39 | |
| *** achadha has joined #tripleo | 21:43 | |
| *** achadha has quit IRC | 21:44 | |
| *** achadha has joined #tripleo | 21:45 | |
| *** thrash is now known as thrash|g0ne | 21:48 | |
| *** jmelvin has quit IRC | 21:49 | |
| *** gfidente|afk is now known as gfidente | 21:50 | |
| EmilienM | gfidente: we are close :D | 21:54 |
| gfidente | EmilienM hey yeah I think it failed before getting to the real thing | 21:55 |
| gfidente | we'll see | 21:55 |
| gfidente | but it's terrible | 21:55 |
| gfidente | that we're late | 21:56 |
| gfidente | I know | 21:56 |
| gfidente | EmilienM actually, do you know if for upgrade patches we're using a special branch topic, always referring to a LP bug, creating as many LP bugs? | 21:57 |
| EmilienM | gfidente: not atfik | 21:59 |
| EmilienM | afik | 21:59 |
| EmilienM | gfidente: i'm having issue to test upgrades locally right now with scenarios but i keep trying | 22:00 |
| EmilienM | gfidente: have we done some tests on upgrading ocata to pike with ceph-ansible on pike already? | 22:00 |
| gfidente | yeah was talking about that with mcornea earlier today about that | 22:01 |
| gfidente | we did some | 22:01 |
| gfidente | and we need patches | 22:01 |
| gfidente | one of which I was working on right now | 22:01 |
| EmilienM | do we know if upgrades to containers doesn't blow up the cloud? | 22:01 |
| EmilienM | has someone tried that? | 22:01 |
| gfidente | the playbook existed in ceph-ansible | 22:01 |
| *** achadha_ has joined #tripleo | 22:01 | |
| gfidente | from a while | 22:01 |
| gfidente | but for me it didn't work yet | 22:01 |
| gfidente | I suppose some versions of it worked | 22:02 |
| gfidente | and it looks like there is CI in ceph for it | 22:02 |
| gfidente | https://github.com/ceph/ceph-ansible/issues/1758 | 22:02 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 22:04 |
| weshay | rlandy, Slower we may need to check if we have a proxy setup in rdo cloud re: containers | 22:04 |
| EmilienM | gfidente: I continue the tests ^ and try to see if pingtest pass | 22:05 |
| *** achadha has quit IRC | 22:05 | |
| gfidente | EmilienM so the existing pingtest submission is testing puppet-ceph | 22:05 |
| gfidente | I think if that passes and we merge it | 22:05 |
| *** achadha_ has quit IRC | 22:05 | |
| gfidente | then the change above will also only pass if pingtest passes | 22:06 |
| gfidente | which sounds fine to me | 22:06 |
| EmilienM | gfidente: yeah I have another patch coming | 22:06 |
| gfidente | for which change? | 22:07 |
| *** cdearborn_ has quit IRC | 22:07 | |
| gfidente | I mean, to change what | 22:07 |
| *** _milan_ has quit IRC | 22:07 | |
| *** _milan_ has joined #tripleo | 22:08 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 22:09 |
| *** jkilpatr has quit IRC | 22:09 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on 2 scenarios container jobs https://review.openstack.org/490129 | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 22:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 22:10 |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 22:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 22:10 |
| EmilienM | gfidente: ok it's a bit a spaguetti but it should test pingtest now on ^ | 22:10 |
| EmilienM | let's see how it goes | 22:10 |
| gfidente | I don't see differences, the rebase yes but not changes | 22:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge https://review.openstack.org/461000 | 22:12 |
| EmilienM | gfidente: no change, just a spaguetti on depends-on :D | 22:13 |
| EmilienM | gfidente: I didn't touch your code | 22:13 |
| EmilienM | gfidente: I'm just trying to test things now | 22:13 |
| *** achadha has joined #tripleo | 22:13 | |
| gfidente | EmilienM whatever | 22:14 |
| gfidente | trusted | 22:14 |
| * gfidente looking at the upgrade thing :D | 22:14 | |
| EmilienM | :) | 22:14 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario003 to run Tempest https://review.openstack.org/494290 | 22:17 |
| *** homeski has quit IRC | 22:17 | |
| EmilienM | gfidente: right now, this is my blocker for testing upgrades in CI | 22:20 |
| EmilienM | http://logs.openstack.org/00/461000/36/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container-upgrades-nv/defe4a7/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-08-16_18_20_12 | 22:20 |
| EmilienM | it looks like it takes pike environment with THT on ocata | 22:20 |
| EmilienM | docker/services/manila-api.yaml doesn't exist on ocata | 22:20 |
| *** nyechiel has quit IRC | 22:21 | |
| EmilienM | mhh wait, I found something weird though | 22:21 |
| EmilienM | https://github.com/openstack/tripleo-heat-templates/blob/stable/ocata/ci/environments/scenario004-multinode-containers.yaml#L17 | 22:21 |
| EmilienM | why do we do this | 22:21 |
| gfidente | because the service is disabled by default | 22:22 |
| EmilienM | so | 22:22 |
| EmilienM | it's weird because I manually backported https://review.openstack.org/#/c/489874/ in the RPM | 22:22 |
| EmilienM | but it sounds like it didn't work | 22:22 |
| EmilienM | ok so I'm reverting the custom patch in the spec file: https://review.rdoproject.org/r/#/c/8575 | 22:24 |
| EmilienM | and i'll try to merge https://review.openstack.org/#/c/489874/ | 22:24 |
| EmilienM | even if we have to temporarly disable voting on the upgrade jobs just the time to have this one | 22:24 |
| gfidente | rebase? | 22:26 |
| EmilienM | gfidente: where? | 22:26 |
| gfidente | 874b6fce-9383-4586-ae02-c13869070352 | 22:26 |
| gfidente | sorry https://review.openstack.org/#/c/489874/7 | 22:26 |
| EmilienM | gfidente: no need to rebase | 22:27 |
| gfidente | ah the merge conflict is in delorean | 22:27 |
| EmilienM | it's HEAD already | 22:27 |
| EmilienM | yeah | 22:27 |
| *** pradk has quit IRC | 22:27 | |
| EmilienM | I'm going to follow that plan ^ | 22:27 |
| *** achadha has quit IRC | 22:33 | |
| *** achadha has joined #tripleo | 22:34 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: overcloud_containers.yaml.j2 map images to services https://review.openstack.org/448328 | 22:35 |
| *** achadha_ has joined #tripleo | 22:36 | |
| *** akrivoka has quit IRC | 22:37 | |
| pabelanger | EmilienM: weshay: mwhahaha: pretty sure the job timeout issues we are now seeing on some tripleo jobs are because of DLRN http://logs.openstack.org/19/494219/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/5230441/console.html#_2017-08-16_14_42_15_518071 alone is trying for 30mins to build-test-packages | 22:39 |
| EmilienM | ouch | 22:39 |
| *** achadha has quit IRC | 22:39 | |
| pabelanger | It would be helpfully to collect the DLRN logs there | 22:39 |
| weshay | geez | 22:39 |
| pabelanger | and see what is going one | 22:39 |
| pabelanger | on* | 22:40 |
| weshay | pabelanger, EmilienM I know adarazs and amoralej|off were looking at it | 22:40 |
| pabelanger | however, I think, because this is infracloud, and DLRN still hitting upstream mirrors just don't have bandwidth to download everything | 22:40 |
| pabelanger | so, really need it using AFS mirrors | 22:40 |
| *** achadha_ has quit IRC | 22:41 | |
| pabelanger | aside from repodata issue today with buildlogs, every time I've seen a gate reset is because job timed out. And I am pretty sure, if we audit these failures, going to point to DLRN | 22:41 |
| pabelanger | weshay: EmilienM: telnet://15.184.66.45:19885 that job is stuck on DLRN | 22:42 |
| pabelanger | same task | 22:42 |
| weshay | NEWMAN | 22:42 |
| pabelanger | build-test-packages | 22:42 |
| pabelanger | about to time out after 2.5 hours | 22:42 |
| pabelanger | which will reset gate | 22:42 |
| pabelanger | weshay: EmilienM: And if that didn't timeout, we'd likely have about 10 merges happing | 22:43 |
| weshay | jebus | 22:43 |
| pabelanger | happening* | 22:44 |
| pabelanger | because everything behind that 1 job, has passed, or about to pass | 22:44 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Swith to the appropriate ceph-ansible playbook on upgrade https://review.openstack.org/494336 | 22:45 |
| pabelanger | okay, I just put a hold on 10478149, want to see what it is doing | 22:45 |
| weshay | I wonder if delorean was using mirrors previously in ci | 22:46 |
| weshay | in tripleo.sh | 22:46 |
| pabelanger | Hmm | 22:46 |
| pabelanger | weshay: in quickstart do you overwrite ~root/authorized_keys? | 22:47 |
| pabelanger | cause I can't ssh into that node | 22:47 |
| weshay | no | 22:47 |
| pabelanger | k, I'll have to hit it from a zuul-launcher | 22:47 |
| pabelanger | 1 sec | 22:47 |
| weshay | that would be a shitty trick | 22:47 |
| *** jkilpatr has joined #tripleo | 22:48 | |
| *** brault has joined #tripleo | 22:49 | |
| pabelanger | ya, something overwrote ~/root/.ssh/authorized_keys | 22:50 |
| pabelanger | I'll figure that out later | 22:50 |
| weshay | here is the old build https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L436-L527 | 22:51 |
| * weshay compares it to the oooq | 22:51 | |
| pabelanger | it hung doing a git clone | 22:52 |
| weshay | <sigh> | 22:53 |
| * weshay updates the bug | 22:53 | |
| pabelanger | looks like maybe cloning from github.com? | 22:53 |
| pabelanger | I guess for package -dist? | 22:53 |
| *** brault has quit IRC | 22:54 | |
| pabelanger | {{ artg_rdoinfo_repo_url }} currently looking to see what that would be | 22:55 |
| pabelanger | https://github.com/redhat-openstack/rdoinfo ? | 22:55 |
| pabelanger | k, looks like a config file needed for DLRN | 22:56 |
| *** achadha has joined #tripleo | 22:58 | |
| weshay | updated https://bugs.launchpad.net/tripleo/+bug/1710678 | 22:59 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 22:59 |
| pabelanger | weshay: so, if it is failing to clone github.com, that is a different issue. Also, we should not be cloning from github.com, its pretty unreliable | 23:01 |
| weshay | sure.. just going through it | 23:01 |
| *** salmankhan has joined #tripleo | 23:01 | |
| *** dsariel has joined #tripleo | 23:02 | |
| pabelanger | weshay: that info should maybe be on trunk.rdoproject.org, going to ask in #rdo now | 23:03 |
| weshay | pabelanger, git should be http://git.openstack.org | 23:03 |
| weshay | ? | 23:03 |
| weshay | ya | 23:03 |
| weshay | nvrmind | 23:04 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci master: Copy Ceph logs from running containers https://review.openstack.org/494340 | 23:05 |
| *** achadha_ has joined #tripleo | 23:05 | |
| pabelanger | weshay: it could be git.o.o but we'd have to mirror that repo into our infra | 23:06 |
| pabelanger | weshay: possible to do, but we'd need to talk about it first in #openstack-infra | 23:06 |
| *** achadha has quit IRC | 23:08 | |
| *** achadha_ has quit IRC | 23:09 | |
| stevebaker | gfidente: hi, this has finally landed https://review.openstack.org/#/c/489083/ | 23:09 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1702955 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710678 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710773 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710938 | 23:10 |
| openstack | Launchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs) | 23:10 |
| *** _milan_ has quit IRC | 23:10 | |
| openstack | Launchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 23:10 |
| openstack | Launchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| *** _milan_ has joined #tripleo | 23:11 | |
| gfidente | stevebaker yeah I tried to use that and the general set mechanism works fine | 23:12 |
| *** achadha has joined #tripleo | 23:12 | |
| gfidente | but then I think panda pointed out in RDO they use the --prefix argument to set the registry address | 23:12 |
| gfidente | and we get the ceph images prepended with that too | 23:12 |
| gfidente | when they actually want only the osp images to be prepended | 23:13 |
| gfidente | plus the naming choice for the ceph images upstream vs downstream is so different that it was hard even to set the variables appropriately | 23:14 |
| gfidente | basically upstream the version is all set in the tag name | 23:14 |
| *** achadha_ has joined #tripleo | 23:16 | |
| stevebaker | gfidente: ok, could you raise a bug which documents all the upstream and downstream image name conventions so we can figure something out which works? | 23:16 |
| *** achadha has quit IRC | 23:16 | |
| radez | slagle: thx for the insight, I'll take a look at my packages and build and see if I've got something mixed up, I'm intending to be deploying Ocata | 23:17 |
| *** achadha_ has quit IRC | 23:20 | |
| *** tosky has quit IRC | 23:33 | |
| *** achadha_ has joined #tripleo | 23:41 | |
| *** achadha_ has quit IRC | 23:46 | |
| *** salmankhan has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!