| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO https://review.openstack.org/564285 | 00:01 |
|---|---|---|
| *** psahoo has joined #tripleo | 00:23 | |
| *** rlandy is now known as rlandy|afk | 00:29 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: WIP/DNM: Do not grant caps if metrics pool name is empty https://review.openstack.org/575571 | 00:30 |
| *** mcornea has quit IRC | 00:32 | |
| *** pradk has joined #tripleo | 00:37 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: WIP/DNM: Do not grant caps if metrics pool name is empty https://review.openstack.org/575571 | 00:40 |
| *** limao has joined #tripleo | 00:53 | |
| *** thrash|g0ne has quit IRC | 01:04 | |
| *** Petersingh has joined #tripleo | 01:10 | |
| *** thrash has joined #tripleo | 01:12 | |
| *** thrash has joined #tripleo | 01:12 | |
| *** lifeless_ has quit IRC | 01:13 | |
| *** lifeless has joined #tripleo | 01:13 | |
| *** lifeless_ has joined #tripleo | 01:24 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo master: [WIP] enable support for openidc federation in keystone https://review.openstack.org/574851 | 01:24 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates master: [WIP] enable support for openidc federation in keystone https://review.openstack.org/574855 | 01:24 |
| *** lifeless has quit IRC | 01:25 | |
| *** psahoo has quit IRC | 01:28 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo stable/pike: install python-neutron-lbaas before starting neutron https://review.openstack.org/575500 | 01:33 |
| *** rlandy|afk is now known as rlandy | 01:36 | |
| *** lifeless has joined #tripleo | 01:40 | |
| *** zaneb has quit IRC | 01:40 | |
| *** Petersingh is now known as Petersingh|afk | 01:40 | |
| *** lifeless_ has quit IRC | 01:41 | |
| *** zaneb has joined #tripleo | 01:42 | |
| *** psahoo has joined #tripleo | 01:42 | |
| *** pmannidi has joined #tripleo | 01:46 | |
| *** Petersingh|afk is now known as Petersingh | 01:48 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: undercloud: manage mysql user + revoke grant during upgrade https://review.openstack.org/575186 | 01:48 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: undercloud: manage mysql user + revoke grant during upgrade https://review.openstack.org/575186 | 01:48 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Check container health as part of the deploy https://review.openstack.org/569153 | 01:48 |
| *** lifeless has quit IRC | 01:54 | |
| *** dxiri has joined #tripleo | 01:55 | |
| *** lifeless has joined #tripleo | 02:00 | |
| *** dxiri has quit IRC | 02:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Delete not-used services-docker files https://review.openstack.org/567810 | 02:08 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Move remnants of environments/services-docker https://review.openstack.org/567811 | 02:08 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Switch remaining env files to containers defaults https://review.openstack.org/567813 | 02:08 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Support containerized DVR in compute role https://review.openstack.org/573199 | 02:09 |
| *** Petersingh is now known as Petersingh|afk | 02:12 | |
| *** ayoung has joined #tripleo | 02:20 | |
| *** dxiri has joined #tripleo | 02:20 | |
| *** dxiri has quit IRC | 02:25 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Allow a containerized mistral-executor to access docker https://review.openstack.org/574905 | 02:29 |
| openstackgerrit | Steve Baker proposed openstack/puppet-tripleo master: Use user and group instead of ensure_resource https://review.openstack.org/575583 | 02:32 |
| *** Petersingh|afk is now known as Petersingh | 02:33 | |
| weshay|ruck | EmilienM, https://review.openstack.org/#/c/572830/ | 02:33 |
| *** brault has joined #tripleo | 02:34 | |
| EmilienM | weshay|ruck: your patch needs more work | 02:34 |
| EmilienM | see comments | 02:34 |
| weshay|ruck | k | 02:34 |
| *** brault has quit IRC | 02:34 | |
| EmilienM | weshay|ruck: you did like we run standalone on pike lol | 02:34 |
| EmilienM | weshay|ruck: this patch fixes standalone, please review | 02:37 |
| EmilienM | https://review.openstack.org/#/c/575564/ | 02:37 |
| EmilienM | weshay|ruck, stevebaker and https://review.openstack.org/#/c/575563/ please | 02:37 |
| EmilienM | (see zuul, it's passing) | 02:37 |
| stevebaker | EmilienM: lgtm | 02:38 |
| EmilienM | weshay|ruck: my tempest patch does nothing, validate-tempest is not executed. Any clue? | 02:39 |
| EmilienM | anything to hack in tripleo-ci repo? | 02:39 |
| weshay|ruck | EmilienM, I'll get tempest going w/ chandan and arx man | 02:40 |
| weshay|ruck | don't worry about it | 02:40 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: use fs config to assign standalone role, tests https://review.openstack.org/572830 | 02:45 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Allow a containerized mistral-executor to access docker https://review.openstack.org/574905 | 02:49 |
| EmilienM | stevebaker: anything we need to review? | 02:49 |
| *** agopi has quit IRC | 02:50 | |
| *** agopi has joined #tripleo | 02:50 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: skydive external_deploy_task migration https://review.openstack.org/561274 | 02:51 |
| stevebaker | EmilienM: no, I need to write a custom fact in ruby, no hurry on my series until that is working | 02:51 |
| EmilienM | ok | 02:51 |
| stevebaker | EmilienM: see the TODO. Do you think this is best done as a custom fact? https://review.openstack.org/#/c/575583/1/manifests/profile/base/mistral/executor.pp | 02:52 |
| EmilienM | looking | 02:52 |
| stevebaker | EmilienM: I need to stat /var/run/docker.sock | 02:52 |
| EmilienM | stevebaker: omg | 02:53 |
| EmilienM | that sounds tricky | 02:53 |
| EmilienM | I mean doable but dangerous | 02:53 |
| EmilienM | but I have nothing better in mind right now | 02:53 |
| stevebaker | EmilienM: me neither, other than some messy explicit discover gid on the host and pass it in as a parameter | 02:59 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: add tempest to standalone https://review.openstack.org/575588 | 03:00 |
| weshay|ruck | EmilienM, ^ | 03:00 |
| EmilienM | oh that | 03:00 |
| weshay|ruck | EmilienM, we need some rc file | 03:00 |
| EmilienM | yeah | 03:00 |
| EmilienM | I know | 03:00 |
| EmilienM | this thing is done for undercloud | 03:01 |
| EmilienM | let me see what it takes for standalone | 03:01 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Streamline variables passed in different environments https://review.openstack.org/573819 | 03:02 |
| weshay|ruck | https://review.openstack.org/#/c/573819/ | 03:02 |
| *** sai_p has quit IRC | 03:04 | |
| EmilienM | weshay|ruck: I talked with safchain today (Skydive dev) and I suggested to create a CI job for it | 03:06 |
| EmilienM | they need keystone + skydive | 03:06 |
| EmilienM | so another great use case for standalone in CI | 03:06 |
| EmilienM | I said he could talk to you | 03:06 |
| EmilienM | while I'm on PTO | 03:07 |
| *** agopi has quit IRC | 03:07 | |
| *** dxiri has joined #tripleo | 03:08 | |
| weshay|ruck | k | 03:09 |
| EmilienM | stevebaker: hey re: https://review.openstack.org/#/c/575552/ | 03:10 |
| EmilienM | stevebaker: I just tested it | 03:10 |
| EmilienM | with skydive | 03:10 |
| EmilienM | and it solved my problems, thanks again | 03:10 |
| EmilienM | so I didn't have to run any prepare command :D | 03:11 |
| EmilienM | just "openstack tripleo deploy" and my patch ^ | 03:11 |
| EmilienM | so discovery works fine, no bug | 03:11 |
| *** dxiri has quit IRC | 03:13 | |
| *** ramishra has joined #tripleo | 03:13 | |
| stevebaker | EmilienM: cool, tripleo deploy should just work | 03:14 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: skydive external_deploy_task migration https://review.openstack.org/561274 | 03:15 |
| *** dxiri has joined #tripleo | 03:17 | |
| *** d0ugal has quit IRC | 03:17 | |
| *** d0ugal_ has joined #tripleo | 03:17 | |
| *** dsneddon has quit IRC | 03:23 | |
| *** dsneddon has joined #tripleo | 03:24 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: standalone: execute undercloud_post to have stackrc https://review.openstack.org/575597 | 03:25 |
| EmilienM | weshay|ruck: ^ that should do the job for now | 03:25 |
| weshay|ruck | rock | 03:25 |
| EmilienM | weshay|ruck: I suspect alex will want to rename the env or something so we don't have "undercloud" | 03:27 |
| EmilienM | but the idea is here | 03:27 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: standalone: execute undercloud_post to have stackrc https://review.openstack.org/575597 | 03:27 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: standalone: switch default NeutronPublicInterface to br-ex https://review.openstack.org/575564 | 03:27 |
| weshay|ruck | what | 03:27 |
| *** rlandy has quit IRC | 03:27 | |
| EmilienM | weshay|ruck: why did you do that? | 03:28 |
| EmilienM | please remove the depends-on | 03:28 |
| EmilienM | on https://review.openstack.org/#/c/575564/ | 03:28 |
| EmilienM | it's useless and the patch was in gate :( | 03:28 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: standalone: switch default NeutronPublicInterface to br-ex https://review.openstack.org/575564 | 03:28 |
| *** psachin has joined #tripleo | 03:28 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: use fs config to assign standalone role, tests https://review.openstack.org/572830 | 03:30 |
| *** tzumainn has joined #tripleo | 03:31 | |
| EmilienM | I've reset the gate to save time, a job was failing anyway, we saved 2h | 03:38 |
| *** skramaja has joined #tripleo | 03:38 | |
| *** agopi has joined #tripleo | 03:42 | |
| *** Haresh has joined #tripleo | 03:45 | |
| *** limao has quit IRC | 03:48 | |
| *** Petersingh is now known as Petersingh|afk | 03:48 | |
| *** salmankhan has quit IRC | 03:49 | |
| *** yamahata has joined #tripleo | 03:56 | |
| *** limao has joined #tripleo | 03:57 | |
| *** ykarel_ has joined #tripleo | 04:08 | |
| *** haleyb has quit IRC | 04:08 | |
| *** haleyb has joined #tripleo | 04:09 | |
| *** shreshtha has joined #tripleo | 04:16 | |
| *** Petersingh|afk is now known as Petersingh | 04:17 | |
| *** jaganathan has joined #tripleo | 04:18 | |
| *** gouthamr has quit IRC | 04:22 | |
| *** stevebaker has quit IRC | 04:22 | |
| *** dmellado has quit IRC | 04:23 | |
| *** tzumainn has quit IRC | 04:43 | |
| *** moshele has joined #tripleo | 04:44 | |
| *** moshele has quit IRC | 04:47 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Don't run setfiles on /boot/efi https://review.openstack.org/575315 | 04:54 |
| *** links has joined #tripleo | 04:55 | |
| *** ratailor has joined #tripleo | 04:56 | |
| *** slaweq has joined #tripleo | 05:00 | |
| *** yprokule has joined #tripleo | 05:04 | |
| *** ratailor has quit IRC | 05:05 | |
| *** pdeore__ has joined #tripleo | 05:08 | |
| *** janki has joined #tripleo | 05:13 | |
| *** dxiri has quit IRC | 05:15 | |
| *** thrash has quit IRC | 05:16 | |
| *** pcaruana has quit IRC | 05:18 | |
| *** thrash has joined #tripleo | 05:20 | |
| *** thrash has joined #tripleo | 05:20 | |
| *** pmannidi_ has joined #tripleo | 05:20 | |
| *** leanderthal has quit IRC | 05:23 | |
| *** leanderthal has joined #tripleo | 05:25 | |
| *** saneax has joined #tripleo | 05:26 | |
| *** tcw has quit IRC | 05:27 | |
| *** tcw has joined #tripleo | 05:27 | |
| *** leanderthal has quit IRC | 05:30 | |
| *** leanderthal has joined #tripleo | 05:36 | |
| *** leanderthal has quit IRC | 05:41 | |
| *** leanderthal has joined #tripleo | 05:43 | |
| *** dparkes has quit IRC | 05:49 | |
| *** ykarel__ has joined #tripleo | 05:50 | |
| *** ykarel_ has quit IRC | 05:53 | |
| *** morazi has quit IRC | 05:57 | |
| *** dparkes has joined #tripleo | 06:01 | |
| *** pgadiya has joined #tripleo | 06:02 | |
| *** mschuppert has joined #tripleo | 06:04 | |
| bandini | anyone else not reaching gerrit@rdo? | 06:05 |
| *** mhenkel_ has joined #tripleo | 06:05 | |
| *** mhenkel_ has quit IRC | 06:05 | |
| *** dparkes has quit IRC | 06:06 | |
| *** mhenkel_ has joined #tripleo | 06:06 | |
| *** jfrancoa has joined #tripleo | 06:06 | |
| bandini | yeah it's unreachable from a bunch of networks | 06:07 |
| *** dparkes has joined #tripleo | 06:07 | |
| ramishra | bandini: I was told on #rdo that rdo-cloud is experiencing some network issue | 06:08 |
| bandini | ramishra: ah ok thanks. so folks are aware | 06:09 |
| *** Petersingh is now known as Petersingh|afk | 06:09 | |
| *** paramite has quit IRC | 06:10 | |
| bandini | anyone around to review this queens cherry-pick ? https://review.openstack.org/#/c/575525/ | 06:10 |
| *** dparkes has quit IRC | 06:13 | |
| *** dmsimard has quit IRC | 06:13 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add dbus-python to overcloud-full image https://review.openstack.org/574674 | 06:14 |
| *** cylopez has joined #tripleo | 06:21 | |
| *** cylopez has left #tripleo | 06:21 | |
| *** Petersingh|afk is now known as Petersingh | 06:24 | |
| *** agopi has quit IRC | 06:24 | |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP nova: Use nova-status upgrade check during upgrade https://review.openstack.org/575623 | 06:26 |
| *** dmsimard has joined #tripleo | 06:27 | |
| *** gkadam has joined #tripleo | 06:38 | |
| *** jfrancoa has quit IRC | 06:38 | |
| *** holser__ has joined #tripleo | 06:39 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-heat-templates master: Adds support to configure enabled and default bios interfaces https://review.openstack.org/575462 | 06:41 |
| *** kopecmartin has joined #tripleo | 06:42 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Uses new format for CephX keys caps https://review.openstack.org/574664 | 06:43 |
| *** pcaruana has joined #tripleo | 06:44 | |
| *** pkovar has joined #tripleo | 06:45 | |
| *** ffiore has joined #tripleo | 06:47 | |
| *** ffiore has quit IRC | 06:49 | |
| *** ffiore has joined #tripleo | 06:50 | |
| *** slaweq has quit IRC | 06:50 | |
| *** cmurphy has joined #tripleo | 06:51 | |
| *** ykarel_ has joined #tripleo | 06:52 | |
| *** dparkes has joined #tripleo | 06:53 | |
| *** ykarel__ has quit IRC | 06:55 | |
| *** zul has quit IRC | 06:56 | |
| *** zul has joined #tripleo | 06:56 | |
| *** slaweq has joined #tripleo | 07:01 | |
| *** pmannidi has quit IRC | 07:03 | |
| *** dmellado has joined #tripleo | 07:04 | |
| *** brault has joined #tripleo | 07:07 | |
| *** ykarel_ is now known as ykarel | 07:07 | |
| *** rcernin has quit IRC | 07:08 | |
| *** ratailor has joined #tripleo | 07:10 | |
| *** ramishra has quit IRC | 07:13 | |
| *** tesseract has joined #tripleo | 07:17 | |
| *** ramishra has joined #tripleo | 07:20 | |
| *** bogdando has joined #tripleo | 07:20 | |
| bogdando | jaosorior, mwhahaha, stevebaker: hi PTAL https://review.openstack.org/#/c/572404/8 | 07:20 |
| *** tosky has joined #tripleo | 07:28 | |
| bandini | bogdando: morning sir, clean cherry pick to queens of the logrotate thing if you have a min https://review.openstack.org/#/c/575525/ ;) | 07:29 |
| *** chem` has joined #tripleo | 07:30 | |
| *** Petersingh is now known as Petersingh|lunch | 07:32 | |
| *** amoralej|off is now known as amoralej | 07:32 | |
| *** florianf has joined #tripleo | 07:33 | |
| *** chem has quit IRC | 07:33 | |
| bogdando | bandini: done, thanks! | 07:33 |
| bandini | bogdando: merci! | 07:34 |
| *** limao has quit IRC | 07:35 | |
| *** jfrancoa has joined #tripleo | 07:35 | |
| *** ksambor has quit IRC | 07:37 | |
| *** ksambor has joined #tripleo | 07:37 | |
| *** radeks has joined #tripleo | 07:38 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add iscsi-boot element https://review.openstack.org/511494 | 07:38 |
| *** gfidente has joined #tripleo | 07:40 | |
| *** suuuper has joined #tripleo | 07:41 | |
| *** dmellado has quit IRC | 07:42 | |
| *** dmellado has joined #tripleo | 07:42 | |
| sileht | https://images.rdoproject.org/ looks dead | 07:47 |
| dmellado | if it's hosted on rdocloud | 07:47 |
| dmellado | houston we've an issue atm | 07:47 |
| dmellado | xD | 07:47 |
| *** jpena|off is now known as jpena | 07:47 | |
| *** ramishra has quit IRC | 07:49 | |
| *** ramishra has joined #tripleo | 07:51 | |
| *** ccamacho has joined #tripleo | 07:51 | |
| *** jbadiapa_ is now known as jbadiapa | 07:54 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Log memcached to stdout/err only https://review.openstack.org/575676 | 07:55 |
| *** chem`` has joined #tripleo | 07:55 | |
| *** chem` has quit IRC | 07:57 | |
| *** verdurin has quit IRC | 07:59 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Log memcached to stdout/err only https://review.openstack.org/575676 | 08:00 |
| *** verdurin has joined #tripleo | 08:00 | |
| *** tzumainn has joined #tripleo | 08:00 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Update for openshift 3.9 https://review.openstack.org/574233 | 08:00 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Run dispatch script for all unmanaged devices https://review.openstack.org/575679 | 08:00 |
| *** d0ugal_ has quit IRC | 08:04 | |
| *** d0ugal has joined #tripleo | 08:04 | |
| *** jpich has joined #tripleo | 08:06 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates stable/queens: Updates: run host_prep_tasks between update tasks and deployment tasks https://review.openstack.org/575680 | 08:07 |
| *** radeks has quit IRC | 08:16 | |
| *** jfrancoa has quit IRC | 08:16 | |
| *** tzumainn has quit IRC | 08:18 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: DNM - Testing swift integration https://review.openstack.org/575683 | 08:19 |
| *** threestrands has quit IRC | 08:20 | |
| flaper87 | bogdando: do you know Sergii's irc nick? | 08:22 |
| flaper87 | :D | 08:22 |
| *** dparkes has quit IRC | 08:24 | |
| bogdando | flaper87, holser__: ^^ | 08:24 |
| holser__ | here | 08:24 |
| bogdando | a perfection hunter ;) | 08:24 |
| holser__ | what's up flaper87 ? | 08:25 |
| flaper87 | holser__: https://review.openstack.org/#/c/575679/ :) | 08:25 |
| *** janki has quit IRC | 08:25 | |
| flaper87 | holser__: just saw your comment and replied but I thought I'd ping you on IRC too | 08:25 |
| holser__ | yeah - it's nitpick | 08:25 |
| *** dparkes has joined #tripleo | 08:25 | |
| holser__ | will +1 after comment | 08:25 |
| flaper87 | holser__: no worries, just wanted to mention that it's prob not worth to -1 on nitpicks. | 08:25 |
| flaper87 | that said, thanks a bunch for the suggestion | 08:25 |
| flaper87 | I'm honestly not an awk expert and it's good to learn something new | 08:26 |
| holser__ | well, the problem I did a lot of bash for very loaded machines | 08:26 |
| holser__ | the problem was forking a new process | 08:26 |
| bogdando | less pipes is goodness | 08:26 |
| flaper87 | indeed | 08:26 |
| bogdando | but indeed, does not worth -1 | 08:26 |
| holser__ | so | | | | - 4 forks | 08:26 |
| *** suuuper has quit IRC | 08:27 | |
| flaper87 | fully agree! just not a huge deal in this case ^^ | 08:27 |
| bogdando | that's why I said much better but not perfect haha | 08:27 |
| holser__ | flaper87 done | 08:27 |
| jistr | weshay|ruck: re https://review.openstack.org/#/c/574417 -- according to what i see in logs (on the other patch i mentioned) the job is doing the upgrade properly | 08:28 |
| *** suuuper has joined #tripleo | 08:29 | |
| flaper87 | holser__: you rock, thanks | 08:29 |
| bogdando | there is a funny topic btw http://lists.openstack.org/pipermail/openstack-dev/2018-May/130802.html , so I opened the patch https://review.openstack.org/#/c/570940/ and the first thing I see there is https://review.openstack.org/#/c/570940/2/reference/principles.rst@55 :D | 08:30 |
| bogdando | although, Eric didn't make it -1 :) | 08:31 |
| bogdando | sorry, it is Friday yet... | 08:32 |
| *** Petersingh|lunch is now known as Petersingh | 08:34 | |
| *** derekh has joined #tripleo | 08:36 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Create L3 start/stop scripts for minor updates. https://review.openstack.org/565760 | 08:37 |
| holser__ | chem`` bogdando | 08:39 |
| *** chem`` is now known as chem | 08:39 | |
| chem | bogdando: hey, I've commented on https://review.openstack.org/#/c/575676/2, I think it's worth a new review, WDYT ? | 08:39 |
| openstackgerrit | Merged openstack/diskimage-builder master: Don't run setfiles on /boot/efi https://review.openstack.org/575315 | 08:42 |
| bogdando | chem: sounds fair, though could we make it a follow up? | 08:43 |
| bogdando | chem: would you go for it? :) | 08:44 |
| chem | bogdando: well, would make it more work to get it in the product ... but oki, I'll do a followup | 08:44 |
| bogdando | thanks! | 08:44 |
| * chem doing it now | 08:44 | |
| *** ykarel is now known as ykarel|lunch | 08:47 | |
| *** akrivoka has joined #tripleo | 08:48 | |
| *** panda|off is now known as panda | 08:54 | |
| *** lifeless has quit IRC | 08:57 | |
| *** social has joined #tripleo | 08:57 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Activate memcached debug only when using MemcachedDebug param. https://review.openstack.org/575693 | 08:59 |
| chem | bogdando: holser__ mwhahaha ^ | 09:00 |
| holser__ | +1 | 09:00 |
| *** gouthamr has joined #tripleo | 09:02 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Activate memcached debug only when using MemcachedDebug param. https://review.openstack.org/575693 | 09:04 |
| *** ratailor_ has joined #tripleo | 09:06 | |
| holser__ | chem +1 | 09:08 |
| holser__ | I thought about upper/lower case of true | 09:08 |
| holser__ | but it's not a big deal, but imho heat should support regexp :D | 09:10 |
| *** ratailor has quit IRC | 09:10 | |
| *** lifeless has joined #tripleo | 09:10 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Add undercloud admin VIP to insec registries https://review.openstack.org/575697 | 09:10 |
| *** jtomasek_ has joined #tripleo | 09:12 | |
| *** jtomasek has quit IRC | 09:12 | |
| *** quiquell|off is now known as quiquell | 09:14 | |
| *** janki has joined #tripleo | 09:16 | |
| *** quiquell is now known as quiquell|rover | 09:18 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Add undercloud admin VIP to insec registries https://review.openstack.org/575697 | 09:19 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-common stable/queens: DNM: test queens patch https://review.openstack.org/575244 | 09:22 |
| *** ykarel|lunch is now known as ykarel | 09:25 | |
| *** stevebaker has joined #tripleo | 09:26 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Get ctlplane interface config from the subnet https://review.openstack.org/569547 | 09:29 |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Fix ansible tmp dir to mistral working directory https://review.openstack.org/575703 | 09:30 |
| quiquell|rover | EmilienM: Queues are high again, maybe the containerized activation is affecting it | 09:33 |
| *** zoli is now known as zoli|lunch | 09:35 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 09:37 |
| *** psachin has quit IRC | 09:38 | |
| *** lifeless has quit IRC | 09:38 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 09:40 |
| *** lifeless has joined #tripleo | 09:45 | |
| bogdando | stevebaker, EmilienM: could you folks please comment https://trello.com/c/osszhcjk/79-autogen-container-image-params#comment-5b1e83e0a6d2883b78a0fc98 ? | 09:46 |
| *** leanderthal has quit IRC | 09:48 | |
| *** Petersingh has quit IRC | 09:52 | |
| *** Petersingh has joined #tripleo | 09:52 | |
| *** Petersingh has quit IRC | 09:53 | |
| *** Petersingh has joined #tripleo | 09:54 | |
| *** Petersingh is now known as Petersingh|afk | 10:00 | |
| openstackgerrit | Dan Sneddon proposed openstack/os-net-config master: [WIP] Add Policy Based Routing Support to os-net-config https://review.openstack.org/575712 | 10:01 |
| quiquell|rover | Any cores for a +2 +1w here ? | 10:01 |
| quiquell|rover | https://review.openstack.org/#/c/575407/ | 10:01 |
| quiquell|rover | Affects collecting logs | 10:01 |
| *** psahoo has quit IRC | 10:03 | |
| *** Petersingh|afk is now known as Petersingh | 10:03 | |
| *** Petersingh is now known as Petersingh|afk | 10:05 | |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| *** rcernin has joined #tripleo | 10:07 | |
| openstackgerrit | Dan Sneddon proposed openstack/os-net-config master: [WIP] Add Policy Based Routing Support to os-net-config https://review.openstack.org/575712 | 10:08 |
| *** leanderthal has joined #tripleo | 10:09 | |
| *** Petersingh|afk is now known as Petersingh | 10:10 | |
| dtantsur | morning folks, can someone please W+1 https://review.openstack.org/#/c/541565/ ? | 10:10 |
| *** paramite has joined #tripleo | 10:12 | |
| *** Petersingh has quit IRC | 10:13 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Disable memcached's cachedump https://review.openstack.org/538987 | 10:13 |
| mschuppert | fultonj: hi, could you +2 https://review.openstack.org/#/c/575380/ ci passed | 10:20 |
| *** lifeless_ has joined #tripleo | 10:22 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add environments/net-noop.yaml https://review.openstack.org/550072 | 10:23 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Ensure WorkflowSteps resource is enabled in ceph-upgrade-prepare https://review.openstack.org/572053 | 10:23 |
| openstackgerrit | Merged openstack/tripleo-common master: Write to /etc/ssh with super user privileges https://review.openstack.org/574721 | 10:23 |
| *** lifeless has quit IRC | 10:23 | |
| *** lifeless_ has quit IRC | 10:39 | |
| *** lifeless has joined #tripleo | 10:41 | |
| *** dtantsur is now known as dtantsur|afk | 10:45 | |
| *** pdeore has joined #tripleo | 10:49 | |
| *** zoli|lunch is now known as zoli | 10:50 | |
| *** brault has quit IRC | 10:50 | |
| *** gfidente has quit IRC | 10:50 | |
| *** pdeore__ has quit IRC | 10:51 | |
| *** lifeless_ has joined #tripleo | 10:55 | |
| *** lifeless has quit IRC | 10:56 | |
| *** pdeore has quit IRC | 10:58 | |
| *** raildo has joined #tripleo | 10:59 | |
| *** lifeless has joined #tripleo | 11:00 | |
| *** lifeless_ has quit IRC | 11:01 | |
| *** pdeore has joined #tripleo | 11:02 | |
| *** yolanda has quit IRC | 11:03 | |
| *** lifeless has quit IRC | 11:05 | |
| *** yolanda has joined #tripleo | 11:06 | |
| *** gfidente has joined #tripleo | 11:08 | |
| *** lifeless has joined #tripleo | 11:10 | |
| *** jpena is now known as jpena|lunch | 11:12 | |
| *** lifeless_ has joined #tripleo | 11:15 | |
| *** lifeless has quit IRC | 11:15 | |
| fultonj | mschuppert: +2'd :) | 11:17 |
| *** shreshtha has quit IRC | 11:25 | |
| *** brault has joined #tripleo | 11:30 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Fix update/upgrade or config change for ODL https://review.openstack.org/574039 | 11:31 |
| mschuppert | fultonj thx! | 11:34 |
| fultonj | mschuppert: np | 11:35 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Disable update of OpenShift and k8s containers https://review.openstack.org/573630 | 11:36 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Map OpenShift resources to services https://review.openstack.org/573225 | 11:36 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Do not fail on non existent ansible group https://review.openstack.org/574187 | 11:36 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Define a 'baremetal' file for nodes config https://review.openstack.org/553465 | 11:36 |
| *** rodolof has joined #tripleo | 11:38 | |
| *** anande has joined #tripleo | 11:38 | |
| *** jpena|lunch is now known as jpena|off | 11:38 | |
| *** rh-jelabarre has joined #tripleo | 11:41 | |
| *** rh-jelabarre has quit IRC | 11:43 | |
| *** rh-jelabarre has joined #tripleo | 11:44 | |
| *** cylopez has joined #tripleo | 11:46 | |
| *** rh-jelabarre has quit IRC | 11:46 | |
| *** rh-jelabarre has joined #tripleo | 11:46 | |
| *** rh-jelabarre has quit IRC | 11:47 | |
| *** rh-jelabarre has joined #tripleo | 11:48 | |
| *** rh-jelabarre has quit IRC | 11:48 | |
| *** vpickard_ is now known as vpickard | 11:49 | |
| *** pdeore has quit IRC | 11:52 | |
| *** vpickard is now known as vpickard_ | 11:53 | |
| *** morazi has joined #tripleo | 11:54 | |
| weshay|ruck | EmilienM, https://review.openstack.org/#/c/571613/ :( | 11:58 |
| weshay|ruck | bogdando, stevebaker ^ | 11:59 |
| weshay|ruck | second time it died in the gate like that | 11:59 |
| bogdando | weshay|ruck: the same issue with overloading registry mirrors? | 12:00 |
| * weshay|ruck looking | 12:00 | |
| bogdando | weshay|ruck, EmilienM, stevebaker: I have a patch in work to improve exp backoff logic for pull and push | 12:00 |
| bogdando | we keep those too short in busy | 12:00 |
| bogdando | I'll push it for review shortly | 12:00 |
| bogdando | short and busy* | 12:01 |
| weshay|ruck | ya.. same thing http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/298ee13/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 12:01 |
| *** edmondsw has joined #tripleo | 12:01 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add upgrade and update logs to logstash indexing https://review.openstack.org/570896 | 12:03 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Run openstack overcloud failures when failures https://review.openstack.org/571897 | 12:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Upgrades: Refactor package removal to step3 https://review.openstack.org/571431 | 12:03 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Revert "Revert "fs050: enable upgrade cleanup"" https://review.openstack.org/571500 | 12:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: undercloud: add missing services for upgrade cleanup https://review.openstack.org/570283 | 12:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Rename Dell EMC XTREMIO iSCSI environment file https://review.openstack.org/570230 | 12:03 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: Conditionally Enable Nuage VRS https://review.openstack.org/572631 | 12:03 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Add XClarity in the list of supported hw types https://review.openstack.org/541570 | 12:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Set Ceph pools rule_name to replicated_rule https://review.openstack.org/574341 | 12:03 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Make sure remotes are fully up before proceeding https://review.openstack.org/569578 | 12:03 |
| weshay|ruck | quiquell|rover, ^ good sign :) | 12:03 |
| weshay|ruck | lots o merge | 12:04 |
| *** jpena|off is now known as jpena | 12:05 | |
| *** moshele has joined #tripleo | 12:06 | |
| *** trown|outtypewww is now known as trown | 12:09 | |
| *** sanjay__u has quit IRC | 12:13 | |
| *** artom has quit IRC | 12:14 | |
| *** skramaja has quit IRC | 12:17 | |
| *** cdearborn has joined #tripleo | 12:18 | |
| *** rlandy has joined #tripleo | 12:19 | |
| *** rh-jelabarre has joined #tripleo | 12:20 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Set Ceph pools rule_name to replicated_rule https://review.openstack.org/575721 | 12:21 |
| *** moshele has quit IRC | 12:23 | |
| *** ratailor_ has quit IRC | 12:23 | |
| beagles | bandini: does this ring any bells with you http://paste.openstack.org/show/723516/ ? | 12:25 |
| beagles | bandini: context - I'm running 'openstack overcloud update run --nodes Controller' using master | 12:26 |
| beagles | bandini: I think this is the last controller node being updated and it is bombing there (heartbreak!) | 12:26 |
| *** lblanchard has joined #tripleo | 12:27 | |
| *** liverpooler has joined #tripleo | 12:28 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Fix Swift storage device setup https://review.openstack.org/575722 | 12:31 |
| *** tcw1 has joined #tripleo | 12:31 | |
| *** tcw has quit IRC | 12:31 | |
| *** jaganathan has quit IRC | 12:32 | |
| *** leanderthal has quit IRC | 12:34 | |
| mhenkel_ | hi all, | 12:36 |
| mhenkel_ | anyone ever had an issue where the workflow creation silently stops with the the only line 'passwords' ? | 12:36 |
| d0ugal | mhenkel_: What CLI command? | 12:37 |
| mhenkel_ | http://paste.openstack.org/show/723545/ | 12:37 |
| d0ugal | mhenkel_: I suspect you are getting a KeyError exception | 12:37 |
| mhenkel_ | this happened between two openstack deploy runs | 12:37 |
| mhenkel_ | I stopped one with CTRL+C | 12:38 |
| d0ugal | mhenkel_: Right - it is a limitation of python-tripleoclient/python-openstackclient | 12:38 |
| mhenkel_ | anything I can do without re-installing the undercloud? | 12:38 |
| d0ugal | basically there is a keyerror in tripleoclient and by default openstackclient prints str(exception) - which when it is a keyerror is just the name of the missing key :/ | 12:38 |
| *** ykarel has quit IRC | 12:39 | |
| *** ykarel has joined #tripleo | 12:39 | |
| d0ugal | mhenkel_: I am not sure what the root cause is. | 12:39 |
| mhenkel_ | ok | 12:39 |
| d0ugal | I think there may have been a bug about it | 12:39 |
| * d0ugal digs | 12:40 | |
| d0ugal | mhenkel_: if you run the command again with "--debug" at the end you will get more information and the full Python traceback | 12:40 |
| mhenkel_ | unfortunately the search parameters for this are quiet wide | 12:40 |
| d0ugal | Indeed | 12:41 |
| mhenkel_ | http://paste.openstack.org/show/723546/ | 12:41 |
| mhenkel_ | this is very clear | 12:41 |
| d0ugal | Okay, so my guess about KeyError was correct :) | 12:41 |
| d0ugal | mhenkel_: so it seems, for some reason the passwords are missing from your plan. | 12:42 |
| d0ugal | I think this is the similar, but different root cause, bug I was thinking of: https://bugs.launchpad.net/tripleo/+bug/1734704 | 12:43 |
| openstack | Launchpad bug 1734704 in tripleo "Tripleoclient failures don't produce enough information" [Medium,In progress] - Assigned to Dougal Matthews (d0ugal) | 12:43 |
| mhenkel_ | checking | 12:43 |
| d0ugal | Similar issue in tripleoclient | 12:43 |
| *** abishop has joined #tripleo | 12:44 | |
| d0ugal | mhenkel_: did you disable password generation? | 12:45 |
| d0ugal | There is probably a way to trigger the password generation to make sure they exist... | 12:45 |
| * d0ugal checks | 12:45 | |
| *** rlandy_ has joined #tripleo | 12:46 | |
| d0ugal | Strange, I can't connect to my undercloud. So I can't check easily :/ | 12:46 |
| *** pradk has quit IRC | 12:46 | |
| *** lifeless_ has quit IRC | 12:46 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: [WIP] Add Network Topology diagram https://review.openstack.org/569125 | 12:46 |
| *** dmellado_ has joined #tripleo | 12:46 | |
| *** gouthamr has quit IRC | 12:46 | |
| *** stevebaker has quit IRC | 12:47 | |
| *** rlandy_ has quit IRC | 12:48 | |
| *** rlandy has quit IRC | 12:48 | |
| *** rlandy_ has joined #tripleo | 12:48 | |
| *** dmellado has quit IRC | 12:49 | |
| *** rlandy_ is now known as rlandy | 12:49 | |
| *** myoung|off is now known as myoung | 12:49 | |
| mhenkel_ | d0gual: can the plan just be re-created? | 12:51 |
| *** dougbtv_ has joined #tripleo | 12:53 | |
| d0ugal | mhenkel_: Yeah, with the CLI commands "openstack overcloud plan delete" and "openstack overcloud plan create" | 12:53 |
| mhenkel_ | thx | 12:54 |
| *** anande has quit IRC | 12:58 | |
| *** rcernin has quit IRC | 13:02 | |
| beagles | does anyone know of any place in updates, upgrades, or even just deploying where something might cause a restart of all running containers on a host? | 13:03 |
| *** pradk has joined #tripleo | 13:04 | |
| *** erlon_ has joined #tripleo | 13:05 | |
| *** paramite has quit IRC | 13:05 | |
| beagles | EmilienM: mwhahaha: bogdando: ccamacho: dciabrin_ ^ | 13:06 |
| beagles | I mean a blanket restart, not a paunch restart because a config changed etc | 13:07 |
| beagles | (the containers I'm concerned with aren't paunch managed at any rate) | 13:07 |
| mhenkel_ | d0ugal: looks like plan re-creation helped. It's at least progressing further now | 13:08 |
| mhenkel_ | thanks! | 13:08 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777130 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,New] - Assigned to Quique Llorente (quiquell) | 13:10 |
| dciabrin_ | beagles, in t-h-t puppet/services/docker.yaml use to docker stop all container before a docker package update. it's moved to ansible starting I2f5bc355d3c952e2daf9ec7442850e24bed275e1 | 13:11 |
| dciabrin_ | beagles, not sure where it is now, or if it still stops containers | 13:12 |
| beagles | dciabrin_: thanks I'll check | 13:12 |
| d0ugal | mhenkel_: np | 13:13 |
| openstackgerrit | Honza Pokorny proposed openstack/tempest-tripleo-ui master: Add basic project structure https://review.openstack.org/575730 | 13:14 |
| erlon_ | @all, folks, I'm trying to build a virtual environment using RH director. I have director installed but I'm not being able to get a heat configuration that can bring my overcloud up. Does anyone has have done this before and have the templates+plus CLI command to share? | 13:14 |
| beagles | dciabrin_: strictly speaking we shouldn't really need to do that anymore - correct? | 13:14 |
| dciabrin_ | beagles, correct, now that the docker bug (hanging on restart) has been fixed, we shouldn't need it anymore | 13:15 |
| erlon_ | I tried using single_nic_vlans.yaml | 13:15 |
| *** tzumainn has joined #tripleo | 13:16 | |
| *** tzumainn has quit IRC | 13:17 | |
| *** tzumainn has joined #tripleo | 13:17 | |
| *** rh-jelabarre has quit IRC | 13:18 | |
| *** amoralej is now known as amoralej|lunch | 13:22 | |
| *** rh-jelabarre has joined #tripleo | 13:25 | |
| *** mcornea has joined #tripleo | 13:27 | |
| *** suuuper has quit IRC | 13:27 | |
| *** links has quit IRC | 13:27 | |
| *** ykarel is now known as ykarel|away | 13:31 | |
| beagles | dciabrin_: I think that is what is causing my issue, thanks | 13:32 |
| dciabrin_ | beagles, ouch :) | 13:32 |
| beagles | dciabrin_: indeed, but at least we know now :) | 13:33 |
| beagles | or at least think we do | 13:33 |
| *** Haresh has quit IRC | 13:34 | |
| *** mmethot has quit IRC | 13:36 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Introduces NovaComputeCpuSharedSet parameter https://review.openstack.org/575449 | 13:38 |
| *** pblaho has quit IRC | 13:39 | |
| *** ykarel|away has quit IRC | 13:40 | |
| beagles | dciabrin_: what do you think the chances of us dropping the "stop all containers" steps from rocky and queens would be? | 13:42 |
| dciabrin_ | beagles, tough question. the problem is that docker ships in RHEL IIRC, so if the RHOS channel is updated before we have the fixed docker in the env, minor update will fail, for another reason. | 13:44 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Add split-stack environment support. https://review.openstack.org/572778 | 13:44 |
| beagles | EmilienM: mwhahaha: that's a question for you as well ^ | 13:44 |
| dciabrin_ | beagles, does the problem only happen in OSP13? in which case, we might be covered by major upgrade if it enforces a RHEL upgrade before proceeding | 13:44 |
| beagles | dciabrin_: ugh yeah | 13:44 |
| dciabrin_ | marios, ^^ | 13:44 |
| EmilienM | beagles: good question for upgrade folks | 13:44 |
| EmilienM | (I haven't been much involved on that area, tbh) | 13:45 |
| beagles | dciabrin_: yeah, I think we are talking queens/13 as this is the only affect neutron and it's agents at the moment | 13:45 |
| beagles | EmilienM: ack | 13:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use the in-built image prepare for undercloud install https://review.openstack.org/571613 | 13:46 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use the in-built image prepare for undercloud install https://review.openstack.org/571613 | 13:46 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use role tripleo-modify-image to update containers https://review.openstack.org/571631 | 13:46 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use role tripleo-modify-image to update containers https://review.openstack.org/571631 | 13:46 |
| EmilienM | gate is 11h | 13:47 |
| EmilienM | I think we should stop approving things until it's resolved | 13:47 |
| *** dansmith is now known as superdan | 13:47 | |
| *** toure|gone is now known as toure | 13:47 | |
| beagles | ack | 13:48 |
| beagles | "impending weekend gate syndrome" | 13:48 |
| *** Haresh has joined #tripleo | 13:49 | |
| *** quiquell|rover is now known as quique|rover|lch | 13:49 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Improve exp backoff for images pull/push/rpm-upd https://review.openstack.org/575742 | 13:49 |
| EmilienM | bandini: ++ thanks for your feedback on storyboard, the search is not great I agree | 13:49 |
| EmilienM | weshay|ruck: is someone actively looking at why the gate is 11h long? | 13:49 |
| bogdando | weshay|ruck, StevenK, EmilienM: https://review.openstack.org/#/c/575742/ I hope that improves the containers situation for CI infra | 13:50 |
| bogdando | StevenK: oh, I meant Steve Baker, sorry | 13:51 |
| StevenK | bogdando: Heh heh, no worries | 13:51 |
| *** saneax has quit IRC | 13:52 | |
| EmilienM | bogdando: do you have an example of logs in CI that indicated it was that? | 13:53 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Log memcached to stdout/err only https://review.openstack.org/575676 | 13:53 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Activate memcached debug only when using MemcachedDebug param. https://review.openstack.org/575693 | 13:53 |
| EmilienM | bogdando: without bug report or example of logs, it's pure guessing now | 13:53 |
| *** dparkes has quit IRC | 13:54 | |
| chem | bogdando: fyi I rebased memcached https://review.openstack.org/#/c/575676 | 13:55 |
| bogdando | EmilienM: I didn't found a bug report forhttp://lists.openstack.org/pipermail/openstack-dev/2018-June/131482.html | 13:56 |
| bogdando | that falls into that context | 13:56 |
| weshay|ruck | EmilienM, I started to look this morning.. the containers patch failed in gate agian | 13:56 |
| weshay|ruck | again | 13:56 |
| bogdando | yeah, wanted to post that link | 13:56 |
| weshay|ruck | I'll free up in a bit, I need quique|rover|lch to cover that now | 13:57 |
| bogdando | EmilienM: 3:01:30 PM GMT+3 - weshay|ruck: ya.. same thing http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/298ee13/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 13:57 |
| bogdando | see the log time stamps for failures | 13:57 |
| bogdando | and we need to compare it between all of the failed multinode jobs | 13:57 |
| bogdando | in that run | 13:57 |
| bogdando | or other runs taken the same time prolly | 13:58 |
| bogdando | I think that data leaves in logstash... | 13:58 |
| EmilienM | bogdando: I'm not sure the retry will really help here | 13:58 |
| bogdando | but have not idea how to query it | 13:58 |
| bogdando | cannot prove that strictly EmilienM | 13:59 |
| bogdando | formally* | 13:59 |
| bogdando | I have no data to operate | 13:59 |
| marios | dciabrin_: beagles so if the question is do we expect rhel updated before the major upgrade the answer is yes (minor update delivers that and we expect operator to do minor before major) | 13:59 |
| *** ratailor has joined #tripleo | 14:00 | |
| bogdando | but I think exponential backoff shifting all that DoS messy CI jobs load into a 3 min intervals randomly, might alleviate | 14:00 |
| bogdando | and multi-process workers in each of the jobs as well | 14:00 |
| bogdando | I designed those intervals be interleaved with random fallbacks with fixed and longer ones | 14:01 |
| *** lblanchard has quit IRC | 14:02 | |
| *** ksambor has quit IRC | 14:03 | |
| *** pradk has quit IRC | 14:03 | |
| *** etingof has quit IRC | 14:05 | |
| bogdando | EmilienM: If I had not mistaken, the backoff for each of the worker will be distributed like (rnd(6s), rnd(18s)), 15s, (rnd(6s), rnd(18s)), 45s, (rnd(6s), rnd(18s)) <= 132s. | 14:05 |
| bogdando | not sure with the very last round | 14:06 |
| bogdando | if it would ever take a place | 14:06 |
| bogdando | if we do not want too much wait for image workflows, we are free to experiment like we want for CI only containers update script | 14:07 |
| *** yamahata has quit IRC | 14:07 | |
| EmilienM | bogdando: https://review.openstack.org/#/c/575564/ | 14:08 |
| EmilienM | bogdando: this is how the containeirzed undercloud is configured with br-ex now | 14:08 |
| EmilienM | bogdando: eth1 was wrong | 14:08 |
| EmilienM | bogdando: quickstart creates br-ex though | 14:08 |
| *** rodolof has quit IRC | 14:08 | |
| bogdando | well, EmilienM I just searched all code and found it literally everyther | 14:08 |
| bogdando | and nothing that uses br-ex | 14:09 |
| *** rodolof has joined #tripleo | 14:09 | |
| bogdando | http://codesearch.openstack.org/?q=NeutronPublicInterface&i=nope&files=&repos= | 14:09 |
| bogdando | so I'm not sure | 14:09 |
| alee__ | d0ugal, rbrady , toure - made changes to https://review.openstack.org/#/c/574782/ as requested | 14:09 |
| EmilienM | bogdando: the job is working now | 14:10 |
| EmilienM | bogdando: it's actually fixing the job, please remove your -1 | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777130 | 14:10 |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,New] - Assigned to Quique Llorente (quiquell) | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| EmilienM | bandini: nice spot for the mirror problem | 14:11 |
| EmilienM | http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-06-14_14_54_30 | 14:11 |
| EmilienM | Exception: Error inspecting image: docker://docker.io/tripleomaster/centos-binary-iscsid:current-tripleo | 14:11 |
| EmilienM | 2018-06-14 14:54:30 | time="2018-06-14T14:54:30Z" level=fatal msg="Error determining repository tags: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-iscsid/tags/list: EOF" | 14:11 |
| EmilienM | I would rather investigate that problem | 14:11 |
| bandini | EmilienM: :) | 14:12 |
| EmilienM | mwhahaha, bandini : I think it happens before we configure docker | 14:12 |
| EmilienM | so the mirror isn't configured yet, let me check | 14:12 |
| mwhahaha | EmilienM: negative it's configured | 14:12 |
| mwhahaha | EmilienM: it's a different set of software | 14:12 |
| EmilienM | mhh weird, docker is configured http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/etc/docker/daemon.json.txt.gz | 14:12 |
| mwhahaha | so i don't think it honors the docker config | 14:12 |
| * mwhahaha files a bug | 14:12 | |
| EmilienM | ok | 14:12 |
| ratailor | Does anyone have any suggestion about this comment here. https://review.openstack.org/#/c/575623/1/docker/services/nova-api.yaml | 14:13 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: [WIP]Add a OpenShift of OpenStack HW validation https://review.openstack.org/575749 | 14:13 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: [WIP]Add a OpenShift on OpenStack HW validation https://review.openstack.org/575749 | 14:13 |
| ratailor | EmilienM, jaosorior ^^ | 14:13 |
| mwhahaha | EmilienM, bandini, bogdando: https://bugs.launchpad.net/tripleo/+bug/1777143 | 14:14 |
| openstack | Launchpad bug 1777143 in tripleo "container tag discovery does not use a mirror" [High,Triaged] | 14:14 |
| mwhahaha | i think we can switch the mirror by configuring the namespace correctly in oooq | 14:14 |
| * mwhahaha isn't 100% on that though | 14:15 | |
| bogdando | EmilienM: I removed -1, could you please elaborate how undercloud already has br-ex configured for NeutronPublicInterface? | 14:15 |
| mwhahaha | bogdando: in CI we add it in the vxlan script | 14:15 |
| mwhahaha | bogdando: see bottom of http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz#_2018-06-14_14_16_47 | 14:16 |
| bogdando | mwhahaha: I cannot see how this relates to heat's NeutronPublicInterface | 14:17 |
| mwhahaha | bogdando: so that's what we end up putting the local-ip on | 14:17 |
| mwhahaha | bogdando: this is just matching what we already do for the undercloud | 14:18 |
| mwhahaha | in CI | 14:18 |
| mwhahaha | this is purely an upstream CI configuration | 14:18 |
| *** etingof has joined #tripleo | 14:18 | |
| mwhahaha | bogdando: see local_interface in http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/undercloud.conf.txt.gz | 14:18 |
| EmilienM | mwhahaha: there is "docker_registry_host" in quickstart | 14:19 |
| mwhahaha | EmilienM: yea but i don't know if skopeo honors that stuff | 14:19 |
| mwhahaha | bogdando: so in undercloud local_interface -> NuetronPublicInterface | 14:19 |
| mwhahaha | bogdando: that's our network config mapping crud | 14:19 |
| EmilienM | and it's used by "openstack overcloud container image prepare" | 14:19 |
| mwhahaha | bogdando: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/undercloud_config.py#L67 | 14:20 |
| *** pradk has joined #tripleo | 14:20 | |
| mwhahaha | EmilienM: but we don't use it when we do the automagic tag discovery in containerized undercloud i bet | 14:20 |
| bogdando | mwhahaha: oh, right! those 'local_interface': 'NeutronPublicInterface' magic mappings | 14:21 |
| *** lblanchard has joined #tripleo | 14:21 | |
| *** florianf has quit IRC | 14:21 | |
| *** florianf has joined #tripleo | 14:22 | |
| *** mvenesio has joined #tripleo | 14:22 | |
| EmilienM | mwhahaha: what worries me is I don't see docker_registry_host overridden with our mirror except for rdo cloud jobs | 14:22 |
| *** amoralej|lunch is now known as amoralej | 14:23 | |
| EmilienM | grep docker_registry_host in tripleo-ci repo | 14:23 |
| *** bdodd_ is now known as bdodd | 14:24 | |
| EmilienM | so the problem is probably in both "tripleo image container prepare" and the old container image prepare as well | 14:24 |
| EmilienM | bandini spotted the one when deploying the undercloud, so the new command | 14:24 |
| *** quique|rover|lch is now known as quiquell|rover | 14:24 | |
| EmilienM | but for the second one I suspect something in quickstart | 14:24 |
| *** rodolof has quit IRC | 14:24 | |
| bogdando | mwhahaha, EmilienM: here is that magic mapping http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/tripleo-heat-installer-templates/undercloud_parameters.yaml.txt.gz . I completely forgot about that mappings, thanks for refreshing my mind :D | 14:27 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add dry run option to toci_quickstart - playbook order in tact https://review.openstack.org/570694 | 14:27 |
| * EmilienM brb 30m | 14:28 | |
| mwhahaha | bogdando: yea so for standalone we're just generating that file by hand for now until we can create a standalone.conf or some sort | 14:28 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Remove zuul-cloner from buildimage job https://review.openstack.org/571951 | 14:28 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Remove zuul-cloner from tripleo-ci job https://review.openstack.org/571995 | 14:28 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Refactor Zuul job using base jobs to consolidate the configuration https://review.openstack.org/572027 | 14:28 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Add TRIPLEO_CI_ROOT variable https://review.openstack.org/574122 | 14:28 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: bootstrap a native zuulv3 tripleo-ci job https://review.openstack.org/575752 | 14:28 |
| openstackgerrit | Brent Eagles proposed openstack/ansible-role-container-registry master: Do not stop all running containers on docker update https://review.openstack.org/575756 | 14:29 |
| openstackgerrit | Carlos Goncalves proposed openstack/puppet-tripleo stable/pike: Install python-neutron-lbaas before starting neutron https://review.openstack.org/575500 | 14:32 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates stable/queens: Do not stop all running containers on docker update https://review.openstack.org/575758 | 14:33 |
| *** yprokule has quit IRC | 14:39 | |
| larsks | mwhahaha: so, in the horizon service template, we're setting horizon::keystone_url to the internal endpoint...but this is used for browser redirects when using web sso, so it should actually be the public endpoint. Should I just submit a patch to change that? Or is there a way to paramaterize which endpoint we use (nested get_param)? | 14:40 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Fix error in the role-specific parameter merging in rhsm service https://review.openstack.org/575761 | 14:40 |
| larsks | (https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/horizon.yaml#L124) | 14:40 |
| mwhahaha | larsks: i'd assume there's a KeystonePublic but you should check with the keystone folks on that | 14:41 |
| *** ramishra has quit IRC | 14:41 | |
| mwhahaha | larsks: yea there's a KeystonePublic so if it's supposed to be the public end point just submit a patch to change it | 14:41 |
| larsks | mwhahaha: I have, and they agree we should use the public endpoint there. But should I replace KeystoneInternal with KeystonePublic, or should I replace it with something like {get_param: KeystoneEndpointName}? | 14:41 |
| larsks | (so that it can be configured conveniently at deploy time) | 14:41 |
| larsks | (and introduce a new parameter, obviously) | 14:42 |
| mwhahaha | larsks: no KeystonePUblic | 14:42 |
| larsks | Awesome. | 14:42 |
| mwhahaha | larsks: because that comes from the endpoint map | 14:42 |
| larsks | Thanks. | 14:42 |
| *** rpioso|afk is now known as rpioso | 14:42 | |
| larsks | Oh, right, but I was asking if we could parameterize which endpointmap key we're requesting. E.g., KeystoneEndpointName would be a parameter that could be KeystoneInternal or KeystonePublic, and then we wwould get that from EndpointMap. | 14:42 |
| mwhahaha | oh not sure about that | 14:43 |
| mwhahaha | unless we're already doing that elsewhere i don't think we should | 14:43 |
| larsks | Okay. I'll look. | 14:44 |
| mwhahaha | If you were going to add a param it would be a condition between internal/external that we could then use to switch the get param bits. but i think that's over thinking it unless there's a reason | 14:44 |
| mwhahaha | you may want to ask jaosorior | 14:44 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates master: [WIP] enable support for openidc federation in keystone https://review.openstack.org/574855 | 14:46 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates master: use keystone public endpoint in horizon https://review.openstack.org/575762 | 14:46 |
| *** ratailor has quit IRC | 14:46 | |
| larsks | mwhahaha: jaosorior: https://review.openstack.org/#/c/575762/ | 14:48 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Improve exp backoff for images pull/push/rpm-upd https://review.openstack.org/575742 | 14:49 |
| weshay|ruck | time="2018-06-15T14:35:19Z" level=fatal msg="pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: no such host" | 14:53 |
| weshay|ruck | EmilienM, this is odd | 14:53 |
| weshay|ruck | gate failure | 14:53 |
| weshay|ruck | guess just a dns fail | 14:53 |
| quiquell|rover | weshay|ruck: http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 14:54 |
| quiquell|rover | something about not finding RPMs | 14:54 |
| *** myoung is now known as myoung|bbl | 14:54 | |
| quiquell|rover | ERROR: ERROR running rpm query in container: | 14:55 |
| *** dparkes has joined #tripleo | 14:57 | |
| weshay|ruck | EmilienM, do you have a sec? | 15:00 |
| weshay|ruck | or bogdando or sbaker | 15:01 |
| EmilienM | yes I'm here now | 15:01 |
| *** pcaruana has quit IRC | 15:01 | |
| weshay|ruck | EmilienM, k.. need your eyes.. it looks like container update is failing to find the container in the undercloud registry when updating | 15:02 |
| weshay|ruck | have a log up need eyes | 15:02 |
| weshay|ruck | my bluejeans channeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee | 15:02 |
| weshay|ruck | eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeel | 15:02 |
| weshay|ruck | ugh.. bluejeans lag | 15:02 |
| *** lblanchard has quit IRC | 15:04 | |
| weshay|ruck | EmilienM, I suspect the diff in containers between the containerized undercloud and whats required by the overcloud? | 15:07 |
| weshay|ruck | http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 15:07 |
| weshay|ruck | meh.. this is not a containerized undercloud nvrmind | 15:08 |
| quiquell|rover | weshay|ruck: Have to drop | 15:08 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Improve exp backoff for images pull/push/rpm-upd https://review.openstack.org/575742 | 15:08 |
| bogdando | weshay|ruck: LD | 15:09 |
| *** quiquell|rover is now known as quiquell|off | 15:09 | |
| bogdando | for the moment I started to believe that might be the case | 15:09 |
| weshay|ruck | bogdando, so weird https://review.openstack.org/#/c/571613/ works in check but fails in gate over and over | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777130 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 15:10 |
| *** weshay|ruck changes topic to "Welcome to Rocky. CI status: RED - rdo-cloud is down, long upstream zuul queue | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest" | 15:10 | |
| bogdando | weshay|ruck: are node pool vms are the same for check vs gate? | 15:11 |
| EmilienM | yes | 15:11 |
| bogdando | may be the latter cannot access registry mirrors? | 15:11 |
| weshay|ruck | bogdando, they could be running on diff providers between check and gate | 15:11 |
| bogdando | or poorly located, slow etc | 15:11 |
| bogdando | meh, kolla images really need to be included into those vms | 15:11 |
| weshay|ruck | the error is that it can't access the container from the undercloud during container-check | 15:11 |
| EmilienM | bogdando: it doesn't help, we modify them | 15:12 |
| bogdando | the delta might be less at least for containers pull | 15:12 |
| bogdando | not for rpm updates, indeed... | 15:12 |
| weshay|ruck | Trying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-iscsid ... | 15:12 |
| weshay|ruck | 2018-06-15 08:45:23 | Pulling repository 192.168.24.1:8787/tripleomaster/centos-binary-iscsid | 15:12 |
| weshay|ruck | 2018-06-15 08:45:23 | /usr/bin/docker-current: Error: image tripleomaster/centos-binary-iscsid:39d396e0ae0d982d63cbcadef57e75e84a052911_033c532c not found. | 15:12 |
| * weshay|ruck bringing up a recreate now | 15:12 | |
| *** gouthamr has joined #tripleo | 15:19 | |
| bogdando | PTAL https://review.openstack.org/#/c/572404/ | 15:23 |
| bogdando | https://review.openstack.org/#/c/575676/ | 15:24 |
| *** mschuppert has quit IRC | 15:29 | |
| bogdando | weshay|ruck, EmilienM: that's weird it pulls from the local registry w/o pushing it there from the upstream source | 15:29 |
| *** dxiri has joined #tripleo | 15:30 | |
| bogdando | it seems | 15:30 |
| bogdando | openstack tripleo container image prepare didn't pull it | 15:30 |
| bogdando | or was it cleaned up? | 15:34 |
| larsks | EmilienM: what's the correct way of setting multi-valued keystone configuration settings in puppet (ones that should result in multiple lines each with the same config key)? | 15:35 |
| larsks | (I tried keystone_config { 'trusted_dashboard': value => $a_list_variable }, and that didn't work...) | 15:37 |
| bogdando | weshay|ruck, EmilienM: http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz shows 0 images in the local cache, I think that's expected for cleanup. So we need to figure out why the image is not in local registry | 15:37 |
| *** gouthamr has quit IRC | 15:38 | |
| larsks | Or mwhahaha ^^^^ if you have any idea. | 15:38 |
| *** gouthamr has joined #tripleo | 15:39 | |
| EmilienM | bogdando: http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 15:39 |
| EmilienM | it's because the container image prepare failed | 15:39 |
| *** rbrady is now known as rbrady-afk | 15:39 | |
| EmilienM | and the undercloud isn't containerized so no containers are in local registry | 15:39 |
| mwhahaha | larsks: the provider has to support those (it may not) | 15:39 |
| mwhahaha | larsks: also, i hate those types of params | 15:39 |
| larsks | Yeah, me too. | 15:39 |
| EmilienM | multi params is the worst thing they created for operators wrt config management | 15:40 |
| mwhahaha | we did add support in the puppet providers and i think some of the neutron ones support it, the keystone ones may not | 15:40 |
| mwhahaha | so it would have to be updated to | 15:40 |
| larsks | mwhahaha: it looks like they just use puppet-openstacklib? | 15:41 |
| mwhahaha | yea but there's a specific pattern for it | 15:41 |
| * mwhahaha doesn't remember it's been a few yuears | 15:41 | |
| mwhahaha | i think we even added it in for lbaas or some crap like that | 15:41 |
| * larsks looks at puppet-neutron | 15:42 | |
| EmilienM | i think we added it to puppet keystone | 15:42 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Update CephX client.openstack keyring to use 'profile rbd' https://review.openstack.org/575776 | 15:43 |
| mwhahaha | nova has it | 15:43 |
| larsks | mwhahaha: in lib/puppet/provider/nova_config/openstackconfig.rb? | 15:44 |
| EmilienM | well, can't find it, probably not then | 15:44 |
| mwhahaha | https://review.openstack.org/#/c/329991/ | 15:44 |
| mwhahaha | you have to use the openstackconfig version of the provider | 15:44 |
| mwhahaha | not ini_setting | 15:44 |
| EmilienM | https://review.openstack.org/#/c/492172/ | 15:45 |
| mwhahaha | it's complicated | 15:45 |
| bogdando | EmilienM: in that log, I can see the container-check command invoked before any failures | 15:45 |
| bogdando | how was it expected to work w/o containers pulled and pushed into local registry? | 15:45 |
| larsks | mwhahaha: okay, thanks. | 15:46 |
| larsks | I'll poke at it. | 15:46 |
| bogdando | EmilienM: or do you mean there is a silent failure after http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-06-15_08_39_07 ? | 15:46 |
| bogdando | it had been doing pull/push for a 6 min, right? didn't make it? | 15:47 |
| bogdando | we need to improve logging :) | 15:47 |
| openstackgerrit | Bob Fournier proposed openstack/tripleo-docs master: Document how to use a composable network with Ironic in Overcloud https://review.openstack.org/570437 | 15:50 |
| EmilienM | bogdando: right, that's one of the problems we need to look | 15:52 |
| EmilienM | but I think there is more | 15:52 |
| EmilienM | I wish we had a list of reasons why gate job are failing now | 15:52 |
| EmilienM | is the CI team can provide that? | 15:52 |
| EmilienM | weshay|ruck: ^ do some work please | 15:52 |
| bogdando | EmilienM: hm, there is a tones of ' HTTP response to HTTPS client' in underlcoud journald | 15:52 |
| EmilienM | weshay|ruck: :P | 15:53 |
| weshay|ruck | EmilienM, interesting.. openstack-infra is saying the proxy cache is not working correctly | 15:53 |
| * EmilienM trolling it's friday !! | 15:53 | |
| bogdando | \o/ | 15:53 |
| bogdando | please do the needful? :D | 15:53 |
| EmilienM | bogdando has a good point with the update issues | 15:53 |
| *** radeks has joined #tripleo | 15:53 | |
| EmilienM | but we also need to figure out https://bugs.launchpad.net/tripleo/+bug/1777143 | 15:53 |
| openstack | Launchpad bug 1777143 in tripleo "container tag discovery does not use a mirror" [High,Triaged] | 15:53 |
| weshay|ruck | oh man | 15:54 |
| weshay|ruck | the jokes | 15:54 |
| weshay|ruck | the jokes | 15:54 |
| EmilienM | weshay|ruck: can you provide links of jobs that failed in the gate during the last 10 hours | 15:55 |
| EmilienM | weshay|ruck: let's look at the error, and file a bug for each of them | 15:55 |
| weshay|ruck | EmilienM, when rdo cloud comes back | 15:55 |
| EmilienM | come on don't tell me we rely on rdo cloud to tell what gate job is failing | 15:55 |
| EmilienM | I quit | 15:55 |
| weshay|ruck | http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 | 15:56 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: bootstrap a native zuulv3 tripleo-ci job https://review.openstack.org/575752 | 15:56 |
| weshay|ruck | is down.. that has the list | 15:56 |
| EmilienM | damn | 15:56 |
| EmilienM | trying to use http://zuul.openstack.org/builds.html | 15:56 |
| EmilienM | but it's damn slow | 15:56 |
| weshay|ruck | EmilienM, read through openstack-infra as well | 15:58 |
| EmilienM | I'm sure we can find it via logstash | 15:58 |
| EmilienM | let me build a query to find out errors in gate | 15:58 |
| weshay|ruck | EmilienM, if we do use the proxy.. it's just a direct pass through | 15:58 |
| weshay|ruck | and they are trying to fix that | 15:58 |
| EmilienM | ah so if docker.io decides to not respond the proxy is useless | 15:59 |
| EmilienM | why do we use a proxy then | 15:59 |
| bogdando | interesting | 16:00 |
| bogdando | how did you figure that out EmilienM? | 16:00 |
| bogdando | that behavior of mirrors? | 16:00 |
| EmilienM | bogdando: join #openstack-infra | 16:00 |
| bogdando | ah, ok | 16:00 |
| *** dmellado_ has quit IRC | 16:02 | |
| bogdando | EmilienM: aside of those HTTP vs HTTPS errors, there is a craptone of 'Attempting next endpoint for pull after error: manifest unknown: manifest unknown' | 16:02 |
| bogdando | and others like Not continuing with pull after error: Error: image tripleomaster/centos-binary-heat-api:39d396e0ae0d982d63cbcadef57e75e84a052911_033c532c not found | 16:02 |
| bogdando | those are failing into those 6 silent minutes | 16:03 |
| bogdando | EmilienM: my bad, they are later | 16:03 |
| bogdando | come after 08:45:09 | 16:04 |
| bogdando | so we really have nothing to inspect for those 6 mins it seems | 16:04 |
| EmilienM | I have a list of failures in the gate : | 16:05 |
| EmilienM | goo.gl/kbdMH7 | 16:05 |
| *** ykarel|away has joined #tripleo | 16:05 | |
| EmilienM | weshay|ruck: ^ | 16:05 |
| weshay|ruck | looking at it now | 16:06 |
| weshay|ruck | that is a very fine point | 16:06 |
| EmilienM | I'm trying to build a better query but you got the idea | 16:07 |
| weshay|ruck | I have one | 16:07 |
| *** ffiore has quit IRC | 16:07 | |
| EmilienM | share it like : build_queue: ^gate$ AND build_name: *tripleo-ci* AND build_status: FAILURE | 16:08 |
| *** tcw1 has quit IRC | 16:08 | |
| EmilienM | so it's pretty clear when we purged the gate | 16:08 |
| EmilienM | and when it came back unstable | 16:08 |
| EmilienM | you can see the revert between the 2 picks | 16:08 |
| EmilienM | and the revert didn't help that much it seems | 16:09 |
| EmilienM | the queue is 13h now | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777130 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 16:10 |
| EmilienM | http://zuul.openstack.org/builds.html is also great to see job health | 16:10 |
| EmilienM | weshay|ruck: ^ we should use more of existing infra tools sometimes | 16:11 |
| EmilienM | instead of creating new things | 16:11 |
| EmilienM | new things are good, but we need to also use existing things | 16:11 |
| EmilienM | anyway, I'm building up a list of failures seen in logstash | 16:11 |
| EmilienM | http://etherpad.openstack.org/p/tripleo-gate-issues-june-2018 | 16:11 |
| *** bogdando has quit IRC | 16:13 | |
| EmilienM | the only thing with http://zuul.openstack.org/builds.html is I can't find how to use regex | 16:13 |
| EmilienM | I guess we just can't | 16:13 |
| weshay|ruck | EmilienM, re: building tools, if the team has to go to 10 different web sites to get clarity of the status | 16:14 |
| weshay|ruck | that is a problem, and that is what we are resolving | 16:14 |
| EmilienM | sure but your stuff is not available now | 16:15 |
| EmilienM | anyway | 16:15 |
| EmilienM | can we get help from CI folks please | 16:15 |
| *** tcw has joined #tripleo | 16:16 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-ha-utils master: WIP - Introduce simulate failure task https://review.openstack.org/575784 | 16:16 |
| *** ayoung has quit IRC | 16:18 | |
| EmilienM | the query isn't good | 16:23 |
| EmilienM | I have successful jobs in the list of results | 16:23 |
| *** holser__ has quit IRC | 16:25 | |
| *** ayoung has joined #tripleo | 16:25 | |
| *** jpena is now known as jpena|off | 16:27 | |
| *** cylopez has quit IRC | 16:28 | |
| *** jpena|off is now known as jpena | 16:29 | |
| *** jpena is now known as jpena|away | 16:29 | |
| *** panda is now known as panda|off | 16:30 | |
| EmilienM | another issue in gate: | 16:30 |
| EmilienM | http://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/job-output.txt.gz#_2018-06-15_14_32_37_611511 | 16:30 |
| EmilienM | building python-networking-ovn rpm this time | 16:30 |
| mwhahaha | :( | 16:30 |
| EmilienM | just read logstash it's magic | 16:31 |
| EmilienM | it tells you where shit broke | 16:31 |
| mwhahaha | ovn packaging is bad | 16:31 |
| mwhahaha | got new files | 16:31 |
| EmilienM | filing a bug for this one | 16:31 |
| mwhahaha | ansible'd | 16:31 |
| mwhahaha | http://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz | 16:31 |
| mwhahaha | why the hell is tripleo update role going in neutroning-ovn-migration | 16:32 |
| mwhahaha | dalvarez: https://review.openstack.org/#/c/574211/ -^ | 16:33 |
| EmilienM | new alert: https://bugs.launchpad.net/tripleo/+bug/1777168 | 16:33 |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 16:33 |
| EmilienM | we probably want to make the job non voting now if nobody is looking at it quickly | 16:33 |
| mwhahaha | yes plz | 16:38 |
| *** jpich has quit IRC | 16:39 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/pike: Set Ceph pools rule_name to replicated_rule https://review.openstack.org/575789 | 16:40 |
| *** derekh has quit IRC | 16:41 | |
| honza | EmilienM: i'm working on the tempest plugin for tripleo-ui, and i'm wondering where to plug it in --- do you know, off the top of your head, if there is a ci job that has both an undercloud and tempest available? | 16:42 |
| *** mhenkel_ has quit IRC | 16:43 | |
| *** gkadam has quit IRC | 16:43 | |
| *** mhenkel_ has joined #tripleo | 16:43 | |
| *** ykarel|away has quit IRC | 16:43 | |
| EmilienM | honza: fs027 | 16:46 |
| EmilienM | I'm going to purge the gate (again) | 16:46 |
| EmilienM | so we can land the patch weshay|ruck will propose to disable voting on scenario007 | 16:46 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate https://review.openstack.org/575790 | 16:47 |
| *** mhenkel_ has quit IRC | 16:47 | |
| *** hamzy_ has joined #tripleo | 16:48 | |
| EmilienM | http://logs.openstack.org/81/568781/4/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/631a562/logs/ara_oooq/ | 16:50 |
| EmilienM | weshay|ruck: ^ | 16:50 |
| *** gouthamr has quit IRC | 16:50 | |
| *** gouthamr has joined #tripleo | 16:50 | |
| *** hamzy has quit IRC | 16:51 | |
| dalvarez | mwhahaha: those roles are to migrate from ML2/OVS to OVN | 16:51 |
| dalvarez | mwhahaha: i proposed the patch to RDO | 16:51 |
| dalvarez | but revoew.rdoproject.org seems down | 16:52 |
| mwhahaha | dalvarez: hmm ok seems odd to have something named tripleo in that repo and not as part of the upgrade bits for us | 16:53 |
| dalvarez | mwhahaha: it's the migration from ml2ovs to ovn from tripleo | 16:54 |
| dalvarez | i proposed the patch to fix the packaging a few days back but we didnt have all the upstream bits merged in networking-ovn | 16:54 |
| dalvarez | now they got merged but the package is broken and review.rdoproject is down :S | 16:54 |
| mwhahaha | dalvarez: yea we probably should have added conditionals to the packaging so it worked in both cases before merging that | 16:55 |
| mwhahaha | now it's completely hosed the packaging for that | 16:55 |
| mwhahaha | and rdocloud is having problem so yea | 16:55 |
| dalvarez | yeah :s didn't know about those conditionals | 16:55 |
| EmilienM | mnaser: just by curiosity, is there anything happening in your cloud right now? | 16:55 |
| EmilienM | like networking issues or something | 16:55 |
| mwhahaha | when adding new directories you can just if [ -d ... ] on the d for inclusion | 16:56 |
| EmilienM | we are seeing that package downloads in our tripleo ci jobs took more time than usual in your cloud | 16:56 |
| dalvarez | mwhahaha: but in the spec it'll see the new files and it'll fail no? | 16:56 |
| mwhahaha | dalvarez: depends on what we're doing there's some workarounds to get it in | 16:57 |
| *** zoli is now known as zoli|gone | 16:57 | |
| *** zoli|gone is now known as zoli | 16:57 | |
| dalvarez | mwhahaha: ack, i checked with amoralej to do this but wasn't aware of any workaround, i will double check with him for next times | 16:57 |
| mwhahaha | dalvarez: https://github.com/rdo-packages/tripleo-common-distgit/blob/rpm-master/openstack-tripleo-common.spec#L117 | 16:57 |
| mwhahaha | dalvarez: you pre-create the dir and add it to the list | 16:58 |
| dalvarez | mwhahaha: gotcha, and same for entry points? this is adding console scripts too | 16:58 |
| dalvarez | like for example touch <file> ? | 16:58 |
| mwhahaha | dalvarez: yea | 16:58 |
| dalvarez | ack | 16:58 |
| mwhahaha | then you could clean them up after | 16:58 |
| mwhahaha | that would prevent this type of failure | 16:59 |
| dalvarez | got it | 16:59 |
| mwhahaha | we'll just disable scenario007 for now | 16:59 |
| *** gfidente has quit IRC | 16:59 | |
| dalvarez | yeah, good one... wasn't aware, hope rdocloud is back again and we can fix it ASAP, the dist-git patch passed CI last week | 16:59 |
| dalvarez | ack | 16:59 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate https://review.openstack.org/575790 | 16:59 |
| mwhahaha | and hope it didn't break anything else :/ | 16:59 |
| *** tesseract has quit IRC | 17:01 | |
| amoralej | dalvarez, mwhahaha what's the problem? | 17:03 |
| mwhahaha | amoralej: networking-ovn packaging is broken and rdocloud is dead | 17:03 |
| dalvarez | amoralej: that the upstream patches got merged in networking-ovn and we can't merge the changes in the spec so ovn jobs are broken | 17:03 |
| amoralej | ahh | 17:03 |
| EmilienM | gate is purged | 17:03 |
| amoralej | i see | 17:03 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate https://review.openstack.org/575790 | 17:05 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate https://review.openstack.org/575790 | 17:10 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777130 | 17:10 |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777168 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 17:10 |
| openstackgerrit | wes hayutin proposed openstack/puppet-tripleo master: scen007 is failing in the gate https://review.openstack.org/575795 | 17:11 |
| *** amoralej is now known as amoralej|off | 17:14 | |
| *** kopecmartin has quit IRC | 17:14 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-common master: scen007 is failing in the gate https://review.openstack.org/575796 | 17:15 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: scen007 is failing in the gate https://review.openstack.org/575798 | 17:18 |
| dalvarez | can anybody take a look at this pls? https://review.openstack.org/#/c/570942/ it's only missing workflow | 17:20 |
| dalvarez | beagles: ^ :? | 17:20 |
| mwhahaha | dalvarez: scenario007 is broken | 17:20 |
| mwhahaha | we cannot merge that right now | 17:20 |
| dalvarez | mwhahaha: ops true! damn sorry | 17:20 |
| *** radeks has quit IRC | 17:21 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: scen007 is failing in the gate https://review.openstack.org/575799 | 17:22 |
| EmilienM | I've purged the check as well | 17:23 |
| EmilienM | so we can merge the infra patch that fix the proxy | 17:23 |
| *** pradk has quit IRC | 17:23 | |
| EmilienM | @all please do not recheck or +A | 17:24 |
| *** ccamacho has quit IRC | 17:24 | |
| EmilienM | I've abandoned a bunch of jobs from the check, i'll restore them shortly | 17:25 |
| EmilienM | I wanted to let the infra patch to land quickly | 17:25 |
| *** pradk has joined #tripleo | 17:25 | |
| *** gouthamr_ has joined #tripleo | 17:27 | |
| *** gouthamr has quit IRC | 17:27 | |
| *** gouthamr_ is now known as gouthamr | 17:27 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: use the tq ansible.cfg for the undercloud install https://review.openstack.org/575535 | 17:33 |
| *** pkovar has quit IRC | 17:33 | |
| *** janki has quit IRC | 17:34 | |
| *** pradk has quit IRC | 17:35 | |
| *** bugzy_ is now known as bugzy | 17:40 | |
| *** cdearborn has quit IRC | 17:43 | |
| *** gkadam has joined #tripleo | 17:44 | |
| *** artom has joined #tripleo | 17:44 | |
| *** leanderthal has joined #tripleo | 17:47 | |
| *** jpena|away is now known as jpena | 17:47 | |
| *** chandankumar is now known as chkumar|pto | 18:01 | |
| *** stevebaker has joined #tripleo | 18:01 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777130 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777168 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 18:10 |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 18:10 |
| *** agopi has joined #tripleo | 18:11 | |
| EmilienM | https://review.openstack.org/#/c/575787 landed, the proxy should be in place soon | 18:12 |
| *** jpena is now known as jpena|off | 18:13 | |
| *** cdearborn has joined #tripleo | 18:17 | |
| *** florianf has quit IRC | 18:26 | |
| *** rmascena has joined #tripleo | 18:27 | |
| *** raildo has quit IRC | 18:30 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate https://review.openstack.org/575790 | 18:30 |
| EmilienM | weshay|ruck: it was failing in check | 18:30 |
| EmilienM | I moved it to check | 18:30 |
| EmilienM | err, it was failing in gate, I moved it to check | 18:30 |
| *** rlandy_ has joined #tripleo | 18:31 | |
| *** rlandy_ has quit IRC | 18:31 | |
| *** rlandy_ has joined #tripleo | 18:31 | |
| *** rlandy has quit IRC | 18:33 | |
| *** gkadam_ has joined #tripleo | 18:33 | |
| *** rlandy_ is now known as rlandy | 18:36 | |
| *** gkadam has quit IRC | 18:37 | |
| beagles | EmilienM: are we still holding off on +A stuff because of queue? | 18:37 |
| *** leanderthal has quit IRC | 18:37 | |
| beagles | he asks as he goes to see what the state of the queue is | 18:37 |
| EmilienM | beagles: yeah please don't approve anything | 18:38 |
| EmilienM | you can use check | 18:38 |
| beagles | EmilienM: gotcha | 18:38 |
| EmilienM | but as long as we haven't seen successful gate jobs, we hold off | 18:39 |
| * beagles nods | 18:39 | |
| *** artom has quit IRC | 18:41 | |
| *** Goneri has joined #tripleo | 18:43 | |
| *** matbu has quit IRC | 18:47 | |
| *** radeks has joined #tripleo | 18:52 | |
| mnaser | EmilienM: nothing to note in specific, do you have urls that you're seeing download dslowly | 18:56 |
| mnaser | EmilienM: if you have some, i can dig in and figure out whats going on | 18:57 |
| *** radeks_ has joined #tripleo | 18:59 | |
| *** radeks has quit IRC | 19:01 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777130 | 19:10 |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777168 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 19:10 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Do not grant caps if metrics pool name is empty https://review.openstack.org/575571 | 19:11 |
| *** pradk has joined #tripleo | 19:11 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Make BIND listen address configurable https://review.openstack.org/575822 | 19:11 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Make BIND /var dir persistent https://review.openstack.org/575823 | 19:11 |
| *** dougbtv_ has quit IRC | 19:14 | |
| *** rlandy has quit IRC | 19:15 | |
| *** sai_p has joined #tripleo | 19:18 | |
| *** rlandy has joined #tripleo | 19:18 | |
| *** weshay|ruck changes topic to "Welcome to Rocky. CI status: RED - rdo-cloud is back, zuul queue cleared| http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest" | 19:19 | |
| weshay|ruck | dsneddon, if you are avail to poke at https://bugs.launchpad.net/tripleo/+bug/1777168 RDO is back and we can get a spec update in | 19:22 |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 19:22 |
| dsneddon | weshay|ruck, I'll take a look | 19:26 |
| dsneddon | weshay|ruck, https://review.openstack.org/#/c/575799 looks good to me, should I merge it? | 19:28 |
| dsneddon | weshay|ruck, Nevermind, Emilien already did... | 19:29 |
| weshay|ruck | ah ha | 19:30 |
| weshay|ruck | thanks dsneddon /me looks | 19:30 |
| *** Goneri has quit IRC | 19:31 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/puppet-tripleo stable/ocata: Conditionally Enable Nuage VRS https://review.openstack.org/575826 | 19:31 |
| openstackgerrit | Sai Ram Peesapati proposed openstack/puppet-tripleo stable/newton: Conditionally Enable Nuage VRS https://review.openstack.org/575827 | 19:31 |
| weshay|ruck | dsneddon, /me looking for a patch on distgit for http://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz | 19:33 |
| weshay|ruck | https://review.rdoproject.org/r/#/q/project:openstack/networking-ovn-distgit | 19:34 |
| *** lifeless has joined #tripleo | 19:34 | |
| *** mvenesio has quit IRC | 19:36 | |
| *** holser__ has joined #tripleo | 19:37 | |
| weshay|ruck | hrm.. I see networking-ovn-migration/playbooks in https://github.com/rdo-packages/networking-ovn-distgit/blob/rpm-master/python-networking-ovn.spec | 19:38 |
| weshay|ruck | trown, what error did you see the build? | 19:38 |
| weshay|ruck | setup.cfg? | 19:39 |
| *** rlandy_ has joined #tripleo | 19:39 | |
| *** Goneri has joined #tripleo | 19:39 | |
| trown | weshay|ruck: hmm let me find it | 19:39 |
| *** rlandy has quit IRC | 19:41 | |
| jrist | review begging https://review.openstack.org/#/c/498936 | 19:41 |
| weshay|ruck | trown, hrm.. https://github.com/rdo-packages/networking-ovn-distgit/blob/rpm-master/python-networking-ovn.spec#L151-L155 | 19:42 |
| weshay|ruck | is not in | 19:42 |
| weshay|ruck | http://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz | 19:42 |
| trown | weshay|ruck: it is in http://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz ... but maybe that pulled older distgit? | 19:42 |
| * weshay|ruck looks at new log | 19:42 | |
| weshay|ruck | trown, /me doesn't see %files migration-tool | 19:44 |
| weshay|ruck | trown, we're not consistent | 19:44 |
| *** radeks_ has quit IRC | 19:45 | |
| trown | I dont think that should matter for a package built in a job | 19:45 |
| openstackgerrit | Dan Sneddon proposed openstack/os-net-config master: [WIP] Add Policy Based Routing Support to os-net-config https://review.openstack.org/575712 | 19:45 |
| weshay|ruck | trown, oh that is probably true | 19:46 |
| * weshay|ruck wonder why it has a old copy of the spec | 19:46 | |
| trown | weshay|ruck: only thing I can think of is that github mirror of rdoproject was not up to date because of cloud outage... maybe it will resolve now? | 19:53 |
| weshay|ruck | ya infra! | 19:54 |
| weshay|ruck | I mean.. yay infra! | 19:54 |
| *** holser___ has joined #tripleo | 20:01 | |
| *** lifeless has quit IRC | 20:02 | |
| *** lifeless has joined #tripleo | 20:03 | |
| *** dxiri has quit IRC | 20:05 | |
| *** holser__ has quit IRC | 20:05 | |
| *** dxiri has joined #tripleo | 20:08 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777168 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 20:10 |
| *** social has quit IRC | 20:13 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: use the tq ansible.cfg for the undercloud install https://review.openstack.org/575535 | 20:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: scen007 is failing in the gate https://review.openstack.org/575798 | 20:18 |
| *** owalsh has quit IRC | 20:20 | |
| *** owalsh has joined #tripleo | 20:20 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DNM - Test change https://review.openstack.org/575837 | 20:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use the in-built image prepare for undercloud install https://review.openstack.org/571613 | 20:24 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Switch to workflow driven image prepare https://review.openstack.org/573476 | 20:24 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Use role tripleo-modify-image to update containers https://review.openstack.org/571631 | 20:24 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Switch to workflow driven image prepare https://review.openstack.org/573476 | 20:24 |
| EmilienM | weshay|ruck: let's try a revert of https://review.openstack.org/#/c/575798/ and a patch that kick off scenario007 | 20:26 |
| EmilienM | see if it works with the patch in the distgit | 20:26 |
| weshay|ruck | k | 20:26 |
| *** tcw has quit IRC | 20:30 | |
| weshay|ruck | EmilienM, this is all we need to test https://review.openstack.org/#/c/575843/ | 20:31 |
| fultonj | does anyone know anything about /opt/stack/new/tripleo-ci/test-environments/worker-config.yaml in rdo CI? | 20:31 |
| fultonj | https://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh | 20:32 |
| fultonj | ^ context | 20:32 |
| fultonj | it's being based in oooq's featureset024 but i can't see it so i don't know what it does | 20:33 |
| fultonj | s/based/passed | 20:33 |
| *** tcw has joined #tripleo | 20:33 | |
| fultonj | this is an OVB job | 20:33 |
| EmilienM | weshay|ruck: oh wait | 20:34 |
| EmilienM | it's pointless | 20:34 |
| weshay|ruck | fultonj, am I blind, don't see it in fs24 | 20:34 |
| EmilienM | ah nevermind | 20:34 |
| EmilienM | weshay|ruck: nevermind what I said. Thanks it's good | 20:35 |
| fultonj | weshay|ruck: https://review.openstack.org/#/c/570469/ | 20:35 |
| fultonj | in rdo ci | 20:35 |
| weshay|ruck | fultonj, worker-config.yaml is not in that dep | 20:36 |
| weshay|ruck | is that the issue? | 20:36 |
| weshay|ruck | why is it getting passed | 20:37 |
| fultonj | weshay|ruck: i am testing a change to fs24 by updating tht | 20:37 |
| fultonj | https://review.openstack.org/#/c/570473/ | 20:37 |
| weshay|ruck | toci_gate_test-orig.sh:374: OVERCLOUD_DEPLOY_ARGS="$OVERCLOUD_DEPLOY_ARGS -e $TRIPLEO_ROOT/tripleo-ci/test-environments/worker-config.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml" | 20:37 |
| fultonj | weshay|ruck: so that source caused this overcloud-deploy command https://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh | 20:38 |
| weshay|ruck | ya.. I see it there | 20:39 |
| fultonj | weshay|ruck: is worker-config.yaml somewhere i can read it so i can see what it might be doing? | 20:39 |
| weshay|ruck | afaict, it should not be | 20:39 |
| weshay|ruck | https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/worker-config.yaml | 20:40 |
| fultonj | ahhh, it's in infra | 20:40 |
| fultonj | thanks weshay|ruck | 20:40 |
| mwhahaha | weshay|ruck: did we ever fix tripleo-docs so it runs stuff? | 20:40 |
| weshay|ruck | ah here it is http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci-quickstart/config/testenv/multinode.yml#n7 | 20:40 |
| weshay|ruck | heh.. | 20:41 |
| weshay|ruck | it's been in there for a year | 20:41 |
| weshay|ruck | https://github.com/openstack-infra/tripleo-ci/commit/ab7914d28de5cc79269d3839de10f6017e4c8c13 | 20:42 |
| weshay|ruck | pile on ci day | 20:42 |
| weshay|ruck | pi day | 20:42 |
| weshay|ruck | or just a day | 20:42 |
| mwhahaha | aka day ending in y | 20:42 |
| mwhahaha | now get back to work | 20:42 |
| fultonj | i'm trying to make fs24 use ceph-ansible but the ceph-storage nodes don't have docker running. | 20:45 |
| weshay|ruck | mwhahaha, so looking, it's missing zuul config entirely | 20:45 |
| weshay|ruck | afaict | 20:45 |
| openstackgerrit | Merged openstack/tripleo-common master: scen007 is failing in the gate https://review.openstack.org/575796 | 20:46 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Add cors class to gnocchi profile https://review.openstack.org/575845 | 20:46 |
| mwhahaha | weshay|ruck: yea, i'm poking at it as well | 20:46 |
| fultonj | i'm looking for evidence that there's a custom role | 20:46 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Include cors to gnocchi profile https://review.openstack.org/575845 | 20:47 |
| weshay|ruck | mwhahaha, docs for openstack/tripleo-quickstart is working w/o any specfic config | 20:47 |
| mwhahaha | weshay|ruck: it's likely that project-config is bad | 20:48 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: DNM test doc https://review.openstack.org/575846 | 20:48 |
| *** lifeless has quit IRC | 20:49 | |
| mwhahaha | weshay|ruck: i think it's the files stuff in project-config | 20:49 |
| mwhahaha | weshay|ruck: we're limiting stuff when i don't think we should be | 20:49 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-docs master: DNM testing doc https://review.openstack.org/575848 | 20:49 |
| *** lifeless has joined #tripleo | 20:50 | |
| mwhahaha | weshay|ruck: i think it's https://review.openstack.org/575849 | 20:51 |
| weshay|ruck | mwhahaha, I blame u | 20:52 |
| mwhahaha | pffft it was like that when i got here | 20:52 |
| * mwhahaha blames everyone else | 20:52 | |
| *** mhenkel_ has joined #tripleo | 20:52 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade stable/queens: Fix docker images validation script https://review.openstack.org/575461 | 20:52 |
| weshay|ruck | that's what we need is a docker validation script | 20:53 |
| mwhahaha | so salty | 20:54 |
| *** rmascena has quit IRC | 20:54 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-docs master: DNM testing doc https://review.openstack.org/575848 | 20:56 |
| fultonj | weshay|ruck: another way to ask my question is, why do I need to start docker manually with a preboot script to run ceph containers when they should already be included in the role? https://review.openstack.org/#/c/570473/2/ci/environments/dev_block_devices.sh | 20:57 |
| weshay|ruck | mwhahaha, job kicked https://review.openstack.org/#/c/575848/2 | 20:57 |
| weshay|ruck | hrm.. then died | 20:58 |
| mwhahaha | yea i think it's the files | 20:58 |
| mwhahaha | so unless you touch one of the ones that was listed, no jobs | 20:58 |
| weshay|ruck | ya | 20:58 |
| weshay|ruck | I have a dep on your change though | 20:58 |
| EmilienM | nice, an opportunity to blame mwhahaha for once | 20:58 |
| EmilienM | he always blames me :( | 20:59 |
| mwhahaha | fultonj: the docker bits should be covered by a the docker service on the role | 20:59 |
| weshay|ruck | fultonj, hrm.. you should not have to start docker like that | 20:59 |
| weshay|ruck | for shiz | 20:59 |
| fultonj | mwhahaha: exactly! | 20:59 |
| mwhahaha | fultonj: can you point me to what you're looking at? | 20:59 |
| fultonj | mwhahaha: so i thought maybe the service was removed from the ceph storage role | 21:00 |
| fultonj | https://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh | 21:00 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Switch scenario00{1,2,3,4,7} to containerized undercloud"" https://review.openstack.org/575330 | 21:00 |
| fultonj | ^ so i looked at how the deployment was done to see if there's "-e no-docker-for-ceph.yaml" or something like it | 21:01 |
| fultonj | two reviews | 21:01 |
| mwhahaha | fultonj: you know that's for pike right? | 21:01 |
| fultonj | https://review.openstack.org/#/q/topic:bug/1772034+(status:open+OR+status:merged) | 21:01 |
| fultonj | mwhahaha: yes | 21:01 |
| mwhahaha | so you're trying to switch it in pike? | 21:01 |
| fultonj | i want to change the pike job to ceph-ansible | 21:02 |
| mwhahaha | we didn't support ceph-ansible in pike did we? | 21:02 |
| fultonj | we did | 21:02 |
| fultonj | so that i can test an upgrade of ceph w/ 3 mons to queens | 21:02 |
| fultonj | fs24 is one of the few jobs w/ 3 controllers | 21:02 |
| mwhahaha | in ovb we use the regular roles | 21:02 |
| fultonj | yeah, AFAICT it's using regular roles | 21:03 |
| fultonj | and no role changing things done via a -e | 21:03 |
| fultonj | ceph-ansible was introduced in pike | 21:03 |
| fultonj | fs24, the rdo job w/ 3 mons to test upgrades from pike to queens still uses puppet-ceph | 21:04 |
| mwhahaha | fultonj: so docker should be managed via https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/roles_data.yaml#L288 but it might be an ordering thing | 21:04 |
| fultonj | so i'm trying to switch it | 21:04 |
| fultonj | interesting... | 21:04 |
| mwhahaha | though docker should be up in step1 | 21:04 |
| mwhahaha | my thought would be that the external workflow gets kicked off before the node gets anything run but that seems od | 21:05 |
| weshay|ruck | EmilienM, so the most recent container runs in prep look good | 21:05 |
| fultonj | the external workflow waits until step2 | 21:05 |
| weshay|ruck | updating a few rpms.. working etc | 21:05 |
| *** gkadam_ has quit IRC | 21:05 | |
| mwhahaha | fultonj: so why doesn't the deploy fail? | 21:07 |
| mwhahaha | fultonj: it seems to just fail on the validation | 21:07 |
| fultonj | you mean at step1 if it fails to bring up docker? | 21:07 |
| mwhahaha | fultonj: no i mean ceph ansible fails with docker? | 21:07 |
| mwhahaha | fultonj: shouldn't the deployment fail if that was the case? | 21:07 |
| mwhahaha | but the deployment is successful | 21:07 |
| fultonj | the deply used to fail until i added a hack to start it manually | 21:08 |
| fultonj | then evertyting is fine | 21:08 |
| mwhahaha | plz don't add a hack, make it hard to debug :D | 21:08 |
| fultonj | mwhahaha: i only added a hack to prove my point that the root cause is docker not running | 21:08 |
| fultonj | the hack will go | 21:08 |
| fultonj | line 22 | 21:08 |
| fultonj | https://review.openstack.org/#/c/570473/2/ci/environments/dev_block_devices.sh | 21:08 |
| fultonj | easy to see it fail | 21:09 |
| fultonj | i can remove line 22 and we can see it fail the old way | 21:09 |
| fultonj | i was just proving it was tripleo not starting docker, as opposed to ceph-ansible having some other issue | 21:10 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777168 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 21:10 |
| *** brault has quit IRC | 21:10 | |
| mwhahaha | fultonj: so do you have link to the logs for a failure w/o the hack? | 21:10 |
| fultonj | mwhahaha: i think results of ps1 (without the hack) would be gone after ps2 was tested | 21:11 |
| mwhahaha | because the docker was started on it's own https://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/overcloud-cephstorage-0/var/log/journal.txt.gz#_May_30_22_12_41 | 21:11 |
| *** pradk has quit IRC | 21:11 | |
| fultonj | i will make ps3 without the hack so we can get some logs | 21:11 |
| fultonj | mwhahaha: sound like ok next step? | 21:11 |
| mwhahaha | let me look at teh ps1 logs | 21:11 |
| fultonj | ahhh | 21:11 |
| fultonj | https://logs.rdoproject.org/73/570473/1/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zb2bfcd4b6e9d49d9aa733634eeb1c9cd/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz | 21:12 |
| fultonj | we do have ps1 logs | 21:12 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Fix DNS firewall rules https://review.openstack.org/575146 | 21:12 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Make BIND listen address configurable https://review.openstack.org/575822 | 21:12 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Make BIND /var dir persistent https://review.openstack.org/575823 | 21:12 |
| mwhahaha | fultonj: can't connect doesn't mean it's not running | 21:12 |
| mwhahaha | fultonj: could be a docker permissions thing | 21:12 |
| fultonj | but starting it as root made it run... | 21:13 |
| * fultonj clicking around https://logs.rdoproject.org/73/570473/1/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zb2bfcd4b6e9d49d9aa733634eeb1c9cd/overcloud-cephstorage-0/var/log/extra/?C=N;O=A | 21:14 | |
| *** agopi has quit IRC | 21:14 | |
| *** agopi has joined #tripleo | 21:15 | |
| mwhahaha | fultonj: so this is ovb, docker should already be installed | 21:15 |
| *** rfolco has quit IRC | 21:15 | |
| fultonj | confirmed it's isntalled | 21:15 |
| fultonj | e.g. in rpm-list.txt.gz | 21:15 |
| *** yamahata has joined #tripleo | 21:16 | |
| mwhahaha | fultonj: so step1 ran | 21:16 |
| fultonj | you'd think i would see it in pstree if it were running | 21:16 |
| mwhahaha | https://logs.rdoproject.org/73/570473/1/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zb2bfcd4b6e9d49d9aa733634eeb1c9cd/overcloud-cephstorage-0/var/log/journal.txt.gz#_May_24_23_57_17 | 21:17 |
| mwhahaha | that's the start of puppet step1 | 21:17 |
| mwhahaha | let me see where that is supposed to start | 21:17 |
| *** slaweq has quit IRC | 21:18 | |
| *** slaweq has joined #tripleo | 21:19 | |
| mwhahaha | fultonj: so yea i don't see a reference to Service[docker] anywhere | 21:19 |
| mwhahaha | that seems odd | 21:19 |
| *** holser___ has quit IRC | 21:21 | |
| *** jtomasek_ has quit IRC | 21:21 | |
| * fultonj nods | 21:21 | |
| mwhahaha | though i don't see where it gets started on the controller either | 21:23 |
| *** slaweq has quit IRC | 21:24 | |
| fultonj | that's a good point | 21:24 |
| fultonj | one thing odd about this job is that it's doing a '-e puppet-pacemaker.yaml' | 21:24 |
| *** akrivoka has quit IRC | 21:24 | |
| fultonj | seems like a blast from the past that shouldn't be in pike, unless I am misremembering | 21:24 |
| mwhahaha | thats normal | 21:24 |
| fultonj | is it? | 21:24 |
| mwhahaha | yea it's an ovb job | 21:25 |
| mwhahaha | we ran ha on those | 21:25 |
| fultonj | shouldn't this job have docker-ha too? | 21:25 |
| mwhahaha | fultonj: uh, i don't think job is containerized | 21:26 |
| fultonj | right! | 21:26 |
| fultonj | https://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh | 21:26 |
| fultonj | no -e docker.yaml! | 21:26 |
| fultonj | omg | 21:26 |
| fultonj | dahhhhhhhhhh | 21:27 |
| fultonj | thank you mwhahaha | 21:27 |
| fultonj | now i know what to do | 21:27 |
| fultonj | containerize it as i was going to | 21:27 |
| mwhahaha | also -e docker-ha.yaml | 21:27 |
| fultonj | exactly | 21:27 |
| mwhahaha | HEH | 21:27 |
| fultonj | we want our pike CI to be containerized right... | 21:28 |
| fultonj | this job | 21:28 |
| mwhahaha | some of it, yea | 21:28 |
| mwhahaha | because we were not completely containerized upstream | 21:28 |
| mwhahaha | so only some of the jobs were switched over | 21:28 |
| mwhahaha | fultonj: so we usualy don't cut over a fs to containers i thought we created new ones | 21:29 |
| mwhahaha | fultonj: i'd have to defer back to EmilienM as to wtf we were doing around that | 21:29 |
| *** edmondsw has quit IRC | 21:29 | |
| fultonj | hmmmm | 21:29 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: scen007 is failing in the gate https://review.openstack.org/575799 | 21:30 |
| *** ipsecguy has quit IRC | 21:33 | |
| *** ipsecguy has joined #tripleo | 21:34 | |
| fultonj | mwhahaha: thanks for helping me figure this out. i will continue on this and get input from EmilienM before it merges by adding him to the review | 21:34 |
| * fultonj stepping out for now | 21:34 | |
| EmilienM | weshay|ruck: is your dashboard up again? | 21:44 |
| EmilienM | I want to check the failures | 21:44 |
| weshay|ruck | yes | 21:44 |
| EmilienM | fultonj: sorry what's the patch? I can look now | 21:45 |
| EmilienM | http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 | 21:45 |
| weshay|ruck | EmilienM, everything merged.. I thought the same thing | 21:45 |
| EmilienM | I have it | 21:45 |
| EmilienM | weshay|ruck: ^ | 21:45 |
| weshay|ruck | still at 13 fails | 21:45 |
| weshay|ruck | it didn't send all the merge messages to this channel | 21:46 |
| EmilienM | weshay|ruck: 13 fails? | 21:46 |
| EmilienM | weshay|ruck: we need to count from when infra pushed the new proxy | 21:46 |
| weshay|ruck | EmilienM, 0 | 21:47 |
| EmilienM | 0 is a good number so far :D | 21:48 |
| EmilienM | I think there is a new problem coming up | 21:51 |
| EmilienM | http://logs.openstack.org/27/572027/3/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/10d0022/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-06-15_19_43_19 | 21:51 |
| EmilienM | I can't believe it wasn't catch by https://review.openstack.org/#/c/571500/ | 21:52 |
| EmilienM | it's hitting the gate as well | 21:52 |
| EmilienM | I'm about to propose a revert and a fix | 21:52 |
| *** dbecker has quit IRC | 21:53 | |
| *** mcornea has quit IRC | 21:53 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Revert "fs050: enable upgrade cleanup""" https://review.openstack.org/575855 | 21:54 |
| weshay|ruck | when did their patch merge ? | 21:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Revert "fs050: enable upgrade cleanup""" https://review.openstack.org/575855 | 21:54 |
| weshay|ruck | and get pushed in | 21:54 |
| EmilienM | weshay|ruck: which one | 21:54 |
| weshay|ruck | 2hr ago? | 21:54 |
| EmilienM | weshay|ruck: oh infra? | 21:54 |
| weshay|ruck | ya | 21:54 |
| EmilienM | like 2.5h ago yea | 21:54 |
| EmilienM | weshay|ruck, mwhahaha ^ can I get quick +A on my revert | 21:55 |
| EmilienM | weshay|ruck: when you +A I'll ask infra to move it to gate directly | 21:56 |
| EmilienM | I see a lot of failures in check with that job | 21:56 |
| *** dmacpher has quit IRC | 21:57 | |
| EmilienM | I'll go ahead and approve it | 21:57 |
| weshay|ruck | EmilienM, this one? https://review.openstack.org/#/c/575330/ | 21:58 |
| EmilienM | weshay|ruck: it's fine, it was https://review.openstack.org/575855 | 21:59 |
| EmilienM | you got it | 21:59 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-docs master: Document single nic configuration for standalone https://review.openstack.org/575859 | 22:00 |
| mwhahaha | EmilienM: -^ | 22:00 |
| *** agopi has quit IRC | 22:00 | |
| mwhahaha | some day we'll get the tripleo-docs jobs back and we can merge that | 22:00 |
| EmilienM | nice | 22:00 |
| *** lifeless has quit IRC | 22:01 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-docs master: Document single nic configuration for standalone https://review.openstack.org/575859 | 22:02 |
| *** owalsh has quit IRC | 22:07 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Update standalone role for a cloud https://review.openstack.org/575862 | 22:08 |
| *** owalsh has joined #tripleo | 22:09 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777168 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 22:10 |
| mwhahaha | anyone remember the openstack command to attach a volume to a server :D | 22:10 |
| * mwhahaha can never remember these things | 22:10 | |
| mwhahaha | or is this one of those things that you still have to use the nova command for | 22:11 |
| mwhahaha | nm openstack serfver add volume | 22:11 |
| *** slaweq has joined #tripleo | 22:11 | |
| EmilienM | oh I think I found the cont/uc upgrade issue | 22:11 |
| EmilienM | mwhahaha: use horizon :P | 22:11 |
| mwhahaha | PFFFFT | 22:12 |
| mwhahaha | (but it does work, i just didn't want to setup sshuttle) | 22:12 |
| EmilienM | mwhahaha: could you spawn and ping a vm using horizon at the end? | 22:12 |
| mwhahaha | yea | 22:12 |
| EmilienM | niceeeee | 22:12 |
| mwhahaha | with that doc update and that role patch -^ it's an openstack cloud in a box in ~35mins | 22:13 |
| mwhahaha | nova/neutron/cinder/glance/swift | 22:13 |
| mwhahaha | all working | 22:13 |
| mwhahaha | no ceph tho | 22:13 |
| * mwhahaha shuns ceph | 22:13 | |
| EmilienM | pretty awesome | 22:13 |
| mwhahaha | SHUUUUUUUNNNNN | 22:13 |
| EmilienM | we can pretty much switch scenarios on that once we have tempest working | 22:13 |
| * EmilienM hides | 22:13 | |
| mwhahaha | config file first | 22:14 |
| EmilienM | the immediate benefit is runtime in CI | 22:14 |
| mwhahaha | we need to ditch that doc bit | 22:14 |
| mwhahaha | it's ugly | 22:14 |
| *** rlandy_ is now known as rlandy | 22:15 | |
| mwhahaha | cinder worked | 22:16 |
| mwhahaha | wooo | 22:16 |
| mwhahaha | i think i'm going to give up for teh week | 22:16 |
| EmilienM | wait | 22:18 |
| mwhahaha | i will not wait | 22:18 |
| EmilienM | don't let me here | 22:18 |
| mwhahaha | IT'S MY LIFE WOOOO | 22:18 |
| mwhahaha | well i'm going to send an email first | 22:18 |
| EmilienM | ok go now | 22:18 |
| EmilienM | that's too much | 22:18 |
| EmilienM | yeah send an email | 22:18 |
| EmilienM | people can start using it imho | 22:18 |
| EmilienM | i've got the skydive folks poking at it today as well | 22:19 |
| *** owalsh has quit IRC | 22:19 | |
| *** slaweq has quit IRC | 22:19 | |
| *** owalsh has joined #tripleo | 22:19 | |
| EmilienM | oh I know why the upgrade job is failing | 22:20 |
| EmilienM | it's because we aren't consistent | 22:20 |
| mwhahaha | :( | 22:20 |
| EmilienM | therefore we need https://github.com/openstack/tripleo-heat-templates/commit/029f500feedd01c12a2e9b24d4bab8f2a6c8beed | 22:20 |
| *** cdearborn has quit IRC | 22:20 | |
| EmilienM | let's merge the revert anyway | 22:21 |
| EmilienM | and let's see why we aren't consistent | 22:21 |
| EmilienM | https://git.openstack.org/cgit/openstack/python-cloudkittyclient/commit/?id=d070f6a68cddf51c57e77107f1b823a8f75770ba | 22:22 |
| EmilienM | lol | 22:22 |
| EmilienM | that kind of patch makes me sad | 22:22 |
| EmilienM | 113 files changed, 4231 insertions, 8399 deletions | 22:22 |
| EmilienM | people need to learn what iteration means :D | 22:23 |
| mwhahaha | iterwhat? | 22:23 |
| EmilienM | amoralej|off, jpena|off here by any chance? | 22:24 |
| EmilienM | I'll take a stab otherwise | 22:24 |
| *** yamahata has quit IRC | 22:24 | |
| *** mhenkel_ has quit IRC | 22:29 | |
| mwhahaha | EmilienM: did you restore all the rest of the stuff or do we need your upgrade revert first | 22:38 |
| EmilienM | mwhahaha: I'll take care of it | 22:39 |
| mwhahaha | k | 22:39 |
| EmilienM | yeah we need my revert | 22:39 |
| EmilienM | sorry for that | 22:39 |
| *** matbu has joined #tripleo | 22:39 | |
| EmilienM | coz we aren't consistent | 22:39 |
| EmilienM | I'm working on it | 22:39 |
| EmilienM | (cloudkittyclient) | 22:39 |
| *** mhenkel_ has joined #tripleo | 22:41 | |
| *** mhenkel_ has quit IRC | 22:45 | |
| *** lifeless has joined #tripleo | 22:49 | |
| *** Goneri has quit IRC | 22:50 | |
| *** edmondsw has joined #tripleo | 22:50 | |
| *** edmondsw has quit IRC | 22:55 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777168 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged] | 23:10 |
| *** slaweq has joined #tripleo | 23:11 | |
| *** hamzy_ has quit IRC | 23:15 | |
| *** slaweq has quit IRC | 23:16 | |
| *** owalsh_ has joined #tripleo | 23:16 | |
| *** owalsh has quit IRC | 23:17 | |
| *** owalsh has joined #tripleo | 23:17 | |
| *** owalsh has quit IRC | 23:18 | |
| *** owalsh_ is now known as owalsh | 23:19 | |
| *** hamzy has joined #tripleo | 23:25 | |
| *** rpioso is now known as rpioso|afk | 23:36 | |
| *** ipsecguy has quit IRC | 23:39 | |
| *** tosky has quit IRC | 23:41 | |
| *** tahini has quit IRC | 23:54 | |
| *** tahini has joined #tripleo | 23:57 | |
| *** agopi has joined #tripleo | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!