| *** jascott1 has quit IRC | 00:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Ignore all dot-files for translation updates https://review.openstack.org/521395 | 00:01 |
|---|---|---|
| ianw | ok, try again... | 00:02 |
| *** stakeda has joined #openstack-infra | 00:06 | |
| *** dingyichen has joined #openstack-infra | 00:08 | |
| *** salv-orlando has quit IRC | 00:14 | |
| ianw | http://logs.openstack.org/57/57bf7f49cf08dbc0c0d16eb7512ecaa15978fbff/post/upstream-translation-update/9121818/job-output.txt.gz#_2017-11-20_00_10_58_698913 | 00:15 |
| ianw | 2017-11-20 00:10:58.698913 | ubuntu-xenial | [INFO] Found source documents: | 00:15 |
| ianw | 2017-11-20 00:10:58.699026 | ubuntu-xenial | [INFO] nova/locale/nova | 00:15 |
| ianw | 2017-11-20 00:10:58.699064 | ubuntu-xenial | [INFO] releasenotes/source/locale/releasenotes | 00:15 |
| ianw | 2017-11-20 00:10:58.940336 | ubuntu-xenial | [INFO] pushing source doc [name=nova/locale/nova size=1381] to server | 00:15 |
| ianw | 2017-11-20 00:11:04.554502 | ubuntu-xenial | [INFO] pushing source doc [name=releasenotes/source/locale/releasenotes size=1111] to server | 00:15 |
| ianw | i'm going to say that's working then | 00:15 |
| *** Apoorva has joined #openstack-infra | 00:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Stop supporting releasenotes/requirements.txt https://review.openstack.org/520834 | 00:31 |
| *** hongbin has quit IRC | 00:31 | |
| ianw | ok, AJaeger's point on on 520834 seems good that it gets out of sync with the tc testing docs. i've proposed an update there and added depends-on, but happy to be overridden | 00:33 |
| ianw | i'm trying to decide what job is failing with 521119 | 00:35 |
| ianw | otherwise repos using oslosphinx fail ... ok | 00:37 |
| *** claudiub|2 has quit IRC | 00:38 | |
| *** Apoorva has quit IRC | 00:45 | |
| *** Apoorva has joined #openstack-infra | 00:46 | |
| *** Apoorva_ has joined #openstack-infra | 00:49 | |
| *** Apoorva has quit IRC | 00:52 | |
| ianw | ok, so something like http://logs.openstack.org/99/521399/1/check/build-openstack-releasenotes/5fba427/ fails without 521119 because no oslosphinx | 00:53 |
| ianw | AJager has a bunch of changes out to fix this and move things not openstackdocstheme, but we need to unbreak | 00:53 |
| *** dsariel has quit IRC | 01:01 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Try to install more dependencies in the release notes job https://review.openstack.org/521119 | 01:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Stop supporting releasenotes/requirements.txt https://review.openstack.org/520834 | 01:07 |
| ianw | ok, so i swapped these around. we can do dhellmann's patch and fix the broken jobs, and then discuss mordred's removal of relasenotes/requirements.txt install separately in the context of updating docs | 01:08 |
| ianw | cpu usage is right up on review.o.o and i'm seeing 502 errors ... will monitor for a bit | 01:13 |
| adriant | yeah review.org is very slow right now :( | 01:14 |
| *** Anticimex has quit IRC | 01:14 | |
| adriant | and back it seems | 01:14 |
| *** salv-orlando has joined #openstack-infra | 01:15 | |
| adriant | nah, it's still spotty | 01:15 |
| *** Anticimex has joined #openstack-infra | 01:15 | |
| ianw | yeah, it will be | 01:16 |
| ianw | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=26&rra_id=all | 01:16 |
| ianw | it's definitely load average and a spike in i/o ... | 01:16 |
| * ianw logs in ... crosses fingers for no weird i/o errors :/ | 01:17 | |
| *** larainema has joined #openstack-infra | 01:17 | |
| ianw | alright, nothing bad looking from attached storage etc | 01:19 |
| ianw | i'm going to restart it | 01:19 |
| *** Apoorva_ has quit IRC | 01:24 | |
| ianw | #status log gerrit restarted; review.o.o 42,571 Mb / 48,551 Mb and persistent system load ~10, definite i/o spike o /dev/xvb but nothing usual to the naked eye | 01:25 |
| openstackstatus | ianw: finished logging | 01:25 |
| *** dhinesh has joined #openstack-infra | 01:27 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Try to install more dependencies in the release notes job https://review.openstack.org/521119 | 01:27 |
| *** cuongnv has joined #openstack-infra | 01:32 | |
| *** kiennt26 has joined #openstack-infra | 01:33 | |
| *** Daisy_ has joined #openstack-infra | 01:33 | |
| *** Daisy_ has quit IRC | 01:33 | |
| *** hongbin has joined #openstack-infra | 01:34 | |
| *** Wei_Liu has joined #openstack-infra | 01:35 | |
| *** dhinesh has quit IRC | 01:40 | |
| *** dhinesh has joined #openstack-infra | 01:41 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Stop supporting releasenotes/requirements.txt https://review.openstack.org/520834 | 01:41 |
| ianw | AJaeger: ok, so I believe we have working translation jobs and working releasenotes+requirements jobs. we can discuss the removal in ^ which depends on the doc change now | 01:42 |
| *** liujiong has joined #openstack-infra | 01:43 | |
| ianw | the stack on top of that i've left alone. that seems like something to work on and test more | 01:43 |
| openstackgerrit | Rui Chen proposed openstack-infra/nodepool feature/zuulv3: Fix nodepool cmd TypeError when no arguemnts https://review.openstack.org/519582 | 01:44 |
| *** Goneri has quit IRC | 01:45 | |
| *** lbragstad has joined #openstack-infra | 01:45 | |
| *** liujiong has quit IRC | 01:48 | |
| *** salv-orlando has quit IRC | 01:49 | |
| *** liujiong has joined #openstack-infra | 01:49 | |
| *** gcb has joined #openstack-infra | 01:52 | |
| *** armax has joined #openstack-infra | 02:07 | |
| *** gongysh has joined #openstack-infra | 02:16 | |
| *** daidv has joined #openstack-infra | 02:21 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add /var/lib/mistral for log collection https://review.openstack.org/518102 | 02:28 |
| *** armax has quit IRC | 02:42 | |
| *** salv-orlando has joined #openstack-infra | 02:46 | |
| *** ramishra has joined #openstack-infra | 02:53 | |
| tbarron | ianw: ok if we workflow +1 https://review.openstack.org/#/c/520880/ now? | 02:59 |
| *** docaedo has left #openstack-infra | 03:00 | |
| *** annp has joined #openstack-infra | 03:01 | |
| *** bobh has joined #openstack-infra | 03:01 | |
| ianw | tbarron: yep, that's the cleanup patch so all good | 03:07 |
| ianw | once all of those are in, we can remove some bits of the infra jobs | 03:08 |
| ianw | oh yeah, sorry that was one i rebased this morning just as testing | 03:08 |
| *** liusheng has joined #openstack-infra | 03:17 | |
| *** udesale has joined #openstack-infra | 03:23 | |
| *** sree has joined #openstack-infra | 03:33 | |
| *** dave-mccowan has quit IRC | 03:39 | |
| *** ykarel_ has joined #openstack-infra | 03:45 | |
| *** bobh has quit IRC | 03:46 | |
| *** armax has joined #openstack-infra | 03:48 | |
| *** rcernin has quit IRC | 03:50 | |
| *** rcernin has joined #openstack-infra | 03:52 | |
| *** bobh has joined #openstack-infra | 03:52 | |
| *** hongbin has quit IRC | 03:57 | |
| *** psachin has joined #openstack-infra | 04:03 | |
| *** aeng has quit IRC | 04:03 | |
| *** ramishra has quit IRC | 04:08 | |
| *** bobh has quit IRC | 04:09 | |
| *** ramishra has joined #openstack-infra | 04:10 | |
| *** links has joined #openstack-infra | 04:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install fedora-gpg-keys for F27 https://review.openstack.org/521414 | 04:18 |
| *** gildub has quit IRC | 04:18 | |
| *** yamamoto has joined #openstack-infra | 04:21 | |
| *** bobh has joined #openstack-infra | 04:22 | |
| *** coolsvap has joined #openstack-infra | 04:22 | |
| *** bobh has quit IRC | 04:30 | |
| *** bobh has joined #openstack-infra | 04:32 | |
| *** gmann is now known as gmann_lunch | 04:35 | |
| *** bobh has quit IRC | 04:37 | |
| *** bobh has joined #openstack-infra | 04:44 | |
| *** bobh has quit IRC | 04:49 | |
| *** bobh has joined #openstack-infra | 04:51 | |
| *** bobh has quit IRC | 04:55 | |
| *** owalsh_ has joined #openstack-infra | 04:57 | |
| *** eroux has quit IRC | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Pre-install curl https://review.openstack.org/521415 | 04:58 |
| *** owalsh has quit IRC | 05:01 | |
| *** aeng has joined #openstack-infra | 05:02 | |
| *** bobh has joined #openstack-infra | 05:02 | |
| *** janki has joined #openstack-infra | 05:03 | |
| *** bobh has quit IRC | 05:07 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Pre-install curl https://review.openstack.org/521415 | 05:11 |
| *** pgadiya has joined #openstack-infra | 05:13 | |
| *** pgadiya has quit IRC | 05:13 | |
| *** bobh has joined #openstack-infra | 05:13 | |
| *** gongysh has quit IRC | 05:15 | |
| *** bobh has quit IRC | 05:18 | |
| *** zerick has joined #openstack-infra | 05:23 | |
| *** jascott1 has joined #openstack-infra | 05:29 | |
| *** d0ugal_ has joined #openstack-infra | 05:30 | |
| *** bobh has joined #openstack-infra | 05:31 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/520394 | 05:31 |
| *** dsariel has joined #openstack-infra | 05:32 | |
| *** d0ugal has quit IRC | 05:32 | |
| *** jascott1 has quit IRC | 05:33 | |
| *** eharney has quit IRC | 05:35 | |
| *** bobh has quit IRC | 05:36 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Dont install python-pip for py3k https://review.openstack.org/519502 | 05:38 |
| *** rcernin has quit IRC | 05:38 | |
| *** bobh has joined #openstack-infra | 05:43 | |
| *** ramishra has quit IRC | 05:46 | |
| *** bobh has quit IRC | 05:48 | |
| *** ramishra has joined #openstack-infra | 05:49 | |
| *** bobh has joined #openstack-infra | 05:56 | |
| *** ramishra has quit IRC | 05:57 | |
| AJaeger | ianw: thanks for fixing the releasenotes and translations! | 05:57 |
| *** ramishra has joined #openstack-infra | 05:59 | |
| *** ykarel_ is now known as ykarel | 05:59 | |
| *** bobh has quit IRC | 06:01 | |
| *** gongysh has joined #openstack-infra | 06:02 | |
| AJaeger | regarding the releasenotes/test-translations: I just added the change to support this last week. So, yes we should remove it from PTI and mordred planned - or already submitted - a change for that. Since we only added it last week, I would not block the removal change on the PTI removal. We can discuss that later... | 06:03 |
| *** threestrands has quit IRC | 06:03 | |
| *** bobh has joined #openstack-infra | 06:06 | |
| *** bobh has quit IRC | 06:11 | |
| AJaeger | so, rechecked some releasenotes jobs - all look fine now from the infra side | 06:18 |
| *** eharney has joined #openstack-infra | 06:19 | |
| *** gouthamr has quit IRC | 06:20 | |
| *** gildub has joined #openstack-infra | 06:21 | |
| *** janki is now known as janki|afk | 06:22 | |
| *** bobh has joined #openstack-infra | 06:23 | |
| *** bobh has quit IRC | 06:28 | |
| *** aeng has quit IRC | 06:30 | |
| *** jtomasek has joined #openstack-infra | 06:33 | |
| *** jbadiapa has joined #openstack-infra | 06:35 | |
| ianw | AJaeger: ok, cool, i didn't see a change out for that so that was why i created https://review.openstack.org/521398 | 06:37 |
| *** salv-orlando has quit IRC | 06:37 | |
| *** salv-orlando has joined #openstack-infra | 06:38 | |
| AJaeger | pabelanger: http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-centos-7/ef3a987/ has retry_limit | 06:40 |
| AJaeger | ianw: if you check, then mordred might not have done it yet... He had quite a few changes in flight... | 06:41 |
| *** bobh has joined #openstack-infra | 06:41 | |
| AJaeger | yeah, periodic translation jobs working again! | 06:42 |
| ianw | Ajaeger: yes, i noticed :) | 06:42 |
| *** bobh has quit IRC | 06:46 | |
| *** bhavik has joined #openstack-infra | 06:47 | |
| *** yamamoto has quit IRC | 06:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Ignore missing .tox/env/logs directorys for copy https://review.openstack.org/521436 | 06:52 |
| *** e0ne has joined #openstack-infra | 06:55 | |
| *** gouthamr has joined #openstack-infra | 06:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Ignore missing .tox/env/logs directories for copy https://review.openstack.org/521436 | 06:55 |
| *** yamamoto has joined #openstack-infra | 06:56 | |
| *** dhinesh has quit IRC | 07:00 | |
| *** bobh has joined #openstack-infra | 07:01 | |
| *** bobh has quit IRC | 07:06 | |
| *** xinliang has quit IRC | 07:06 | |
| ianw | pabelanger / anyone : f27 test-build host up 38.145.34.178, have a poke and see if anything obviously wrong | 07:06 |
| *** cuongnv has quit IRC | 07:08 | |
| *** Hal has quit IRC | 07:08 | |
| *** gongysh has quit IRC | 07:11 | |
| *** cuongnv has joined #openstack-infra | 07:16 | |
| *** gongysh has joined #openstack-infra | 07:16 | |
| *** jtomasek has quit IRC | 07:16 | |
| *** xinliang has joined #openstack-infra | 07:18 | |
| *** florianf has joined #openstack-infra | 07:22 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: Add tempest-horizon to #openstack-horizon https://review.openstack.org/521442 | 07:23 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: Add openstack-tox-pep8 job to tempest-horizon https://review.openstack.org/521443 | 07:25 |
| *** gcb has quit IRC | 07:25 | |
| *** gcb has joined #openstack-infra | 07:26 | |
| *** andreas_s has joined #openstack-infra | 07:27 | |
| *** d0ugal_ has quit IRC | 07:31 | |
| *** jtomasek has joined #openstack-infra | 07:34 | |
| *** janki|afk is now known as janki | 07:35 | |
| *** nsingh has quit IRC | 07:35 | |
| *** nsingh has joined #openstack-infra | 07:35 | |
| *** bobh has joined #openstack-infra | 07:37 | |
| *** bobh has quit IRC | 07:42 | |
| *** bobh has joined #openstack-infra | 07:45 | |
| *** shardy has joined #openstack-infra | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:47 | |
| *** e0ne has quit IRC | 07:48 | |
| *** bobh has quit IRC | 07:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove obsolete murano-firstapp docs job https://review.openstack.org/521127 | 07:51 |
| *** AJaeger has quit IRC | 07:52 | |
| *** bobh has joined #openstack-infra | 07:54 | |
| *** AJaeger has joined #openstack-infra | 07:56 | |
| *** eumel8 has joined #openstack-infra | 07:58 | |
| *** bobh has quit IRC | 07:59 | |
| eumel8 | morning | 07:59 |
| *** dhinesh has joined #openstack-infra | 08:01 | |
| *** pcaruana has joined #openstack-infra | 08:04 | |
| *** niraj_singh has joined #openstack-infra | 08:05 | |
| *** nsingh has quit IRC | 08:05 | |
| *** dhinesh has quit IRC | 08:06 | |
| *** bobh has joined #openstack-infra | 08:06 | |
| *** alexchadin has joined #openstack-infra | 08:11 | |
| *** ramishra has quit IRC | 08:12 | |
| *** d0ugal_ has joined #openstack-infra | 08:14 | |
| *** ramishra has joined #openstack-infra | 08:14 | |
| *** alexchadin has quit IRC | 08:16 | |
| *** alexchadin has joined #openstack-infra | 08:17 | |
| *** bobh has quit IRC | 08:19 | |
| *** alexchadin has quit IRC | 08:19 | |
| *** danpawlik has quit IRC | 08:19 | |
| *** alexchadin has joined #openstack-infra | 08:19 | |
| *** gouthamr has quit IRC | 08:25 | |
| *** bobh has joined #openstack-infra | 08:28 | |
| *** bobh has quit IRC | 08:33 | |
| *** hashar has joined #openstack-infra | 08:33 | |
| *** bobh has joined #openstack-infra | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 08:34 | |
| *** bobh has quit IRC | 08:38 | |
| *** amoralej|off is now known as amoralej | 08:38 | |
| AJaeger | mordred: regarding your stack on zuul-jobs, my suggestion: revert basically https://review.openstack.org/521253 - but let's merge really the current version over (not using pip --user), remove the jobs then from openstack-zuul-jobs (to not have two versions with same name) - and then base the rest of the stack on top of the moved roles. We could also iterate on openstack-zuul-jobs with changes that | 08:39 |
| AJaeger | we can merge directly. | 08:39 |
| *** dingyichen has quit IRC | 08:42 | |
| *** salv-orlando has quit IRC | 08:42 | |
| *** gmann_lunch is now known as gmann | 08:42 | |
| *** salv-orlando has joined #openstack-infra | 08:43 | |
| *** salv-orlando has quit IRC | 08:47 | |
| *** jpena|off is now known as jpena | 08:48 | |
| *** Daisy_ has joined #openstack-infra | 08:49 | |
| *** numans has joined #openstack-infra | 08:49 | |
| *** sshnaidm|off is now known as sshnaidm | 08:51 | |
| *** dbecker has joined #openstack-infra | 08:53 | |
| *** d0ugal_ has quit IRC | 08:53 | |
| *** claudiub|2 has joined #openstack-infra | 08:53 | |
| *** d0ugal has joined #openstack-infra | 08:53 | |
| *** d0ugal has quit IRC | 08:53 | |
| *** d0ugal has joined #openstack-infra | 08:53 | |
| *** yamamoto has quit IRC | 08:54 | |
| *** Hal has joined #openstack-infra | 08:56 | |
| *** apetrich has quit IRC | 08:56 | |
| *** apetrich has joined #openstack-infra | 08:56 | |
| Jeffrey4l | is there a squid cache proxy in the infra environment which i can use to speed up downloading rpms? | 08:56 |
| *** salv-orlando has joined #openstack-infra | 08:58 | |
| *** owalsh_ is now known as owalsh | 08:58 | |
| *** makowals has joined #openstack-infra | 09:03 | |
| *** dhajare has joined #openstack-infra | 09:04 | |
| *** e0ne has joined #openstack-infra | 09:05 | |
| *** jpich has joined #openstack-infra | 09:06 | |
| *** spectr has joined #openstack-infra | 09:07 | |
| *** spectr has quit IRC | 09:07 | |
| *** numans has quit IRC | 09:08 | |
| frickler | Jeffrey4l: not directly afaik, but the infra mirrors are set up as caching proxy for some repos, maybe you can add what you need here? https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n153 | 09:09 |
| Jeffrey4l | got. thanks. | 09:10 |
| *** numans has joined #openstack-infra | 09:10 | |
| *** spectr has joined #openstack-infra | 09:11 | |
| *** spectr has quit IRC | 09:12 | |
| *** slaweq has joined #openstack-infra | 09:14 | |
| *** pilgrimstack has joined #openstack-infra | 09:14 | |
| *** danpawlik has joined #openstack-infra | 09:18 | |
| *** efoley has joined #openstack-infra | 09:20 | |
| *** yamamoto has joined #openstack-infra | 09:20 | |
| *** Daisy_ has quit IRC | 09:22 | |
| *** Daisy_ has joined #openstack-infra | 09:22 | |
| *** d0ugal has quit IRC | 09:24 | |
| *** d0ugal has joined #openstack-infra | 09:25 | |
| *** lucas-afk is now known as lucasagomes | 09:25 | |
| *** Wei_Liu has quit IRC | 09:25 | |
| *** Wei_Liu has joined #openstack-infra | 09:26 | |
| *** derekh has joined #openstack-infra | 09:26 | |
| *** pilgrimstack has quit IRC | 09:26 | |
| *** Daisy_ has quit IRC | 09:26 | |
| openstackgerrit | Merged openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/520378 | 09:26 |
| *** Daisy_ has joined #openstack-infra | 09:27 | |
| *** pilgrimstack has joined #openstack-infra | 09:27 | |
| *** jascott1 has joined #openstack-infra | 09:29 | |
| *** dhajare has quit IRC | 09:31 | |
| *** makowals has quit IRC | 09:31 | |
| *** makowals has joined #openstack-infra | 09:32 | |
| *** jascott1 has quit IRC | 09:33 | |
| *** efoley has quit IRC | 09:34 | |
| *** efoley has joined #openstack-infra | 09:34 | |
| *** Daisy_ has quit IRC | 09:35 | |
| *** yamamoto has quit IRC | 09:36 | |
| *** gmann is now known as gmann_afk | 09:37 | |
| *** yamamoto has joined #openstack-infra | 09:37 | |
| *** bobh has joined #openstack-infra | 09:41 | |
| *** pilgrimstack1 has joined #openstack-infra | 09:44 | |
| *** yamamoto has quit IRC | 09:45 | |
| *** bobh has quit IRC | 09:46 | |
| *** pilgrimstack has quit IRC | 09:46 | |
| *** dhajare has joined #openstack-infra | 09:48 | |
| *** slaweq has quit IRC | 09:48 | |
| *** alker has joined #openstack-infra | 09:51 | |
| *** alker has quit IRC | 09:52 | |
| *** e0ne has quit IRC | 09:52 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove setting of version/release from releasenotes https://review.openstack.org/520792 | 09:52 |
| *** alker has joined #openstack-infra | 09:52 | |
| *** bhavik has quit IRC | 09:53 | |
| *** slaweq has joined #openstack-infra | 09:54 | |
| *** e0ne has joined #openstack-infra | 09:54 | |
| *** Daisy_ has joined #openstack-infra | 09:54 | |
| *** electrofelix has joined #openstack-infra | 09:55 | |
| *** kjackal has joined #openstack-infra | 09:55 | |
| *** alker1 has joined #openstack-infra | 09:56 | |
| *** alker1 has quit IRC | 09:56 | |
| *** alker1 has joined #openstack-infra | 09:58 | |
| *** alker has quit IRC | 09:59 | |
| *** Daisy_ has quit IRC | 09:59 | |
| *** ccamacho has joined #openstack-infra | 10:02 | |
| *** bobh has joined #openstack-infra | 10:02 | |
| *** annp has quit IRC | 10:03 | |
| *** bhavik has joined #openstack-infra | 10:03 | |
| *** efoley has quit IRC | 10:03 | |
| *** bobh has quit IRC | 10:06 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Setting default log url for tempestmail https://review.openstack.org/518723 | 10:07 |
| *** alker has joined #openstack-infra | 10:08 | |
| *** alker1 has quit IRC | 10:09 | |
| *** tosky has joined #openstack-infra | 10:10 | |
| *** andreas_s has quit IRC | 10:10 | |
| *** andreas_s has joined #openstack-infra | 10:11 | |
| *** liujiong has quit IRC | 10:13 | |
| *** bobh has joined #openstack-infra | 10:13 | |
| *** slaweq has quit IRC | 10:14 | |
| *** slaweq has joined #openstack-infra | 10:15 | |
| *** andreas_s has quit IRC | 10:15 | |
| *** yamamoto has joined #openstack-infra | 10:17 | |
| *** bobh has quit IRC | 10:17 | |
| *** dhajare has quit IRC | 10:18 | |
| *** slaweq has quit IRC | 10:20 | |
| *** andreas_s has joined #openstack-infra | 10:21 | |
| *** cuongnv has quit IRC | 10:21 | |
| *** dhill_ has quit IRC | 10:23 | |
| *** slaweq has joined #openstack-infra | 10:24 | |
| *** bobh has joined #openstack-infra | 10:25 | |
| *** dtantsur|afk is now known as dtantsur | 10:29 | |
| *** jaosorior has joined #openstack-infra | 10:30 | |
| *** bobh has quit IRC | 10:30 | |
| *** alker has quit IRC | 10:31 | |
| *** sree has quit IRC | 10:32 | |
| *** kiennt26 has quit IRC | 10:32 | |
| *** gongysh has quit IRC | 10:33 | |
| *** sree has joined #openstack-infra | 10:33 | |
| *** bobh has joined #openstack-infra | 10:33 | |
| *** andreas_s has quit IRC | 10:34 | |
| *** alker has joined #openstack-infra | 10:35 | |
| *** bhavik1 has joined #openstack-infra | 10:37 | |
| *** sree has quit IRC | 10:37 | |
| *** bobh has quit IRC | 10:38 | |
| *** ldnunes has joined #openstack-infra | 10:38 | |
| *** efoley has joined #openstack-infra | 10:38 | |
| AJaeger | config-core, could you review https://review.openstack.org/521127 to remove a broken post job - and https://review.openstack.org/520764 as cleanup, please? | 10:39 |
| *** bhavik has quit IRC | 10:39 | |
| *** bhavik1 is now known as bhavik | 10:39 | |
| *** andreas_s has joined #openstack-infra | 10:39 | |
| *** andreas_s has quit IRC | 10:43 | |
| *** andreas_s has joined #openstack-infra | 10:43 | |
| *** efoley has quit IRC | 10:44 | |
| *** efoley has joined #openstack-infra | 10:45 | |
| AJaeger | and one more change for murano, please: https://review.openstack.org/#/c/520267 - remove in-tree jobs | 10:51 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: remove unused extra release notes templates https://review.openstack.org/520764 | 10:51 |
| AJaeger | yolanda: please put 520267 on your queue as well | 10:52 |
| *** bhavik has quit IRC | 10:55 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Ignore missing .tox/env/logs directories for copy https://review.openstack.org/521436 | 10:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding jobs to new networking-l2gw-tempest-plugin https://review.openstack.org/521061 | 10:56 |
| *** ccamacho has quit IRC | 10:56 | |
| *** ccamacho has joined #openstack-infra | 10:57 | |
| AJaeger | thanks, frickler and yolanda ! | 10:57 |
| *** jpena is now known as jpena|off | 10:57 | |
| *** jpena|off is now known as jpena | 10:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add tempest-horizon to #openstack-horizon https://review.openstack.org/521442 | 10:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-tox-pep8 job to tempest-horizon https://review.openstack.org/521443 | 10:59 |
| claudiub|2 | AJaeger: hello. any news regarding the release-notes-jobs-neutron project template? | 11:00 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add puppet-openstackci to infra spec helper unit jobs https://review.openstack.org/521328 | 11:01 |
| *** kmalloc has joined #openstack-infra | 11:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy Murano jobs https://review.openstack.org/520267 | 11:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove obsolete murano-firstapp docs job https://review.openstack.org/521127 | 11:02 |
| *** szaher has quit IRC | 11:02 | |
| *** wolverineav has joined #openstack-infra | 11:03 | |
| eumel8 | AJaeger, frickler, yolanda: we want to move on with Zanata upgrade. Maybe you have time to merge https://review.openstack.org/#/c/506795/ . so I ask this week again :) | 11:04 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove Solum legacy job https://review.openstack.org/520963 | 11:05 |
| AJaeger | claudiub|2: not needed anymore, the nornmal one should just work with recent fixes | 11:05 |
| AJaeger | claudiub|2: which repo? | 11:05 |
| AJaeger | eumel8: frickler and myself are not core reviewer of those puppet repos | 11:05 |
| eumel8 | AJaeger: I know, but there are so few cores for infra puppet, then I try to spread my request. | 11:07 |
| claudiub|2 | AJaeger: networking-hyperv. I previously had this patch: https://review.openstack.org/#/c/519435/ | 11:08 |
| claudiub|2 | AJaeger: ah, nvm, I see you've sent a patch to networking-hyperv. :) | 11:09 |
| AJaeger | claudiub|2: should work even without that - but that will allow us some other fixes, so please merge it | 11:10 |
| AJaeger | claudiub|2: So, releasenotes should work *now* for you | 11:10 |
| claudiub|2 | AJaeger: yep, thanks. :D | 11:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove Murano legacy job https://review.openstack.org/520268 | 11:14 |
| *** andreas_s has quit IRC | 11:17 | |
| *** andreas_s has joined #openstack-infra | 11:18 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Install ssh keys of tripleo ci admins on nodes https://review.openstack.org/518638 | 11:18 |
| *** dtantsur is now known as dtantsur|bbl | 11:21 | |
| *** bobh has joined #openstack-infra | 11:21 | |
| *** andreas_s has quit IRC | 11:22 | |
| *** gcb has quit IRC | 11:24 | |
| *** raissa has joined #openstack-infra | 11:26 | |
| *** bobh has quit IRC | 11:26 | |
| *** alexchadin has quit IRC | 11:28 | |
| *** pilgrimstack1 has quit IRC | 11:31 | |
| *** rfolco has joined #openstack-infra | 11:32 | |
| *** openstackgerrit has quit IRC | 11:32 | |
| *** bobh has joined #openstack-infra | 11:34 | |
| *** andreas_s has joined #openstack-infra | 11:37 | |
| *** bobh has quit IRC | 11:39 | |
| *** slaweq has quit IRC | 11:39 | |
| *** alker has quit IRC | 11:40 | |
| *** bobh has joined #openstack-infra | 11:45 | |
| *** andreas_s has quit IRC | 11:45 | |
| *** openstackgerrit has joined #openstack-infra | 11:46 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Collect unbound DNS server configs and logs https://review.openstack.org/520007 | 11:46 |
| *** andreas_s has joined #openstack-infra | 11:46 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/openstack-zuul-jobs master: Jenkins plugins to xenial for java 8 https://review.openstack.org/518287 | 11:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove Murano legacy job https://review.openstack.org/520268 | 11:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-murano-firstapp-docs https://review.openstack.org/521497 | 11:48 |
| *** slaweq has joined #openstack-infra | 11:49 | |
| *** bobh has quit IRC | 11:50 | |
| *** szaher has joined #openstack-infra | 11:51 | |
| *** andreas_s has quit IRC | 11:51 | |
| *** d0ugal has quit IRC | 11:51 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/project-config master: Experimental Xenial jobs for Jenkins plugins https://review.openstack.org/518333 | 11:51 |
| *** gongysh has joined #openstack-infra | 11:52 | |
| *** jkilpatr has quit IRC | 11:52 | |
| *** slaweq has quit IRC | 11:53 | |
| *** d0ugal has joined #openstack-infra | 11:53 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/gearman-plugin master: Add maven/java8 jdk as test bindeps https://review.openstack.org/518284 | 11:54 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zmq-event-publisher master: Add maven/java8 jdk as test bindeps https://review.openstack.org/518290 | 11:54 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 11:56 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Set unbound logging to debug https://review.openstack.org/521500 | 11:57 |
| sshnaidm | pabelanger, ^^ | 11:57 |
| *** udesale has quit IRC | 11:57 | |
| sshnaidm | cores, please review in your time ^^ | 11:57 |
| *** andreas_s has joined #openstack-infra | 12:02 | |
| *** alker has joined #openstack-infra | 12:04 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Set unbound logging to debug https://review.openstack.org/521500 | 12:08 |
| *** lucasagomes is now known as lucas-hungry | 12:08 | |
| *** alker has quit IRC | 12:09 | |
| *** efried_zzz is now known as efried | 12:09 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set ubound to debug level and dump cache https://review.openstack.org/520055 | 12:12 |
| AJaeger | hashar: there's no need for an experimental job to just test that the job works. Zuul v3 jobs can be done self-testing, so test it yourself - and once it's ready, we can review and merge. If you need help with that, please just ask | 12:14 |
| *** pbourke has quit IRC | 12:14 | |
| *** pbourke has joined #openstack-infra | 12:16 | |
| *** caphrim007_ has quit IRC | 12:18 | |
| *** caphrim007 has joined #openstack-infra | 12:18 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zmq-event-publisher master: Add maven/java8 jdk as test bindeps https://review.openstack.org/518290 | 12:24 |
| *** jkilpatr has joined #openstack-infra | 12:25 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zmq-event-publisher master: Add maven/java8 jdk as test bindeps https://review.openstack.org/518290 | 12:26 |
| hashar | AJaeger: Zuul v3 is full of magic :] | 12:27 |
| *** dsariel has quit IRC | 12:27 | |
| *** ccamacho has left #openstack-infra | 12:28 | |
| *** sdague has joined #openstack-infra | 12:28 | |
| *** janki has quit IRC | 12:29 | |
| *** udesale has joined #openstack-infra | 12:29 | |
| *** bobh has joined #openstack-infra | 12:30 | |
| *** links has quit IRC | 12:31 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/gearman-plugin master: Add maven/java8 jdk as test bindeps https://review.openstack.org/518284 | 12:31 |
| *** efoley_ has joined #openstack-infra | 12:32 | |
| *** efoley has quit IRC | 12:32 | |
| *** gongysh has quit IRC | 12:33 | |
| *** sree has joined #openstack-infra | 12:34 | |
| *** bobh has quit IRC | 12:35 | |
| AJaeger | hashar: nice magic ;) | 12:35 |
| AJaeger | you could move the jobs also in-tree as explained in https://docs.openstack.org/infra/manual/zuulv3.html#moving-legacy-jobs-to-projects | 12:36 |
| *** gildub has quit IRC | 12:37 | |
| *** sree has quit IRC | 12:38 | |
| *** yamamoto has quit IRC | 12:39 | |
| *** bobh has joined #openstack-infra | 12:39 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/gearman-plugin master: Add maven/java8 jdk as test bindeps https://review.openstack.org/518284 | 12:41 |
| *** dsariel has joined #openstack-infra | 12:42 | |
| hashar | AJaeger: does that mean that the jobs would no more be defined in project-config but solely in the code repo? | 12:43 |
| *** pilgrimstack has joined #openstack-infra | 12:43 | |
| hashar | AJaeger: seems it is just about amending the .zuul.yaml file I have added to the test changes :) | 12:44 |
| *** andreas_s has quit IRC | 12:44 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: mirror: Use alternative openSUSE mirror https://review.openstack.org/521508 | 12:44 |
| *** andreas_s has joined #openstack-infra | 12:44 | |
| hwoarang | dirk: AJaeger^ what do you thnk? | 12:44 |
| *** bobh has quit IRC | 12:44 | |
| AJaeger | hashar: yes, you could define jobs solely in that repo. And the page I referenced explains that. | 12:45 |
| AJaeger | hwoarang: I'm not a mirror expert ;( | 12:45 |
| AJaeger | hwoarang: but let me review... | 12:45 |
| *** andreas_s has quit IRC | 12:49 | |
| hwoarang | AJaeger: no problem. this is only a temp fix until the main mirror is back | 12:49 |
| *** jpena is now known as jpena|lunch | 12:50 | |
| *** LindaWang1 has joined #openstack-infra | 12:50 | |
| *** bobh has joined #openstack-infra | 12:51 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/project-config master: Switch Jenkins plugins to xenial for java 8 https://review.openstack.org/518333 | 12:52 |
| *** sree has joined #openstack-infra | 12:54 | |
| *** sree_ has joined #openstack-infra | 12:55 | |
| *** sree_ is now known as Guest72696 | 12:56 | |
| *** andreas_s has joined #openstack-infra | 12:56 | |
| *** bobh has quit IRC | 12:56 | |
| *** rhallisey has joined #openstack-infra | 12:57 | |
| *** ramishra has quit IRC | 12:58 | |
| *** dprince has joined #openstack-infra | 12:58 | |
| *** sree has quit IRC | 12:59 | |
| *** salv-orlando has quit IRC | 12:59 | |
| *** gcb has joined #openstack-infra | 12:59 | |
| *** salv-orlando has joined #openstack-infra | 13:00 | |
| *** andreas_s has quit IRC | 13:00 | |
| *** slaweq has joined #openstack-infra | 13:01 | |
| *** yamamoto has joined #openstack-infra | 13:01 | |
| *** alker has joined #openstack-infra | 13:02 | |
| *** alker has quit IRC | 13:02 | |
| *** pilgrimstack has quit IRC | 13:03 | |
| *** salv-orlando has quit IRC | 13:04 | |
| *** bobh has joined #openstack-infra | 13:05 | |
| *** andreas_s has joined #openstack-infra | 13:06 | |
| *** ijw has joined #openstack-infra | 13:07 | |
| *** bobh has quit IRC | 13:10 | |
| *** andreas_s has quit IRC | 13:11 | |
| *** bobh has joined #openstack-infra | 13:12 | |
| *** alexchadin has joined #openstack-infra | 13:13 | |
| *** dhajare has joined #openstack-infra | 13:13 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** ijw_ has joined #openstack-infra | 13:14 | |
| *** dhajare has quit IRC | 13:14 | |
| *** lucas-hungry is now known as lucasagomes | 13:15 | |
| *** bobh has quit IRC | 13:17 | |
| *** ijw has quit IRC | 13:17 | |
| *** links has joined #openstack-infra | 13:18 | |
| *** andreas_s has joined #openstack-infra | 13:18 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/openstack-zuul-jobs master: Jenkins plugins to xenial for java 8 https://review.openstack.org/518287 | 13:19 |
| openstackgerrit | Antoine Musso proposed openstack-infra/openstack-zuul-jobs master: Jenkins plugins to xenial for java 8 https://review.openstack.org/518287 | 13:19 |
| dirk | Hwoarang: it should be quicker to get our mirroring fixed :-( | 13:20 |
| hwoarang | dirk: i don't know the status package says it's already 11 days | 13:20 |
| *** coolsvap has quit IRC | 13:20 | |
| *** trown|outtypewww is now known as trown | 13:21 | |
| dirk | hwoarang: yeah, how difficult can it be to remove a stale lock file ;-) | 13:21 |
| hwoarang | maybe we dont even need a full blown mirror in the end. we dont use source/debug packages, isos or obs repos | 13:21 |
| hwoarang | dirk: the status page says that the NIC is also broken | 13:22 |
| hwoarang | so i have no clue | 13:22 |
| *** andreas_s has quit IRC | 13:22 | |
| *** andreas_s has joined #openstack-infra | 13:22 | |
| *** rossella_s has quit IRC | 13:22 | |
| *** ijw has joined #openstack-infra | 13:23 | |
| *** andreas_s has quit IRC | 13:23 | |
| *** markvoelker has quit IRC | 13:23 | |
| *** markvoelker has joined #openstack-infra | 13:24 | |
| *** andreas_s has joined #openstack-infra | 13:24 | |
| *** rossella_s has joined #openstack-infra | 13:25 | |
| *** bobh has joined #openstack-infra | 13:25 | |
| *** rlandy has joined #openstack-infra | 13:25 | |
| dirk | Hwoarang: ah! | 13:26 |
| *** ijw_ has quit IRC | 13:26 | |
| *** rlandy is now known as rlandy|rover | 13:26 | |
| *** alker has joined #openstack-infra | 13:27 | |
| *** alker has quit IRC | 13:27 | |
| *** andreas_s has quit IRC | 13:27 | |
| *** andreas_s has joined #openstack-infra | 13:27 | |
| *** dhinesh has joined #openstack-infra | 13:28 | |
| *** andreas_s has quit IRC | 13:29 | |
| *** dpawlik has joined #openstack-infra | 13:29 | |
| *** andreas_s has joined #openstack-infra | 13:29 | |
| *** bobh has quit IRC | 13:29 | |
| *** andreas_s has quit IRC | 13:30 | |
| *** andreas_s has joined #openstack-infra | 13:30 | |
| *** danpawlik has quit IRC | 13:30 | |
| *** dhinesh has quit IRC | 13:33 | |
| *** bobh has joined #openstack-infra | 13:33 | |
| *** dave-mccowan has joined #openstack-infra | 13:36 | |
| *** dsariel has quit IRC | 13:36 | |
| *** pblaho has joined #openstack-infra | 13:37 | |
| openstackgerrit | boden proposed openstack-infra/project-config master: periodic py jobs for vmware-nsx ocata and pike https://review.openstack.org/521538 | 13:40 |
| *** slaweq has quit IRC | 13:41 | |
| *** ijw has quit IRC | 13:41 | |
| *** udesale has quit IRC | 13:42 | |
| *** Daisy_ has joined #openstack-infra | 13:43 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: mirror: Use alternative openSUSE mirror https://review.openstack.org/521508 | 13:43 |
| *** kgiusti has joined #openstack-infra | 13:45 | |
| *** slaweq has joined #openstack-infra | 13:45 | |
| *** dave-mcc_ has joined #openstack-infra | 13:46 | |
| *** dave-mccowan has quit IRC | 13:48 | |
| *** Daisy_ has quit IRC | 13:48 | |
| *** bobh has quit IRC | 13:48 | |
| *** dsariel has joined #openstack-infra | 13:49 | |
| *** Daisy_ has joined #openstack-infra | 13:50 | |
| *** jpena|lunch is now known as jpena | 13:54 | |
| *** Daisy_ has quit IRC | 13:55 | |
| *** Guest72696 has quit IRC | 13:58 | |
| *** sree has joined #openstack-infra | 13:59 | |
| *** salv-orlando has joined #openstack-infra | 13:59 | |
| *** eharney has quit IRC | 14:00 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use prepared instackenv.json if exists https://review.openstack.org/520023 | 14:00 |
| *** sree has quit IRC | 14:03 | |
| Jeffrey4l | inc0, could u review https://review.openstack.org/519939 | 14:05 |
| Jeffrey4l | and https://review.openstack.org/519969 | 14:06 |
| *** bobh has joined #openstack-infra | 14:09 | |
| *** slaweq has quit IRC | 14:09 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 14:12 |
| *** mriedem has joined #openstack-infra | 14:14 | |
| *** yamamoto has joined #openstack-infra | 14:14 | |
| *** Goneri has joined #openstack-infra | 14:16 | |
| *** andreas_s has quit IRC | 14:17 | |
| *** ramishra has joined #openstack-infra | 14:17 | |
| *** andreas_s has joined #openstack-infra | 14:18 | |
| *** yamamoto has quit IRC | 14:20 | |
| *** yamamoto has joined #openstack-infra | 14:20 | |
| *** bobh has quit IRC | 14:21 | |
| *** rhallisey has quit IRC | 14:22 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 14:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 14:24 |
| mordred | morning AJaeger - ^^ there's the final form of the zuul-jobs stuff squahsed into a single patch | 14:24 |
| *** rhallisey has joined #openstack-infra | 14:25 | |
| *** bobh has joined #openstack-infra | 14:25 | |
| *** [HeOS] has quit IRC | 14:28 | |
| *** bobh has quit IRC | 14:30 | |
| *** andreas_s has quit IRC | 14:31 | |
| *** gouthamr has joined #openstack-infra | 14:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Base build-openstack-releasenotes on build-reno-releasenotes https://review.openstack.org/521144 | 14:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Switch build-openstack-sphinx-docs to build-sphinx-docs https://review.openstack.org/521145 | 14:32 |
| *** pbourke has quit IRC | 14:32 | |
| *** alexchadin has quit IRC | 14:32 | |
| *** dhinesh has joined #openstack-infra | 14:35 | |
| *** bobh has joined #openstack-infra | 14:35 | |
| *** dhill_ has joined #openstack-infra | 14:39 | |
| *** andreas_s has joined #openstack-infra | 14:39 | |
| *** jamesdenton has joined #openstack-infra | 14:39 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config master: [rally] Remove 2 legacy jobs https://review.openstack.org/520372 | 14:40 |
| *** bobh has quit IRC | 14:40 | |
| dmsimard | mordred: did you mean to submit another patchset on https://review.openstack.org/#/c/511843/ ? I see "done" on some comments but not a third patchset. | 14:42 |
| *** bobh has joined #openstack-infra | 14:42 | |
| *** bobh has quit IRC | 14:43 | |
| *** bobh has joined #openstack-infra | 14:43 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove Murano legacy job https://review.openstack.org/520268 | 14:45 |
| *** fultonj has joined #openstack-infra | 14:47 | |
| *** pbourke has joined #openstack-infra | 14:47 | |
| *** andreas_s has quit IRC | 14:48 | |
| dmsimard | mordred: you also mention a follow up in https://review.openstack.org/#/c/511823/ but I can't find it :/ | 14:48 |
| *** dhinesh has quit IRC | 14:48 | |
| *** andreas_s has joined #openstack-infra | 14:48 | |
| *** mat128 has joined #openstack-infra | 14:48 | |
| *** ykarel is now known as ykarel|away | 14:49 | |
| *** dtantsur|bbl is now known as dtantsur | 14:51 | |
| *** ramishra has quit IRC | 14:51 | |
| AJaeger | mordred: will review later tonight, thanks | 14:52 |
| dmsimard | The different DNS issues that we have... do we know if it's about Google or OpenDNS rate-limiting us ? Would it be worth standing up resolvers in each nodepool region and then have VMs use those ? | 14:52 |
| dmsimard | Even if those resolvers end up just forwarding to Google/OpenDNS and caching | 14:52 |
| *** andreas_s has quit IRC | 14:53 | |
| *** ramishra has joined #openstack-infra | 14:53 | |
| *** andreas_s has joined #openstack-infra | 14:53 | |
| *** eharney has joined #openstack-infra | 14:55 | |
| *** ykarel|away has quit IRC | 14:56 | |
| sshnaidm | pabelanger, hi, did you raise the question in infra mtg? https://review.openstack.org/#/c/517928/ | 14:58 |
| dmsimard | Wow, trying to find legitimate DNS lookup errors in logstash is hard.. projects are using garbage hostnames all over the place | 14:59 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Add subunit2sql check database server and workers https://review.openstack.org/520192 | 14:59 |
| *** mat128 has quit IRC | 14:59 | |
| *** udesale has joined #openstack-infra | 15:00 | |
| *** yamahata has joined #openstack-infra | 15:01 | |
| *** sree has joined #openstack-infra | 15:01 | |
| *** links has quit IRC | 15:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Base build-openstack-releasenotes on build-reno-releasenotes https://review.openstack.org/521144 | 15:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Switch build-openstack-sphinx-docs to build-sphinx-docs https://review.openstack.org/521145 | 15:04 |
| *** mat128 has joined #openstack-infra | 15:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use the new sphinx PTI for publish jobs https://review.openstack.org/521557 | 15:04 |
| *** dave-mcc_ is now known as dave-mccowan | 15:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove ensure-reno role https://review.openstack.org/521143 | 15:07 |
| *** LindaWang1 has quit IRC | 15:07 | |
| *** alker has joined #openstack-infra | 15:07 | |
| *** alker has quit IRC | 15:08 | |
| *** sree has quit IRC | 15:08 | |
| *** gcb has quit IRC | 15:08 | |
| *** sree has joined #openstack-infra | 15:08 | |
| *** alker has joined #openstack-infra | 15:09 | |
| *** alker has quit IRC | 15:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make build-python-release job https://review.openstack.org/513925 | 15:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove old python-sdist job https://review.openstack.org/513926 | 15:10 |
| mordred | dmsimard: yah - it's a shame that more people don't use the example.com domain for fake hostnames | 15:10 |
| *** hongbin has joined #openstack-infra | 15:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: [rally] Remove 2 legacy jobs https://review.openstack.org/520372 | 15:11 |
| electrofelix | I recall asking before about as to the reasons behind removing the ability to push back the merge created in zuul merger to the source (GitHub or Gerrit), and some of it was related to issues when scaling out zuul horizontally | 15:11 |
| mordred | dmsimard: also - I think we've talked about per-region resolvers that we then point unbound to - I'm not sure where we ever got on that - I certainly don't think it's totally insane | 15:11 |
| *** alker has joined #openstack-infra | 15:12 | |
| *** ihrachys has joined #openstack-infra | 15:12 | |
| electrofelix | If that was to be solved by ensuring the same zuul merger was used when queuing up changes belonging to the same project, would support for pushing back merges be accepted back into zuul? | 15:12 |
| *** alker has quit IRC | 15:12 | |
| dmsimard | mordred: I'm currently digging to try and see if there's something in common for the DNS failures. So far it looks like there's a lot of legit issues. I'll come back with some data later. | 15:13 |
| mordred | electrofelix: I believe there is work on the todo list for using the refs created by the merger on the executor | 15:13 |
| *** links has joined #openstack-infra | 15:13 | |
| electrofelix | is that in a story already? | 15:14 |
| mordred | electrofelix: and I think having the ability to have zuul push its merge commit is a desired thing | 15:14 |
| mordred | electrofelix: one sec, lemme look ... | 15:14 |
| mordred | SpamapS: ^^ I think you were involved in that discussion most recently | 15:14 |
| electrofelix | mordred: I understand there are some security concerns, but this has been biting us a bit and requiring a lot of workaround effort | 15:15 |
| *** jaypipes has joined #openstack-infra | 15:15 | |
| electrofelix | so it might be something I can get some support from management to fix permanently ;-) | 15:15 |
| fungi | EmilienM: here's a representative sample of the top 100 jobs from the last month or so of job logs (total size in bytes across all runs by name): http://paste.openstack.org/show/626813/ | 15:16 |
| mordred | electrofelix: cool! jeblair is back from vacation today iirc, so maybe when he's up it would be a good thing to chat about - it's obviously something that touches architecture in a fairly deep way, but I'm pretty sure we at least want zuul to have the option to push the merge it made, even if openstack doesn't use that option | 15:17 |
| AJaeger | mordred: did you run some DNM jobs against your new stack to see that everything works? | 15:17 |
| mordred | AJaeger: I was just about to go recheck the dnm patches | 15:17 |
| *** alker has joined #openstack-infra | 15:17 | |
| AJaeger | mordred: I abandonend mine - if I should restore some, tell me... | 15:17 |
| *** alker has quit IRC | 15:17 | |
| *** udesale has quit IRC | 15:18 | |
| fungi | EmilienM: analysis of the complete dataset indicates that jobs with "tripleo" in their name or pipeline account for 41% of the total we're archiving | 15:18 |
| mordred | AJaeger: I just marked both of them with the updated-pti topic (which is the topic I used for this stack now) | 15:19 |
| mtreinish | has anyone seen this failure mode on the infra beaker jobs? http://logs.openstack.org/12/520212/1/check/legacy-openstackci-beaker-ubuntu-trusty/1839158/job-output.txt.gz#_2017-11-19_18_59_54_977581 | 15:19 |
| mtreinish | my patches seem to be failing on it pretty frequently | 15:19 |
| mordred | AJaeger: maybe restore yours and mark them with that topic too - just so we can make double-sure | 15:19 |
| fungi | EmilienM: so actually it's gotten worse overall since the old sample i have from august 5, where "tripleo" jobs accounted for 33% of the bytes in our log archive | 15:20 |
| *** marst has joined #openstack-infra | 15:20 | |
| AJaeger | mordred: will do | 15:20 |
| *** alker has joined #openstack-infra | 15:22 | |
| *** rbrndt has joined #openstack-infra | 15:22 | |
| AJaeger | mordred: done, so 3 more tests up ;) | 15:23 |
| fungi | that analysis does raise an interesting question though... why is build-openstack-sphinx-docs using so much room? it takes up nearly 3% of our log archive. i guess we run it a lot more often than other jobs | 15:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Half-Revert "Revert "Add ensure-reno and ensure-babel roles"" https://review.openstack.org/521558 | 15:24 |
| *** d0ugal has quit IRC | 15:24 | |
| *** bhavik1 has joined #openstack-infra | 15:24 | |
| fungi | yeah, 6.9% of the jobs we run (by count) are build-openstack-sphinx-docs so i guess that's a big part of why | 15:25 |
| fungi | it's fairly ubiquitous | 15:25 |
| AJaeger | mordred: argh, your change https://review.openstack.org/521145 has retry_limit | 15:26 |
| fungi | 1 out of every 14 or 15 jobs we run is build-openstack-sphinx-docs | 15:26 |
| AJaeger | fungi, it includes the html now as well | 15:26 |
| fungi | ahh | 15:26 |
| fungi | right, phasing out docs-draft.o.o | 15:26 |
| fungi | makes sense | 15:26 |
| AJaeger | fungi, remember /html is not subdir of logs, so phasing out +1 | 15:26 |
| AJaeger | but 1 out of 15 jbos - that's lot | 15:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Half-Revert "Revert "Add ensure-reno and ensure-babel roles"" https://review.openstack.org/521558 | 15:27 |
| *** d0ugal has joined #openstack-infra | 15:27 | |
| AJaeger | fungi, IMHO we need to build docs, if code changes - for autodoc | 15:27 |
| *** links has quit IRC | 15:27 | |
| fungi | agreed | 15:27 |
| fungi | i wasn't suggesting we should run it less often, just trying to figure out whay causes it to rank so high | 15:27 |
| *** rhallisey has quit IRC | 15:28 | |
| fungi | it's a generic enough job that most projects can use it without redefining, it's rarely skipped (for good reason) and it incorporates the complete draft output from the test build | 15:28 |
| fungi | so makes sense | 15:29 |
| *** rhallisey has joined #openstack-infra | 15:29 | |
| fungi | oh, also, most projects keep their documentation in-tree, which is a good thing, so that means more runs. and projects are moving their pieces of the larger manuals in-tree as well... do those also use the same job? | 15:30 |
| *** pilgrimstack has joined #openstack-infra | 15:30 | |
| AJaeger | fungi: yes, single job for all docs mainly (releasenotes, api docs are separate) | 15:31 |
| fungi | awesome | 15:32 |
| AJaeger | mordred: http://logs.openstack.org/77/520677/2/check/build-openstack-releasenotes/2ad2375/ - nova failure | 15:33 |
| AJaeger | mordred: retry_limit for openstackdocstheme http://logs.openstack.org/63/520363/2/check/build-openstack-releasenotes/80deb6e/ | 15:33 |
| *** pilgrimstack has quit IRC | 15:34 | |
| *** bhavik1 has quit IRC | 15:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use ensure-babel to make sure babel is installed https://review.openstack.org/521105 | 15:40 |
| mordred | AJaeger: awesome. thanks | 15:40 |
| *** efoley_ is now known as efoley | 15:40 | |
| *** alker has quit IRC | 15:41 | |
| *** pilgrimstack has joined #openstack-infra | 15:41 | |
| mordred | fungi: once this stack is working and complete ... I'd like to discuss doing $something to enable use of pip install --user on our test nodes | 15:42 |
| fungi | mordred: sure, i can see a generic role for that being useful too | 15:42 |
| mordred | fungi: it's not really an option in an easy way currently because ~/.local/bin isn't in the zuul user's PATH - but I'm not sure the 'right' way to ensure that it is in such a way that zuul-jobs content could depend on it | 15:43 |
| mordred | fungi: maybe a generic role which is "enable ~/.local" - and then just have roles in zuul-jobs that depend on ~/.local working just depend on that role too? | 15:43 |
| mordred | fungi: and we could have that role add ~/.local/bin to path in ~/.bashrc on linux - and thus have the opportunity for someone to add whatever is needed for windows perhaps | 15:44 |
| fungi | perhaps... i don't really like the ~/.local/bin addition when ~/bin is already a fairly widely-implemented standard, but i suppose that train has sailed with pip now | 15:45 |
| *** edmondsw has joined #openstack-infra | 15:47 | |
| AJaeger | mordred: https://review.openstack.org/520363 finished - both jobs failed ;( | 15:48 |
| *** dhinesh has joined #openstack-infra | 15:49 | |
| AJaeger | mordred: btw. I have now a script for the rechecks, it contains lines like ssh -p 29418 jaegerandi@review.openstack.org gerrit review 520363,2 -m "'recheck - testing again'" | 15:50 |
| fungi | if it's for batch rechecking, the process mark in gertty is great for that | 15:52 |
| dmsimard | mtreinish: I'm thinking I would really like elastic-recheck to graph some failures without necessarily commenting on the patch to ask for a recheck, but it sounds like what I want would be logstash dashboards instead.. | 15:52 |
| dmsimard | mtreinish: should we perhaps rely on logstash for graphing/dashboarding ? | 15:53 |
| dmsimard | grafyaml for logstash anyone ? | 15:53 |
| *** bnemec-pto is now known as bnemec | 15:54 | |
| *** dhinesh has quit IRC | 15:54 | |
| fungi | logstash queries are slow/expensive, so an intermediary etl solution is still needed | 15:54 |
| mtreinish | dmsimard: I mean that's what http://status.openstack.org/elastic-recheck/ is | 15:55 |
| mtreinish | the first part not relying on logstash | 15:55 |
| AJaeger | mordred: let me rebase https://review.openstack.org/#/c/521105/ for you | 15:55 |
| mtreinish | and for that we just run a command in cron to generate json used for the graphs | 15:55 |
| mtreinish | because as fungi says logstash queries are slow/expensive | 15:55 |
| dmsimard | mtreinish: right, I know the elastic-recheck but that also implies elastic-recheck adding a comment in the reviews for a recheck, right ? | 15:55 |
| *** pilgrimstack has quit IRC | 15:56 | |
| fungi | not necessarily for the successes | 15:56 |
| mtreinish | dmsimard: no they're independent | 15:56 |
| fungi | also, separate bits of software | 15:56 |
| mtreinish | dmsimard: the comments are done by a bot, just don't run the bot | 15:56 |
| dmsimard | mtreinish: eh, but openstack-infra runs the bot :) | 15:56 |
| *** caphrim007 has quit IRC | 15:56 | |
| mtreinish | dmsimard: the graphs are just cron jobs calling standalone command | 15:56 |
| dmsimard | You want me to graph those on my machine or something? :p | 15:56 |
| mtreinish | dmsimard: oh you want infra to generate the graphs | 15:56 |
| dmsimard | well yeah, it'd be to highlight some issues on elastic-recheck page | 15:57 |
| mtreinish | dmsimard: there is also an arg for queries (i'd have to look it up) to disable commenting | 15:57 |
| *** caphrim007 has joined #openstack-infra | 15:57 | |
| dmsimard | for example, there might be a legitimate issue that is not yet fixed so we don't want people to needlessly recheck | 15:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use the new sphinx PTI for publish jobs https://review.openstack.org/521557 | 15:58 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use ensure-babel to make sure babel is installed https://review.openstack.org/521105 | 15:58 |
| mtreinish | dmsimard: in my experience people recheck regardless of whether a bot says there is a bug or not | 15:58 |
| AJaeger | mordred: please review whether playbooks/translation/pre.yaml in 521105 is fine | 15:58 |
| mtreinish | dmsimard: often when their patch introduces the failure too :/ | 15:58 |
| dmsimard | mtreinish: that makes me a sad panda | 15:58 |
| SpamapS | mordred: yeah, having the sha stay the same would have the nice benefit of being able to re-use artifacts built using it. | 16:00 |
| *** yamamoto has quit IRC | 16:00 | |
| mtreinish | dmsimard: anyway if you add 'suppress-notification: true' to the query the bot shouldn't report it on gerrit | 16:00 |
| SpamapS | Something that follows a trigger all the way to job success and notifier. | 16:00 |
| dmsimard | mtreinish: good to know, thanks | 16:00 |
| *** caphrim007 has quit IRC | 16:01 | |
| *** andrea___ has joined #openstack-infra | 16:02 | |
| *** andreas_s has quit IRC | 16:05 | |
| *** andrea___ has quit IRC | 16:06 | |
| *** yamamoto has joined #openstack-infra | 16:06 | |
| *** andreas_s has joined #openstack-infra | 16:06 | |
| clarkb | mtreinish: re beaker failures I pushed up patches saturday to fix that I think | 16:08 |
| fungi | did we merge a fix for "bundler: command not found: rspec" in legacy-openstackci-beaker-ubuntu-trusty? http://logs.openstack.org/89/484989/2/gate/legacy-openstackci-beaker-ubuntu-trusty/326fe2a/job-output.txt.gz#_2017-11-17_22_10_42_561586 | 16:08 |
| fungi | oh, hah | 16:08 |
| fungi | i think clarkb just answered my question too ;) | 16:08 |
| fungi | clarkb: not sure if you saw my du sample analysis for logs.o.o which ran over the weekend, but tripleo jobs currently make up 41% of our job log content on the server | 16:09 |
| clarkb | dmsimard: mordred every other time we've debugged dns failures its not been a google/opendns rate limit but some network issue between here nd there | 16:09 |
| clarkb | fungi: I briefly skimmed it. Thanks for putting that together | 16:10 |
| fungi | took a few days to complete | 16:10 |
| AJaeger | clarkb: do you mean https://review.openstack.org/521328 - that merged today | 16:10 |
| *** yamamoto has quit IRC | 16:11 | |
| *** ramishra has quit IRC | 16:11 | |
| fungi | that was analyzing a random ~20% sample of the change-triggered (check, check-tripleo, gate, experimental...) job logs | 16:11 |
| clarkb | AJaeger: 521327 | 16:12 |
| clarkb | that isnt passing tests though | 16:12 |
| *** andreas_s has quit IRC | 16:13 | |
| *** jkilpatr has quit IRC | 16:13 | |
| clarkb | I think because we dont test the code proposed | 16:13 |
| clarkb | but instead fetch from the git mirror | 16:13 |
| *** andreas_s has joined #openstack-infra | 16:14 | |
| EmilienM | dmsimard, pabelanger: hey, can you tell me what I did wrong in https://review.openstack.org/#/c/521278/2/zuul.d/layout.yaml ? it seems like the py35 job is still executed in zuul | 16:16 |
| *** andreas_s has quit IRC | 16:18 | |
| *** kjackal has quit IRC | 16:20 | |
| AJaeger | EmilienM: you need to backport to all branches | 16:21 |
| *** kjackal has joined #openstack-infra | 16:21 | |
| EmilienM | AJaeger: from master? | 16:21 |
| EmilienM | AJaeger: ok | 16:21 |
| AJaeger | EmilienM: https://docs.openstack.org/infra/manual/zuulv3.html#stable-branches | 16:21 |
| AJaeger | EmilienM: is os-net-config still in project-config? That would explain it as well... | 16:22 |
| EmilienM | ok | 16:22 |
| EmilienM | AJaeger: yeah it is | 16:22 |
| AJaeger | EmilienM: but py35 is not in project-config. | 16:22 |
| *** sree has quit IRC | 16:22 | |
| AJaeger | EmilienM: wait it is - and that's right. It should stay there. Add in project-config the branches lines | 16:23 |
| *** andreas_s has joined #openstack-infra | 16:24 | |
| andreaf | mordred, jeblair: pip freeze is not happy when a repo without an origin is installed in dev mode - any workaround for that? | 16:24 |
| AJaeger | EmilienM: now reviewed your os-net-config change and commented | 16:25 |
| EmilienM | AJaeger: ok | 16:25 |
| clarkb | andreaf: its a repo withojt any remote iirc, I have a PR to pip to change that and it isnt fatal just a warning | 16:27 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Disable py35 jobs for os-net-config/newton https://review.openstack.org/521576 | 16:28 |
| EmilienM | AJaeger: thx btw | 16:28 |
| mtreinish | clarkb: ok cool, so I should recheck? or are we waiting for it to land? | 16:28 |
| andreaf | clarkb: yeah it's just a warning but still annoying :D thanks for making a PR for that | 16:28 |
| clarkb | mtreinish: need to fix the job to install the helper from zuul source and not the git mirror I think | 16:29 |
| AJaeger | EmilienM: disabling a template would be nice but I don't think it works | 16:29 |
| EmilienM | AJaeger: ahah | 16:29 |
| *** jkilpatr has joined #openstack-infra | 16:30 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Disable py35 jobs for os-net-config/newton https://review.openstack.org/521576 | 16:30 |
| *** sean-k-mooney has joined #openstack-infra | 16:31 | |
| sean-k-mooney | o/ | 16:31 |
| clarkb | mtreinish: I will work on that post breakfast | 16:31 |
| clarkb | sean-k-mooney: hello | 16:31 |
| mtreinish | clarkb: thanks | 16:32 |
| sean-k-mooney | qq i just noticed that the nova specs repo is not being mirrored correctly to github as of september 21st | 16:32 |
| clarkb | sean-k-mooney: the repo is corrupt so github wont accept it | 16:32 |
| sean-k-mooney | corrupt? | 16:33 |
| *** andreas_s has quit IRC | 16:33 | |
| clarkb | ya I forget specific details gut github doesnt like it | 16:33 |
| *** andreas_s has joined #openstack-infra | 16:34 | |
| sean-k-mooney | hum ok so even a force push to sync it to the openstack copy would not resolve it. | 16:34 |
| clarkb | there is a proposed fix but involves extendedgerrit downtime and rewriting git history in the repo iirc | 16:34 |
| clarkb | which we havent had time to do yet | 16:34 |
| sean-k-mooney | hum ok i was just wondering if ye were aware. sound like ye are | 16:34 |
| dhellmann | AJaeger , fungi : I would appreciate your input on this thread when you have some time: http://lists.openstack.org/pipermail/openstack-dev/2017-November/124718.html | 16:34 |
| *** pcaruana has quit IRC | 16:35 | |
| clarkb | sean-k-mooney: I think it replicates to our git mirrors properly | 16:36 |
| AJaeger | dhellmann: read it - no comments from my side, not aware of anything to block this. But I don't know about permissions | 16:36 |
| *** yamamoto has joined #openstack-infra | 16:36 | |
| sean-k-mooney | the fact that we don't render rst automatically actually makes it easier to like people to specific section of the spec if i use the openstack git instead of github anyway | 16:36 |
| sean-k-mooney | clarkb: yes it does | 16:36 |
| dhellmann | AJaeger : I think as a release manager I have the permissions, I was more concerned about process. | 16:36 |
| *** jtomasek has quit IRC | 16:36 | |
| *** jtomasek has joined #openstack-infra | 16:37 | |
| AJaeger | dhellmann: ad-hoc process ;) Let me answer :) | 16:38 |
| dhellmann | AJaeger : I'm just not aware of any precedent for restoring a stable branch, I guess, and if there is one I want to follow it :-) | 16:38 |
| *** salv-orlando has quit IRC | 16:38 | |
| *** andreas_s has quit IRC | 16:38 | |
| *** salv-orlando has joined #openstack-infra | 16:39 | |
| fungi | tag deletion is not something we do, since that won't get picked up on remote update/pull anyway | 16:39 |
| fungi | but branch creation should be fine | 16:39 |
| *** bhavik1 has joined #openstack-infra | 16:39 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Move TripleO jobs in-tree (part 2) https://review.openstack.org/521315 | 16:39 |
| AJaeger | mordred: docs + releasenotes fail on https://review.openstack.org/520677 | 16:39 |
| *** yamamoto has quit IRC | 16:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable py35 jobs for os-net-config/newton https://review.openstack.org/521576 | 16:41 |
| fungi | dhellmann: dhellmann probably the only adjustment is that if you want the new branch state tagged with something before redeletion, it'll need to be named something other than "mitaka-eol" | 16:42 |
| dhellmann | fungi : under the new policy we won't be deleting the branch | 16:43 |
| *** psachin has quit IRC | 16:43 | |
| *** jpena is now known as jpena|brb | 16:43 | |
| fungi | dhellmann: one of the work items says "Re-close the branch." | 16:43 |
| dhellmann | hmm | 16:43 |
| fungi | so not sure what is meand in that case | 16:43 |
| fungi | er, meant | 16:43 |
| dhellmann | maybe I'm misremembering | 16:43 |
| *** salv-orlando has quit IRC | 16:43 | |
| *** jascott1 has joined #openstack-infra | 16:44 | |
| *** Apoorva has joined #openstack-infra | 16:44 | |
| *** hashar is now known as hasharAway | 16:45 | |
| fungi | what was the reason for keeping the recreated branch around indefinitely? the spec says content will still remain frozen. are reviewers just going to take care of abandoning any new changes proposed against stable/mitaka, or will the acl be adjusted to reject uploads for that branch, or what? | 16:45 |
| *** andreas_s has joined #openstack-infra | 16:45 | |
| fungi | branch deletion is normally how we implement that to reduce review burden | 16:45 |
| jeblair | (and indicate to users that patches are not accepted) | 16:46 |
| dhellmann | I was trying to remove steps from the eol process for the docs and allow for critical fixes to happen (like if we need to rebuild something because of a theme change) | 16:46 |
| dhellmann | the repo is effectively branchless now | 16:46 |
| *** pkovar has joined #openstack-infra | 16:47 | |
| jeblair | (since writing a patch and uploading it, and only then (or even later) receiving a message saying you never should have written it in the first place is not the best experience) | 16:47 |
| dhellmann | in any case, we could decide to re-close the branch if that's what we want to do | 16:48 |
| jeblair | last time this came up i think we just suggested "foo-eol2" as a tag format? | 16:48 |
| dhellmann | we spent a lot of time talking about this in denver, it's a shame neither of you could be there | 16:48 |
| *** jascott1 has quit IRC | 16:48 | |
| dhellmann | sure, that would be fine | 16:48 |
| dhellmann | when jimmy finishes the work to add "support level" badges to the theme, we'll need to retrigger builds in all of the branches so we can update the published content to add those badges | 16:49 |
| dhellmann | so we'll need to hold off on closing it until at least then | 16:49 |
| *** bhavik1 has quit IRC | 16:50 | |
| *** [HeOS] has joined #openstack-infra | 16:50 | |
| jeblair | the spec is very helpful since i wasn't able to participate | 16:50 |
| EmilienM | AJaeger: that worked fine, fyi. Thanks again | 16:50 |
| *** bhavik1 has joined #openstack-infra | 16:50 | |
| jeblair | dhellmann: ack. i'll start mulling on whether there could be another way to handle that sort of thing. | 16:51 |
| *** Apoorva has quit IRC | 16:51 | |
| dhellmann | the idea is to build the badges from openstack-manuals on master so we have a $series.svg or something and then have the theme link to the right $series file (but not contain the content) | 16:52 |
| *** dsariel has quit IRC | 16:52 | |
| dhellmann | that way the right content is loaded on page view and we don't have to rebuild old stuff more than once | 16:52 |
| *** pilgrimstack has joined #openstack-infra | 16:52 | |
| *** Apoorva has joined #openstack-infra | 16:53 | |
| dhellmann | bbiab | 16:54 |
| *** andreas_s has quit IRC | 16:54 | |
| *** petevg_afk is now known as petevg | 16:54 | |
| *** andreas_s has joined #openstack-infra | 16:56 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix nodepool cmd TypeError when no arguemnts https://review.openstack.org/519582 | 16:56 |
| *** witek has joined #openstack-infra | 16:57 | |
| *** andreas_s has quit IRC | 16:57 | |
| *** andreas_s has joined #openstack-infra | 16:57 | |
| *** armax has quit IRC | 16:59 | |
| *** bhavik1 has quit IRC | 16:59 | |
| *** jascott1 has joined #openstack-infra | 17:00 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Add subunit2sql check database server and workers https://review.openstack.org/520192 | 17:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 17:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update fetch sphinx output to use sphinx vars https://review.openstack.org/521590 | 17:01 |
| mordred | AJaeger: okie. I think that should fix the errors from the depends-on patches | 17:01 |
| mordred | AJaeger: I've rechecked the DNM patches | 17:03 |
| *** pilgrimstack has quit IRC | 17:05 | |
| *** yamamoto has joined #openstack-infra | 17:05 | |
| *** yamamoto has quit IRC | 17:05 | |
| *** andreas_s has quit IRC | 17:06 | |
| *** caphrim007 has joined #openstack-infra | 17:08 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Add novajoin to collect-logs.yml list https://review.openstack.org/521591 | 17:09 |
| *** jkilpatr has quit IRC | 17:09 | |
| *** jkilpatr has joined #openstack-infra | 17:10 | |
| *** caphrim007_ has joined #openstack-infra | 17:10 | |
| *** caphrim007_ has quit IRC | 17:12 | |
| *** caphrim007_ has joined #openstack-infra | 17:12 | |
| *** caphrim007_ has quit IRC | 17:12 | |
| *** caphrim007 has quit IRC | 17:12 | |
| *** caphrim007 has joined #openstack-infra | 17:13 | |
| *** Apoorva has quit IRC | 17:15 | |
| *** hemna_ has quit IRC | 17:17 | |
| *** hemna_ has joined #openstack-infra | 17:17 | |
| *** iyamahat has joined #openstack-infra | 17:18 | |
| *** Hal has quit IRC | 17:20 | |
| clarkb | mordred: AJaeger fungi I'm working to fix this puppet-openstackci issue and remind me, ZUUL_REF and WORKSPACE are no longer env vars by default? | 17:21 |
| clarkb | or at least ZUUL_REF isn't right? | 17:21 |
| clarkb | what is a good way from within a shell script to know if you are running under zuul or not? | 17:21 |
| *** electrofelix has quit IRC | 17:21 | |
| *** lucasagomes is now known as lucas-afk | 17:23 | |
| clarkb | ZUUL_UUID for now I guess | 17:23 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstackci master: Check for presence of Zuul with Zuul v3 https://review.openstack.org/521596 | 17:24 |
| *** yamahata has quit IRC | 17:24 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_infra_spec_helper master: Cap rake below 12.3.0 https://review.openstack.org/521327 | 17:25 |
| *** iyamahat has quit IRC | 17:25 | |
| clarkb | lets see if ^ works. I expect that 521596 will fail because it won't have 521327 merged | 17:25 |
| clarkb | if thats the case I think there is a way to solve this without a force merge which I'll start working on now | 17:25 |
| *** jpena|brb is now known as jpena | 17:26 | |
| *** dhinesh has joined #openstack-infra | 17:28 | |
| *** bobh has quit IRC | 17:28 | |
| dmsimard | Is there any known recurring issue on OVH clouds ? I know they had some network/power outages last week but they're resolved now, right ? | 17:29 |
| mordred | dmsimard: I'm not aware of any current outage | 17:30 |
| mordred | dmsimard: which does not mean there isn't one - only that I'm not aware of one | 17:30 |
| dmsimard | mordred: it's not an outright outage but seeing flapping tempest tests | 17:30 |
| dmsimard | which are only happening on OVH clouds | 17:30 |
| dmsimard | https://i.imgur.com/Q6M7Q1n.png | 17:30 |
| *** rossella_s has quit IRC | 17:30 | |
| dmsimard | brought it up to #openstack-qa earlier http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2017-11-20.log.html#t2017-11-20T15:35:57 | 17:31 |
| dmsimard | odyssey4me, evrardjp: ^ seems to impact your jobs too, have you been able to pinpoint any problems ? | 17:32 |
| dmsimard | interestingly, not seeing any issues on projects other than OSA and packstack.. maybe it's something specific to how those installers deploy things? :/ | 17:32 |
| dmsimard | It's random/flapping too. | 17:33 |
| *** rossella_s has joined #openstack-infra | 17:33 | |
| clarkb | dmsimard: my guess is nested virt | 17:33 |
| dmsimard | clarkb: pretty sure we set up qemu for the hypervisor but I can double check | 17:33 |
| clarkb | devstack-gate says nope to nested virt but if your installers dont that could be the issue | 17:33 |
| *** bobh has joined #openstack-infra | 17:33 | |
| clarkb | and there is known issue with nested virt in ovh | 17:33 |
| jpena | dmsimard, clarkb: we set up the hypervisor as qemu, I've checked it | 17:34 |
| jpena | let me find the config file | 17:34 |
| dmsimard | virt_type=qemu | 17:34 |
| dmsimard | http://logs.openstack.org/44/517644/13/check/packstack-integration-scenario002-tempest/94ffa8e/logs/etc/nova/nova.conf.txt.gz | 17:34 |
| clarkb | I talked to them about it sydney and they were interested in debugging at least | 17:34 |
| pabelanger | AJaeger: we need to land https://review.openstack.org//520178/ to fix wheel-builds | 17:34 |
| clarkb | dmsimard: check your libvirt logs to make sure it stuck? | 17:34 |
| dmsimard | clarkb: problems with nested virt even with qemu ? | 17:34 |
| clarkb | no qemu should be fine | 17:34 |
| dmsimard | clarkb: there's no kvm hypervisor afaict in http://logs.openstack.org/44/517644/13/check/packstack-integration-scenario002-tempest/94ffa8e/logs/libvirt/qemu/instance-00000002.log.txt.gz | 17:35 |
| dmsimard | maybe something to do with the version of libvirt/qemu/kvm though, but it seems like the problem occurs on both ubuntu and centos so maybe unlikely | 17:36 |
| *** armax has joined #openstack-infra | 17:36 | |
| *** edmondsw has quit IRC | 17:37 | |
| clarkb | we also use UCA on xenial for newer libvirt to avoid a memory bug | 17:37 |
| odyssey4me | dmsimard ovh-gra1 is well known to be the region with the slowest test execution for us | 17:37 |
| odyssey4me | we very often get timeouts from there | 17:37 |
| *** jpich has quit IRC | 17:38 | |
| clarkb | but if it is a problem on centos too then not likely the issue | 17:38 |
| *** edmondsw has joined #openstack-infra | 17:38 | |
| odyssey4me | IIRC it's also a european region, so connectivity to some things is not as good as most other regions | 17:38 |
| dmsimard | odyssey4me: gra1 is in europe yeah | 17:38 |
| *** bobh has quit IRC | 17:38 | |
| dmsimard | odyssey4me: you know, that's actually a good point. Latency over the duration of a build from the executor to the test node probably adds up | 17:39 |
| dmsimard | doesn't explain the tempest failures, but rather the timeouts | 17:39 |
| *** salv-orlando has joined #openstack-infra | 17:39 | |
| dmsimard | or, well, in part -- of course the hardware is probably different :p | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 17:41 |
| *** edmondsw has quit IRC | 17:42 | |
| clarkb | ok cool the puppet-openstackci fixes worked as is, don't need a hack around merge ordering | 17:43 |
| clarkb | thanks for the reviews/approvals too | 17:43 |
| *** bobh has joined #openstack-infra | 17:43 | |
| *** Apoorva has joined #openstack-infra | 17:44 | |
| *** efried is now known as efried_rollin | 17:44 | |
| *** salv-orlando has quit IRC | 17:45 | |
| *** Apoorva has quit IRC | 17:45 | |
| *** rwsu has joined #openstack-infra | 17:45 | |
| *** iyamahat has joined #openstack-infra | 17:46 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Run ovb-container only on pike and after https://review.openstack.org/521330 | 17:47 |
| *** bobh has quit IRC | 17:48 | |
| jeblair | pabelanger, AJaeger, mordred: i suspect we may need to restart the zuul executors to pick up support for override-checkout (based on timestamps, i think we may have missed that by 1 day, or less). that would explain why override-checkout wasn't doing the right thing for tox-py35-on-zuul. it does not explain why override-branch didn't work. | 17:49 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_infra_spec_helper master: Fix installation path in test runner script https://review.openstack.org/521609 | 17:49 |
| pabelanger | ack | 17:49 |
| clarkb | mordred: jeblair ^ is I think the last piece in the puppet-openstackci test fixing. That addresses it for local runs | 17:49 |
| jeblair | pabelanger, AJaeger, mordred: i'd like to proceed by restarting the zuul executors, then re-enabling tox-py35-on-zuul (still with override-checkout) and then seeing what that produces. | 17:50 |
| *** Apoorva has joined #openstack-infra | 17:50 | |
| pabelanger | jeblair: wfm. will refresh coffee and support | 17:52 |
| jeblair | stopping executors now | 17:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Make scenario007-container voting https://review.openstack.org/521614 | 17:55 |
| *** bobh has joined #openstack-infra | 17:56 | |
| jeblair | Shrews: this restart will have all the executors running the threaded socketserver | 17:58 |
| *** Apoorva_ has joined #openstack-infra | 17:59 | |
| *** pcaruana has joined #openstack-infra | 18:00 | |
| *** trown is now known as trown|lunch | 18:01 | |
| *** bobh has quit IRC | 18:01 | |
| *** derekh has quit IRC | 18:01 | |
| *** Apoorva has quit IRC | 18:02 | |
| *** yamahata has joined #openstack-infra | 18:03 | |
| *** bobh has joined #openstack-infra | 18:04 | |
| *** slaweq has joined #openstack-infra | 18:05 | |
| *** yamamoto has joined #openstack-infra | 18:05 | |
| *** slaweq_ has joined #openstack-infra | 18:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add support for warning-is-error to sphinx role https://review.openstack.org/521618 | 18:07 |
| mordred | AJaeger: ^^ I think that should take care of your warning-is-error concern | 18:07 |
| mordred | jeblair: ack on executor restart | 18:07 |
| jeblair | 2 executors had stuck console streaming subprocesses, i've killed them manually | 18:08 |
| jeblair | restarting executors now | 18:08 |
| *** bobh has quit IRC | 18:09 | |
| *** slaweq has quit IRC | 18:10 | |
| *** yamamoto has quit IRC | 18:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Revert "Stop running tox-py35-on-zuul for now" https://review.openstack.org/521622 | 18:12 |
| jeblair | actually.. i bet i can do this speculatively, i'll WIP that for now | 18:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: test tox-py35-on-zuul https://review.openstack.org/521623 | 18:14 |
| clarkb | it looks like we haven't lost so many logstash workers since the karaf exclusion and we are down to 127k logstash jobs (from somewhere like 300l) | 18:15 |
| jeblair | clarkb: what's the karaf exclusion? it sounds like something that starfleet hq would mention in a mission briefing to picard... | 18:16 |
| clarkb | jeblair: screen-karaf.txt matched our new lgostash regex but its really a java process that produces logs of 300MB or so (uncompressed) and was causing logstash workers to OOM | 18:17 |
| clarkb | jeblair: since it is the wrong log format anyways I just excluded it from our regex and things are happy now | 18:17 |
| jeblair | neat! | 18:17 |
| *** bobh has joined #openstack-infra | 18:17 | |
| clarkb | karaf devs can help sort out the debug level filtering and logstash rules for processing their logs properly if they want that | 18:18 |
| *** camunoz has joined #openstack-infra | 18:19 | |
| *** slaweq_ has quit IRC | 18:19 | |
| *** jpena is now known as jpena|off | 18:20 | |
| *** bobh has quit IRC | 18:22 | |
| jeblair | mordred: it looks like the tox-py35-on-zuul job is working as expected after the executor restart. that suggests that the problem with override-checkout was out-of-date executors. i don't know what would have caused the override-branch version of the job to fail, but i haven't looked into it at all (perhaps it was something unrelated and branch checkouts were a red-herring?). do you think it's worth looking into? if so, do you ... | 18:23 |
| jeblair | ... have a link to a failure? | 18:23 |
| clarkb | infra-root and config-core I'm going to draft up that "infra is back and please feel free to annoy us" emial we talked about at the summit (if you weren't at the summit basically an email to dev list letting people know the majority of firefighting is over and they should feel free to reach out to us if they need to) other than the long weekend coming up for those of us in the USA any reason to not | 18:23 |
| clarkb | send that out today? | 18:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add angular to fetch-dependencies.sh https://review.openstack.org/521625 | 18:23 |
| fungi | clarkb: sounds awesome, thanks! | 18:24 |
| jeblair | clarkb: that sounds good except maybe due to long weekend (and possible slackness this week) it would be better to send out in 1 week? | 18:24 |
| mordred | jeblair: I'm happy to consider it a red herring for now - the patch I had that was triggering it should have had nothing to do with it | 18:24 |
| *** bobh has joined #openstack-infra | 18:24 | |
| clarkb | jeblair: ya I can send it out on monday if we feel that is more appropriate | 18:24 |
| fungi | clarkb: jeblair: fair point, i'll be gone again half this week due to travel/family obligations | 18:25 |
| fungi | so supposing that may be the case for many of our usa-based members | 18:25 |
| jeblair | clarkb: (i was just noticing how under capacity we are -- it's a rare monday we can restart executors with virtually no impact -- that made me think about whether activity is lower this wk) | 18:26 |
| clarkb | ya I expect this week to be quieter | 18:26 |
| *** dtantsur is now known as dtantsur|afk | 18:27 | |
| jeblair | http://grafana.openstack.org/dashboard/db/zuul-status?from=1511198840344&to=1511202440344 | 18:27 |
| jeblair | (that shows the restart) | 18:27 |
| *** bobh has quit IRC | 18:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Revert "Stop running tox-py35-on-zuul for now" https://review.openstack.org/521622 | 18:30 |
| jeblair | mordred, AJaeger: ^ that turns it back on | 18:30 |
| *** kgiusti has quit IRC | 18:30 | |
| *** bobh has joined #openstack-infra | 18:32 | |
| *** tosky has quit IRC | 18:35 | |
| *** Apoorva_ has quit IRC | 18:35 | |
| clarkb | hrm http://logs.openstack.org/96/521596/1/gate/legacy-openstackci-beaker-ubuntu-trusty/4a8d981/job-output.txt.gz did end up failing afterall in the gate but not in check? | 18:35 |
| *** Apoorva has joined #openstack-infra | 18:36 | |
| *** e0ne has quit IRC | 18:36 | |
| clarkb | I think that means I may need to get https://review.openstack.org/521609 infirst | 18:36 |
| clarkb | does that means i need to squash things? | 18:36 |
| *** vhosakot has joined #openstack-infra | 18:36 | |
| *** bobh has quit IRC | 18:37 | |
| *** kgiusti has joined #openstack-infra | 18:37 | |
| *** rbrndt has quit IRC | 18:38 | |
| pabelanger | clarkb: or recheck once 521327 lands? | 18:39 |
| clarkb | pabelanger: well 521327 depends on that change | 18:39 |
| openstackgerrit | Luz Cazares proposed openstack-infra/project-config master: Add py35 jobs for refstack-client https://review.openstack.org/520814 | 18:39 |
| pabelanger | +3 521609 | 18:39 |
| clarkb | this is the how to avoid a force merge situation from earlier that I thought I could avoid | 18:39 |
| clarkb | I think I need to squash 521327 and 521609 together | 18:40 |
| clarkb | then that should work on its own | 18:40 |
| pabelanger | kk | 18:40 |
| clarkb | then land 521596 | 18:40 |
| *** salv-orlando has joined #openstack-infra | 18:40 | |
| *** salv-orlando has quit IRC | 18:42 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_infra_spec_helper master: Cap rake below 12.3.0 https://review.openstack.org/521327 | 18:42 |
| *** salv-orlando has joined #openstack-infra | 18:42 | |
| clarkb | if ^ passes tests (I think it should) then we merge that as is then get the zuul side fix in after) | 18:42 |
| *** bobh has joined #openstack-infra | 18:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Install angular.js for zuul web https://review.openstack.org/521630 | 18:44 |
| mordred | clarkb: ++ | 18:44 |
| *** efoley has quit IRC | 18:47 | |
| *** efoley has joined #openstack-infra | 18:47 | |
| *** bobh has quit IRC | 18:47 | |
| clarkb | http://logs.openstack.org/96/521596/1/check/legacy-openstackci-beaker-ubuntu-trusty/7d22d8d/job-output.txt.gz#_2017-11-20_17_38_30_554124 <- that is the one that ran in check and passed and it installs the repo from source properly | 18:50 |
| *** pkovar has quit IRC | 18:51 | |
| clarkb | is this a weird depends on behavior maybe? | 18:51 |
| clarkb | with check and gate acting differently? | 18:51 |
| pabelanger | think so | 18:52 |
| dmsimard | fungi: hey, that nobarrier mount flag from last week.. I'm curious, are we setting nobarrier/noatime on nodepool VMs ? Should we ? I don't think we care if an outage corrupts the disk of an ephemeral VM meant for CI purposes. | 18:52 |
| fungi | dmsimard: that's a great q. i can check | 18:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add support for warning-is-error to sphinx role https://review.openstack.org/521618 | 18:53 |
| dmsimard | nobarrier/noatime combined tend to have a significant impact on i/o performance if you don't care about the integrity of the data in case of an outage of the underlying storage | 18:53 |
| *** ldnunes has quit IRC | 18:54 | |
| pabelanger | didn't we consider at one point add eatmydata to nodes to help with IO preformance | 18:55 |
| clarkb | pabelanger: yes, though testing showed it didn't help because we are or were largely cpu bound | 18:55 |
| clarkb | job runtime was the same with and without eatmydata | 18:55 |
| clarkb | for things like nova unitests and tempest | 18:55 |
| pabelanger | cool, never knew we actually tested it | 18:56 |
| dmsimard | eatmydata ? /me google | 18:56 |
| clarkb | pabelanger: mordred ok https://review.openstack.org/#/c/521327/ does pass if I can get new reviews on that that would be great | 18:56 |
| clarkb | I'll recheck the zuul side fix once ^ is in | 18:56 |
| *** Daisy_ has joined #openstack-infra | 18:56 | |
| dmsimard | disables fsync heh | 18:56 |
| pabelanger | +2 | 18:56 |
| clarkb | dmsimard: it uses ldpreload to ya do that | 18:56 |
| fungi | dmsimard: "options: rw,relatime,data=ordered" according to http://logs.openstack.org/99/519699/2/gate/legacy-tempest-dsvm-py35/4c14530/zuul-info/host-info.primary.yaml | 18:57 |
| dmsimard | okay | 18:58 |
| fungi | dmsimard: we could tweak that in our dib elements i suppose | 18:58 |
| dmsimard | I wonder if there's a good way to benchmark with and without, considering there's different providers with different hardware | 18:58 |
| clarkb | dmsimard: iirc we did it on held nodes | 18:59 |
| dmsimard | We're considering running with that in parts of RDO CI because the storage is kind of slow :( | 18:59 |
| clarkb | dmsimard: so the node was fixed | 18:59 |
| dmsimard | clarkb: on real workloads ? or just synthetic fio or something ? | 19:00 |
| clarkb | but ya should be safe to noatime and such on our test images.I can't think of anywhere we use them where we care about the data on them beyond the job running | 19:00 |
| clarkb | dmsimard: real job loads, running tempest and nova unitests and the like | 19:00 |
| *** Daisy_ has quit IRC | 19:00 | |
| dmsimard | and there was no difference ? huh. | 19:00 |
| dmsimard | that sort of surprises me, even if you're CPU bound | 19:00 |
| dmsimard | it has an impact especially on writes, so things like package installation, database migrations and such should run faster | 19:01 |
| dmsimard | ¯\_(ツ)_/¯ | 19:01 |
| *** IRCNickname has joined #openstack-infra | 19:03 | |
| fungi | clarkb: did we actually test nobarrier, or just noatime? | 19:05 |
| clarkb | fungi: we tested eatmydata | 19:05 |
| fungi | ahh | 19:05 |
| *** trown|lunch is now known as trown | 19:05 | |
| clarkb | whcih is ldpreloaded fsync nop | 19:05 |
| fungi | right, i recall that now | 19:05 |
| clarkb | its possible nobarrier and noatime would have a much more appreciable difference? | 19:05 |
| *** bobh has joined #openstack-infra | 19:05 | |
| fungi | unlikely, but i suppose it could be tested | 19:06 |
| fungi | we also use a lot of different providers than we did back then, so what was not a performance gain at the time might be now | 19:06 |
| *** ldnunes has joined #openstack-infra | 19:06 | |
| clarkb | ya | 19:07 |
| mordred | project-config-core: TIL that 'default(omit)' only works with module args, not other variable setting or substitution places | 19:07 |
| dmsimard | mordred: yes | 19:07 |
| dmsimard | it omits that module arg if there is no value for that var | 19:07 |
| clarkb | mordred: https://review.openstack.org/#/c/521327/ rereview please? I think that will finally fix this issue with puppet testing | 19:07 |
| dmsimard | but using omit in defaults/main.yml, for exmaple, doesn't work | 19:07 |
| mordred | dmsimard: yah. I now understand this :) | 19:08 |
| dmsimard | ++ | 19:08 |
| mordred | clarkb: done | 19:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 19:09 |
| *** bobh has quit IRC | 19:10 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Stop running tox-py35-on-zuul for now" https://review.openstack.org/521622 | 19:11 |
| jeblair | dmsimard, clarkb, fungi: yes, my recollection is that, after testing eatmydata, we surmised that our providers were using a storage configuration that negated the need for that, but that was a long time ago and newer providers may not follow suit, so reopening this seems worthwhile. | 19:12 |
| clarkb | jeblair: oh ya just like we do with infracloud where we tell kvm to just be happy with writes regardless of whether or not they have hit disk | 19:12 |
| *** tosky has joined #openstack-infra | 19:13 | |
| *** andreas_s has joined #openstack-infra | 19:13 | |
| *** andreas_s has quit IRC | 19:18 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 19:18 |
| *** rossella_s has quit IRC | 19:18 | |
| *** bobh has joined #openstack-infra | 19:20 | |
| *** rossella_s has joined #openstack-infra | 19:21 | |
| *** efoley has quit IRC | 19:21 | |
| dmsimard | ok, I'll figure out where the right place is for those parameters and send a patch | 19:21 |
| dmsimard | noatime/nobarrier is universal, not sure about eatmydata (not in centos, not sure about suse, etc.) | 19:22 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper master: Cap rake below 12.3.0 https://review.openstack.org/521327 | 19:23 |
| *** rlandy|rover is now known as rlandy|rover|bia | 19:24 | |
| *** bobh has quit IRC | 19:24 | |
| mordred | dmsimard: we should totally get libeatmydata added to centos on general principle | 19:25 |
| dmsimard | mordred: it's probably in epel fwiw, just not in base rhel :p | 19:26 |
| * dmsimard looks | 19:26 | |
| dmsimard | huh, nothing even in fedora ? https://koji.fedoraproject.org/koji/search?match=glob&type=package&terms=*eatmydata* | 19:26 |
| dmsimard | first google hit is a fuel bug! https://bugs.launchpad.net/fuel/+bug/1457976 | 19:27 |
| openstack | Launchpad bug 1457976 in Fuel for OpenStack "eatmydata package for centos" [Medium,Invalid] - Assigned to Vladimir Kozhukalov (kozhukalov) | 19:27 |
| *** Keitaro has quit IRC | 19:27 | |
| pabelanger | dmsimard: nosync | 19:27 |
| jeblair | dmsimard, mordred: we probably don't want eatmydata on in general for real -- it might be too invasive to apply to everyone. but it's a great tool in specific cases and very good for this testing so would be good to have available. | 19:27 |
| *** rbrndt has joined #openstack-infra | 19:28 | |
| jeblair | dmsimard: (first hit in your google bubble at least ;) | 19:28 |
| dmsimard | jeblair: hah, surely. | 19:28 |
| mordred | jeblair: yah- agree | 19:29 |
| *** rwsu has quit IRC | 19:30 | |
| fungi | right, the ldpreload eatmydata does could also alter some other behaviors in ways we haven't thought of, while mount options are fairly innocuous | 19:31 |
| *** bobh has joined #openstack-infra | 19:32 | |
| sshnaidm | pabelanger, I did as you asked: https://review.openstack.org/#/c/521500/ | 19:33 |
| *** kjackal has quit IRC | 19:36 | |
| *** kjackal has joined #openstack-infra | 19:36 | |
| *** bobh has quit IRC | 19:37 | |
| dmsimard | sshnaidm, pabelanger: It seems like there is a lot of DNS problems in general, the query I'm using in logstash is -> message:"Name or service not known" AND NOT message:"trigger.you" AND NOT message:"trigger.me" AND NOT message:"somehost" AND NOT message:"hostname:" | 19:37 |
| dmsimard | I briefly mentioned it earlier today, I'm spending a bit of time to try and understand if there is something going on we can address | 19:38 |
| pabelanger | yes, I don't mind enabling debug logs in unbound to help with that | 19:38 |
| dmsimard | pabelanger: in the image for all jobs ? | 19:38 |
| sshnaidm | dmsimard, I was looking by "can not be resolved".. it'll be difficult to understand without unbound logs | 19:38 |
| pabelanger | dmsimard: sure, I don't think it will be a lot of data | 19:38 |
| AJaeger | mordred: http://logs.openstack.org/63/520363/2/check/build-openstack-sphinx-docs/df31762/job-output.txt.gz#_2017-11-20_19_33_19_121402 | 19:39 |
| clarkb | andreaf: fyi https://github.com/pypa/pip/pull/4760 | 19:41 |
| dmsimard | pabelanger: I'll defer to you if you think it's appropriate, I don't have a strong opinion hence not -2 | 19:41 |
| clarkb | dmsimard: pabelanger fwiw last time I dug into this the vast majority of dns lookup errors are expected | 19:42 |
| clarkb | so its quite noisy in logstash unfortunately | 19:42 |
| *** bobh has joined #openstack-infra | 19:42 | |
| dmsimard | clarkb: expected ? | 19:42 |
| clarkb | dmsimard: they are lookups for names that don't exist in dnsso fail | 19:43 |
| dmsimard | clarkb: oh, you mean "normal" errors, not actual legitimate lookup failures | 19:43 |
| clarkb | (because testing) | 19:43 |
| dmsimard | yeah ok | 19:43 |
| dmsimard | clarkb: that's why my query (see above) filters out some "normal" errors, there's still quite a bit in there that seem legitimate. | 19:43 |
| clarkb | I found it much easier to focus on a single fail and go from there | 19:43 |
| clarkb | basically one I new was an error | 19:43 |
| *** Keitaro has joined #openstack-infra | 19:44 | |
| dmsimard | clarkb: stuff like pool.ntp.org, or different yum/apt mirrors | 19:44 |
| clarkb | dmsimard: ya the yum/apt mirrors were actually failing legitimetely too | 19:44 |
| clarkb | dmsimard: there was a non zero number of fedora mirrors that had been removed from dns | 19:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Base build-openstack-releasenotes on build-reno-releasenotes https://review.openstack.org/521144 | 19:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Switch build-openstack-sphinx-docs to build-sphinx-docs https://review.openstack.org/521145 | 19:45 |
| dmsimard | yeah.. I see copr-be.cloud.fedoraproject.org a bit | 19:46 |
| dmsimard | I see what you mean, since logstash pulls down syslog/journalctl, it tends to find what ends up being non-fatal errors | 19:46 |
| smcginnis | So looking at moving some jobs for Cinder. There's irrelevant-files, but is there a way to give a job only _relevant_ files? | 19:46 |
| *** bobh has quit IRC | 19:47 | |
| clarkb | dmsimard: ya though that one is currently in dns at least | 19:47 |
| dmsimard | smcginnis: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.files | 19:47 |
| clarkb | dmsimard: but I distinctly remember finding errors then checking with dig and finding out the records just didn't exist | 19:47 |
| smcginnis | dmsimard: Oh, perfect. Thanks! | 19:47 |
| dmsimard | smcginnis: be wary of using files and irrelevant-files simultaneously, gets a bit confusing | 19:47 |
| clarkb | dmsimard: I wonder if they derped their dns though | 19:47 |
| smcginnis | dmsimard: I'll be careful not to cross the streams. | 19:48 |
| *** Sukhdev has joined #openstack-infra | 19:48 | |
| pabelanger | clarkb: dmsimard: but seems like tripleo is only reporting DNS failures, so think (or guessing) maybe we are just seeing them on centos | 19:49 |
| dmsimard | pabelanger: the logstash query I linked is not centos-exclusive, all distros are represented | 19:49 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DNM, update logging for undercloud containers https://review.openstack.org/518972 | 19:49 |
| *** amoralej is now known as amoralej|off | 19:50 | |
| clarkb | dmsimard: ubuntu-xenial | requests.exceptions.ConnectionError: HTTPConnectionPool(host='netloc', port=80): Max retries exceeded with url: /path/to/file.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd2dcd2ba58>: Failed to establish a new connection: [Errno -2] Name or service not known',)) <- is a funny one that matches your query :) | 19:50 |
| mordred | AJaeger: yah - still crunching on it - getting closer though | 19:51 |
| mordred | AJaeger: also - don'tknow if you saw, but I also put up a patch to handle the warning-is-error issue | 19:51 |
| dmsimard | clarkb: yeah I found it after and edited it out :( | 19:51 |
| clarkb | dmsimard: the ntp errors appear to be legit failures, there is no 0.ubuntu.pool.ntp.org | 19:52 |
| fungi | clarkb: dmsimard: seems to be used as a dummy value throughout http://git.openstack.org/cgit/openstack/glance_store/tree/glance_store/tests/unit/test_http_store.py | 19:52 |
| fungi | the /path/to/file.tar.gz i mean | 19:52 |
| dmsimard | there's a lot of dummy DNS everywhere | 19:52 |
| AJaeger | mordred: saw it - liked what you did ;) | 19:52 |
| fungi | http://git.openstack.org/cgit/openstack/glance_store/tree/glance_store/tests/unit/test_http_store.py#n52 | 19:52 |
| fungi | uri = "http://netloc/path/to/file.tar.gz" | 19:52 |
| clarkb | dmsimard: and no 1.ubuntu.pool.ntp.org | 19:53 |
| dmsimard | clarkb: so is that a config issue in how we're setting up ntp then ? | 19:53 |
| pabelanger | clarkb: dmsimard: IIRC, we don't manage ntp server selection from puppet-ntp. Maybe we should start doing so and force all OSes to use same source | 19:53 |
| clarkb | dmsimard: either that or we hit cnames to that or something | 19:53 |
| dmsimard | pabelanger: puppet-ntp doesn't run in the nodepool VMs though | 19:53 |
| pabelanger | oh right | 19:53 |
| pabelanger | not any more | 19:53 |
| clarkb | dmsimard: its entirely possible for errors like that to be completely usptream of us | 19:54 |
| pabelanger | So, ya, we could update our elements to set same source across all images | 19:54 |
| dmsimard | clarkb: 1.ubuntu.pool.ntp.org works for me | 19:54 |
| dmsimard | http://paste.openstack.org/raw/626851/ | 19:54 |
| dmsimard | anyway, we can enable debug logging in unbound and look at what kind of issue we see in the logs.. | 19:55 |
| clarkb | oh I may have derped lcoal dns | 19:55 |
| pabelanger | yah, if too noise, we can always revert once we fixed dns failures | 19:55 |
| clarkb | dmsimard: ya though keep in mind unbound logging doesn't seem to tell you what went wrong | 19:55 |
| clarkb | it tells you what it is queryign then you ahve to ifner from that | 19:56 |
| dmsimard | I remember filing a bug against dnsmasq in Ubuntu where installing/upgrading dnsmasq would totally kill DNS resolution for a couple seconds | 19:56 |
| clarkb | its annoying | 19:56 |
| dmsimard | maybe there's something similar for unbound | 19:56 |
| pabelanger | clarkb: I'm more wondering if caching on centos isn't working for some reason | 19:56 |
| clarkb | pabelanger: you can query unbound for its cache list | 19:56 |
| dmsimard | pabelanger: we can easily tell by holding a node ? | 19:56 |
| clarkb | dmsimard: ya | 19:56 |
| pabelanger | maybe we should do that first | 19:57 |
| pabelanger | let me see | 19:57 |
| *** kjackal has quit IRC | 19:59 | |
| clarkb | trunk.rdoproject.org has a TTL of 300 so depending on how much that is used in these jobs it may not end up being cached much | 19:59 |
| pabelanger | unbound-control dump_cache I take it? | 19:59 |
| clarkb | pabelanger: that sounds right | 20:00 |
| clarkb | yum.oracle.com has a 20 second ttl | 20:00 |
| clarkb | they are doing global dns to akamai with that looks like | 20:00 |
| *** hasharAway is now known as hashar | 20:00 | |
| dmsimard | I suppose we can raise the TTL, we probably lowered that when we migrated the trunk repos a while back | 20:01 |
| clarkb | search.maven.org is a 60 second ttl | 20:01 |
| pabelanger | github.com is 30 second ttl | 20:01 |
| pabelanger | so, that might explain why we see a lot of DNS failures to github | 20:02 |
| dmsimard | pretty sure ttl is mostly a suggestion anyway and isn't necessarily respected by recursive servers | 20:02 |
| clarkb | copr-be is a 1 hour ttl | 20:02 |
| dmsimard | maybe there's a setting in unbound to disregard TTL, not familiar with it | 20:02 |
| pabelanger | well, dump_cache exists it in 30 secods | 20:02 |
| clarkb | ya unbound will respect the ttl iirc | 20:03 |
| *** rossella_s has quit IRC | 20:03 | |
| clarkb | does seem like a lot of these failures are to things that wouldn't likely be cached | 20:03 |
| clarkb | the exception is copr-be | 20:03 |
| clarkb | or at least not cached for very long | 20:03 |
| dmsimard | pabelanger, clarkb: http://paste.openstack.org/raw/626853/ | 20:03 |
| dmsimard | we could probably enforce a cache-min-ttl | 20:03 |
| dmsimard | of something still short but reasonable, 10 minutes ? | 20:04 |
| clarkb | dmsimard: I'm not sure we want to though, that would just replace dns errors with server doesn't exist errors | 20:04 |
| clarkb | (when github/akamai/etc move things around) | 20:04 |
| hwoarang | fungi: hello! what do you think of https://review.openstack.org/#/c/521508/ in order to restore the mirror sync for openSUSE? | 20:04 |
| dmsimard | clarkb: depends on why the lookup is failing in the first place | 20:05 |
| hwoarang | fungi: the main rsync.o.o is still down and I couldn't obtain an ETA for the fix | 20:05 |
| dmsimard | clarkb: if it's failing because of rate limiting, for example, keeping our cache for longer would lower the requests sent out | 20:05 |
| dmsimard | there's also configuration to not cache failures | 20:06 |
| dmsimard | at least in powerdns/bind | 20:06 |
| pabelanger | I was thinking more like 300 min ttl, but ya, agree it could cause issues that clarkb pointed out | 20:06 |
| dmsimard | pabelanger: powerdns and bind have settings for not caching failures, trying to see if there's one for unbound | 20:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Half-Revert "Revert "Add ensure-reno and ensure-babel roles"" https://review.openstack.org/521558 | 20:07 |
| dmsimard | yeah there is, val-bogus-ttl: http://paste.openstack.org/raw/626855/ | 20:08 |
| dmsimard | The time interval prevents repeated revalidation of bogus data. | 20:08 |
| clarkb | pabelanger: fungi https://review.openstack.org/#/c/520672/ passes tests now that puppet openstackci jobs work. Can you rereview that? I updated the patchset to make the depends on more explicit (which was probably a mistake considering I could recheck now if I hadn't done that) | 20:08 |
| *** jtomasek has quit IRC | 20:09 | |
| dmsimard | This is a bit hard to test since it's embedded in the image. I'd be willing to give it a shot, though. | 20:10 |
| openstackgerrit | Jamey McCabe proposed openstack-infra/irc-meetings master: Updating LCOO schedule to new time and channel https://review.openstack.org/517449 | 20:10 |
| *** e0ne has joined #openstack-infra | 20:11 | |
| *** [HeOS] has quit IRC | 20:11 | |
| *** gildub has joined #openstack-infra | 20:12 | |
| pabelanger | dmsimard: thanks | 20:12 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Check for presence of Zuul with Zuul v3 https://review.openstack.org/521596 | 20:12 |
| dmsimard | pabelanger: we can do a small scale experiment just to test our theory by increasing the TTL for trunk.rdo ? | 20:12 |
| dmsimard | I can totally put it at 14400 | 20:13 |
| *** dsariel has joined #openstack-infra | 20:13 | |
| dmsimard | Otherwise I'd tend to do this modification outside of the nodepool image (base job?) in order to make it easy to revert if need be | 20:14 |
| *** claudiub|2 has quit IRC | 20:14 | |
| pabelanger | dmsimard: are you seeing timeouts on trunk.rdo? | 20:14 |
| clarkb | dmsimard: an hour should be plenty given the length of our jobs | 20:15 |
| dmsimard | Some of the DNS lookup failures are on trunk.rdoproject.org, yeah | 20:15 |
| clarkb | dmsimard: basically any ttl longer than a job's lifetime doesn't help any as far as unbound caching since the node will be deleted | 20:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| pabelanger | dmsimard: clarkb: okay, yah. Lets bump TTL on trunk.rdo DNS to 1 hour | 20:15 |
| dmsimard | clarkb: oh, sure, it's just that 14400 is the "default" for our DNS records | 20:15 |
| pabelanger | and see how that runs for the next few days | 20:15 |
| clarkb | last time we had dns errors I did increase the TTLs on our names that were failing and it seemed to help | 20:15 |
| pabelanger | FWIW: cache-min-ttl: 300 does work as expected | 20:16 |
| mordred | AJaeger: I rechecked https://review.openstack.org/#/c/521145 after pushing https://review.openstack.org/521142 above - based on where/how it failed last time, I think we're really close ... | 20:16 |
| pabelanger | github.com was cached to 300 in unbound | 20:16 |
| *** [HeOS] has joined #openstack-infra | 20:17 | |
| dmsimard | pabelanger: yeah, val-bogus-ttl should control the duration of failure cache too | 20:19 |
| dmsimard | defaults to 60 apparently | 20:19 |
| *** salv-orlando has quit IRC | 20:24 | |
| *** salv-orlando has joined #openstack-infra | 20:25 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:25 | |
| *** rlandy|rover|bia is now known as rlandy|rover | 20:25 | |
| clarkb | hwoarang: +2 to the rsync host chagne for opensuse mirroring | 20:27 |
| fungi | hwoarang: seems fine. i didn't test, but i assume ftp.gwdg.de supports rsync protocol? | 20:28 |
| *** salv-orlando has quit IRC | 20:28 | |
| *** edmondsw has joined #openstack-infra | 20:29 | |
| AJaeger | mordred: curious to see how it looks tomorrow - I'm signing off now. | 20:29 |
| * AJaeger waves good-bye | 20:29 | |
| *** bobh has joined #openstack-infra | 20:30 | |
| *** edmondsw has quit IRC | 20:33 | |
| *** bobh has quit IRC | 20:35 | |
| *** salv-orlando has joined #openstack-infra | 20:36 | |
| *** pcaruana has quit IRC | 20:39 | |
| clarkb | pabelanger: dmsimard I think if we want to increase the min ttl we should be careful to watch and see that we don't have a spike in errors talking to the servers after. It is entirely possible that orgs like github recycle their http servers frequently | 20:39 |
| dmsimard | clarkb: yeah we don't need to put a ridiculous TTL either, but something reasonable higher than 60 seconds | 20:40 |
| dmsimard | I'm just getting the ack before raising the TTL on rdoproject.org and it'll give us a good idea as to whether or not it helps | 20:41 |
| clarkb | dmsimard: sounds good | 20:41 |
| dmsimard | Also, the frequency at which they recycle their servers that end up failing one of our jobs is probably lower than current DNS failures | 20:42 |
| pabelanger | clarkb: dmsimard: I've added to comments on https://bugs.launchpad.net/tripleo/+bug/1721702 which related to our discussions | 20:42 |
| openstack | Launchpad bug 1721702 in tripleo "Possible DNS issue in the CI" [Medium,Triaged] | 20:42 |
| *** efried_rollin is now known as efried | 20:43 | |
| *** bobh has joined #openstack-infra | 20:43 | |
| jeblair | clarkb: can you +3 516110 please? | 20:44 |
| openstackgerrit | Merged openstack-infra/system-config master: modules: mirror: Use alternative openSUSE mirror https://review.openstack.org/521508 | 20:47 |
| clarkb | jeblair: done | 20:47 |
| openstackgerrit | Merged openstack-infra/puppet-log_processor master: Reduce log worker internal queue size https://review.openstack.org/520672 | 20:47 |
| *** bobh has quit IRC | 20:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 20:49 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Remove legacy-cinder-tox-* jobs from Cinder https://review.openstack.org/521655 | 20:49 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-cinder-tox-* jobs https://review.openstack.org/521656 | 20:50 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Updating LCOO schedule to new time and channel https://review.openstack.org/517449 | 20:50 |
| clarkb | fungi: https://review.openstack.org/#/c/520672/ was one you previously reviewed with an almost completely trivial patchset update if you want to give that another look | 20:50 |
| hwoarang | fungi: yeah it supports rsync | 20:52 |
| hwoarang | clarkb: thank you! | 20:52 |
| hwoarang | fungi: let me know if get any logs when that script runs again | 20:53 |
| hwoarang | thank you | 20:53 |
| pabelanger | clarkb: looking at some old patches, mind looking at https://review.openstack.org/461223/ removes some SSH keys from old root users | 20:54 |
| *** rwsu has joined #openstack-infra | 20:54 | |
| clarkb | pabelanger: will puppet work with empty string ssh keys like that? | 20:55 |
| clarkb | pabelanger: I think we use the ssh key resource and I'm wondering if it will error | 20:55 |
| * clarkb digs up docs | 20:56 | |
| openstackgerrit | Merged openstack-infra/shade master: Complete move of baremetal machine tests https://review.openstack.org/502333 | 20:56 |
| *** felipemonteiro has joined #openstack-infra | 20:57 | |
| fungi | hwoarang: sure, in the meantime i guess timestamps at http://mirror.gra1.ovh.openstack.org/opensuse/update/leap/42.3/oss/ should update to something newer than 2017-11-05 once it's working | 20:57 |
| clarkb | pabelanger: I think that will result in key lines that say 'ssh-rsa foo@bar' which is fine I guess especially since its for only that user | 20:58 |
| clarkb | docs say only invalid content is whitespace | 20:58 |
| clarkb | and we disabled those users | 20:58 |
| clarkb | I've approved the change | 20:58 |
| pabelanger | kk | 20:59 |
| *** rwsu has quit IRC | 21:00 | |
| *** jkilpatr_ has joined #openstack-infra | 21:00 | |
| *** dprince has quit IRC | 21:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Normalize daemon process handling https://review.openstack.org/517381 | 21:00 |
| *** jkilpatr has quit IRC | 21:01 | |
| *** rwsu has joined #openstack-infra | 21:01 | |
| *** jkilpatr_ has quit IRC | 21:01 | |
| *** jkilpatr_ has joined #openstack-infra | 21:01 | |
| openstackgerrit | Jeremy Stanley proposed openstack-dev/pbr master: Support Description-Content-Type metadata https://review.openstack.org/521658 | 21:02 |
| *** Apoorva_ has joined #openstack-infra | 21:02 | |
| *** krtaylor has quit IRC | 21:03 | |
| fungi | clarkb: i already reviewed/approved 520672 and it merged at 20:47z according to scrollback | 21:04 |
| *** bobh has joined #openstack-infra | 21:04 | |
| clarkb | fungi: bah sorry I didn't refresh browser properly | 21:05 |
| fungi | np, just didn't want you waiting forever thinking it wasn't merged | 21:05 |
| *** rwsu has quit IRC | 21:05 | |
| *** rwsu has joined #openstack-infra | 21:05 | |
| *** rwsu has quit IRC | 21:06 | |
| *** Apoorva has quit IRC | 21:06 | |
| pabelanger | jeblair: do you mind adding https://review.openstack.org/514483/ to your review pipeline, that's our first step at removing zuul-env from DIB images. cc clarkb (you've voted) | 21:06 |
| *** rwsu has joined #openstack-infra | 21:06 | |
| jeblair | pabelanger: wfm | 21:09 |
| *** fultonj has left #openstack-infra | 21:09 | |
| *** bobh has quit IRC | 21:09 | |
| openstackgerrit | Merged openstack/os-client-config master: Remove setting of version/release from releasenotes https://review.openstack.org/520790 | 21:10 |
| *** rfolco has quit IRC | 21:11 | |
| *** e0ne has quit IRC | 21:12 | |
| *** erlon has joined #openstack-infra | 21:12 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add docs to replace a cinder volume https://review.openstack.org/449232 | 21:13 |
| *** bobh has joined #openstack-infra | 21:14 | |
| *** rhallisey has quit IRC | 21:15 | |
| openstackgerrit | Merged openstack-infra/puppet-exim master: Process mailman addresses last https://review.openstack.org/484989 | 21:15 |
| *** krtaylor has joined #openstack-infra | 21:16 | |
| *** dave-mccowan has quit IRC | 21:18 | |
| *** bobh has quit IRC | 21:18 | |
| *** kgiusti has quit IRC | 21:19 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove SSH keys from oubiwann and rockstar (disabled accounts) https://review.openstack.org/461223 | 21:19 |
| *** florianf has quit IRC | 21:20 | |
| *** claudiub|2 has joined #openstack-infra | 21:20 | |
| *** bobh has joined #openstack-infra | 21:23 | |
| *** eumel8 has quit IRC | 21:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 21:26 |
| clarkb | puppet is updating the logstash worker code on the worker nodes now. I'm going to restart the services on them shortly to pick up the more memory friendly queue size change | 21:27 |
| *** bobh has quit IRC | 21:28 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Enable support for Gentoo overlays https://review.openstack.org/516823 | 21:28 |
| *** rwsu has quit IRC | 21:29 | |
| pabelanger | clarkb: ack | 21:36 |
| *** Sukhdev has quit IRC | 21:41 | |
| *** felipemonteiro has quit IRC | 21:44 | |
| *** eharney has quit IRC | 21:45 | |
| *** linkmark has joined #openstack-infra | 21:45 | |
| *** bobh has joined #openstack-infra | 21:46 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Collect unbound DNS server configs and logs https://review.openstack.org/520007 | 21:48 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Set ubound to debug level and dump cache https://review.openstack.org/520055 | 21:48 |
| *** rcernin has joined #openstack-infra | 21:48 | |
| dmsimard | clarkb, pabelanger: TTL on rdoproject.org raised from 300 to 3600 | 21:49 |
| *** bobh has quit IRC | 21:50 | |
| openstackgerrit | Merged openstack-infra/puppet-haveged master: Add bindep.txt file https://review.openstack.org/461568 | 21:51 |
| *** aeng has joined #openstack-infra | 21:52 | |
| *** bobh has joined #openstack-infra | 21:54 | |
| *** dsariel has quit IRC | 21:55 | |
| pabelanger | dmsimard: ++ | 21:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Normalize daemon process handling https://review.openstack.org/517381 | 21:55 |
| clarkb | ok I've restarted all the logstash workers | 21:56 |
| *** bobh has quit IRC | 21:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 21:59 |
| jeblair | pabelanger, clarkb: i'm planning on moving the zuul pidfile locations in 517381. puppet-zuul init scripts hardcode those, and there's no v3 switch on that yet | 21:59 |
| *** mat128 has quit IRC | 21:59 | |
| *** bobh has joined #openstack-infra | 22:00 | |
| jeblair | how should we proceed? maybe template those in puppet and set the variable based on the v3 flag? | 22:00 |
| clarkb | jeblair: or maybe use /etc/default/zuul ? | 22:00 |
| clarkb | (puppet can write that out, but then its not hardcoded in the init script on disk) | 22:00 |
| jeblair | oh hm | 22:00 |
| pabelanger | looking | 22:01 |
| jeblair | that should just override the current 'default' value too | 22:01 |
| jeblair | later we can change the default and drop it | 22:01 |
| pabelanger | clarkb: we'd need per service default files for that right? currently all source /etc/default/zuul | 22:02 |
| pabelanger | and would share same PID | 22:02 |
| clarkb | or per service variables for the path | 22:03 |
| clarkb | ZUUL_EXECUTOR_PID ZUUL_SCHEDULER_PID and so on | 22:03 |
| jeblair | actually | 22:03 |
| jeblair | all v3 services source /etc/default/zuul-foo | 22:03 |
| clarkb | even better | 22:03 |
| jeblair | only the v2 scheduler sources /etc/default/zuul | 22:03 |
| pabelanger | yah, that is right | 22:03 |
| pabelanger | so, that should work if we add PIDFILE | 22:03 |
| jeblair | i'll write up a change real quick | 22:04 |
| clarkb | is it zuul meeting time now too? | 22:04 |
| jeblair | derp yes | 22:04 |
| jeblair | in #openstack-meeting-alt | 22:04 |
| *** hashar has quit IRC | 22:06 | |
| *** Apoorva_ has quit IRC | 22:06 | |
| *** shardy has quit IRC | 22:06 | |
| *** trown is now known as trown|outtypewww | 22:07 | |
| *** rwsu has joined #openstack-infra | 22:10 | |
| *** rossella_s has joined #openstack-infra | 22:10 | |
| *** Goneri has quit IRC | 22:12 | |
| *** bobh has quit IRC | 22:13 | |
| *** e0ne has joined #openstack-infra | 22:16 | |
| *** edmondsw has joined #openstack-infra | 22:17 | |
| *** rtjure has quit IRC | 22:21 | |
| *** edmondsw has quit IRC | 22:22 | |
| *** rtjure has joined #openstack-infra | 22:23 | |
| *** jamesmcarthur has joined #openstack-infra | 22:24 | |
| mordred | AJaeger: ZOMG. latest iteration works for both docs and releasenotes: | 22:25 |
| mordred | AJaeger: http://logs.openstack.org/54/521254/2/check/build-openstack-sphinx-docs/0cfe4ac/job-output.txt.gz | 22:25 |
| *** jkilpatr_ has quit IRC | 22:26 | |
| fungi | ooh! | 22:31 |
| *** jlk has quit IRC | 22:31 | |
| *** jlk has joined #openstack-infra | 22:32 | |
| *** jlk has quit IRC | 22:32 | |
| *** jlk has joined #openstack-infra | 22:32 | |
| *** e0ne has quit IRC | 22:32 | |
| mordred | fungi: right? there is still one thing that's not behaving as I expect from looking at the output- but it's not a fatal error, so I think we're VERY close | 22:33 |
| *** e0ne has joined #openstack-infra | 22:33 | |
| *** jaypipes has quit IRC | 22:34 | |
| *** jamesmcarthur has quit IRC | 22:36 | |
| *** jamesmcarthur has joined #openstack-infra | 22:38 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Add subunit2sql check database server and workers https://review.openstack.org/520192 | 22:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add general sphinx and reno jobs and role https://review.openstack.org/521142 | 22:38 |
| *** Apoorva has joined #openstack-infra | 22:39 | |
| *** Apoorva has quit IRC | 22:40 | |
| *** Apoorva has joined #openstack-infra | 22:40 | |
| *** pilgrimstack has joined #openstack-infra | 22:44 | |
| *** jamesmcarthur has quit IRC | 22:46 | |
| *** rwsu has quit IRC | 22:47 | |
| *** jamesmcarthur has joined #openstack-infra | 22:48 | |
| *** Sukhdev has joined #openstack-infra | 22:48 | |
| *** e0ne has quit IRC | 22:49 | |
| *** d0ugal has quit IRC | 22:50 | |
| *** lbragstad has quit IRC | 22:51 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Revert "Install ssh keys of tripleo ci admins on nodes" https://review.openstack.org/521667 | 22:54 |
| *** raissa has quit IRC | 22:54 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Make build-python-release job https://review.openstack.org/513925 | 22:56 |
| *** pilgrimstack has quit IRC | 22:56 | |
| *** aeng has quit IRC | 23:02 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci master: Add help links to /var/lib/mistral https://review.openstack.org/521671 | 23:03 |
| *** jkilpatr_ has joined #openstack-infra | 23:03 | |
| *** lbragstad has joined #openstack-infra | 23:04 | |
| mordred | infra-root, project-config-core: the first patches in updated-pti have now passed their own tests plus depends-on changes | 23:05 |
| *** wolverineav has quit IRC | 23:05 | |
| *** wolverineav has joined #openstack-infra | 23:05 | |
| *** bobh has joined #openstack-infra | 23:06 | |
| ianw | mordred: do we have one that has graphviz-y type magic in it? | 23:07 |
| *** d0ugal has joined #openstack-infra | 23:07 | |
| *** Goneri has joined #openstack-infra | 23:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update fetch sphinx output to use sphinx vars https://review.openstack.org/521590 | 23:09 |
| *** sdague has quit IRC | 23:09 | |
| mordred | ianw: not sure - we've got nova and neutron and a networking-bgpvpn patch | 23:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add support for warning-is-error to sphinx role https://review.openstack.org/521618 | 23:11 |
| mordred | hrm - we should maybe add a depends-on on the warning-is-error patch ^^ - lemme do that real quick and recheck again | 23:12 |
| ianw | ok nova at least has some funky graphs so if that's coming out that's good : http://git.openstack.org/cgit/openstack/nova/tree/doc/source/reference/vm-states.rst | 23:13 |
| *** jrist is now known as jrist-afk | 23:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add support for warning-is-error to sphinx role https://review.openstack.org/521618 | 23:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Switch build-openstack-sphinx-docs to build-sphinx-docs https://review.openstack.org/521145 | 23:15 |
| mordred | ianw: cool. so - I updated theopenstack-zuul-jobs depends-on to include the patch that adds support for warning-is-error - if that passes through the various rechecks, I'll ping folkks again ... | 23:16 |
| mordred | that said - https://review.openstack.org/521142 and https://review.openstack.org/#/c/521144 are both ready for review | 23:16 |
| *** jamesmcarthur has quit IRC | 23:18 | |
| *** aeng has joined #openstack-infra | 23:20 | |
| *** yamamoto has joined #openstack-infra | 23:21 | |
| *** marst has quit IRC | 23:21 | |
| *** d0ugal has quit IRC | 23:24 | |
| *** rossella_s has quit IRC | 23:26 | |
| *** rossella_s has joined #openstack-infra | 23:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Zuul v3: add per-service default files https://review.openstack.org/521680 | 23:27 |
| jeblair | clarkb, pabelanger: ^ | 23:28 |
| jeblair | we probably want to merge those around the same time, as there's no cross-compatability | 23:28 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Stop indexing libvirt logs https://review.openstack.org/521681 | 23:28 |
| clarkb | config-core ^ another logstash throughput cleanup related change | 23:29 |
| *** mriedem is now known as mriedem_away | 23:36 | |
| ianw | mordred: sorry to harp on about this contraints file missing thing, just it already bit me several times | 23:37 |
| *** linkmark has quit IRC | 23:38 | |
| ianw | mordred: and is there a reason not to use dependencies: ? it's all tightly coupled so it seems like it might be a clearer expression of what's going on | 23:39 |
| *** rossella_s has quit IRC | 23:40 | |
| *** Sukhdev has quit IRC | 23:40 | |
| clarkb | config-core infra-root https://etherpad.openstack.org/p/infra-team-back-in-office thinking about something like that for sending out on monday (a week from today) | 23:41 |
| mordred | ianw: wait - which thing? I feel like I maybe missed a thing you're harping on? | 23:41 |
| ianw | mordred: https://review.openstack.org/#/c/521142/32/roles/find-constraints/tasks/main.yaml | 23:42 |
| jeblair | clarkb: ++ | 23:42 |
| mordred | ianw: ah - nod. so - we could certianly add an assertion to make sure of the thing you mention there (and good point) | 23:44 |
| mordred | ianw: for the other thing, you mean in meta/main.yaml? | 23:48 |
| ianw | mordred: yeah, it's just that say build-releasenotes is really quite dependent on having ensure-sphinx run? i mean, i guess you could use it there if you had sphinx & gettext etc installed some other way, but it doesn't seem the intention | 23:49 |
| mordred | ianw: oh- well - those two are separate because one is for pre-run and one is for run | 23:50 |
| ianw | ahhh | 23:50 |
| ianw | yeah forgot about that | 23:51 |
| mordred | ianw: I definitely grok/agree with the urge :) | 23:51 |
| clarkb | I've got to pop out shortly but review on https://review.openstack.org/#/c/521681/ would make my day (I think that may be last major piece to reliable logstash indexing) | 23:51 |
| *** aeng has quit IRC | 23:51 | |
| mordred | clarkb: +2 from me | 23:53 |
| fungi | clarkb: announcement for monday lgtm | 23:54 |
| *** wolverineav has quit IRC | 23:56 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update Fedora defaults to 27 https://review.openstack.org/521684 | 23:56 |
| *** armax has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
