| *** agopi_ is now known as agopi | 00:04 | |
| tonyb | It looks like the bas SQLAlchemy is back ? http://logs.openstack.org/85/611885/1/check/requirements-tox-py36-check-uc/c5772a6/job-output.txt.gz#_2018-10-21_21_47_41_775188 | 00:09 |
|---|---|---|
| tonyb | and the wheel build went 'wonky' fro openstacksdk http://logs.openstack.org/85/611885/1/check/cross-horizon-py27/85785fd/job-output.txt.gz#_2018-10-21_21_47_23_414691 | 00:09 |
| *** dave-mccowan has quit IRC | 00:14 | |
| *** pcrews has joined #openstack-infra | 00:14 | |
| ianw | tonyb: hrm ... did they run in parallel ok is first question | 00:18 |
| tonyb | ianw: I don't know | 00:18 |
| ianw | having a look... | 00:20 |
| ianw | this would be the bionic build i guess | 00:21 |
| tonyb | ianw: Yeah I guess so for py36 | 00:21 |
| ianw | ok, logs would be http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-ubuntu-bionic/6c88da7/ | 00:22 |
| ianw | hrm, not sure about this comment any more -> https://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/copy-wheels/tasks/main.yaml#n6 | 00:25 |
| ianw | hrm, seemed to run sequentially. :/ | 00:27 |
| tonyb | phooey | 00:28 |
| ianw | zip gives up on the file, with that index error or whatever | 00:29 |
| ianw | 7za however says | 00:29 |
| ianw | WARNINGS: | 00:29 |
| ianw | There are data after the end of archive | 00:29 |
| ianw | Physical Size = 88761 | 00:29 |
| ianw | Tail Size = 22 | 00:29 |
| tonyb | Oh wait, the bionic build logs are for 0.17.2 but the version shoudl have been 0.18.0 (I think) | 00:30 |
| *** dave-mccowan has joined #openstack-infra | 00:31 | |
| ianw | tonyb: what is that you're looking at the versions of? | 00:32 |
| tonyb | ianw: Yeah for the openstacksdk thing the chnage under review is https://review.openstack.org/#/c/611885/ to update from 0.17.2 -> 0.18.0 but the build logs you linked to above shows it building 0.17.2 | 00:32 |
| ianw | interesting ... 7za has a good mode where it will parse things for you. it splits out the wheel zip file, and then there's an extra "something" -> http://paste.openstack.org/show/732562/ | 00:33 |
| tonyb | ianw: So the tag job for openstacksdk shoudl have built and publishd the 0.18.o wheel right? | 00:33 |
| ianw | which looks mighty like an empty .zip header tagged onto the end of the .whl | 00:34 |
| tonyb | ianw: Yeah that is what it looks like | 00:34 |
| ianw | tonyb: i'm not sure where you're seeing 0.17.2 in http://logs.openstack.org/85/611885/1/check/requirements-tox-py36-check-uc/c5772a6/job-output.txt.gz#_2018-10-21_21_47_41_775188 ? | 00:37 |
| tonyb | ianw: nm I think I'm confusing myself | 00:38 |
| tonyb | ianw: I shouldn't have dropped 2 pronlems at the same time. | 00:39 |
| ianw | yeah, if you drop the last 22 bytes with "dd if=../SQLAlchemy_Utils-0.33.6-py2.py3-none-any.whl of=test.whl bs=1 count=88761" you have a valid wheel | 00:40 |
| tonyb | ianw: Okay nice find. | 00:44 |
| *** jamesmcarthur has joined #openstack-infra | 00:45 | |
| *** longkb has joined #openstack-infra | 00:46 | |
| ianw | hrm, i dunno. pkzip headers should be 50 4b 03 04 ... we have 50 4b 05 06 | 00:46 |
| tonyb | ianw: https://en.wikipedia.org/wiki/List_of_file_signatures lists 50 4b 05 06 as a zipfile | 00:55 |
| tonyb | ianw: but doesn't explain why it'd be different | 00:56 |
| tonyb | ianw: Ahh but https://en.wikipedia.org/wiki/Zip_(file_format) lists the 22 bytes for an empty zip file | 00:57 |
| ianw | hrm, yeah under that is "(empty archive)" | 00:58 |
| *** jamesmcarthur has quit IRC | 00:58 | |
| ianw | yeah, on that page it has PK\x05\x06 (empty) as the magic number | 00:59 |
| tonyb | Ahh okay | 00:59 |
| tonyb | I missed that | 00:59 |
| ianw | well everything matches up with it being an empty zip file, appended on the end of the wheel | 01:01 |
| tonyb | ianw: Yup, but i have no idea why | 01:02 |
| ianw | hrm, me either, but at least we have a suspect now ... | 01:02 |
| ianw | obvious searches for similar bugs are not being fruitful atm | 01:03 |
| *** jamesmcarthur has joined #openstack-infra | 01:04 | |
| tonyb | 2~Looking at logstash seems to have started on Oct 16th .. unless that's just the begining of the retention period | 01:05 |
| ianw | ok, didn't ring any bels for dstufft in pypa, so it's unlikely to be something known i guess | 01:09 |
| *** dave-mccowan has quit IRC | 01:10 | |
| ianw | i can't replicate it, i think we need to bring up an infra node and try it in situ to see what might be different | 01:11 |
| tonyb | Yeah. sounds good. I was goign to suggest added some additional logging to the build process ... nut sure what yet apart from SHAs for the generated wheels and zip -T or similar | 01:14 |
| ianw | it's so specifically corrupt, with the blank file right at the end, it doesn't seem like it could be random | 01:15 |
| ianw | i wonder, if i put a failed hold on the job, kick it off, then manually go in and kill the connection, hopefully the nodes will be held. that's probably the "easiest" way | 01:16 |
| tonyb | ianw: That's deep magic of which I knwo notthing ;P | 01:17 |
| ianw | :) just trying to ensure we get some nodes that look exactly like where this happens | 01:18 |
| *** mrsoul has joined #openstack-infra | 01:19 | |
| *** imacdonn has quit IRC | 01:23 | |
| *** felipemonteiro has joined #openstack-infra | 01:23 | |
| ianw | ok, i got a hold on the bionic job and enqueued it | 01:23 |
| *** imacdonn has joined #openstack-infra | 01:23 | |
| ianw | 198.72.124.24 & 198.72.124.239 are the two hosts | 01:23 |
| *** bobh has quit IRC | 01:24 | |
| *** dave-mccowan has joined #openstack-infra | 01:25 | |
| ianw | ok i think i got the job to die, hopefully the nodes hold | 01:37 |
| ianw | sigh, building it almost exactly the same way, no error | 01:43 |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** bobh has joined #openstack-infra | 01:43 | |
| tonyb | :( | 01:46 |
| *** bobh has quit IRC | 01:47 | |
| *** bobh has joined #openstack-infra | 01:51 | |
| ianw | interesting, the freshly built one also has the same empty zip at the end | 01:54 |
| ianw | what's a bit weird is i get 3 wheels when building sqlalchemy_utils | 01:56 |
| ianw | -rw-rw-r-- 1 zuul zuul 1.1M Oct 22 01:44 SQLAlchemy-1.2.12-cp36-cp36m-linux_x86_64.whl | 01:56 |
| ianw | -rw-rw-r-- 1 zuul zuul 87K Oct 22 01:44 SQLAlchemy_Utils-0.33.6-py2.py3-none-any.whl | 01:56 |
| ianw | -rw-rw-r-- 1 zuul zuul 11K Oct 22 01:44 six-1.11.0-py2.py3-none-any.whl | 01:56 |
| *** jamesmcarthur has joined #openstack-infra | 01:57 | |
| *** annp has joined #openstack-infra | 01:58 | |
| ianw | tonyb: do you happen to have everything checked out? | 02:02 |
| ianw | by everything, i mean everything in requirements. i'm wondering if two things are running in parallel and end up making the same wheel file | 02:02 |
| *** jamesmcarthur has quit IRC | 02:03 | |
| tonyb | ianw: I have things cloned locally and mostly on master but I can pretty much checkout whatever is needed to test it | 02:03 |
| ianw | hrm, probably the logs tell us ... cross-grepping SQLAlchemy_Utils.*.whl ... hrm | 02:04 |
| tonyb | That is strange though getting multiple wheels | 02:04 |
| ianw | i'm not sure we expected "pip wheel" to also output other wheels | 02:05 |
| ianw | Build Wheel archives for your requirements and dependencies | 02:06 |
| ianw | :( AFAICS, nothing else has a dependency on SQLAlchemy-Utils | 02:11 |
| tonyb | ianw: Probably not directly ... let me try to work it out ... | 02:12 |
| tonyb | ianw: aodh certainly does ... look in setup,cfg | 02:14 |
| tonyb | http://git.openstack.org/cgit/openstack/aodh/tree/setup.cfg | 02:14 |
| *** ykarel|away has joined #openstack-infra | 02:15 | |
| ianw | tonyb: yeah, i mean just in the wheel build process, however | 02:16 |
| ianw | so of the things in upper-constraints.txt, nothing else seems to depend on it | 02:17 |
| *** felipemonteiro has quit IRC | 02:17 | |
| ianw | which seems to suggest that even running all the wheel builds in parallel, the file shouldn't be created twice | 02:18 |
| *** hongbin has joined #openstack-infra | 02:19 | |
| ianw | the empty zip entry at the end seems to maybe be a red herring, it seems to always be there | 02:19 |
| tonyb | ianw: Hmm okay ... I get | 02:19 |
| tonyb | Package : sqlalchemy-utils [sqlalchemy-utils] (used by 4 projects) | 02:19 |
| tonyb | Re-Release : 1 projects | 02:19 |
| tonyb | openstack/taskflow [cycle-with-intermediary] | 02:19 |
| tonyb | Included in : 2 projects | 02:19 |
| tonyb | openstack/aodh [cycle-with-intermediary] | 02:19 |
| tonyb | openstack/panko [cycle-with-intermediary] | 02:19 |
| tonyb | Also affects : 1 projects | 02:19 |
| tonyb | openstack/molteniron [None] | 02:19 |
| tonyb | but none of them are actually in requirements.txt | 02:20 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add support for enabling the ARA callback plugin in install-ansible https://review.openstack.org/611228 | 02:20 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add playbook for deploying the ARA web application https://review.openstack.org/611232 | 02:20 |
| *** felipemonteiro has joined #openstack-infra | 02:31 | |
| *** felipemonteiro has quit IRC | 02:31 | |
| *** hongbin has quit IRC | 02:35 | |
| *** hongbin has joined #openstack-infra | 02:35 | |
| *** bobh has quit IRC | 02:37 | |
| *** haleyb has quit IRC | 02:37 | |
| *** bobh has joined #openstack-infra | 02:41 | |
| *** hongbin_ has joined #openstack-infra | 02:41 | |
| *** hongbin has quit IRC | 02:43 | |
| *** psachin has joined #openstack-infra | 02:54 | |
| ianw | well i have now built all the wheels manually, under parallel for both python2 & 3 and i can't replicate the corruption | 03:01 |
| tonyb | :( | 03:10 |
| *** dave-mccowan has quit IRC | 03:11 | |
| *** bobh has quit IRC | 03:18 | |
| *** bobh has joined #openstack-infra | 03:20 | |
| *** carl_cai has quit IRC | 03:23 | |
| *** bobh has quit IRC | 03:25 | |
| *** felipemonteiro has joined #openstack-infra | 03:32 | |
| *** jamesmcarthur has joined #openstack-infra | 03:41 | |
| *** jamesmcarthur has quit IRC | 03:46 | |
| *** ykarel|away is now known as ykarel | 03:50 | |
| *** zhangfei has joined #openstack-infra | 03:54 | |
| *** udesale has joined #openstack-infra | 03:55 | |
| *** udesale has quit IRC | 03:55 | |
| *** udesale has joined #openstack-infra | 03:56 | |
| AJaeger | ianw: what fun ;( | 04:09 |
| AJaeger | ianw: did you delete the broken wheel - or push a fixed one up? | 04:09 |
| ianw | AJaeger: umm, not yet, i can do, just trying to work on some debugging | 04:09 |
| *** zhangfei has quit IRC | 04:10 | |
| ianw | tonyb: was it only bionic? i've removed that one | 04:20 |
| tonyb | That was the only one I saw fail | 04:21 |
| tonyb | ianw: Thanks for removing it | 04:22 |
| *** hongbin_ has quit IRC | 04:22 | |
| *** ramishra has joined #openstack-infra | 04:23 | |
| *** hongbin has joined #openstack-infra | 04:24 | |
| *** kaiokmo has quit IRC | 04:28 | |
| *** kaiokmo has joined #openstack-infra | 04:30 | |
| *** hongbin has quit IRC | 04:38 | |
| ianw | tonyb: my first thought was we could get pip to help us here and give us the hash of what it outputs ... https://github.com/pypa/pip/pull/5908 | 04:53 |
| *** kaiokmo has quit IRC | 04:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: wheel-build : temporary pip with checksum output https://review.openstack.org/612234 | 05:11 |
| *** apetrich has quit IRC | 05:23 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:29 | |
| *** felipemonteiro has quit IRC | 05:37 | |
| *** apetrich has joined #openstack-infra | 05:39 | |
| *** quiquell|off is now known as quiquell | 06:04 | |
| AJaeger | thanks, ianw | 06:07 |
| *** maciejjozefczyk has joined #openstack-infra | 06:09 | |
| *** dpawlik has joined #openstack-infra | 06:20 | |
| *** ifat_afek has joined #openstack-infra | 06:20 | |
| *** pcaruana has joined #openstack-infra | 06:21 | |
| *** xinliang has joined #openstack-infra | 06:24 | |
| *** psachin has quit IRC | 06:29 | |
| *** janki has joined #openstack-infra | 06:30 | |
| *** fresta_ has joined #openstack-infra | 06:37 | |
| *** fresta has quit IRC | 06:40 | |
| *** jbadiapa has joined #openstack-infra | 06:42 | |
| *** chkumar|off is now known as chandankumar | 06:45 | |
| *** e0ne has joined #openstack-infra | 06:45 | |
| *** yamamoto has quit IRC | 06:49 | |
| *** yamamoto has joined #openstack-infra | 06:49 | |
| *** yamamoto has quit IRC | 06:53 | |
| *** yamamoto has joined #openstack-infra | 06:53 | |
| *** ifat_afek has quit IRC | 06:54 | |
| *** yamamoto has quit IRC | 06:55 | |
| *** ifat_afek has joined #openstack-infra | 06:56 | |
| *** yamamoto has joined #openstack-infra | 06:58 | |
| *** pcaruana has quit IRC | 06:58 | |
| *** rcernin has quit IRC | 06:59 | |
| *** yamamoto has quit IRC | 07:01 | |
| *** yamamoto_ has joined #openstack-infra | 07:01 | |
| *** ccamacho has joined #openstack-infra | 07:03 | |
| *** ginopc has joined #openstack-infra | 07:04 | |
| *** rcernin has joined #openstack-infra | 07:05 | |
| *** rcernin has quit IRC | 07:07 | |
| *** rascasoft has joined #openstack-infra | 07:08 | |
| *** zhangfei has joined #openstack-infra | 07:12 | |
| *** jtomasek has joined #openstack-infra | 07:13 | |
| *** pcaruana has joined #openstack-infra | 07:13 | |
| *** florianf has joined #openstack-infra | 07:19 | |
| *** alexchadin has joined #openstack-infra | 07:29 | |
| *** longkb has quit IRC | 07:29 | |
| *** longkb has joined #openstack-infra | 07:29 | |
| *** xek has joined #openstack-infra | 07:31 | |
| *** jpena|off is now known as jpena | 07:32 | |
| *** udesale has quit IRC | 07:33 | |
| *** udesale has joined #openstack-infra | 07:33 | |
| *** ifat_afek has quit IRC | 07:34 | |
| *** jamesmcarthur has joined #openstack-infra | 07:42 | |
| *** tosky has joined #openstack-infra | 07:42 | |
| *** ifat_afek has joined #openstack-infra | 07:44 | |
| *** jamesmcarthur has quit IRC | 07:46 | |
| *** ifat_afek has quit IRC | 07:48 | |
| *** kjackal has joined #openstack-infra | 07:49 | |
| *** lpetrut has joined #openstack-infra | 07:50 | |
| *** alexchadin has quit IRC | 07:58 | |
| *** rpittau has joined #openstack-infra | 08:02 | |
| *** ifat_afek has joined #openstack-infra | 08:03 | |
| *** shardy has joined #openstack-infra | 08:09 | |
| *** panda has joined #openstack-infra | 08:13 | |
| *** shardy has quit IRC | 08:16 | |
| *** finucannot is now known as stephenfin | 08:20 | |
| *** zhangfei has quit IRC | 08:20 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 08:22 | |
| *** shardy has joined #openstack-infra | 08:31 | |
| *** gfidente has joined #openstack-infra | 08:37 | |
| *** eernst has joined #openstack-infra | 08:38 | |
| *** bobh has joined #openstack-infra | 08:45 | |
| *** yamamoto_ has quit IRC | 08:46 | |
| *** yamamoto has joined #openstack-infra | 08:46 | |
| *** bobh has quit IRC | 08:50 | |
| *** eernst has quit IRC | 08:50 | |
| *** yamamoto_ has joined #openstack-infra | 08:57 | |
| *** ykarel_ has joined #openstack-infra | 08:57 | |
| *** yamamoto has quit IRC | 09:00 | |
| *** ykarel has quit IRC | 09:00 | |
| *** yamamoto_ has quit IRC | 09:01 | |
| *** yamamoto has joined #openstack-infra | 09:01 | |
| ianw | amorin: gra1 seems to be back to not really booting vm's | 09:03 |
| ianw | the last port clearing run cleared out 600+ ports, which is what we see when the vm's aren't booting and cycle quickly | 09:04 |
| *** lpetrut has quit IRC | 09:06 | |
| *** ykarel_ is now known as ykarel | 09:08 | |
| *** slaweq has joined #openstack-infra | 09:08 | |
| *** derekh has joined #openstack-infra | 09:11 | |
| *** ykarel is now known as ykarel|lunch | 09:14 | |
| *** xinliang has quit IRC | 09:18 | |
| *** alexchadin has joined #openstack-infra | 09:19 | |
| *** gary_perkins has joined #openstack-infra | 09:20 | |
| *** markmcd has joined #openstack-infra | 09:21 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack-infra/bindep master: Add OracleLinux support https://review.openstack.org/536355 | 09:24 |
| *** ifat_afek has quit IRC | 09:26 | |
| *** electrofelix has joined #openstack-infra | 09:27 | |
| *** alexchadin has quit IRC | 09:30 | |
| *** dtantsur|afk is now known as dtantsur | 09:30 | |
| *** sambetts|afk is now known as sambetts | 09:30 | |
| cgoncalves | ianw, hi. it seems that the EPEL mirror patch didn't fix the issue we were seeing in octavia :/ | 09:36 |
| *** zzzeek_ has quit IRC | 09:37 | |
| cgoncalves | ianw, oh, actually the problem is a different one. http://logs.openstack.org/30/608830/3/check/octavia-v2-dsvm-scenario-centos-7/6050cda/controller/logs/devstacklog.txt.gz#_2018-10-22_07_11_53_326 | 09:38 |
| cgoncalves | also here http://logs.openstack.org/18/609418/6/check/octavia-v2-dsvm-scenario-centos-7/09cd4fe/controller/logs/devstacklog.txt.gz#_2018-10-22_07_14_34_164 | 09:38 |
| *** ociuhandu has joined #openstack-infra | 09:39 | |
| *** zzzeek has joined #openstack-infra | 09:40 | |
| *** ociuhandu has quit IRC | 09:40 | |
| *** bnemec has joined #openstack-infra | 09:45 | |
| *** yamamoto has quit IRC | 09:46 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| *** ifat_afek has joined #openstack-infra | 09:49 | |
| ianw | cgoncalves: do you use dib from master? haven't released with that epel fix yet | 09:50 |
| ianw | cgoncalves: ooohh, yeah we don't mirror that. is there a reason not use use centos-minimal element? that we build from packages directly | 09:51 |
| *** xek has quit IRC | 09:51 | |
| *** yamamoto has quit IRC | 09:52 | |
| *** bnemec has quit IRC | 09:53 | |
| *** roman_g has joined #openstack-infra | 09:53 | |
| cgoncalves | ianw, not sure there's a reason and/or we ever tested it instead of centos7 | 09:53 |
| cgoncalves | ianw, for ubuntu we switched some months ago to ubuntu-minimal | 09:54 |
| ianw | yeah, it will stop having to download the upstream .qcow2 | 09:54 |
| *** psachin has joined #openstack-infra | 09:56 | |
| cgoncalves | hmm let me try it :) | 09:56 |
| ianw | lmn, i'm end of day here ,but if there's any issues drop me a mail and i'll look into it :) | 09:59 |
| *** ykarel|lunch is now known as ykarel | 10:02 | |
| *** psachin has quit IRC | 10:06 | |
| *** alexchadin has joined #openstack-infra | 10:12 | |
| *** shardy has quit IRC | 10:13 | |
| *** xek has joined #openstack-infra | 10:14 | |
| *** yamamoto has joined #openstack-infra | 10:15 | |
| *** shardy has joined #openstack-infra | 10:16 | |
| *** kjackal has quit IRC | 10:19 | |
| *** dave-mccowan has joined #openstack-infra | 10:22 | |
| *** pbourke has quit IRC | 10:23 | |
| cgoncalves | ianw, cool, thank you! | 10:29 |
| *** carl_cai has joined #openstack-infra | 10:30 | |
| *** kjackal has joined #openstack-infra | 10:36 | |
| *** pbourke has joined #openstack-infra | 10:47 | |
| *** annp has quit IRC | 10:49 | |
| *** jamesdenton has joined #openstack-infra | 10:52 | |
| *** aojea has joined #openstack-infra | 10:55 | |
| *** bobh has joined #openstack-infra | 10:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: wheel-build : temporary pip with checksum output https://review.openstack.org/612234 | 10:59 |
| *** bobh has quit IRC | 11:02 | |
| *** owalsh has quit IRC | 11:13 | |
| *** rh-jelabarre has joined #openstack-infra | 11:15 | |
| *** kaiokmo has joined #openstack-infra | 11:18 | |
| *** priteau has joined #openstack-infra | 11:19 | |
| *** markvoelker has joined #openstack-infra | 11:22 | |
| *** jpena is now known as jpena|lunch | 11:23 | |
| *** gfidente has quit IRC | 11:25 | |
| *** pcaruana has quit IRC | 11:26 | |
| *** markvoelker has quit IRC | 11:26 | |
| *** ansmith has quit IRC | 11:27 | |
| *** bobh has joined #openstack-infra | 11:27 | |
| *** udesale has quit IRC | 11:30 | |
| *** dtantsur is now known as dtantsur|brb | 11:37 | |
| *** owalsh has joined #openstack-infra | 11:41 | |
| *** pcaruana has joined #openstack-infra | 11:48 | |
| *** gfidente has joined #openstack-infra | 11:55 | |
| *** panda is now known as panda|lunch | 11:55 | |
| *** longkb has quit IRC | 11:56 | |
| *** quiquell is now known as quiquell|lunch | 11:57 | |
| *** aojeagarcia has joined #openstack-infra | 11:59 | |
| *** aojea has quit IRC | 12:01 | |
| *** aojea has joined #openstack-infra | 12:02 | |
| *** aojeagarcia has quit IRC | 12:05 | |
| *** bhavikdbavishi has quit IRC | 12:06 | |
| *** kopecmartin has quit IRC | 12:08 | |
| *** aojea has quit IRC | 12:09 | |
| *** janki has quit IRC | 12:10 | |
| *** pcaruana has quit IRC | 12:12 | |
| *** pcaruana has joined #openstack-infra | 12:12 | |
| *** efried_pto is now known as efried | 12:14 | |
| *** bobh has quit IRC | 12:16 | |
| *** icey has quit IRC | 12:19 | |
| *** jpena|lunch is now known as jpena | 12:23 | |
| *** quiquell|lunch is now known as quiquell | 12:24 | |
| *** xek has quit IRC | 12:24 | |
| *** xek has joined #openstack-infra | 12:25 | |
| *** smcginnis_vaca is now known as smcginnis | 12:26 | |
| *** zul has joined #openstack-infra | 12:28 | |
| *** rlandy has joined #openstack-infra | 12:35 | |
| *** jcoufal has joined #openstack-infra | 12:36 | |
| *** tpsilva has joined #openstack-infra | 12:38 | |
| *** alexchadin has quit IRC | 12:38 | |
| *** jesusaur has quit IRC | 12:46 | |
| *** owalsh_ has joined #openstack-infra | 12:48 | |
| *** yamamoto has quit IRC | 12:50 | |
| *** yamamoto has joined #openstack-infra | 12:50 | |
| *** icey has joined #openstack-infra | 12:50 | |
| *** owalsh has quit IRC | 12:51 | |
| slaweq | hi infra, I found in some Neutron jobs issue like http://logs.openstack.org/23/577223/22/check/build-openstack-releasenotes/1fb3112/job-output.txt.gz#_2018-10-20_23_12_20_242159 | 12:51 |
| *** alexchadin has joined #openstack-infra | 12:51 | |
| slaweq | do You know about it maybe? | 12:51 |
| *** ansmith has joined #openstack-infra | 12:52 | |
| *** bobh has joined #openstack-infra | 12:52 | |
| *** kgiusti has joined #openstack-infra | 12:52 | |
| *** jroll has quit IRC | 12:53 | |
| *** jroll has joined #openstack-infra | 12:54 | |
| *** yamamoto has quit IRC | 12:55 | |
| *** kopecmartin has joined #openstack-infra | 12:55 | |
| *** panda|lunch is now known as panda | 12:56 | |
| *** owalsh_ is now known as owalsh | 12:57 | |
| *** owalsh has quit IRC | 12:58 | |
| *** ifat_afek has quit IRC | 13:00 | |
| *** ykarel_ has joined #openstack-infra | 13:03 | |
| *** dtantsur|brb is now known as dtantsur | 13:04 | |
| *** boden has joined #openstack-infra | 13:04 | |
| *** ykarel has quit IRC | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:06 | |
| *** weshay_pto has quit IRC | 13:07 | |
| openstackgerrit | Merged openstack-infra/zuul master: Handle missing node during hold check https://review.openstack.org/610969 | 13:08 |
| *** rpioso|afk is now known as rpioso | 13:09 | |
| *** alexchadin has quit IRC | 13:10 | |
| *** smarcet has joined #openstack-infra | 13:11 | |
| *** mriedem has joined #openstack-infra | 13:11 | |
| *** hwoarang has quit IRC | 13:12 | |
| *** owalsh has joined #openstack-infra | 13:13 | |
| *** jesusaur has joined #openstack-infra | 13:15 | |
| *** hwoarang has joined #openstack-infra | 13:16 | |
| *** jamesmcarthur has joined #openstack-infra | 13:17 | |
| AJaeger | slaweq: I just hear releasenotes - was fixed yesterday. | 13:22 |
| AJaeger | slaweq: please recheck | 13:22 |
| slaweq | AJaeger: thx for info, I will try then :) | 13:22 |
| *** fuentess has joined #openstack-infra | 13:23 | |
| AJaeger | slaweq: change I6c9f303dd5f284d7df68a4b764b84a05309f6693 fixed it - if it's a missing "msgmerge" | 13:23 |
| *** ykarel_ is now known as ykarel | 13:23 | |
| slaweq | AJaeger: yes, it was that, thx | 13:25 |
| *** eharney has joined #openstack-infra | 13:27 | |
| *** munimeha1 has joined #openstack-infra | 13:35 | |
| *** elbragstad is now known as lbragstad | 13:39 | |
| *** felipemonteiro has joined #openstack-infra | 13:48 | |
| *** smarcet has quit IRC | 13:53 | |
| *** xek has quit IRC | 13:58 | |
| *** yamamoto has quit IRC | 13:58 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Make functional src jobs actually install from source https://review.openstack.org/612168 | 13:59 |
| *** yamamoto has joined #openstack-infra | 13:59 | |
| *** gfidente has quit IRC | 14:01 | |
| *** lpetrut has joined #openstack-infra | 14:03 | |
| *** agopi has quit IRC | 14:03 | |
| *** yamamoto has quit IRC | 14:04 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 14:04 |
| *** dpawlik has quit IRC | 14:07 | |
| *** maciejjozefczyk has quit IRC | 14:08 | |
| *** dpawlik has joined #openstack-infra | 14:08 | |
| *** maciejjozefczyk has joined #openstack-infra | 14:08 | |
| *** SteelyDan is now known as dansmith | 14:11 | |
| *** haleyb has joined #openstack-infra | 14:12 | |
| *** ykarel is now known as ykarel|afk | 14:13 | |
| *** carl_cai has quit IRC | 14:19 | |
| *** ykarel_ has joined #openstack-infra | 14:29 | |
| *** ykarel|afk has quit IRC | 14:32 | |
| *** graphene has joined #openstack-infra | 14:32 | |
| *** yamamoto has joined #openstack-infra | 14:41 | |
| *** jamesmcarthur has quit IRC | 14:41 | |
| *** graphene has quit IRC | 14:42 | |
| *** graphene has joined #openstack-infra | 14:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Cleanup down ports https://review.openstack.org/609829 | 14:44 |
| *** jamesmcarthur has joined #openstack-infra | 14:46 | |
| spotz | fungi or anyone else who might know, how can I search for an etherpad from the Vancouver summit? Is there a way? | 14:47 |
| Shrews | spotz: https://wiki.openstack.org/wiki/Forum/Vancouver2018#List_of_Brainstorming_Etherpads_.28Historical.29 maybe? | 14:50 |
| spotz | Shrews: You rock! | 14:52 |
| spotz | Was looking for the mentoring one to link to the Berlin session:) | 14:52 |
| *** smarcet has joined #openstack-infra | 14:52 | |
| * Shrews gives all credit to google search | 14:52 | |
| *** jamesmcarthur has quit IRC | 14:55 | |
| spotz | I try to leave ethers open, I mean there should be a way to search through them but I've never figured it out | 14:55 |
| *** jamesmcarthur has joined #openstack-infra | 14:55 | |
| *** quiquell is now known as quiquell|off | 14:57 | |
| *** ramishra has quit IRC | 14:59 | |
| *** roman_g has quit IRC | 15:00 | |
| *** roman_g has joined #openstack-infra | 15:00 | |
| *** felipemonteiro has quit IRC | 15:02 | |
| *** david-lyle has joined #openstack-infra | 15:07 | |
| *** pcrews has quit IRC | 15:09 | |
| *** pcaruana has quit IRC | 15:12 | |
| *** e0ne has quit IRC | 15:15 | |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config master: Add starlingx-docs-core group https://review.openstack.org/612419 | 15:18 |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New Repo - OpenStack-Helm Images https://review.openstack.org/611892 | 15:20 |
| *** gfidente has joined #openstack-infra | 15:20 | |
| fungi | spotz: etherpad explicitly doesn't allow searching for pads by name or comment because the way it implements private and read-only pad urls is by using unguessable names for them | 15:22 |
| fungi | s/comment/content/ | 15:22 |
| spotz | fungi: That's what I figured | 15:23 |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New Repo: OpenStack-Helm Docs https://review.openstack.org/611893 | 15:23 |
| *** e0ne has joined #openstack-infra | 15:28 | |
| *** e0ne has quit IRC | 15:32 | |
| *** xek has joined #openstack-infra | 15:33 | |
| *** sthussey has joined #openstack-infra | 15:35 | |
| *** manjeets has joined #openstack-infra | 15:39 | |
| mwhahaha | Hey is there anyway to determine why jobs aren't triggering for a change? https://review.openstack.org/#/c/610343/ there is a .zuul.yaml in that branch | 15:43 |
| *** jamesmcarthur has quit IRC | 15:47 | |
| clarkb | mwhahaha: I think it is because there are no templates with those names defined on the stable/luminous branch | 15:47 |
| clarkb | mwhahaha: puppet-openstack-integration only has branches for openstack stable releases | 15:48 |
| *** ifat_afek has joined #openstack-infra | 15:49 | |
| mwhahaha | hrm | 15:49 |
| mwhahaha | let me go track that down. seems odd cause we did merge stuff in luminous | 15:49 |
| mwhahaha | so something else has changed | 15:49 |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove sqlalchemy from requirements https://review.openstack.org/611821 | 15:50 |
| clarkb | mwhahaha: is it possible the puppet job definitions went from an unbranched setup (master only) to a branched setup? If unbranched zuul assumes that applies globally but if brnached scopes it to the specific branch aiui | 15:50 |
| *** bnemec has joined #openstack-infra | 15:51 | |
| mwhahaha | i don't think we've changed anything like that | 15:51 |
| mwhahaha | i'll figure it out, it's likely the zuul config somewhere | 15:52 |
| openstackgerrit | Pierre Riteau proposed openstack-infra/project-config master: Create blazar-specs repo https://review.openstack.org/602783 | 15:55 |
| mwhahaha | ah i see it, we moved the integration jobs to be branch specific cause we're testing multiple ceph versions. though the check/unit jobs should still run | 15:55 |
| mwhahaha | odd | 15:55 |
| *** smarcet has quit IRC | 15:58 | |
| *** bnemec has quit IRC | 15:59 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:59 | |
| *** smarcet has joined #openstack-infra | 15:59 | |
| *** gyee has joined #openstack-infra | 16:00 | |
| *** ifat_afek has quit IRC | 16:01 | |
| mwhahaha | clarkb: is there a way to reference a different branches config? I think the issue is that puppet-ceph being the weirdo uses stable/luminous or stable/jewel so i'll probably want to just inherit from like stable/rocky out of puppet-openstack-integration and define the jobs in the luminous branch | 16:02 |
| mwhahaha | but i don't want to redefine everything | 16:02 |
| *** ifat_afek has joined #openstack-infra | 16:02 | |
| *** xek has quit IRC | 16:02 | |
| clarkb | There are ways to indicate master should be tested with eg devel (openstacksdk does this when testing with ansible) but I'm not sure about mapping that for the jobs themselves | 16:02 |
| clarkb | mordred: ^ you may know having dealt with this around ansible | 16:03 |
| clarkb | you can do specific branch matchers I guess | 16:05 |
| clarkb | looking for docs | 16:06 |
| clarkb | mwhahaha: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.branches | 16:07 |
| *** david-lyle has quit IRC | 16:07 | |
| *** dklyle has quit IRC | 16:07 | |
| mwhahaha | yea that's what we were doing in some places, i'll mess around with it | 16:07 |
| clarkb | mwhahaha: note the override checkout section is I think what you want | 16:08 |
| clarkb | run a job in one branch of a project that doesn't match the branch of another project | 16:08 |
| mwhahaha | meh this is going to get annoying | 16:09 |
| mwhahaha | k thanks i'll give that a shot | 16:10 |
| clarkb | hrm no pabelanger. I'll run this by anyone willing to listen then. Looking at moving zuul and nodepool to the zk cluster I think the steps we want to take are move one of the nodepool builders to cluster and wait for it to build images against the clouds. Then stop nl* nb* and zuul scheduler, update their config to point at new zk, start those services. Clean up aliens and delete nodepool.o.o | 16:12 |
| clarkb | Shrews: ^ do you know if that is safe for the nodepool builders to be running against disjoint zookeepers? Mostly worried that they will try to delete each other's images as aliens | 16:13 |
| clarkb | I don't think they will do that though | 16:13 |
| *** graphene has quit IRC | 16:15 | |
| mordred[m] | clarkb yeah ... There's examples of branch things in the openstacksdk ansible jobs | 16:16 |
| clarkb | mwhahaha: ^ that may give good examples | 16:16 |
| mordred[m] | clarkb for zk ... I'd just move all the builders to the cluster and let them build/upload. | 16:17 |
| *** graphene has joined #openstack-infra | 16:17 | |
| mordred[m] | We'll stop getting new images on existing system, byr thats ok | 16:17 |
| mordred[m] | The move Zuul and the launchers once the builders are good | 16:18 |
| *** rkukura has joined #openstack-infra | 16:18 | |
| clarkb | ++ | 16:19 |
| mordred[m] | And then go back and clean up the old images and any orphaned servers with a script | 16:19 |
| corvus | howdy! i should be around for a while now | 16:23 |
| *** shardy has quit IRC | 16:24 | |
| clarkb | I'm working on the three changes to make ^ happen and I'll also reboot the new cluster to pick up patches and stuff then we can decide when we want to work to deploy this | 16:25 |
| *** ykarel__ has joined #openstack-infra | 16:25 | |
| mordred[m] | ++ | 16:26 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 16:27 | |
| *** ykarel_ has quit IRC | 16:28 | |
| *** smarcet has quit IRC | 16:28 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Switch nodepool builders to zk cluster https://review.openstack.org/612441 | 16:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Switch nodepool launchers to use new zk cluster https://review.openstack.org/612442 | 16:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Switch zuul scheduler to new zk cluster https://review.openstack.org/612443 | 16:30 |
| clarkb | those are the three changes we need. I've also learned that I don't think we document how to connect to zk in the zuul docs so I'm off to figure that out now | 16:30 |
| clarkb | nevermind its under components | 16:31 |
| clarkb | I guess we set zookeeper hosts globally even though I think only the scheduelr talks to zk? | 16:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Switch zuul scheduler to new zk cluster https://review.openstack.org/612443 | 16:33 |
| *** jcoufal has quit IRC | 16:35 | |
| corvus | clarkb, mordred[m]: cool, that sounds like a good plan | 16:36 |
| corvus | clarkb: yes, that's slightly forward looking documentation (other components should use it eventually) | 16:36 |
| corvus | documentation/configuration design | 16:36 |
| clarkb | fwiw my next step is actually trying to confirm that the zk cluster is operating as a cluster properly | 16:37 |
| clarkb | the logs indicate that maybe this isn't the case | 16:37 |
| clarkb | but maybe that was fixed? the error is old and the myid complain doesn't seem to be valid anymore | 16:38 |
| clarkb | Mode: standalone that does seem to confirm it thought | 16:39 |
| *** kopecmartin is now known as kopecmartin|off | 16:41 | |
| clarkb | reading teh config files they don't seem to list out what the other zk servers are for them to work out a leader election. Also zk02 is afk similar to how adns1 was I think | 16:41 |
| clarkb | corvus: ^ speaking of. adns1 went out to lunch at some point and the zone expired on ns1 and ns2 for zuul-ci.org. Rebooting adns1 throught the openstack api fixed this | 16:41 |
| clarkb | corvus: We already have a 10 day expiry on that zone, wondering if maybe we should set up an external dig verification against adns1 to ensure the server is reachable? | 16:41 |
| corvus | clarkb: yeah that sounds reasonable :) | 16:42 |
| *** ykarel__ is now known as ykarel|away | 16:42 | |
| clarkb | have ns1 and ns2 both run a daily dig against their configured zones ro something | 16:43 |
| corvus | mwhahaha: i also wrote this feature for the use case you're talking about: https://zuul-ci.org/docs/zuul/user/config.html#pragma | 16:43 |
| corvus | mwhahaha: it lets you map stable/queens to stable/luminus (or whatever the right words are :). i don't know if it ended up being used or not | 16:44 |
| *** smarcet has joined #openstack-infra | 16:44 | |
| mwhahaha | corvus: hmm ok i can try that too, i'm almost done duplicating everything anyway :D | 16:44 |
| *** smarcet has quit IRC | 16:44 | |
| corvus | mwhahaha: hopefully we can break the copy-pasta cycle at some point :) | 16:45 |
| *** efried has quit IRC | 16:49 | |
| *** efried has joined #openstack-infra | 16:49 | |
| *** ykarel|away has quit IRC | 16:50 | |
| *** e0ne has joined #openstack-infra | 16:52 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Configure zk\d to act as a cluster https://review.openstack.org/612447 | 16:53 |
| clarkb | mordred[m]: corvus ^ that should configure zk to be an actual cluster. I'm working on reboots and getting zk02 beack into happy state | 16:53 |
| *** jpena is now known as jpena|off | 16:55 | |
| corvus | clarkb: do those numbers mean anything? | 16:56 |
| *** ifat_afek has quit IRC | 16:57 | |
| clarkb | corvus: they do. in the config it writes out #server.1=zookeeper1:2888:3888 and server.2 and so on | 16:58 |
| *** derekh has quit IRC | 17:00 | |
| corvus | are they just arbitrary labels? | 17:01 |
| *** dklyle has joined #openstack-infra | 17:02 | |
| Shrews | clarkb: i don't think disjoint zk servers will be safe | 17:02 |
| clarkb | corvus: no I think the 1,2,3 are zk config specific | 17:03 |
| clarkb | corvus: all of the zk examples use 1,2, and 3 | 17:03 |
| clarkb | Shrews: mordred[m]'s suggestion avoids the problem entirely and we just switch all builders to new zk and have launchers use existing images for a bit | 17:04 |
| Shrews | k. still catching up on sb from lunch | 17:04 |
| *** eharney has quit IRC | 17:04 | |
| *** e0ne has quit IRC | 17:06 | |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| *** felipemonteiro has joined #openstack-infra | 17:11 | |
| *** armax has joined #openstack-infra | 17:12 | |
| Shrews | clarkb: i'm sort of curious how the builders will handle the orphaned images. i forget, tbh. taking a look at the code now... | 17:12 |
| *** felipemonteiro has quit IRC | 17:15 | |
| *** eharney has joined #openstack-infra | 17:15 | |
| clarkb | ok all three zookeeper servers are running and have been rebooted to apply kernel updates | 17:15 |
| Shrews | I think they'll be left alone on the provider and will have to be manually removed. We'll also need to cleanup the builds on the nb* machines manually to free up that disk space | 17:16 |
| clarkb | should just need to get that change above in, 612447, for them to act like a cluster | 17:16 |
| clarkb | Shrews: that seems doable | 17:16 |
| Shrews | approved | 17:18 |
| *** jamesmcarthur has joined #openstack-infra | 17:18 | |
| *** sambetts is now known as sambetts|afk | 17:18 | |
| *** bnemec has joined #openstack-infra | 17:19 | |
| *** jamesmcarthur has quit IRC | 17:22 | |
| *** pcaruana has joined #openstack-infra | 17:27 | |
| *** e0ne has joined #openstack-infra | 17:30 | |
| *** agopi has joined #openstack-infra | 17:32 | |
| *** irclogbot_2 has joined #openstack-infra | 17:34 | |
| clarkb | reminder we are set to have a meeting with foundation about opendev messaging in ~25 minutes | 17:34 |
| clarkb | we'll be in here | 17:34 |
| * clarkb double checks with foundation that it is still visible on their calendars | 17:35 | |
| corvus | clarkb: cool, i'm slightly discombobulated, but can attend, do you have a link handy? | 17:37 |
| clarkb | corvus: for eg an etherpad? not yet, but I should probably go ahead and create one since that was useful for the last meeting we had | 17:38 |
| clarkb | https://etherpad.openstack.org/p/infra-to-opendev-messaging | 17:39 |
| corvus | oh, right, it's an irc meeting, not a phone type meeting, so erm, just right here i guess? :) | 17:39 |
| clarkb | yup | 17:39 |
| clarkb | And we can use that etherpad for note taking or brainstorming as necessary | 17:39 |
| corvus | ++ | 17:39 |
| *** graphene has quit IRC | 17:40 | |
| *** ccamacho has quit IRC | 17:41 | |
| *** graphene has joined #openstack-infra | 17:41 | |
| *** smarcet has joined #openstack-infra | 17:43 | |
| *** jamesmcarthur has joined #openstack-infra | 17:44 | |
| *** electrofelix has quit IRC | 17:44 | |
| *** davidlenwell has joined #openstack-infra | 17:46 | |
| *** diablo_rojo has joined #openstack-infra | 17:47 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:47 | |
| *** jamesmcarthur has quit IRC | 17:48 | |
| *** ifat_afek has joined #openstack-infra | 17:49 | |
| *** ifat_afek has quit IRC | 17:53 | |
| *** jcoufal has joined #openstack-infra | 17:54 | |
| *** irclogbot_2 has quit IRC | 17:56 | |
| *** pcaruana has quit IRC | 17:57 | |
| * fungi is around and relatively alert for the meeting | 17:57 | |
| * mordred is around - but is on an airplane, so may or may not be super useful | 17:58 | |
| clarkb | mordred: are you ever not on an airplane? | 17:59 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added get my speaker profile per summit https://review.openstack.org/612461 | 17:59 |
| clarkb | I remain in awe of how peopel like mordred and jbryce handle the travel they do | 17:59 |
| Shrews | mordred: have ever tried *not* being on an airplane? | 18:00 |
| clarkb | ok it is 1800 UTC now | 18:00 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added get my speaker profile per summit https://review.openstack.org/612461 | 18:00 |
| clarkb | jbryce: is on IRC, I'm told wes and lsell intend on making it. I wonder if the +r is going to cause problems actually. Should've considered that :/ | 18:00 |
| mordred | Shrews: no - is that an option? | 18:00 |
| mordred | clarkb: oh, yeah. stupid +r | 18:02 |
| clarkb | I'm told that +r is indeed causing a problem so we should be pateint :) | 18:03 |
| corvus | no worries -- happy to help if needed | 18:04 |
| *** ildikov has joined #openstack-infra | 18:05 | |
| *** irclogbot_2 has joined #openstack-infra | 18:06 | |
| *** bhavikdbavishi has quit IRC | 18:07 | |
| *** bhavikdbavishi has joined #openstack-infra | 18:08 | |
| *** graphene has quit IRC | 18:10 | |
| *** graphene has joined #openstack-infra | 18:11 | |
| *** lsell has joined #openstack-infra | 18:11 | |
| *** graphene has quit IRC | 18:12 | |
| *** iamweswilson has joined #openstack-infra | 18:12 | |
| lsell | hi i think i made it in, sorry for being late clarkb | 18:13 |
| clarkb | thats ok, unexpected turbulence due to recent irc changes | 18:13 |
| iamweswilson | here! | 18:13 |
| * mordred waves to lsell | 18:13 | |
| * mordred waves to iamweswilson | 18:13 | |
| *** graphene has joined #openstack-infra | 18:13 | |
| corvus | yay! | 18:14 |
| clarkb | I think claire and mark are still trying to connect | 18:14 |
| *** agopi has quit IRC | 18:15 | |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config master: Add starlingx-docs-core group https://review.openstack.org/612419 | 18:16 |
| *** apetrich has quit IRC | 18:17 | |
| *** sparkycollier has joined #openstack-infra | 18:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 18:20 |
| *** ifat_afek has joined #openstack-infra | 18:24 | |
| *** e0ne has quit IRC | 18:24 | |
| clarkb | ok we have lsell iamweswilson and jbryce joining us | 18:24 |
| clarkb | I think at least claire has indicated we should go ahead and start and mark may join us still? | 18:24 |
| *** smarcet has quit IRC | 18:24 | |
| fungi | sparkycollier is already here | 18:25 |
| clarkb | #startmeeting opendev-messaging | 18:25 |
| openstack | Meeting started Mon Oct 22 18:25:04 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:25 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:25 |
| clarkb | ah | 18:25 |
| *** openstack changes topic to " (Meeting topic: opendev-messaging)" | 18:25 | |
| openstack | The meeting name has been set to 'opendev_messaging' | 18:25 |
| clarkb | #link https://etherpad.openstack.org/p/infra-to-opendev-messaging Early thoughts on what we want to communicate opendev is and how that may affect opendev services | 18:25 |
| clarkb | Long story short we want to effectively communicate what Opendev is to avoid perpetuating the existing confusions around the things that openstack infra does | 18:26 |
| mordred | ++ | 18:26 |
| clarkb | In particular one concern is that if we say "openstack infra is now opendev" we haven't changed the message that we are the openstack infra team. We have only said we have a new name | 18:26 |
| jbryce | i read through the etherpad and think the basics in there make sense. i think one question i still have is how broad is the "bigger than openstack" world we want to address | 18:26 |
| clarkb | On the flip side we don't want to communicate that openstack is now on its own, they are still a user and an important one | 18:27 |
| fungi | corvus had some good words to that effect during our discussion at the ptg, and i'm trying to remember how he phrased it now | 18:27 |
| fungi | (that it's not simply a rebranding exercise) | 18:27 |
| corvus | fungi: i have forgotten everything before saturday :( | 18:28 |
| iamweswilson | from an initial messaging standpoint, would something like "Community hosted tools and infrastructure for developing and maintaining open source software." capture the sentiment or would it be required that we call out OpenStack in the services offered? | 18:28 |
| clarkb | jbryce: in my head I've seen as an implementation detail that it would be good to keep scope small to start with the intent of going broader | 18:28 |
| corvus | iamweswilson: that hits it for me | 18:28 |
| mordred | iamweswilson: ++ | 18:29 |
| jbryce | my suggestion would be that we start with what we wanted OpenDev to be and flesh out the messaging and the brand, and then find the use cases that we need to address individually (like the openstack one) | 18:29 |
| clarkb | jbryce: zuul and airship and starlingx are friendly projects that would benefit from this work that will understand the bumps along the way when we hit them | 18:29 |
| sparkycollier | so anyone developing any open source software? (not arguing just asking) | 18:29 |
| iamweswilson | good clarification sparkycollier | 18:30 |
| *** apetrich has joined #openstack-infra | 18:30 | |
| fungi | i think we want to underscore that we're about more than "just" open source software, but also provide open tools to help with open collaboration | 18:30 |
| lsell | spaykcollier: my guess is that would be the intent over time, but it will take time for us to practically reach that goal | 18:31 |
| mordred | sparkycollier: conceptually yes - although I don't think we want the flood gates to open on that immediately ... largely because we need to figure out how nodepool capacity is funded | 18:31 |
| mordred | yah - what lsell said | 18:31 |
| clarkb | lsell: ++ and I do think that keeping the scope smaller to start will help keep things at a reasonable size while we figure out the transition | 18:31 |
| lsell | i'm thinking that the first step is to communicate to existing community / users about the changes, including the long-term vision and any short-term implications for their work | 18:31 |
| fungi | if you're looking for somewhere to publish the freely-licensed source code for this thing you've written, but aren't really looking to get together with like-minded people on improving it together, then this would work for you but there are a lot of other services out there which might be more what you're looking for | 18:31 |
| mordred | fungi: ++ | 18:31 |
| sparkycollier | in other words, we're not just trying to get cloud/infrastructure software? Like if you develop open source for mobile phones or drones or tractors that's all welcome? (I'm not arguing against just wanting to be clear) | 18:32 |
| fungi | sparkycollier: in short, yes | 18:32 |
| sparkycollier | agreed Fungi - I guess I'm searching for what wouldn't be a fit... seems like it's more like things that don't really want "our" collaboration model (as opposed to is/is not "cloud:) | 18:33 |
| lsell | i'm guessing one of the goals in the short term is also to recruit more contributors, in order to serve that larger audience | 18:33 |
| sparkycollier | thanks! | 18:33 |
| clarkb | sparkycollier: I think what we'll see is that the tools we provide likely aren't suited to tractor software dev so they will go elsewhere, but if they wanted to try and make that use case work with us then they should bne able to | 18:33 |
| clarkb | lsell: yes | 18:33 |
| fungi | keep in mind that we likely wouldn't go out of our way to set up whitelabeled per-project services for most projects, just provide a means for them to use the services we offer under the opendev.org domain | 18:33 |
| iamweswilson | lsell: + | 18:33 |
| clarkb | lsell: in particular if we look at it from the current openstack-infra team perspective that may indicate to airship that they shouldn't get involved | 18:33 |
| clarkb | when we think it would be awesome for them to interact as necessary to facilitate their needs | 18:33 |
| lsell | and recruiting more contributors benefits the openstack community, which is an important part of the message :) | 18:34 |
| clarkb | (note airship has been doing a good job of being involved, that may have been a bad example) | 18:34 |
| jbryce | we should add these goals to the etherpad (and any others that people think of) | 18:34 |
| mordred | clarkb: I think more that the companies who are interested in, say, airship, might have a vested interest in helping with opendev even if they don't care about, say, openstack | 18:35 |
| fungi | sparkycollier: to me it's less about software for a particular industry or a particular technology, and more about what your collaboration style is as a project | 18:35 |
| mordred | for some values of airship and openstack | 18:35 |
| mordred | fungi: ++ | 18:35 |
| lsell | one other tactical question, in the etherpad it mentioned it will take some time to transition git and gerrit...are you thinking weeks, months, a year? just trying to gauge when we want to communicate what | 18:36 |
| fungi | i expect some services to take us upwards of a year to migrate | 18:36 |
| fungi | others may disagree | 18:36 |
| clarkb | it is related to how much pain existing users are willing to experience | 18:36 |
| corvus | lsell: months at best i'd guess. year is plausible. | 18:36 |
| clarkb | I expect a good chunk of work to happen early train cycle to ease the pain openstack experiences | 18:37 |
| clarkb | but maybe openstack is willing to ride it out earlier and we can psuh on big moves sooner | 18:37 |
| corvus | so part of the desire here is to be able to start communicating/discussing about that transition without tripping over the fact that that we haven't said 'opendev' except around the virtual water-cooler :) | 18:38 |
| clarkb | corvus: ++ | 18:38 |
| iamweswilson | is the thought to start communicating the name change more broadly while services are being over or wait until all is complete? | 18:38 |
| iamweswilson | corvus: answered my Q | 18:38 |
| clarkb | iamweswilson: ya we want to communicate it today so that we can take a step by step approach without confusing or surprising users | 18:38 |
| fungi | yeah, we just wanted to have a clear message we can all get behind first, i think | 18:38 |
| fungi | so we're not all going in different directions with how we're describing this, and causing more confusion | 18:39 |
| iamweswilson | makes sense | 18:40 |
| jbryce | i'm trying to turn these into goals on line 6 of the etherpad if anyone wants to check my work | 18:40 |
| clarkb | jbryce: the goals you are capturing lgtm | 18:40 |
| jbryce | well....starting on line 6 | 18:40 |
| * fungi hates the term "best practices" (we could just say practices?) but otherwise i like it | 18:41 | |
| *** ifat_afek has quit IRC | 18:42 | |
| iamweswilson | what about "better than not good" practices? | 18:42 |
| clarkb | ha | 18:42 |
| jbryce | so i think we should do a messaging doc like we've done with the pilot projects to settle in on a few phrases we can use in all the comms | 18:42 |
| jbryce | things like the one-liner from iamweswilson | 18:42 |
| fungi | (i spent enough time in places where it just meant the otherwise ineffective things you do to avoid being sued or fired) | 18:42 |
| jbryce | but also maybe one that's a few sentences | 18:42 |
| clarkb | jbryce: a few sentences would be valuable for existing users I expect | 18:43 |
| iamweswilson | looking for my thumbsup enoji | 18:43 |
| clarkb | since they already have some familiarity with the tooling and get the top level one liner | 18:43 |
| jbryce | and then we can add in a few q&a that we know are important like the relationship to openstack projects, some of the next steps, timing, impact to existing services | 18:43 |
| corvus | jbryce: i like it | 18:44 |
| jbryce | and then use those to start communicating broadly | 18:44 |
| fungi | iamweswilson's one-liner makes for a great intro, given lots of people won't read past the first sentence anyway | 18:44 |
| corvus | was very helpful with zuul | 18:44 |
| lsell | sounds like a good idea. make sure we capture the goals, any anticipated concerns, etc. today and then we can start a draft messaging doc for review | 18:44 |
| clarkb | me too. I think that avoids trying to capture everything in a single message and instead rely on qa to dig into details | 18:44 |
| fungi | and then the details can follow it nicely there | 18:44 |
| iamweswilson | jbryce: agreed. It would be great to surface that into a simple landing page to start with on the marketing side | 18:44 |
| jbryce | i also think that we should think about some marketing to potential contributors | 18:44 |
| lsell | in terms of timing and channels, it sounds like a community email in the relatively near future is first on the list, and then do we want to communicate this again at the berlin summit? | 18:44 |
| mordred | lsell: we might also want to bring it up in the board meeting? | 18:45 |
| fungi | jbryce: by "marketing to potential contributors" you mean people to help us run it? or projects to start making use of it? | 18:45 |
| jbryce | as we start talking about the transition, it will probably re-expose the whole concept of the infra team to some people (and first time exposure for others), so making it clear that they can come participate and what that looks like would be good | 18:45 |
| clarkb | lsell: I worry berlin might be too early and give the impression we are ready for the hoard. But the board meeting might be a good venue there | 18:45 |
| jbryce | the former | 18:45 |
| lsell | are we expecting opendev.org to host the simple landing page or will that live elsewhere? | 18:45 |
| jbryce | fungi: the former of your options | 18:45 |
| fungi | jbryce: thanks, yes that would be amazing | 18:45 |
| clarkb | jbryce: ++ that is a good point. I know we have had confusion over the fact that our infrastructure is community run | 18:45 |
| lsell | good point on the board meeting, maybe we should schedule time for opendev during the project updates section in the afternoon | 18:46 |
| jbryce | i know we have the docs, but i think we need something kind of higher level | 18:46 |
| jbryce | and i think it would be useful to actually feature some of the people who work on it as examples...almost like user stories except for from a infra engineer standpoint instead | 18:46 |
| fungi | lsell: i expect some fairly simple landing page at the root of opendev.org but probably lots of other documentation related to the service too. i don't think we've discussed it at length though | 18:46 |
| corvus | lsell: i like the idea of having a page where folks have quick access to this info and updates... maybe we can use opendev.org for that for now, and as it becomes less important, evolve that into the "about" page under opendev.org ? | 18:46 |
| jbryce | infraneer? | 18:47 |
| iamweswilson | corvus: agreed | 18:47 |
| lsell | also agree on the landing page | 18:47 |
| mordred | ++ | 18:47 |
| fungi | infranaut? | 18:48 |
| lsell | i think the benefit to talking about it in berlin -- being clear about what's happening now and the vision for the next year -- is that people are still very confused by all the projects hosted on openstack git | 18:48 |
| jbryce | infranaut is good = ) | 18:48 |
| mordred | also - genericaly, having some words about what the infra team is and is not might be helpful to define the concept for people at their companies too ... I know there is frequently confusion for folks about "ci" teams being synonymous with QA teams | 18:48 |
| corvus | jbryce: that's a good idea -- that can also help us with the message that this is something anyone can contribute to, even as a part-time thing (which will be obvious when we feature a bunch of stories about us part-timers :) | 18:48 |
| lsell | we got this feedback in china last week, they were talking about the big tent and how many projects are in openstack. so we have an opportunity to almost bring up the misconception and talk about how we're addressing it | 18:48 |
| iamweswilson | lsell: ++ | 18:49 |
| fungi | some of us also still liked the possibility of using the "collaboratory" term to describe the general suite of services we're providing | 18:49 |
| clarkb | mordred: the user stories might help address the what it is part of that? | 18:49 |
| corvus | lsell: cool, as long as we know that we won't have much to show for it from a technical pov by berlin | 18:49 |
| corvus | technical/implementation even | 18:49 |
| iamweswilson | the message in Berlin would be less of "now you can all use OpenDev" and more about what's to come | 18:50 |
| clarkb | corvus: right that is my concern with broadly advertising it at berlin | 18:50 |
| fungi | hard to balance having a message around what we want to do with the risk of it seeming like vaporware | 18:50 |
| corvus | yep, there are pitfalls there, so we may have to walk a fine line | 18:50 |
| lsell | fungi: interesting about collaboratory. what would be the difference in meaning between opendev services and collaboratory services? | 18:50 |
| clarkb | fungi: it does help that we already do things for openstack (and others) | 18:50 |
| fungi | opendev is the name, collaboratory is a term | 18:51 |
| *** HenryG has quit IRC | 18:51 | |
| *** flaper87 has quit IRC | 18:51 | |
| *** niedbalski has quit IRC | 18:51 | |
| fungi | so "the opendev collaboratory" would be a collaboratory of services provided by opendev | 18:51 |
| *** HenryG has joined #openstack-infra | 18:51 | |
| *** flaper87 has joined #openstack-infra | 18:51 | |
| clarkb | We have ~9 minutes left on the allotted hour. Happy to go longer if people can, but want to make sure we capture the next steps if people can't | 18:51 |
| iamweswilson | I might not be dug in enough, but the two terms confuse me a bit. | 18:52 |
| corvus | collaboratory is a nice word that encapsulates what we're trying to do from a process pov. however, if you don't already know that, you still have to explain what a collaboratory is. so it might be helpful to sort of fix the idea in people's minds, but we still pretty much have to explain the whole deal. | 18:52 |
| iamweswilson | clarkb: agreed | 18:52 |
| lsell | yes worth having bigger discussion, but sounds likeit won't come into play until farther down he line when there are more services | 18:52 |
| *** jamesmcarthur_ has quit IRC | 18:54 | |
| lsell | sounds like next steps are to work on a collaborative messaging doc, similar to what we did for zuul, and also a draft email for community lists. perhaps foundation marketing folks can take the lead on the messaging doc and then infra folks can take the lead on the email draft? | 18:54 |
| *** kazsh has quit IRC | 18:55 | |
| clarkb | lsell: I expect the email draft may end up quoting significant parts of the messaging doc with some email specific introductions. But I'm happy to work on the email side of things | 18:55 |
| lsell | should we plan to review the draft messaging document next week, or how quickly do we need to move? | 18:55 |
| iamweswilson | lsell: those lgtm. They're all captured in the etherpad as well | 18:55 |
| fungi | that seems like a fine timeline to me | 18:55 |
| clarkb | lsell: next week is probably about the right speed. I think corvus may get dns running this week? so we'd be moving at roughly the same pace there? | 18:55 |
| fungi | and yeah, getting our new dns infrastructure up isn't really a user-facing change so safe to work on before we have something to more clearly communicate | 18:56 |
| *** lpetrut has quit IRC | 18:56 | |
| *** jamesmcarthur has joined #openstack-infra | 18:56 | |
| iamweswilson | assuming we can agree on Berlin messaging, we should probably have something on opendev,org by then, right? Even if a simple logo and one-liner? | 18:56 |
| clarkb | iamweswilson: yes that should be doable | 18:56 |
| fungi | i think that's reasonable | 18:57 |
| iamweswilson | I'm happy to help with that | 18:57 |
| clarkb | iamweswilson: DNS is step 0 then we can reuse existing infra hosting like with zuul-ci.org to get something up | 18:57 |
| *** kazsh has joined #openstack-infra | 18:57 | |
| corvus | ++ to all above :) | 18:57 |
| jbryce | sounds like we've got a good set of steps to move it forward. this is exciting! = ) | 18:59 |
| clarkb | alright seems like we may be winding down. I can help with drafting both the main doc and the email and coordinate between infra/opendev and the foundation. | 18:59 |
| fungi | thanks!!! i think this was very helpful | 18:59 |
| iamweswilson | perfect! thanks all | 18:59 |
| lsell | thanks for setting this up! | 18:59 |
| clarkb | thank you everyone | 19:00 |
| clarkb | #endmeeting | 19:00 |
| *** openstack changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 19:00 | |
| openstack | Meeting ended Mon Oct 22 19:00:07 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/opendev_messaging/2018/opendev_messaging.2018-10-22-18.25.html | 19:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/opendev_messaging/2018/opendev_messaging.2018-10-22-18.25.txt | 19:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/opendev_messaging/2018/opendev_messaging.2018-10-22-18.25.log.html | 19:00 |
| *** iamweswilson has quit IRC | 19:00 | |
| clarkb | lsell: should I bug you as far as that doc goes? and feel free to bug me about making progress on the email? | 19:00 |
| lsell | sounds good | 19:01 |
| clarkb | thinking out loud a bit more, we may want to hold off on transitioning any services to the new branding until after the berlin summit? Since it should be well communicated at that point? | 19:02 |
| *** kopecmartin|off has quit IRC | 19:03 | |
| mordred | clarkb: seems like a good idea - communicate first, then we can do the dns swap / reirect for things that are easy | 19:04 |
| fungi | plenty of opprotunity for testing -dev non-production services in the interim if we want | 19:05 |
| *** jamesmcarthur has quit IRC | 19:06 | |
| *** bhavikdbavishi has quit IRC | 19:06 | |
| *** jamesmcarthur has joined #openstack-infra | 19:06 | |
| *** jcoufal has quit IRC | 19:09 | |
| *** jamesmcarthur has quit IRC | 19:11 | |
| corvus | yep, i mean, moving anything public facing would be a bit of a rush at this point anyway, so that's an easy "ok" from me :) | 19:11 |
| *** smarcet has joined #openstack-infra | 19:13 | |
| *** irclogbot_2 has quit IRC | 19:14 | |
| *** jcoufal has joined #openstack-infra | 19:15 | |
| *** irclogbot_2 has joined #openstack-infra | 19:15 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed typos https://review.openstack.org/612479 | 19:21 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed typos https://review.openstack.org/612479 | 19:25 |
| mwhahaha | corvus: the pragma stuff didn't seem to work | 19:28 |
| mwhahaha | or maybe i'm doing it backwards | 19:29 |
| corvus | mwhahaha: link to change? | 19:31 |
| mwhahaha | corvus: https://review.openstack.org/#/c/612436/13 | 19:31 |
| *** jamesmcarthur has joined #openstack-infra | 19:32 | |
| AJaeger | mwhahaha: implied != impied | 19:32 |
| openstackgerrit | Merged openstack-infra/system-config master: Configure zk\d to act as a cluster https://review.openstack.org/612447 | 19:32 |
| * mwhahaha flips tables | 19:32 | |
| clarkb | I'll keep an eye on ^ and if the cluster becomes a cluster we can discuss timing details for the switchover | 19:33 |
| corvus | mwhahaha: ah yeah that's not right :) that's intended to make the job definitions automatically work for multiple branches | 19:33 |
| corvus | mwhahaha: based on my *limited* understanding of the problem, my guess is that it should be added to puppet-openstack-integration, which is i think where those jobs are defined? | 19:33 |
| mwhahaha | corvus: yea i'm trying that now | 19:33 |
| mwhahaha | vi https://review.openstack.org/#/c/612481/ | 19:34 |
| *** rlandy has quit IRC | 19:35 | |
| *** jamesmcarthur has quit IRC | 19:36 | |
| AJaeger | mwhahaha: and now make 612436 depends-on that change - without the implied lines, just some "random" change... | 19:36 |
| corvus | mwhahaha: yeah, i think that's more like it. then have a change depends-on that, and set "debug: true" in the project definition under check ( https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.debug ), and if the jobs still don't run, zuul will tell you why | 19:36 |
| mwhahaha | they look like the right ones are getting queued now | 19:37 |
| mwhahaha | so i think that's it, i'll see if it actually works | 19:37 |
| corvus | neat. then nevermind about the debug thing :) | 19:37 |
| anteaya | I don't see it in the backscroll so perhaps it wasn't discussed, but I do hope that I am able to continue to communicate with folks doing infra work on the #openstack-infra irc channel long term | 19:37 |
| mwhahaha | corvus: last question i guess would be do i need to leave implied-branch-matches to false in https://review.openstack.org/#/c/612436/14/.zuul.yaml or is that necessary because of the branch name difference | 19:38 |
| corvus | mwhahaha: i think you should not add that; i think the only pragma needed is in p-o-i. that should make the jobs and templates defined in p-o-i apply equally to both sets of branch names. since those templates are defined in p-o-i, it should Just Work. | 19:41 |
| mwhahaha | ok soudns good | 19:41 |
| *** rlandy has joined #openstack-infra | 19:43 | |
| *** felipemonteiro has joined #openstack-infra | 19:44 | |
| clarkb | ansible is not running right now `openstack plugin: public endpoint for compute service in Sto2 region not found` | 19:45 |
| clarkb | I'm going to eat lunch but can look at fixing that if no one lese has yet. I had to do similar for their london region last monday | 19:45 |
| mordred | clarkb: poo. lemme look ... what was the london fix? | 19:46 |
| *** jamesmcarthur has joined #openstack-infra | 19:47 | |
| *** felipemonteiro has quit IRC | 19:47 | |
| *** agopi has joined #openstack-infra | 19:48 | |
| clarkb | mordred: I removed the region from our clouds.yaml configs | 19:49 |
| clarkb | we can add it back later if reeneabled | 19:49 |
| *** graphene has quit IRC | 19:50 | |
| *** agopi_ has joined #openstack-infra | 19:50 | |
| *** jamesmcarthur has quit IRC | 19:51 | |
| *** graphene has joined #openstack-infra | 19:51 | |
| mordred | ah - gotcha | 19:51 |
| mordred | oh - the auth urls are changing for city ... | 19:53 |
| *** agopi has quit IRC | 19:53 | |
| clarkb | we arent running stuff there now so we can disable and reenable once they are all upgraded | 19:53 |
| corvus | this is a weird situation. i know why we want ansible to break in this case, but i also know why we *don't* want it to break in *this* case. maybe we should have two clouds.yaml files and a separate playbook for the mirrors? | 19:54 |
| mriedem | zuul seems to not want to queue this up https://review.openstack.org/#/c/463526/ | 19:55 |
| corvus | also, gertty can't sync it. | 20:01 |
| corvus | there's no mention of it in zuul's logs; i just ran stream-events and left a recheck comment and it did not emit an event | 20:02 |
| corvus | there are tracebacks in the gerrit log about it | 20:03 |
| corvus | java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 818dc055b971cd2b78260fd17d0b9065 | 20:03 |
| clarkb | we had to rewrite nova-specs because it was a corrupted repo iirc | 20:04 |
| corvus | this is in nova-specs. istr some problems with that... yeah that... | 20:04 |
| clarkb | chances are that change needs a rebase onto current repo state | 20:04 |
| corvus | so is the repo ok now, but this change predates the fix? | 20:04 |
| clarkb | that is my hunch | 20:04 |
| mriedem | i rebased it a few minutes ago | 20:04 |
| clarkb | we did fix it | 20:04 |
| corvus | maybe it's that an earlier patchset is broken? | 20:05 |
| mriedem | maybe, gerrit UI won't show the older patch sets in the drop down | 20:05 |
| corvus | if that's the case, we'd probably need to "fix" the change by rewriting it's git history. it would probably be better if we didn't attempt that. if it's at all acceptable, i'd suggest we just start a new change and leave a comment pointing at this one for history. | 20:06 |
| mriedem | i can do that | 20:07 |
| clarkb | ++ | 20:07 |
| corvus | mriedem: whew. thanks. :) | 20:07 |
| mriedem | i just hope we don't have to do that for like 30 other nova specs | 20:07 |
| *** jamesdenton has quit IRC | 20:09 | |
| corvus | checking that now | 20:09 |
| *** agopi_ is now known as agopi | 20:12 | |
| corvus | mriedem: assuming gertty is a reasonable indicator of the problem, i subscribed to nova-specs and synced it. it synced 78 open changes and that one was the only problem. | 20:20 |
| corvus | now 79 specs | 20:21 |
| mriedem | ack, thanks for checking | 20:26 |
| *** irclogbot_2 has quit IRC | 20:27 | |
| mriedem | is there a way to change the owner of a change in gerrit? | 20:28 |
| mriedem | i changed the owner to the original | 20:28 |
| mriedem | *author | 20:28 |
| mriedem | but the owner is me | 20:28 |
| *** cfriesen has joined #openstack-infra | 20:28 | |
| cfriesen | if I get a -1 from Zuul in the "verified" section of a review, is there a way to see what it didn't like? | 20:28 |
| *** felipemonteiro has joined #openstack-infra | 20:28 | |
| tonyb | mriedem: nope the change owner isn't changeable. You'd need to abandon the first and upload a new chnage with a new change-id | 20:31 |
| anteaya | cfriesen: there should be a link to the logs on the gerrit page | 20:32 |
| mriedem | tonyb: yeah i see now, i guess i should have had the original owner re-propose | 20:34 |
| mriedem | now it looks like i want sriov bonding in nova | 20:34 |
| mriedem | :( | 20:34 |
| tonyb | :/ | 20:34 |
| cfriesen | anteaya: there weren't any logs, but it's now updated to show "cannot merge" which is straightforward enough. :) | 20:35 |
| anteaya | cfriesen: do you have an understanding as to why it cannot merge? | 20:37 |
| anteaya | or do you want to enjoy spending some time arriving at that understanding | 20:38 |
| anteaya | far be it from me to impose on your esthetic of enjoyment | 20:38 |
| cfriesen | anteaya: I think I've got it at this point, thanks | 20:38 |
| anteaya | welcome | 20:38 |
| *** kjackal has quit IRC | 20:40 | |
| clarkb | mordred: mordred[m]: any luck with clouds.yaml? | 20:41 |
| clarkb | mordred: I am back from lunch now and can take a look if the plane is not conducive | 20:42 |
| mordred[m] | clarkb I just landed, it would be more expedient for you to do it, but I can in a bit | 20:44 |
| clarkb | mordred[m]: is there an auth url I need to switch to or should I remove sto2 like I removed london? | 20:44 |
| mordred[m] | I tried the new form but it didn't work .. I'd remove for now | 20:45 |
| clarkb | ok | 20:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Cleanup Citycloud Sto2 https://review.openstack.org/612502 | 20:47 |
| clarkb | If I can get reviews on that I will direct enqueue it to the gate since it will unblock ansible and puppet | 20:47 |
| clarkb | infra-root ^ fyi | 20:48 |
| ianw | mriedem / corvus : (sorry, reading scrollback) phew nova-specs are ok. never really got an answer to that, and nobody ever added to the bug i filed suggesting they'd seen it too. a real mystery | 20:50 |
| ianw | the gerrit bug | 20:50 |
| *** jcoufal has quit IRC | 20:52 | |
| *** agopi has quit IRC | 20:52 | |
| *** kgiusti has left #openstack-infra | 20:53 | |
| clarkb | corvus: reading more about the server.number we may need that to match up with the myid files which are 01, 02, 03 I think | 20:55 |
| clarkb | the docs say myid should be a number between 1 and 255. Makes me wonder if the 01,02,03 will be interpreted as 1,2,3 anyway | 20:56 |
| clarkb | I'll fiddle with it, its a learning experience :) | 20:56 |
| *** ansmith has quit IRC | 20:56 | |
| corvus | clarkb: ack | 20:57 |
| cfriesen | is there a time delay between zuul setting "verified +1" and actually doing the merge? I have a commit that shows "needs verified label" at the same time as "Verified +1" | 21:07 |
| *** kopecmartin has joined #openstack-infra | 21:07 | |
| clarkb | cfriesen: you need a verified +2 to merge | 21:08 |
| *** bobh has quit IRC | 21:08 | |
| cfriesen | clarkb: so it's doing gate checks now? | 21:08 |
| clarkb | cfriesen: the check tests apply a +1, then the gate applies a +2. It is after the gate tests pass that zuul merges things | 21:08 |
| cfriesen | got it, thanks | 21:08 |
| clarkb | cfriesen: if it has a code review +2 and approved workflow vote it should be in the gate | 21:08 |
| *** bobh has joined #openstack-infra | 21:09 | |
| clarkb | cfriesen: https://zuul.openstack.org/status should give you the up to date status | 21:10 |
| *** bobh has quit IRC | 21:13 | |
| *** armax has quit IRC | 21:14 | |
| *** graphene has quit IRC | 21:18 | |
| *** graphene has joined #openstack-infra | 21:19 | |
| clarkb | ianw: was the sqlalchemy utils issue yesterday written to the afs wheel mirror? | 21:20 |
| clarkb | ianw: or was it constrained to some requirements job that builds all the wheels? iwasn't entirely sure skimming the scrollback | 21:20 |
| ianw | clarkb: umm, well again we got a corrupt .whl written out to afs. so i put an autohold on, started the periodic job and manually killed it, then tried as best i could to replicate what the gate does building the wheels in situ | 21:22 |
| ianw | i could not get a corrupt wheel out of that process. also it seems the output is quite "specifically" corrupt ... it's not like it suddenly half-way turns into /dev/random output | 21:23 |
| clarkb | ianw: are you leaning towards a broken wheel build itself then? | 21:23 |
| clarkb | that is not deterministic for some reason. This is a weird one | 21:23 |
| ianw | yeah. that's why i came up with https://github.com/pypa/pip/pull/5908 so that pip outputs the sha256 hash of the wheel it produces. that should be logged in our builds | 21:24 |
| *** slaweq has quit IRC | 21:24 | |
| ianw | so if we get a corrupt wheel, we can look at it's hash and go through all the build logs and see if pip thinks it outputted this file | 21:24 |
| clarkb | ++ | 21:24 |
| ianw | although i don't really like either answer :) either it did, and we have some weird, non-deterministic problem there, or it didn't, and somehow the file gets corrupt after pip, but before vos release | 21:25 |
| ianw | no reports of corruption today, right? http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-ubuntu-bionic/516ed62/ | 21:26 |
| ianw | i'll check the wheels | 21:26 |
| clarkb | ianw: haven't seen reports | 21:27 |
| ianw | urgh, my pip pull request fails on windows. desire to debug very low | 21:27 |
| *** graphene has quit IRC | 21:29 | |
| *** graphene has joined #openstack-infra | 21:30 | |
| *** stevebaker has quit IRC | 21:31 | |
| *** dmellado has quit IRC | 21:32 | |
| *** gouthamr has quit IRC | 21:32 | |
| ianw | so all the http://mirror/wheel/*/sqlalchemy-utils/SQLAlchemy_Utils-0.33.6-py2.py3-none-any.whl files are currently valid zip files | 21:33 |
| ianw | i might kick off a periodic build now anyway, so i can monitor it with the new change | 21:33 |
| ianw | clarkb: ovh-gra1 seems to have regressed to "everything failing" ... i'm not sure if we turn it off or just leave it ... | 21:36 |
| clarkb | ianw: ya... thinking maybe we turn it off ? and rely on amorin and dpawlik and ovh to continue to work it? | 21:37 |
| *** spsurya has quit IRC | 21:38 | |
| ianw | yeah, it doesn't seem there's much we can do externally | 21:38 |
| mordred[m] | We can always panic | 21:39 |
| *** rh-jelabarre has quit IRC | 21:41 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul-jobs master: Add a markdownlint job and role https://review.openstack.org/607691 | 21:44 |
| *** gfidente has quit IRC | 21:45 | |
| ianw | mordred[m]: my hitchhiker's guide says very clearly "don't panic" :) | 21:45 |
| *** priteau has quit IRC | 21:49 | |
| openstackgerrit | Ildiko Vancsa proposed openstack-infra/irc-meetings master: Removing the Upstream Institute meetings https://review.openstack.org/612525 | 21:49 |
| clarkb | I've updated /etc/openstack/clouds.yaml on bridge.o.o to remove Sto2 as I expect hte fix to merge prior to the 2200UTC ansible run (and it needs to be fixed for ansible to run to apply the fix) | 21:49 |
| mordred[m] | ianw good point | 21:49 |
| *** graphene has quit IRC | 21:51 | |
| *** armax has joined #openstack-infra | 21:51 | |
| *** graphene has joined #openstack-infra | 21:52 | |
| openstackgerrit | Merged openstack-infra/system-config master: Cleanup Citycloud Sto2 https://review.openstack.org/612502 | 21:57 |
| clarkb | yay now to see if ansible will run. I did check that the other two regions were still in the catalog | 21:57 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul-jobs master: Add a markdownlint job and role https://review.openstack.org/607691 | 21:59 |
| *** mriedem is now known as mriedem_away | 22:07 | |
| *** bnemec has quit IRC | 22:18 | |
| *** boden has quit IRC | 22:23 | |
| *** graphene has quit IRC | 22:37 | |
| clarkb | puppet didn't write the zookeeper config as I expected. `server.1=["1", "zk01.openstack.org"]:2888:3888` is wrong | 22:43 |
| *** felipemonteiro has quit IRC | 22:44 | |
| clarkb | ok its a puppet module version thing. we use a much older version than I looked at docs for and its not semvering? | 22:45 |
| *** carl_cai has joined #openstack-infra | 22:45 | |
| *** rcernin has joined #openstack-infra | 22:46 | |
| *** agopi has joined #openstack-infra | 22:47 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Removing the Upstream Institute meetings https://review.openstack.org/612525 | 22:55 |
| *** cfriesen has quit IRC | 22:56 | |
| * clarkb is learning things about zookeeper, but not why it is refusing to start as a cluster | 22:59 | |
| clarkb | there is a significan't lack of logging :/ | 22:59 |
| *** gyee has quit IRC | 23:00 | |
| *** tpsilva has quit IRC | 23:01 | |
| clarkb | ok figured it out I think. It binds to 127.0.1.1 beacuse that is what we have in /etc/hosts for the fqdn. Then no other servers can talk to that beacuse its localhost | 23:07 |
| *** eharney has quit IRC | 23:08 | |
| *** tosky has quit IRC | 23:09 | |
| clarkb | the internet suggests you set ip addresses instaed. so I will do that | 23:09 |
| clarkb | I'll confirm it works and push an update to site.pp | 23:10 |
| *** shoogz has quit IRC | 23:18 | |
| *** ansmith has joined #openstack-infra | 23:18 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Fix zk cluster members listing https://review.openstack.org/612535 | 23:23 |
| clarkb | infra-root ^ I have hand applied that and it works | 23:23 |
| clarkb | I'm going to disable zk0*.openstack.org in ansible/puppet now | 23:23 |
| *** shoogz has joined #openstack-infra | 23:23 | |
| *** mriedem_away has quit IRC | 23:27 | |
| *** rcernin_ has joined #openstack-infra | 23:28 | |
| *** rcernin has quit IRC | 23:30 | |
| *** smarcet has quit IRC | 23:30 | |
| *** rpioso is now known as rpioso|afk | 23:32 | |
| *** dklyle has quit IRC | 23:32 | |
| *** david-lyle has joined #openstack-infra | 23:32 | |
| *** david-lyle has quit IRC | 23:35 | |
| *** dklyle has joined #openstack-infra | 23:35 | |
| *** ssbarnea has quit IRC | 23:40 | |
| mordred | clarkb: I understand what's going on with citycloud now | 23:41 |
| mordred | patch coming | 23:41 |
| clarkb | mordred: k, if you can review the zk thing above after, that would be great | 23:43 |
| clarkb | I think that gets us to working zk cluster that we can use sometime later this week to get rid of nodepool.o.o | 23:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update clouds.yaml for citycloud with new auth info https://review.openstack.org/612538 | 23:50 |
| mordred | clarkb: woot! there's the citycloud fix ^^ | 23:50 |
| mordred | I will not look at zk | 23:50 |
| mordred | clarkb: seems reasonable | 23:51 |
| clarkb | os-c-c will do string substitution? | 23:52 |
| clarkb | but also Lon1 and Sto2 are not in the region list any longer so we'd have to add that back? and do the other regions support the new auth urls at this point? | 23:52 |
| mordred | yeah - largely to support this usecase - having per-region auth_urls is a thing people do | 23:52 |
| mordred | clarkb: the original patch to update the vendor profile came from tobias there ... | 23:53 |
| clarkb | ah | 23:53 |
| mordred | although this is the fix: https://review.openstack.org/#/c/612536/ | 23:53 |
| mordred | clarkb: also - the web dashboard lists the region-specific hosts | 23:53 |
| mordred | so I'm guessing they're good across the board | 23:53 |
| mordred | and yes - we'll need to add Sto2 and Lon1 back in | 23:54 |
| mordred | want me to do that in the same patch? | 23:54 |
| *** eumel8 has quit IRC | 23:54 | |
| clarkb | mordred: ya I think we should add it in the same patch | 23:54 |
| clarkb | mordred: but also double check La1 and Kna1 work with the per region stuff? | 23:54 |
| mordred | clarkb: verified | 23:55 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul-jobs master: Add a markdownlint job and role https://review.openstack.org/607691 | 23:55 |
| clarkb | mordred: we also want to add back in cloud launcher but that can probably go in a followup change | 23:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update clouds.yaml for citycloud with new auth info https://review.openstack.org/612538 | 23:56 |
| mordred | clarkb: oh - I already restored it :) | 23:56 |
| mordred | gah. I should put Lon back in too | 23:56 |
| clarkb | mordred: that works too | 23:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update clouds.yaml for citycloud with new auth info https://review.openstack.org/612538 | 23:58 |
| ianw | clarkb / tonyb : well, a fresh periodic build just finished also doesn't seem to have triggered any corruption in the file either. might have to just put it in monitor mode until we get something | 23:59 |
| clarkb | mordred: I need to followup with daniel about using those regions again | 23:59 |
| * clarkb makes a note to write that email tomorrow | 23:59 | |
| clarkb | ianw: :( | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!