| clarkb | I need to go sort out what dinner will be now. But will check on the vos release periodically and if it ends before I fall asleep I will unlock the lock file | 00:01 |
|---|---|---|
| *** sthussey has quit IRC | 00:01 | |
| *** owalsh has quit IRC | 00:01 | |
| *** dchen has joined #openstack-infra | 00:03 | |
| *** betherly has joined #openstack-infra | 00:08 | |
| *** jamesmcarthur has quit IRC | 00:09 | |
| *** mattw4 has quit IRC | 00:12 | |
| *** diablo_rojo has joined #openstack-infra | 00:12 | |
| *** betherly has quit IRC | 00:13 | |
| *** jamesmcarthur has joined #openstack-infra | 00:16 | |
| *** gyee has quit IRC | 00:19 | |
| *** dchen has quit IRC | 00:21 | |
| *** diablo_rojo has quit IRC | 00:25 | |
| *** diablo_rojo__ has joined #openstack-infra | 00:25 | |
| *** mriedem has quit IRC | 00:38 | |
| *** jamesmcarthur has quit IRC | 00:38 | |
| *** ekultails has quit IRC | 00:40 | |
| *** markvoelker has joined #openstack-infra | 00:41 | |
| *** markvoelker has quit IRC | 00:51 | |
| *** diablo_rojo__ has quit IRC | 00:54 | |
| *** igordc has quit IRC | 01:00 | |
| *** yamamoto has joined #openstack-infra | 01:01 | |
| *** markvoelker has joined #openstack-infra | 01:17 | |
| *** spsurya has joined #openstack-infra | 01:17 | |
| *** betherly has joined #openstack-infra | 01:21 | |
| *** yamamoto has quit IRC | 01:24 | |
| *** yamamoto has joined #openstack-infra | 01:25 | |
| *** betherly has quit IRC | 01:26 | |
| *** altlogbot_0 has quit IRC | 01:37 | |
| *** dchen has joined #openstack-infra | 01:37 | |
| *** altlogbot_1 has joined #openstack-infra | 01:38 | |
| *** markvoelker has quit IRC | 01:38 | |
| *** jamesmcarthur has joined #openstack-infra | 01:44 | |
| clarkb | vos release is still running (3 hours now) | 01:47 |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:52 | |
| *** rlandy|bbl is now known as rlandy | 01:55 | |
| *** rlandy has quit IRC | 01:57 | |
| *** yamamoto has quit IRC | 02:02 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:04 | |
| *** bhavikdbavishi has quit IRC | 02:06 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:06 | |
| *** apetrich has quit IRC | 02:08 | |
| *** weshay_dentist is now known as weshay | 02:13 | |
| *** ricolin has joined #openstack-infra | 02:16 | |
| *** jamesmcarthur has joined #openstack-infra | 02:18 | |
| *** betherly has joined #openstack-infra | 02:23 | |
| *** gregoryo has joined #openstack-infra | 02:25 | |
| *** betherly has quit IRC | 02:27 | |
| *** yamamoto has joined #openstack-infra | 02:28 | |
| *** slaweq has quit IRC | 02:35 | |
| *** markvoelker has joined #openstack-infra | 02:42 | |
| *** betherly has joined #openstack-infra | 02:43 | |
| *** betherly has quit IRC | 02:48 | |
| *** bhavikdbavishi has quit IRC | 02:49 | |
| *** hongbin has joined #openstack-infra | 02:49 | |
| *** rascasoft has quit IRC | 02:49 | |
| *** rascasoft has joined #openstack-infra | 02:52 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** jamesmcarthur has quit IRC | 02:59 | |
| *** betherly has joined #openstack-infra | 03:04 | |
| *** jamesmcarthur has joined #openstack-infra | 03:04 | |
| *** whoami-rajat has joined #openstack-infra | 03:07 | |
| *** betherly has quit IRC | 03:09 | |
| *** tdasilva_ has quit IRC | 03:14 | |
| *** tdasilva_ has joined #openstack-infra | 03:14 | |
| *** markvoelker has quit IRC | 03:15 | |
| *** jamesmcarthur has quit IRC | 03:17 | |
| *** yolanda has quit IRC | 03:21 | |
| *** yolanda has joined #openstack-infra | 03:22 | |
| *** ramishra has joined #openstack-infra | 03:34 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:35 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** thiago__ has joined #openstack-infra | 04:02 | |
| *** tdasilva_ has quit IRC | 04:04 | |
| *** ykarel|away has joined #openstack-infra | 04:05 | |
| AJaeger | infra-root, I just got lots of post_Failures - is our logserver down again? | 04:06 |
| ianw | AJaeger: hrm, can log in at least | 04:09 |
| AJaeger | all red again on zuul.opendev.org ;( | 04:09 |
| ianw | [Tue Aug 6 02:02:38 2019] EXT4-fs (dm-2): Remounting filesystem read-only | 04:09 |
| AJaeger | yeah ;( | 04:10 |
| ianw | Tue Aug 6 04:10:03 UTC 2019 | 04:10 |
| ianw | i guess i do the usual of moving it and fscking it ... on it | 04:10 |
| AJaeger | thanks, ianw | 04:10 |
| AJaeger | want to send an alert or something? | 04:11 |
| ianw | hopefully should be just a minute while i symlink it locally | 04:11 |
| ianw | for reference : the log -> http://paste.openstack.org/show/755550/ | 04:11 |
| *** hongbin has quit IRC | 04:11 | |
| ianw | ok, /srv/static/logs symlinked to /opt/tmp_srv_static_logs | 04:13 |
| ianw | logs should be caching there, i'll start a fsck in a screen | 04:13 |
| ianw | #status logs.openstack.org log volume offline; /srv/static/logs symlinked to /opt/tmp_srv_static_logs and fsck running in a screen | 04:15 |
| openstackstatus | ianw: unknown command | 04:15 |
| ianw | #status log logs.openstack.org log volume offline; /srv/static/logs symlinked to /opt/tmp_srv_static_logs and fsck running in a screen | 04:15 |
| openstackstatus | ianw: finished logging | 04:15 |
| ianw | will just let that go. with all the excitement around swift logs right now, i think we just bumble through with this until ti's replaced | 04:16 |
| AJaeger | yeah ;( | 04:17 |
| *** udesale has joined #openstack-infra | 04:38 | |
| *** jamesmcarthur has quit IRC | 04:40 | |
| ianw | i have to go afk for about a hour, it's at 40 something % ... will check in then | 04:41 |
| *** Lucas_Gray has joined #openstack-infra | 04:41 | |
| *** jhesketh has joined #openstack-infra | 04:54 | |
| *** janki has joined #openstack-infra | 05:04 | |
| *** jamesmcarthur has joined #openstack-infra | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:15 | |
| *** thiago__ has quit IRC | 05:15 | |
| *** thiago__ has joined #openstack-infra | 05:15 | |
| *** sshnaidm is now known as sshnaidm|afk | 05:19 | |
| *** markvoelker has joined #openstack-infra | 05:28 | |
| *** markvoelker has quit IRC | 05:33 | |
| *** Lucas_Gray has quit IRC | 05:35 | |
| *** tkajinam has quit IRC | 05:38 | |
| *** ramishra has quit IRC | 05:38 | |
| *** tkajinam has joined #openstack-infra | 05:38 | |
| *** jchhatbar has joined #openstack-infra | 05:38 | |
| *** ykarel|away is now known as ykarel | 05:39 | |
| *** jchhatbar has quit IRC | 05:41 | |
| *** janki has quit IRC | 05:41 | |
| *** jchhatbar has joined #openstack-infra | 05:42 | |
| *** jchhatbar has quit IRC | 05:43 | |
| *** smarcet has joined #openstack-infra | 05:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/674730 | 06:09 |
| *** jamesmcarthur has joined #openstack-infra | 06:11 | |
| *** yamamoto has quit IRC | 06:12 | |
| *** jamesmcarthur has quit IRC | 06:16 | |
| *** jtomasek has joined #openstack-infra | 06:17 | |
| *** yamamoto has joined #openstack-infra | 06:18 | |
| *** ramishra has joined #openstack-infra | 06:19 | |
| *** yamamoto has quit IRC | 06:24 | |
| *** yamamoto has joined #openstack-infra | 06:30 | |
| *** markvoelker has joined #openstack-infra | 06:36 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/674730 | 06:41 |
| *** yamamoto has quit IRC | 06:42 | |
| *** jamesmcarthur has joined #openstack-infra | 06:46 | |
| *** apetrich has joined #openstack-infra | 06:47 | |
| *** yamamoto has joined #openstack-infra | 06:47 | |
| *** jamesmcarthur has quit IRC | 06:51 | |
| *** odicha has joined #openstack-infra | 06:53 | |
| *** ginopc has joined #openstack-infra | 06:57 | |
| gmann | ianw: AJaeger : is log up ? | 06:58 |
| *** kopecmartin|off is now known as kopecmartin | 06:58 | |
| *** xek has joined #openstack-infra | 07:00 | |
| *** jamesmcarthur has joined #openstack-infra | 07:00 | |
| *** xek has quit IRC | 07:03 | |
| *** jamesmcarthur has quit IRC | 07:05 | |
| AJaeger | gmann: ianw fixed it, so safe to recheck... | 07:06 |
| *** slaweq has joined #openstack-infra | 07:07 | |
| gmann | thx. | 07:07 |
| AJaeger | to be accurate: Fix is on going but he has a temporary stop-gap in place, see above in backscroll | 07:08 |
| ianw | yes new logs should be ok | 07:08 |
| ianw | it's 90% finished fscking, so it says | 07:09 |
| ianw | i got eye drops at the optometrist that make close up things a bit blurry ... not great timing :) | 07:09 |
| *** markvoelker has quit IRC | 07:09 | |
| ianw | see what finishes first, the fsck or my vision returning :) | 07:09 |
| AJaeger | ianw: can we wait until fungi is up? | 07:18 |
| *** jbadiapa has joined #openstack-infra | 07:18 | |
| *** tosky has joined #openstack-infra | 07:24 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/674730 | 07:27 |
| *** jbadiapa has quit IRC | 07:27 | |
| *** tesseract has joined #openstack-infra | 07:30 | |
| *** jaosorior has quit IRC | 07:34 | |
| *** pgaxatte has joined #openstack-infra | 07:38 | |
| *** jamesmcarthur has joined #openstack-infra | 07:39 | |
| *** ralonsoh has joined #openstack-infra | 07:43 | |
| *** jamesmcarthur has quit IRC | 07:43 | |
| *** iurygregory has joined #openstack-infra | 07:44 | |
| *** rpittau|afk is now known as rpittau | 07:46 | |
| *** lucasagomes has joined #openstack-infra | 07:48 | |
| *** gregoryo has quit IRC | 07:50 | |
| *** cdent has joined #openstack-infra | 08:00 | |
| *** ykarel is now known as ykarel|lunch | 08:02 | |
| *** rcernin has quit IRC | 08:04 | |
| ianw | ok, final stats if anyone is interested http://paste.openstack.org/show/755553/ | 08:04 |
| *** ociuhandu has joined #openstack-infra | 08:05 | |
| ianw | alright, /srv/static/logs now back to being mounted, all old logs should be back | 08:05 |
| ianw | i'll sync the logs from the intermediate few hours now | 08:05 |
| *** ociuhandu has quit IRC | 08:05 | |
| *** pkopec has joined #openstack-infra | 08:06 | |
| *** tkajinam has quit IRC | 08:06 | |
| *** ccamacho has joined #openstack-infra | 08:07 | |
| *** ccamacho has quit IRC | 08:08 | |
| *** jamesmcarthur has joined #openstack-infra | 08:08 | |
| *** ccamacho has joined #openstack-infra | 08:08 | |
| *** lpetrut has joined #openstack-infra | 08:10 | |
| *** aedc has quit IRC | 08:11 | |
| *** electrofelix has joined #openstack-infra | 08:11 | |
| AJaeger | thanks, ianw ! | 08:12 |
| *** jamesmcarthur has quit IRC | 08:12 | |
| AJaeger | ianw: something for the next days: CAn we delete content of http://logs.openstack.org/lost+found/ ? Or what should be done there? | 08:13 |
| AJaeger | and please delete http://logs.openstack.org/help/ - that is completely bogus | 08:13 |
| ianw | oh, i'm sure we can ... nothing worth recovering i'd say | 08:13 |
| *** jamesmcarthur has joined #openstack-infra | 08:14 | |
| AJaeger | infra-root, any idea why we keep log files from 2018? Like http://logs.openstack.org/93/539993/2/check/ | 08:16 |
| AJaeger | looks like empty directory - but still... | 08:16 |
| AJaeger | so, log directories, not dirs... | 08:17 |
| *** ociuhandu has joined #openstack-infra | 08:17 | |
| ianw | hrm, dunno, maybe i've managed to copy back in something old? | 08:18 |
| *** jamesmcarthur has quit IRC | 08:20 | |
| AJaeger | or we had an error in the delete script so that files were deleted but not parent directories? | 08:21 |
| AJaeger | nothing urgent to fix... | 08:21 |
| *** dtantsur|afk is now known as dtantsur | 08:22 | |
| *** smarcet has quit IRC | 08:25 | |
| ianw | AJaeger: i'm running the removal script on just /93 | 08:27 |
| ianw | it's definitely clearing stuff out | 08:27 |
| *** ociuhandu has quit IRC | 08:30 | |
| AJaeger | why is that not done as part of the cronjob? | 08:30 |
| AJaeger | still seeing 2018 directories at top level at http://logs.openstack.org/93/?C=M;O=A | 08:32 |
| AJaeger | I guess the script is not done yet... | 08:32 |
| *** jaosorior has joined #openstack-infra | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:35 | |
| *** jaosorior has quit IRC | 08:37 | |
| ianw | yeah, it's going. it definitely should be part of the cron job; maybe there was just old stuff in /srv/tmp_static_logs? but i thought i checked it was clear | 08:38 |
| ianw | maybe i didn't, i couldn't be certain. and then i've copied it back? | 08:38 |
| *** fghaas has joined #openstack-infra | 08:39 | |
| *** jaosorior has joined #openstack-infra | 08:41 | |
| *** jamesmcarthur has joined #openstack-infra | 08:44 | |
| *** panda|pubholiday is now known as panda | 08:49 | |
| *** ociuhandu has joined #openstack-infra | 08:51 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Provide buildset.uuid in /builds API result https://review.opendev.org/674759 | 08:58 |
| *** Lucas_Gray has joined #openstack-infra | 08:59 | |
| ianw | it's finished and not cleared out those directories, i wonder why | 08:59 |
| ianw | lrwxrwxrwx 1 jenkins jenkins 9 Feb 5 2018 539993/3/gate/openstack-ansible-functional-centos-7/68dfd3d/logs/etc/host/network-scripts/ifup-isdn -> ifup-ippp | 09:02 |
| ianw | i think the script doesn't clear out symlinks | 09:02 |
| *** ociuhandu has quit IRC | 09:05 | |
| *** gfidente has joined #openstack-infra | 09:08 | |
| *** owalsh_ is now known as owalsh | 09:10 | |
| AJaeger | mmh, http://logs.openstack.org/93/539993/3/gate/openstack-ansible-functional-centos-7/68dfd3d/logs/etc/host/network-scripts/ does not show any content | 09:12 |
| AJaeger | So, no links displayed either? | 09:13 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix uneven distribution of jobs https://review.opendev.org/674762 | 09:14 |
| openstackgerrit | Ian Wienand proposed opendev/puppet-openstackci master: Log archive maintenance; also clear out symlinks https://review.opendev.org/674764 | 09:16 |
| ianw | AJaeger: yeah, that's what makes it a bit confusing; ^ i'm running that on /93 directory and stuff is disappearing | 09:16 |
| ianw | i guess we just never looked and noticed before | 09:16 |
| *** jamesmcarthur has quit IRC | 09:18 | |
| AJaeger | Thanks, ianw ! | 09:25 |
| openstackgerrit | Merged opendev/irc-meetings master: Free up unused BGPVPN meeting slot https://review.opendev.org/674308 | 09:26 |
| *** yamamoto has quit IRC | 09:27 | |
| *** Lucas_Gray has quit IRC | 09:27 | |
| *** yamamoto has joined #openstack-infra | 09:28 | |
| *** Lucas_Gray has joined #openstack-infra | 09:29 | |
| ianw | haha, so 549393/1/check/build-tox-manuals-checkbuild/266d1cd/publish-docs/www/robots.txt doesn't get removed because it matches the robots.txt exclusion | 09:32 |
| *** yamamoto has quit IRC | 09:33 | |
| ianw | i'll leave that as an exercise for anyone interested in advanced find command line magic :) but mostly all the 2018 stuff is cleared | 09:33 |
| ianw | i gotta stop staring at a bright screen ... ttfn | 09:34 |
| *** AJaeger has quit IRC | 09:38 | |
| *** AJaeger has joined #openstack-infra | 09:45 | |
| *** ociuhandu has joined #openstack-infra | 09:47 | |
| *** bhavikdbavishi has quit IRC | 09:48 | |
| *** rascasoft has quit IRC | 09:50 | |
| *** thiago__ has quit IRC | 09:50 | |
| *** yamamoto has joined #openstack-infra | 09:51 | |
| *** rascasoft has joined #openstack-infra | 09:52 | |
| *** jamesmcarthur has joined #openstack-infra | 09:55 | |
| *** elod has quit IRC | 09:58 | |
| *** jamesmcarthur has quit IRC | 09:59 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** mgoddard has quit IRC | 10:03 | |
| *** kjackal has joined #openstack-infra | 10:04 | |
| *** elod has joined #openstack-infra | 10:05 | |
| *** yamamoto has joined #openstack-infra | 10:05 | |
| *** yamamoto has quit IRC | 10:06 | |
| *** mgoddard has joined #openstack-infra | 10:07 | |
| *** ykarel_ has joined #openstack-infra | 10:17 | |
| *** ykarel|lunch has quit IRC | 10:19 | |
| *** pgaxatte has quit IRC | 10:20 | |
| *** kjackal has quit IRC | 10:23 | |
| *** kjackal has joined #openstack-infra | 10:25 | |
| *** ykarel_ is now known as ykarel | 10:27 | |
| *** ykarel_ has joined #openstack-infra | 10:31 | |
| *** rfolco|ruck is now known as rfolco|doctor | 10:33 | |
| *** ykarel has quit IRC | 10:34 | |
| *** elod has quit IRC | 10:36 | |
| *** ykarel_ is now known as ykarel | 10:42 | |
| *** rascasoft has quit IRC | 10:43 | |
| *** rascasoft has joined #openstack-infra | 10:46 | |
| *** elod has joined #openstack-infra | 10:48 | |
| *** yamamoto has joined #openstack-infra | 10:48 | |
| *** kjackal has quit IRC | 10:55 | |
| *** jamesmcarthur has joined #openstack-infra | 10:55 | |
| *** ramishra has quit IRC | 11:00 | |
| *** jamesmcarthur has quit IRC | 11:00 | |
| *** jaosorior has quit IRC | 11:02 | |
| *** ramishra has joined #openstack-infra | 11:02 | |
| *** tdasilva has joined #openstack-infra | 11:03 | |
| *** kjackal has joined #openstack-infra | 11:09 | |
| *** ociuhandu has quit IRC | 11:13 | |
| *** ociuhandu has joined #openstack-infra | 11:16 | |
| *** ociuhandu has quit IRC | 11:21 | |
| *** jaosorior has joined #openstack-infra | 11:31 | |
| *** jamesmcarthur has joined #openstack-infra | 11:31 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| *** jamesmcarthur has joined #openstack-infra | 11:35 | |
| *** ociuhandu has joined #openstack-infra | 11:35 | |
| *** ociuhandu has quit IRC | 11:40 | |
| *** yamamoto has quit IRC | 11:42 | |
| *** rlandy has joined #openstack-infra | 11:46 | |
| *** ykarel is now known as ykarel|afk | 11:47 | |
| *** ociuhandu has joined #openstack-infra | 11:49 | |
| *** panda is now known as panda|lunch | 11:49 | |
| *** ociuhandu has quit IRC | 12:01 | |
| *** zbr is now known as zbr|lunch | 12:01 | |
| *** Lucas_Gray has quit IRC | 12:02 | |
| *** markvoelker has joined #openstack-infra | 12:04 | |
| *** Lucas_Gray has joined #openstack-infra | 12:04 | |
| *** markvoelker has quit IRC | 12:06 | |
| *** pgaxatte has joined #openstack-infra | 12:06 | |
| *** markvoelker has joined #openstack-infra | 12:06 | |
| *** rh-jelabarre has joined #openstack-infra | 12:10 | |
| *** rh-jelabarre has quit IRC | 12:10 | |
| *** rh-jelabarre has joined #openstack-infra | 12:11 | |
| *** ykarel|afk is now known as ykarel | 12:11 | |
| mordred | ianw: awesome. I betcha fungi has the find wizard nuggets | 12:12 |
| *** auristor has quit IRC | 12:13 | |
| *** yamamoto has joined #openstack-infra | 12:14 | |
| openstackgerrit | Merged opendev/irc-meetings master: Free up unused Designate meeting slot https://review.opendev.org/674301 | 12:16 |
| *** jamesmcarthur has quit IRC | 12:20 | |
| *** Lucas_Gray has quit IRC | 12:22 | |
| *** auristor has joined #openstack-infra | 12:23 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix uneven distribution of jobs https://review.opendev.org/674762 | 12:27 |
| *** udesale has quit IRC | 12:32 | |
| *** udesale has joined #openstack-infra | 12:32 | |
| *** hongbin has joined #openstack-infra | 12:34 | |
| *** zbr|lunch is now known as zbr | 12:38 | |
| *** ykarel_ has joined #openstack-infra | 12:42 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** ykarel has quit IRC | 12:44 | |
| *** elod is now known as elod_off | 12:50 | |
| *** ykarel_ is now known as ykarel | 12:54 | |
| *** aaronsheffield has joined #openstack-infra | 12:56 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:58 | |
| *** mriedem has joined #openstack-infra | 13:00 | |
| *** eharney has joined #openstack-infra | 13:00 | |
| *** ekultails has joined #openstack-infra | 13:02 | |
| *** aedc has joined #openstack-infra | 13:03 | |
| *** Lucas_Gray has joined #openstack-infra | 13:07 | |
| *** ociuhandu has joined #openstack-infra | 13:08 | |
| *** sthussey has joined #openstack-infra | 13:11 | |
| *** panda|lunch is now known as panda | 13:12 | |
| *** ykarel_ has joined #openstack-infra | 13:16 | |
| *** jamesmcarthur has joined #openstack-infra | 13:17 | |
| *** ykarel has quit IRC | 13:18 | |
| *** ykarel_ is now known as ykarel|afk | 13:19 | |
| *** yamamoto has joined #openstack-infra | 13:19 | |
| *** iurygregory has quit IRC | 13:20 | |
| *** iurygregory has joined #openstack-infra | 13:20 | |
| rm_work | dtroyer: what channel is good for client discussion? :D | 13:23 |
| rm_work | nm got it | 13:24 |
| dtroyer | rm_work: #openstac-sdks as pretty much all of the client-type folks in it | 13:24 |
| *** hongbin has quit IRC | 13:24 | |
| *** smarcet has joined #openstack-infra | 13:27 | |
| AJaeger | fungi, mordred, what do you think of https://review.opendev.org/674764 ? | 13:31 |
| mordred | AJaeger: lgtm - I definitely want an ack from fungi though | 13:32 |
| fungi | yeah, sorry, mostly trying to prep for an advisory going out in an hour or so but will try to look real quick | 13:33 |
| *** yamamoto has quit IRC | 13:34 | |
| mordred | fungi: I don't think there is a rush - I just mostly always want you to look at things related to the find command :) | 13:34 |
| fungi | yep, looks like it should be safe. good catch! | 13:36 |
| *** ykarel_ has joined #openstack-infra | 13:38 | |
| *** ykarel|afk has quit IRC | 13:40 | |
| *** ociuhandu has quit IRC | 13:40 | |
| fghaas | Hello there, familiar faces. :) I've been looking to find documentation on rate limits that apply to accessing the OpenDev Git repos via HTTPS, but have been unsuccessful thus far. Could someone point me to such documentation, or else let me know if no such rate limiting exists? | 13:43 |
| AJaeger | thanks! | 13:43 |
| *** Lucas_Gray has quit IRC | 13:43 | |
| AJaeger | fghaas: what evil thinks do you want to do? ;) | 13:43 |
| *** yamamoto has joined #openstack-infra | 13:44 | |
| *** yamamoto has quit IRC | 13:44 | |
| *** yamamoto has joined #openstack-infra | 13:45 | |
| mordred | fghaas: we do not impose rate limits on accessing them via https | 13:45 |
| *** ykarel__ has joined #openstack-infra | 13:45 | |
| * mordred hasn't seen fghaas in quite some time | 13:46 | |
| *** yamamoto has quit IRC | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:46 | |
| *** yamamoto has quit IRC | 13:46 | |
| fghaas | AJaeger: :) I have no evil in mind; I just heard from an openstack-ansible user that they ran into connection drops from opendev earlier this morning. Not being aware of any rate limits and never having run into similar issues, my hunch was that the problem is on their end. I guess you're supporting that hypothesis? | 13:46 |
| *** Lucas_Gray has joined #openstack-infra | 13:47 | |
| fghaas | ("this morning" being your morning and mine; i.e. CEST) | 13:47 |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| mordred | yes. we are not doing things that would cause those to happen | 13:47 |
| mordred | at least, not purposely | 13:47 |
| fghaas | mordred: perfect, thanks! | 13:47 |
| *** ykarel_ has quit IRC | 13:47 | |
| *** ykarel__ is now known as ykarel | 13:48 | |
| * fghaas concurs; we should fix that — any plans to return to Stockholm in October mordred? | 13:48 | |
| mordred | fghaas: why yes! I'll be doing a zuul workshop | 13:48 |
| fghaas | well there we go! That's not far off then. | 13:48 |
| mordred | fghaas: I'll also be in the nordics from Aug 17 until then, so we'll at least be in a closer timezone | 13:49 |
| fghaas | oh, whereabouts exactly? | 13:49 |
| mordred | fghaas: starting in Goteborg for the Gerrit User Conference, then Sandy and I are going to drive up the Norwegian Coast to Tromso | 13:50 |
| *** iurygregory_ has joined #openstack-infra | 13:50 | |
| *** iurygregory has quit IRC | 13:50 | |
| *** yamamoto has quit IRC | 13:51 | |
| *** iurygregory_ is now known as iurygregory | 13:52 | |
| mordred | fghaas: map https://waterwanders.com/map/ and tour dates https://waterwanders.com/tour :) | 13:52 |
| fghaas | Excellent plan, that. | 13:53 |
| clarkb | while we may not rate limit the cloud might? rax definitely did based on flavor size not sure about vexxhost. Also we have been battling an OOM situation that seems caused by nova handling being worth ~1.1GB per request | 13:54 |
| clarkb | so details on what the problem is may aid debugging potential issues | 13:55 |
| paladox | mordred ah your joining the Gerrit User Conference? :) | 13:55 |
| mordred | paladox: yes indeed - I'll even be presenting and stuff | 13:55 |
| openstackgerrit | Merged opendev/puppet-openstackci master: Log archive maintenance; also clear out symlinks https://review.opendev.org/674764 | 13:56 |
| paladox | :) (i'll be participating in the hackathon remotely) | 13:56 |
| fghaas | clarkb: thanks for weighing in; the info I have is rather sparse though I'm afraid. Basically it seems to manifest itself as intermittent "fatal: unable to access 'https://opendev.org/openstack/cinder/': Couldn't connect to server" errors when OSA tries to git clone for its venv build. Happens for other repos too when it happens, but not for all of them — which raised the possibility of the reason being something like a source | 13:59 |
| fghaas | IP based rate limit. | 13:59 |
| fghaas | Like I said, I really don't want anyone to chase a red herring, a rate limit or unreliable uplink on their end is still the best hypothesis that fits the available data, as far as I can see. | 14:00 |
| clarkb | if unable to connect at all I doubt that is the OOm issue unless we had a gitea crash | 14:00 |
| fghaas | clarkb: right. | 14:00 |
| clarkb | fghaas: if you have a timestamp we can compare against any possible OOMs/crashes (lastone I know of was last week) | 14:01 |
| fghaas | I don't, but I can ask and if I get any meaningful info I'd chuck a message onto the list; would that be OK? | 14:01 |
| clarkb | sure | 14:02 |
| fghaas | gotcha, will do. Thanks again! | 14:02 |
| AJaeger | infra-root, is Linaro-london setup now fine? https://review.opendev.org/671445 | 14:05 |
| *** roman_g has joined #openstack-infra | 14:07 | |
| *** dpawlik has quit IRC | 14:10 | |
| *** liuyulong has joined #openstack-infra | 14:10 | |
| *** ykarel has quit IRC | 14:11 | |
| *** ociuhandu has joined #openstack-infra | 14:11 | |
| *** ykarel has joined #openstack-infra | 14:12 | |
| *** altlogbot_1 has quit IRC | 14:12 | |
| *** iurygregory has quit IRC | 14:14 | |
| *** altlogbot_3 has joined #openstack-infra | 14:14 | |
| *** iurygregory has joined #openstack-infra | 14:15 | |
| *** ociuhandu has quit IRC | 14:17 | |
| *** Lucas_Gray has quit IRC | 14:17 | |
| *** rfolco|doctor is now known as rfolco | 14:19 | |
| *** rfolco is now known as rfolco|ruck | 14:19 | |
| *** Lucas_Gray has joined #openstack-infra | 14:19 | |
| *** ramishra has quit IRC | 14:23 | |
| *** rascasoft has quit IRC | 14:25 | |
| *** ociuhandu has joined #openstack-infra | 14:27 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 14:28 |
| *** rascasoft has joined #openstack-infra | 14:29 | |
| *** michael-beaver has joined #openstack-infra | 14:30 | |
| *** Lucas_Gray has quit IRC | 14:31 | |
| *** Lucas_Gray has joined #openstack-infra | 14:32 | |
| *** smarcet has quit IRC | 14:40 | |
| dtroyer | I think I need a bit of debugging help… I am trying to create our release branch in the StarlingX repos, using https://opendev.org/starlingx/tis-repo as the test. 'git push gerrit test-branch' is being rejected. I am able to push a signed tag. More details are in http://paste.openstack.org/show/755569/ | 14:41 |
| dtroyer | I think this is the first time we've branched since the opendev transition, I can't think of how that changes anything other than the URLs involved… I am using freshly cloned repos. | 14:43 |
| clarkb | dtroyer: due to the way gerrit acls work you have to have force push accesa to create a new branch via git push. Instead if you have the create ref acl you can click buttons in the web ui to create abranch or usethe api | 14:43 |
| clarkb | (and we really prefer avoiding force push perms as it implies a bunch of other stuff) | 14:43 |
| dtroyer | clarkb: hmm…ok. Is that a change from say the beginning of the year? MY script from the last time doesn't force | 14:44 |
| clarkb | dtroyer: no I think it hasbeeen this way for a while. You dont have to git push -f, it is justhow gerrit acls work iirc | 14:44 |
| clarkb | crwating and deleting branch refs in gerrit via git requires that permission aiui | 14:45 |
| clarkb | the workaround is to use the ui or api with the create ref permission | 14:45 |
| *** yamamoto has joined #openstack-infra | 14:45 | |
| dtroyer | I'm wondering how this has worked in the past? | 14:46 |
| sean-k-mooney | dtroyer: you may have inherited mor permission on other repos? | 14:46 |
| dtroyer | sean-k-mooney: it's the same set of repos and the same script | 14:47 |
| sean-k-mooney | oh ok | 14:47 |
| dtroyer | I know it worked last Oct/Nov when we branched the first release, and at least around Feb for a couple of feature branches | 14:47 |
| sean-k-mooney | dtroyer: where you previously in a gerrit admin group | 14:47 |
| clarkb | though rereading the docs they only suggest adding force perms if adding create | 14:47 |
| dtroyer | no that I know of | 14:47 |
| dtroyer | so it sounds like the correct way forward is to use the Gerrit API to do this? | 14:48 |
| *** yamamoto has quit IRC | 14:50 | |
| sean-k-mooney | dtroyer: are you trying to do a release the old way by locally signing and pushing a tag? | 14:50 |
| dtroyer | The StarlingX release process is similar but much simpler than the OpenStack process. I'm trying to create the release branches | 14:51 |
| *** Lucas_Gray has quit IRC | 14:52 | |
| dtroyer | a signed tag is included (and works) but can be ignored for this problem I think | 14:52 |
| clarkb | tags are a separate set of permissions | 14:52 |
| sean-k-mooney | dtroyer: i assume starlingx does not use the release repo or have its own to automate some of this | 14:53 |
| sean-k-mooney | the workflow i used to use was use gerrit to create branch | 14:53 |
| clarkb | rereading thw docs I thinj I understand better | 14:53 |
| clarkb | to push new commits you need push perms | 14:53 |
| sean-k-mooney | switch to branch locally tag and sign then push tag | 14:53 |
| clarkb | to create a ref only you need create perms | 14:53 |
| sean-k-mooney | but i havent really done it that way in quite a while | 14:53 |
| clarkb | if your git push contains refs unknown to geerrit the you mist have both | 14:54 |
| clarkb | I think you only have create | 14:54 |
| fungi | clarkb: dtroyer there is also a gerrit api method for branch creation, so the webui is not *strictly* required for that | 14:54 |
| sean-k-mooney | since that requires you to also manually build and sign a release on pypi | 14:54 |
| clarkb | fungi correct | 14:54 |
| fungi | ahh, i see that came up later in discussion | 14:54 |
| clarkb | using the apis you are forced to create thw ref using an existing commit | 14:54 |
| sean-k-mooney | fungi: do we still allow ssh access so you can do that via the commandline | 14:54 |
| clarkb | I think that the newbranch must contain unknown commits | 14:55 |
| sean-k-mooney | to make it scripable without curl | 14:55 |
| fungi | sean-k-mooney: yes, ssh api is available as is rest api | 14:55 |
| dtroyer | clarkb: dang, yes it did. ok, that's the problem, and I'm unsure of why that got created out of order (the .gitreview update) | 14:56 |
| *** eernst has joined #openstack-infra | 14:56 | |
| sean-k-mooney | dtroyer: ah ya that makes sense | 14:56 |
| sean-k-mooney | the .gitreview update is usally the thing i tag too. | 14:57 |
| dtroyer | thanks for the help… sean-k-mooney: I'm tagging the branch point so we can also easily get the delta on master since the branch. I'm sure there is another way to do that too... | 14:58 |
| clarkb | vos release on fedora mirror did eventually compelte | 14:58 |
| fungi | yeah, you would push the tag for the commit on master then branch from that tag and then add the .gitreview change | 14:58 |
| clarkb | I don't know how long it took | 14:58 |
| *** andreykurilin_ has joined #openstack-infra | 14:58 | |
| sean-k-mooney | well you can just compare the branches with git log | 14:59 |
| clarkb | I am leaning towards unlocking the lockfile and letting the cron run and if it breaks again deal with it at thatpoint? | 14:59 |
| *** slittle1 has joined #openstack-infra | 14:59 | |
| clarkb | corvus: ^ any concern with that? Now that I know its ~12 hours for the krb creds I'm less concerned | 14:59 |
| fungi | clarkb: you could try a second manual run to see how long it runs? | 14:59 |
| clarkb | fungi: ya the one I don't know how long it took was the second one | 14:59 |
| clarkb | the first one took about 7 hours I think | 15:00 |
| fungi | btu yes, assuming it's fast i would just remove the lock and let it go | 15:00 |
| clarkb | no second one took about 4 hours before I gave up on it and went to sleep | 15:00 |
| clarkb | or rather stopped computing | 15:00 |
| clarkb | fungi: I think the sheer size of the volume means it is slow | 15:01 |
| clarkb | it is currently > 600GB | 15:01 |
| corvus | clarkb: 1sec | 15:01 |
| clarkb | which is about 200GB lighter than a few days ago | 15:01 |
| *** diablo_rojo has joined #openstack-infra | 15:02 | |
| fungi | vos release on that volume with minimal/no changes would be much faster, i thought | 15:02 |
| clarkb | the second run did update one aarch64 image for atomic | 15:02 |
| clarkb | (there is enough churn in there that with the previous slow vos releases things have updated | 15:02 |
| corvus | clarkb: sounds good; we might be able to determine the runtime, but since it's unlikely to be > 7 hours, i don't think it would change anything | 15:04 |
| *** Lucas_Gray has joined #openstack-infra | 15:14 | |
| *** ccamacho has quit IRC | 15:15 | |
| *** pgaxatte has quit IRC | 15:15 | |
| *** ykarel is now known as ykarel|away | 15:19 | |
| openstackgerrit | Merged zuul/zuul master: Fix uneven distribution of jobs https://review.opendev.org/674762 | 15:23 |
| pabelanger | infra-root: I have some time this week to help with https://review.opendev.org/673304/ which updates the zuul tenant to use ansible 2.8 for all jobs. Do you think we could setup a time to land it? I'm 99.98% sure nothing should break. | 15:25 |
| *** nfakhir has quit IRC | 15:26 | |
| *** odicha has quit IRC | 15:26 | |
| clarkb | I'm ok with landing it whenever as long as people have a few cycles to help debug anything that pops up that day (even if the result is "here is how you pin to 2.7) | 15:26 |
| clarkb | I will be out tomorrow, but can help today and thursday/friday | 15:26 |
| pabelanger | yah, my thoughts were if a job failed under 2.8, is we could just roll that job back to use 2.7. But from what I see, everything is likely to work | 15:28 |
| pabelanger | but want to make sure people know of the change and are able to help if something is broke | 15:29 |
| *** gyee has joined #openstack-infra | 15:29 | |
| corvus | pabelanger, clarkb: this is for the zuul tenant, there's like 12 jobs, we should just land it | 15:32 |
| corvus | just announce that in #zuul and do it :) | 15:32 |
| clarkb | oh if it is for zuul then ya I think we can just land it | 15:32 |
| clarkb | zuulians are more than capable of debugging those problems I expect | 15:33 |
| clarkb | done | 15:33 |
| pabelanger | Yah, just the zuul tenant. Doing openstack is a little more work then I can sign up for right now | 15:33 |
| pabelanger | thanks! | 15:33 |
| corvus | pabelanger: that's okay; if no one signs up for doing more work, we'll just send an email and switch it | 15:34 |
| pabelanger | ++ | 15:34 |
| *** cdent has left #openstack-infra | 15:35 | |
| *** smarcet has joined #openstack-infra | 15:35 | |
| *** ociuhandu has quit IRC | 15:36 | |
| *** mattmceuen has joined #openstack-infra | 15:38 | |
| openstackgerrit | Merged openstack/project-config master: Default zuul tenant to use ansible 2.8 for jobs https://review.opendev.org/673304 | 15:47 |
| *** igordc has joined #openstack-infra | 15:48 | |
| *** ociuhandu has joined #openstack-infra | 15:48 | |
| mattmceuen | GM team - could I please be added to the two new airship project groups, so I can seed in the right team members? | 15:48 |
| mattmceuen | https://review.opendev.org/#/admin/groups/2036,members | 15:48 |
| mattmceuen | https://review.opendev.org/#/admin/groups/2037,members | 15:48 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 15:49 |
| clarkb | mattmceuen: done | 15:50 |
| mattmceuen | thanks clarkb! | 15:50 |
| fungi | what's a gm? | 15:51 |
| pabelanger | good morning? | 15:51 |
| fungi | aha! | 15:51 |
| fungi | i was wondering for a sec who the gm team was ;) | 15:51 |
| * fungi sighs | 15:52 | |
| fungi | moar coffee | 15:52 |
| pabelanger | or maybe guten morgen :D | 15:52 |
| mattmceuen | yes! or evening, if you're lucky enough to be at that point of the workday ;-) | 15:52 |
| *** kjackal has quit IRC | 15:52 | |
| AJaeger | nearly 6 pm for "Guten Abend" ;) | 15:52 |
| fungi | indeed | 15:52 |
| mattmceuen | wow - versatile little abbreviation, that | 15:52 |
| fungi | time to head to the bier garten | 15:52 |
| fungi | s/ // | 15:53 |
| mattmceuen | To cut down on ambiguity, from now on I'll just use | 15:53 |
| mattmceuen | g | 15:53 |
| * mattmceuen you can fill in your TZ-appropriate second letter | 15:53 | |
| clarkb | I'm releasing the fedora mirror lockfile now | 15:58 |
| clarkb | will keep an eye on it today | 15:58 |
| *** lucasagomes has quit IRC | 15:59 | |
| *** Wryhder has joined #openstack-infra | 16:00 | |
| *** Lucas_Gray has quit IRC | 16:01 | |
| *** Wryhder is now known as Lucas_Gray | 16:01 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 16:02 |
| *** ociuhandu has quit IRC | 16:03 | |
| *** ginopc has quit IRC | 16:04 | |
| *** weifan has joined #openstack-infra | 16:06 | |
| *** eernst has quit IRC | 16:06 | |
| *** mattw4 has joined #openstack-infra | 16:09 | |
| *** igordc has quit IRC | 16:12 | |
| *** Lucas_Gray has quit IRC | 16:12 | |
| *** iurygregory has quit IRC | 16:13 | |
| *** owalsh_ has joined #openstack-infra | 16:17 | |
| *** mattw4 has quit IRC | 16:19 | |
| *** ricolin has quit IRC | 16:20 | |
| *** mattw4 has joined #openstack-infra | 16:20 | |
| *** owalsh has quit IRC | 16:22 | |
| *** weifan has quit IRC | 16:23 | |
| *** weifan has joined #openstack-infra | 16:24 | |
| *** hongbin has joined #openstack-infra | 16:24 | |
| *** tosky has quit IRC | 16:25 | |
| Shrews | infra-root: the dfw, iad, and ord regions of rax have been significantly cleaned of the leaked image objects. Some objects still show up in the object listing, yet don't appear to actually exist if I do a 'show' or 'delete' on them (e.g., any of the objects in IAD in the 'images' container). I'm going to assume that's some sort of swift bug and not worry about those | 16:27 |
| clarkb | Shrews: swift is technically eventually consistent | 16:28 |
| clarkb | maybe check it sometime in the future? | 16:28 |
| mordred | yeah | 16:28 |
| *** hongbin has quit IRC | 16:29 | |
| *** weifan has quit IRC | 16:29 | |
| Shrews | mordred: fyi, i may be enlisting your assistance in tracking down the sdk bit that leaves these objects lying around | 16:29 |
| *** jamesmcarthur has quit IRC | 16:31 | |
| *** eernst has joined #openstack-infra | 16:31 | |
| *** owalsh_ has quit IRC | 16:31 | |
| *** owalsh has joined #openstack-infra | 16:31 | |
| *** markvoelker has quit IRC | 16:32 | |
| *** tdasilva has quit IRC | 16:32 | |
| mordred | Shrews: neat! | 16:32 |
| * Shrews detects hints of sarcasm with a mellow overtone of hatred :) | 16:34 | |
| mordred | Shrews: line 297 in openstack/image/v2/_proxy.py | 16:34 |
| mordred | Shrews: _should_ be deleting the object when we're done with it | 16:34 |
| Shrews | mordred: neat! | 16:35 |
| Shrews | will try to debug that after lunch and meetings | 16:35 |
| mordred | Shrews: cool. it's also possible there is too much spaghetti in there | 16:36 |
| *** gfidente has quit IRC | 16:38 | |
| *** rpittau is now known as rpittau|afk | 16:41 | |
| rm_work | review.opendev.org feeling slow/unresponsive for others? | 16:42 |
| rm_work | been this way for like an hour or so | 16:42 |
| rm_work | not sure if it's been reported / is being dealt with / is just me | 16:42 |
| rm_work | (I don't think it's just me) | 16:42 |
| clarkb | I've used it lightly this morning and it seemed fine. I'll check java meolody | 16:43 |
| clarkb | memory use has increased over the course of the day and the garbage collector is being invoked more frequently | 16:44 |
| *** markvoelker has joined #openstack-infra | 16:44 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Remove support for ansible 2.5 https://review.opendev.org/650431 | 16:44 |
| clarkb | does look like the old runaway state yet, but we should monitor it | 16:44 |
| AJaeger | rm_work: yes, looks kind of slow... | 16:44 |
| rm_work | taking about 15s for me to load zuul data, sticks on "working..." at the top | 16:45 |
| clarkb | *does not look like the old runaway state yet I mean | 16:45 |
| rm_work | which makes the rest of the page unresponsive for a while | 16:45 |
| clarkb | rm_work: the gerrit js is likely blocking on a resposne from the api for your request | 16:45 |
| rm_work | although i feel like it was a little worse a bit ago, so might just be a usage spike | 16:46 |
| clarkb | rm_work: it is likely tied to when it is GCing | 16:46 |
| rm_work | yeah k | 16:46 |
| clarkb | which has started being measurable since about 15:45UTC | 16:46 |
| clarkb | so the last hour or so | 16:46 |
| rm_work | ;) | 16:47 |
| clarkb | we've seen gerrit recover from such states in the past and also seen it not recover from such states in the past | 16:47 |
| clarkb | the active thread count is also way up | 16:47 |
| * rm_work crosses fingers | 16:47 | |
| clarkb | (does that imply a larger number of requests?) | 16:47 |
| *** e0ne has quit IRC | 16:48 | |
| rm_work | keeping a gerrit window open keeps a websocket connection open, right? for the auto-refresh stuff? | 16:48 |
| clarkb | http hits/ minute hasn't chnged too much I think that means thread count may be higher because we are taking longer to response | 16:48 |
| clarkb | rm_work: I'm not sure if that is a websocket or a poll. I don't think our web proxy handles websockets directly so it may be a poll? | 16:48 |
| fungi | `gerrit show-queue` is longish too | 16:48 |
| rm_work | hmm k | 16:48 |
| rm_work | i got hit up by the security team a couple days ago because they said my machine had "unusual ipv6 traffic" to review01.openstack.org | 16:49 |
| rm_work | lol | 16:49 |
| clarkb | rm_work: the sad thing about that is that ipv6 http traffic is apparently abnormal :( | 16:49 |
| *** ociuhandu has joined #openstack-infra | 16:49 | |
| clarkb | the GC time graph shows it falling off fwiw | 16:49 |
| fungi | it's only unusual because you're actually doing work and not reading social media all day ;) | 16:49 |
| rm_work | (I keep like 30 review tabs open at a time, guessing it was the autorefresh polling/socket) | 16:49 |
| rm_work | and i just leave it like that 24/7 heh | 16:50 |
| clarkb | cpu graph also shows it falling off in the short term | 16:50 |
| fungi | yeah, likely so. i think the version of ff i'm running suspends js for inactive tabs | 16:50 |
| fungi | and then resumes it when i bring up the tab | 16:50 |
| rm_work | i'm on chrome, wonder if it does or not | 16:50 |
| rm_work | the way FF does it seems smart ;) | 16:50 |
| clarkb | I am hopeful this will self correct looking at these graphs | 16:51 |
| rm_work | need to use chrome for hangouts >_< but could use both | 16:51 |
| fungi | rm_work: well, it's a tradeoff, definitely means more lag to get js autoupdating/notifications/et cetera when switching tabs | 16:51 |
| fungi | but saves a lot of cpu load | 16:51 |
| rm_work | also chrome is using 8.2G of RAM on my 16G macbook, so | 16:52 |
| fungi | yikes | 16:52 |
| clarkb | /changes/?q=is:watched+is:merged&n=25&O=81&S=25 GET is a slwo query someone ran | 16:52 |
| rm_work | my macbook is pretty much constantly at ~9G of active swap | 16:52 |
| rm_work | this is my life | 16:52 |
| rm_work | they won't get me one of the 32G ones :D | 16:52 |
| clarkb | what is O=81? | 16:52 |
| mordred | rm_work: why would you ever get a person in a technical role a computer with memory in it | 16:52 |
| * rm_work shrugs | 16:53 | |
| clarkb | n=25 is give me 25 results and S=25 is give me the results after the first 25 iirc | 16:53 |
| mordred | clarkb: offset? | 16:53 |
| mordred | oh. yeah - that's what s is I guess | 16:53 |
| clarkb | mordred: I thought that was S but I'm pulling up api docs now | 16:53 |
| mordred | I don't know what o is | 16:53 |
| rm_work | ah 12G swap right now rofl | 16:53 |
| rm_work | not even running vmware <_< | 16:53 |
| AJaeger | how can https://review.opendev.org/#/c/645331 be in merge-conflict? Or is that a side-effect of gerrit's slowness? That change is on top of a change to another one that is on the current branch, so nothing to rebase... | 16:54 |
| clarkb | ya not finding any info on the O parameter | 16:57 |
| clarkb | o is used to apply search options but they are named not numbers | 16:57 |
| clarkb | maybe it is a paging thing | 16:57 |
| fungi | AJaeger: it could still be in merge conflict if other changes have merged to the branch since the parent and conflict with the change | 16:57 |
| AJaeger | fungi: nothing merged to that branch at all... | 16:57 |
| clarkb | AJaeger: both zuul and gerrit seem to think it is in conflict | 16:57 |
| *** ociuhandu has quit IRC | 16:57 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Remove support for ansible 2.5 https://review.opendev.org/650431 | 16:58 |
| fungi | zuul just trusts gerrit to tell if if things are in merge conflict, right? | 16:58 |
| *** ociuhandu has joined #openstack-infra | 16:58 | |
| AJaeger | it was not in conflict this morning - and I see nothing merged on that branch | 16:58 |
| clarkb | I agree that the parent of that change has a parent which is the stable/stein HEAD | 16:59 |
| fungi | yeah, possible this is a mistake in gerrit's merge conflict reporting | 16:59 |
| clarkb | so that should be fast forwardable | 16:59 |
| clarkb | fungi: odd that zuul would say that too though | 16:59 |
| fungi | or something has corrupted its on-disk repository? | 16:59 |
| AJaeger | interesting, now https://review.opendev.org/#/c/645331/ looks fine - no message about conflicts... | 16:59 |
| AJaeger | wait, there it is - takes a few seconds to come ;( | 17:00 |
| fungi | load average on gerrit is 10 right now | 17:00 |
| AJaeger | I need to wait for "Working..." | 17:00 |
| *** udesale has quit IRC | 17:00 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 17:00 |
| clarkb | it does cleanly merge locally as expected | 17:00 |
| AJaeger | gerrit does not allow rebasing of that change - and neither of the parent... | 17:00 |
| AJaeger | so, another point that there's no conflict | 17:01 |
| fungi | a load average of 10 is busy but probably not unusual as there are 16 vcpus | 17:01 |
| fungi | and java is multithreaded across them | 17:01 |
| clarkb | fungi: ya the issue is more the GCing I think | 17:01 |
| clarkb | since that can do GIL type locking | 17:01 |
| clarkb | we can try killing this query I pasted above | 17:02 |
| fungi | nothing strange (or even recent) in dmesg, so i doubt it's disk issues | 17:02 |
| clarkb | not sure how safe that is via melody though | 17:02 |
| fungi | but yeah, possible the merge conflict is gerrit giving up waiting on whatever internal api call it makes to determine that | 17:02 |
| clarkb | that query has used ~9000 seconds of cpu time | 17:03 |
| clarkb | there are several others like it I see now after expanding that block of threds in melody | 17:04 |
| fungi | but that may be a symptom and not a cause | 17:04 |
| clarkb | possible though they predate the occurence of garabage collection | 17:05 |
| clarkb | and the next nearest request time is in the 100 second range | 17:05 |
| clarkb | so they are major outliers | 17:05 |
| yoctozepto | ah, I see you already notice gerrit got slow in the half an hour or so | 17:05 |
| clarkb | I also see requests with O=a | 17:06 |
| clarkb | corvus: ^ do you know what O query parameter in gerrit api means? (maybe you have encountered it with gertty?) | 17:06 |
| *** dtantsur is now known as dtantsur|afk | 17:07 | |
| *** ralonsoh has quit IRC | 17:08 | |
| clarkb | Based on the cpu and gc graphs in melody I think this may be slowly getting better. I think our options are to 1) wait it out and see if it is actually getting better, 2) kill the ~5 long lived http requests via melody and see if that fixes it, or 3) restart gerrit | 17:12 |
| rm_work | at 9000s i doubt anyone is still actively waiting for the results? >_> | 17:12 |
| clarkb | rm_work: ya my concern isn't for the client but for how gerrit may handle threads being killed out from under it in its http server | 17:12 |
| clarkb | 2) might result in 3) | 17:12 |
| rm_work | that's past "go for lunch and see if it is done when you get back" time even | 17:12 |
| rm_work | ah | 17:13 |
| rm_work | hmm | 17:13 |
| *** weifan has joined #openstack-infra | 17:14 | |
| clarkb | digging around in theg errit source I'm not having any luck finding where it handles the O parameter either | 17:19 |
| *** weifan has quit IRC | 17:19 | |
| clarkb | used memory is starting to fall off now | 17:19 |
| clarkb | the thread I noticed before is gone | 17:20 |
| clarkb | but some of its firneds are still there | 17:20 |
| *** betherly has joined #openstack-infra | 17:21 | |
| clarkb | the O might be noise. If I query is:watched is:merged for myself I get an O=81 parameter according to firefox dev tools | 17:22 |
| pabelanger | clarkb: zuul tenant is running ansible 2.8 now, however it seems we are using ansible 2.8.0, not ansible 2.8.3. How do we manage ansible in executors now? | 17:22 |
| clarkb | (also that query is fast for me) | 17:22 |
| *** weifan has joined #openstack-infra | 17:22 | |
| *** weifan has quit IRC | 17:22 | |
| *** ociuhandu has quit IRC | 17:22 | |
| clarkb | pabelanger: I believe when we update zuul we are supposed to also update the venvs? I think tobiash wrote that tooling | 17:22 |
| *** weifan has joined #openstack-infra | 17:23 | |
| *** ociuhandu has joined #openstack-infra | 17:23 | |
| *** weifan has quit IRC | 17:23 | |
| *** weifan has joined #openstack-infra | 17:23 | |
| clarkb | and now active threads are falling way off | 17:23 |
| clarkb | pretty sure it is recovering on its own | 17:23 |
| clarkb | I need to go do a bike ride now if I want to do that before the heat of the day and our meeting | 17:23 |
| pabelanger | clarkb: k, will check | 17:23 |
| *** betherly has quit IRC | 17:25 | |
| *** weifan has quit IRC | 17:26 | |
| *** tdasilva has joined #openstack-infra | 17:26 | |
| *** rpioso is now known as rpioso|afk | 17:27 | |
| *** ociuhandu has quit IRC | 17:27 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Don't install centos repos on RHEL https://review.opendev.org/674572 | 17:28 |
| *** ociuhandu has joined #openstack-infra | 17:28 | |
| rm_work | cool well thanks for looking into it :) sorry to waste your time | 17:28 |
| rm_work | glad it's recovering | 17:28 |
| clarkb | rm_work: no problem, It is good for us to make sure it isnt in a spiral of doom | 17:30 |
| *** chandankumar is now known as raukadah | 17:30 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:31 | |
| AJaeger | so, any ideas what to do with the merge-conflict change? Just wait and hope it self-heals? | 17:34 |
| *** jonher has quit IRC | 17:39 | |
| *** jonher has joined #openstack-infra | 17:40 | |
| *** lpetrut has quit IRC | 17:44 | |
| AJaeger | I rebased twice - now all fine ... | 17:44 |
| *** diablo_rojo has quit IRC | 17:45 | |
| *** ociuhandu has quit IRC | 17:48 | |
| *** betherly has joined #openstack-infra | 17:51 | |
| *** spsurya has quit IRC | 17:52 | |
| *** diablo_rojo has joined #openstack-infra | 17:55 | |
| *** betherly has quit IRC | 17:56 | |
| mordred | corvus, clarkb: gitea 1.9.0 has been released. should we take a stab at updating? | 17:57 |
| mordred | corvus: I think we're still carrying a patch on your branch that's slated to go into 1.10 iirc | 17:57 |
| *** bhavikdbavishi has joined #openstack-infra | 17:58 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Remove support for ansible 2.5 https://review.opendev.org/650431 | 18:01 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 18:01 |
| *** ociuhandu has joined #openstack-infra | 18:02 | |
| *** lpetrut has joined #openstack-infra | 18:03 | |
| *** ociuhandu has quit IRC | 18:04 | |
| *** ociuhandu has joined #openstack-infra | 18:05 | |
| *** igordc has joined #openstack-infra | 18:08 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to 1.9.0 https://review.opendev.org/674899 | 18:08 |
| mordred | corvus, clarkb: actually - looking at the files, we've been running from the ref of the docs patch I pushed up - which is included in 1.9.0 - so I think we're pretty solid on first glance at just being able to update | 18:09 |
| *** ociuhandu has quit IRC | 18:09 | |
| corvus | mordred: sounds good | 18:09 |
| mordred | corvus: also - hopefully our testing would show us if we're wrong or not :) | 18:09 |
| *** armax has joined #openstack-infra | 18:15 | |
| *** bhavikdbavishi has quit IRC | 18:15 | |
| *** bhavikdbavishi has joined #openstack-infra | 18:15 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 18:15 |
| *** andreykurilin_ has quit IRC | 18:17 | |
| *** e0ne has joined #openstack-infra | 18:17 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: render console in js with listview https://review.opendev.org/674663 | 18:19 |
| corvus | this is fun -- http://zuul.opendev.org/t/zuul/build/e4ea8eae961a449fba40f8413ee0d19e/log/job-output.txt#703 | 18:21 |
| corvus | "Something failed during the generation of the HTML report. Please verify the execution of the role for details." | 18:22 |
| corvus | but of course, the ara report generation failed, so we can't use it to examine the problem | 18:22 |
| corvus | and the console json omits that because it's in the last play (which is incomplete) | 18:22 |
| corvus | so we have no other debugging information than "Something failed" | 18:23 |
| *** igordc has quit IRC | 18:25 | |
| mordred | corvus: oh that's joyous | 18:28 |
| *** rpioso|afk is now known as rpioso | 18:30 | |
| corvus | if anyone has time to figure that out, i'd appreciate it; i have to afk for a bit | 18:30 |
| *** ykarel|away has quit IRC | 18:31 | |
| *** e0ne has quit IRC | 18:31 | |
| *** smarcet has quit IRC | 18:32 | |
| *** tdasilva has quit IRC | 18:33 | |
| *** tdasilva has joined #openstack-infra | 18:33 | |
| clarkb | I think the executor logs will have the ansible logs for ara task | 18:35 |
| mordred | corvus, clarkb: I looked at the executor logs on ze03 | 18:37 |
| mordred | I see the error message - but there is no additional information | 18:37 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 18:42 |
| rm_work | i assume when i make a job experimental, i don't need to add "voting: false" because that's assumed? | 18:44 |
| clarkb | rm_work: correct the experimental pipeline does not do voting, only reporting of results | 18:45 |
| rm_work | kk | 18:45 |
| fungi | the only pipelines where voting is relevant for us are check and gate | 18:45 |
| *** igordc has joined #openstack-infra | 18:46 | |
| *** jbadiapa has joined #openstack-infra | 18:47 | |
| clarkb | according to melody memory and cpu and active threads did fall out significantly. Then memory and cpu have started to creep back up again. Active threads is still relatively low and gc is less frequent | 18:50 |
| clarkb | there are still 3 very long lived requests of the form I pasted above earlier | 18:50 |
| *** tesseract has quit IRC | 18:52 | |
| *** bhavikdbavishi has quit IRC | 18:55 | |
| *** eernst has quit IRC | 18:55 | |
| clarkb | We will have our team meeting in ~5 minutes over in #openstack-meeting. | 18:55 |
| *** eernst has joined #openstack-infra | 18:56 | |
| *** jamesmcarthur has joined #openstack-infra | 18:57 | |
| *** eernst has quit IRC | 19:00 | |
| pabelanger | ianw: have you look at fedora-30 DIB yet? | 19:00 |
| *** jbadiapa has quit IRC | 19:03 | |
| *** e0ne has joined #openstack-infra | 19:04 | |
| *** jtomasek has quit IRC | 19:05 | |
| *** mriedem has quit IRC | 19:08 | |
| *** jamesmcarthur has quit IRC | 19:08 | |
| *** eernst has joined #openstack-infra | 19:08 | |
| *** mriedem has joined #openstack-infra | 19:09 | |
| *** eernst has quit IRC | 19:13 | |
| *** e0ne has quit IRC | 19:13 | |
| *** mattmceuen has left #openstack-infra | 19:13 | |
| *** e0ne has joined #openstack-infra | 19:14 | |
| *** e0ne has quit IRC | 19:15 | |
| *** jbadiapa has joined #openstack-infra | 19:16 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Switch swift logs test job to OVH https://review.opendev.org/674918 | 19:23 |
| *** eernst has joined #openstack-infra | 19:29 | |
| *** kopecmartin is now known as kopecmartin|off | 19:29 | |
| *** betherly has joined #openstack-infra | 19:30 | |
| *** jbadiapa has quit IRC | 19:31 | |
| *** eernst has quit IRC | 19:33 | |
| *** betherly has quit IRC | 19:35 | |
| openstackgerrit | Andreas Jaeger proposed opendev/puppet-openstackci master: Handle robots.txt on logs.o.o https://review.opendev.org/674920 | 19:35 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 19:36 |
| AJaeger | ianw, fungi, https://review.opendev.org/674920 should handle the not yet deleted robots.txt on logs.o.o. | 19:37 |
| *** Goneri has joined #openstack-infra | 19:37 | |
| *** smarcet has joined #openstack-infra | 19:38 | |
| *** lpetrut has quit IRC | 19:39 | |
| *** eernst has joined #openstack-infra | 19:41 | |
| *** jamesmcarthur has joined #openstack-infra | 19:44 | |
| *** eernst has quit IRC | 19:46 | |
| corvus | clarkb: https://review.opendev.org/674918 next thing for swift logs | 19:49 |
| clarkb | corvus: done | 19:49 |
| *** betherly has joined #openstack-infra | 19:50 | |
| *** e0ne has joined #openstack-infra | 19:51 | |
| *** jamesmcarthur has quit IRC | 19:54 | |
| *** Goneri has quit IRC | 19:55 | |
| *** betherly has quit IRC | 19:55 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 20:00 |
| *** dirk has quit IRC | 20:02 | |
| *** dirk has joined #openstack-infra | 20:05 | |
| openstackgerrit | Merged opendev/base-jobs master: Switch swift logs test job to OVH https://review.opendev.org/674918 | 20:07 |
| *** lpetrut has joined #openstack-infra | 20:08 | |
| *** slaweq has quit IRC | 20:09 | |
| *** diablo_rojo has quit IRC | 20:12 | |
| *** tdasilva has quit IRC | 20:14 | |
| *** jamesmcarthur has joined #openstack-infra | 20:14 | |
| *** jamesmcarthur has quit IRC | 20:15 | |
| *** tdasilva has joined #openstack-infra | 20:15 | |
| *** jamesmcarthur has joined #openstack-infra | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** smarcet has quit IRC | 20:21 | |
| *** eharney has quit IRC | 20:22 | |
| *** factor has joined #openstack-infra | 20:22 | |
| *** jamesmcarthur has joined #openstack-infra | 20:25 | |
| *** slaweq has joined #openstack-infra | 20:25 | |
| *** kjackal has joined #openstack-infra | 20:28 | |
| *** slaweq has quit IRC | 20:30 | |
| *** yamamoto has joined #openstack-infra | 20:36 | |
| *** eharney has joined #openstack-infra | 20:37 | |
| *** yamamoto has quit IRC | 20:41 | |
| *** smarcet has joined #openstack-infra | 20:41 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Collect gitea sshd logs https://review.opendev.org/674930 | 20:43 |
| clarkb | I think ^ may be all that is required for sshd logs | 20:43 |
| clarkb | should be self testing too | 20:43 |
| *** mriedem is now known as mriedem_afk | 20:46 | |
| *** e0ne has quit IRC | 20:48 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Render console in js https://review.opendev.org/674368 | 20:49 |
| *** priteau has joined #openstack-infra | 20:51 | |
| *** tdasilva_ has joined #openstack-infra | 20:53 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Experiment: Don't auto-expand past play https://review.opendev.org/674931 | 20:55 |
| *** tdasilva has quit IRC | 20:56 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 20:56 |
| *** whoami-rajat has quit IRC | 20:56 | |
| *** armax has quit IRC | 21:00 | |
| *** strigazi has joined #openstack-infra | 21:01 | |
| *** armax has joined #openstack-infra | 21:01 | |
| clarkb | gerrit memory use has more than halved and there are no more of those super old long running http request threads | 21:08 |
| *** lpetrut has quit IRC | 21:08 | |
| *** igordc has quit IRC | 21:12 | |
| *** mriedem_afk is now known as mriedem | 21:20 | |
| *** betherly has joined #openstack-infra | 21:22 | |
| *** betherly has quit IRC | 21:26 | |
| *** igordc has joined #openstack-infra | 21:31 | |
| paladox | I think you had the replication problems when the config auto-reloaded, right? | 21:37 |
| clarkb | infra-root https://review.opendev.org/#/c/674930/ passed the simple test I gave it if you want to review a fix for gitea sshd logs | 21:38 |
| *** markvoelker has quit IRC | 21:38 | |
| clarkb | paladox: no we haev problems with replication when restarting gerrit (and would prefer to make that an option when restarting gerrit) | 21:38 |
| clarkb | since if gerrit is off typically you arne't updated git repos | 21:38 |
| clarkb | (therefore replication is not necesary) | 21:38 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for test_novnc invalid token bug 1669468 https://review.opendev.org/674936 | 21:39 |
| openstack | bug 1669468 in OpenStack Compute (nova) "tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc fails intermittently in neutron multinode nv job" [Medium,Confirmed] https://launchpad.net/bugs/1669468 | 21:39 |
| *** jamesmcarthur has quit IRC | 21:39 | |
| paladox | clarkb oh, we just experenced a bug in replication :( (it broke replicating to github so we had to restart gerrit) | 21:40 |
| clarkb | oh I remember what you are thinking of | 21:41 |
| clarkb | paladox: the issue was gerrit doesn't understand non rsa host keys | 21:41 |
| clarkb | so either ed21559 or ecdsa hostkeys broke replication. Once I acceped the rsa hostkey it was fine | 21:41 |
| paladox | That's supported now clarkb | 21:41 |
| paladox | also we use rsa keys :) | 21:41 |
| * paladox added the ecdsa/ed21559 support to gerrit | 21:43 | |
| paladox | https://github.com/GerritCodeReview/gerrit/commit/f2d4fac3f35febc56fb6b64b482bfb35a4de9451 | 21:43 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use description for playbook name https://review.opendev.org/674938 | 21:47 |
| *** ekultails has quit IRC | 21:52 | |
| *** pkopec has quit IRC | 22:04 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for test_novnc invalid token bug 1669468 https://review.opendev.org/674936 | 22:04 |
| openstack | bug 1669468 in OpenStack Compute (nova) "tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc fails intermittently in neutron multinode nv job" [Medium,Confirmed] https://launchpad.net/bugs/1669468 | 22:04 |
| *** dchen has quit IRC | 22:09 | |
| *** betherly has joined #openstack-infra | 22:13 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Stack listview in build console https://review.opendev.org/674942 | 22:14 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add vexxhost backup server https://review.opendev.org/674549 | 22:15 |
| *** betherly has quit IRC | 22:17 | |
| *** smarcet has left #openstack-infra | 22:19 | |
| *** jamesmcarthur has joined #openstack-infra | 22:20 | |
| *** efried has left #openstack-infra | 22:21 | |
| *** priteau has quit IRC | 22:21 | |
| clarkb | anyone know if the boot.iso for fedora atomic is necessary? http://mirror.dfw.rax.openstack.org/fedora/atomic/stable/Fedora-Atomic-29-20181025.1/AtomicHost/x86_64/os/images/ codesearch says sushy might use a boot.iso, not sure if it is that image though | 22:24 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Combine all plays on build page https://review.opendev.org/674945 | 22:24 |
| clarkb | there is also http://mirror.dfw.rax.openstack.org/fedora/atomic/stable/Fedora-Atomic-29-20181025.1/AtomicHost/x86_64/iso/ which an exclude on *.iso files would clean up for us | 22:24 |
| clarkb | should free up a bit more space on the fedora mirror | 22:24 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Enable all swift log targets for testing https://review.opendev.org/674947 | 22:30 |
| corvus | infra-root: ^ i think we've seen cors work on all 6 swift targets | 22:30 |
| *** betherly has joined #openstack-infra | 22:34 | |
| *** betherly has quit IRC | 22:38 | |
| clarkb | vos release for fedora did finally finish. Took almost 6 hours | 22:39 |
| *** armax has quit IRC | 22:42 | |
| openstackgerrit | Merged opendev/base-jobs master: Enable all swift log targets for testing https://review.opendev.org/674947 | 22:43 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Flatten host tasks into tasks on build page https://review.opendev.org/674951 | 22:49 |
| *** tkajinam has joined #openstack-infra | 22:51 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove 'Details' button from build console https://review.opendev.org/674953 | 22:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Move hostname to additionalInfo on build page https://review.opendev.org/674954 | 22:56 |
| *** igordc has quit IRC | 22:56 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't auto-expand past play on build page https://review.opendev.org/674931 | 22:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use description for playbook name https://review.opendev.org/674938 | 22:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Stack listview in build console https://review.opendev.org/674942 | 22:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Combine all plays on build page https://review.opendev.org/674945 | 22:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Flatten host tasks into tasks on build page https://review.opendev.org/674951 | 22:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove 'Details' button from build console https://review.opendev.org/674953 | 22:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Move hostname to additionalInfo on build page https://review.opendev.org/674954 | 22:59 |
| *** rlandy has quit IRC | 22:59 | |
| *** yamamoto has joined #openstack-infra | 23:00 | |
| corvus | infra-root: if anyone has time to figure out what's wrong with ara static html generation as seen in jobs that have reported on https://review.opendev.org/674359 that would be great; that's probably the only blocker for logs-in-swift now | 23:03 |
| corvus | actually, i think anyone can debug that, i don't think that's a root issue | 23:03 |
| *** betherly has joined #openstack-infra | 23:05 | |
| *** aaronsheffield has quit IRC | 23:05 | |
| clarkb | corvus: do we not get links to the job-output.txt file when that happens? | 23:08 |
| clarkb | I'm not seeing anything at http://zuul.opendev.org/t/zuul/build/855da8d8efc140299fac005da4e09f88 or http://zuul.opendev.org/t/zuul/build/4f82f3f585cc4fe1aec41f3df41b3271 | 23:08 |
| corvus | clarkb: that's an unrelated post failure from yesterday | 23:09 |
| corvus | it's the latest, successful builds that are the problem | 23:09 |
| clarkb | oh the successful ones huh | 23:09 |
| *** betherly has quit IRC | 23:09 | |
| corvus | yeah, ara failures don't cause post-fails | 23:10 |
| clarkb | thinking out loud here, the logs.o.o ara reports are based on the wsgi app + database so do not do html generation statically | 23:12 |
| clarkb | is it possible that the ara version on the executors got updated and cannot generate the html static site anymore? | 23:12 |
| corvus | that is plausible (but i think we pinned to <1.0 because 1.0 can't do static generation (yet)) | 23:13 |
| *** markvoelker has joined #openstack-infra | 23:13 | |
| clarkb | ya I thought we pinned too but maybe something snuck through the ansible 2.8 update or similar | 23:13 |
| clarkb | I'm working to check on this theory now | 23:13 |
| corvus | clarkb: i'm inclined to send an announcement tomorrow saying that we're switching to the build page on monday, followed shortly thereafter (time tbd) by storing logs in swift. i don't think the second one needs advance notice. | 23:15 |
| clarkb | corvus: ok | 23:17 |
| *** markvoelker has quit IRC | 23:18 | |
| clarkb | I'm a bit confused over where we install the venvs | 23:19 |
| clarkb | on ze01 there is no ansible_root set in the executor config, but /var/lib/zuul/ansible-bin does not exist? | 23:19 |
| *** igordc has joined #openstack-infra | 23:20 | |
| fungi | clarkb: /usr/lib/zuul/ansible/ | 23:22 |
| clarkb | fungi: that doesn't seem to be venvs though nor complete ansible installs. That is just the zuul ansible bits I think | 23:22 |
| clarkb | (but also we don't set the config to install there so how does it end up there if that is indeed the location?) | 23:23 |
| fungi | it's far too easy to read /usr/lib/zuul as /var/lib/zuul when you have it in your mind that zuul is modifying content there | 23:23 |
| fungi | for example /usr/lib/zuul/ansible/2.7/bin/ looks like the executables for a venv to me | 23:23 |
| clarkb | oh indeed however state_dir defaults to /var/lib/zuul and ansible install dir appends ansible-bin to that | 23:24 |
| clarkb | any idea how we end up using ^ instead? | 23:24 |
| fungi | for sure ansible isn't providing /usr/lib/zuul/ansible/2.7/bin/python3 itself | 23:24 |
| clarkb | or maybe ansible_install_root is not what I think it is | 23:24 |
| fungi | i think that's where it stages ansible source, but i never dug into the ansible-manage bits long enough to confirm | 23:25 |
| *** betherly has joined #openstack-infra | 23:25 | |
| clarkb | the docs don't seem to document it iether (just mention running the manage command) | 23:26 |
| *** rcernin has joined #openstack-infra | 23:26 | |
| clarkb | ok https://opendev.org/zuul/zuul/src/branch/master/zuul/lib/ansible.py#L44-L49 sorts that out | 23:28 |
| clarkb | however https://opendev.org/zuul/zuul/src/branch/master/zuul/lib/ansible.py#L173-L175 should pass through the install_path from https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L2400-L2408 so now I am confused again | 23:30 |
| *** betherly has quit IRC | 23:30 | |
| clarkb | in any case the version of ara there appears to be 0.16.3 | 23:30 |
| clarkb | the global install is 0.16.5 | 23:33 |
| *** armax has joined #openstack-infra | 23:33 | |
| *** dchen has joined #openstack-infra | 23:34 | |
| clarkb | I wonder if ara 0.16 stopped working with ansible 2.8? | 23:36 |
| clarkb | looks like it is only tested with 2.7 and older | 23:36 |
| clarkb | dmsimard: ^ you may know off the top of your head | 23:36 |
| ianw | i feel like i remember us having to upgrade, sometime around when i was looking at 1.0 release | 23:37 |
| ianw | I7ed75d253857f86b68f67023af6897af4e1b4f50 i think | 23:38 |
| ianw | https://review.opendev.org/#/c/662662/ | 23:38 |
| ianw | so yeah, 0.16.4 for ansible 2.8 afaik | 23:38 |
| corvus | clarkb, fungi, ianw, jhesketh: how's this look? https://etherpad.openstack.org/p/h3rBMQyfwJ | 23:39 |
| corvus | (maybe we can actually get that out today rather than tomorrow) | 23:39 |
| clarkb | corvus: email looks good | 23:40 |
| *** mriedem has quit IRC | 23:40 | |
| clarkb | I think I see the bug in ansible install management | 23:41 |
| *** mattw4 has quit IRC | 23:42 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
| clarkb | if you use the command to install ansible venvs you get /usr/lib but if you use manage_ansible = true config option you get /var/lib/zuul/ansible-bin | 23:42 |
| ianw | corvus: LGTM | 23:42 |
| clarkb | I feel like we should maybe unite those so that users can switch back and forth | 23:42 |
| *** mattw4 has joined #openstack-infra | 23:42 | |
| ianw | personally i found it hard to find the "logs" tab at first, i think it would be great as just a panel on the main page. but that's just, like, my opinion | 23:43 |
| corvus | maybe we need a line break after the tabs | 23:43 |
| clarkb | corvus: I think our next step is to update ara to 0.16.5 in all the venvs given what ianw found? | 23:44 |
| ianw | corvus: yes, delineation would probably help too and be minimally invasive | 23:44 |
| corvus | ianw: i originally had it all in one page, but it sorta gets lost, especially if there are failed tasks | 23:45 |
| corvus | (eg http://zuul.opendev.org/t/openstack/build/75d1e8d4ffaf477db00520d7bfd77246) | 23:45 |
| clarkb | except now I'm not sure how ara gets into those venvs in the first place | 23:45 |
| corvus | clarkb: i think it might either be built in to the command or we put it in the env variable that says install extra pkgs | 23:46 |
| clarkb | https://opendev.org/opendev/puppet-zuul/src/branch/master/manifests/executor.pp#L45 is our extra packages var (just gear) I'll check the other possibility | 23:46 |
| *** jamesmcarthur has joined #openstack-infra | 23:47 | |
| clarkb | found it change incoming | 23:48 |
| *** kjackal has quit IRC | 23:48 | |
| jhesketh | corvus: lgtm :-) | 23:49 |
| jhesketh | also great work to everybody who has been getting this through ! | 23:49 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update ara to >=0.16.5 to support ansible 2.8 https://review.opendev.org/674958 | 23:50 |
| corvus | it has taken the whole village :) | 23:50 |
| ianw | corvus: yeah, i mean if you're taking suggestions, i think presenting the summary first is ok, but the "artifacts" as a separate thing below makes you think "oh my logs are artifacts" and then you click and don't see the logs. so maybe that should be in a tab too. and then the task status bits could probably use a header | 23:50 |
| clarkb | corvus: ianw ^ I think when that change merges to zuul our puppetry will rerun the ansible manage command which should update our ara version in the venvs | 23:50 |
| ianw | i also understand shuffling UX is never ending :) | 23:51 |
| clarkb | I've also scribbled a note to myself to confirm (and fix if confirmed) my suspicious about different behavior around zuul's manage ansible command and the zuul config setting to manage ansible | 23:52 |
| clarkb | but I won't get a chance to look at that until thursday | 23:52 |
| corvus | ianw: yeah, also we're working on adding another tab, so that could change things too | 23:52 |
| ianw | corvus: i'm sure once it becomes live, the one thing you will not be short of is opinions :) | 23:54 |
| *** jamesmcarthur_ has joined #openstack-infra | 23:54 | |
| *** jamesmcarthur has quit IRC | 23:54 | |
| corvus | maybe we should schedule this for the day i fly to sweden | 23:55 |
| clarkb | ha | 23:55 |
| *** betherly has joined #openstack-infra | 23:56 | |
| corvus | ianw: maybe we can put it all in one page, with some nice navigation links at the top to jump down to the sections. | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!