| ianw | it looks like the rsync is working from logs, but i think the job failed the same way in several different changes ... i'll aduit it and switch to nv if so till someone can poke | 00:00 |
|---|---|---|
| *** ijw has quit IRC | 00:01 | |
| ianw | auristor: if around ... i'd take suggestions on why our fedora mirror volume has suddently started taking 10+ hours to release every single time, even with minimal changes. it seems like maybe something has got stuck and it does a complete release every time? | 00:01 |
| ianw | i'm not sure we've seen that particular behaviour before? | 00:02 |
| clarkb | ok I tried to get some reviews in this afternoon. I'm out tomorrow have a good weekend everyone | 00:06 |
| ianw | ttyl! | 00:07 |
| *** ramishra has quit IRC | 00:10 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** jamesmcarthur has joined #openstack-infra | 00:12 | |
| *** stewie925 has quit IRC | 00:14 | |
| *** slaweq has quit IRC | 00:15 | |
| *** markvoelker has joined #openstack-infra | 00:15 | |
| *** markvoelker has quit IRC | 00:20 | |
| *** dpawlik has joined #openstack-infra | 00:20 | |
| *** jamesmcarthur has quit IRC | 00:22 | |
| *** dpawlik has quit IRC | 00:24 | |
| *** jamesmcarthur has joined #openstack-infra | 00:25 | |
| *** jamesmcarthur has quit IRC | 00:37 | |
| *** jamesmcarthur has joined #openstack-infra | 00:37 | |
| *** hongbin has quit IRC | 00:44 | |
| fungi | ianw: clarkb: yeah, i commented out the obs mirroring in the script temporarily to get the rest of the mirror into a sane state | 00:45 |
| *** mriedem has quit IRC | 00:45 | |
| fungi | at dirk's request | 00:45 |
| *** jamesmcarthur has quit IRC | 00:46 | |
| *** zhurong has quit IRC | 00:48 | |
| *** jamesmcarthur has joined #openstack-infra | 00:50 | |
| *** jamesmcarthur has quit IRC | 00:55 | |
| fungi | #status log logstash-worker16 rebooted by the provider at 00:19 UTC due to a host problem (provider ticket 190829-ord-0001222) | 00:58 |
| openstackstatus | fungi: finished logging | 00:58 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Make opensuse-150 non-voting for a while https://review.opendev.org/679367 | 00:59 |
| ianw | do we really need to test 15.0 & 15.1 ... ? i get quite lost with the suse versions | 00:59 |
| fungi | no clue | 01:01 |
| clarkb | no I think plan is to only test latest 15 | 01:02 |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| *** zhurong has joined #openstack-infra | 01:22 | |
| *** jamesmcarthur has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:27 | |
| *** jamesmcarthur has quit IRC | 01:27 | |
| *** jamesmcarthur has joined #openstack-infra | 01:32 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:00 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:03 | |
| *** bhavikdbavishi has quit IRC | 02:04 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:04 | |
| *** markvoelker has joined #openstack-infra | 02:05 | |
| *** apetrich has quit IRC | 02:10 | |
| *** markvoelker has quit IRC | 02:10 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** bhavikdbavishi has quit IRC | 02:15 | |
| *** ricolin has joined #openstack-infra | 02:16 | |
| *** slaweq has quit IRC | 02:16 | |
| *** dpawlik has joined #openstack-infra | 02:21 | |
| *** dpawlik has quit IRC | 02:25 | |
| *** jamesmcarthur has quit IRC | 02:36 | |
| *** jamesmcarthur has joined #openstack-infra | 02:37 | |
| *** jamesmcarthur has quit IRC | 02:45 | |
| *** nicolasbock has quit IRC | 02:52 | |
| auristor | ianw: rsync is updating the metadata of files even when they don't change | 02:55 |
| ianw | auristor: but this never used to happen ... | 03:00 |
| *** hongbin has joined #openstack-infra | 03:02 | |
| auristor | did your version of rsync change? | 03:03 |
| auristor | the relevant behavior of the openafs client and fileserver hasn't changed in 27 years | 03:04 |
| ianw | heh yeah i'm certainly not saying it's an openafs bug ... but i'm finding it hard to get insight into what it thinks has changed | 03:06 |
| auristor | You can examine the FileAuditLog files. If there are many AFS_SRX_StStat entries during the rsync then that is the file metadata being set | 03:06 |
| ianw | i will look; i can't see it touching all 400+ gb of data in the r | 03:08 |
| ianw | r/w volume | 03:08 |
| ianw | i mean, 8-10 hours of copying means it's copying *a lot* | 03:09 |
| auristor | an incremental volume dump includes all metadata and data for any file or directory that has been modified regardless of the size of the change | 03:10 |
| auristor | its not like rsync in which only the changed data ranges are transferred | 03:11 |
| ianw | right, but when you look at the run @ 2019-08-28 08:41:01 --> http://files.openstack.org/mirror/logs/rsync-mirrors/fedora.log.1 | 03:12 |
| ianw | rsync runs with "-v" ... i don't think it even transferred everything | 03:12 |
| ianw | s/everything/anything/ | 03:12 |
| ianw | http://paste.openstack.org/show/767469/ is it extracted for future reference | 03:14 |
| *** bhavikdbavishi has joined #openstack-infra | 03:14 | |
| auristor | please check the FileAuditLog for the run. It might not be setting data but it is probably altering metadata | 03:14 |
| ianw | we don't have that turned on right now | 03:15 |
| auristor | what is the volume name? | 03:15 |
| ianw | mirror.fedora | 03:16 |
| ianw | i wonder if it's in the old logs | 03:16 |
| auristor | vos exa mirror.fedora -cell openstack.org -noauth -ext | 03:17 |
| auristor | look at the Raw Read/Write Stats | 03:17 |
| *** psachin has joined #openstack-infra | 03:17 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:17 | |
| *** bhavikdbavishi has quit IRC | 03:19 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:19 | |
| ianw | auristor: ok, so if i dump that before and after the rsync, and check "diff network" that will basically show the writes the update process did? | 03:21 |
| ianw | ok, we also need rsync with "-i" to show "mention all items that are changed in any way." | 03:25 |
| ianw | "-v" isn't quite enough | 03:25 |
| auristor | I believe those stats are daily | 03:25 |
| ianw | ok ... is there anything else that might be helpful to dump around the rsync updates? | 03:28 |
| auristor | I'm quite sure that is what the problem is because auristorfs has logic in both the client and the fileserver to filter out extraneous rsync metadata changes | 03:31 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: fedora-mirror: Use rsync itemize output https://review.opendev.org/679372 | 03:32 |
| *** diablo_rojo has quit IRC | 03:32 | |
| ianw | auristor: ^ well, TIL another thing about rsync flags ... | 03:33 |
| ianw | if it comes to it, i can run the updates under strace and save that somewhere and get a definitive answer on what's touched ... | 03:33 |
| ianw | and i can also re-enable file audit logging if still seeming weird | 03:35 |
| ianw | auristor: thanks ,as usual, for getting us moving in the right direction :) | 03:36 |
| auristor | any time | 03:40 |
| *** udesale has joined #openstack-infra | 03:52 | |
| *** dklyle has quit IRC | 04:00 | |
| *** david-lyle has joined #openstack-infra | 04:00 | |
| *** rfolco has quit IRC | 04:03 | |
| *** slaweq has joined #openstack-infra | 04:04 | |
| *** yamamoto has joined #openstack-infra | 04:04 | |
| *** slaweq has quit IRC | 04:08 | |
| *** dave-mccowan has quit IRC | 04:14 | |
| *** hongbin has quit IRC | 04:19 | |
| *** jhesketh has quit IRC | 04:20 | |
| *** dpawlik has joined #openstack-infra | 04:22 | |
| openstackgerrit | Merged openstack/diskimage-builder master: yum-minimal: install fedora-release-cloud https://review.opendev.org/676581 | 04:24 |
| *** dpawlik has quit IRC | 04:26 | |
| openstackgerrit | Merged opendev/system-config master: fedora-mirror: Use rsync itemize output https://review.opendev.org/679372 | 04:30 |
| openstackgerrit | Merged openstack/diskimage-builder master: rpm-distro: ensure we selinux relabel underlying directories https://review.opendev.org/676855 | 04:31 |
| *** larainema has joined #openstack-infra | 04:54 | |
| *** ccamacho has joined #openstack-infra | 05:00 | |
| *** markvoelker has joined #openstack-infra | 05:00 | |
| *** raukadah is now known as chandankumar | 05:02 | |
| *** ykarel|away has joined #openstack-infra | 05:02 | |
| *** ykarel|away is now known as ykarel | 05:03 | |
| *** tkajinam has quit IRC | 05:05 | |
| *** markvoelker has quit IRC | 05:06 | |
| *** rcosnita has joined #openstack-infra | 05:10 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** ykarel is now known as ykarel|afk | 05:12 | |
| *** slaweq has quit IRC | 05:15 | |
| *** hamzy has quit IRC | 05:16 | |
| *** e0ne has joined #openstack-infra | 05:19 | |
| *** ramishra has joined #openstack-infra | 05:20 | |
| *** stakeda has quit IRC | 05:20 | |
| *** e0ne has quit IRC | 05:20 | |
| *** ramishra has quit IRC | 05:25 | |
| *** jaosorior has joined #openstack-infra | 05:27 | |
| *** dpawlik has joined #openstack-infra | 05:30 | |
| *** tkajinam has joined #openstack-infra | 05:41 | |
| *** ramishra has joined #openstack-infra | 05:46 | |
| *** igordc has quit IRC | 05:46 | |
| *** yamamoto has quit IRC | 05:48 | |
| *** yamamoto has joined #openstack-infra | 05:48 | |
| *** e0ne has joined #openstack-infra | 05:51 | |
| *** yamamoto has quit IRC | 05:53 | |
| *** jtomasek has joined #openstack-infra | 05:54 | |
| *** jtomasek has quit IRC | 05:54 | |
| *** jtomasek has joined #openstack-infra | 05:55 | |
| *** bhavikdbavishi has quit IRC | 06:01 | |
| *** slaweq has joined #openstack-infra | 06:08 | |
| mordred | paladox: wow -that fix is fun. the story here with plugins with their own deps is really messy isn't it? | 06:12 |
| *** slaweq has quit IRC | 06:13 | |
| *** xek has joined #openstack-infra | 06:18 | |
| mordred | paladox: although - that patch gives me an idea on an improvement to the whole thing - I'll make a patch in just a bit | 06:18 |
| *** pkopec has joined #openstack-infra | 06:21 | |
| *** e0ne has quit IRC | 06:24 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fedora 30 functional and boot tests https://review.opendev.org/676582 | 06:27 |
| *** e0ne has joined #openstack-infra | 06:39 | |
| *** rcernin has quit IRC | 06:48 | |
| *** slaweq has joined #openstack-infra | 06:49 | |
| *** slaweq has quit IRC | 06:54 | |
| *** yamamoto has joined #openstack-infra | 06:57 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Allow extra repositories to be added to images https://review.opendev.org/676708 | 07:02 |
| *** slaweq has joined #openstack-infra | 07:05 | |
| *** trident has quit IRC | 07:10 | |
| *** trident has joined #openstack-infra | 07:18 | |
| *** trident has quit IRC | 07:22 | |
| *** tosky has joined #openstack-infra | 07:29 | |
| *** trident has joined #openstack-infra | 07:30 | |
| *** yamamoto has quit IRC | 07:30 | |
| *** lpetrut has joined #openstack-infra | 07:30 | |
| *** jpena|off is now known as jpena | 07:36 | |
| *** jhesketh has joined #openstack-infra | 07:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:39 | |
| *** gfidente has joined #openstack-infra | 07:41 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 07:42 | |
| *** bhavikdbavishi has quit IRC | 07:43 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 07:43 | |
| *** ykarel|afk is now known as ykarel | 07:44 | |
| *** dtantsur|afk is now known as dtantsur | 07:46 | |
| *** dchen has quit IRC | 08:02 | |
| *** apetrich has joined #openstack-infra | 08:03 | |
| *** xenos76 has joined #openstack-infra | 08:04 | |
| *** lucasagomes has joined #openstack-infra | 08:06 | |
| *** jaosorior has quit IRC | 08:07 | |
| *** dpawlik has quit IRC | 08:12 | |
| *** ramishra has quit IRC | 08:17 | |
| *** markvoelker has joined #openstack-infra | 08:21 | |
| *** ralonsoh has joined #openstack-infra | 08:26 | |
| *** markvoelker has quit IRC | 08:27 | |
| *** tkajinam has quit IRC | 08:27 | |
| *** ramishra has joined #openstack-infra | 08:27 | |
| *** derekh has joined #openstack-infra | 08:35 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add support for smart reconfigurations https://review.opendev.org/652114 | 08:37 |
| *** dpawlik has joined #openstack-infra | 08:39 | |
| *** e0ne has quit IRC | 08:40 | |
| *** e0ne has joined #openstack-infra | 08:40 | |
| *** rcosnita has quit IRC | 08:43 | |
| *** dpawlik has quit IRC | 08:43 | |
| *** e0ne has quit IRC | 08:46 | |
| *** ykarel is now known as ykarel|lunch | 08:50 | |
| *** dpawlik has joined #openstack-infra | 08:52 | |
| *** e0ne has joined #openstack-infra | 09:03 | |
| *** jaosorior has joined #openstack-infra | 09:10 | |
| *** hamzy has joined #openstack-infra | 09:10 | |
| *** ykarel_ has joined #openstack-infra | 09:16 | |
| *** ykarel|lunch has quit IRC | 09:18 | |
| *** xek has quit IRC | 09:20 | |
| *** apetrich has quit IRC | 09:29 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks and static dirs https://review.opendev.org/678875 | 09:37 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 09:37 |
| *** dpawlik has quit IRC | 09:46 | |
| *** ykarel_ is now known as ykarel | 09:51 | |
| paladox | mordred yeh | 09:53 |
| *** e0ne has quit IRC | 10:01 | |
| *** e0ne has joined #openstack-infra | 10:10 | |
| *** icarusfactor has quit IRC | 10:11 | |
| *** icarusfactor has joined #openstack-infra | 10:11 | |
| *** dpawlik has joined #openstack-infra | 10:12 | |
| *** apetrich has joined #openstack-infra | 10:12 | |
| *** icarusfactor has quit IRC | 10:12 | |
| *** icarusfactor has joined #openstack-infra | 10:13 | |
| *** factor__ has joined #openstack-infra | 10:17 | |
| *** icarusfactor has quit IRC | 10:20 | |
| *** trident has quit IRC | 10:23 | |
| *** udesale has quit IRC | 10:23 | |
| *** noonedeadpunk has quit IRC | 10:23 | |
| *** notmyname has quit IRC | 10:23 | |
| *** mtreinish has quit IRC | 10:23 | |
| *** cgoncalves has quit IRC | 10:23 | |
| *** bnemec has quit IRC | 10:23 | |
| *** owalsh|away has quit IRC | 10:23 | |
| *** Wasaac has quit IRC | 10:23 | |
| *** frickler has quit IRC | 10:23 | |
| *** clayg has quit IRC | 10:23 | |
| *** tobiash has quit IRC | 10:23 | |
| *** pleia2 has quit IRC | 10:23 | |
| *** tobias-urdin has quit IRC | 10:23 | |
| *** flaper87 has quit IRC | 10:23 | |
| *** harlowja has quit IRC | 10:24 | |
| *** efried has quit IRC | 10:24 | |
| *** pfallenop has quit IRC | 10:24 | |
| *** gagehugo has quit IRC | 10:24 | |
| *** SpamapS has quit IRC | 10:24 | |
| *** Diabelko has quit IRC | 10:24 | |
| *** radez has quit IRC | 10:24 | |
| *** yoctozepto has quit IRC | 10:24 | |
| *** eandersson has quit IRC | 10:24 | |
| *** evrardjp has quit IRC | 10:24 | |
| *** amorin has quit IRC | 10:24 | |
| *** lifeless has quit IRC | 10:24 | |
| *** otherwiseguy has quit IRC | 10:24 | |
| *** weshay_pto has quit IRC | 10:24 | |
| *** ociuhandu has joined #openstack-infra | 10:24 | |
| *** clayg has joined #openstack-infra | 10:24 | |
| *** trident has joined #openstack-infra | 10:24 | |
| *** frickler has joined #openstack-infra | 10:24 | |
| *** irclogbot_3 has quit IRC | 10:25 | |
| *** mtreinish has joined #openstack-infra | 10:26 | |
| *** Wasaac has joined #openstack-infra | 10:26 | |
| *** cgoncalves has joined #openstack-infra | 10:26 | |
| *** noonedeadpunk has joined #openstack-infra | 10:26 | |
| *** efried has joined #openstack-infra | 10:27 | |
| *** tobiash has joined #openstack-infra | 10:27 | |
| *** evrardjp has joined #openstack-infra | 10:27 | |
| *** yoctozepto has joined #openstack-infra | 10:27 | |
| *** owalsh has joined #openstack-infra | 10:27 | |
| *** notmyname has joined #openstack-infra | 10:27 | |
| *** irclogbot_1 has joined #openstack-infra | 10:28 | |
| *** bnemec has joined #openstack-infra | 10:28 | |
| *** gagehugo has joined #openstack-infra | 10:28 | |
| *** lifeless has joined #openstack-infra | 10:29 | |
| *** pfallenop has joined #openstack-infra | 10:30 | |
| *** amorin has joined #openstack-infra | 10:33 | |
| *** otherwiseguy has joined #openstack-infra | 10:33 | |
| *** Diabelko has joined #openstack-infra | 10:40 | |
| *** harlowja has joined #openstack-infra | 10:41 | |
| *** SpamapS has joined #openstack-infra | 10:41 | |
| *** ociuhandu has quit IRC | 10:42 | |
| *** jaosorior has quit IRC | 10:43 | |
| *** rcosnita has joined #openstack-infra | 10:56 | |
| *** pkopec has quit IRC | 10:57 | |
| *** bhavikdbavishi has quit IRC | 10:59 | |
| *** jaosorior has joined #openstack-infra | 10:59 | |
| *** ociuhandu has joined #openstack-infra | 11:02 | |
| *** pkopec has joined #openstack-infra | 11:04 | |
| *** ykarel is now known as ykarel|afk | 11:05 | |
| *** nicolasbock has joined #openstack-infra | 11:05 | |
| *** pkopec has joined #openstack-infra | 11:07 | |
| *** rcosnita has quit IRC | 11:13 | |
| *** ociuhandu has quit IRC | 11:14 | |
| *** pkopec has quit IRC | 11:17 | |
| paladox | mordred we did this change https://gerrit.wikimedia.org/r/#/c/operations/software/gerrit/+/533336/ | 11:18 |
| *** pkopec has joined #openstack-infra | 11:20 | |
| *** slaweq has quit IRC | 11:26 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** ricolin has quit IRC | 11:35 | |
| *** rh-jelabarre has joined #openstack-infra | 11:39 | |
| *** dpawlik has quit IRC | 11:39 | |
| *** tesseract has joined #openstack-infra | 11:42 | |
| *** ociuhandu has joined #openstack-infra | 11:50 | |
| *** markvoelker has joined #openstack-infra | 11:51 | |
| *** ociuhandu has quit IRC | 11:57 | |
| *** ykarel|afk is now known as ykarel | 11:57 | |
| *** dave-mccowan has joined #openstack-infra | 12:02 | |
| *** rlandy has joined #openstack-infra | 12:07 | |
| *** rlandy is now known as rlandy|ruck | 12:07 | |
| *** jamesmcarthur has joined #openstack-infra | 12:10 | |
| *** yolanda has quit IRC | 12:12 | |
| *** dpawlik has joined #openstack-infra | 12:12 | |
| *** dpawlik has quit IRC | 12:16 | |
| *** rfolco has joined #openstack-infra | 12:25 | |
| *** dpawlik has joined #openstack-infra | 12:25 | |
| *** jamesmcarthur has quit IRC | 12:27 | |
| *** ociuhandu has joined #openstack-infra | 12:27 | |
| *** derekh has quit IRC | 12:27 | |
| *** xek has joined #openstack-infra | 12:31 | |
| *** ociuhandu has quit IRC | 12:33 | |
| *** jeliu_ has joined #openstack-infra | 12:34 | |
| *** sthussey has joined #openstack-infra | 12:34 | |
| *** aaronsheffield has joined #openstack-infra | 12:39 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| *** yolanda has joined #openstack-infra | 12:41 | |
| *** larainema has quit IRC | 12:43 | |
| *** mriedem has joined #openstack-infra | 12:51 | |
| *** apetrich has quit IRC | 12:51 | |
| *** xek has quit IRC | 12:52 | |
| *** derekh has joined #openstack-infra | 12:59 | |
| *** apetrich has joined #openstack-infra | 13:00 | |
| mordred | paladox: ++ | 13:02 |
| *** jamesmcarthur has joined #openstack-infra | 13:05 | |
| *** e0ne has quit IRC | 13:05 | |
| *** aedc has joined #openstack-infra | 13:10 | |
| *** e0ne has joined #openstack-infra | 13:14 | |
| *** aedc has quit IRC | 13:16 | |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Do not delete cracklib from /usr/share https://review.opendev.org/676983 | 13:19 |
| openstackgerrit | Merged zuul/zuul master: Add release note for bug fix to test correct commit https://review.opendev.org/679285 | 13:21 |
| *** eharney has joined #openstack-infra | 13:31 | |
| *** edmondsw_ has joined #openstack-infra | 13:31 | |
| *** ykarel is now known as ykarel|afk | 13:33 | |
| mordred | infra-root: interesting REST endpoint for gerrit: https://review.opendev.org/a/config/server/caches/ - shows cache size and hit rate information | 13:33 |
| *** xek has joined #openstack-infra | 13:33 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: fix connections list github base_url https://review.opendev.org/678950 | 13:36 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Remove immutable attributes before cleanup https://review.opendev.org/668766 | 13:39 |
| *** edmondsw_ has quit IRC | 13:41 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add phoronix-test-suite job https://review.opendev.org/679082 | 13:45 |
| *** EmilienM is now known as EvilienM | 13:49 | |
| *** Goneri has joined #openstack-infra | 13:49 | |
| *** pkopec has quit IRC | 13:49 | |
| *** pkopec has joined #openstack-infra | 13:54 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: add merge_jobs options to disable gearman merge jobs https://review.opendev.org/676974 | 13:54 |
| *** ykarel|afk is now known as ykarel | 13:55 | |
| *** jamesmcarthur has quit IRC | 13:57 | |
| *** jamesmcarthur has joined #openstack-infra | 13:58 | |
| *** ociuhandu has joined #openstack-infra | 14:01 | |
| *** ociuhandu has quit IRC | 14:02 | |
| fungi | i need to disappear for an hour or so to go to an appointment, but will be back soon hopefully | 14:04 |
| *** aedc has joined #openstack-infra | 14:06 | |
| *** dpawlik has quit IRC | 14:11 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Write errors from ansible execution into json log https://review.opendev.org/676723 | 14:26 |
| *** jamesmcarthur has quit IRC | 14:27 | |
| *** jamesmcarthur has joined #openstack-infra | 14:33 | |
| *** ociuhandu has joined #openstack-infra | 14:34 | |
| *** e0ne has quit IRC | 14:35 | |
| *** jamesmcarthur has quit IRC | 14:35 | |
| *** jamesmcarthur has joined #openstack-infra | 14:36 | |
| *** ociuhandu has quit IRC | 14:39 | |
| *** e0ne has joined #openstack-infra | 14:39 | |
| *** ykarel is now known as ykarel|away | 14:52 | |
| openstackgerrit | Merged zuul/zuul master: Make json outfile seeking clearer https://review.opendev.org/678740 | 15:00 |
| *** ijw has joined #openstack-infra | 15:05 | |
| *** michael-beaver has joined #openstack-infra | 15:07 | |
| *** ociuhandu has joined #openstack-infra | 15:07 | |
| *** ijw has quit IRC | 15:11 | |
| *** ijw has joined #openstack-infra | 15:11 | |
| *** ijw has quit IRC | 15:13 | |
| *** ijw has joined #openstack-infra | 15:14 | |
| *** eandersson has joined #openstack-infra | 15:30 | |
| *** tobias-urdin has joined #openstack-infra | 15:30 | |
| *** jaosorior has quit IRC | 15:31 | |
| *** gyee has joined #openstack-infra | 15:33 | |
| *** shilpasd has joined #openstack-infra | 15:36 | |
| shilpasd | AJaeger_: Hi, help to understand zuul failure on https://review.opendev.org/#/c/669079/ in-spite of changes done at https://review.opendev.org/#/c/669079/6/blazar/tests/utils/openstack/test_placement.py | 15:37 |
| shilpasd | verified locally and tox suite is passing, only change i have cleared .tox/ locally | 15:37 |
| *** jaosorior has joined #openstack-infra | 15:37 | |
| shilpasd | is missing anything at tox.ini? https://github.com/openstack/blazar/blob/master/tox.ini | 15:38 |
| *** jaosorior has quit IRC | 15:42 | |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Allow configurable gzip binary name https://review.opendev.org/669225 | 15:46 |
| *** tesseract has quit IRC | 15:52 | |
| *** chandankumar is now known as raukadah | 15:58 | |
| *** lucasagomes has quit IRC | 15:59 | |
| *** shilpasd has quit IRC | 16:01 | |
| *** e0ne has quit IRC | 16:04 | |
| *** ginopc has quit IRC | 16:05 | |
| *** ccamacho has quit IRC | 16:05 | |
| *** jpena is now known as jpena|off | 16:07 | |
| *** e0ne has joined #openstack-infra | 16:09 | |
| *** eharney has quit IRC | 16:13 | |
| *** ijw_ has joined #openstack-infra | 16:15 | |
| *** ijw_ has quit IRC | 16:15 | |
| *** ijw_ has joined #openstack-infra | 16:16 | |
| *** mriedem is now known as mriedem_afk | 16:16 | |
| *** ijw_ has quit IRC | 16:17 | |
| *** ijw_ has joined #openstack-infra | 16:17 | |
| *** ijw has quit IRC | 16:18 | |
| *** xek has quit IRC | 16:21 | |
| *** markvoelker has quit IRC | 16:22 | |
| *** markvoelker has joined #openstack-infra | 16:28 | |
| *** e0ne has quit IRC | 16:30 | |
| *** dtantsur is now known as dtantsur|afk | 16:36 | |
| *** igordc has joined #openstack-infra | 16:39 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add phoronix-test-suite job https://review.opendev.org/679082 | 16:43 |
| *** jeliu_ has quit IRC | 16:50 | |
| *** bnemec is now known as beekneemech | 16:53 | |
| *** eharney has joined #openstack-infra | 16:58 | |
| *** derekh has quit IRC | 17:00 | |
| *** gfidente has quit IRC | 17:02 | |
| fungi | shilpsad seems to have left ~20 minutes after asking a question, which makes me question whether it's worth my time to try to identify the cause of their problem | 17:18 |
| *** ralonsoh has quit IRC | 17:19 | |
| *** ricolin has joined #openstack-infra | 17:28 | |
| *** rmcall has quit IRC | 17:36 | |
| *** rcosnita has joined #openstack-infra | 17:37 | |
| *** rcosnita has quit IRC | 17:38 | |
| *** pkopec has quit IRC | 17:40 | |
| *** ociuhandu has quit IRC | 17:51 | |
| *** ociuhandu has joined #openstack-infra | 17:57 | |
| *** radez has joined #openstack-infra | 17:59 | |
| *** ricolin has quit IRC | 18:00 | |
| *** ociuhandu has quit IRC | 18:02 | |
| *** jtomasek has quit IRC | 18:07 | |
| *** jtomasek has joined #openstack-infra | 18:09 | |
| *** ykarel|away has quit IRC | 18:11 | |
| *** psachin has quit IRC | 18:12 | |
| johnsom | I am getting a "network error" when I try to access a log file from a job: https://zuul.opendev.org/t/openstack/build/1bcb6753acee47cf938f142cc6c4a286/log/controller/logs/octavia-amphora_log.txt.gz is the zuul link to the log, where the zuul page is here: https://zuul.opendev.org/t/openstack/build/1bcb6753acee47cf938f142cc6c4a286/logs | 18:17 |
| *** iokiwi has quit IRC | 18:17 | |
| *** adriant has quit IRC | 18:17 | |
| *** adriant has joined #openstack-infra | 18:18 | |
| *** jamesmcarthur has quit IRC | 18:18 | |
| johnsom | It looks like the "network error" pop up is coming from zuul as opposed to my browser, etc. It's a little flag in the upper right corner. | 18:22 |
| *** ramishra has quit IRC | 18:24 | |
| fungi | johnsom: i'm not getting a network error. could you have a browser plugin which is blocking cross-domain requests for some files? | 18:32 |
| fungi | the data embedded is coming from files in a random swift object store at a provider's domain, and even though cors headers are being set that's not sufficient for some particularly thorough browser filters | 18:34 |
| johnsom | https://usercontent.irccloud-cdn.com/file/cY36gpSJ/image.png | 18:36 |
| *** jamesmcarthur has joined #openstack-infra | 18:36 | |
| johnsom | That is what I see. It seems to only be this log. If I open the link in the dialog it opens fine. | 18:36 |
| johnsom | It running a standard chrome on Fedora. I don't think any of these extension would mess with this. | 18:37 |
| johnsom | Let me try firefox. I can always reproduce this, so easy to test. | 18:37 |
| fungi | yeah, that warning is the one we added because browser extensions were blocking some file types for cross-domain requests | 18:38 |
| johnsom | Yeah, it works on firefox. Funny that this one file would not work on chrome | 18:38 |
| fungi | (before it was just silently empty) | 18:38 |
| fungi | could also be built-in anti-tracker functionality in some browsers. hard to say. they get really touchy about javascript requesting files from urls with different domains in them | 18:39 |
| *** jamesmcarthur has quit IRC | 18:39 | |
| johnsom | Yeah, for good reason really. | 18:39 |
| fungi | and its been hard to pin down which browser/extension/domain/filetype combinations are involved | 18:40 |
| fungi | and just setting cors headers saying they're allowed isn't sufficient in some cases, apparently | 18:40 |
| fungi | which is unfortunate, since this is precisely the functionality http-connected data storage was designed for | 18:41 |
| *** jamesmcarthur has joined #openstack-infra | 18:42 | |
| fungi | so you were experiencing this on chrome, i guess? hard to tell from the screenshot | 18:42 |
| fungi | also were you browsing zuul.openstack.org or zuul.opendev.org? | 18:42 |
| johnsom | Yeah, Version 76.0.3809.100 (Official Build) (64-bit) | 18:42 |
| fungi | thanks. this information may help in aggregate | 18:43 |
| johnsom | This exact URL: https://zuul.opendev.org/t/openstack/build/1bcb6753acee47cf938f142cc6c4a286/log/controller/logs/octavia-amphora_log.txt.gz | 18:43 |
| johnsom | Disabling all of the extensions (video conference plugins) makes no difference. | 18:43 |
| fungi | yeah, so that ought to be covered by cors | 18:43 |
| johnsom | Funny, it loads in "private mode" | 18:44 |
| fungi | we suspect it may be cross-domain requests for files with .gz extensions which are weirding out some browsers | 18:44 |
| johnsom | This one loads just fine: https://zuul.opendev.org/t/openstack/build/1bcb6753acee47cf938f142cc6c4a286/log/controller/logs/screen-o-cw.txt.gz | 18:45 |
| *** jamesmcarthur has quit IRC | 18:45 | |
| johnsom | From the same patch | 18:45 |
| johnsom | This one also fails: https://zuul.opendev.org/t/openstack/build/1bcb6753acee47cf938f142cc6c4a286/log/controller/logs/octavia-tenant-traffic_log.txt.gz | 18:46 |
| johnsom | Both of the failing files are "non-standard" format. They aren't oslo logging files. I wonder if that has something to do with it. | 18:46 |
| openstackgerrit | Ronelle Landy proposed zuul/zuul-jobs master: Only use RHEL8 deps repo on Red Hat systems newer than 7 https://review.opendev.org/679126 | 18:47 |
| johnsom | octavia-amphora_log.txt.gz:1 Access to XMLHttpRequest at 'https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/logs_58/679358/2/check/octavia-v2-dsvm-scenario/1bcb675/controller/logs/octavia-amphora_log.txt.gz' from origin 'https://zuul.opendev.org' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. | 18:53 |
| johnsom | From the chrome developer tools log | 18:54 |
| fungi | weird | 18:54 |
| fungi | we're supposedly setting cors headers for those on upload | 18:54 |
| johnsom | We use this: | 18:56 |
| johnsom | https://github.com/openstack/octavia-tempest-plugin/blob/master/zuul.d/jobs.yaml#L113 | 18:56 |
| *** dave-mccowan has quit IRC | 19:02 | |
| openstackgerrit | Ronelle Landy proposed zuul/zuul-jobs master: Only use RHEL8 deps repo on Red Hat systems newer than 7 https://review.opendev.org/679126 | 19:05 |
| *** jamesmcarthur has joined #openstack-infra | 19:10 | |
| openstackgerrit | Ronelle Landy proposed zuul/zuul-jobs master: Only use RHEL8 deps repo on Red Hat systems newer than 7 https://review.opendev.org/679126 | 19:11 |
| *** jeliu_ has joined #openstack-infra | 19:11 | |
| *** xek has joined #openstack-infra | 19:11 | |
| *** Swami has joined #openstack-infra | 19:18 | |
| *** Swami_ has joined #openstack-infra | 19:18 | |
| *** Swami_ has quit IRC | 19:19 | |
| *** Swami has quit IRC | 19:19 | |
| *** mriedem_afk is now known as mriedem | 19:24 | |
| *** jamesmcarthur has quit IRC | 19:28 | |
| *** jamesmcarthur has joined #openstack-infra | 19:29 | |
| *** slaweq has joined #openstack-infra | 19:32 | |
| *** jamesmcarthur has quit IRC | 19:33 | |
| *** jeliu_ has quit IRC | 19:42 | |
| *** Goneri has quit IRC | 19:45 | |
| *** jamesmcarthur has joined #openstack-infra | 19:52 | |
| *** xek has quit IRC | 19:57 | |
| *** pleia2 has joined #openstack-infra | 20:06 | |
| *** adriant has quit IRC | 20:17 | |
| *** adriant has joined #openstack-infra | 20:18 | |
| *** Goneri has joined #openstack-infra | 20:23 | |
| *** ociuhandu has joined #openstack-infra | 20:25 | |
| *** ociuhandu has quit IRC | 20:30 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Remvove openSUSE 15.0 testing and rename 15.1 to 15 https://review.opendev.org/679367 | 20:37 |
| *** aaronsheffield has quit IRC | 20:39 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Remove Ubuntu Trusty testing https://review.opendev.org/679511 | 20:39 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Rename openSUSE 15.1 testing to 15 https://review.opendev.org/679367 | 20:44 |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Enable testing against Python 3.6/3.7 (switch to stestr) https://review.opendev.org/667666 | 20:46 |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Update testing to opensuse 15 https://review.opendev.org/679512 | 20:46 |
| *** rmcall has joined #openstack-infra | 20:50 | |
| *** markvoelker has quit IRC | 20:51 | |
| *** rosmaita has left #openstack-infra | 20:54 | |
| *** michael-beaver has quit IRC | 21:07 | |
| *** eharney has quit IRC | 21:18 | |
| *** hwoarang has quit IRC | 21:25 | |
| *** hwoarang has joined #openstack-infra | 21:28 | |
| *** markvoelker has joined #openstack-infra | 21:30 | |
| *** xek has joined #openstack-infra | 21:34 | |
| *** markvoelker has quit IRC | 21:35 | |
| *** xek has quit IRC | 21:43 | |
| *** jamesmcarthur has quit IRC | 21:47 | |
| *** signed8b_ has joined #openstack-infra | 21:49 | |
| *** ijw_ has quit IRC | 21:50 | |
| *** signed8bit has quit IRC | 21:52 | |
| *** slaweq has quit IRC | 21:53 | |
| *** Goneri has quit IRC | 22:08 | |
| *** mriedem has quit IRC | 22:09 | |
| *** rlandy|ruck has quit IRC | 22:11 | |
| *** Goneri has joined #openstack-infra | 22:13 | |
| *** jtomasek has quit IRC | 22:19 | |
| *** vesper has quit IRC | 22:46 | |
| *** vesper11 has joined #openstack-infra | 22:47 | |
| *** rh-jelabarre has quit IRC | 22:53 | |
| *** dayou has quit IRC | 22:55 | |
| *** dayou has joined #openstack-infra | 22:56 | |
| *** tosky has quit IRC | 23:14 | |
| *** jhesketh has quit IRC | 23:18 | |
| *** xek has joined #openstack-infra | 23:19 | |
| *** aedc has quit IRC | 23:33 | |
| *** xek has quit IRC | 23:34 | |
| *** gyee has quit IRC | 23:37 | |
| *** Goneri has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
