Monday, 2019-06-03

*** slaweq has joined #openstack-infra00:11
*** bradm has joined #openstack-infra00:22
*** slaweq has quit IRC00:24
*** markvoelker has quit IRC00:28
*** rfolco has quit IRC00:31
*** bobh has joined #openstack-infra00:37
*** bobh has quit IRC00:42
*** Goneri has joined #openstack-infra00:46
*** tkajinam has quit IRC01:00
*** tkajinam has joined #openstack-infra01:02
*** rcernin has joined #openstack-infra01:09
*** slaweq has joined #openstack-infra01:11
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265701:17
*** markvoelker has joined #openstack-infra01:24
*** Goneri has quit IRC01:24
*** slaweq has quit IRC01:24
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265701:28
*** Goneri has joined #openstack-infra01:28
*** Goneri has quit IRC01:34
*** tkajinam_ has joined #openstack-infra01:45
*** tkajinam has quit IRC01:48
*** owalsh has quit IRC01:49
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265701:53
openstackgerritIan Wienand proposed opendev/system-config master: bridge.opendev.org: use Ansible 2.8.0  https://review.opendev.org/66266201:53
*** whoami-rajat has quit IRC01:54
*** owalsh has joined #openstack-infra01:57
*** markvoelker has quit IRC01:58
*** zhurong has joined #openstack-infra01:59
*** slaweq has joined #openstack-infra02:11
*** owalsh_ has joined #openstack-infra02:18
*** owalsh has quit IRC02:20
*** hwoarang has quit IRC02:23
*** slaweq has quit IRC02:25
*** hwoarang has joined #openstack-infra02:25
*** bobh has joined #openstack-infra02:46
*** bobh has quit IRC02:51
*** jhesketh has joined #openstack-infra02:52
*** markvoelker has joined #openstack-infra02:55
*** owalsh_ is now known as owalsh02:56
*** hwoarang has quit IRC03:00
*** hwoarang has joined #openstack-infra03:09
*** slaweq has joined #openstack-infra03:12
openstackgerritIan Wienand proposed opendev/system-config master: bridge.opendev.org: use Ansible 2.8.0 stable  https://review.opendev.org/66266203:24
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265703:24
*** slaweq has quit IRC03:25
*** markvoelker has quit IRC03:29
openstackgerritIan Wienand proposed opendev/system-config master: bridge.opendev.org: use Ansible 2.8.0 stable  https://review.opendev.org/66266203:40
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265703:40
*** ykarel|away has joined #openstack-infra03:53
openstackgerritIan Wienand proposed opendev/system-config master: bridge.opendev.org: use Ansible 2.8.0 stable  https://review.opendev.org/66266204:07
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265704:07
*** rascasoft has quit IRC04:14
*** rascasoft has joined #openstack-infra04:16
*** whoami-rajat has joined #openstack-infra04:19
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265704:22
*** ramishra has joined #openstack-infra04:24
*** ramishra_ has joined #openstack-infra04:28
openstackgerritTristan Cacqueray proposed zuul/zuul master: executor: run cleanup playbook on stop  https://review.opendev.org/66188104:29
*** ramishra has quit IRC04:30
*** altlogbot_2 has quit IRC04:36
*** altlogbot_0 has joined #openstack-infra04:37
*** janki has joined #openstack-infra04:38
*** raukadah is now known as chandankumar04:41
*** hwoarang has quit IRC04:44
*** hwoarang has joined #openstack-infra04:46
*** ramishra has joined #openstack-infra04:51
*** ramishra_ has quit IRC04:53
*** sshnaidm is now known as sshnaidm|off05:03
*** pcaruana has joined #openstack-infra05:08
*** slaweq has joined #openstack-infra05:11
*** slaweq has quit IRC05:16
*** quiquell has joined #openstack-infra05:22
*** markvoelker has joined #openstack-infra05:25
*** udesale has joined #openstack-infra05:26
*** jtomasek has joined #openstack-infra05:32
*** dtantsur|afk is now known as dtantsur05:44
*** ykarel|away is now known as ykarel05:44
*** piotrowskim has joined #openstack-infra05:45
*** efried has quit IRC05:46
*** bobh has joined #openstack-infra05:50
*** bobh has quit IRC05:56
*** markvoelker has quit IRC05:58
*** roman_g has joined #openstack-infra06:00
*** jbadiapa has joined #openstack-infra06:02
*** lpetrut has joined #openstack-infra06:06
*** slaweq has joined #openstack-infra06:11
*** dpawlik has joined #openstack-infra06:16
*** ykarel is now known as ykarel|afk06:20
*** slaweq has quit IRC06:24
*** efried has joined #openstack-infra06:24
*** hwoarang has quit IRC06:26
*** hwoarang has joined #openstack-infra06:28
*** ccamacho has joined #openstack-infra06:29
*** pgaxatte has joined #openstack-infra06:29
*** slaweq has joined #openstack-infra06:39
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265706:51
hrwanyone know how much space centos mirror takes?06:51
*** markvoelker has joined #openstack-infra06:55
*** tosky has joined #openstack-infra06:58
*** kopecmartin|off is now known as kopecmartin06:59
*** lpetrut has quit IRC07:11
*** pkopec has joined #openstack-infra07:12
*** rpittau|afk is now known as rpittau07:19
hrwlooks like my centos+altarch mirror script works07:21
hrwwaiting for it to fetch aarch64, ppc64, x86-6407:21
*** markvoelker has quit IRC07:29
*** lpetrut has joined #openstack-infra07:31
*** xek has joined #openstack-infra07:40
*** _Cyclone_ has quit IRC07:41
*** jbryce has quit IRC07:41
*** davecore has quit IRC07:41
*** dougwig has quit IRC07:41
*** hamzy has quit IRC07:41
*** abelur has quit IRC07:41
*** rm_work has quit IRC07:41
*** icey has quit IRC07:41
*** mnaser has quit IRC07:41
*** bauzas has quit IRC07:41
*** snierodz has quit IRC07:41
*** timburke has quit IRC07:41
*** mugsie has quit IRC07:41
*** calebb has quit IRC07:41
*** evrardjp has quit IRC07:41
*** irclogbot_0 has quit IRC07:44
*** jaosorior has joined #openstack-infra07:44
*** dansmith has quit IRC07:45
*** kazsh has quit IRC07:45
*** irclogbot_1 has joined #openstack-infra07:45
*** dansmith has joined #openstack-infra07:47
*** prometheanfire has quit IRC07:47
*** amorin has quit IRC07:47
*** portdirect has quit IRC07:47
*** EvilienM has quit IRC07:47
*** tinwood has quit IRC07:47
*** ab-a has quit IRC07:47
*** lathiat has quit IRC07:47
*** dtantsur has quit IRC07:47
*** egonzalez has quit IRC07:47
*** kazsh has joined #openstack-infra07:47
openstackgerritTristan Cacqueray proposed zuul/zuul master: executor: run cleanup playbook on stop  https://review.opendev.org/66188107:49
ianwhrw: yep, we track it all at http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=107:50
ianwso what we currently mirror is ~40gb07:50
hrwuf. still have enough space then07:51
hrwhttp://paste.debian.net/1085517/ is WIP version of centos-mirror-update.sh to take care also about altarch07:52
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265707:52
*** prometheanfire has joined #openstack-infra07:52
*** portdirect has joined #openstack-infra07:52
*** ab-a has joined #openstack-infra07:52
*** EvilienM has joined #openstack-infra07:52
*** tinwood has joined #openstack-infra07:52
*** lathiat has joined #openstack-infra07:52
*** dtantsur has joined #openstack-infra07:52
*** egonzalez has joined #openstack-infra07:52
ianwhrw: hrm paste.debian.net doesn't like my request (503) but sounds good!07:53
*** _Cyclone_ has joined #openstack-infra07:53
*** jbryce has joined #openstack-infra07:53
*** davecore has joined #openstack-infra07:53
*** dougwig has joined #openstack-infra07:53
*** hamzy has joined #openstack-infra07:53
*** calebb has joined #openstack-infra07:53
*** icey has joined #openstack-infra07:53
*** abelur has joined #openstack-infra07:53
*** rm_work has joined #openstack-infra07:53
*** mnaser has joined #openstack-infra07:53
*** bauzas has joined #openstack-infra07:53
*** snierodz has joined #openstack-infra07:53
*** timburke has joined #openstack-infra07:53
*** mugsie has joined #openstack-infra07:53
*** evrardjp has joined #openstack-infra07:53
hrwhttps://pastebin.com/LKsN6P8507:54
*** jpena|off is now known as jpena07:55
*** bgmccollum has quit IRC07:55
*** ykarel|afk is now known as ykarel07:55
*** bgmccollum has joined #openstack-infra07:58
*** yoctozepto has quit IRC07:59
*** yoctozepto has joined #openstack-infra07:59
*** yoctozepto has quit IRC08:03
hrw+ rsync -rptDvz --delete --delete-excluded '--exclude=*.iso' '--exclude=*.img' --exclude=atomic --exclude=centosplus --exclude=EFI --exclude=experimental --exclude=fasttrack --exclude=LiveOS --exclude=isos --exclude=paas --exclude=sclo '--exclude=*/drpms' --include '*.xml*' --include '*.sqlite*' --include '*.noarch.rpm' --include '*.srpm' --copy-links --exclude=armhfp --exclude=i386 --exclude=power908:04
hrw--exclude=ppc64 --include '*.aarch64.rpm' --include '*.ppc64le.rpm' rsync://mirror.freethought-internet.co.uk/centos-altarch//7/ MIRROR/altarch/7/08:04
hrwugly but works08:04
*** iurygregory has joined #openstack-infra08:05
iurygregorymorning all08:05
hrwhi iurygregory08:05
*** kjackal has joined #openstack-infra08:05
hrwhm. most of --include removed08:06
*** ramishra has quit IRC08:07
*** yoctozepto has joined #openstack-infra08:09
*** apetrich has joined #openstack-infra08:10
*** yolanda has joined #openstack-infra08:11
*** dchen has quit IRC08:12
*** ociuhandu has joined #openstack-infra08:13
*** ociuhandu has quit IRC08:13
*** ociuhandu has joined #openstack-infra08:13
*** priteau has joined #openstack-infra08:15
*** jchhatbar has joined #openstack-infra08:20
*** janki has quit IRC08:20
*** chason has joined #openstack-infra08:23
*** markvoelker has joined #openstack-infra08:26
*** yoctozepto_ has joined #openstack-infra08:30
*** yoctozepto has quit IRC08:30
*** happyhemant has joined #openstack-infra08:31
openstackgerritMasayuki Igawa proposed openstack/os-testr master: Deprecate ostestr command  https://review.opendev.org/57363608:32
openstackgerritIan Wienand proposed opendev/system-config master: [wip] backups  https://review.opendev.org/66265708:34
*** raissa has quit IRC08:35
openstackgerritThomas Bechtold proposed openstack/os-testr master: Do not require coverage and hacking when running tests  https://review.opendev.org/50405008:35
*** rcernin has quit IRC08:36
openstackgerritMark Meyer proposed zuul/zuul master: Extend event reporting  https://review.opendev.org/66213408:37
*** ykarel is now known as ykarel|lunch08:37
*** raissa has joined #openstack-infra08:38
*** raissa has quit IRC08:38
*** jchhatbar has quit IRC08:40
*** janki has joined #openstack-infra08:40
*** raissa has joined #openstack-infra08:40
*** tkajinam_ has quit IRC08:46
*** zhurong has quit IRC08:48
openstackgerritMatthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions  https://review.opendev.org/57690708:50
openstackgerritMatthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI  https://review.opendev.org/63619708:50
openstackgerritMatthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST  https://review.opendev.org/63631508:50
openstackgerritMatthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration  https://review.opendev.org/63985508:50
openstackgerritMatthieu Huin proposed zuul/zuul master: Web: plug the authorization engine  https://review.opendev.org/64088408:51
openstackgerritMatthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.opendev.org/64109908:51
openstackgerritMatthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry  https://review.opendev.org/64240808:51
*** e0ne has joined #openstack-infra08:53
*** tinwood has quit IRC08:55
*** tinwood has joined #openstack-infra08:57
*** kaisers has quit IRC08:58
*** markvoelker has quit IRC08:59
*** kaisers has joined #openstack-infra08:59
*** gfidente has joined #openstack-infra09:01
*** tinwood has quit IRC09:01
*** tinwood has joined #openstack-infra09:04
openstackgerritMerged opendev/irc-meetings master: Update repository name in eavesdrop.o.o main page  https://review.opendev.org/66200909:19
*** ramishra has joined #openstack-infra09:20
*** dtantsur has quit IRC09:20
*** dtantsur has joined #openstack-infra09:25
*** Lucas_Gray has joined #openstack-infra09:30
*** ykarel|lunch is now known as ykarel09:34
*** kjackal has quit IRC09:42
*** kjackal has joined #openstack-infra09:44
openstackgerritMasayuki Igawa proposed openstack/os-testr master: Deprecate ostestr command  https://review.opendev.org/57363609:44
openstackgerritMarcin Juszkiewicz proposed opendev/system-config master: centos-mirror-update: handle non-x86 architectures  https://review.opendev.org/66272709:46
hrwhope I did that correct09:47
hrwaltarch for aarch64 and ppc64le takes ~same as centos/x86-6409:48
*** markvoelker has joined #openstack-infra09:55
*** ykarel_ has joined #openstack-infra10:05
*** aspiers has joined #openstack-infra10:08
*** ykarel has quit IRC10:08
*** ykarel_ is now known as ykarel10:22
*** markvoelker has quit IRC10:28
*** kjackal has quit IRC10:36
*** Lucas_Gray has quit IRC10:52
*** dpawlik has quit IRC10:54
*** Lucas_Gray has joined #openstack-infra10:55
*** janki has quit IRC10:57
*** janki has joined #openstack-infra10:58
*** kjackal has joined #openstack-infra11:00
*** trident has quit IRC11:04
*** trident has joined #openstack-infra11:06
openstackgerritMarcin Juszkiewicz proposed opendev/system-config master: centos-mirror-update: handle non-x86 architectures  https://review.opendev.org/66272711:07
*** tosky has quit IRC11:11
hrwreformatted to satisfy bashate11:12
*** dpawlik has joined #openstack-infra11:15
*** tobasco has joined #openstack-infra11:25
*** markvoelker has joined #openstack-infra11:25
*** tobasco is now known as tobias-urdin11:26
*** jpena is now known as jpena|lunch11:31
*** janki has quit IRC11:33
*** udesale has quit IRC11:40
*** udesale has joined #openstack-infra11:40
*** dave-mccowan has joined #openstack-infra11:48
*** hwoarang has quit IRC11:50
*** hwoarang has joined #openstack-infra11:53
*** dave-mccowan has quit IRC11:55
*** rosmaita has joined #openstack-infra11:56
*** tesseract has joined #openstack-infra11:57
*** markvoelker has quit IRC11:58
*** rfolco has joined #openstack-infra11:58
*** janki has joined #openstack-infra11:59
*** rh-jelabarre has joined #openstack-infra12:00
*** rlandy has joined #openstack-infra12:06
*** tosky has joined #openstack-infra12:06
*** janki has quit IRC12:13
*** priteau has quit IRC12:15
*** hwoarang has quit IRC12:19
*** hwoarang has joined #openstack-infra12:20
*** rfolco has quit IRC12:21
*** rfolco has joined #openstack-infra12:24
*** e0ne has quit IRC12:25
*** eharney has joined #openstack-infra12:25
*** e0ne has joined #openstack-infra12:25
*** apetrich has quit IRC12:26
*** apetrich has joined #openstack-infra12:27
*** jpena|lunch is now known as jpena12:29
*** rfarr_ has joined #openstack-infra12:39
*** rascasoft has quit IRC12:39
*** hwoarang has quit IRC12:41
*** abelur has quit IRC12:41
*** rfarr__ has joined #openstack-infra12:41
*** rascasoft has joined #openstack-infra12:41
*** icey has quit IRC12:42
*** hwoarang has joined #openstack-infra12:42
*** dougwig has quit IRC12:42
*** rm_work has quit IRC12:43
*** dougwig has joined #openstack-infra12:43
*** jbryce has quit IRC12:43
*** icey has joined #openstack-infra12:43
*** rfarr_ has quit IRC12:44
*** portdirect has quit IRC12:44
*** abelur has joined #openstack-infra12:44
*** priteau has joined #openstack-infra12:44
*** prometheanfire has quit IRC12:45
*** jbryce has joined #openstack-infra12:46
*** prometheanfire has joined #openstack-infra12:47
*** rm_work has joined #openstack-infra12:47
*** davecore has quit IRC12:48
*** davecore has joined #openstack-infra12:51
*** aaronsheffield has joined #openstack-infra12:55
openstackgerritDavid Moreau Simard proposed openstack/devstack-gate master: Pin ARA to <1.0.0  https://review.opendev.org/66274612:57
*** portdirect has joined #openstack-infra12:58
*** priteau has quit IRC13:03
*** owalsh_ has joined #openstack-infra13:04
*** zzehring has quit IRC13:05
*** owalsh has quit IRC13:06
*** zzehring has joined #openstack-infra13:06
*** zzehring has quit IRC13:06
*** zul has joined #openstack-infra13:07
*** kaisers has quit IRC13:07
*** beisner has quit IRC13:08
*** zzehring has joined #openstack-infra13:08
*** beisner has joined #openstack-infra13:10
*** kaisers has joined #openstack-infra13:10
*** lseki has joined #openstack-infra13:12
*** jaosorior has quit IRC13:15
*** jcoufal has joined #openstack-infra13:18
*** sthussey has joined #openstack-infra13:18
*** EvilienM is now known as EmilienM13:24
*** sreejithp has joined #openstack-infra13:27
*** happyhemant has quit IRC13:33
*** mnaser has quit IRC13:34
*** happyhemant has joined #openstack-infra13:35
*** rfarr__ has quit IRC13:35
*** pkopec has quit IRC13:35
*** diablo_rojo has joined #openstack-infra13:36
*** mnaser has joined #openstack-infra13:37
*** panda has quit IRC13:39
*** tdasilva has joined #openstack-infra13:40
*** bobh has joined #openstack-infra13:41
*** panda has joined #openstack-infra13:42
*** rfarr has joined #openstack-infra13:43
*** mriedem has joined #openstack-infra13:44
*** bobh_ has joined #openstack-infra13:45
fricklerinfra-root: four job failures on this patch, all with read errors from dfw.rax mirror, if someone has time to take a deeper look https://review.opendev.org/66273413:47
*** bobh has quit IRC13:48
*** yamamoto has joined #openstack-infra13:55
*** ykarel_ has joined #openstack-infra14:00
*** ykarel has quit IRC14:03
mwhahahayea we're also seeing dfw mirror problems as well14:07
corvusfrickler, infra-root: proxy_80_error.log:[Mon Jun 03 11:24:12.158009 2019] [cache_disk:warn] [pid 1362:tid 139790028625664] (28)No space left on device: [client 2001:4800:7819:105:be76:4eff:fe03:bce1:51378] AH00722: could not write to vary file /var/cache/apache2/proxy/aptmpVnkcWD14:07
corvus/dev/mapper/main-proxycache   63G   60G   24K 100% /var/cache/apache214:08
corvuson ord, the fs is: /dev/mapper/main-proxycache   99G   82G   12G  88% /var/cache/apache214:09
corvusweird, the lvsize is 120GB14:11
corvusoh, no, sorry, that's the afs cache, the proxy cache is 6414:11
corvusthere is, however, 64G free in the vg group, so we can extend the proxycache to 128 and resize the fs14:12
corvusit's not clear to me why 64 was used instead of 128 in the first place, does anyone know?14:12
corvuswe don't seem to have the mirrors documented: https://docs.openstack.org/infra/system-config/systems.html14:13
corvusexcept https://docs.openstack.org/infra/system-config/reprepro.html14:14
corvusbut that only deals with afs and doesn't mention the proxy14:14
*** michael-beaver has joined #openstack-infra14:17
fungithe way the apache cache cleanup breathes, it needs a fair amount more space on the filesystem than its cleanup deletes down to14:17
fungii don't know that we ever settled on an exact percentage14:18
fungifluctuations in the rate of content churn can make it highly variable14:18
fungiallocating the additional 64gb to it sounds reasonable14:20
corvusok.  i'll do that14:20
fungii casn try to write up something about the proxy cache nowish14:21
fungier, can14:21
*** quiquell has quit IRC14:22
fungii'll rename the reprepro doc and expand on it14:23
*** lpetrut has quit IRC14:23
corvus#status log resized proxycache volume from 64 to 120GB on rax.dfw.opendev.org because it was 100% used (other mirrors report ~80G used)14:23
openstackstatuscorvus: finished logging14:23
fungisince we also do rsync mirroring14:23
*** lpetrut has joined #openstack-infra14:23
corvusi think that's all that should be necessary ^ frickler, mwhahaha14:23
*** Goneri has joined #openstack-infra14:24
fungiyeah, apache should just go back to writing there now that it's not getting enospc14:24
fricklercorvus: thanks, seems we instruct apache to target 70G, so 64G was surely too small: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/manifests/mirror.pp#L34714:24
corvusah, yep!14:26
*** ykarel_ is now known as ykarel14:26
*** yamamoto has quit IRC14:26
*** yamamoto has joined #openstack-infra14:27
*** yamamoto has quit IRC14:27
fungias part of documenting this, i'll try to put together a quick survey of what we have allocated across the various mirror servers14:28
*** yamamoto has joined #openstack-infra14:28
*** yamamoto has quit IRC14:33
*** lpetrut has quit IRC14:35
*** msmarcal has quit IRC14:38
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: registry test job  https://review.opendev.org/66132714:40
*** pgaxatte has quit IRC14:40
*** bobh_ has quit IRC14:43
*** igordc has joined #openstack-infra14:50
*** yamamoto has joined #openstack-infra14:51
*** owalsh has joined #openstack-infra14:52
*** owalsh_ has quit IRC14:54
*** sreejithp has quit IRC14:56
*** yamamoto has quit IRC14:57
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: registry test job  https://review.opendev.org/66132714:59
*** ccamacho has quit IRC15:01
*** jamesmcarthur has joined #openstack-infra15:01
*** ykarel_ has joined #openstack-infra15:01
hrwfungi: can you take a look at https://review.opendev.org/662727 patch? changed centos mirror script like we discussed yesterday15:03
*** ccamacho has joined #openstack-infra15:04
*** ykarel has quit IRC15:04
fungihrw: already saw it, haven't had time to review yet but thanks for working on that. your summary in here seemed on the right track15:04
hrwfungi: thanks15:05
*** josephrsandoval has joined #openstack-infra15:05
*** eharney has quit IRC15:06
*** dklyle has joined #openstack-infra15:07
*** ykarel_ is now known as ykarel15:07
*** jamesmcarthur_ has joined #openstack-infra15:07
*** igordc has quit IRC15:08
*** igordc has joined #openstack-infra15:08
*** tosky has quit IRC15:08
*** irclogbot_1 has quit IRC15:10
*** jamesmcarthur has quit IRC15:10
*** altlogbot_0 has quit IRC15:11
*** irclogbot_3 has joined #openstack-infra15:11
*** altlogbot_2 has joined #openstack-infra15:12
*** ykarel is now known as ykarel|away15:12
*** e0ne has quit IRC15:15
zulgot a question when i do git review -s in the x/browbeat tree it fails with something like " ssh://smalleni@review.openstack.org:29418/openstack/browbeat.git"15:16
zulis there anything we need to do to the git tree?15:16
*** priteau has joined #openstack-infra15:17
AJaegerzul, looking at https://opendev.org/x/browbeat/src/branch/master/.gitreview it is correct15:18
AJaegerDo you see that in your checked out tree as well?15:18
zulAJaeger: yep15:19
*** eharney has joined #openstack-infra15:19
zulAJaeger: on new clones at least15:20
*** whoami-rajat has quit IRC15:20
AJaegerzul, so a "git review -s" on new clones should not get the fail you have - so, update first ..15:21
zulAJaeger: gotcha thnaks15:21
*** xek has quit IRC15:23
*** mgoddard has joined #openstack-infra15:29
*** priteau has quit IRC15:32
*** chandankumar is now known as raukadah15:43
*** sreejithp has joined #openstack-infra15:44
*** zbr has quit IRC15:47
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: registry test job  https://review.opendev.org/66132715:48
*** zbr has joined #openstack-infra15:55
*** ykarel|away has quit IRC15:56
*** josephrsandoval has quit IRC15:59
mriedemis there a bug by chance for that rax dfw full disk issue?16:02
mriedemlooking to e-r that16:02
clarkbmriedem: I haven't seen one so probably not16:02
mriedemok i'll do that16:02
clarkb(pretty sure I still get emails from lp about those?)16:03
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add a registry test job  https://review.opendev.org/66132716:05
openstackgerritJames E. Blair proposed openstack/project-config master: Add registry test to zuul-jobs  https://review.opendev.org/66279416:05
mriedemlhttps://bugs.launchpad.net/openstack-gate/+bug/183147416:06
openstackLaunchpad bug 1831474 in OpenStack-Gate "pip install failing on rax-dfw nodes with "('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))"" [Undecided,New]16:06
openstackgerritMatt Riedemann proposed opendev/elastic-recheck master: Add query for pip install rax-dfw bug 1831474  https://review.opendev.org/66279716:09
openstackbug 1831474 in OpenStack-Gate "pip install failing on rax-dfw nodes with "('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))"" [Undecided,New] https://launchpad.net/bugs/183147416:09
*** Lucas_Gray has quit IRC16:09
*** gyee has joined #openstack-infra16:15
*** dtantsur is now known as dtantsur|afk16:15
*** yamamoto has joined #openstack-infra16:22
openstackgerritMark Goddard proposed opendev/irc-meetings master: Hold kolla meetings in #openstack-kolla  https://review.opendev.org/66280116:25
*** tesseract has quit IRC16:27
*** rpittau is now known as rpittau|afk16:28
*** auristor has quit IRC16:30
*** yamamoto has quit IRC16:30
*** rkukura has joined #openstack-infra16:33
*** auristor has joined #openstack-infra16:34
*** panda has quit IRC16:35
*** dosaboy has joined #openstack-infra16:36
*** panda has joined #openstack-infra16:37
openstackgerritMerged opendev/elastic-recheck master: Add query for pip install rax-dfw bug 1831474  https://review.opendev.org/66279716:37
openstackbug 1831474 in OpenStack-Gate "pip install failing on rax-dfw nodes with "('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))"" [Undecided,Confirmed] https://launchpad.net/bugs/183147416:37
* mordred waves to the fine humans from an aeroplane16:46
cloudnullo/16:47
* mordred hands cloudnull a pie he found laying around16:48
fungimordred: i think i can see you up there!16:54
* fungi waves back16:54
*** kjackal has quit IRC16:54
*** gfidente is now known as gfidente|afk16:55
*** weifan has joined #openstack-infra17:00
*** whoami-rajat has joined #openstack-infra17:03
*** jpena is now known as jpena|off17:04
* AJaeger waves to mordred17:05
*** rossella_s has quit IRC17:06
fricklercorvus: this seems to have been half an hour after you increased the disk size. do we need to clean up broken cached files or will that resolve itself soon enough? http://logs.openstack.org/34/662734/1/check/tempest-full/bbbaf01/job-output.txt.gz#_2019-06-03_14_49_23_86508917:11
corvusfrickler: oof; that i don't know.17:12
*** kjackal has joined #openstack-infra17:19
openstackgerritMonty Taylor proposed zuul/zuul master: Read multiple *.conf files from directory  https://review.opendev.org/66097717:20
*** rlandy has quit IRC17:22
*** rlandy has joined #openstack-infra17:22
*** hwoarang has quit IRC17:24
*** jamesmcarthur_ has quit IRC17:25
*** rkukura has quit IRC17:27
*** jamesmcarthur has joined #openstack-infra17:29
*** aakarsh has joined #openstack-infra17:29
*** hwoarang has joined #openstack-infra17:31
aakarshHi corvus, since the repo move looks like one of the project our team works on has issues syncing repo and github.17:33
*** jamesmcarthur_ has joined #openstack-infra17:33
aakarshany pointers on what could be wrong?17:33
aakarshsay for example https://review.opendev.org/#/c/662792/ got merged17:33
aakarshhttps://github.com/openstack/browbeat/tree/master17:34
aakarshseems to have not updated since march17:34
AJaegeraakarsh, yes - as announced on openstack-discuss list17:34
* mordred is landing - will be back online in a bit17:35
AJaegeraakarsh: sorry, can't find link directly17:35
aakarshAJaeger, were there any changes that were needed for the project in new gerrit for syncing with github?17:36
*** jamesmcarthur has quit IRC17:36
AJaegeraakarsh: finally found it, see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html17:37
AJaegeraakarsh: note that you will not be able to sync to openstack/browbeat - you need to find another location17:37
aakarshso we'd need to probably put browbeat in another org?17:38
aakarshand then sync it there?17:38
AJaegercorrect17:39
aakarshah gotcha thanks a ton AJaeger17:39
*** e0ne has joined #openstack-infra17:41
*** udesale has quit IRC17:41
*** jamesmcarthur_ has quit IRC17:49
evrardjphello17:57
evrardjpDid I mis an annoucement with some kind of failures going on?17:57
*** ociuhandu_ has joined #openstack-infra17:58
clarkbevrardjp: there was an unhappy mirror in rax dfw. But other than that I think that is the only known issue (and now fixed)17:58
evrardjpit seems I have trouble hitting logs, and promote pipeline failed me recently: https://review.opendev.org/#/c/659578/1017:58
clarkb(thank you corvus for fixing that)17:58
evrardjpclarkb: I see17:58
evrardjpit's possible it's on my side for the logs17:58
clarkbcorvus: ^ansible_date_time is undefined for promote-docker-image17:59
clarkbcorvus: has that changed with the test job updates maybe?17:59
evrardjpclarkb: it used to work though :)17:59
clarkb(also I'm here now after doing some emergency fixing to my front stoop)17:59
evrardjpoh wait17:59
evrardjpit is possible it doesn't, but I ensured the thing was available on build18:00
evrardjpnever tried on promote it's a new thing18:00
*** ociuhandu_ has quit IRC18:01
clarkbI guess you have to gather facts for ansible_date_time to be present18:01
evrardjpclarkb: oh indeed logs are fine now. and yes I see ansible_date_time18:01
evrardjpclarkb: didn't I do that?18:01
*** ociuhandu has quit IRC18:01
evrardjpthat seem a very weird oversight from me...18:01
evrardjpor maybe I am just reusing a play, let me check :)18:02
clarkbit may be that we don't gather facts on localhost by default which that playbook runs against?18:02
openstackgerritEd Leafe proposed opendev/irc-meetings master: Change Placement meeting to Office Hour  https://review.opendev.org/66281418:02
evrardjpclarkb: yes it's just inheriting from promote-docker-image18:03
evrardjpclarkb: I suppose it's this?18:04
clarkbya and that playbook doesn't say gather_facts: false18:04
evrardjphttps://opendev.org/zuul/zuul-jobs/src/branch/master/playbooks/docker-image/promote.yaml18:04
evrardjpyes18:04
clarkbevrardjp: yes I believe it is that playbook18:04
evrardjpso yes it should gather_facts by default18:04
clarkbcorvus: ^ any idea why it isn't gathering facts by default?18:04
evrardjpclarkb: the thing is, for that case, I suppose I can't have a pre-run play18:06
*** e0ne has quit IRC18:06
clarkbcorrect pre, run, post are separate ansible invocations18:06
evrardjpI suppose pre-run and run are different "ansible runs"?18:06
evrardjpk18:06
evrardjpalso that raises the question if promote can technically be geographically separated from the check/gates machines18:07
clarkbyes18:07
clarkbnot sure I understand that question18:08
evrardjpI wonder if I should do differently then, because the date will not be the same on promote machine18:08
evrardjpI think it's an okay gap though, we are not in a big hurry for this18:08
clarkbthe date is to make up a password hash18:08
*** e0ne has joined #openstack-infra18:08
clarkbso shouldn't be an issue18:09
evrardjpnot in this case, but it shouldn't be an issue either18:09
*** ociuhandu has joined #openstack-infra18:10
evrardjpI guess I will wait for corvus's insights on that fact gathering :)18:10
*** markvoelker has joined #openstack-infra18:13
openstackgerritJean-Philippe Evrard proposed zuul/zuul-jobs master: Explicitly store facts for promote  https://review.opendev.org/66281718:13
clarkbevrardjp: ya I know we are using promote jobs for zuul itself and I haven't heard of that problem18:14
clarkbso guessing there is some missing element18:14
*** ociuhandu has quit IRC18:15
fungihow is the date being used for making a password hash?18:15
evrardjpI don't know that part :p18:15
clarkbone sec I'll get a link18:15
evrardjpthat's not me :)18:15
clarkbfungi: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/run-buildset-registry/tasks/main.yaml#L28-L3118:16
evrardjpclarkb: as you can see a little above, I added Jim on that review, so he can tell us more about that promote18:16
clarkbfungi: a hash is generated from the current date time then we use a portion of that as the password. This is for the temporary buildset registry18:17
evrardjpoh in my case it's different18:17
evrardjpI am using the date to freeze image by date. https://review.opendev.org/#/c/659578/10/zuul.d/openvswitch.yaml18:18
evrardjpit worked on the build/publish but not on promote18:18
clarkbevrardjp: that is for the build18:18
clarkbbut also why is freezing by the change insufficient?18:18
evrardjpclarkb: I am passing the same vars for promote18:18
fungiclarkb: i guess that's okay, as long as we're comfortable with it being trivially brute-forceable for someone to work out that registry password in a fraction of a secnod18:19
clarkbok so I've gotten confused then the thing above probably doesn't run in promote and that is why it works for zuul?18:19
clarkbfungi: I'm not actually sure if we are ok with that or not. Would have to think about it (I'm just discovering that is how the password is generated now)18:19
fungiat least i assume someone streaming the log can see what time the "Generate registry password" task fires18:20
*** e0ne has quit IRC18:20
evrardjpclarkb: I am not sure to understand the "that is for the build" comment18:20
*** e0ne has joined #openstack-infra18:21
clarkbevrardjp: the link you linked does not change the promote job, only the build job18:21
fungiso that narrows the window of possible guesses into the tens/hundreds of thousands18:21
evrardjpclarkb: well it kinda does, because of the yaml macro18:21
evrardjpso when I introduced this for build, I also have this introduced for promote18:21
clarkbevrardjp: I see you reuse that block in the promote job18:21
evrardjpcorrect18:22
evrardjpthe build worked, the promote didn't18:22
clarkbso ya my guess is that the localhost only task doesn't father facts by default18:22
*** bobh has joined #openstack-infra18:22
evrardjpclarkb: if https://review.opendev.org/#/c/662817/ doesn't make it work, we can be sure it's something else :p18:23
evrardjpmaybe I should have added a setup: pre_task here18:23
zbr@infra-root: who will be present at Shanghai summit?18:23
evrardjpjust to be on the safe side18:23
clarkbevrardjp: the problem with that is you'll need a host to run it18:23
evrardjpand?18:23
clarkbevrardjp: I think you are probably better off using the change based system if you can (this is known to work for zuul)18:23
clarkbevrardjp: and those jobs are specifically designed to not need a test node18:24
clarkbzbr: I expect fungi and I will be there18:24
evrardjpclarkb: this is fine18:24
fungizbr: yeah, i'm expecting to be18:24
fungiassuming my visa is granted18:24
evrardjpwell I mean it's just renaming the thing that uploaded as changeid_tagname to tagname when you think about it18:24
evrardjpit's just that tagname contains a date18:25
clarkbevrardjp: maybe its fine for you but running a VM (or even a container) to do the equivalent of `date` is silly imo18:25
clarkbmaybe we can update ansible to just lookup date via python internally and expose that as a var separate from host facts18:26
evrardjpnot sure to see the difference between running curl and running date at the end :)18:26
evrardjp(compute resources wise)18:26
*** ramishra has quit IRC18:26
evrardjpI think that's where I can confused18:26
clarkbevrardjp: the current jobs only run on the executor18:26
clarkbevrardjp: there are no test nodes involved18:26
evrardjpI still get that18:26
zbrclarkb: fungi: time to ask my mgr about that.18:26
evrardjpand?18:27
clarkbBut as a side effect I think that may mean we don't run gather facts on that host18:27
clarkbdue to how zuul treats the executors specially18:27
clarkbif you want to gather facts and use ansible_date_time you will need to update the job to have a test node18:27
clarkb(I think)18:27
clarkbso now you've gone from no node job to job with VM to run `date`18:27
evrardjpclarkb: we could technically have a different task inside this role to run setup:18:27
evrardjpand you will still get this :)18:27
clarkbevrardjp: only if zuul allows it18:27
evrardjpI see18:28
clarkband I think it doesn't which is why it doesn't work as is18:28
clarkbbecause the default in ansible is to gather facts18:28
clarkband we don't tell it not to18:28
clarkbso the assumption I have is that zuul is not gathering facts intentionally because it doesn't want to expose details about the executors that it shouldn't expose18:28
evrardjpgot it!18:29
evrardjpnow I get it :)18:29
clarkbyou may be able to do a network request to get a timestamp using the ansible http module18:29
evrardjp(I got it earlier that it was restricted, but now I understand why you would want to avoid it)18:29
clarkb(just thinking out loud for ways to get a timestamp)18:29
evrardjplookup('pipe','date') is kinda ugly18:30
evrardjpand not sure if that's allowed either18:30
clarkbyou can't exec commands on the executor18:30
evrardjpk18:30
evrardjpbut you can exec python? :p18:30
*** e0ne has quit IRC18:31
evrardjpI thought I had a wonderful idea there! :p18:31
evrardjphahahah18:32
evrardjpI got another idea18:32
evrardjpI am fine with not using a date, but in that case I would need not the date, but the sha of the thing I would just have merged, would that be easier?18:33
clarkbI think that is available as a zuul var so ya probably18:33
evrardjpnot the sha before the merge, but the final merged state18:34
clarkbI think that is available (I don't know for sure because we don't capture the inventory file on localhost only jobs)18:34
*** rkukura has joined #openstack-infra18:35
evrardjpI think we should wait for corvus to give us an idea of what we can achieve I guess18:35
clarkbevrardjp: why is the change insufficient? (curious what the use case is)18:35
evrardjpthe change id_tagname will get erased over time18:36
evrardjpif you are talking about that18:37
clarkbok so you are subverting the garbage collection process? iirc we did that so you don't end up with hundreds of thousands of tags18:37
openstackgerritTobias Henkel proposed zuul/zuul master: Parallelize githib event processing  https://review.opendev.org/66281818:37
clarkbbut maybe that can be optional? I'd have to go reread how that works18:37
*** bobh has quit IRC18:38
evrardjpclarkb: well date was fine for us, because it was clear when it was really published18:39
evrardjpelse you have to go gerrit, check the date the change was merged.18:40
evrardjpI think it's fine in both cases18:40
*** weifan has quit IRC18:40
clarkbevrardjp: you can also ask the docker registry for the date18:40
clarkb(it is an attribute of the tag)18:40
*** weifan has joined #openstack-infra18:40
evrardjpoh18:41
evrardjpthat would be interesting18:41
evrardjpso changing the role there, have a new var: publish_with_date or something18:42
evrardjpand trigger that behaviour18:42
clarkbits there by default18:42
evrardjpit would make that very cleanly18:42
clarkbif you go to docker hub and look at a tag it tells you how old it is18:42
evrardjpoh I thought you meant something else18:42
clarkbchange_662112_latest tag for zuul/zuul was set June 03, 2019 8:22am PDT18:43
*** altlogbot_2 has quit IRC18:43
evrardjpyes I guess18:43
*** altlogbot_3 has joined #openstack-infra18:44
evrardjpthat kinda breaks my parsing rules I have written downstream, but I can deal with that :)18:44
evrardjpI would still prefer knowing what's possible/not possible, for clearer understanding18:44
*** weifan has quit IRC18:45
evrardjpbut in that case, to have a frozen forever, I should just edit the promote role to not remove the change_<changeid>_<tagname>18:45
clarkbya we can probably stand to say more details at https://zuul-ci.org/docs/zuul/developer/ansible.html#restricted-execution18:45
*** e0ne has joined #openstack-infra18:46
openstackgerritTobias Henkel proposed zuul/zuul master: Parallelize github event processing  https://review.opendev.org/66281818:46
*** _erlon_ has joined #openstack-infra18:46
*** kopecmartin is now known as kopecmartin|off18:47
evrardjpclarkb: please note that this runs: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/promote-docker-image/tasks/promote-cleanup.yaml#L718:49
evrardjpwhich kinda makes me super sad :p18:49
evrardjp(I mean the date is _there_ !)18:49
clarkbevrardjp: I think that exec works because it is in a trusted context18:50
clarkbso there may be ways to get around this for the date18:50
evrardjpbut I like your idea of providing this as a default for everyone18:51
clarkbalso it isn't using a fact18:51
clarkbit is asking python for the current date from that python interpreter (which ansible could do too, but I'm sure it doesn't yet)18:51
openstackgerritMerged zuul/zuul-jobs master: Add a registry test job  https://review.opendev.org/66132718:52
evrardjpclarkb: yes it's totally different way to run18:52
evrardjpI don't see lookup as being thing filtered on the link above18:52
evrardjpbut I think I prefered to not have the lookup anyway18:53
clarkbevrardjp: it works because it is in a trusted context I think18:53
evrardjpI can't remember why nowadays18:53
AJaegerclarkb: we should merge https://review.opendev.org/662794 as well to test the registry test job, please review18:53
clarkbAJaeger: yup just approved18:53
AJaegerclarkb: ;) Thanks!18:53
clarkbAJaeger: I had it queued up and was just waiting for that change to merge before I approved18:53
evrardjpbut I like the idea of adding something in the role to add a task in there: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/promote-docker-image/tasks/promote-retag-inner.yaml#L11-L20 to just add the tag with a -date18:54
clarkbsince I don't think they share a queue18:54
evrardjp(optionally)18:54
clarkbevrardjp: ya I think that would work too18:54
clarkband I've just approved the test jobs for these roles so we should be able to test that chagne too :)18:54
evrardjpyou want me to push that up, so we can have corvus step in and say which way is best?18:55
clarkbsure18:55
evrardjpok will do18:56
openstackgerritMerged openstack/project-config master: Add registry test to zuul-jobs  https://review.opendev.org/66279419:05
*** my_nd has joined #openstack-infra19:06
clarkbAJaeger: ^ should be all set now :)19:09
openstackgerritJean-Philippe Evrard proposed zuul/zuul-jobs master: Append date to tag optionally  https://review.opendev.org/66282819:11
*** e0ne has quit IRC19:13
evrardjpclarkb: for testing do you want me to flip the boolean on true temporarily?19:13
evrardjpnot really sure how you want to test that optional feature :p19:13
*** tdasilva has quit IRC19:14
AJaegerclarkb: cool19:15
my_ndHi there, do you know if there are limitations or specials configuration for installing Whitecloud Rio using kolla-ansible in Ubuntu Server 18.04 LTS with processors  AMD Opteron(tm) 6380 working in compute nodes and KVM as a hypervisor.19:15
my_ndHi there, do you know if there are limitations or specials configuration for installing Openstack Rocky using kolla-ansible in Ubuntu Server 18.04 LTS with processors  AMD Opteron(tm) 6380 working in compute nodes and KVM as a hypervisor.19:15
*** imacdonn has quit IRC19:18
*** imacdonn has joined #openstack-infra19:19
*** eharney has quit IRC19:19
*** altlogbot_3 has quit IRC19:21
mwhahahastill seeing dfw mirror issues, http://logs.openstack.org/85/661585/3/check/tripleo-ci-centos-7-undercloud-containers/f6948cf/job-output.txt.gz#_2019-06-03_15_31_14_45167319:21
mwhahahaactually nm that look to be from this morning19:22
fungimy_nd: you probably want to ask in #openstack-kolla19:22
* mwhahaha breaks out his UTC converter19:22
fungimy_nd: see https://wiki.openstack.org/wiki/IRC for a list of irc channels related to openstack19:22
my_ndthanks!19:23
corvusevrardjp, clarkb: i really like the original idea of using {{currentdate}} (since it lets the project decide the policy); but aside from the fact problem, i also think it would be subject to race conditions if gate/promote straddled a date change.  i'd love to think of a way to make that work, but if we can't, i think the general approach in 662828 would probably work.19:23
fungimwhahaha: that was a little over 4 hours ago19:23
mwhahahafungi: yea i the job results just came back, needed to figure out the time19:23
fungimwhahaha: we did see some possible lingering effects where apache was trying and failing to return what may be corrupt/truncated cache contents19:24
fungiwe expect those to age out but not sure how rapidly19:24
corvusevrardjp, clarkb: i think we just don't gather facts for efficiency...?19:25
clarkbcorvus: possibly, though do we want to expose the execuyor info? (I need to go reread what facts are gathered)19:25
corvusevrardjp, clarkb: we might be able to explicitly gather facts?19:25
corvuspabelanger is expert in this ^ :)19:26
*** altlogbot_1 has joined #openstack-infra19:26
*** dklyle has quit IRC19:29
*** dklyle has joined #openstack-infra19:29
*** weifan has joined #openstack-infra19:34
openstackgerritEd Leafe proposed opendev/irc-meetings master: Change Placement meeting to Office Hour  https://review.opendev.org/66281419:37
openstackgerritTobias Henkel proposed zuul/zuul master: Parallelize github event processing  https://review.opendev.org/66281819:49
*** e0ne has joined #openstack-infra19:54
*** e0ne has quit IRC19:57
*** hamzy_ has joined #openstack-infra20:06
*** jamesmcarthur has joined #openstack-infra20:06
*** e0ne has joined #openstack-infra20:09
*** mugsie_ has joined #openstack-infra20:09
*** evrardjp_ has joined #openstack-infra20:09
*** jcoufal has quit IRC20:11
*** bauzas_ has joined #openstack-infra20:12
*** snierodz_ has joined #openstack-infra20:12
mriedemare we ok to recheck on the rax-dfw failures now? or is it a crap shoot?20:13
*** e0ne has quit IRC20:13
clarkbmriedem: I think you should be fine20:13
mriedemwas there a fix i can link into lp?20:14
*** timburke_ has joined #openstack-infra20:14
*** _Cyclone_ has quit IRC20:15
*** hamzy has quit IRC20:15
*** bauzas has quit IRC20:15
*** snierodz has quit IRC20:15
*** timburke has quit IRC20:15
*** mugsie has quit IRC20:15
*** calebb has quit IRC20:15
*** evrardjp has quit IRC20:15
*** bauzas_ is now known as bauzas20:15
*** _Cyclone_ has joined #openstack-infra20:15
clarkbmriedem: corvus expanded the disk (via lvm) and then expanded the filesystem on the disk. So no change20:16
*** jtomasek has quit IRC20:16
*** imacdonn has quit IRC20:17
*** snierodz_ has quit IRC20:17
*** trident has quit IRC20:18
*** Qiming has quit IRC20:18
mriedemack20:19
*** snierodz has joined #openstack-infra20:20
*** trident has joined #openstack-infra20:20
*** Qiming has joined #openstack-infra20:20
*** dpawlik has quit IRC20:21
*** jamesmcarthur has quit IRC20:25
corvusinfra-root: i'm planning on creating tarballs.opendev.org; currently tarballs.openstack.org is on the static fileserver, should i do what i did for docs.opendev.org and put it in afs and serve from files.o.o?20:31
corvuswe could do the same thing we do for doc publication and release the volume every 5m, so there would be a slight delay before -lastest was really the latest, but we'd get redundancy20:32
clarkbthe redundancy aspect is a nice win with afs20:33
clarkbI think the next big OS upgrade will come with a more difficult afs upgrade since the versions bump a bit more (somethign to keep in mind while we put more stuff in afs)20:34
clarkbthere is enough stuff in afs now that I don't think adding eg tarballs there would be a huge deal though20:34
corvusi've learned enough about ceph to understand the upgrade path is not easier.  :)20:35
corvusclarkb: hopefully one of our cloud providers will have ceph-via-manila by then :)20:36
openstackgerritMonty Taylor proposed zuul/zuul master: Read multiple *.conf files from directory  https://review.opendev.org/66097720:37
*** jamesmcarthur has joined #openstack-infra20:37
clarkbcorvus: that would be nice20:37
fungicorvus: i've been wanting for a while to see all the stuff on the tarballs site migrate to afs (along with a bunch of the web content subdomains like releases, security, service-types, specs, summit...)20:37
fungibasically the only things on there which aren't candidates to move into afs are logs and logs-dev20:38
corvusand we have a plan for those20:38
fungiand a bunch of redirects we could move to files.o.o or ask the community again about dropping20:38
*** altlogbot_1 has quit IRC20:45
*** sgw1 has quit IRC20:45
*** Goneri has quit IRC20:49
*** slaweq has quit IRC20:52
*** jbadiapa has quit IRC20:52
*** nicolasbock has joined #openstack-infra20:53
*** sreejithp has quit IRC21:04
*** sgw has joined #openstack-infra21:04
*** rlandy is now known as rlandy|brb21:04
mordredcorvus: just a cname might be sufficient - the upload jobs already upload automatically namespaced by org/repo - and we already have a single collection of org/repos21:06
*** pcaruana has quit IRC21:07
mordred(like, in this case, I don't think we need an additional per-tenant namespace added, since zuul/zuul is a unique thing within opendev regardless of zuul tenant21:08
* mordred is landing again - may drop off in a minute or two21:08
corvusmordred: tarballs.openstack.org is namespaced by short name, not full repo name (eg 'nova' not 'openstack/nova')21:11
corvusmordred: over in #zuul earlier, i was trying to say that we don't need to namespace by zuul tenant, we need to namespace by full project name21:11
mordredcorvus: ah - gotcha. yes - we do need to namespace by full project name21:12
mordredcorvus: we should also ponder how renames works with artifacts published to places namespaced by named21:13
mordrednames21:13
corvusmordred: indeed.  redirects would be nice.  maybe someone will contribute that.  but i don't think we should block on having that.21:14
mordredoh - TOTALLY - just saying we shoudl think about it ... should have said "later"21:15
*** rlandy|brb is now known as rlandy21:18
*** jamesmcarthur has quit IRC21:19
*** slaweq has joined #openstack-infra21:20
*** whoami-rajat has quit IRC21:22
*** slaweq has quit IRC21:25
*** eharney has joined #openstack-infra21:29
*** sshnaidm|off has quit IRC21:34
openstackgerritMerged zuul/zuul master: Set the DiskAccountant log as warning level  https://review.opendev.org/66099621:36
dmsimardinfra-root: I'm hoping to release ara 1.0 this week (as early as tomorrow if nothing goes wrong) and I tried to catch unpinned installations to avoid unexpected upgrades -- there's still puppet-zuul ( https://review.opendev.org/#/c/661329/ ) and devstack-gate ( https://review.opendev.org/#/c/662746/ ) left21:38
*** sshnaidm has joined #openstack-infra21:40
*** sshnaidm is now known as sshnaidm|off21:41
*** Goneri has joined #openstack-infra21:47
*** yamamoto has joined #openstack-infra21:50
*** sgw has left #openstack-infra21:51
clarkbianw mordred so nb03 is in a bad state. I think because it can't delete the old images it is carrying for the arm64 clouds that went away21:52
clarkbat first I was thinking that we had to completely clean the clouds.yaml and nodepool configs up but I've since realized that I can probably just delete the images we don't need/want out from under nodepool21:52
*** eharney has quit IRC21:52
clarkbthen when we clean up the clouds.yaml and nodepool config we'll get the db side of things21:53
clarkbany objection to me rm'ing those uneeded image files now?21:53
clarkb(its disk is full sorry should've mentioned that in first message)21:53
*** mattw4 has joined #openstack-infra21:54
*** yamamoto has quit IRC21:54
ianwcorvus: i think that disk issue was my mistake in setting it up.  i must have done 50% for the first partition, then i mean to do 100%FREE for the next but if i did 50% again that would explain why 25% was left free21:56
corvusianw: ah!  that makes sense, thanks.  glad there wasn't anything deeper going on :)21:57
ianwclarkb: no issues with cleaning up nb03 ... i can take a look in about 30 minutes if you'd like21:58
*** rcernin has joined #openstack-infra21:58
clarkbianw: would you like to take a look before I delete the old images on disk?21:58
clarkbpretty sure they are kept around for arm64ci-nrt and linaro clouds21:58
clarkbthat don't really exist anymore so should be safe to remove them and we'll have to do manual cleanup anyway21:58
*** mattw4 has quit IRC22:01
*** mattw4 has joined #openstack-infra22:02
ianwyeah, i cleared out dib_tmp and that gave back like 100g, but there are 4 bionic images22:03
openstackgerritMerged opendev/puppet-zuul master: Pin ARA to <1.0.0  https://review.opendev.org/66132922:04
ianwit's probably worth a reboot to 100% clear out mounts, etc22:04
clarkbok I'll delete the images in the down clouds and then do a reboot22:05
clarkbI'll start with stopping nodepool-builder so ti doesn't start going when I clear disk out22:05
ianwdmsimard: i was just playing with that yesterday actually; 2.8 has a feature i'd like for backups (ssh key mgmt) so it would be good if we could update bridge -> https://review.opendev.org/66266222:05
ianwbut yeah, that required 0.16.4 for ara22:05
ianwit had me stumped for a bit :)  then i realised the devel job also brings in latest ara22:06
*** rfarr has quit IRC22:18
clarkbok rebooting nb03 now22:18
*** rh-jelabarre has quit IRC22:19
clarkbnow why isn't it doing new builds22:20
clarkbshould we try manually triggering a build?22:20
dmsimardianw: oh, yeah, I like the new openssh_keypair module :D22:21
clarkboh cool it just started building a new stretch image22:22
clarkb| debian-stretch-arm64-0000013908 | debian-stretch-arm64 | nb03    | qcow2         | building | 00:00:02:21 | <- info for following up later to see if it built22:22
*** aakarsh has quit IRC22:24
openstackgerritJeremy Stanley proposed zuul/zuul master: Safely add Ansible password lookup plugin  https://review.opendev.org/66287022:26
*** my_nd has quit IRC22:28
openstackgerritMonty Taylor proposed zuul/zuul master: Read multiple *.conf files from directory  https://review.opendev.org/66097722:31
clarkbOther items on my backlog list include sending a flag day warning for https://review.opendev.org/#/c/656195/ to stop using bindep fallback everywhere but legacy jobs. Also would there be objection to an ansible mysql db backup role that we can use for the gitea servers?22:32
clarkbI probably wont' get to ^ today but if that still seems like it is probably useful I'll probably start on the backup role tomorrow22:32
clarkbinfra-root ^22:33
corvusthat seems like a fine idea22:33
*** imacdonn has joined #openstack-infra22:34
fungii'm in favor22:35
corvusclarkb: coincidentally, i just set up my first containerized mariadb backup at home; here are the commands i used in case it's any help: http://paste.openstack.org/show/752454/22:37
clarkbcorvus: thanks22:40
clarkbif I used nextcloud for more than weird phone backups I would have to set up mariadb backups at home too22:40
clarkbbut I've decided to not use the service for anything more than phone sync so just backup the data dirs instead22:40
corvushow did you guess?  that's exactly what i set it up for22:41
corvushowever, i've gone all in, and am using it for syncing music, photos, contacts, and calendar entries, with my phone and li's.  it's working great so far, and i've found it very easy to use.22:41
fungineat!22:42
fungii'm spending a few moments over here imagining what it's like to keep important data on a phone22:42
corvusi'm fully in agreement with mordred that it would be a great platform for the bod, working groups, etc, for file sharing, if there's a need.22:43
corvusfungi: for me the point is to make sure the important data make it off the phone :)22:43
fungigood call22:44
fungithe main feature i feel my current phone is missing is an easier way to wipe what little data it accumulates (text messages, call records)22:44
corvusanyway, i'm happy to answer questions anytime about my setup if anyone's curious22:45
*** hwoarang has quit IRC22:45
*** diablo_rojo has quit IRC22:46
clarkbcorvus: I've been using it for that now for about a year and a half22:46
clarkbspecifically the phone sync and backups22:46
corvusfungi: "fastboot erase userdata" will do that for an android device :)22:46
clarkbfungi: larissa takes soo many pictures of the kids now22:47
clarkband I've been giventhe task of backing them up22:47
*** hwoarang has joined #openstack-infra22:47
*** imacdonn has quit IRC22:47
clarkbcorvus: my setup is to rsync the data dirs daily into a staging area, then borg backs up the staging area offsite22:48
clarkbso I kind of have on site backups too if nextcloud/docker go sideways22:48
corvusclarkb: yeah, that's probably good enough for most things.  i figured the db backups might make recovery a little easier, but given that almost everything is files, probably not a big deal.22:48
*** aakarsh has joined #openstack-infra22:49
clarkband my remote is a raspi with 2TB disk attached to it at my brothers place. We are on the same ISP about 3 miles apart and are only one router hop apart so get full advertised bw between locations22:49
clarkbnot resistant to meteor strike but should handle house fire22:49
corvushrm, my offsite backup wouldn't handle a much larger meteor either.22:51
clarkbThe other upside to that is I can unplug the disk and bring it home for recovery22:51
*** tkajinam has joined #openstack-infra22:52
*** aaronsheffield has quit IRC22:53
*** hwoarang has quit IRC22:53
*** hwoarang has joined #openstack-infra22:55
clarkbfungi: re your gitea symlink observation should we start tracking all those little things that we think could be better?22:56
clarkbI guess we could file individual bugs in github for them against gitea22:56
clarkbfwiw the arm64 stretch build is updating the repo cache22:57
clarkbthis is not incredibly fast when you've been out to lunch for almost a month22:57
clarkbbut it is still running which is a good sign22:57
*** zbr has quit IRC23:00
ianwdmsimard: yeah; only found one bug so far :) https://github.com/ansible/ansible/pull/5729523:06
*** imacdonn has joined #openstack-infra23:08
*** eernst has joined #openstack-infra23:16
*** eernst has quit IRC23:17
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Allow download-artifact to download multiple files  https://review.opendev.org/66287623:22
*** eernst has joined #openstack-infra23:22
*** dchen has joined #openstack-infra23:22
openstackgerritJames E. Blair proposed opendev/base-jobs master: WIP: publish artifacts to tarballs.opendev.org  https://review.opendev.org/66287723:23
clarkband now the image build is into the mkfs step23:24
corvusclarkb, fungi, mordred: ^ that's a sketch; it's not complete yet but i think it shows the direction i'm heading.  i figured if we're doing this, we should use promote.23:24
corvuswell, maybe a bit more than a sketch.  call it 75% complete.23:25
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Allow download-artifact to download multiple files  https://review.opendev.org/66287623:25
corvusforgot git add ^23:25
fungiclarkb: i take back what i said about the synlinks... apparently the little icons on the far left indicate what is a regular file and what is a symlink23:27
*** eernst has quit IRC23:27
fungitook me finding the pr where they implemented it two years ago to notice they were there ;)23:27
* fungi must be going blind23:27
fungihttps://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/lookup/23:27
clarkbfungi: there doesn't seem to be a way to see the target though?23:28
clarkbfungi: could still be improved I think23:28
fungiyeah, that's a fair point23:28
fungiand it shows you the raw object content git uses to record the symlink target without any indication on the file detail view what file type it is: https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/lookup/chef_databag.py23:29
fungiso the information *is* there but i agree the ui isn't quite what it could be23:29
openstackgerritJeremy Stanley proposed zuul/zuul master: Safely add Ansible password lookup plugin  https://review.opendev.org/66287023:32
*** sthussey has quit IRC23:33
fungicorvus: so... a lot of artifacts are going to be based off tag events... do you think the same mechanisms can be leveraged in both kinds of pipelines?23:36
fungialso for branch tip artifacts, aren't these in the promote pipeline technically being built from zuul's synthetic branch tip rather than the final published branch state?23:37
fungithough i guess that's probably the same for the docker image builds in promote23:38
clarkband now it is creating a qcow2 image so I think this may succeed23:38
fungicorvus: mostly wondering how pbr's git sha inclusion will match up for these23:38
corvusfungi: second question first: yes, and i think that's "good enough" for branch-tip artifacts, but not good enough for tags, so different jobs are needed (to build and publish from the actual tag in one step).23:39
fungiwe'll no longer be able to compare pbr version metadata against branch history, i expect23:39
fungibut maybe that's an acceptable loss23:40
fungi(i rely on it with some regularity, but that could be seen as a crutch, i suppose)23:41
corvusfungi: can you be more specific?23:41
corvusi'm not sure exactly what functionality we might be losing here23:41
*** jamesmcarthur has joined #openstack-infra23:43
fungipbr freeze23:43
*** rcernin has quit IRC23:43
*** mattw4 has quit IRC23:44
*** rcernin has joined #openstack-infra23:44
corvusfungi: so installing http://tarballs.openstack.org/zuul/zuul-master.whl and running pbr freeze on it?23:44
corvushrm, that doesn't seem valid23:45
fungicorvus: yeah, some semblance of http://paste.openstack.org/show/752457/ but when installed from a wheel (so comparing to a git checkout on, say, my workstation)23:45
corvusi'm dumb -- how do you install a branch-tip wheel from zuul?23:47
fungii mean, we don't have to do continuous deployments from those wheels ourselves, just saying if someone does then pbr will claim it's been built from a commit id which is not available in the repo23:47
corvuspip install zuul-master.whl23:47
corvusERROR: zuul-master.whl is not a valid wheel filename.23:47
fungii think it expects platform information encoded, at a minimum23:48
corvusi guess "pip install zuul-master.tar.gz" will work23:48
fungiahh, yeah, or the sdist, they'll both contain the same metadata23:48
*** Adri2000 has quit IRC23:48
fungiwheel and sdist both23:48
corvusso yeah, we may get a git commit sha which is not present in the repo23:48
corvushowever, the ".dev85" part should be traceable to a commit23:49
fungilike i said, people who care about that can just pip install directly from the git state anyway23:49
*** jamesmcarthur has quit IRC23:49
fungiso not a huge loss, just worth being aware of23:49
fungilike, i don't know if pabelanger's windmill case relies on that in any way23:49
*** Adri2000 has joined #openstack-infra23:50
fungiand yes, the .dev count will match23:50
fungiunless gerrit somehow decides to apply a merge commit on a change which zuul thinks is fast-forwardable or something, which seems unlikely23:51
corvuswell, this is only for the latest one anyway, we don't permanently archive every commit  (that would be crazy! it would take up so much space! [cc clarkb] :)23:51
*** jamesmcarthur has joined #openstack-infra23:51
fungioh, yeah, if they're ephemeral i care even less23:51
corvusso this isn't suitable for the pabelanger use case23:51
fungishould we have automation signing these branch tip artifacts? i'm ambivalent on that as well23:53
corvusi don't think we do right now, and i don't propose we change that.  i probably over-thought the job there.  i could drop it, otoh, the job will work fine with that there.23:54
corvusmay be worth dropping just so it doesn't look broken.23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!