Wednesday, 2019-05-22

openstackgerritIan Wienand proposed opendev/system-config master: : use python3 for ansible
*** bobh has joined #openstack-infra00:08
*** gyee has quit IRC00:10
*** hamzy has joined #openstack-infra00:10
*** altlogbot_1 has quit IRC00:10
*** Goneri has joined #openstack-infra00:10
*** altlogbot_0 has joined #openstack-infra00:11
*** bobh has quit IRC00:18
*** mattw4 has quit IRC00:20
*** mattw4 has joined #openstack-infra00:20
*** rcernin has quit IRC00:21
*** sreejithp_ has quit IRC00:21
*** rcernin has joined #openstack-infra00:21
*** sreejithp has joined #openstack-infra00:22
*** sreejithp_ has joined #openstack-infra00:25
*** mattw4 has quit IRC00:25
*** ijw has joined #openstack-infra00:26
*** sreejithp has quit IRC00:28
*** ijw has quit IRC00:28
*** ijw has joined #openstack-infra00:29
*** michael-beaver has quit IRC00:29
*** ijw has quit IRC00:29
*** ijw has joined #openstack-infra00:29
*** armax has joined #openstack-infra00:30
*** ijw has quit IRC00:35
*** ijw_ has joined #openstack-infra00:35
*** igordc has quit IRC00:36
*** sreejithp_ has quit IRC00:37
*** sreejithp has joined #openstack-infra00:37
*** ijw_ has quit IRC00:37
*** ijw has joined #openstack-infra00:38
*** zhurong has joined #openstack-infra00:39
*** jamesdenton has joined #openstack-infra00:44
*** bobh has joined #openstack-infra00:48
openstackgerritIan Wienand proposed opendev/system-config master: Replace skip with errors=ignore
*** ijw has quit IRC00:54
*** ijw has joined #openstack-infra00:55
*** ijw has quit IRC00:58
*** ijw has joined #openstack-infra00:58
*** bobh has quit IRC01:08
*** ijw has quit IRC01:11
*** ijw has joined #openstack-infra01:11
*** ijw has joined #openstack-infra01:12
*** betherly has quit IRC01:17
*** diablo_rojo has quit IRC01:21
*** rcernin has quit IRC01:22
*** rcernin has joined #openstack-infra01:22
openstackgerritMerged opendev/system-config master: : use python3 for ansible
*** gregoryo has joined #openstack-infra01:28
*** happyhemant has quit IRC01:28
*** bobh has joined #openstack-infra01:30
*** Goneri has quit IRC01:31
*** Goneri has joined #openstack-infra01:33
*** ijw has quit IRC01:38
*** bobh has quit IRC01:42
*** apetrich has quit IRC01:57
*** rlandy has joined #openstack-infra02:06
*** sreejithp has quit IRC02:07
ianwinteresting, i don't think the mirror dns changes have applied02:13
*** rh-jelabarre has quit IRC02:13
*** rlandy has quit IRC02:15
ianwMay 22 02:02:40 adns1 named[21836]: client @0x7f2434560ee0 tsig ( transfer of '': AXFR started: TSIG tsig (serial 1555627256)02:15
ianwcurrent serial is 155839985702:15
openstackgerritIan Wienand proposed openstack-infra/ master: Put dots on end of
openstackgerritMerged openstack-infra/ master: Put dots on end of
*** dangtrinhnt has quit IRC02:21
*** dangtrinhnt has joined #openstack-infra02:22
*** dangtrinhnt has quit IRC02:23
*** ykarel|away has joined #openstack-infra02:24
ianwummm, i'm very worried dns is broken somehow02:30
clarkbianw is the adns servernot sending transfers?02:33
clarkbor could be the base.yaml split broke dns updates02:34
ianw /var/lib/bind/zones/ on adns1 looks ok02:36
clarkbis the symptom that your updates arent happening?02:39
ianwa minute ago i couldn't lookup www.opendev.org02:40
ianwbut yeah, doesn't seem to exist either02:40
clarkbI cantlookup www but resolves02:42
clarkbdo we just not have a www record?02:42
ianwhrm, ok, maybe i've just got myself worried over a non-existent record :)02:42
clarkbI confirm that mirror record seems to not exist though02:43
clarkbimplying no updates?02:43
clarkbis bind running on adns1?02:43
*** ricolin has joined #openstack-infra02:44
ianwit is, it's reporting02:44
ianwserial: 155849142402:44
ianwsigned serial: 155562725602:44
clarkboh a signing issue?02:44
ianwthe signed serial seems out of date; seems like it should be after the last serial02:45
clarkbya I think that implies we stopped signing (could be related to base.yaml sllit?)02:45
*** nicolasbock has quit IRC02:45
ianwMay 22 02:42:40 adns1 named[21836]: zone (unsigned): ixfr-from-differences: new serial (1558470372) out of range [1558470373 - 3705954019]02:45
ianwMay 22 02:42:40 adns1 named[21836]: zone (unsigned): not loaded due to errors.02:45
ianwargh, oh wait, that's acme02:47
*** jamesmcarthur has joined #openstack-infra02:50
ianwMay 22 02:51:08 adns1 named[11161]: zone (unsigned): loading from master file /var/lib/bind/zones/ failed: CNAME and other data02:51
ianwMay 22 02:51:08 adns1 named[11161]: zone (unsigned): not loaded due to errors.02:51
ianwi must have typo'd something ?02:52
clarkbdiff against the last working serial?02:53
ianwdns_master_load: ./ CNAME and other data02:53
ianwdns_master_load: ./ CNAME and other data02:53
ianwi think that's the error02:53
clarkboh right02:53
clarkbcnames must be unique02:54
*** xinranwang has joined #openstack-infra02:56
ianwi'm not sure how they're not?02:56
ianwoh, these IN CAA records i guess ...02:57
ianwthat must mean you can't have an IN CAA record to a CNAME, only an A / AAAA ?02:57
clarkbthe CAA recirds ya02:58
clarkbSo either we switvh to A/AAAA for drop the CAA records02:58
ianwif a domain name is a CNAME ... then the CA looks for record sets at the CNAME target02:59
ianwi think we point it at the 01 servers02:59
openstackgerritIan Wienand proposed openstack-infra/ master: Use A/AAAA records for CAA
ianwalso, i will look at getting named-checkzone running as a ci job, which would have caught this03:02
logan-catching up on backscroll.. glad to see that firewall fix appears to have worked clarkb. and ianw: i was also surprised not to find discussion on the journal size topic, especially now that whole disk images are becoming more common for things like bare metal deploys. i'll be interested to see how it affects the image sizes :)03:02
ianwclarkb / fungi : ^^ dns fix ... fungi I think I said to move the CAA records and was completely wrong about that, mea culpa03:02
clarkbianw +2'd03:03
clarkblogan-: ya and you cant change it without unmounting the fs apparently03:03
clarkblogan-: which means for / you are kinda stuck with what the image sets03:03
ianwyeah i feel like it has to be contiguous so it goes down, but not up03:03
*** Goneri has quit IRC03:05
clarkblogan- I think the firewall wouldve been way more obvious if ipv4 didnt magically work03:06
clarkboh well lesson learned and hopeefully bug fixed03:06
*** bhavikdbavishi has joined #openstack-infra03:08
*** factor has quit IRC03:12
*** factor has joined #openstack-infra03:12
openstackgerritMerged openstack-infra/ master: Use A/AAAA records for CAA
ianwgetting some lunch, will come back and check on all that ^03:17
*** whoami-rajat has joined #openstack-infra03:20
*** bhavikdbavishi1 has joined #openstack-infra03:31
*** bhavikdbavishi has quit IRC03:32
*** bhavikdbavishi1 is now known as bhavikdbavishi03:32
*** ramishra has joined #openstack-infra03:38
*** psachin has joined #openstack-infra03:38
*** armax has quit IRC03:43
*** armax has joined #openstack-infra03:44
*** jamesmcarthur has quit IRC03:45
*** ykarel|away is now known as ykarel03:58
*** armax has quit IRC04:01
*** sreejithp has joined #openstack-infra04:03
*** jamesmcarthur has joined #openstack-infra04:04
*** jamesmcarthur has quit IRC04:06
*** sreejithp has quit IRC04:07
*** Lucas_Gray has joined #openstack-infra04:11
*** udesale has joined #openstack-infra04:11
*** markvoelker has joined #openstack-infra04:14
*** markvoelker has quit IRC04:18
ianwzone (unsigned): journal rollforward failed: journal out of sync with zone04:39
ianwit says that happens because you manually edited a zone; i don't believe i did :/04:39
*** slaweq has joined #openstack-infra04:40
ianwremoved the jnl files, restarted bind manually and it seems up04:46
openstackgerritMerged zuul/zuul master: pass-to-parent: Fix passing multiple secrets to parent
*** ykarel has quit IRC04:53
*** ykarel has joined #openstack-infra05:11
*** Lucas_Gray has quit IRC05:18
*** dpawlik has joined #openstack-infra05:18
*** ajo has quit IRC05:27
*** portdirect has quit IRC05:27
*** nickv1985 has quit IRC05:28
*** jrosser has quit IRC05:29
*** mwhahaha has quit IRC05:30
*** adriancz has quit IRC05:31
*** rm_work has quit IRC05:31
*** dustinc has quit IRC05:31
*** fresta has quit IRC05:32
*** fresta has joined #openstack-infra05:33
*** ajo has joined #openstack-infra05:33
*** nickv1985 has joined #openstack-infra05:33
*** mwhahaha has joined #openstack-infra05:33
*** adriancz has joined #openstack-infra05:33
*** portdirect has joined #openstack-infra05:33
*** jrosser has joined #openstack-infra05:34
*** dustinc has joined #openstack-infra05:34
AJaegerconfig-core, please review and
*** rm_work has joined #openstack-infra05:34
*** quiquell has joined #openstack-infra05:37
*** lpetrut has joined #openstack-infra05:40
*** imacdonn has quit IRC05:41
*** imacdonn has joined #openstack-infra05:41
*** roman_g has quit IRC05:43
*** sarob has joined #openstack-infra05:43
*** raukadah is now known as chandankumar05:44
AJaegerzuul-jobs reviewers, config-core, please have a look at
*** jcoufal has joined #openstack-infra05:56
*** diablo_rojo has joined #openstack-infra06:01
*** kopecmartin|off is now known as kopecmartin06:06
*** markvoelker has joined #openstack-infra06:15
*** sarob has quit IRC06:17
*** dpawlik has quit IRC06:27
*** pgaxatte has joined #openstack-infra06:27
*** janki has joined #openstack-infra06:27
*** dtantsur|afk is now known as dtantsur06:28
*** e0ne has joined #openstack-infra06:29
*** liuyulong has quit IRC06:38
*** jcoufal has quit IRC06:39
openstackgerritIan Wienand proposed opendev/system-config master: letsencrypt : use date call for serial number
ianwinfra-root: ^ i might have to emergency merge that, because we're in a loop failing to authenticate the new mirror to letsencrypt (using up our failure quota).  although eventually i guess ansible_date_time will update, not sure how long it will remain cached06:46
jheskethianw: seems reasonable to me06:47
AJaegerianw: agreed, go ahead and emergency merge...06:48
ianwjhesketh: 2 hours of my life debugging through the stack why the zone wasn't reloading i won't get back :)06:49
*** markvoelker has quit IRC06:49
jheskethgood find though :-)06:49
AJaeger#thanks ianw for debugging and fixing zone update problems06:50
openstackstatusAJaeger: Added your thanks to Thanks page (
* AJaeger is optimistic that this fixes it ;)06:50
ianwAJaeger: :) ... hit up in a few hours, and we'll see :)06:51
*** diablo_rojo has quit IRC06:52
*** e0ne has quit IRC06:57
*** ralonsoh has joined #openstack-infra07:02
*** ralonsoh has quit IRC07:03
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: remove SafeLoader left-over from ZuulJSONEncoder
*** ralonsoh has joined #openstack-infra07:03
*** ginopc has joined #openstack-infra07:08
*** jcoufal has joined #openstack-infra07:09
openstackgerritIldiko Vancsa proposed opendev/system-config master: Adding new ML for the NBMP group
*** pcaruana has joined #openstack-infra07:17
*** jcoufal has quit IRC07:17
*** tesseract has joined #openstack-infra07:18
*** rpittau|afk is now known as rpittau07:19
*** tosky has joined #openstack-infra07:21
*** roman_g has joined #openstack-infra07:25
*** hwoarang has quit IRC07:29
*** sarob has joined #openstack-infra07:34
openstackgerritMerged zuul/zuul-jobs master: add-build-sshkey: remove previously authorized build-sshkey
*** hwoarang has joined #openstack-infra07:37
*** apetrich has joined #openstack-infra07:39
*** jpena|off is now known as jpena07:44
openstackgerritMerged opendev/system-config master: letsencrypt : use date call for serial number
*** bhavikdbavishi has quit IRC07:44
*** markvoelker has joined #openstack-infra07:46
*** ykarel is now known as ykarel|lunch07:52
*** xinranwang has quit IRC07:55
*** hwoarang has quit IRC07:58
*** hwoarang has joined #openstack-infra08:00
*** ociuhandu has quit IRC08:00
*** yboaron_ has joined #openstack-infra08:00
*** ociuhandu has joined #openstack-infra08:02
*** sreejithp has joined #openstack-infra08:04
*** sarob has quit IRC08:07
*** sreejithp has quit IRC08:08
*** kashyap has left #openstack-infra08:08
*** dpawlik has joined #openstack-infra08:08
*** lucasagomes has joined #openstack-infra08:10
*** e0ne has joined #openstack-infra08:12
*** pkopec has joined #openstack-infra08:15
*** pkopec has quit IRC08:16
*** pkopec has joined #openstack-infra08:17
*** markvoelker has quit IRC08:18
*** gregoryo has quit IRC08:20
*** tkajinam has quit IRC08:21
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: stage-output: fix dotfiles handling
*** ccamacho has joined #openstack-infra08:23
*** Lucas_Gray has joined #openstack-infra08:24
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: stage-output: fix dotfiles handling
*** bhavikdbavishi has joined #openstack-infra08:31
*** gfidente|afk is now known as gfidente08:33
*** ykarel|lunch is now known as ykarel08:38
*** hwoarang has quit IRC08:44
*** hwoarang has joined #openstack-infra08:46
ianwyay, it worked!
ianwi had to manually reload bind9 and clear the journal after that seq number update.  i don't know why08:47
*** panda is now known as panda|rover08:50
*** derekh has joined #openstack-infra08:51
*** priteau has joined #openstack-infra08:55
*** electrofelix has joined #openstack-infra09:00
*** Lucas_Gray has quit IRC09:04
openstackgerritMerged opendev/irc-meetings master: Update Kolla meeting chair to mgoddard
openstackgerritMerged opendev/irc-meetings master: Update neutron-ci agenda and chairs
*** Lucas_Gray has joined #openstack-infra09:10
*** rcernin has quit IRC09:10
*** markvoelker has joined #openstack-infra09:15
openstackgerritAdam Spiers proposed openstack/cookiecutter master: Fix outdated references to
*** ricolin has quit IRC09:40
openstackgerritNir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8
*** pgaxatte has quit IRC09:42
openstackgerritNir Magnezi proposed openstack/diskimage-builder master: Deprecate rhel7 in favor of rhel
*** lifeless_ is now known as lifeless09:46
*** flaper87 has quit IRC09:48
*** markvoelker has quit IRC09:49
*** bhavikdbavishi has quit IRC10:23
*** bhavikdbavishi has joined #openstack-infra10:23
*** bhavikdbavishi has quit IRC10:28
*** yboaron_ has quit IRC10:28
*** yboaron_ has joined #openstack-infra10:29
*** diablo_rojo has joined #openstack-infra10:36
fungiianw: sorry for not catching that! that was probably the reason i proposed the caa for graphite01 instead of graphite originally, but yes i should have remembered that when you asked me to change it10:40
fungiand also if we tried restarting bind on the adns in the job, that should fail in such situations (cname and other data would cause it to refuse to load a required zone in that case)10:41
*** bhavikdbavishi has joined #openstack-infra10:43
*** markvoelker has joined #openstack-infra10:46
*** diablo_rojo has quit IRC10:56
*** toabctl has joined #openstack-infra10:58
*** nicolasbock has joined #openstack-infra11:06
*** ykarel is now known as ykarel|afk11:12
*** panda|rover is now known as panda|rover|eat11:16
*** markvoelker has quit IRC11:18
openstackgerritMark Meyer proposed zuul/zuul master: Upgrade formatting of the patch series.
*** pkopec has quit IRC11:29
*** pkopec has joined #openstack-infra11:30
*** udesale has quit IRC11:30
*** sreejithp has joined #openstack-infra11:30
*** udesale has joined #openstack-infra11:31
*** jpena is now known as jpena|lunch11:31
*** ginopc has quit IRC11:32
*** sshnaidm|afk is now known as sshnaidm11:35
*** sreejithp has quit IRC11:35
*** rfolco is now known as rfolco|dentist11:39
*** iurygregory_ has joined #openstack-infra11:39
*** iurygregory has quit IRC11:40
*** _erlon_ has joined #openstack-infra11:45
*** bhavikdbavishi has quit IRC11:49
*** bhavikdbavishi has joined #openstack-infra11:51
*** bhavikdbavishi has quit IRC11:56
*** rlandy has joined #openstack-infra11:57
*** apetrich has quit IRC12:01
*** rh-jelabarre has joined #openstack-infra12:01
*** panda|rover|eat is now known as panda|rover12:06
*** eharney has quit IRC12:10
*** bobh has joined #openstack-infra12:10
*** apetrich has joined #openstack-infra12:14
*** markvoelker has joined #openstack-infra12:16
*** ykarel|afk is now known as ykarel12:18
*** apetrich has quit IRC12:19
*** bobh has quit IRC12:19
*** rfarr__ has joined #openstack-infra12:21
*** hwoarang has quit IRC12:23
*** hwoarang has joined #openstack-infra12:25
*** rfarr_ has joined #openstack-infra12:27
*** psachin has quit IRC12:27
*** iurygregory_ is now known as iurygregory12:28
*** markvoelker has quit IRC12:29
*** rfarr__ has quit IRC12:29
*** jpena|lunch is now known as jpena12:33
*** persia has quit IRC12:34
*** apetrich has joined #openstack-infra12:35
*** persia has joined #openstack-infra12:35
*** janki has quit IRC12:35
openstackgerritAdam Coldrick proposed opendev/storyboard master: Install compatible version of python-openid when using Python 2
*** bhavikdbavishi has joined #openstack-infra12:52
*** jamesmcarthur has joined #openstack-infra12:54
*** ricolin has joined #openstack-infra12:55
*** zul has joined #openstack-infra13:02
*** igordc has joined #openstack-infra13:06
*** priteau has quit IRC13:09
*** bhavikdbavishi has quit IRC13:10
*** adriancz has quit IRC13:10
openstackgerritMerged opendev/storyboard master: Install compatible version of python-openid when using Python 2
*** mriedem has joined #openstack-infra13:14
*** lseki has joined #openstack-infra13:21
*** pkopec has quit IRC13:23
*** aaronsheffield has joined #openstack-infra13:25
*** ginopc has joined #openstack-infra13:28
*** rfolco|dentist is now known as rfolco13:29
*** Goneri has joined #openstack-infra13:29
*** dtantsur is now known as dtantsur|brb13:29
*** sreejithp has joined #openstack-infra13:33
*** eharney has joined #openstack-infra13:36
*** sreejithp has quit IRC13:37
*** sreejithp has joined #openstack-infra13:38
*** hwoarang has quit IRC13:45
*** liuyulong has joined #openstack-infra13:46
*** hwoarang has joined #openstack-infra13:46
*** kaiokmo has joined #openstack-infra13:49
*** igordc has quit IRC13:51
*** sarob has joined #openstack-infra13:53
zigofungi: Where may I see the logs of the Buster build in the infra?13:55
*** bobh has joined #openstack-infra13:55
zigo(thanks for the +2w btw)13:55
*** rfarr_ has quit IRC13:57
fungijust a sec, finding it13:59
fungi2019-05-21 23:43:56.486 | Build completed successfully14:00
funginow... hopefully it's right14:00
mordredfungi: who needs right when you have right now?14:00
fungizigo: also you can fetch if you want to inspect it, though be forewarned it's 9.1gib14:01
fungi(mostly thanks to the git caches we include)14:02
zigofungi: Great, thanks so much! :)14:03
zigoI don't really mind 9GB from here in the office... :P14:03
*** ginopc has quit IRC14:04
fungiyou can also find checksums in the same directory if you want them14:08
zigofungi: Is the image safe to boot in my own cloud?14:09
zigoThere's no cloud-init, right?14:09
fungiit uses glean, which will expect you to provide metadata via configdrive14:09
fungithough for network access it can rely on dhcp and/or ipv6 ra14:10
zigoRight, so I just need to use the --config-drive param.14:10
openstackgerritAdam Spiers proposed openstack/cookiecutter master: Fix outdated references to
*** quiquell has quit IRC14:12
fungiand ssh keys you provide should automatically be authorized for the root user, i believe14:12
*** lpetrut has quit IRC14:12
*** ginopc has joined #openstack-infra14:14
openstackgerritAdam Spiers proposed openstack/cookiecutter master: Add a testenv for lower-constraints
*** jamesmcarthur has quit IRC14:16
openstackgerritMatt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects
*** jamesmcarthur has joined #openstack-infra14:18
*** jamesmcarthur has quit IRC14:20
*** quiquell has joined #openstack-infra14:21
*** roman_g has quit IRC14:22
mriedemi noticed the compute api-ref was last published on march 26
mriedemis that when the opendev migration happened?14:25
*** sarob has quit IRC14:27
zigofungi: Would you use glean as a package if I do that?14:28
*** dpawlik has quit IRC14:29
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver -
*** dtantsur|brb is now known as dtantsur14:29
*** bobh has quit IRC14:30
*** ykarel is now known as ykarel|afk14:31
*** jamesmcarthur has joined #openstack-infra14:41
*** armax has joined #openstack-infra14:41
*** jamesmcarthur has quit IRC14:41
*** jamesmcarthur has joined #openstack-infra14:42
*** udesale has quit IRC14:44
*** udesale has joined #openstack-infra14:44
*** mriedem is now known as hansmoleman14:44
*** tosky has quit IRC14:45
*** sarob has joined #openstack-infra14:46
*** tosky has joined #openstack-infra14:46
*** quiquell has quit IRC14:53
*** pkopec has joined #openstack-infra14:54
*** pkopec is now known as pkopec|afk14:54
fungizigo: probably not since we want to be able to continuously deploy it from source. i guess we could autobuild debian packages for it on each commit, but would probably want to do the same with rpms for centos/suse, emerge for gentoo, et cetera14:56
fungiand we want a consistent version installed on distros of various vintages14:56
*** lpetrut has joined #openstack-infra14:57
fungibasically because we sometimes find that to bring a new provider online we have to make adjustments to glean or add features to support unique environments14:57
fungiand so we need to be able to add that support to every image for every distro/version we build14:57
fungiusing python packages for it at least gives us a single format which works on a variety of distros of different ages14:58
*** roman_g has joined #openstack-infra14:59
*** hansmoleman is now known as mriedem15:00
*** ginopc has quit IRC15:05
*** ykarel|afk is now known as ykarel15:06
*** rfarr__ has joined #openstack-infra15:07
*** jtomasek has joined #openstack-infra15:11
*** roman_g has quit IRC15:12
logan-2nd job in ovh-bhs i have seen in the last 24h that hung for a long time trying to clone from
*** roman_g has joined #openstack-infra15:14
openstackgerritSean McGinnis proposed openstack/openstack-zuul-jobs master: Fix typo in tempest-dsvm-full-hdfs
zigofungi: <--- Packaged anyway! :)15:16
zigoI may try it ...15:16
fungizigo: yeah, feel free to package it for debian, i just don't think the opendev infrastructure would likely be able to take advantage of that is all15:19
fungii need to step out for lunch and some errands, but expect to be back in plenty of time for the storyboard meeting15:19
corvusi'm back at home and resettling15:20
*** hwoarang has quit IRC15:20
*** diablo_rojo has joined #openstack-infra15:20
*** hwoarang has joined #openstack-infra15:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate builds with event id
*** ginopc has joined #openstack-infra15:26
*** dciabrin_ is now known as dciabrin15:28
*** eharney has quit IRC15:28
*** gyee has joined #openstack-infra15:32
*** ginux has joined #openstack-infra15:35
*** ginopc has quit IRC15:35
*** ginux is now known as ginopc15:35
*** nhicher has quit IRC15:35
mordredcorvus: yay!15:36
mordredcorvus: you'll be happy to hear that we figured out the gitea 1.8.0 behavior change and have fixed it15:37
mordredcorvus: so is now ready to land15:37
*** lpetrut has quit IRC15:38
mordredcorvus: (that also led to a PR to gitea docs which has been landed)15:39
*** ginopc has quit IRC15:39
*** david-lyle has joined #openstack-infra15:39
*** dklyle has quit IRC15:40
corvusmordred: oh awesome!15:40
*** david-lyle is now known as dklyle15:41
corvusmordred: related: my gitea patch for refs has landed upstream:
*** lucasagomes has quit IRC15:43
corvusmordred: so maybe we should land the 1.8.0 patch, then maybe look into running master?15:43
*** lucasagomes has joined #openstack-infra15:43
*** lucasagomes has quit IRC15:43
*** lmiccini has quit IRC15:43
*** lucasagomes has joined #openstack-infra15:44
mordredcorvus: yes to 1.8.0 ... master seems maybe scary - but also maybe not super horrible since we wouldn't build new images if they don't pass the docker-compose test15:44
mordredit's possible template updates could cause our UI to regress unexpectedly15:44
*** lmiccini has joined #openstack-infra15:44
mordredbut that probably won't happen *frequently*15:44
openstackgerritSean McGinnis proposed openstack/openstack-zuul-jobs master: Fix typo in tempest-dsvm-full-hdfs
corvusmordred: or we could just pull the refs patch into our 1.8.0 branch15:45
corvusmordred: mostly, i didn't want to do something that would interact badly with upstream, but since they accepted it, i think adding it to our local fork until 1.9.0 would be fine15:45
mordredcorvus: yeah. or even pull master into our 1.8.0 branch as a one-time sync and then move back to tags when they make them15:46
mordredbut yeah - I'd love to go ahead and start running your refs patch so we can start on replicating refs15:46
mordredcorvus: we could see if the gitea folks would be interested in zuul running system-config-run-gitea on gitea PRs :)15:47
corvusok i'll put that on the list; meanwhile, 655522+315:47
mordredalong with running master15:47
*** kjackal has joined #openstack-infra15:47
corvuser, i'll put "refs patch" on the list for later this week15:48
corvusspeaking of talking to the gitea folks, i haven't gotten a response to my email to lunny15:48
*** sarob has quit IRC15:49
corvusmay have to try wechat or something15:49
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Fix confusing statement in tempest-dsvm-full-hdfs
mordredoh - just had a thought - should we redo the metadata for the Zuul app on github to call it "OpenDev Zuul" ?15:50
*** ramishra has quit IRC15:51
mordredI'm going to do that ... although I think we might also want to create a new app, since the URL for people to use to add the app to their things is "" - but we can handle that later15:51
mordredcorvus: we can upload a logo - should we upload the zuul logo or the opendev logo? I think maybe the opendev logo15:54
*** dklyle has quit IRC15:55
*** david-lyle has joined #openstack-infra15:55
*** iurygregory has quit IRC15:55
*** Lucas_Gray has quit IRC15:55
corvusmordred: ++opendev15:55
mordredcorvus: oh! that makes the url ... neat15:56
corvusthat looks lovely15:58
*** derekh has quit IRC15:59
openstackgerritMonty Taylor proposed opendev/system-config master: Update references to OpenDev Zuul
*** ykarel is now known as ykarel|away16:01
*** jpena is now known as jpena|off16:02
mordredinfra-root: it still shows "openstack-infra" as the owner of the app - should we maybe make a new gh org, "opendev-zuul", and transfer the App into that org? (we can't make an opendev one because there's that dude whose name is opendev)16:04
corvusmordred: or "opendev-org" or "opendevorg" (ala dockerhub?)16:05
*** jtomasek has quit IRC16:05
mordredah - yeah  - that's smarter16:05
*** hwoarang has quit IRC16:07
*** hwoarang has joined #openstack-infra16:09
*** tesseract has quit IRC16:10
*** jbadiapa has quit IRC16:10
*** igordc has joined #openstack-infra16:12
*** tesseract has joined #openstack-infra16:14
*** tesseract has quit IRC16:14
*** ykarel|away has quit IRC16:15
*** dklyle has joined #openstack-infra16:15
*** david-lyle has quit IRC16:15
openstackgerritMonty Taylor proposed opendev/system-config master: Update references to OpenDev Zuul
mordredcorvus: done and updated docs ^^16:16
mordred(I had already created the opendevorg org apparently. I have now added openstackadmin to it as owner)16:17
*** rpittau is now known as rpittau|afk16:18
mordredinfra-root: while I'm doing github things - should we rename the openstackadmin account to opendevadmin?16:19
corvusmordred: it seems like we're probably going to have less and less openstacky stuff over time, so i lean toward yes, but we probably want more input on that16:20
*** jpena|off is now known as jpena16:22
AJaegermriedem: the index.rst of was last updated on 26th of March - but published on the 7th of May the last time (view page info). Still a bit long ago, let's check logs...16:22
AJaegermriedem: see for last run16:23
*** jbadiapa has joined #openstack-infra16:24
*** david-lyle has joined #openstack-infra16:24
AJaegermriedem: I think and will fix it - need config-core review...16:24
*** kopecmartin is now known as kopecmartin|off16:25
*** dklyle has quit IRC16:25
mordredcorvus: agree16:26
mriedemAJaeger: ok thanks16:26
mordredcorvus: also - I could see the openstack project wanting an account like openstackadmin to manage replication things and whatnot16:27
clarkbcorvus: we also think we fixedthe limestone docker problems16:33
clarkbfirewall neededport 5001 open toi16:33
* clarkb goes back to sitting on river16:33
mordredclarkb: before you go sit on the river - any opinions on renaming openstackadmin to opendevadmin in github?16:34
clarkbcan you rename an account? or would be making a new one?16:35
clarkbmaking a new one seems fine just dont forget to make it owner on all the things16:35
corvusclarkb: huh?  i thought we did that?16:36
corvusclarkb: great news! :)16:37
mordredclarkb: you can rename it16:37
corvus(how in the world did we all miss that the first time :)16:38
*** udesale has quit IRC16:38
mordredclarkb: and since it doesn't have repos directly under it, as I read it it should have no impact16:38
*** mattw4 has joined #openstack-infra16:39
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver -
*** david-lyle has quit IRC16:40
*** udesale has joined #openstack-infra16:43
prometheanfireiirc logs are not kept forever, is there a way I can save this without just coppying the text?16:44
prometheanfirewant to submit it to fasteners upstream16:45
prometheanfireI meant still hosted and all16:46
prometheanfirebrowsable, linkable16:46
*** dklyle has joined #openstack-infra16:48
*** Lucas_Gray has joined #openstack-infra16:48
*** e0ne has quit IRC16:51
*** diablo_rojo has quit IRC16:51
ricolinHI all, I'm facing a issue when I try to add a test job to self-healing-sig repo. For some reason, I can't find new directory which I added it that patch   Anyone can guide me on what I doing wrong here?
*** Swami has joined #openstack-infra16:53
mordredricolin: I can't see anything, but then I've paged most of devstack-gate out of my brain ... AJaeger, do you still know how that works enough to spot an error?16:55
mordredprometheanfire: no, we don't have any mechanism for that at the moment, although it's an interesting use case16:56
openstackgerritMerged zuul/zuul master: Install latest git-review from PyPI in quickstart
openstackgerritMerged opendev/system-config master: Use local fork of gitea and upgrade to 1.8.0
prometheanfiremordred: yarp, I did make in the mean time17:00
openstackgerritMerged zuul/zuul-jobs master: Delete files in dest that don't exist
*** bobh has joined #openstack-infra17:06
*** hjensas has joined #openstack-infra17:06
notmynamessl for down for everyone or just me?17:06
* hjensas has ssl issues with as well.17:07
bnemecnotmyname: I'm also here wondering about that17:07
notmynameboth git and a browser aren't working for me17:08
bnemecIt was working for me 10 minutes ago, FWIW.17:10
*** electrofelix has quit IRC17:10
pabelangerjust merged, 11mins ago17:10
pabelangermaybe related17:10
pabelangerinfra-root: ^ seems down17:10
corvusneat, 522 seems very likely to be related17:11
corvusadb916bc6863        opendevorg/gitea:latest    "/usr/bin/entrypoint…"   11 minutes ago      Restarting (1) 54 seconds ago                       giteadocker_gitea-web_117:11
*** udesale has quit IRC17:12
corvus2019/05/22 17:11:40 ...s/setting/setting.go:738:NewContext() [F] error saving generating JWT secret to custom config: open /custom/conf/app.ini: permission denied17:12
*** Lucas_Gray has quit IRC17:12
corvusthat's very strange since that patch should have taken care of that17:12
mordredthat's the thing we fixed in 52217:12
*** ricolin has quit IRC17:13
corvusi do not see the oauth2 thing in the app.ini on disk17:13
mordredI don't either17:13
corvusi think we should manually edit that for now, then figure out what went wrong17:14
mordredI agree17:14
*** chandankumar is now known as raukadah17:14
mordredwanna divy up hosts?17:14
corvusmordred: sure, i'll take 1-4, you take 5,7,817:14
corvus6 is out17:14
*** bhavikdbavishi has joined #openstack-infra17:15
mordredcorvus: 5-8 done17:16
corvusmordred: once edited, the auto restart should take care of things17:16
mordrednow - why did this break?17:17
*** jpena is now known as jpena|off17:17
mordredcorvus: I'm still getting site down17:17
corvusthere we go17:18
mordredam I just too impatient?17:18
corvusmordred: gitea01 still took 3 minutes to start up... i'm not sure what it's doing here:17:19
corvus2019/05/22 17:15:46 models/repo.go:136:NewRepoContext() [I] Git Version: 2.20.117:19
corvus2019/05/22 17:18:20 routers/init.go:37:checkRunMode() [I] Run Mode: Production17:19
corvusmaybe we should look into that sometime17:19
mordredme either17:19
corvusbut i think we're good now17:19
corvusbnemec, notmyname, hjensas: ^ thanks!17:19
notmynameI'm still getting an error with `git fetch orgiin`17:19
notmyname$ git fetch origin17:19
notmynamefatal: unable to access '': LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to
hjensasI was able to clone a repo, but then when 'git remote update' I get: fatal: unable to access '': OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to
*** bobh has quit IRC17:20
hjensasoh, on retry it succeded.17:20
notmynamejust started working here too17:21
mordredcorvus: it's a race that will be fixed by running playbooks with zuul17:21
corvusmy guess is the load balancer was putting backends back in too early, since they were accepting connections but not responding...17:21
mordredcorvus: system-config on bridge is not updated yet17:21
mordredbut the docker images were published17:21
mordredso the cron-job ansible run told docker compose to pull and restart - but the ansible wasn't ready for it yet17:22
corvusmordred: ah!  this is a tricky time for us.17:22
openstackgerritMerged openstack/cookiecutter master: Fix outdated references to
mordredcorvus: yah17:22
corvusmordred: in some cases (like this) we could make config/image changes separately.  obviously that's fragile, but could be something to keep in mind until we finish the transition17:23
*** lucasagomes has quit IRC17:23
mordredyeah. we could also maybe go ahead and start running the services that are split out from zuul and remove them from teh cron, now that we've landed the first split-out patch17:24
corvusyeah, i think we need the python version of gitea project creation for that, right?17:25
*** kskels has joined #openstack-infra17:25
*** bdodd has quit IRC17:25
corvusor, at least, that will allow us to do the most sensible and least hacky version of that.17:25
mordredoh - yeah17:25
kskelshi clarkb an infra team - is there a way to re-trigger/recheck zuul post job? or would it be waiting on another merge coming in?17:26
mordredcorvus: I was going to work on that after upgrading status.o.o off of trusty17:26
*** jtomasek has joined #openstack-infra17:27
corvuskskels: if it's not too much of a burden to wait for another commit, that would be best.  but if it's important and there isn't going to be another merge soon, we can retrigger (it's a manual process)17:27
kskelsit would likely be a wait.. if you can re-trigger without too much work then I can test and see if adding "write" permissions to gerrit user worked.. then I could move on to apply the same for all other like 14 airship projects..
kskelsif it's too much trouble.. I'll wait. Thank you!!17:29
kskelsis the PS17:30
corvuskskels: oh, i'm sorry i thought maybe there was an infrastructure problem or something... we're trying to get out of the business of being responsible for github replication, so retriggering jobs for that doesn't seem like something we should be doing17:32
corvus(if we were going to be responsible for github replication, it would be better for us to have just kept the old system with gerrit)17:32
kskelsno worries, thank you!17:33
*** eharney has joined #openstack-infra17:33
*** kskels has left #openstack-infra17:34
*** Swami has quit IRC17:35
*** samueldmq has joined #openstack-infra17:50
*** bobh has joined #openstack-infra17:57
*** yboaron_ has quit IRC18:00
*** yboaron_ has joined #openstack-infra18:01
*** nhicher has joined #openstack-infra18:04
dtroyerWe're seeing an error at via browser but not git clone, etc:  "template: repo/view_list:82:38: executing "repo/view_list" at <$commit.Summary>: invalid value; expected string"    I have not found this in any other repos so far…still checking18:10
*** Goneri has quit IRC18:11
*** bdodd has joined #openstack-infra18:11
corvusdtroyer: fascinating  mordred ^18:11
corvus(i'm digging in to it)18:12
mordredI agree - that does not look good18:12
corvusdtroyer: any chance you can confirm whether this worked before, let's say, 2 hours ago? :)18:13
dtroyerI've asked in #starlingx, not heard back yet...18:13
corvusview_list.tmpl does not seem to be one of the ones we have modified locally18:13
corvusi don't see anything weird about the first line of the 50 most recent commit messages in that repo18:16
dtroyercorvus: it was first reported about 20 minutes ago…we're not sure when it was last known to work...18:16
*** jtomasek has quit IRC18:23
*** jtomasek_ has joined #openstack-infra18:23
corvusthe logs don't seem to have any info other than there was a 500 error for that url18:25
logan-would someone mind enqueueing in gate please? it got stuck for over an hour earlier in the gate trying to clone from, and eventually timed out, and then i was waiting for a recheck to go through the check queue and one of the tests has been waiting  in check for a retry node for over an hour because of the issues a bit ago.18:26
*** bobh has quit IRC18:28
corvusdtroyer, mordred: i can reproduce that with a local gitea, will try to find more info18:28
mordredcorvus: oh good!18:29
*** tosky has quit IRC18:30
*** tosky has joined #openstack-infra18:30
openstackgerritMatt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects
*** gfidente is now known as gfidente|afk18:35
corvusdtroyer, mordred: i'm rebuilding gitea with more debug info and will pick this back up after lunch18:36
dtroyercorvus: ok, thank you18:36
*** dpawlik has joined #openstack-infra18:36
*** jamesmcarthur has quit IRC18:38
*** bhavikdbavishi has quit IRC18:41
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver -
*** slaweq has quit IRC18:44
*** sshnaidm is now known as sshnaidm|afk18:45
openstackgerritTobias Henkel proposed zuul/zuul master: Log github requests with annotated events
fungidtroyer: corvus: mordred: that looks similar to this ref which turned up broken in a release audit but for which i wasn't able to find a cause:
fungiit's the tag ref corresponding to the most recent successful build at
fungimaybe the cause for both is related18:58
fungigranted it's a different function and a different variable, but a similar resulting error18:59
*** smarcet has joined #openstack-infra19:01
mordredfungi: when I look at that page in gitea - it shows: tags/1.5.1^019:01
mordredwhich makes me think maybe a null got appended to the tag?19:01
*** ramishra has joined #openstack-infra19:02
mordredoh - hrm. maybe that's just a gitea display weirdness - shows up here too:19:02
openstackgerritAdam Coldrick proposed opendev/system-config master: Update key for SotK
openstackgerritTobias Henkel proposed zuul/zuul master: Log github requests with annotated events
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate logs around build completion and cancellation
*** raissa has joined #openstack-infra19:12
openstackgerritMerged opendev/system-config master: Update references to OpenDev Zuul
*** e0ne has joined #openstack-infra19:15
*** EmilienM is now known as EmilienM|bbl19:15
*** EmilienM|bbl is now known as EmilienM19:16
*** bobh has joined #openstack-infra19:16
*** sarob has joined #openstack-infra19:17
*** flaper87 has joined #openstack-infra19:21
fungiany other infra-root available for a quick ssh key change in ?19:21
Shrewsfungi: +3d19:22
*** smarcet has quit IRC19:23
*** jtomasek_ has quit IRC19:24
*** smarcet has joined #openstack-infra19:25
*** jtomasek has joined #openstack-infra19:25
*** quiquell has joined #openstack-infra19:26
*** quiquell has quit IRC19:26
*** slaweq has joined #openstack-infra19:27
*** ramishra has quit IRC19:27
openstackgerritDouglas Mendizábal proposed openstack/project-config master: End Project Gating
*** sarob has quit IRC19:32
*** smarcet has quit IRC19:33
*** Goneri has joined #openstack-infra19:37
*** jamesmcarthur has joined #openstack-infra19:39
*** jtomasek has quit IRC19:42
*** jtomasek_ has joined #openstack-infra19:43
*** smarcet has joined #openstack-infra19:47
AJaegermordred: ricolin is touching legacy devstack-gate hooks - I swapped those out as well but will quickly look. Rewriting from scratch is my recommendation ;)19:50
AJaegerconfig-core, the api jobs are broken due to sphinx/python 3, I think and will fix it - could you review, please?19:51
*** jcoufal has joined #openstack-infra20:00
*** jcoufal has quit IRC20:00
*** jcoufal has joined #openstack-infra20:01
*** jtomasek_ has quit IRC20:02
rpiosoI'm proposing new entries in driverlog --
*** jtomasek has joined #openstack-infra20:03
rpiosoHow can I determine the Ironic CI's username/ID, success pattern, and failure pattern?20:04
openstackgerritDouglas Mendizábal proposed openstack/project-config master: End Project Gating
*** ralonsoh has quit IRC20:07
*** liuyulong has quit IRC20:11
fungirpioso: are you the one running the "ironic ci" you describe? the username would be whatever the username is for the gerrit account it's using to leave comments20:15
fungior so i assume. we don't manage driverlog (i think the mirantis folks running stackalytics do?)20:15
logan- just failed with RETRY_LIMIT on a job that has been retrying for hours (see my message above). i was able to catch the console on the last one while it was running and it seems like the node just died in the middle of it
*** bobh has quit IRC20:16
fungilooks like it was in the middle of installing python packages from remote git sources into containers?20:18
logan-not installing anything, just cloning repos there20:18
logan-i just rechecked it again, but it seems like maybe something is still not quite right with the opendev git20:19
corvuslogan-: a problem with the opendev git servers would not cause a zuul worker node to go offline20:20
*** pcaruana has quit IRC20:20
*** michael-beaver has joined #openstack-infra20:21
corvuslogan-: i don't suppose you have the top of the build log handy?20:21
logan-corvus: sadly no :( i accidentally clicked a link in there and the back button does not restore it :(20:22
corvusit may be useful for someone to check the logs on ze07 and see which node provider that was in20:22
logan-rax-ord or rax-iad I cannot remember which20:23
logan-related question though, does nodepool schedule the retries to the same cloud?20:23
logan-that IP traceroutes to rax-ord btw20:24
corvuslogan-: generally no (unless it depends on a paused job; that's the only time zuul expresses a provider preference to nodepool)20:25
*** pkopec|afk has quit IRC20:25
logan-the executor chart on looks weird20:25
logan-nodepool clouds like show a similar drop to almost 0 used nodes around the same time20:26
corvusthat sounds like a zookeeper connection drop, which usually means a memory problem20:27
corvusthe scheduler was last restarted on may 16 at commit ee4b6b120:28
openstackgerritDirk Mueller proposed openstack/openstack-zuul-jobs master: Switch to using TOX_CONSTRAINTS_FILE
corvusinfra-root: is any other infra-root around to deal with zuul while i continue to deal with gitea?20:28
openstackgerritMatt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects
Shrewscorvus: just a scheduler restart? i can deal with that20:30
Shrewslemme check the mem use first20:30
corvusShrews: thx20:30
openstackgerritDirk Mueller proposed zuul/zuul-jobs master: Switch to TOX_CONSTRAINTS_FILE
Shrewshim, seems likely:
openstackgerritMatt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects
Shrewscorvus: any need to wait on anything after stopping the scheduler? or is init.d/zuul-scheduler restart enough?20:34
*** whoami-rajat has quit IRC20:34
*** e0ne has quit IRC20:34
corvusShrews: nope, but there's a bug in pid handling for zuul web, so you'll need to manually remove its pidfile and restart it20:35
corvus(and its also a bug that it needs to be restarted)20:35
*** jcoufal has quit IRC20:36
* mordred has eaten lunch and is back ... anything I need to jump in on?20:36
Shrewshim, zuul-scheduler does not seem to want to stop20:38
*** mattw4 has quit IRC20:39
*** e0ne has joined #openstack-infra20:40
*** mattw4 has joined #openstack-infra20:40
rpiosofungi: No, I don’t run it. However, I am a beneficiary of it.20:40
*** sarob has joined #openstack-infra20:40
fungiyeah, sorry, stepped away for a moment but back now20:40
fungiso we think there was a zk connection drop in addition to whatever is wrong with that gitea git ref?20:41
Shrewsok, there it goes20:41
*** e0ne has quit IRC20:41
logan-appears to be a zuul scheduler memory leak fungi
fungioh, huh yep that's a dramatic increase today20:42
*** smarcet has quit IRC20:43
Shrewsfungi: yeah, restarted scheduler and web20:43
fungiwe've definitely seem memory pressure cause sudden zk connectivity issues20:43
Shrewsfungi: anything else too restart?20:43
fungii think that'll be all you need to restart20:44
Shrewsi guess the fingergw is ok20:44
fungiand looks like this was the first restart in ~6 days? so interesting that memory use just skyrocketed in the last 24 hours20:44
pabelangerShrews: yah, usually save queues, restart, then reenqueue gate, then check20:44
Shrewspabelanger: ah, knew i forgot something (the queues)20:45
pabelangerHmm, I think they are still on disk20:45
pabelangerwe have a crontab still?20:45
fungiyeah, there is a periodic dump to /var i think, looking20:45
pabelangerShrews: gives some good info too20:47
pabelangerbut we don't seem to have docs about crontab status that fungi linked20:47
fungitimestamp on /var/run/zuul/ is 20:41z so we want the last dump before that20:47
fungiwhich i think is either /var/lib/zuul/backup/openstack-zuul-tenant_status_1558557601.json or /var/lib/zuul/backup/openstack-zuul-tenant_status_1558557661.json20:48
funginope. earlier20:49
fungi/var/lib/zuul/backup/openstack-zuul-tenant_status_1558557361.json is the last one with a nontrivial file size20:49
Shrewsfungi: how do we use that?20:50
fungigreat question... shell history says `python file:///var/lib/zuul/backup/openstack-zuul-tenant_status_1558557361.json openstack gate >` et cetera20:51
fungis/gate/check/g to export the check pipeline similarly20:51
*** dpawlik has quit IRC20:51
fungithose should give you and scripts you can run to reenqueue whatever was running at the time it was stopped (or moments before anyway)20:52
Shrewsthat script does not appear to be a thing20:52
Shrewsat least not in /opt/zuul/tools20:53
fungii think it'll be in /opt/zuul/tools/20:53
fungipossible that /opt/zuul/tools/ can handle file:// urls now20:54
fungi(or always could?)20:54
pabelangeryah, always could20:55
Shrewshrm, don't think it handles files20:58
corvusi think there's a modified version of that in my homedir20:59
fungiurllib2.URLError: <urlopen error [Errno 20] Not a directory: '/var/lib/zuul/backup/openstack-zuul-tenant_status_1558557361.json/api/info'>21:00
fungiyeah, seems the vanilla one indeed does not21:00
corvusdmsimard has a year-wip patch to support files
fungithe reason for the -1 there looks incorrect, at least without some alteration to the script21:02
corvusi don't know what needs to be done, but the end goal should be that "file://" is supported21:03
Shrewsthat seems to work21:03
dmsimardit was the distinction between --file and --url in the patch that was problematic because really file:// is an url21:03
dmsimardso we could do away with the args altogether and always use urls21:03
corvusyes that21:03
fungigot it21:04
Shrewsfungi: using that, i have and scripts from openstack-zuul-tenant_status_1558557361.json21:04
Shrewsre-enqueuing using those now...21:05
*** smarcet has joined #openstack-infra21:11
corvusdtroyer, mordred: re integ and gitea -- i have more information: when gitea is displaying the commit message summary of each file in the root of the repo directory, it collects a list of git tree entries and associated commits.  for some of the entries in the root of integ, the commit object is nil, and that is not supposed to happen.  i do not (yet) know why that is the case.  i have also found21:11
corvusthat our gitea is perfectly happy to display the commit information for the troublesome objects in other contexts (for instance, "doc/" is one of the problematic entries on the first page, but works fine).  i have also found that while i can reproduce this with a local build of our gitea image, i can not reproduce it with which21:11
corvusis running upstream master:
dtroyercorvus: do we need to do some repo surgery to clean it up?21:12
corvusdtroyer: i have no information at this point to suggest that there is a problem with the repo itself21:13
dtroyercoreycb: ok, thanks21:13
corvusdtroyer, mordred: given all that, i am somewhat inclined to build an opendev gitea image based on master and see if it works, and if it does, upgrade opendev to that (since that is the most forward looking thing).  alternatively, i can attempt to find the difference between our version of 1.8.0 and upstream master (perhaps there is a fix upstream for this bug, or, perhaps one of our local changes21:14
corvuson 1.8.0 has broken it)21:14
mordredcorvus: I am in favor of the idea of the gitea image based on master since we were talking about doing that earlier today anyway21:14
corvusdtroyer: this is where it would be really helpful to know if it was working before today (as it would eliminate any changes between the last two local builds)21:14
dtroyercorvus: I know it was OK on Friday, I'll see if anyone can confirm for this morning21:15
corvusdtroyer: i think that may be enough21:15
corvusdtroyer: if it worked friday and not today, then the breaking change is between opendev-1.7.6 and opendev-1.8.0, so it does lend credence to the idea that something upstream broke and there is a fix in master21:16
* mordred agrees with corvus21:17
corvusfound it21:17
fungiaha, so it's runaway template parsing on commit message content?21:18
Shrewscheck and gate jobs re-enqueued21:20
Shrewsnot sure if we lost any21:20
fungiShrews: close enough, i'm sure21:20
fungithe error on is also from "template" but has existed since migration, leading me to wonder if there's a similar bug in repo/diff/box21:21
dtroyerShrews: FWIW the two I was watching have re-appeared21:21
fungimaybe for longer21:21
Shrewsdtroyer: awesome21:21
*** smarcet has quit IRC21:23
corvusthat commit alone is insufficient to fix things, so i'm trying a build of upstream master now21:24
mordredI still don't see any errors on that page21:24
mordredfungi: ^^ that link totally works fine for me with no errors21:24
*** jamesmcarthur has quit IRC21:26
fungimordred: wow! so it does. i still had this response cached in a browser tab after ttx pointed it out before the summit... template: repo/diff/box:101:80: executing "repo/diff/box" at <$.BeforeSourcePath>: invalid value; expected string21:27
fungiand force refreshing that tab continues to return that error21:27
fungibut opening it in a browser on another computer loads fine...21:28
fungiplugging that url into a new tab also loads fine21:28
fungifollowing the "cc9bbec" link in the change column for the most recent successful build listed at is also working now21:29
fungibut was how the error was previously observed21:29
fungiso maybe it was a bug in opendev-1.7.6 which was fixed in opendev-1.8.0?21:30
fungiand somethnig about the persistent javascriptness of gitea is causing that old tab to still return the error message21:31
fungiyeah, after loading the same url in a new tab successfully, reloading the old tab now gets me the correct content... weird21:32
mordredfungi: it's best not to ask questions - the computers don't want you to know their secrets, and they'll stop at nothing to keep it that way21:32
*** slaweq has quit IRC21:35
aspiersinteresting, stackalytics broke on nova
aspierswho's the Mirantis contact?21:36
aspiersor whoever maintains it21:37
fungifolks have just been e-mailing openstack-discuss and hoping someone sees it21:37
fungithey replied on the ml over the weekend that they'd finally done a redeploy/reindex so maybe that was what triggered it? could be worth replying to that message21:38
aspiersOK, thanks21:38
aspiersfungi: done21:39
*** kjackal has quit IRC21:40
fungihah, i was looking for the most recent message in that thread to link for you... and it's yours! ;)21:40
corvusmordred, fungi, dtroyer: local build of upstream master seems to have the problem fixed.  mordred: i've poked around a bit at various urls; do you have a more thorough procedure for determining necessary template updates?21:40
aspiersfungi: quickest list spammer in the west ... of Europe21:41
*** mattw4 has quit IRC21:41
openstackgerritMerged opendev/system-config master: Update key for SotK
*** mattw4 has joined #openstack-infra21:42
*** Goneri has quit IRC21:43
*** gfidente|afk has quit IRC21:43
*** jamesmcarthur has joined #openstack-infra21:45
corvusby the way, we got "blame" support in gitea with the upgrade to 1.8.021:45
aspierscorvus: wooo, nice21:46
*** jtomasek has quit IRC21:47
*** mriedem has quit IRC21:48
ianwsorry i'll be starting today a little later; is there anything i need to help with for the next 30 minutes or so?21:49
*** rh-jelabarre has quit IRC21:50
aspierspersia: have you seen ? this seems quite like that thing you were proposing to me ages ago21:51
aspierscorvus: I miss equivalent functionality to the 't' shortcut in GitHub21:52
openstackgerritJames E. Blair proposed opendev/system-config master: Upgrade gitea to current master (pre-1.9.0)
*** mriedem has joined #openstack-infra21:52
aspiersI find that frequently useful21:52
corvusianw: i think all fires are out21:53
corvusdtroyer, mordred, fungi: ^ gitea upgrade21:53
corvusaspiers: what's a "t"?21:53
*** Goneri has joined #openstack-infra21:53
aspierscorvus: try pressing it :)21:53
corvuspretend, for the sake of argument, i'm not a big github user21:53
aspiers for example21:53
aspierssubstring path matching21:54
corvusaspiers: do you like hacking in go?  i've found the gitea authors very accepting of new features :)21:54
aspierssadly no go (haha) experience yet21:55
aspiersI'm interpreting "i'm not a big github user" as a boast cunningly designed to make me jealous21:55
aspiersone day maybe I'll be able to say that21:56
*** Weifan has joined #openstack-infra21:56
mordredcorvus: +2 from me21:57
dtroyercorvus: thanks for a quick response!  also, I look forward to our new 'git blame' overlord^H^H^H^H^H^H^H^Hfunctionality…21:58
*** rlandy is now known as rlandy|bbl21:59
WeifanHi, I have a question about the github repo.22:01
WeifanOriginally our repo was under openstack, and now it is moved to x.22:01
WeifanWe decided that we would just keep the one on opendev, and stop updating github.22:01
WeifanHowever, we want to add a redirect/link on the original github repo to opendev.22:01
WeifanIs there any guidlines for it?22:01
*** sreejithp has quit IRC22:02
fungiwe can transfer to openstack/whatever github repo to a different github org under your control22:03
*** sreejithp has joined #openstack-infra22:03
*** Weifan has quit IRC22:03
fungigithub has no features i'm aware of for redirecting to somewhere other than github22:03
openstackgerritMerged zuul/zuul master: Annotate node request processing with event id
*** Weifan has joined #openstack-infra22:04
fungiWeifan: not sure if you saw what i was saying before you disconnected22:04
fungiWeifan: we can transfer to openstack/whatever github repo to a different github org under your control22:04
fungiWeifan: github has no features i'm aware of for redirecting to somewhere other than github22:04
Weifanso how should we start the proccess?22:04
corvuswe can certainly delete the x/ repo22:05
*** rfarr__ has quit IRC22:05
corvusi'm not sure if we could do something like merging a "retirement" commit to it though22:06
Weifannah.. we want to keep the x22:06
Weifanwe just want to retire the github repo22:06
corvussorry, i meant delete the x/ from github22:06
corvusoh, er, we didn't create any x/ repos on github22:06
Weifani think we people search for our plugin, they find github first, so we want to put link there22:07
corvusWeifan: we can delete the openstack/ version of the repo22:07
fungiyeah, that would be very simple22:07
corvusi believe that will happen for all of the repos which moved from openstack/ to x/ soon anyway22:07
mordredmaybe this is a "archive the openstack/ version and update the description to point to the url"22:07
weshaysorry to distrub, is the retry_limit issue known?
mordredso that people searching for something on github - or who are used to the old github mirror have a path to find the correct thing22:08
corvusmordred: my understanding is that one of the goals was to remove all repos from openstack/ which were not openstack22:08
mordredcorvus: I'm guessing the gh archive feature probably isn't removal enough22:09
corvusmordred: i would assume that means no archived non-openstack repos... but i admit that's a grey area?22:09
mordredyeah. maybe we should experiment with what archiving a repo looks like22:09
mordredand see if providing an archive-with-pointer would be a good or a bad idea22:09
Weifanwhen are you planning to delete the repos? personally i feel transfer is better as we would have more control, but i am not sure how complicated the process is22:09
corvusmordred: yeah, from an optics pov, it seems like it may not reflect what openstack wants to project, as i understand it  (as in, it would be a whole lot of retired repos unrelated to openstack)22:10
*** slaweq has joined #openstack-infra22:11
corvusWeifan: i don't think a date is set yet, but i think it should be done within a few months at the most.22:11
fungiyeah, the up-side to gh archiving is that it doesn't prevent us from transferring an org in case a project pops up months after asking for that22:11
corvusfungi: then hopefully mordred's idea will work out :)22:12
Weifani am afraid that if people find that the repo is deleted without noticed or without update for months, they would assume the repo is no longer maintained22:12
fungidownside of course is that the repo still exists and just grows a yellow-on-brown banner in their webui indicating the repo is archived (based on screenshots i saw)22:12
mordredcorvus, fungi: maybe transferring to somewhere like stackforge/ - then archiving - would achieve all of the goals - remove things from openstack/ namespace, still have the content show up in searches for peple, then have a pointer to the opendev repos22:12
fungihrm, another option...22:13
corvusmordred: that could work22:13
fungiwe could (eventually) delete repos, but add new empty repos and transfer them on request?22:13
fungithat would effectively replace the nonexistent repo with a redirect to another org22:14
*** raissa has quit IRC22:14
mordredwell - I think if we moved them to somewhere people are unlikely to browse to - with a gh transfer - then old links will still work, and we can do the retire-commit trick before we archive witha  pointer22:14
*** sreejithp has quit IRC22:14
openstackgerritJames E. Blair proposed opendev/system-config master: Switch to upstream and upgrade gitea to current master (pre-1.9.0)
mordredand, amusingly enough, non-openstack-things-in-openstack would have been in stackforge/ if we'd stayed with that model, so it's not SUPER inapproriate22:15
corvusmordred, fungi: ^ that commit is an alternate to the previous one22:15
*** slaweq has quit IRC22:15
mordredcorvus: actually - I think I like that with a pin22:16
mordredcorvus: it does give us an ugly version - but we ARE running off of a sha from master, so it's accurate22:16
rpiosoDoes the gitea web UI offer a permalink feature similar to GitHub's?22:16
mordredand it makes it easy for us to bump the sha if we need to between now and 1.9.022:16
mordredrpioso: what do you mean?22:16
mordreddo you  mean the redirect thing on repo renames?22:17
logan-rpioso: it does, in a slightly less intuitive way. view a file, click "permalink" in the file header, and then select lines22:18
rpiosomordred: It provides a URI to code which is "permanently" valid.22:18
corvuslogan-: as a non-native github user, i found that more intuitive than "select the lines, then click '...' and then click 'copy permalink'"22:20
corvusi have never used "copy permalink" before22:20
* mordred too - and me either22:20
corvusso you just performed a ui experiment on me.  it took me between 1 and 2 minutes of poking through the ui to find that.22:21
fungiintuitive for me (also as a non-github user) is to browse the tree as of a particular commit and ling from there22:21
corvussorry i didn't know it was going to be a ui experiment, otherwise i would have started a recording :)22:21
rpiosologan: Thank you!22:21
corvusfungi: yeah, that's how i've done that before22:22
rpiosofungi: And that appears to be what Gitea's web UI does. The Tree: displays the commit's SHA.22:25
*** jamesmcarthur has quit IRC22:26
fungimakes sense to me. that's how you refer to a permanent state in a git repository22:31
fungi(where permanent depends on it at least being in the history of a head, tag or named ref so the garbage collector doesn't eventually clean it up)22:32
*** kskels has joined #openstack-infra22:32
rpiosofungi: I refer to that as "permanent" :-)22:32
*** Goneri has quit IRC22:32
kskelsclarkb: fyi - github replication works now! this is great -thanks for the support and samples from ara project!!22:33
fungirpioso: i guess i just tend to dwell on the impermanence of all things22:33
rpiosofungi: It's software.22:34
mordredkskels: \o/22:37
Weifanfungi: if we want to transfer the repo, how should we process? or should we just wait for all the repos to be "archived"?22:39
mordredWeifan: where do you want to transfer the repo?22:39
Weifantransfer to our organization22:40
Weifan  this one, to bigswitch22:40
mordredah. nod. I'm not sure what we've decided on process-wise for doing that ... fungi, do we have a process decided for that?22:41
*** diablo_rojo has joined #openstack-infra22:42
*** diablo_rojo has quit IRC22:47
fungiWeifan: mordred: we just did this for airship's repos, via this mailing list thread:
*** kskels has left #openstack-infra22:57
*** mriedem has quit IRC22:59
*** igordc has quit IRC23:01
Weifanfungi: in the email, it says that openstackadmin needs to be added as an adimin of the organization. Is this necessary? I believe it is already an admin of the repo that we want to transfer.23:02
fungiWeifan: the account used to perform the transfer has to temporarily be an admin of both source and destination orgs, because of github's permission model23:03
fungisince it's a one-step process and there's no separate proposal and acceptance phases to the transaction23:04
Weifanshouldn't it be sending an email or notification the receiver?23:05
fungiit does not do any of those things23:06
*** tkajinam has joined #openstack-infra23:06
fungiit basically allows you to transfer the repository if you're an admin of the old and new org, otherwise it refuses23:06
fungihrm, at least that's the way the process we'd been using works23:08
Weifanis it any different transferring to a private account?23:08
fungibut this documentation implies a proposal/acceptance:
mordredfungi: yeah - I just did an app transfer and when I did it it said "this will have to wait for the receiving party to accept"23:09
mordredand an email was sent to the admin email of the receiving org23:09
mordredfungi: so it's possible this got better since last we looked23:09
fungiagreed... i wonder if this mechanism is accessible via the api23:10
mordredwe could try transferring networking-bigswitch without the openstackadmin grant first to test it ... ooh, it would be awesome if it was accessible via the API23:10
*** tosky has quit IRC23:11
fungithe way we'd been doing it was via scripted api interaction23:11
*** slaweq has joined #openstack-infra23:11
*** tosky has joined #openstack-infra23:11
Weifanyep..we can try that23:11
*** slaweq has quit IRC23:15
mordredfungi: BOOYAH23:19
*** rcernin has joined #openstack-infra23:20
fungithis must be a very recent improvement23:21
fungimakes things easier on us23:21
*** sarob has quit IRC23:23
*** sarob has joined #openstack-infra23:23
*** sarob has quit IRC23:23
*** mattw4 has quit IRC23:26
*** zhurong has quit IRC23:30
openstackgerritTristan Cacqueray proposed zuul/zuul master: Skip file matcher for pipeline using timer trigger
Weifanso, is there anything needed for me to proceed, or should I just inform our admin to to watch for the incoming email?23:34
*** tosky has quit IRC23:34
openstackgerritTristan Cacqueray proposed zuul/zuul master: ansible-config: pin ara to <1.0.0
*** michael-beaver has quit IRC23:41
mordredWeifan: I thnik that - although it might be a couple of days (we should update our script first, and it's dinner time)23:42
mordred(just inform your admin to watch for the incoming email)23:42
Weifanmordred: thanks:)23:42
*** michael-beaver has joined #openstack-infra23:51
dirkconfig-core: could I get a 2nd review on ?23:59

Generated by 2.15.3 by Marius Gedminas - find it at!