Thursday, 2019-05-23

sean-k-mooneywas there a zuul restart earlier?00:00
fungiyes, there was00:00
sean-k-mooneyok that makes sense00:00
fungi#status log zuul scheduler was restarted at 20:41z00:00
sean-k-mooneyi kicked off something 6 hours ago and its still hadnt started00:00
fungiseems we forgot to log that00:01
fungiopenstackstatus is also ignoring me00:01
sean-k-mooneyno worries im going to reproduce it locally and ill check the job in the morning00:01
openstackstatusfungi: finished logging00:02
fungioh, there it goes00:02
fungithat was a lengthy delay00:02
sean-k-mooneyit might need a reboot too :)00:02
fungii think it's just tired00:02
mordredhave you tried turning it off and on again?00:04
*** gyee has quit IRC00:04
sean-k-mooneymordred: while i think of it if you have an opipion on the nameing of the nova evacuate command equivalent in osc feel free to chime in
sean-k-mooneyill likely send a mail to the list about it but you expressed an interst at the fourm session so just an fyi00:05
* sean-k-mooney or rather mordred didnt run seaming out of the room when evacuate was mentions so is interested by default00:06
*** jamesmcarthur has joined #openstack-infra00:08
*** eernst has joined #openstack-infra00:17
*** jamesmcarthur has quit IRC00:18
*** jamesmcarthur has joined #openstack-infra00:19
*** Weifan has quit IRC00:21
*** aaronsheffield has quit IRC00:24
*** ekultails has quit IRC00:24
*** jamesmcarthur has quit IRC00:25
pabelangeroh, ha. zuul-scheduler was only restarted today, not executors. That explains why we don't have full 2.8 support00:33
pabelangerfor ansible00:34
*** hwoarang has quit IRC00:36
*** hwoarang has joined #openstack-infra00:37
*** jamesmcarthur has joined #openstack-infra00:52
*** eernst has quit IRC00:54
*** eernst has joined #openstack-infra00:54
*** slaweq has joined #openstack-infra00:57
*** jamesmcarthur has quit IRC00:58
*** eernst has quit IRC00:58
*** lseki has quit IRC01:00
*** slaweq has quit IRC01:01
*** sarob has joined #openstack-infra01:06
fungiyes, i'm happy to help with executor restarts tomorrow, but it's a bit late in the evening for me now01:09
pabelangerwfm, also will be around to assist too01:15
openstackgerritMerged opendev/system-config master: Switch to upstream and upgrade gitea to current master (pre-1.9.0)
fungiinfra-root: we should probably keep an eye on with that ^ merging01:16
fungihopefully it deploys smoothly and there are no new regressions01:17
*** jamesmcarthur has joined #openstack-infra01:28
*** whoami-rajat has joined #openstack-infra01:30
*** jamesmcarthur has quit IRC01:33
*** ricolin has joined #openstack-infra01:36
*** gregoryo has joined #openstack-infra01:41
*** jamesmcarthur has joined #openstack-infra01:45
fungiGitea Version: 6eb53ac570ab9af51fc9cbd79f1db782edce57e001:46
fungilooks like it's deployed now01:46
fungidtroyer: corvus: mordred: is showing normal content for me now01:49
fungipoking around the site, stuff's looking normal so far01:49
*** jamesmcarthur has quit IRC01:49
fungiunless someone else notices a problem, i think we're good01:50
*** zhurong has joined #openstack-infra01:50
*** apetrich has quit IRC01:57
*** michael-beaver has quit IRC02:01
*** slaweq has joined #openstack-infra02:11
logan-the new gitea upgrade has blame view! awesome02:11
*** jamesmcarthur has joined #openstack-infra02:12
*** slaweq has quit IRC02:15
*** jamesmcarthur has quit IRC02:17
*** ykarel|away has joined #openstack-infra02:20
dtroyerfungi, corvus: yup, looks good here too, thanks!02:25
*** irclogbot_0 has quit IRC02:26
*** irclogbot_3 has joined #openstack-infra02:29
*** jamesmcarthur has joined #openstack-infra02:35
*** bobh has joined #openstack-infra02:39
*** sreejithp has joined #openstack-infra02:39
*** sreejithp_ has joined #openstack-infra02:41
*** sreejithp has quit IRC02:44
openstackgerritMerged openstack/pbr master: Update Sphinx requirement
*** nicolasbock has quit IRC02:51
*** _erlon_ has quit IRC02:52
*** jamesmcarthur has quit IRC03:05
*** bobh has quit IRC03:12
*** rlandy|bbl is now known as rlandy03:14
*** bhavikdbavishi has joined #openstack-infra03:16
*** sarob has quit IRC03:20
*** rlandy has quit IRC03:20
*** jamesmcarthur has joined #openstack-infra03:26
*** bhavikdbavishi has quit IRC03:45
*** bhavikdbavishi has joined #openstack-infra03:45
openstackgerritdongwenjuan proposed x/gearman-plugin master: add project parameters
*** gregoryo has quit IRC03:47
openstackgerritTristan Cacqueray proposed zuul/zuul master: Add nodesets API route
*** jamesmcarthur has quit IRC03:48
*** samueldmq has quit IRC03:50
*** yikun_ has quit IRC03:50
*** sreejithp_ has quit IRC03:51
*** jamesmcarthur has joined #openstack-infra03:56
*** bhavikdbavishi has quit IRC04:06
*** bhavikdbavishi has joined #openstack-infra04:07
*** slaweq has joined #openstack-infra04:11
*** jamesmcarthur has quit IRC04:13
*** slaweq has quit IRC04:16
*** ykarel|away has quit IRC04:17
*** auristor has quit IRC04:23
*** udesale has joined #openstack-infra04:28
*** ykarel|away has joined #openstack-infra04:32
*** auristor has joined #openstack-infra04:32
AJaegerfrickler: good morning! the api jobs are broken due to sphinx/python 3, I think and will fix it as side effect - could you review, please?04:37
*** auristor has quit IRC04:39
*** jamesmcarthur has joined #openstack-infra04:40
*** auristor has joined #openstack-infra04:42
*** jamesmcarthur has quit IRC04:51
*** kjackal has joined #openstack-infra04:55
*** jamesmcarthur has joined #openstack-infra04:57
*** kjackal has quit IRC05:00
*** jamesmcarthur has quit IRC05:04
*** roman_g has quit IRC05:05
*** lpetrut has joined #openstack-infra05:11
*** yboaron_ has quit IRC05:11
*** jamesmcarthur has joined #openstack-infra05:12
openstackgerritFelix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI
*** ykarel|away is now known as ykarel05:20
*** sarob has joined #openstack-infra05:23
*** raukadah is now known as chandankumar05:25
*** quiquell has joined #openstack-infra05:27
*** sarob has quit IRC05:27
*** ccamacho has quit IRC05:29
*** janki has joined #openstack-infra05:31
*** zbr has joined #openstack-infra05:34
openstackgerritFelix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI
*** imacdonn has quit IRC05:41
*** imacdonn has joined #openstack-infra05:42
AJaegerconfig-core, please review to help the requirements team move forward05:48
*** jamesmcarthur has quit IRC05:57
*** jamesmcarthur has joined #openstack-infra05:58
*** slaweq has joined #openstack-infra05:59
*** hwoarang has quit IRC06:01
*** hwoarang has joined #openstack-infra06:02
*** slaweq has quit IRC06:03
*** sarob has joined #openstack-infra06:04
*** kopecmartin|off is now known as kopecmartin06:04
*** slaweq has joined #openstack-infra06:11
*** dpawlik has joined #openstack-infra06:16
*** jchhatbar has joined #openstack-infra06:19
openstackgerritIan Wienand proposed openstack-infra/ master: [dnm] testing
*** janki has quit IRC06:20
openstackgerritFelix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI
*** gregoryo has joined #openstack-infra06:25
openstackgerritIan Wienand proposed openstack-infra/ master: [dnm] testing
*** sarob has quit IRC06:37
*** Lucas_Gray has joined #openstack-infra06:39
*** tesseract has joined #openstack-infra06:41
*** yboaron_ has joined #openstack-infra06:43
*** Lucas_Gray has quit IRC06:46
*** Lucas_Gray has joined #openstack-infra06:48
*** roman_g has joined #openstack-infra06:48
*** yboaron_ has quit IRC06:48
*** yboaron_ has joined #openstack-infra06:49
*** jamesmcarthur has quit IRC06:54
*** Lucas_Gray has quit IRC06:55
*** pgaxatte has joined #openstack-infra06:58
*** adriancz has joined #openstack-infra06:59
*** rcernin has quit IRC07:00
*** markvoelker has joined #openstack-infra07:03
*** pcaruana has joined #openstack-infra07:07
*** markvoelker has quit IRC07:08
*** ginopc has joined #openstack-infra07:09
*** sarob has joined #openstack-infra07:11
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver -
openstackgerritMerged openstack/openstack-zuul-jobs master: Use openstack-tox-docs for api-* jobs
openstackgerritMerged openstack/openstack-zuul-jobs master: Switch to using TOX_CONSTRAINTS_FILE
openstackgerritMerged openstack/project-config master: Handle TOX_CONSTRAINTS_FILE as well as UPPER_CONSTRAINTS_FILE
AJaegerconfig-core, let's finish up the two changes above with merging , please07:19
*** apetrich has joined #openstack-infra07:19
openstackgerritIan Wienand proposed openstack-infra/ master: [dnm] testing
*** rpittau|afk is now known as rpittau07:20
*** rf0lc0 has joined #openstack-infra07:23
*** dhellmann_ has joined #openstack-infra07:25
*** rfolco has quit IRC07:26
*** dhellmann has quit IRC07:26
*** geguileo has quit IRC07:27
openstackgerritMerged openstack/project-config master: Switch api publish jobs to tox-docs
*** jchhatbar has quit IRC07:27
*** dhellmann_ is now known as dhellmann07:29
*** e0ne has joined #openstack-infra07:31
*** tosky has joined #openstack-infra07:32
openstackgerritMonty Taylor proposed opendev/system-config master: Update opendev website to be more present tense
ianwlooks good07:33
*** ccamacho has joined #openstack-infra07:35
AJaegerthanks, ianw07:36
*** ykarel is now known as ykarel|afk07:37
*** janki has joined #openstack-infra07:44
*** sarob has quit IRC07:44
*** jpena|off is now known as jpena07:45
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] zone file vaildator
openstackgerritIan Wienand proposed openstack-infra/ master: [dnm] testing
AJaegermriedem, is looking green again wiht the fixes merged and is also updated. So, should be all fine again...07:46
*** apetrich has quit IRC07:48
openstackgerritMarkus Hosch proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check
openstackgerritMerged zuul/zuul-jobs master: Switch to TOX_CONSTRAINTS_FILE
*** sreejithp has joined #openstack-infra07:52
*** tosky has quit IRC07:55
openstackgerritMerged opendev/system-config master: Remove unused commit-filter script
*** tosky has joined #openstack-infra07:55
*** apetrich has joined #openstack-infra07:55
*** gregoryo has quit IRC07:56
*** gregoryo has joined #openstack-infra07:56
*** sreejithp has quit IRC07:56
*** gregoryo has quit IRC07:56
openstackgerritMerged opendev/system-config master: Drop tools/
zigoianw: AJaeger: In puppet-openstack-integration, there's a Puppetfile, I used to replace :ref => 'master' by for example :ref => 'refs/changes/46/660846/1' to get a particular patch which is still under review, so I can continue to work on the next patch. This doesn't seem to work anymore. What's the trick?07:59
*** apetrich has quit IRC07:59
*** jtomasek has joined #openstack-infra07:59
zigoShould I change the Git URL somehow?07:59
*** apetrich has joined #openstack-infra08:00
*** iurygregory has joined #openstack-infra08:01
ianwzigo: hrm, wouldn't that be pulling from (not what used to be
ianwif it's not, it could be i think08:02
*** ykarel|afk is now known as ykarel08:03
zigoianw: I can try to use that URL then.08:03
*** lucasagomes has joined #openstack-infra08:04
*** trident has quit IRC08:04
*** ralonsoh has joined #openstack-infra08:04
*** jtomasek has quit IRC08:05
*** trident has joined #openstack-infra08:05
*** bhavikdbavishi has quit IRC08:11
*** tkajinam has quit IRC08:14
*** lyarwood has joined #openstack-infra08:15
*** ykarel is now known as ykarel|lunch08:16
lyarwoodIs anyone aware of any host ssh key changes to I'm getting `RSA host key for has changed and you have requested strict checking.` errors suddenly.08:17
*** tobias-urdin has joined #openstack-infra08:19
*** pkopec has joined #openstack-infra08:20
*** janki has quit IRC08:23
*** sarob has joined #openstack-infra08:30
*** Lucas_Gray has joined #openstack-infra08:40
*** panda|rover has quit IRC08:40
*** panda has joined #openstack-infra08:42
openstackgerritFelix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter
*** derekh has joined #openstack-infra08:46
openstackgerritFelix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter
*** bhavikdbavishi has joined #openstack-infra08:49
openstackgerritdongwenjuan proposed openstack/openstack-zuul-jobs master: install maven for building gearman-plugin
*** jamesmcarthur has joined #openstack-infra08:51
*** jamesmcarthur has quit IRC08:55
*** gfidente has joined #openstack-infra08:58
*** bhavikdbavishi1 has joined #openstack-infra08:59
*** bhavikdbavishi has quit IRC08:59
*** bhavikdbavishi1 is now known as bhavikdbavishi08:59
*** sarob has quit IRC09:02
*** jamesmcarthur has joined #openstack-infra09:04
*** markvoelker has joined #openstack-infra09:04
AJaegerlyarwood, we renamed to and it has a new host key...09:08
*** panda is now known as panda|rover09:11
*** pgaxatte has quit IRC09:12
openstackgerritFelix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter
openstackgerritStephen Finucane proposed openstack/pbr master: Remove neutron-lbaas
*** ricolin has quit IRC09:15
lyarwoodAJaeger: right but that was a while ago and the redirect has been working for my existing remotes until this morning09:16
fricklerlyarwood: do you see this for connects to gerrit (port 29418) or to ssh (port 22)09:18
lyarwoodAJaeger: nvm, looks like it's just one repo where things are borked so maybe this is just a local issue09:18
lyarwoodfrickler: yeah the host key was to port 22, so that's my fault09:20
*** jtomasek has joined #openstack-infra09:20
lyarwoodand the actual issue is the repo has moved with the opendev rename, apologies for missing this09:20
dirkfrickler: pabelanger: ianw: any chance you could review and its dependencies? things are starting to break because 15.1 was released but the ci jobs are now not having their repositories anymore09:21
dirkit would be good to have a mirror of 15.1 repos shortly09:21
openstackgerritLee Yarwood proposed opendev/elastic-recheck master: Add query for Cinder volume attachment bug 1820155
openstackbug 1812913 in Cinder "duplicate for #1820155 VolumeAttachment is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed" [High,Fix released] - Assigned to Gorka Eguileor (gorka)09:24
*** jtomasek has quit IRC09:25
*** ginopc has quit IRC09:33
*** Lucas_Gray has quit IRC09:34
*** pgaxatte has joined #openstack-infra09:34
*** ykarel|lunch is now known as ykarel09:34
*** markvoelker has quit IRC09:38
*** electrofelix has joined #openstack-infra09:38
rpittauAJaeger: hi! I think this change broke a couple of things, for example the doc/build dir do not exist because tox run with the api-ref env, not docs09:39
*** jamesmcarthur has quit IRC09:44
*** jamesmcarthur has joined #openstack-infra09:45
*** derekh has quit IRC09:54
*** derekh has joined #openstack-infra09:54
openstackgerritTristan Cacqueray proposed zuul/zuul master: trigger: add job filter event
openstackgerritTristan Cacqueray proposed zuul/zuul master: webtrigger: add initial driver and event
openstackgerritTristan Cacqueray proposed zuul/zuul master: webtrigger: add web route and rpclistener
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: add build button to trigger job
openstackgerritFelix Schmidt proposed zuul/zuul master: Show retry attempts in UI
fricklerrpittau: dtantsur: AJaeger: ack, patch upcoming, we need to keep the sphinx_build_dir variable09:58
*** jamesmcarthur has quit IRC09:58
rpittaufrickler: thanks!09:58
openstackgerritJens Harbott (frickler) proposed openstack/openstack-zuul-jobs master: Fix build-api-* jobs
*** jamesmcarthur has joined #openstack-infra10:10
*** electrofelix has quit IRC10:12
*** electrofelix has joined #openstack-infra10:13
openstackgerritMerged opendev/system-config master: Properly mirror the lxc container images as well
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Zone file validation role
*** jamesmcarthur has quit IRC10:16
openstackgerritIan Wienand proposed openstack-infra/ master: Add zone-check job
openstackgerritIan Wienand proposed openstack-infra/ master: [dnm] sample of zone misconfiguration to fail testing
*** hjensas is now known as hjesas|afk10:19
*** jamesmcarthur has joined #openstack-infra10:21
openstackgerritIan Wienand proposed openstack-infra/ master: Add zone-check job
*** jamesmcarthur has quit IRC10:26
dtantsurfrickler: thank you!10:31
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Zone file validation role
fricklerianw: could you take a quick look at the api job fix ? otherwise I'd just self-approve it in order to hopefully fix the regression merged earlier10:34
openstackgerritrenminmin proposed openstack/diskimage-builder master: Add DIB_LOCAL_IMAGE to use ubuntu local cloud image
ianwfricker: thanks, lgtm10:35
*** markvoelker has joined #openstack-infra10:35
*** jamesmcarthur has joined #openstack-infra10:39
*** paladox has quit IRC10:41
openstackgerritSorin Sbarnea proposed openstack/pbr master: Set subparser argument required
openstackgerritSorin Sbarnea proposed openstack/pbr master: Add openstack-tox-py37 job
*** jamesmcarthur has quit IRC10:43
*** paladox has joined #openstack-infra10:45
ianwfungi / clarkb / corvus : something like would have prevented the cname conflicts i merged the other day.  i think it's good for hygiene because debugging is a bit of a pain, and impossible for !root10:46
openstackgerritTobias Henkel proposed zuul/zuul master: Log github requests with annotated events
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate logs around build completion and cancellation
*** jamesmcarthur has joined #openstack-infra10:48
*** jamesmcarthur has quit IRC10:52
*** diablo_rojo has joined #openstack-infra10:53
openstackgerritMerged openstack/openstack-zuul-jobs master: Fix build-api-* jobs
openstackgerritFelix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter
*** jpena is now known as jpena|lunch11:01
*** ginopc has joined #openstack-infra11:04
*** diablo_rojo has quit IRC11:07
*** markvoelker has quit IRC11:08
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source
openstackgerritMark Meyer proposed zuul/zuul master: Upgrade formatting of the patch series.
*** nicolasbock has joined #openstack-infra11:11
*** janki has joined #openstack-infra11:11
*** udesale has quit IRC11:13
*** udesale has joined #openstack-infra11:13
*** jamesmcarthur has joined #openstack-infra11:16
*** Lucas_Gray has joined #openstack-infra11:22
openstackgerritFlavio Percoco proposed zuul/zuul master: Read multiple *.conf files from directory
fricklerrpittau: dtantsur: latest api-ref jobs seem to be passing again, please let us know if you notice further issues11:27
dtantsurthx, sure11:28
*** bhavikdbavishi has quit IRC11:29
*** dave-mccowan has joined #openstack-infra11:32
*** eharney has quit IRC11:34
*** sshnaidm|afk is now known as sshnaidm11:42
*** bobh has joined #openstack-infra11:47
*** sreejithp has joined #openstack-infra11:52
*** _erlon_ has joined #openstack-infra11:54
*** sreejithp has quit IRC11:57
*** panda|rover is now known as panda|rover|eat11:59
*** markvoelker has joined #openstack-infra11:59
*** tosky has quit IRC12:01
*** tosky has joined #openstack-infra12:03
*** hrw has joined #openstack-infra12:03
hrwcan someone tell me who I should pester to get merged? it is x/networking-infoblox which looks like no one cares about but nicely blocks any Py3 work12:04
*** janki has quit IRC12:04
hrwand this is Py3 one line fix12:04
*** janki has joined #openstack-infra12:05
*** dpawlik has quit IRC12:06
hrwfor now I just cut it off from Kolla to make it unblocking and as long as no one will complain I will be happy to ignore that x/ project12:07
*** tosky__ has joined #openstack-infra12:08
*** rfarr__ has joined #openstack-infra12:08
*** rfarr_ has joined #openstack-infra12:10
*** dpawlik has joined #openstack-infra12:10
*** tosky has quit IRC12:11
*** dpawlik has quit IRC12:12
*** rfarr__ has quit IRC12:13
*** dpawlik has joined #openstack-infra12:13
*** dhill_ has joined #openstack-infra12:14
*** sreejithp has joined #openstack-infra12:16
*** jamesmcarthur has quit IRC12:18
*** jamesmcarthur has joined #openstack-infra12:18
*** eharney has joined #openstack-infra12:18
*** rf0lc0 is now known as rfolco12:18
*** sreejithp_ has joined #openstack-infra12:20
*** rh-jelabarre has joined #openstack-infra12:20
*** sreejithp has quit IRC12:21
*** sreejithp_ has quit IRC12:22
*** sreejithp has joined #openstack-infra12:25
AJaegerfrickler: thanks for fixing api-ref! dtantsur, frickler, sorry for breakage!12:28
*** rlandy has joined #openstack-infra12:29
*** jpena|lunch is now known as jpena12:30
*** jamesmcarthur has quit IRC12:30
*** tosky has joined #openstack-infra12:33
*** tosky__ has quit IRC12:36
*** sreejithp has quit IRC12:36
*** tosky__ has joined #openstack-infra12:38
*** tosky has quit IRC12:40
*** tosky__ is now known as tosky12:40
*** lseki has joined #openstack-infra12:41
*** eernst has joined #openstack-infra12:47
*** ykarel is now known as ykarel|afk12:48
*** bobh has quit IRC12:49
rpittauAJaeger frickler thanks and no worries! One change that was failing is now passing, so all good :)12:51
fungihrw: it looks like these are the people capable of approving your change on that project:
hrwfungi: thx12:51
AJaegerhrw: you can check the core group, go on the option icon, look at access etc - here's a direct link:,access ., But if those do not care, ignore them...12:51
AJaegerrpittau, yeah, I fixed the post job and broke the check/gate ;( Glad that everything is ggreen again...12:52
*** jamesmcarthur has joined #openstack-infra12:52
fungi(for the record, the way i found the core review group for that repo was following the same steps AJaeger mentions)12:52
AJaegerhrw: or use fungi's link directly...12:52
hrwok. added missing ones.12:53
*** e0ne has quit IRC12:57
*** panda|rover|eat is now known as panda|rover13:00
*** mriedem has joined #openstack-infra13:03
*** jamesmcarthur has quit IRC13:05
*** eernst has quit IRC13:06
*** Goneri has joined #openstack-infra13:06
*** dave-mccowan has quit IRC13:12
*** ykarel|afk is now known as ykarel13:12
openstackgerritFabien Boucher proposed zuul/zuul master: Set the DiskAccountant log when size > self.limit as warning level
openstackgerritMatt Riedemann proposed opendev/elastic-recheck master: Remove query for bug 1640920
openstackbug 1640920 in Cinder "list manageable snapshots fails with lazy load error on snapshot model attr" [Undecided,New] - Assigned to Gorka Eguileor (gorka)13:14
*** dave-mccowan has joined #openstack-infra13:14
*** smarcet has joined #openstack-infra13:17
*** aaronsheffield has joined #openstack-infra13:18
*** janki has quit IRC13:31
*** ginopc has quit IRC13:33
*** ginopc has joined #openstack-infra13:34
*** samueldmq has joined #openstack-infra13:36
openstackgerritMerged opendev/elastic-recheck master: Remove query for bug 1640920
openstackbug 1640920 in Cinder "list manageable snapshots fails with lazy load error on snapshot model attr" [Undecided,New] - Assigned to Gorka Eguileor (gorka)13:37
*** sreejithp has joined #openstack-infra13:40
*** sreejithp_ has joined #openstack-infra13:42
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Fix case sensitivity in codeowners check
*** sreejithp has quit IRC13:45
*** hrw has left #openstack-infra13:47
*** ricolin has joined #openstack-infra13:49
corvusfungi, mordred: now that gitea should be able to handle the extra refs, we need to get them into the git repos.  we could just let gerrit replication take care of things, but it will still take a while to replicate nova, so i think we may want to handle it specially.  what do you think about this procedure?
*** quiquell has quit IRC13:58
openstackgerritMerged zuul/zuul-jobs master: Zone file validation role
*** smarcet has quit IRC14:01
*** yolanda__ has joined #openstack-infra14:02
*** iurygregory has quit IRC14:02
*** yolanda__ is now known as yolanda14:03
corvusfungi, mordred: according to it would take 2 hours to push nova without pre-seeding it, but should only take a few seconds to do so afterwords.  should we think about pre-seeding any other repos?14:08
openstackgerritJames E. Blair proposed opendev/system-config master: Push all refs to gitea
*** pcaruana has quit IRC14:10
fungicorvus: if gerrit reindex times are any measure, neutron and openstack-manuals14:10
corvusfungi: i think they are good measures14:11
fungithose are consistently the top three left reindexing after all other repos are done14:11
corvusi believe it's directly linked to the number of changes14:11
corvus(so, eg, deb-nova should not be a problem)14:11
fungiyes, which is why i thought of it14:11
fungiseems like a relevant comparison14:12
fungicorvus: +refs/notes/*:refs/notes/* too or do you want to save those for a separate pass?14:14
*** eernst has joined #openstack-infra14:14
corvusfungi: i forgot them, will add, thx14:14
fungijust checking14:15
*** sreejithp has joined #openstack-infra14:16
*** smarcet has joined #openstack-infra14:16
corvusfungi, mordred: i took the liberty of validating that this works on gitea06 (which is out of the load balancer due to a corrupt filesystem and due to be replaced, but works well enough to test these commands)14:17
fungiyeah, i wondered if we should try it there14:17
fungigood idea14:18
fungigranted, it may go faster there as it's not serving any requests14:18
*** sreejithp__ has joined #openstack-infra14:18
fungiso has less competition for system resources14:18
corvusi think it goes slower whenever it hits bad fs blocks :)14:18
fungibut at least provides us with a lower bound on runtime14:18
fungioh, hah ;)14:18
corvusit took "a few minutes"14:19
fungitrivial then, in the grand scheme14:19
*** sreejithp_ has quit IRC14:19
*** eernst has quit IRC14:19
funginot "a few hours" or "a few days" anyway14:19
corvusyeah.  and i really do expect it to be faster, because even "git show $something" is unusually slow.14:19
*** sreejithp has quit IRC14:21
*** smarcet has quit IRC14:24
*** michael-beaver has joined #openstack-infra14:24
corvusthis may work, if you reload a couple of times:
corvusthat's an intermediate patchset (not the final merged version)14:26
corvusit worked 1/3 times for me; given the state of 06, i think that counts as success14:27
*** dpawlik has quit IRC14:27
fungii guess taking a while to return a result is the expected misbehavior when it runs into fs trouble?14:28
fungiaha, it did eventually load on the first try14:28
corvusfungi: yeah, i believe so14:28
fungiyeah, server seems... "sluggish"14:28
fungimaybe it needs more coffee14:28
fungi(and a less-corrupt filesystem)14:29
*** pcaruana has joined #openstack-infra14:29
* mordred waves to fungi and corvus14:34
mordredcorvus: ++ to the scrollback and pre-seeding refs for things14:35
corvusmordred, fungi: what's the down low on changing the gerrit replication config?  is a full restart needed before it sees changes to the file?14:36
fungimodifying the file on disk will now cause gerrit to automatically reload it (unless we reverted that)14:37
fungibut the frogurt is cursed14:37
fungiqueued replication tasks will be dropped when that happens14:37
corvusi guess that's okay since this process ends with a replicate --all14:37
*** smarcet has joined #openstack-infra14:38
*** auristor has quit IRC14:38
corvusokay, so we should keep the change WIP until the pre-seeding is done, then merge it, then wait for it to be in place, then replicate14:38
fungijust somethnig to keep in mind more generally when altering that file14:38
*** auristor has joined #openstack-infra14:38
fungisounds right to me14:38
mordredyes, that sounds like the right process14:39
corvusgiven that, i'll start the pre-seeding right after breakfast.  i'll do one host first and we can check it out to double check the process didn't break anything (in case there's a difference between gitea06 and a real production host)14:39
*** pgaxatte has quit IRC14:39
*** smarcet has quit IRC14:44
fungialso, this will be awesome once it lands:
fungithough it may need a followup to handle refs/notes/gerrit in addition to just refs/notes/commits14:46
*** sarob has joined #openstack-infra14:46
corvusi like how well thought out that pr is14:47
fungioh! maybe not... a recent update to that pr made the notesref configurable14:47
fungiso actually we should be able to just set it in configuration14:48
mordredyeah- the discussion on the PR is also nice14:48
openstackgerritStephen Finucane proposed openstack/pbr master: Resolve some issue with tox.ini, setup.cfg
fungii tended to rely somewhat often on cgit's behavior of displaying relevant git notes, so if anyone else who was doing the same this is one more checkbox ticked for feature parity14:50
*** dave-mccowan has quit IRC14:52
*** dave-mccowan has joined #openstack-infra14:53
*** kopecmartin is now known as kopecmartin|afk14:53
*** hjesas|afk is now known as hjensas14:55
*** auristor has quit IRC14:58
*** auristor has joined #openstack-infra14:59
smcginnisIs there something that can be done to fix the "No account linked to committer's email address" in our gitea instance, or is it just going to be that way since we don't have fully integrated auth?15:03
*** auristor has quit IRC15:03
corvussmcginnis: can you specify where you see that?15:04
smcginniscorvus: I think since I sign my commits:
*** auristor has joined #openstack-infra15:04
corvussmcginnis: i wasn't aware of that until now.  i'm not sure if we can do anything about it, but i think it's worth looking into.15:06
mordredI think that'll require having user accounts in the gitea - but I agree, I also think it's worth looking in to15:06
corvusconveniently, the pr fungi linked earlier has a screenshot of what it looks like when it's working:
pabelangerit does look like one works:
smcginnisWith GitHub, you have your account that includes somewhere to upload your keys for validation. We might not have that.15:07
pabelangernot sure how that happened15:07
*** lpetrut has quit IRC15:08
corvusalso, frickler seems to have made an account?15:08
mordredthey don't show up in explore users15:08
corvuseach server is going to be different15:09
mordredthis is ... interesting15:09
pabelangerOh, yah, if I refresh I can see it bounce users15:10
fricklercorvus: oh, I did that while exploring things a while back and saw that it had disappeared again immediately, didn't look into it any further back then. that it's per backend server does indeed explain things15:11
*** yboaron_ has quit IRC15:11
corvusfrickler: any chance you remember how you did it so we can prevent it from happening?15:11
mordredI just tried going to (which is what the registration link _would_ be if we didn't have it disabled) and it's properly showing "registration is disabled"15:12
mordredmaybe there was a moment where our config was incomplete and it wasnt' disabled or something?15:12
corvusor 2 moments15:13
corvusyatinkarel was april 2115:13
fricklermordred: corvus: yeah, I think that it's disabled now. I think it should be fine to just remove it if you have a tool for that15:14
corvusyatinkarel is on 1,2,5,7 j-harbott on 5,815:14
corvusyatinkarel jobined on may 1 and april 2115:15
corvusso at least 3 days where that was possible15:16
*** chandankumar is now known as raukadah15:17
corvusand yatinkarel made 1 repo:
clarkbcorvus: re replicating extra refs we should make sure we wont fill the disks on the gitea servers15:17
*** ian-pittwood has joined #openstack-infra15:17
mordredclarkb: speaking of - when we rebuild the gitea servers - I think we want to be sure to clone one of the existing servers' mysql and filesystem rather than rebuilding an empty fs since the repl of the refs/ stuff takes a second15:20
corvusclarkb: good point.  the size difference is currently 2.6G with 13G free, so i think we'll be okay15:20
*** smarcet has joined #openstack-infra15:21
*** ian-pittwood has quit IRC15:21
*** jamesmcarthur has joined #openstack-infra15:21
*** smarcet has quit IRC15:22
*** ccamacho has quit IRC15:22
corvusinfra-root: could someone take on the task of investigating how the yatinkarel account was created?  perhaps clues in the logs or the database may suggest something.15:22
corvusi'd like to press on with the refs15:23
fungiif we're going to copy mysql filesystems around, we maybe ought to sync one new server fresh and then copy that to the others? to address the lingering users and repos which got created early on15:23
fungibut yes, first we probably need to know how the one was created ~3 weeks ago15:23
fungiykarel: do you happen to be around?15:23
mordredcorvus: I'll go troll through the logs15:24
corvusfungi: i'm sure we can delete the account through the web ui, but we should do that after investigating15:24
ykarelfungi, yes15:24
ykarelanything for me?15:24
mordredcorvus: we do not have logs that go back that far15:26
corvusykarel: we noticed that you created an account (perhaps several times) on the gitea servers at, and we thought we had disabled the ability to do that.  can you tell us how you did it so we can close that loophole?15:26
mordredcorvus: only since the latest restart ... so we should also probably add to the list to do something about how logging is logged to places15:26
ykarelcorvus, yes there was login button, i used that, /me looks again15:27
clarkbmordred: I'd really prefer we continue to treat them as stateless as much as possible15:27
corvusmordred: i don't think we have any log rotation configured15:27
clarkbmordred: we can take one out of the lb, replace it, sync, then add to the lb15:27
corvusmordred: which logs are you looking at?15:27
mordredcorvus: the docker logs15:27
corvusmordred: look in /var/gitea/logs15:28
corvus(at this point, the logging bug is that anything is output to docker)15:28
ykarelcorvus, explicitly
corvusykarel: did you use openid?15:29
ykarelcorvus, yes15:29
corvusykarel: thanks, i think that explains it :)15:29
fungineat, neither of those look particularly disabled15:29
corvusfungi: logging in with a password is okay as long as you can't create an account15:29
ykarelcorvus, okk15:29
*** Lucas_Gray has quit IRC15:29
corvusfungi: but i think logging in with an openid carries an implicit registration15:30
mordredclarkb: what I'm saying is that syncing from gerrit is likely to be highly inefficient. so I agree that they should be treated as stateless - but was more suggesting that we perform the "sync" step via a mechanism other that "gerrit replicate"15:30
corvusykarel: fyi, we don't intend to support logins on that system, so we'll delete that account (and the scripts repo) soon15:30
corvusclarkb: nova alone will take >2 hours15:30
*** priteau has joined #openstack-infra15:31
ykarelcorvus, okk np from me, corvus actually some urls got redirect to login page, i logged in thinking those urls will work15:31
mordredcorvus: we have: ENABLE_OPENID_SIGNIN = true15:31
mordredENABLE_OPENID_SIGNUP = true15:31
mordredin our config - should we change those to false?15:31
corvusmordred: ++15:31
mordredpatch coming15:31
openstackgerritMonty Taylor proposed opendev/system-config master: Disable openid login and signup
*** smarcet has joined #openstack-infra15:32
*** jamesmcarthur_ has joined #openstack-infra15:32
*** diablo_rojo has joined #openstack-infra15:33
fricklercorvus: mordred: now that I see it, this openid page looks slightly familiar to me, so there's a good chance that I used that one, too15:35
corvusgood, hopefully that's the only leak then :)15:35
*** jamesmcarthur has quit IRC15:36
clarkbmordred: ya we also need the redirect state so not totally stateless15:37
mordredclarkb: yah. oh - yeah - in fact, that's a _really_ good point15:38
clarkbcorvus: gitea06 has been offline since the ptg. Not sure a fewhours is a major deal?15:38
corvusclarkb: i don't follow15:38
corvusi think i understand15:38
mordredclarkb: if we just do an rsync of git repo data and a mysqldump/restore from a working server - that shoudl get us redirect state and whatnot15:39
*** ykarel is now known as ykarel|away15:39
corvusclarkb: yes, we could allow replication to happen normally and then add it to the load balancer after it completes (say, the next day).  but your point about the db probably supercedes that.15:39
*** hwoarang has quit IRC15:40
clarkbjust pointing out 2 hours to sync nova usnt that bad compared to being out of commission for a couple weeks15:41
*** hwoarang has joined #openstack-infra15:41
*** hamzy has quit IRC15:44
corvusi've copied the repo data to all gitea hosts and am running the nova git fetch on 01 now15:46
*** ykarel|away has quit IRC15:46
clarkbcorvus: that was out of band git push of refs/changes and refs/notes ?15:47
corvusclarkb: fetch, but yes:
clarkbah yes the big three repos15:48
*** sarob has quit IRC15:48
*** whoami-rajat has quit IRC15:50
*** rpittau is now known as rpittau|afk15:51
openstackgerritMonty Taylor proposed opendev/system-config master: Use gitea links in gerrit
mordredcorvus: once the refs are replicated and we're happy with it all, there's the rebased version of "use gitea not gitweb"15:54
*** diablo_rojo has quit IRC15:54
openstackgerritMonty Taylor proposed opendev/system-config master: Finish removing references
corvusoh, i think we need to pack those refs15:59
corvusgit upload-pack and friends are taking a long time because of all the stat calls16:00
mordredmmm. yes. pack refs16:02
corvusokay, i've done that on gitea01.  that makes take about the same time as
toskyquestion on zuul-jobs: before I add more python code to solve the problem highlighted during the review of, I would like to ask: why are dotfiles not shown on by default?16:03
clarkbtosky: mod auto directory renders those indexes for us16:04
clarkber mod auto index in paache16:04
clarkbpossible that it ignores "hidden" files?16:04
toskyclarkb: that seems to be the case; I don't have recent builds but I'm pretty sure that without that change I wasn't able to see them16:05
tosky(now, the change is not complete, but that's a different story)16:05
corvusinfra-root: does anyone else want to poke around the nova repo on gitea01, or should i proceed with the process in for all 3 repos on all servers?16:05
corvus(and i think we're going to want a cron to pack-refs periodically)16:06
clarkbcorvus: reading the process it seemed sound. The only bit that I wondered about was where is the source of the nova repo? rsync from gerrit?16:06
corvusclarkb: scp tarball from /opt/lib on gerrit16:06
*** smarcet has quit IRC16:07
*** roman_g has quit IRC16:07
clarkbtosky: the default indexignore value does not seem to ignore all dotfiles just the current dir single dot dotfile16:08
clarkbtosky: possible our distro is overriding that though16:08
clarkbI think one of the gotchas here is exposing .htpasswd files?16:08
fungiit tends to be considered a safety precaution for that reason, yes16:08
mordredcorvus: renders16:08
*** kjackal has joined #openstack-infra16:08
*** ykarel|away has joined #openstack-infra16:08
toskyI'm sure that there are good reasons for closing it by default, but, unless there are some super-secret files, exposing them under shouldn't be a big issue, I think16:09
clarkbmordred: corvus nova loads really slowly for me on gitea0116:09
*** pcaruana has quit IRC16:09
corvusclarkb: it loads in about the same time for me on gitea0216:09
fungigitea01 lgtm16:09
mordredyes. I agree - it does load slow on 01 - but I also agree, it loads slow on 02 too16:10
mordredat some point we should maybe talk to our gitea friends about ways to make that better16:10
mordredwe have a lovely test-case repo they can play with :)16:11
fungii need to step away to grab lunch and pick up some groceries. need ingredients for stuffing bratwurst this weekend, but will be back as soon as i can16:11
clarkbI can't seem to find the path to render refs/changes/47/651647/1 without the commit sha16:11
clarkbmaybe we don't need that?16:11
clarkb^ that ref maps to the commit url above16:12
mordredyeah - I also can't find such a thing - it's possible it doesn't exist16:12
mordredI believe gerrit will link to changes via the gitweb interface using sha16:12
corvusyeah, i don't think it's needed for the gerrit links16:12
mordredso I don't think we need it immediately16:12
corvushopefully we can git fetch it?16:12
* clarkb tries16:12
mordredit would be a nice improvement to the ui for it to be able to work16:13
*** Romik has joined #openstack-infra16:13
*** pkopec has quit IRC16:14
*** hwoarang has quit IRC16:14
clarkbseems to be working though I got a warning no common commits message with my somewhat older nova clone16:15
*** factor has quit IRC16:15
clarkbI would've expected there to be common commits somwhere16:15
corvusi'm starting on the other servers now16:15
*** hwoarang has joined #openstack-infra16:16
*** gyee has joined #openstack-infra16:16
clarkbwell that explains it my repo named "nova" is actually openstack/openstack for some reason16:16
*** mattw4 has joined #openstack-infra16:16
clarkbI should wipe out my local repos and start over16:16
*** lucasagomes has quit IRC16:18
mordredclarkb: yeah - I did that with the opendev shift16:18
mordredfigured that way I could also get rid of local clones I wasn't using16:19
clarkbya I should probably go ahead and do that myself16:21
clarkbtosky: I think the gotcha with changing that setting is if it comes from the distro apache install then it is likely apache wide and not per vhost (still working to confirm that but I need to dig out my ssh key)16:22
clarkbtosky: in which case while safe for logs, we do host other things on that server which we may not want to have exposed in a similar manner16:23
corvusnova is done on all servers16:23
clarkbmordred: oh cool so we could reset it just for logs16:24
mordredhowever ...16:24
mordredare there any issues serving dotfiles as swift objects?16:24
clarkb and are the two vhosts to change for that. Often we'll update logs-dev first, check the behavior then update logs next16:24
clarkbmordred: I expect swift will do what tosky wants since it has fewer built in assumptions like hiding htpassword behind a .16:25
*** dpawlik has joined #openstack-infra16:25
timburkefrom swift's perspective, definitely -- an object starting with a dot, or containing a '/.' anywhere in its path, is just another object16:26
corvusneutron done on all servers16:26
clarkb would be the section to modify16:27
timburkethough i seem to recall that things like the log hosting are actually on radosgw, in which case i have no idea about what extra limitations there may be16:27
corvusokay, all 3 repos complete on all 7 servers16:28
mordredcorvus: that is exciting!16:28
corvusi have +3d
mordredtimburke: woot!16:28
openstackgerritBen Nemec proposed openstack/pbr master: Read description file as utf-8
corvusnext step is to wait for that to go into effect, then run 'replication start -- all'16:29
openstackgerritMerged opendev/system-config master: Disable openid login and signup
corvusi'm going to afk for a bit while that works its way through16:30
openstackgerritClark Boylan proposed opendev/puppet-openstackci master: The logs for a job my contain dotfiles, index them
clarkbmordred: fungi tosky ^ we can test it out on logs-dev16:31
*** kjackal has quit IRC16:38
*** kskels has joined #openstack-infra16:39
kskelsHi infra team! I am having issue with airship/pegleg post job showing CONFIG_ERROR in the build set.. I can't seem to find any logs, etc.16:40
kskelsis there any tips you could share..16:40
toskyclarkb: thanks!16:40
toskynow I must confess than I don't know what logs-dev is used for :)16:41
openstackgerritMerged openstack/pbr master: Add openstack-tox-py37 job
*** armax has quit IRC16:41
clarkbtosky: it is a vhost that allows us to test changes to the vhost. Basically take any url and use instead to use it16:42
clarkbkskels: which post job?16:43
kskelsI believe it doesn't really even get to the post job16:43
kskelshere is the .zuul file so I've been adding the git mirror job additionally to exisitng post jobs16:44
kskelsthe config works well for ariship/treasuremap repo - so bit puzzled where to even look for error logs as it doesn't get to the job itself16:45
kskelsshows no builds16:45
*** zbr has quit IRC16:45
clarkbkskels: looks like you used the same secret in both repos, but the secret encryption key is repo specific16:46
toskyclarkb: oh, different frontend for the same data? Interesting16:47
clarkbmy hunch here is that pegleg can't decrypt the secret with its key16:47
clarkbtosky: yes16:47
kskelsoh.. you are right! :)16:47
kskelsthe cert encryption was indeed project specific16:47
kskelsthank you..16:47
mordredclarkb: seems like the sort of thing that should get reported somehwere - not 100% sure where though16:50
*** whoami-rajat has joined #openstack-infra16:51
*** roman_g has joined #openstack-infra16:51
mordredclarkb, corvus: maybe when someone submits a zuul config change with a secret in it, if the secret is undecryptable for that project, an error message about "this secret was unable to be decrypted" or something similar?16:52
*** Romik has quit IRC16:52
*** dtantsur is now known as dtantsur|afk16:54
clarkbmordred: ya. THe only question would be if that violates security concerns somehow (like when you login to a service and fail they don't tell you what you got wrong)16:55
*** ricolin has quit IRC16:55
*** tjgresha has quit IRC16:55
*** tjgresha has joined #openstack-infra16:55
openstackgerritMerged openstack/pbr master: Fix white space handling in file names
*** derekh has quit IRC17:00
*** pcaruana has joined #openstack-infra17:00
*** ykarel|away has quit IRC17:02
openstackgerritMerged openstack/pbr master: Set subparser argument required
*** udesale has quit IRC17:03
openstackgerritMerged opendev/system-config master: Push all refs to gitea
*** ginopc has quit IRC17:08
*** tosky has quit IRC17:08
*** Weifan has joined #openstack-infra17:10
*** tosky has joined #openstack-infra17:12
corvuswell that wasn't super fast, was it? :)17:12
mordredcorvus: I mean - it wasn't the fastest17:13
mordredbut it also could have been worse17:13
*** dpawlik has quit IRC17:17
*** jpena is now known as jpena|off17:19
*** tosky has quit IRC17:22
openstackgerritDirk Mueller proposed opendev/system-config master: Move openSUSE Tumbleweed into a caching mirror instead
openstackgerritDirk Mueller proposed opendev/system-config master: Start mirroring openSUSE Leap 15.1
*** tosky__ has joined #openstack-infra17:23
*** ykarel has joined #openstack-infra17:24
*** smarcet has joined #openstack-infra17:26
*** dpawlik has joined #openstack-infra17:26
*** hamzy has joined #openstack-infra17:28
*** e0ne has joined #openstack-infra17:29
*** auristor has quit IRC17:30
*** auristor has joined #openstack-infra17:33
*** sarob has joined #openstack-infra17:33
*** hamzy has quit IRC17:40
*** hamzy has joined #openstack-infra17:40
clarkbmordred: on we actually set force_ipv4 on vexxhost now too to fix the dns resolution errors that test jobs were having17:43
clarkbmordred: I don't know that we need to set that in that file because builders don't launch nodes and otherwise I expect it will be fine (whereas in rax there are glean considerations)17:43
clarkbmordred: but wanted to call that out before I revote and get some input17:44
clarkbinfra-root can you review which is our fixup change to the rename playbook. Important if we want to rename repos next week.17:45
*** hamzy_ has joined #openstack-infra17:46
*** hamzy has quit IRC17:49
corvusclarkb: done17:49
corvusand the new replication config just landed on gerrit17:50
clarkbI'm going to pop out for a bit to dig out some food then I'll be double checking on ask.o.o before I declare puppet-4 done17:50
*** ralonsoh has quit IRC17:51
clarkbI guess I can do an ansible command to check for which puppet && test -e /opt/puppetlabs17:51
corvusi just watched the gerrit replication queue and saw it replicate to openstack-helm-infra, and verified that there are now refs/changes refs for that repo on the gitea servers17:52
corvusso i think we're ready to trigger the really big replication17:52
evrardjpclarkb: could you by any chance reapply a +w on please?17:53
evrardjpor anyone, sorry, it was the name that was in the middle of my screen17:53
corvusinfra-root: i've started replication of all refs of all projects to all gitea servers; this may take a while and therefore we will likely see a replication lag until the process is complete17:56
corvusit started with ~18000 replication tasks17:57
*** lpetrut has joined #openstack-infra18:03
*** kskels has left #openstack-infra18:07
corvusetc 3.6 hours (~21:45 utc)18:07
*** lpetrut has quit IRC18:07
openstackgerritTobias Henkel proposed zuul/zuul master: Allow to select the merge method in GitHub
*** rlandy is now known as rlandy|brb18:16
openstackgerritMerged opendev/system-config master: Omnibus rename repo playbook fixes
*** hamzy__ has joined #openstack-infra18:26
*** hamzy_ has quit IRC18:26
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Use openstack-tox-docs for deploy-guide job
*** raissa has joined #openstack-infra18:30
*** ykarel has quit IRC18:32
*** hamzy__ has quit IRC18:34
*** smarcet has quit IRC18:35
*** mriedem has quit IRC18:37
*** smarcet has joined #openstack-infra18:37
openstackgerritAndreas Jaeger proposed openstack/project-config master: Use tox for publish-deploy-guide
*** rlandy|brb is now known as rlandy18:42
*** hamzy has joined #openstack-infra18:43
*** mriedem has joined #openstack-infra18:43
fungilooks like we're suddenly failing bashate "E042 local declaration hides errors" on
fungispecifically i think it's "current" and "elapsed_ms" it's bothered bu18:44
*** priteau has quit IRC18:44
*** hamzy has quit IRC18:47
*** electrofelix has quit IRC18:47
fungitrying to figure out what it doesn't like about them18:47
*** hamzy has joined #openstack-infra18:47
fungiaccording to the source code it means it doesn't like =$( assignment18:49
fungior =`18:49
fungithere have been no new releases of bashate for nearly a year so no idea why this is only showing up now18:50
fungiahh, it wants the local declaration to be done separately from the variable assignment18:52
*** e0ne has quit IRC18:53
fungithe lines it's complaining about in that script haven't been touched since september18:54
*** e0ne has joined #openstack-infra18:54
fungii'm mystified19:00
*** armstrong has joined #openstack-infra19:01
clarkbfungi: have a link to a log?19:01
clarkbfungi: but usually what it means is you have to do local foo\nfoo=$(ls)19:01
clarkbfungi: there is a bash behvaior issue where if you do local foo=$(some exec) you don't get the local behavior you expected on failure?19:02
clarkbsomething like that19:02
fungiand yeah, i see how to make it not complain, more wondering what caused it to suddenly start19:02
fungii don't think the version of bashate we're using has changed in almost a year, the lines it's complaining about haven't been touched in 8 months...19:03
mtreinishfungi: I don't think there has been a bashate update in over a year19:04
fungitox.ini was last altered a couple months back19:04
fungiand didn't touch the bashate invocation either19:04
clarkbfungi: I think those are just warnings?19:04
clarkb is the error failing the job I think19:04
*** hamzy has quit IRC19:05
fungi84 bashate warning(s) found, 2 bashate error(s) found19:05
*** tesseract has quit IRC19:05
fungioh! so it is19:06
fungiokay, so problem with that patch19:06
fungievrardjp: " E040 Syntax error: syntax error near unexpected token `distribution/leap/15.0/jeos'19:06
*** Adri2000 has quit IRC19:06
clarkbinfra-root cmurphy in reviewing if we have any remaining puppet-3 hosts and and are still puppet 3. The first two because they are arm64 hosts and the third because it is one we don't touch much19:06
clarkbI think we can call puppet-4 done when we update the backup server? Are we actively puppeting the backup server?19:06
openstackgerritTobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github
*** Adri2000 has joined #openstack-infra19:08
clarkbfungi: I don't know why those "errors" are treated as warnings. Maybe because those files aren't touched by the change or we've ignored those rules?19:08
fungii can't tell how you're expected to differentiate warnings from errors in bashate output19:08
fungiit's all "ENNN some text"19:09
fungilikely i'm just reading it wrong19:09
openstackgerritClark Boylan proposed opendev/system-config master: Remove groups configuration
openstackgerritClark Boylan proposed opendev/system-config master: Remove the inventory entry
clarkbfungi: ianw ^ can you review those since you have been involved in the related changes?19:13
clarkbnoticed those were still in the inventory when checking puppet-4 completeness19:13
clarkbAll three hosts are unreachable19:14
clarkbmissed some stuff will get new ps up19:18
openstackgerritClark Boylan proposed opendev/system-config master: Remove groups configuration
openstackgerritClark Boylan proposed opendev/system-config master: Remove the inventory entry
clarkbthat should be more complete19:20
clarkb(also I couldn't find existing changes for that stuff, but point me at them if they exist and I can abandon my changes)19:21
clarkbcorvus: we are down to 6.8GB of disk on gitea0119:26
*** hamzy has joined #openstack-infra19:26
clarkbthough 1.3GB of that is your temp dir19:27
clarkbso really 8.2GB free? I guess that isn't too bad19:27
fungii think we can survive on that until we rebuild them19:27
clarkbstill about 10k replication tasks to go19:28
clarkbbut ya I expect we'll get to a point that is still sustainable19:28
clarkbmordred: did you see my question earlier about the nodepool builder clouds change?19:28
clarkbour backup server is in the disabled group in system-config/inventory/groups.yaml19:29
clarkbso I'm not sure what the best way forward as far as puppet 4ing it is19:29
mordredclarkb: oh - yeah. I agree, I don't think it actually matters. BUT - maybe we should update it anyway for completeness19:31
mordredI can respin if you want19:31
clarkbmordred: ya I'm thinking that might ease confusion in the future if we wonder why it isn't consistent. So maybe a respin is worthwhile19:31
mordredkk. coming right up19:32
corvusclarkb: i'm about to delete the tempdirs, so we'll get some back soon19:32
openstackgerritTobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github
clarkbcorvus: ++19:33
openstackgerritMonty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml
openstackgerritTobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github
clarkbalso I think the flavor I had in mind for the rebuilt gitea nodes should be 4 cpu instead of 219:35
clarkbthese big replication events do want cpu available19:35
corvusclarkb: well, most of the time the load average is <119:36
clarkbya that is why I went with the 2vcpu flavor with 8GB ram (we do seem to use about 6-7GB of ram19:36
clarkbwe can likely get away with that it iwll just make replication in bulk even slower (which should happen infrequently)19:36
openstackgerritDirk Mueller proposed opendev/system-config master: Move openSUSE Tumbleweed into a caching mirror instead
openstackgerritDirk Mueller proposed opendev/system-config master: Start mirroring openSUSE Leap 15.1
corvusclarkb: even though our LA is high (>7) we're still 50% idle cpu (and somehow, almost no iowait?)  we might be able to increase the number of replication threads even with our current setup19:38
clarkbcorvus: ya the 50% of 8 vcpus is where I got the maybe it should be 4vcpus number19:38
corvusclarkb: oh, you said 4 instead of 2 :)19:39
*** e0ne has quit IRC19:39
clarkbcorvus: when you were out I had suggested a 2vcpu flavor19:40
clarkbbut now thinking 4 would be better19:40
corvusoh. i lacked that context.19:40
openstackgerritTobias Henkel proposed zuul/zuul master: Support squash merge in Github
clarkb(no new nodes have been booted yet as we are trying to spin up nodepool built minimal images for control plane)19:40
corvusyes, i don't think we should go down to 2.  4 might be okay.19:40
clarkbcorvus: basically 8GB memory seems about right based on cacti showing we use a lot of cache in memory. 8vcpu seemed like a lot. And we probably want double the disk19:41
clarkbalso I dug up info in the docs and to resize a volume you have to detach it from an instance first. So we can't resize the existing volumes I don't think19:43
clarkbBut I think we also decided we wanted to avoid that so that we could boot on new images with appropriately sized ext4 journals anyway19:43
clarkbso next steps here are merging mordred's changes to have nodepool build us control plane images, updating them to have bigger journals, then replacing gitea instances one by one19:43
corvusclarkb: do we have inappropriately sized journals?19:44
mordredclarkb: actually - next step is doing the secrets update on bridge19:44
mordredclarkb: but then yet, merging that change19:44
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: allow the use of non-bzip compressed stages for building gentoo
clarkbcorvus: logan- discovered that dib built images had/have a 64MB journal. mkfs.ext4 will use a much larger journal for the size of disks we are running. For 80GB filesystems the journal default in mkfs is 512MB19:44
mordredclarkb: shall I go ahead and copy the secrets?19:44
clarkbmordred: ya I think we should make that copy so we are ready when the changes go in19:45
mordredkk. copying now19:45
clarkbcorvus: at 128GB the journal is 1GB in size and that seems to be the size for anything bigger than that19:45
clarkb256MB for 32GB to 64GB fses19:45
clarkband it scales down to some number like 8MB for really small fses19:45
clarkbIt does make me wonder if the small journal may have contributed to gitea06's fs problems19:46
clarkba bigger journal wouldn't have made the problems go away but may have helped it recover?19:46
corvusdid we use a nodepool image for that?19:47
corvusor rather a dib image19:47
clarkbcorvus: aiui mordred used a locally built dib image (no nodepool but yes dib)19:47
fungicorvus: clarkb: also, having too small a journal for the fs churn implied significant i/o resource contention, i think?19:48
mordredyah. I've forgotten *why* I did that - there was a reason at the time. I think it might have been missing stuff in the stock bionic image and I didn't want to deal with trying to bootstrap package installs via cloud-init19:48
clarkbmordred: iirc you got mad at the broken transition to the network yaml configuration stuff19:48
*** gfidente has quit IRC19:48
clarkbmordred: so you built a new image that was post transition19:49
mordredinfra-root: cloud creds copied to group_vars/control-plane-clouds.yaml19:49
mordredclarkb: oh right!19:49
fungiso suspicion was small ext4 journal sizes chosed by dib were suspect in reduced performance we were seeing in limestone19:49
clarkbcorvus: latest dib allows us to set the journal size when fs is created19:50
clarkbcorvus: we now do that for our test node images and set it to 512MB19:50
clarkb(to match mkfs default)19:51
logan-fungi: not necessarily provider specific, but i'd suspect certain jobs may have been doing more IO churn than necessary due to journal contention19:56
logan-DIB basically just chooses an arbitrary number, 64MB, since it has no way to know what the actual filesystem size will be after the image is booted and growpart runs19:58
clarkbI imagine other cloud images do similar19:59
fungilogan-: i remember you had linked to something originally which had interesting things to say about i/o performance implications of under-sized journals, bit am now not finding it from
clarkbif someone had time to inspect say ubuntus images might be interesting but enotime19:59
fungidid i dream it?19:59
logan-it was on the project-config patch19:59
logan-and the link seems to have disappeared :(20:00
logan-the content at the link*20:00
fungiahh, right20:00
fungii wonder what they did with that article20:01
fungiit was indeed there at the time, i remember reading it20:01 has it
openstackgerritJames E. Blair proposed opendev/system-config master: Add cron to pack refs on gitea servers
corvusclarkb, fungi, mordred: ^ that should tie off the pack-refs work20:02
fungireminds me of something20:03
clarkbcorvus will look in a asec20:03
*** Goneri has quit IRC20:08
clarkbcorvus: fungi has a good quetion on the pack refs change20:08
*** armax has joined #openstack-infra20:09
fungiyeah, i wasn't able to find any *.t directories so am thinking it may just be a typo20:09
corvusheh, yeah, i think that was a copy-paste error20:12
openstackgerritJames E. Blair proposed opendev/system-config master: Add cron to pack refs on gitea servers
corvusprobably resized the terminal and lost some chars without noticing20:12
clarkbcorvus: did you check that git and find are both installed in the container image?20:14
clarkb(I don't actually know myself)20:15
corvusclarkb: yeah, i ran that on gitea0620:16
*** jamesmcarthur_ has quit IRC20:16
clarkbchange is approved20:16
clarkbanyone else able to review and child to put that stuff behind us?20:17
*** tosky__ is now known as tosky20:18
*** kjackal has joined #openstack-infra20:22
clarkbback down to ~6.7GB free on gitea0120:29
clarkbabout 9k replication events20:29
*** whoami-rajat has quit IRC20:31
corvusclarkb: oh, i forgot to remove my tarballs.  just did that, free on 01 is 7.8G now20:33
corvusthe others are all >8G20:33
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: allow the use of non-bzip compressed stages for building gentoo
*** dpawlik has quit IRC20:34
openstackgerritDavid Shrewsbury proposed zuul/zuul master: WIP: Store hold requests in zookeeper
*** slaweq has quit IRC20:41
*** pcaruana has quit IRC20:47
clarkbfungi: did you want to review the security impact of (re tosky's dotfile logs thing)20:50
openstackgerritMerged opendev/system-config master: Remove groups configuration
openstackgerritMerged opendev/system-config master: Remove the inventory entry
clarkbfungi: also maybe we should fix all of those warnings from bashate to make the errors more obvious? I'll take a stab at that cleanup20:53
*** nicolasbock has quit IRC20:58
*** nicolasbock has joined #openstack-infra20:58
*** kjackal has quit IRC21:00
openstackgerritClark Boylan proposed opendev/system-config master: Cleanup bashate errors to make them easier to understand
clarkbfungi: infra-root ^ that is a reduce confusion change by cleaning up bashate output21:02
fungiahh, thanks21:06
*** jtomasek has joined #openstack-infra21:12
openstackgerritAdam Spiers proposed openstack/cookiecutter master: Add a testenv for lower-constraints
*** raissa has quit IRC21:13
*** raissa has joined #openstack-infra21:13
*** raissa has quit IRC21:14
*** raissa has joined #openstack-infra21:14
*** raissa has quit IRC21:15
*** raissa has joined #openstack-infra21:15
*** raissa has quit IRC21:15
*** armstrong has quit IRC21:20
*** eharney has quit IRC21:20
dmsimardsome changes have been merged but aren't showing up on gitea ?21:24
clarkbdmsimard: we are doing a full sync to pick up refs/chagnes and refs/notes on gitea so there is a backlog21:25
dmsimard doesn't show several changes that merged today
*** jtomasek has quit IRC21:25
dmsimardclarkb: ah, okay, I thought I was going crazy there for a moment21:25
fungi9072 tasks remaining21:27
corvusdmsimard: you may want to add infra-root as a nick highlight if you haven't21:27
openstackgerritMerged openstack/project-config master: Create 'Backport-Candidate' for Octavia repos
*** smarcet has quit IRC21:35
dmsimardcorvus: ++, I've been a bit disconnected21:37
*** Goneri has joined #openstack-infra21:38
openstackgerritMerged openstack/project-config master: End Project Gating
corvuswe should probably ask for project names in commit messages like that ^ :)21:38
*** rh-jelabarre has quit IRC21:39
openstackgerritMerged openstack/project-config master: Retire ansible-role-k8s-* repos
dmsimardmade me look too.21:40
*** bobh has joined #openstack-infra21:42
openstackgerritMerged openstack/project-config master: Switch bindep fallback testing to openSUSE Leap 15.x
clarkbcorvus: mordred want to review a change from pabelanger ot trim docker image sizes a bit21:43
openstackgerritMerged opendev/puppet-openstackci master: The logs for a job my contain dotfiles, index them
corvusi kind of want to say that maybe we talked about keeping the caches around in the builder image to make building more efficient21:44
corvuslet's hold approval of that for mordred's review21:45
*** bobh has quit IRC21:45
corvusokay i guess mordred is okay with it :)21:45
mordredwe need the pip cache for the packages the builder image installs21:45
mordredbut not the pip cache or the apt-cache for bindep itself21:46
clarkbmordred: corvus that change removes /var/lib/apt-lists21:46
mordredso I think those are alright21:46
mordredthe followup removes pip cache for bindep21:46
clarkbfungi: can you squash into its parent then we can merge that in for historical purposes?21:46
corvusmordred: you may want to +W it then21:46
mordredincidentally, once we've got better testing of the image build jobs, there are some techniques we can start using to make this whole "don't forget to delete dirs" thing easier21:47
corvusyeah, i'm planning on picking up work on image testing next week21:47
* clarkb is trying to clear review backlogs21:48
fungiclarkb: what's the reason for the squash? will it not merge otherwise? happy to regardless21:49
clarkbfungi: the commit message says "This should be squashed into its parent"21:49
clarkbI don't know why corvus thought that was necessary21:49
fungiwell, for actual running anyway21:50
fungibut sure, will combine them21:50
clarkb clean bashate \o/21:55
fungidoes 653138 crash anyone else's gertty?21:58
corvusnot i21:59
corvuswhat's the traceback?21:59
*** armax has quit IRC22:03
clarkbcorvus: do we need to maybe consider packing refs in the foreground to free up more disk on the gitea servers? (back down to 6.2GB with about 8800 replication tasks to go)22:05
clarkb(its hard to gauge how much disk is necessary via the replication tasks count since each repo isn't the same)22:05
corvusclarkb: hrm, i wouldn't expect it to be a problem based on df in /opt on review0122:06
clarkbcorvus: I think those repos are all packed22:07
*** armax has joined #openstack-infra22:07
*** Goneri has quit IRC22:08
openstackgerritMerged opendev/system-config master: Clean up apt cache
corvusclarkb: ah, i think you're right and they are packed periodically (probably repacked)22:11
corvuswe may want to look into doing repacks on gitea instead of pack-refs actually22:11
corvusclarkb: think i should just run the cron pack-refs command?22:12
clarkbcorvus: ya pack files are compressed so I'm thinking that should free up a bit of disk (which I think we want given its hard to tell how much headroom we have)22:12
corvusclarkb: no this is just pack-refs22:12
corvussorry, there's 2 things under discussion and streams may be getting crossed22:13
clarkboh pack-refs only sticks the ref pointers in a file and doesn't pack objects got it22:13
*** mriedem has quit IRC22:13
corvusyeah, and pack-refs is the only thing we've done on gitea servers so far22:13
clarkbI think we need to pack objects or we may run out of disk sometime between now and 8k replication events from now22:13
clarkb(it is possible that won't happen as I said very hard to tell)22:14
corvusany idea where the ref packing for the /opt mirrors happens?22:14
*** rfarr_ has quit IRC22:14
clarkbya one sec22:15
corvusah found it22:15
corvus17 4 * * 0 find /opt/lib/git -type d -name "*.git" -print -exec git --git-dir="{}" gc \;22:15
fungicrontab for gerrit2 user22:15
clarkbhrm I can't link it because opendev saying things like secure connection failed22:16
*** sreejithp__ has quit IRC22:16
fungii guess git gc also packs objects?22:16
clarkbfungi: yes22:16
corvusand packs refs22:16
*** sreejithp has joined #openstack-infra22:16
clarkb broken for anyone else or just me?22:16
fungitaking a while to load for me22:17
fungii wonder if i'm having ipv6 connectivity issues22:18
logan-doesn't load for me, also took a good 30 seconds or so22:18
logan-but / is fast upon refresh22:18
logan-system-config loaded after a minute or two22:19
corvuswe may be running into contention with the replication22:19
fungieventually loaded for me22:19
fungino errors22:20
corvus is erroring22:20
fungidoesn't seem to be a v6 to v4 fallback delay, i can connect over v6 to it just fine22:20
clarkbthere is/was a container restart22:20
clarkbon gitea0122:20
clarkbso that may be spinup time?22:20
fungioh, maybe for the openid disablement?22:21
corvus67693ca76264        opendevorg/gitea:latest    "/usr/bin/entrypoint…"   5 minutes ago       Restarting (2) 53 seconds ago                       giteadocker_gitea-web_122:21
corvusthats 0722:21
corvusso yeah, that's probably what's happening22:21
corvusmaybe we should orchestrate restarts with haproxy22:21
corvus(since we can theoretically do that now with ansible)22:21
clarkbcorvus: also something freed up some disk (did you run the pack-refs command?)22:22
corvusclarkb: nop22:22
clarkbI wonder if gitea does stuff like that on startup22:22
fungiwhich could possibly introduce startup delays22:22
corvusi dunno what it does on startup.. 07 looks idle22:23
corvuspanic: Failed to execute 'git config --global core.quotepath false': error: could not lock config file /data/git/.gitconfig: File exists22:23
corvuspossible side effect of the container being stopped in the middle of a git command running?22:24
clarkbor running the fetch commands earlier?22:24
corvusit's only happening on one host though22:24
clarkbgitea01's .gitconfig is super simple22:25
clarkbI half expect we can move that aside on gitea07 and it would be happy22:25
corvusit's the lock file it's complaining about22:25
corvusthere are no long-running git processes running on gitea07 so i will just remove the .lock file22:26
clarkb.gitconfig.lock for anyone else wondering (in the same dir)22:26
clarkbcorvus: oh ya on 01 that .lock file has come and gone22:27
clarkbso guessing if we stop the container when the .lock is present it gets unhappy on startup22:27
corvus07 is starting up now22:27
corvusand running22:27
clarkbmaybe our init container needs to rm that file before starting gitea?22:28
openstackgerritMerged opendev/system-config master: Use --no-cache-dir with pip
openstackgerritJames E. Blair proposed opendev/system-config master: Add cron to gc on gitea servers
corvusclarkb, fungi: ^ shall i run that now on all gitea hosts?22:31
fungiseems prudent, yes22:32
clarkbshould we test it on one first?22:32
openstackgerritJeremy Stanley proposed opendev/system-config master: Add opendev migration repo rename scripts
clarkb(mostly curious if gitea will be happy with that happening under it)22:32
corvussure i'll do that on 0722:32
clarkbsounds good22:32
openstackgerritJames E. Blair proposed opendev/system-config master: Add cron to gc on gitea servers
clarkbcorvus: re ^ you might also want to drop the -t since ansible may not ssh with a tty?22:35
corvusclarkb: "-t" works from the command line on bridge; "-it" does not22:36
corvuscron could be different, but i think this might work22:36
clarkb07 is slow (due to load I expect) but I can request opendev/system-config from it and it renders pages22:37
corvusclarkb: it actually *just* finished gcing system-config22:37
corvusand yeah, still seems to work for me...22:37
clarkbthat could explain it :) but ya I was able to browse around system-config so I think it probably is happy with the GCing under it22:38
corvusclarkb: happily, you may have even read from the repo during the gc which is a great test22:38
clarkb(if a little slow when also doing the replicate world step)22:38
clarkband that is freeing up disk on 0722:38
corvusi will run the same on the other hosts now22:39
*** sreejithp has quit IRC22:39
clarkbnote that 06 may still be quite unhappy so errors there probably don't mean much22:40
corvusoh yeah, i won't do it on 0622:40
corvusi also didn't do the fetches on 0622:40
corvusreplication is very fast on 06; it seems to have been failing fast22:41
corvusgc running everywhere now22:41
clarkbI expect this will make disk usage much more closely match review.o.o:/opt which should mean we'll have plenty of disk22:45
fungialso possible pushes are way faster when there's no ci system making constant fetch requests22:45
clarkbgitea01 is doing the openstack ansible repos now (for a sense of where in the alphabetical queueing it is)22:47
toskyclarkb: should I already see the new setting on
clarkbtosky: I think it is delayed due to the replication backlog in gerrit22:51
clarkbtosky: so it might be a little while before it applies (I checked on the server and the config isn't applied yet)22:51
clarkbunder 8k replications events finally :)22:52
toskyclarkb: oh, that's fine, I was worrying that it didn't work :)22:54
toskyI will recheck in few hours (my morning)22:54
*** tkajinam has joined #openstack-infra22:57
*** smarcet has joined #openstack-infra22:57
*** panda|rover has quit IRC22:59
*** panda has joined #openstack-infra22:59
*** harlowja has quit IRC23:01
*** michael-beaver has quit IRC23:03
*** tosky has quit IRC23:05
*** rlandy has quit IRC23:06
clarkbnow down to 7.7k replication tasks. Seems to be moving a bit quicker now (possibly because it got through most of the big repos)23:06
*** sarob has quit IRC23:07
clarkb07 is done GC'ing23:11
clarkbended up with 11GB free23:11
clarkber no 13. 01 has 11 and is still running23:11
clarkbso that looks a lot more like what we expect23:12
clarkbwe probably want to rerun the GC'ing after the replication completes (as things that haven't yet replicated won't be packed)23:13
clarkbdepending on how much disk space we end up with in the end it may be fine to wait for the crno to run23:14
openstackgerritTristan Cacqueray proposed zuul/zuul master: encrypt_secret: display the full_url on error
*** rcernin has joined #openstack-infra23:18
*** smarcet has quit IRC23:19
*** Weifan has quit IRC23:22
*** _erlon_ has quit IRC23:24
clarkbGC has ended on gitea01 now too so is probably done nearly across the board23:27
*** smarcet has joined #openstack-infra23:27
*** michael-beaver has joined #openstack-infra23:35
openstackgerritIan Wienand proposed zuul/zuul-jobs master: validate-zone-db : add job and make more generic
openstackgerritIan Wienand proposed openstack-infra/ master: Add zone-check job
openstackgerritIan Wienand proposed openstack-infra/ master: [dnm] sample of zone misconfiguration to fail testing
ianwfungi: i've found that when i save a zone.db file emacs bumps the serial number automagically23:46
clarkbunder 7k replication events now23:47
clarkbthings are looking good now on that front. I'm about to pop out for dadops things23:48
clarkbianw: so I did a sanity check that we didn't miss any puppet-3 nodes. The two arm64 instances (mirror and builder) are puppet3. Your replacement work for the mirrors will handle one of those and the last instance I found was our backup host23:48
openstackgerritIan Wienand proposed zuul/zuul-jobs master: validate-zone-db : add job and make more generic
clarkbianw: the backup host is currently in the disabled group in inventory/groups.yaml implying we don't touch it often. Any idea if that should be updated to puppet-4?23:49
clarkbor is that a task for the next time we build a backup host?23:49
ianwclarkb: yeah, i think that's on purpose because we don't want any risk of it being wiped by some automagic23:49
ianwbut ? ... it's just always been like that since i got here :)23:49
clarkbok I'll have to think about that one then23:50
clarkbianw: also not sure if you saw the email from linaro. TLDR is someone new will be working with us in the next few weeks (probably) and they want to spin up a new cloud hosted on packethost23:50
*** mattw4 has quit IRC23:50
clarkboh and they have more capacity in london if we want to push on that more23:50
ianwwe could make a new backup host ... does it do more than install bup really?  seems ansible-isable23:51
*** trident has quit IRC23:51
clarkbianw: I think it is install users and firewall and bup so really basic23:51
clarkband now I must walk the kids to the playground to keep them from falling asleep early23:52
ianwclarkb: yeah, saw that, will take a look at where we're at.23:52
ianwclarkb: did you see ?23:52
clarkbianw: I did23:52
ianwi'll attach some volumes to make it have space similar to the other mirrors; then i think we can probably CNAME it into rotation and see how it goes for a bit23:53
ianw(when we're around to monitor it :)23:53
*** trident has joined #openstack-infra23:53
clarkbianw: then we can add vhosts for tls'd docker hub proxy?23:53
ianwyep, i mean could do that now and just point everything at the one mirror for testing?23:54
clarkbianw: ya that might be a good idea.23:54
clarkbget tseting out of the way early as possible23:54
ianwi'll ad the extra volumes and lvm setup today and we can take it from there23:54
*** Weifan has joined #openstack-infra23:57
*** lseki has quit IRC23:58
*** Weifan has quit IRC23:59
*** Weifan has joined #openstack-infra23:59

Generated by 2.15.3 by Marius Gedminas - find it at!