Tuesday, 2019-05-21

clarkbok my day is about over now. Sorry I wasn't very useful beyond doing some reviews and cleaning out the cgit cluster00:01
clarkbmordred: do you intend on replacing status.o.o on xenial node still? If not I may try to pick thatup this week00:02
openstackgerritIan Wienand proposed opendev/system-config master: Create opendev mirrors  https://review.opendev.org/65828100:02
*** jamesmcarthur has joined #openstack-infra00:04
*** sshnaidm|afk has quit IRC00:04
*** sshnaidm has joined #openstack-infra00:06
*** panda|rover has quit IRC00:10
*** panda has joined #openstack-infra00:11
mordredclarkb: yes - I do - but if you run out of things to do before I get to it, I won't be offended00:11
clarkbmordred: k I may focus on gitea06 replacement first then00:12
clarkbianw: opendev mirrors change looks great00:12
ianwclarkb: one other slightly related one is the updates in zone file for ssl proposed by fungi -> https://review.opendev.org/#/c/652280/00:13
ianwif that's all ok, will follow that for mirror nodes too from start00:14
fungiand https://review.opendev.org/652281 to turn on hsts00:14
*** smarcet has joined #openstack-infra00:16
ianwfungi: https://review.opendev.org/#/c/658281/30/playbooks/roles/mirror/templates/mirror.vhost.j2 should contain the ssl config you proposed as standard too00:16
openstackgerritMerged openstack-infra/zone-opendev.org master: Add RFC 6844 CAA RR for graphite01  https://review.opendev.org/65228000:17
clarkbfungi: ianw for https://review.opendev.org/#/c/652281/1 considering the validity period of the ssl cert itself is only 90 days should we use a more conservative period for insisting on ssl?00:17
clarkbI guess reducing the period increases the attack surface as it has to renew that ttl?00:18
*** sreejithp has joined #openstack-infra00:18
openstackgerritJimmy McArthur proposed zuul/zuul-website master: Revert "Revert promotional message banner and event list"  https://review.opendev.org/66023200:19
ianwclarkb: yeah i think the less you check that flag, the better00:20
ianwalthough now i think about it, i wonder if with the mirrors maybe we don't want it, if we get weird interactions where we're still specifying http:// and then it's saying only use https://00:21
fungiclarkb: the idea with setting it to a year is that it only matters you can age out that setting if you decide that you want to switch the site from https to http later00:21
ianwnot sure if apt etc even obey it00:21
clarkbfungi: yup which you might do if you cannot renew the cert00:21
fungiconversely, the longer you make it, the more it reduces the opportunity for downgrade attacks00:22
openstackgerritIan Wienand proposed opendev/system-config master: Create opendev mirrors  https://review.opendev.org/65828100:24
ianwclarkb: i'll bring up, say, a rax dfw node for an initial opendev test mirror?  i guess the process is launch.py it, add to opendev.org zone, add to inventory.yaml00:26
fungianyway, hsts for graphite is not really critical, i was using that change more as an example00:27
clarkbyup shouldn't be any different from older launches except for the dns management00:27
fungifor services we know we won't ever want to serve over http for safety reasons, it could be a nice little extra bit of insurance00:27
clarkbfollowup ps adds the file that I should've checked was added before I +2'd the last ps :)00:30
openstackgerritTristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0""  https://review.opendev.org/65999100:32
*** sreejithp has quit IRC00:32
*** jamesmcarthur has quit IRC00:35
clarkband with that I need to call it a day. ianw thank you for pushing on this stuff00:37
ianwclarkb: thanks!  later00:37
openstackgerritMerged opendev/puppet-graphite master: Enable RFC 6797 HSTS signaling  https://review.opendev.org/65228100:39
*** smarcet has quit IRC00:39
*** markvoelker has joined #openstack-infra00:41
ianwMay 21 00:41:06 mirror01 systemd-resolved[988]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP00:41
ianwi dunno what that means, during launching a node00:41
clarkbmaybe it tried to resolve itself and failed?00:42
*** markvoelker has quit IRC00:45
*** mattw4 has quit IRC00:46
fungiwhich also mentions https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/182241600:48
openstackLaunchpad bug 1818527 in systemd (Ubuntu) "duplicate for #1822416 Stub resolver cache is corrupted" [Undecided,Confirmed]00:48
openstackgerritIan Wienand proposed openstack-infra/zone-opendev.org master: Add mirror.dfw.rax.opendev.org  https://review.opendev.org/66023500:52
*** zhurong has joined #openstack-infra00:52
ianwhttp://logs.openstack.org/13/660013/1/gate/openstackci-beaker-puppet-4/acc41ed/job-output.txt.gz#_2019-05-20_21_46_46_358317 gpg failure again; not sure if the long key patch was merged though00:59
openstackgerritIan Wienand proposed opendev/system-config master: Bringup mirror01.dfw.rax.opendev.org  https://review.opendev.org/66023701:00
*** jamesmcarthur has joined #openstack-infra01:05
openstackgerritIan Wienand proposed opendev/system-config master: Create opendev mirrors  https://review.opendev.org/65828101:09
openstackgerritIan Wienand proposed opendev/system-config master: Bringup mirror01.dfw.rax.opendev.org  https://review.opendev.org/66023701:09
*** jamesmcarthur_ has joined #openstack-infra01:10
*** jamesmcarthur has quit IRC01:11
ianwone thing with running the playbooks in a loop is that i don't think we get streaming output during the testing01:26
*** rh-jelabarre has quit IRC01:31
*** roman_g has quit IRC01:38
openstackgerritMerged opendev/system-config master: launch.py : fix typo calling legacy dns print function  https://review.opendev.org/66001301:38
*** gregoryo has joined #openstack-infra01:50
openstackgerritTristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0""  https://review.opendev.org/65999101:53
openstackgerritTristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0""  https://review.opendev.org/65999101:58
*** whoami-rajat has joined #openstack-infra02:00
*** hwoarang has quit IRC02:04
*** hwoarang has joined #openstack-infra02:05
*** sreejithp has joined #openstack-infra02:13
*** sreejithp_ has joined #openstack-infra02:15
*** nicolasbock has quit IRC02:16
*** sreejithp has quit IRC02:19
*** ykarel|away has joined #openstack-infra02:29
*** ricolin has joined #openstack-infra02:44
*** ykarel|away is now known as ykarel02:44
openstackgerritTristan Cacqueray proposed zuul/zuul master: DNM: this should cause a React exception  https://review.opendev.org/66023102:51
*** dave-mccowan has quit IRC02:55
*** hongbin has joined #openstack-infra03:00
*** tkajinam has quit IRC03:03
*** tkajinam has joined #openstack-infra03:04
*** kaisers has quit IRC03:17
*** tkajinam has quit IRC03:20
*** jamesmcarthur_ has quit IRC03:23
*** psachin has joined #openstack-infra03:28
*** tkajinam has joined #openstack-infra03:28
*** bhavikdbavishi has joined #openstack-infra03:42
*** udesale has joined #openstack-infra03:47
*** yolanda_ has quit IRC03:49
*** jamesmcarthur has joined #openstack-infra03:54
*** armax has quit IRC03:54
*** jamesmcarthur has quit IRC03:59
*** sreejithp_ has quit IRC04:03
*** zerick has quit IRC04:03
*** zerick has joined #openstack-infra04:04
*** hongbin has quit IRC04:21
*** markvoelker has joined #openstack-infra04:35
*** janki has joined #openstack-infra04:37
*** markvoelker has quit IRC04:39
*** ramishra has joined #openstack-infra04:40
*** slaweq has joined #openstack-infra04:49
*** yboaron_ has joined #openstack-infra04:57
*** slaweq has quit IRC04:59
*** jamesmcarthur has joined #openstack-infra05:01
*** pcaruana has joined #openstack-infra05:02
*** ykarel is now known as ykarel|away05:03
*** raukadah is now known as chandankumar05:06
*** jamesmcarthur has quit IRC05:07
*** ykarel|away has quit IRC05:08
*** pcaruana has quit IRC05:11
*** slaweq has joined #openstack-infra05:16
AJaegerconfig-core, please review https://review.opendev.org/659963 and https://review.opendev.org/659978 https://review.opendev.org/66002205:18
*** slaweq has quit IRC05:20
*** ykarel|away has joined #openstack-infra05:20
*** Lucas_Gray has joined #openstack-infra05:23
*** e0ne has joined #openstack-infra05:23
*** e0ne has quit IRC05:24
*** bhavikdbavishi1 has joined #openstack-infra05:28
*** bhavikdbavishi has quit IRC05:28
*** bhavikdbavishi1 is now known as bhavikdbavishi05:28
*** ykarel|away is now known as ykarel05:28
*** quiquell has joined #openstack-infra05:38
*** quiquell has quit IRC05:38
*** quiquell has joined #openstack-infra05:39
*** imacdonn has quit IRC05:40
*** imacdonn has joined #openstack-infra05:41
*** ykarel_ has joined #openstack-infra05:42
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Fix broken setRefs whith missing objects  https://review.opendev.org/62166705:43
*** udesale has quit IRC05:44
*** ykarel has quit IRC05:44
*** udesale has joined #openstack-infra05:44
openstackgerritTobias Henkel proposed zuul/zuul master: Match tag items against containing branches  https://review.opendev.org/57855705:47
openstackgerritTobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches  https://review.opendev.org/64027205:47
openstackgerritTobias Henkel proposed zuul/zuul master: Update cached repo during job startup only if needed  https://review.opendev.org/64822905:53
openstackgerritTobias Henkel proposed zuul/zuul master: Add support for smart reconfigurations  https://review.opendev.org/65211405:57
openstackgerritTobias Henkel proposed zuul/zuul master: Add --check-config option to zuul scheduler  https://review.opendev.org/54216005:57
*** ykarel__ has joined #openstack-infra06:01
*** ykarel_ has quit IRC06:03
*** ykarel__ is now known as ykarel06:04
*** dangtrinhnt has quit IRC06:04
*** zbr_ has quit IRC06:05
*** dangtrinhnt has joined #openstack-infra06:06
*** bhavikdbavishi has quit IRC06:10
*** bhavikdbavishi has joined #openstack-infra06:12
*** slaweq has joined #openstack-infra06:12
*** lpetrut has joined #openstack-infra06:15
*** Lucas_Gray has quit IRC06:18
*** ramishra has quit IRC06:19
*** jistr is now known as jistr|mtg06:20
openstackgerritTobias Henkel proposed zuul/zuul master: Match tag items against containing branches  https://review.opendev.org/57855706:20
openstackgerritTobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches  https://review.opendev.org/64027206:20
*** yboaron_ has quit IRC06:23
openstackgerritTobias Henkel proposed zuul/zuul master: Update cached repo during job startup only if needed  https://review.opendev.org/64822906:26
*** kaisers has joined #openstack-infra06:34
*** jamesmcarthur has joined #openstack-infra06:35
*** markvoelker has joined #openstack-infra06:36
*** jamesmcarthur has quit IRC06:39
ianw#status log ask.openstack.org migrated to new xenial server06:39
openstackstatusianw: finished logging06:39
ianwi have had to manually redo the solr indexes; it doesn't seem puppet handles this (maybe a periodic update might have got it?)06:40
ianwanyway, it seems to be working enough ...06:40
*** ramishra has joined #openstack-infra06:51
*** pgaxatte has joined #openstack-infra06:55
*** ccamacho has joined #openstack-infra06:56
*** sreejithp has joined #openstack-infra07:03
*** dpawlik has joined #openstack-infra07:03
*** trident has quit IRC07:03
*** trident has joined #openstack-infra07:05
*** tesseract has joined #openstack-infra07:06
*** sreejithp has quit IRC07:07
*** ginopc has joined #openstack-infra07:08
*** markvoelker has quit IRC07:09
openstackgerritMerged openstack/project-config master: Remove unused job publish-openstack-sphinx-docs-direct  https://review.opendev.org/65996307:09
*** kopecmartin|off is now known as kopecmartin07:12
openstackgerritTristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint  https://review.opendev.org/62134407:18
*** roman_g has joined #openstack-infra07:18
*** rcernin has quit IRC07:19
*** ralonsoh has joined #openstack-infra07:19
*** pcaruana has joined #openstack-infra07:21
*** jamesmcarthur has joined #openstack-infra07:25
*** piotrowskim has joined #openstack-infra07:26
*** apetrich has joined #openstack-infra07:28
*** jamesmcarthur has quit IRC07:30
*** ramishra has quit IRC07:33
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org: switch backup user to ask01-bup  https://review.opendev.org/66026907:34
*** ramishra has joined #openstack-infra07:34
*** amoralej|off is now known as amoralej07:35
*** ykarel is now known as ykarel|lunch07:35
*** udesale has quit IRC07:35
*** udesale has joined #openstack-infra07:36
*** pcaruana has quit IRC07:39
*** yikun_ has joined #openstack-infra07:40
*** tosky has joined #openstack-infra07:41
*** rpittau|afk is now known as rpittau07:41
*** e0ne has joined #openstack-infra07:42
*** bhavikdbavishi has quit IRC07:51
*** jpena|off is now known as jpena07:56
*** e0ne has quit IRC07:57
*** happyhemant has joined #openstack-infra07:58
*** e0ne has joined #openstack-infra08:01
*** markvoelker has joined #openstack-infra08:06
*** ramishra_ has joined #openstack-infra08:08
*** lucasagomes has joined #openstack-infra08:08
*** ramishra has quit IRC08:11
*** tkajinam has quit IRC08:12
*** ramishra_ is now known as ramishra08:15
*** diablo_rojo has joined #openstack-infra08:17
openstackgerritMerged opendev/system-config master: Add #starlingx to statusbot channels  https://review.opendev.org/65965208:19
openstackgerritMerged opendev/system-config master: ask.openstack.org: switch backup user to ask01-bup  https://review.opendev.org/66026908:19
*** dtantsur|afk is now known as dtantsur08:20
*** priteau has joined #openstack-infra08:20
*** priteau has quit IRC08:24
*** betherly has joined #openstack-infra08:25
*** priteau has joined #openstack-infra08:28
*** ykarel|lunch is now known as ykarel08:30
*** xek has joined #openstack-infra08:36
*** xek has quit IRC08:38
*** markvoelker has quit IRC08:39
openstackgerritMerged opendev/system-config master: Remove ask-staging* from disabled list  https://review.opendev.org/65942108:39
openstackgerritMerged opendev/system-config master: ask.o.o : workaround old puppet-solr package  https://review.opendev.org/65947308:39
*** dabukalam has joined #openstack-infra08:41
zigoclarkb: Oh, great, thanks ! :)08:42
*** derekh has joined #openstack-infra08:43
*** jistr|mtg is now known as jistr08:46
*** xek has joined #openstack-infra08:48
*** bhavikdbavishi has joined #openstack-infra08:56
openstackgerritStephen Finucane proposed openstack/pbr master: Add openstack-tox-py37 job  https://review.opendev.org/61065908:56
openstackgerritStephen Finucane proposed openstack/pbr master: Resolve some issue with tox.ini, setup.cfg  https://review.opendev.org/66029708:56
openstackgerritMatthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions  https://review.opendev.org/57690709:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI  https://review.opendev.org/63619709:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST  https://review.opendev.org/63631509:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration  https://review.opendev.org/63985509:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Web: plug the authorization engine  https://review.opendev.org/64088409:02
*** diablo_rojo has quit IRC09:03
openstackgerritMatthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.opendev.org/64109909:03
openstackgerritMatthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry  https://review.opendev.org/64240809:03
*** sreejithp has joined #openstack-infra09:03
*** dtantsur is now known as dtantsur|brb09:06
*** sreejithp has quit IRC09:07
*** openstackstatus has quit IRC09:13
*** openstackstatus has joined #openstack-infra09:14
*** ChanServ sets mode: +v openstackstatus09:14
*** gregoryo has quit IRC09:15
*** pkopec has joined #openstack-infra09:21
*** jamesmcarthur has joined #openstack-infra09:26
*** electrofelix has joined #openstack-infra09:30
*** jamesmcarthur has quit IRC09:31
*** markvoelker has joined #openstack-infra09:35
*** diablo_rojo has joined #openstack-infra09:38
*** ramishra has quit IRC09:46
diablo_rojoMaybe a dumb question/ situation, but bear with me. I used to have ssh access to storyboard-dev. I *thought* that I had transferred my ssh key from my old laptop to my new one, but I legit cant find the file in system-config where that info had been kept. Am I blind? Or did that all change?09:48
*** ramishra has joined #openstack-infra09:49
*** pkopec has quit IRC10:08
*** dpawlik has quit IRC10:09
*** markvoelker has quit IRC10:09
*** dpawlik has joined #openstack-infra10:09
*** pkopec has joined #openstack-infra10:10
*** Lucas_Gray has joined #openstack-infra10:11
openstackgerritStephen Finucane proposed openstack/pbr master: Use more verbose logging for test failure  https://review.opendev.org/66033910:14
openstackgerritMark Goddard proposed opendev/irc-meetings master: Update Kolla meeting chair to mgoddard  https://review.opendev.org/66034410:26
*** dtantsur|brb is now known as dtantsur10:30
*** udesale has quit IRC10:46
*** amoralej is now known as amoralej|lunch11:01
*** snapiri has quit IRC11:03
*** markvoelker has joined #openstack-infra11:06
*** panda is now known as panda|rover|eat11:13
*** nicolasbock has joined #openstack-infra11:15
*** sreejithp has joined #openstack-infra11:20
*** dave-mccowan has joined #openstack-infra11:26
*** jamesmcarthur has joined #openstack-infra11:27
*** jamesmcarthur has quit IRC11:33
*** markvoelker has quit IRC11:39
*** sreejithp has quit IRC11:40
*** sreejithp has joined #openstack-infra11:40
*** jamesmcarthur has joined #openstack-infra11:43
*** sreejithp has quit IRC11:44
*** bobh has joined #openstack-infra11:45
*** bobh has quit IRC11:46
openstackgerritThomas Goirand proposed openstack/project-config master: Add a Debian Buster image.  https://review.opendev.org/64557411:47
*** rascasoft has quit IRC11:52
*** janki has quit IRC11:54
*** jamesmcarthur has quit IRC11:55
*** jamesmcarthur has joined #openstack-infra12:00
*** jpena is now known as jpena|lunch12:00
mordreddiablo_rojo: https://opendev.org/opendev/system-config/src/branch/master/playbooks/group_vars/all.yaml#L153-L15712:02
mordreddiablo_rojo: and https://opendev.org/opendev/system-config/src/branch/master/playbooks/group_vars/storyboard-dev.yaml#L412:03
mordreddiablo_rojo: so I agree - you should have access to storyboard-dev, assuming you have the ssh key - but if you don't, there's where you can update the public key12:03
*** sreejithp has joined #openstack-infra12:06
*** sreejithp has quit IRC12:12
diablo_rojomordred, thank you! I will get that updated.12:15
*** jamesmcarthur has quit IRC12:16
*** jamesmcarthur has joined #openstack-infra12:16
*** udesale has joined #openstack-infra12:16
*** bhavikdbavishi has quit IRC12:23
*** amoralej|lunch is now known as amoralej12:25
pabelangerclarkb: re kata / github yesterday, I can confirm I was seeing delayed events from github yesterday, sometime upwards of 20mins.12:27
pabelangerso, maybe zuul.o.o was seeing the same thing from kata PRs12:27
*** panda|rover|eat is now known as panda|rover12:27
openstackgerritMerged opendev/storyboard master: Clarify the rationale for StoryBoard's unique design  https://review.opendev.org/65763312:30
*** pkopec has quit IRC12:31
*** jamesmcarthur has quit IRC12:32
*** diablo_rojo has quit IRC12:33
*** lseki has joined #openstack-infra12:34
*** psachin has quit IRC12:34
*** priteau has quit IRC12:36
*** rlandy has joined #openstack-infra12:37
*** priteau has joined #openstack-infra12:39
*** jamesmcarthur has joined #openstack-infra12:42
*** Goneri has joined #openstack-infra12:43
openstackgerritMonty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml  https://review.opendev.org/64002712:44
openstackgerritMonty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files  https://review.opendev.org/66037412:44
mordredclarkb: ^^ I updated the control plane patch to match current reality - then made a followup patch to clean up the unused clouds from everywhere12:45
*** rfarr has joined #openstack-infra12:45
openstackgerritMonty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml  https://review.opendev.org/64002712:49
openstackgerritMonty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files  https://review.opendev.org/66037412:49
*** rascasoft has joined #openstack-infra12:50
*** priteau has quit IRC12:51
openstackgerritMonty Taylor proposed openstack/project-config master: Build ubuntu bionic minimal images for control plane  https://review.opendev.org/64004412:51
mordredclarkb, AJaeger: ^^ I also updated the nodepool config file patch that depends on it to rename the cloud entries. "openstackci" seems like an antiquated way of saying "control plane" - so I thought maybe just saying "control-plane" would be better12:52
*** jpena|lunch is now known as jpena12:52
*** rascasoft has quit IRC12:56
*** tobias-urdin has quit IRC13:03
*** pkopec has joined #openstack-infra13:05
*** mriedem has joined #openstack-infra13:07
*** cgoncalves has quit IRC13:08
*** eharney has quit IRC13:11
*** janki has joined #openstack-infra13:11
*** hwoarang has quit IRC13:21
openstackgerritMonty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files  https://review.opendev.org/66037413:22
openstackgerritMonty Taylor proposed opendev/system-config master: Add the mirror-only cloud regions to nodepool builder  https://review.opendev.org/66038413:22
*** hwoarang has joined #openstack-infra13:23
*** cgoncalves has joined #openstack-infra13:23
*** sreejithp has joined #openstack-infra13:26
openstackgerritMonty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files  https://review.opendev.org/66037413:30
openstackgerritMonty Taylor proposed opendev/system-config master: Add the mirror-only cloud regions to nodepool builder  https://review.opendev.org/66038413:30
openstackgerritMonty Taylor proposed openstack/project-config master: Add control plane mirror-only cloud regions  https://review.opendev.org/66038813:30
*** rkukura has quit IRC13:31
*** rossella_s has quit IRC13:34
*** ramishra has quit IRC13:37
*** Lucas_Gray has quit IRC13:38
*** jamesmcarthur has quit IRC13:39
*** Lucas_Gray has joined #openstack-infra13:39
*** ramishra has joined #openstack-infra13:42
*** bhavikdbavishi has joined #openstack-infra13:45
*** jamesmcarthur has joined #openstack-infra13:48
*** jchhatbar has joined #openstack-infra13:57
*** rf0lc0 has joined #openstack-infra13:57
*** kmarc has joined #openstack-infra13:57
*** mnencia_ has joined #openstack-infra13:58
*** calbers_ has joined #openstack-infra13:58
*** jamesmcarthur_ has joined #openstack-infra13:59
*** Wryhder has joined #openstack-infra13:59
*** rh-jelabarre has joined #openstack-infra14:00
*** ekultails has joined #openstack-infra14:00
*** gary_perkins_ has joined #openstack-infra14:00
*** gfidente|afk is now known as gfidente14:00
*** lifeless_ has joined #openstack-infra14:00
*** tobiash_ has joined #openstack-infra14:01
*** Anticime1 has joined #openstack-infra14:01
*** pgaxatte1 has joined #openstack-infra14:01
*** ktsuyuzaki has joined #openstack-infra14:04
*** pgaxatte1 has quit IRC14:05
*** ddurst- has joined #openstack-infra14:05
*** aspiers_ has joined #openstack-infra14:05
*** pgaxatte1 has joined #openstack-infra14:05
*** tosky__ has joined #openstack-infra14:05
*** Lucas_Gray has quit IRC14:06
*** janki has quit IRC14:06
*** mriedem has quit IRC14:06
*** derekh has quit IRC14:06
*** tosky has quit IRC14:06
*** zhurong has quit IRC14:06
*** jamesdenton has quit IRC14:06
*** kota_ has quit IRC14:06
*** zer0c00l has quit IRC14:06
*** SpamapS has quit IRC14:06
*** rfolco has quit IRC14:06
*** markmcclain has quit IRC14:06
*** lifeless has quit IRC14:06
*** dosaboy has quit IRC14:06
*** amoralej has quit IRC14:06
*** andreykurilin has quit IRC14:06
*** mnencia has quit IRC14:06
*** bradm has quit IRC14:06
*** Wryhder is now known as Lucas_Gray14:06
*** mnencia_ is now known as mnencia14:06
*** jamesmcarthur has quit IRC14:06
*** udesale has quit IRC14:06
*** electrofelix has quit IRC14:06
*** ralonsoh has quit IRC14:06
*** pgaxatte has quit IRC14:06
*** sshnaidm has quit IRC14:06
*** toabctl has quit IRC14:06
*** harlowja has quit IRC14:06
*** calbers has quit IRC14:06
*** dayou_ has quit IRC14:06
*** aspiers has quit IRC14:06
*** smcginnis has quit IRC14:06
*** ddurst has quit IRC14:06
*** gary_perkins has quit IRC14:06
*** Anticimex has quit IRC14:06
*** strigazi has quit IRC14:06
*** edmondsw_ has quit IRC14:06
*** tobiash has quit IRC14:06
*** kmarc_ has quit IRC14:06
*** calbers_ is now known as calbers14:06
*** aspiers_ is now known as aspiers14:06
*** rpittau is now known as rpittau|afk14:06
*** udesale has joined #openstack-infra14:07
*** electrofelix has joined #openstack-infra14:07
*** mriedem has joined #openstack-infra14:08
*** rf0lc0 is now known as rfolco14:12
*** mattw4 has joined #openstack-infra14:12
*** aaronsheffield has joined #openstack-infra14:12
*** eharney has joined #openstack-infra14:12
*** sshnaidm has joined #openstack-infra14:12
*** ralonsoh has joined #openstack-infra14:12
*** dayou_ has joined #openstack-infra14:13
*** SpamapS has joined #openstack-infra14:13
*** harlowja has joined #openstack-infra14:14
*** armax has joined #openstack-infra14:26
*** lpetrut has quit IRC14:27
*** _erlon_ has joined #openstack-infra14:27
*** ykarel is now known as ykarel|afk14:27
*** bhavikdbavishi has quit IRC14:28
*** rascasoft has joined #openstack-infra14:33
*** mattw4 has quit IRC14:35
*** liuyulong has joined #openstack-infra14:39
zigoclarkb: mordred: fungi: ianw: https://review.opendev.org/#/c/645574/ <--- Hopefully, this will work after debootstrap is fixed!14:43
zigoPlease review.14:43
*** rascasoft has quit IRC14:44
*** chandankumar is now known as raukadah14:46
clarkbzigo: ianw did most of the work. I just helped ensure it got in happily at the end14:55
clarkbmordred: cool I'll take a look at those changes after my morning meeting14:55
zigoclarkb: Thanks a lot. :P14:56
clarkbpabelanger: it happens due to needing to lookup PRs via commit hashes and there is no direct api for that so zuul ends up scanning all the PRs. In most cases this is actually a one time cost because zuul scans the PRs and then caches the results. The problems arise with post merge testing which ansible uses extensively because the cache is out of data if multiple non fast forwardable PRs merge in14:57
clarkbsequence (as only the first can have a cached hash, subsequent PRs all get new hashes that are uncached)14:57
clarkbs/out of data/out of date/14:58
*** tosky__ is now known as tosky14:59
*** jchhatbar is now known as janki15:02
*** hamzy has quit IRC15:07
*** LordGT has joined #openstack-infra15:08
fungiskopeo (0.1.37-1~dev~ubuntu16.04.2~ppa2) xenial; urgency=medium * autobuilt 0c90e57 -- Lokesh Mandvekar (Bot) <...>  Tue, 21 May 2019 03:44:55 +000015:23
fungithat's what we've been waiting for15:23
*** priteau has joined #openstack-infra15:24
* fungi un-wips and approves his revert15:24
* fungi checks apt cache show on ze01 to be sure15:24
clarkbI think we'll be ready to approve https://review.opendev.org/#/c/647877/3 today too (though we should probably double check new ask is happy with ianw first)15:25
fungiapt show skopeo|grep ^Version:15:26
fungiVersion: 0.1.37-1~dev~ubuntu16.04.2~ppa215:26
fungilooks like we're ready15:26
clarkbwhat is the current version?15:26
clarkb(that version looks a lot like the one I remember from yseterday)15:26
fungiadding -a to the apt show command reveals 0.1.36-1~dev~ubuntu16.04.2~ppa19.115:26
fungiwhich jives with the installed version reported by dpkg -l too15:27
fungithe .1 in the package revision is what differentiates my hand-built package from the one in the ppa15:28
*** sreejithp_ has joined #openstack-infra15:28
fungiso that was the source package for 0.1.36-1~dev~ubuntu16.04.2~ppa19 patched with tristanC's pr and then built in a local xenial chroot15:28
clarkbgot it15:28
*** sreejithp__ has joined #openstack-infra15:30
clarkbmordred: https://review.opendev.org/#/c/640027/11 requires an update to the secret ansible vars right? for the control plane clouds group?15:31
*** sreejithp has quit IRC15:31
*** sreejithp_ has quit IRC15:32
clarkbfungi: you've been more up to date on the arm and linaro cloud situation iirc care to look at https://review.opendev.org/#/c/660374/4 just to double check we are doing the right things there?15:33
dmsimardI have a weird issue when trying to use "virtualenv_command: /usr/bin/python3 -m venv" instead of "virtualenv_python: python3", has anyone else see something like it ? Looks like pip gives up if it can't find a wheel for a package.15:34
dmsimardWith virtualenv_python it works: http://logs.openstack.org/14/660414/1/check/ansible-role-ara-api-ubuntu/e4dbc27/ara-report/result/e061c0a3-3ca0-47fe-8aa7-c9825f84de0b/15:34
dmsimardWith virtualenv_command it doesn't: http://logs.openstack.org/20/660420/1/check/ansible-role-ara-api-ubuntu/fae8f2f/ara-report/result/1d5255ec-83bc-4a87-9aa3-7590daa19a9b/15:34
dmsimardFWIW the virtualenv_command works on Fedora but not on Ubuntu15:35
dmsimardusing virtualenv_command on Fedora: http://logs.openstack.org/20/660420/1/check/ansible-role-ara-api-fedora/7071797/ara-report/result/a35a86bf-fb73-4447-b03b-5dcb2e72f2a2/15:35
*** hamzy has joined #openstack-infra15:35
fungidmsimard: part of the problem may be that python3 -m venv is going to use the venv module shipped with the python stdlib, while virtualenv is a separate python package versioned independently of the interpreter15:36
clarkbworht noting that python -m venv != virtualenv15:36
dmsimardyeah python3-venv is installed on ubuntu prior to trying that15:36
dmsimardI'll try reproducing without Ansible at all15:36
fungiright, so the venv module will have features contemporary with the python release from which it's shipped15:37
clarkbfungi: dmsimard ya I think that is likely it. the venv module is going to be ancient (whatever shipped with the distro) whereas virtualenv should be up to date on our images15:37
clarkbin particular this may be the issue I managed to get pip to fix15:37
clarkbwhere it bails out early rather than trying additional indexes15:37
fungiahh, right, it'll also be using the pip version vendored with the python interpreter rather than the version vendored into latest virtualenv package15:38
clarkbdebian (via zigo) fixed this in their pip packaging15:38
clarkbbut I don't know that ubuntu has15:38
fungiso less likely this is actually a venv vs virtualenv behavior issue, and more likely it's just a question of which version of pip is being invoked15:38
*** pgaxatte1 has quit IRC15:40
*** LordGT has quit IRC15:41
fungihttps://metadata.ftp-master.debian.org/changelogs//main/p/python-pip/python-pip_18.1-5_changelog shows clarkb's fix in the version for sid/buster15:41
fungithe job in question ran on our ubuntu-bionic image15:42
dmsimardI'm unable to reproduce on a local 18.04 vm, I'll hold a node to troubleshoot15:43
*** michael-beaver has joined #openstack-infra15:43
fungilooks like that version has been pulled into ubuntu starting in 19.0415:44
clarkbmordred: comments on https://review.opendev.org/#/c/640044/3 I think our minimal image is still a bit too minimal15:44
fungithe version in bionic-updates has this changelog, which doesn't indicate any backport of that fix (or any update at all in over a year): http://changelogs.ubuntu.com/changelogs/pool/universe/p/python-pip/python-pip_9.0.1-2.3~ubuntu1/changelog15:45
fungiwe could file an sru request if that hasn't been done already15:45
*** ykarel|afk is now known as ykarel|away15:45
fungithough i have no idea how easily that patch can be backported to the pip 9.0.1 codebase15:46
*** kskels has joined #openstack-infra15:46
*** smarcet has joined #openstack-infra15:46
mordredclarkb: I hear you on python and haveged - I'd argue that the other things are better if we express them in the base role. BUT - I don't feel *strongly* about that15:48
mordredclarkb: and yes - 640027 still requires an update to secret ansible vars15:48
mordredclarkb: maybe what we shoudl do is do that move in 2 parts ... copy to the new file now, then land the change, then delete the old file15:49
clarkbmordred: ya that should work safely15:49
clarkbmordred: ya I don't mean to make that list an absolute suggestion. Was more here is all the stuff we are likely to want on our servers in production that we don't currently get15:50
clarkbhaveged is importand as is python for base functionality though15:50
fungii think you could get a newer pip by using python3.7 in bionic, though there doesn't seem to be a distinct python3.7-pip package (there is a python3.7-venv package but neither it nor libpython3.7-stdlib seem to include the pip module)15:50
fungiand anyway, the source package was last updated in october15:50
*** Lucas_Gray has quit IRC15:51
clarkbfor example we use iptables rules on all servers15:51
clarkbmaybe we should have the base package for that in the image15:51
clarkbntp is in that boat too15:51
dmsimardfungi: let me get back to you on that once I have more details and a way to reproduce it15:51
*** ykarel|away has quit IRC15:51
*** raissa has joined #openstack-infra15:51
dmsimardfungi: yeah, it definitely reproduces in the nodepool vm but doesn't in my local vm15:52
fungidmsimard: may want to confirm which version of pip is being invoked15:53
clarkbdmsimard: does your local vm talk to our mirrors?15:53
openstackgerritMerged opendev/system-config master: Revert "Pin skopeo to unbreak skopeo+bubblewrap"  https://review.opendev.org/65990815:53
dmsimardclarkb: does not, it's a vanilla ubuntu server 18.0415:53
clarkbdmsimard: the issue here is there is no wheel package for that package on our mirror (this is expected). Pip is supposed to look at all theo ther indexes like pypi for that package; hwoever pip has had broken versions (that I helped fix since) where it won't do that15:53
clarkbdmsimard: ok thats why you aren't reproducing locally then15:53
dmsimardyeah if I move /etc/pip.conf elsewhere it works15:54
*** e0ne has quit IRC15:55
*** janki has quit IRC15:55
clarkbthe fix is to either use up to date pip (with my fixing) or don't use package mirrors that don't have all the packages you need15:55
fungispecifically it's our wheel caches which are the issue, right?15:56
funginot our pypi proxy?15:56
dmsimardcan you specify a fallback index ?15:56
clarkbfungi: our wheel caches not being a complete mirror of pypi trigger the bug in pypi15:56
dmsimardor, well, more than one index15:56
clarkbdmsimard: the bug is that pip doesn't use those alternative indexes15:56
clarkbdmsimard: it is supposed to (and I helped fix it so that it does)15:56
*** smcginnis has joined #openstack-infra15:57
*** ccamacho has quit IRC15:57
fungiclarkb: right, but point being a job could work around this by avoiding use of our wheel cache while still benefiting from our pypi proxy15:57
clarkbfungi: yes, or using an up to date pip15:57
*** kskels has left #openstack-infra15:57
*** ginopc has quit IRC15:59
dmsimardso the pip.conf as set up by infra has "index-url = http://mirror.iad.rax.openstack.org/pypi/simple" and "extra-index-url = http://mirror.iad.rax.openstack.org/wheel/ubuntu-18.04-x86_64"16:00
dmsimardI tried using "index-url = http://mirror.iad.rax.openstack.org/pypi/simple" and "extra-index-url = https://pypi.python.org/simple/" and that works16:01
clarkbdmsimard: yup beacuse both are complete mirrors of pypi16:01
clarkb(they don't lack any packages like our wheel mirror)16:01
dmsimardclarkb: pip still pulls the wheels from our mirror though16:01
clarkbdmsimard: not from our wheel mirror16:01
clarkb(because you removed it)16:02
dmsimardclarkb: am I reading this wrong? http://paste.openstack.org/show/751889/16:02
fungithat's the pypi proxy, not the wheel cache16:02
*** rascasoft has joined #openstack-infra16:02
clarkbdmsimard: no those are wheels that are put on pypi by those packages16:03
clarkbdmsimard: our wheel mirror builds wheels for all packages on specific platforms16:03
clarkbdmsimard: because many packages on pypi don't supply wheels16:03
clarkblibvirt is one example16:03
fungiwell, more that many packages on pypi can't provide generic wheels16:03
fungipure python packages can easily, some of the more simple non-pure python packages can build manylinux1 compliant wheels16:04
dmsimardok, I understand now :D16:04
fungibut other more complex packages need to build platform-specific wheels16:04
clarkbfungi: which fedora 30 has broken btw due to changes in libcrypt16:04
fungiand we pre-build platform-specific wheels to save every job from having to build those locally from the sdist on the fly, which can take a very long time in some cases16:05
fungiwhich is what's being served at the /wheel/ubuntu-18.04-x86_64 paths16:05
fungi(and similar for other platforms)16:05
roman_gHello dear Infra team. How could we get e-mail notification for failed post-merge jobs?16:06
clarkbthe underlying bug in distro pip was due to broken monkey patching done by pip for distros so that they could unvendor requests. This caused pip to try and catch the wrong exception which led to uncaught exceptions from requests when the lookup fails because package doesn't exist (and no other indexes were checked as a result)16:06
*** rascasoft has quit IRC16:07
openstackgerritNate Johnston proposed opendev/irc-meetings master: Update neutron-ci agenda and chairs  https://review.opendev.org/66044316:07
clarkbroman_g: zuul has an email reporter, but that has to be added to the pipeline (so typically isn't great if different jobs need to report to different email addrs)16:07
clarkbroman_g: openstack health has rss/atom feeds for openstack jobs. I think this has been the suggested method for subscribing to those events16:07
clarkbwe may want to have zuul provide that directly in a generic manner16:08
dmsimardfungi, clarkb: I'll work around this for now by avoiding the wheel index -- thanks for your help :D16:09
*** panda|rover has quit IRC16:09
roman_gclarkb: good, thanks. Could we may be get openstackgerrit bot notify us in the channel?16:09
roman_gor other bot16:10
*** panda has joined #openstack-infra16:10
*** smarcet has quit IRC16:11
clarkbroman_g: I think openstack's gerritbot can actually do that16:11
clarkbit can trigger on verified -1 and -2 votes iirc16:11
clarkboh except in post there is no -1 or -2 hrm16:11
roman_gyes, yes.16:11
*** smarcet has joined #openstack-infra16:12
roman_git's Zuul's responsibility to report something16:12
clarkbroman_g: also do you know about the zuul builds dashboard?16:12
roman_gclarkb: yes, I do. But prefer to have either e-mail, or something in IRC, or better both.16:13
roman_gspecifically for post-merge16:14
roman_gbecause once we WF+1 - no one cares if those jobs were successful or not. No one would get back to Zuul dashboard and manually check post-merge jobs.16:15
*** gyee has joined #openstack-infra16:15
clarkbyup unfortunately this is true even when we sent a ton of email to people on those failures (the openstack-qa list for years got all the periodic and post test job failure reports iirc)16:15
clarkbwhich is why I think we haven't put a lot of effort into improving the feature16:15
clarkbthat said I think a subscription model like rss or atom would be a great reporter addition16:16
clarkbbecause then anyone could subscribe as necessary and add their own filters16:16
roman_gif we get e-mail with failure report, then we could set up a filter in e-mail client and watch for failures16:16
roman_gbtw, where is the RSS?16:17
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile  https://review.opendev.org/66019116:17
*** rosmaita has quit IRC16:18
clarkbroman_g: rss is currently provided by http://status.openstack.org/openstack-health/#/16:18
clarkbroman_g: the problem with email is in determining the targets16:18
clarkbroman_g: we probably don't want anyone to arbitrarily decide where email should be sent (because spam)16:18
clarkbroman_g: so currently that is a per pipeline setting (which is config with a bit more oversight)16:19
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile  https://review.opendev.org/66019116:19
clarkbthe benefit of rss/atom is that it is subscription based so zuul could publish to an endpoint and people can opt in to subscribe16:19
*** igordc has joined #openstack-infra16:20
roman_gclarkb: we are planning to have mailing list for failures & automated reports (e.g. for periodic jobs)16:20
fungiyikes, wiki.o.o has a load average of 90 righth now16:20
*** smarcet has quit IRC16:21
openstackgerritAlex Schultz proposed openstack/project-config master: Retire ansible-role-k8s-* repos  https://review.opendev.org/66044616:21
clarkbroman_g: so that will work as long as you have your own pipelines (which I think isn't a major issue if we get people split into tenants, but prior to that we'd probably not want to add a ton of project specific pipelines)16:21
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile  https://review.opendev.org/66019116:21
roman_gclarkb: yes, that is a bit problematic (i'd need to add param to common jobs)16:22
roman_gbut still doable16:22
clarkbroman_g: it isn't a job parameter, it is a piepline parameter16:22
fungioh yeah, the wiki server's being flooded with requests right now16:22
clarkb(pipelines determine what reports are made)16:22
*** smarcet has joined #openstack-infra16:22
roman_gahha, reported is a pipeline param... good16:23
clarkbfungi: new wave of spam?16:23
roman_gthank you for the consultation, clarkb. I need to digest it, and will then start working.16:23
fungilooks like getting flooded with random attempts at php script compromises16:23
fungii see tons of requests for things like /wiki/info/infoShowAction.do?method=getAccessoryByManager&id=../../../../../../../../etc/passwd%00.jpg&accessory=116:24
openstackgerritBen Nemec proposed openstack/pbr master: Install more dependencies for integration testing  https://review.opendev.org/66044916:24
dmsimardclarkb, fungi: just confirming that taking out the extra-index-url from pip.conf resolved my issue, thanks <316:26
clarkbdmsimard: fwiw I would use virtualenv over -m venv due to issues like this16:27
clarkbrather than editing the pip configs16:27
clarkb(because you get up to date pip that way)16:27
*** ricolin has quit IRC16:27
dmsimardclarkb: unfortunately doing the other way around brings other problems so I'm trying to find something that works for both :D16:27
fungi#status log blocked all traffic to wiki.openstack.org from using an iptables drop rule to quell a denial of service condition16:29
openstackstatusfungi: finished logging16:29
fungiload average for wiki.o.o is back under 1 again now16:32
fungii'll keep an eye on the access log for a while in case it starts back up again16:32
*** smarcet has quit IRC16:36
*** mattw4 has joined #openstack-infra16:36
openstackgerritSean McGinnis proposed openstack/project-config master: Drop jobs for devstack-plugins-sheepdog  https://review.opendev.org/66045216:36
*** raissa has quit IRC16:36
*** raissa has joined #openstack-infra16:37
*** raissa has quit IRC16:37
*** raissa has joined #openstack-infra16:38
*** raissa has quit IRC16:38
*** raissa has joined #openstack-infra16:39
*** raissa has quit IRC16:39
*** raissa has joined #openstack-infra16:40
*** raissa has quit IRC16:40
*** ramishra has quit IRC16:42
clarkbmwhahaha: fedora 28 is the version of fedora that tripleo is using to test python3 transition work right? And ya'll expect to need that until we have centos 8 images?16:43
* clarkb trying to figure out which images we can clean up with the impedning removal of trusty16:43
mordredclarkb: I'll update the control plane images patches in just a sec - debugging the gitea 1.8.0 patch atm ...16:45
mordredI'm getting this:16:45
mordred2019/05/21 16:44:17 ...s/setting/setting.go:738:NewContext() [F] error saving generating JWT secret to custom config: open /custom/conf/app.ini: permission denied16:46
mordredso apparently in 1.8.0 they're wanting to do that thing where they write something to a config file :(16:46
clarkbmordred: I'm guessing the jwt secret is something they want to generate and not somethign we can stick in our images ;)16:47
clarkbmordred: maybe the simplest fix is to change the perms on the file?16:47
*** smarcet has joined #openstack-infra16:47
*** lucasagomes has quit IRC16:48
*** dtantsur is now known as dtantsur|afk16:48
mordredclarkb: well - I'm investigating to see if it's one of those things where maybe we can set the setting in our app.ini ourselves first16:48
mordredlike - maybe they generate if you dont have one already16:48
Shrewsmordred:  'chmod -R o+rw /' will fix it16:48
openstackgerritSean McGinnis proposed openstack/project-config master: Remove devstack-plugin-sheepdog  https://review.opendev.org/66045416:49
mordredShrews: that's a great suggestion16:49
clarkbmordred: ya but its also a secret right?16:49
*** whoami-rajat has quit IRC16:49
clarkbI guess we can bind mount that in then have our init thing sed it into the file?16:49
clarkb(or just bind mount the entire file I suppose)16:50
*** armax has quit IRC16:52
*** udesale has quit IRC16:53
*** smarcet has quit IRC16:54
* clarkb finds breakfast16:55
mordredclarkb: ok. turns out the issue is that what's in our file isn't a base64 encoded secret16:55
mordredclarkb: we just have a random string of characters16:55
mordredthis is also true of our real secret on bridge16:55
mordredactually - it turns out this is also not the actual issue16:58
mwhahahaclarkb: yes but we're trying to get an alternative. Might want to check with weshay16:59
weshaywhat's the context?17:01
clarkbweshay: I'm taking stock of the images we may be able to clean up and wanted to check if fedora 28 was still used by tripleo17:01
fungikeeping f28 images until centos8 is out17:01
clarkbin particular as the python3 test bed until centos 8 happens17:01
fungi(rather than switching to f29 or f30 i suppose)17:02
clarkbpart of the motiviation here is zigo wants to add buster images and monty wants to add a control plane bionic image and we already can't support all the images we have when we lose a builder due to disk sizes17:02
*** kashyap has joined #openstack-infra17:05
weshayya..  so yes we are still using it atm.  However I wouldn't stand in your way if you needed to kill it.   We're waiting on some items for rhel-8 in rdo zuul but hopefully it won't take too long, few more week.   That should help bootstrap centos-8 when it becomes a thing17:05
kashyapHi folks, can someone remind me -- where do we capture guest boot log, if at all, during CI run?17:05
weshayso if you guys can hold out a few more weeks that would be super, but if we can't, that is ok too17:06
clarkbI don't think we haev to kill it. Was mostly checking if there was an easy win there17:06
kashyapE.g. should I find it somewhere here? -- http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/17:06
clarkbkashyap: do you have an example job run (it will depend on the job itself)17:06
kashyapclarkb: Check above ^17:06
clarkbkashyap: http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/libvirt/qemu/ there17:06
weshayk, thanks when we have 8 I'll definitely come back and say NUKE IT17:06
clarkbkashyap: oh the boot log17:06
clarkbkashyap: for that I think it is captured in the tempest log17:06
kashyapclarkb: It doesn't contain the full serial console log of the guest boot17:06
kashyap(Yeah; boot)17:06
clarkbkashyap: it will if ssh conenctivity failed iirc17:07
kashyapclarkb: And where can I find it?17:07
clarkbit may not do it unless that condition is tripped17:07
*** mattw4 has quit IRC17:07
kashyapAh, I see.  Trying to figure out why on earth guest fails to shutdown cleanly:17:07
kashyapDamned if I could put a finger on the cause17:08
clarkbkashyap: did you check syslog too?17:08
kashyapThe return code from libvirt's shutdown request is '0'; that's "boring" (as in, expected)17:08
kashyapclarkb: This one? -- http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/syslog.txt.gz17:08
clarkbya or even http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/devstack.journal.xz.gz17:08
*** dpawlik has quit IRC17:08
clarkb(we capture the systemwide journal too17:08
kashyap(I filtered the first one by ERROR; and didn't get anything.)17:09
* kashyap clicks on the second17:09
clarkbhttp://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/devstack.journal.README.txt.gz is the notes on how to deal with that exported journal17:09
*** quiquell has quit IRC17:10
clarkbhttps://wiki.centos.org/About/Building_8 looks like there isn't anything for us to do on our end with centos 8 yet17:10
kashyapclarkb: Thanks; useful notes17:10
clarkbmordred: | ac2c4d17-8d6f-4e3c-a9eb-57c155f0a949 | v2-standard-2  |  8192 |    0 |         0 |     2 | True      | is the flavor that I think new gitea servers can run under with 80GB volumes instead of 40 GB17:12
*** igordc has quit IRC17:12
*** igordc has joined #openstack-infra17:13
kashyapclarkb: Do you know WTF is this: "Bad message":17:13
kashyap$> journalctl --file ./devstack.journal -u 'devstack@n-api'17:13
kashyapFailed to open files: Bad message17:13
kashyap$> file devstack.journal17:13
kashyapdevstack.journal: data17:13
kashyapI can `less` it, fine, though.17:13
clarkbkashyap: did you run /lib/systemd/systemd-journal-remote <(xzcat ./devstack.journal.xz) -o output.journal ?17:13
kashyapBad me17:14
clarkbyou need to do that format conversion for journalctl --file to work (and you do --file on the output of ^)17:14
kashyapI'm insufferably blind17:14
*** whoami-rajat has joined #openstack-infra17:15
openstackgerritMonty Taylor proposed opendev/system-config master: Use local fork of gitea and upgrade to 1.8.0  https://review.opendev.org/65552217:18
mordredclarkb: ok. I *think* that will fix it17:18
openstackgerritMatt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects  https://review.opendev.org/66046117:18
clarkbsilly new features17:19
*** dpawlik has joined #openstack-infra17:20
fungithe executors are still on the old skopeo packages... i wonder if this is because we ensure=>present instead of ensure=>latest and so wouldn't normally see it happen until unattended-upgrades runs later?17:20
*** mattw4 has joined #openstack-infra17:20
clarkbfungi: ya I think we ran into that with the debootstrap fix yesterday17:21
clarkbapt-get install debootstrap was enough for apt to figure out it could update17:21
*** jpena is now known as jpena|off17:24
*** dpawlik has quit IRC17:24
clarkbas a sanity check nb01 hasn't leaked any images. All the images in /opt/nodepool_dib are active images according to nodepool dib-image-list17:26
clarkbI think we'll be ok to add more images but it will start getting tight17:26
*** armax has joined #openstack-infra17:27
clarkb/opt is an lvm'd cinder volume so we can add another 1TB volume, add it to lvm vg then extend the filesystem if we need to17:27
*** eharney has quit IRC17:27
fungiwondering if i should go ahead and manually trigger the skopeo upgrades now while more folks are around rather than wait for them to upgrade at random times over the next 24 hours17:29
clarkbfungi: ++17:29
fungiclarkb: nb03 will be the tricky one since we don't have cinder in that cloud so need to rebuild on a custom flavor with the desired rootfs size17:30
clarkbthinking out loud here on the gitea06 plan. I think we get mordred's control plane clouds change in, then add ubuntu-bionic-minimal image to control planes, then add journalctl fix from logan-'s change to those images, then we rebuild gitea06 on flavor v2-standard-2 with an 80GB disk17:30
clarkbfungi: nb03 is ok because it only builds like 2 images17:31
*** kopecmartin is now known as kopecmartin|off17:31
clarkbthen if in the process of adding these new images we run out of disk we add 1TB extra disk to lvm17:31
openstackgerritTobias Henkel proposed zuul/zuul master: Add example statsd_exporter mapping  https://review.opendev.org/66047217:33
mordredclarkb: I think that's a good plan17:34
*** tobiash_ is now known as tobiash17:34
mordredcorvus: I just pushed up a gitea PR - watch out, I might catch up to you!17:35
clarkbmordred I want to double check with ianw today before approving logan-'s journal size fix (note not journalctl as mentioned previously) since he had some concern about its interaction with growroot17:35
clarkbbut I expect we'll get that in today and it will start making test nodes happier17:35
*** electrofelix has quit IRC17:36
clarkbI also wanted to make sure ianw is ready for us to puppet-4 ask.o.o, but I expect we'll get that in today too completing our puppet-4 upgrades17:36
clarkbslow but steady progress17:36
openstackgerritTobias Henkel proposed zuul/nodepool master: Add statsd_exporter mapping  https://review.opendev.org/66047317:37
*** ralonsoh has quit IRC17:37
*** smarcet has joined #openstack-infra17:37
*** ykarel|away has joined #openstack-infra17:37
mordredclarkb: working on the element for the packages ...17:41
*** jamesmcarthur_ has quit IRC17:42
*** priteau has quit IRC17:42
mordredclarkb: in the infra-package-needs element we enable acpid under openrc but not under systemd (we enable ntpd under systemd)17:42
mordredthis seems weird to me17:42
openstackgerritMonty Taylor proposed openstack/project-config master: Build ubuntu bionic minimal images for control plane  https://review.opendev.org/64004417:43
*** rkukura has joined #openstack-infra17:44
mordredclarkb: how does that look? ^^17:44
clarkbya I think we want acpid across the board (it makes things like nova server restarts work)17:44
clarkbmordred: I'm guessing acpid is enabled by default when installed on other platforms?17:45
*** igordc has quit IRC17:46
clarkbmordred: that looks great17:46
mordredclarkb: awesome17:47
clarkbmordred: oh except one thing, I don't think we should use our mirrors to build them17:47
mordredclarkb: yeah - also - I don't think we currently have any intention to use this for anything other than ubuntu or anything <bionic17:47
mordredclarkb: totally. fix coming17:48
clarkbmordred: left a comment to make sure we capture that too, but other tahn that I think this is about ready17:48
openstackgerritMonty Taylor proposed openstack/project-config master: Build ubuntu bionic minimal images for control plane  https://review.opendev.org/64004417:49
*** smarcet has quit IRC17:49
openstackgerritMonty Taylor proposed openstack/project-config master: Add control plane mirror-only cloud regions  https://review.opendev.org/66038817:49
clarkbone thing I'll have to remind myself is those control plane images should be quite small17:53
clarkbprobably only a couple gigabytes17:53
clarkbso their cost is low compared to the test node images17:53
mordredclarkb: now - the _fun_ part is going to be using them from launch-node17:53
mordredsince the image name in glance will change daily17:53
mordredclarkb: we'll probably want to make launch-node smarter17:54
clarkbya though I personally set image name every time I use laucnh node anyways17:54
clarkbtoo many times I've been bitten by the default not being what I want17:54
clarkbthat it was easier to always set it and know I got the right thing17:54
mordredoh - totally.17:54
mordredI was more thinking we coudl set IMAGE='ubuntu-bionic-minimal' and have launch-node do the list and find the newer of the two images prefixed with that17:55
mordredmostly because it'll allow us to interact with launch-node as humans - and launch-node can find the correct image pretty easily I'd imagine17:55
mordred(although, that said - if we add that, we could set the default to "ubuntu-bionic-minimal" - and it would be correct on all of our clouds!)17:56
clarkbmordred: fungi http://logs.openstack.org/22/655522/2/check/system-config-build-image-gitea/677bc01/job-output.txt.gz#_2019-05-21_17_31_19_348141 is that possibly related to updating skopeo?17:57
mordredclarkb: uh17:57
*** jamesmcarthur has joined #openstack-infra17:58
fungii haven't upgraded it yet, so hopefully not?17:59
clarkbah ok17:59
fungiwas eating a lateish lunch first18:00
*** rossella_s has joined #openstack-infra18:00
mordredoh yeah. lunch. I'mma gonna do that18:01
clarkboh right /var/registry/data is empty because we use swift now18:01
*** jamesmcarthur has quit IRC18:02
clarkbwhat is odd there is prior to that we push the image itself which seemed to work18:03
clarkbI'm going to try a recheck to see if this is one of our many network related issues and inconsistent or if it is something else18:03
fungiinfra-root: heads up, the culprit i blocked from wiki.o.o earlier also hit www.openstack.org hard in the last few minutes... keep an eye out for other servers misbehaving i guess18:04
clarkbwhois says it is a chinese clothing company. I'm guessing they have been compromised18:05
clarkbthey have a phone number listed if someone wants to warn them18:06
openstackgerritIldiko Vancsa proposed opendev/system-config master: [WIP] Adding new ML for the NBMP group  https://review.opendev.org/66048518:08
*** jamesmcarthur has joined #openstack-infra18:12
*** panda has quit IRC18:14
*** panda has joined #openstack-infra18:15
*** igordc has joined #openstack-infra18:18
*** eharney has joined #openstack-infra18:20
*** rosmaita has joined #openstack-infra18:21
*** xek_ has joined #openstack-infra18:21
*** jamesmcarthur_ has joined #openstack-infra18:22
clarkbthat intermediate registry push for gitea images seem to happen consistently on limestone at least18:24
*** xek has quit IRC18:24
clarkbodd that it affects the tag push and not the image push18:24
*** jamesmcarthur has quit IRC18:24
*** mattw4 has quit IRC18:24
clarkbmakes me wonder if there is something wrong with the intermediate registry18:24
fungiperhaps in ways which only occur when being accessed over ipv6? what could that be though?18:25
clarkbcould imply our socat tunnel is broken18:26
clarkbI've asked docker for the registry logs and they have been scrolling for quite some time18:26
fungibut it uses that for the image push too, right?18:26
*** hwoarang has quit IRC18:27
fungiis it a persistent socket connection or started on-demand and terminated again?18:27
*** hwoarang has joined #openstack-infra18:28
*** mattw4 has joined #openstack-infra18:28
clarkbcorrect should be used for the image push too. Should be persistent forking socat18:28
clarkbThere are no logs for gitea accesses when the two job runs failed18:29
clarkbtime="2019-05-21T14:50:01.506433721Z" is most recently logged gitea access by the intermediate registry18:29
clarkbso maybe it is never getting to the intermedaite registry and some problem between limestone and there exists18:30
*** xek__ has joined #openstack-infra18:30
fungithe intermediate registry is our persistent server, right?18:31
fungiwhat's the hostname?18:31
clarkbok now I notice that the image build itself failed18:31
clarkbfor the incorrect username or password issue18:32
fungiahh, so it's not just affecting tag pushes then18:32
clarkbpossible that the image push task (which is no_log=true) isn't actually pushing the image as a result but then does try to tag it and fails because it wasn't pushed?18:32
clarkbI can look into that more after our meeting as it may be that the error we get is just confusing us due to procedural mistakes18:32
*** xek_ has quit IRC18:33
fungiyeah, that's an interesting possibility18:34
clarkbI'm beginning to suspect something like that is happening18:35
logan-i've been watching the conntrack table size and it looks like it hovers between 8k-12k active connections, which doesn't seem like enough to trigger snat port contention. nothing in dmesg indicating full conntrack tables or that sort of thing18:35
clarkblogan-: another possibility I hadn't considered yet/before was docker registry proxy could be trying to talk via ipv6 because it doesn't like the 10/8 address or something18:36
clarkblogan-: and just failing outright18:36
clarkbI need to pop out for a bit before the infra meeting but can help look into this more after the meeting18:37
fungiat this point i'll save the skopeo upgrades for immediately following the meeting so we don't have to deal with it breaking something during18:38
*** bhavikdbavishi has joined #openstack-infra18:42
*** xek has joined #openstack-infra18:49
*** xek__ has quit IRC18:50
clarkbso I don't forget the other item to consider is firewalls18:56
*** sshnaidm is now known as sshnaidm|afk18:56
clarkbI thought we had punched all the holes necessary but we may be missing some still18:57
openstackgerritMatt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects  https://review.opendev.org/66046118:59
fungiclarkb: ooh, yeah maybe we forgot v6 equivalent of some v4 rule(s)?18:59
fungithat could explain why we see it where we do19:00
*** gyee has quit IRC19:07
openstackgerritAlex Schultz proposed openstack/project-config master: Retire ansible-role-k8s-* repos  https://review.opendev.org/66044619:07
fungi#thanks cmurphy for driving our massive puppet upgrade19:12
openstackstatusfungi: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks)19:12
*** gyee has joined #openstack-infra19:19
*** bhavikdbavishi has quit IRC19:27
*** whoami-rajat has quit IRC19:29
*** pkopec has quit IRC19:34
openstackgerritMerged opendev/system-config master: Update ask.openstack.org to puppet 4  https://review.opendev.org/64787719:35
*** ykarel|away has quit IRC19:36
AJaegerconfig-core, please review https://review.opendev.org/#/c/659977 to update a few docs jobs19:36
mordredfungi: re: mediawiki + sqlite - a lot of the projects that make Dockefiles and images seem focused on enabling simple use, rather than production use19:42
mordredfungi: I thnik that's one of the times when we might look at what they did, go - "thanks for the starting point" - and then make our own19:42
fungiyeah, it's... very short (which is a great demonstration i guess): https://github.com/wikimedia/mediawiki-docker/blob/master/1.32/Dockerfile19:44
*** jamesmcarthur_ has quit IRC19:44
mordredfungi: oh - there's another thing - the dockerfiles we've been writing make use of a "modern" technique - which is multi-stage dockerfiles19:45
mordredfungi: this allows you to have one "stage" do build-type things, and then copy artifacts from that stage into the final image - so that you don't need to install -dev packages and whatnot in your actual application image19:46
fungiahh, so simplifies cleanup19:47
clarkbright part of the issue is docker images would balloon and get huge due to accumulated intermediate artifacts19:48
*** Goneri has quit IRC19:48
*** logan- has quit IRC19:55
openstackgerritMerged openstack/project-config master: Drop jobs for devstack-plugins-sheepdog  https://review.opendev.org/66045219:55
Shrewsmordred: question for you on https://review.opendev.org/64002720:00
fungicontinuing the ask.o.o discussion, i guess a major point in the services we run as alternatives to other popular services is that they're built on free software... is the source code for stackexchange open? if so then i see zero benefit in competing with it20:01
mordredclarkb, fungi, ianw: I think I was trying to get at 2 different things - one is that various of our opendev tenants might, from time to time, want to run a specific service that is not generally opendev - and we've got several openstack examples that exist via history - but we should figure out what, if any, part in running such services in the future we want to have as opendev20:01
mordredand the second is that, since ask is not a general-pupose opendev tool, I thnik it's reasonable to ask that folks who care about it existing show up with resources to make that happen20:02
*** Goneri has joined #openstack-infra20:02
mordredShrews: good catch20:02
*** jamesmcarthur has joined #openstack-infra20:02
clarkbfungi: I don't think it is. askbot was the answer to that problem20:02
ianwfungi: the source isn't, but all comments are creative common licensed and i think you can bulk download them20:03
fungiyeah, in that case there could be a reason to host an open source alternative to stackexchange, unless there's someone out there who already is doing that we can recommend instead20:03
fungiokay, preparing to upgrade skopeo20:05
clarkbfungi: https://askbot.com/20:05
clarkbbtw I think I know what the docker image problem is20:05
clarkbwill have patch as soon as I figure out syntax20:05
fungiclarkb: doesn't look like askbot.com is providing free hosting of a general ask site20:06
*** rlandy is now known as rlandy|biab20:06
fungiinstead it looks like you can do a 30-day free trial of a paid ask site specific to your project20:06
openstackgerritMonty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml  https://review.opendev.org/64002720:06
openstackgerritMonty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files  https://review.opendev.org/66037420:06
openstackgerritMonty Taylor proposed opendev/system-config master: Add the mirror-only cloud regions to nodepool builder  https://review.opendev.org/66038420:06
mordredclarkb: ^^ respun due to review from Shrews20:06
openstackgerritClark Boylan proposed opendev/base-jobs master: Open port 5001 for the registry proxy  https://review.opendev.org/66050420:08
fungiokay, preparing to upgrade skopeo20:08
mordredfungi: yah - I agree, I could see a reason to run one on that reason - but if we did, I'd think we'd want to either make a general purpose one (ask.opendev.org) and/or be willing to generally provide white-labeled ones for more than just openstack20:08
clarkbfungi: mordred logan- Shrews ^ I think 660504 does the firewall stuff we need20:08
openstackgerritSean McGinnis proposed openstack/openstack-zuul-jobs master: Remove legacy sheepdog tests  https://review.opendev.org/66050520:08
mordredclarkb: FOR THE LOVE OF20:08
clarkbfungi: ya its paid20:09
fungimordred: i would probably not even want to consider white-labeled ones, just a generic opendev one where people can ask questions about anything related to projects hosted in opendev20:09
clarkbfungi: but it is oepn source20:09
ianwfungi: seems similar to the translate discussion.  it might be a better use of resources to pay a company providing hosting & support of an opensource product for services, rather than allocate developer time to self-hosting20:09
ianwif it's a situation where paying for that hosting is supporting development of the project, it seems win-win20:10
fungiianw: yep, in that case it's a matter of getting osf to work out contract details and long-term financial commitments for those i guess20:10
clarkbfwiw we had a similar setup with them in the past20:10
clarkbI don't know what happened to move away from that20:10
clarkbreed probably knows20:11
openstackgerritSean McGinnis proposed openstack/openstack-zuul-jobs master: Fix typo in tempest-dsvm-full-hdfs  https://review.opendev.org/66050720:11
clarkbthank you #openstack-infra for being my rubber ducky while I talked myself through this docker thing20:12
openstackgerritMerged openstack/project-config master: Switch some repos to PTI docs jobs  https://review.opendev.org/65997720:12
clarkbianw: you mentioend wanted to check growroot with https://review.opendev.org/#/c/659215/ any reason to not approve that now?20:13
mordredclarkb: well - I think our original deal with the ask folks was that they'd run it but it would run on our infrastructur20:13
ianwoh i was just wondering what happens with growing the disk and journals20:13
clarkbfungi: fwiw I see our mailing list(s) as the better venue for that type of discussion. Ask suffers from lack of moderation in addition to service side problems20:13
openstackgerritMerged openstack/project-config master: Remove broken docs jobs from monasca-agent  https://review.opendev.org/65997820:14
fungiclarkb: yep, i agree. maybe mailman 3 is the answer20:14
clarkbianw: wondering if it is a noop because growroot will also grow the journal?20:14
mordredwhile I think with the translate suggetsion it's just "let someone else completely run a hosted service for us"20:14
mordredfungi: mailman3 is always the answer20:14
fungi#status upgraded skopeo on all zuul executors to 0.1.37-1~dev~ubuntu16.04.2~ppa220:14
openstackstatusfungi: unknown command20:14
clarkbmordred: yup they did that for a bit then at some point we ended up hosting it afterall20:14
fungi#status log upgraded skopeo on all zuul executors to 0.1.37-1~dev~ubuntu16.04.2~ppa220:14
openstackstatusfungi: finished logging20:14
mordredclarkb: yah.20:14
fungiclarkb: so... 660504 firewall rules ftw?20:15
ianwclarkb: yeah, i mean that's the way it works now right, grows up to the 80gb of the vm at boot time.  does the journal grow with that too?20:16
clarkbianw: May 21 20:04:09 ask01 ansible-apt: Invoked with dpkg_options=force-confdef,force-confold autoremove=False force=False force_apt_get=False install_recommends=None package=None autoclean=False purge=False allow_unauthenticated=False state=present upgrade=None update_cache=None default_release=None only_upgrade=False20:16
clarkbdeb=https://release-archives.puppet.com/apt/pool/xenial/PC1/p/puppet-agent/puppet-agent_1.10.14-1xenial_amd64.deb cache_valid_time=020:16
clarkbianw: I think that implies that we didn't actually upgrade puppet20:16
clarkb(this may be why the upgrade process was relying on install.sh20:17
openstackgerritMerged opendev/puppet-mediawiki master: Replace transitional package names for Xenial  https://review.opendev.org/65135220:17
ianwi did a bit of research yesterday, but i couldn't find anything on what the best practice is for growing disks and journals20:17
ianwclarkb: hrm ... did they end up re-creating a repo?20:17
mordredfwiw - I pushed up https://github.com/go-gitea/gitea/pull/7011 earlier to correct a piece of documentation - just in case someone wants to check the docs when looking at the gitea 1.8.0 patch20:17
clarkbianw: re growroot we should be able to interrogate a test node for its journal size (I imagine)20:18
clarkbianw: re ask puppet 4 install_puppet.sh was a bit more sophisticated than just install this deb20:18
clarkbianw: its possible that puppet doesn't want to update because of some missing option20:18
ianwok, to my repo question : https://tickets.puppetlabs.com/browse/CPR-685 is the answer an it's "in progress"20:19
clarkbI would've expected a dpkg -i equivalent to result in just install this package please20:20
clarkbbut maybe it won't because puppet 3 is a conflict or something?20:20
fungiif there are unmet dependencies it won't20:21
clarkbfungi: there aren't any that I know of and this was tested by our test jobs20:21
clarkb(we install puppet 4 via the same manner on test jobs where there is no preinstalled puppet)20:22
clarkbso maybe one thing we can try is uninstall puppet-3 by hand and have ansible install puppet 420:22
fungiyou can apt install a local file path to the downloaded package if you want to make sure that declared dependencies get fetched and installed20:22
fungioh, also if the package name is different, they could conflict20:22
*** diablo_rojo_phon has joined #openstack-infra20:22
clarkbI think it is puppet from distro for puppet-3 and puppet-agent for puppet-4 deb20:23
clarkbso that may be it20:23
fungibtw, for anybody who hasn't yet seen a demo of the checks plugin for gerrit, go to https://gerrit-review.googlesource.com/c/gerrit/+/224831 and then select the "checks" tab where the list of changed files is displayed20:23
diablo_rojo_phonclarkb: fungi sorry am at dinner and missed the meeting, yes we do want to start doing releases. We talked about it at the PTG.20:23
fungidiablo_rojo_phon: cool! thanks for confirming20:24
clarkbfungi: ianw so what do we think about uninstalling the puppet package supplied by the distro?20:24
mtreinishfungi: is that the new gerrit interface we have to look forward to at some point in the future?20:25
clarkbMay 21 19:37:15 ask01 puppet-user[12470]: Invalid parameter state on Package[jetty] at /opt/system-config/production/modules/openstack_project/manifests/ask.pp:34 on node ask01.openstack.org20:25
fungii think we probably have to before we can install the newer differently-named one. unless it provides the appropriate metadata to indicate it replaces the distro package, in which case we still need something smarter than dpkg -i (like apt install of the local file path maybe)20:25
clarkbpuppet is unhappy either way so probably not urgent (though getting this done would be nice)20:25
ianwclarkb: that seems reasonable ...20:25
clarkbok I'll start with that uninstal20:26
* mtreinish doesn't like it (but that's not really a surprise, still miss the old view in gerrit)20:26
openstackgerritMerged opendev/base-jobs master: Open port 5001 for the registry proxy  https://review.opendev.org/66050420:26
fungimtreinish: yes, that's their new polyglot-based "polygerrit" ui20:26
mtreinishthe checks tab is cool though20:26
clarkbI don't like that the checks tab is a tab you have to click on20:26
clarkbpeople will wonder why can't my change merge20:27
clarkbI ran `apt-get purge puppet` to remove puppet then `apt-get autoremove` to remove augeas-lenses facter hiera libaugeas0 puppet-common ruby-augeas ruby-deep-merge ruby-json ruby-nokogiri ruby-rgen ruby-safe-yaml ruby-selinux ruby-shadow virt-what20:28
fungiclarkb: also there's no grouping mechanism, so we'd need to prepend all the job names for every different ci system, and i don't think there's currently a way to prevent one ci system from replacing the results of another because they don't consider the possibility you don't trust the ci systems reporting on your change or that there would be more than one20:28
ianwclarkb: hrm ... "root@ask01:/tmp# dpkg -i puppet-agent_1.10.14-1xenial_amd64.deb" seemed like it wanted to work ...20:28
clarkbianw: was this after I removed things?20:28
ianwohhh, right20:29
clarkbfungi: I think the spec for it does allow for different systems at least20:29
clarkbfungi: it has been a while since I read the spec though20:29
ianwclarkb: should i run a kick.sh?20:29
clarkbmaybe, I think kick.sh may not do what we want anymore iwth the base.yaml split?20:30
ianwit runs remote_puppet_adhoc, which should do the puppeting?20:31
clarkbianw: but that doesn't do the install20:31
clarkboh it does20:31
clarkbI take that back20:31
clarkbso ya I think a kick.sh should work20:31
ianwyeah, i think it installs and copies hiera and all that20:31
ianwrunning now20:32
ianwTASK [puppet : Sanity check puppet version] **********************************************************************************************************************************20:33
ianwfatal: [ask01.openstack.org]: FAILED! => {"changed": false, "msg": "Failed as requested from task"}20:33
clarkbianw: /opt/puppetlabs exists now20:33
fungiso i guess it was the package name difference20:34
fungimost likely they tried to both provide at least one file at an identical path and dpkg refused to install the new one because it would overwrite a file it knew belonged to the existing one20:35
clarkb/opt/puppetlabs/bin is empty though20:36
clarkb/opt/puppetlabs/puppet/bin/puppet.dpkg-new exists though20:36
ianwyeah, it failed in the "Sanity check puppet version" check per above ... but didn't really say why20:36
clarkbdid the halfway dpkg -i get us into a halfway state?20:36
clarkbianw: one sec I'll link you the code20:36
ianwhrm, yeah just ran a  dpkg --configure -a , maybe i messed it up.20:37
clarkbianw: https://opendev.org/opendev/ansible-role-puppet/src/branch/master/tasks/main.yaml#L37-L39 is the check and https://opendev.org/opendev/ansible-role-puppet/src/branch/master/tasks/main.yaml#L25-L29 is what seems to fail20:37
ianwlet's try again20:37
clarkbif you run that puppet version command it fails because the code isn't there and in the other bin dir is all the dpkg.new files20:38
ianwfatal: [ask01.openstack.org]: FAILED! => {"changed": false, "msg": "E:The package puppet-agent needs to be reinstalled, but I can't find an archive for it."20:38
ianwbase-server : Install base packages20:38
fungipuppet-agent package is in iHR state20:39
clarkbI need to eat lunch now but suspect the issue is the package isn't fully installed and once it is it'll be happy20:39
*** e0ne has joined #openstack-infra20:39
clarkbI also rechecked the gitea change after the iptables port opening merged20:39
*** rlandy|biab is now known as rlandy20:39
ianwok, i'm just dpkg -i ing it manually20:40
ianwok, puppet failed, but it ran20:41
fungiseems it's in ii state now, which is what we want20:42
ianwno parameter named 'state' at /opt/system-config/production/modules/openstack_project/manifests/ask.pp:34 on Package[jetty]20:42
fungithe previous iHR state expands to Desired=Install,Status=Half-inst,Err?=Reinst-required20:44
ianwi guess that should be "ensure"20:45
*** hamzy has quit IRC20:45
fungiyeah, ensure=>present is probably what we want there?20:46
fungii'm unfamiliar with puppet's dpkg package provider though20:46
openstackgerritIan Wienand proposed opendev/system-config master: ask.o.o : fix jetty package install  https://review.opendev.org/66051720:46
ianwfungi: ^ :)20:46
clarkbfungi: this is ansibles dpkg via apt module20:47
fungiit looks like a puppet resource to me20:47
clarkboh for jetty20:49
*** xek has quit IRC20:49
clarkbsorry thought you were refferring to the puppet dpkg install20:49
*** dave-mccowan has quit IRC20:50
funginah, we moved on whilst you sammiched20:50
fungi(or whatever lunch was)20:50
clarkbstill sammiching :)20:50
fungis/was/is/ then ;)20:50
ianwclarkb / mordred: so if we can get https://review.opendev.org/#/q/status:open+topic:letsencrypt in (clarkb just one syntax fix since your last +2 on the base change) we can have mirror.dfw.rax.opendev.org up pretty much now (modulo fixing unforseen issues) ... then i'm happy to move forward based on the nodepool built images as that comes online20:52
ianwspeaking of food ... breakfast ... bib :)20:53
*** rfarr_ has joined #openstack-infra20:54
*** e0ne has quit IRC20:56
*** rfarr has quit IRC20:56
*** gfidente has quit IRC21:02
*** gfidente has joined #openstack-infra21:03
fungiianw: i like the macro approach in 658281, i'd never noticed that feature of apache before21:07
clarkbthe gitea image build job succeeded but the jobs ran on ovh not limestone so not confirmation the port issue was the thing21:08
clarkbbut is confirmation that mordred fixed the gitea problem21:08
clarkber no need the run-gitea job to run I guess21:09
*** jamesmcarthur has quit IRC21:11
mordredfungi, ianw: WOAH21:12
mordredclarkb: yay fixing gitea problem!21:13
mordredclarkb: we can finally land the "make the gitea front page text in the present tense" patch :)21:13
fungiyay for being able to land patches again21:13
openstackgerritMonty Taylor proposed opendev/system-config master: Update opendev website to be more present tense  https://review.opendev.org/65394621:13
clarkbmordred: yup21:13
*** logan- has joined #openstack-infra21:13
clarkbmordred: also I think sparkycollier has some edits he wants to propose (I was assisting with git set up yesterday)21:14
mordredclarkb: cool! I'm excited about getting edits from sparkycollier21:14
cmurphyask.o.o puppet is unhappy?21:14
clarkbcmurphy: yes but we think we've got it under control https://review.opendev.org/#/c/660517/121:14
clarkbthat was a preexisting issue not puppet-4 related21:14
mordredclarkb: like - he's going to submit a new patch from scratch? so we don't have to wait on this one, yeah?21:15
fungimore of a nobody-looks-at-this-thing-like-ever issue21:15
*** kjackal has joined #openstack-infra21:15
fungithe ask.o.o puppet issue i mean21:15
clarkbmordred: correct21:15
cmurphyi'm about to be driving for a while so hopefully nothing puppetty explodes too badly21:16
*** diablo_rojo has joined #openstack-infra21:19
clarkbI've rechecked a few zuul changes that seemed to have hit the registry proxy issue21:20
fungilooks like we're 2x+2 on the letsencrypt changes, and the earliest in the sequence is in the gate now21:25
fungiholding off on approving the second since i don't think it shares a change queue with the others21:25
*** rfarr__ has joined #openstack-infra21:26
openstackgerritMerged openstack/project-config master: Finish moving python-jenkins jobs in-tree  https://review.opendev.org/66002221:27
*** slaweq has quit IRC21:27
*** Lucas_Gray has joined #openstack-infra21:28
*** rfarr_ has quit IRC21:28
fungii'm going to try and knock out some yardwork but will be in and out21:30
zigofungi: Hi there! Any chance you have time to review adding a Buster image? https://review.opendev.org/#/c/645574/21:35
openstackgerritMerged opendev/system-config master: ask.o.o : fix jetty package install  https://review.opendev.org/66051721:43
*** Lucas_Gray has quit IRC21:47
clarkb^ should apply in about 30 minutes21:49
clarkbnot seeing any buildset registry proxies on limestone since the fix went in. At least it doesn't seem to break anything for the other regions21:49
logan-my bnc dropped for a bit. what was the fix?21:50
clarkblogan-: https://review.opendev.org/#/c/660504/ is the current suspect21:51
*** mriedem has quit IRC21:51
logan-docker really does hate ipv6 heh21:52
*** Lucas_Gray has joined #openstack-infra21:52
weshayhttps://review.opendev.org/660529  warlock again21:57
*** sreejithp__ has quit IRC21:59
*** sreejithp has joined #openstack-infra21:59
clarkbianw: May 21 22:00:21 ask01 puppet-user[9397]: (/Stage[main]/Openstack_project::Ask/File[/root/jetty_1.0_all.deb]) Could not evaluate: Could not retrieve file metadata for puppet://modules/openstack_project/ask/jetty_1.0_all.deb: getaddrinfo: Name or service not known22:01
clarkbI think you need three slashes on the puppet:///22:01
openstackgerritKendall Nelson proposed opendev/system-config master: Adds new key for diablo_rojo  https://review.opendev.org/66053022:01
diablo_rojoclarkb, fungi, corvus whoever is around, if its not too much trouble to give that a look ^^ I wanted to try to get the swift test migration done before their meeting tomorrow (I think its tomorrow lol)22:02
ianwclarkb: ok ... i'm not even going to think about why that used to work but doesn't now22:02
openstackgerritClark Boylan proposed opendev/system-config master: Add more /s to puppet:/// url for jetty shim deb  https://review.opendev.org/66053122:03
clarkbianw: ^ like that22:03
notmynamediablo_rojo: yes. tomorrow at 2100utc (2pm pacific)22:03
ianwthanks :)22:04
clarkblogan-: fungi mordred ok we caught one on limestone22:04
*** sreejithp has quit IRC22:04
clarkbthe tests for https://review.opendev.org/63619722:04
diablo_rojonotmyname, ....I'll figure out what time it is where I am lol22:05
diablo_rojoclarkb, thanks!22:05
*** rcernin has joined #openstack-infra22:05
*** gfidente is now known as gfidente|afk22:05
clarkbdiablo_rojo: 11pm for you I think22:05
clarkbdiablo_rojo: an hour ago22:05
clarkbbut tomorrow22:05
*** tesseract has quit IRC22:07
mordredclarkb: woot!22:08
clarkbouch linux 5.1 fstrim reportedly discarding the wrong blocks22:09
*** dciabrin_ has quit IRC22:10
*** rfarr__ has quit IRC22:11
*** dciabrin_ has joined #openstack-infra22:11
diablo_rojoOh wow okay got it.22:13
diablo_rojothanks for the conversion clarkb :)22:13
clarkbdiablo_rojo: well it was 2200UTC22:23
clarkband cest is +2 so was relatively easy for me to do the maths after looking at my irc buffer22:23
clarkbI'm cloning the linux kernel now to figure out if the current kernel I am running is going to destroy my disk22:23
*** kjackal has quit IRC22:23
clarkbmaybe I should just turn it off and go fishing and hope things are better when I return on thursday22:24
mordredclarkb: I think that's the best plan22:26
openstackgerritMerged openstack/project-config master: Add a Debian Buster image.  https://review.opendev.org/64557422:30
clarkbI think it trimmed yesterday due to fstrim.timer22:32
clarkband that was after I rebooted into the new kernel22:32
clarkbso maybe I'm already broken and I just don't know it yet22:32
*** sarob has joined #openstack-infra22:32
clarkbor it isn't a universal error with btrfs + dm + lvm + linux 5.122:33
fungidebian/sid being toward the end of a lengthy release freeze for buster has luckily kept me on 4.19 for a bit22:34
*** sarob has quit IRC22:35
clarkbhttp://logs.openstack.org/97/636197/20/check/zuul-build-image/e75436b/ara-report/result/0d0ebdbc-93b6-4d06-9e2d-30724fbfca1a/ is the iptables hole punch on limestone and that job as well as the consuming job succeeded22:36
clarkbhopefully that deals with a major issue in the system22:36
* fungi cheers22:37
* mordred hands everyone baby hippos22:37
clarkbwoo ya the bisected change culprit was in 5.0-rc622:39
clarkbso my 5.1.2 almost certainly is running with it22:39
clarkbit should be known that my fileserver runs a stable ubuntu release22:40
*** rcernin has quit IRC22:40
clarkbmy desktop and laptop are more like thinclients that I won't cry too much over if fstrim eats the disk22:40
clarkbI'll just have to reinstall22:40
*** rcernin has joined #openstack-infra22:41
clarkbhttps://www.redhat.com/archives/dm-devel/2019-May/msg00102.html is the fix22:41
*** tosky has quit IRC22:42
fungii rely on backups22:48
fungibut yeah, not having to restore data from a backup is definitely preferable22:49
clarkbfungi: ya I have those too, now with offsites22:49
ianwclarkb: sorry, right, back to the ext4 thing ... it's unclear to me if resize2fs ultimately changes the journal size on growth22:49
clarkbianw: can we inspect the fs on a running post growroot'd instance?22:49
fungii basically have that insofar as i stick encrypted backups of my local systems on my remote cloud servers and vice cersa22:49
clarkbhttp://blog.dailystuff.nl/2012/07/getting-ext34-journal-size/ oh wow you have to find the inode where the journal starts?22:50
*** armax has quit IRC22:50
clarkbdumpe2fs is easy mode apparently22:50
ianwUser:     0   Group:     0   Size: 6710886422:51
*** tjgresha has joined #openstack-infra22:51
ianwthat's on one of our nodes  /dev/xvda1       38G  5.3G   31G  15% /22:52
clarkbdumpe2fs on an ovh bhs1 node says 64M for journal after growing to 75G22:53
clarkbI think that implies we don't grow the journal when we grow the fs22:53
clarkbthat blog post also says never change its size unless it is a clean unmounted fs22:54
*** auristor has quit IRC22:55
clarkbalso 128MB seems to be a common journal size22:55
clarkbwe might want to change the dib default to that?22:56
clarkbthat does make me wonder if 512M is huge22:57
*** auristor has joined #openstack-infra22:57
ianwper https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n340 256 is probably as big as our fs gets22:59
*** tkajinam has joined #openstack-infra22:59
clarkbwe are in between 64GB and 128GB23:00
clarkbso 512 is indeed the default23:01
openstackgerritMerged opendev/system-config master: Create opendev mirrors  https://review.opendev.org/65828123:01
openstackgerritMerged opendev/system-config master: Adds new key for diablo_rojo  https://review.opendev.org/66053023:01
clarkbI say we go for the change as is, it fits into the defaults and should make the journal on ext4 happier23:01
clarkbwe can always tune it back down if the balloons the image size23:01
clarkbI'm guessing this is why dib makes it so small23:01
fungii have to assume it compresses well when freshly initialized23:02
diablo_rojothanks fungi and clarkb!23:02
clarkbfungi: ya but we can't compress or trim the vhd images23:03
clarkbfungi: we compress qcow2 so should be fine there and we trim raw images so also fine there23:03
clarkbianw: if no objections feel free to approve the change (I would but I'm about to afk for a day so figure I shouldn't be responsible)23:04
clarkbhttps://review.opendev.org/#/c/659215/ is the change for anyone following along23:05
ianwok, i approved it.  there's some numbers out there on things like moving the journal to a faster ssd, etc, but not really much i can see on sizing23:06
ianwyou'd think people dealing with cloud-init and growing root file systems would have discussed it before, but if they have, i can't find it23:07
*** calbers has quit IRC23:07
clarkbianw: we are trailblazers!23:07
clarkblets hope we don't get swept by the warriors23:07
*** jklare has quit IRC23:09
*** sreejithp has joined #openstack-infra23:09
*** sreejithp_ has joined #openstack-infra23:11
*** mattw4 has quit IRC23:11
*** mattw4 has joined #openstack-infra23:12
*** sreejithp has quit IRC23:14
fungisports joke, i guess23:16
clarkbyes the san francisco warriors just destroyed the portland trailblazers23:16
fungiwhich of our clouds still rely on vhd images, btw... just rax?23:16
clarkbfungi: correct23:16
*** _erlon_ has quit IRC23:17
openstackgerritMerged openstack/project-config master: Increase nodepool image journal size  https://review.opendev.org/65921523:17
openstackgerritMerged openstack-infra/zone-opendev.org master: Add mirror.dfw.rax.opendev.org  https://review.opendev.org/66023523:17
*** Lucas_Gray has quit IRC23:20
fungiokay, slightly too real... i ripped a variety of sound effects discs along with the rest of my cd collection when building my home jukebox/radio station... taking a break from pushing around the reel mower, catching up on e-mail and listening to said radio... what comes up in rotation? sound clip of a power lawn mower23:21
*** jklare has joined #openstack-infra23:24
*** aaronsheffield has quit IRC23:25
*** calbers has joined #openstack-infra23:26
ianwso you just have random sound effects in your radio shuffle?23:27
ianwi don't know if the humor goes outside an australian context but as a sound effects fan you might like sizzletown https://www.podbean.com/site/EpisodeDownload/PB8F29CFECIHY23:29
fungiheh, yes random clips of sound effects are salted throughout my random shuffle. it gets entertaining at times, as evidenced23:32
openstackgerritMerged opendev/system-config master: Add more /s to puppet:/// url for jetty shim deb  https://review.opendev.org/66053123:32
fungialso occasionally artist interviews, sound clips from films (john carpenter seemed to do this on a number of the soundtracks he composed for his films) and short episodes from some radio serials23:33
*** icarusfactor has joined #openstack-infra23:34
fungibut the collection is approaching a solid 5 weeks of uninterrupted audio though, so this stuff accounts for less than 1% probably. it's *almost* entirely music (depending on how you define music at least)23:34
fungii have no idea what tony martin's sizzletown is, but going to find out once i'm done with the yard ;)23:36
*** factor has quit IRC23:36
*** icarusfactor is now known as factor23:37
* tonyb tried to work out why irssi needed to tell me y'all are talking sound effects .. but it did23:38
tonybOh .... I see now nm23:39
funginow i get it, "tony martin" is tonyb's stage name!23:42
tonybfungi: for sure ;P23:42
openstackgerritMerged opendev/system-config master: Bringup mirror01.dfw.rax.opendev.org  https://review.opendev.org/66023723:42
*** lseki has quit IRC23:43
*** Goneri has quit IRC23:52
*** rlandy has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!