Friday, 2018-06-22

mnaserwould it be ok to add reverse proxy cache to something that is not fully mirrored?00:00
mnaseri.e. mariadb for ubuntu is mirrored but not other distros00:00
mnaserit would just be easier to use the reverse proxy for the other rather than having a super complicated set of filters00:00
*** owalsh has quit IRC00:00
clarkbmnaser: for things like that it might make more sense to proxy cache all of it? The reason we have the cache is that we don't have the resouces to mirror the internet and that gives us a flexible way of targetin specific things that will likely change over time00:01
clarkbwhereas base distro repos and pypi in particular have been sometin we've needed/wanted all along00:01
mnaserclarkb: well the idea is i would have to go find all the consumers of the existing mirror00:01
mnaserand i have no idea how to do that00:01
clarkbcodesearch is probably a good place to start00:01
*** felipemonteiro has joined #openstack-infra00:02
*** heyongli has quit IRC00:02
mnaserwe could totally edit NODEPOOL_MARIADB_MIRROR and hope people actaully use it too00:02
mnasercodesearch shows no one referencing it00:02
*** heyongli has joined #openstack-infra00:02
mnasernor is anyone referencing NODEPOOL_MARIADB_MIRROR00:02
mnaserlol00:02
clarkbI want to say we set it up for osa because ubuntu had mysqland they were deploying maria only00:03
clarkbwhereas other distros had maria which is why we didn't mirror other distro packages00:03
mnaserosa isn't consuming it unfortunately (i'm trying to improve gating for osa)00:04
mnaserso percona is good but mariadb is the current issue00:04
clarkband you can't just use distro packages for maria on not ubuntu?00:04
*** eernst has quit IRC00:05
ianwi feel like there might be chat about this on openstack-dev a while ago, when i asked if anyone was using that00:05
mnaserclarkb: at this point that's too fundamnetal of a change to make, but ideally i would like us to use distro packages only00:05
funginote that some distros call their mariadb packages "mysql" for backwards compatibility reasons00:05
mnaserbut that's a much bigger thing :-)00:05
mnaserright now rdo ships mariadb, so does opensuse and ubuntu, so we could do that00:06
mnaserbut centos gates have been broken for the past 2 weeks and i still haven't gotten through them all, OSA is filled with places where they had not been using mirrors00:06
mnaserand a lot of rechecks being thrown00:06
openstackgerritMohammed Naser proposed openstack-infra/system-config master: Add MariaDB to cache  https://review.openstack.org/57732500:08
mnaserfungi, clarkb, ianw: check above and maybe it might be productive to grep logs for ubuntu-mariadb ?00:08
mnaserif no one is hitting it, i'll propose to remove it so that we can change NODEPOOL_MARIADB_MIRROR to say the right thing00:09
mnaserbut according to codesearch, nothing is using it00:09
*** heyongli has quit IRC00:12
*** heyongli has joined #openstack-infra00:12
openstackgerritMohammed Naser proposed openstack-infra/system-config master: Add MariaDB to cache  https://review.openstack.org/57732500:13
ianwmnaser: here's the thread http://lists.openstack.org/pipermail/openstack-dev/2018-March/128322.html00:14
mnaserianw: given that the only user commented is OSA and i'm going to 'refactor' into using the cache, i think it's safe to drop it00:15
ianwmnaser: feel like doing that :)00:17
ianw?00:17
*** gyee has quit IRC00:17
mnaserianw: yes, do you want a patch with a bunch of state => absent and then follow up deleting them?00:17
mnaseror just delete right away and itll e done manually00:17
ianwthe volume has to be removed manually, so yeah, fine to just remove it00:18
ianwand we can cleanup00:18
mnaserok cool00:18
openstackgerritMohammed Naser proposed openstack-infra/system-config master: Drop ubuntu-mariadb mirror  https://review.openstack.org/57732800:21
mnaserianw: ^ (should be) done, no more references to mariadb except the proxy ones00:22
*** heyongli has quit IRC00:22
*** heyongli has joined #openstack-infra00:23
ianwoh that's right, it was that the two runs were corrupting each other that lead me to looking at this in the first place00:24
*** heyongli has quit IRC00:33
*** heyongli has joined #openstack-infra00:33
*** dbecker has joined #openstack-infra00:36
*** rpioso is now known as rpioso|afk00:38
*** diablo_rojo has quit IRC00:41
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add a clouds.yaml checker  https://review.openstack.org/57560700:42
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add identity_api_version to packethost  https://review.openstack.org/57733100:42
*** heyongli has quit IRC00:43
*** heyongli has joined #openstack-infra00:43
*** jesslampe has joined #openstack-infra00:48
*** heyongli has quit IRC00:53
*** heyongli has joined #openstack-infra00:53
*** stakeda has joined #openstack-infra00:59
*** stevebaker has quit IRC00:59
*** stevebaker has joined #openstack-infra01:01
*** heyongli has quit IRC01:03
*** heyongli has joined #openstack-infra01:04
*** hemna_ has quit IRC01:04
*** jesslampe has quit IRC01:07
*** spsurya has joined #openstack-infra01:07
*** dbecker has quit IRC01:12
*** heyongli has quit IRC01:14
*** heyongli has joined #openstack-infra01:14
*** pcrews has quit IRC01:17
*** pcrews has joined #openstack-infra01:18
*** rh-jelabarre has quit IRC01:21
*** yamahata has quit IRC01:22
*** heyongli has quit IRC01:24
*** heyongli has joined #openstack-infra01:24
*** iyamahat_ has quit IRC01:28
*** heyongli has quit IRC01:34
*** heyongli has joined #openstack-infra01:34
*** aeng has quit IRC01:43
*** heyongli has quit IRC01:44
*** heyongli has joined #openstack-infra01:45
openstackgerritIan Wienand proposed openstack-infra/system-config master: tools/sorted_modules.py : make python3 safe  https://review.openstack.org/57560601:47
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add a clouds.yaml checker  https://review.openstack.org/57560701:47
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add identity_api_version to packethost  https://review.openstack.org/57733101:47
*** heyongli has quit IRC01:55
*** heyongli has joined #openstack-infra01:55
*** heyongli has quit IRC02:05
*** heyongli has joined #openstack-infra02:05
*** felipemonteiro has quit IRC02:09
*** dayou has quit IRC02:13
*** heyongli has quit IRC02:15
*** heyongli has joined #openstack-infra02:15
*** dayou has joined #openstack-infra02:16
*** hongbin has joined #openstack-infra02:18
openstackgerritMerged openstack-infra/system-config master: Fix acceptance test setup script for puppet 4  https://review.openstack.org/57569802:18
*** felipemonteiro has joined #openstack-infra02:19
*** felipemonteiro has quit IRC02:21
*** annp has joined #openstack-infra02:23
openstackgerritIan Wienand proposed openstack-infra/project-config master: Consolidate nodepool graphs  https://review.openstack.org/57704502:23
*** heyongli has quit IRC02:25
*** namnh has joined #openstack-infra02:26
*** heyongli has joined #openstack-infra02:26
*** heyongli has quit IRC02:36
*** heyongli has joined #openstack-infra02:36
openstackgerritLogan V proposed openstack/diskimage-builder master: Allow extra grub modules to be installed  https://review.openstack.org/57733802:37
*** psachin has joined #openstack-infra02:40
*** heyongli has quit IRC02:46
*** heyongli has joined #openstack-infra02:46
*** roman_g has quit IRC02:49
*** heyongli has quit IRC02:56
*** heyongli has joined #openstack-infra02:56
*** heyongli has quit IRC03:06
*** heyongli has joined #openstack-infra03:07
*** hongbin has quit IRC03:11
*** heyongli has quit IRC03:17
*** heyongli has joined #openstack-infra03:17
*** heyongli has quit IRC03:27
*** heyongli has joined #openstack-infra03:27
*** ykarel|away has joined #openstack-infra03:35
*** heyongli has quit IRC03:37
*** heyongli has joined #openstack-infra03:38
*** sree has joined #openstack-infra03:39
*** sree has quit IRC03:39
*** sree has joined #openstack-infra03:40
*** lpetrut has joined #openstack-infra03:44
*** udesale has joined #openstack-infra03:45
*** heyongli has quit IRC03:47
*** heyongli has joined #openstack-infra03:48
*** iyamahat has joined #openstack-infra03:55
*** heyongli has quit IRC03:58
*** heyongli has joined #openstack-infra03:58
*** ykarel|away is now known as ykarel04:02
*** heyongli has quit IRC04:08
*** heyongli has joined #openstack-infra04:08
*** yamahata has joined #openstack-infra04:13
*** heyongli has quit IRC04:18
*** heyongli has joined #openstack-infra04:18
*** lpetrut has quit IRC04:25
*** heyongli has quit IRC04:28
*** heyongli has joined #openstack-infra04:29
*** heyongli has quit IRC04:39
*** heyongli has joined #openstack-infra04:39
*** Bhujay has joined #openstack-infra04:40
*** cshastri has joined #openstack-infra04:45
*** pgadiya has joined #openstack-infra04:46
*** pgadiya has quit IRC04:46
*** spsurya has quit IRC04:48
*** heyongli has quit IRC04:49
*** heyongli has joined #openstack-infra04:49
*** heyongli has quit IRC04:59
*** heyongli has joined #openstack-infra04:59
*** janki has joined #openstack-infra05:01
*** hashar has joined #openstack-infra05:03
*** janki has quit IRC05:06
*** heyongli has quit IRC05:09
*** heyongli has joined #openstack-infra05:10
*** dhajare has joined #openstack-infra05:12
openstackgerritMerged openstack-infra/project-config master: Add fedora vexxhost flavor for Kata  https://review.openstack.org/57731805:15
*** felipemonteiro has joined #openstack-infra05:17
*** heyongli has quit IRC05:20
*** heyongli has joined #openstack-infra05:20
*** heyongli has quit IRC05:30
*** heyongli has joined #openstack-infra05:30
*** heyongli has quit IRC05:40
*** heyongli has joined #openstack-infra05:40
*** iyamahat has quit IRC05:45
*** iranzo has joined #openstack-infra05:47
*** janki has joined #openstack-infra05:48
*** heyongli has quit IRC05:50
*** heyongli has joined #openstack-infra05:51
*** nicolasbock has joined #openstack-infra05:53
*** masber has joined #openstack-infra05:57
*** felipemonteiro has quit IRC05:59
*** ramishra has joined #openstack-infra06:00
*** heyongli has quit IRC06:01
*** heyongli has joined #openstack-infra06:01
*** kjackal has joined #openstack-infra06:02
*** lpetrut has joined #openstack-infra06:11
*** heyongli has quit IRC06:11
*** heyongli has joined #openstack-infra06:11
*** lpetrut_ has joined #openstack-infra06:13
*** lpetrut has quit IRC06:16
*** heyongli has quit IRC06:21
*** heyongli has joined #openstack-infra06:22
*** andreas_s has joined #openstack-infra06:24
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned  https://review.openstack.org/57687606:27
*** samueldmq has quit IRC06:28
*** caphrim007_ has joined #openstack-infra06:28
*** caphrim007 has quit IRC06:28
*** samueldmq has joined #openstack-infra06:28
*** sweston has quit IRC06:28
*** sweston has joined #openstack-infra06:28
*** edwarnicke has quit IRC06:29
*** edwarnicke has joined #openstack-infra06:29
*** pcaruana has joined #openstack-infra06:30
*** heyongli has quit IRC06:31
*** heyongli has joined #openstack-infra06:32
*** ykarel_ has joined #openstack-infra06:34
*** ykarel has quit IRC06:36
*** ykarel_ is now known as ykarel06:38
*** heyongli has quit IRC06:42
*** heyongli has joined #openstack-infra06:42
*** armaan has joined #openstack-infra06:43
*** iyamahat has joined #openstack-infra06:49
*** heyongli has quit IRC06:52
*** heyongli has joined #openstack-infra06:52
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Improve logging of GithubTriggerEvents  https://review.openstack.org/57501406:53
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant two  https://review.openstack.org/57735406:53
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant one  https://review.openstack.org/57735406:53
*** e0ne has joined #openstack-infra06:54
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant two  https://review.openstack.org/57735506:54
*** jpena|off is now known as jpena06:55
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: github: prevent AttributeError when missing branch protection  https://review.openstack.org/57735707:02
*** heyongli has quit IRC07:02
*** heyongli has joined #openstack-infra07:02
*** e0ne has quit IRC07:03
*** florianf|afk is now known as florianf07:04
*** Goneri has joined #openstack-infra07:05
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant one  https://review.openstack.org/57735407:06
*** ccamacho has joined #openstack-infra07:06
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant two  https://review.openstack.org/57735507:06
*** rcernin has quit IRC07:08
*** tesseract has joined #openstack-infra07:11
*** heyongli has quit IRC07:12
*** heyongli has joined #openstack-infra07:13
*** armaan has quit IRC07:15
*** armaan has joined #openstack-infra07:15
*** tosky has joined #openstack-infra07:18
*** heyongli has quit IRC07:23
*** heyongli has joined #openstack-infra07:23
*** dayou_ has joined #openstack-infra07:23
*** dayou has quit IRC07:26
*** heyongli has quit IRC07:33
*** heyongli has joined #openstack-infra07:33
*** jistr|off is now known as jistr07:34
*** dpawlik has joined #openstack-infra07:35
ssbarneahow does "announce-release" work? https://docs.openstack.org/infra/openstack-zuul-jobs/jobs.html#job-announce-release doesn't give any insights.07:38
ssbarneawhere is the announce send irc/mailing list, can these be customized?07:39
*** amoralej|off is now known as amoralej07:39
*** dpawlik has quit IRC07:39
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned  https://review.openstack.org/57687607:39
*** heyongli has quit IRC07:43
*** heyongli has joined #openstack-infra07:44
*** dpawlik has joined #openstack-infra07:44
*** gfidente has joined #openstack-infra07:52
*** gfidente has joined #openstack-infra07:52
*** zoli|gone is now known as zoli07:53
*** heyongli has quit IRC07:53
*** heyongli has joined #openstack-infra07:54
*** jpich has joined #openstack-infra07:57
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver  https://review.openstack.org/53555607:59
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver  https://review.openstack.org/53555708:01
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider  https://review.openstack.org/57066708:02
*** d0ugal_ has quit IRC08:03
*** heyongli has quit IRC08:04
*** heyongli has joined #openstack-infra08:04
*** d0ugal has joined #openstack-infra08:04
*** peereb has joined #openstack-infra08:05
*** rcernin has joined #openstack-infra08:05
*** sshnaidm is now known as sshnaidm|off08:05
*** roman_g has joined #openstack-infra08:05
*** shardy has joined #openstack-infra08:08
*** namnh has quit IRC08:09
*** armaan has quit IRC08:09
*** zhangfei has joined #openstack-infra08:09
*** armaan has joined #openstack-infra08:10
*** heyongli has quit IRC08:14
*** heyongli has joined #openstack-infra08:14
*** owalsh_ is now known as owalsh08:17
*** ykarel is now known as ykarel|lunch08:23
*** heyongli has quit IRC08:24
*** heyongli has joined #openstack-infra08:24
*** Bhujay has quit IRC08:25
*** armaan has quit IRC08:25
*** armaan has joined #openstack-infra08:25
*** yamahata has quit IRC08:32
*** heyongli has quit IRC08:34
*** heyongli has joined #openstack-infra08:35
*** andreas_s has quit IRC08:36
*** andreas_s has joined #openstack-infra08:36
*** Bhujay has joined #openstack-infra08:38
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref)  https://review.openstack.org/57690708:38
*** dtantsur|afk is now known as dtantsur08:39
*** andreas_s has quit IRC08:40
*** andreas_s has joined #openstack-infra08:40
*** rcernin has quit IRC08:41
*** derekh has joined #openstack-infra08:43
*** heyongli has quit IRC08:45
*** heyongli has joined #openstack-infra08:45
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref)  https://review.openstack.org/57690708:51
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix branch parsing in _event_pull  https://review.openstack.org/57737108:54
*** heyongli has quit IRC08:55
*** heyongli has joined #openstack-infra08:55
*** e0ne has joined #openstack-infra09:02
*** lpetrut__ has joined #openstack-infra09:03
*** zoli is now known as zoli|lunch09:03
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix branch parsing in _event_pull  https://review.openstack.org/57737109:04
*** heyongli has quit IRC09:05
*** heyongli has joined #openstack-infra09:06
*** lpetrut_ has quit IRC09:07
*** ykarel|lunch is now known as ykarel09:07
*** ykarel has quit IRC09:14
*** heyongli has quit IRC09:15
*** heyongli has joined #openstack-infra09:16
*** shardy has quit IRC09:16
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref)  https://review.openstack.org/57690709:17
*** shardy has joined #openstack-infra09:17
*** dosaboy has quit IRC09:19
*** dosaboy has joined #openstack-infra09:23
*** caphrim007_ has quit IRC09:26
*** heyongli has quit IRC09:26
*** heyongli has joined #openstack-infra09:26
*** sree has quit IRC09:26
*** caphrim007 has joined #openstack-infra09:26
*** sree has joined #openstack-infra09:33
*** heyongli has quit IRC09:36
*** heyongli has joined #openstack-infra09:37
*** sree has quit IRC09:37
*** zhangfei has quit IRC09:44
*** zhangfei has joined #openstack-infra09:44
*** heyongli has quit IRC09:46
*** heyongli has joined #openstack-infra09:46
*** ykarel has joined #openstack-infra09:49
*** sthussey has quit IRC09:50
*** d0ugal has quit IRC09:55
*** d0ugal has joined #openstack-infra09:56
*** heyongli has quit IRC09:56
*** heyongli has joined #openstack-infra09:57
*** armaan has quit IRC09:57
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref)  https://review.openstack.org/57690710:01
*** salv-orlando has joined #openstack-infra10:04
*** heyongli has quit IRC10:07
*** heyongli has joined #openstack-infra10:07
*** zhangfei has quit IRC10:08
*** salv-orlando has quit IRC10:09
*** heyongli has quit IRC10:17
*** heyongli has joined #openstack-infra10:17
*** zhangfei has joined #openstack-infra10:21
*** heyongli has quit IRC10:27
*** heyongli has joined #openstack-infra10:27
*** yamamoto has quit IRC10:29
fricklerinfra-root: integrated gate seems very backed up, most recent failure is this post error with remote host id changed: http://logs.openstack.org/15/577315/1/gate/swift-tox-func-ec/648b878/job-output.txt.gz#_2018-06-22_10_19_13_71218110:29
*** annp has quit IRC10:30
*** boden has joined #openstack-infra10:33
*** heyongli has quit IRC10:37
*** heyongli has joined #openstack-infra10:38
fricklerhmm, that seems to have been a one off, most other failures I found were either generic timeouts or some flaky cinder stuff10:38
*** stakeda has quit IRC10:40
*** psachin` has joined #openstack-infra10:41
*** psachin has quit IRC10:42
*** heyongli has quit IRC10:48
*** heyongli has joined #openstack-infra10:48
*** zoli|lunch is now known as zoli10:49
*** honza has quit IRC10:55
*** armaan has joined #openstack-infra10:56
*** heyongli has quit IRC10:58
*** heyongli has joined #openstack-infra10:58
*** honza has joined #openstack-infra11:00
*** honza is now known as Guest7176311:00
*** auristor has quit IRC11:03
*** salv-orlando has joined #openstack-infra11:05
*** heyongli has quit IRC11:08
*** lpetrut__ has quit IRC11:08
*** heyongli has joined #openstack-infra11:08
*** salv-orlando has quit IRC11:09
*** jpena is now known as jpena|lunch11:10
*** lpetrut has joined #openstack-infra11:14
*** yamamoto has joined #openstack-infra11:14
*** auristor has joined #openstack-infra11:15
*** yamamoto has quit IRC11:16
*** heyongli has quit IRC11:18
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref)  https://review.openstack.org/57690711:19
*** heyongli has joined #openstack-infra11:19
*** neiloy has quit IRC11:23
*** neiloy has joined #openstack-infra11:23
*** yamamoto has joined #openstack-infra11:27
*** jcoufal has joined #openstack-infra11:27
*** yamamoto has quit IRC11:28
*** heyongli has quit IRC11:29
*** heyongli has joined #openstack-infra11:29
*** ldnunes has joined #openstack-infra11:31
*** vivsoni has quit IRC11:32
*** udesale_ has joined #openstack-infra11:32
*** udesale_ has quit IRC11:33
*** sree_ has joined #openstack-infra11:34
*** udesale_ has joined #openstack-infra11:34
*** udesale has quit IRC11:34
*** sree_ is now known as Guest9997511:34
*** yamamoto has joined #openstack-infra11:35
*** Guest99975 has quit IRC11:39
*** heyongli has quit IRC11:39
*** heyongli has joined #openstack-infra11:39
*** dhajare has quit IRC11:42
*** yamamoto has quit IRC11:42
*** udesale_ has quit IRC11:42
*** andreas_s has quit IRC11:43
*** andreas_s has joined #openstack-infra11:44
*** jaypipes has joined #openstack-infra11:46
jaypipesmorning y'all. I'm wondering what the new Gerrit status of "Ready to Submit" means? Example patch that has been approved (+W'd) but is stuck in "Ready to Submit": https://review.openstack.org/#/c/571375/11:46
*** tianhui has joined #openstack-infra11:48
*** heyongli has quit IRC11:49
*** heyongli has joined #openstack-infra11:49
*** Goneri has quit IRC11:50
*** panda|off is now known as panda11:56
*** heyongli has quit IRC11:59
*** heyongli has joined #openstack-infra12:00
*** trown|outtypewww is now known as trown12:00
fricklerjaypipes: if I read things correctly, that patch is based on a patch that was abandoned (https://review.openstack.org/#/c/571082/1). it is ready to be submitted, because it has zuul +2, but cannot be merged because of that other patch12:01
fricklerjaypipes: doing a rebase should fix that, but will probably re-trigger zuul jobs12:01
*** jpena|lunch is now known as jpena12:02
*** andreas_s has quit IRC12:03
fricklerjaypipes: oh, it is also based on an old version of https://review.openstack.org/#/c/571147/1 and not PS3, which was merged. so yeah, needs rebase12:03
*** andreas_s has joined #openstack-infra12:03
jaypipesfrickler: ah, ok. ty sir!12:05
*** amoralej is now known as amoralej|lunch12:07
*** yamamoto has joined #openstack-infra12:08
*** heyongli has quit IRC12:10
*** heyongli has joined #openstack-infra12:10
dmelladohi folks, it seems that we're having IPv6 issues on the gates12:11
dmelladocould that be related to some specific hosts?12:11
dmelladosome of our gates are failing as they try to connect to IPv6 centos mirrors12:12
dmelladowithout having ipv6 configured12:12
dmelladoi.e. http://paste.openstack.org/show/724105/12:12
dmelladohttp://logs.openstack.org/62/558762/21/experimental/kuryr-kubernetes-tempest-daemon-containerized-octavia-l2/4976ae7/ara-report/result/6725a0aa-a58d-4ddf-ab7b-fdff4e8e4aeb/12:12
dmelladoAJaeger: frickler any known issue on this^^12:12
*** andreas_s has quit IRC12:12
*** andreas_s has joined #openstack-infra12:17
*** heyongli has quit IRC12:20
*** heyongli has joined #openstack-infra12:20
*** yamamoto has quit IRC12:21
fricklerdmellado: well, that node did have IPv6 available, see http://logs.openstack.org/62/558762/21/experimental/kuryr-kubernetes-tempest-daemon-containerized-octavia-l2/4976ae7/zuul-info/zuul-info.controller.txt12:23
fricklerdmellado: but not sure why it failed12:23
dmelladofrickler: I was wondering about one thing12:23
dmelladoI'm using a centos container inside it12:23
dmelladowhich seems to use mirrors.centos.org12:23
dmelladoshould this be improved by using AFS inside it?12:23
dmelladojpena: suggested that it might help12:23
fricklerdmellado: hmm, you should try to copy the mirror config that we provide into the container, but no idea how that would work in detail12:25
dmelladofrickler: do you have any pointer to that?12:25
dmelladoI'd like to at least give it a try12:25
fricklerdmellado: this is the role that should set things up on the test nodes http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/configure-mirrors12:28
*** sthussey has joined #openstack-infra12:28
*** rlandy has joined #openstack-infra12:28
fricklerdmellado: but if your host node is ubuntu, you won't have the centos repo config there, so not sure how to get those12:29
*** kgiusti has joined #openstack-infra12:30
*** heyongli has quit IRC12:30
*** heyongli has joined #openstack-infra12:30
*** andreas_s has quit IRC12:33
*** sree has joined #openstack-infra12:34
*** andreas_s has joined #openstack-infra12:34
pas-haHi guys, are there any plans for upstream CI to test OpenStack on Ubuntu 18.04? AFAIU not for Rocky for sure (too late in the cycle already) but may be there are plans for Stein?12:34
dmelladofrickler: hmm yeah12:36
dmelladoI was actually thinking about maybe fetching the repo info there12:36
*** andreas_s has quit IRC12:36
*** jcoufal has quit IRC12:36
dmelladowith a local change in the container12:36
*** andreas_s has joined #openstack-infra12:36
dmelladoin any case I'm not sure about how to deal with the ipv6 host12:37
dmelladoI recall having some issues back there with osic12:37
* dmellado goes for lunh, bbiab12:37
dmelladolunch12:37
fricklerpas-ha: there are some jobs already using bionic, see e.g. http://logs.openstack.org/97/576697/1/check/tempest-full-py36/06ea595/12:38
*** sree has quit IRC12:38
*** andreas_s_ has joined #openstack-infra12:39
openstackgerritSorin Sbarnea proposed openstack-infra/project-config master: migrate python-jenkins config to repo  https://review.openstack.org/57725012:39
*** heyongli has quit IRC12:40
*** heyongli has joined #openstack-infra12:41
*** andreas_s has quit IRC12:42
*** armaan has quit IRC12:44
*** armaan has joined #openstack-infra12:45
*** cshastri has quit IRC12:46
*** heyongli has quit IRC12:51
*** heyongli has joined #openstack-infra12:51
*** zhongjun__ has quit IRC12:52
*** camunoz has joined #openstack-infra12:53
*** yamamoto has joined #openstack-infra12:55
*** lpetrut has quit IRC12:58
*** heyongli has quit IRC13:01
*** heyongli has joined #openstack-infra13:01
*** armaan has quit IRC13:10
*** armaan has joined #openstack-infra13:10
*** heyongli has quit IRC13:11
*** heyongli has joined #openstack-infra13:11
*** vabada has joined #openstack-infra13:16
*** lpetrut has joined #openstack-infra13:17
*** HenryG has quit IRC13:18
*** jistr is now known as jistr|mtg13:20
*** HenryG has joined #openstack-infra13:21
*** heyongli has quit IRC13:21
*** heyongli has joined #openstack-infra13:22
*** myoung|off is now known as myoung|ruck13:22
*** amoralej|lunch is now known as amoralej13:22
*** dhill_ has joined #openstack-infra13:26
*** efried_pto is now known as fried_rice13:27
*** felipemonteiro has joined #openstack-infra13:27
*** rpioso|afk is now known as rpioso13:29
fungissbarnea: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/release/announce.yaml seems to run https://git.openstack.org/cgit/openstack/releases/tree/tools/announce.sh which calls into the release-notes entrypoint eventually ending up at https://git.openstack.org/cgit/openstack/releases/tree/openstack_releases/release_notes.py where the mail template (including default13:29
fungito address) is defined13:30
*** caphrim007 has quit IRC13:31
*** heyongli has quit IRC13:32
*** heyongli has joined #openstack-infra13:32
*** ramishra has quit IRC13:33
myoung|ruckpabelanger: o/ ahoy and good morning13:33
fungipas-ha: as frickler points out you can run some experimental jobs on 18.04 (bionic) now, but official openstack deliverables won't switch to it as a default until the beginning of the stein cycle, because 16.04 (xenial) was still the latest ubuntu lts at the time the rocky development cycle began and switching default test platforms mid-cycle is extremely disruptive13:33
*** armaan has quit IRC13:35
*** armaan has joined #openstack-infra13:35
*** jistr|mtg is now known as jistr13:37
*** ykarel has quit IRC13:40
*** ykarel has joined #openstack-infra13:40
*** njohnston has quit IRC13:40
*** heyongli has quit IRC13:42
*** heyongli has joined #openstack-infra13:42
*** jistr is now known as jistr|mtg13:43
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Add new modprobe element  https://review.openstack.org/57717013:46
*** mriedem has joined #openstack-infra13:46
*** r-daneel has joined #openstack-infra13:48
*** r-daneel has quit IRC13:48
pas-hafungi: frickler thanks!13:51
*** r-daneel has joined #openstack-infra13:52
*** heyongli has quit IRC13:52
*** heyongli has joined #openstack-infra13:52
*** lbragstad is now known as elbragstad13:52
*** agopi has joined #openstack-infra13:54
*** jcoufal has joined #openstack-infra13:59
*** heyongli has quit IRC14:02
*** heyongli has joined #openstack-infra14:03
*** hongbin has joined #openstack-infra14:07
*** jistr|mtg is now known as jistr14:09
*** mriedem has quit IRC14:09
*** peereb has quit IRC14:10
*** armaan has quit IRC14:10
*** armaan has joined #openstack-infra14:10
*** heyongli has quit IRC14:13
*** heyongli has joined #openstack-infra14:13
*** ykarel is now known as ykarel|away14:14
*** pblaho has quit IRC14:16
*** ykarel|away has quit IRC14:16
*** armaan has quit IRC14:18
*** armaan has joined #openstack-infra14:19
*** janki has quit IRC14:19
*** eernst has joined #openstack-infra14:21
*** udesale has joined #openstack-infra14:21
*** eernst has quit IRC14:21
*** eernst has joined #openstack-infra14:21
*** heyongli has quit IRC14:23
*** heyongli has joined #openstack-infra14:23
openstackgerritMerged openstack/diskimage-builder master: Remove redundant word  https://review.openstack.org/57646614:23
*** andreas_s_ has quit IRC14:29
*** andreas_s has joined #openstack-infra14:30
*** andreas_s has quit IRC14:33
*** heyongli has quit IRC14:33
*** heyongli has joined #openstack-infra14:33
*** andreas_s has joined #openstack-infra14:34
*** sree has joined #openstack-infra14:35
*** andreas_s has quit IRC14:37
*** andreas_s has joined #openstack-infra14:37
*** andreas_s has quit IRC14:38
*** andreas_s has joined #openstack-infra14:39
*** sree has quit IRC14:39
*** felipemonteiro_ has joined #openstack-infra14:40
pabelangermyoung|ruck: hello14:41
*** lpetrut has quit IRC14:42
*** heyongli has quit IRC14:43
*** felipemonteiro has quit IRC14:44
*** heyongli has joined #openstack-infra14:44
*** andreas_s has quit IRC14:45
*** yamahata has joined #openstack-infra14:46
*** lpetrut has joined #openstack-infra14:51
*** andreas_s has joined #openstack-infra14:52
*** heyongli has quit IRC14:54
*** tianhui_ has joined #openstack-infra14:54
*** heyongli has joined #openstack-infra14:54
*** sree has joined #openstack-infra14:56
*** tianhui has quit IRC14:57
*** neiloy has quit IRC14:58
*** neiloy has joined #openstack-infra14:58
*** lpetrut has quit IRC14:59
*** afazekas has quit IRC15:00
*** afazekas has joined #openstack-infra15:00
*** felipemonteiro_ has quit IRC15:00
*** eernst has quit IRC15:00
*** sree has quit IRC15:01
*** felipemonteiro_ has joined #openstack-infra15:01
myoung|ruckpabelanger: checking in re: https://trello.com/c/THLXPAqw/806-migrate-rdo-sf-jobs-to-zuulv3-openstack-infra-communication, weshay_PTO is out today - covering for him.  how goes it?15:01
*** r-daneel has quit IRC15:02
pabelangermyoung|ruck: wrong channel, we are doing migration in #rdo, since it doesn't affect openstack-infra.  I can follow up there?15:02
myoung|ruckpabelanger: of course, my bad.  apologies.15:02
* myoung|ruck moves to #rdo15:02
pabelangernp!15:02
*** eernst has joined #openstack-infra15:03
*** heyongli has quit IRC15:04
*** heyongli has joined #openstack-infra15:04
*** slaweq has quit IRC15:06
* fungi started to wonder why the openstack infra channel was getting links to an rdo trello board15:07
*** slaweq has joined #openstack-infra15:07
openstackgerritAndrey Kurilin proposed openstack-infra/openstack-zuul-jobs master: Remove playbooks for redundant rally legacy jobs  https://review.openstack.org/57714115:08
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref)  https://review.openstack.org/57690715:10
*** andreas_s has quit IRC15:11
clarkbalright I'm going to start requesting certificates. Will start with etherpad.o.o and run through the entire process with that one since I am familiar with the puppetry around it and once happy with that I'll probably do the other 15 in a more parallelized assembly line fashion15:12
clarkbthen do static.o.o last15:13
*** heyongli has quit IRC15:14
*** heyongli has joined #openstack-infra15:14
*** andreas_s has joined #openstack-infra15:15
fungiplease give the foundation web dev team a heads up when you do openstackid.org15:18
fungior even immediately before, just so they're not taken by surprise15:18
clarkbwill do15:19
*** Bhujay has quit IRC15:20
*** andreas_s has quit IRC15:20
fungithanks for doing all these!15:21
*** andreas_s has joined #openstack-infra15:21
*** jamesmcarthur has joined #openstack-infra15:23
*** heyongli has quit IRC15:24
clarkbfirst thing I notice is that for etherpad we have an fqdn and group hiera file. I think only the fqdn file is used based on my read of the puppet site.pp. I will update both but we should probably  try to clean up hiera as a followup15:25
*** heyongli has joined #openstack-infra15:25
clarkbinfra-root assume I have the hiera lock this morning for ssl cert updates, ping me if you need a hiera change15:25
mordredclarkb: ++15:25
*** andreas_s has quit IRC15:26
*** jcoufal has quit IRC15:28
*** janki has joined #openstack-infra15:29
fungithanks for the heads up, clarkb15:30
*** jistr is now known as jistr|afk15:30
fungidid we not finish migrating etherpad to the group file?15:30
clarkbI don't think so, there is no group = etherpad in site.pp15:30
fungi:(15:31
fungiyeah, i guess that was in service of building an etherpad01.o.o on xenial which hasn't happened yet15:31
clarkbthe files match each other currently with the new cert. I'm going to make tea while waiting for puppet to update the cert on the server. Then confirm everything works well end to end15:31
clarkbalso there are outstanding packethost hiera edits that I will leave as is for now15:32
*** jistr|afk is now known as jistr15:32
openstackgerritMerged openstack-infra/project-config master: use python3 to run launchpad commenting script  https://review.openstack.org/57729115:33
fungii expect that's ianw's staged credential changes he mentioned the other day15:33
clarkbya loosk like an update of the project names15:33
clarkbetherpad should update in ~10 minutes if I am reading ansible puppet logs correctly15:34
clarkbfungi: I guess I'll just mv the new crt file into place in the certs dir. Any value in keeping the old one? I expect not15:35
*** heyongli has quit IRC15:35
*** ykarel|away has joined #openstack-infra15:35
*** heyongli has joined #openstack-infra15:35
*** rpittau has quit IRC15:35
fungiclarkb: nope, we have it in git history there anyway15:35
fungiso technically, yes there is potential value in keeping the old one, but we already do it without you needing to put in any additional effort15:36
*** dpawlik has quit IRC15:37
*** gyee has joined #openstack-infra15:41
*** pbourke has quit IRC15:41
*** pbourke has joined #openstack-infra15:42
*** caphrim007 has joined #openstack-infra15:43
*** caphrim007 has quit IRC15:44
*** kgiusti has left #openstack-infra15:44
*** caphrim007 has joined #openstack-infra15:44
*** caphrim007 has quit IRC15:44
*** heyongli has quit IRC15:45
*** heyongli has joined #openstack-infra15:45
*** jaypipes has quit IRC15:46
*** caphrim007 has joined #openstack-infra15:47
*** zhangfei has quit IRC15:49
clarkbit appears that puppet updating the cert file doesn't tell apache to pick up that change. I am going to manually reload and then restart to have it see the new cert15:50
*** jistr is now known as jistr|off15:50
clarkbreload was sufficientl https://etherpad.openstack.org is done now15:50
fungiahh, right, we don't notify the service from those files15:51
fungii recall needing to do similar15:52
*** heyongli has quit IRC15:55
*** heyongli has joined #openstack-infra15:55
*** r-daneel has joined #openstack-infra15:56
*** r-mibu has quit IRC15:57
*** panda is now known as panda|off15:59
*** tesseract has quit IRC16:02
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Static driver: fix node registration  https://review.openstack.org/57746416:03
*** iyamahat has quit IRC16:03
*** iyamahat has joined #openstack-infra16:04
*** pcaruana has quit IRC16:04
*** heyongli has quit IRC16:05
*** heyongli has joined #openstack-infra16:06
*** boden has quit IRC16:06
*** nicolasbock has quit IRC16:06
mnaserany infra-root around to push https://review.openstack.org/#/c/577325/ through?16:08
mnaserthe change after it drops the current unused mirror.. but that needs a bit more 'manual' work to be done and it's friday so i won't bother anyone with that.. yet :-)16:08
*** Bhujay has joined #openstack-infra16:10
*** kgiusti has joined #openstack-infra16:10
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Log more information about what events trigger a reconfig  https://review.openstack.org/57587116:13
openstackgerritMerged openstack-infra/puppet-openstack_infra_spec_helper master: Use system-config script to install puppet  https://review.openstack.org/48194316:14
*** Bhujay has quit IRC16:14
*** Bhujay has joined #openstack-infra16:14
*** heyongli has quit IRC16:16
notmynameclarkb: FWIW, the reason I want https://review.openstack.org/#/c/564650/ to land is to make https://review.openstack.org/#/c/577467/ work well with alpine instead of ubuntu16:16
*** heyongli has joined #openstack-infra16:16
AJaegerinfra-root, what do you think of ianw's work on nodepool yaml file consolidation? It looks good to me but I like some other eyes on it...16:20
AJaegerinfra-root, change is https://review.openstack.org/57704516:20
*** slaweq has quit IRC16:21
*** bobh has joined #openstack-infra16:22
*** boden has joined #openstack-infra16:22
*** ykarel|away has quit IRC16:25
*** slaweq has joined #openstack-infra16:25
*** yamahata has quit IRC16:26
*** heyongli has quit IRC16:26
*** heyongli has joined #openstack-infra16:26
*** bobh has quit IRC16:28
clarkbask, docs, developer, and ethercalc are the next batch. Hiera is updated. Will reload apache on those hosts as soon as new files are written16:28
*** zoli is now known as zoli|gone16:29
mordred++16:29
*** iyamahat has quit IRC16:30
*** fried_rice is now known as fried_rolls16:31
fungiawesome16:31
*** jpich has quit IRC16:31
*** e0ne has quit IRC16:32
clarkbnotmyname: ok when I'm done with ssl cert refreshes I'll remove the test and repush16:33
notmynameclarkb: thanks16:34
*** jamesmcarthur has quit IRC16:35
*** heyongli has quit IRC16:36
*** heyongli has joined #openstack-infra16:36
*** kgiusti has quit IRC16:36
clarkbAJaeger: seems like a reasonable thing to do. I wonder if we couldn't get grafyaml to support something like that more natively btu we don't need that to implement ianw's idea16:37
*** dtantsur is now known as dtantsur|afk16:37
*** slaweq has quit IRC16:38
*** jesslampe has joined #openstack-infra16:38
*** e0ne has joined #openstack-infra16:41
*** e0ne has quit IRC16:42
mordredclarkb, AJaeger: ++16:43
*** jpena is now known as jpena|off16:43
*** iyamahat has joined #openstack-infra16:45
*** iyamahat has quit IRC16:46
*** elod has quit IRC16:46
*** heyongli has quit IRC16:46
*** yamahata has joined #openstack-infra16:47
*** heyongli has joined #openstack-infra16:47
*** iranzo has quit IRC16:47
*** psachin` has quit IRC16:48
openstackgerritMerged openstack-infra/zuul master: Don't process lock files as refs in tests  https://review.openstack.org/57041716:49
*** elod has joined #openstack-infra16:49
*** kgiusti has joined #openstack-infra16:49
*** derekh has quit IRC16:51
*** felipemonteiro__ has joined #openstack-infra16:51
openstackgerritMerged openstack-infra/zuul master: Clarify usage of the zuul_success variable  https://review.openstack.org/57505016:54
*** felipemonteiro_ has quit IRC16:55
*** iyamahat has joined #openstack-infra16:55
*** dpawlik has joined #openstack-infra16:56
clarkbAlright that batch of 4 is done and lgtm. I'm on to the next batch of 5 (so that I can do two more batches then the special ones at the end)16:56
openstackgerritMerged openstack-infra/zuul master: Log more information about what events trigger a reconfig  https://review.openstack.org/57587116:56
*** heyongli has quit IRC16:57
*** heyongli has joined #openstack-infra16:57
*** rpioso is now known as rpioso|eat16:59
*** dpawlik has quit IRC17:00
clarkbOne thing I notice about refstack is we write our actual cert to the snakeoil file location17:02
clarkbits working fine, but probably somethign worth cleaning up17:02
*** trown is now known as trown|lunch17:03
fungisounds like a puppet misconfig (may just need to explicitly set the filenames for those files as class parameters)17:04
fungiin most of our modules we only use the snakeoil path when no file content is provided and then default to another filename if file content is provided17:04
fungimaking it unnecessary to specify a filename unless you just want to override the location17:05
*** heyongli has quit IRC17:07
*** heyongli has joined #openstack-infra17:07
*** salv-orlando has joined #openstack-infra17:10
*** salv-orlando has quit IRC17:14
*** sree has joined #openstack-infra17:17
*** heyongli has quit IRC17:17
*** heyongli has joined #openstack-infra17:17
*** armaan has quit IRC17:21
*** sree has quit IRC17:22
*** armaan has joined #openstack-infra17:22
*** electrofelix has quit IRC17:23
*** armaan has quit IRC17:24
*** armaan has joined #openstack-infra17:24
*** jamesmcarthur has joined #openstack-infra17:26
*** heyongli has quit IRC17:27
*** heyongli has joined #openstack-infra17:28
*** camunoz has quit IRC17:29
*** jamesmcarthur has quit IRC17:29
*** jamesmcarthur has joined #openstack-infra17:29
*** gfidente has quit IRC17:31
*** amoralej is now known as amoralej|off17:33
clarkbgit.o.o cert should be updated17:36
*** jamesmcarthur has quit IRC17:36
*** heyongli has quit IRC17:38
*** heyongli has joined #openstack-infra17:38
fungiyup. "Expires on: 22 June 2019"17:38
fungilgtm17:38
clarkbfungi: mtreinish for firehose can I safely just restart mosquitto?17:38
fungiyes17:39
*** yamamoto has quit IRC17:42
clarkbfungi for the wiki is that going to be a manual cert update on the server?17:42
fungiyes, edit the files in /etc referred to in the vhost config on the server17:43
*** camunoz has joined #openstack-infra17:44
fungithough we should also keep hiera updated for future-proofing17:44
clarkb++ I'll be sure to edit hiera first then update /etc17:44
fungiout of curiosity, have you needed to update any chain certs?17:45
clarkbfungi: no17:45
clarkbI am diffing all of them first but hasn't been necesary yet17:45
fungiexcellent17:45
clarkband I'm letting puppet do all of the updates so that I know the hiera changes function17:47
clarkbwiki will be an exception to that17:47
fungiright17:47
fungibut overall, yes that's how i did them last time too17:47
clarkbI'm finding 4-5 in a batch a good number as I get that done by about the time the next puppet run begins17:48
*** heyongli has quit IRC17:48
clarkbthenI can verify the previous batch and start the next17:48
*** heyongli has joined #openstack-infra17:48
*** agopi has quit IRC17:48
*** agopi has joined #openstack-infra17:49
*** anteaya has joined #openstack-infra17:49
clarkbfungi: can you test mosquitto for me. service restart didn't work I had to stop start17:49
fungisure17:49
fungiecho | openssl s_client -connect firehose.openstack.org:8883 2>/dev/null | openssl x509 -text | grep 'Not After'17:53
fungiNot After : Jun 22 23:59:59 2019 GMT17:53
fungisame on port 44317:53
clarkbcool thanks17:54
fungialso `mosquitto_sub -h firehose.openstack.org --topic '#'` shows we're still streaming events17:54
*** lpetrut has joined #openstack-infra17:55
clarkbas a heads up review.o.o is in the next batch17:55
fungilooks like germqtt died though17:56
fungisocket.error: [Errno 111] Connection refused17:57
*** Bhujay has quit IRC17:57
fungiraised from /usr/lib/python2.7/socket.py line 575 in create_connection17:57
fungiJun 22 17:48:57 so right when mosquitto was restarted i guess17:57
fungii've restarted it now17:58
mtreinishclarkb: yeah firehose is safe to restart. Germqtt (and lpmqtt) have a bug where they don't handle a socket error correctly, but everything handles it seamlessly17:58
mtreinishhttps://review.openstack.org/561952 should fix that for germqtt17:58
fungilpmqtt seems to have been unphased17:58
fungier, unfazed too ;)17:58
*** heyongli has quit IRC17:58
*** heyongli has joined #openstack-infra17:58
*** felipemonteiro__ has quit IRC17:58
mtreinishI think there are other issues with lpmqtt in general, I haven't seen any events from it in a long time17:59
*** felipemonteiro__ has joined #openstack-infra17:59
mtreinishwhich is why it hasn't crashed, because it's only a bug when it tries to publish and the server is down17:59
*** lpetrut has quit IRC18:00
fungioh, yeah /var/spool/mail/lpmqtt is 5666203 lines long18:00
fungiso it hasn't been popping from that in a while18:00
fungimmm, it's checking but complaining about a message in the inbox which is not from lp18:01
*** r-daneel has quit IRC18:02
fungiJun 22 15:20:02 firehose01 lpmqtt[18654]: Initial filtering of Message ID: <20160919125818.5763.83678@juju-prod45-mojo-sso-machine-1>18:02
fungiJun 22 15:20:02 firehose01 lpmqtt[18654]: <20160919125818.5763.83678@juju-prod45-mojo-sso-machine-1> is not from LP18:02
fungii wonder whether i should clear out that mbox and let it start fresh? there are messages in it all the way back to Date: Tue, 19 Dec 2017 21:42:53 -000018:03
mtreinishoh, I wonder if they changed there mail format18:04
*** trown|lunch is now known as trown18:04
*** jamesmcarthur has joined #openstack-infra18:05
fungii'm going to move the mbox aside and let it start accumulating a new one and see if that changes anything18:05
mtreinishhttps://github.com/openstack-infra/lpmqtt/blob/master/lpmqtt/lp.py#L74-L9618:05
mtreinishif I'm reading that right it should log that it's not lp and just move on18:07
*** eernst has quit IRC18:07
fungii've rotated and compressed the old mbox. it's 37m if you want a copy for testing purposes18:07
mtreinishand we don't delete it or mark it as read if it's not lp (which will always log it on each iteration)18:07
*** heyongli has quit IRC18:08
*** heyongli has joined #openstack-infra18:09
*** salv-orlando has joined #openstack-infra18:10
mtreinishfungi: if you could just pop that message with the id from the log off and post that in a paste that would probably be enough18:11
fungii couldn't actually find it in the mbox18:11
mtreinishit definitely looks like it's coming from a canonical server, don't think anyone else would put juju in the hostname18:11
mtreinishhmm, ok18:11
fungileading me to wonder whether the mbox file is what we're actually serving over pop318:12
fungia new message just showed up in the mbox, so i suppose it is18:13
mtreinishI've never run Cyrus so I don't know where it stores things18:13
mtreinishon my dovecot instances I always explicit set the maildir, so that's not really helpful :)18:13
fungisure, and you're using maildir format not mbox18:14
fungithat fancy new maildir ;)18:14
*** anteaya has quit IRC18:15
*** salv-orlando has quit IRC18:15
fungithough the message received by firehose01.o.o at 18:11:22 is still in /var/spool/mail/lpmqtt18:18
fungihow often does it check messages?18:18
*** heyongli has quit IRC18:19
*** heyongli has joined #openstack-infra18:19
mtreinishtbh, I don't remember. Let me check the code18:22
fungii have a feeling it's less than a 10-minute interval at any rate18:22
fungiso something's happened to cause it to accumulate messages in /var/spool/mail/lpmqtt at least as of decemberish18:23
mtreinishlooks like https://github.com/openstack-infra/lpmqtt/blob/master/lpmqtt/daemon.py#L124 60 secs by default unless the imap server says something before that18:24
mtreinishit's a config flag though, let me check if we set that to something in the puppet18:24
fungiwe don't. the only non-credential options set in /etc/lpmqtt.conf are use_ssl = false, folder = INBOX and delete-old = true18:25
mtreinishheh, I don't see that option exposed in puppet-lpmqtt at all18:26
mtreinishyeah so the idle timeout is 60 secs so it should iterate and check messages manually at least that often18:26
fungiwhich means it's either not finding these or it's failing to delete them18:27
*** eernst has joined #openstack-infra18:28
fungihttp://paste.openstack.org/show/724148 is the oldest message which has been sitting in /var/spool/mail/lpmqtt since i rotated that file18:28
mtreinishit only deletes messages it thinks are from lp right before it returns the dict to send over mqtt: https://github.com/openstack-infra/lpmqtt/blob/master/lpmqtt/lp.py#L106-L11118:28
openstackgerritMerged openstack-infra/project-config master: Add Vitrage to StoryBoard  https://review.openstack.org/57665118:29
*** heyongli has quit IRC18:29
*** heyongli has joined #openstack-infra18:29
*** e0ne has joined #openstack-infra18:31
clarkbfungi: is openstackid being puppeted again?18:33
fungiclarkb: yes, as of a couple days ago18:34
clarkbreview.o.o is puppeting now18:35
clarkbthe above project-config merge is causing it to run a little more slowly than I would like but I shall be paitent18:36
fungithanks, i'm actually waiting on that one to get applied to the storyboard server so i can start their import ;)18:37
*** kjackal has quit IRC18:39
*** heyongli has quit IRC18:39
*** anteaya has joined #openstack-infra18:39
clarkbfungi: storyboard will be in this run as well (though much later in the else playbook) are you ok for me to reload apache when it writes the new cert?18:39
*** heyongli has joined #openstack-infra18:40
*** jamesmcarthur has quit IRC18:40
clarkbreview.o.o puppet is done, reloading apache now18:41
*** iranzo has joined #openstack-infra18:41
fungiyeah, that's fine18:41
fungipuppet isn't going to do that for you anyway18:42
pabelangerHmm, did we not have a linter that automatically sorted our yaml files?18:42
fungidepends on the file18:42
pabelangerthought AJaeger might have written one18:42
mtreinishfungi: tracing throught the code manually it works in identifying that's from lp. My only thought is that it's something with the formatting from the imaplib response on the fetch() call18:42
*** yamamoto has joined #openstack-infra18:42
mtreinishbecause there is a bunch of weird things like use data[0][1] for the message contents and data[0][0] for the id18:43
mtreinishand I don't remember the format from the lib, but I know that I would have found that by manually poking things18:43
fungimtreinish: i'm starting to wonder if something's up with cyrus-imapd18:44
mtreinishis there anything in the logs for it?18:45
clarkbfungi: for openstackid.org hostmaster@openstack.org isn't an option, do we have access to the mailbox for openstackid.org?18:46
*** jamesmcarthur has joined #openstack-infra18:46
fungiclarkb: huh, wow. looks like csc isn't listing our contacts in the domain registrations any longer according to whois18:47
clarkbfungi: ya just noticed that18:48
*** yamamoto has quit IRC18:48
*** rosmaita has joined #openstack-infra18:48
fungithey probably think they're doing us a service by protecting our domain contacts from spammers18:48
clarkbthe SOA record has hostmaster.openstack.org in it though18:49
clarkbI guess namecheap doesn't look that up though18:49
*** heyongli has quit IRC18:49
fungiwe could probably set up a mailbox/forwarder somewhere and add an mx record to there18:50
*** heyongli has joined #openstack-infra18:50
mtreinishfungi: I wonder if it's gdpr related18:50
fungibut that seems like a lot of hassle18:50
fungimtreinish: it almost certainly is18:50
clarkbthere is http based validation where I can stick a file in a known location or dns based where I have to create a cname record18:50
clarkbthe dns option doesn't actually look like what we want18:51
clarkbbut http may be doable18:51
*** janki has quit IRC18:52
*** camunoz has quit IRC18:52
fungii just noticed that the registrar i use for my personal domains started doing the same thing18:53
fungiseeing if they give me an option to fix it back18:54
fungiDue to the GDPR, we are no longer able to publish contact information in Whois without explicit consent from the contact. Without consent, the message "REDACTED FOR PRIVACY" will be displayed in each field. This may prevent the validation of domain transfers and SSL certificates.18:55
clarkbI can't help but think this is a major overreach. Aiui you only have to remove that data if asked18:55
clarkbwe didn't ask18:55
clarkbya I think thats way more CYA than actual application of the regulations18:56
*** fried_rolls is now known as efried_pto18:56
fungiso my registrar added a consent management system for which you have to actively have them send a consent request to each e-mail address18:56
fungiand then follow a confirmation url they send you, and then check several checkboxes for each domain18:57
*** heyongli has quit IRC19:00
*** heyongli has joined #openstack-infra19:00
clarkbdue to openstackid.org being special I'm going to do static.o.o next19:02
clarkbthe last batch is done and lgtm19:02
fungiexcellent19:05
*** heyongli has quit IRC19:10
*** heyongli has joined #openstack-infra19:10
*** salv-orlando has joined #openstack-infra19:11
*** yolanda__ has joined #openstack-infra19:12
fungiclarkb: i'm finding articles which suggest that the registrars, as a whole, would rather see whois go away entirely so they're using "gdpr compliance" as an excuse to kill it off19:13
clarkbfungi: re static.o.o hiera has cert data for releases.o.o which is different than static.o.o's cert is that just no longer used?19:15
clarkbreleases is an altname on the new and old static.o.o certs19:16
*** salv-orlando has quit IRC19:16
fungiclarkb: yeah, as long as the releases site is using the standard template now, that's just cruft which can be cleaned up19:16
clarkbok I won't worry about that now but noting it on the etherpad as a lceanup19:16
fungiand i expect it is, or else we'd have seen it expire last year19:16
*** yolanda_ has quit IRC19:16
clarkbya vhost is set to use the sahred cert19:17
*** jmorgan1 has quit IRC19:17
*** yolanda__ has quit IRC19:18
*** diablo_rojo has joined #openstack-infra19:19
clarkbstatic.o.o hiera is updated with the new cert and key19:20
*** heyongli has quit IRC19:20
*** heyongli has joined #openstack-infra19:20
*** iranzo has quit IRC19:22
openstackgerritColleen Murphy proposed openstack-infra/puppet-nodepool master: Update libxslt package name for Xenial  https://review.openstack.org/57751919:27
fungithanks!19:28
*** ihar has quit IRC19:28
*** ihar has joined #openstack-infra19:29
*** anteaya has quit IRC19:30
fungimtreinish: i notice that the only mail.log entries associated with imap logins are spaced roughly 30 minutes apart and look like http://paste.openstack.org/show/724157/19:30
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Update Gemfile for Zuulv3  https://review.openstack.org/57752119:30
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Add beaker tests for nodepool  https://review.openstack.org/57752219:30
*** heyongli has quit IRC19:30
fungimtreinish: i'll see if i can figure out why it's failing to squat (not sure i even want to know?)19:30
*** heyongli has joined #openstack-infra19:31
*** cmurphy is now known as cmurphy_vacation19:31
*** udesale has quit IRC19:35
*** rpioso|eat is now known as rpioso19:39
*** heyongli has quit IRC19:41
*** heyongli has joined #openstack-infra19:41
*** yamamoto has joined #openstack-infra19:44
clarkbalright reloading apache on static.o.o now19:48
*** yamamoto has quit IRC19:48
*** udesale has joined #openstack-infra19:49
clarkblgtm19:50
clarkbtime for lunch then back to finish up with openstackid19:50
*** heyongli has quit IRC19:51
*** heyongli has joined #openstack-infra19:51
*** udesale has quit IRC19:58
mtreinishfungi: hmm, maybe the interval is 30min not 60secs. That rings a bell because I remember the mqtt side being pretty bursty20:00
mtreinishI just don't recall where in the code that number is coming from20:00
mtreinishheh, yeah failing to squat has all sorts of interpretations :p20:01
*** heyongli has quit IRC20:01
*** heyongli has joined #openstack-infra20:01
*** jamesdenton has quit IRC20:02
*** jmorgan1 has joined #openstack-infra20:03
*** heyongli has quit IRC20:11
mtreinishfungi: I might be able to change the lpmqtt code to not use a search call. Although the only way I can think of doing it is to just get all messages instead of just unseen messages (which will only work in the delete case)20:12
*** heyongli has joined #openstack-infra20:12
*** salv-orlando has joined #openstack-infra20:12
*** jamesmcarthur has quit IRC20:13
fungimtreinish: apparently those squat failures just mean we're not building a search index, yeah20:14
*** jamesmcarthur has joined #openstack-infra20:15
*** salv-orlando has quit IRC20:16
mtreinishfungi: any idea on how to fix that, or should I look at modifying lpmqtt to avoid using search?20:19
fungiwell, we could run the indexing service, according to the documentation i found, we just don't seem to have it turned on20:20
*** heyongli has quit IRC20:22
*** heyongli has joined #openstack-infra20:22
mtreinishhmm, this wasn't always a problem I wonder what changed. Maybe we were running it in the past, or it isn't really needed20:23
kmallocfungi: if our pypi mirror(s) somehow lost a release of a package, what is the correct way to fix it?20:23
kmallocfungi: looks like pycryptodomex 3.6.2 is not in our mirror.. or one of them.20:23
kmallocbut 3.6.3 is, and 3.6.2 is on pypi itself.20:23
smcginniskmalloc: They decided to remove it from pypi, if what I saw earlier was correct.20:25
smcginniskmalloc: Yeah, 3.6.2 is gone: https://test.pypi.org/project/pycryptodomex/3.6.2/20:26
kmallocah20:27
smcginniskmalloc: A requirements patch is going through right now I think to blacklist that version.20:27
kmallocahhh20:27
kmallocsmcginnis: it was back when i looked :P20:27
kmallocguess i hit a cache20:27
*** jamesmcarthur has quit IRC20:27
smcginnis:)20:27
smcginniskmalloc: Just checked, this has probably an hour two yet in the gate queue - https://review.openstack.org/#/c/577494/20:28
*** jamesmcarthur has joined #openstack-infra20:28
smcginnisSo I misspoke. Not blacklisted, but at least letting us move past it.20:28
kmallocyeah20:29
kmalloci see it20:29
kmallocwoo20:29
clarkbis this something that needs to be promoted to the head of the queue?20:30
smcginnisclarkb: Looks like the jobs are running. Not sure if you can do anything to affect that?20:30
kmallocclarkb: it is already on it's way running through jobs.20:30
kmallocsmcginnis: yeah i found out about that trying to bump flask ;)20:31
smcginnisAh, didn't look on the status of that one.20:31
smcginniskmalloc: Cool, see now you rebased it on top of that bump. Perfect!20:32
kmallocyep!20:32
*** heyongli has quit IRC20:32
fungiright, our pypi mirror respects deletions from pypi. our persistent wheel cache does not, so if we need to clear any wheels from there we have to do that manually20:32
*** heyongli has joined #openstack-infra20:32
kmallocsmcginnis: i have no words how much better life is when i can do testing: "with app.client() as c: c.<do-request_things>" instead of all the headaches i deal with now spinning up a wsgiref app along side.20:33
*** jamesmcarthur has quit IRC20:33
smcginniskmalloc: Nice!! I really need to take a look at some of that work and see what I can steal^w borrow there.20:34
kmallocsmcginnis: well, you might want to wait until i get an api or two moved, there is a lot of wonky compat stuff. i'll be writing up a blog post once i get more than "well, it's some code shuffling and it could do something interesting"20:34
kmallocbut it is def. moving forward.20:35
smcginniskmalloc: ++20:35
clarkbok I'm back from lunch and gonna do the dns dance to verify cert signing requset for openstackid.org now20:35
kmallocclarkb: yay dns dance... =/20:35
smcginnisI would imagine keeping backwards compat is going to be the trickiest part of the whole thing with flask.20:35
clarkbkmalloc: apparently registrars are masking whois info across the board now so you can' rely on contact info from that anymore20:36
clarkbyay misapplying eu regulations20:36
*** armaan has quit IRC20:36
kmallocclarkb: yeah, i noticed namecheap did that20:36
smcginnisOn the plus side, that used to be something they would all charge for.20:36
*** armaan has joined #openstack-infra20:36
clarkbsmcginnis: sort of, the thing you could apy for was for them to proxy that contact stuff for you20:37
clarkbthis is wholesale removal from what I've seen20:37
clarkbproxying the contact info would likely have worked in this case20:37
smcginnisAh20:37
*** rh-jelabarre has joined #openstack-infra20:38
kmallocyeah namecheap is just default proxying.20:38
kmalloca lot of places are just wiping it out20:38
*** sree has joined #openstack-infra20:41
*** heyongli has quit IRC20:42
*** heyongli has joined #openstack-infra20:43
*** yamamoto has joined #openstack-infra20:45
*** sree has quit IRC20:46
clarkbnow I get to wait for some unspecified time period for them to check my dns records instead20:48
*** yamamoto has quit IRC20:49
fungii'm gonna go grab dinner and run some errands, but will check back in a while20:51
clarkbI'm spinning up other work now fwiw20:51
clarkbsince it could be hours for them to verify this signing request20:51
clarkbI take that back it just completed20:52
*** heyongli has quit IRC20:52
*** heyongli has joined #openstack-infra20:53
openstackgerritClark Boylan proposed openstack-infra/bindep master: Use the distro package to determine distro  https://review.openstack.org/56465020:53
clarkbnotmyname: fungi ^ test removed20:53
*** eernst has quit IRC20:53
*** e0ne has quit IRC20:54
*** kgiusti has left #openstack-infra20:57
*** kjackal has joined #openstack-infra20:57
*** rfolco_ has quit IRC21:00
clarkb#status log Removed docs-draft CNAME record to static.o.o as the doc drafts are no longer hosted separately21:00
openstackstatusclarkb: finished logging21:00
*** heyongli has quit IRC21:03
*** heyongli has joined #openstack-infra21:03
*** trown is now known as trown|outtypewww21:07
*** ihar has quit IRC21:08
*** ihar has joined #openstack-infra21:10
*** salv-orlando has joined #openstack-infra21:13
*** heyongli has quit IRC21:13
*** heyongli has joined #openstack-infra21:13
*** yolanda has joined #openstack-infra21:15
*** ldnunes has quit IRC21:16
*** salv-orlando has quit IRC21:17
*** heyongli has quit IRC21:23
*** heyongli has joined #openstack-infra21:23
clarkb#status log SSL cert rotation for June 2018 completed.21:25
openstackstatusclarkb: finished logging21:25
*** boden has quit IRC21:31
*** heyongli has quit IRC21:33
*** heyongli has joined #openstack-infra21:34
*** felipemonteiro__ has quit IRC21:37
*** jamesmcarthur has joined #openstack-infra21:40
*** rh-jelabarre has quit IRC21:42
openstackgerritClark Boylan proposed openstack-infra/openstack-zuul-jobs master: Add fedora 28 support to kata runsh job  https://review.openstack.org/57756321:43
*** heyongli has quit IRC21:44
*** heyongli has joined #openstack-infra21:44
clarkbmnaser: ^ fyi21:44
mnaseryay a role21:45
mnaserlet me see21:45
*** yamamoto has joined #openstack-infra21:46
*** jamesmcarthur has quit IRC21:46
mnaserclarkb: looks godo to me, although some cleanups can be done to speed up the ansible stuff but shouldn't be too much of an issue21:46
*** caphrim007_ has joined #openstack-infra21:48
*** caphrim007_ has quit IRC21:49
clarkbmnaser: I've just rechecked my existing PR to make sure that doesn't regress ubuntu21:50
clarkbif it doesn't I'll get a change up to add the fedora job to the proxy repo and we can recheck again21:50
clarkbactually that might require we merge the above change first so that might end up being a monday thing21:50
*** yamamoto has quit IRC21:51
*** caphrim007 has quit IRC21:52
mnaserclarkb: I guess given this only affects Kata and nothing else we could merge, it does look good to me21:52
*** heyongli has quit IRC21:54
*** heyongli has joined #openstack-infra21:54
clarkbmnaser: any idea where golang for centos is installed from?21:55
clarkbno rush we will figure out fedora first but I think thats the biggest centos question from me21:55
*** anteaya has joined #openstack-infra21:56
mnaserclarkb: afaik they pull upstream21:59
openstackgerritClark Boylan proposed openstack-infra/project-config master: Run kata-runsh-fedora-28 on kata proxy  https://review.openstack.org/57756622:01
*** neiloy has quit IRC22:02
*** roman_g has quit IRC22:02
*** heyongli has quit IRC22:04
*** heyongli has joined #openstack-infra22:04
*** hamzy_ has joined #openstack-infra22:06
*** hamzy has quit IRC22:08
*** edmondsw has joined #openstack-infra22:09
*** edmondsw has quit IRC22:13
*** salv-orlando has joined #openstack-infra22:13
*** heyongli has quit IRC22:14
*** heyongli has joined #openstack-infra22:15
*** rh-jelabarre has joined #openstack-infra22:15
*** tpsilva has quit IRC22:17
*** rosmaita has quit IRC22:17
*** salv-orlando has quit IRC22:18
*** roman_g has joined #openstack-infra22:23
*** florianf has quit IRC22:24
*** heyongli has quit IRC22:25
*** heyongli has joined #openstack-infra22:25
clarkbconfig-core http://logs.openstack.org/74/74/c3a54a71613ec4736adcdb363be6c17b01ccd9f1/third-party-check/kata-runsh/6aef7d8/ara-report/file/91d4770e-f7bf-4c67-a731-a195c2a89f76/ shows that https://review.openstack.org/577563 doesn't break ubuntu, if we can get that in then https://review.openstack.org/577566 it will help us make more progress with kata ci22:27
*** hongbin has quit IRC22:33
*** heyongli has quit IRC22:35
*** heyongli has joined #openstack-infra22:35
openstackgerritClark Boylan proposed openstack-infra/system-config master: Don't use snakeoil files with real certs  https://review.openstack.org/57757122:39
clarkbfungi: ^ is the cleanup for refstack that we talked about22:40
clarkbcleaning up the releases cert from hiera will be a monday task I think since I want to research that a bit more and do other general hiera cleanups22:41
*** heyongli has quit IRC22:45
*** heyongli has joined #openstack-infra22:45
*** yamamoto has joined #openstack-infra22:47
*** yamamoto has quit IRC22:53
*** myoung has joined #openstack-infra22:53
*** rnoriega_ has joined #openstack-infra22:54
*** pabelanger_ has joined #openstack-infra22:54
*** _ari__ has joined #openstack-infra22:54
*** haleyb_ has joined #openstack-infra22:54
*** hashar has quit IRC22:54
*** rasca- has joined #openstack-infra22:54
*** otherwiseguy_ has joined #openstack-infra22:55
fungiif there were some easy way to find unreferenced heira keys, that would be awesome22:55
*** heyongli has quit IRC22:55
*** heyongli has joined #openstack-infra22:56
*** _ari_ has quit IRC22:56
*** weshay_PTO has quit IRC22:57
*** weshay has joined #openstack-infra22:57
*** pabelanger has quit IRC22:57
*** rasca has quit IRC22:57
*** otherwiseguy has quit IRC22:57
*** rnoriega has quit IRC22:58
*** haleyb has quit IRC22:58
*** myoung|ruck has quit IRC22:58
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add fedora 28 support to kata runsh job  https://review.openstack.org/57756323:01
*** myoung is now known as myoung|off23:03
*** rlandy has quit IRC23:05
*** heyongli has quit IRC23:06
*** heyongli has joined #openstack-infra23:06
*** salv-orlando has joined #openstack-infra23:14
*** heyongli has quit IRC23:16
*** heyongli has joined #openstack-infra23:16
*** dhill_ has quit IRC23:18
*** salv-orlando has quit IRC23:19
*** heyongli has quit IRC23:26
*** heyongli has joined #openstack-infra23:26
*** heyongli has quit IRC23:36
*** heyongli has joined #openstack-infra23:37
*** tosky has quit IRC23:41
clarkbfungi: mnaser https://review.openstack.org/#/c/577566/ any chance we can get that one in too?23:44
clarkbI guess it isn't a huge rush with the week ending but this way I can get fedora running over the weekend if I end up working23:44
mnaserclarkb: lgtm23:45
clarkb(I shouldn't do that but its the sort of thing that is good for doing while watching the world cup)23:45
pabelanger_clarkb: +3, left comment23:45
*** pabelanger_ is now known as pabelanger23:45
clarkbpabelanger: I don't think kata would necessarily work that way? I dunno they are only doing fedora27 now in existing ci23:46
*** heyongli has quit IRC23:47
clarkbpabelanger: the idea is get the base set of tests working via thirdparty check then have them run it through its paces and then decide if tehy want a proper tenant23:47
*** heyongli has joined #openstack-infra23:47
pabelangerokay, once kata is ready, they should consider fedora-latest for their tenant too, might be less jobs changing, but think it will be easier, if we are going to replace that image every 6 months23:48
*** yamamoto has joined #openstack-infra23:49
openstackgerritMerged openstack-infra/project-config master: Run kata-runsh-fedora-28 on kata proxy  https://review.openstack.org/57756623:50
*** yamamoto has quit IRC23:54
*** rpioso is now known as rpioso|afk23:57
*** heyongli has quit IRC23:57
*** heyongli has joined #openstack-infra23:57
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove playbooks for redundant rally legacy jobs  https://review.openstack.org/57714123:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!