Tuesday, 2020-07-07

openstackgerritIan Wienand proposed opendev/system-config master: Cleanup grafana.openstack.org  https://review.opendev.org/73962500:04
*** ryohayakawa has joined #opendev00:07
*** hrw has quit IRC00:16
*** hrw has joined #opendev00:21
*** hrw has quit IRC00:36
*** hrw has joined #opendev00:36
mnasercan i get a quick peek at https://review.opendev.org/#/c/739602/ ?00:38
openstackgerritMerged opendev/system-config master: Graphite container deployment  https://review.opendev.org/73812500:41
openstackgerritMohammed Naser proposed openstack/project-config master: add vexxhost/openstack-tools  https://review.opendev.org/73962700:45
*** hashar has joined #opendev00:46
openstackgerritMerged openstack/project-config master: Add openstack/devstack-plugin-ceph in vexxhost tenant  https://review.opendev.org/73960200:48
*** DSpider has quit IRC00:59
*** hashar has quit IRC01:11
openstackgerritMerged openstack/project-config master: Add "Review-Priority" label to qa projects  https://review.opendev.org/73955001:11
ianwok, deploy has now failed because the openedge mirror is down01:12
fungioh, we should add it to the emergency disable list01:17
ianwyeah, have done, not sure if it's a longer term thing and we want to remove from the inventory from prior conversation01:17
fungiunless we still have a disable list in system-config group vars01:17
donnydWorst case it will be back in Oct02:18
*** ysandeep|away is now known as ysandeep02:26
openstackgerritmelissaml proposed openstack/diskimage-builder master: Switch from unittest2 compat methods to Python 3.x methods  https://review.opendev.org/73964503:12
*** factor has quit IRC03:26
*** factor has joined #opendev03:27
*** factor has joined #opendev03:28
*** ykarel|away is now known as ykarel04:04
*** sgw1 has quit IRC04:39
*** bhagyashris is now known as bhagyashris|brb05:50
*** marios has joined #opendev06:01
*** bhagyashris|brb is now known as bhagyashris06:16
openstackgerritIan Wienand proposed opendev/system-config master: Cleanup grafana.openstack.org  https://review.opendev.org/73962507:01
openstackgerritIan Wienand proposed opendev/system-config master: graphite: statsd timer settings  https://review.opendev.org/73966507:05
*** dtantsur|afk is now known as dtantsur07:18
*** tosky has joined #opendev07:36
*** hashar has joined #opendev07:42
*** fressi has quit IRC07:53
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** kevinz has joined #opendev08:13
openstackgerritSorin Sbarnea (zbr) proposed opendev/base-jobs master: Avoid rsync chgrp errors on promote  https://review.opendev.org/73967408:19
*** DSpider has joined #opendev08:24
openstackgerritMerged openstack/diskimage-builder master: Use kpartx option to update partition mappings  https://review.opendev.org/72882408:27
openstackgerritRiccardo Pittau proposed openstack/diskimage-builder master: Convert multi line if statement to case  https://review.opendev.org/73447908:31
zbrAny infra-root online to check my AFS rsync patch? ^08:33
*** ysandeep is now known as ysandeep|lunch08:34
*** priteau has joined #opendev08:49
kevinzHi Infra, the Linaro-us mirror looks died08:51
kevinzerror: Status code: 403 for https://mirror.regionone.linaro-us.opendev.org/centos/8/AppStream/aarch64/os/repodata/repomd.xml (IP: 2604:1380:4111:3e54:f816:3eff:fea4:899d) (https://mirror.regionone.linaro-us.opendev.org/centos/8/AppStream/aarch64/os/repodata/repomd.xml)08:53
kevinzAjaeger:ianw: This mirror has strangely shutdowned...And I reboot it but there is no files on the link: http://mirror01.regionone.linaro-us.opendev.org/08:57
hasharzbr: I am not an infra root but I have done a quick review.  Those options can be passed as parameters to the ansible module ( see https://docs.ansible.com/ansible/2.7/modules/synchronize_module.html )09:02
zbrtrue, leftovers from when i was testing more advanced options :D09:03
openstackgerritSorin Sbarnea (zbr) proposed opendev/base-jobs master: Avoid rsync chgrp errors on promote  https://review.opendev.org/73967409:13
AJaegerkevinz: I cannot help, let's ask infra-root to help with the linaro mirror ^09:16
*** fressi has joined #opendev09:25
fricklerkevinz: infra-root: there's a kernel panic from the afs module on mirror.regionone.linaro-us.opendev.org , I'm trying another reboot, if that doesn't help, we'll need some kernel expert I guess09:26
fricklerkevinz: seems to be working better now09:44
*** ysandeep|lunch is now known as ysandeep09:48
*** ryohayakawa has quit IRC09:53
*** tosky has quit IRC10:00
*** tosky has joined #opendev10:01
*** hashar has quit IRC10:07
hrwfrickler: thanks10:34
*** bhagyashris is now known as bhagyashris|brb10:37
fricklerinfra-root: fyi, that's the kernel oops, it looks to me like it might have choked on some corrupted cache file after the previous unclean shutdown. probably difficult to reproduce but maybe worth forwarding to afs developers? http://paste.openstack.org/show/795602/10:45
*** priteau has quit IRC11:00
*** tkajinam has quit IRC11:04
*** bhagyashris|brb is now known as bhagyashris11:12
openstackgerritMerged zuul/zuul-jobs master: Drop support for ansible 2.6  https://review.opendev.org/72715811:32
*** ysandeep is now known as ysandeep|afk11:41
*** hashar has joined #opendev11:52
*** priteau has joined #opendev11:55
*** ysandeep|afk is now known as ysandeep12:06
*** priteau has quit IRC12:12
*** priteau has joined #opendev12:19
fungijust a heads up, i'm around but catching up after a few days off, and today is grocery day plus a number of meetings, so i can help with urgent issues but likely not much else before tomorrow13:03
*** sgw1 has joined #opendev13:11
*** rh-jelabarre has joined #opendev13:21
*** wrenchyfrenchy has left #opendev13:39
fungiokay, stepping out to fetch our grocery order, shouldn't be more than an hour. back soon13:44
openstackgerritThierry Carrez proposed openstack/project-config master: Define maintain-github-mirror job  https://review.opendev.org/73822813:53
openstackgerritThierry Carrez proposed openstack/project-config master: Define maintain-github-mirror job  https://review.opendev.org/73822814:01
kevinzfrickler: thanks a lot :-D14:04
*** bhagyashris is now known as bhagyashris|dinn14:09
*** _marc-antoine_ has joined #opendev14:27
openstackgerritSlawek Kaplonski proposed opendev/irc-meetings master: Update neutron team meeting time  https://review.opendev.org/73978014:40
openstackgerritSorin Sbarnea (zbr) proposed opendev/base-jobs master: Avoid rsync chgrp errors on promote  https://review.opendev.org/73967414:41
*** mlavalle has joined #opendev14:46
*** fressi has quit IRC14:52
*** bhagyashris|dinn is now known as bhagyashris14:56
openstackgerritDrew Walters proposed openstack/project-config master: Publish Airship governance  https://review.opendev.org/73979015:17
openstackgerritgugug proposed openstack/diskimage-builder master: Remove glance-registry  https://review.opendev.org/73979615:26
openstackgerritgugug proposed openstack/diskimage-builder master: Remove glance-registry  https://review.opendev.org/73979615:30
*** ysandeep is now known as ysandeep|away15:32
openstackgerritgugug proposed openstack/diskimage-builder master: Remove glance-registry  https://review.opendev.org/73979615:33
*** dtantsur is now known as dtantsur|brb15:49
zbrcorvus: i discovered with pabelanger that fedora-31 lots ability to create swaps over last night :) affecting all jobs on ansible zuul.15:54
zbri was trying to identify if same issue happened on opendev too15:55
corvuszbr: there's a "build_node" field in logstash, but i don't see any fedora-31 entries for it15:56
corvuszbr: but i can check the zuul scheduler logs15:56
*** ykarel is now known as ykarel|away15:57
corvuszbr: here are the builds from today: http://paste.openstack.org/show/795628/15:58
corvuszbr: and yesterday: http://paste.openstack.org/show/795629/15:59
corvuszbr: looks like it's just a couple of jobs, so you can probably query the zuul dashboard for those jobs and find the builds (rather than going to each build id individually)16:00
*** marios has quit IRC16:02
*** ysandeep|away is now known as ysandeep16:21
*** icarusfactor has joined #opendev16:29
openstackgerritMerged zuul/zuul-jobs master: Allow deleting workspace after running terraform destroy  https://review.opendev.org/73877116:29
*** factor has quit IRC16:32
*** factor has joined #opendev16:32
*** icarusfactor has quit IRC16:34
*** factor has quit IRC16:34
*** chandankumar is now known as raukadah16:38
*** dtantsur|brb is now known as dtantsur16:41
*** _marc-antoine_ has quit IRC17:05
*** dtantsur is now known as dtantsur|afk17:38
*** hashar has quit IRC17:47
*** priteau has quit IRC17:52
hrwis there a way to add other fonts to etherpad.opendev.org list of fonts?18:09
hrwI can write userscript to do it on client side but prefer to check first is it doable on server side ;D18:10
fungihrw: possible it can be done with a plugin, if you want to look at the available etherpad plugins18:10
fungiwe can add plugins to our image build fairly easily, and hold a test node for more thorough interactive exercisinfg18:10
hrwfungi: thanks, will check18:13
fungialso possible it's configurable server-side, i haven't explored the etherpad config file that deeply, nor what new features came with the theming enhancements in recent versions18:14
hrwfungi: is config somewhere on opendev.org git?18:19
hrwcause when I look at etherpad plugins list I feel bad18:19
fungiheh, yeah just a sec, it's in opendev/system-config but i'll find it18:20
fungihrw: right now we only have the headings plugin added: https://opendev.org/opendev/system-config/src/branch/master/docker/etherpad/Dockerfile#L20-L2418:23
hrwlooked at plugins. will write userscript18:31
fungiheh, okay18:37
fungii totally understand18:38
fungisome aspects of etherpad are not as configurable as others18:38
fungiout of curiosity, is the change you need to make an accessibility related one? maybe we could help with a pr upstream, and/or some css changes on our deployment18:47
ianwmeeting in 2 minutes @ #opendev-meeting18:58
*** diablo_rojo has joined #opendev19:00
*** hashar has joined #opendev19:34
openstackgerritMonty Taylor proposed opendev/system-config master: Don't install the track-upstream cron on review-test  https://review.opendev.org/73984019:40
openstackgerritMichael Johnson proposed openstack/diskimage-builder master: Fix DIB scripts python version  https://review.opendev.org/73984419:53
ianwhuh, arm64 job on 739840 failed with "fatal: unable to access 'https://github.com/infraly/k8s-on-openstack/': GnuTLS recv error (-54): Error in the pull function." ... transient error i hope20:07
fungithat seems arbitrary enough at least20:08
fungior maybe the request had to pass through kentucky20:08
corvushrm, starting to smell like wildfire here...20:19
ianwwere the iptables drops that are currently on gitea-lb persisted anywhere?20:34
ianwok there is a ~clarkb/china_unicom_ranges20:37
clarkbianw I did a for loop over the contents of that file. Check my user history gor the exact command20:38
ianwyeah, i did :)  just thinking about how we can test now20:39
fungiwell, there are many options for testing20:44
fungiwe could tcpdump to see which of those cidrs has a lot of requests coming from it and then just delete the rule covering it20:44
fungibut in reality i don't think we need to fiddle with the current ruleset to test20:45
ianwyeah, just switch one of the backends in haproxy to 308120:45
fungithere are similar requests coming from other netblocks (last we checked)20:45
fungiwhich is why we can still see a significant open connections count in cacti20:45
fungiand elevated (but now not dire) processor utilization across the backends20:46
ianwonce it deploys i'll poke at a manual redirect and see what happens, then propose a config file update20:47
openstackgerritMerged opendev/system-config master: gitea: Add reverse proxy option  https://review.opendev.org/73872121:07
openstackgerritMerged opendev/system-config master: gitea: crawler UA reject rules  https://review.opendev.org/73872521:10
openstackgerritMerged opendev/system-config master: graphite: statsd timer settings  https://review.opendev.org/73966521:10
*** tobiash has quit IRC22:12
*** DSpider has quit IRC22:22
*** avass has quit IRC22:25
*** calcmandan has quit IRC22:42
*** calcmandan has joined #opendev22:42
*** jentoio has quit IRC22:43
*** jentoio has joined #opendev22:45
*** seongsoocho has quit IRC22:47
*** gmann has quit IRC22:47
*** mwhahaha has quit IRC22:50
*** mnaser has quit IRC22:50
*** jbryce has quit IRC22:50
*** knikolla has quit IRC22:50
*** johnsom has quit IRC22:51
*** jrosser has quit IRC22:51
*** hillpd has quit IRC22:51
*** aannuusshhkkaa has quit IRC22:51
*** rpittau has quit IRC22:51
*** wendallkaters has quit IRC22:52
*** rm_work has quit IRC22:52
*** rajinir_ has quit IRC22:52
*** jentoio has quit IRC22:52
*** Open10K8S has quit IRC22:53
*** lseki has quit IRC22:53
*** zbr has quit IRC22:53
*** diablo_rojo_phon has quit IRC22:53
*** donnyd has quit IRC22:53
*** tkajinam has joined #opendev22:54
*** mlavalle has quit IRC22:59
*** tosky has quit IRC23:01
openstackgerritIan Wienand proposed opendev/system-config master: gitea: install proxy  https://review.opendev.org/73986523:14
*** hashar has quit IRC23:15
gouthamrhello o/ anyone aware of any problems with rax-iad? We have a test job in openstack/manila that fails only on that cloud with likely an OOM issue: rabbitmq halts, systemd services are restarted (i notice a reboot in the logs when this happens)23:37
fungii'd be surprised if it happens in rax-iad but not rax-dfw or rax-ord23:40
fungiany idea how long it's been going on?23:40

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!