Tuesday, 2020-09-29

« Monday, 2020-09-28 Index Wednesday, 2020-09-30 »
*** hamalq has quit IRC00:32
*** Qitao has joined #opendev01:01
*** DSpider has quit IRC01:05
*** Topner has joined #opendev01:11
*** qchris has quit IRC01:11
*** Qitao has quit IRC01:14
*** Topner has quit IRC01:15
ianwFileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/.local/share/virtualenv/wheel/3.5/embed/1/setuptools.json'01:19
ianwsigh, i don't really want to see if virtualenv/setuptools/etc have released something, but i'm betting they have01:20
*** qchris has joined #opendev01:25
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: only run docker-setup.yaml when installed  https://review.opendev.org/74706201:33
openstackgerritIan Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json  https://review.opendev.org/74683401:33
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround  https://review.opendev.org/74706301:33
openstackgerritMerged zuul/zuul-jobs master: Make sure that we pass list in loop  https://review.opendev.org/75461201:35
ianwhuh, i can't repeat it01:36
*** hamalq has joined #opendev01:37
*** diablo_rojo has quit IRC02:33
*** hamalq has quit IRC02:45
*** fressi has joined #opendev03:01
*** ysandeep|away is now known as ysandeep03:42
*** fressi has quit IRC03:47
*** ykarel|away has joined #opendev04:02
*** ykarel|away has quit IRC04:11
*** ykarel|away has joined #opendev04:12
openstackgerritIan Wienand proposed opendev/system-config master: graphite: copy storage-schemas from puppet-graphite  https://review.opendev.org/75489304:22
openstackgerritIan Wienand proposed opendev/system-config master: graphite: add cleanup job  https://review.opendev.org/75489404:22
*** ykarel|away is now known as ykarel04:30
*** zigo has quit IRC04:38
*** ysandeep is now known as ysandeep|afk04:52
openstackgerritIan Wienand proposed opendev/system-config master: graphite: copy storage-schemas from puppet-graphite  https://review.opendev.org/75489304:52
openstackgerritIan Wienand proposed opendev/system-config master: graphite: add cleanup job  https://review.opendev.org/75489404:52
*** marios has joined #opendev04:56
*** mtreinish has quit IRC05:09
*** sparkp1ug has joined #opendev05:14
*** sparkp1ug has quit IRC05:26
*** mtreinish has joined #opendev05:26
*** mtreinish has quit IRC05:54
*** mtreinish has joined #opendev05:56
*** ysandeep|afk is now known as ysandeep05:59
openstackgerritIan Wienand proposed opendev/system-config master: graphite: add cleanup job  https://review.opendev.org/75489406:20
*** eolivare has joined #opendev06:25
*** slaweq has joined #opendev06:25
openstackgerritAndreas Jaeger proposed openstack/project-config master: Complete retirement of x/osops-* repos  https://review.opendev.org/75114106:30
*** slaweq has quit IRC06:31
*** mlavalle has quit IRC06:34
*** mlavalle has joined #opendev06:34
*** noonedeadpunk has quit IRC06:37
*** noonedeadpunk has joined #opendev06:37
openstackgerritMerged zuul/zuul-jobs master: Fix promote cleanup  https://review.opendev.org/75486506:42
*** slaweq has joined #opendev06:53
*** hamalq has joined #opendev07:02
*** hamalq has quit IRC07:08
ianwinfra-root: if you have time for https://review.opendev.org/#/q/status:open+project:opendev/system-config+branch:master+topic:graphite-final that would help me cut over to the new graphite.  i'm rsyncing the old logs now; i can shutdown old, do a catch-up rsync then cutover07:08
*** slaweq has quit IRC07:09
*** slaweq has joined #opendev07:10
*** dirk has quit IRC07:13
*** dirk has joined #opendev07:15
*** hamalq has joined #opendev07:24
*** hamalq has quit IRC07:29
*** andrewbonney has joined #opendev07:44
*** tosky has joined #opendev07:45
*** dtantsur|afk is now known as dtantsur07:58
*** ysandeep is now known as ysandeep|lunch07:59
*** moppy has quit IRC08:01
*** hamalq has joined #opendev08:01
*** moppy has joined #opendev08:01
*** hamalq has quit IRC08:06
*** fressi has joined #opendev08:13
*** hashar has joined #opendev08:20
*** ysandeep|lunch is now known as ysandeep08:29
*** sshnaidm|off is now known as sshnaidm08:33
*** priteau has joined #opendev08:45
*** Topner has joined #opendev09:14
*** zbsarash1 has quit IRC09:17
*** Topner has quit IRC09:19
openstackgerritzbr proposed zuul/zuul-jobs master: Partial address ansible-lint E208  https://review.opendev.org/74848009:31
*** sparkp1ug has joined #opendev09:40
*** sparkp1ug has quit IRC09:42
*** sparkp1ug has joined #opendev09:43
*** sparkp1ug has quit IRC09:43
*** sparkp1ug has joined #opendev09:43
*** DSpider has joined #opendev09:50
*** auristor has quit IRC09:53
*** slaweq has quit IRC10:17
*** lpetrut has joined #opendev10:21
*** slaweq has joined #opendev10:34
lourot export BYOBU_CHARMAP=x ; . ~/.bashrc10:47
*** sparkp1ug has quit IRC10:48
lourotsorry, wrong shortcut that ended up pasting random things...10:48
*** hashar has quit IRC10:50
*** eolivare has quit IRC11:24
*** auristor has joined #opendev11:35
*** hashar has joined #opendev11:38
*** Tengu has quit IRC11:53
*** Tengu has joined #opendev11:55
*** ysandeep is now known as ysandeep|brb11:56
*** sparkp1ug has joined #opendev12:06
*** sparkplug has joined #opendev12:08
*** eolivare has joined #opendev12:11
*** sparkp1ug has quit IRC12:11
*** sparkplug_ has joined #opendev12:32
*** sparkp1ug has joined #opendev12:35
*** sparkplug has quit IRC12:35
openstackgerritAndreas Jaeger proposed openstack/project-config master: Cleanup unneeded lines for os-loganalyze  https://review.opendev.org/75499012:35
*** ysandeep|brb is now known as ysandeep12:54
*** priteau has quit IRC13:10
*** iurygregory has quit IRC13:15
*** iurygregory has joined #opendev13:30
*** slaweq has quit IRC13:33
*** slaweq has joined #opendev13:35
*** sshnaidm has quit IRC13:48
*** sshnaidm has joined #opendev13:58
*** roman_g has joined #opendev14:00
*** sparkplug has joined #opendev14:18
*** lpetrut has quit IRC14:22
*** sparkp1ug has quit IRC14:26
*** sparkplug has quit IRC14:26
*** sparkplug_ has quit IRC14:26
*** sparkp1ug has joined #opendev14:28
*** sparkplug has joined #opendev14:28
*** sparkplug_ has joined #opendev14:28
*** hashar has quit IRC14:29
*** zbsarash1 has joined #opendev14:30
*** ykarel is now known as ykarel|away14:42
*** ysandeep is now known as ysandeep|away14:48
*** zbsarash1 has quit IRC14:55
openstackgerritQianBiao Ng proposed openstack/diskimage-builder master: Fix: ramdisk image build issues  https://review.opendev.org/75505615:01
openstackgerritQianBiao Ng proposed openstack/diskimage-builder master: Fix: ramdisk image build issues  https://review.opendev.org/75505615:04
*** iurygregory has quit IRC15:20
openstackgerritAndrii Ostapenko proposed zuul/zuul-jobs master: Fix certificate issue with use buildset registry  https://review.opendev.org/74158415:21
openstackgerritAndrii Ostapenko proposed zuul/zuul-jobs master: Fix certificate issue with use buildset registry  https://review.opendev.org/74158415:24
*** iurygregory has joined #opendev15:28
clarkbinfra-root as a heads up the fedora-30 cleanup has happened on the RW volume but not been synced to RO yet due to what appear to be problems rsyncing the rawhid content15:43
clarkbI think our upstream mirror's .tmp rsync files/dirs are being seen by our rsync whcih fails to fetch them due to permissions issues15:43
clarkbprobably we'll be fine if it run it a few more times. If it doesn't fix itself in a day or so then we can bother with working around it15:44
fungii wonder if we can add .tmp to excludes?15:44
clarkboh thats a good idea15:44
clarkbrsync: opendir "/linux/development/rawhide/Everything/x86_64/os/Packages/b/.~tmp~" (in fedora) failed: Permission denied (13)15:44
clarkbits errors like that so exclude .~tmp~ ?15:44
fungiworth a try15:45
fungii honestly have no idea whether that'll work, but i can't imagine we want to mirror those files if they do happen to be readable15:45
clarkbya I'm sure we don't want them15:46
clarkbfailing might be advantageous if an usptream rsync is happening though. This way we don't end up with a half baked mirror state15:46
openstackgerritzbr proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container  https://review.opendev.org/72962315:46
clarkbya thinking about it more not failing when upstream is half synced could be bad for us15:49
clarkbwith rpm mirrors we don't have a validating step like we do with reprepro so are relying on upstream mirrors being consistent when we publish them15:50
*** hashar has joined #opendev15:51
fungigreat point. also debian's mirroring process involves touching a lockfile while a mirror is being updated, so if other mirrors see that file present they should refrain from syncing. i wonder if rpm mirroring has a similar signal15:52
*** hamalq has joined #opendev16:00
*** ykarel|away has quit IRC16:09
*** fressi has quit IRC16:11
*** marios is now known as marios|out16:40
clarkbpopping out for that bike ride now.16:51
*** dtantsur is now known as dtantsur|afk16:52
*** hamalq has quit IRC16:58
*** aannuusshhkkaa has quit IRC17:02
*** andrewbonney has quit IRC17:32
*** jentoio has quit IRC17:36
*** erbarr has quit IRC17:37
*** bbezak has quit IRC17:37
*** dviroel has quit IRC17:37
*** lseki has quit IRC17:37
*** gmann has quit IRC17:37
*** mnaser has quit IRC17:37
*** mnasiadka has quit IRC17:37
*** seongsoocho has quit IRC17:37
*** donnyd has quit IRC17:37
*** ShadowJonathan has quit IRC17:37
*** jrosser has quit IRC17:38
*** mwhahaha has quit IRC17:38
*** hemanth_n has quit IRC17:38
*** ildikov has quit IRC17:38
*** davidlenwell has quit IRC17:38
*** johnsom has quit IRC17:38
*** knikolla has quit IRC17:38
*** Open10K8S has quit IRC17:38
*** jbryce has quit IRC17:38
*** hillpd has quit IRC17:38
*** gouthamr has quit IRC17:39
*** rm_work has quit IRC17:39
*** johnsom has joined #opendev17:40
*** jentoio has joined #opendev17:47
*** lseki has joined #opendev17:48
*** dviroel has joined #opendev17:48
*** erbarr has joined #opendev17:49
*** mwhahaha has joined #opendev17:49
*** mnasiadka has joined #opendev17:49
*** knikolla_ has joined #opendev17:49
*** knikolla_ is now known as knikolla17:49
*** Open10K8S has joined #opendev17:50
*** gmann_ has joined #opendev17:54
*** donnyd has joined #opendev17:54
*** mnaser has joined #opendev17:55
*** gmann_ is now known as gmann17:56
*** hillpd has joined #opendev17:57
*** davidlenwell has joined #opendev17:58
*** jbryce has joined #opendev17:59
*** ShadowJonathan has joined #opendev18:00
*** bbezak has joined #opendev18:04
*** hemanth_n has joined #opendev18:04
*** seongsoocho has joined #opendev18:05
*** jrosser has joined #opendev18:05
*** ildikov has joined #opendev18:05
*** roman_g has quit IRC18:14
*** marios|out has quit IRC18:18
*** gouthamr has joined #opendev18:23
*** gouthamr__ has joined #opendev18:26
*** sparkplug_ has quit IRC18:36
*** sparkp1ug has quit IRC18:37
*** sparkplug has quit IRC18:37
*** sparkp1ug has joined #opendev18:38
*** sparkplug has joined #opendev18:38
*** sparkplug_ has joined #opendev18:39
*** diablo_rojo_phon has joined #opendev18:48
*** diablo_rojo has joined #opendev18:57
*** hemanth_n_ has joined #opendev19:06
*** hemanth_n has quit IRC19:07
*** hemanth_n_ is now known as hemanth_n19:07
*** ildikov has quit IRC19:07
*** ildikov has joined #opendev19:08
*** eolivare has quit IRC19:20
*** gouthamr has quit IRC19:39
*** gouthamr__ is now known as gouthamr19:39
*** tosky has quit IRC19:51
ianwfungi: sorry to harp but if you have a sec for https://review.opendev.org/#/q/status:open+project:opendev/system-config+branch:master+topic:graphite-final that will give me the rest of the day for a final sync of graphite and cut-over to new server20:01
ianwi have synced as of last night my time, which you can see @ https://graphite02.opendev.org/20:02
ianwso i think the new server is working fine20:02
clarkbianw: did you see the earlier discussion about the rawhide mirroring?20:03
ianwclarkb: sorry, no, haven't scrolled back20:03
clarkbany thoughts on whether it is unsafe to exclude those .~tmp~ dirs when rsyncing?20:03
clarkb(also I wonder why we rsync rawhide?)20:03
ianwon why, there was a recent request for it iirc20:04
ianwit's not something we've been doing for a long time20:04
ianwthe .~tmp~ sounds like a broken mirror to me20:05
ianwhttps://fedoraproject.org/wiki/Infrastructure/Mirroring/Tiering#Tier_1_rsync_configuration20:05
ianwexclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs*20:05
ianwi guess if that lists it, it must be a bit of a known thing20:06
fungiianw: yep, let me get caught back up on some openstack election stuff and that's next on my list20:07
clarkbianw: oh interesting. maybe we should add that to our excludes tehn. I was worried about doing that if it would cause us to synch half sycned mirrors20:09
*** ildikov has quit IRC20:16
*** ildikov has joined #opendev20:17
openstackgerritIan Wienand proposed opendev/zone-opendev.org master: Point graphite.opendev.org to new server  https://review.opendev.org/75511820:17
openstackgerritClark Boylan proposed opendev/system-config master: Exclude private dirs when mirroring fedora  https://review.opendev.org/75512320:37
clarkbianw: ^ something like that maybe?20:37
ianwclarkb: LGTM, thanks!20:41
*** hashar has quit IRC20:44
openstackgerritMerged opendev/system-config master: graphite: copy storage-schemas from puppet-graphite  https://review.opendev.org/75489320:48
clarkbinfra-root I've taken another quick look at review-test and come up with https://etherpad.opendev.org/p/review-test-todos20:50
clarkbcan you look over that and maybe check the things listed at the top there on review-test just to be sure its safe to do the things listed in the todo section?20:50
ianwclarkb: mail, maybe?  it won't start sending emails will it?20:53
clarkbianw: it likely will20:53
clarkbI'm not sure what the best way to disable that is as I think gerrit always wants to send mail20:54
clarkbmaybe we can iptables it off?20:54
ianwor just stop exim?20:56
clarkboh ya gerrit is sending to localhost and exim spools from there? then ya that may be easiset20:56
openstackgerritMerged opendev/system-config master: graphite: add cleanup job  https://review.opendev.org/75489420:59
ianwif there's no objections, soon i'll stop graphite01.opendev.org, do a final rsync to graphite02.opendev.org and then switch the dns cname, and monitor that the stats are coming into the new server21:01
clarkb++21:02
*** sparkp1ug has quit IRC21:08
*** sparkplug has quit IRC21:11
*** sparkplug has joined #opendev21:11
*** lseki has quit IRC21:52
*** johnsom has quit IRC21:52
*** Open10K8S has quit IRC21:52
*** diablo_rojo_phon has quit IRC21:52
*** hemanth_n has quit IRC21:52
*** dviroel has quit IRC21:52
*** davidlenwell has quit IRC21:52
*** jbryce has quit IRC21:53
*** erbarr has quit IRC21:53
*** ildikov has quit IRC21:53
*** hillpd has quit IRC21:53
*** lseki_ has joined #opendev21:53
*** knikolla has quit IRC21:53
*** jentoio has quit IRC21:53
*** rpittau|afk has quit IRC21:53
*** ShadowJonathan has quit IRC21:54
*** johnsom has joined #opendev21:54
*** mnasiadka has quit IRC21:54
*** erbarr has joined #opendev21:55
*** mnasiadka has joined #opendev21:55
*** dviroel has joined #opendev21:55
*** jbryce has joined #opendev21:55
*** knikolla has joined #opendev21:56
*** hillpd has joined #opendev21:56
*** davidlenwell has joined #opendev21:56
*** Open10K8S has joined #opendev21:56
*** jentoio has joined #opendev21:56
*** ildikov has joined #opendev21:57
*** rpittau|afk has joined #opendev21:57
*** ShadowJonathan has joined #opendev21:58
*** hemanth_n has joined #opendev22:07
*** sparkplug_ has left #opendev22:08
*** sparkp1ug has joined #opendev22:12
*** sparkp1ug has quit IRC22:13
ianwi'm doing the sync now.  i'll swap the dns entry now; the container is disabled on the new server22:16
*** sparkplug has quit IRC22:21
*** slaweq has quit IRC22:27
openstackgerritMerged opendev/zone-opendev.org master: Point graphite.opendev.org to new server  https://review.opendev.org/75511822:51
*** mlavalle has quit IRC23:01
*** rm_work has joined #opendev23:03
*** rm_work has quit IRC23:03
ianwi wonder if zuul has to restart to pickup the new CNAME for graphite?23:08
clarkboh I bet it does, the statsd lib does a bunch of work on import iirc23:09
clarkb:/23:09
clarkband its the scheduler that does all the stats sending too iirc23:10
clarkbfungi: ^ fyi since you're doing election things which may interact with a zuul restart23:10
ianwi think the executors also sending stuff, i'm going to restart one and see if stats come in from it23:11
clarkbk23:11
clarkbalso note we just updated the zuul images to python3.8 rather than 3.723:11
clarkbno expected problems from that but calling it out as a restart will pick it up23:11
ianwyeah i am seeing stats from a tcpdump from a restarted ze0123:12
clarkbI can help with a full zuul restart if we want to do that nowish23:15
clarkbwe should make sure the elections folks and openstack release team know though23:15
clarkbboth are doing things this week23:15
clarkbianw: I believe the playbook that does the restarting should work. You will need to grab queues separately though (either from zuul before the restart or from the local backups we do of that content)23:16
ianwis the executor process now docker-compose down/up -d23:16
ianwoh, ok, which playbook?23:16
clarkbianw: yes, its playbooks/zuul_restart.yaml which calls zuul_stop.yaml then zuul_start.yaml23:17
fungiclarkb: lemme make sure all the post jobs have finished23:17
clarkband those call the roles/zuul-*/tasks/stop.yaml and start.yaml things23:17
ianwif nothing else it would be good for me to get experience with this :)23:18
ianwis there anything else people are waiting on?23:18
ianwin terms of zuul features that would be picked up by restart23:18
clarkbnot that I am aware of23:18
ianwhttps://review.opendev.org/#/q/status:open+project:zuul/zuul+branch:master+topic:synchronize-find23:18
ianwwould be good to be able to re-write that zuul-jobs sphinx job23:18
fungiclarkb: looks like we're all caught up publishing from merged elections changes, so a restart isn't disruptive to the openstack election officials right now23:19
clarkbhrm I feel like that one will need me to review it not at the end of a day (since it has security implications)23:19
ianwyeah, that's fine23:19
*** DSpider has quit IRC23:20
ianwit was ze02 i restarted sorry; i'm seeing stats come to the server.  trying to validate they're making it to the db23:20
ianwhttps://graphite02.opendev.org/render/?width=647&height=381&from=-15minutes&lineMode=connected&connectedLimit=&drawNullAsZero=true&target=stats.zuul.executor.ze02_openstack_org.builds&target=stats.zuul.executor.ze03_openstack_org.builds23:26
ianwi think that shows the stats getting into the db23:27
clarkbits got something it sees23:27
ianwso i've done /opt/zuul/tools/zuul-changes.py https://zuul.opendev.org > queues.sh23:32
ianwthat seems to get everything23:32
ianwhttps://docs.opendev.org/opendev/system-config/latest/zuul.html#restarting-the-scheduler suggests using the queue names though23:32
clarkbianw: I think that is out of date23:33
clarkbwe modified it to grab everything more recently iirc23:33
ianwyeah, i guess my penance for doing this will be to update that :)23:33
clarkbthe othe rthing to keep in mind is you don't want to run the output of that until after the scheduler has finished coming back up23:33
ianwyep23:33
ianwso, any objections to me running the restart playbook?23:33
clarkbnot from me. things look quiet23:34
clarkbfungi checked off on elections side and relases don't seem tohave anything happening right now23:34
ianwPlaybook run took 0 days, 0 hours, 0 minutes, 33 seconds23:36
clarkboh ya now that we switched to docker we don't let things drain out23:37
clarkbit just hard stops23:37
clarkb(should be ok but that is why its quick)23:37
fungieasier to do since they all share a cgroup which we pull out from under them23:38
ianwa lot of "2020-09-29 23:36:50,345 WARNING zuul.Scheduler: Tenant openstack isn't loaded" which i guess is because it's getting events but config isn't loaded23:38
clarkbianw: yes that is the case iirc23:38
fungii guess it doesn't pause processing the event queue while it waits for cat jobs to complete?23:38
fungioh, this is before the tenant config is loaded even23:39
fungi?23:39
clarkbyes23:40
ianwit's zooming through that now23:40
clarkbusually takes about 5 minutes23:40
clarkblooks like it may be up now23:43
ianwyep, i'll redo the queue23:43
ianwhttps://grafana01.opendev.org/d/9VdOrLMMk/zuul-status?orgId=1 looks to have things coming in23:44
ianwi guess nodepool tool23:44
ianwtoo23:44
ianwdoh i really should have thought of that before23:45
clarkbI think the nodepool restart playbook should work too23:45
clarkbrunning jobs won't be affected by that iirc23:46
clarkbso its a bit more graceful than zuul23:46
ianwok, i ran that just now, before any jobs get too far into anything23:47
ianwthanks for the help -- i think we now have containerised graphite/grafana stack!23:48
ianwi'll shutdown 01 and make a note to cleanup23:48
ianwand work on updating the restart instructions23:48
clarkbthank you!23:48
clarkbalso we're running zuul with python3.8 again23:50
ianwtime for a cup of tea23:50
clarkbor should be anyway23:50
ianwthen i might watch this "debate"23:50
ianwoh, hang on, i'm 1 hour out i think23:50
clarkbyes 70 minutes away23:51
fungii don't think i can bear it23:59
« Monday, 2020-09-28 Index Wednesday, 2020-09-30 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!