Tuesday, 2020-08-04

ianwkevinz: so the linaro-us mirror is shutoff again00:06
ianwmirror01.regionone.linaro-us.opendev.org | SHUTOFF00:06
ianwit's back after i started it, but i don't know why it keeps disappearing00:14
*** ryohayakawa has joined #opendev00:17
ianwhttps://github.com/pyca/cryptography/pull/5341/checks?check_run_id=942805616 ... checks api is reporting on pyca00:21
funginothing in syslog/wtmp to suggest when or why it went down? (we can probably approximate shutdown time to within 5 minutes based on snmp blackout time in cacti)00:21
corvusianw: looks legit00:21
ianwfungi: last entry is "Aug  3 10:53:18 mirror01 kernel: [291253.841199] afs: volume location server in cell openstack.org is back up (code 0)"00:22
corvusianw: i wonder if we'll have a stale 'status' on that pr after the check run is done00:22
fungilooks like it went dark between 10:50 and 10:55 utc today00:23
fungijudging from cacti's graphs00:23
ianwwe could put a netconsole on it, and see if we get a oops (i'm guessing yes)00:24
ianwor, update it to focal and see if it still happens too, before we spend too much time debugging an old kernel00:24
ianwcorvus: yeah, i guess the old one will hang around, although it's a +1 from the last good run00:25
fungicacti shows a fairly large outbound traffic spike shortly before it died00:25
openstackgerritIan Wienand proposed opendev/system-config master: launch-node : add sshfp records  https://review.opendev.org/74346101:04
ianwfungi: if you have a sec for a backup review on https://review.opendev.org/#/c/743445/1 adds zuul to the "new" backup server01:11
fungilgtm, thanks for fixing that!01:25
ianwi'll start the oe mirror and make sure the launch script spits out sshfp records with it01:27
openstackgerritMerged opendev/system-config master: launch-node : add sshfp records  https://review.opendev.org/74346101:33
openstackgerritMerged opendev/system-config master: Backup inventory - match zuul01.openstack.org  https://review.opendev.org/74344501:48
*** ysandeep|away is now known as ysandeep02:32
openstackgerritAdrian Turjak proposed openstack/project-config master: Return gnocchi back to openstack  https://review.opendev.org/74459202:51
*** fressi has joined #opendev04:20
*** raukadah is now known as chkumar|rover04:23
*** ysandeep is now known as ysandeep|afk04:33
*** xiaolin has quit IRC05:10
*** marios has joined #opendev05:13
*** marios is now known as marios|ruck05:43
*** DSpider has joined #opendev05:54
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/74409606:10
*** ysandeep|afk is now known as ysandeep06:18
openstackgerritMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/74409606:37
*** hashar has joined #opendev06:49
openstackgerritMerged openstack/project-config master: Add notifications for openstack-stable channel  https://review.opendev.org/74405006:51
*** dtantsur|afk is now known as dtantsur07:16
*** tosky has joined #opendev07:48
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** AJaeger has joined #opendev08:08
*** fressi has quit IRC08:19
*** jhesketh has quit IRC08:24
openstackgerritMerged openstack/project-config master: Add Ceph iSCSI charm to OpenStack charms  https://review.opendev.org/74447908:32
*** fressi has joined #opendev08:37
*** priteau has joined #opendev08:43
*** fressi has joined #opendev09:09
*** lpetrut has joined #opendev09:12
openstackgerritMerged openstack/project-config master: Revert "Remove os_congress gating"  https://review.opendev.org/74253209:24
*** bolg has quit IRC09:28
*** hashar has quit IRC09:42
*** auristor has quit IRC10:01
*** tkajinam has quit IRC10:12
*** sshnaidm_ has joined #opendev10:12
*** sshnaidm has quit IRC10:15
*** jhesketh has joined #opendev10:21
*** sshnaidm_ is now known as sshnaidm10:26
openstackgerritDaniel Bengtsson proposed openstack/diskimage-builder master: Update the tox minversion parameter.  https://review.opendev.org/73875410:37
*** tosky_ has joined #opendev11:20
*** tosky has quit IRC11:20
*** tosky_ is now known as tosky11:20
*** marios|ruck has quit IRC11:48
openstackgerritThierry Carrez proposed openstack/project-config master: Retire Zuul's Kata tenant  https://review.opendev.org/74468711:58
*** xiaolin has joined #opendev12:05
*** tosky has quit IRC12:14
*** tosky_ has joined #opendev12:14
*** tosky_ is now known as tosky12:15
*** xiaolin has quit IRC12:15
*** hashar has joined #opendev12:16
*** ryo_hayakawa has joined #opendev12:28
*** ryohayakawa has quit IRC12:29
*** ryo_hayakawa has quit IRC13:02
*** ryo_hayakawa has joined #opendev13:03
*** auristor has joined #opendev13:05
*** ryo_hayakawa has quit IRC13:10
*** ysandeep is now known as ysandeep|mtg13:18
*** fressi has quit IRC13:25
*** ysandeep|mtg is now known as ysandeep14:00
*** mlavalle has joined #opendev14:00
*** fressi has joined #opendev14:14
*** ysandeep is now known as ysandeep|off14:16
*** hashar has quit IRC14:29
*** fressi has quit IRC14:34
*** chkumar|rover is now known as raukadah15:10
*** lpetrut has quit IRC15:15
*** sdmitriev has quit IRC15:16
*** openstackgerrit has quit IRC15:20
*** fressi has joined #opendev15:36
*** fressi has left #opendev15:36
*** lpetrut has joined #opendev15:43
*** hashar has joined #opendev15:45
*** lpetrut has quit IRC16:21
*** mlavalle has quit IRC16:22
*** mlavalle has joined #opendev16:23
*** sshnaidm is now known as sshnaidm|afk16:48
*** fressi has joined #opendev16:50
*** priteau has quit IRC16:51
*** priteau has joined #opendev16:51
*** fressi has quit IRC16:58
*** dtantsur is now known as dtantsur|afk17:00
*** hashar is now known as hashardinner17:16
clarkbI've approved ^ I don't expect any issues but this is the first time we've retired a tenant17:17
*** openstackgerrit has joined #opendev17:26
openstackgerritMerged openstack/project-config master: Retire Zuul's Kata tenant  https://review.opendev.org/74468717:26
*** priteau has quit IRC17:33
*** lpetrut has joined #opendev17:49
*** lpetrut has quit IRC17:50
clarkbhttps://zuul.opendev.org/tenants doesn't show the kata tenant so I expect that all went well18:27
fungiyeah, seems fine and all18:31
*** auristor has quit IRC19:14
*** mtreinish has quit IRC19:27
*** auristor has joined #opendev19:46
openstackgerritClark Boylan proposed opendev/system-config master: Increate nodepool builder upload workers from 4 to 8  https://review.opendev.org/74478019:57
clarkbfungi: ^ thats the upload workers fix. I think we want to land that after the current round of uploads completes as they are almost done19:58
clarkbianw: ^ fyi19:58
*** sgw2 has joined #opendev20:02
sgw2Morning gang, I need to delete a tag in the stx-tools repo, it was applied slightly prematurely. I don't appear the have Force Push rights to allow me to delete the 4.0.0 tag (refs/tags/4.0.0).20:03
clarkbsgw2: typically we strongly recommend against deleting tags because any downstream pullers won't have their tag updated20:05
clarkbthat means you can end up with weird repo states. Instead we recommend pushing a 4.0.1 or similar to address the problem in a roll forward fashion. that way all remote repos have a consistent state20:05
sgw2I understand, this should have limited scope as the time window has only been an couple of hours.20:05
clarkbis a 4.0.1 tag inappropriate for some reason?20:06
fungialso ci systems will likely have pulled copies of that tag already20:06
fungiand won't know to replace them later if the tag name is reused20:06
sgw2What ci system for starlingx, unless Zull deals with something20:07
fungibecause tags aren't altered on git pull20:07
sgw2Zuul sorry.20:07
clarkbyes zuul would be one of them20:08
fungiyeah, i mainly don't know if you have any ci/cd automation triggered from tags, just pointing out that any automation fetching refs from your repos will likely wind up with incorrect tags if you delete and later replace them20:08
clarkb(there are potentially others)20:08
sgw2no automation other than our Jenkins builds which have not fired yet.20:08
fungialso anyone who has run something like `git remote update` will forever have the old tag locally (even if you replace it with a new tag later) unless they know to manually delete it20:09
fungiso for the most part we've considered tags permanent, and can't predict what the result will be if we delete one20:10
clarkbright its usually better to roll forward as that is predictable20:10
clarkbwhich is why I was asking what the concerns are with that20:10
sgw2I understand the challenges and as I said this is a very small window.  I will double check with our build/release team.20:11
clarkbif we do it we'll need to rm the zuul executors local repos for stx-tools20:15
clarkbcorvus: ^ can that be done with zuul running or do we also need to stop the executors?20:15
clarkb(or I guess we may also be able to manually surgery the repos instead of having zuul reclone)20:15
fungii can escalate my gerrit account privs to delete refs/tags/4.0.0 from the starlingx/tools repo momentarily20:15
corvusclarkb: i certainly wouldn't do it with an stx job running20:15
corvusclarkb: but i think aside from that, it should be okay20:15
clarkbcorvus: k20:16
clarkbfungi: lets see what sgw2 after asking their build/release team. I can help with executor and merger repo cleanups20:17
sgw2So it is more complex than just a tag deletion, that's extra info to inform our team.20:18
fungiit's definitely extra work. we don't even give ourselves permission to delete tags (or push --force for that matter) by default20:19
fungipart of why we have the second bullet in the "note" block under Request For Enhancement20:20
fungier, under https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#tagging-a-release20:20
fungi"Tags can’t be effectively deleted once pushed, so make absolutely certain they’re correct (ideally by locally testing release artifact generation commands and inspecting the results between the tag and push steps above)."20:21
*** hashardinner has quit IRC20:34
openstackgerritIan Wienand proposed opendev/system-config master: openedge mirror: remove for replacement  https://review.opendev.org/74478520:37
ianwclarkb/fungi: ^ i think we should do that before i remove it from the emergency list, so we don't have non-responding hosts in inventory20:38
clarkbfungi: latest patchset of the identity service spec lgtm20:41
ianwthanks, once that merges i'll restart the oe mirror with the bigger instance, and try getting the sshfp keys from the host21:09
openstackgerritMerged opendev/system-config master: openedge mirror: remove for replacement  https://review.opendev.org/74478521:17
donnydugg... I have taskers the general just came in and handed me that have to be done today.. be back in a bit21:34
*** DSpider has quit IRC21:39
ianwhrm, deployment didn't seem to go too well anyway21:52
ianwinfra-prod-base failed21:52
ianwreview-test has a full fs, which killed the run : /dev/xvda1       39G   39G     0 100% /21:54
ianwit's /usr/local/bin/track-upstream21:54
ianwi thought we fixed that21:54
ianwhttps://review.opendev.org/#/c/739840/ ... we did drop it from cron, in theory21:56
ianwok, that has not applied because the gerrit playbook is failing on review-test22:04
ianwAnsibleUndefinedVariable: 'gerrit_vhost_name' is undefined22:04
clarkbianw: ya I was hoping mordred would be around for an update on that host since we ran into trouble with it during project renames too22:07
ianwit seems /etc/ansible/hosts/host_vars/review-test.opendev.org.yaml is not in sync?22:08
ianware we supposed to be reading that directly from system-config?22:08
clarkbianw: I think it was intentionally different to avoid creating confusion with a truly prod server22:09
clarkbso it should be in sync to a degree but not completely?22:09
clarkbinfra-root I've got a change ready to go for gerritbot on eavesdrop instead of review.o.o and running out of a container. Before I push it is there any concern using our actual channel config if I'm supplying bogus freenode nick/password and gerrit user connection details?22:09
ianwclarkb: so ... inventory/service/host_vars/review-test.opendev.org.yaml:gerrit_vhost_name: review-test.opendev.org isn't deployed to /etc/ansible/hosts?22:10
clarkbactually what I can do is comment out the inclusion of the role from the playbook then we can review it and decide if that is safe22:10
clarkbianw: no we should include the contents of both /etc/ansible/hosts and system-config side by side22:11
clarkbrather than write system-config data into /etc/ansible/hosts on bridge iirc22:11
clarkbmaybe that is broken?22:11
ianw... i don't want to go too far down this rabbit hole right now.  for now, i've just manually deleted the crontab entry22:11
clarkbthat seems reasonable. It would be good to sync up with mordred on that as we keep running into weird behaviors related to it22:12
ianwok, having another go at the oe mirror22:13
openstackgerritClark Boylan proposed opendev/system-config master: Add ansible role to manage gerritbot  https://review.opendev.org/74479522:15
ianwdonnyd: does not look like it's my day.  launching the server just went into error status, and now i can't delete it either22:15
clarkbinfra-root ^ thoughts on the testing questions there would be great22:15
clarkbI put TODOs in the places where I had questions22:16
clarkbfungi: ^ fyi since you manually updated that services config22:17
ianwdonnyd: i thought it might be the 250g instance, but similarly doesn't work for 80gb22:18
ianwdifferent error though : 504: Server Error for url: https://api.us-east.open-edge.io:8774/v2.1/os-keypairs, 504 Gateway Time-out: The server didn't respond in time.22:18
*** qchris has quit IRC22:22
*** qchris has joined #opendev22:35
*** tosky has quit IRC22:50
*** dpawlik2 has quit IRC22:55
*** guillaumec has quit IRC22:55
*** frickler has quit IRC22:55
*** tkajinam has joined #opendev22:55
*** dpawlik2 has joined #opendev22:57
*** guillaumec has joined #opendev22:57
*** frickler has joined #opendev22:57
*** mlavalle has quit IRC22:59
clarkbianw: donnyd just noticed that an ubuntu focal upload to openedge failed23:05
clarkbprobably not super urgent but once we've got the mirror up we may want to look into upload reliability23:05
ianwyeah, given the launch issues i'm guessing it would all be related23:06
fungiclarkb: i'll try to review in the morning, but initial concern is that we may want to prevent it from trying to connect to freenode just so we're not pestering them with bogus login attempts23:12
clarkbfungi: ya I can make that configurable too and point it somewhere invalid23:12
clarkbthough I think we may test our other bots against freenode23:13
fungia really neat test might be to install something like inspircd from distro package and configure it to listen on the loopback23:13
* fungi runs the debian-packaged inspircd and has for years, rather simple to set up23:14
fungialso supports things like sasl auth and comes with nickserv/chanserv and the like, so we could test some fairly complex bot interactions eventually if we wanted23:15
fungii could try to find time to add something like that, though it probably won't be this week23:17
clarkbya maybe we do what we can for now then can add that in later23:18
fungisounds reasonable23:19
openstackgerritMerged opendev/system-config master: Increate nodepool builder upload workers from 4 to 8  https://review.opendev.org/74478023:23

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!