Friday, 2020-06-19

*** mlavalle has quit IRC00:15
*** DSpider has quit IRC00:18
openstackgerritMerged opendev/system-config master: Make a review-test that we run ansible on
openstackgerritMerged opendev/system-config master: Add playbook for syncing state from review to review-test
ianwagain there's a bunch of hung ssh to
ianwdoes the same for me, ssh just hangs, rebooting it00:42
ianw"hung process" stuff on console, nothing else00:42
*** aannuusshhkkaa has quit IRC01:00
*** factor__ has quit IRC01:53
*** factor__ has joined #opendev01:53
*** factor__ has quit IRC02:02
openstackgerritIan Wienand proposed opendev/system-config master: mirror-update: restore -t to rsync commands
openstackgerritIan Wienand proposed opendev/ master: Remove mirror-update01
*** sgw has quit IRC02:42
openstackgerritMerged opendev/ master: Remove mirror-update01
*** sgw has joined #opendev03:00
ianwlogstashworker09 is also causing problems.  that just reejcts ssh connections quickly03:22
ianwthat has oom messages on it03:22
ianw#status log reboot and, both had become unresponsive03:26
openstackstatusianw: finished logging03:26
openstackgerritMerged opendev/system-config master: mirror-update: restore -t to rsync commands
*** diablo_rojo has quit IRC03:40
*** ysandeep|away is now known as ysandeep04:05
*** sgw has quit IRC04:08
*** sgw has joined #opendev04:09
AJaegerianw: could you review , please?04:42
ianwAJaeger: ok .. it seems fine but why did tox go missing?04:45
AJaegerguess it was on the images before?04:46
*** ykarel|away is now known as ykarel04:51
openstackgerritMerged openstack/project-config master: Add ensure-tox to pti-python-tarball
*** sgw has quit IRC05:03
*** sgw has joined #opendev05:10
openstackgerritIan Wienand proposed opendev/system-config master: Remove from graphite iptables
*** tosky has joined #opendev06:30
*** rpittau|afk is now known as rpittau06:44
iurygregoryfrickler, hey you around? Do you know if there was any progress on the missing logs after the ansible update? =)06:51
frickleriurygregory: yes, the instance I found should be fixed by , but it is possible that other code parts are affected, too. let us know if you find any other weirdness07:07
fricklerbut at least I'm know getting the logs in devstack again that were missing earlier07:08
iurygregoryfrickler, cool!07:08
openstackgerritMerged opendev/system-config master: Remove from graphite iptables
*** DSpider has joined #opendev07:21
*** hashar has joined #opendev07:30
*** bhagyashris|away is now known as bhagyashris07:39
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:03
*** ykarel is now known as ykarel|lunch08:45
*** mugsie has quit IRC08:48
*** mugsie has joined #opendev08:48
iurygregoryfrickler, we have the ironic logs back \o/ tks!09:03
*** tkajinam has quit IRC09:21
*** clarkb has quit IRC09:23
*** clarkb has joined #opendev09:28
*** ysandeep is now known as ysandeep|brb09:29
*** priteau has joined #opendev09:35
*** ykarel|lunch is now known as ykarel09:48
openstackgerritMerged openstack/project-config master: Complete retirement of mox3
*** calcmandan has quit IRC10:11
*** calcmandan has joined #opendev10:12
*** rpittau is now known as rpittau|bbl10:22
*** calcmandan has quit IRC10:30
*** calcmandan has joined #opendev10:32
openstackgerritHervé Beraud proposed openstack/project-config master: Add Backport-Candidate label for oslo deliverables
AJaegerinfra-root, the deploy job  infra-prod-manage-projects failed on
fricklerthe failure seems to be only on review-test, so not too critical I hope11:33
*** tkajinam has joined #opendev12:04
*** ysandeep|brb is now known as ysandeep12:11
*** rpittau|bbl is now known as rpittau12:19
openstackgerritJavier Peña proposed opendev/system-config master: openafs-client: Ensure make is installed for CentOS
openstackgerritJavier Peña proposed opendev/system-config master: Support CentOS for AFS mirror
openstackgerritHervé Beraud proposed openstack/project-config master: Add Backport-Candidate label for oslo deliverables
openstackgerritJavier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror
openstackgerritHervé Beraud proposed openstack/project-config master: Add Backport-Candidate label for oslo deliverables
*** ykarel is now known as ykarel|afk13:28
*** auristor has quit IRC13:39
*** auristor has joined #opendev13:44
*** rpittau is now known as rpittau|brb13:54
openstackgerritDonny Davis proposed openstack/project-config master: Reduce max servers on OE - air handler needs repairs
*** mlavalle has joined #opendev13:54
openstackgerritJavier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror
openstackgerritJavier Peña proposed opendev/system-config master: Make the set-hostname role compatible with non-Debian distros
donnydSo I need to reduce the number of GP builds so I can turn off some of the hypervisors until I get my air handler issues resolved. Heat is currently overrunning the air system13:54
mordredfrickler: oh - that's ok, but I should maybe exclude review-test from that13:55
mordreddonnyd: that is reasonable :)13:55
donnydNot sure where the issue with the air handler is at - but I am working it and hopefully will have it resolved soon13:56
donnydnot cutting the custom jobs though, I know people are dependent on them ATM13:56
AJaegerdonnyd: will single-core approve13:58
fungidonnyd: you even increased the custom node count slightly13:59
fungipresumably that was intentional13:59
AJaegerfungi, do you want to push 737010 to gate directly?13:59
fungiyeah, i can14:00
*** rpittau|brb is now known as rpittau|afk14:00
AJaegerfungi: or should we change the 12 back to 10 on the custom nodes first?14:01
AJaegerwe can leave as is as well...14:01
fungii assume it was increased to 12 intentionally14:01
donnydI changed it to 12 so airship can run more than one job14:01
AJaegerfungi: you're right ;)14:01
donnydi do believe they are running 4 nodes at a time, and the math on 10 as a max is bad14:02
donnydso now they should be able to do 3 at a time14:02
*** guillaumec has joined #opendev14:03
fungimakes sense14:04
donnydno emergency, just need to start taking some action to fix this for good this time around14:06
*** tosky has quit IRC14:08
openstackgerritMerged openstack/project-config master: Reduce max servers on OE - air handler needs repairs
*** tkajinam has quit IRC14:13
*** ysandeep is now known as ysandeep|away14:14
fungionce the deploy jobs run for that, you should be all set14:20
fungilooks like it's just the nodepool deploy job and it's in progress14:22
*** hashar has quit IRC14:27
*** ykarel|afk is now known as ykarel14:28
*** rchurch has quit IRC14:29
*** rchurch has joined #opendev14:33
*** chandankumar is now known as raukadah14:45
mnasermordred: i need to get around updating your shell script to work with zsh now14:58
openstackgerritMonty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook
mordredmnaser: you accepted the update? I did not :)15:05
mordred(although I will obviously accept that patch)15:05
mordredinfra-root: that patch ^^ is based on running the sync playbook by hand - could I get a quick +A on it?15:05
*** sgw has quit IRC15:06
*** sgw has joined #opendev15:08
openstackgerritMonty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook
mordredinfra-root: ^^ sorry, missed one thing15:09
clarkbmordred: I left a note on ps1, I don't think its really a thing, but its early for me and figured I'd mention it15:09
mordredclarkb: yeah, I agree - running that in the wrong direction would be ++bad15:11
mordred(although this is definitely a good argument for using rsync in a shell and not synchronize: :) )15:12
openstackgerritMonty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook
mordredclarkb: one more thing. sorry.15:13
mordredclarkb: do you remember why we write mysql creds to /root/.gerrit_db.cnf instead of /root/.my.cnf?15:19
clarkbmordred: I think they started as backup specific and we may have restricted them relative to a full root connection but they ended up equivalent over time?15:20
*** mlavalle has quit IRC15:21
mordredclarkb: nod15:21
mordredclarkb: I'll have one more patch coming for you in just a sec15:21
*** mlavalle has joined #opendev15:25
openstackgerritMonty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook
*** sgw has quit IRC15:32
*** rchurch has quit IRC15:32
*** calcmandan has quit IRC15:32
*** moppy has quit IRC15:32
*** sgw has joined #opendev15:32
*** rchurch has joined #opendev15:32
*** calcmandan has joined #opendev15:32
*** moppy has joined #opendev15:32
*** Eighth_Doctor has quit IRC15:36
clarkbmordred: is another scary thing because we could be writing back over the production db?15:39
mordredclarkb: yah. I have verified that the creds on review-test are correctly not the creds on review0115:39
clarkbI guess in my half awake brain I'm trying to come up with "what is the proper separation between the two hosts and checks that -test is configured properly before landing the scary bits"15:39
clarkblike maybe we need two playbooks? a sync of the data, then a mv of the data into fake production15:40
mordredclarkb: would that really be any safer? we'd still worry that the creds on review-test were pointing at prod15:41
mordred(I'm with you - it's a terrifying playbook - I'm just not sure any of the thoughts I've had so far are any less terrifying)15:41
mordredfungi, corvus: if you have a sec, would appreciate thoughts on this ^^15:42
clarkbmordred: ya I'm not sure myself.15:42
clarkbthe plays are very explicit about where they should run which is probably 95% of what we need15:43
clarkband its probably better to write it down and have it reviewed than try to ad hoc15:43
clarkbthinking about the two step process if we copied from prod:/prod/path to test:/test/path then had a separate step of apply test:/test/path to test:/prod/path we'd likely fail on prod simply because the prod:/test/path doesn't exist at all15:45
clarkbBut that is still vulnerable to applying things on the wrong host which is the biggest issue15:45
clarkband so it probably doesn't win much overall15:45
corvusmordred: ack; i'll mull over breakfast15:46
fungimy first impression is we do the best we can, but first make sure our backups are recent (and intact)15:49
fungiyou can never be 100% positive you're not going to accidentally blow away production data, that's one of the main reasons we make backups15:49
*** tosky has joined #opendev15:50
fungiso a bit of uncertainty is probably fine, even expected, and so time spent avoiding disaster should ideally be less than the time required to deal with the fallout from that potential disaster multiplied by the chance of it occuring15:53
AJaegerhere's a simple change to update system-config after refstack rename, it could use another review, please -
AJaegerthanks, clarkb15:59
AJaegerfungi, your twine change has enough +2s, please self-approve anytime16:00
clarkb exists now. I'm working on updating my change to run that version16:00
corvusclarkb, fungi, mordred: yeah, i think the best things we can do are review carefully and make sure backups are current.  should we do a restore test for review.o.o?16:01
corvus(just a simple one -- not a new server, just try to extract the current state on the backup server)16:01
clarkbthinking out loud here, maybe thats the way to sync to review-test16:01
clarkbfrom backup and not directly from prod16:01
openstackgerritJavier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror
openstackgerritJavier Peña proposed opendev/system-config master: Use the ansible service task to reload iptables on CentOS
openstackgerritJavier Peña proposed opendev/system-config master: Make the base role and playbook compatible with CentOS
*** ykarel is now known as ykarel|away16:03
openstackgerritClark Boylan proposed opendev/system-config master: Update to gitea 1.12.0
clarkbinfra-root ^ fyi16:10
clarkbI'm going to pop out on a bike ride before the sun makes things too warm16:11
fungii was hoping to mow the lawn today, first time in a week there's not been standing water in the yard, but rain's coming back :/16:12
openstackgerritMerged opendev/system-config master: Update refstack reference after rename
mordredclarkb: hrm. that's not a bad idea16:16
mordredclarkb: are we backing up the git repos though?16:17
clarkbWe should be16:21
clarkbwe exclude not include backup input16:21
fungiyeah, i was just poking around on review.o.o checking for the excludes16:22
fungithough i don't recall us specifically excluding the path to the git repos16:22
mordredstill - it might be a little more dangerous - because we'd have to make sure we ran ansible on review-test after the backup restore to be sure we haven't written review.o.o credentials or config16:22
mordredlike - that actually makes me more apprehensive than the current sync script16:23
mordred(althogh there is obvious benefit)16:23
clarkbya may want to limit it to just the git repos and the sql dump and nothing else16:23
clarkbbasically replace your rsyncs with bup16:23
mordredalso - fwiw, I think our log rotation is doing something weird with mysql backups16:24
mordred-rw-r--r--  1 root    root             0 Jun 19 06:25 gerrit.sql.gz16:24
fungii suppose we can be reasonably sure gids/uids match between those servers, so won't need to chown the restored repos?16:24
fungimordred: it is weird, but it's explainable16:25
mordredgerrit.sql.gz is zero bytes - but the others are all correct16:25
fungilogrotate creates an empty file16:25
fungithen mysqldump later writes to that file16:25
mordrednod. just as long as we're not backing up only the gerrit.sql.gz to bup :)16:25
fungiand then logrotate renames it out of the way and creates a new empty file16:25
fungiso yes, the most recent mysqldump will be in gerrit.sql.gz.1 or whatever16:26
fungibut yeah, this seems like a worthwhile experiment because at the very least it'll help us confirm whether our backups are good16:32
fungialso if we want to be extra safe we can run another mysqldump and bup pass immediately before we try the automation, just to be sure we have a little data loss as possible if something does blow up the production server somehow16:32
fungier, AS little data loss as possible16:33
openstackgerritJavier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror
*** ChanServ has quit IRC16:59
openstackgerritJavier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror
openstackgerritJavier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror
openstackgerritJavier Peña proposed opendev/system-config master: Make the base role and playbook compatible with CentOS
openstackgerritJavier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror
mtreinishclarkb, fungi, mordred: do you have any experience with: ? Someone just submitted a bunch of PRs to projects I maintain (for example ) that proposes switching to it17:10
mtreinishI tend to be skeptical of anything that relies on pyproject.toml because it's been pretty fragile in my experience for what little I've used it17:11
fungimtreinish: i haven't used it at all recently. i think maybe setuptools_scm was one of the contemporary solutions we looked at years ago before writing pbr17:15
mordredyup. we replaced use of it with pbr17:16
mordredor - at least its predecessor17:17
fungii now don't recall what our problems with it were, but odds are it's improved in the many years since17:17
mtreinishheh, well that was my first reaction looking at it, "this is like pbr" :)17:17
mordredyeah - I think they mocked us for writing pbr, then went and wrote the same thing but slightly different iside their little in-crowd of people17:17
mordredI dont' have any negative feelings about that at all clearly :)17:18
fungibut yeah, pbr was effectively a fork of distutils2, right?17:18
mordredinfra-root: I'm afk for the next several hours - gotta go get the kittens vaccinated17:18
fungiwe obviously picked the wrong in-crowd17:18
fungimordred: have fun17:18
mordredmtreinish: but in direct answer - I have not played with that in earnest in quite some time because pbr is pretty good at what it does and I don't see the need to17:18
mtreinishmordred: ok, thanks17:19
mtreinishhave fun with your cats17:19
mordredmtreinish: that said - I do think anything that uses declarative metadata is better than having _anything_ in setup.py17:19
mordredso I'd be inclined to accept that pr because it is a definite improvement over code in setup.py17:20
mordredmtreinish: thanks! there are ... many of them :)17:20
mordred(and then you can tell us whether setuptools_scm works :) )17:20
mtreinishI definitely agree with that. I just get worried with changes for retworkx and a couple of other projects with compiled extensions (cython and c++ instead of rust) that rely on build time dependencies being present before is run17:22
mtreinishbecause it was really finicky to get things in a place where you could install those from sdist without it erroring because something was missing (despite having a build dep set in the metadata)17:22
mtreinishwhich is why the in that pr subprocesses out to pip install setuptools-rust, which was the best answer I found17:23
*** ChanServ has joined #opendev17:37
*** sets mode: +o ChanServ17:37
*** mlavalle has quit IRC17:40
openstackgerritRafael Folco proposed openstack/diskimage-builder master: DNM: Debug py3 on dib 7
*** jaicaa_ has quit IRC18:11
*** mlavalle has joined #opendev18:11
*** jaicaa has joined #opendev18:19
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Fix and test multiarch docker builds in a release pipeline
openstackgerritSergii Golovatiuk proposed zuul/zuul-jobs master: Install source Python for supported OSes
*** calcmandan has quit IRC18:31
*** calcmandan has joined #opendev18:32
clarkbinfra-root I'm thinking it might be a good idea to deploy on a single gitea backend18:35
clarkblargely to see what the memory use looks like with the new caching18:35
clarkbI'm not quite sure how to do that yet, but if I can get some reviews on the chnage (without approvals) I can sort that out this afternoon18:35
clarkbI think it will be something like put gitea01 in the emergency file, edit the docker compose file to pull the image for that change, docker-compose down then up?18:36
fungithat sounds like a reasonable goal... maybe you can stick gitea01 into the emergency disable list and locally edit the compose file?18:36
fungiheh, yep18:36
clarkbalso I'm going to grab some quick render times for nova and neutron and friends to compare against post upgrade18:37
clarkb put some timing notes there. Its not terribly scientific but should give us an idea if its better or not18:47
clarkbfungi: the other way we could do it is put 02-08 into emergency and then land the change and have it update 01 only19:00
fungiyep, i thought about that possibility as well19:01
fungithough it does result in more to back out if something doesn't go as expected19:02
clarkbya probably just editing 01 pre merge is simplest19:03
clarkbfungi: maybe you can give that change a look over and if you and someone else think it looks good I'll start on 0119:07
fungiyeah, sorry, had to step away to shape pizza dough, back for a while now19:31
clarkbno worries I'm trying to figure out what lunch looks likehere and now I want pizza19:32
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Fix zuul-jobs-test-ensure-python-pyenv files
clarkbwe're seeing peoplecontinue to use meetpad19:45
clarkbwhih does make me think that sunsetting pbx is the right move19:46
clarkb(ironic used it recently and interop is planningto use it)19:46
fungiyeah, makes sense. also means we should maybe talk about whether to keep the account for the dial-in line, and how we might want to incorporate it into jitst-meet if so20:01
clarkbfungi: those comments explaining deltas from upstream html templates are good feedback I'll see about adding that to a followup20:20
*** aannuusshhkkaa has joined #opendev20:24
fungiyeah, not critical, just things which sprang to mind as i was reading through, certainly nothing to hold up that patch20:25
*** _mlavalle_1 has joined #opendev20:25
*** mlavalle has quit IRC20:26
*** priteau has quit IRC20:36
clarkbmordred: when cats are happy do you think you could give the gitea 1.12.0 change a look over again? If it still looks good I'll get gitea01 updated20:38
*** hashar has joined #opendev20:54
*** josephillips has joined #opendev21:04
*** factor has joined #opendev21:04
josephillipsif i want to perform a fork from a actual openstack project can i keep it on opendev?21:04
josephillipsor you recommend use github for this21:04
clarkbjosephillips: I think our strong suggestion would be to work with that project directly21:05
clarkbwe've historically not prevented forks for packaging purposes (though that also means we've got like 2 extra copies of a bunch of repos that are completely dead now)21:05
josephillipsi wish work on charms repo21:06
josephillipsbut for things i probably will use only for us21:06
clarkbanother thing to consider is if the fork is intended to be a long term situation. If not then working with the existing project becomes more important imo21:07
clarkbjosephillips: what is the goal here?21:07
josephillipswell modify the original charm to add features that will be requiered for own infrastructure , my goal is keep a fork in opendev and merge from new changes to my fork when they commit to they master branch21:08
clarkbjosephillips: and that can't be done with the existing project?21:09
josephillipsi tried but they told me that these changes need to be discussed with the cannonical team etc etc21:10
josephillipsand this why i prefer keep a fork21:10
josephillipsand probably will be thing of months21:10
clarkbgot it. In that case a proper fork is probably not the end of the world. Another potential option depending on how much you think you'll diverge is to use proposed state in gerrit21:10
clarkb(which is definitely less nice thant git clone and go)21:11
josephillipsfor something that i already have working on my infrastructure im just looking a way to keep updated my charm and when if someone else look utils then can use it too21:11
clarkbjosephillips: ya so you can fetch gerrit changes and use them directly but that won't be already on the master branch and ready to go after a clone (would need a separate fetch and checkout)21:12
clarkbthat's the very lightweight fork option where you know they won't merge things in the near term but you can still consume the change from refs/changes/xy/abcxy21:12
josephillipsyou mean commit to the original project my changes when they perform the review just will rejected but keep the changes on this way?21:14
clarkbI don't know that it is the best option here, but wanted to point out it is an option21:15
josephillipsyeah i know21:15
josephillipsim just thinking if is fair with the reviewers do that21:15
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add tests for upload-docker-image
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Fix and test multiarch docker builds in a release pipeline
corvusclarkb: i'm thinking of landing zk now:
corvusload isn't crazy; so i'll land that then shutdown and bringup all the zk hosts real quick21:31
clarkbcorvus: rgr. I'm around if I can help21:32
clarkbI started looking at my gitea plan more and realized we don't have the change specific tag on github?21:32
clarkber docker hob21:32
clarkbso not sure that plan is going to work and am poking around at it to see what options I've got21:33
corvusclarkb: yeah, we clean those up21:33
clarkbI think build the image myself or do the 02-08 in emergency and have just 01 update21:33
fungii'm mostly around, though going to see if i can still get to the lawn21:33
corvusclarkb: you want to manually deploy a gitea image we built in check or gate?21:33
corvusclarkb: if so, you can use the intermediate registry21:34
corvusclarkb: link me to the change that built the image you want21:34
clarkbcorvus: the previously discussed plan would've been check, but now I'm thinking just approving the change with 02-08 in emergency is the path forward and that will use gate's promoted image as usual21:34
corvusclarkb: let's follow through the check thing first i think it might work21:34
clarkbcorvus: k21:34
corvusclarkb: docker://
corvusclarkb: docker://
clarkbcorvus: cool so I can docker pull that21:35
corvusclarkb: those are from
corvusclarkb: right click on the artifacts and copy their urls21:35
corvusclarkb: yep21:35
clarkbthen the local image table will find it when docker-compose restarts it21:35
corvusshould be able to confirm by just running locally on a workstation too21:35
clarkbya I'll start there21:36
corvus"docker run"  does something for me :)21:36
clarkbjosephillips: to recap I think the prefernece is to at least have the groups talk about it first and ensure we aren't avoiding potential for collaboration or adoption of projects that need help. It sounds like you've done that and the two ideas are still independent so a fork isn't the end of the world21:37
corvusmnaser: ^ fyi since you're doing more with images -- the intermediate registry can be a useful debugging tool; lets you manually run what was built in check or gate.21:37
clarkbcorvus: heh I was going to set up a docker-compose file. I think I'll still do that but good to know the super simple case is happy21:37
mnaserooou, that's REALLY neat21:37
corvushonestly, if openstack uses images more and more, this could be an excellent way for devs to reproduce problems.21:37
corvusmnaser: yeah, it was one of two main reasons we wrote zuul-registry21:38
corvusthat plus the split-horizon thing with it shadowing multiple upstream registries21:38
corvusour first intermediate registry, which was just the upstream docker registry, couldn't handle the split authentication needed for auth-write / anon-read21:39
corvusclarkb: well i mean the 'docker run' pulled an image for me, i didn't try to use gitea or anything :)21:40
clarkbya I'm putting those images into the docker compose file to see if that just works (tm) (I'm also commenting out the volumes because I don't want to set those up on my laptop but this should at least confirm docker-compose does what we expect)21:43
clarkbcool docker-compose is happy with that but it results in errors due to perms but thats fine that is due to me commenting out the bind mounts21:46
clarkbcorvus: fungi: should I go ahead and put gitea01 in emergency now and run gitea off the images corvus linked above? or would we like to see mordred review the change first?21:47
clarkbI'm happy to wait for zk thing sfirst too21:48
fungii think it's fine to go ahead, it's just a test anyway, no real commitment and if something goes sideways we can roll back21:49
clarkbfungi: yes, but rollback may mean rebuilding on the previous version21:49
clarkbI'm not sure how safe it is to go backwards with the db21:49
clarkbbut we do have 7 so we can do that with little impact21:49
fungiright, worst case raze the earth as far as 01 is concerned21:51
corvusi agree with fungi; and no need to wait for zk21:52
clarkbk putting 01 in emergency now21:53
*** hashar has quit IRC21:54
clarkbstopping services on 01 now21:55
clarkbgitea01 services are up21:59
clarkbit seems to be working21:59
clarkbloading nova root first time took 8 seconds. Second time 1 second give or take22:00
clarkbpreviously it was just under 4 seconds22:00
clarkbdefinitely seems like cold is still slow but warm is quicker22:01
clarkbwich isn't surprising given its a cache that is used22:01
clarkbthe other thing I want to monitor is memory use22:02
clarkbwhich may be a check cacti on monday item and if happy land the change22:02
clarkb is the graph to monitor22:04
clarkb#status Log put gitea01 in the emergency file and manually upgraded gitea to 1.12.0 there. This allows us to check the performance updates and memory use after the addition of caching.22:05
openstackstatusclarkb: finished logging22:05
fungiyeah, i don't have confidence our workload between now and monday will be sufficient to really identify memory increases22:06
clarkbat the very least it hasn't immediately imploded and it seems to be quicker for our large repos. I'll call that a win :)22:09
clarkb is a new thing that we might want to pull out of the UI22:11
clarkbits not a direct problem but it doesn't show anything useful either and people may read into that too much22:11
openstackgerritMerged opendev/system-config master: Zookeeper: listen on plain and TLS ports
clarkboh actually it may be more useful than I thought22:13
clarkbya I think we can keep it22:13
corvusoh right, this change runs all the deploy jobs22:17
corvusso it's going to be hours until it gets around to running infra-prod-service-zuul22:18
corvusmordred: ^ what did we say we were going to do about that?22:18
corvusi guess i'm done for the day, and maybe we can restart zk sometime this weekend :/22:19
clarkbcorvus: I think the idea was to start being specific about which bits of host vars and inventory map to which jobs so that changes to one small corner of that information doens't trigger everything22:20
corvusclarkb: i agree we should yank activity22:20
clarkbcorvus: ya its like almost useful22:21
corvusclarkb: yeah, it's technically correct22:23
*** _mlavalle_2 has joined #opendev22:23
corvusbut i'm not sure 'excluding merges' is the right way to characterize our development :)22:23
corvusthe branch list is not sorted; that seems weird.22:24
corvusrather, it's not alpha-sorted22:24
*** _mlavalle_1 has quit IRC22:24
corvusthat's not a change though, just something i noticed now22:25
openstackgerritClark Boylan proposed opendev/system-config master: Small repo template cleanups in Gitea
clarkbfungi: corvus ^ I did the repo template cleanups in a followup to help make it more clear that the first thing is what we're testing22:27
fungithanks, i needed an excuse to sit down and not push the mower for a few minutes anyway22:28
*** _mlavalle3 has joined #opendev22:29
*** _mlavalle_2 has quit IRC22:31
*** _mlavalle3 has quit IRC22:57
*** DSpider has quit IRC22:58
*** tosky has quit IRC23:45

Generated by 2.17.2 by Marius Gedminas - find it at!