Wednesday, 2023-03-08

clarkbya its py36 and newer which is newer than xenial00:00
clarkbsmoething something we should just use nox00:03
clarkb :)00:03
clarkbthe afs grafana dashboard reflects the quota changes I made now00:05
ianwclarkb: we are already setting up the verified tag in our test as a submit-requirement ->
ianwi can probably do something like edit the base file from gerrit to turn the code-review into a submit-requirement00:12
JayFclarkb: if it makes a difference to you; I'll be out of town that day thru Thursday of the following week.00:12
ianwor maybe add a verified tag with similar00:12
JayFclarkb: although I don't think I have any magic knowledge that other Ironic cores involved (rpittau or iurygregory) wouldn't know00:12
ianwbut i'm wondering if it's worth us really maintaining that.  does it really give us more coverage than the verified tag we have?00:12
clarkbJayF: no, that just helps reinforce its likely to be a good quiet time00:13
clarkbianw: verified is probably fine. Looking at htat I thought I set the function to be noop/noblock but it isn't set there00:13
ianwactually interesting.  that should be able to push to 3.700:14
clarkbianw: I thought that had landed. Maybe we should switch to NoBlock and send it in?00:14
JayFclarkb: when is that official enough  I can tell ironic community? Now? 00:15
ianwclarkb: ++00:15
clarkbianw: I think what I found was the checks for function only look if you set a function value. But if ou don't set one it doesn't check it and you get the default00:15
opendevreviewClark Boylan proposed opendev/system-config master: Switch Gerrit test Verified label function to NoOp
clarkbianw: ^00:15
clarkbJayF: its probabl official enough now with the note that as we get closer if we aren't ready for some reason we'll delay00:15
ianwclarkb: maybe just switch the NoOp/NoBlock in the subject before it runs ci?00:16
clarkbbut we've got a month to sort things out00:16
clarkbianw: I'm not sure I know how to parse that statement00:16
ianwmake the subject "Switch Gerrit test Verified label function to NoBlock" just to be accurate to the change i mean 00:17
clarkboh sorry00:17
opendevreviewClark Boylan proposed opendev/system-config master: Switch Gerrit test Verified label function to NoBlock
clarkbThis is what I get for using Gerrit's edit functionality00:18
clarkbianw: ^ done00:18
clarkbfungi: I went ahead and approved the git-review change since I don't really know of anyone else regularly reviewing changes there00:19
ianwok, if we're in agreement there's probably not much point doing more work in the gate all-projects, i think getting probably fungi to just look over and
ianwand then i can probably try pushing that to our all-projects at some time mid-morning for me, when it's quiet but not too quiet00:21
ianwit should make no difference, but if it does we can revert quick and re-evaluate00:21
ianwassuming it works, i'll feel better about moving on with the bigger updates to all the other acl's00:21
fungisure, taking a look at those now00:29
fungilooks like it needs 876236 too00:29
fungiwhich i just approved now since it's purely documentation edits00:30
fungiapproved both of them00:31
fungiianw: the diff in that paste lgtm, please proceed at your convenience!00:32
ianwthanks!  i think i'll try it tomorrow morning 00:34
opendevreviewMerged opendev/git-review master: Upgrade testing to Gerrit 3.4.4
opendevreviewMerged opendev/system-config master: doc/gerrit : update copyCondition
opendevreviewMerged opendev/system-config master: doc/gerrit : update to submit-requirements
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Provide deploy-microshift role
opendevreviewSaggi Mizrahi proposed opendev/git-review master: Add worktree support
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Provide ensure-microshift role
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Provide ensure-microshift role
*** jpena|off is now known as jpena08:39
ianwfor reference the f37 sync finished and i've dropped the locks09:04
bbezakHi, I can't fetch any build in zuul -, and particular builds are "not found"09:42
bbezakor "This build does not exist"09:45
bbezaklooks ok now10:23
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Provide ensure-microshift role
opendevreviewRadosÅ‚aw Piliszek proposed openstack/project-config master: Add the NebulOuS tenant
opendevreviewSaggi Mizrahi proposed opendev/git-review master: Add worktree support
*** odyssey4me is now known as odyssey4me_13:05
fungibbezak: there may have been a temporary object storage outage in one of the clouds we use to store job logs13:19
*** elodilles is now known as elodilles_afk13:20
fungiwe rely on the swift services in rackspace and ovh public clouds for that, so if one of those was down then ~50% of recent builds would have had no data in the webui13:20
fungigraphs for inmotion-iad3 look much better today. rax-ord is still rather rough though... it does sort of look like something, quite possibly us, is dragging down api response times there13:39
fungii suppose we can try cranking up the timeouts more, though that means potentially even longer that jobs have to wait to get node assignments13:40
opendevreviewJeremy Stanley proposed openstack/project-config master: Wait longer for rax-ord nodes and ease up API rate
*** elodilles_afk is now known as elodilles14:58
johnsomFYI, meetbot seems to be not functioning in #openstack-lbaas today16:02
fungihave you tried without an extra space before the #?16:10
fungiit looks like both of you did " #startmeeting ..." instead of "#startmeeting ..."16:11
fungijohnsom: ^16:11
johnsomfungi Doh! good catch, thanks. Cut/Paste error16:12
fungiarguably meetbot could be improved to be a little more lenient with leading whitespace16:16
bbezakthx fungi for explanation16:26
clarkbfungi: can you review as a followup to the testing of gerrit with our proposed acl changes discussion yestyerday16:42
clarkbshould be an easy one. Basically converts the verified label to a submit requirement in a similar manner to what will be done in prod on our test node16:43
clarkb is the next gitea replacement related change. THis one will need a little bit of monitoring.  Ithink I should be able to do that later today if you don't feel like approving16:44
fungioh, yep16:44
clarkbI had previousl half converted the verifed label but then we realized not setting the function made it default to MaxWithBlock so the new submit requirement wasn't doing anything16:45
fungiyeah, i sort of understand the problem there16:47
clarkband then is another change that would probably be good to land before we upgrade gerrit. But the urgency here is low (this updates our gerrit image builds to use new gerrit tag versions for plugins but those are all equivalent to the old code except for one location which we were already overriding to something else that was16:47
fungiand it's only for the test config anyway16:47
fungi(the earlier change i mean)16:48
clarkbgitea09's load average is currently well over 8 (it has 8 vcpus)17:06
clarkbmy initial inclination is that implies we should add a couple more giteas17:06
clarkbhowever, if all that load is coming form the same nat address then this might not help so I may need to look more closely at the logs before making that decision17:07
clarkbbut its good data for the question of whether or not we add more serves17:07
fungior fewer larger servers i guess17:09
clarkbmy local mapping is to gitea09 and it seemsto be snappy in my browser so this may not be as big a problem as I initially feared17:13
fungiclarkb: not sure if you saw, but the next experiment i've proposed for rax-ord is 87687417:14
clarkbI missed that. Looking17:14
clarkbapproved. Seems like a reaosnable thing to test17:14
fungiseems like we could be our own noisy neighbor problem there, but hard to tell still17:14
johnsomfungi Yeah, I might propose a patch for that. I have two other things I need to look at first, but probably worth the effort.17:14
clarkbya we might want to drop max servers to like 10 and see if the other metrics become more normal17:15
clarkbbut lets see what being patient with boot times does17:15
opendevreviewMerged openstack/project-config master: Wait longer for rax-ord nodes and ease up API rate
clarkbfungi: ^ the nodepool fix for slow restarts of the provider mnaager should have deployed yesterday to17:23
clarkbthis change will be a good exercise of that.17:23
fungioh, yep right17:24
clarkbfungi: you good with me self approving at this point? Considering all he nodes currently serving content are jammy now I think it is a good idea to land17:25
fungii went ahead and approved it17:28
opendevreviewMerged opendev/system-config master: Switch Gerrit test Verified label function to NoBlock
clarkblooking at gitea graphs for the other servers gitea11 is also quite busy and the others haven't been slouching either17:32
clarkbI think this has me leaning towards booting another couple of servers17:32
clarkbI've got an appointment this morning, but can kick that off later today17:32
*** jpena is now known as jpena|off17:33
fungiit doesn't look dire anyway17:33
clarkbya I don't think its dire, but does point to these servers maybe being a bit too little when demand is high17:33
clarkbbut I suspect replacing 8 smaller servers with 6 larger ones may be sufficient so I'll just do 2 I guess17:34
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: WIP Correct boot path to cover FIPS usage cases
fungimakes sense17:36
clarkbThe good news is I've got the process pretty well sorted out now. It shouldn't take long other than witing for gerrit replication steps to occur17:38
clarkblookingat cpu graphs I wonder too if this is due to iowait17:43
clarkbin theory splitting this into more hosts will reduce io demands to a single VMs disk17:43
fungiquite possibly17:47
fungidoesn't really look like iowait to me17:49
fungiat least not according to top17:49
clarkbfungi: cacti reports up to 5% iowait on gitea11 ercently17:49
fungia lot of the reported cpu is "st" (time stolen from this vm by the hypervisor)17:50
clarkbfungi: thats via top?17:51
fungithe "wa" column (time waiting for I/O completion) is comparatively low17:51
fungii'm guessing this is how modern kernels expose processor overcommit info17:52
clarkbya and thats out of 100% not 800% in top right?17:52
clarkbadding more hosts should mitigate that too unless we end up on the same hypervisor?17:52
fungiright. the 1 key will toggle per-processor view/percent17:52
clarkbI'm not in a rush to add more hosts if we want ot monitor it more17:53
clarkbI do need to pop out for that appointment now though. But let me know what you think17:53
funginow, this *could* be iowait on the hypervisor which doesn't show up as iowait on the guest, i don't really know17:53
fungiwell, i still agree that adding another couple of backends is good to try and see if it helps. we can always delete them again if we decide it didn't17:54
fungialso we might be able to see if mnaser or guilhermesp can help us fine-tune things if it really is iowait hidden from us on the hypervisor side17:56
fungifor example we might be able to isolate the i/o to more performant storage with a separate volume, or set up anti-affinity rules to make sure the backends wind up on different hosts, or whatever else makes sense17:58
fungimight be this is a difference in disk throughput, since we switched from bfv to local storage for the rootfs18:01
fungiin rax-ord/nodepool news, the config update deployed but i think nl01 is still running older code and will need a manual restart18:05
fungi2023-03-07 19:46:03,043 DEBUG nodepool.StateMachineProvider.rax-ord: Stopping18:05
fungiand no "stopped" in the log18:05
fungiso it's been in the process of stopping for about 22.5 hours18:05
fungii expect we'll need to down/up the launcher container like we did on nl02, but i'll wait until more folks are around since this isn't urgent18:08
fungiclarkb: when you're back, could also pick your brain on git-review testing. apparently the output you had to update for changes in newer gerrit isn't stable, and the fields can be returned in an arbitrary order. in one test we were looking for "Processing changes: new: 1" but got "Processing changes: refs: 1, new: 1\nremote: Processing changes: refs: 1, new: 1\nremote: Processing changes: refs:18:17
fungi1, new: 1, done [...]"18:17
clarkbok that went quickly.18:19
clarkbfungi: the fix for the nodepool thing still runs the long shutdown process but it does so in another thread and ignores it basically18:20
clarkbfungi: so the stopped may not show up for a while but the new manager should run with the new settings anyway18:20
clarkbthe thing to look for is probably the boot timeouts to see if they are at 10 minutes or 1518:20
fungiahh, okay18:21
clarkbfungi: for git-review can you be more specific. The old side "Processing changes: new: 1" is still showing up?18:21
fungii don't think the boot timeout values get logged, but can likely be inferred from timestamps on a node that does time out18:21
fungiclarkb: oh, i just realized that the failure is likely a test introduced in the change being reviewed which copied from existing tests prior to your update18:22
clarkbah ok. Ya the old stuff needs to be updated I think. But the new content should be stable from what I've seen so far18:22
fungii'm working to update it now, just mildly confused that it only triggered on the py39 job and not others18:23
clarkbok I'm booting gitea13 and gitea14 now18:25
fungii thought you were at an appointment? back already?18:26
clarkbya we switched it to an online appointment because one of the kid sis sick today and that cut out the travel time which made it go much more quickly18:26
fungigot it18:27
clarkbusually its half na hour of driver and half an hour of meeting. Today just half an hour of meeting18:27
clarkbheh finally hit a quota exceeded error for instance count. I guess that means I'm going to delete gitea08 first18:28
clarkbany objections to me doing that now?18:28
funginope, go for it18:28
funginoticing git remote origin operations and browsing gitea are slow to respond, not sure if it's a problem local to me or the result of the servers being more heavily loaded18:29
clarkbI think more of them are getting loaded so probably not just you. We can also reenable the old servers. I'll do that now18:30
clarkbgitea01-04 have been reenabled in haproxy18:31
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: WIP Correct boot path to cover FIPS usage cases
clarkblooks like system load is falling. I'm slightly worried that we might be our own noisy neighbors here and that putting the old servers back helps because they are on different hardware18:36
clarkbI'll proceed with gitea13 and 14 since that is what we can control, but maybe before deleting gitea01-07 (I have to delete 08 to make room) we try to clarify some of this behavior?18:37
clarkblooks like a ton of cat-file commands which isn't normal fetching18:37
opendevreviewMerged opendev/system-config master: Convert gitea99 test node to Jammy
clarkbfungi: looking at logs I think we're being crawled18:40
clarkband its through the load balancer so not something finding new webservers for the new giteas (yay non standard ports)18:41
clarkbok ya truning on the other ndoes definitely caused them to spike too18:54
clarkbwhats curious is they seem to have run more quickly with the load though that may just be due to better distribution18:54
fungiyeah, could be a combination of crawler botnet hitting all the backends plus rh corporate nat getting balanced to one backend18:55
clarkbI've got a quick sort and count going against the haproxy log to see if anything stands out18:55
*** odyssey4me_ is now known as odyssey4me18:56
clarkbbut ya I guess I should delete gitea08 to free up room for gitea14. Then we can add two more and see where that gets us18:56
clarkbbeing cautious to not delete the other old servers too soon18:56
*** odyssey4me is now known as odyssey4me_19:04
*** odyssey4me_ is now known as odyssey4me19:04
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: WIP Correct boot path to cover FIPS usage cases
fungiclarkb: not urgent, but the git-review change also failed on this (unchanged from master):
fungithat leads me to suspect the messages may be nondeterministic since otherwise we'd expect it to have failed on your earlier change19:07
clarkbdeleting gitea08 did not delete its volume19:11
clarkbits bfv so that volume really only makes sense in the context of having the server so I will delete the volume now too19:11
opendevreviewJeremy Stanley proposed opendev/system-config master: Block another bogus crawler botnet UA
opendevreviewJeremy Stanley proposed opendev/system-config master: Block another bogus crawler botnet UA
clarkb#status log Deleted gitea08 and its associated boot volume as part of gitea server replacements19:13
opendevstatusclarkb: finished logging19:13
clarkbfungi: comment on 87688919:14
opendevreviewJeremy Stanley proposed opendev/system-config master: Block another bogus crawler botnet UA
*** odyssey4me is now known as odyssey4me_19:22
fungiclarkb: thanks, fixed and also commented with a reference to the apache docs which describe the =19:22
*** odyssey4me_ is now known as odyssey4me19:22
opendevreviewJeremy Stanley proposed opendev/git-review master: Add CC similarly to reviewers
clarkb+2 but I didn't approve yet. Maybe when that lands we should remove gitea01-04 again and see if we hold stable or not19:31
clarkbgitea14 is almost done bootstrapping. I'll get changes up for DNS and adding them to the inventory shortly19:31
*** odyssey4me is now known as odyssey4me_19:31
*** odyssey4me_ is now known as odyssey4me19:32
clarkbanyone know why when I do `uniq -c` on a large number of records I have to first sort the input to get accurate results? Is uniq operating on really small buffers?19:33
clarkbI wonder if we didn't OOM bceause we've got more memory now but we would've if we were still running the old servers19:36
clarkbIf that is the case then this ended up being a more graceful handling of things I guess19:36
*** odyssey4me is now known as odyssey4me_19:37
*** odyssey4me_ is now known as odyssey4me19:37
clarkbfungi: Internet seems to say steal is basically when the host isn't providing the resources expected. So ya we could be our own noisy neighbors here etc. I don't think this is due to IO though. Ist due to other things needing the cpu19:40
clarkbit seems much higher on gitea09 and gitea11 than 10 and 1219:42
fungiclarkb: afaik, uniq only deduplicates adjacent lines19:43
fungii usually |sort|uniq -c instead19:44
fungiuniq's manpage concurs with my recollection19:44
clarkbfungi: so gitea14 reuses gitea08's IP addr. Should I commit the removeal of gitea08 from dns as a separate change then add gitea14 in or just do it as one?19:46
clarkbI don't think there is a difference to dns propagation as thats all ttl based19:46
fungii'd just do it in one change19:49
opendevreviewClark Boylan proposed opendev/ master: Add gitea13 and gitea14 to DNS
opendevreviewClark Boylan proposed opendev/system-config master: Add gitea13 and gitea14 to inventory
clarkbOnce those land we wait for giteas to be deployed then I can do the brain transplants then we can set up replication19:53
clarkbIn the meantime we can leave the 4 old servers behind haproxy along with the 4 new ones19:53
fungisounds good19:53
clarkbmaybe drop the 4 old ones again when the apache rule lands to see if that helped19:53
clarkbfwiw current cpu steal on gitea13 and 14 looks low like 10 and 1219:55
fungianti-affinity rules could help then, if possible19:56
fungithough 6-way anti-affinity might be tough on placement19:56
clarkbI don't think we have access to that from the nova apis as a regular user20:06
clarkbfungi: has a +1 now. We can probably wait for ianw to take a look since the immediate fire has been dealt with20:07
*** odyssey4me is now known as odyssey4me_20:15
*** odyssey4me_ is now known as odyssey4me20:15
*** odyssey4me is now known as odyssey4me_20:17
*** odyssey4me_ is now known as odyssey4me20:17
opendevreviewMerged opendev/ master: Add gitea13 and gitea14 to DNS
clarkbok other than being worried about cpu steal making the new servers potentially worse than the old ones I think I've run down what I can for now. I'm going to grab lunch20:18
Clark[m]fungi: re git-review maybe it is deterministic based on the command you run?20:30
Clark[m]fungi: probably the best thing is to just look for new: 2 or new:1 instead20:31
Clark[m]We check separately for the success message and the commit titles. I think that checking the minimal count with those extra checks is sufficient 20:31
fungiwell, the command we're running hasn't changed though20:32
fungiat least i'm not sure why test_multiple_changes() would have suddenly started to fail on 849219 when it worked for your change with newer gerrit20:34
fungiand also it only failed on tox-py39 but not 36,37,3820:35
Clark[m]Oh it failed. That's what I missed. I thought you were comparing to what I added at the end of the test which is the new version20:38
Clark[m]Maybe ref: 1 only happens when git receives a new ref?20:38
Clark[m]And could be a timing issue for that where other tests may have already written the commit to Gerrit it or something 20:39
Clark[m]But that info comes directly from Gerrit and git-review passes it through so I don't think we need to be super specific about it in the tests20:39
fungiagreed, i think we can just simplify/shorten the strings we're looking for20:40
fungii'll whip something up20:40
Clark[m]fungi: at the end of that multiple changes test I added asserts to look for both commit messages implying both were pushed as well as the SUCCESS message20:44
opendevreviewJeremy Stanley proposed opendev/git-review master: Simplify test output strings for new Gerrit
*** odyssey4me is now known as odyssey4me_20:47
ianwthat UA patch looks right to me.  yeah the "=" at the front means you can just paste in the full line, that's the idea20:47
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: WIP Correct boot path to cover FIPS usage cases
clarkbfungi: ya that change seems sufficient given we're still checking the count and that gerrit data is passed through21:06
clarkbI've done more checking and gitea09+gitea14 share a hostId as do gitea11+gitea13. Unfortunate because those were the nodes with the cpu steal happening21:33
clarkb10 and 12 are distinct21:33
clarkbso ya potential for being our own noisy neighbor and something to pay attention to I guess21:33
clarkbianw: I went ahead and approved the UA change21:37
clarkbnot sure if it ws intentional to not approve it. But you've got a few minutes to -W it if so21:37
fungii suppose we could keep relaunching servers until we get unique hostids across them all, but that seems borderline abusive and we should be conscientious stewards of these resources21:52
fungiand also host migrations in the future might undo that without our noticing anyway21:53
opendevreviewMerged opendev/system-config master: Add gitea13 and gitea14 to inventory
ianwi'm going to add myself to project bootstrappers and push the s-r change to all-projects as discussed yesterday.  if it's not a complete no-op i'll revert22:07
ianwok, it's done22:08
ianwchange pages look ok22:08
ianwthe conditions all look about right to me22:09
fungilgtm, thanks for doing that!22:09
ianwhrm, status says "n/a".  not sure it did that before22:09
ianwdoes anyone have an old page up (maybe don't refresh it just yet)22:10
opendevreviewMerged opendev/system-config master: Block another bogus crawler botnet UA
ianw looks right.  it's got the crossed grey circle22:11
ianwand says like 2 (of 3) about the SR22:11
clarkbI pulled up my old page but it seemed to autorefresh22:12
clarkbit says n/a there22:12
clarkbianw: another check would be trying to vote on something you don't have rights to22:12
clarkbthat looks fine for me22:12
clarkb(I pulled up a random nova change and can onl +/-1 code review)22:13
ianw ... some have n/a and some don't.  trying to see if there's a pattern ...22:13
ianwmaybe it's if all SR are unsatisfied?22:14
ianw is one for example.  it has a +1 one from zuul 22:15
clarkbah ya I've got some that say 2 of 3 that have code-review +2 and verified +1 but lack the workflow +122:15
clarkboh wait its 2 unsatisfied of 322:15
ianwbut then does as well, and that i see without n/a22:15
clarkbsince verified also requires a +222:15
clarkbianw: could it be an indexing thing maybe?22:16
clarkbianw: perhaps make a comment only (no vote) update to a chagne and see if it toggles22:16
clarkbI believe any action like that will cause the change's index content to be updated22:17
ianwyeah, i don't think we can really get around that.  we're basically never having the s-r satisfied for "humans" in our case22:17
clarkbI don't think its a big deal22:17
clarkband the n/a doesn't seem to be a problem either. Just a matter of understanding why I guess22:17
ianwyeah, i think that's it22:19
ianwi commented on and now it shows up as 0 (of 3)22:19
clarkbcool that will sort of naturally correct itself for active changes22:20
ianw(btw, that little stack probably wants reviews.  i'm a bit stuck on it because it seems like submodules have weird semantics, and i'm not sure what zuul should do with them)22:20
ianw++ i think that's right22:20
clarkbianw: I've been deferring to corvus on the submodule stuff as I think he groks it reasonably well (and I do not)22:21
fungii break out in hives when i see a submodule22:21
ianw#status log switched Gerrit ACL's to submit-requirements.  You may see a status of "n/a" in, this should resolve as changes are updated and reindexed by Gerrit22:22
opendevstatusianw: finished logging22:22
ianwdo you think it's worth filing a bug?  i wonder if the reindex can be forced?22:23
fungiyou can still reindex the way we used to during upgrades, right?22:23
clarkbyes you should be able to22:24
clarkbtrigger an online reindex22:24
fungi`gerrit index start changes --force` according to my notes22:24
fungito refresh the changes index22:25
clarkbthats a gerrit ssh command (different than the offline gerrit way reindex command)22:25
ianwwould this reindex the right things for this though?  if that makes sense ... basically would it fix the problem22:25
clarkbI think the idea is it reindexes everything and should. But you're right we won't know without testing. I don't know if you can limit that command to a specific change or projet22:26
clarkbbut really I don't think it is a big deal22:26
clarkbthe reason I don't is that those values will never mean that gerrit upstream inteded for us due to zuul hitting the +2 and submit button22:26
clarkbthey show you that so you can go and hit submit manually on any chagnes of yours that are ready22:27
ianwyeah, i'm not even what to suggest about that22:28
ianweven if verified was some sort of "automated submit-requirement", you still need to add the +w to get things going22:30
ianwi guess really we want things to be marked as "ready for commit" when they just have a +2 code-review22:31
ianwbut even then it's really "ready to signal to zuul that gate testing can be done" which just doesn't seem to fit the model at all22:31
clarkbI think +2 code review and +1 verified are the rough equivalent22:32
clarkbbecause ya that tells reviewers "this is one step away from merging"22:32
ianwwell at least we can point to our live system now, although it does feel like it not affecting google means we might have to live with it or employ a full time java stack developer22:35
clarkbwe might be able to convince them to remove the collumn with a config option but ya actually changing behavior would likely require someone towrite the change(s)22:36
hasharI was about to ask why does not exist and I found the fixed link in this channel topic: no www! :D22:36
clarkbwe could add a redirect for it but meh22:37
hashardon't bother, Gerrit submit requirements is definitely more important :]22:37
clarkbhashar: its the last major piece in prep for our 3.7.x upgrad22:38
clarkbI think that will be the first time in like a decade we will be running the latest release of Gerrit22:39
clarkbthen 3.8 will release a month later and we'll be behind but that is ok22:39
hasharfor the reindexing, you can pass a change number and it will only index that one (`gerrit index changes 1234`)22:40
hasharand I think there is a command to do it on a per project basis22:40
hasharif you can find sometime to write a report of your 3.7 upgrade on upstream list, I am sure it will benefit others in the future22:41
opendevreviewMerged opendev/git-review master: Simplify test output strings for new Gerrit
clarkbhashar: thats good feedback. That said every upgrade since 3.2 has been straightforward. I think 3.7 prescirbes an offline reindex though so may be the most complicated one since22:42
clarkbhashar: we also test the upgrades with every change to our gerrit config management22:43
clarkbhashar: our CI builds the current version that we deploy to prod and the next version and we have a job that deploys the current version and then upgrades it to the next one22:43
clarkbthat has been useful to sanity check gerrit isn't adding things to our config file(s) and so on22:43
hasharI am lagging a few years behind though, I still copy paste a few commands but have some bits to automate those22:45
hasharthen , we still run on baremetal rather than container images22:45
clarkbhashar: we manually do upgrades in prod still though we could theoretically adapt what we do in testing to prod too22:45
clarkbbut it has helped us build a lot of confidence in the newer versions and how to get to them22:46
ianwi feel like we may be quite unique in keeping our ACL's separate and git-ops-ing them22:46
hasharand we have Puppet on top of that for config management which adds another layer of fun :]22:46
ianweven with the migration tool, it says "we will run this at google" in the CL ... implying that they just update the ACL's in gerrit but don't push them externally22:46
hasharthe ACL management is main complaint. We have locked them down and left them untouched for the most important/sensible repo (in short only admins can do anything about it)22:47
hasharbut for the rest, it is a bit of a mess :\22:47
ianwthat's kind of the reason why we have to do more work with submit-requirements, because we have to migrate our stuff in project-config22:47
clarkbthats a good point. Otherwise we'll be out of sync with our external view of the acls22:47
ianw(put in about the n/a thing.  if someone who knows says "yes, run XYZ" i'll propose a docs update)22:48
clarkbianw: speaking of 3.7.x is a totally not urgent update to our gerrit image builds to sync up with latest plugin versions22:51
ianwseeing as the s-r all-projects has gone well enough, i'm more confident to work my way through now22:53
ianwi remember going around "at the time".  in my mind "at the time" was about 2 years ago.  that is dated ... 200322:59
ianwi think i'm the dated thing in this story :)22:59
clarkbif we wanted to add it we would need to update our ssl certs but then we could add a simple redirect to apache23:00
ianwyeah, i'm not fussed.  i just remembered a time when it was a thing people were talking about23:01
clarkbwhen you manually provision certs from namecheap (what we did before LE) they automatically added www for you23:03
opendevreviewJeremy Stanley proposed opendev/git-review master: Add CC similarly to reviewers
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: WIP Correct boot path to cover FIPS usage cases
opendevreviewMerged openstack/project-config master: gerrit/acl : remove deprecated copy conditions
opendevreviewMerged openstack/project-config master: gerrit/acl : handle submit requirements in normalise tool
ianw^ hrm i guess that has to deploy behind which is running everything23:19
clarkboh sorry about that. But ya udpating the hosts inventory file in particular makes all the things go23:20
clarkbianw: that said I think manage-projects always pull project-config from master so the manage-projects job for 876892 may run it for yo?23:21
clarkbdefinitely worth checking after that run is done23:21
ianwyeah good point, i'll watch that closely23:21
ianw(no need for sorry; sorry i haven't got them running in parallel yet as well :)23:22
ianwi still think that's close but i've probably got a day of context switching it back in23:22

Generated by 2.17.3 by Marius Gedminas - find it at!