Monday, 2020-11-30

*** tosky has quit IRC00:39
*** DSpider has quit IRC01:05
*** ykarel|away has joined #opendev04:34
*** ykarel|away has quit IRC04:36
*** chandan_kumar has joined #opendev04:37
*** chandan_kumar is now known as chandankumar04:37
*** ysandeep is now known as ysandeep|brb05:24
*** brinzhang has joined #opendev05:59
*** marios has joined #opendev06:10
*** ysandeep|brb is now known as ysandeep06:34
*** sboyron has joined #opendev06:49
*** ralonsoh has joined #opendev07:06
*** whoami-rajat__ has joined #opendev07:21
*** slaweq has joined #opendev07:27
*** eolivare has joined #opendev07:37
*** hashar has joined #opendev07:48
prometheanfireis gertty working, getting 401s, reset my http password just in case, running git master gertty, same issue in py37 and 38
*** andrewbonney has joined #opendev08:05
*** tosky has joined #opendev08:37
*** mgoddard has joined #opendev08:54
*** DSpider has joined #opendev09:15
*** rpittau|afk is now known as rpittau09:27
*** fressi has joined #opendev10:07
*** mgoddard has quit IRC10:47
*** brinzhang_ has joined #opendev10:49
*** brinzhang has quit IRC10:53
*** cgoncalves has quit IRC11:18
*** cgoncalves has joined #opendev11:21
*** dtantsur|afk is now known as dtantsur11:31
*** mgoddard has joined #opendev11:55
*** mgoddard has quit IRC12:07
*** tosky has quit IRC12:24
*** tosky has joined #opendev12:25
*** hashar has quit IRC12:30
*** sshnaidm|off is now known as sshnaidm12:38
*** hashar has joined #opendev12:46
fungiprometheanfire: did you switch the server entry in your config from digest auth to basic auth?13:04
fungiprometheanfire: see item #3 in
*** d34dh0r53 has joined #opendev13:58
*** redrobot has joined #opendev13:59
fungiheads up everyone, pip 20.3 showed up today and turns on the new dep solver by default. keep an eye out for any unexpected changes in behavior13:59
*** fressi has quit IRC14:12
*** ysandeep is now known as ysandeep|brb14:24
*** ysandeep|brb is now known as ysandeep14:45
*** d34dh0r53 has quit IRC14:48
*** hashar has quit IRC14:51
fungiinfra-root: possible data integrity issue in gerrit reported in #openstack-infra just now, this is what it looks like from the server side when a push is failing:
*** d34dh0r53 has joined #opendev14:59
fungifirst occurrence of any "Missing tree" in the error_log was at 14:24:52 today and it looks like all of them are for that same 466cd024745bbeffebf3f8e598759fcc9b75df4d id15:00
fungidouble-checking those assertions now15:00
funginope, there were also two for b5ecb7fe4fa73b1cd1d6e87142ce359c5124be90 in openstack/neutron15:03
fungiso separate projects15:03
fungi`git show <id>` in the corresponding bare repos in ~gerrit2 returns tree contents for both of those15:07
fungialso the user was eventually able to push their changes, it looks like this may have been transient15:08
fungithe most recent missing tree error logged was at 14:57:0215:08
prometheanfirefungi: thanks, missed that15:16
*** gibi has joined #opendev15:16
openstackgerritMatthew Thode proposed ttygroup/gertty master: update examples for new opendev gerrit version
fungiprometheanfire: i think there might already be a duplicate of that already waiting for corvus to review when he returns15:21
fungiinfra-root: also seeing the gerrit load average climbing again like we observed early last week15:23
fungiit's over 60 at the moment15:24
fungiand again cpus are ~80% idle most of that time15:24
fungiwith the rest somewhat evenly split between user and system activity15:24
*** ysandeep is now known as ysandeep|away15:26
fungiseeing lots of change queries from (mostly openstack/cinder) third-party ci systems in the show-queue output again too15:29
openstackgerritSean McGinnis proposed openstack/project-config master: [check-release-approval] Handle PTL-less projects
fungigibi: at 15:34:22z we logged a missing tree 466cd024745bbeffebf3f8e598759fcc9b75df4d in openstack/nova15:37
gibiyes that was me again15:38
gibithen on a second try it worked15:38
fungithat's the same tree id it was complaining about earlier15:38
fungisystem load is scarily high on the server though so i'm wondering if it's timing out some internal read task and that's bubbling up as a git error15:39
gibicould be the case as I see that the push is slow even when it succeeds15:39
fungihaving it only hit nova and neutron so far (and the first more often than the second) isn't surprising if whatever action is failing has performance relative to repository size15:41
fungiwe've got a smallish backlog of send-email tasks in the queue, though that's probably just a symptom of overall resource starvation on the server15:43
fungithat backlog seems to be falling steadily over the past few minutes15:44
clarkbfungi: I wonder if that is related (note they are seeing it on 2.16)15:44
clarkbfungi: I think we shoudl look at melody again and show-queue?15:46
fungithat doesn't mention if they were using notedb, so hard to know if it's related15:46
clarkbfungi: ya15:46
clarkb"The projects cache just exploded, with 10% of the projects being out of the cache coverage."15:47
clarkbwas in one of luca's replies. I'm not sure how that was determined though or how we check if we are in the same situation15:47
clarkbfungi: `gerrit show-caches --show-jvm --show-threads` ? I think that is what luca was looking at to make that determiniation15:49
fungitakes a while to return15:50
clarkbprojects                      |  1024               |   7.9ms | 90%     |15:50
clarkbwas from the upstream thread so ya I think that was the source of the info15:50
fungiprojects                      |  1024               |   9.9ms | 99%     |15:51
fungiis what it gave me15:51
clarkbfungi: any other caches look like low hit rates though?15:51
fungichangeid_project              |  1024               |         | 24%     |15:51
fungithat's probably our culprit right there15:51
clarkbcache.changeid_project.memoryLimit is the config value to set to change that15:52
clarkb1024 at 24% would mean that ~4096 may be enough? but maybe we do 8192 to be cautious?15:53
fungiseems reasonable15:53
clarkbfungi: also note that 1024 is the default limit for many of these cachces15:53
clarkbso if others are showing 1024 entries maybe we should double them?15:54
fungilots of them are:15:54
fungigroups_bysubgroup, groups_byuuid(2400), permission_sort, projects15:55
*** brinzhang0 has joined #opendev15:55
fungithose are the exclusively-memory-resident caches at 1024 entries or greater15:55
clarkbgroups_byuuid is unlimited apparently15:55
fungithere are also disk-backed caches with a lot more entries15:55
fungithe diff cache for example has a 65% cache hit rate, but it's disk-based and there may be little we can do to improve that15:56
fungialso it will likely improve once the cache warms more15:56
clarkbya I imagine that one will warm as it goes.15:57
*** brinzhang_ has quit IRC15:58
fungiokay, so we want to try upping to changeid_project=8192 groups_bysubgroup=2048 permission_sort=2048 projects=204815:58
clarkbbased on that thread I linked luca seems to think poor cache hit rates are a bad thing (not surprising) so maybe we start with setting cache.changeid_project.memoryLimit to 8196 and cache.permission_sort.memoryLimit to 2048 and cache.projects.memoryLimit to 2048 ? The groups_* I think may already have high limits based on docs15:58
clarkbfungi: ya that sounds right (though I'm not sure we need to go _bysubgroup15:59
clarkbunless you see it stuck at 1024 implying it isn't one of the unlimited caches15:59
clarkbin that case lets bump it to 2048 too15:59
fungigroups_bysubgroup             |  1024               |   4.6ms | 99%     |15:59
clarkbya lets set that to 2048 then15:59
fungiare you writing that change or shall i?15:59
clarkbfungi: I've not yet loaded my ssh keys16:00
fungion it16:00
clarkbif you want to start on it whiel I do that, that woudl be great16:00
*** mlavalle has joined #opendev16:00
gibifungi: now got an internal error when pushed a longer commit chain16:08
*** d34dh0r53 has quit IRC16:09
openstackgerritJeremy Stanley proposed opendev/system-config master: Increase some of Gerrit's in-memory cache pools
fungigibi: thanks! we're hoping that ^ improves things16:10
clarkbfungi: do you want to manually apply that and restart? or shoudl I approve it and we wait for it to apply through normal process?16:11
clarkbfungi: re 8196 I meant to type 8192 :) but an off by 4 number shouldn't matter much here :)16:12
openstackgerritJeremy Stanley proposed opendev/system-config master: Increase some of Gerrit's in-memory cache pools
fungioops! that's what i get for copying and pasting from your numbers instead of mine ;)16:13
fungifixed in ps216:13
fungii hadn't even noticed16:13
clarkbfungi: I've got ssh keys loaded if you want to manually apply that and restart fwiw16:14
fungiclarkb: and yeah, we may as well apply it manually if you think it looks right. we've got users experiencing extremely sub-par performance and errors (i saw it take nearly a minute to push the first patchset for that change)16:14
clarkbfungi: yes that change lgtm16:15
fungistatus notice The Gerrit service on is being restarted quickly to troubleshoot high load and poor query caching performance, downtime should be less than 5 minutes16:15
clarkbthat lgtm too16:15
fungithat work for a notice to users?16:15
fungicool, jumping in the root screen session now16:15
clarkbI'm there too if you want to drive I'll follow along16:16
*** d34dh0r53 has joined #opendev16:16
fungiscreen -x 12385116:16
fungiyeah. i'm in16:16
clarkbput a #hi in the window I see to make sure we're looking at the same buffer16:17
*** eolivare has quit IRC16:17
fungiyeah, i saw it ;)16:17
*** brinzhang0 has quit IRC16:18
clarkbfungi: that looks correct to me16:19
fungithanks, writing16:19
*** brinzhang0 has joined #opendev16:19
fungi#status notice The Gerrit service on is being restarted quickly to troubleshoot high load and poor query caching performance, downtime should be less than 5 minutes16:19
openstackstatusfungi: sending notice16:19
-openstackstatus- NOTICE: The Gerrit service on is being restarted quickly to troubleshoot high load and poor query caching performance, downtime should be less than 5 minutes16:19
clarkbfungi: I say go for it whenever you are ready16:20
fungiokay, ready for me to down the container?16:20
fungicool, doing it now16:20
*** osmanlicilegi has joined #opendev16:20
fungiand it's on its way back up now16:20
openstackstatusfungi: finished sending notice16:22
clarkbfungi: `WARN : Logging context is not empty` <- is it made about not being able to rotate the replication log or something?16:23
fungicould be16:24
clarkbseems like it is logging to the replication log just fine though16:26
fungisystem load average is fairly low at the moment16:26
clarkbI'm going to run the show caches command to see what that looks like out of curiousity16:26
fungiyeah, i just started that as well16:27
fungicaches are still warming, other than projects which has already topped out16:27
fungii guess something's querying all our projects regularly16:27
fungii guess we could increase that even more. presumably it will never have more entries than the total number of repos in the system?16:28
clarkbfungi: "Caches the project description records, from the refs/meta/config branch of each project."16:29
clarkbI think the answer to that question is yes16:29
clarkbnot that changeid_project doesn't have a documentation entry describing its purpose16:29
clarkbfungi: looks like load is climing16:32
clarkbbut maybe that is due to needing to rewarm the caches?16:33
clarkbanother thought I had was that maybe we should look at deploying java 11 just to get up to speed there and rule out gc performance as a potential cause16:33
fungimakes sense16:35
fungialso load isn't that high right now comparatively16:36
fungiit's been hovering around 8 when i've checked after the restart16:36
clarkbya it is rising and falling16:36
clarkbdefinitely better than before16:36
fungimmm... back up to 35 now16:40
fungichangeid_project cache has only 246 entries, cache hit rate for it is 72% at the moment16:41
clarkbfungi: its starting to fall back down again now fiw16:42
clarkbit seems bursty16:42
*** hashar has joined #opendev17:00
fungichangeid_project              |  2183               |         | 38%     |17:01
openstackgerritMerged zuul/zuul-jobs master: Fix typo with container_images siblings logic
fungigranted that's a better hit rate than we had before the restart, and that pool is only ~ a quarter full so far17:01
fungibut we may still want to consider bumping it up even more17:02
clarkbya and I think a lot of that population is happening in response to actions in the system17:02
clarkbbasically that cache is cold right now and things people aer doing are populating it17:02
clarkbProbably too early totell if we'll plateau around the earlier prediced ~4k17:02
clarkbfungi: maybe lets wait and see if we go above 4k? and if so then ya bump it up further17:03
fungialso groups_byuuid has reached 2048 now17:03
clarkbI expect that 38% should climb as more things use what was recently added to the cache17:03
clarkbfungi: interestinly that one should be 32k I think17:03
fungicould just be coincidental17:04
clarkbfungi: I wonder if by setting a bound on bysubgroup < 32k we've set a lower bound for that one too17:04
clarkbsomething to watch, its hit rate is still good17:04
*** marios is now known as marios|out17:11
fungiborg is running now too and eating an entire cpu, so that's contributing some of the load17:13
fungithe last "Missing tree" exception in gerrit's error_log was 16:39:44z, so we're still seeing it after the restart17:22
*** hamalq has joined #opendev17:23
*** mgoddard has joined #opendev17:30
clarkbfungi: that was not long after the restart though, maybe we still had cold caches?17:32
*** venkatakrishnath has joined #opendev17:32
clarkbfungi: that one was associated to what gibi was doing17:34
clarkbgibi: ^ have you had more recent issues or is it happier now?17:34
venkatakrishnathFacing the issue 'Create' rights to create new references while submitting code to openstack-cinder through Gerrit. More details
venkatakrishnathCan some one help17:35
*** mgoddard has quit IRC17:35
clarkbvenkatakrishnath: line 11 is the hint you need. git-review needs to be upgraded17:36
venkatakrishnathUpgraded to 1.28.017:36
clarkbah the next issue is you can't push drafts17:36
clarkbwe haven't allowed drafts beacuse they create confusion and problems that people don't expect. Just push it as a regular change. You can mark it WIP in the web ui from there if you like17:37
venkatakrishnathI have cleared all drafts on my username and here is the latest error message
clarkbvenkatakrishnath: you are pushing to refs/drafts/ (see line 28 of the latest paste) this is not allowed17:39
clarkbthis is because you are using git-review -D17:39
clarkbstop using -D17:39
clarkbfungi: gibi fwiw I was able to push a change to nova without getting that error17:41
venkatakrishnathI used to check with Draft stuff to confirm git-review command is working.  working now without -D17:41
clarkbvenkatakrishnath: we've not enabled that functioanlity in years17:42
*** rpittau is now known as rpittau|afk17:42
clarkbit was just too confusing for CI and people thought it was private when it wasn't so we disabled it17:42
*** hashar is now known as hasharAway17:44
fungivenkatakrishnath: clarkb: yes, git review -D hasn't worked with our gerrit since merged almost 5 years ago17:45
venkatakrishnathclarkb: I have been submitting code since this August and following that. Thanks for the information17:45
openstackgerritJeremy Stanley proposed opendev/system-config master: Disable Gerrit's automatic Git GC on push
fungiclarkb: ^17:54
clarkbfungi: my only nit is the whitespace difference :P17:55
clarkbfungi: do you want to change that?17:55
fungioh, sure thing17:56
openstackgerritJeremy Stanley proposed opendev/system-config master: Disable Gerrit's automatic Git GC on push
*** andrewbonney has quit IRC17:59
clarkbfungi: thats weird ps2 shows the lines above as not having tabs anymroe but there is no diff there. I wonder if that is a rendering bug18:00
fungicould be18:00
clarkb+2 from me anyway18:00
fungialso that file has a mix of spaces and tabs already18:00
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Zuul Cache role
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Zuul Cache role
*** venkatakrishnath has quit IRC18:07
*** mlavalle has quit IRC18:11
fungichangeid_project              |  4571               |         | 43%18:12
fungiwe're over 4096 entries now18:12
fungiload average has fallen back to reasonable levels18:12
*** dtantsur is now known as dtantsur|afk18:13
fungithat cache continues to grow and the hit rate continues to fall... i'm starting to wonder if it's effectively cacheable18:22
*** marios|out has quit IRC18:24
melwittoof, gonna take awhile to get used to this new gerrit ui18:24
melwittmine eyes @_@18:24
fungitried the dark theme under user prefs?18:24
melwittnot yet18:25
fungidoesn't blast my retinas with quite as much radiation at least18:25
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Zuul Cache role
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Zuul Cache role
*** mgoddard has joined #opendev19:19
fungichangeid_project              |  7450               |         | 40%19:26
fungii have a feeling we're going to top out our new 8192 limit and not see a >50% cache hit ratio19:26
clarkbfungi: ya we should probably consider another restart to apply the receive auto gc disable and bump up that cache value further?19:27
fungiso increase changeid_project to... 16384? and maybe bump the projects cache to 4096?19:29
clarkb16k or even 32k maybe? We appear to have plenty of memroy according to show-caches19:29
clarkbbut ya that and the receive autogc disable?19:30
clarkbcorvus: ^ not sure if you want to review any of that19:30
fungii'll update the cache change with those values now19:30
*** mgoddard has quit IRC19:33
openstackgerritJeremy Stanley proposed opendev/system-config master: Increase some of Gerrit's in-memory cache pools
openstackgerritJeremy Stanley proposed opendev/system-config master: Disable Gerrit's automatic Git GC on push
fungiclarkb: corvus: ^19:34
clarkbfungi: lgtm19:35
fungichangeid_project              |  7900               |         | 41%19:38
fungiso it's just about topped out19:38
clarkbfungi: I'm about to grab lunch. Do you want to do those updates and restart nowish or wait for later?19:43
*** tosky has quit IRC19:45
fungii would maybe give it a little longer so we can observe behavior with warmed caches, and also because frequent restarts aren't helping users get work done. i think the urgency is somewhat lessened as the system load graph is looking a lot more sane and i haven't seen more "missing tree" exceptions since 16:39:44z19:46
clarkbok I'll grab lunch then19:46
fungimaybe in a few hours once activity levels are trailing off again, a restart would be slightly less disruptive too19:47
clarkbfungi: I suppose its possible that we'll evict from the cache and still be reasonably happy at the current level19:49
clarkbwhereas before maybe the level was too low/too much eviction19:50
fungichangeid_project              |  8192               |         | 44%20:24
fungiat least the hit rate has climbed a bit20:24
fungiand that's nearly double the hit rate we saw at ~steady state with a smaller cache20:25
fungiload has been nominal for over two hours now as well20:26
fungiwell under the cpu count20:26
*** ralonsoh has quit IRC20:36
fungiand back down to 41% cache hit ratio now20:46
clarkbbut load seems to be fairly stable? I wonder if that cache size is large enough that evictions aren't super costly and we udpate as we need to?20:47
clarkbstill would've expected a much better hit rate given teh previous data20:47
fungiyeah, load average is still reasonable20:47
*** tosky has joined #opendev20:48
fungiit's been running somewhere between 2 and 15 that i've seen20:48
fungi5-minute average just dipped below 1 moments ago20:49
fungiwatched it dip as low as 0.72 for a bit20:50
fungibut it's back up around 4 again now20:51
*** sboyron has quit IRC20:54
*** sboyron has joined #opendev20:56
*** whoami-rajat__ has quit IRC21:08
*** DSpider has quit IRC21:46
openstackgerritMatthew Treinish proposed opendev/subunit2sql master: Fix compatibility with latest oslo.config
fungiclarkb: seems the hit ratio for changeid_project continues to hover in the 40-44% range for the past few hours. also we haven't maxxed out any of the other newer cache memory limits besides project, so i've staged those increases and the autogc false in the root screen session we've been using if you want to double-check it22:30
clarkbfungi: the grep you've got there looks good to me22:30
fungistatus notice The Gerrit service on is being restarted quickly to make further query caching and Git garbage collection adjustments, downtime should be less than 5 minutes22:32
fungithat work for the restart notice?22:32
clarkbyes that lgtm too22:32
fungii can restart whenever folks are ready, zuul seems relatively idle22:33
clarkbI'm deep into writing emails right now but dno't wait on me22:35
fungi#status notice The Gerrit service on is being restarted quickly to make further query caching and Git garbage collection adjustments, downtime should be less than 5 minutes22:35
openstackstatusfungi: sending notice22:35
fungii'll give that a minute and then down/up -d the container22:35
-openstackstatus- NOTICE: The Gerrit service on is being restarted quickly to make further query caching and Git garbage collection adjustments, downtime should be less than 5 minutes22:35
fungiand it's down22:36
fungiand it's starting again22:36
fungisorry TheJulia, hopefully it'll perform better with this configuration update22:37
TheJuliano worries22:37
fungiwe're fighting internal gerrit queue backlogs and cache tuning needs for the new version now that it's being put under additional load as lots of people are returning from a week off22:38
openstackstatusfungi: finished sending notice22:38
TheJuliac'est la vie22:38
TheJuliaand it is back22:39
fungiit's one of those look at the full caches and hit ratios, guess how big they need to be, reconfigure and restart, let things settle for a while, look at the caches again, realize you guessed poorly, lather, rinse, repeat22:39
*** slaweq has quit IRC22:40
TheJuliafun :(22:40
TheJuliaand then GC fun too22:40
fungiTheJulia: yeah, this is a classic quote from the gerrit docs: "By default, git-receive-pack will run auto gc after receiving data from git-push and updating refs. You can stop it by setting this variable to false. This is recommended in gerrit to avoid the additional load this creates." or in other words "this option defaults to true, but we recommend you set it to false"22:42
TheJuliaheh, that sounds familiar22:42
fungiit goes on to suggest a periodic repack instead, which we already do, so it was somewhat redundant anyway22:43
*** slaweq has joined #opendev22:45
mordred"the default behavior of git pull is to to pull and merge, which you probably don't want, so instead please remember to always type --ff-only when you type pull"22:49
*** slaweq has quit IRC22:50
clarkbthey at least made that the default in new gerrit now iirc22:50
fungijokes on them, i always just `git remote update && reset --hard origin/master` instead22:50
mordredfungi: me too!22:50
mordredfungi: I use reset --hard instead of checkout most of the time :)22:51
TheJuliahard resets ftw22:51
mordredwho needs named branches when you have git reflog?22:51
fungioh, i do also sometimes `git checkout -B mytopicbranch --no-track origin/master`22:52
TheJuliawhich reminds me, I need to go dig through the my reflog on one of my repos after I finish my current change22:52 fill out your weekly progress report? ;)22:52
fungisounds like something i'd do22:52
TheJuliano, could hvae sworn I pushed up a decently sized change22:53
TheJuliawell, one with words that was more than a quick edit22:53
TheJuliaand I don't know what happened ot it22:53
fungitried a message:"words you thought you used" search in gerrit?22:54
TheJuliano, I haven't22:54
TheJuliaI know the repo, so I guess I just never pushed it up22:54
TheJuliaor gerrit ate it which is something I don't want to think about22:55
TheJuliaso I'll go with "I forgot to run git review"22:55
fungialso occasionally i can't find my changes because someone actually approved them. and then i get happily surprised22:55
mordredmaybe you pushed it up to a honeypot gerrit by mistake22:55
mordredfungi: ++22:55
TheJuliahuh, that is a good point, I thoguht I checked all changes already22:55
fungialso (thankfully less often) because someone abandoned them22:56
fungiTheJulia: also if you know what file you meant to change you can search for project:some/project file:path/to/thatfile22:56
mordredwhat I love is when I write a big patch, push it up, then can't find it, then rewrite it and push it up - then find the old patch because the two are in conflict and show up in the gerrit conflicts list22:57
fungioh, yep, i've definitely had that happen. or more often i forget i pushed a fix and then push the same fix again, with varying degrees of agreement vs the original22:57
TheJuliaoh yeah, it is a specs repo, so not much happens there anyway22:57
*** hasharAway has quit IRC22:58
TheJuliaI just need to finish reorganizing my desk since I started this past weekend and never finished22:59
* TheJulia grumbles about needing landscape screens now22:59
mordredTheJulia: my first reaction to landscape screens was assuming you were talking about gardening23:00
fungithat sounds like a very healthy first reaction23:00
TheJuliaIt does23:00
clarkbI'm about to send out the infra meeting agenda for tomorrow. Anything else to add to it?23:01
TheJulianot the "my screen is now too narrow to all fit and make optical sense of what I'm looking at"23:01
fungii have two 2560x1440 screens in portrait orientation and just keep a browser in one, so it gets 1440px width to do what it needs, and i've seen that be generally happy23:02
fungii was previously doing that with 1920x1080 screens but a lot of stuff that's happy with 1440px width isn't so good at 1080px width23:03
TheJuliayeah, these are 1080 so sadness23:05
TheJuliaI've got new arms on the way so I can change up the mounting. It should work out a lot better23:05
clarkbagenda sent as I didn't hear any need for updates23:06
TheJuliaclarkb: out of curiosity, any resolution to ui glitching with browser graphics acceleration?23:07
clarkbTheJulia: no, sorry I don't know that we have filed a bug yet.23:07
clarkbit does sound like Google has no intention of enabling hardware acceleration on linux by default though23:08
clarkbI half expect the response will be "the browser does't support this so we can't"23:08
*** tkajinam has quit IRC23:10
TheJuliaThat is... extreme sadness23:11
TheJuliasince I can actually feel the impact on my browser and hear my CPU fan having to throttle more23:11
clarkbI don't know if it is possible with chrome but firefox allows you to toggle that per user session I think23:11
*** tkajinam has joined #opendev23:11
clarkbmaybe you can have the gerrit window without acceleration and the everything else window with it23:12
fungiwas anyone able to determine if it was similar to the chromium crashes for webrtc?23:12
clarkbI didn't23:12
fungijust wondering if the next release will quietly fix it again23:12
clarkbI actually don't have chromium just chrome and firefox. I use firefox for 99% of my browsing then chrome for when that doesn't work (flash/nonfree codecs/etc)23:15
clarkbfungi: should we approve your changes to reflect the config we're running in prod? I don't know that corvus or ianw will end up reviewing them today?23:16
fungiyeah, probably a good idea just to keep configuration management in sync with configuration reality23:18
clarkbfungi: did you want to push the +A buttno then?23:19
fungioh, yup, can do23:20
fungiclarkb: this week is probably a good time for 763473 as well?23:24
clarkbfungi: I think we took care of that when we removed the extra images we were building23:25
clarkbdouble check against the current system-cofnig state23:25
fungioh, good call, i forgot we merged that already23:25
clarkbwe should only be building 3.2 now off the regular 3.2 branch23:25
clarkbsorry I should've called out your change when that was happening but we had a few things in the fire at the time23:25
fungiyeah, no sweat, just abandoned it with a mention of the change which addressed it23:28
TheJuliaquick question, w/r/t new gerrit. Any bulk way to nuke draft comments?23:58

Generated by 2.17.2 by Marius Gedminas - find it at!