Wednesday, 2021-02-17

diablo_rojofungi, I did them today00:01
fungidiablo_rojo: perfect, should be all set then00:06
fungisee gouthamr's comments above as well00:06
diablo_rojofungi, cool! I will get that email sent out then.00:06
diablo_rojoYeah I saw the comment! Still processing what that means, looking at the patch now.00:06
fungijust that meetings needed to be updated for correct channels00:14
diablo_rojoReviewed them both now.00:31
*** zbr has quit IRC00:33
*** zbr has joined #opendev00:45
*** dviroel has quit IRC00:59
clarkbthe gatling-git output is really confusing, it seems some small numebr of changes were successfully pushed. Another set failed due to a lack of change ids. and then the bulk failed due to no common ancestry01:03
clarkbI wonder if this is just a problem with races or something01:03
clarkbweird to have the no change id errors in a small subset because you'd expect those to consistently be generated or not be consistently generated01:03
clarkbmaybe tomorrow I'll ask on the gerrit slack channel01:04
openstackgerritClark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit
*** brinzhang has joined #opendev01:10
*** zbr7 has joined #opendev02:23
*** zbr has quit IRC02:23
*** zbr7 is now known as zbr02:23
*** hemanth_n has joined #opendev02:58
openstackgerritIan Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container
*** hamalq has quit IRC04:41
*** diablo_rojo has quit IRC05:35
*** marios has joined #opendev06:00
*** whoami-rajat__ has joined #opendev06:12
*** zbr has quit IRC06:13
*** zbr has joined #opendev06:14
*** mnaser has quit IRC06:32
*** mnaser has joined #opendev06:34
*** ralonsoh has joined #opendev06:41
*** slaweq has joined #opendev07:07
openstackgerritIan Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container
*** sboyron has joined #opendev07:56
*** rpittau|afk is now known as rpittau08:12
*** andrewbonney has joined #opendev08:13
*** eolivare has joined #opendev08:21
*** DSpider has joined #opendev08:43
*** ysandeep|away is now known as ysandeep|ruck08:50
*** cloudnull has quit IRC08:59
*** jpena|off is now known as jpena08:59
*** cloudnull has joined #opendev09:02
*** tosky has joined #opendev09:14
*** tosky_ has joined #opendev09:20
*** tosky has quit IRC09:24
*** tosky_ is now known as tosky09:25
*** cloudnull has quit IRC09:37
*** cloudnull has joined #opendev09:40
*** dtantsur|afk is now known as dtantsur10:01
*** zoharm has joined #opendev10:11
openstackgerritIan Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container
*** rferraz has joined #opendev10:19
*** prometheanfire has quit IRC10:42
kopecmartinhi, does have an api endpoint supporting f.e. file iteration from a python code? similar to github where i can query a repo for files getting json data -
kopecmartinnever mind, found it
*** prometheanfire has joined #opendev11:02
openstackgerritMerged opendev/irc-meetings master: Fix meeting channel for diversity wg
*** sshnaidm has quit IRC11:15
*** sshnaidm has joined #opendev11:18
*** dviroel has joined #opendev11:37
*** klonn has joined #opendev11:56
*** eolivare has quit IRC12:09
*** fdegir5 is now known as fdegir12:24
*** jpena is now known as jpena|lunch12:30
*** Dmitrii-Sh has quit IRC12:32
*** DSpider has quit IRC12:45
*** DSpider has joined #opendev12:57
*** klonn has quit IRC12:59
*** eolivare has joined #opendev13:19
*** jpena|lunch is now known as jpena13:23
mnasiadkaHi, is there any opendev Ansible role I can use, to get chrony/ntp configured?13:25
openstackgerritMerged opendev/irc-meetings master: Fix the openstacksdk meeting schedule
*** hemanth_n has quit IRC13:56
*** ysandeep|ruck is now known as ysandeep|mtg14:01
*** fressi has quit IRC14:07
*** fressi has joined #opendev14:09
*** lpetrut has joined #opendev14:20
fungimnasiadka: i think we already configure ntpd in the images we boot. are you seeing logs with incorrect timestamps in builds?14:50
mnasiadkafungi: Nah, trying to deploy ceph using cephadm - and it's complaining about lack of chrony on the images - because it's checking a systemd unit, so if ntp is configured I'll just make it quiet :)14:52
*** jhesketh has quit IRC14:52
fungiahh, has ntpd lost its sheen, that they don't check to see if it might be in use instead of chrony?14:54
*** brinzhang has quit IRC14:55
*** brinzhang has joined #opendev14:55
mnasiadkafungi: well, ask Ceph guys - can you point me to playbooks that Zuul uses to configure ntp?14:55
fungizuul doesn't configure ntp, we install it in the images nodepool boots, it'll be in nodepool elements, not playbooks14:56
fungii'l go looking14:56
mnasiadkaAh, ok14:56
*** brinzhang has quit IRC14:57
fungithis tells nodepool to install the "ntp" package in all images it builds:
*** brinzhang has joined #opendev14:58
fungiand we also install ntpdate (the line just after it)14:58
*** slaweq has quit IRC15:00
fungimnasiadka: looks like we do expect a chronyd service on centos:
*** fressi has quit IRC15:00
fungiand some variation on ntp in other distros15:00
fungiwhat image are you testing on?15:01
mnasiadkacentos8, so will dig a bit more why cephadm complains on missing chronyd systemd unit15:01
*** slaweq has joined #opendev15:04
fungii suppose timesyncd is too new to be in centos 815:05
fungimnasiadka: here's a current image build log for centos-8:
fungishows we're definitely running `systemctl enable chronyd` during image creation15:07
fungii'm stepping out to run some errands but should be back by ~16:00 utc15:10
*** ysandeep|mtg is now known as ysandeep|ruck15:44
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift and upload-logs-gcs
*** ysandeep|ruck is now known as ysandeep|dinner16:08
*** lpetrut has quit IRC16:25
clarkbfungi: do you know how to find new instances of the growroot issue in kna1? Want to check if we caught any and wonder if you've got a quick way to find those16:25
fungigrenade jobs failing with errors about apt-get not having enough space to download packages was the initial sighting16:25
fungii'll see if i can dig up the exact message16:26
fungithe up side there is the jobs get far enough to have their logs indexed in logstash16:26
*** hashar has joined #opendev16:29
clarkbfungi: the jobs do fail though right? I wonder if my luck might be good enough just looking at failed jobs in kna1 in the last 12 hours16:34
fungiyes, it's a failure, happens during the run phase16:34
clarkbcool let me see if I can follow that breadcrumb16:35
*** hashar has quit IRC16:35
fungilook for failed grenade jobs in particular16:35
*** hashar has joined #opendev16:38
clarkbdoesn't look like there were any grenade failures there in the last 12 hours but there are 19 other failures I can spot check16:45
clarkbone failure was due to connectivity issues to the mirror just over an hour ago, I can reach the mirror now at least16:46
clarkbtwo other failures were for unrelated issues (one was an actual python test failure, another an ansible var being set wrong and devstack being unhappy about it)16:47
fungiE: You don't have enough free space in /var/cache/apt/archives/.16:49
fungithat's the message to search for16:49
fungiwas turning up semi-regularly in grenade failures16:49
fungiand only in airship-kna116:49
clarkbthere was another mirror contact failure about 4.5 hours ago too16:50
fungialso only on ubuntu-bionic, but i expect that's because master branch grenade jobs use bionic still16:50
* clarkb looks for lack of freespace message16:50
clarkbfungi: the most recent occurence of that message was about 23 hours ago so not recent enough for the growroot log collection16:51
clarkbI'll continue to look at these 19 failurse and see if any are helpful16:51
*** slaweq has quit IRC16:53
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Upgrade ansible-lint to 5.0
clarkbrm_work johnsom digging through failures to find a specific failure type so that I can debug it and I noticed neutron-lbaas-to-octavia-migration seems to fail consistently because it tries to install latest dib on python217:05
clarkbis that a job we should just stop running or perhaps update? Thought I would mention it as I don't think it has passed in a long time17:05
*** mlavalle has joined #opendev17:05
clarkbit is running in the periodic pipeline17:06
johnsomI think cgoncalves is working on the DIB-being-branchless issues with python2 and the stable branches17:07
johnsomfor example17:07
cgoncalvespython 2 and 3.517:08
johnsomIt's been a bit of a chain of things being broken with DIB and stable branches17:09
clarkbthere is a fix for dib for that iirc17:09
clarkbI thought it got released17:09
clarkbthe job failure I noticed is for rocky though not stein or train17:09
johnsomRocky should be EOL17:09
clarkbits still running periodic jobs :)17:10
johnsomrm_work Weren't you working on the EOL for rocky, etc.?17:10
cgoncalvesrocky octavia EOL'd months ago17:11
johnsomI'm just wondering if neutron-lbaas got missed, or the EOL job was not finished.17:11
fungiwe haven't deleted the branches yet17:12
fungiif that plays into it at all17:12
clarkbfungi: that job name doesn't show up in codesearch so it must live on a stable branch zuul config17:12
rm_workyeah I was focused on Octavia17:12
fungimaybe people are still proposing changes against those branches17:12
clarkbseems likely that is the source of the issue17:12
rm_workI thought neutron-lbaas already got EOL'd, but maybe not17:12
fungiwell, they could just be tagged eol and not have been deleted yet17:13
clarkbfungi: I've yet to find a growroot that looks unhappy. I think I'll leave it for a bit and hceck again later to see if more recent runs have caught one17:13
* fungi finds the list17:13
fungi is the set of deletions requested so far, no lbaas in there (just stable/ocata for octavia)17:14
*** jpena is now known as jpena|brb17:14
johnsomrm_work I thought you submitted for EOL up to at least Rocky across the board.17:15
rm_workmaybe, looking (i'm really bad at historical gerrit searching)17:16
fungiit's possible things have been added since that list was generated several weeks ago17:16
johnsomI see that Octavia has EOL tags up to Stein17:16
*** dtantsur is now known as dtantsur|afk17:17
fungiaha, the ml thread that list was attached to was specifically requesting stable/ocata branch deletions17:17
* johnsom disappears back under a pile of other work now that the correct folks are engaged17:18
*** klonn has joined #opendev17:21
clarkbI'm wary of reading into the dstat data during gatling runs against gerrit 3.2 and 3.3 too much because this is all running in cloud and we can be our own noisy neighbors, etc. Due to the reliance on the buildset registry both jobs ran in the same cloud region at near the same time though so the comparison is probably more valid that if they ran on random clouds.17:21
clarkbTwo things I notice are that 3.2 has higher system load and lower memory use17:22
clarkb3.3 uses a bit more memory but system load is better17:22
clarkbthe system isn't push super hard though, which might be the next thing to look at with gatling after figuring out the push errors17:22
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Upgrade ansible-lint to 5.0
*** eolivare has quit IRC17:28
*** ysandeep|dinner is now known as ysandeep|ruck17:29
*** marios is now known as marios|out17:35
*** jpena|brb is now known as jpena17:36
clarkbI think the reason my pushes are failing is that this framework doesn't clone the specified repo first and then update from HEAD. It just edits some local commits and then pushes straight to the repo. I think that means if I create a second empty repo just for gatling it should be happier /me tries this17:40
*** rpittau is now known as rpittau|afk17:41
openstackgerritClark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit
*** cloudnull has quit IRC17:49
*** cloudnull has joined #opendev17:49
*** ralonsoh has quit IRC17:54
*** klonn has quit IRC18:08
*** rferraz has quit IRC18:08
*** ysandeep|ruck is now known as ysandeep|away18:10
*** jpena is now known as jpena|off18:12
*** auristor has quit IRC18:21
*** slaweq has joined #opendev18:23
*** klonn has joined #opendev18:25
*** andrewbonney has quit IRC18:33
clarkbI feel like this growroot thing is going to be a heisenbug18:33
clarkbas soon as we start actively checking logs to determine what is going on the problem disappears18:33
*** auristor has joined #opendev18:36
*** klonn has quit IRC18:42
*** klonn has joined #opendev18:43
*** marios|out has quit IRC18:48
clarkbhrm now gatling complains about change ids missing and the remote not advertising ref master19:10
clarkbI guess that is progress19:11
*** slaweq has quit IRC19:20
openstackgerritClark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit
clarkbif ^ works then I guess I'll file bugs upstream19:31
clarkbbut thats a massive hack to force change id generation19:31
kopecmartinianw: hi, I fixed issues I found in refstack - , I don't know what to do with the url/endpoint ('api' is missing)19:34
*** DSpider has quit IRC19:50
clarkbkopecmartin: for the api url change maybe it is something apache proxying is modifying?19:50
clarkbkopecmartin: also left a note about hwo to work around DNS not being updated yet19:51
clarkbleft a couple of thoughts about the api url thing too (after grepping in config mgmt19:54
*** DSpider has joined #opendev19:58
*** DSpider has quit IRC19:59
*** klonn has quit IRC20:03
*** elod has quit IRC20:17
*** elod has joined #opendev20:19
*** hashar has quit IRC20:23
clarkboh hrm I see, the errors about no ref being advertised is due to me trying to pull the empty repo's master branch20:27
openstackgerritClark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit
*** slaweq has joined #opendev20:46
*** LowKey has joined #opendev21:08
openstackgerritIan Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container
clarkbianw: note that I think the current server's memory constraints will make ^ difficult21:23
ianwkopecmartin / clarkb: yeah, i have no idea really if was correct, it would be worth understanding the configs better probably21:24
ianwclarkb: yeah, i just wanted to flesh out the idea initially.  i pulled stats on the usage ( and the db barely manages to trip 1 reference per second now21:25
clarkbit will be related to how quickly people review files in the web ui, but ya that seems about right. I'm more worried about memory needed to keep tables in memory21:26
ianwclarkb: yeah, i agree.  the other thing is we have stuff like lxc running, which is not started on focal by default but takes up about the same space as mysql (~800mb)21:27
clarkbianw: oh interesting, is that something we should go ahead and turn off now ?21:28
ianwprobably, i was going to investigate but was very hesitant to touch anything on production :)21:28
*** cloudnull has quit IRC21:28
clarkbit didn't even occur to me that there could be unneeded system level things there to look for21:28
ianwroot       1110      1  0  2018 ?        00:19:16 /usr/bin/lxcfs /var/lib/lxcfs/21:29
ianwroot@review01:~# pmap 111021:29
ianw total           760408K21:29
ianwso i definitely buy the contention argument.  a bit like the http proxy on gitea to filter useragents, i figure at minimum it might be something good to have in our pocket anyway21:30
clarkbya, and may also simplify testing?21:31
ianwyeah, and it was inspired by the practicalities of us doing a pretty bad job at keeping the trove instance updated21:32
ianwwhich has bitten us now with the backups issue, that luckily we noticed but isn't going to fix21:32
openstackLaunchpad bug 1914695 in mysql-5.7 (Ubuntu) "mysqldump --all-databases not dumping any databases with 5.7.33" [Undecided,Fix released]21:32
ianw(well unless we do it)21:33
*** cloudnull has joined #opendev21:33
ianwanyway, it doesn't work yet :)  you can be sure i'll have a far too long changelog when it does and we can discuss :)21:34
clarkbsort of related is my gatling work. I can't get pushes to work at all due to missing change ids (I even hacked the code to try and force that flag on :/) the interesting thing so far is it generates load and we can look at dstat metrics from there21:34
clarkband since the buildset registry forces all the related jobs to run in the same cloud region we get what should be somewhat comparable numbers though I've not gaterhed enough data to check that yet21:35
clarkbwith a dataset of one gerrit 3.3 uses more memory but hsa lower system loda compared to 3.221:35
*** Dmitrii-Sh has joined #opendev21:39
dmsimardo/ heads up: ansible==3.0.0 releasing tomorrow21:44
clarkbI think we pin our ansible installation on bridge so that isn't likely to create much headache there. Similarly zuul controls the ansible versions tightly21:45
clarkbmostly likely to see problems in places like devstack-gate maybe21:45
dmsimardyep, despite the major version bump it still relies on the same 2.10 ansible-base so it's really just the ansible package moving to semver and updating included collections21:46
*** gmann is now known as gmann_afk21:51
fungithanks for the warning!22:00
*** slaweq has quit IRC22:02
*** zoharm has quit IRC22:02
*** whoami-rajat__ has quit IRC22:09
*** jhesketh has joined #opendev22:32
openstackgerritIan Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var
*** sboyron has quit IRC23:07
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var
*** gmann_afk is now known as gmann23:22
openstackgerritMartin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI
*** prometheanfire has quit IRC23:27
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var
openstackgerritIan Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container
*** prometheanfire has joined #opendev23:51

Generated by 2.17.2 by Marius Gedminas - find it at!