| diablo_rojo | fungi, I did them today | 00:01 |
|---|---|---|
| fungi | diablo_rojo: perfect, should be all set then | 00:06 |
| fungi | see gouthamr's comments above as well | 00:06 |
| diablo_rojo | fungi, cool! I will get that email sent out then. | 00:06 |
| diablo_rojo | Yeah I saw the comment! Still processing what that means, looking at the patch now. | 00:06 |
| fungi | just that meetings needed to be updated for correct channels | 00:14 |
| diablo_rojo | Reviewed them both now. | 00:31 |
| *** zbr has quit IRC | 00:33 | |
| *** zbr has joined #opendev | 00:45 | |
| *** dviroel has quit IRC | 00:59 | |
| clarkb | the gatling-git output is really confusing, it seems some small numebr of changes were successfully pushed. Another set failed due to a lack of change ids. and then the bulk failed due to no common ancestry | 01:03 |
| clarkb | I wonder if this is just a problem with races or something | 01:03 |
| clarkb | weird to have the no change id errors in a small subset because you'd expect those to consistently be generated or not be consistently generated | 01:03 |
| clarkb | maybe tomorrow I'll ask on the gerrit slack channel | 01:04 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 01:04 |
| *** brinzhang has joined #opendev | 01:10 | |
| *** zbr7 has joined #opendev | 02:23 | |
| *** zbr has quit IRC | 02:23 | |
| *** zbr7 is now known as zbr | 02:23 | |
| *** hemanth_n has joined #opendev | 02:58 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container https://review.opendev.org/c/opendev/system-config/+/775961 | 04:25 |
| *** hamalq has quit IRC | 04:41 | |
| *** diablo_rojo has quit IRC | 05:35 | |
| *** marios has joined #opendev | 06:00 | |
| *** whoami-rajat__ has joined #opendev | 06:12 | |
| *** zbr has quit IRC | 06:13 | |
| *** zbr has joined #opendev | 06:14 | |
| *** mnaser has quit IRC | 06:32 | |
| *** mnaser has joined #opendev | 06:34 | |
| *** ralonsoh has joined #opendev | 06:41 | |
| *** slaweq has joined #opendev | 07:07 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container https://review.opendev.org/c/opendev/system-config/+/775961 | 07:53 |
| *** sboyron has joined #opendev | 07:56 | |
| *** rpittau|afk is now known as rpittau | 08:12 | |
| *** andrewbonney has joined #opendev | 08:13 | |
| *** eolivare has joined #opendev | 08:21 | |
| *** DSpider has joined #opendev | 08:43 | |
| *** ysandeep|away is now known as ysandeep|ruck | 08:50 | |
| *** cloudnull has quit IRC | 08:59 | |
| *** jpena|off is now known as jpena | 08:59 | |
| *** cloudnull has joined #opendev | 09:02 | |
| *** tosky has joined #opendev | 09:14 | |
| *** tosky_ has joined #opendev | 09:20 | |
| *** tosky has quit IRC | 09:24 | |
| *** tosky_ is now known as tosky | 09:25 | |
| *** cloudnull has quit IRC | 09:37 | |
| *** cloudnull has joined #opendev | 09:40 | |
| *** dtantsur|afk is now known as dtantsur | 10:01 | |
| *** zoharm has joined #opendev | 10:11 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container https://review.opendev.org/c/opendev/system-config/+/775961 | 10:18 |
| *** rferraz has joined #opendev | 10:19 | |
| *** prometheanfire has quit IRC | 10:42 | |
| kopecmartin | hi, does opendev.org have an api endpoint supporting f.e. file iteration from a python code? similar to github where i can query a repo for files getting json data - https://api.github.com/repos/openstack/interop/contents | 10:48 |
| kopecmartin | never mind, found it https://opendev.org/api/swagger#/repository | 10:51 |
| *** prometheanfire has joined #opendev | 11:02 | |
| openstackgerrit | Merged opendev/irc-meetings master: Fix meeting channel for diversity wg https://review.opendev.org/c/opendev/irc-meetings/+/775953 | 11:14 |
| *** sshnaidm has quit IRC | 11:15 | |
| *** sshnaidm has joined #opendev | 11:18 | |
| *** dviroel has joined #opendev | 11:37 | |
| *** klonn has joined #opendev | 11:56 | |
| *** eolivare has quit IRC | 12:09 | |
| *** fdegir5 is now known as fdegir | 12:24 | |
| *** jpena is now known as jpena|lunch | 12:30 | |
| *** Dmitrii-Sh has quit IRC | 12:32 | |
| *** DSpider has quit IRC | 12:45 | |
| *** DSpider has joined #opendev | 12:57 | |
| *** klonn has quit IRC | 12:59 | |
| *** eolivare has joined #opendev | 13:19 | |
| *** jpena|lunch is now known as jpena | 13:23 | |
| mnasiadka | Hi, is there any opendev Ansible role I can use, to get chrony/ntp configured? | 13:25 |
| openstackgerrit | Merged opendev/irc-meetings master: Fix the openstacksdk meeting schedule https://review.opendev.org/c/opendev/irc-meetings/+/775933 | 13:27 |
| *** hemanth_n has quit IRC | 13:56 | |
| *** ysandeep|ruck is now known as ysandeep|mtg | 14:01 | |
| *** fressi has quit IRC | 14:07 | |
| *** fressi has joined #opendev | 14:09 | |
| *** lpetrut has joined #opendev | 14:20 | |
| fungi | mnasiadka: i think we already configure ntpd in the images we boot. are you seeing logs with incorrect timestamps in builds? | 14:50 |
| mnasiadka | fungi: Nah, trying to deploy ceph using cephadm - and it's complaining about lack of chrony on the images - because it's checking a systemd unit, so if ntp is configured I'll just make it quiet :) | 14:52 |
| *** jhesketh has quit IRC | 14:52 | |
| fungi | ahh, has ntpd lost its sheen, that they don't check to see if it might be in use instead of chrony? | 14:54 |
| *** brinzhang has quit IRC | 14:55 | |
| *** brinzhang has joined #opendev | 14:55 | |
| mnasiadka | fungi: well, ask Ceph guys - can you point me to playbooks that Zuul uses to configure ntp? | 14:55 |
| fungi | zuul doesn't configure ntp, we install it in the images nodepool boots, it'll be in nodepool elements, not playbooks | 14:56 |
| fungi | i'l go looking | 14:56 |
| mnasiadka | Ah, ok | 14:56 |
| *** brinzhang has quit IRC | 14:57 | |
| fungi | this tells nodepool to install the "ntp" package in all images it builds: https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/infra-package-needs/package-installs.yaml#L14 | 14:58 |
| *** brinzhang has joined #opendev | 14:58 | |
| fungi | and we also install ntpdate (the line just after it) | 14:58 |
| *** slaweq has quit IRC | 15:00 | |
| fungi | mnasiadka: looks like we do expect a chronyd service on centos: https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/infra-package-needs/post-install.d/80-enable-infra-services | 15:00 |
| *** fressi has quit IRC | 15:00 | |
| fungi | and some variation on ntp in other distros | 15:00 |
| fungi | what image are you testing on? | 15:01 |
| mnasiadka | centos8, so will dig a bit more why cephadm complains on missing chronyd systemd unit | 15:01 |
| *** slaweq has joined #opendev | 15:04 | |
| fungi | i suppose timesyncd is too new to be in centos 8 | 15:05 |
| fungi | mnasiadka: here's a current image build log for centos-8: https://nb01.opendev.org/centos-8-0000140932.log | 15:07 |
| fungi | shows we're definitely running `systemctl enable chronyd` during image creation | 15:07 |
| fungi | i'm stepping out to run some errands but should be back by ~16:00 utc | 15:10 |
| *** ysandeep|mtg is now known as ysandeep|ruck | 15:44 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift and upload-logs-gcs https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 15:46 |
| *** ysandeep|ruck is now known as ysandeep|dinner | 16:08 | |
| *** lpetrut has quit IRC | 16:25 | |
| clarkb | fungi: do you know how to find new instances of the growroot issue in kna1? Want to check if we caught any and wonder if you've got a quick way to find those | 16:25 |
| fungi | grenade jobs failing with errors about apt-get not having enough space to download packages was the initial sighting | 16:25 |
| fungi | i'll see if i can dig up the exact message | 16:26 |
| fungi | the up side there is the jobs get far enough to have their logs indexed in logstash | 16:26 |
| *** hashar has joined #opendev | 16:29 | |
| clarkb | fungi: the jobs do fail though right? I wonder if my luck might be good enough just looking at failed jobs in kna1 in the last 12 hours | 16:34 |
| fungi | yes, it's a failure, happens during the run phase | 16:34 |
| clarkb | cool let me see if I can follow that breadcrumb | 16:35 |
| *** hashar has quit IRC | 16:35 | |
| fungi | look for failed grenade jobs in particular | 16:35 |
| *** hashar has joined #opendev | 16:38 | |
| clarkb | doesn't look like there were any grenade failures there in the last 12 hours but there are 19 other failures I can spot check | 16:45 |
| clarkb | one failure was due to connectivity issues to the mirror just over an hour ago, I can reach the mirror now at least | 16:46 |
| clarkb | two other failures were for unrelated issues (one was an actual python test failure, another an ansible var being set wrong and devstack being unhappy about it) | 16:47 |
| fungi | E: You don't have enough free space in /var/cache/apt/archives/. | 16:49 |
| fungi | that's the message to search for | 16:49 |
| fungi | was turning up semi-regularly in grenade failures | 16:49 |
| fungi | and only in airship-kna1 | 16:49 |
| clarkb | there was another mirror contact failure about 4.5 hours ago too | 16:50 |
| fungi | also only on ubuntu-bionic, but i expect that's because master branch grenade jobs use bionic still | 16:50 |
| * clarkb looks for lack of freespace message | 16:50 | |
| clarkb | fungi: the most recent occurence of that message was about 23 hours ago so not recent enough for the growroot log collection | 16:51 |
| clarkb | I'll continue to look at these 19 failurse and see if any are helpful | 16:51 |
| *** slaweq has quit IRC | 16:53 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Upgrade ansible-lint to 5.0 https://review.opendev.org/c/zuul/zuul-jobs/+/773245 | 16:58 |
| clarkb | rm_work johnsom digging through failures to find a specific failure type so that I can debug it and I noticed neutron-lbaas-to-octavia-migration seems to fail consistently because it tries to install latest dib on python2 | 17:05 |
| clarkb | is that a job we should just stop running or perhaps update? Thought I would mention it as I don't think it has passed in a long time | 17:05 |
| *** mlavalle has joined #opendev | 17:05 | |
| clarkb | it is running in the periodic pipeline | 17:06 |
| johnsom | I think cgoncalves is working on the DIB-being-branchless issues with python2 and the stable branches | 17:07 |
| johnsom | clarkb https://review.opendev.org/c/openstack/octavia/+/774239 | 17:07 |
| johnsom | for example | 17:07 |
| cgoncalves | python 2 and 3.5 | 17:08 |
| johnsom | It's been a bit of a chain of things being broken with DIB and stable branches | 17:09 |
| clarkb | there is a fix for dib for that iirc | 17:09 |
| clarkb | I thought it got released | 17:09 |
| clarkb | the job failure I noticed is for rocky though not stein or train | 17:09 |
| johnsom | Rocky should be EOL | 17:09 |
| clarkb | https://7607c456e9cafc55c180-e9f29c7afce5197c5c20e02f6b6da59e.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron-lbaas/stable/rocky/neutron-lbaas-to-octavia-migration/7eeed91/job-output.txt | 17:09 |
| clarkb | its still running periodic jobs :) | 17:10 |
| johnsom | rm_work Weren't you working on the EOL for rocky, etc.? | 17:10 |
| cgoncalves | rocky octavia EOL'd months ago | 17:11 |
| johnsom | I'm just wondering if neutron-lbaas got missed, or the EOL job was not finished. | 17:11 |
| fungi | we haven't deleted the branches yet | 17:12 |
| fungi | if that plays into it at all | 17:12 |
| clarkb | fungi: that job name doesn't show up in codesearch so it must live on a stable branch zuul config | 17:12 |
| rm_work | hmmm | 17:12 |
| rm_work | yeah I was focused on Octavia | 17:12 |
| fungi | maybe people are still proposing changes against those branches | 17:12 |
| clarkb | seems likely that is the source of the issue | 17:12 |
| rm_work | I thought neutron-lbaas already got EOL'd, but maybe not | 17:12 |
| fungi | well, they could just be tagged eol and not have been deleted yet | 17:13 |
| clarkb | fungi: I've yet to find a growroot that looks unhappy. I think I'll leave it for a bit and hceck again later to see if more recent runs have caught one | 17:13 |
| * fungi finds the list | 17:13 | |
| fungi | http://paste.openstack.org/show/801992/ is the set of deletions requested so far, no lbaas in there (just stable/ocata for octavia) | 17:14 |
| *** jpena is now known as jpena|brb | 17:14 | |
| johnsom | rm_work I thought you submitted for EOL up to at least Rocky across the board. | 17:15 |
| rm_work | maybe, looking (i'm really bad at historical gerrit searching) | 17:16 |
| fungi | it's possible things have been added since that list was generated several weeks ago | 17:16 |
| johnsom | I see that Octavia has EOL tags up to Stein | 17:16 |
| *** dtantsur is now known as dtantsur|afk | 17:17 | |
| fungi | aha, the ml thread that list was attached to was specifically requesting stable/ocata branch deletions | 17:17 |
| * johnsom disappears back under a pile of other work now that the correct folks are engaged | 17:18 | |
| *** klonn has joined #opendev | 17:21 | |
| clarkb | I'm wary of reading into the dstat data during gatling runs against gerrit 3.2 and 3.3 too much because this is all running in cloud and we can be our own noisy neighbors, etc. Due to the reliance on the buildset registry both jobs ran in the same cloud region at near the same time though so the comparison is probably more valid that if they ran on random clouds. | 17:21 |
| clarkb | Two things I notice are that 3.2 has higher system load and lower memory use | 17:22 |
| clarkb | 3.3 uses a bit more memory but system load is better | 17:22 |
| clarkb | the system isn't push super hard though, which might be the next thing to look at with gatling after figuring out the push errors | 17:22 |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Upgrade ansible-lint to 5.0 https://review.opendev.org/c/zuul/zuul-jobs/+/773245 | 17:28 |
| *** eolivare has quit IRC | 17:28 | |
| *** ysandeep|dinner is now known as ysandeep|ruck | 17:29 | |
| *** marios is now known as marios|out | 17:35 | |
| *** jpena|brb is now known as jpena | 17:36 | |
| clarkb | I think the reason my pushes are failing is that this framework doesn't clone the specified repo first and then update from HEAD. It just edits some local commits and then pushes straight to the repo. I think that means if I create a second empty repo just for gatling it should be happier /me tries this | 17:40 |
| *** rpittau is now known as rpittau|afk | 17:41 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 17:43 |
| *** cloudnull has quit IRC | 17:49 | |
| *** cloudnull has joined #opendev | 17:49 | |
| *** ralonsoh has quit IRC | 17:54 | |
| *** klonn has quit IRC | 18:08 | |
| *** rferraz has quit IRC | 18:08 | |
| *** ysandeep|ruck is now known as ysandeep|away | 18:10 | |
| *** jpena is now known as jpena|off | 18:12 | |
| *** auristor has quit IRC | 18:21 | |
| *** slaweq has joined #opendev | 18:23 | |
| *** klonn has joined #opendev | 18:25 | |
| *** andrewbonney has quit IRC | 18:33 | |
| clarkb | I feel like this growroot thing is going to be a heisenbug | 18:33 |
| clarkb | as soon as we start actively checking logs to determine what is going on the problem disappears | 18:33 |
| fungi | yeah | 18:34 |
| *** auristor has joined #opendev | 18:36 | |
| *** klonn has quit IRC | 18:42 | |
| *** klonn has joined #opendev | 18:43 | |
| *** marios|out has quit IRC | 18:48 | |
| clarkb | hrm now gatling complains about change ids missing and the remote not advertising ref master | 19:10 |
| clarkb | I guess that is progress | 19:11 |
| *** slaweq has quit IRC | 19:20 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 19:31 |
| clarkb | if ^ works then I guess I'll file bugs upstream | 19:31 |
| clarkb | but thats a massive hack to force change id generation | 19:31 |
| kopecmartin | ianw: hi, I fixed issues I found in refstack - https://etherpad.opendev.org/p/refstack-docker , I don't know what to do with the url/endpoint ('api' is missing) | 19:34 |
| *** DSpider has quit IRC | 19:50 | |
| clarkb | kopecmartin: for the api url change maybe it is something apache proxying is modifying? | 19:50 |
| clarkb | kopecmartin: also left a note about hwo to work around DNS not being updated yet | 19:51 |
| clarkb | left a couple of thoughts about the api url thing too (after grepping in config mgmt | 19:54 |
| *** DSpider has joined #opendev | 19:58 | |
| *** DSpider has quit IRC | 19:59 | |
| *** klonn has quit IRC | 20:03 | |
| *** elod has quit IRC | 20:17 | |
| *** elod has joined #opendev | 20:19 | |
| *** hashar has quit IRC | 20:23 | |
| clarkb | oh hrm I see, the errors about no ref being advertised is due to me trying to pull the empty repo's master branch | 20:27 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 20:28 |
| *** slaweq has joined #opendev | 20:46 | |
| *** LowKey has joined #opendev | 21:08 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container https://review.opendev.org/c/opendev/system-config/+/775961 | 21:22 |
| clarkb | ianw: note that I think the current server's memory constraints will make ^ difficult | 21:23 |
| ianw | kopecmartin / clarkb: yeah, i have no idea really if https://review.opendev.org/c/opendev/system-config/+/775053 was correct, it would be worth understanding the configs better probably | 21:24 |
| ianw | clarkb: yeah, i just wanted to flesh out the idea initially. i pulled stats on the usage (https://imgur.com/a/QGJV7Fw) and the db barely manages to trip 1 reference per second now | 21:25 |
| clarkb | it will be related to how quickly people review files in the web ui, but ya that seems about right. I'm more worried about memory needed to keep tables in memory | 21:26 |
| ianw | clarkb: yeah, i agree. the other thing is we have stuff like lxc running, which is not started on focal by default but takes up about the same space as mysql (~800mb) | 21:27 |
| clarkb | ianw: oh interesting, is that something we should go ahead and turn off now ? | 21:28 |
| ianw | probably, i was going to investigate but was very hesitant to touch anything on production :) | 21:28 |
| *** cloudnull has quit IRC | 21:28 | |
| clarkb | it didn't even occur to me that there could be unneeded system level things there to look for | 21:28 |
| ianw | root 1110 1 0 2018 ? 00:19:16 /usr/bin/lxcfs /var/lib/lxcfs/ | 21:29 |
| ianw | root@review01:~# pmap 1110 | 21:29 |
| ianw | total 760408K | 21:29 |
| ianw | so i definitely buy the contention argument. a bit like the http proxy on gitea to filter useragents, i figure at minimum it might be something good to have in our pocket anyway | 21:30 |
| clarkb | ya, and may also simplify testing? | 21:31 |
| ianw | yeah, and it was inspired by the practicalities of us doing a pretty bad job at keeping the trove instance updated | 21:32 |
| ianw | which has bitten us now with the backups issue, that luckily we noticed but https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1914695 isn't going to fix | 21:32 |
| openstack | Launchpad bug 1914695 in mysql-5.7 (Ubuntu) "mysqldump --all-databases not dumping any databases with 5.7.33" [Undecided,Fix released] | 21:32 |
| ianw | (well unless we do it) | 21:33 |
| *** cloudnull has joined #opendev | 21:33 | |
| ianw | anyway, it doesn't work yet :) you can be sure i'll have a far too long changelog when it does and we can discuss :) | 21:34 |
| clarkb | :) | 21:34 |
| clarkb | sort of related is my gatling work. I can't get pushes to work at all due to missing change ids (I even hacked the code to try and force that flag on :/) the interesting thing so far is it generates load and we can look at dstat metrics from there | 21:34 |
| clarkb | and since the buildset registry forces all the related jobs to run in the same cloud region we get what should be somewhat comparable numbers though I've not gaterhed enough data to check that yet | 21:35 |
| clarkb | with a dataset of one gerrit 3.3 uses more memory but hsa lower system loda compared to 3.2 | 21:35 |
| *** Dmitrii-Sh has joined #opendev | 21:39 | |
| dmsimard | o/ heads up: ansible==3.0.0 releasing tomorrow | 21:44 |
| clarkb | I think we pin our ansible installation on bridge so that isn't likely to create much headache there. Similarly zuul controls the ansible versions tightly | 21:45 |
| clarkb | mostly likely to see problems in places like devstack-gate maybe | 21:45 |
| dmsimard | yep, despite the major version bump it still relies on the same 2.10 ansible-base so it's really just the ansible package moving to semver and updating included collections | 21:46 |
| *** gmann is now known as gmann_afk | 21:51 | |
| fungi | thanks for the warning! | 22:00 |
| *** slaweq has quit IRC | 22:02 | |
| *** zoharm has quit IRC | 22:02 | |
| *** whoami-rajat__ has quit IRC | 22:09 | |
| *** jhesketh has joined #opendev | 22:32 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container https://review.opendev.org/c/opendev/system-config/+/775961 | 22:44 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var https://review.opendev.org/c/zuul/zuul-jobs/+/776290 | 23:02 |
| *** sboyron has quit IRC | 23:07 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var https://review.opendev.org/c/zuul/zuul-jobs/+/776290 | 23:15 |
| *** gmann_afk is now known as gmann | 23:22 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 23:25 |
| *** prometheanfire has quit IRC | 23:27 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var https://review.opendev.org/c/zuul/zuul-jobs/+/776290 | 23:44 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : use mariadb container https://review.opendev.org/c/opendev/system-config/+/775961 | 23:48 |
| *** prometheanfire has joined #opendev | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!