| fungi | thanks! | 00:03 |
|---|---|---|
| *** mlavalle has quit IRC | 00:09 | |
| johnsom | Any advice about what I should do when I get "failed to push some refs"? | 00:11 |
| johnsom | https://www.irccloud.com/pastebin/4qFPQk1Z/ | 00:11 |
| ianw | johnsom: in what context? first thought is that you're trying to update a change that doesn't need updating? | 00:13 |
| clarkb | johnsom: you aren't pushing a merge commit are you? | 00:13 |
| johnsom | I was doing the normal "git review" after fixing a requirements.txt | 00:13 |
| johnsom | There is a screen and a half of spinning lines above it. I can paste it all if you would like | 00:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add bionic-plain non-voting testing https://review.opendev.org/717655 | 00:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header https://review.opendev.org/717656 | 00:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 00:14 |
| johnsom | https://www.irccloud.com/pastebin/chJB2kXw/ | 00:15 |
| ianw | "internal error" there looks ... interesting | 00:15 |
| johnsom | Full thing | 00:15 |
| ianw | because ^^ i am also pushing a stack right now that seems to have stopped | 00:15 |
| johnsom | gerrit bot announced it. I'm just ... concerned | 00:15 |
| clarkb | johnsom: gerrit bot announced it meaning the chagne was pushed? have a link to it? | 00:16 |
| johnsom | https://review.opendev.org/711376 | 00:16 |
| fungi | possible that happened right at utc midnight? | 00:16 |
| johnsom | The two files I changed reflect my change | 00:16 |
| clarkb | johnsom: thats a lot more than just a requirements update | 00:16 |
| mnaser | zuul-preview has been updated to fix the security issue that it had. is there a way to know if/when it has been redeployed? | 00:17 |
| clarkb | johnsom: can you use more words to express that? the change has a lot of things in it. much more than 2 files | 00:17 |
| johnsom | Sorry, my revision, patch set 8, only updated the requirements.txt and lower-constraints.txt files. | 00:18 |
| johnsom | Bumped both to octavia-lib 2.0.0 | 00:18 |
| clarkb | johnsom: ok, from gerrits and gits perspective you push the entire patch each time | 00:18 |
| ianw | ! [remote rejected] HEAD -> refs/for/master%topic=ensure-pip (internal error) | 00:18 |
| ianw | i just got the same thing | 00:18 |
| fungi | infra-root: per mnaser's question, should it be safe to boot zuul-preview.o.o back up (it's presently in shutdown state) and then immediately disable apache until configuration management gets applied? | 00:18 |
| clarkb | [2020-04-07 00:18:20,275] [SSH git-receive-pack '/zuul/zuul-jobs.git' (iwienand)] WARN com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits while processing changes for project zuul/zuul-jobs | 00:19 |
| clarkb | ianw: ^ was that you? | 00:19 |
| clarkb | oh ya its got your username in it | 00:19 |
| ianw | very likely :) | 00:19 |
| clarkb | WARN com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker killed after 240205ms(timeout 205ms, cancelled) | 00:19 |
| fungi | "Error in ReceiveCommits while processing changes for project" sounds like we could have a problem with the local repositories, i suppose | 00:19 |
| clarkb | fungi: it seems like its taking too long so gerrit is stopping it | 00:20 |
| clarkb | melody doesn't show garbage collecting which is a commonish reason for things being slow | 00:21 |
| fungi | right, maybe we've got something else timing out... db connections? | 00:21 |
| ianw | this all has my local timestamp of 10:14 ... same as openstackgerrit's comments above https://review.opendev.org/#/q/topic:ensure-pip+(status:open+OR+status:merged) | 00:21 |
| clarkb | ianw: was that all of your commits or only some? | 00:21 |
| ianw | however, openstackgerrit only mentioned 3 of the commits | 00:21 |
| clarkb | but ya I'm wondering if this is related to db backups | 00:21 |
| mnaser | fwiw on sunday we observed some weird behaviour with gearman disconnecting from zuul. i don't think this is related, but it may signal *something* | 00:21 |
| fungi | looks like mysqldump ran from 00:00 to 00:11z | 00:21 |
| ianw | that stack was rebased ontop of the recent renames in zuul-jobs, so all should have been updated | 00:22 |
| clarkb | stackalytics bot is doing a ton of queries right now too | 00:22 |
| clarkb | ianw: johnsom octavia and zuul-jobs are the only two things that show "Error in ReceiveCommits while processing changes for project" | 00:23 |
| clarkb | my hunch is that db backup and/or stacklytics queries impacting performance of pushes on the short term | 00:23 |
| clarkb | possibly those two things happenign together | 00:24 |
| fungi | 240205ms is just over 4 minutes, so that still doesn't put initialization of the call inside the mysqldump window | 00:24 |
| clarkb | fungi: could be request backlog due to db locks though | 00:24 |
| * clarkb pushes a change to see if it is still happening | 00:25 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add bionic-plain non-voting testing https://review.opendev.org/717655 | 00:25 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header https://review.opendev.org/717656 | 00:25 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 00:25 |
| ianw | interesting ... i rebased on a wording change | 00:25 |
| ianw | it appears to have hung again | 00:25 |
| fungi | cacti doesn't show any significant spikes in activity | 00:26 |
| clarkb | opendev/infra-manual change I just pushed up was fine | 00:26 |
| ianw | read(3, "a\325\216\32\255a\235W\343\251\245,O7\10\5\27\33\375H\r\215\222\377\31\3753\205\322*\345\351"..., 8192) = 112 | 00:26 |
| ianw | select(7, [3], [5], NULL, {tv_sec=300, tv_usec=0}) = 1 (out [5], left {tv_sec=299, tv_usec=999933}) | 00:26 |
| ianw | write(5, "0031\2\rProcessing changes: update"..., 49) = 49 | 00:26 |
| clarkb | however it was based on HEAD and one line | 00:26 |
| ianw | is what it's doing over and over | 00:26 |
| ianw | (the ssh connection) | 00:26 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 00:27 |
| ianw | /usr/bin/ssh -p 29418 iwienand@review.opendev.org git-receive-pack '/zuul/zuul-jobs.git' | 00:27 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 00:28 |
| clarkb | ianw: no new errors in the log yet at least (and it looks like they are coming through slowly | 00:28 |
| clarkb | so ya possibly related to that repo? | 00:28 |
| fungi | though johnsom's was for a different repo? | 00:28 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 00:28 |
| clarkb | fungi: yes for octavia | 00:29 |
| johnsom | openstack/octavia | 00:29 |
| * clarkb tries octavia | 00:29 | |
| ianw | yeah, last push openstackgerrit didn't comment on 3 of 6 | 00:29 |
| ianw | this time it did, slowly | 00:29 |
| johnsom | Let me know if you want me to try something, I'm just hanging out and writing a google doc | 00:29 |
| fungi | fwiw, we have seen patchsets "go missing" when pushed during the mysqldump window, because gerrit gives up waiting for the db write to complete and so never records the push | 00:30 |
| clarkb | I can push to octavia with no slowness or problems | 00:30 |
| ianw | fungi: but the one above at 10:25 was well outside the window? | 00:30 |
| clarkb | ianw's recent push just tripped the error log string that I was checking for | 00:30 |
| clarkb | my ability to push to octavia wihtout issue makes me think it isn't a repo problem. Could be construction of the change series? | 00:31 |
| fungi | ianw: is it for the same change or a different change? | 00:31 |
| clarkb | ianw: do you mind if I fetch your stack and trivially update it and push that? | 00:31 |
| clarkb | ianw: actually now I don't know which change to update. https://review.opendev.org/#/c/717657/5 that one maybe? | 00:32 |
| ianw | clarkb: umm, the bottom one is ... | 00:33 |
| clarkb | ianw: the bionic update one? | 00:34 |
| ianw | oh, the top one would be better | 00:34 |
| clarkb | ianw: ya but I worry it won't exercise things as much since it just one change being pushed | 00:34 |
| clarkb | but thats why I am asking for permission before I invalidate test results :) | 00:34 |
| ianw | git review -d 717657 | 00:34 |
| ianw | that should be all of it | 00:34 |
| clarkb | ianw: ya I've pulled that change down and updated the one 6 ahead of it | 00:35 |
| clarkb | you ok with my pushing new patchsets up for all 6? | 00:35 |
| ianw | yep | 00:35 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add bionic-plain non-voting testing https://review.opendev.org/717655 | 00:35 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header https://review.opendev.org/717656 | 00:35 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 00:35 |
| clarkb | seems to be doing it to me too | 00:35 |
| clarkb | thats good I guess? | 00:35 |
| clarkb | johnsom: was your change in a stack of changes? | 00:36 |
| clarkb | ok so gerrit knwos about my newly pushed patchests but git review doesn't return yet | 00:36 |
| johnsom | It was the first one with two dependent on top | 00:36 |
| clarkb | and gerritbot doesn't say anything yet | 00:36 |
| ianw | it *has* seemed to update them all though? they all have the same timestamp now | 00:37 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 00:37 |
| clarkb | ianw: yup | 00:37 |
| clarkb | but my git review has not returned | 00:37 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 00:38 |
| clarkb | johnsom: ianw: in ianws stack the first three seem to be processed quickly and don't add any new files | 00:38 |
| clarkb | the fourth change is slower and does add new files. I notice that johnsom's change also adds files | 00:38 |
| clarkb | potentially somethign to do with the contruction of the change itself? | 00:39 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 00:39 |
| clarkb | there is that second mysql database that gerrit uses that tracks files iirc | 00:39 |
| clarkb | perhaps it could be related to that (that db being slow to update possibly?) | 00:39 |
| fungi | the h2 db? | 00:41 |
| clarkb | https://review.opendev.org/717887 was just pushed up and adds a new file. That was not slow at all | 00:41 |
| clarkb | fungi: its not an h2 db, its in mysql | 00:41 |
| clarkb | https://review.opendev.org/717887 probably rules out the new file being the cause | 00:41 |
| ianw | [2020-04-07 00:36:20,564] [ReceiveCommits-16] ERROR com.googlesource.gerrit.plugins.its.storyboard.StoryboardClient : Failed to post, response: 500 (Internal Server Error) | 00:41 |
| fungi | hrm... accountPatchReviewDb? | 00:41 |
| ianw | maybe something to do with that ... is that the first one that has a Story: # | 00:42 |
| clarkb | ianw: oh interesting | 00:42 |
| clarkb | fungi: ya that sounds correct | 00:42 |
| clarkb | fwiw `gerrit show-queue -w` doesn't show any oustanding queue items | 00:42 |
| fungi | oh, could this the the its-storyboard plugin misbehaving somehow? | 00:42 |
| clarkb | ya I'm betting the storyboard thread is the one to pull on | 00:42 |
| johnsom | My patch has a story | 00:43 |
| clarkb | I need to go help make dinner but can check in in a bit | 00:43 |
| ianw | logging in to see what that 500 might be about | 00:43 |
| fungi | storyboard api still seems responsive | 00:44 |
| ianw | [Tue Apr 07 00:40:00.314321 2020] [wsgi:error] [pid 9734:tid 140188982429440] 2020-04-07 00:40:00.313 9734 ERROR wsme.api [-] Server-side error: "This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted | 00:44 |
| ianw | within this transaction.". Detail: | 00:44 |
| ianw | hang on, it was 00:36 | 00:44 |
| ianw | same msg @ Tue Apr 07 00:36:20.561390 2020 | 00:45 |
| ianw | http://paste.openstack.org/show/791710/ | 00:45 |
| fungi | mysqld is running | 00:46 |
| fungi | [Mon Apr 6 23:11:25 2020] Out of memory: Kill process 4923 (apache2) score 745 or sacrifice child | 00:46 |
| fungi | i suspect this is related | 00:47 |
| ianw | oh great, one of those errors where the only hits you get is from sites indexing the code that is throwing it | 00:47 |
| fungi | ianw: i had one last week where i searched for an obscure error in some (not ours) software and the only hit was an irc log where we were scratching our heads about it 5 years ago | 00:47 |
| fungi | i'm going to restart services on storyboard.o.o | 00:48 |
| ianw | it would explain the behaviour though of gerrit seeming to get all the changes, but follow-up processing being weird | 00:49 |
| fungi | the server is currently early 4gb into swap | 00:49 |
| fungi | looks like mysql is eating 4gb and an apache worker has another 4gb | 00:49 |
| fungi | may actually just reboot the whole server to make sure everything comes up clean and i don't miss anything | 00:49 |
| fungi | it's rebooting now | 00:50 |
| fungi | #status log rebooted storyboard.openstack.org following a slew of out-of-memory conditions which killed various processes | 00:51 |
| openstackstatus | fungi: finished logging | 00:51 |
| fungi | server's back up | 00:51 |
| ianw | i'll try the stack again | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add bionic-plain non-voting testing https://review.opendev.org/717655 | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header https://review.opendev.org/717656 | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 00:52 |
| fungi | yeah, looks like it was thrashing so hard even snmp queries were going unanswered | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 00:52 |
| ianw | yay \o/ | 00:52 |
| ianw | mystery solved i guess | 00:53 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66356&rra_id=all | 00:53 |
| fungi | yikes | 00:53 |
| fungi | looks like things just started going nuts around 20:00 | 00:53 |
| johnsom | o/ I'm not too worried that storyboard didn't get update for this patch, so I'm off to make dinner. Thanks folks | 00:55 |
| fungi | thanks for bringing it to our attention johnsom! | 00:55 |
| fungi | i suspect the pattern we're seeing on that graph is something causing an apache worker to eat all the memory, then it gets killed, then another apache worker does the same, over and over | 00:56 |
| fungi | since the storyboard api is running in mod_wsgi though that could be pretty much any api interaction at fault | 00:57 |
| fungi | the first oom in the syslog starts at 20:27:00 | 01:03 |
| fungi | as expected, it reached 100% utilization of all 8gb ram and 8gb swap | 01:04 |
| fungi | the pid killed that time was an apache2 process with 3438675 total_vm (1839417 rss, 1103106 swapents) | 01:07 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 01:08 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 01:08 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 01:08 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 01:08 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 01:15 |
| fungi | not seeing any unusual traffic jumping out at me from the access logs | 01:15 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 01:15 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 01:15 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 01:15 |
| fungi | also those wsgi "IOError: failed to write data" messages are probably unrelated, we get a handful of those every day in the logs for as far back as our log retention goes | 01:18 |
| fungi | the user-facing impact (gerrit et ceters) likely didn't start until 23:49:39z when the first sqlalchemy traceback appears in the apache error log | 01:20 |
| fungi | [Mon Apr 06 23:49:39.702752 2020] [wsgi:error] [pid 9734:tid 140188890109696] [client 216.54.31.86:5548] DBError: Can't reconnect until invalid transaction is rolled back, referer: https://storyboard.openstack.org/ | 01:21 |
| fungi | anyway, not sure what else to look at for the moment, other than to keep an eye on the cacti graphs for the server and see if we get any recurrence | 01:22 |
| *** jentoio has joined #opendev | 01:26 | |
| *** ysandeep|away is now known as ysandeep|rover | 02:02 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 02:59 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 02:59 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 03:00 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 03:00 |
| *** prometheanfire has quit IRC | 03:11 | |
| *** prometheanfire has joined #opendev | 03:12 | |
| kevinz | ianw: talk here:-) https://a1f557d8e7f24035a796-8b5cc018a02fed487ca2a6d59a4ee9d5.ssl.cf1.rackcdn.com/716839/1/check-arm64/dib-functests-arm64-bionic/129eb08/logs/ubuntu-minimal_bionic-arm64-build-succeeds.FAIL.log | 03:12 |
| kevinz | Connection failed [IP: 139.178.85.143 80] | 03:12 |
| *** ysandeep|rover is now known as ysandeep|afk | 03:13 | |
| kevinz | ianw: I remember it has happened before in Linaro London | 03:13 |
| ianw | kevinz: oh yes sorry frickler mentioned that, i meant to follow up | 03:13 |
| ianw | that's the mirror? | 03:14 |
| kevinz | ianw: no worries, yes that is the mirror | 03:14 |
| ianw | yeah ... i guess it doesn't have AAAA records? | 03:14 |
| kevinz | ianw: i can access it locally | 03:14 |
| prometheanfire | anyone able to take a min to review this glean fix https://review.opendev.org/717339 | 03:15 |
| prometheanfire | fixes some resolved functionality | 03:15 |
| ianw | kevinz: hrm, it does have AAAA records so i wonder why we try ipv4? | 03:16 |
| kevinz | ianw: Aha, I thinks it may due to the settings. I change the setting in DNS to use host dns config | 03:17 |
| kevinz | ianw: I change this because Kolla need this IPv4 dns for docker building, as IPv6 for some apt/yum resources are not stable. | 03:18 |
| kevinz | ianw: let me think how to fix this. I can set os-jobs subnet dns to host dns server, and leave os-controls subnet dns as default. | 03:20 |
| *** ysandeep|afk is now known as ysandeep|rover | 04:18 | |
| *** ykarel|away is now known as ykarel | 04:20 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 04:21 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 04:21 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 04:21 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 04:22 |
| prometheanfire | ianw: thanks for the review, unfortunately this bug was turned up in a more basic gentoo install, the previous stage4 wrote a file (with google dns servers) to /etc/resolv.conf, hiding the error, stage3 doesn't have that file | 04:26 |
| prometheanfire | it seems like what we need is... if file is exists and is not symlink, write file. if file exists and is symlink (not dangling), write file. if file does not exist and is symlink, do NOT write file (resolved). if file does not exist and is not symlink, write file (unconfigured system) | 04:29 |
| prometheanfire | ianw: https://review.opendev.org/#/c/257173/ could work I guess, was hoping to avoid that | 04:35 |
| prometheanfire | the problem with it is that it disables resolved | 04:36 |
| *** prometheanfire has quit IRC | 04:40 | |
| *** prometheanfire has joined #opendev | 04:40 | |
| ianw | i ... don't quite know :) perhaps glean can not decide where the file should point to at all? it should all be up to the image to provide the right thing? | 04:44 |
| prometheanfire | ianw: if only systemd sucked a little less :P | 04:46 |
| fungi | yeah, problem is systemd's "resolved" eventually writes out a file with dns info, i guess, but until it does /etc/resolv.conf is a dangling symlink? | 04:46 |
| prometheanfire | ya, that's basically it | 04:47 |
| fungi | but if glean writes to the symlink target then resolved will decide it shouldn't touch it? | 04:47 |
| prometheanfire | no, the target's directory does not exist | 04:47 |
| ianw | more if that's a symlink, then dhclient/networkmanager won't write to it IIRC | 04:47 |
| prometheanfire | /run/systemd/resolved | 04:47 |
| prometheanfire | tmpdir that's created when resolved starts, after glean has run | 04:48 |
| fungi | i see | 04:48 |
| ianw | but also, for like RAX where we have to write out the nameservers from configdrive, i think there's chicken/egg? | 04:48 |
| prometheanfire | not sure about that, I use config drive at home, so maybe I'm experiencing the same thing | 04:49 |
| prometheanfire | I thought glean had to use config drive | 04:49 |
| ianw | yes it doesn't do http meta-data reading, but it's more about if the environment supports dhcp or not | 04:50 |
| prometheanfire | sure | 04:52 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds https://review.opendev.org/717177 | 04:52 |
| prometheanfire | well, included that, won't help the gcc build time, but should help it look like it boots | 04:53 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update registry test to use ensure-podman and ensure-docker https://review.opendev.org/716752 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: prefer venv to virtualenv https://review.opendev.org/717882 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 04:53 |
| prometheanfire | now, if only I could get hurricane electric to stop sending me duplicate packets | 04:54 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire OSA repo_build and pip_install projects https://review.opendev.org/717775 | 04:55 |
| prometheanfire | well, gonna go offline, if you have a reply, I guess reply in a bug (should be back in 5-10 min) | 04:55 |
| ianw | i'm quite willing to believe that glean is either doing the wrong thing, or can not make the right decision in this case, and the base image should configure /etc/resolv.conf in such a way that if glean writes to it it "does the right thing" | 04:57 |
| *** prometheanfire has quit IRC | 05:01 | |
| *** prometheanfire has joined #opendev | 05:12 | |
| prometheanfire | well, he is sucking hard eggs | 05:12 |
| openstackgerrit | Merged zuul/zuul-jobs master: Update registry test to use ensure-podman and ensure-docker https://review.opendev.org/716752 | 05:14 |
| *** ralonsoh has joined #opendev | 06:02 | |
| *** dpawlik has joined #opendev | 06:30 | |
| *** rpittau|afk is now known as rpittau | 06:56 | |
| *** DSpider has joined #opendev | 07:06 | |
| *** sgw has quit IRC | 07:11 | |
| AJaeger | config-core, please review https://review.opendev.org/717715 https://review.opendev.org/717655 https://review.opendev.org/717713 | 07:19 |
| *** tosky has joined #opendev | 07:24 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 07:44 | |
| frickler | AJaeger: I wouldn't merge 717655 on its own, rather wait for the stack to be a bit more complete | 08:02 |
| *** ralonsoh has quit IRC | 08:05 | |
| openstackgerrit | Merged openstack/project-config master: Retire repo_build and pip_install roles https://review.opendev.org/717715 | 08:07 |
| openstackgerrit | Merged zuul/zuul-jobs master: docs: fix a typo in `run-test-command` https://review.opendev.org/717713 | 08:15 |
| *** ralonsoh has joined #opendev | 08:17 | |
| openstackgerrit | mathieu bultel proposed openstack/project-config master: Add validations-common and validations-libs to pypi https://review.opendev.org/717976 | 08:26 |
| *** ysandeep|lunch is now known as ysandeep|rover | 08:44 | |
| *** ykarel is now known as ykarel|lunch | 08:48 | |
| *** roman_g has joined #opendev | 09:02 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Support ssh-enabled windows hosts in add-build-sshkey https://review.opendev.org/653712 | 09:12 |
| *** ykarel|lunch is now known as ykarel | 09:41 | |
| *** ykarel is now known as ykarel|meeting | 10:02 | |
| *** rpittau is now known as rpittau|bbl | 10:25 | |
| *** ykarel|meeting is now known as ykarel | 10:37 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 11:24 |
| *** ysandeep|rover is now known as ysandeep|afk | 11:29 | |
| *** hashar has joined #opendev | 11:31 | |
| *** hashar has quit IRC | 11:42 | |
| *** ysandeep|afk is now known as ysandeep|rover | 11:56 | |
| *** rpittau|bbl is now known as rpittau | 12:06 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 12:18 |
| *** hashar has joined #opendev | 12:31 | |
| mnaser | infra-root: can we enable zuul-preview again? The security issue has been patched now | 12:44 |
| *** sgw has joined #opendev | 13:12 | |
| *** roman_g has quit IRC | 13:21 | |
| mordred | mnaser: yeah - probably so. I'm slow this morning so I wanna wait for corvus to be up and verify he's happy | 13:49 |
| mnaser | mordred: ok cool :) | 13:52 |
| *** mlavalle has joined #opendev | 13:58 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add bionic-plain non-voting testing https://review.opendev.org/717655 | 14:14 |
| corvus | mnaser, mordred: yep, looks like the image is published; i'll remove zp01 from the emergency file and manually start it back up | 14:17 |
| corvus | #status log removed zp01 from emergency file since open-proxy issue is resolved in container image | 14:17 |
| openstackstatus | corvus: finished logging | 14:18 |
| mordred | corvus: woot | 14:19 |
| corvus | the, ah, host doesn't appear responsive | 14:19 |
| corvus | it was in 'shutoff' state; i'm restarting it | 14:24 |
| corvus | mnaser: should be up and running the updated image now | 14:25 |
| *** ysandeep|rover is now known as ysandeep|away | 14:33 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 14:34 |
| *** pramchan has joined #opendev | 14:38 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 14:38 |
| mnaser | corvus: cool, thanks, i'll try it out in a bit :) | 14:42 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 15:11 |
| johnsom | clarkb Morning, I assume you are done with https://review.opendev.org/717885? | 15:25 |
| clarkb | johnsom: yes, I'll abandon it | 15:26 |
| johnsom | Thank you. Doing our freeze week review scrubs. grin | 15:26 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 15:45 |
| frickler | corvus: ffmuc folks have built updated jitsi apps, I tested .deb and x64.appimage and they seem to work fine against meetpad for me https://github.com/freifunkMUC/jitsi-meet-electron/releases/tag/v2.0.5 | 15:55 |
| AJaeger | config-core, infra-root, please review https://review.opendev.org/717814 https://review.opendev.org/717815 https://review.opendev.org/717813 https://review.opendev.org/718124 https://review.opendev.org/717822 - this is all changing install-* to ensure-* roles | 15:57 |
| AJaeger | config-core, and one more review, please : https://review.opendev.org/717976 - adding pypi jobs for two new repos | 16:00 |
| *** dpawlik has quit IRC | 16:00 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman https://review.opendev.org/716578 | 16:03 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Improving logging of prod playbook jobs https://review.opendev.org/718161 | 16:05 |
| clarkb | mordred: ^ what do you think of that? | 16:05 |
| clarkb | AJaeger: looking | 16:05 |
| mordred | clarkb: I think that's great | 16:07 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| corvus | frickler: cool, i think we're probably going to update meetpad to point at etherpad.opendev.org when that's finished and then see how things are working | 16:09 |
| clarkb | frickler: corvus are we thinking the app might be more stable than web browser? | 16:09 |
| clarkb | or just a useful tool to add to the toolchest? | 16:09 |
| corvus | clarkb: oh, i wasn't thinking that; i thought that was just an 'fyi' from frickler saying he tested it | 16:09 |
| clarkb | ah | 16:10 |
| *** ykarel is now known as ykarel|away | 16:10 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 16:11 |
| mordred | speaking of - maybe that'll be green this time | 16:11 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: fetch-javascript-output: add metadata.type https://review.opendev.org/718163 | 16:11 |
| openstackgerrit | Merged openstack/project-config master: Add validations-common and validations-libs to pypi https://review.opendev.org/717976 | 16:19 |
| fungi | also looks like we might ought to try running tip of the develop branch for now (like etherpad-dev) until 1.8.1 releases: https://github.com/ether/etherpad-lite/issues/3781 | 16:21 |
| fungi | which is probably any day now | 16:21 |
| clarkb | fungi: I think we are going to use their upstream docker image | 16:22 |
| fungi | (which might end up being released as 1.8.3, i'm unclear on what the decision is there) | 16:22 |
| fungi | oh, in that case i guess their images are build from the tagged releases? | 16:22 |
| fungi | er, built | 16:22 |
| clarkb | yes appears that way | 16:22 |
| openstackgerrit | Merged openstack/project-config master: Use ensure-* roles https://review.opendev.org/717813 | 16:23 |
| openstackgerrit | Merged opendev/base-jobs master: Use ensure-* roles https://review.opendev.org/717815 | 16:26 |
| fungi | in that case, i guess we should just be on the lookout for a new etherpad image for the upcoming point release | 16:26 |
| *** lpetrut has joined #opendev | 16:27 | |
| mordred | yeah - and bump the version in the compose file | 16:30 |
| mordred | they don't seem to release a 1.8 image tag - just the full point release | 16:30 |
| AJaeger | clarkb: thx for reviews | 16:31 |
| openstackgerrit | Merged zuul/zuul-jobs master: fetch-javascript-output: add metadata.type https://review.opendev.org/718163 | 16:36 |
| *** pramchan has quit IRC | 16:49 | |
| mordred | clarkb: https://review.opendev.org/#/c/716442/ is green (at least the run-etherpad job is) so is ready for re-review when you get a sec | 16:56 |
| *** lpetrut has quit IRC | 16:56 | |
| mordred | clarkb: I've also got a mysqldump of production I'll apply to the local db once it's up and running so we can test things | 16:56 |
| mordred | clarkb: I figure rollout process will be turn off old, swap dns, dump db, restore dump on new server - will have an effective downtime of the dns ttl for folks | 16:57 |
| fungi | probably worth announcing, not only because it will be down briefly, but also in case folks notice anything broken with older pads | 16:58 |
| mordred | yeah | 16:58 |
| fungi | i did notice one of clarkb's test pads on etherpad-dev stopped working normally after upgrading | 16:58 |
| mordred | also - we should probably poke at the new install for at least a few days before we announce a cutover | 16:59 |
| clarkb | fungi: the clarkb-test pad was broken a long time ago | 16:59 |
| fungi | this was clarkb-test2, but we had tortured that one fairly severely and it was around for years, so not entirely surprising | 16:59 |
| fungi | nevermind, it's working again now | 17:00 |
| fungi | it was failing to load with the new colibris theme | 17:00 |
| fungi | reverting to the no-skin theme seems to have got it back to normal | 17:01 |
| fungi | so maybe a bug in the theme itself | 17:01 |
| clarkb | wouldn't surprise me if formatting of pads is affected in bad ways by new themes | 17:01 |
| fungi | yeah, me either | 17:02 |
| fungi | theming support in it is relatively new anyway | 17:02 |
| mnaser | if there’s a change in an image that opendev consumes, does it automatically update it every from run? | 17:03 |
| clarkb | mordred: I'm still not sure I understand the utf8 thing becuse in the mysqld portion of the config you haven't set it to utf8mb4 https://review.opendev.org/#/c/716442/19/playbooks/roles/etherpad/files/my.cnf | 17:03 |
| mnaser | (zuul preview change merged and just wondering if we have to bump | 17:04 |
| mnaser | something somewhere) | 17:04 |
| clarkb | mnaser: yes, I believe all of our images auto update | 17:04 |
| mordred | clarkb: right. that's specifying the type on the server side of the connection. I'm setting it there because that's what's set on our current trove instance | 17:09 |
| mordred | clarkb: separately, the schemas and tables have their own character sets | 17:09 |
| clarkb | mordred: so that doesn't affect the default encoding when you createa a new table? | 17:10 |
| clarkb | (because the issue aiui is that etherpad doesn't specify the encoding so we have to help it) | 17:10 |
| clarkb | but maybe thats since been fixed in the software and is a non issue | 17:11 |
| mordred | clarkb: yeah - but our tables will be created from the mysqldump - which will | 17:11 |
| mordred | yeah | 17:11 |
| mordred | well - I have no idea what etherpad will do if we let it do its own thing | 17:11 |
| clarkb | mordred: back when we had problems with it it was using utf8 but not understanding that utf8 is only 3 bytes wide | 17:11 |
| clarkb | so we did an end around etherpad and converted to utf8mb4 | 17:11 |
| mordred | clarkb: http://paste.openstack.org/show/791754/ | 17:12 |
| mordred | this is what is in the mysqldump | 17:12 |
| mordred | which is how the table wil get created | 17:12 |
| fungi | though it may still be worth solving for greenfield deployments | 17:12 |
| fungi | that can always come later | 17:12 |
| mordred | I mean - we could set it to mb4 in the file - I'm pretty sure it won't make any difference ... but it is a change from current prod and I'm not 100% sure how to test that it's not going to break something | 17:13 |
| mordred | I'm 99% sure it won't | 17:14 |
| * mordred doesn't feel strongly - mostly just explaining thinking - happy to change it if that's what we want | 17:15 | |
| clarkb | I don't feel strongly and as long as the tables explciitly set the 4 byte type in their schemas we shouldn't regress | 17:16 |
| mordred | clarkb: also - if it breaks, we can always change the settings and re-restore the dump :) | 17:17 |
| * mordred is going to pull the trigger | 17:17 | |
| mordred | clarkb, fungi : also - maybe it's time to land some more of these: https://review.opendev.org/#/q/topic:infra-prod-zuul+status:open ? | 17:33 |
| clarkb | mordred: is there a change to siwtch to a more frequent periodic pipeline yet? I'm looking at the next one in the stack and it is for zuul-preview https://review.opendev.org/#/c/717053/1 and if we want image updates (which just happened for zuul-preview) we need it to run semi frequently on a timer | 17:36 |
| clarkb | oh thats https://review.opendev.org/#/c/717064/3/.zuul.yaml should we maybe stick that lower in the stack? | 17:36 |
| mordred | clarkb: want me to rebase it for that? or we could just land the whole stack up til that one too | 17:39 |
| clarkb | mordred: I worry that if we run into problems as we go that we might miss that? doesn't look like much of the stack has reviews atthis point so we don't lose much in a rebase (do have to wait for jobs to rerun though) | 17:41 |
| clarkb | we can land some and see where we end up I guess | 17:42 |
| fungi | though to take the zuul-preview case as an example, we'd eventually trigger the deployment from promote after the image upload too, right? | 17:43 |
| fungi | so that we eventually reach the instant-gratification model | 17:43 |
| clarkb | mordred: for review I'm looking at the dep list at https://review.opendev.org/#/c/717054/1/.zuul.yaml do we need to set up dependences between manage-rpojects too? | 17:44 |
| fungi | i guess that depends on the earlier discussion about triggering deployments from events for other projects | 17:44 |
| clarkb | fungi: we won't because zuul image builds happen in a different tenant | 17:44 |
| fungi | ahh, yeah that too | 17:44 |
| clarkb | which is fine we can continue to just have an hourly cron for external dep updates | 17:44 |
| fungi | so zuul-preview is not really a case of an internal dep, right | 17:45 |
| fungi | because we're not building our own images, we're using the zuul images | 17:45 |
| mordred | yeah | 17:46 |
| clarkb | mordred: left the question about deps on https://review.opendev.org/#/c/717054/1 after soem thought I think the semaphore avoids any major issues I can think with that | 17:46 |
| mordred | clarkb: re: manage-projects ... hrm | 17:46 |
| mordred | yeah | 17:46 |
| clarkb | mordred: so I +2'd it with a note | 17:46 |
| mordred | clarkb: I've got a rebased stack locally if you want me to push it up (with the hourly pushed earlier) | 17:47 |
| openstackgerrit | Merged opendev/system-config master: Mention new mailing lists https://review.opendev.org/717824 | 17:47 |
| clarkb | mordred: maybe push it up between afs and puppet else? | 17:47 |
| clarkb | seems like puppet else is another good pause point since thats a bit of a change functioanlly | 17:47 |
| mordred | nah - I rebased it for right after zuul-preview | 17:47 |
| clarkb | that works too if you want, I've only gotten to review so ar | 17:47 |
| mordred | so we can start off with zuul-preview going hourly | 17:47 |
| mordred | kk. one sec | 17:48 |
| clarkb | ++ | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zuul and nodepool related deploys hourly https://review.opendev.org/717064 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run review and review-dev in zuul https://review.opendev.org/717054 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gitea in zuul https://review.opendev.org/717055 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run AFS in zuul https://review.opendev.org/717056 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run remote-puppet-else in zuul https://review.opendev.org/717057 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove run_all.sh and ansible cron job https://review.opendev.org/717058 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove ansible-cron role https://review.opendev.org/717059 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trigger everything on inventory changes https://review.opendev.org/717114 | 17:48 |
| clarkb | and I'll have to switch gears to meeting prep soon enough so biting off a reasonable chunk here seems worthwhile | 17:48 |
| mordred | fungi: https://review.opendev.org/#/c/717064 look ok to you? | 17:57 |
| fungi | mordred: yep, that's pretty straightforward | 18:05 |
| mordred | woot | 18:05 |
| openstackgerrit | Merged opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 18:06 |
| openstackgerrit | Merged opendev/system-config master: Run zuul-preview in zuul https://review.opendev.org/717053 | 18:06 |
| *** hashar has quit IRC | 18:10 | |
| mnaser | infra-root: http://site.925bfe37815144d0859f260605d5fb98.zuul.zuul-preview.opendev.org is giving me an internal server error -- anyone happen to be able to pull out the error message? | 18:19 |
| * mordred looking | 18:20 | |
| clarkb | mnaser: https://zuul.opendev.org/t/zuul/build/925bfe37815144d0859f260605d5fb98 I haven't looked at the error message but that doesn't seem to have a url | 18:21 |
| clarkb | (is that a rendering issue?) | 18:22 |
| mordred | wow. it straight up got a core | 18:22 |
| mordred | [Tue Apr 07 17:15:30.780412 2020] [core:notice] [pid 15:tid 140579200193792] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND -e info' | 18:22 |
| mordred | also - we should bind-mount a logs dir | 18:22 |
| clarkb | mordred: the zuul-preview does just assume that artifacts[i][url] will be present if the type matches | 18:23 |
| clarkb | mordred: which could result in a core dump? | 18:23 |
| clarkb | (I point that out becuse we don't render a url entry for that artifact in the link above) | 18:23 |
| mordred | I agree | 18:24 |
| mnaser | oh, uh | 18:24 |
| mnaser | the type wasn't even picked up -- https://zuul.opendev.org/api/tenant/zuul/build/30de77395f8b4ade9b1489cf4de24559 | 18:24 |
| mordred | remote: https://review.opendev.org/718186 Check to make sure artifact has a url | 18:25 |
| mordred | clarkb, corvus : ^^ | 18:25 |
| clarkb | mnaser: thats a different build | 18:25 |
| mnaser | oh, yes, you're right | 18:25 |
| clarkb | https://zuul.opendev.org/api/tenant/zuul/build/925bfe37815144d0859f260605d5fb98 does how the url | 18:26 |
| clarkb | so its a rendering issue | 18:26 |
| mordred | nod | 18:26 |
| clarkb | ok not that problem then :) | 18:26 |
| mnaser | https://zuul.opendev.org/api/tenant/zuul/build/30de77395f8b4ade9b1489cf4de24559 | 18:26 |
| clarkb | mordred: we probably do still want to check the validity of the inputs though for nicer logging :) | 18:26 |
| mnaser | maybe i am doing something silly | 18:26 |
| clarkb | mnaser: why do you keep looking at that other build? | 18:27 |
| mnaser | uhh, no idea. | 18:27 |
| mnaser | ah, ok | 18:28 |
| clarkb | mnaser: that second build is from yesterday | 18:28 |
| mordred | clarkb: yeah - I think that patch is still correct, but I don't know that it fixes this | 18:28 |
| clarkb | I'm not sure it is new enough to have the info you want | 18:28 |
| mnaser | my browser had "preserve log" option selected | 18:28 |
| mnaser | so it wasn't wiping the request history when i was F5ing | 18:29 |
| mordred | this zuul-preview has been running for 4 days | 18:29 |
| mordred | gah | 18:29 |
| mordred | this zuul-preview has been running for 4 hours | 18:29 |
| mordred | so - perhaps we haven't restarted since the new patch landed? | 18:29 |
| clarkb | mordred: ok so maybe just needs to be updated to the new code | 18:29 |
| mnaser | oh so maybe it hasn't picked up the change | 18:29 |
| mnaser | it _should_ have been updated by now | 18:29 |
| mnaser | oh, maybe it's in the emergency file? | 18:29 |
| mordred | we're running on sha 5a13fb00de7a5387af917943982a9be832bf5c200c432b8e66ad5325a8128916 | 18:29 |
| mordred | (docker sha) | 18:30 |
| clarkb | mnaser: corvus removed it, but we're also changing it from cron to cd (and need cron for image deployment) | 18:30 |
| clarkb | its possible it just got caught in that | 18:30 |
| mordred | latest image is 216be64cf86cd46cb56ee577909b0a189e33b3825e0194615b6b23f4104e5330 | 18:30 |
| mordred | maybe I should do a compose up? | 18:31 |
| mnaser | oh ok, sorry i haven't followed that effort too closely | 18:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Bind-mount apache log dir to zuul-preview https://review.opendev.org/718187 | 18:33 |
| mordred | clarkb: want me to do a manual restart? | 18:34 |
| clarkb | mordred: we are landing the cron right? shoudl we wait for that to ensure its working? | 18:34 |
| mordred | maybe so, yeah | 18:34 |
| mordred | mnaser: this is all part of "make things so that people who are not infra-root can see more of what's going on" | 18:35 |
| clarkb | mordred: are the uid and gid stable in https://review.opendev.org/#/c/718187/1/playbooks/roles/zuul-preview/tasks/main.yaml ? | 18:35 |
| clarkb | mordred: does that depend on the order software is installed? | 18:35 |
| mnaser | mordred: i figured so, that's pretty good. it'll make life easier | 18:35 |
| mordred | clarkb: they're just the uid/gid of the www-data user - I'm not sure how stable that is | 18:36 |
| clarkb | mordred: ya I'm trying to noodle on that. We can't say user: www-data beacuse that is form the hosts context | 18:36 |
| mordred | yup | 18:36 |
| clarkb | but the container could be rebuilt and change the uid | 18:36 |
| mordred | yup | 18:36 |
| mordred | I mean - I don't think it's LIKELY that that happens | 18:36 |
| clarkb | thinking out loud here, would it be better to have logs go to journald? | 18:37 |
| clarkb | (via stdout/stderr?) | 18:37 |
| mordred | clarkb: it's not a contiguous set of ids | 18:37 |
| mordred | clarkb: so it's not like "33 is the next in the list after 32" | 18:37 |
| mordred | I thnk it might be a base debian user | 18:37 |
| clarkb | ah so maybe ubuntu does fix that | 18:37 |
| mnaser | i think www-data being 33 is like a _thing_ | 18:37 |
| clarkb | or debian in this case | 18:37 |
| mordred | yeah | 18:37 |
| clarkb | k | 18:37 |
| mordred | in fact, given that we also run on ubuntu - we could PROBABLY re-write that as www-data | 18:37 |
| mordred | in fact, why don't we do that | 18:38 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add IRC logs and ML subscribe links to opendev.org https://review.opendev.org/718188 | 18:38 |
| mordred | NEVERMIND | 18:38 |
| clarkb | fungi: wasn't there a change that already did ^ | 18:38 |
| mnaser | im not sure if the apache2 pkgs add that | 18:38 |
| mordred | logs are opened as root before the the user changes | 18:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Bind-mount apache log dir to zuul-preview https://review.opendev.org/718187 | 18:39 |
| clarkb | oh its links this time got it | 18:39 |
| mordred | dont' need to set ownership at all | 18:39 |
| fungi | clarkb: yeah, i realized in looking at the page just now that we give people the e-mail address of, like, the announcements ml, but that's probably not as useful on its own | 18:39 |
| fungi | leaving figuring out how to find where to subscribe that as an exercise for the reader was probably not ideal | 18:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zuul and nodepool related deploys hourly https://review.opendev.org/717064 | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run review and review-dev in zuul https://review.opendev.org/717054 | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gitea in zuul https://review.opendev.org/717055 | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run AFS in zuul https://review.opendev.org/717056 | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run remote-puppet-else in zuul https://review.opendev.org/717057 | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove run_all.sh and ansible cron job https://review.opendev.org/717058 | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove ansible-cron role https://review.opendev.org/717059 | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trigger everything on inventory changes https://review.opendev.org/717114 | 18:41 |
| mordred | mnaser: https://review.opendev.org/717064 is the thing for zuul-preview to run hourly | 18:41 |
| clarkb | as I prep for the meeting today, what are thoughts on #startmeeting name? stick with infra for now then rename later in a batch to keep logs around or? | 18:41 |
| mordred | clarkb: yeah | 18:42 |
| mnaser | mordred: ok cool, will i have to wait for that to land and all? | 18:42 |
| mordred | clarkb: or - we could do opendev today | 18:42 |
| mordred | clarkb: and keep the old ones as infra | 18:42 |
| mordred | if we want to track the historical name | 18:42 |
| * mordred could go either way | 18:42 | |
| clarkb | mordred: actually since the schedule still calls us infra (it probably has the channel wrong too I'll update that) I think sticking with infra today may be best (its where people will look for logs I expect) | 18:43 |
| clarkb | thenwe can cahnge it in the future if necessary | 18:43 |
| mordred | mnaser: yeah - sorry - I think since zuul-preview changes slowly this is actually a good example case to make sure the automation is working | 18:43 |
| mordred | clarkb: ++ | 18:43 |
| mordred | mnaser: otherwise we might have to do this all again next time :) | 18:43 |
| mordred | (and sorry for the delay) | 18:43 |
| mnaser | that's fine, so step #1 is wait for that change to land and step #2 -- give it an hour or two? | 18:44 |
| mordred | yeah | 18:44 |
| mnaser | ok cool, i'll work on some other stuff and then let this run | 18:45 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run review and review-dev in zuul https://review.opendev.org/717054 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gitea in zuul https://review.opendev.org/717055 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run AFS in zuul https://review.opendev.org/717056 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run remote-puppet-else in zuul https://review.opendev.org/717057 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove run_all.sh and ansible cron job https://review.opendev.org/717058 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove ansible-cron role https://review.opendev.org/717059 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trigger everything on inventory changes https://review.opendev.org/717114 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update typo on infra-prod-service-letsencrypt https://review.opendev.org/718189 | 18:46 |
| mordred | clarkb, fungi : please to see https://review.opendev.org/718189 | 18:47 |
| mordred | https://review.opendev.org/#/c/717053 <-- second to last zuul comment is what got me there | 18:48 |
| mordred | clarkb: also - I had to rebase a merge conflict out so could you re:+2 https://review.opendev.org/#/c/717054 ? | 18:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove leftover /var/run dir creation https://review.opendev.org/717132 | 18:53 |
| clarkb | mordred: done | 18:58 |
| corvus | mordred, clarkb: apache logs go to stdout, so docker logs gets them | 19:00 |
| mordred | corvus: not the error logs | 19:01 |
| mordred | corvus: at least - I found things by looking in /var/log/apache2 inside the ocntainer that docker logs did not show me | 19:01 |
| corvus | oh neat | 19:01 |
| mordred | yeah | 19:02 |
| corvus | well, i could go either way on bind mounting them or also redirecting them to stdout | 19:02 |
| corvus | does something rotate them? | 19:02 |
| mordred | no - we'd need to add that | 19:02 |
| corvus | so right now, the error logs could fill up the container | 19:03 |
| corvus | though it only has 2 lines in it, so it would happen very slowly | 19:04 |
| mordred | corvus: yeah - eventually in the fullness of time | 19:04 |
| mordred | and I mean we should only error log when we throw a core | 19:04 |
| corvus | oh it's meeting time | 19:05 |
| mordred | corvus: crap - I forgot - what's teh new location? | 19:05 |
| corvus | #opendev-meeting | 19:05 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 19:07 |
| ianw | "For an example of OpenID Connect at work, look at Google+ Sign-In, Google’s flagship social-identity offering" ... that didn't age well | 19:39 |
| fungi | heh | 19:39 |
| *** ralonsoh has quit IRC | 19:40 | |
| mordred | haha | 19:44 |
| clarkb | I had to delete an index with a corrupted translog from elasticsearch. It was just bouncing that shard around to all the nodes trying to recover it and failing | 19:54 |
| clarkb | once I did that the cluster went green and now it is rebalancing onto es02 (which had crashed) | 19:54 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add vexxhost/ansible-role-base-server https://review.opendev.org/718199 | 19:56 |
| openstackgerrit | Merged opendev/system-config master: Run zuul and nodepool related deploys hourly https://review.opendev.org/717064 | 20:00 |
| clarkb | I need to find lunch now | 20:00 |
| fungi | yeah, firing up the wok now | 20:02 |
| mordred | mnaser: ^^ | 20:04 |
| mordred | mnaser: we should see the zuul-preview job run within the next hour | 20:05 |
| mnaser | mordred: cool! i'm excited :D | 20:05 |
| * mnaser feels like they're opening all the old boxes of stuff that was never quite polished | 20:05 | |
| * mordred appreciates the adding of the polish! | 20:09 | |
| openstackgerrit | Merged opendev/system-config master: Remove leftover /var/run dir creation https://review.opendev.org/717132 | 20:09 |
| mordred | fungi, ianw : not to re-adjudicate things we've already thought about several times .... | 20:19 |
| mordred | but I just had a thought | 20:19 |
| openstackgerrit | Merged opendev/system-config master: Update typo on infra-prod-service-letsencrypt https://review.opendev.org/718189 | 20:19 |
| mordred | the new python-stow element does a nice job of providing a non-invasive mechansim for pre-installing python versions that is still behind a decent enough interface that we can check for it in zuul-jobs | 20:20 |
| mordred | what if we did similar for tox and pip -so that ensure-tox could check to see if there is a tox in stow and if so "install" it via stow | 20:21 |
| mordred | (I thin this builds on top of waht ianw has been talking about with the pip/virtualenv rework) | 20:21 |
| mordred | because we could do "python -m venv /usr/local/stow/tox ; /usr/local/stow/tox/bin/pip install tox" - then have ensure-tox look to see if can use stow to install that tox. | 20:22 |
| mordred | WAIT | 20:22 |
| mordred | nevermind | 20:22 |
| mordred | ignore all of those words | 20:22 |
| mordred | that won't work at all - our previous plans are still better | 20:22 |
| * mordred goes back into hiding | 20:22 | |
| *** hashar has joined #opendev | 20:38 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 20:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 20:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-pip: export ensure_pip_virtualenv_command https://review.opendev.org/718224 | 20:39 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] use ensure-pip in fetch-subunit-output test https://review.opendev.org/718225 | 20:39 |
| ianw | mordred: if i had to distill my current thinking, which i reserve the right to change :) ... | 20:40 |
| ianw | zuul-jobs roles currently use pip: and frequently the virtualenv: field of pip: without consideration of pip, it's dependencies, or virtualenv being installed | 20:41 |
| ianw | so, we add a ensure-pip role that installs pip, from packages, and exports a variable that works as a virtualenv_command (usually, "/usr/bin/python3 -m venv") | 20:42 |
| ianw | if "/usr/bin/python3 -m venv" doesn't exist on the host (and at this point, I can't think of an environment we support that it doesn't) it exports "virtualenv" and every role is in the same position it is in now; running with the hope/assumption that virtualenv is installed and working | 20:44 |
| fungi | presumably we need to recommend somewhere that people incorporate the python3-venv package into their debian derivative (ubuntu, et cetera) images, since it generally isn't installed by default | 20:46 |
| clarkb | the outdoors are not quite as warm as I had hoped | 20:46 |
| ianw | fungi: i think that (and curretly done in wip changes) the ensure-pip role can reasonably install that package as a pip dependency | 20:47 |
| fungi | got it | 20:48 |
| ianw | fungi: although, on our systems, it does not install as such, because it's already there on dib images | 20:48 |
| fungi | granted, people don't need system-context pip installed to have system-context venv | 20:48 |
| ianw | what i *don't* want to get this role into is the game of installing virtualenv ... because i feel this is ambiguous. "virtualenv" is ambiguous between python2 and 3 | 20:49 |
| fungi | granted, if folks want venv and no system-wide pip, we can always break that role up further | 20:50 |
| clarkb | "Elsa never gets cold" I've been told to deal wiht it being chilly | 21:09 |
| corvus | clarkb: i left a little -1 on https://review.opendev.org/718161 | 21:09 |
| clarkb | corvus: k will update | 21:09 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Improving logging of prod playbook jobs https://review.opendev.org/718161 | 21:10 |
| clarkb | corvus: ^ fixed | 21:10 |
| corvus | clarkb: did you attempt to explain how you and elsa may be different? or would that only further highlight your failings? | 21:10 |
| clarkb | corvus: ya I think that would only further highlight my failings | 21:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 21:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-pip: export ensure_pip_virtualenv_command https://review.opendev.org/718224 | 21:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] fetch-zuul-cloner: use ensure-pip https://review.opendev.org/717882 | 21:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] use ensure-pip in fetch-subunit-output test https://review.opendev.org/718225 | 21:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 21:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 21:11 |
| corvus | "A-- not elsa" | 21:11 |
| mordred | corvus: if I get the error: Exception: Job infra-prod-service-nodepool depends on infra-prod-update-system-config which was not run. | 21:26 |
| mordred | corvus: because we don't have that in the opendev-prod-hourly pipeline - obviously I can (and should) add it to the pipeline | 21:27 |
| mordred | corvus: however - there are three other soft depends - it's safe to not add them to the pipeline because they're soft? or do I need to change the dependencies in that pipeline invocation? | 21:27 |
| corvus | mordred: i think you need to change the deps | 21:28 |
| mordred | corvus: ok. | 21:28 |
| corvus | i believe that error comes when it's constructing the job graph for the item, so it wants to see everything in the graph, even if the links are optional | 21:29 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add sytem-config-update and remove other deps from hourly https://review.opendev.org/718230 | 21:30 |
| mordred | corvus, clarkb : ^^ that should fix our hourly pipeline entries | 21:30 |
| clarkb | mordred: that is making me wonder if we should put all the deps into the pipelines? | 21:32 |
| openstackgerrit | melanie witt proposed openstack/project-config master: Update grafana for ceph to reflect current jobs https://review.opendev.org/718232 | 21:32 |
| clarkb | as variants I mean | 21:32 |
| mordred | clarkb: yeah- we've been waffling back and forth on that as to which thing is better | 21:40 |
| mordred | clarkb: I think I'm starting to waffle back to "let's put them all in the pipelines" - because the deps really do frequently depend on the pipeline it's in in this case | 21:41 |
| mordred | that said - we'll lose the ability to get the common deps that are shared by the majority of these jobs | 21:41 |
| mordred | so there will be a lot more yaml anchor :) | 21:41 |
| mordred | clarkb: I'll make a patch that does that on the end of the stac | 21:42 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] fetch-zuul-cloner: use ensure-pip https://review.opendev.org/717882 | 21:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] use ensure-pip in fetch-subunit-output test https://review.opendev.org/718225 | 21:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 21:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 21:45 |
| openstackgerrit | Merged opendev/system-config master: Add IRC logs and ML subscribe links to opendev.org https://review.opendev.org/718188 | 21:49 |
| corvus | mordred: the etherpad.opendev.org patch merged; what's next on that? | 21:55 |
| mordred | corvus: 2 things | 21:56 |
| mordred | corvus: we need to run the playbook - because the job had an error in its dependencies | 21:56 |
| mordred | corvus: (this is coming to be a running refrain) | 21:56 |
| mordred | but I think we've fixed that now | 21:57 |
| clarkb | mordred: have you been doing that via zuul enqueue or running the playbook on bridge directly? | 21:57 |
| mordred | and then there is a db dump from production I was going to run in | 21:57 |
| mordred | clarkb: so far neither - when we fix things I've either waited for a periodic run or a new natural triggering event | 21:57 |
| mordred | in this case it's likely more expedient to run the playbook by hand | 21:57 |
| clarkb | rgr | 21:58 |
| mordred | so - shall I do that real quick? | 21:58 |
| corvus | mordred: ++ | 21:58 |
| mordred | running | 21:58 |
| mordred | I have stopped running it | 21:59 |
| mordred | I'm going to send up a patch | 21:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Only run etherpad playbook on new server https://review.opendev.org/718235 | 22:00 |
| mordred | clarkb, corvus : ^^ that's important :) | 22:00 |
| fungi | okay, done making/eating dinner, back to pitch in | 22:00 |
| corvus | then i think we can test it out -- then do we shut down prod etherpad, add a cname, and do the db dump/load? maybe that should be scheduled? or we could just do it real quick like some afternoon evening? | 22:00 |
| fungi | just in time it seems! | 22:00 |
| clarkb | corvus: it sounded like fungi was saying shceduling it would be a good idea | 22:00 |
| corvus | mordred: comment on that | 22:00 |
| corvus | looks like etherpad.openstack.org has a 3600 ttl; we should lower that to 300 | 22:01 |
| corvus | then we can take a 5 min outage to deal with the ttl and the db dump | 22:02 |
| fungi | scheduling mainly so we can announce it, in case there are unexpected behavior changes with the new version. but maybe we can just announce it when we do it and tell folks to let us know if there are problems | 22:02 |
| corvus | i'll go lower the ttl in rax | 22:02 |
| mordred | corvus: I'm not sure which syntax is correct there - let's just do it with etherpad01 for now - and switch it back to the etherpad group when it's done | 22:02 |
| corvus | mordred: k | 22:02 |
| mordred | I think that's liekly to happen before we grow an 02 | 22:02 |
| mordred | fwiw - nothing new was installed on etherpad01.openstack.org - I caught the ansible quickly enough | 22:02 |
| mordred | in positive news - that patch should run correctly in zuul | 22:03 |
| mordred | in case I'm AFK when taht playbook runs and someone wants to run in the dump - it's in /opt/db on etherpad01.opendev.org | 22:04 |
| mordred | we should be able to do cat /opt/db/etherpad.sql | docker-compose exec mariadb mysql | 22:05 |
| mordred | speaking of - I'm now going to AFK for the evening quarantine walk | 22:06 |
| openstackgerrit | Merged opendev/system-config master: Improving logging of prod playbook jobs https://review.opendev.org/718161 | 22:07 |
| corvus | #status log lowered etherpad.openstack.org dns ttl to 300 seconds | 22:07 |
| openstackstatus | corvus: finished logging | 22:07 |
| corvus | mordred: +2 on the whole infa-prod-zuul stack | 22:12 |
| clarkb | corvus: mordred why doesn't the puppet remote else job depend on update system-config? | 22:27 |
| clarkb | left that as a question on the change (I think we do want that dep since a lot of our puppet is in system-config) | 22:29 |
| fungi | i thought we already covered that | 22:29 |
| fungi | reminding myself what the answer was... | 22:29 |
| fungi | oh, i think it was because it will have already run first on its own but i'll need to revisit why that was | 22:31 |
| clarkb | fungi: thats the case across different changes but in this case you would update puppet for something and you'd need system-config to be updated on bridge for that specific change I think | 22:31 |
| openstackgerrit | Merged opendev/system-config master: Add sytem-config-update and remove other deps from hourly https://review.opendev.org/718230 | 22:32 |
| corvus | clarkb: oh that's a good point... i think maybe you're right, so i removed my +2 | 22:33 |
| clarkb | it conflicts with that change that just merged anyway so new ps if necessary isn't a big deal :) | 22:33 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: use ensure-pip role https://review.opendev.org/717663 | 22:35 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 22:35 |
| *** DSpider has quit IRC | 22:51 | |
| openstackgerrit | Merged opendev/system-config master: Only run etherpad playbook on new server https://review.opendev.org/718235 | 22:52 |
| *** hashar has quit IRC | 22:56 | |
| ianw | clarkb: do you remember what the inspiration for the tox env installed via element was -> https://review.opendev.org/#/c/713017/ ? | 23:08 |
| clarkb | ianw: I think it was just to have common tools preinstalled without polluting the global namespace | 23:11 |
| clarkb | ianw: so we can have tox there ready to go without breaking the system package manager | 23:11 |
| ianw | clarkb: is there a reason we can't do that in zuul-jobs? | 23:12 |
| clarkb | ianw: I'd like to avoid hitting pypi that much | 23:12 |
| clarkb | we don't necessarily need to, but if we can identify common things and take some of that load off of pypi I think it is a good thing | 23:13 |
| clarkb | (particularly after their plea to have more mirrors out there) | 23:13 |
| ianw | we should be going via our cache, though? | 23:13 |
| clarkb | ianw: the indexes have a really short ttl so we hit them (that is where we typically see our issues these days) | 23:13 |
| *** mlavalle has quit IRC | 23:15 | |
| clarkb | the biggest downside is probably the cases where upstream releasing new versions that break us causesome disruption | 23:16 |
| clarkb | and we need to spin out new images rather than update a flag on the jobs | 23:16 |
| ianw | ok, gonna have to think on this ... thanks | 23:18 |
| clarkb | fwiw I can probably be convinced the other way around is better too. It is entirely possible my thinking is not sustainable :) | 23:19 |
| clarkb | for us in this particular case I think its balancing job runtime and pypi requests against being able to quickly change the version of what is installed | 23:20 |
| *** tosky has quit IRC | 23:26 | |
| fungi | i get the desire to be pragmatic and pre-cache certain bits of software on images which are heavily used by lots of the jobs we run, but i do wonder where the line is and how we decide when this tool or that tool should be preinstalled | 23:42 |
| fungi | especially as opendev's horizon's broaden and openstack (or python software in general) becomes less of a percentage of what we're testing | 23:43 |
| fungi | is there a point at which we drop tox, or at which we start adding compilers and interpreters for other languages? | 23:44 |
| clarkb | ya I think tox is a bit special becaus its basically the version of make most of our users use (and I think we preinstlal make make as well) | 23:44 |
| clarkb | its definitely an optimization for a significan't chunk of the jobs we run | 23:45 |
| clarkb | fungi: personally I've long wished we would just use make :) | 23:45 |
| clarkb | I even wrote a makefile to approximate tox things | 23:45 |
| fungi | yeah, tox seems like an overly-python-specific make without the flexibility or decades of stability behind it | 23:46 |
| mnaser | hmm | 23:59 |
| mnaser | http://site.925bfe37815144d0859f260605d5fb98.zuul.zuul-preview.opendev.org still 500s -- would someone be nice enough to check if the playbook was updated? | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!