| opendevreview | Merged opendev/system-config master: Create ircbot container https://review.opendev.org/c/opendev/system-config/+/793704 | 01:04 |
|---|---|---|
| *** bhagyashris_ has joined #opendev | 01:12 | |
| *** bhagyashris has quit IRC | 01:20 | |
| *** whayutin_ has joined #opendev | 01:31 | |
| *** weshay|ruck has quit IRC | 01:38 | |
| *** timburke has quit IRC | 02:00 | |
| opendevreview | Merged opendev/system-config master: limnoria/meetbot setup on eavesdrop01.opendev.org https://review.opendev.org/c/opendev/system-config/+/793719 | 02:04 |
| *** boistordu_ex has quit IRC | 02:18 | |
| ianw | it looks like base is failing to deploy with issues with apt-get autoremove | 02:34 |
| ianw | dpkg: error processing package shim-signed (--configure): | 02:35 |
| *** ysandeep has joined #opendev | 02:38 | |
| ianw | Setting up grub-efi-amd64-signed (1.167~18.04.3+2.04-1ubuntu44.1) ... | 02:39 |
| ianw | Installing for x86_64-efi platform. | 02:39 |
| ianw | grub-install: error: cannot find EFI directory. | 02:39 |
| ianw | which probably isn't surprising given we don't boot efi | 02:40 |
| prometheanfire | sqlalchemy bump just merged, so if gates fail, may be that (tested via cross tests, but...) | 02:51 |
| *** ysandeep has quit IRC | 02:52 | |
| ianw | https://launchpad.net/ubuntu/+source/grub2-signed/1.167~18.04.3 | 03:00 |
| ianw | http://launchpadlibrarian.net/541948832/grub2-signed_1.167~18.04.1_1.167~18.04.3.diff.gz | 03:00 |
| ianw | i feel like it must have something to do with that | 03:00 |
| *** whayutin_ has quit IRC | 03:27 | |
| *** whayutin has joined #opendev | 03:31 | |
| ianw | i have filed https://bugs.launchpad.net/ubuntu/+source/grub2-signed/+bug/1931518 | 04:01 |
| opendevmeet | Launchpad bug 1931518 in grub2-signed (Ubuntu) "grub-efi-amd64-signed 1.167~18.04.1+2.04-1ubuntu44 has broken non-EFI install" [Undecided,New] | 04:01 |
| ianw | our images have an unmounted /boot/efi directory which might be confusing things | 04:07 |
| *** ykarel has joined #opendev | 04:11 | |
| *** gthiemon1e has joined #opendev | 04:31 | |
| *** gthiemonge has quit IRC | 04:31 | |
| *** gthiemon1e has quit IRC | 04:32 | |
| *** gthiemonge has joined #opendev | 04:32 | |
| *** timburke has joined #opendev | 04:47 | |
| frickler | so the gate still looks pretty borked. iiuc we have a new zuul release but nobody got round to deploy it? should I give it a try? I'm also wondering whether in the current situation one would actually want to restore queues or better give it a fresh start | 04:52 |
| *** marios has joined #opendev | 05:00 | |
| *** marios is now known as marios|ruck | 05:01 | |
| *** whayutin has quit IRC | 05:03 | |
| *** jentoio has quit IRC | 05:06 | |
| *** jentoio has joined #opendev | 05:07 | |
| *** opendevreview has quit IRC | 05:26 | |
| *** ysandeep has joined #opendev | 05:42 | |
| *** timburke has quit IRC | 05:47 | |
| *** opendevreview has joined #opendev | 06:07 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: limnoria: fix some minor typos https://review.opendev.org/c/opendev/system-config/+/795729 | 06:07 |
| *** bhagyashris_ is now known as bhagyashris | 06:08 | |
| ianw | frickler: yeah, i'm afraid i just didn't keep up enough on that one. i do believe that corvus' fix merged | 06:12 |
| ianw | i might be worried about introducing other changes though without someone with more overview | 06:13 |
| frickler | ianw: yeah, the fix was merged and 4.5.0, but I didn't see any indication as to why it wasn't deployed, so I'm wary about that, too | 06:14 |
| frickler | ... and 4.5.0 released ... | 06:14 |
| ianw | i'm afraid i'm about out of time. i haven't managed to get meetbot switched over to eavesdrop01.opendev.org but i've identified and fixed up a few small issues to get there tomorrow hopefully | 06:18 |
| ianw | frickler: i don't know if i'm of any help. if you want to restart, i can be around in an hour or two | 06:21 |
| frickler | ianw: no, I don't think I want to touch it really, will wait for corvus' feedback, thx | 06:24 |
| frickler | I wouldn't have time to do extended debugging in case it wouldn't just work after a restart, too. and at least check jobs still mostly seem to be working fine | 06:26 |
| *** ralonsoh has joined #opendev | 06:54 | |
| *** rpittau|afk is now known as rpittau | 06:54 | |
| *** hashar has joined #opendev | 06:56 | |
| *** jpena|off is now known as jpena | 06:58 | |
| *** tosky has joined #opendev | 07:03 | |
| *** amoralej|off has joined #opendev | 07:04 | |
| *** david-lyle has joined #opendev | 07:04 | |
| *** amoralej|off is now known as amoralej | 07:06 | |
| *** lucasagomes has joined #opendev | 07:07 | |
| *** dklyle has quit IRC | 07:08 | |
| *** andrewbonney has joined #opendev | 07:20 | |
| opendevreview | Merged opendev/system-config master: limnoria: fix some minor typos https://review.opendev.org/c/opendev/system-config/+/795729 | 07:43 |
| *** boistordu_ex has joined #opendev | 09:07 | |
| *** artom has quit IRC | 09:16 | |
| *** artom has joined #opendev | 09:17 | |
| opendevreview | Ananya proposed opendev/elastic-recheck master: Configure ER bot https://review.opendev.org/c/opendev/elastic-recheck/+/795756 | 10:03 |
| *** jpena is now known as jpena|lunch | 11:37 | |
| yoctozepto | infra-root: ethercalc seemingly down | 11:40 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Rename x/tap-as-a-service to openstack/tap-as-a-service https://review.opendev.org/c/openstack/project-config/+/790093 | 11:42 |
| *** ysandeep is now known as ysandeep|afk | 11:44 | |
| *** hjensas|afk has quit IRC | 11:49 | |
| *** whoami-rajat has quit IRC | 11:57 | |
| yoctozepto | it's back | 12:11 |
| *** ysandeep|afk is now known as ysandeep | 12:11 | |
| *** whayutin has joined #opendev | 12:14 | |
| *** jpena|lunch is now known as jpena | 12:34 | |
| *** amoralej is now known as amoralej|lunch | 12:34 | |
| *** yoctozepto is now known as Guest1570 | 12:36 | |
| *** Guest1570 is now known as yoctozepto | 12:36 | |
| *** ysandeep is now known as ysandeep|mtg | 13:08 | |
| corvus | frickler, ianw: i think we just ran out of time yesterday; i can deploy it today, though i'm also surprised if the situation got worse; it should have been a one-time event unless the cloud killed another executor vm | 13:13 |
| *** amoralej|lunch is now known as amoralej | 13:23 | |
| fungi | corvus: i don't know that it's gotten worse, the report in #openstack-infra was about changes stuck for more than a day, so probably leftovers from the same incident | 13:38 |
| fungi | "elapsed time 28 hours" as of 05:25 utc so yeah, that was the same incident | 13:39 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 13:40 |
| corvus | ok, thought i saw someone mention something about the gate pipeline | 13:40 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 13:41 |
| fungi | yeah, not sure what frickler meant by "the gate still looks pretty borked" in that case, there wasn't much explanation of which issue was suspected | 13:45 |
| fungi | my internet service is still down (yay!) so looking at stuff over 2g/3g cellular is a bit of a pain | 13:47 |
| fungi | there's a hung openstack/neutron item in the openstack tenant's gate pipeline for ~36 hours, so again same incident, but it doesn't seem to be blocking anything else from merging | 13:48 |
| *** artom_ has joined #opendev | 13:50 | |
| *** artom has quit IRC | 13:56 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 14:06 |
| *** kuku has joined #opendev | 14:14 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 14:19 |
| frickler | corvus: fungi: sorry for not being more specific. there was a large backlog in tripleo gate, though that looks better now. and then https://review.opendev.org/c/openstack/devstack-gate/+/795383 still didn't get into gate, although clarkb had a comment about that yesterday, but I can't see any trace of that in gerrit | 14:26 |
| opendevreview | Alex Schultz proposed opendev/system-config master: Add cdn0{1,2}.quay.io https://review.opendev.org/c/opendev/system-config/+/744038 | 14:27 |
| *** timburke has joined #opendev | 14:47 | |
| opendevreview | Hervé Beraud proposed openstack/project-config master: Adding missing zuul for etcd3gw https://review.opendev.org/c/openstack/project-config/+/795824 | 14:49 |
| *** lucasagomes has quit IRC | 15:08 | |
| *** tkajinam has quit IRC | 15:19 | |
| clarkb | frickler: https://review.opendev.org/c/openstack/devstack-gate/+/795383 went into the gate when I approved it | 15:26 |
| clarkb | frickler: it was at the end of the queue, maybe when it was properly evaluated it got evicted? I wonder if we have no jobs that apply when .rst files are updated? | 15:27 |
| clarkb | if we deploy 4.5.0 then we'll automatically deal with those changes that are stuck from the original event | 15:28 |
| clarkb | I have reapproved 795383 to see what it does | 15:28 |
| clarkb | 795383 shows up in the pipeline now, but without any jobs, lets see what it does | 15:29 |
| fungi | well, deploying 4.5.0 means restarting the scheduler, which means reenqueuing the changes, which means sure they're dealt with at that point | 15:30 |
| clarkb | fungi: yup exactly | 15:30 |
| fungi | the bug which led to them getting stuck would require another executor crashing/disconnecting/rebooting | 15:30 |
| fungi | to have a recurrence | 15:30 |
| fungi | i've seen no evidence of new incidents there | 15:30 |
| clarkb | looking at 795383 it shows no jobs. I strongly suspect that we've removed any job which will run against only an rst change | 15:31 |
| fungi | also, cable provider helpfully scheduled someone to come take a look at my internet outage... sunday afternoon | 15:31 |
| fungi | guess i'm going to be at half-capacity for a few more days | 15:32 |
| clarkb | the alternative for those stuck changes is to abandon and restore them or we can deuqeue enqueue? | 15:34 |
| frickler | clarkb: ah, we had that set of patches to move d-g jobs from project-config into the repo, maybe it missed something. so that would indeed be a completely unrelated issue then | 15:34 |
| clarkb | infra-root should I go get my keys loaded and dequeue enqueue those changes? | 15:34 |
| *** ysandeep|mtg is now known as ysandeep | 15:37 | |
| fungi | clarkb: if you want to, feel free, they're not blocking anything though and their owners can also push new patchsets or abandon/restore to achieve the same effect if it's urgent | 15:43 |
| kuku | Any solutions to update bios firmware with ironic ? | 15:45 |
| clarkb | kuku: you're probably best off asking the ironic team. We run the developer tools that help make ironic but don't have much direct experience with the tool | 15:46 |
| clarkb | fungi: it seems like a honey trap at this point causing people confusion. I'll go ahead and sort them out | 15:46 |
| fungi | kuku: you can usually find them in #openstack-ironic | 15:48 |
| clarkb | dequeue then enqueue seems to have worked for the oldest change in check. I'll do the others now | 15:51 |
| corvus | sorry, just had some surprise stuff come up; i should be able to restart later, but if you want to pick those off now that's fine | 15:56 |
| clarkb | corvus: no worries, I've taken care of those in check. There are a couple in periodic and one in check-arm64 that are a bit older than the incident so I think they may be stuck for other reasons | 15:56 |
| *** marios|ruck is now known as marios|out | 16:02 | |
| fungi | my guess there would either be no available nodes, or we've had more nodeset locks fail to get released by the launcher | 16:03 |
| clarkb | I'm looking into the d-g thing more closely now since I'm fairly sure the issue is lack of jobs to run on .rst file updates based on the state of the gate for it right now | 16:04 |
| clarkb | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L126 ya that seems to be common. Now to see if I can find one that doesn't exclude .rst that can be added | 16:08 |
| clarkb | I would expect devstack-gate-hooks to run since it doesn't seem to filter on files and its parent doesn't either | 16:12 |
| clarkb | oh it is only in check. Cool we can fix this | 16:13 |
| *** jpena is now known as jpena|off | 16:17 | |
| *** marios|out has quit IRC | 16:21 | |
| *** rpittau is now known as rpittau|afk | 16:24 | |
| *** ysandeep is now known as ysandeep|out | 16:38 | |
| melwitt | can anyone tell me what's the location/url of the gerrit rest api for review.opendev.org? | 16:41 |
| clarkb | melwitt: https://review.opendev.org/ and https://review.opendev.org/a/ for authenticated requests | 16:41 |
| clarkb | melwitt: note they prefix the json responses with invalid json that you have to chomp off | 16:43 |
| clarkb | )]}'\n is what they prefix with so 5 bytes you can just remove | 16:43 |
| melwitt | clarkb: oh, ok thanks! | 16:44 |
| clarkb | and for /a/ you have to use that url prefix when you are doing actions that require auth. If you supply auth to the non /a/ url you get 403s back and it is very confusing | 16:46 |
| clarkb | its also fine to use /a/ when not needing auth but you have to supply the auth if you use /a/ and can't query those urls anonymously | 16:47 |
| clarkb | fungi: you had asked about changes related to https://review.opendev.org/c/opendev/system-config/+/792247 recently, if your Internets are working well enough could you review that onet oo? | 16:54 |
| clarkb | *too | 16:54 |
| fungi | melwitt: also be aware you need to use http basic auth with this gerrit version (older versions used digest auth instead) | 16:56 |
| *** amoralej is now known as amoralej|off | 16:56 | |
| fungi | and if you want to look at an example piece of software which interfaces with the gerrit rest api, ttygroup/gertty is excellent... it does just about anything you can imagine via the api | 16:57 |
| melwitt | ok thanks | 16:57 |
| *** amoralej|off has quit IRC | 16:59 | |
| melwitt | oh good, I will need that :) | 17:01 |
| fungi | clarkb: yep, reviewing now. i basically ended up wiring the entire house into a cellular modem as a workaround, which is not great but i can manage | 17:07 |
| clarkb | wow, when I have those isseus I just pick a device to tether with and don't bother with doing the whole home. But it happens less for me so you probably want to have a more robust fallback | 17:10 |
| fungi | i switch between computers a lot, so being tied to one machine isn't as efficient for me | 17:10 |
| fungi | also tethering one computer via wifi means it's cut off from the other systems here | 17:11 |
| *** ykarel has quit IRC | 17:22 | |
| mnaser | if any other infra-root is around to +2 https://review.opendev.org/c/opendev/system-config/+/795602 so that ianw can +A in their AM and help add those mirrors? | 17:22 |
| clarkb | looking | 17:24 |
| *** david-lyle is now known as dklyle | 17:26 | |
| corvus | sanity check: it looks like we don't release jeepyb? is that correct? | 17:28 |
| clarkb | corvus: looking | 17:28 |
| corvus | i don't see it on pypi, and i see zero tags | 17:28 |
| clarkb | corvus: yup agreed it seems we don't release jeepyb, we just deploy it. But jeepyb consumes gerritlib from releases so depending on where things change we do sometimes make releases of gerritlib | 17:29 |
| corvus | k, thx | 17:29 |
| opendevreview | Merged opendev/system-config master: Remove system-config-legacy-logstash-filters job https://review.opendev.org/c/opendev/system-config/+/792247 | 17:29 |
| *** ysandeep|out has quit IRC | 17:43 | |
| opendevreview | Merged zuul/zuul-jobs master: Bump default Helm version to 2.17.0 https://review.opendev.org/c/zuul/zuul-jobs/+/795329 | 17:44 |
| corvus | manage-projects is a really good candidate for a container image deliverable (like we did for grafyaml) | 18:05 |
| clarkb | corvus: we bundle it onto the gerrit container now forsome reasonI don't recall | 18:06 |
| clarkb | its pssible that was for simplicitly and it isn't necessary | 18:07 |
| corvus | yeah, i'm assuming we're doing something semi-intrusive (like using the gerrit server keys). but i have a non-opendev use for just manage-projects, and that's well suited for running completely external | 18:07 |
| clarkb | oh I remember! | 18:07 |
| clarkb | it is because jeepyb also contains the gerrit hooks | 18:07 |
| clarkb | and gerrit expects those in its path | 18:07 |
| clarkb | For manage-projects we could split that out I think | 18:07 |
| clarkb | then also continue to have jeepyb in the gerrit image for hooks | 18:08 |
| corvus | ah yep the hooks; that'll do it | 18:12 |
| fungi | yeah, would make sense to install twice (once in gerrit container, once in manage-projects container) | 18:12 |
| corvus | i mean, even we're trying to get rid of the hooks | 18:13 |
| fungi | if someone reimplements those remaining hook scripts as zuul jobs, we can completely drop it from the gerrit container | 18:13 |
| clarkb | yup | 18:13 |
| fungi | yeah, precisely | 18:13 |
| *** slittle1 has quit IRC | 18:14 | |
| *** slittle1 has joined #opendev | 18:15 | |
| fungi | in addition to my current tin cans and string internet backup, the gutter installers finally arrived this afternoon, so i get to listen to hammering, drilling, and sawing sheet metal just on the other side of the wall | 18:39 |
| fungi | at least i don't have any conference calls this afternoon | 18:40 |
| clarkb | we can probably arrange one if you'd like to share the fun :) | 18:40 |
| fungi | oh, you still need to test you new dac, right? | 18:40 |
| fungi | "can you hear the reciprocating saw now? how about now?" | 18:41 |
| clarkb | I tested it with a local call. I'm satisfied it is working | 18:41 |
| fungi | cool | 18:41 |
| clarkb | also I'm on the couch today because parents are in my office | 18:41 |
| clarkb | so conference call today may not use it | 18:42 |
| clarkb | ianw: I have reviewed the statusbot change. Couple of notes there as well as I think there may be missing files (need to git add and repush?) | 18:59 |
| clarkb | ianw: also a comment on https://review.opendev.org/c/opendev/system-config/+/795707 but I +2'd it anyay as it should work as is | 19:13 |
| *** hashar has quit IRC | 20:00 | |
| *** andrewbonney has quit IRC | 20:00 | |
| *** ralonsoh has quit IRC | 20:56 | |
| *** SpamapS has quit IRC | 21:35 | |
| ianw | clarkb: thanks! | 21:43 |
| ianw | clarkb: my thought with the volume is that it is one thing that has very little context outside the container; logs and images we want visibility into, but the only reason for this is to workaround overlayfs issues | 21:47 |
| *** SpamapS has joined #opendev | 21:52 | |
| SpamapS | Well I think I'm here now? | 21:53 |
| ianw | SpamapS: it's too early for philosophy here :) | 21:54 |
| SpamapS | assuming time is linear, you're probably right. | 21:55 |
| SpamapS | Weirdly though, I can't seem to rejoin #ceph or #debian-mysql now that I'm on Matrix. :-P | 21:56 |
| opendevreview | Danni Shi proposed openstack/diskimage-builder master: Add a keylime-agent element and a tpm-emulator https://review.opendev.org/c/openstack/diskimage-builder/+/789601 | 22:03 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run statusbot from eavesdrop01.opendev.org https://review.opendev.org/c/opendev/system-config/+/795213 | 22:09 |
| corvus | spamaps: looks like #debian-mysql needs nick registration; you can give your registered nick password to the bridge, and it will authenticate for you. start a PM with @oftc-irc:matrix.org and say "!help" | 22:19 |
| ianw | i'm running syncs of meeting logs and channel logs now from eavesdrop01.openstack.org -> opendev.org | 22:19 |
| corvus | spamaps: "!storepass" is the command to save the password | 22:20 |
| opendevreview | Merged opendev/system-config master: Add Fedora 34 mirrors https://review.opendev.org/c/opendev/system-config/+/795602 | 22:27 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!