| *** bhagyashris_ has joined #openstack-infra | 00:31 | |
| *** bhagyashris has quit IRC | 00:38 | |
| *** benj_ has quit IRC | 01:37 | |
| *** yamak16 has joined #openstack-infra | 01:59 | |
| *** benj_ has joined #openstack-infra | 01:59 | |
| *** ramishra has quit IRC | 02:34 | |
| *** zxiiro has quit IRC | 02:36 | |
| *** ramishra has joined #openstack-infra | 02:41 | |
| *** brtknr has quit IRC | 03:00 | |
| *** brtknr has joined #openstack-infra | 03:00 | |
| *** rlandy|bbl is now known as rlandy | 03:14 | |
| *** rlandy has quit IRC | 03:14 | |
| *** ykarel|away has joined #openstack-infra | 04:37 | |
| *** ade_lee has quit IRC | 05:04 | |
| *** abhishekk has joined #openstack-infra | 05:11 | |
| *** abhishekk is now known as akekane|home | 05:12 | |
| *** ykarel|away is now known as ykarel | 05:14 | |
| *** slaweq_ has joined #openstack-infra | 06:00 | |
| *** slaweq has left #openstack-infra | 06:02 | |
| *** slaweq_ has quit IRC | 06:03 | |
| *** slaweq_ has joined #openstack-infra | 06:03 | |
| *** vishalmanchanda has joined #openstack-infra | 06:15 | |
| *** dklyle has quit IRC | 06:16 | |
| *** soniya29 has joined #openstack-infra | 06:21 | |
| *** soniya29 has quit IRC | 06:22 | |
| *** soniya29 has joined #openstack-infra | 06:25 | |
| *** soniya29 has quit IRC | 06:25 | |
| *** soniya29 has joined #openstack-infra | 06:26 | |
| *** bhagyashris_ is now known as bhagyashris | 06:33 | |
| *** soniya29 has quit IRC | 06:50 | |
| *** soniya29 has joined #openstack-infra | 06:55 | |
| *** rpittau|afk is now known as rpittau | 07:05 | |
| *** tosky has joined #openstack-infra | 07:23 | |
| *** andrewbonney has joined #openstack-infra | 07:32 | |
| *** slaweq has joined #openstack-infra | 07:36 | |
| *** slaweq_ has quit IRC | 07:41 | |
| *** akekane|home has quit IRC | 07:41 | |
| *** jpena|off is now known as jpena | 07:54 | |
| *** lucasagomes has joined #openstack-infra | 08:04 | |
| *** slaweq_ has joined #openstack-infra | 08:05 | |
| *** slaweq_ has quit IRC | 08:11 | |
| *** lucasagomes has quit IRC | 08:19 | |
| *** lucasagomes has joined #openstack-infra | 08:26 | |
| *** lucasagomes has quit IRC | 08:48 | |
| *** lucasagomes has joined #openstack-infra | 08:49 | |
| *** soniya29 has quit IRC | 08:57 | |
| *** lucasagomes has quit IRC | 09:05 | |
| *** jpward has quit IRC | 09:11 | |
| *** lucasagomes has joined #openstack-infra | 09:25 | |
| *** lucasagomes has quit IRC | 09:37 | |
| *** lucasagomes has joined #openstack-infra | 09:43 | |
| *** lucasagomes has quit IRC | 09:54 | |
| *** lucasagomes has joined #openstack-infra | 09:59 | |
| *** yamak16 has quit IRC | 10:21 | |
| *** soniya29 has joined #openstack-infra | 10:28 | |
| *** lucasagomes has quit IRC | 10:53 | |
| *** lucasagomes has joined #openstack-infra | 10:58 | |
| *** derekh has joined #openstack-infra | 11:05 | |
| *** soniya29 has quit IRC | 11:22 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** osmanlicilegi has quit IRC | 11:38 | |
| *** rlandy has joined #openstack-infra | 11:49 | |
| *** cgoncalves has quit IRC | 11:59 | |
| *** osmanlicilegi has joined #openstack-infra | 12:00 | |
| *** cgoncalves has joined #openstack-infra | 12:01 | |
| *** osmanlicilegi has quit IRC | 12:11 | |
| *** zul has joined #openstack-infra | 12:12 | |
| *** jpena|lunch is now known as jpena | 12:20 | |
| *** osmanlicilegi has joined #openstack-infra | 12:35 | |
| *** lucasagomes has quit IRC | 12:38 | |
| *** osmanlicilegi has quit IRC | 12:46 | |
| *** bhagyashris_ has joined #openstack-infra | 12:50 | |
| *** soniya29 has joined #openstack-infra | 12:52 | |
| *** rlandy is now known as rlandy|training | 12:54 | |
| *** osmanlicilegi has joined #openstack-infra | 12:55 | |
| *** lucasagomes has joined #openstack-infra | 12:56 | |
| *** bhagyashris has quit IRC | 12:56 | |
| *** bhagyashris_ is now known as bhagyashris | 12:58 | |
| *** cgoncalves has quit IRC | 13:14 | |
| *** cgoncalves has joined #openstack-infra | 13:15 | |
| *** rpittau is now known as rpittau|afk | 13:37 | |
| *** tdasilva_ has quit IRC | 13:37 | |
| *** tdasilva_ has joined #openstack-infra | 13:37 | |
| *** rlandy|training is now known as rlandy | 13:43 | |
| *** abhishekk has joined #openstack-infra | 13:44 | |
| *** soniya29 has quit IRC | 14:14 | |
| *** lucasagomes has quit IRC | 14:18 | |
| *** lucasagomes has joined #openstack-infra | 14:21 | |
| *** lucasagomes has quit IRC | 14:34 | |
| *** lucasagomes has joined #openstack-infra | 14:35 | |
| *** gmann is now known as gmann_afk | 14:40 | |
| *** lucasagomes has quit IRC | 14:46 | |
| *** dklyle has joined #openstack-infra | 14:46 | |
| *** lucasagomes has joined #openstack-infra | 14:50 | |
| *** gmann_afk is now known as gmann | 15:13 | |
| *** ykarel is now known as ykarel|away | 15:27 | |
| *** ade_lee has joined #openstack-infra | 15:37 | |
| ade_lee | hi all -- I've been running into an issue with centos 8 jobs for a few weeks now -- https://zuul.opendev.org/t/openstack/build/a6dce3efd4e246cc9a05012c90aa554a/log/controller/logs/devstacklog.txt#1134 | 15:39 |
|---|---|---|
| ade_lee | this is happening for jobs where the nodeset is devstack-single-node-centos-8 | 15:39 |
| ade_lee | any idea how this could be fixed? | 15:40 |
| ade_lee | fungi, clarkb ^^ not sure if you guys are the right ones to ask - or who would be .. | 15:40 |
| ade_lee | it was working until a few weeks back .. | 15:41 |
| fungi | ade_lee: any idea what the "openstack-wallaby" it wants to download metadata for is? | 15:45 |
| *** lucasagomes has quit IRC | 15:45 | |
| fungi | like, is that a channel from rdo or something? | 15:45 |
| fungi | i'm unfortunately not all that well versed in centos package management | 15:45 |
| fungi | it doesn't seem to say what "mirrorlist" it's complaining about either | 15:46 |
| fungi | maybe someone with deeper understanding of devstack would know | 15:46 |
| clarkb | wallaby is new right? is it possible that there is some sort of templating going on based on newer releases and that is incomplete for wallaby? but ya seems like a devstack question? | 15:47 |
| *** lucasagomes has joined #openstack-infra | 15:48 | |
| fungi | might get more traction with the devstack maintainers in #openstack-qa | 15:48 |
| ykarel|away | clarkb, ade_lee wallaby is centos8-stream only | 15:50 |
| ykarel|away | so if that's centos8 based job and looking for repo in centos8 it will fail | 15:50 |
| ykarel|away | http://mirror.centos.org/centos/8-stream/cloud/x86_64/openstack-wallaby/ vs http://mirror.centos.org/centos/8/cloud/x86_64/openstack-wallaby/ | 15:51 |
| fungi | got it, so switch the nodeset to be based on stream | 15:52 |
| fungi | maybe it should just be updated here https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L55 | 15:53 |
| fungi | since it's a devstack-specific nodeset, maybe the devstack maintainers want to decide that the devstack-single-node-centos-8 nodeset should use the centos-8-stream label | 15:54 |
| fungi | or maybe they want to create a separate devstack-single-node-centos-8-stream nodeset which jobs can switch to | 15:54 |
| fungi | though the only direct uses of that nodeset in master branches seem to be the cinder-tempest-plugin-lvm-lio-barbican-centos-8 and devstack-platform-centos-8 job definitions | 15:56 |
| fungi | ade_lee: anyway, worth re-raising as a discussion in #openstack-qa | 15:56 |
| ykarel|away | yes for wallaby release, good to move to stream node and see how it goes ^ | 15:56 |
| ade_lee | fungi, well - I'm trying to add a few fips related jobs -- so there will be a few more | 15:56 |
| ykarel|away | for other projects like packstack,tripleo,puppet openstack we already using c8-stream | 15:57 |
| fungi | ade_lee: in that case, better it's fixed before adding more jobs | 15:57 |
| clarkb | I suspect that for master devstack stream is the only centos-8 worth supporting now? | 15:57 |
| ade_lee | fungi, ack | 15:57 |
| clarkb | but that will be up to the qa team I suspect | 15:57 |
| ykarel|away | wallaby+ RDO is centos8-stream only | 15:57 |
| ykarel|away | and moving forward it will be c9-stream whenever it get's available | 15:58 |
| ade_lee | ykarel|away, ok - so to fix this -- I just need to change https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L55 to centos8-stream ? | 15:59 |
| ade_lee | I'll bring it up in openstack-qa - but I can put up a patch to see if this fixes things | 15:59 |
| fungi | ade_lee: or create a new nodeset similar to that one which uses stream and switch to using the new nodeset | 15:59 |
| clarkb | ade_lee: it might be most accurate to update the nodeset then update the jobs | 15:59 |
| ykarel|away | ade_lee, for testing u can add nodeset in ur project itself | 15:59 |
| ykarel|away | and later can change in devstack post discussion | 16:00 |
| fungi | yes, that too, or use depends-on to a wip devstack change for it | 16:00 |
| ykarel|away | yes exactly what fungi said | 16:00 |
| ykarel|away | just use label: centos-8-stream | 16:00 |
| ykarel|away | instead of centos-8 | 16:00 |
| ykarel|away | in custom nodeset u create | 16:01 |
| ade_lee | ok thanks - I'll give that a try and use a wip devstack change and bring up discussion in openstack-qa | 16:01 |
| ykarel|away | ack | 16:01 |
| * ykarel|away out | 16:01 | |
| ade_lee | thanks all | 16:02 |
| fungi | yw | 16:02 |
| *** lucasagomes has quit IRC | 16:04 | |
| *** rlandy is now known as rlandy|brb | 16:10 | |
| *** ykarel|away has quit IRC | 16:18 | |
| *** zul has quit IRC | 16:22 | |
| *** derekh has quit IRC | 16:34 | |
| *** jpena is now known as jpena|off | 16:38 | |
| *** rlandy|brb is now known as rlandy | 16:42 | |
| *** akekane_ has joined #openstack-infra | 16:44 | |
| *** abhishekk has quit IRC | 16:44 | |
| *** jlvillal has quit IRC | 16:48 | |
| *** abhishekk has joined #openstack-infra | 16:51 | |
| *** akekane_ has quit IRC | 16:52 | |
| dansmith | man, gerrit is seriously constipated today | 17:10 |
| yoctozepto | it is | 17:11 |
| fungi | pretty good load spike in progress at the moment: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=26&rra_id=all | 17:13 |
| dansmith | remote: error: internal error | 17:13 |
| dansmith | just got that from git-review just now | 17:13 |
| dansmith | after many minutes of spinning | 17:13 |
| fungi | yeah, that'll happen if it times out an internal request due to excessive load | 17:14 |
| fungi | i caught it yesterday when trying to push a patch series | 17:14 |
| dansmith | I think something is broken now, | 17:14 |
| dansmith | because if I re-review, it tells me nothing has changed so I can't submit | 17:14 |
| fungi | though all my changes did end up in gerrit, it just didn't emit any events so gerritbot and zuul didn't act on the uploads | 17:14 |
| dansmith | ah, okay maybe that's what happened | 17:14 |
| fungi | likely same for yours, yeah | 17:14 |
| fungi | check that the changes exist in gerrit, and add recheck comments on them for now | 17:15 |
| dansmith | my set was only 15 patches long | 17:15 |
| fungi | heh | 17:15 |
| dansmith | it does look like it ended up in there though | 17:15 |
| clarkb | fwiw I strongly suspect those long patch chains in large repos are the source of the constipation | 17:16 |
| fungi | i'll try to get to a stopping point on the patch i'm writing now and switch to digging in gerrit logs for problematic queryspam | 17:16 |
| clarkb | orat least that is what trips it | 17:16 |
| fungi | odds are it's on a knife edge for maxing out its memory utilization and something is ocasionally pushing it into a gc scramble | 17:17 |
| dansmith | that'd be unfortunate | 17:17 |
| clarkb | basically anytime we run into thise and i check zuulthere are long nova chains | 17:17 |
| clarkb | could be coincidence but it has been very consistent when I've checked | 17:17 |
| dansmith | clarkb: ooh, caught you.. this is a long glance chain! :P | 17:18 |
| fungi | totally different then ;) | 17:18 |
| dansmith | obvi | 17:18 |
| clarkb | heh I mean I think any large repo can probably do it. Glance isn't huge but isn't small iirc | 17:19 |
| *** abhishekk has quit IRC | 17:22 | |
| dansmith | yeah, I know :) | 17:24 |
| dansmith | just had to take the #fakenews opportunity | 17:24 |
| *** ramishra has quit IRC | 17:38 | |
| *** andrewbonney has quit IRC | 17:46 | |
| *** thiago__ has joined #openstack-infra | 18:07 | |
| *** thiago__ has quit IRC | 18:11 | |
| *** thiago__ has joined #openstack-infra | 18:11 | |
| *** tdasilva_ has quit IRC | 18:13 | |
| *** thiago__ has quit IRC | 18:17 | |
| *** thiago__ has joined #openstack-infra | 18:17 | |
| *** thiago__ has quit IRC | 18:22 | |
| *** thiago__ has joined #openstack-infra | 18:23 | |
| dansmith | looks like in addition to no irc bot announce, zuul didn't start running jobs on those patches either | 18:28 |
| *** thiago__ is now known as tdasilva | 18:33 | |
| clarkb | I think ianw has been looking at eavesdrop things due to the oftc move, but we should probably prioritize the gerrit server swap to get all that extra memory | 18:40 |
| clarkb | ianw: ^ for next week, maybe we dive into that | 18:41 |
| fungi | dansmith: yep, that's why at 17:15z i recommended you add recheck comments to all of them | 18:42 |
| dansmith | oh, totes missed the "and zuul" part too, mah bad | 18:42 |
| fungi | no worries | 18:42 |
| fungi | i think what happens is the change upload completes but takes longer than gerrit's internal watcher is willing to wait, so it's already given up and never gets around to sending any notifications on the event stream (which is what both zuul and gerritbot are listening to) | 18:43 |
| dansmith | yeah, I figured | 18:46 |
| dansmith | I guess if the upload fails on git-review, I expect the "transaction" to be aborted | 18:46 |
| dansmith | but clearly they're too cool for "databases" now | 18:46 |
| fungi | well, yeah got repository writes are probably not transactional | 18:47 |
| fungi | s/got/git/ | 18:47 |
| dansmith | I thought they were, but I'm guessing they're waiting for an fsync() or something without the ability to abort if the waiter times out like you'd get from a db session | 18:49 |
| dansmith | i.e. I expect they're safe for interrupted writes, but without the "okay I wrote all the data, ready to commit?" phase | 18:49 |
| *** tinwood has quit IRC | 18:55 | |
| *** tinwood has joined #openstack-infra | 18:58 | |
| *** tdasilva_ has joined #openstack-infra | 19:00 | |
| *** tdasilva has quit IRC | 19:00 | |
| fungi | i just mean they way they have it implemented, they probably can't safely roll back writes. no idea honestly | 19:05 |
| opendevreview | Ade Lee proposed openstack/openstack-zuul-jobs master: Add FIPS job https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/790534 | 19:23 |
| *** vishalmanchanda has quit IRC | 19:54 | |
| *** gfidente|afk has quit IRC | 20:42 | |
| *** thiago__ has joined #openstack-infra | 20:55 | |
| *** tdasilva_ has quit IRC | 20:55 | |
| *** gmann is now known as gmann_afk | 21:51 | |
| *** tosky has quit IRC | 22:01 | |
| *** rlandy has quit IRC | 23:01 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!