Friday, 2019-11-15

clarkbianw: fwiw that has been done with thus server once before :(00:03
ianw12084 write(2, "fatal: Needed a single revision\n", 32 <unfinished ...>00:05
ianwit really seems like gitea is sending git something it doesn't like00:05
clarkbthe recent gitea upgrade did switch to go-git00:05
clarkband you said git itself fsck'd clean00:05
clarkbweird that the other 7 hosts are fine though00:06
*** dave-mccowan has quit IRC00:10
*** slaweq has joined #openstack-infra00:11
*** diablo_rojo__ has joined #openstack-infra00:15
*** slaweq has quit IRC00:16
*** KeithMnemonic has joined #openstack-infra00:16
*** dave-mccowan has joined #openstack-infra00:16
ianw12027 write(2, "fatal: the remote end hung up unexpectedly\n", 43 <unfinished ...>00:17
*** KeithMnemonic has quit IRC00:17
ianw*that* appears to be the error00:17
*** KeithMnemonic has joined #openstack-infra00:17
ianw12027 execve("/usr/bin/git", ["/usr/bin/git", "upload-pack", "--stateless-rpc", "/data/git/repositories/x/docker-machine-openstack.git"]00:17
ianwthis actually matches the code @ https://github.com/go-gitea/gitea/blob/master/routers/repo/http.go#L41800:17
ianwi wonder if this points to packet loss on the host, or soemthing like that?00:22
*** KeithMnemonic has quit IRC00:23
*** igordc has quit IRC00:24
*** igordc has joined #openstack-infra00:25
ianwtonyb: what git client were/are you using?00:25
tonybianw: F30 whatever comes with that00:26
ianwwhat's the version number?00:26
tonybgit version 2.21.000:26
ianwok, git on gitea is 2.20.000:26
ianwdo you happen to have the error you saw at all?00:27
tonybianw: I didn't error just sat there for hours00:27
ianwinteresting ...00:27
ianwmaybe it is networking then?00:28
ianwdo you have any packet loss pinging gitea06?00:28
*** slaweq has joined #openstack-infra00:28
tonybnope00:30
*** rfolco has joined #openstack-infra00:31
tonyb10 packets transmitted, 10 received, 0% packet loss, time 19ms00:31
tonybthat's with 1500byte packets00:31
ianwhttps://github.com/go-gitea/gitea/issues/900600:32
*** igordc has quit IRC00:32
tonybHmm okay, It could have been networking I guess00:32
*** slaweq has quit IRC00:33
tonybianw: Perhaps put it back in rotation and if it happens again we can try and get more data at that point in time00:34
ianwoh, as soon as you put it back, the errors start appearing00:34
* tonyb wonders if I can really really reset my repo to 5 days ago and try again00:35
ianwif we had a reproduction i might be able to tcpdump it00:38
tonybYeah00:38
*** mriedem has quit IRC00:45
*** slaweq has joined #openstack-infra00:50
*** slaweq has quit IRC00:55
*** tosky has quit IRC00:59
*** dave-mccowan has quit IRC01:01
*** ociuhandu has joined #openstack-infra01:01
*** yamamoto has joined #openstack-infra01:05
*** rfolco has quit IRC01:06
*** slaweq has joined #openstack-infra01:06
*** yamamoto has quit IRC01:09
*** jdelaros1 has quit IRC01:11
*** rpittau|afk has quit IRC01:11
*** mnasiadka has quit IRC01:11
*** jamespage has quit IRC01:11
*** dougwig has quit IRC01:11
ianwi've run a tcpdump, but it's all just binary blobs of ssl data ... i can't see any obvious dropped connections looking in wireshark01:11
*** jdelaros1 has joined #openstack-infra01:11
*** rpittau|afk has joined #openstack-infra01:11
*** crodriguez has quit IRC01:11
*** cjohnston has quit IRC01:11
*** npochet has quit IRC01:11
tonybianw: Are you seeing the hang?01:11
*** guilhermesp has quit IRC01:11
*** MarkMaglana has quit IRC01:11
*** beisner has quit IRC01:11
*** mwhahaha has quit IRC01:11
*** lamt has quit IRC01:12
*** TheJulia has quit IRC01:12
*** nickv1985 has quit IRC01:12
*** icey has quit IRC01:12
*** mnasiadka has joined #openstack-infra01:12
*** lsell has quit IRC01:12
*** sparkycollier has quit IRC01:12
*** thedac has quit IRC01:12
*** npochet has joined #openstack-infra01:12
*** cjohnston has joined #openstack-infra01:12
*** crodriguez has joined #openstack-infra01:12
ianwno, but it's all behind the load balance, to add another layer of compelxity01:12
*** ofosos has quit IRC01:12
*** MarkMaglana has joined #openstack-infra01:12
*** icey has joined #openstack-infra01:13
*** jamespage has joined #openstack-infra01:13
*** mwhahaha has joined #openstack-infra01:13
*** dougwig has joined #openstack-infra01:13
*** nickv1985 has joined #openstack-infra01:13
*** TheJulia has joined #openstack-infra01:13
*** guilhermesp has joined #openstack-infra01:13
*** sparkycollier has joined #openstack-infra01:13
*** beisner has joined #openstack-infra01:13
*** thedac has joined #openstack-infra01:14
*** lsell has joined #openstack-infra01:14
ianwtonyb: if you can replicate it with a direct connection to gitea06, maybe we can see something?01:14
tonybianw: I'm trying to reproduce it, but after I grabbed the objects via ssh I haven't been able to see it again01:14
tonybianw: right now I'm trying to create a nova repo that looks like it was last updated in October01:15
*** slaweq has quit IRC01:15
*** onovy has quit IRC01:16
*** armax has quit IRC01:17
ianwyeah, it works cloning fresh for me -- it does run upload-pack01:18
*** onovy has joined #openstack-infra01:18
*** slaweq has joined #openstack-infra01:18
*** diablo_rojo__ has quit IRC01:20
ianwit could well be the load-balancer -> gitea06 connection01:20
tonybOh, I guess that's true01:21
*** ociuhandu has quit IRC01:23
*** slaweq has quit IRC01:23
donnydSo it's directly possible that I don't know enough about swift, but I cannot for the life of me figure out what broke this morning01:26
donnydUp until today everything has been running without issue, and everything has been browseable no problem01:27
donnydthen I started getting this random error > Web Listing Disabled01:27
donnydswift is one of the project I am familiar enough with to break badly01:28
clarkbdonnyd: I think the web listibg thibg is due to index.html objects 404ing01:28
clarkbif the objects dont 404 it works01:28
clarkbmaybe chwck the proxies as I believe they do the initial location determination01:28
clarkbianw: tonyb direct requests to gitea06 may be able to rule out the proxy01:33
tonybclarkb: I am tryign direct to gitea0601:34
tonybclarkb: I get different results when going through the LB, but I can't reproduce the failure :(01:34
donnydbut I don't really understand why the proxy wouldn't be able to find the objects01:36
donnydyou are right it is 404 when I request the file01:36
fungiyeah, it wasn't *just* the indices which were a problem01:36
fungithose just happen to be what people hit first to find other files01:36
fungiif i directly hit urls for other files which were uploaded there, those were also giving intermittent 40401:37
*** slaweq has joined #openstack-infra01:37
donnydis there a way to do a scrub or anything like that?01:38
*** ociuhandu has joined #openstack-infra01:38
donnydI did have a drive failure01:38
donnydbut there is still a clean copy on the other server01:38
donnydthere are 10 drives in each server... so I am not exactly sure how to fix it.. but I will keep searching01:41
*** ociuhandu has quit IRC01:42
tonybclarkb, ianw: I can't reproduce it today :(01:43
*** yamamoto has joined #openstack-infra01:44
tonybhttp://paste.openstack.org/show/786152 is the script I used for testing FWIW01:44
*** yamamoto has quit IRC01:44
*** slaweq has quit IRC01:45
ianwtonyb: thanks, i'll update the story.  but it's feeling a bit like tilting at windmills when the error message from gitea doesn't correlate itself to a request01:46
*** yamamoto has joined #openstack-infra01:48
donnydfungi: clarkb: So should we put FN back online once the failed drive is finished syncing?01:50
donnydthis swift part that is01:50
fungidonnyd: if that solves the 404s then sure01:50
donnydwell it seems like the 404's are for objects that are supposed to have been written.. but I don't know that they actually were.. shouldn't it pull the object from a known good copy?01:51
*** slaweq has joined #openstack-infra01:52
donnydI don't know enough about how swift works to effectively T/S this issue beyond.. yep it works.. If it was built on ceph it would be more in my wheelhouse01:53
*** rlandy|rover|bia is now known as rlandy|rover01:56
*** slaweq has quit IRC01:56
*** yamamoto has quit IRC01:57
fungioddly, the 404s are intermittent for the same urls01:58
*** rlandy|rover has quit IRC01:59
fungiso it's like there are replicas in swift and sometimes it's picking missing replicas?01:59
fungihard to say01:59
ianw#status log further investigation hasn't revealed solutions for gitea06's issues.  https://storyboard.openstack.org/#!/story/2006849 is updated.  host should probably be rebuilt.  will wait to see if any comments on https://github.com/go-gitea/gitea/issues/900602:01
ianwno statusbot huh02:01
*** ociuhandu has joined #openstack-infra02:02
donnydfungi: well that does make some level of sense because I am seeing some of this type of stuff in the logs02:02
donnyd....02:03
donnydNov 15 01:58:54 swift1 object-server[5711]: <f+++++++++ ffc/40087c899f0f6870656acb80dee9fffc/1573111913.06629.data02:03
donnydNov 15 01:58:54 swift1 object-server[5711]: Successful rsync of /srv/node/sdk/objects/256/785 at 10.0.10.12::object/sdh/objects/256 (227.381)02:03
*** openstackstatus has joined #openstack-infra02:03
*** ChanServ sets mode: +v openstackstatus02:03
ianw#status log further investigation hasn't revealed solutions for gitea06's issues.  https://storyboard.openstack.org/#!/story/2006849 is updated.  host should probably be rebuilt.  will wait to see if any comments on https://github.com/go-gitea/gitea/issues/900602:03
openstackstatusianw: finished logging02:03
donnydso maybe it just needs to finish syncing all the objects around... but not sure why it wouldn't know about it to begin with02:03
donnydfungi: do you have a url that is doing that handy?02:04
fungisure, just a sec02:04
fungidonnyd: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_8b9/694235/1/gate/kolla-build-ubuntu-source/8b9769d/02:05
*** ociuhandu has quit IRC02:06
*** ociuhandu has joined #openstack-infra02:07
*** kjackal has quit IRC02:07
*** slaweq has joined #openstack-infra02:11
*** ociuhandu has quit IRC02:12
donnydok so I figured out how to use the audit tool and I found some bad things.. but i am working on how I can fix it02:14
*** slaweq has quit IRC02:15
*** kjackal has joined #openstack-infra02:20
*** yamamoto has joined #openstack-infra02:25
*** yamamoto has quit IRC02:25
*** yamamoto has joined #openstack-infra02:26
*** ociuhandu has joined #openstack-infra02:26
*** Lucas_Gray has joined #openstack-infra02:27
*** darvon has quit IRC02:32
*** ociuhandu has quit IRC02:36
*** darvon has joined #openstack-infra02:38
*** rkukura has quit IRC02:40
openstackgerritIan Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon  https://review.opendev.org/69346402:52
*** igordc has joined #openstack-infra02:53
*** gyee has quit IRC02:55
*** zzzeek has quit IRC02:56
*** zzzeek has joined #openstack-infra02:57
*** zzzeek has quit IRC02:58
*** zzzeek has joined #openstack-infra02:59
*** rosmaita has left #openstack-infra03:02
*** ociuhandu has joined #openstack-infra03:07
*** ociuhandu has quit IRC03:12
*** ociuhandu has joined #openstack-infra03:13
*** yamamoto has quit IRC03:17
*** yamamoto has joined #openstack-infra03:18
*** gfidente has quit IRC03:26
*** kjackal has quit IRC03:33
*** ociuhandu has quit IRC03:36
*** goldyfruit_ has quit IRC03:53
*** ociuhandu has joined #openstack-infra04:01
*** ociuhandu has quit IRC04:06
*** psachin has joined #openstack-infra04:15
*** raukadah is now known as chandankumar04:21
*** rh-jelabarre has quit IRC04:32
*** yamamoto has quit IRC04:34
*** ykarel|away has joined #openstack-infra04:37
*** ociuhandu has joined #openstack-infra04:40
*** ociuhandu has quit IRC04:45
*** soniya29 has joined #openstack-infra04:56
*** hwoarang has quit IRC04:59
*** ociuhandu has joined #openstack-infra05:00
*** hwoarang has joined #openstack-infra05:00
*** Lucas_Gray has quit IRC05:02
*** yamamoto has joined #openstack-infra05:06
*** yamamoto has quit IRC05:11
openstackgerritIan Wienand proposed zuul/nodepool master: Dockerfile: create APP_DIR  https://review.opendev.org/69364605:12
openstackgerritIan Wienand proposed zuul/nodepool master: docker: Base nodepool images on diskimage-builder base image  https://review.opendev.org/69417705:12
openstackgerritIan Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon  https://review.opendev.org/69346405:12
*** ykarel|away is now known as ykarel05:15
*** kjackal has joined #openstack-infra05:18
*** dchen has quit IRC05:21
*** udesale has joined #openstack-infra05:22
*** yamamoto has joined #openstack-infra05:25
*** ociuhandu has quit IRC05:28
*** ociuhandu has joined #openstack-infra05:30
*** ociuhandu has quit IRC05:36
AJaegerconfig-core, please review https://review.opendev.org/#/c/688822/ https://review.opendev.org/694244 https://review.opendev.org/#/c/690718/ https://review.opendev.org/691140  https://review.opendev.org/#/c/693455/ and https://review.opendev.org/69346205:37
openstackgerritMerged openstack/project-config master: Add tracking for Train jobs  https://review.opendev.org/68882205:54
openstackgerritMerged openstack/project-config master: Grant permissions for murano-dashboard to horizon-core  https://review.opendev.org/69114005:55
openstackgerritMerged openstack/project-config master: New charm for manila-ganesha  https://review.opendev.org/69346205:55
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/69445506:08
*** ociuhandu has joined #openstack-infra06:15
*** kjackal has quit IRC06:19
*** dciabrin_ has joined #openstack-infra06:28
*** ddurst- has joined #openstack-infra06:35
*** gouthamr_ has joined #openstack-infra06:35
*** Jeffrey4l_ has joined #openstack-infra06:36
*** ociuhandu has quit IRC06:37
*** ykarel has quit IRC06:37
*** rcernin has quit IRC06:37
*** trident has quit IRC06:37
*** jpena|off has quit IRC06:37
*** jaicaa has quit IRC06:37
*** ddurst has quit IRC06:39
*** mgoddard has quit IRC06:39
*** hamzy has quit IRC06:39
*** gouthamr has quit IRC06:39
*** antonym has quit IRC06:39
*** fdegir has quit IRC06:39
*** dciabrin has quit IRC06:39
*** amorin has quit IRC06:39
*** brtknr has quit IRC06:39
*** elod_off has quit IRC06:39
*** yankcrime has quit IRC06:39
*** lastmikoi has quit IRC06:39
*** Jeffrey4l has quit IRC06:39
*** brwyatt has quit IRC06:39
*** evrardjp has quit IRC06:39
*** osmanlicilegi has quit IRC06:39
*** tobias-urdin has quit IRC06:39
*** itxaka has quit IRC06:39
*** Zara has quit IRC06:39
*** spotz has quit IRC06:39
*** dabukalam has quit IRC06:39
*** openstackstatus has quit IRC06:40
*** itxaka has joined #openstack-infra06:42
*** Zara has joined #openstack-infra06:42
*** tobias-urdin has joined #openstack-infra06:42
*** dabukalam has joined #openstack-infra06:42
*** spotz has joined #openstack-infra06:42
*** ykarel has joined #openstack-infra06:48
*** rcernin has joined #openstack-infra06:48
*** trident has joined #openstack-infra06:48
*** jpena|off has joined #openstack-infra06:48
*** jaicaa has joined #openstack-infra06:48
*** mgoddard has joined #openstack-infra06:48
*** osmanlicilegi has joined #openstack-infra06:48
*** hamzy has joined #openstack-infra06:48
*** antonym has joined #openstack-infra06:48
*** fdegir has joined #openstack-infra06:48
*** amorin has joined #openstack-infra06:48
*** lastmikoi has joined #openstack-infra06:48
*** brtknr has joined #openstack-infra06:48
*** elod_off has joined #openstack-infra06:48
*** brwyatt has joined #openstack-infra06:48
*** evrardjp has joined #openstack-infra06:48
*** dpawlik has joined #openstack-infra06:59
*** dpawlik has quit IRC07:15
*** ykarel has quit IRC07:15
*** rcernin has quit IRC07:15
*** trident has quit IRC07:15
*** jpena|off has quit IRC07:15
*** jaicaa has quit IRC07:15
*** mgoddard has quit IRC07:15
*** hamzy has quit IRC07:15
*** antonym has quit IRC07:15
*** fdegir has quit IRC07:15
*** amorin has quit IRC07:15
*** brtknr has quit IRC07:15
*** elod_off has quit IRC07:15
*** lastmikoi has quit IRC07:15
*** brwyatt has quit IRC07:15
*** evrardjp has quit IRC07:15
*** osmanlicilegi has quit IRC07:15
*** ykarel has joined #openstack-infra07:16
*** rcernin has joined #openstack-infra07:16
*** trident has joined #openstack-infra07:16
*** jpena|off has joined #openstack-infra07:16
*** jaicaa has joined #openstack-infra07:16
*** mgoddard has joined #openstack-infra07:16
*** osmanlicilegi has joined #openstack-infra07:16
*** hamzy has joined #openstack-infra07:16
*** antonym has joined #openstack-infra07:16
*** fdegir has joined #openstack-infra07:16
*** amorin has joined #openstack-infra07:16
*** lastmikoi has joined #openstack-infra07:16
*** brtknr has joined #openstack-infra07:16
*** elod_off has joined #openstack-infra07:16
*** brwyatt has joined #openstack-infra07:16
*** evrardjp has joined #openstack-infra07:16
*** dpawlik has joined #openstack-infra07:17
*** pgaxatte has joined #openstack-infra07:18
*** jtomasek has quit IRC07:19
*** jtomasek has joined #openstack-infra07:22
openstackgerritMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/69445507:24
*** ociuhandu has joined #openstack-infra07:27
*** KeithMnemonic has joined #openstack-infra07:29
*** ociuhandu has quit IRC07:29
*** ociuhandu has joined #openstack-infra07:30
*** KeithMnemonic has quit IRC07:34
*** ociuhandu has quit IRC07:41
*** pkopec has joined #openstack-infra07:42
*** ociuhandu has joined #openstack-infra07:42
*** ociuhandu has quit IRC07:44
*** rcernin has quit IRC07:44
*** ociuhandu has joined #openstack-infra07:45
*** slaweq has joined #openstack-infra07:46
*** trident has quit IRC07:49
openstackgerritMerged openstack/openstack-zuul-jobs master: Run neutron-grenade job only for branches before Ussuri  https://review.opendev.org/69424407:49
*** slaweq_ has joined #openstack-infra07:54
*** ociuhandu has quit IRC07:55
*** dpawlik has quit IRC07:56
*** slaweq has quit IRC07:56
*** rpittau|afk is now known as rpittau07:56
*** trident has joined #openstack-infra07:58
*** dchen has joined #openstack-infra07:58
*** tkajinam has quit IRC08:00
*** ykarel has quit IRC08:02
*** ykarel has joined #openstack-infra08:02
slaweq_frickler: hi08:03
*** slaweq_ is now known as slaweq08:03
*** dpawlik has joined #openstack-infra08:04
slaweqfrickler: do You know what is procedure to move projects (neutron-interconnection) from official openstack/ namespace to x/ namespace?08:04
slaweqfrickler: we want to move out of stadium projects https://github.com/openstack/neutron-interconnection08:05
*** dpawlik has quit IRC08:08
*** dpawlik has joined #openstack-infra08:10
AJaegerslaweq: let me get you a link...08:11
AJaegerslaweq: follow Raum 2.2.1008:11
AJaegerargh, pasto ;(08:11
slaweq:)08:11
AJaegerslaweq: follow https://docs.openstack.org/infra/manual/creators.html#project-renames08:11
slaweqAJaeger: thx a lot08:13
AJaegerslaweq: we need to schedule such renames (at least in the past, we might be able to do better now), so be sure to add to the referenced wiki page08:14
slaweqAJaeger: sure, I will08:14
*** tesseract has joined #openstack-infra08:15
*** yamamoto has quit IRC08:15
*** gibi_off has quit IRC08:19
*** slaweq_ has joined #openstack-infra08:19
*** slaweq has quit IRC08:19
*** pcrews has joined #openstack-infra08:26
*** pgaxatte has quit IRC08:26
*** pgaxatte has joined #openstack-infra08:27
*** ralonsoh has joined #openstack-infra08:32
*** ociuhandu has joined #openstack-infra08:36
*** lucasagomes has joined #openstack-infra08:36
*** lucasagomes has quit IRC08:38
*** tosky has joined #openstack-infra08:39
openstackgerritSlawek Kaplonski proposed openstack/project-config master: Rename neutron-interconnection from openstack/ to x/ namespace  https://review.opendev.org/69447808:45
slaweq_AJaeger: should I add this project to upcomming renames list now or after patch ^^ will be merged?08:45
*** slaweq_ is now known as slaweq08:45
AJaegerslaweq: add it now, please. We need to wiki update to schedule merging your change...08:47
slaweqAJaeger: sure08:47
*** jpena|off is now known as jpena08:47
AJaegerslaweq: do you have a governance change up as well?08:47
slaweqAJaeger: ahh, not yet08:47
slaweqI will do it in few minutes08:47
*** kopecmartin|pto is now known as kopecmartin08:48
AJaegerslaweq: will there be ongoing work on the repo - or do you wnat to retire it completely instead?08:48
slaweqAJaeger: we just want to move this repo out of stadium08:49
AJaegerok08:49
slaweqto make it unofficial project08:49
slaweqAJaeger: ok, governance change https://review.opendev.org/#/c/694480/08:53
slaweqand wiki page updated08:53
slaweqI hope I didn't miss anything08:53
*** iurygregory has joined #openstack-infra09:04
*** gfidente has joined #openstack-infra09:05
*** gfidente has quit IRC09:06
*** gfidente has joined #openstack-infra09:13
fungislaweq: AJaeger: that repo needs to be forked, not renamed: https://governance.openstack.org/tc/resolutions/20190711-mandatory-repository-retirement.html09:14
fungithe tc agreed that having official deliverables "become unofficial" can take consumers by surprise even years later, and that forcing them to be retired in-place and then started up again under a different namespace sends a stronger downstream signal09:16
slaweqfungi: so should I change commit message of https://review.opendev.org/#/c/694480/2 to something like "Retire neutron-interconnection project"?09:19
fungislaweq: probably, but it's more advice for when it comes time to write the project-config change since we won't be renaming it you'll want to import the content in a new repo and retire the original09:21
fungior just do the retirement part and let whoever wants to continue/resume development propose the new repo09:22
slaweqfungi: ok, so this retirement patch shouldn't depends on project-config rename patch anymore, correct?09:23
fungithe project-config patch should not rename anything09:23
fungiit should just retire the repo (and optionally add a new one importing the old content)09:24
fungiand follow the retirement process linked from that tc resolution09:25
*** ykarel is now known as ykarel|lunch09:27
slaweqfungi: ok, so I will follow steps as descirbed in https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project09:28
*** carl_cai has joined #openstack-infra09:33
*** ociuhandu has quit IRC09:35
*** ramishra has quit IRC09:36
slaweqfungi: I changed commit message in https://review.opendev.org/#/c/694480/ and also sent patch https://review.opendev.org/694490/ to add README file in repo now09:38
slaweqfungi: as this project was never really implemented, there is no any requirements or zuul jobs for it so I ommited those steps from the governance guide09:38
slaweqplease review those patches and let me know if I'm still missing something there09:39
slaweqand thx a lot for help09:39
*** psachin has quit IRC09:39
fungimy pleasure09:41
*** aedc has joined #openstack-infra09:42
*** ykarel|lunch is now known as ykarel|09:46
*** ykarel| is now known as ykarel09:46
*** ramishra has joined #openstack-infra09:51
AJaegerfungi: oh - not familiar with the new process yet. Could you update the infra manual, please?09:53
*** xek__ has joined #openstack-infra09:54
fungiAJaeger: it's not an infra process, just the tc deciding no more renaming repos out of the openstack namespace09:54
AJaegerfungi: I think we should still mention it for this case somehow...09:56
fungiinfra can still rename repos, but openstack has decided renaming official deliverables out of their namespace is not good for the ecosystem09:56
AJaegerfungi: understood - different process depending on namespace and move09:57
fungimainly because now that we have redirects on renaming, folks consuming those repos are increasingly unlikely to notice the project has moved09:57
fungibut also it leaves a clearer breadcrumb for consumers to find09:58
fungiarguably in this case retirement was the better choice anyway, as the commit message indicates the project died and has no new activity09:59
*** yankcrime has joined #openstack-infra09:59
fungiso we'd i think in all cases people retire a project to signal it's dead, rather than just moving it to another namespace where it's easier to ignore10:00
fungier, we'd i think in all cases prefer people retire a project to signal it's dead10:01
*** ramishra has quit IRC10:07
AJaegerfungi: I agree, retiring looks better here. slaweq, please consider it10:12
*** udesale has quit IRC10:14
*** udesale has joined #openstack-infra10:15
fungithe real upshot of the tc resolution, once folks get accustomed to it, is that we should just stop seeing renames out of the openstack namespace, but also we could just pay attention to whatever happens to the openstack governance change since it's up to the openstack tc to enforce their processes10:15
fungianyway, i'm going to fight this jetlag and try to get back to sleep for a bit10:16
openstackgerritZara proposed opendev/system-config master: Add some details on where to find things to homepage  https://review.opendev.org/69149110:16
*** ykarel is now known as ykarel|afk10:22
*** lpetrut has joined #openstack-infra10:26
*** aedc has quit IRC10:28
*** aedc has joined #openstack-infra10:28
openstackgerritDaniel Bengtsson proposed openstack/reviewstats master: Stop configuring install_command in tox.  https://review.opendev.org/69450610:28
*** ociuhandu has joined #openstack-infra10:31
*** slaweq has quit IRC10:34
*** ociuhandu has quit IRC10:37
*** elod_off is now known as elod10:38
*** ociuhandu has joined #openstack-infra10:38
*** ociuhandu has quit IRC10:46
*** yamamoto has joined #openstack-infra10:47
*** tesseract has quit IRC10:52
*** tesseract has joined #openstack-infra10:52
*** ykarel_ has joined #openstack-infra10:56
*** yamamoto_ has joined #openstack-infra10:57
*** ykarel|afk has quit IRC10:57
*** yamamoto has quit IRC10:59
*** ramishra has joined #openstack-infra11:05
*** soniya29 has quit IRC11:05
toskynot sure whether it's an infra issue, but I don't see other changes: a glance backport for stein started failing because apparently the functional tox environment now uses python 311:10
toskydid anything change from ~ October 24 and now?11:11
tosky(in the base environment/images)11:11
toskylink to the job: https://review.opendev.org/#/c/691308/ - the first run passed11:11
*** soniya29 has joined #openstack-infra11:14
*** lucasagomes has joined #openstack-infra11:38
openstackgerritMerged opendev/irc-meetings master: Remove Neutron performance sub-team meeting  https://review.opendev.org/69441911:44
*** dpawlik has quit IRC11:47
*** dpawlik has joined #openstack-infra11:53
*** ramishra has quit IRC11:53
*** slaweq has joined #openstack-infra11:55
*** jpena is now known as jpena|lunch12:00
*** lucasagomes has quit IRC12:06
fricklertosky: that may be an unintended side effect of https://review.opendev.org/686524 I guess. the solution would be for glance to hardcode the basepython version like they did in master already https://review.opendev.org/66515312:11
fricklerif basepython is unspecified, tox uses the one it was installed with itself12:11
*** goldyfruit_ has joined #openstack-infra12:12
*** goldyfruit_ has quit IRC12:16
toskyfrickler: but isn't this going to break a lot of stable jobs?12:17
toskyI'd argue that a more stable solution is needed12:17
fricklertosky: I'm not sure we would want to continue to use python2 if we can avoid it in building our base images, so I'd say fixing decaying stable jobs would be the correct way forward, but maybe other infra-roots see this differently12:23
*** pgaxatte has quit IRC12:24
toskyfrickler: sorry, but stable jobs are not "decaying", they are stable and having to patch every one of them with a change that is needed only because something else in the future changed is not a good solution12:25
toskyit's an (unintended) breakage and a regression12:25
toskystable jobs should continue to work as before12:25
*** dpawlik has quit IRC12:28
fricklertosky: I've added this topic for discussion to our next infra meeting, maybe if you have time you can join that (next Tue 19:00 UTC) https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting12:29
*** ykarel_ is now known as ykarel12:29
*** dciabrin_ is now known as dciabrin12:30
*** ociuhandu has joined #openstack-infra12:32
toskyfrickler: sure, thanks12:33
frickler#status log restarted apache on ask.o.o which had died on reload again12:33
*** dpawlik has joined #openstack-infra12:33
fricklerinfra-root: ^^ it went well for a couple of days but now it was failing again, if noone objects I'd merge https://review.opendev.org/694105 and have us see whether that helps.12:34
openstackgerritAndreas Jaeger proposed openstack/infra-manual master: Mention TC resolution on moving projects out of openstack  https://review.opendev.org/69452812:36
AJaegerfungi: I suggest we link to the resolution ^12:36
*** ociuhandu has quit IRC12:37
*** ociuhandu has joined #openstack-infra12:38
*** ociuhandu has quit IRC12:39
*** ociuhandu has joined #openstack-infra12:39
*** rh-jelabarre has joined #openstack-infra12:44
*** ociuhandu has quit IRC12:45
*** ricolin has joined #openstack-infra12:50
*** aedc_ has joined #openstack-infra12:52
*** aedc has quit IRC12:55
*** rfolco has joined #openstack-infra12:59
*** rh-jelabarre is now known as jlabarre_dr-appt13:01
*** rlandy has joined #openstack-infra13:03
*** rlandy is now known as rlandy|ruck13:04
*** dpawlik has quit IRC13:06
*** aedc_ has quit IRC13:06
*** rfolco is now known as rfolco_pto13:08
*** pgaxatte has joined #openstack-infra13:10
openstackgerritMerged openstack/infra-manual master: Mention TC resolution on moving projects out of openstack  https://review.opendev.org/69452813:11
*** ociuhandu has joined #openstack-infra13:16
*** mriedem has joined #openstack-infra13:23
*** rosmaita has joined #openstack-infra13:27
*** rfolco_pto has quit IRC13:33
*** dpawlik has joined #openstack-infra13:45
openstackgerritDaniel Bengtsson proposed openstack/diskimage-builder master: Stop configuring install_command in tox.  https://review.opendev.org/69454013:47
*** ociuhandu has quit IRC13:58
*** jpena|lunch is now known as jpena14:01
*** gfidente has quit IRC14:03
*** dpawlik has quit IRC14:03
*** kjackal has joined #openstack-infra14:05
*** dpawlik has joined #openstack-infra14:05
*** ociuhandu has joined #openstack-infra14:05
*** jlabarre_dr-appt is now known as rh-jelabarre14:13
*** apetrich has quit IRC14:22
openstackgerritSimon Westphahl proposed zuul/zuul master: Fix issue search in FakeGithubClient  https://review.opendev.org/69455314:24
*** goldyfruit has joined #openstack-infra14:30
*** ykarel is now known as ykarel|afk14:32
*** ociuhandu has quit IRC14:32
*** soniya29 has quit IRC14:32
*** ssbarnea has joined #openstack-infra14:33
*** chandankumar is now known as raukadah14:39
*** KeithMnemonic has joined #openstack-infra14:41
*** apetrich has joined #openstack-infra14:43
*** surpatil has joined #openstack-infra14:53
*** gfidente has joined #openstack-infra14:59
*** eharney has joined #openstack-infra15:04
*** armax has joined #openstack-infra15:04
*** carl_cai has quit IRC15:09
*** dpawlik has quit IRC15:10
*** jaosorior has joined #openstack-infra15:13
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove lib-forward-testing-plugin  https://review.opendev.org/69457015:15
*** rlandy|ruck has quit IRC15:15
*** weshay is now known as weshay|ruck15:15
AJaegerconfig-core, please review a new role for installing podman: https://review.opendev.org/#/c/693455/15:16
AJaegerclarkb: please see the comment on https://review.opendev.org/690718 - want to change or self-approve?15:17
*** aedc has joined #openstack-infra15:17
*** ricolin has quit IRC15:18
*** surpatil has quit IRC15:22
*** surpatil has joined #openstack-infra15:22
*** yamamoto_ has quit IRC15:26
*** surpatil has quit IRC15:30
*** pgaxatte has quit IRC15:33
openstackgerritDavid Hill proposed openstack/diskimage-builder master: Add grub-efi-x86_64 pkg-map  https://review.opendev.org/69457915:37
*** ociuhandu has joined #openstack-infra15:37
*** KeithMnemonic1 has joined #openstack-infra15:41
*** ociuhandu has quit IRC15:42
*** ykarel|afk is now known as ykarel15:45
openstackgerritMerged opendev/system-config master: Restart apache2 on logrotate on ask.o.o  https://review.opendev.org/69410515:47
openstackgerritSlawek Kaplonski proposed openstack/project-config master: Retire neutron-interconnection project  https://review.opendev.org/69447815:48
AJaegerslaweq: read https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project - you need to merge the "retired" README to the repo first - and to do that, need to add noop-jobs to project-config.15:49
AJaegerslaweq: otherwise your change above looks fine15:50
openstackgerritSlawek Kaplonski proposed openstack/project-config master: Add noop-jobs template for neutron-interconnection project  https://review.opendev.org/69458215:55
slaweqAJaeger: done ^^ :)15:55
*** goldyfruit_ has joined #openstack-infra15:55
slaweqso in general I have 5 patches to retire the project: https://review.opendev.org/#/q/topic:neutron-interconnection-retire+(status:open+OR+status:merged)15:55
slaweqI hope it's all fine now :)15:55
*** ociuhandu has joined #openstack-infra15:55
*** goldyfruit has quit IRC15:57
AJaegerslaweq: 694478 will need to remove noop-jobs again. Sorry, just commented there without seing your message here. Will review them all together now...16:05
AJaegerslaweq: https://review.opendev.org/#/c/694582 is fine - config-core, please review16:05
openstackgerritSlawek Kaplonski proposed openstack/project-config master: Retire neutron-interconnection project  https://review.opendev.org/69447816:07
AJaegermnaser: for static.o.o, please review https://review.opendev.org/681582 and the stack on top of it. 681582 can merge directly IMHO. I let you drive the rest and any changes to it...16:09
slaweqAJaeger: is it better now ^^? I put 694478 on top of 694582 and added Depends-On on https://review.opendev.org/#/c/694490/ for it16:09
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles.  https://review.opendev.org/69111416:09
slaweqif I'm still missing something, please just left me comments in gerrit, I will address them later16:09
slaweqnow I have to drop off16:10
slaweqthx a lot for all Your help with this16:10
AJaegerslaweq: you're welcome, have a great weekend!16:10
AJaegerslaweq: I left a comment - you need to remove noop-jobs again16:10
*** rkukura has joined #openstack-infra16:10
slaweqthx, You too AJaeger :)16:10
slaweqAJaeger: but now 694478 is removing noop jobs already16:11
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles.  https://review.opendev.org/69111416:11
openstackgerritSlawek Kaplonski proposed openstack/project-config master: Retire neutron-interconnection project  https://review.opendev.org/69447816:11
slaweqAJaeger: sorry, now ^^ it's rebased :)16:12
slaweqI forgot to push it to gerrit16:12
slaweqok, I need to go16:12
slaweqo/16:12
*** jamesmcarthur has joined #openstack-infra16:12
*** hwoarang has quit IRC16:12
AJaegerall good ;)16:12
*** hwoarang has joined #openstack-infra16:13
*** hwoarang has quit IRC16:14
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles.  https://review.opendev.org/69111416:14
*** hwoarang has joined #openstack-infra16:14
AJaegerconfig-core, a tiny cleanup - removal of obsolete jobs / template - for review, please https://review.opendev.org/69457016:15
*** EmilienM is now known as EmilienM|PTO16:16
*** goldyfruit___ has joined #openstack-infra16:16
*** piotrowskim has quit IRC16:18
*** goldyfruit_ has quit IRC16:18
*** jamesmcarthur has quit IRC16:19
*** rkukura has quit IRC16:19
openstackgerritMerged openstack/project-config master: Add noop-jobs template for neutron-interconnection project  https://review.opendev.org/69458216:19
*** tbarron has quit IRC16:20
*** jamesmcarthur has joined #openstack-infra16:20
*** kjackal has quit IRC16:21
*** jamesmcarthur has quit IRC16:21
*** michael-beaver has joined #openstack-infra16:22
*** armax has quit IRC16:24
*** dpawlik has joined #openstack-infra16:24
*** armax has joined #openstack-infra16:27
*** hashar has joined #openstack-infra16:28
mnaserAJaeger: are the afs volumes setup?16:29
mnaserinfra-root might help answer thjat so i know if i can start the governance work16:30
mnaserAJaeger: and maybe we can move these to in-repo instead of in project-config to make life easier too?16:32
AJaegermnaser: no - that's why I have a WIP on https://review.opendev.org/68158316:32
AJaegermnaser: but https://review.opendev.org/681582 is just the base job, so can merge anytime...16:32
clarkbsorry I would helpbut really suffering from post conference crud and trying to take it easy16:33
AJaegerclarkb: no urgency for any of these - get some rest!16:34
mnaserok i think that change makes sense overall for the base16:34
mnaserthe job name feels inconsistent but maybe im not getting it16:35
AJaegermnaser: why do you think it's inconsistent?16:35
AJaegermnaser: which part?16:36
*** jamesmcarthur has joined #openstack-infra16:36
AJaegermnaser: I'm bad at naming, so happy to explain and not offended if you rename - but depending on what you struggle with I can explain my reasoning16:36
mnaserAJaeger: i guess the fact the base job for the old one ius 'base-promote-tox-docs-static' and the new one is 'promote-tox-docs-site-base'16:38
*** gfidente has quit IRC16:38
mnaseri.e. id imagine `base-promote-tox-docs-site` makes a little bit more consistent sense16:38
mnaserbut opendev jobs have swapped around (suffixed with base)16:38
AJaegermost jobs have -base as suffix in that file, only the ones we will replace are the odd-ones. So, I would argue to keep what I proposed.16:41
mnasersounds good to me then!16:42
*** yamamoto has joined #openstack-infra16:42
mnaser+2d16:42
*** ykarel is now known as ykarel|away16:46
*** yamamoto has quit IRC16:51
*** rpittau is now known as rpittau|afk16:52
*** bnemec is now known as beekneemech16:52
*** kjackal has joined #openstack-infra16:52
*** udesale has quit IRC16:53
*** eharney has quit IRC16:55
*** ykarel has joined #openstack-infra16:56
*** ykarel|away has quit IRC16:58
*** rkukura has joined #openstack-infra17:00
*** gyee has joined #openstack-infra17:01
*** goldyfruit_ has joined #openstack-infra17:02
*** iurygregory has quit IRC17:03
*** dpawlik has quit IRC17:03
*** goldyfruit___ has quit IRC17:05
*** tbarron has joined #openstack-infra17:06
*** jamesmcarthur has quit IRC17:07
*** lpetrut has quit IRC17:10
*** jaosorior has quit IRC17:11
*** aedc has quit IRC17:16
*** kjackal has quit IRC17:18
*** mriedem is now known as mriedem_afk17:19
*** dpawlik has joined #openstack-infra17:19
*** ykarel is now known as ykarel|away17:20
*** armstrong has joined #openstack-infra17:20
*** hashar has quit IRC17:22
*** hashar has joined #openstack-infra17:22
*** dpawlik has quit IRC17:25
*** dpawlik has joined #openstack-infra17:35
*** ykarel|away has quit IRC17:35
*** kjackal has joined #openstack-infra17:36
*** jaosorior has joined #openstack-infra17:39
*** hashar has quit IRC17:39
*** dpawlik has quit IRC17:39
*** aedc has joined #openstack-infra17:55
*** jpena is now known as jpena|off17:58
mnaserinfra-root: anyone around to make a small delete favour?18:16
mordredmnaser: whazzup18:16
*** KeithMnemonic1 has quit IRC18:19
*** ralonsoh has quit IRC18:20
*** diablo_rojo__ has joined #openstack-infra18:25
*** jaosorior has quit IRC18:26
*** eharney has joined #openstack-infra18:26
*** kjackal has quit IRC18:33
*** diablo_rojo__ has quit IRC18:39
*** jaosorior has joined #openstack-infra18:40
*** ddurst- is now known as ddurst18:44
*** ociuhandu has quit IRC18:47
*** jaosorior has quit IRC18:50
*** kjackal has joined #openstack-infra18:52
openstackgerritDavid Shrewsbury proposed zuul/zuul master: WIP: test  https://review.opendev.org/69461918:55
openstackgerritDavid Shrewsbury proposed zuul/zuul master: WIP: test  https://review.opendev.org/69461918:55
*** pcaruana has quit IRC18:56
*** ociuhandu has joined #openstack-infra19:00
*** KeithMnemonic1 has joined #openstack-infra19:00
*** tesseract has quit IRC19:04
*** ociuhandu has quit IRC19:10
ShrewsOk, so 694619 seems to fix the zuul-stream-functional jobs (except for 2.8 which still fails in post for some reason). What I *believe* is happening is that the zsf pre.yaml playbook copies the ansible inventory.yaml for the _executor_ to the test "controller" node. The controller then runs the 2.x test against the nodes, and fails. Since the executor version of the inventory uses 'auto' for the interpreter (totally valid since it is19:13
Shrewsusing ansible 2.8), that fails for the controller inventory which uses the older ansible versions. So we can either modify the inventory on-the-fly somehow in the pre.yaml, or force the executor to use the same version of ansible the controller will use.19:13
ShrewstristanC: pabelanger: mordred: ^^^^19:13
*** tosky has quit IRC19:14
Shrewsmy WIP above does the latter19:14
Shrewsthat's a lot of words. hopefully some of them made sense19:15
Shrewsand even though i posted all of that in the wrong channel, it doesn't seem to matter since everyone still seems to be in post-conference recovery   :/19:20
*** trident has quit IRC19:20
*** mriedem_afk is now known as mriedem19:23
*** trident has joined #openstack-infra19:29
*** armstrong has quit IRC19:30
*** mriedem has quit IRC19:37
*** mriedem has joined #openstack-infra19:42
openstackgerritDavid Moreau Simard proposed zuul/zuul master: DNM: Get output from ara command  https://review.opendev.org/69462219:48
*** lennyb has quit IRC20:00
*** kjackal has quit IRC20:05
*** kjackal has joined #openstack-infra20:08
*** pkopec has quit IRC20:08
*** goldyfruit___ has joined #openstack-infra20:09
*** goldyfruit_ has quit IRC20:12
openstackgerritDavid Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2  https://review.opendev.org/69462220:18
*** yolanda has quit IRC20:35
*** yolanda has joined #openstack-infra20:43
*** kjackal has quit IRC20:46
*** goldyfruit_ has joined #openstack-infra20:46
*** goldyfruit___ has quit IRC20:48
*** yamamoto has joined #openstack-infra20:49
*** slaweq has quit IRC20:50
*** yamamoto has quit IRC20:53
*** slaweq has joined #openstack-infra21:11
*** slaweq has quit IRC21:16
*** kjackal has joined #openstack-infra21:30
*** yamamoto has joined #openstack-infra21:48
*** kopecmartin is now known as kopecmartin|off22:01
*** kaisers has joined #openstack-infra22:30
*** kaisers1 has quit IRC22:32
*** yamamoto has quit IRC22:49
*** KeithMnemonic1 has quit IRC22:50
*** michael-beaver has quit IRC22:52
*** kjackal has quit IRC23:01
*** KeithMnemonic has quit IRC23:08
*** slaweq has joined #openstack-infra23:11
*** slaweq has quit IRC23:16
*** rcernin has joined #openstack-infra23:17
*** jamesmcarthur has joined #openstack-infra23:18
*** Goneri has quit IRC23:41
*** jamesmcarthur has quit IRC23:43
*** jamesmcarthur has joined #openstack-infra23:47
*** mriedem has quit IRC23:50
*** jamesmcarthur has quit IRC23:55
*** rh-jelabarre has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!