| *** ociuhandu has joined #openstack-infra | 00:01 | |
| *** redrobot has quit IRC | 00:03 | |
| *** rcernin has quit IRC | 00:03 | |
| *** gyee has quit IRC | 00:03 | |
| *** dklyle has quit IRC | 00:03 | |
| *** dchen has quit IRC | 00:03 | |
| *** ianychoi has quit IRC | 00:03 | |
| *** tbachman has quit IRC | 00:03 | |
| *** yoctozepto has quit IRC | 00:03 | |
| *** snapiri has quit IRC | 00:03 | |
| *** gfidente|afk has quit IRC | 00:03 | |
| *** benj_ has quit IRC | 00:03 | |
| *** yamamoto has quit IRC | 00:03 | |
| *** dwilde has quit IRC | 00:03 | |
| *** cloudnull has quit IRC | 00:03 | |
| *** openstackgerrit has quit IRC | 00:03 | |
| *** elod has quit IRC | 00:03 | |
| *** bdodd has quit IRC | 00:03 | |
| *** SotK has quit IRC | 00:03 | |
| *** irclogbot_0 has quit IRC | 00:03 | |
| *** amoralej|off has quit IRC | 00:03 | |
| *** antonym has quit IRC | 00:03 | |
| *** spotz has quit IRC | 00:03 | |
| *** soniya29 has quit IRC | 00:03 | |
| *** arne_wiebalck has quit IRC | 00:03 | |
| *** rledisez has quit IRC | 00:03 | |
| *** hamalq has quit IRC | 00:03 | |
| *** rlandy|bbl has quit IRC | 00:03 | |
| *** auristor has quit IRC | 00:03 | |
| *** jaicaa has quit IRC | 00:03 | |
| *** tkajinam has quit IRC | 00:03 | |
| *** dviroel has quit IRC | 00:03 | |
| *** HenryG has quit IRC | 00:03 | |
| *** rajinir has quit IRC | 00:03 | |
| *** jamespage has quit IRC | 00:03 | |
| *** StevenK has quit IRC | 00:03 | |
| *** guilhermesp has quit IRC | 00:03 | |
| *** srwilkers has quit IRC | 00:03 | |
| *** gagehugo has quit IRC | 00:03 | |
| *** rm_work has quit IRC | 00:03 | |
| *** walshh_ has quit IRC | 00:03 | |
| *** donnyd has quit IRC | 00:03 | |
| *** persia has quit IRC | 00:03 | |
| *** seongsoocho has quit IRC | 00:03 | |
| *** cjohnston has quit IRC | 00:03 | |
| *** thedac has quit IRC | 00:03 | |
| *** mattmceuen has quit IRC | 00:03 | |
| *** portdirect has quit IRC | 00:03 | |
| *** coreycb has quit IRC | 00:03 | |
| *** beisner has quit IRC | 00:03 | |
| *** gmann has quit IRC | 00:03 | |
| *** melwitt has quit IRC | 00:03 | |
| *** amorin has quit IRC | 00:03 | |
| *** ttx has quit IRC | 00:03 | |
| *** mordred has quit IRC | 00:03 | |
| *** Adri2000 has quit IRC | 00:03 | |
| *** larsks has quit IRC | 00:03 | |
| *** rlandy|bbl has joined #openstack-infra | 00:04 | |
| *** hamalq has joined #openstack-infra | 00:05 | |
| *** auristor has joined #openstack-infra | 00:05 | |
| *** jaicaa has joined #openstack-infra | 00:05 | |
| *** tkajinam has joined #openstack-infra | 00:05 | |
| *** dviroel has joined #openstack-infra | 00:05 | |
| *** HenryG has joined #openstack-infra | 00:05 | |
| *** rajinir has joined #openstack-infra | 00:05 | |
| *** jamespage has joined #openstack-infra | 00:05 | |
| *** StevenK has joined #openstack-infra | 00:05 | |
| *** guilhermesp has joined #openstack-infra | 00:05 | |
| *** srwilkers has joined #openstack-infra | 00:05 | |
| *** walshh_ has joined #openstack-infra | 00:05 | |
| *** beisner has joined #openstack-infra | 00:05 | |
| *** gagehugo has joined #openstack-infra | 00:05 | |
| *** rm_work has joined #openstack-infra | 00:05 | |
| *** donnyd has joined #openstack-infra | 00:05 | |
| *** persia has joined #openstack-infra | 00:05 | |
| *** seongsoocho has joined #openstack-infra | 00:05 | |
| *** cjohnston has joined #openstack-infra | 00:05 | |
| *** portdirect has joined #openstack-infra | 00:05 | |
| *** thedac has joined #openstack-infra | 00:05 | |
| *** mattmceuen has joined #openstack-infra | 00:05 | |
| *** coreycb has joined #openstack-infra | 00:05 | |
| *** gmann has joined #openstack-infra | 00:05 | |
| *** melwitt has joined #openstack-infra | 00:05 | |
| *** amorin has joined #openstack-infra | 00:05 | |
| *** ttx has joined #openstack-infra | 00:05 | |
| *** mordred has joined #openstack-infra | 00:05 | |
| *** Adri2000 has joined #openstack-infra | 00:05 | |
| *** larsks has joined #openstack-infra | 00:05 | |
| *** redrobot has joined #openstack-infra | 00:06 | |
| *** yamamoto has joined #openstack-infra | 00:06 | |
| *** rcernin has joined #openstack-infra | 00:06 | |
| *** dwilde has joined #openstack-infra | 00:06 | |
| *** gyee has joined #openstack-infra | 00:06 | |
| *** cloudnull has joined #openstack-infra | 00:06 | |
| *** dklyle has joined #openstack-infra | 00:06 | |
| *** amoralej|off has joined #openstack-infra | 00:06 | |
| *** dchen has joined #openstack-infra | 00:06 | |
| *** ianychoi has joined #openstack-infra | 00:06 | |
| *** tbachman has joined #openstack-infra | 00:06 | |
| *** elod has joined #openstack-infra | 00:06 | |
| *** spotz has joined #openstack-infra | 00:06 | |
| *** bdodd has joined #openstack-infra | 00:06 | |
| *** yoctozepto has joined #openstack-infra | 00:06 | |
| *** snapiri has joined #openstack-infra | 00:06 | |
| *** gfidente|afk has joined #openstack-infra | 00:06 | |
| *** SotK has joined #openstack-infra | 00:06 | |
| *** irclogbot_0 has joined #openstack-infra | 00:06 | |
| *** antonym has joined #openstack-infra | 00:06 | |
| *** soniya29 has joined #openstack-infra | 00:06 | |
| *** arne_wiebalck has joined #openstack-infra | 00:06 | |
| *** rledisez has joined #openstack-infra | 00:06 | |
| *** benj_ has joined #openstack-infra | 00:06 | |
| *** mordred has quit IRC | 00:07 | |
| *** ociuhandu has quit IRC | 00:07 | |
| *** tosky has quit IRC | 00:07 | |
| *** cyberpear has quit IRC | 00:09 | |
| *** wolsen has quit IRC | 00:09 | |
| *** guilhermesp has quit IRC | 00:09 | |
| *** donnyd has quit IRC | 00:09 | |
| *** coreycb has quit IRC | 00:09 | |
| *** gmann has quit IRC | 00:09 | |
| *** donnyd has joined #openstack-infra | 00:10 | |
| *** guilhermesp has joined #openstack-infra | 00:10 | |
| *** irclogbot_0 has quit IRC | 00:10 | |
| *** coreycb has joined #openstack-infra | 00:11 | |
| *** gmann has joined #openstack-infra | 00:11 | |
| *** irclogbot_0 has joined #openstack-infra | 00:11 | |
| *** cyberpear has joined #openstack-infra | 00:12 | |
| *** gfidente|afk has quit IRC | 00:30 | |
| *** hamalq has quit IRC | 00:40 | |
| *** mordred has joined #openstack-infra | 00:43 | |
| *** jamesmcarthur has joined #openstack-infra | 00:44 | |
| *** mordred has quit IRC | 00:51 | |
| *** mordred has joined #openstack-infra | 01:03 | |
| *** jamesmcarthur has quit IRC | 01:25 | |
| *** jamesmcarthur has joined #openstack-infra | 01:30 | |
| *** jamesmcarthur has quit IRC | 01:30 | |
| *** jamesmcarthur has joined #openstack-infra | 01:30 | |
| *** wolsen has joined #openstack-infra | 01:36 | |
| *** cloudnull has quit IRC | 01:49 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** jamesmcarthur has joined #openstack-infra | 02:03 | |
| *** rlandy|bbl is now known as rlandy | 02:05 | |
| *** jamesmcarthur has quit IRC | 02:07 | |
| *** jamesmcarthur has joined #openstack-infra | 02:31 | |
| *** cloudnull has joined #openstack-infra | 02:35 | |
| *** jamesmcarthur has quit IRC | 02:35 | |
| *** ianychoi has quit IRC | 02:42 | |
| *** ociuhandu has joined #openstack-infra | 02:43 | |
| *** cloudnull has quit IRC | 02:46 | |
| *** ociuhandu has quit IRC | 02:47 | |
| *** jamesmcarthur has joined #openstack-infra | 02:51 | |
| *** ysandeep|out is now known as ysandeep | 03:06 | |
| *** rlandy has quit IRC | 03:08 | |
| *** rcernin has quit IRC | 03:11 | |
| *** rcernin has joined #openstack-infra | 03:12 | |
| *** cloudnull has joined #openstack-infra | 03:21 | |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** ramishra has quit IRC | 03:51 | |
| *** ricolin_ has joined #openstack-infra | 03:52 | |
| *** ramishra has joined #openstack-infra | 03:52 | |
| *** cloudnull has quit IRC | 03:53 | |
| *** cloudnull has joined #openstack-infra | 03:55 | |
| *** cloudnull has quit IRC | 04:02 | |
| *** cloudnull has joined #openstack-infra | 04:05 | |
| *** armax has quit IRC | 04:07 | |
| *** gyee has quit IRC | 04:12 | |
| *** ianychoi has joined #openstack-infra | 04:29 | |
| adriant | I'm curious, is there a 'good' reason any storyboard uses shift+enter to actually submit an issue comment when everywhere else on the internet shift+enter is newline? | 04:39 |
|---|---|---|
| *** dave-mccowan has quit IRC | 04:40 | |
| adriant | I keep accidentally submitting comments early before I mean to because of that, and it is immensely frustrating because everywhere else that key press combination is always new line, and storyboard doesn't let me go back and edit those comments. :/ | 04:41 |
| *** ysandeep is now known as ysandeep|afk | 04:49 | |
| adriant | https://storyboard.openstack.org/#!/story/2000504 < seems to incorrectly mark 'edit comments' as merged even though the patch in question is about comment filtering :( | 04:52 |
| *** jamesmcarthur has quit IRC | 05:13 | |
| adriant | oh wait, no, it was merged, ugh, what did I click on | 05:13 |
| adriant | Oh, reverted | 05:14 |
| *** ysandeep|afk is now known as ysandeep | 05:21 | |
| adriant | Odd, so it looks like the edit comments patch was added back again with https://review.opendev.org/c/opendev/storyboard-webclient/+/343539 but never enabled? | 05:33 |
| * adriant is confused about storyboard | 05:35 | |
| *** psachin has quit IRC | 05:40 | |
| *** zbr has quit IRC | 06:09 | |
| *** zbr has joined #openstack-infra | 06:11 | |
| *** psachin has joined #openstack-infra | 06:22 | |
| *** rcernin has quit IRC | 07:05 | |
| *** jamesmcarthur has joined #openstack-infra | 07:11 | |
| *** eolivare has joined #openstack-infra | 07:12 | |
| *** jamesmcarthur has quit IRC | 07:15 | |
| *** jcapitao has joined #openstack-infra | 07:43 | |
| *** rcernin has joined #openstack-infra | 07:43 | |
| *** ysandeep is now known as ysandeep|lunch | 07:47 | |
| *** rcernin has quit IRC | 07:48 | |
| *** rcernin has joined #openstack-infra | 07:49 | |
| *** rcernin has quit IRC | 07:54 | |
| *** sboyron has joined #openstack-infra | 07:54 | |
| *** dklyle has quit IRC | 07:55 | |
| *** ralonsoh has joined #openstack-infra | 07:55 | |
| *** dchen has quit IRC | 08:07 | |
| *** rpittau|afk is now known as rpittau | 08:09 | |
| *** andrewbonney has joined #openstack-infra | 08:12 | |
| *** amoralej|off is now known as amoralej | 08:13 | |
| *** xek has joined #openstack-infra | 08:25 | |
| *** rcernin has joined #openstack-infra | 08:26 | |
| *** rcernin has quit IRC | 08:31 | |
| *** yamamoto has quit IRC | 08:33 | |
| *** jpena|off is now known as jpena | 08:34 | |
| *** tosky has joined #openstack-infra | 08:42 | |
| *** ysandeep|lunch is now known as ysandeep | 08:47 | |
| *** lpetrut has joined #openstack-infra | 08:50 | |
| *** hashar has joined #openstack-infra | 08:52 | |
| *** lucasagomes has joined #openstack-infra | 09:04 | |
| *** ramishra has quit IRC | 09:04 | |
| *** ramishra has joined #openstack-infra | 09:06 | |
| *** yamamoto has joined #openstack-infra | 09:13 | |
| *** nightmare_unreal has joined #openstack-infra | 09:17 | |
| *** yamamoto has quit IRC | 09:18 | |
| *** rcernin has joined #openstack-infra | 09:28 | |
| *** ociuhandu has joined #openstack-infra | 09:33 | |
| *** rcernin has quit IRC | 09:33 | |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** ociuhandu has quit IRC | 09:44 | |
| *** ociuhandu has joined #openstack-infra | 09:50 | |
| *** jamesmcarthur has joined #openstack-infra | 10:01 | |
| *** jamesmcarthur has quit IRC | 10:34 | |
| *** dtantsur|afk is now known as dtantsur | 10:38 | |
| *** rcernin has joined #openstack-infra | 10:45 | |
| *** ociuhandu has quit IRC | 10:46 | |
| *** psachin has quit IRC | 10:58 | |
| *** gfidente has joined #openstack-infra | 11:01 | |
| *** ociuhandu has joined #openstack-infra | 11:17 | |
| *** rcernin has quit IRC | 11:17 | |
| *** ociuhandu has quit IRC | 11:26 | |
| *** ociuhandu has joined #openstack-infra | 11:27 | |
| *** rcernin has joined #openstack-infra | 11:30 | |
| *** jcapitao is now known as jcapitao_lunch | 11:37 | |
| *** yamamoto has joined #openstack-infra | 11:37 | |
| *** hashar is now known as hasharLunch | 12:04 | |
| *** jamesdenton has quit IRC | 12:07 | |
| *** jamesdenton has joined #openstack-infra | 12:08 | |
| *** geguileo has joined #openstack-infra | 12:15 | |
| geguileo | Hi, it seems like the centos-8 nodeset is using an incorrect wheel mirror which is making some jobs fails... | 12:20 |
| geguileo | it's trying https://mirror.bhs1.ovh.opendev.org/wheel/centos-8.3-x86_64 | 12:20 |
| geguileo | and it should be https://mirror.bhs1.ovh.opendev.org/wheel/centos-8-x86_64 | 12:20 |
| geguileo | Can someone point me to where I can fix that | 12:21 |
| *** rcernin has quit IRC | 12:31 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** ricolin_ has quit IRC | 12:42 | |
| *** rcernin has joined #openstack-infra | 12:45 | |
| *** rlandy has joined #openstack-infra | 12:46 | |
| *** rlandy is now known as rlandy|ruck | 12:46 | |
| *** jcapitao_lunch is now known as jcapitao | 12:54 | |
| *** zzzeek has quit IRC | 13:00 | |
| *** rcernin has quit IRC | 13:00 | |
| *** zzzeek has joined #openstack-infra | 13:00 | |
| *** rcernin has joined #openstack-infra | 13:05 | |
| *** slittle1 has quit IRC | 13:10 | |
| *** slittle1 has joined #openstack-infra | 13:11 | |
| *** nightmare_unreal has quit IRC | 13:14 | |
| *** yamamoto has quit IRC | 13:20 | |
| *** amoralej is now known as amoralej|lunch | 13:28 | |
| *** derekh has quit IRC | 13:29 | |
| *** derekh has joined #openstack-infra | 13:30 | |
| *** slittle1 has quit IRC | 13:30 | |
| *** rcernin has quit IRC | 13:30 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| sean-k-mooney | hi is review.opendev.org slow at the moment? or in general over the last few days | 13:39 |
| sean-k-mooney | i started up my trid party ci a few days ago and its timing out clone nova now | 13:39 |
| sean-k-mooney | i just bumped the timeout in the zull config to 10 mins to see if that helps | 13:40 |
| sean-k-mooney | looks like it took real 5m42.879s | 13:40 |
| sean-k-mooney | user 1m30.738s | 13:40 |
| sean-k-mooney | sys 0m12.767s | 13:40 |
| sean-k-mooney | so i guess i was just crossing the default 5min timeout windows | 13:41 |
| sean-k-mooney | so that likely will be all that is needed to fix it just wondering if that was an infra issue or just a side effect of the nova being 1.01GB in size | 13:42 |
| frickler | sean-k-mooney: you should not clone from review.opendev.org in general if it can be avoided. clone from opendev.org and then add review as remote | 13:44 |
| sean-k-mooney | this is for zuul | 13:47 |
| sean-k-mooney | its cloning the repo form the gerrit connection | 13:47 |
| sean-k-mooney | "git fetch -f --prune --tags origin" did not complete in 300 secs | 13:49 |
| sean-k-mooney | then zuul deleted the nova repo | 13:49 |
| sean-k-mooney | and then it clonned it again | 13:49 |
| sean-k-mooney | but that took over 300 second too | 13:49 |
| sean-k-mooney | and failed | 13:49 |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| frickler | sean-k-mooney: hmm, then maybe ask in #zuul how the full clone can be avoided. we do use pre-cloned copies of all repos baked into our images which are updated daily, but they be overkill for a small 3rd party ci | 13:52 |
| frickler | s/they be/that may be/ | 13:53 |
| sean-k-mooney | well the full clone is on the zuul executor/merger not the instances | 13:53 |
| sean-k-mooney | so its caching it for me but every so often it fails an operation and deletes it and clones fresh when it cant recover | 13:54 |
| sean-k-mooney | i set the timeout to 10 mins so that should help | 13:54 |
| sean-k-mooney | i normally only see this as an issue when my copy gets out of date e.g. if i have taken the ci offline for a while | 13:54 |
| sean-k-mooney | i had it offline for the last 3 weeks and only brought it back on monday | 13:55 |
| sean-k-mooney | but i forgot to do a pull on all the repos first | 13:55 |
| sean-k-mooney | nova is the only one that was having issue due to its size | 13:55 |
| sean-k-mooney | anyway no worries just wanted to make sure ye were not having a bandwith issue or ddos currently | 13:57 |
| frickler | hmm, o.k., even then it might be a useful feature for zuul to be able to do a fresh clone from a better source | 13:57 |
| sean-k-mooney | e.g. use a mirror for the inital clone | 13:57 |
| sean-k-mooney | ya possible | 13:57 |
| sean-k-mooney | frickler: for what its worth since i have my http password set it is using https to clone instead of ssh | 13:59 |
| *** yamamoto has quit IRC | 13:59 | |
| sean-k-mooney | so it kind of can do that excpet we dont have teh ablity to say where to clone form when doing http | 14:04 |
| *** amoralej|lunch is now known as amoralej | 14:24 | |
| *** dave-mccowan has joined #openstack-infra | 14:27 | |
| *** ociuhandu_ has joined #openstack-infra | 14:28 | |
| fungi | adriant: we don't enable comment editing on our storyboard deployment, but the feature exists for sites where the users/administrators don't care that someone could go back and alter a comment and change the entire meaning of the subsequent discussion | 14:30 |
| *** ociuhandu has quit IRC | 14:31 | |
| fungi | adriant: as for the webclient on the whole, in yesterday's storyboard meeting we started talking about redoing it, because there's really no upgrade path from angularjs, so whether we go to angular(2) or react or something like vue, it's going to need to be mostly recreated from the ground up | 14:31 |
| *** jamesmcarthur has joined #openstack-infra | 14:32 | |
| fungi | geguileo: it's probably happening here? https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/vars/CentOS.yaml i guess ansible_distribution is now including the .3 for some reason? | 14:34 |
| dansmith | is there any possibility that the zuul comments are going to be unwrapped at some point in the new gerrit? | 14:34 |
| dansmith | they seem to have gone full widescreen with the UI, but zuul's comments with all the links are really hard to read | 14:35 |
| *** jamesmcarthur has quit IRC | 14:36 | |
| fungi | sean-k-mooney: the gerrit server doesn't look like it's under particularly heavy load, so at least not like the other recent occasions where it started getting really slow: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=26&rra_id=all | 14:37 |
| fungi | sean-k-mooney: i believe we increased the clone timeout on our mergers/executors and recommended others do the same | 14:38 |
| fungi | sean-k-mooney: frickler: https://review.opendev.org/766400 | 14:38 |
| fungi | dansmith: the new gerrit ui is basically doing what the old ui did by default. for old gerrit we wrote a hacky js overlay to parse and rewrite comments which looked like zuul reports. ianw has upstreamed a new gerrit plugin to perform similarly, but we need a gerrit restart to pull it in and are hesitant to do that when the system is under such heavy use like this week | 14:41 |
| dansmith | fungi: ack.. can zuul post an html comment that just links the test names where they need to go instead of name: link ? | 14:46 |
| dansmith | or maybe md or rst or whatever the language is | 14:46 |
| fungi | i don't recall for sure why we don't do that, but i suspect it's that gerrit may not allow html in review comments | 14:47 |
| dansmith | it allows some minor formatting at least, so I figured there'd be a way | 14:47 |
| fungi | (i haven't tried personally) | 14:47 |
| fungi | but the plugin is also bringing back the results table, which a lot of folks seemed to miss from our old overlay | 14:48 |
| *** rpittau is now known as rpittau|afk | 14:48 | |
| dansmith | yeah | 14:49 |
| dansmith | I hated the table because it reshapes the UI a couple seconds after page load, | 14:49 |
| dansmith | which means I go to click somewhere (not even on the table) and it shifts underneath me | 14:49 |
| dansmith | and I *hate* that behavior :) | 14:49 |
| sean-k-mooney | fungi: ya the docs suggest raising it for large repos of which nova is one | 14:50 |
| fungi | dansmith: yeah, i'm not a fan of pregressive rendering either. this one will put it in a sub-tab like files and findings currently uses | 14:50 |
| sean-k-mooney | fungi: yep https://review.opendev.org/c/opendev/system-config/+/766400/ was the same behavior i saw | 14:51 |
| dansmith | fungi: oh then perfect | 14:51 |
| fungi | the upstream ci integration gerrit's working on will use a similar sub-tab there, so we're trying to be consistent | 14:51 |
| sean-k-mooney | and i also set it to 600 sec. and it seams to be fine now | 14:51 |
| fungi | sean-k-mooney: we speculated that the repository size increase from the gerrit conversion to notedb is what made the repos start taking a bit longer to clone | 14:54 |
| sean-k-mooney | ya so in my case it took 5:34sec manually i think | 14:55 |
| sean-k-mooney | so it just crossed that treashold | 14:55 |
| sean-k-mooney | although i suspect before there were ocation where it might have failed when ye were under heavy load | 14:56 |
| sean-k-mooney | 10mins should leave plenty of buffer | 14:56 |
| fungi | also once the cache is warmed you don't really see it because the executors/mergers don't want to re-clone unless the repo is corrupted or missing | 14:56 |
| sean-k-mooney | yep | 14:57 |
| fungi | normally they just pull new refs in | 14:57 |
| sean-k-mooney | ya and they do that incrementally as the gerrit merge events come in | 14:57 |
| sean-k-mooney | this was just an issue as i had my ci offline over the break | 14:57 |
| sean-k-mooney | when i inially deployed it i had to manually clone nova too | 14:58 |
| sean-k-mooney | i just fogot to do a fetch on them before starting it again | 14:58 |
| fungi | we basically ddos'ed our own gerrit by clearing all the repository caches on all our zuul executors and mergers (20 in total) and then none of them could clone nova within the 5-minute timeout so just kept hammering it over and over | 14:58 |
| sean-k-mooney | hum looks like the cirros image hosting is down | 15:00 |
| sean-k-mooney | http://download.cirros-cloud.net/0/5/1-x86_64-disk.img | 15:00 |
| sean-k-mooney | is failing to connect | 15:00 |
| sean-k-mooney | ya http://cirros-cloud.net/ is down | 15:01 |
| sean-k-mooney | hum apprently its just me. | 15:02 |
| frickler | sean-k-mooney: newer cirros images are available on github, too. cirros-cloud.net may be rate-limited, too, cheap hosting ;) | 15:02 |
| frickler | https://github.com/cirros-dev/cirros/releases | 15:03 |
| sean-k-mooney | i need to figure out how to use the file cache role to cache it locally | 15:03 |
| sean-k-mooney | but ya i can change it to github fo rnow | 15:03 |
| sean-k-mooney | im just ussing devstack default | 15:03 |
| sean-k-mooney | would it be worth updting that | 15:04 |
| frickler | no, cirros-cloud.net is still the official primary source. we work around the limits by pre-caching those images within our own CI images, too | 15:04 |
| sean-k-mooney | ya i can cache them on the executor and have the pre job sync it | 15:05 |
| fungi | or set up a local caching proxy | 15:06 |
| sean-k-mooney | i am plannign to set up some local mirros anyway so i might just reverse proxy that and have apache cache it too | 15:06 |
| sean-k-mooney | yep | 15:06 |
| sean-k-mooney | i have a devpi pip cache deployed but havent enabled it yet | 15:07 |
| *** lpetrut has quit IRC | 15:07 | |
| sean-k-mooney | planing ot do the same for .debs and rpms soon and image is not had to add | 15:07 |
| *** ociuhandu_ has quit IRC | 15:09 | |
| *** gfidente has quit IRC | 15:12 | |
| *** jamesmcarthur has joined #openstack-infra | 15:14 | |
| *** ociuhandu has joined #openstack-infra | 15:22 | |
| *** openstackgerrit has joined #openstack-infra | 15:29 | |
| openstackgerrit | Merged openstack/project-config master: update gentoo from python 3.6 to python 3.8 https://review.opendev.org/c/openstack/project-config/+/770828 | 15:29 |
| *** gfidente has joined #openstack-infra | 15:46 | |
| *** dklyle has joined #openstack-infra | 15:50 | |
| *** tbachman has left #openstack-infra | 15:58 | |
| *** yamamoto has joined #openstack-infra | 15:58 | |
| *** armax has joined #openstack-infra | 16:00 | |
| dansmith | man the new gerrit is sooo slow | 16:03 |
| dansmith | I'm really warming to the UI changes, but dang | 16:03 |
| *** yamamoto has quit IRC | 16:04 | |
| sean-k-mooney | dansmith: did you notice you can expand out each fo the files on the top level review page without going into them | 16:05 |
| sean-k-mooney | still not sure if its a useful feature or not | 16:05 |
| dansmith | I'm just talking about leaving a single vote taking like 30+ seconds to submit | 16:05 |
| sean-k-mooney | ya i was just wondering what you tought of the that new feature | 16:06 |
| sean-k-mooney | i havent decied if i like it yet or not | 16:06 |
| sean-k-mooney | for longer review i dont think its useful but shorter seams kind of neat | 16:06 |
| *** CrayZee has joined #openstack-infra | 16:10 | |
| *** CrayZee has quit IRC | 16:11 | |
| fungi | consistently slow or just now slow? looks like we're getting slammed with something at the moment: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=26&rra_id=all | 16:11 |
| *** snapiri has quit IRC | 16:14 | |
| fungi | going to delete the older of our two gentoo-17-0-systemd images now that 770828 has merged, but looks like the images we have are 295 and 294 days old so there's a good chance something else is also broken with image builds for it | 16:19 |
| *** slaweq has joined #openstack-infra | 16:19 | |
| fungi | so it's presumably already looping trying to replace those | 16:19 |
| fungi | i'll see if i can spot a problem in the logs | 16:19 |
| fungi | er, that was intended for #opendev, but whatevs | 16:20 |
| *** ysandeep is now known as ysandeep|dinner | 16:21 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Un-pause Gentoo image builds https://review.opendev.org/c/openstack/project-config/+/771031 | 16:32 |
| *** ysandeep|dinner is now known as ysandeep|away | 16:33 | |
| *** ricolin_ has joined #openstack-infra | 16:35 | |
| *** ricolin_ has quit IRC | 16:35 | |
| *** dtantsur is now known as dtantsur|afk | 16:41 | |
| *** iurygregory has quit IRC | 16:41 | |
| *** ociuhandu_ has joined #openstack-infra | 16:53 | |
| *** ociuhandu has quit IRC | 16:56 | |
| *** ociuhandu_ has quit IRC | 16:58 | |
| *** lucasagomes has quit IRC | 17:00 | |
| *** iurygregory has joined #openstack-infra | 17:06 | |
| dansmith | sean-k-mooney: sorry, just saw this.. I feel like that feature is cool, but I haven't yet used it... maybe for patches that just add files? | 17:08 |
| *** hasharLunch is now known as hashar | 17:08 | |
| dansmith | fungi: it's been consistently slow all week.. 30+ seconds to leave a single vote, often at least that long to submit a change with git-review | 17:08 |
| fungi | got it, if it's slow right now with system load down below 10, then it's probably not only load related | 17:09 |
| *** ociuhandu has joined #openstack-infra | 17:11 | |
| dansmith | just tried a comment and it was much faster than an hour ago | 17:12 |
| *** slaweq has quit IRC | 17:12 | |
| fungi | an hour ago the system load was up around 40 | 17:12 |
| fungi | something was hitting the server hard | 17:13 |
| *** xek has quit IRC | 17:13 | |
| *** ociuhandu has quit IRC | 17:16 | |
| dansmith | ack, cool | 17:16 |
| *** jcapitao has quit IRC | 17:17 | |
| fungi | i've been focused on afs-related fires though so didn't have a chance to look into what was driving that | 17:18 |
| dansmith | well, it has definitely varied over the week, but more often than not, I've switched away to something else waiting for the vote dialog to finish | 17:19 |
| dansmith | but maybe that was all load-attack related | 17:19 |
| fungi | we've definitely seen the new version of gerrit get bogged down when handling lots of large queries, so odds are it's a script/tool/bot/ci system/something along those lines and not an intentional attack | 17:23 |
| fungi | previously we found there was a researcher at some university in canada trying to pull copies of every change in our system since the beginning of time | 17:24 |
| fungi | took a while to track them down and get in touch with them through the university administration to suggest alternative approaches | 17:25 |
| fungi | initially they were connecting from home and from some cloud provider, and this was all anonymous queries so we had no account associated with it | 17:26 |
| fungi | wasn't until we'd blocked those ip addresses and saw the same queries start coming from a university computer lab network that we had something more to go on | 17:29 |
| sean-k-mooney | dansmith: it can do side by side or inline diffes but ya its really only usful for really small changes or new files | 17:29 |
| sean-k-mooney | fungi: have ye figured out how to supprot that reaserach that was tryign to look into how code reviews work | 17:30 |
| clarkb | now that we are on notedb the best way would be to go through git and our mirrors | 17:31 |
| clarkb | all of the content is in the repos now | 17:31 |
| clarkb | and you can get it without talking to gerrit | 17:31 |
| *** amoralej is now known as amoralej|off | 17:31 | |
| *** jpena is now known as jpena|off | 17:31 | |
| sean-k-mooney | ya true | 17:31 |
| sean-k-mooney | on a related note has gertty been updated to work with new gerrit. | 17:32 |
| sean-k-mooney | i havent used it in quite a while but ever few months i liek to try it again | 17:32 |
| clarkb | didn't it always work with new gerrit? | 17:32 |
| clarkb | (that was something I checked on before the upgrade) | 17:32 |
| sean-k-mooney | i dont think it works now | 17:32 |
| sean-k-mooney | ill have to reinstall it and see. | 17:33 |
| clarkb | I believe fungi and corvus use it and have used it snce the upgrade | 17:33 |
| sean-k-mooney | but i tried it after the upgrade and it didnt seame to be able to get the repos | 17:33 |
| clarkb | did you use basic auth instead of digest? | 17:33 |
| sean-k-mooney | it might be somthing on my end so | 17:33 |
| melwitt | one thing I'm bummed about with the new version is that it doesn't show cherry picks merged/unmerged status on the right hand side anymore | 17:33 |
| sean-k-mooney | clarkb: i just hae my passward set but now auth type so it might be defaulting to digest so that could be it | 17:35 |
| sean-k-mooney | ah yes ye updated the config template at some point https://opendev.org/ttygroup/gertty/src/branch/master/examples/opendev-gertty.yaml#L12 | 17:36 |
| sean-k-mooney | it might help if i used https://review.opendev.org/ instead of https://review.openstack.org/ | 17:37 |
| *** dave-mccowan has quit IRC | 17:38 | |
| *** hashar has quit IRC | 17:38 | |
| openstackgerrit | Merged openstack/project-config master: Un-pause Gentoo image builds https://review.opendev.org/c/openstack/project-config/+/771031 | 17:41 |
| sean-k-mooney | clarkb: i deleted the sqlight db and change the auth and it might be workign now | 17:44 |
| sean-k-mooney | thanks for the pointer in the right direction | 17:44 |
| corvus | sean-k-mooney: cool, let me know if you have probs; most of the new stuff should be supported | 17:44 |
| sean-k-mooney | i ported my home directory form a differen distro and i was uing a much older verion so im not really surpiesed i had issues withthe existing db and config | 17:47 |
| sean-k-mooney | that siad http://paste.openstack.org/show/801675/ is not super promising | 17:50 |
| fungi | sean-k-mooney: don't use /p in the url, i think? | 17:53 |
| fungi | what do you have your gerrit url set to? | 17:53 |
| sean-k-mooney | url: https://review.opendev.org/ | 17:53 |
| fungi | you might need a newer gertty, it's possible that was fixed since the last tagged release on pypi | 17:54 |
| fungi | tried installing from https://opendev.org/ttygroup/gertty master branch? | 17:54 |
| sean-k-mooney | i have 1.6 but i can grab the master copy | 17:54 |
| fungi | yeah, looks like https://review.opendev.org/763885 was the fix for that | 17:56 |
| *** ralonsoh has quit IRC | 17:56 | |
| sean-k-mooney | yep running it with -d and lookign at the log it looks like its cloneing the repos correclty now | 17:57 |
| sean-k-mooney | cool | 17:57 |
| sean-k-mooney | so is this using the notdb info now or is it still using the rest api for the comments | 17:58 |
| sean-k-mooney | it would need to use the api to submit them but just wondering about other peopels comments | 17:58 |
| clarkb | it still uses the rest api | 17:58 |
| sean-k-mooney | they used to be stored in teh sqlight db right | 17:58 |
| clarkb | I think they will still be in sqlite | 17:58 |
| sean-k-mooney | ok good to know | 17:59 |
| sean-k-mooney | well i guess ill leave that sync so whiel i go finish up for the day. | 18:00 |
| *** eolivare has quit IRC | 18:02 | |
| *** eolivare has joined #openstack-infra | 18:02 | |
| *** derekh has quit IRC | 18:03 | |
| fungi | to be clear, notedb is basically an implementation detail of gerrit's metadata storage, it's not really an interface for interacting with gerrit | 18:06 |
| fungi | it can be convenient to treat it as a read-only api for some purposes | 18:08 |
| *** jamesmcarthur has quit IRC | 18:08 | |
| sean-k-mooney | fungi: yep i know i was wondering if ye were using it as a data source | 18:09 |
| sean-k-mooney | by the wway will gertty ratelimit itesfly whil its syncing | 18:10 |
| *** gyee has joined #openstack-infra | 18:14 | |
| erbarr | hello, I'm following docs.openstack.org/devstack/latest/networking.html#locally-accessible-guests using the one network device I want to setup but stack.sh is failing right after the port gets added to br-ex and the arping command gets executed, I'm using focal, any ideas? doing the Shared Guest Interface | 18:18 |
| fungi | erbarr: you might have more luck asking in #openstack-qa since devstack is a quality assurance team deliverable | 18:19 |
| erbarr | fungi: thanks, tried there fist | 18:19 |
| erbarr | i guess is late friday | 18:20 |
| fungi | that's probably some of it, but also don't be discouraged if it takes people longer than 10 minutes to see you've asked a question ;) | 18:21 |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| *** gfidente has quit IRC | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:33 | |
| *** icey has quit IRC | 18:48 | |
| *** andrewbonney has quit IRC | 19:00 | |
| erbarr | fungi: yea, i understand, thanks man I appreciate it | 19:16 |
| *** slaweq has joined #openstack-infra | 19:22 | |
| *** rcernin has joined #openstack-infra | 19:27 | |
| openstackgerrit | Ghanshyam proposed openstack/project-config master: Combine acl file for all interop source code repo https://review.opendev.org/c/openstack/project-config/+/771066 | 19:28 |
| *** rcernin has quit IRC | 19:31 | |
| *** dmellado has quit IRC | 19:47 | |
| *** dmellado has joined #openstack-infra | 19:48 | |
| dansmith | fungi: much slow again | 19:49 |
| clarkb | load jumped up a bit but not significantly | 19:50 |
| clarkb | fwiw I think a good portion of this is simply the difference between using a database tuned for making updates to disk and using git instead | 19:51 |
| clarkb | a while back I used cacti to show that we're basically not IO bound when slow | 19:51 |
| clarkb | since we do significantly more IO on the same device when we do things like backups | 19:51 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Set up access for #openinfra channel https://review.opendev.org/c/openstack/project-config/+/771073 | 19:55 |
| clarkb | it is possible that the peak IO ops/bw isn't always available | 19:58 |
| clarkb | and we slow if the volume access slows? | 19:58 |
| dansmith | clarkb: are you saying new gerrit keeps more things in git that used to be stored in a real db? | 19:58 |
| clarkb | dansmith: yes, almost everything is stored in git now and not in a real db | 19:59 |
| clarkb | the only thing stored in a real db now is what files you've reviewed | 19:59 |
| dansmith | that seems like "it worked in my docker" stuff | 19:59 |
| clarkb | google is using it for much larger gerrits than ours, but its all googley I'm sure | 19:59 |
| dansmith | yeah, running in a 1T ramdisk prolly ;P | 20:00 |
| dansmith | btw, | 20:00 |
| dansmith | I just did my 2021 corporate harassment training this morning.. I'm pretty sure you're not allowed to say "googley" | 20:00 |
| fungi | yeah, i expect google also doesn't use a normal file backend for their git repositories | 20:02 |
| *** auristor has quit IRC | 20:03 | |
| fungi | anyway, there are probably more tuning opportunities. we've done a few things which have helped, but there's still more we haven't given a shot yet | 20:04 |
| *** eolivare has quit IRC | 20:04 | |
| fungi | in part we were unsure how much improvement we were seeing, because we were also doing it at the lowest activity time of the year, and needed to see that ramp back up first | 20:05 |
| *** slaweq has quit IRC | 20:11 | |
| clarkb | googliness is apparently the word google uses | 20:11 |
| *** auristor has joined #openstack-infra | 20:23 | |
| fungi | not to be confused with goodliness | 20:26 |
| *** rcernin has joined #openstack-infra | 20:57 | |
| *** rcernin has quit IRC | 21:01 | |
| *** jamesmcarthur has joined #openstack-infra | 21:10 | |
| *** jamesmcarthur has quit IRC | 21:15 | |
| *** rlandy|ruck has quit IRC | 21:27 | |
| *** sboyron has quit IRC | 21:41 | |
| *** rcernin has joined #openstack-infra | 21:53 | |
| *** rcernin has quit IRC | 21:58 | |
| *** yamamoto has joined #openstack-infra | 22:01 | |
| *** rcernin has joined #openstack-infra | 22:04 | |
| *** yamamoto has quit IRC | 22:06 | |
| *** yamamoto has joined #openstack-infra | 22:09 | |
| *** jamesmcarthur has joined #openstack-infra | 22:18 | |
| *** jamesmcarthur has quit IRC | 22:18 | |
| *** ociuhandu has joined #openstack-infra | 22:20 | |
| *** jamesmcarthur has joined #openstack-infra | 22:22 | |
| *** ociuhandu has quit IRC | 22:25 | |
| *** erbarr has quit IRC | 22:38 | |
| *** erbarr has joined #openstack-infra | 22:38 | |
| *** logan- has quit IRC | 22:40 | |
| *** logan- has joined #openstack-infra | 22:53 | |
| *** yoctozepto5 has joined #openstack-infra | 22:54 | |
| *** yoctozepto has quit IRC | 22:55 | |
| *** yoctozepto5 is now known as yoctozepto | 22:55 | |
| *** mwhahaha has quit IRC | 22:57 | |
| *** vdrok has quit IRC | 22:57 | |
| *** hogepodge has quit IRC | 22:57 | |
| *** mwhahaha has joined #openstack-infra | 22:57 | |
| *** vdrok has joined #openstack-infra | 22:58 | |
| *** hogepodge has joined #openstack-infra | 22:58 | |
| *** yamamoto has quit IRC | 23:00 | |
| *** verdurin has quit IRC | 23:00 | |
| *** coreycb has quit IRC | 23:01 | |
| *** jrosser has quit IRC | 23:01 | |
| *** cyberpear has quit IRC | 23:01 | |
| *** coreycb has joined #openstack-infra | 23:02 | |
| *** jrosser has joined #openstack-infra | 23:03 | |
| *** cyberpear has joined #openstack-infra | 23:03 | |
| *** logan- has quit IRC | 23:04 | |
| *** yamamoto has joined #openstack-infra | 23:09 | |
| *** verdurin has joined #openstack-infra | 23:12 | |
| *** yoctozepto6 has joined #openstack-infra | 23:16 | |
| *** yoctozepto has quit IRC | 23:17 | |
| *** yoctozepto6 is now known as yoctozepto | 23:17 | |
| *** logan- has joined #openstack-infra | 23:19 | |
| *** dulek has quit IRC | 23:20 | |
| *** eandersson has quit IRC | 23:20 | |
| *** ysandeep|away has quit IRC | 23:20 | |
| *** aarents has quit IRC | 23:20 | |
| *** rakhmerov has quit IRC | 23:20 | |
| *** ddurst has quit IRC | 23:20 | |
| *** ddurst has joined #openstack-infra | 23:21 | |
| *** aarents has joined #openstack-infra | 23:21 | |
| *** eandersson has joined #openstack-infra | 23:21 | |
| *** ysandeep has joined #openstack-infra | 23:22 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
| *** yoctozepto4 has joined #openstack-infra | 23:42 | |
| *** yoctozepto has quit IRC | 23:44 | |
| *** yoctozepto4 is now known as yoctozepto | 23:44 | |
| *** jamesmcarthur has joined #openstack-infra | 23:45 | |
| *** dwilde has quit IRC | 23:49 | |
| *** jamesmcarthur has quit IRC | 23:50 | |
| *** dwilde has joined #openstack-infra | 23:50 | |
| *** jamesmcarthur has joined #openstack-infra | 23:57 | |
| *** yoctozepto4 has joined #openstack-infra | 23:58 | |
| *** yoctozepto has quit IRC | 23:59 | |
| *** yoctozepto4 is now known as yoctozepto | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!