| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 00:00 |
|---|---|---|
| *** jamesmcarthur has joined #openstack-infra | 00:04 | |
| *** diablo_rojo has quit IRC | 00:06 | |
| *** rfolco has joined #openstack-infra | 00:07 | |
| *** jamesmcarthur has quit IRC | 00:08 | |
| *** tosky has quit IRC | 00:10 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 00:13 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 00:16 |
| rpioso | git clone is of openstack/requirements.git is taking an unusually long amount of time. It seems to be stuck. Same happened with ironic.git a few minutes ago. Both from the same host. ping of opendev.org succeeds. Ideas? | 00:29 |
| clarkb | rpioso: this is a fresh clone of both repos? | 00:30 |
| rpioso | clarkb: Yes | 00:31 |
| clarkb | ok both work for me currently against whichever backend I am hitting. Now to try the individual backends | 00:31 |
| rpioso | clarkb: It worked normally at least once against ironic. | 00:32 |
| clarkb | rpioso: if you go to https://opendev.org in your browser and inspect the certification which giteaXX.opendev.org does it say you are talking to ? | 00:32 |
| clarkb | rpioso: this should be run from the same ip that was doing the git clone | 00:32 |
| rpioso | clarkb: I'm performing a devstack deployment on Ubuntu Bionic. It doesn't appear to have chrome, firefox, nor google-chrome. Is there a default browser installed? | 00:34 |
| clarkb | rpioso: that will depend on how you installed your system | 00:35 |
| clarkb | I wouldn't know | 00:36 |
| clarkb | you can use openssl s_client instead | 00:36 |
| clarkb | I am able to clone ironic and requirements from all 8 backends | 00:38 |
| *** weshay is now known as weshay_pto | 00:38 | |
| *** ociuhandu has joined #openstack-infra | 00:39 | |
| clarkb | if you can identify the backend you are talking to via the cert details that would be helpful then we can dig into that host to see if it has problem | 00:39 |
| rpioso | clarkb: Not familiar with that approach. I get several errors when I invoke "sudo openssl s_client opendev.org", with the first one being "139963203715520:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:110:" | 00:40 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 00:40 |
| clarkb | rpioso: `openssl s_client -host opendev.org -port 443` | 00:40 |
| clarkb | then look for CN = gitea0X.opendev.org | 00:41 |
| rpioso | subject=CN = gitea02.opendev.org | 00:41 |
| rpioso | Let me kick off another devstack deployment and see if it gets stuck again. | 00:42 |
| clarkb | k let me run some clones in a loop to try and reproduce. Note you can hit specific backends via https://gitea0X.opendev.org:3000 | 00:42 |
| clarkb | cacti shows a definite busy period from about 1600UTC to 2000UTC (roughly) | 00:45 |
| clarkb | but not a super busy period. I would expect the server to be happy during that time | 00:45 |
| rpioso | clarkb: The new deployment successfully cloned ironic. It's stuck on requirements. Before kicking it off, I confirmed CN still equaled gitea02.opendev.org. | 00:48 |
| clarkb | rpioso: ok my local attempts haven't failed yet | 00:48 |
| *** ociuhandu has quit IRC | 00:48 | |
| clarkb | I'm running while git clone ; rm -rf git_dir ; done | 00:48 |
| clarkb | against requirements | 00:48 |
| clarkb | on gitea02 | 00:48 |
| clarkb | I'll need to get my keys out to see if the server logs say anything | 00:49 |
| rpioso | It got by requirements \o/ | 00:49 |
| *** rfolco has quit IRC | 00:50 | |
| clarkb | ok is it just slow then? | 00:50 |
| rpioso | Now it's failing at a more common location in stack.sh -- "Instlling package prerequisites" | 00:50 |
| rpioso | s/Instlling/Installing/ | 00:50 |
| clarkb | I'm going to stop my loop now since it hasn't failed yet | 00:50 |
| rpioso | clarkb: Cool! Repeatedly invoking stack.sh got me past that failure. I hope it's idempotent. It's cranking along now. We'll see. | 00:51 |
| rpioso | Thank you! | 00:51 |
| clarkb | mordred: looking at cacti graphs it seems that gitea 1.10 is far more tcp open efficient | 00:52 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66631&rra_id=all looks like 1.9 regressed there? | 00:52 |
| clarkb | and now 1.10 is happier | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 01:15 |
| donnyd | corvus: uggg.. somebody should really start doing something about this fn cloud and its many failures | 01:15 |
| *** goldyfruit_ has joined #openstack-infra | 01:20 | |
| *** larainema has joined #openstack-infra | 01:27 | |
| *** rfolco has joined #openstack-infra | 01:29 | |
| *** ociuhandu has joined #openstack-infra | 01:39 | |
| *** ociuhandu has quit IRC | 01:43 | |
| *** nicolasbock has quit IRC | 01:55 | |
| *** Xuchu has joined #openstack-infra | 02:05 | |
| *** ricolin has joined #openstack-infra | 02:11 | |
| *** rfolco has quit IRC | 02:12 | |
| *** ykarel|away has joined #openstack-infra | 02:27 | |
| *** slaweq has joined #openstack-infra | 02:30 | |
| *** ykarel|away has quit IRC | 02:32 | |
| *** slaweq has quit IRC | 02:35 | |
| *** ociuhandu has joined #openstack-infra | 02:38 | |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| *** ociuhandu has quit IRC | 02:47 | |
| *** slaweq has joined #openstack-infra | 02:51 | |
| *** igordc has quit IRC | 02:54 | |
| *** slaweq has quit IRC | 02:56 | |
| *** gyee has quit IRC | 03:03 | |
| *** jamesmcarthur has quit IRC | 03:04 | |
| *** diablo_rojo has joined #openstack-infra | 03:06 | |
| *** apetrich has quit IRC | 03:09 | |
| *** tinwood has quit IRC | 03:10 | |
| *** tinwood has joined #openstack-infra | 03:12 | |
| *** jamesmcarthur has joined #openstack-infra | 03:19 | |
| *** armax has quit IRC | 03:19 | |
| *** rlandy|bbl is now known as rlandy | 03:19 | |
| *** rlandy has quit IRC | 03:23 | |
| *** jamesmcarthur has quit IRC | 03:35 | |
| *** tkajinam is now known as tkajinam|lunch | 03:40 | |
| *** gfidente|afk is now known as gfidente | 03:43 | |
| *** diablo_rojo has quit IRC | 03:46 | |
| *** udesale has joined #openstack-infra | 03:48 | |
| *** jamesmcarthur has joined #openstack-infra | 03:51 | |
| *** ykarel|away has joined #openstack-infra | 03:51 | |
| *** ykarel|away is now known as ykarel | 03:51 | |
| *** gfidente has quit IRC | 03:53 | |
| *** jamesmcarthur has quit IRC | 03:54 | |
| *** jamesmcarthur has joined #openstack-infra | 03:55 | |
| *** rosmaita has left #openstack-infra | 03:55 | |
| *** jamesmcarthur has quit IRC | 04:00 | |
| *** armax has joined #openstack-infra | 04:03 | |
| *** armax has quit IRC | 04:03 | |
| *** rh-jelabarre has quit IRC | 04:18 | |
| *** jamesmcarthur has joined #openstack-infra | 04:25 | |
| *** jamesmcarthur has quit IRC | 04:32 | |
| *** slaweq has joined #openstack-infra | 04:33 | |
| *** slaweq has quit IRC | 04:38 | |
| *** rishabhhpe has joined #openstack-infra | 05:07 | |
| *** slaweq has joined #openstack-infra | 05:25 | |
| *** slaweq has quit IRC | 05:30 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** pcaruana has joined #openstack-infra | 05:38 | |
| *** pcaruana has quit IRC | 05:42 | |
| *** raukadah is now known as chkumar|ruck | 05:45 | |
| *** slaweq has joined #openstack-infra | 05:48 | |
| *** eernst has quit IRC | 05:53 | |
| *** slaweq has quit IRC | 05:54 | |
| rishabhhpe | Hi Team , I am facing an issue while uploading a dib-image to the provider .. i had pasted the error and my nodepool.yaml in below link -: http://paste.openstack.org/show/787137/ ..... please do let me know if i am missing anything | 05:59 |
| *** armax has joined #openstack-infra | 06:01 | |
| ianw | rishabhhpe: it's saying "Exception: Unable to find flavor: general1-8" | 06:02 |
| ianw | so it seems you need to choose another flavor name | 06:02 |
| *** rcernin has quit IRC | 06:09 | |
| *** slaweq has joined #openstack-infra | 06:13 | |
| *** udesale has quit IRC | 06:13 | |
| *** jaicaa has quit IRC | 06:14 | |
| *** udesale has joined #openstack-infra | 06:14 | |
| *** armax has quit IRC | 06:15 | |
| *** ociuhandu has joined #openstack-infra | 06:16 | |
| *** jaicaa has joined #openstack-infra | 06:16 | |
| *** slaweq has quit IRC | 06:18 | |
| *** jtomasek has joined #openstack-infra | 06:19 | |
| *** surpatil has joined #openstack-infra | 06:23 | |
| *** Xuchu has quit IRC | 06:34 | |
| *** ociuhandu has quit IRC | 06:36 | |
| *** Xuchu has joined #openstack-infra | 06:38 | |
| *** ykarel_ has joined #openstack-infra | 06:42 | |
| *** ykarel has quit IRC | 06:45 | |
| *** dciabrin_ has joined #openstack-infra | 06:48 | |
| *** dciabrin has quit IRC | 06:49 | |
| *** ccamacho has quit IRC | 06:54 | |
| *** ralonsoh has joined #openstack-infra | 06:58 | |
| *** ykarel__ has joined #openstack-infra | 07:01 | |
| *** udesale has quit IRC | 07:02 | |
| *** ociuhandu has joined #openstack-infra | 07:03 | |
| *** rishabhhpe has quit IRC | 07:03 | |
| *** udesale has joined #openstack-infra | 07:03 | |
| *** ykarel_ has quit IRC | 07:03 | |
| *** udesale has quit IRC | 07:06 | |
| *** udesale has joined #openstack-infra | 07:06 | |
| *** ykarel__ is now known as ykarel | 07:08 | |
| *** pgaxatte has joined #openstack-infra | 07:17 | |
| *** pcaruana has joined #openstack-infra | 07:17 | |
| *** ramishra has quit IRC | 07:21 | |
| *** ociuhandu has quit IRC | 07:27 | |
| *** ociuhandu has joined #openstack-infra | 07:27 | |
| *** ramishra has joined #openstack-infra | 07:28 | |
| *** jamesmcarthur has joined #openstack-infra | 07:34 | |
| *** ykarel is now known as ykarel|lunch | 07:37 | |
| *** jamesmcarthur has quit IRC | 07:38 | |
| *** pkopec has joined #openstack-infra | 07:59 | |
| *** rcernin has joined #openstack-infra | 08:01 | |
| *** iurygregory has joined #openstack-infra | 08:01 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Don't process builds not longer in job graph https://review.opendev.org/697420 | 08:06 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Don't process builds not longer in job graph https://review.opendev.org/697420 | 08:09 |
| *** witek has joined #openstack-infra | 08:18 | |
| *** tesseract has joined #openstack-infra | 08:21 | |
| *** dchen has quit IRC | 08:23 | |
| *** ykarel|lunch is now known as ykarel | 08:27 | |
| *** udesale has quit IRC | 08:31 | |
| *** udesale has joined #openstack-infra | 08:32 | |
| *** rpittau|afk is now known as rpittau | 08:32 | |
| *** pabelanger has quit IRC | 08:34 | |
| *** mudpuppy has quit IRC | 08:34 | |
| *** hemna_ has quit IRC | 08:34 | |
| *** mattoliverau has quit IRC | 08:34 | |
| *** tosky has joined #openstack-infra | 08:36 | |
| *** gfidente has joined #openstack-infra | 08:37 | |
| *** pabelanger has joined #openstack-infra | 08:38 | |
| *** mudpuppy has joined #openstack-infra | 08:38 | |
| *** hemna_ has joined #openstack-infra | 08:38 | |
| *** mattoliverau has joined #openstack-infra | 08:38 | |
| *** gfidente has quit IRC | 08:41 | |
| *** SurajPatil has joined #openstack-infra | 08:41 | |
| *** surpatil has quit IRC | 08:44 | |
| *** iurygregory has quit IRC | 08:45 | |
| *** surpatil has joined #openstack-infra | 08:46 | |
| *** iurygregory has joined #openstack-infra | 08:47 | |
| *** SurajPatil has quit IRC | 08:49 | |
| *** slaweq_ has joined #openstack-infra | 08:50 | |
| *** jpena|off is now known as jpena | 08:50 | |
| *** rishabhhpe has joined #openstack-infra | 08:50 | |
| *** apetrich has joined #openstack-infra | 08:51 | |
| rishabhhpe | Hi , While launching an instance from nodepool i am getting the error pasted in below link -: http://paste.openstack.org/show/787142/ ...........can anyone please review if i am missing something ..i had attached my nodepool.yaml along with error. | 08:52 |
| *** udesale has quit IRC | 08:58 | |
| *** slaweq_ has quit IRC | 08:59 | |
| frickler | rishabhhpe: networks must be a list, not a hash, i.e. s/name: 'public'/public/ , see also the docs at https://zuul-ci.org/docs/nodepool/configuration.html | 09:00 |
| *** Xuchu has quit IRC | 09:04 | |
| *** lpetrut has joined #openstack-infra | 09:04 | |
| *** lucasagomes has joined #openstack-infra | 09:05 | |
| sshnaidm | can someone clear SSH sessions to gerrit for use os-tripleo-ci ? I have an error about max 64 connections | 09:07 |
| *** ociuhandu has quit IRC | 09:09 | |
| *** tkajinam|lunch has quit IRC | 09:09 | |
| *** udesale has joined #openstack-infra | 09:12 | |
| *** SurajPatil has joined #openstack-infra | 09:15 | |
| *** surpatil has quit IRC | 09:17 | |
| *** surpatil has joined #openstack-infra | 09:18 | |
| *** iurygregory is now known as iurygregory_cour | 09:20 | |
| *** iurygregory_cour is now known as iury_course | 09:20 | |
| *** SurajPatil has quit IRC | 09:20 | |
| openstackgerrit | Merged opendev/system-config master: bridge.o.o: update to latest Ansible https://review.opendev.org/695099 | 09:30 |
| *** ccamacho has joined #openstack-infra | 09:42 | |
| *** derekh has joined #openstack-infra | 09:43 | |
| *** pabelanger has quit IRC | 09:54 | |
| *** mudpuppy has quit IRC | 09:54 | |
| *** hemna_ has quit IRC | 09:54 | |
| *** mattoliverau has quit IRC | 09:54 | |
| *** ccamacho has quit IRC | 09:54 | |
| *** lpetrut has quit IRC | 09:54 | |
| *** ramishra has quit IRC | 09:54 | |
| *** ykarel has quit IRC | 09:54 | |
| *** jtomasek has quit IRC | 09:54 | |
| *** tinwood has quit IRC | 09:54 | |
| *** ricolin has quit IRC | 09:54 | |
| *** goldyfruit_ has quit IRC | 09:54 | |
| *** dpawlik has quit IRC | 09:54 | |
| *** hamzy has quit IRC | 09:54 | |
| *** liuyulong has quit IRC | 09:54 | |
| *** dmellado has quit IRC | 09:54 | |
| *** jaicaa has quit IRC | 09:54 | |
| *** Adri2000 has quit IRC | 09:54 | |
| *** Roamer` has quit IRC | 09:54 | |
| *** mugsie has quit IRC | 09:54 | |
| *** donnyd has quit IRC | 09:54 | |
| *** tobias-urdin has quit IRC | 09:54 | |
| *** cloudnull has quit IRC | 09:54 | |
| *** lathiat has quit IRC | 09:54 | |
| *** adam_g has quit IRC | 09:54 | |
| *** lseki has quit IRC | 09:54 | |
| *** beisner has quit IRC | 09:54 | |
| *** Buggys has quit IRC | 09:54 | |
| *** ddurst has quit IRC | 09:54 | |
| *** owalsh has quit IRC | 09:54 | |
| *** nickv1985 has quit IRC | 09:54 | |
| *** TheJulia has quit IRC | 09:54 | |
| *** cjohnston has quit IRC | 09:54 | |
| *** npochet has quit IRC | 09:54 | |
| *** mnasiadka has quit IRC | 09:54 | |
| *** rpittau has quit IRC | 09:54 | |
| *** HenryG has quit IRC | 09:54 | |
| *** gsantomaggio has quit IRC | 09:54 | |
| *** EmilienM has quit IRC | 09:54 | |
| *** frickler has quit IRC | 09:54 | |
| *** rledisez has quit IRC | 09:54 | |
| *** Diabelko has quit IRC | 09:54 | |
| *** seongsoocho has quit IRC | 09:54 | |
| *** masayukig has quit IRC | 09:54 | |
| *** abelur has quit IRC | 09:54 | |
| *** setuid has quit IRC | 09:54 | |
| *** srwilkers has quit IRC | 09:54 | |
| *** surpatil has quit IRC | 09:54 | |
| *** iury_course has quit IRC | 09:54 | |
| *** pgaxatte has quit IRC | 09:54 | |
| *** eharney has quit IRC | 09:54 | |
| *** fdegir has quit IRC | 09:54 | |
| *** jklare has quit IRC | 09:54 | |
| *** trident has quit IRC | 09:54 | |
| *** AJaeger has quit IRC | 09:54 | |
| *** dmsimard has quit IRC | 09:54 | |
| *** d34dh0r53 has quit IRC | 09:54 | |
| *** openstackgerrit has quit IRC | 09:54 | |
| *** SotK has quit IRC | 09:54 | |
| *** jpena has quit IRC | 09:54 | |
| *** antonym has quit IRC | 09:54 | |
| *** amorin has quit IRC | 09:54 | |
| *** brtknr has quit IRC | 09:54 | |
| *** evrardjp has quit IRC | 09:54 | |
| *** pkopec has quit IRC | 09:54 | |
| *** larainema has quit IRC | 09:54 | |
| *** lourot has quit IRC | 09:54 | |
| *** andreykurilin has quit IRC | 09:54 | |
| *** mgoddard has quit IRC | 09:54 | |
| *** fnordahl has quit IRC | 09:54 | |
| *** StevenK has quit IRC | 09:54 | |
| *** cmurphy has quit IRC | 09:54 | |
| *** logan- has quit IRC | 09:54 | |
| *** hwoarang has quit IRC | 09:54 | |
| *** dtantsur|afk has quit IRC | 09:54 | |
| *** kevinz has quit IRC | 09:54 | |
| *** markmcclain has quit IRC | 09:54 | |
| *** smcginnis has quit IRC | 09:54 | |
| *** zxiiro has quit IRC | 09:54 | |
| *** admcleod has quit IRC | 09:54 | |
| *** onovy has quit IRC | 09:54 | |
| *** haleyb has quit IRC | 09:54 | |
| *** calebb has quit IRC | 09:54 | |
| *** yankcrime has quit IRC | 09:54 | |
| *** radez has quit IRC | 09:54 | |
| *** nicholas has quit IRC | 09:54 | |
| *** arif-ali has quit IRC | 09:54 | |
| *** rakhmerov has quit IRC | 09:54 | |
| *** fresta has quit IRC | 09:54 | |
| *** rpioso has quit IRC | 09:54 | |
| *** kota_ has quit IRC | 09:54 | |
| *** davidlenwell has quit IRC | 09:54 | |
| *** cyberpear has quit IRC | 09:54 | |
| *** petevg has quit IRC | 09:54 | |
| *** SergeyLukjanov has quit IRC | 09:54 | |
| *** bradm has quit IRC | 09:54 | |
| *** amotoki has quit IRC | 09:54 | |
| *** zaro has quit IRC | 09:54 | |
| *** Anticimex has quit IRC | 09:54 | |
| *** coreycb has quit IRC | 09:54 | |
| *** clayg has quit IRC | 09:54 | |
| *** davecore has quit IRC | 09:54 | |
| *** knikolla has quit IRC | 09:54 | |
| *** ildikov has quit IRC | 09:54 | |
| *** ttx has quit IRC | 09:54 | |
| *** itxaka has quit IRC | 09:54 | |
| *** Zara has quit IRC | 09:54 | |
| *** spotz has quit IRC | 09:54 | |
| *** dabukalam has quit IRC | 09:54 | |
| *** lucasagomes has quit IRC | 09:54 | |
| *** tosky has quit IRC | 09:54 | |
| *** tesseract has quit IRC | 09:54 | |
| *** pcaruana has quit IRC | 09:54 | |
| *** ralonsoh has quit IRC | 09:54 | |
| *** lennyb has quit IRC | 09:54 | |
| *** yolanda has quit IRC | 09:54 | |
| *** jamesdenton has quit IRC | 09:54 | |
| *** jlvillal has quit IRC | 09:54 | |
| *** jonher has quit IRC | 09:54 | |
| *** factor has quit IRC | 09:54 | |
| *** Jeffrey4l has quit IRC | 09:54 | |
| *** elod has quit IRC | 09:54 | |
| *** tbarron has quit IRC | 09:54 | |
| *** iokiwi has quit IRC | 09:54 | |
| *** irclogbot_3 has quit IRC | 09:55 | |
| *** bnemec has quit IRC | 09:55 | |
| *** benj_ has quit IRC | 09:55 | |
| *** harlowja has quit IRC | 09:55 | |
| *** brwyatt has quit IRC | 09:55 | |
| *** corvus has quit IRC | 09:55 | |
| *** ianw has quit IRC | 09:55 | |
| *** efried has quit IRC | 09:55 | |
| *** icey has quit IRC | 09:55 | |
| *** otherwiseguy_ has quit IRC | 09:55 | |
| *** dosaboy has quit IRC | 09:55 | |
| *** stephenfin has quit IRC | 09:55 | |
| *** aspiers has quit IRC | 09:55 | |
| *** gagehugo has quit IRC | 09:55 | |
| *** zer0c00l has quit IRC | 09:55 | |
| *** szaher has quit IRC | 09:55 | |
| *** larsks has quit IRC | 09:55 | |
| *** gothicmindfood has quit IRC | 09:55 | |
| *** jmccrory has quit IRC | 09:55 | |
| *** strigazi has quit IRC | 09:55 | |
| *** rishabhhpe has quit IRC | 09:55 | |
| *** rcernin has quit IRC | 09:55 | |
| *** Goneri has quit IRC | 09:55 | |
| *** Tengu has quit IRC | 09:55 | |
| *** dirk has quit IRC | 09:55 | |
| *** persia has quit IRC | 09:55 | |
| *** jroll has quit IRC | 09:55 | |
| *** zzehring has quit IRC | 09:55 | |
| *** gouthamr has quit IRC | 09:55 | |
| *** bauzas has quit IRC | 09:55 | |
| *** sshnaidm has quit IRC | 09:55 | |
| *** pots has quit IRC | 09:55 | |
| *** darvon has quit IRC | 09:55 | |
| *** lxkong has quit IRC | 09:55 | |
| *** sparkycollier has quit IRC | 09:55 | |
| *** jamespage has quit IRC | 09:55 | |
| *** MarkMaglana has quit IRC | 09:55 | |
| *** jdelaros1 has quit IRC | 09:55 | |
| *** jrist has quit IRC | 09:55 | |
| *** fungi has quit IRC | 09:55 | |
| *** rajinir has quit IRC | 09:55 | |
| *** kambiz has quit IRC | 09:55 | |
| *** ullbeking has quit IRC | 09:55 | |
| *** beagles has quit IRC | 09:55 | |
| *** Ng has quit IRC | 09:55 | |
| *** gmann has quit IRC | 09:55 | |
| *** diablo_rojo_phon has quit IRC | 09:55 | |
| *** dustinc has quit IRC | 09:55 | |
| *** DinaBelova has quit IRC | 09:55 | |
| *** derekh has quit IRC | 09:55 | |
| *** udesale has quit IRC | 09:55 | |
| *** witek has quit IRC | 09:55 | |
| *** dciabrin_ has quit IRC | 09:55 | |
| *** osmanlicilegi has quit IRC | 09:55 | |
| *** yoctozepto has quit IRC | 09:55 | |
| *** paladox has quit IRC | 09:55 | |
| *** auristor has quit IRC | 09:55 | |
| *** calbers has quit IRC | 09:55 | |
| *** rkukura has quit IRC | 09:55 | |
| *** jistr has quit IRC | 09:55 | |
| *** bstinson has quit IRC | 09:55 | |
| *** panda has quit IRC | 09:55 | |
| *** adriant has quit IRC | 09:55 | |
| *** nhicher has quit IRC | 09:55 | |
| *** kopecmartin has quit IRC | 09:55 | |
| *** tonyb has quit IRC | 09:55 | |
| *** kaisers has quit IRC | 09:55 | |
| *** tobiash has quit IRC | 09:55 | |
| *** kukacz_ has quit IRC | 09:55 | |
| *** bdodd has quit IRC | 09:55 | |
| *** Qiming has quit IRC | 09:55 | |
| *** weshay_ has quit IRC | 09:55 | |
| *** aluria has quit IRC | 09:55 | |
| *** exsdev has quit IRC | 09:55 | |
| *** chkumar|ruck has quit IRC | 09:55 | |
| *** aarents has quit IRC | 09:55 | |
| *** cgoncalves has quit IRC | 09:55 | |
| *** dansmith has quit IRC | 09:55 | |
| *** tobberydberg has quit IRC | 09:55 | |
| *** mnencia has quit IRC | 09:55 | |
| *** pfallenop has quit IRC | 09:55 | |
| *** gary_perkins has quit IRC | 09:55 | |
| *** freerunner has quit IRC | 09:55 | |
| *** ericyoung has quit IRC | 09:55 | |
| *** timburke has quit IRC | 09:55 | |
| *** tristanC has quit IRC | 09:55 | |
| *** arxcruz has quit IRC | 09:55 | |
| *** jhesketh has quit IRC | 09:55 | |
| *** noonedeadpunk has quit IRC | 09:55 | |
| *** rodrigods has quit IRC | 09:55 | |
| *** tomaw has quit IRC | 09:55 | |
| *** lastmikoi has quit IRC | 09:55 | |
| *** andreaf has quit IRC | 09:55 | |
| *** csatari has quit IRC | 09:55 | |
| *** johnsom has quit IRC | 09:55 | |
| *** portdirect has quit IRC | 09:55 | |
| *** jungleboyj has quit IRC | 09:55 | |
| *** snierodz has quit IRC | 09:55 | |
| *** apetrich has quit IRC | 09:55 | |
| *** zbr_ has quit IRC | 09:55 | |
| *** dklyle has quit IRC | 09:55 | |
| *** vesper11 has quit IRC | 09:55 | |
| *** clarkb has quit IRC | 09:55 | |
| *** jrosser has quit IRC | 09:55 | |
| *** thedac has quit IRC | 09:55 | |
| *** kmarc has quit IRC | 09:55 | |
| *** dulek has quit IRC | 09:55 | |
| *** ssbarnea has quit IRC | 09:55 | |
| *** verdurin has quit IRC | 09:55 | |
| *** prometheanfire has quit IRC | 09:55 | |
| *** tdasilva has quit IRC | 09:55 | |
| *** mgagne has quit IRC | 09:55 | |
| *** dtroyer has quit IRC | 09:55 | |
| *** lifeless has quit IRC | 09:55 | |
| *** weshay_pto has quit IRC | 09:55 | |
| *** zzzeek has quit IRC | 09:55 | |
| *** lsell has quit IRC | 09:55 | |
| *** guilhermesp has quit IRC | 09:55 | |
| *** mwhahaha has quit IRC | 09:55 | |
| *** dougwig has quit IRC | 09:55 | |
| *** crodriguez has quit IRC | 09:55 | |
| *** tonyb[m] has quit IRC | 09:55 | |
| *** odyssey4me has quit IRC | 09:55 | |
| *** mordred has quit IRC | 09:55 | |
| *** sdoran has quit IRC | 09:55 | |
| *** arne_wiebalck has quit IRC | 09:55 | |
| *** evgenyl has quit IRC | 09:55 | |
| *** philroche has quit IRC | 09:55 | |
| *** vdrok has quit IRC | 09:55 | |
| *** mnaser has quit IRC | 09:55 | |
| *** cjloader has quit IRC | 09:55 | |
| *** rm_work has quit IRC | 09:55 | |
| *** dancek has quit IRC | 09:55 | |
| *** melwitt has quit IRC | 09:55 | |
| *** Shrews has quit IRC | 09:55 | |
| *** ChanServ has quit IRC | 09:55 | |
| *** mattoliverau has joined #openstack-infra | 10:04 | |
| *** hemna_ has joined #openstack-infra | 10:04 | |
| *** mudpuppy has joined #openstack-infra | 10:04 | |
| *** pabelanger has joined #openstack-infra | 10:04 | |
| *** ykarel has joined #openstack-infra | 10:04 | |
| *** electrofelix has joined #openstack-infra | 10:04 | |
| *** jklare has joined #openstack-infra | 10:04 | |
| *** dmellado has joined #openstack-infra | 10:04 | |
| *** liuyulong has joined #openstack-infra | 10:04 | |
| *** hamzy has joined #openstack-infra | 10:04 | |
| *** dpawlik has joined #openstack-infra | 10:04 | |
| *** goldyfruit_ has joined #openstack-infra | 10:04 | |
| *** ricolin has joined #openstack-infra | 10:04 | |
| *** tinwood has joined #openstack-infra | 10:04 | |
| *** jtomasek has joined #openstack-infra | 10:04 | |
| *** ramishra has joined #openstack-infra | 10:04 | |
| *** lpetrut has joined #openstack-infra | 10:04 | |
| *** ccamacho has joined #openstack-infra | 10:04 | |
| *** srwilkers has joined #openstack-infra | 10:04 | |
| *** setuid has joined #openstack-infra | 10:04 | |
| *** abelur has joined #openstack-infra | 10:04 | |
| *** masayukig has joined #openstack-infra | 10:04 | |
| *** Diabelko has joined #openstack-infra | 10:04 | |
| *** rledisez has joined #openstack-infra | 10:04 | |
| *** frickler has joined #openstack-infra | 10:04 | |
| *** seongsoocho has joined #openstack-infra | 10:04 | |
| *** EmilienM has joined #openstack-infra | 10:04 | |
| *** gsantomaggio has joined #openstack-infra | 10:04 | |
| *** HenryG has joined #openstack-infra | 10:04 | |
| *** rpittau has joined #openstack-infra | 10:04 | |
| *** mnasiadka has joined #openstack-infra | 10:04 | |
| *** npochet has joined #openstack-infra | 10:04 | |
| *** cjohnston has joined #openstack-infra | 10:04 | |
| *** nickv1985 has joined #openstack-infra | 10:04 | |
| *** TheJulia has joined #openstack-infra | 10:04 | |
| *** owalsh has joined #openstack-infra | 10:04 | |
| *** ddurst has joined #openstack-infra | 10:04 | |
| *** Buggys has joined #openstack-infra | 10:04 | |
| *** beisner has joined #openstack-infra | 10:04 | |
| *** lseki has joined #openstack-infra | 10:04 | |
| *** adam_g has joined #openstack-infra | 10:04 | |
| *** lathiat has joined #openstack-infra | 10:04 | |
| *** cloudnull has joined #openstack-infra | 10:04 | |
| *** tobias-urdin has joined #openstack-infra | 10:04 | |
| *** donnyd has joined #openstack-infra | 10:04 | |
| *** mugsie has joined #openstack-infra | 10:04 | |
| *** Adri2000 has joined #openstack-infra | 10:04 | |
| *** Roamer` has joined #openstack-infra | 10:04 | |
| *** jaicaa has joined #openstack-infra | 10:04 | |
| *** evrardjp has joined #openstack-infra | 10:04 | |
| *** brtknr has joined #openstack-infra | 10:04 | |
| *** amorin has joined #openstack-infra | 10:04 | |
| *** antonym has joined #openstack-infra | 10:04 | |
| *** jpena has joined #openstack-infra | 10:04 | |
| *** SotK has joined #openstack-infra | 10:04 | |
| *** openstackgerrit has joined #openstack-infra | 10:04 | |
| *** d34dh0r53 has joined #openstack-infra | 10:04 | |
| *** dmsimard has joined #openstack-infra | 10:04 | |
| *** AJaeger has joined #openstack-infra | 10:04 | |
| *** trident has joined #openstack-infra | 10:04 | |
| *** fdegir has joined #openstack-infra | 10:04 | |
| *** eharney has joined #openstack-infra | 10:04 | |
| *** pgaxatte has joined #openstack-infra | 10:04 | |
| *** iury_course has joined #openstack-infra | 10:04 | |
| *** surpatil has joined #openstack-infra | 10:04 | |
| *** witek has joined #openstack-infra | 10:04 | |
| *** derekh has joined #openstack-infra | 10:04 | |
| *** lucasagomes has joined #openstack-infra | 10:04 | |
| *** apetrich has joined #openstack-infra | 10:04 | |
| *** rishabhhpe has joined #openstack-infra | 10:04 | |
| *** tosky has joined #openstack-infra | 10:04 | |
| *** tesseract has joined #openstack-infra | 10:04 | |
| *** rcernin has joined #openstack-infra | 10:04 | |
| *** pcaruana has joined #openstack-infra | 10:04 | |
| *** ralonsoh has joined #openstack-infra | 10:04 | |
| *** dciabrin_ has joined #openstack-infra | 10:04 | |
| *** osmanlicilegi has joined #openstack-infra | 10:04 | |
| *** lennyb has joined #openstack-infra | 10:04 | |
| *** Goneri has joined #openstack-infra | 10:04 | |
| *** spotz has joined #openstack-infra | 10:04 | |
| *** dabukalam has joined #openstack-infra | 10:04 | |
| *** Zara has joined #openstack-infra | 10:04 | |
| *** itxaka has joined #openstack-infra | 10:04 | |
| *** zbr_ has joined #openstack-infra | 10:04 | |
| *** yolanda has joined #openstack-infra | 10:04 | |
| *** yoctozepto has joined #openstack-infra | 10:04 | |
| *** Tengu has joined #openstack-infra | 10:04 | |
| *** tomaw has joined #openstack-infra | 10:04 | |
| *** dklyle has joined #openstack-infra | 10:04 | |
| *** paladox has joined #openstack-infra | 10:04 | |
| *** auristor has joined #openstack-infra | 10:04 | |
| *** calbers has joined #openstack-infra | 10:04 | |
| *** rkukura has joined #openstack-infra | 10:04 | |
| *** vesper11 has joined #openstack-infra | 10:04 | |
| *** jamesdenton has joined #openstack-infra | 10:04 | |
| *** jhesketh has joined #openstack-infra | 10:04 | |
| *** jlvillal has joined #openstack-infra | 10:04 | |
| *** jonher has joined #openstack-infra | 10:04 | |
| *** dirk has joined #openstack-infra | 10:04 | |
| *** jistr has joined #openstack-infra | 10:04 | |
| *** factor has joined #openstack-infra | 10:04 | |
| *** clarkb has joined #openstack-infra | 10:04 | |
| *** Jeffrey4l has joined #openstack-infra | 10:04 | |
| *** persia has joined #openstack-infra | 10:04 | |
| *** jrosser has joined #openstack-infra | 10:04 | |
| *** thedac has joined #openstack-infra | 10:04 | |
| *** jroll has joined #openstack-infra | 10:04 | |
| *** bstinson has joined #openstack-infra | 10:04 | |
| *** kmarc has joined #openstack-infra | 10:04 | |
| *** dulek has joined #openstack-infra | 10:04 | |
| *** tbarron has joined #openstack-infra | 10:04 | |
| *** elod has joined #openstack-infra | 10:04 | |
| *** lastmikoi has joined #openstack-infra | 10:04 | |
| *** iokiwi has joined #openstack-infra | 10:04 | |
| *** panda has joined #openstack-infra | 10:04 | |
| *** zzehring has joined #openstack-infra | 10:04 | |
| *** ssbarnea has joined #openstack-infra | 10:04 | |
| *** gouthamr has joined #openstack-infra | 10:04 | |
| *** bauzas has joined #openstack-infra | 10:04 | |
| *** arne_wiebalck has joined #openstack-infra | 10:04 | |
| *** kopecmartin has joined #openstack-infra | 10:04 | |
| *** verdurin has joined #openstack-infra | 10:04 | |
| *** adriant has joined #openstack-infra | 10:04 | |
| *** nhicher has joined #openstack-infra | 10:04 | |
| *** sshnaidm has joined #openstack-infra | 10:04 | |
| *** tonyb has joined #openstack-infra | 10:04 | |
| *** kaisers has joined #openstack-infra | 10:04 | |
| *** irclogbot_3 has joined #openstack-infra | 10:04 | |
| *** tobiash has joined #openstack-infra | 10:04 | |
| *** bnemec has joined #openstack-infra | 10:04 | |
| *** benj_ has joined #openstack-infra | 10:04 | |
| *** harlowja has joined #openstack-infra | 10:04 | |
| *** pots has joined #openstack-infra | 10:04 | |
| *** brwyatt has joined #openstack-infra | 10:04 | |
| *** corvus has joined #openstack-infra | 10:04 | |
| *** prometheanfire has joined #openstack-infra | 10:04 | |
| *** tdasilva has joined #openstack-infra | 10:04 | |
| *** mgagne has joined #openstack-infra | 10:04 | |
| *** dtroyer has joined #openstack-infra | 10:04 | |
| *** kukacz_ has joined #openstack-infra | 10:04 | |
| *** ianw has joined #openstack-infra | 10:04 | |
| *** aarents has joined #openstack-infra | 10:04 | |
| *** efried has joined #openstack-infra | 10:04 | |
| *** bdodd has joined #openstack-infra | 10:04 | |
| *** tonyb[m] has joined #openstack-infra | 10:04 | |
| *** lifeless has joined #openstack-infra | 10:04 | |
| *** Qiming has joined #openstack-infra | 10:04 | |
| *** icey has joined #openstack-infra | 10:04 | |
| *** weshay_pto has joined #openstack-infra | 10:04 | |
| *** weshay_ has joined #openstack-infra | 10:04 | |
| *** darvon has joined #openstack-infra | 10:04 | |
| *** lxkong has joined #openstack-infra | 10:04 | |
| *** otherwiseguy_ has joined #openstack-infra | 10:04 | |
| *** zzzeek has joined #openstack-infra | 10:04 | |
| *** lsell has joined #openstack-infra | 10:04 | |
| *** sparkycollier has joined #openstack-infra | 10:04 | |
| *** guilhermesp has joined #openstack-infra | 10:04 | |
| *** dougwig has joined #openstack-infra | 10:04 | |
| *** mwhahaha has joined #openstack-infra | 10:04 | |
| *** jamespage has joined #openstack-infra | 10:04 | |
| *** MarkMaglana has joined #openstack-infra | 10:04 | |
| *** crodriguez has joined #openstack-infra | 10:04 | |
| *** jdelaros1 has joined #openstack-infra | 10:04 | |
| *** odyssey4me has joined #openstack-infra | 10:04 | |
| *** aluria has joined #openstack-infra | 10:04 | |
| *** dosaboy has joined #openstack-infra | 10:04 | |
| *** exsdev has joined #openstack-infra | 10:04 | |
| *** chkumar|ruck has joined #openstack-infra | 10:04 | |
| *** jrist has joined #openstack-infra | 10:04 | |
| *** fungi has joined #openstack-infra | 10:04 | |
| *** rajinir has joined #openstack-infra | 10:04 | |
| *** mordred has joined #openstack-infra | 10:04 | |
| *** stephenfin has joined #openstack-infra | 10:04 | |
| *** jungleboyj has joined #openstack-infra | 10:04 | |
| *** aspiers has joined #openstack-infra | 10:04 | |
| *** gagehugo has joined #openstack-infra | 10:04 | |
| *** jmccrory has joined #openstack-infra | 10:04 | |
| *** cgoncalves has joined #openstack-infra | 10:04 | |
| *** zer0c00l has joined #openstack-infra | 10:04 | |
| *** szaher has joined #openstack-infra | 10:04 | |
| *** kambiz has joined #openstack-infra | 10:04 | |
| *** andreaf has joined #openstack-infra | 10:04 | |
| *** dansmith has joined #openstack-infra | 10:04 | |
| *** larsks has joined #openstack-infra | 10:04 | |
| *** tobberydberg has joined #openstack-infra | 10:04 | |
| *** gothicmindfood has joined #openstack-infra | 10:04 | |
| *** ullbeking has joined #openstack-infra | 10:04 | |
| *** mnencia has joined #openstack-infra | 10:04 | |
| *** strigazi has joined #openstack-infra | 10:04 | |
| *** pfallenop has joined #openstack-infra | 10:04 | |
| *** gary_perkins has joined #openstack-infra | 10:04 | |
| *** diablo_rojo_phon has joined #openstack-infra | 10:04 | |
| *** freerunner has joined #openstack-infra | 10:04 | |
| *** ericyoung has joined #openstack-infra | 10:04 | |
| *** timburke has joined #openstack-infra | 10:04 | |
| *** tristanC has joined #openstack-infra | 10:04 | |
| *** arxcruz has joined #openstack-infra | 10:04 | |
| *** beagles has joined #openstack-infra | 10:04 | |
| *** gmann has joined #openstack-infra | 10:04 | |
| *** Ng has joined #openstack-infra | 10:04 | |
| *** dustinc has joined #openstack-infra | 10:04 | |
| *** DinaBelova has joined #openstack-infra | 10:04 | |
| *** csatari has joined #openstack-infra | 10:04 | |
| *** johnsom has joined #openstack-infra | 10:04 | |
| *** sdoran has joined #openstack-infra | 10:04 | |
| *** portdirect has joined #openstack-infra | 10:04 | |
| *** noonedeadpunk has joined #openstack-infra | 10:04 | |
| *** snierodz has joined #openstack-infra | 10:04 | |
| *** rodrigods has joined #openstack-infra | 10:04 | |
| *** Shrews has joined #openstack-infra | 10:04 | |
| *** melwitt has joined #openstack-infra | 10:04 | |
| *** dancek has joined #openstack-infra | 10:04 | |
| *** rm_work has joined #openstack-infra | 10:04 | |
| *** mnaser has joined #openstack-infra | 10:04 | |
| *** vdrok has joined #openstack-infra | 10:04 | |
| *** cjloader has joined #openstack-infra | 10:04 | |
| *** philroche has joined #openstack-infra | 10:04 | |
| *** evgenyl has joined #openstack-infra | 10:04 | |
| *** ChanServ has joined #openstack-infra | 10:04 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 10:04 | |
| *** ttx has joined #openstack-infra | 10:04 | |
| *** pkopec has joined #openstack-infra | 10:04 | |
| *** larainema has joined #openstack-infra | 10:04 | |
| *** lourot has joined #openstack-infra | 10:04 | |
| *** andreykurilin has joined #openstack-infra | 10:04 | |
| *** mgoddard has joined #openstack-infra | 10:04 | |
| *** fnordahl has joined #openstack-infra | 10:04 | |
| *** cmurphy has joined #openstack-infra | 10:04 | |
| *** StevenK has joined #openstack-infra | 10:04 | |
| *** kevinz has joined #openstack-infra | 10:04 | |
| *** logan- has joined #openstack-infra | 10:04 | |
| *** hwoarang has joined #openstack-infra | 10:04 | |
| *** bradm has joined #openstack-infra | 10:04 | |
| *** dtantsur|afk has joined #openstack-infra | 10:04 | |
| *** markmcclain has joined #openstack-infra | 10:04 | |
| *** smcginnis has joined #openstack-infra | 10:04 | |
| *** zxiiro has joined #openstack-infra | 10:04 | |
| *** admcleod has joined #openstack-infra | 10:04 | |
| *** onovy has joined #openstack-infra | 10:04 | |
| *** haleyb has joined #openstack-infra | 10:04 | |
| *** calebb has joined #openstack-infra | 10:04 | |
| *** yankcrime has joined #openstack-infra | 10:04 | |
| *** radez has joined #openstack-infra | 10:04 | |
| *** arif-ali has joined #openstack-infra | 10:04 | |
| *** nicholas has joined #openstack-infra | 10:04 | |
| *** rakhmerov has joined #openstack-infra | 10:04 | |
| *** fresta has joined #openstack-infra | 10:04 | |
| *** rpioso has joined #openstack-infra | 10:04 | |
| *** kota_ has joined #openstack-infra | 10:04 | |
| *** davidlenwell has joined #openstack-infra | 10:04 | |
| *** cyberpear has joined #openstack-infra | 10:04 | |
| *** petevg has joined #openstack-infra | 10:04 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:04 | |
| *** amotoki has joined #openstack-infra | 10:04 | |
| *** zaro has joined #openstack-infra | 10:04 | |
| *** Anticimex has joined #openstack-infra | 10:04 | |
| *** ildikov has joined #openstack-infra | 10:04 | |
| *** coreycb has joined #openstack-infra | 10:04 | |
| *** clayg has joined #openstack-infra | 10:04 | |
| *** davecore has joined #openstack-infra | 10:04 | |
| *** knikolla has joined #openstack-infra | 10:04 | |
| *** gfidente has joined #openstack-infra | 10:05 | |
| *** logan- has quit IRC | 10:05 | |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| *** logan- has joined #openstack-infra | 10:06 | |
| *** hashar has joined #openstack-infra | 10:14 | |
| *** osmanlicilegi has quit IRC | 10:18 | |
| *** osmanlicilegi has joined #openstack-infra | 10:19 | |
| *** osmanlicilegi has quit IRC | 10:22 | |
| *** osmanlicilegi has joined #openstack-infra | 10:27 | |
| *** ykarel is now known as ykarel|afk | 10:27 | |
| *** dchen has joined #openstack-infra | 10:32 | |
| *** ociuhandu has joined #openstack-infra | 10:39 | |
| *** pcaruana has quit IRC | 10:42 | |
| *** hashar has quit IRC | 10:54 | |
| *** ykarel|afk is now known as ykarel | 11:07 | |
| *** rfolco has joined #openstack-infra | 11:17 | |
| *** rfolco has quit IRC | 11:17 | |
| *** rfolco has joined #openstack-infra | 11:18 | |
| *** rishabhhpe has quit IRC | 11:18 | |
| *** witek has quit IRC | 11:29 | |
| *** ociuhandu has quit IRC | 11:31 | |
| *** derekh has quit IRC | 11:33 | |
| *** SurajPatil has joined #openstack-infra | 11:33 | |
| *** surpatil has quit IRC | 11:36 | |
| *** SurajPatil has quit IRC | 11:38 | |
| openstackgerrit | Mattias Jernberg proposed opendev/git-review master: Allow the default of notopic to be configurable https://review.opendev.org/697448 | 11:40 |
| *** ociuhandu has joined #openstack-infra | 11:44 | |
| *** witek has joined #openstack-infra | 11:46 | |
| *** udesale has joined #openstack-infra | 11:51 | |
| *** iury_course has quit IRC | 11:55 | |
| *** iurygregory has joined #openstack-infra | 11:56 | |
| *** iurygregory is now known as iury_course | 11:56 | |
| *** Lucas_Gray has joined #openstack-infra | 12:07 | |
| *** zbr_ is now known as zbr|out | 12:13 | |
| *** Lucas_Gray has quit IRC | 12:20 | |
| *** jpena is now known as jpena|lunch | 12:22 | |
| *** Lucas_Gray has joined #openstack-infra | 12:22 | |
| *** larainema has quit IRC | 12:26 | |
| noonedeadpunk | hi folks. We have a pretty usefull job, which sometimes lasts more than 3 hours which results in timeout limit. For some roles it fails almost always. | 12:32 |
| noonedeadpunk | So it's upgrade job which verifies that OSA upgrade is good. It's pretty inportant to be sure that patches are not breaking upgrade path. But they deploy osa twice, which is not so fast... | 12:44 |
| noonedeadpunk | I was wondering if we can do smth regarding that - like changing flavor with encreased ram or allow this exact job running more that 3h.... | 12:44 |
| noonedeadpunk | Or maybe you have some suggestions regarding this case... | 12:45 |
| *** rcernin has quit IRC | 12:52 | |
| *** jamesmcarthur has joined #openstack-infra | 12:53 | |
| *** pcaruana has joined #openstack-infra | 12:54 | |
| *** jklare has quit IRC | 12:57 | |
| *** jklare has joined #openstack-infra | 13:00 | |
| *** rlandy has joined #openstack-infra | 13:01 | |
| Shrews | noonedeadpunk: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.timeout | 13:02 |
| *** ociuhandu has quit IRC | 13:06 | |
| *** eharney has quit IRC | 13:06 | |
| *** jamesmcarthur has quit IRC | 13:07 | |
| *** rh-jelabarre has joined #openstack-infra | 13:07 | |
| noonedeadpunk | Shrews: there's max limit now defined in 3hours https://review.opendev.org/#/c/697435/ | 13:07 |
| noonedeadpunk | oh, so you mean we have this limit defined... | 13:09 |
| sshnaidm | sorry for bothering again, but can someone clear SSH sessions to Gerrit for use os-tripleo-ci ? I have an error about max 64 connections | 13:10 |
| donnyd | I have brought this up before, but I'm not entirely sure how to fix it. I think last time we had to restart my nodepool worker. The instances in "Ready" are eating a significant amount of quota from what nodepool will build on FN. | 13:10 |
| donnyd | Its at 9 instances, but quota is much higher. So right now it will only build 25-27 instances | 13:11 |
| donnyd | but it should be more like 35 | 13:12 |
| *** jamesmcarthur has joined #openstack-infra | 13:12 | |
| donnyd | In the big picture, 10 test nodes isn't a lot.. but for little FN it's like 30% of what it can do :( | 13:13 |
| *** jpena|lunch is now known as jpena | 13:14 | |
| Shrews | noonedeadpunk: ah, ok, that's the default max timeout. it could be changed, but would have to be done for the entire openstack tenant. | 13:20 |
| *** goldyfruit_ has quit IRC | 13:20 | |
| noonedeadpunk | Shrews: Yeah, that's not so good and reasonable... What about changing flavor to smth with more that 8Gb of ram? | 13:21 |
| noonedeadpunk | as an alternative we're considering just dropping such job for centos7 until we add support of centos8 which hopefully will be faster... | 13:22 |
| Shrews | if the job is resource constrained, then a different flavor might be faster. but i have no idea if that is the case | 13:23 |
| *** Goneri has quit IRC | 13:24 | |
| *** jamesmcarthur has quit IRC | 13:25 | |
| *** jamesmcarthur has joined #openstack-infra | 13:26 | |
| *** ociuhandu has joined #openstack-infra | 13:27 | |
| *** udesale has quit IRC | 13:30 | |
| noonedeadpunk | Shrews: actually I even find ooms from time to time during tempest runs, and when reproducing test from vm with the same flavor it really lucks resources... | 13:30 |
| *** jamesmcarthur has quit IRC | 13:31 | |
| *** witek has quit IRC | 13:34 | |
| *** udesale has joined #openstack-infra | 13:39 | |
| donnyd | noonedeadpunk: FN already has expanded resources. We have instances for centos and ubuntu that have 16Gb of memory | 13:40 |
| Shrews | donnyd: not entirely sure i understand your question. there are different quotas at play in nodepool: the max-servers setting and the quotas reported back from the provider (cores, ram, etc) | 13:41 |
| donnyd | just use the following labels - centos-7-expanded or ubuntu-bionic-expanded | 13:41 |
| noonedeadpunk | donnyd: oh, that's great,thanks! | 13:42 |
| noonedeadpunk | will try this out | 13:42 |
| donnyd | I think FN is the only one with those labels, so it will execute there.. but at least it can help you sort out if more memory fixes the issue | 13:42 |
| AJaeger | noonedeadpunk: keep in mind that this will tie you to a single cloud - if that one is down, you have bad luck | 13:43 |
| donnyd | I wouldn't use it as a long term solution, maybe just something to deterministically figure out if more memory will in fact fix the issue you are having | 13:44 |
| donnyd | But the purpose of the label is for exactly issues like this | 13:44 |
| donnyd | nova used them last cycle for a couple things if I am not mistaken | 13:44 |
| noonedeadpunk | can we add flavor on another one cloud (like vexxhost?:)) if this will work out? | 13:45 |
| AJaeger | noonedeadpunk: that's better a question to discuss later with the rest of the team. I expect the answer to be: No - with reference to https://docs.openstack.org/infra/manual/testing.html | 13:46 |
| donnyd | we can ask mnaser, but I think for the most part infra didn't want these labels to be used permanently. I believe the feeling is the job should fit in the standard 8gb flavor. don't want to speak on behalf on infra though | 13:46 |
| donnyd | Its only there to aid in getting a critical job through or to be used for testing | 13:47 |
| *** Lucas_Gray has quit IRC | 13:47 | |
| donnyd | noonedeadpunk: I am thing AJaeger is right, and it surely has been asked before. With that said, use the label and see if it fixes the issue. | 13:48 |
| mordred | infra-root: I've contracted plague so I may not be my normal energetic self today | 13:48 |
| noonedeadpunk | Anyway I hope that centos8 is going to be faster - just centos 7 jobs are 1.5 times slower than ubuntu ones while doing exactly the same thing... | 13:48 |
| Shrews | mordred: ugh. have you tried not being sick? | 13:49 |
| noonedeadpunk | thanks for the answers anyway | 13:49 |
| *** ykarel is now known as ykarel|afk | 13:50 | |
| noonedeadpunk | will check if extended resources help to think futher about this | 13:50 |
| mordred | Shrews: yes, in fact, I did try that. I clearly failed | 13:51 |
| donnyd | mordred: unacceptable | 13:51 |
| mordred | Shrews: I guess that makes me both sick and a failure | 13:51 |
| mordred | donnyd: I know it | 13:51 |
| donnyd | mordred: I am thinking you are correct | 13:51 |
| donnyd | I get at least one solid laugh out of this community every day, mordred you have already made mine | 13:52 |
| mordred | donnyd: \o/ *cough* | 13:52 |
| *** mriedem has joined #openstack-infra | 13:54 | |
| mordred | clarkb: yay on tcp open! we should send that to lunny just to say "nice work" | 13:56 |
| *** derekh has joined #openstack-infra | 13:56 | |
| *** jamesmcarthur has joined #openstack-infra | 13:57 | |
| mordred | (I pinged him with the link in their discord) | 13:59 |
| *** diablo_rojo has joined #openstack-infra | 14:00 | |
| *** armax has joined #openstack-infra | 14:00 | |
| *** witek has joined #openstack-infra | 14:04 | |
| frickler | infra-root: we are seeing some RST rendering issue on https://opendev.org/openstack/cookbook-openstack-common , see the table at the end (we moved from .md to .rst after noticing other issues with the rendering there) | 14:04 |
| *** jamesmcarthur has quit IRC | 14:04 | |
| frickler | the table shows up fine on github and also when I look at it locally with https://pypi.org/project/restview/ | 14:05 |
| frickler | do we know what gitea uses for rendering? | 14:05 |
| mordred | frickler: /usr/bin/pandoc -f rst | 14:06 |
| mordred | frickler: check playbooks/roles/gitea/templates/app.ini.j2 in system-config for the markup.pandoc section | 14:07 |
| *** jtomasek has quit IRC | 14:08 | |
| *** eharney has joined #openstack-infra | 14:09 | |
| noonedeadpunk | donnyd: unfortunatelly I got "The nodeset "centos-7-expanded" was not found."... | 14:11 |
| *** jtomasek has joined #openstack-infra | 14:11 | |
| *** gfidente has quit IRC | 14:12 | |
| *** gfidente has joined #openstack-infra | 14:12 | |
| *** cgoncalves has quit IRC | 14:13 | |
| noonedeadpunk | oh, I did it wrong | 14:14 |
| frickler | mordred: great, thx, with that I can reproduce the issue. is there a particular reason to use pandoc instead of other renderers? gitea proposes rst2html.py from docutils which also produces a correct rendering for me https://docs.gitea.io/en-us/external-renderers/ | 14:15 |
| *** ociuhandu has quit IRC | 14:16 | |
| AJaeger | frickler: add an empty line - want me to send a patch? | 14:18 |
| *** Goneri has joined #openstack-infra | 14:18 | |
| frickler | AJaeger: oh, that works, thanks. if you want to send a patch, you are welcome, we haven't had a contribution from suse in a while ;) | 14:21 |
| AJaeger | frickler: HUGE and significatn patch sent ;) | 14:23 |
| AJaeger | https://review.opendev.org/697479 | 14:24 |
| frickler | AJaeger: well, this was the last blocker for me to start branching stable/rocky, so in fact not as insignificant as it may look, thank you | 14:25 |
| AJaeger | ;) | 14:27 |
| AJaeger | glad to be of help | 14:27 |
| *** rlandy is now known as rlandy|mtg | 14:28 | |
| ricolin | Hi guys, besides the new comming Linaro servers, so we also got other ARM server in our nodepool already? | 14:29 |
| *** rosmaita has joined #openstack-infra | 14:33 | |
| *** zbr|out has quit IRC | 14:33 | |
| *** zbr has joined #openstack-infra | 14:34 | |
| dtantsur | folks, could we do something about "This change depends on a change that failed to merge." when the parent patch fails in the gate? | 14:35 |
| dtantsur | like, instead of -2 and recheck, hold it with no verified and re-enter the gate when the parent re-enters it? | 14:35 |
| *** jamesmcarthur has joined #openstack-infra | 14:36 | |
| *** pcaruana has quit IRC | 14:38 | |
| *** ociuhandu has joined #openstack-infra | 14:40 | |
| *** jamesmcarthur has quit IRC | 14:40 | |
| *** zbr_ has joined #openstack-infra | 14:41 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for NetworkAmbiguous fail bug 1844568 https://review.opendev.org/697486 | 14:41 |
| openstack | bug 1844568 in tempest "[compute] "create_test_server" if networks is undefined and more than one network is present" [Medium,In progress] https://launchpad.net/bugs/1844568 - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | 14:41 |
| *** zbr has quit IRC | 14:44 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store event id for buildsets in database https://review.opendev.org/697488 | 14:47 |
| *** goldyfruit_ has joined #openstack-infra | 14:48 | |
| *** zbr_ has quit IRC | 14:49 | |
| openstackgerrit | Amy Marrich (spotz) proposed opendev/irc-meetings master: New UC meeting time https://review.opendev.org/697489 | 14:53 |
| *** zbr has joined #openstack-infra | 14:55 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store event id for buildsets in database https://review.opendev.org/697488 | 14:55 |
| *** ociuhandu has quit IRC | 14:57 | |
| *** cgoncalves has joined #openstack-infra | 14:57 | |
| *** ociuhandu has joined #openstack-infra | 14:57 | |
| *** goldyfruit___ has joined #openstack-infra | 15:00 | |
| *** goldyfruit_ has quit IRC | 15:02 | |
| *** rpittau is now known as rpittau|afk | 15:03 | |
| *** rlandy|mtg is now known as rlandy | 15:09 | |
| *** dpawlik has quit IRC | 15:16 | |
| TheJulia | Any chance someone can take a look at https://review.opendev.org/#/c/697159/? | 15:16 |
| TheJulia | I have some folks that want to go ahead and start packaging that downstream | 15:17 |
| *** ociuhandu has quit IRC | 15:21 | |
| *** ociuhandu has joined #openstack-infra | 15:22 | |
| mordred | TheJulia: comment from frickler on it - that group name looks weird | 15:24 |
| clarkb | sshnaidm: can you figure put why you are leaking connections? and they should tineout after 1 hour | 15:27 |
| fungi | ricolin: yes, we have quota for 8 servers in linaro-london currently, booting arm64 images for two versions of ubuntu and two versions of debian: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L394-L432 | 15:27 |
| clarkb | sshnaidm: is this account connecting with zuul? if so is that zuul install up to date or is it older? there was a paramiko issue zuul had to address at one point | 15:27 |
| ricolin | fungi, are we assigned any task to those server at this time? | 15:28 |
| TheJulia | mordred: ack, thanks | 15:28 |
| clarkb | ricolin: this is why I suggested chatting with linaro. They already supply arm64 nodes to us | 15:28 |
| ricolin | clarkb, okay I thought you're taking about the same Linaro team as I was | 15:30 |
| ricolin | do we have any contact window from Linaro London | 15:31 |
| clarkb | ricolin: hrw iirc | 15:32 |
| clarkb | dtantsur: I dont think zuul's current config options can express that | 15:34 |
| ricolin | clarkb, Okay I will to chat with Marcin | 15:34 |
| dtantsur | clarkb: that's quite sad. is it something that can be considered for a roadmap? | 15:34 |
| clarkb | dtantsur: I expect that the zuul team may suggest openstack stop requiring "clean check" which I think will avoid the problems you have with the current setup | 15:35 |
| clarkb | the issue being you dont want to wait for both changes to get verified +1 again | 15:36 |
| dtantsur | I'm fine with the one that failed to go through check again | 15:36 |
| dtantsur | I don't quite enjoy making the 3 that depend on it (in this case) to do the same | 15:36 |
| clarkb | that said from an openstack perspective this is why I constantly push on "please fix job instability" | 15:36 |
| dtantsur | well.... | 15:37 |
| *** udesale has quit IRC | 15:37 | |
| clarkb | and if I had to vocally say it I wouldve already lost my voice at least once doing it | 15:37 |
| *** gfidente has quit IRC | 15:37 | |
| dtantsur | I'd love somebody to fix e.g. PXE instability in nested virt | 15:37 |
| *** jamesmcarthur has joined #openstack-infra | 15:37 | |
| dtantsur | or similar vague issues we see from time to time | 15:37 |
| dtantsur | anyway, what I suggest is a bit less nuclear than bringing 'reverify' back | 15:39 |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Add ovn-octavia-provider driver project https://review.opendev.org/697076 | 15:39 |
| clarkb | the issue is from zuuls perspective (under currently expressable config) the child hasfailed when the parent does | 15:39 |
| clarkb | it os a different type of failure and we could possibly configure zuul to treat it differently but that requires updating zuul | 15:40 |
| dtantsur | right | 15:41 |
| *** jamesmcarthur has quit IRC | 15:42 | |
| openstackgerrit | Ilya Etingof proposed openstack/project-config master: Add `sushy-oem-idrac` project https://review.opendev.org/697159 | 15:42 |
| *** witek has quit IRC | 15:54 | |
| *** iury_course has quit IRC | 15:55 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for NetworkAmbiguous fail bug 1844568 https://review.opendev.org/697486 | 15:56 |
| openstack | bug 1844568 in tempest "[compute] "create_test_server" if networks is undefined and more than one network is present" [Medium,In progress] https://launchpad.net/bugs/1844568 - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | 15:56 |
| *** chkumar|ruck is now known as raukadah | 15:57 | |
| *** ykarel|afk is now known as ykarel|away | 16:01 | |
| *** ykarel|away has quit IRC | 16:05 | |
| AJaeger | frickler: time to branch - https://opendev.org/openstack/cookbook-openstack-common looks fine to me ;) | 16:08 |
| *** dave-mccowan has joined #openstack-infra | 16:08 | |
| openstackgerrit | Merged openstack/project-config master: Add `sushy-oem-idrac` project https://review.opendev.org/697159 | 16:10 |
| mordred | is sushy-oem-idrac highlighted weirdly in anyone else's client in those lines above? | 16:13 |
| clarkb | not mine | 16:13 |
| clarkb | but possible due to the `wrapper` ? | 16:13 |
| mordred | oh - wrapper was just highlighted the same way | 16:14 |
| mordred | what is that? | 16:14 |
| mordred | ahhhh | 16:14 |
| mordred | it's the backticks | 16:14 |
| mordred | *fascinating* | 16:14 |
| mordred | what a strange life choice glowingbear has made | 16:15 |
| clarkb | mordred: re being sick. I was getting over the last thing then my kids and my cousins kids ended up in the same place for thanksgiving and now we are all sick again | 16:15 |
| clarkb | also now we know how to get mordred's attention | 16:15 |
| AJaeger | Just got a RETRY_LIMIT, see https://zuul.opendev.org/t/openstack/build/05427596f97046299133a986dfa90d38 | 16:16 |
| AJaeger | "E: Failed to fetch http://mirror.dfw.rax.opendev.org/ubuntu/dists/bionic-updates/main/binary-amd64/Packages.gz File has unexpected size (1139750 != 1139760). Mirror sync in progress? [IP: 2001:4800:7819:105:be76:4eff:fe04:9b8a 80]" | 16:16 |
| mordred | clarkb: that sounds so wonderful | 16:17 |
| clarkb | AJaeger: I think that is the afs cache sync bug. If it persists we can delete things out of the cache via some afs command I don't remember | 16:17 |
| clarkb | I think it may be the case that we hit that now when we read the index, vos release happens, then we read the package file | 16:18 |
| clarkb | so it doesn't actually persist past the next index read | 16:18 |
| clarkb | and it is specific to Packages.gz becasue that filename is fixed unlike the actual package names (whcih are version specific filenames) | 16:18 |
| AJaeger | clarkb: ok, let's wait and see whether we get more reports | 16:20 |
| *** ykarel|away has joined #openstack-infra | 16:20 | |
| *** mattw4 has joined #openstack-infra | 16:21 | |
| *** ociuhandu has quit IRC | 16:23 | |
| frickler | AJaeger: thanks for double checking. we just need to update about a dozen other repos in the same way now ... but I hope we can branch tomorrow | 16:23 |
| AJaeger | frickler: hope it will all work out ! | 16:27 |
| *** igordc has joined #openstack-infra | 16:27 | |
| *** witek has joined #openstack-infra | 16:28 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [DNM][WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 16:32 |
| *** jamesmcarthur has joined #openstack-infra | 16:38 | |
| *** tesseract has quit IRC | 16:39 | |
| *** gyee has joined #openstack-infra | 16:41 | |
| *** pgaxatte has quit IRC | 16:41 | |
| *** slaweq_ has joined #openstack-infra | 16:41 | |
| *** jamesmcarthur has quit IRC | 16:42 | |
| *** openstackgerrit has quit IRC | 16:42 | |
| *** openstackgerrit has joined #openstack-infra | 16:43 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: enqueue: make trigger deprecated https://review.opendev.org/695446 | 16:43 |
| *** goldyfruit_ has joined #openstack-infra | 16:43 | |
| *** nicolasbock has joined #openstack-infra | 16:43 | |
| *** jpena is now known as jpena|brb | 16:45 | |
| *** goldyfruit___ has quit IRC | 16:46 | |
| *** mattw4 has quit IRC | 16:49 | |
| *** electrofelix has quit IRC | 16:49 | |
| *** ricolin has quit IRC | 16:54 | |
| *** dave-mccowan has quit IRC | 16:58 | |
| *** mattw4 has joined #openstack-infra | 16:58 | |
| openstackgerrit | Drew Walters proposed opendev/system-config master: lists: Add Airship VMP mailing lists https://review.opendev.org/689271 | 16:59 |
| *** dave-mccowan has joined #openstack-infra | 17:00 | |
| *** jamesmcarthur has joined #openstack-infra | 17:00 | |
| *** ijw_ has joined #openstack-infra | 17:02 | |
| *** slaweq_ has quit IRC | 17:02 | |
| *** ociuhandu has joined #openstack-infra | 17:02 | |
| *** dave-mccowan has quit IRC | 17:10 | |
| *** gfidente has joined #openstack-infra | 17:10 | |
| *** goldyfruit___ has joined #openstack-infra | 17:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: Add podman support https://review.opendev.org/695051 | 17:14 |
| *** goldyfruit_ has quit IRC | 17:15 | |
| *** dtantsur is now known as dtantsur|afk | 17:19 | |
| *** lennyb has quit IRC | 17:20 | |
| *** witek has quit IRC | 17:20 | |
| *** etingof has joined #openstack-infra | 17:22 | |
| *** lastmikoi has quit IRC | 17:24 | |
| *** jpena|brb is now known as jpena | 17:25 | |
| *** lennyb has joined #openstack-infra | 17:26 | |
| *** dpawlik has joined #openstack-infra | 17:28 | |
| *** lastmikoi has joined #openstack-infra | 17:28 | |
| *** dpawlik has quit IRC | 17:30 | |
| *** lennyb has quit IRC | 17:32 | |
| openstackgerrit | Merged opendev/irc-meetings master: New UC meeting time https://review.opendev.org/697489 | 17:35 |
| *** jamesmcarthur has quit IRC | 17:36 | |
| *** lpetrut has quit IRC | 17:40 | |
| *** lucasagomes has quit IRC | 17:43 | |
| *** jamesmcarthur has joined #openstack-infra | 17:46 | |
| *** ccamacho has quit IRC | 17:46 | |
| *** rfolco is now known as rfolco|brb | 17:48 | |
| *** jamesmcarthur has quit IRC | 17:50 | |
| *** jamesmcarthur has joined #openstack-infra | 17:58 | |
| *** jamesmcarthur has quit IRC | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| *** mattw4 has quit IRC | 18:03 | |
| *** lpetrut has joined #openstack-infra | 18:07 | |
| *** ykarel|away has quit IRC | 18:07 | |
| *** jamesmcarthur has joined #openstack-infra | 18:10 | |
| *** jamesmcarthur has quit IRC | 18:11 | |
| *** mattw4 has joined #openstack-infra | 18:15 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Set correct python version for non-chroot scripts https://review.opendev.org/697211 | 18:15 |
| *** ociuhandu has quit IRC | 18:17 | |
| *** jpena is now known as jpena|off | 18:18 | |
| *** michael-beaver has joined #openstack-infra | 18:19 | |
| *** armax has quit IRC | 18:25 | |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| cmurphy | could i request a new glean release? git log 1.16.0..master --oneline => 403646c Add support for SLES | 18:28 |
| clarkb | cmurphy: yes I can take a look at doing that after late breakfast | 18:28 |
| mordred | cmurphy: I think you just did request it | 18:28 |
| cmurphy | mordred: well would you look at that | 18:29 |
| mordred | ;) | 18:29 |
| cmurphy | thanks clarkb | 18:29 |
| *** jamesmcarthur has quit IRC | 18:32 | |
| *** Goneri has quit IRC | 18:35 | |
| *** rfolco|brb is now known as rfolco | 18:35 | |
| *** ralonsoh has quit IRC | 18:37 | |
| *** gfidente is now known as gfidente|afk | 18:38 | |
| *** dave-mccowan has joined #openstack-infra | 18:38 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Record job build attempts in inventory https://review.opendev.org/697548 | 18:43 |
| *** Goneri has joined #openstack-infra | 18:44 | |
| openstackgerrit | Monty Taylor proposed opendev/storyboard master: Build container images https://review.opendev.org/611191 | 18:58 |
| openstackgerrit | Monty Taylor proposed opendev/storyboard master: Change py3 testing to py36 https://review.opendev.org/697549 | 18:58 |
| mordred | fungi, SotK: ^^ updated the storyboard contianer patch - and another thing | 19:05 |
| fungi | thanks! | 19:05 |
| SotK | thanks, just reviewed the other thing :) | 19:06 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: Add podman support https://review.opendev.org/695051 | 19:07 |
| *** ociuhandu has joined #openstack-infra | 19:07 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Record job build attempts in inventory https://review.opendev.org/697548 | 19:11 |
| *** tobias-urdin has quit IRC | 19:16 | |
| *** ijw_ has quit IRC | 19:17 | |
| clarkb | cmurphy: tag pushed | 19:18 |
| clarkb | cmurphy: release things should happen shortly | 19:18 |
| cmurphy | tyty | 19:18 |
| openstackgerrit | Merged opendev/system-config master: Update gitea docs https://review.opendev.org/694427 | 19:23 |
| clarkb | https://pypi.org/project/glean/1.17.0/ there it is | 19:29 |
| *** ociuhandu has quit IRC | 19:31 | |
| *** dave-mccowan has quit IRC | 19:36 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: Add podman support https://review.opendev.org/695051 | 19:44 |
| *** pkopec has quit IRC | 19:52 | |
| *** hashar has joined #openstack-infra | 19:53 | |
| *** jklare has quit IRC | 19:59 | |
| *** jklare has joined #openstack-infra | 20:01 | |
| *** rcernin has joined #openstack-infra | 20:02 | |
| *** dosaboy has quit IRC | 20:04 | |
| *** igordc has quit IRC | 20:05 | |
| *** rcernin has quit IRC | 20:18 | |
| openstackgerrit | Merged opendev/infra-specs master: Declare victory on StoryBoard https://review.opendev.org/697201 | 20:21 |
| rlandy | AJaeger: clarkb: we also seeing the rax mirror issue ... | 20:22 |
| rlandy | 2019-12-05 16:19:21.042766 | ubuntu-bionic | E: Failed to fetch http://mirror.iad.rax.openstack.org/ubuntu/dists/bionic-updates/main/binary-amd64/Packages.gz File has unexpected size (1139750 != 1139760). Mirror sync in progress? [IP: 2001:4802:7807:103:be76:4eff:fe20:53f8 80] | 20:22 |
| rlandy | reporting again as mentioned above | 20:22 |
| clarkb | rlandy: looks to be in the same time range as AJaeger's? | 20:23 |
| clarkb | fungi ^ is it possible there are races around the index file contents and the packages.gz files as I hypthesised before? | 20:23 |
| rlandy | clarkb: so just a hitch? | 20:24 |
| rlandy | EmilienM reported seeing this failure | 20:24 |
| clarkb | rlandy: my current hunch is it is a race betweens afs file updates and apt's fetching of said files | 20:24 |
| clarkb | there us a different race that our use of afs addresses but I think this one may slip through | 20:25 |
| rlandy | are we expecting a fix? or just retest and hope for better times ahead? | 20:25 |
| clarkb | I think it should work now. It only happens in a small period of time around the fs update | 20:27 |
| clarkb | and doesnt happen every time | 20:27 |
| rlandy | checking logstash | 20:27 |
| clarkb | as for fixing if I am correct in my hunch then this is a part of apts inherent racing | 20:27 |
| clarkb | it may also be an afs bug | 20:28 |
| rlandy | k - logstash shows a good few failures around the same time | 20:32 |
| *** jklare has quit IRC | 20:58 | |
| fungi | clarkb: there shouldn't be races around generation of the indices and performing vos release. remember this is a constructed package repository with mirrors of packages, not a mirror of a package repository | 20:58 |
| fungi | it _is_ possible though that there are cache corruption problems on some of the servers... was it in the same provider each time? | 20:59 |
| fungi | i remember we saw this a bunch when testing kafs, though that got rolled back | 21:00 |
| fungi | we only have one process which writes to that volume, and it does so with a flock to prevent other concurrent runs overlapping it, and performs a vos release only once it's finished | 21:01 |
| fungi | and we're serving the read-only copy, not the read-write volume | 21:02 |
| *** jklare has joined #openstack-infra | 21:02 | |
| fungi | looking back, AJaeger reported one in rax-dfw around 16:16z | 21:03 |
| fungi | so same provider but different region (and so different server/cache) | 21:04 |
| clarkb | fungi: I mean a race between files being updated and apt pulling them | 21:05 |
| clarkb | fungi: apt-get update somehow gets old version of signing file with file sizes then gets new version of packages.gz | 21:05 |
| clarkb | or vice versa | 21:05 |
| fungi | that could be... looking to see how that length is recorded | 21:05 |
| clarkb | the packages themselves should be fine because we don't delete old packages so if you've got an old index you still get the old packages just fine | 21:06 |
| fungi | i was interpreting it as a decompression error, but i may have misread | 21:06 |
| clarkb | but could be a race in the metadata itself | 21:06 |
| clarkb | fungi: iirc its checking filesize against what is listed in some index | 21:06 |
| *** eharney has quit IRC | 21:06 | |
| fungi | possibly http://mirror.iad.rax.openstack.org/ubuntu/dists/bionic-updates/main/Contents-amd64.gz but i'm checking now | 21:07 |
| fungi | taking a loooong time to retrieve for some reason | 21:07 |
| fungi | getting around 700kb/sec average | 21:08 |
| clarkb | I think that is "normal" for uncached files | 21:08 |
| clarkb | though you'd expect that to be a hot file and be cached | 21:08 |
| fungi | in that case i guess apt isn't hitting that file, since it's timestamped from 7 hours ago | 21:08 |
| clarkb | or we aren't caching it | 21:09 |
| *** dosaboy has joined #openstack-infra | 21:09 | |
| fungi | but also, no, that file doesn't record a length for binary-amd64/Packages.gz so i'll keep looking | 21:09 |
| fungi | yeah, nevermind, it's in http://mirror.iad.rax.openstack.org/ubuntu/dists/bionic-updates/Release | 21:10 |
| fungi | 1e560614eb300b783d16637560a69d76 1139750 main/binary-amd64/Packages.gz | 21:10 |
| clarkb | http://mirror.iad.rax.openstack.org/ubuntu/dists/bionic-updates/Release that file | 21:10 |
| clarkb | ya | 21:10 |
| fungi | and yep, `wget -qO- http://mirror.iad.rax.openstack.org/ubuntu/dists/bionic-updates/main/binary-amd64/Packages.gz|wc -c` gives me 1139750 | 21:11 |
| clarkb | if I do an apt-get update on my local xenial server it grabs the InRelease files then the Packages files and in quick succession | 21:11 |
| clarkb | is it possible that apt-get grabs the release file first with old data, then gets the packages file with new data and new size and we see failures | 21:12 |
| clarkb | that gap isn't big in local testing but does exist | 21:12 |
| fungi | the file with the length in it and the file to which it refers are timestamped only a couple minutes apart | 21:12 |
| clarkb | fungi: right this depends on the timing of vos release completion I think | 21:12 |
| *** jtomasek has quit IRC | 21:12 | |
| clarkb | basically you'd need a vos release to complete in the middle of an apt-get update | 21:13 |
| clarkb | which for the time span we see the errors for probably doesn't line up well | 21:13 |
| fungi | seems possible. checking to see when vos release started/ended | 21:13 |
| *** dave-mccowan has joined #openstack-infra | 21:13 | |
| fungi | the vos release for that volume ran from 14:47:05 to 14:50:24 | 21:14 |
| clarkb | which is about 1.25 hours prior to when we see the issue | 21:15 |
| fungi | it ran again between 16:38:45 and 16:39:19 | 21:15 |
| fungi | (cronjob for it runs every two hours) | 21:16 |
| fungi | but yeah, the timestamps for those indices were prior to the 14:47:05 vos release, so didn't update in the 16:38:45 or later releases | 21:17 |
| clarkb | we should probably also double check that the mirrors are talking to the RO volume | 21:17 |
| fungi | DocumentRoot /var/www/mirror | 21:19 |
| fungi | lrwxrwxrwx 1 root root 32 Oct 13 2017 /var/www/mirror/ubuntu -> /afs/openstack.org/mirror/ubuntu | 21:19 |
| fungi | so that's the read-only path | 21:19 |
| clarkb | ok, chances are we are running into an afs read issue? | 21:21 |
| clarkb | and getting an incomplete file size? or the web server is doing an incomplete transfer | 21:21 |
| fungi | extra bytes if i'm reading the error correctly | 21:22 |
| *** jaosorior has joined #openstack-infra | 21:22 | |
| *** sgw has joined #openstack-infra | 21:22 | |
| fungi | File has unexpected size (1139750 != 1139760). | 21:22 |
| fungi | 1e560614eb300b783d16637560a69d76 1139750 main/binary-amd64/Packages.gz | 21:22 |
| clarkb | hrm that makes it more interesting as I would expect fewer not more | 21:22 |
| fungi | yeah, truncated read is easier to explain | 21:22 |
| fungi | currently the Release file and a wget|wc -c agree on 1139750 | 21:24 |
| fungi | so we need an explanation for why it got an 1139760 byte Packages.gz there | 21:24 |
| clarkb | fungi: what if Packages.gz at 1139760 bytes is an old version and we got a new (current) Release file? | 21:25 |
| *** rfolco has quit IRC | 21:25 | |
| clarkb | so no read problems other than we got a file from older vos release | 21:25 |
| fungi | except the timestamp for when both were written was prior to the vos release starting | 21:25 |
| fungi | Packages.gz written at 2019-12-05 14:32 and Release written a couple minutes later at 14:34 then vos release initiated from 14:47:05 to 14:50:24 and we didn't get the bad read until 16:19:21 and the next vos release didn't start until 16:38:45 | 21:28 |
| clarkb | yeah super odd | 21:28 |
| fungi | so i guess possible but the timeline isn't anywhere close to backing up that theory, we'd have to operate on an assumption that we've got bad timestamps somewhere too | 21:29 |
| clarkb | or that the cache somehow reverts to an older state long after a cache update event should happen | 21:29 |
| fungi | right, plenty of pathological explanations but nothing passes an occam's razor sniff test for me yet | 21:30 |
| clarkb | ya | 21:30 |
| fungi | what logstash query were you using? curious to look at the cluster | 21:30 |
| clarkb | and the file size is such that I doubt it is a 500 error response or similar from apache | 21:30 |
| clarkb | I haven't done a search myself yet. was just looking at the two example instances | 21:30 |
| * clarkb makes a query | 21:31 | |
| fungi | right, being within 10 bytes on a 1.1mb file is more than mere coincidence | 21:31 |
| clarkb | fungi: message:"File has unexpected size" seems to work | 21:31 |
| fungi | i half wonder if there's a transparent "web accelerator" in between the nodes and the mirror, buried somewhere in rackspace's routing infrastructure, which served a cached copy or something (this is http after all) | 21:32 |
| clarkb | looks like it happened over an almost 2 hour time period that I now have to do timezone math on to figure out | 21:32 |
| fungi | i do not have to do timezone math, taking a look | 21:32 |
| clarkb | ok the first occurence does seem to lin up with the 14:47-14:50 vos release | 21:33 |
| clarkb | then it stops at ~16:35 | 21:33 |
| clarkb | I think that implies it was broken after that first vos release until the subsequent release | 21:33 |
| fungi | yep, i concur | 21:34 |
| clarkb | I guess we now need to determine if the vos release and associated cache updates were successful and the actual disk content was bad or if the vos release itself had problems. The file timetsamps imply to me that the vos release itself had problem | 21:34 |
| fungi | note that these are not all the same files, different expected and found lengths | 21:34 |
| clarkb | the files don't seem to be region specific either | 21:36 |
| clarkb | the data in the release file seems to match what we currently have so maybe we assume that data was correct | 21:37 |
| clarkb | that would then imply the Packages.gz contents were the ones out of sync (too large) | 21:37 |
| fungi | rax-dfw and rax-iad (we already had examples from each) but also ovh-bhs1 | 21:37 |
| fungi | though i'm interestingly not seeing failures from any other providers like i would expect | 21:37 |
| *** Goneri has quit IRC | 21:38 | |
| clarkb | fungi: I don't think it is a bionic issue as the ovh mirror is xenial iirc | 21:38 |
| fungi | nope, adjusting the query window to go back further i also see rax-ord in the mix | 21:39 |
| *** pkopec has joined #openstack-infra | 21:39 | |
| clarkb | fungi: in the same time window or another occurence? | 21:39 |
| fungi | same window, just closer to the start | 21:39 |
| fungi | we had more than 500 hits so ls was limiting query results | 21:39 |
| fungi | i narrowed the window to closer to the start of the cluster | 21:40 |
| clarkb | " Very occasionally problems might arise due to cache inconsistencies. The following commands (subcommands of fs) can be used to remove stale cache data." dartmouth seems to maybe know about this | 21:40 |
| clarkb | and it lists the fs flush and fs flushvolume which we've used in the past to help address this | 21:41 |
| *** ociuhandu has joined #openstack-infra | 21:41 | |
| fungi | i need to go meet some folks for dinner, but might be interesting to ponder why we only saw it impacting rax and ovh builds, but saw more than enough hits that we should have statistically gotten a few other providers in there | 21:41 |
| fungi | i'll be back in an hour or two | 21:41 |
| clarkb | fungi: my guess is that since caches are per server inconsistencies may arise on them independently | 21:42 |
| clarkb | perhaps due to missed udp packets? | 21:42 |
| fungi | perhaps | 21:42 |
| clarkb | I don't know if there is any handhsaking down around those packets | 21:42 |
| * fungi <- out | 21:42 | |
| *** pkopec has quit IRC | 21:43 | |
| *** persia has quit IRC | 21:43 | |
| *** persia has joined #openstack-infra | 21:45 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Log item warning messages at info level https://review.opendev.org/697571 | 21:45 |
| clarkb | ianw: corvus ^ any of this sound familiar to you? | 21:45 |
| *** ijw has joined #openstack-infra | 21:46 | |
| ianw | sorry, not something i've heard of :/ | 21:47 |
| corvus | clarkb: do i need to read back to 20:22? | 21:47 |
| clarkb | corvus: ya 20:22 gives the background | 21:47 |
| clarkb | 20:58 may be sufficient though | 21:48 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile : install sudo for nodepool-builder https://review.opendev.org/694709 | 21:48 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: add DEBUG environment flag https://review.opendev.org/694845 | 21:48 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 21:48 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 21:48 |
| clarkb | https://lists.openafs.org/pipermail/openafs-info/2012-November/039023.html <- taking the udp dropped packets idea brings that up | 21:49 |
| clarkb | maybe we should increase udp buffer size? | 21:49 |
| *** snierodz has quit IRC | 21:50 | |
| *** stephenfin has quit IRC | 21:50 | |
| clarkb | if we think that is a reasonable next step I can start looking at that after my bike ride | 21:51 |
| *** jaosorior has quit IRC | 21:53 | |
| corvus | remember that rx is basically tcp-before-tcp-existed-implemented-on-udp -- so with dropped packets, we're not supposed to end up with incorrect data or anything -- just poor performance. aiui. | 21:54 |
| corvus | so yeah, there should be handshaking | 21:54 |
| *** stephenfin has joined #openstack-infra | 21:57 | |
| clarkb | ok, so maybe worth doing for performance reasons but wont help this issue necessarily | 21:58 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: install sudo for nodepool-builder https://review.opendev.org/694709 | 22:03 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: add DEBUG environment flag https://review.opendev.org/694845 | 22:03 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 22:03 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 22:03 |
| *** goldyfruit___ has quit IRC | 22:05 | |
| openstackgerrit | Saul Wold proposed openstack/project-config master: starlingx: add oidc-auth-armada-app https://review.opendev.org/697576 | 22:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix potential wedge with provides/requires/dependencies https://review.opendev.org/697579 | 22:19 |
| *** ociuhandu has quit IRC | 22:20 | |
| sgw | Morning folks, I just sent a review to project-config for adding a new starlingx/oidc-auth-armada-app, when it merges, we will need a Gerrit group added "starlingx-oidc-auth-armada-app-core", please add robert.church@windriver.com as initial reviewer | 22:20 |
| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 22:27 |
| *** mattw4 has quit IRC | 22:27 | |
| *** rcernin has joined #openstack-infra | 22:28 | |
| *** jamesmcarthur has joined #openstack-infra | 22:31 | |
| ianw | ok, crazy thought maybe ... we do have a global cluster that serves our AFS contents over https -- possibly we should put a load-balancer infront of our mirrors and serve things like governance.openstack.org via that? | 22:39 |
| corvus | ianw: yes, or we can just scale out on separate servers so we're not hitting our web presence with our ci system | 22:42 |
| ianw | yeah, that was why i had the generic load-balancer idea in the original spec | 22:44 |
| ianw | btw https://review.opendev.org/#/c/677903/ is reviewable, which makes haproxy more generic | 22:44 |
| *** jamesmcarthur has quit IRC | 22:45 | |
| *** jamesmcarthur has joined #openstack-infra | 22:45 | |
| ianw | i was thinking the mirrors are probably not under enough load that a bit of static interactive web traffic would notice. just a thought :) | 22:45 |
| *** hashar has quit IRC | 23:00 | |
| *** ijw has quit IRC | 23:04 | |
| *** tkajinam has joined #openstack-infra | 23:07 | |
| *** mattw4 has joined #openstack-infra | 23:15 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add roles for a basic static server https://review.opendev.org/697587 | 23:20 |
| *** mattw4 has quit IRC | 23:20 | |
| ianw | https://zuul.opendev.org/t/zuul/stream/28896b23208645fca2ddd176bb1b4c79?logfile=console.log ... looks like we lost at least one streamer | 23:24 |
| *** sshnaidm is now known as sshnaidm|afk | 23:24 | |
| ianw | ze04 by the looks of things | 23:25 |
| *** jamesmcarthur has quit IRC | 23:28 | |
| *** nicolasbock has quit IRC | 23:29 | |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| fungi | ianw: i'm not clear on how that's any different from serving it from files.o.o (after maybe making an additional files.o.o to do failover between) | 23:33 |
| fungi | or is that pretty much what you said? | 23:34 |
| fungi | we could i guess serve it from files.o.o with one of the mirror servers as a backup in the lb pool | 23:34 |
| ianw | fungi: yeah, it was more that we *probably* don't need dedicated servers for performance reasons | 23:35 |
| fungi | oh, sure | 23:35 |
| ianw | but "keep things separate for admin purposes" reasons is different :) | 23:35 |
| *** ociuhandu has quit IRC | 23:35 | |
| *** ijw has joined #openstack-infra | 23:36 | |
| rpioso | clarkb: I'm still experiencing lengthy, often timed out, git clone issues. Is there a way to specify a specific server? If so, which ones may I try? | 23:36 |
| clarkb | rpioso: you can use https://gitea0X.opendev.org:3000 with X values 1-8 inclusive | 23:37 |
| clarkb | rpioso: fwiw I was unable to reproduce your experience against gitea02 yesterday and did hundreds of clones | 23:37 |
| *** tosky has quit IRC | 23:38 | |
| clarkb | would be curious to see if other hosts are also slow or not as they are all hosted adjacent to each other. If they are all slow that could point to a networking issue between gitea and you | 23:38 |
| fungi | rpioso: do you have ipv6 connectivity to the internet, or only ipv4? | 23:38 |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| fungi | worth noting that we only publish ipv4 records in dns for the individual gitea servers, but publish both ipv4 and ipv6 addresses for the load balancer | 23:40 |
| fungi | so if you're seeing slow performance to the lb but not to any individual gitea servers and you have ipv6 connectivity... | 23:40 |
| *** ijw has quit IRC | 23:40 | |
| rpioso | fungi: Checking ... | 23:41 |
| fungi | note that git accepts -4 and -6 command-line options to choose the ip address family | 23:42 |
| rpioso | fungi: Looks like it has both ipv4 and ipv6 connectivity to the internet. | 23:42 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openafs master: Increase udp buffer sizes when installing afs client https://review.opendev.org/697588 | 23:42 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Increase udp buffer sizes when installing afs client https://review.opendev.org/697589 | 23:43 |
| clarkb | two chagnes for udp buffer size bumps. One gets the puppeted servers the other the ansibled | 23:43 |
| fungi | so if you're seeing slow performance to https://opendev.org/... you can try something like `git clone -4 https://opendev.org/...` to see if ipv4 is also slow | 23:43 |
| fungi | that might at least tell you if your ipv6 connectivity is terrible compared to ipv4 | 23:44 |
| clarkb | reboots will be required for those changes to take effect or we can manually run sysctl -w | 23:44 |
| * rpioso kicks off the infinite clone loop | 23:44 | |
| *** rlandy is now known as rlandy|afk | 23:45 | |
| rpioso | fungi: Looks like clone -4 is slow. Gonna try -6. | 23:46 |
| fungi | odds are -6 is the default for you anyway if you have ipv6 connectivity | 23:46 |
| rpioso | fungi "fatal: unable to access 'https://opendev.org/openstack/ironic/': Couldn't connect to server" | 23:47 |
| clarkb | you probably don't have ipv6 at all then | 23:47 |
| fungi | yeah | 23:48 |
| rpioso | The interface has both ipv4 and ipv6 addresses. | 23:48 |
| clarkb | rpioso: is your ipv6 address global? | 23:48 |
| fungi | easier way to tell is to check your v6 routing table for a default route | 23:49 |
| fungi | `ip -6 route show | grep default` | 23:49 |
| rpioso | fungi: Notta | 23:50 |
| fungi | then you probably just have linklocal v6 addressing | 23:50 |
| fungi | that tends to be on by default | 23:50 |
| rpioso | fungi: Gotcha | 23:50 |
| fungi | so ignore all the ipv4 vs ipv6 discussion, you're seeing poor performance and you connect over ipv4. that's something to go on anyway | 23:50 |
| fungi | at least your connections to the individual backends will be via the same routes and address family | 23:51 |
| fungi | so it does make ruling out some classes of problems slightly easier | 23:51 |
| *** jamesmcarthur has quit IRC | 23:52 | |
| rpioso | The interesting thing is that the slowness is intermittent. Several clones in a row can complete in 10-14 seconds each. And then the next one takes forever, say two minutes or more. Sometimes it'll take so long that it'll time out. There can be a few of those in a row, too. Rinse and repeat. | 23:52 |
| clarkb | fungi: mnaser ttx sent an updated governance proposal email trying to incporate feedback you've sent. Thank you! | 23:52 |
| fungi | rpioso: that's when the government has decided to intercept your connection (just kidding, i hope!) | 23:53 |
| clarkb | rpioso: fungi probably the next step is to check if it is only gitea02 or if all 8 exhibit this problem | 23:53 |
| clarkb | if all 8 then it points more to networking problem imo | 23:53 |
| clarkb | since they share a common network | 23:53 |
| fungi | yeah, if it's all 8 of them, then we can start looking for signs of packet loss along the outbound and return routes between your clients and vexxhost where these servers live | 23:55 |
| rpioso | Failure: git clone https://gitea02.opendev.org:3000/openstack/ironic.git /opt/stack/ironic | 23:55 |
| fungi | and if it's none of them, and only shows up when going through the load balancer, that's yet another thing we probably need to dig into | 23:55 |
| rpioso | fatal: unable to access 'https://gitea02.opendev.org:3000/openstack/ironic.git/': Failed to connect to gitea02.opendev.org port 3000: Connection refused | 23:55 |
| fungi | rpioso: that sounds like something close to your client is doing packet filtering | 23:56 |
| fungi | is there a corporate firewall or something which might block connectivity to 3000/tcp on remote addresses? | 23:56 |
| rpioso | fungi: Likely | 23:56 |
| fungi | i can run the same git clone command from home and have no special access | 23:57 |
| fungi | so it's nothing we're blocking server-side | 23:57 |
| clarkb | and ssl even validates now :) | 23:57 |
| rpioso | I can try that when I get home. | 23:57 |
| rpioso | The other thing is that I have another system which can successfully and speedily clone like that battery manufacturer bunny :-) | 23:58 |
| rpioso | Their network connections are different. | 23:59 |
| fungi | rpioso: yeah, hate to say it but i would suspect something on or topologically near the client which is experiencing this problem | 23:59 |
| clarkb | rpioso: checking from home doesnt necessarily help | 23:59 |
| clarkb | if it is a network path issue | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!