Monday, 2019-08-12

*** guoqiao has quit IRC00:00
*** yolanda has quit IRC00:12
*** yolanda has joined #openstack-infra00:13
*** jamesmcarthur has joined #openstack-infra00:45
*** jamesmcarthur has quit IRC01:13
*** gregoryo has joined #openstack-infra01:13
*** jamesmcarthur has joined #openstack-infra01:20
*** cmorpheus is now known as cmurphy01:24
*** jamesmcarthur has quit IRC01:30
*** dchen has quit IRC01:33
*** dchen has joined #openstack-infra01:34
*** yamamoto has joined #openstack-infra01:45
*** stevebaker2 has quit IRC02:03
*** apetrich has quit IRC02:08
*** jamesmcarthur has joined #openstack-infra02:12
*** yamamoto has quit IRC02:26
*** yamamoto has joined #openstack-infra02:26
*** yolanda has quit IRC02:32
*** yolanda has joined #openstack-infra02:37
*** jamesmcarthur has quit IRC02:38
*** yolanda has quit IRC02:43
*** yolanda has joined #openstack-infra02:44
*** jamesmcarthur has joined #openstack-infra02:47
*** mordred has joined #openstack-infra02:48
mordredinfra-root: heya - I have just been informed that I will not have internet starting at 6am Atlantic Standard Time. The expected time for it to return is 10AM - but I'll obviously let folks know whence it returneth.02:49
*** rfolco has quit IRC02:49
*** bobh has joined #openstack-infra02:50
*** jamesmcarthur has quit IRC02:56
*** jamesmcarthur has joined #openstack-infra02:59
*** dchen has quit IRC03:03
*** dchen has joined #openstack-infra03:06
*** bobh has quit IRC03:10
*** dchen has quit IRC03:13
*** dchen has joined #openstack-infra03:14
fungii'll be in cars, airplanes and airports most of tomorrow, but will pop in when possible03:17
fungiwell, today utc03:17
* fungi enters power-saving standby mode for a few hours03:17
*** yolanda has quit IRC03:17
*** yolanda has joined #openstack-infra03:22
*** jamesmcarthur has quit IRC03:25
*** psachin has joined #openstack-infra03:35
*** ykarel|away has joined #openstack-infra03:35
*** udesale has joined #openstack-infra04:04
*** dchen has quit IRC04:09
*** dchen has joined #openstack-infra04:09
*** dchen has quit IRC04:11
*** ykarel|away has quit IRC04:11
*** dchen has joined #openstack-infra04:11
*** toabctl has quit IRC04:21
*** toabctl has joined #openstack-infra04:24
*** ykarel|away has joined #openstack-infra04:27
*** ykarel|away is now known as ykarel04:34
*** bhavikdbavishi has joined #openstack-infra05:00
*** dchen has quit IRC05:01
*** dchen has joined #openstack-infra05:02
*** raukadah is now known as chkumar|ruck05:10
*** sshnaidm is now known as sshnaidm|afk05:29
*** pcaruana has joined #openstack-infra05:29
*** pcaruana has quit IRC05:37
*** bhavikdbavishi1 has joined #openstack-infra05:38
*** bhavikdbavishi has quit IRC05:40
*** bhavikdbavishi1 is now known as bhavikdbavishi05:40
*** odicha has joined #openstack-infra05:44
*** jamesmcarthur has joined #openstack-infra05:47
*** dchen has quit IRC05:47
*** janki has joined #openstack-infra05:48
*** pcaruana has joined #openstack-infra05:50
*** dchen has joined #openstack-infra05:50
*** iurygregory has joined #openstack-infra05:57
*** jamesmcarthur has quit IRC06:03
*** jamesmcarthur has joined #openstack-infra06:06
*** dpawlik has joined #openstack-infra06:08
*** e0ne has joined #openstack-infra06:16
*** e0ne has quit IRC06:17
*** jamesmcarthur has quit IRC06:21
*** ramishra has joined #openstack-infra06:28
*** markvoelker has joined #openstack-infra06:31
*** slaweq has joined #openstack-infra06:33
*** markvoelker has quit IRC06:36
*** dtantsur|afk is now known as dtantsur06:42
*** jamesmcarthur has joined #openstack-infra06:45
*** AJaeger has quit IRC06:47
*** jamesmcarthur_ has joined #openstack-infra06:51
*** jamesmcarthur_ has quit IRC06:53
*** jamesmcarthur_ has joined #openstack-infra06:53
*** jamesmcarthur_ has quit IRC06:53
*** Goneri has joined #openstack-infra06:53
*** jamesmcarthur_ has joined #openstack-infra06:54
*** jamesmcarthur has quit IRC06:54
*** ramishra_ has joined #openstack-infra06:57
*** ramishra has quit IRC06:59
*** jamesmcarthur_ has quit IRC07:04
*** AJaeger has joined #openstack-infra07:04
*** jamesmcarthur has joined #openstack-infra07:05
*** ginopc has joined #openstack-infra07:06
*** dchen has quit IRC07:09
*** dchen has joined #openstack-infra07:09
*** jamesmcarthur has quit IRC07:10
*** rcernin has quit IRC07:13
*** tesseract has joined #openstack-infra07:14
*** tosky has joined #openstack-infra07:17
*** kopecmartin|off is now known as kopecmartin07:18
*** jpena|off is now known as jpena07:25
*** panda has quit IRC07:30
*** ykarel is now known as ykarel|lunch07:33
*** panda has joined #openstack-infra07:33
*** lucasagomes has joined #openstack-infra07:37
*** roman_g has joined #openstack-infra07:46
*** zbr is now known as zbr|flu07:47
*** e0ne has joined #openstack-infra07:49
*** ramishra_ has quit IRC07:54
*** rpittau|afk is now known as rpittau07:54
*** ramishra has joined #openstack-infra07:58
*** yamamoto has quit IRC08:00
*** hrw has joined #openstack-infra08:09
hrwmorning08:10
hrwfungi: any luck with buster-updates mirroring?08:10
*** yamamoto has joined #openstack-infra08:10
*** dchen has quit IRC08:12
*** yamamoto has quit IRC08:13
hrwhttps://review.opendev.org/#/c/671445/ changed flavour for linaro-london arm64 nodes from 8 vcpu to 16 vcpu. When nodes will get updated to 16 vcpu?08:13
*** yamamoto has joined #openstack-infra08:14
*** dchen has joined #openstack-infra08:15
*** janki has quit IRC08:16
*** janki has joined #openstack-infra08:17
*** pkopec has joined #openstack-infra08:17
*** jchhatbar has joined #openstack-infra08:18
*** yamamoto has quit IRC08:19
*** dchen has quit IRC08:19
*** yamamoto has joined #openstack-infra08:19
*** yamamoto has quit IRC08:20
*** dchen has joined #openstack-infra08:20
*** dchen has quit IRC08:20
ianwhrw: that should have applied straight away08:20
hrwianw: http://zuul.openstack.org/stream/8727f9d6bd344c79a2635b708853dea2?logfile=console.log shows 'CPUs: 8'08:21
hrwon debian-stretch-arm64-linaro-london-000922904308:21
*** janki has quit IRC08:22
ianw2019-08-12 08:05:40,331 DEBUG nodepool.driver.NodeRequestHandler[nl03-8304-PoolWorker.linaro-london-main]: Locked existing node 0009229043 for request 300-000473508008:22
ianwhrm, possibly it was an old node08:23
hrwanyway in 3h it will be known. as it will timeout on 8 cpus08:23
hrwianw: can you check how it looks with other nodes?08:25
*** kjackal has joined #openstack-infra08:31
*** derekh has joined #openstack-infra08:37
*** gregoryo has quit IRC08:40
*** ramishra_ has joined #openstack-infra08:42
*** tesseract has quit IRC08:42
*** AJaeger has quit IRC08:42
*** slaweq has quit IRC08:42
*** mordred has quit IRC08:42
*** gsantomaggio has quit IRC08:42
*** persia has quit IRC08:42
*** mordred has joined #openstack-infra08:43
*** AJaeger has joined #openstack-infra08:43
*** slaweq has joined #openstack-infra08:43
*** tesseract has joined #openstack-infra08:43
*** jchhatbar has quit IRC08:44
*** ramishra has quit IRC08:45
*** openstackgerrit has quit IRC08:45
*** ykarel|lunch is now known as ykarel08:46
*** gsantomaggio has joined #openstack-infra08:47
*** persia has joined #openstack-infra08:47
*** irclogbot_0 has quit IRC08:49
*** adriancz has joined #openstack-infra08:52
*** irclogbot_2 has joined #openstack-infra08:52
*** gfidente has joined #openstack-infra08:54
*** ociuhandu has joined #openstack-infra09:05
*** jaosorior has joined #openstack-infra09:07
*** ociuhandu has quit IRC09:17
*** ociuhandu has joined #openstack-infra09:22
*** yamamoto has joined #openstack-infra09:36
*** ociuhandu has quit IRC09:41
noonedeadpunkfolks, seems that gerrit irc bot is down...09:47
noonedeadpunkcan you kindly check that out?09:47
*** diga has joined #openstack-infra09:49
*** sshnaidm|afk is now known as sshnaidm09:52
*** rascasoft has quit IRC09:56
*** rascasoft has joined #openstack-infra09:59
*** ociuhandu has joined #openstack-infra09:59
*** ociuhandu has quit IRC10:01
*** ociuhandu has joined #openstack-infra10:01
*** onovy has joined #openstack-infra10:07
onovyhi, what is procedure to discontinue project? i'm maintainer of swauth and i'm going to discontinue it/offer it to somebody else10:08
*** markvoelker has joined #openstack-infra10:09
AJaegeronovy: please follow https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project10:13
*** markvoelker has quit IRC10:14
*** kaisers has quit IRC10:16
*** kaisers has joined #openstack-infra10:18
*** ociuhandu has quit IRC10:20
*** ociuhandu has joined #openstack-infra10:21
*** bhavikdbavishi has quit IRC10:22
*** bhavikdbavishi has joined #openstack-infra10:24
arxcruzHello guys, what's the plans to move zuul_ansible_version to 2.8 ?10:25
*** bhavikdbavishi has quit IRC10:33
fungi2019-08-12 08:45:22     <--     openstackgerrit (~openstack@review01.openstack.org) has quit (Ping timeout: 272 seconds)10:35
fungi#status log restarted gerritbot after it suffered a ping timeout at 08:45 utc10:36
openstackstatusfungi: finished logging10:36
*** bhavikdbavishi has joined #openstack-infra10:36
*** ykarel is now known as ykarel|afk10:36
*** kjackal has quit IRC10:41
onovyAJaeger:thanks10:42
*** pcaruana has quit IRC10:43
*** pcaruana has joined #openstack-infra10:43
ianwhrw: from http://zuul.openstack.org/nodes the nodes are really old.  looks like a new one building10:51
*** kjackal has joined #openstack-infra10:52
hrwianw: is there a way to kill old not-used-at-time nodes to make sure that when they be recreated it will be new flavour?10:53
ianwyes, it's a manual process but i can delete the old nodes10:54
hrwianw: please do. thanks10:54
*** jaosorior has quit IRC10:54
ianw| 0010049614 | linaro-london        | debian-stretch-arm64 | 4dc92d5a-0c2c-44cc-b76e-e5807b1946a6 | None            | None                                   | building | 00:00:28:38  | locked   |10:58
ianwhrw: done, but ^ has been building for a while.  i won't get a chance to look right now, but i can tomorrow if nodes aren't building correctly.  that seems a long time :/10:59
hrwianw: thanks11:00
fungihrw: i tracked down the cause of the buster-updates problem, but the solution isn't straightforward. i need to try to hack some sort of parameterization into the configure-mirrors role to be able to turn off adding the -updates suite for specific releases and not others11:00
hrwfungi: ok. thanks11:01
fungiand i'm on airplanes all day today so not sure how far i'll get (well, 3k miles but as far as fixing that particular problem i mean)11:01
hrwfungi: safe travel then11:01
fungithanks!11:02
hrwairtime days are out of work for me11:02
*** ykarel|afk is now known as ykarel11:02
hrwbook, movie, music, podcasts.11:02
fungithe root of the issue is that if reprepro is asked to mirror a suite which has no packages, it doesn't generate empty indices, instead it just skips it11:02
hrwah.11:03
fungiand buster-updates won't have any packages until the first buster stable point release (due september 7)11:04
*** rosmaita has quit IRC11:05
hrwwill wait then11:05
hrweasiest solution ;)11:05
fungiwell, we need to solve this anyway, because the -backports suite also has a tendency to not exist early on after a release11:06
fungi(though it's working now for buster, it will be an issue again for bullseye, or sid if we decide to add nodes for unstable)11:06
fungianyway, i'm offline for a while. loooong drive to the airport starts now11:07
*** janki has joined #openstack-infra11:08
*** whoami-rajat has joined #openstack-infra11:08
hrwyeah11:10
*** dchen has joined #openstack-infra11:11
*** dchen has quit IRC11:12
*** dchen has joined #openstack-infra11:13
*** rosmaita has joined #openstack-infra11:17
*** udesale has quit IRC11:18
*** guoqiao_ has quit IRC11:20
*** electrofelix has joined #openstack-infra11:23
*** tdasilva has joined #openstack-infra11:24
*** ociuhandu has quit IRC11:27
*** ociuhandu has joined #openstack-infra11:27
*** ociuhandu_ has joined #openstack-infra11:28
*** ociuhandu has quit IRC11:32
*** ginopc has quit IRC11:32
*** jpena is now known as jpena|lunch11:40
*** ociuhandu_ has quit IRC11:44
*** rascasoft has quit IRC11:50
*** janki has quit IRC11:51
*** rascasoft has joined #openstack-infra11:53
*** janki has joined #openstack-infra11:53
ianwroot@debian-stretch-arm64-linaro-london-0010050906:~# cat /proc/cpuinfo  | grep processor | wc -l11:53
ianw1611:53
ianwhrw: ^ looks like we're good ... just took a while to get going11:53
hrwianw: lovely!11:54
hrwrestarted 662456 then11:54
ianwarxcruz: i feel like that may be something pabelanger has been looking into?11:57
arxcruzpabelanger: ^ :D11:59
*** rlandy has joined #openstack-infra12:00
*** rlandy is now known as rlandy|rover12:00
*** ociuhandu has joined #openstack-infra12:03
*** jaosorior has joined #openstack-infra12:03
*** kjackal has quit IRC12:03
*** rfolco has joined #openstack-infra12:03
*** kjackal has joined #openstack-infra12:05
*** rosmaita has left #openstack-infra12:05
*** rh-jelabarre has joined #openstack-infra12:08
*** ociuhandu has quit IRC12:08
*** aprice has quit IRC12:14
*** vdrok has quit IRC12:14
*** aprice has joined #openstack-infra12:14
*** vdrok has joined #openstack-infra12:14
*** bobh has joined #openstack-infra12:15
*** yamamoto has quit IRC12:15
*** yamamoto has joined #openstack-infra12:15
*** ykarel is now known as ykarel|afk12:16
*** iurygregory has quit IRC12:16
*** rosmaita has joined #openstack-infra12:16
*** priteau has joined #openstack-infra12:20
*** bobh has quit IRC12:20
*** ykarel|afk has quit IRC12:28
*** markvoelker has joined #openstack-infra12:28
*** jtomasek_ has joined #openstack-infra12:28
*** jtomasek has quit IRC12:30
*** jpena|lunch is now known as jpena12:30
*** bobh has joined #openstack-infra12:32
*** bobh has quit IRC12:34
*** rascasoft has quit IRC12:35
*** ociuhandu has joined #openstack-infra12:36
*** electrofelix has quit IRC12:37
*** rascasoft has joined #openstack-infra12:38
*** ociuhandu has quit IRC12:41
*** electrofelix has joined #openstack-infra12:47
*** jchhatbar has joined #openstack-infra12:47
*** jchhatbar has quit IRC12:48
*** ykarel|afk has joined #openstack-infra12:48
*** jchhatbar has joined #openstack-infra12:48
*** janki has quit IRC12:50
*** _Cyclone_ has joined #openstack-infra12:51
*** jchhatbar has quit IRC12:52
*** ykarel|afk is now known as ykarel12:52
slaweqfrickler: infra-root: hi, can You maybe hold for me nodes on which job tobiko-tripleo-containers-update will be run? It is passing for me locally all the times and it's failing in gate always. I really don't know why and I would like to understand it finally. Access to those nodes would be very helpful for me :)12:57
*** hrw has left #openstack-infra12:59
fricklerslaweq: sure, do you have a specific patch to test against or do you want a random instance of that job?13:01
*** jroll has quit IRC13:01
slaweqfrickler: yes, it is new job proposed in https://review.opendev.org/#/c/651828/3113:01
slaweqI just rechecked it to run this new job again13:02
*** jroll has joined #openstack-infra13:02
*** beagles has joined #openstack-infra13:05
*** slaweq has quit IRC13:08
*** janki has joined #openstack-infra13:09
*** pkopec has quit IRC13:10
*** slaweq has joined #openstack-infra13:11
pabelangerianw: arxcruz: what was I looking into?13:11
*** pkopec has joined #openstack-infra13:12
*** mriedem has joined #openstack-infra13:18
*** frickler has quit IRC13:18
*** priteau has quit IRC13:18
*** jcoufal has joined #openstack-infra13:19
*** priteau has joined #openstack-infra13:19
*** ekultails has joined #openstack-infra13:21
*** aaronsheffield has joined #openstack-infra13:22
*** openstackgerrit has joined #openstack-infra13:22
openstackgerritMerged openstack/diskimage-builder master: Fixes DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matching when empty  https://review.opendev.org/67529413:22
*** ginopc has joined #openstack-infra13:29
*** tdasilva has quit IRC13:30
*** udesale has joined #openstack-infra13:30
*** chkumar|ruck is now known as raukadah13:30
*** yamamoto has quit IRC13:31
*** frickler has joined #openstack-infra13:34
openstackgerritRonelle Landy proposed opendev/elastic-recheck master: Add query for "RuntimeError: Failed to retrieve repo file"  https://review.opendev.org/67589813:41
fricklerslaweq: sorry, my bouncer restarted for unknown reasons. I did set up a hold for your job, but a dependency seems to be in merge conflict, no job running currently afaict13:44
slaweqfrickler: I will rebased what is needed and will rerun them soon13:45
*** ociuhandu has joined #openstack-infra13:50
*** odicha has quit IRC13:53
*** beekneemech is now known as bnemec13:54
*** janki has quit IRC13:57
openstackgerritTristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint  https://review.opendev.org/62134414:05
*** yamamoto has joined #openstack-infra14:06
*** udesale has quit IRC14:06
*** tdasilva has joined #openstack-infra14:07
*** udesale has joined #openstack-infra14:08
*** udesale has quit IRC14:08
*** Ng has joined #openstack-infra14:08
*** udesale has joined #openstack-infra14:08
*** ykarel is now known as ykarel|afk14:10
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Remove download_artifact_name parameter  https://review.opendev.org/67332714:11
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Use human-readable names for artifact returns  https://review.opendev.org/67238214:13
*** eharney has joined #openstack-infra14:15
*** dpawlik has quit IRC14:17
*** slaweq has quit IRC14:20
corvuswhen those 2 changes merge, i plan to approve the change which switches the reporting url14:21
*** slaweq has joined #openstack-infra14:21
corvusclarkb: you may be interested in https://review.opendev.org/67571114:24
openstackgerritRonelle Landy proposed opendev/elastic-recheck master: Add query for "RuntimeError: Failed to retrieve repo file"  https://review.opendev.org/67589814:27
openstackgerritJames E. Blair proposed opendev/base-jobs master: Switch to download_artifact_type  https://review.opendev.org/67591214:30
openstackgerritMerged zuul/zuul-jobs master: Remove download_artifact_name parameter  https://review.opendev.org/67332714:30
corvusclarkb, AJaeger: ^ oops, i thought we merged a change for  https://review.opendev.org/675912 already but i guess not14:30
mordredinfra-root: I has internet again! yay!14:31
corvusmordred: yay!  https://review.opendev.org/675912 is the next change for the log url reporting switch :)14:32
mordredcorvus: I justed +2 it14:32
Shrewsmordred: what a wonderful birthday present for you!  happy cake day14:33
fungimordred: island internet is the best internet14:33
fungialso the cake is a lie14:33
AJaegercorvus: +2A14:33
mordredShrews: see followup from fungi on the fallability of cake14:33
mordredand thanks :)14:34
AJaegerhappy birthday, mordred !14:34
mordredAJaeger: thanks!14:34
fungi(also the frogurt is cursed)14:35
fungia fröhlich mordredtag to all14:36
mordredharibo macht mordred fröh14:37
donnydso apparently this weekend my object store servers already blew up.. so the object service is going to be down until I am able to sort it out.14:38
openstackgerritMerged opendev/base-jobs master: Test FN swift in base-test-swift  https://review.opendev.org/67571114:39
donnydI wil ltry to get it back up and running tonight after work14:40
corvusdonnyd: you have excellent timing :)14:40
donnydI know right.. I was thinking to myself.. seriously did that just happen14:40
donnydI was trying to do something fancy for performance reasons.. and lets just say it didn't go well14:41
openstackgerritJames E. Blair proposed opendev/base-jobs master: Revert "Test FN swift in base-test-swift"  https://review.opendev.org/67592014:41
corvusi'll just self-approve that ^14:41
donnydLOL, thanks corvus14:41
donnydsry14:41
fungidonnyd: blew up as in the magic smoke got out?14:43
openstackgerritMerged opendev/base-jobs master: Switch to download_artifact_type  https://review.opendev.org/67591214:43
*** ykarel|afk has quit IRC14:43
*** sthussey has joined #openstack-infra14:43
slaweqfrickler: now this my job is in queue on zuul14:43
*** ykarel|afk has joined #openstack-infra14:43
mordredfungi: I keep spare bags of magic smoke in my travel kit14:43
donnydfungi: no fortunately this time there was no magic smoke14:43
fungiwell, as long as the magic smoke stays in, things should be fine14:43
donnydI was using zfs on root for the swift servers. In performance testing it did extremely well. So when they went OOM and had to be cold booted... zfs on root was quite unhappy... and then kernel paniced when they tried to boot14:44
donnydWoops14:44
donnydTrying to eke every last little bit of performance out of these things is interesting... and is a solid learning experience...  but sometimes things don't work as expected14:46
donnydI am pretty sure the defaults in ZoL are to use 50% of system memory for anything over 4GB... so they shouldn't have went OOM... but they did14:46
Shrewscorvus: i'm noticing that swift objects can stick around even after delete for a loooong time. We have objects in rax iad over a year old that we cannot delete from the container. Will be interested to see if the same happens with logs14:47
mordredalso will be interested to see how aging things out works vs. explicitly deleting14:48
Shrewsthose we cannot delete report 404 if you try to delete them, so they appear to be gone yet still present. object ghosts i guess14:48
corvusShrews: "neat"14:48
donnydI also know very little about swift, am used to using ceph to do these types of things14:48
donnydceph just works for the most part and I have really never had any issues with it. Only once when cephfs was young and lost all my data....14:49
donnydBut as I understand we need swift swift, not ceph swift14:49
*** yamamoto has quit IRC14:49
corvusdonnyd: new info there: it's looking like the radosgw may be sufficient for our needs after all14:49
*** yamamoto has joined #openstack-infra14:49
corvusi don't really understand it, because observed behavior seems to contradict the documentation14:49
donnydOh that would be really fantastic14:49
corvusbut maybe i just don't understand the documentation14:49
donnydCeph has been on these exact servers for several years without issues (other than CI performance).. so Switching them back can happen in a flash14:50
corvusdonnyd: at least, whatever vexxhost is doing seems like it will work for us.  so if you wanted to try it, i think it might be worth it.  but since we still don't fully understand it, i guess there's a small risk we may find that something is missing and we have to spend time hunting down what it is.14:51
corvus(to be less vague, this is around the behavior of the access-control-allow-origin header)14:51
fungiwell, also i wouldn't pin stuck/failing deletes in rackspace on swift itself... who knows what vintage and additional patches they could be running with14:51
tdasilvaShrews: you should contact RAX about 'ghosts objects' (AKA: dark data). Deleted objects should be deleted...14:51
fungii hear for many people swift also "just works" ;)14:52
tdasilvadonnyd: feel free to join #openstack-swift with any swift questions, folks there would be happy to help14:52
donnydOh it was well below the swift layer... swift works just fine14:52
donnydIt was a zfs on root issue14:53
openstackgerritMerged opendev/base-jobs master: Revert "Test FN swift in base-test-swift"  https://review.opendev.org/67592014:53
donnydOk tdasilva, I will surely check it out14:54
donnydI do have a few questions that are swift appropriate14:55
*** ginopc has quit IRC14:55
*** ykarel|afk is now known as ykarel|away14:57
fungiShrews: oh, i ended up booking my shanghai flights through rdu so will be spending halloween and november 9 in durham if you want to try to meet up somewhere for dinner/drinks one of those nights14:59
Shrewsfungi: absolutely. i'm sure olaph would be down too14:59
fungihe may have plans for the first of those nights, but definitely!15:00
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Collect container logs from test-registry job  https://review.opendev.org/67592615:00
*** josephrsandoval has joined #openstack-infra15:01
*** mattw4 has joined #openstack-infra15:01
fungistrangely it was some us$400 cheaper to book through rdu, spend two nights in a hotel and do two one-way car rentals than it was to fly in and out of norfolk15:01
donnydfungi: Oh I am sure swift just works fine for everyone that doesn't run their system OOM with ZFS15:02
donnydAnd my issue wasn't swift related at all.15:02
fungisee, i *knew* zfs was good for something! ;)15:02
donnydMaybe this next time around I will just do things the same way normal people would and not worry about performance as much15:03
donnydzfs is great when you need to cache writes on a faster device because your spinny rust is too slow for the workload15:03
fungii know for a fact swift tests every change on xfs in memory-constrained environments15:03
donnydI probably should just run it how the docs say to.. and I am likely to have a lot less in the way of issues15:04
*** ykarel|away has quit IRC15:04
donnydjust trying to get the best possible performance that I can on the gear I have..15:04
*** ykarel|away has joined #openstack-infra15:04
donnydMore fun an interesting things are happening for FN... mnaser recommenced I give local storage a try15:05
*** armax has joined #openstack-infra15:05
*** mattw4 has quit IRC15:06
donnydSo i got 2 nvme drives for each hypervisor and then figured out a way to get the blades to recognize them... apparently some came from the factory in my model of blade that would support nvme drives15:06
donnydso I have all the parts and just need to find time to get it togehter15:06
openstackgerritMerged zuul/zuul-jobs master: Use human-readable names for artifact returns  https://review.opendev.org/67238215:07
* fungi climbs onto next rocketbus and prepares for liftoff15:08
*** ociuhandu has quit IRC15:08
*** ociuhandu has joined #openstack-infra15:10
donnydalso most the images loaded on FN  had made their way to swift.. so waiting on the next image run from the cron job atm15:10
*** cjloader has joined #openstack-infra15:13
*** ociuhandu has quit IRC15:15
slaweqfrickler: now this my job started to run15:17
slaweqfrickler: can You maybe hold those nodes for me?15:17
slaweqor anyone else from infra-root ^^ ?15:17
slaweqI'm talking about tobiko-tripleo-containers-update job :)15:18
mordredslaweq: just a sec ...15:20
*** ociuhandu has joined #openstack-infra15:20
mordredslaweq: I believe frickler already put on the autohold15:20
slaweqmordred: thx a lot for checking that15:20
mordredslaweq: openstack | opendev.org/x/tobiko | tobiko-tripleo-containers-update | refs/changes/28/651828/.* |   1   | frickler/slaweq debugging15:21
*** dave-mccowan has joined #openstack-infra15:21
slaweqyes, this is it15:21
slaweqmordred: frickler: thx for both of You :)15:21
mordred\o/15:22
*** slaweq has quit IRC15:28
*** slaweq has joined #openstack-infra15:30
openstackgerritMerged openstack/project-config master: Report the build page url in Zuul  https://review.opendev.org/67564115:34
corvusthat's going to need an ansible-puppet run to be put in place15:34
*** josephrsandoval has quit IRC15:35
corvusso probably a half hour or so15:35
*** bdodd has joined #openstack-infra15:39
*** diga has quit IRC15:40
*** priteau has quit IRC15:45
*** yamamoto has quit IRC15:47
*** factor has joined #openstack-infra15:48
*** yamamoto has joined #openstack-infra15:49
*** AJaeger is now known as AJaeger_15:51
*** jpena is now known as jpena|off15:53
*** gyee has joined #openstack-infra15:54
*** udesale has quit IRC15:56
*** dave-mccowan has quit IRC15:56
*** pkopec has quit IRC16:00
*** e0ne has quit IRC16:01
*** lucasagomes has quit IRC16:03
*** ykarel|away has quit IRC16:08
*** bhavikdbavishi has quit IRC16:08
*** tdasilva has quit IRC16:09
*** Goneri has quit IRC16:10
*** tdasilva has joined #openstack-infra16:10
*** bhavikdbavishi has joined #openstack-infra16:10
*** rpittau is now known as rpittau|afk16:15
*** kjackal has quit IRC16:27
*** yamamoto has quit IRC16:28
*** ijw has joined #openstack-infra16:28
*** yamamoto has joined #openstack-infra16:28
*** ijw_ has joined #openstack-infra16:29
*** rascasoft has quit IRC16:29
*** ijw has quit IRC16:30
*** ijw_ has joined #openstack-infra16:30
*** rascasoft has joined #openstack-infra16:31
*** ijw_ has quit IRC16:32
*** dklyle_ is now known as dklyle16:34
*** ijw has joined #openstack-infra16:34
*** yamamoto has quit IRC16:34
corvuslooks like the zuul report url change is in place.  example: https://review.opendev.org/#/c/675923/16:34
*** ijw has quit IRC16:34
*** ijw has joined #openstack-infra16:35
*** ijw has quit IRC16:35
clarkbSeems to work for me16:36
clarkbI guess the thing to keep an eye on is the zuul-web process keeping up with the sql lookups?16:36
*** ijw has joined #openstack-infra16:36
openstackgerritJames E. Blair proposed zuul/zuul master: Revert "Try out reporting the build page"  https://review.opendev.org/67594216:37
*** dtantsur is now known as dtantsur|afk16:37
mordredcorvus: \o/16:37
*** ijw has quit IRC16:38
*** ijw has joined #openstack-infra16:39
*** ijw has quit IRC16:40
*** ijw has joined #openstack-infra16:40
*** pkopec has joined #openstack-infra16:41
*** Vadmacs has joined #openstack-infra16:45
*** gfidente has quit IRC16:50
raukadahclarkb: mriedem can we get this review merged https://review.opendev.org/#/c/675898/ we are hitting it multiple time, thanks!16:54
openstackgerritMerged zuul/zuul-jobs master: Collect container logs from test-registry job  https://review.opendev.org/67592616:58
*** derekh has quit IRC17:01
mriedemlooking17:02
openstackgerritJames E. Blair proposed zuul/zuul master: Cap GitPython <3.0.0  https://review.opendev.org/67599117:04
openstackgerritJames E. Blair proposed zuul/zuul master: Add release note for Pagure driver  https://review.opendev.org/67541017:04
openstackgerritJames E. Blair proposed zuul/zuul master: Move admin-rules setting in tenants doc  https://review.opendev.org/67541117:05
openstackgerritJames E. Blair proposed zuul/zuul master: Make auth docs more boring  https://review.opendev.org/67541217:05
openstackgerritJames E. Blair proposed zuul/zuul master: Don't capitalize Token in docs  https://review.opendev.org/67541317:05
openstackgerritBen Nemec proposed openstack/devstack-gate master: WIP: Use OSCaaS to speed up devstack runs  https://review.opendev.org/67541417:07
openstackgerritMerged openstack/pbr master: Use more verbose logging for test failure  https://review.opendev.org/66033917:07
*** ociuhandu_ has joined #openstack-infra17:09
*** pkopec is now known as pkopec|afk17:09
*** mattw4 has joined #openstack-infra17:10
*** yamamoto has joined #openstack-infra17:12
*** weifan has joined #openstack-infra17:12
*** ociuhandu has quit IRC17:13
openstackgerritMerged opendev/elastic-recheck master: Add query for "RuntimeError: Failed to retrieve repo file"  https://review.opendev.org/67589817:14
*** ociuhandu_ has quit IRC17:14
raukadahrlandy|rover: ^617:16
rlandy|roverraukadah: woohoo - our gate troubles are getting some air time17:17
*** weifan has quit IRC17:17
clarkbI noticed the logstash gearman queue was growing. Lookingat elasticsearch cluster Some nodes weren't seeing es07 so I have restarted the es server on those host17:18
clarkbseems the hosts that weren't seeing 07 do see it now17:19
clarkband es07 had all stale indexes so I've run the index cleanup cron manually to get ahead of that and force rebalancing to happen17:20
clarkbI expect it will be much happier once es07 is sharing the load again17:20
*** weifan has joined #openstack-infra17:20
corvusi wonder how much of this is automated by the es k8s operotar17:21
*** electrofelix has quit IRC17:21
clarkbMy guess is the delete data part isn't (because you want oversight of that tpyically). But it should be able to restart any nodes that drop out of the cluster api status reports17:21
*** ramishra_ has quit IRC17:26
*** ociuhandu has joined #openstack-infra17:28
*** ociuhandu has quit IRC17:32
*** diablo_rojo has joined #openstack-infra17:36
*** whoami-rajat has quit IRC17:38
*** gtmanfred has left #openstack-infra17:42
*** kjackal has joined #openstack-infra17:44
slaweqmordred: ping, if You can, please add my ssh key to nodes from https://zuul.opendev.org/t/openstack/build/4f945e05817741c998a5924eeb02760917:46
slaweqmordred: my ssh key is http://paste.openstack.org/show/756355/17:47
slaweqmordred: thx in advance17:47
mordredclarkb: I can't log in to the held nodes (ssh connection is hanging - did we change something?) 10.209.97.96 10.209.98.79 ^^17:49
corvusmordred: those are rfc1918 addresses17:50
mordredoh. HAHAHAHAH17:51
* mordred goes back to bed17:51
mordredhow about I try the non rtc1918 addresses17:52
clarkbgotta use the free range IPs17:52
*** tdasilva has quit IRC17:53
mordredslaweq: root@23.253.56.191 root@104.130.74.5817:53
*** jamesmcarthur has joined #openstack-infra17:53
*** tdasilva has joined #openstack-infra17:53
mordredclarkb: you portlanders with your organic and your free range - I want factory IPs - they're cheaper17:53
corvusin case anyone has tried to visit https://zuul.opendev.org/t/openstack/build/4f945e05817741c998a5924eeb027609/logs and wondered why it's a bit slow17:55
mordredcorvus: fwiw, the console log from slaweq above is another that exhibits the "chrome stops rendering part of the way down" bug - I'm still not seeing any errors to go along with it17:55
corvusit's because that job uploaded 55705 files to the log server17:55
mordredcorvus: oh - you might have more infors17:55
mordredOH WOW17:55
mordredthat's a lot of logs17:56
corvusmordred: i doubt that's the cause of the console log issue17:56
mordredyah17:56
mordredme either17:56
mordredmy browser failed at loading teh logs tab17:56
corvusslaweq: do you need all 55,705 of those? :)17:56
*** kjackal has quit IRC17:56
corvusi'm wondering if gcp_compute_backend_service_facts.pyc.gz is strictly necessary17:57
mordredcorvus: I think I'd fall on the side of probably not17:59
corvusslaweq: it looks like 50490 of those are under ./logs/undercloud/home/zuul/tobiko/.tox/17:59
corvuslike ./logs/undercloud/home/zuul/tobiko/.tox/tox/include/python2.7/pymacconfig.h.gz17:59
corvusslaweq: so maybe you could just stop archiving the .tox directory?18:00
mordred++18:00
corvusthat'd get us down to a mere 5700 log files :)18:01
mordredthis 5700 log files is cut in half ...18:01
*** sgw has quit IRC18:03
clarkbfwiw the logstash process log files ballooned to many GB in size filling some of the worker disks. This appears to be a side effect of elasticsearch being sad (they are logged errors about things not indexing)18:03
clarkbI'm keeping the last 1000 lines of logfile and deleting the rest as well as restarting logstash to make them happy too18:04
*** ociuhandu has joined #openstack-infra18:04
*** yamamoto has quit IRC18:06
slaweqcorvus: nope, those files are not necessary, I'm still trying to make this job working and I will remove upliad of this directory18:06
corvusslaweq: \o/18:07
slaweqmordred: thx a lot for hold nodes18:07
mordredsure nuff!18:07
slaweqI will investigate it tonight and will let You know I will be done with it18:07
*** rkukura has joined #openstack-infra18:10
*** ijw_ has joined #openstack-infra18:11
*** markvoelker has quit IRC18:13
*** ijw has quit IRC18:15
*** ijw_ has quit IRC18:15
*** markvoelker has joined #openstack-infra18:20
clarkbdoes anyone else's grafana dashboard lack a scrollbar on the right?18:20
clarkbI have to click into the window and use arrow keys :/18:20
*** ijw has joined #openstack-infra18:24
*** ijw has quit IRC18:25
fungimine has a scrollback but it's very thin and nonobvious18:26
fungier, scrollbar18:27
*** ijw has joined #openstack-infra18:27
*** ijw has quit IRC18:28
*** ijw has joined #openstack-infra18:29
*** jamesmcarthur has quit IRC18:30
*** ijw_ has joined #openstack-infra18:30
*** jamesmcarthur has joined #openstack-infra18:32
clarkboh hrm ya its a color pallete problem18:32
clarkbthat and my eyeballs don't work so great18:32
*** ijw has quit IRC18:33
*** ijw_ has quit IRC18:34
*** ijw has joined #openstack-infra18:34
*** jamesmcarthur has quit IRC18:34
*** jamesmcarthur has joined #openstack-infra18:34
mnaserinfra-root: https://review.opendev.org/#/c/669298/ is this supposed to go in some sort of queue?18:35
*** dpawlik has joined #openstack-infra18:36
clarkbmnaser: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames18:36
mnaseroh look like that's on the list already18:36
clarkbwe tend to wait for a few to collect there (I honestly expected far more post opendev'ing) then we schedule the gerrit outage and move things around18:37
mnasercool, its just so that we land https://review.opendev.org/#/c/669306/ at some point18:37
clarkbmaybe the expectation for more renames was wrong and we should move forwardwith the list we have18:37
fungithat seems reasonable to me18:38
*** markvoelker has quit IRC18:39
*** markvoelker has joined #openstack-infra18:40
*** weifan has quit IRC18:52
*** ociuhandu has quit IRC18:54
*** kopecmartin is now known as kopecmartin|off18:55
*** ociuhandu has joined #openstack-infra18:55
*** ociuhandu has quit IRC19:00
*** pkopec_ has joined #openstack-infra19:03
*** pkopec|afk has quit IRC19:05
*** jamesmcarthur has quit IRC19:08
*** ociuhandu has joined #openstack-infra19:12
*** eharney has quit IRC19:14
*** factor has quit IRC19:17
*** ociuhandu has quit IRC19:25
*** diablo_rojo has quit IRC19:25
*** ociuhandu has joined #openstack-infra19:26
*** igordc has joined #openstack-infra19:26
*** ociuhandu has quit IRC19:32
*** tesseract has quit IRC19:32
*** psachin has quit IRC19:37
*** igordc has quit IRC19:41
*** ijw has quit IRC19:44
*** bhavikdbavishi has quit IRC19:49
*** dave-mccowan has joined #openstack-infra19:54
*** pkopec_ has quit IRC19:54
*** Vadmacs has quit IRC19:55
corvusaside from the "slow to render 57,000 log files" bug, i haven't seen or heard of any issues with the new log url reporting; has anyone else?20:00
*** eharney has joined #openstack-infra20:01
*** yamamoto has joined #openstack-infra20:04
*** jcoufal has quit IRC20:04
*** ccamacho has quit IRC20:05
clarkbI have not20:08
*** igordc has joined #openstack-infra20:11
*** sthussey has quit IRC20:12
*** weifan has joined #openstack-infra20:14
slaweqmordred: ping, You can destroy nodes which You hold for me: root@23.253.56.191 root@104.130.74.58 - thx a lot20:16
slaweqinfra-root ^^ maybe someone else can clean them if mordred is busy :)20:17
slaweqthx a lot for all Your help. I hope that it will now work better20:17
slaweqI think I found an issue in my job20:17
*** weifan has quit IRC20:19
*** e0ne has joined #openstack-infra20:26
*** jtomasek_ has quit IRC20:26
*** kjackal has joined #openstack-infra20:35
openstackgerritMerged zuul/zuul master: Cap GitPython <3.0.0  https://review.opendev.org/67599120:36
*** tdasilva has quit IRC20:37
*** ijw has joined #openstack-infra20:40
*** dpawlik has quit IRC20:42
*** guoqiao has joined #openstack-infra20:54
*** pcaruana has quit IRC20:56
*** mattw4 has quit IRC21:03
*** mattw4 has joined #openstack-infra21:04
*** guoqiao has quit IRC21:24
*** markvoelker has quit IRC21:24
*** tdasilva has joined #openstack-infra21:25
*** tdasilva has quit IRC21:25
*** mattw4 has quit IRC21:31
*** mattw4 has joined #openstack-infra21:32
*** weifan has joined #openstack-infra21:32
*** ociuhandu has joined #openstack-infra21:36
*** mattw4 has quit IRC21:38
*** mattw4 has joined #openstack-infra21:39
*** ociuhandu has quit IRC21:42
*** slaweq has quit IRC21:46
*** iokiwi has joined #openstack-infra21:48
*** e0ne has quit IRC21:49
*** dchen has quit IRC21:51
*** tosky has quit IRC21:55
*** guoqiao has joined #openstack-infra21:59
*** diablo_rojo has joined #openstack-infra22:03
*** yamamoto has quit IRC22:03
*** mriedem is now known as mriedem_afk22:03
*** markvoelker has joined #openstack-infra22:03
clarkbI have deleted the nodes slaweq was using22:07
clarkb(well asked nodepool to delete them anyway)22:07
*** lbragstad has joined #openstack-infra22:09
*** lbragstad has quit IRC22:17
openstackgerritMerged zuul/zuul master: Add release note for Pagure driver  https://review.opendev.org/67541022:26
*** ociuhandu has joined #openstack-infra22:30
*** diablo_rojo has quit IRC22:32
openstackgerritMerged zuul/zuul master: Move admin-rules setting in tenants doc  https://review.opendev.org/67541122:34
*** ociuhandu has quit IRC22:35
*** markvoelker has quit IRC22:35
*** ijw has quit IRC22:35
*** ijw has joined #openstack-infra22:36
*** ekultails has quit IRC22:49
openstackgerritMerged zuul/zuul master: Make auth docs more boring  https://review.opendev.org/67541222:58
*** weifan has quit IRC22:59
*** mattw4 has quit IRC23:00
*** mattw4 has joined #openstack-infra23:00
*** markvoelker has joined #openstack-infra23:00
*** markvoelker has quit IRC23:05
*** weifan has joined #openstack-infra23:10
*** rcernin has joined #openstack-infra23:11
*** aaronsheffield has quit IRC23:11
*** rlandy|rover is now known as rlandy|rover|bbl23:12
*** weifan has quit IRC23:14
*** weifan has joined #openstack-infra23:14
*** dave-mccowan has quit IRC23:17
*** markvoelker has joined #openstack-infra23:18
*** mriedem_afk has quit IRC23:19
*** weifan has quit IRC23:19
*** weifan has joined #openstack-infra23:24
*** kjackal has quit IRC23:28
openstackgerritMerged zuul/zuul master: Revert "Try out reporting the build page"  https://review.opendev.org/67594223:31
*** eernst has joined #openstack-infra23:32
*** weifan has quit IRC23:32
*** weifan has joined #openstack-infra23:33
*** weifan has quit IRC23:35
*** bdodd has quit IRC23:35
*** eernst has quit IRC23:35
*** dave-mccowan has joined #openstack-infra23:37
openstackgerritMerged opendev/system-config master: backup: minor fixes  https://review.opendev.org/67551723:43
*** dchen has joined #openstack-infra23:47
*** mattw4 has quit IRC23:48
*** dchen has quit IRC23:49
*** dchen has joined #openstack-infra23:49
*** diablo_rojo_phon has joined #openstack-infra23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!