Tuesday, 2020-02-18

ianwdoes look blank00:00
clarkbya I think I may have gotten the provider name wrong. double checking against graphite.o.o now00:02
*** lennyb has quit IRC00:02
*** lennyb has joined #openstack-infra00:04
openstackgerritClark Boylan proposed openstack/project-config master: Fix airship citycloud grafana query  https://review.opendev.org/70827800:04
clarkbthat should fix it I think00:04
clarkbnow to debug why that fix to gear is unhappy00:07
*** dychen has quit IRC00:07
*** dychen has joined #openstack-infra00:08
*** ahosam has quit IRC00:11
*** slaweq has joined #openstack-infra00:11
ianwthe tension between grafyaml and nice code config and grafana's very polished UI for generating dashboards is quite large00:11
ianwhttps://github.com/weaveworks/grafanalib00:13
ianwinteresting now i look00:13
*** dingyichen has joined #openstack-infra00:13
*** slaweq has quit IRC00:16
*** dychen has quit IRC00:17
ianwoh, i've forgotten the acme-challenge entry in openstack.org for tarballs, updating00:37
*** tetsuro has joined #openstack-infra00:49
ianwi have rsynced the openstack tarballs volume to afs.  since the jobs should be publishing in both places, they should now remain in sync00:50
clarkbexciting00:51
openstackgerritClark Boylan proposed opendev/gear master: Revert "Revert "Add BSD/Darwin support.""  https://review.opendev.org/70826700:53
clarkbI think ^ should result in happy gear with epoll or poll00:53
*** tetsuro has quit IRC00:57
*** dchen has quit IRC01:06
*** rlandy has quit IRC01:10
*** slaweq has joined #openstack-infra01:11
*** slaweq has quit IRC01:17
*** Goneri has quit IRC01:25
*** dave-mccowan has quit IRC01:32
*** rkukura has quit IRC01:33
*** rkukura has joined #openstack-infra01:35
ianwok, if anyone wants to override /etc/hosts with "23.253.245.150 tarballs.openstack.org tarballs.opendev.org" that should be working01:39
*** rkukura has quit IRC01:41
clarkbI can check after dinner01:42
*** armax has quit IRC01:43
*** tetsuro has joined #openstack-infra01:44
*** larainema has joined #openstack-infra01:44
*** tetsuro has quit IRC01:45
*** tetsuro has joined #openstack-infra01:48
*** rkukura has joined #openstack-infra01:51
*** tetsuro has quit IRC01:52
*** tetsuro has joined #openstack-infra01:54
kevinzianw: Morning!01:58
*** Lucas_Gray has joined #openstack-infra01:59
kevinzianw: I wonder how could I trigger a os-jobs in linaro-us? I submitted a patch here: https://review.opendev.org/#/c/708102/8,but it looks that it comes to  linaro-london02:01
*** smarcet has joined #openstack-infra02:03
*** tetsuro has quit IRC02:03
ianwkevinz: hrm, we can't really pick a region, although if required we could turn off linaro-london and that would force it to use US02:04
openstackgerritMerged opendev/system-config master: Add project.tarballs to periodic AFS release  https://review.opendev.org/70673202:05
kevinzianw: could we reduce the number to 4 at linaro-london, to force the newly creation jobs to linaro-us?02:06
ianwkevinz: the switch to toggle would be @ https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L39102:07
kevinzsure, I will submit a change02:08
*** hongbin has joined #openstack-infra02:09
clarkbianw: that /etc/hosts entry seems to work. tarballs.openstack.org redirects to tarballs.opendev.org/openstack and tarballs.opendev.org/ root is browseable and LE cert is in place02:10
clarkband the redirect seems to work over https02:10
ianwclarkb: thanks for confirming02:10
*** slaweq has joined #openstack-infra02:11
*** ociuhandu has joined #openstack-infra02:12
*** ociuhandu has quit IRC02:16
*** slaweq has quit IRC02:16
*** cdearborn has quit IRC02:17
*** igordc has quit IRC02:17
*** Lucas_Gray has quit IRC02:25
openstackgerritKevin Zhao proposed openstack/project-config master: reduce machine number from Linaro London  https://review.opendev.org/70828802:30
*** rfolco has quit IRC02:33
*** rfolco has joined #openstack-infra02:33
*** rfolco has quit IRC02:38
openstackgerritMerged openstack/project-config master: reduce machine number from Linaro London  https://review.opendev.org/70828802:47
*** jamesmcarthur has joined #openstack-infra02:55
*** ramishra has joined #openstack-infra03:08
*** slaweq has joined #openstack-infra03:11
*** slaweq has quit IRC03:16
*** ijw_ has quit IRC03:22
kevinzianw: it looks that the test nodes are not scheduled to linaro-us http://zuul.openstack.org/status03:22
*** smarcet has quit IRC03:22
*** yamamoto has joined #openstack-infra03:23
kevinzthey just waiting in queue..03:23
ianwkevinz: hrm, ok let me check the launcher03:24
kevinzianw: thx03:25
*** ijw has joined #openstack-infra03:25
*** ijw has quit IRC03:29
*** psachin has joined #openstack-infra03:41
ianwkevinz: Exception: Unable to find flavor with min ram: 8000 ... maybe03:47
kevinzianw: aha,,,I will manually create it. thx03:47
ianwthere is m1.xlarge?03:48
ianwwhat are we telling it to use?03:48
ianwahhh ... os.large03:48
ianwreplicate that flavor in the us ci region?03:49
ianwdidn't think of that03:49
kevinz7ce03675-5d14-4b1b-85a4-e641d21320db | m1.xlarge | 8192 |   80 |         0 |     8 | True03:50
kevinzyes, in us cloud, we have m1.xlarge, suffcient03:50
kevinzso, I will submit a change to replace it.03:51
ianwwe do set the flavor name to "os.large" https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L39503:51
ianwistr something to do with this, is os.large 16 cpus maybe?03:51
kevinzOK, I create a same flavor os.large in linaro-us, that don't need any change in code03:52
ianw| 69af1b47-5557-42fc-bdcb-9cf48dd5e607 | os.large             |  8200 |   80 |        80 |     8 | False     |03:52
kevinzyes, 8C8G 80GB03:52
ianwm1.xlarge looks similar, execpt for 0 ephemeral storage as opposed to 80g03:53
kevinzyes03:53
ianwi think for general sanity copying os.large is best idea so it's 100% consistent03:53
kevinz312093a9-e19b-45a3-b194-11decb37f05d | os.large  | 8200 |   80 |        80 |     8 | True      |03:56
kevinznew one created in linaro-us03:56
ianwcool ... *possibly* nodepool-launcher needs a restart or we need to wait a bit for cache to time out and it to notice it03:58
ianwkevinz: maybe try submitting a new job?04:05
kevinzianw: I will start a new one :D04:05
ianwException deleting instance f15b6a42-ac33-462c-9f6e-56dba3724792 from vexxhost-sjc1 ... this seem to be coming up a lot too04:06
*** udesale has joined #openstack-infra04:10
*** slaweq has joined #openstack-infra04:11
*** jamesmcarthur has quit IRC04:14
*** slaweq has quit IRC04:16
*** jamesmcarthur has joined #openstack-infra04:19
kevinzianw: submit a new one, now 6 jobs in queue04:20
*** yamamoto has quit IRC04:29
*** tetsuro has joined #openstack-infra04:32
*** yamamoto has joined #openstack-infra04:35
*** redrobot has quit IRC04:43
*** imacdonn has quit IRC04:43
*** imacdonn has joined #openstack-infra04:44
ianwhrm04:53
ianw2020-02-18 04:52:05,424 ERROR nodepool.driver.NodeRequestHandler[nl03-11177-PoolWorker.linaro-us-main]: Declining node request 100-0007475839 due to exception in NodeRequestHandler:04:55
ianwTue Feb 18 04:55:4204:55
ianwi think maybe a restart before we start investigating other things04:56
ianwyeah, it seems like it's building like 5+ nodes now04:58
ianwkevinz: also; check http://grafana.openstack.org/d/vcvSN5Siz/nodepool-linaro?orgId=1 if you didn't know about it :)05:00
*** ramishra has quit IRC05:01
kevinzGreat! Thanks! I saw them running now05:02
kevinzianw: restarting is always a good way :D05:02
ianwyep, turn it on and off again :)05:03
*** ramishra has joined #openstack-infra05:08
*** hongbin has quit IRC05:10
*** slaweq has joined #openstack-infra05:11
*** slaweq has quit IRC05:16
*** ykarel|away is now known as ykarel05:16
*** dave-mccowan has joined #openstack-infra05:17
*** yamamoto has quit IRC05:20
*** jamesmcarthur has quit IRC05:21
*** yamamoto has joined #openstack-infra05:22
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-infra05:34
*** dancek has quit IRC05:37
ianw#status log added 100gb to yum-puppetlabs mirror to stop it hitting afs quota05:37
openstackstatusianw: finished logging05:37
ianwalthough, i'd say nobody is using it because it's been broken for quite a while05:37
*** dancek has joined #openstack-infra05:38
*** dancek has quit IRC05:46
*** ykarel is now known as ykarel|mtg05:47
*** dancek has joined #openstack-infra05:49
*** udesale_ has joined #openstack-infra05:51
*** udesale has quit IRC05:52
openstackgerritIan Wienand proposed openstack/project-config master: Update AFS granfana page  https://review.opendev.org/70830205:52
openstackgerritMerged openstack/project-config master: Fix airship citycloud grafana query  https://review.opendev.org/70827805:59
openstackgerritMerged zuul/zuul master: Cleanup handlers defined in OS_LOG_DEFAULTS  https://review.opendev.org/70794206:05
*** lmiccini has joined #openstack-infra06:06
*** slaweq has joined #openstack-infra06:11
*** slaweq has quit IRC06:16
openstackgerritMerged zuul/zuul master: Don't set OS_LOG_DEFAULTS if unset  https://review.opendev.org/70778306:21
*** redrobot has joined #openstack-infra06:27
*** dayou has quit IRC06:34
*** dayou has joined #openstack-infra06:34
ianwkevinz: do you want to revert the london change back?06:49
*** yamamoto has quit IRC06:51
openstackgerritMerged openstack/project-config master: Update AFS granfana page  https://review.opendev.org/70830207:00
*** xek has joined #openstack-infra07:08
*** ykarel|mtg is now known as ykarel07:10
*** slaweq has joined #openstack-infra07:11
*** dklyle has quit IRC07:15
*** dklyle has joined #openstack-infra07:16
*** slaweq has quit IRC07:16
*** yamamoto has joined #openstack-infra07:16
*** lpetrut has joined #openstack-infra07:20
*** pgaxatte has joined #openstack-infra07:23
openstackgerritMerged zuul/zuul master: cleanup: install docker-compose via role  https://review.opendev.org/70790307:24
kevinzianw: I think we can temperily keep this change, since linaro London don't have good performance07:32
*** ccamacho has quit IRC07:34
*** ccamacho has joined #openstack-infra07:34
*** yamamoto has quit IRC07:34
ianwsounds good07:34
*** yamamoto has joined #openstack-infra07:34
*** dpawlik has joined #openstack-infra07:38
*** Lucas_Gray has joined #openstack-infra07:40
*** rpittau|afk is now known as rpittau07:58
*** slaweq has joined #openstack-infra07:59
*** tesseract has joined #openstack-infra08:02
*** arif-ali has quit IRC08:06
*** arif-ali has joined #openstack-infra08:08
*** dingyichen has quit IRC08:15
*** roman_g has quit IRC08:18
*** tkajinam has quit IRC08:31
*** tosky has joined #openstack-infra08:32
*** amoralej|off is now known as amoralej08:32
*** ralonsoh has joined #openstack-infra08:35
*** gfidente has joined #openstack-infra08:36
*** pkopec has joined #openstack-infra08:45
*** jpena|off is now known as jpena08:52
*** priteau has joined #openstack-infra08:58
*** lucasagomes has joined #openstack-infra09:08
*** rcernin has quit IRC09:12
*** ramishra has quit IRC09:14
*** ramishra has joined #openstack-infra09:16
*** derekh has joined #openstack-infra09:25
openstackgerritSorin Sbarnea proposed opendev/system-config master: Improve 3rd-party logging guidelines  https://review.opendev.org/70832309:28
openstackgerritSorin Sbarnea proposed opendev/system-config master: Improve 3rd-party logging guidelines  https://review.opendev.org/70832309:29
*** ccamacho has quit IRC09:34
*** Lucas_Gray has quit IRC09:35
*** Lucas_Gray has joined #openstack-infra09:41
*** sshnaidm|afk is now known as sshnaidm09:59
*** Wryhder has joined #openstack-infra10:01
*** Lucas_Gray has quit IRC10:01
*** Wryhder is now known as Lucas_Gray10:01
*** migawa is now known as migawa|AFK10:04
*** dtantsur|afk is now known as dtantsur10:05
openstackgerritMattias Jernberg proposed opendev/git-review master: Allow the default of notopic to be configurable  https://review.opendev.org/69744810:10
openstackgerritIlya Etingof proposed openstack/project-config master: Add publish-to-pypi template for sushy-cli  https://review.opendev.org/70832910:19
openstackgerritMerged zuul/zuul master: Fix github app authentication to work with checks API endpoints (2)  https://review.opendev.org/70810910:20
dtantsurhi folks! could you check https://review.opendev.org/#/c/708329/ please?10:20
dtantsurwe missed it when creating sushy-cli and now cannot release10:20
*** fdegir has quit IRC10:34
*** fdegir has joined #openstack-infra10:35
*** Lucas_Gray has quit IRC10:51
*** Lucas_Gray has joined #openstack-infra10:56
*** udesale_ has quit IRC10:57
*** Lucas_Gray has quit IRC10:58
*** petevg has quit IRC11:08
*** ykarel is now known as ykarel|afk11:09
*** Lucas_Gray has joined #openstack-infra11:20
*** Lucas_Gray has quit IRC11:20
*** adriancz has joined #openstack-infra11:22
*** Lucas_Gray has joined #openstack-infra11:24
*** ykarel|afk is now known as ykarel11:26
*** Lucas_Gray has quit IRC11:33
*** Lucas_Gray has joined #openstack-infra11:34
*** Lucas_Gray has quit IRC11:41
*** andrewbonney has joined #openstack-infra11:45
*** Lucas_Gray has joined #openstack-infra11:51
*** migawa|AFK is now known as migawa11:54
*** rfolco has joined #openstack-infra11:55
openstackgerritMerged openstack/project-config master: Add publish-to-pypi template for sushy-cli  https://review.opendev.org/70832911:57
*** amoralej is now known as amoralej|lunch11:59
*** gfidente has quit IRC12:02
*** rpittau is now known as rpittau|bbl12:07
*** ykarel is now known as ykarel|afk12:09
*** gfidente has joined #openstack-infra12:14
*** tkajinam has joined #openstack-infra12:23
*** yamamoto has quit IRC12:28
*** fdegir has quit IRC12:29
*** yamamoto has joined #openstack-infra12:31
*** fdegir has joined #openstack-infra12:31
*** adriant has quit IRC12:33
*** adriant4 has joined #openstack-infra12:33
*** iokiwi has quit IRC12:38
*** iokiwi has joined #openstack-infra12:38
*** Wryhder has joined #openstack-infra12:39
*** jpena is now known as jpena|lunch12:39
*** Lucas_Gray has quit IRC12:41
*** Wryhder is now known as Lucas_Gray12:41
*** smarcet has joined #openstack-infra12:42
openstackgerritTobias Henkel proposed zuul/zuul master: Increase cherrypy thread pool  https://review.opendev.org/70837712:43
*** ykarel|afk is now known as ykarel12:46
*** adriant4 has quit IRC12:48
*** adriant has joined #openstack-infra12:49
*** lucasagomes has quit IRC12:51
*** smarcet has quit IRC12:52
*** lucasagomes has joined #openstack-infra12:55
*** rlandy has joined #openstack-infra12:56
*** smarcet has joined #openstack-infra12:57
*** beagles is now known as beagles_mtg13:01
*** rh-jelabarre has joined #openstack-infra13:02
*** rpittau|bbl is now known as rpittau13:06
openstackgerritFabien Boucher proposed zuul/zuul master: Add reference pipelines file for Github driver  https://review.opendev.org/67271213:08
*** amoralej|lunch is now known as amoralej13:08
openstackgerritFabien Boucher proposed zuul/zuul master: A Zuul reporter for Elasticsearch  https://review.opendev.org/64492713:12
openstackgerritFabien Boucher proposed zuul/zuul master: URLTrigger driver time based  https://review.opendev.org/63556713:14
*** ramishra has quit IRC13:15
openstackgerritRico Lin proposed opendev/irc-meetings master: Rename meeting name for Automation SIG  https://review.opendev.org/70838413:20
*** ramishra has joined #openstack-infra13:25
*** smarcet has quit IRC13:31
*** ccamacho has joined #openstack-infra13:36
openstackgerritJan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper  https://review.opendev.org/70571613:40
*** jpena|lunch is now known as jpena13:42
*** ykarel is now known as ykarel|afk13:44
*** Goneri has joined #openstack-infra13:46
openstackgerritTobias Henkel proposed zuul/zuul master: Increase cherrypy thread pool size  https://review.opendev.org/70837713:51
*** yamamoto has quit IRC13:52
*** udesale has joined #openstack-infra14:02
*** jamesmcarthur has joined #openstack-infra14:03
*** igordc has joined #openstack-infra14:04
*** aaronsheffield has joined #openstack-infra14:07
*** yamamoto has joined #openstack-infra14:10
*** smarcet has joined #openstack-infra14:14
*** owalsh_ is now known as owalsh14:26
openstackgerritThierry Carrez proposed openstack/project-config master: check-release-approval: retry once when Gerrit fails  https://review.opendev.org/70839714:27
*** Tengu has quit IRC14:30
*** Tengu has joined #openstack-infra14:31
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: Use CentOS8 dependencies repo to install openvswitch  https://review.opendev.org/70840214:36
*** cdearborn has joined #openstack-infra14:36
cdearbornmorning folks - i really messed up a couple of backport patches.  i've tried everything i can think of to fix the mess with no luck.  can someone help?14:39
cdearbornthe scenario is: i cherry picked a patch to stable/stein and stable/queens14:39
cdearbornthe patch was reviewed, and WF+114:39
cdearbornit merged14:40
cdearborni accidentally did a git review on the patch again14:40
cdearbornthat appeared to unmerge it14:40
cdearbornnow the CI won't run on the patch, even when WF+1 again14:40
cdearbornand recheck does nothing14:41
cdearbornany idea how i can get these 2 patches back into the merged state?14:41
*** udesale has quit IRC14:41
clarkbcdearborn: can you link to the changes in question? fwiw git review cannot (un)merge things asit only pushes to gerrits magic proposal refs14:42
cdearbornk14:42
*** zigo has quit IRC14:42
cdearbornoriginal patch: https://review.opendev.org/#/c/629613/14:42
cdearborncherry pick 1: https://review.opendev.org/#/c/660215/14:42
cdearborncherry pick 2: https://review.opendev.org/#/c/706383/14:43
cdearborni just abandoned the 2 cherry picks14:43
cdearbornbut can undo that14:43
openstackgerritTobias Henkel proposed zuul/zuul master: Log duration of tenant reconfigurations  https://review.opendev.org/70840314:43
clarkbonly the first change merged. neither of the cherry picks appear to have merged14:45
cdearbornok14:45
clarkbI also dont see any job results on any patchset of the cherry picks14:46
cdearbornk14:46
clarkbI think that you probably dont have jobs configured there and so nothing runs (and that prevents merging)14:46
clarkbthe two cherry picks should be managed independent of each other though. They areon different branches and are distinct changes14:47
cdearbornyes, completely agree14:47
cdearbornso we relatively recently made zuulv3 changes - maybe something is messed up there?14:48
cdearbornhttps://github.com/openstack/python-dracclient/blob/master/zuul.d/project.yaml14:48
clarkbcdearborn: because python-dracclient is branched zuul applies that config only to the branvh that config lives in14:49
clarkbyou need zuul configs on the stable branches too14:50
*** Lucas_Gray has quit IRC14:51
cdearbornohhhhhhhh - that makes complete sense.  will get that addressed.  thank you very much for your help clarkb!14:51
clarkbcdearborn: if the zuul configs are updated on the stable branches you should be able to recheck andmerge those backports14:52
openstackgerritSimon Westphahl proposed zuul/zuul master: wip: Proof of concept for matrix jobs  https://review.opendev.org/70840614:53
cdearbornclarkb: perfect - obviously a lot of fog of war going on here.  thx again for your help!14:53
*** Lucas_Gray has joined #openstack-infra14:55
*** jamesmcarthur has quit IRC14:56
*** jamesmcarthur has joined #openstack-infra14:57
*** michael-beaver has joined #openstack-infra14:59
*** igordc has quit IRC15:00
*** roman_g has joined #openstack-infra15:02
*** Goneri has quit IRC15:03
smcginnisHowdy infra folks (cc ttx) - we've had a few release job failures this morning. They all look like they just need to be rerun. Four puppet forge server errors and one where we got a username prompt on "git review -s".15:03
smcginnisI've added them all starting on line 365 here: https://etherpad.openstack.org/p/ussuri-relmgt-tracking15:04
*** jamesmcarthur has quit IRC15:04
smcginnisDoes anyone have a few minutes to get those reenqueued?15:04
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: Use CentOS8 dependencies repo to install openvswitch  https://review.opendev.org/70840215:05
clarkbsmcginnis: I can do it in after meetings this morning if no one beats me to it15:07
smcginnisThanks!15:07
*** Goneri has joined #openstack-infra15:11
*** lpetrut has quit IRC15:11
smcginnisfungi: Related to that ML thread. ^^15:21
*** jamesmcarthur has joined #openstack-infra15:24
openstackgerritLee Yarwood proposed openstack/diskimage-builder master: WIP Add Fedora 31 support and test job  https://review.opendev.org/70841615:27
*** smarcet has quit IRC15:30
*** jaosorior has joined #openstack-infra15:34
fungismcginnis: thanks! clarkb and i can arm wrestle for it15:34
*** petevg has joined #openstack-infra15:35
smcginnis:)15:36
*** pgaxatte has quit IRC15:36
*** eharney has quit IRC15:38
*** Lucas_Gray has quit IRC15:41
*** takamatsu has joined #openstack-infra15:41
*** Lucas_Gray has joined #openstack-infra15:43
*** udesale has joined #openstack-infra15:51
*** armax has joined #openstack-infra15:53
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: [DNM] Use centos8 deps repo  for RHEL8 nodes  https://review.opendev.org/70843015:53
*** tesseract has quit IRC15:55
*** smarcet has joined #openstack-infra16:01
*** beagles_mtg is now known as beagles16:02
*** rpittau is now known as rpittau|afk16:06
*** udesale_ has joined #openstack-infra16:06
*** udesale has quit IRC16:07
*** zigo has joined #openstack-infra16:08
*** rosmaita has quit IRC16:09
*** gshippey has joined #openstack-infra16:12
*** ccamacho has quit IRC16:13
*** rosmaita has joined #openstack-infra16:14
*** udesale_ has quit IRC16:17
fungigonna nab early lunch and can then reenqueue releases in an hour-ish16:21
fungiback soon16:21
clarkbsmcginnis: was sushi-cli the git review -s prompt?16:22
clarkbthats odd, if it doesn't happen consistently. but I suppose could be a side effect of a network issue16:22
smcginnisclarkb: Yep, that was the one.16:23
*** udesale has joined #openstack-infra16:23
smcginnisSeemed really odd to me too, considering how often we do that operation in our jobs.16:23
clarkbroman_g: not sure if you saw my questions yesterday, but wanted to followup on your ci status when you had a cahnce (just to make sure things are working now or if sitll having trouble we can address that)16:24
smcginnisMy assumption was a network glitch too.16:24
clarkbsmcginnis: git review assumes a failure may be due to wrong username because its such a common error due to mismatch between local username and remote16:24
*** psachin has quit IRC16:27
*** Lucas_Gray has quit IRC16:29
clarkbsmcginnis: ok I've enqueued the sushi-cli one16:29
clarkblooking at puppet now16:30
*** jamesmcarthur_ has joined #openstack-infra16:32
*** jamesmcarthur has quit IRC16:36
clarkbsmcginnis: fungi ok I think I've taken care of it16:37
smcginnisclarkb: Hmm, something wrong with that I guess. Same failure with sushy-cli - https://581819ea67919485b97e-6002fae613cad806f99007086c39ea60.ssl.cf2.rackcdn.com/81e6ec96d584c89df4cdf94fa8c530d0ca2c8e0c/release-post/tag-releases/8fdc2ae/job-output.txt16:44
*** jackedin has joined #openstack-infra16:45
amoralejAJaeger, wrt https://review.opendev.org/#/c/708402/ apparently openvswitch installation is taking 20 mins in gentoo16:45
*** jackedin has quit IRC16:45
amoralejany idea?16:46
clarkbsmcginnis: ssh://release@review.openstack.org:29418/openstack/sushy-cli.git did not work. Description: Host key verification failed.16:46
clarkbsmcginnis: is it possible the host key being loaded is for review.opendev.org?16:46
*** jpena is now known as jpena|brb16:46
clarkbamoralej: is it compiling ovs?16:46
clarkbamoralej: if so we can probably set that job to non voting, but also probably better question for #zuul though prometheanfire might not see it there16:47
smcginnisclarkb: Maybe? How would we check that? And why would this one be different than others?16:47
amoralejack16:47
prometheanfiresup?16:47
amoralejclarkb, not idea how gentoo installs that, it may be compiling it16:47
clarkbprometheanfire: https://zuul.opendev.org/t/zuul/build/4bac5bb45c924547855c33861f921253 job on gentoo testing multinode networking setup seems to take 20 minutes to install ovs (possibly compiling it?)16:47
clarkbprometheanfire: that causes the job to timeout. Maybe we can install precompiled package instead somehow?16:48
smcginnisclarkb: Ah, found it I think.16:48
prometheanfireprobably16:48
clarkbsmcginnis: I think that may be part of a secret?16:48
clarkbsmcginnis: but I don't know16:48
prometheanfiresys-kernel/vanilla-kernel-bin is a precompiled kernel package16:48
smcginnissushy-cli has host=review.openstack.org16:48
clarkbsmcginnis: oh in .gitreview16:48
smcginnisI think everywhere else we have host=review.opendev.org16:48
clarkbthat would do it16:48
prometheanfirenot sure exactly what it's installing (could be a dep taking a long time to build, ovs doesn't take that long iirc16:48
smcginnisdtantsur: ^^16:48
dtantsurugh16:49
dtantsurlemme fix it16:49
smcginnisSo I think that means we will not be able to release that one.16:49
smcginnisdtantsur: Just put up a patch if you can approve it.16:49
smcginnisWe will need to do another release and just increment the bugfix rev.16:50
dtantsursmcginnis: done. I guess we cannot re-release even if nothing really has been done?16:50
smcginnisWell, actually...16:50
clarkbyou have to change the target revision but the tag name can stay 0.1.0 I think16:50
smcginnisIt never got tagged and released, so I guess we could revert the sushy-cli release patch, then resubmit with the new hash.16:51
smcginnisMoving back over to #openstack-release since this is no longer an infra thing.16:51
prometheanfireit doesn't look like we can see the primary/secondary logs?16:51
smcginnisThanks clarkb16:51
clarkbprometheanfire: probably a side effect of timing out?16:52
prometheanfireya16:52
prometheanfireguess I need to try a manual build, or something16:52
mordredsmcginnis, clarkb: http://codesearch.openstack.org/?q=host%3Dreview.openstack.org&i=nope&files=&repos= - we might want to follow up on those16:53
* prometheanfire will ask mgorny if the binary kernel supports building modules against it16:53
clarkbprometheanfire: oh it could be that we have to build a kernel in order to build the ovs module?16:53
clarkband that would take time for sure if so16:53
AJaegerprometheanfire: increase the timeout temporarily for the job and see what takes so long?16:53
mordredlooks like sushi-cli is the only "real" instance16:54
prometheanfireclarkb: typically, no16:54
prometheanfireAJaeger: ya, not sure what it's building and that's the quick way to know16:54
prometheanfirethe image should have sources installed already (and used to build it's running kernel)16:55
smcginnismordred: Great idea to check. That's more than I would have expected at this point.16:55
mordredsmcginnis: seems mostly to be docs - sushy-cli and then the cookiecutter repos16:55
smcginnisOh right. That's good.16:56
openstackgerritSean McGinnis proposed opendev/puppet-infra-cookiecutter master: Fix gitreview host name  https://review.opendev.org/70845216:58
*** lmiccini has quit IRC16:58
*** udesale_ has joined #openstack-infra16:58
dtantsurmordred: I guess we used the cookiecutter16:59
*** udesale has quit IRC16:59
*** gyee has joined #openstack-infra17:00
openstackgerritLee Yarwood proposed openstack/diskimage-builder master: WIP Add Fedora 31 support and test job  https://review.opendev.org/70841617:01
*** eernst has joined #openstack-infra17:02
openstackgerritSean McGinnis proposed openstack/cookiecutter master: Fix gitreview host name  https://review.opendev.org/70845617:02
openstackgerritSean McGinnis proposed openstack/specs-cookiecutter master: Fix gitreview host name  https://review.opendev.org/70845717:03
*** dtantsur is now known as dtantsur|afk17:06
*** eernst has quit IRC17:07
*** igordc has joined #openstack-infra17:09
clarkbwith that sorted out I'm going to find food so that I' not starving during the infra meeting :)17:10
*** udesale_ has quit IRC17:12
*** lucasagomes has quit IRC17:13
*** udesale has joined #openstack-infra17:14
*** jpena|brb is now known as jpena17:23
*** udesale has quit IRC17:27
*** jaosorior has quit IRC17:28
*** yamamoto has quit IRC17:33
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-infra17:34
*** armax has quit IRC17:39
openstackgerritMerged zuul/zuul-operator master: Update sdk to v0.13.0 and install dhall-to-json  https://review.opendev.org/70210317:39
openstackgerritKendall Nelson proposed openstack/cookiecutter master: Add include for CONTRIBUTING.rst  https://review.opendev.org/70773517:42
openstackgerritLee Yarwood proposed openstack/diskimage-builder master: WIP Add Fedora 31 support and test job  https://review.opendev.org/70841617:49
*** gfidente has quit IRC17:54
*** weifan has joined #openstack-infra17:55
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add zuul-operator-promote-image job  https://review.opendev.org/70418717:56
*** derekh has quit IRC18:04
*** weifan has quit IRC18:05
zbrclarkb: fungi: maybe you can send some feedback on https://review.opendev.org/#/c/708323/ on 3rd party logging guidelines.18:06
clarkbzbr: +2 thanks. That seems like a good update for practicality and useability18:08
*** weifan has joined #openstack-infra18:08
*** smarcet has quit IRC18:09
*** weifan has quit IRC18:13
*** weifan has joined #openstack-infra18:14
*** weifan has quit IRC18:18
*** jpena is now known as jpena|off18:33
*** tosky has quit IRC18:35
*** ralonsoh has quit IRC18:36
*** priteau has quit IRC18:39
*** amoralej is now known as amoralej|off18:41
openstackgerritMerged opendev/system-config master: Improve 3rd-party logging guidelines  https://review.opendev.org/70832318:42
*** eernst has joined #openstack-infra18:44
*** larainema has quit IRC18:53
openstackgerritMerged openstack/specs-cookiecutter master: Fix gitreview host name  https://review.opendev.org/70845718:55
openstackgerritMerged openstack/cookiecutter master: Fix gitreview host name  https://review.opendev.org/70845618:55
mordreddiablo_rojo: your cookiecutter patch above - that works in GH? like - the include gets processed right?18:56
*** matt_kosut has joined #openstack-infra18:57
diablo_rojomordred, I think so? I will go check that thing. I think I know of a repo where we have an example of this in practice..18:57
mordreddiablo_rojo: cool. thanks! (neat concept)18:57
diablo_rojoI hope it works lol.18:58
mordreddiablo_rojo: I have complete and utter faith that it does. except for the skepticism18:58
diablo_rojomordred, It seems like it should work.. in theory..then again Github is silly.18:58
weshay|ruckianw, qq.. any plans for devstack centos-8, and do you need help?19:03
weshay|ruckzbr, ^19:04
zbro/19:04
*** matt_kosut has quit IRC19:14
*** priteau has joined #openstack-infra19:19
mordredianw: so based on that - I'm not sure why the .cer isn't updated19:21
*** priteau has quit IRC19:23
ianwmordred: i wonder if *maybe* ansible runs were broken for some other reason at that point, so maybe it got that far, but then never issued the cert, but now doesn't notice?19:24
ianwmordred: the quickest thing might be to just move the certs out of the way so that it think it needs to regenerate them19:24
mordredianw: hrm19:26
fungimordred: diablo_rojo: my original recommendation was the exact opposite of https://review.opendev.org/70773519:26
fungimaybe github knows how to do that, but someone cloning the repo and opening CONTRIBUTING.rst with their text editor or cat doesn't automatically provide useful info19:27
mordredfungi: were you suggesting putting the data in CONTRIBUTING then linking it in the docs?19:27
fungisphinx on the other hand is great at using the include19:27
openstackgerritIan Wienand proposed openstack/project-config master: nodepool-elements: Use venv for utilities  https://review.opendev.org/70775019:27
mordredyeah19:27
fungiso putting the include directive in the doc source is far more correct, while leaving the content in the more discoverable file19:27
fungii left a review comment to that effect19:30
*** ijw has joined #openstack-infra19:37
*** ijw has quit IRC19:42
ianw2020-02-12 07:49:22,716 p=root u=4548 | RUNNING HANDLER [letsencrypt-create-certs : Reload apache] *********************19:44
ianw2020-02-12 07:49:22,717 p=root u=4548 | Wednesday 12 February 2020  07:49:22 +0000 (0:00:02.619)       0:01:15.213 ****19:44
ianw2020-02-12 07:49:23,854 p=root u=4548 | fatal: [static01.opendev.org]: FAILED! => {"changed": false, "msg": "Unable to reload service apache2: Job for apache2.service failed because the control process exited with error code.\nSee \"systemctl status apache2.service\" and \"journalctl -xe\" for details.\n"}19:44
ianwmordred: ^ i think that's a smoking gun19:44
ianwthat was my screw up ... i didn't have the handlers in place for tarballs.openstack.org so it wasn't deploying the cert properly19:44
ianwi think that your change got messed up in there and due to that never got reloaded19:45
*** ijw has joined #openstack-infra19:45
*** auristor has quit IRC19:46
mordredianw: AH weird19:47
mordredianw: so we shoudl just delete the .cer and re-run ansible probably?19:47
ianwmordred: possibly just restart it?19:48
ianwthat explains to me why it never restarted at least19:48
*** eernst has quit IRC19:48
mordredianw: well - but it also has an old cert file19:49
*** auristor has joined #openstack-infra19:49
ianwthe log period is /var/log/ansible/ansible.log.6.gz19:50
ianwmordred: 2020-02-12 19:20:23,776 p=root u=3050 | fatal: [review01.openstack.org]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'dict'. Error was a <class 'ansible.errors.AnsibleError'>, original message: with_dict expects a dict"}19:54
ianwahhh so something went wrong19:54
mordredhrm.19:56
mordredianw: I don't see anywhere that we're using with_dict :(19:57
mordredoh - lookup plugin19:57
ianwthat's also review not review-dev ... still looking through the logs at that period19:57
ianwhttp://paste.openstack.org/show/789728/ -- it noticed it needed updating and got the txt token19:58
openstackgerritJeremy Stanley proposed zuul/zuul master: Flesh out the glossary significantly  https://review.opendev.org/70439119:58
*** Goneri has quit IRC19:58
ianwmordred: i think it's static that gummed up the works.  i had not put in the _acme-challenge redirect for static.openstack.org, so that certificate issue was constantly failing20:00
ianwwe see20:01
ianw2020-02-12 19:21:02,393 p=root u=3050 | TASK [letsencrypt-create-certs : Run acme.sh driver for static01-tarballs-opendev-org certificate issue] ***20:01
ianwbut never the same for review-dev20:01
mordredianw: nod. so it's not likely to be an actual issue in our ansible as much as a point in time issue20:01
ianwwell, i think that we could definitely handle errors better20:02
mordredyah20:02
ianwi have had https://github.com/acmesh-official/acme.sh/pull/2606 out for a long time with no response20:02
ianwthat would help20:03
ianwat this point, i would remove the certs and let them get regenerated on the next run20:03
mordredianw: kk. I'm moving it out of the way20:03
mordredand we'll see how it goes20:04
openstackgerritAndreas Jaeger proposed openstack/project-config master: Switch specs publishing to AFS  https://review.opendev.org/70850020:04
AJaegerianw: what do you think ^20:04
mordredianw: now - I believe you're supposed to be on PTO? So thanks - and go away!20:04
ianwheh yes but nobody else is awake yet :)20:04
AJaegerianw: sorry for pinging, enjoy your vacation!20:05
mordredianw: hehe :)20:05
*** eernst has joined #openstack-infra20:05
mordredianw: I do the same thing ...20:05
AJaegerinfra-root, could we merge https://review.opendev.org/#/c/708452/1 to fix the puppet-infra-cookiecutter .gitreview, please?20:05
diablo_rojofungi, so on that cookiecutter patch..you think I should keep it how it was originally? With the contributing.rst including the CONTRIBUTING.rst20:06
diablo_rojo?20:06
fungithat was what i had originally suggested, yes20:07
diablo_rojoOkay so I did have that right.. I read the conversation like ten times to make sure I had it like that..20:07
diablo_rojoBut at the same time I do think it would be helpful to have the stuff show up when people look at the github repo20:07
ianwAJaeger: lgtm, the volume is there ... not sure if i created apache configs yet?20:07
fungiCONTRIBUTING.rst is the canonical location people expect that information at in the repository. the include directive is for gluing it into the sphinx sources so that sphinx integrates the content when building documentation in other formats (html, et cetera)20:07
diablo_rojo(assuming the include works how I think it does)20:07
openstackgerritIan Wienand proposed opendev/system-config master: Make haproxy role more generic  https://review.opendev.org/67790320:08
openstackgerritIan Wienand proposed opendev/system-config master: Add a service load balancer  https://review.opendev.org/67815920:08
diablo_rojofungi, so maybe make mention of the contributing.rst doc in CONTRIBUTING.rst and put the include in contributing.rst?20:08
AJaegerianw: before we merge, we need to rsync all content over... Note that I have no parallel publishing.20:09
*** eernst has quit IRC20:09
AJaegerianw: if you want parallel publishing, we need to rework...20:10
diablo_rojofungi, like 'For actual details on contributing to this repo go here' in CONTRIBUTING.rst and link to contributing.rst20:10
diablo_rojoand then have the include in contributing.rst to CONTRIBUTING.rst?20:10
fungidiablo_rojo: if there's something worth mentioning about doc/source/contributor/contributing.rst which isn't already conveyed in the CONTRIBUTING.rst file then i guess you could mention it, though i imagined it would just be an empty shell with a .. include:: ../../../CONTRIBUTING.rst20:10
AJaegerianw: so, want to be safe and complex - or go for it?20:10
fungidiablo_rojo: is there something projects are going to want in doc/source/contributor/contributing.rst which doesn't belong in CONTRIBUTING.rst?20:11
ianwAJaeger: no, i think we can just do it quickly.  first though we need to be serving that directory from a site in https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/static/files20:11
diablo_rojofungi, but then the template should be CONTRIBUTING.rst and currently its not.. its contributing.rst. Unless my brain is broken or I don't understand the cookiecutter repo (which I feel is likely at this point)20:12
ianwAJaeger: so we should depend-on a change to do that.  i can do an initial rsync for testing, then we can quickly merge the publishing update and move the dns close together20:12
AJaegerianw: ok, let me update sotryboard20:12
* diablo_rojo needs more caffeine20:12
AJaegerianw: can you do the sstem-config one, please?20:13
fungidiablo_rojo: i think that may be because someone recently added a template there?20:13
* mordred goes to find lunch20:13
openstackgerritAndreas Jaeger proposed openstack/project-config master: Switch specs publishing to AFS  https://review.opendev.org/70850020:14
fungidiablo_rojo: looking back through the repo history, i think i'm actually arguing for a revert of https://review.opendev.org/69173720:17
fungiwe used to have {{cookiecutter.repo_name}}/doc/source/contributor/contributing.rst as an empty shell including the ../../../CONTRIBUTING.rst file to get its content20:17
fungier, i mean revert of https://review.opendev.org/69600120:18
openstackgerritIan Wienand proposed opendev/system-config master: [wip] add specs.openstack.org  https://review.opendev.org/70850720:19
fungiit looks like zaneb argued in that change that some contributor documentation doesn't belong in the CONTRIBUTING.rst file. i disagree but missed the discussion in the midst of holiday travel20:19
diablo_rojoOkay so.. the template shouldnt have been in  {{cookiecutter.repo_name}}/doc/source/contributor/contributing.rst there.. like I had done.. it should be in {{cookiecutter.repo_name}}/CONTRIBUTING.rst ?20:20
ianwAJaeger: ^^ that should be the basics20:20
diablo_rojowell I guess my original approach was the top level CONTRIBUTING.rst but thats for the actual cookie cutter repo so it should have been the file one level lower.20:21
fungidiablo_rojo: in my opinion, yes. contributor documentation (whatever contributor documentation a project thinks is relevant) should be in their CONTRIBUTING.rst file and then their doc/source/contributor/contributing.rst is ideally just glue to make it so sphinx can find CONTRIBUTING.rst and incorporate it into documentation bulds20:21
fungibuilds20:21
diablo_rojofungi, that makes sense to me and I am happy to go and fix that.20:21
fungiwell, there are people who argued in 696001 that CONTRIBUTING.rst and doc/source/contributor/contributing.rst should contain different information, so that probably needs discussion20:22
fungii think projects ought to be able to do either way and still meet the cycle goal, but i'd rather the cookiecutter example didn't recommend folks split contributor information between those files20:23
diablo_rojofungi, I think the way you are proposing makes the most sense. I can put up a patch at least to rehash that argument. I think I would rather not revert the template patch, but start a new one with putting the template in the right place?20:25
AJaegerianw: I'll add a depends-on, thanks20:25
fungidiablo_rojo: yeah, i'm adding another comment20:25
openstackgerritAndreas Jaeger proposed openstack/project-config master: Switch specs publishing to AFS  https://review.opendev.org/70850020:26
*** smarcet has joined #openstack-infra20:26
openstackgerritKendall Nelson proposed openstack/cookiecutter master: Fix Contributing template location  https://review.opendev.org/70851120:40
diablo_rojofungi, ^^ should be what you were talking about?20:40
*** beagles has quit IRC20:44
fungidiablo_rojo: almost, {{cookiecutter.repo_name}}/CONTRIBUTING.rst instead i think?20:45
*** andrewbonney has quit IRC20:45
diablo_rojole sigh20:45
diablo_rojoYes that would be correct.20:45
diablo_rojothat top level CONTRIBUTING keeps tripping me up20:45
fungiworking on the cookiecutter templates is a little mind-bending at times, yeah20:45
*** michael-beaver has quit IRC20:49
AJaegerianw, mordred, mnaser, https://review.opendev.org/708514 is step one for service-type publishing to use AFS promote job.20:49
*** jamesmcarthur_ has quit IRC20:52
*** jamesmcarthur has joined #openstack-infra20:53
openstackgerritKendall Nelson proposed openstack/cookiecutter master: Fix Contributing template location  https://review.opendev.org/70851120:53
diablo_rojofungi, ^ finally20:53
* diablo_rojo goes to get lunch and caffeine20:53
fungimmm, caffeine lunch20:57
*** jamesmcarthur has quit IRC20:58
openstackgerritAndreas Jaeger proposed openstack/project-config master: Add promote job for service-types-authority  https://review.opendev.org/70851721:06
AJaegerianw: and service-types-authority as well ^21:07
AJaegermnaser: I'll take this over ^21:08
*** Goneri has joined #openstack-infra21:08
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove publish-service-types-authority  https://review.opendev.org/70851821:10
AJaegerconfig-core, please review https://review.opendev.org/70851721:10
*** jamesmcarthur has joined #openstack-infra21:12
*** jamesmcarthur has quit IRC21:13
*** jamesmcarthur has joined #openstack-infra21:13
openstackgerritAndreas Jaeger proposed openstack/project-config master: Switch specs publishing to AFS  https://review.opendev.org/70850021:16
AJaegerianw: so, taken care of specs and service-types - could you update system-config for service-types-authority as well, please?21:18
* AJaeger calls it a day and waves good night21:18
clarkbAJaeger: good night, thank you for the help21:19
clarkbianw: https://review.opendev.org/#/c/708507/1 lgtm, I know its your day off so no rush but wanted to let you know I think you are good to rsync that content (if not done already)21:25
*** takamatsu has quit IRC21:32
openstackgerritMerged zuul/nodepool master: Dockerfile: create a nodepool user with uid 10001  https://review.opendev.org/70549721:32
openstackgerritwes hayutin proposed openstack/diskimage-builder master: WIP: Validate virtualenv and pip  https://review.opendev.org/70710421:45
*** aaronsheffield has quit IRC22:03
*** matt_kosut has joined #openstack-infra22:12
*** rh-jelabarre has quit IRC22:14
*** rcernin has joined #openstack-infra22:15
openstackgerritMerged zuul/zuul master: Handle draft pull requests in canMerge  https://review.opendev.org/70415022:24
openstackgerritMerged zuul/zuul master: Disable pip cache when managing ansible  https://review.opendev.org/70418422:25
*** rfolco has quit IRC22:26
clarkbI've submitted the ptg form for infra/opendev22:37
clarkbrequested space for between half a day and 2 days and noted we can be flexible. I think what we found in shanghai was that our small group could manage adhoc meetups pretty well, but having an official presence of some sort was helpful for people that were looking for us22:37
*** slaweq has quit IRC22:38
*** migawa is now known as migawa|AFK|AFK22:39
fungisgtm22:44
mordredclarkb, fungi: got a sec for an easy one: https://review.opendev.org/#/c/707668/ ?22:46
mordredclarkb: ++22:46
*** haleyb is now known as haleyb|away22:46
fungiapprovidado22:47
mordredfungi: tyty22:47
*** jamesmcarthur has quit IRC22:49
*** rcernin has quit IRC22:49
*** rcernin has joined #openstack-infra22:50
openstackgerritMerged openstack/project-config master: Add promote job for service-types-authority  https://review.opendev.org/70851722:53
openstackgerritMerged zuul/zuul master: Flesh out the glossary significantly  https://review.opendev.org/70439122:53
*** Goneri has quit IRC22:54
openstackgerritMerged openstack/diskimage-builder master: Fix cache-url -f  https://review.opendev.org/70590222:55
*** dave-mccowan has quit IRC22:55
*** dave-mccowan has joined #openstack-infra22:59
*** migawa|AFK|AFK is now known as migawa22:59
*** pkopec has quit IRC22:59
*** jamesmcarthur has joined #openstack-infra23:01
*** slaweq has joined #openstack-infra23:11
fungiclarkb: smcginnis: just to be clear, the previous puppet module release failures got rerun and are all fine now, but the sushy-cli failure needed a new release change... there are no outstanding openstack release failures which need resolving now right?23:15
clarkbfungi: that is my understanding23:15
fungiokay, cool, removing from my todo list23:15
*** slaweq has quit IRC23:16
openstackgerritMerged zuul/zuul master: Log duration of AnsibleJob.execute()  https://review.opendev.org/70458923:19
*** jamesmcarthur has quit IRC23:19
openstackgerritMerged opendev/system-config master: Add curl to python-builder image  https://review.opendev.org/70766823:20
*** iurygregory has quit IRC23:21
clarkbmordred: looks like we have the votes to land the gitea upgrade now23:31
clarkbmordred: I need to mow the yard before it gets dark but can be help watch around that23:31
clarkbor we can do it tomrrow (but I'll be distracted by meetings much of the day)23:31
mordredclarkb: let's do it tomorrow23:34
mordredclarkb: I'm pretty much at EOD and have a parade and a ball tonight23:34
clarkboh its that time of year in your new part of the world isn't it23:34
mordredthat's right!23:34
mordredoh - speaking of - I meant to mention in the meeting23:34
mordredI will not be here next Tue23:34
*** dchen has joined #openstack-infra23:36
*** rlandy has quit IRC23:41
openstackgerritMerged zuul/zuul master: ansible manager: only failed if last ansible failed  https://review.opendev.org/70469923:44
*** smarcet has quit IRC23:45
mordredclarkb: also - I will likely get a late start tomorrow - but I will be here23:47
clarkbmordred: k, ping me when you want to get started and we'll see where I'm at in meetings23:48
mordredcoolio23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!