Thursday, 2018-08-30

*** annabelleB has quit IRC00:32
*** annabelleB has joined #openstack-release00:37
*** annabelleB has quit IRC00:51
*** bobh has joined #openstack-release01:19
*** ricolin has joined #openstack-release01:34
*** bobh has quit IRC01:41
*** hongbin has joined #openstack-release01:54
*** ekcs has quit IRC02:00
*** bobh has joined #openstack-release02:15
*** bobh has quit IRC02:16
*** bobh has joined #openstack-release02:16
*** bobh has quit IRC02:20
*** ykarel has joined #openstack-release03:24
*** ykarel_ has joined #openstack-release03:31
*** ykarel has quit IRC03:33
*** rosmaita has quit IRC03:34
*** dave-mccowan has quit IRC03:38
*** ykarel__ has joined #openstack-release03:58
*** ykarel_ has quit IRC03:58
openstackgerritYikun Jiang (Kero) proposed openstack/releases master: Release oslo.utils 3.36.5 for Rocky  https://review.openstack.org/59798304:10
*** ykarel__ has quit IRC04:22
*** radeks_ has joined #openstack-release04:48
*** hongbin has quit IRC04:56
*** ykarel__ has joined #openstack-release04:58
*** ykarel__ is now known as ykarel05:00
*** e0ne has joined #openstack-release05:31
*** udesale has joined #openstack-release05:53
*** openstackgerrit has quit IRC06:07
*** udesale has quit IRC06:07
*** udesale has joined #openstack-release06:08
*** e0ne has quit IRC06:19
*** e0ne has joined #openstack-release06:20
*** e0ne has quit IRC06:20
*** radeks_ has quit IRC06:21
*** radeks has joined #openstack-release06:23
*** alexchadin has joined #openstack-release06:23
*** shardy has joined #openstack-release06:32
*** alexchadin has quit IRC07:04
*** alexchadin has joined #openstack-release07:04
*** alexchadin has quit IRC07:05
*** radeks has quit IRC07:12
*** radeks has joined #openstack-release07:13
*** pcaruana has joined #openstack-release07:14
*** radeks has quit IRC07:16
*** lxkong has quit IRC07:19
*** tosky has joined #openstack-release07:38
*** olivierb_ has joined #openstack-release07:47
*** jpich has joined #openstack-release07:56
*** electrofelix has joined #openstack-release08:21
*** ykarel is now known as ykarel|lunch08:30
*** e0ne has joined #openstack-release08:48
*** dtantsur|afk is now known as dtantsur08:49
*** ykarel|lunch is now known as ykarel09:18
*** priteau has joined #openstack-release09:22
*** udesale has quit IRC09:26
*** flaper87 has quit IRC09:34
*** flaper87 has joined #openstack-release09:36
*** alexchadin has joined #openstack-release09:57
*** ricolin has quit IRC10:29
*** udesale has joined #openstack-release10:33
*** udesale has quit IRC10:44
*** udesale has joined #openstack-release10:44
*** claudiub has joined #openstack-release10:45
*** shardy is now known as shardy_lunch10:57
*** priteau has quit IRC11:00
*** priteau has joined #openstack-release11:14
*** rnm has joined #openstack-release11:34
*** aojea has joined #openstack-release11:36
*** rnm is now known as rmart0411:36
*** aojea has quit IRC11:36
*** aojea has joined #openstack-release11:37
*** rmart04 has quit IRC11:40
*** rnm has joined #openstack-release11:40
*** rnm has quit IRC11:42
*** rnm has joined #openstack-release11:42
lennybzigo: are you building packages for stable/rocky already? I see that https://stretch-rocky.debian.net/job/networking-mlnx/ was not triggered yet.11:43
*** rnm is now known as rmart0411:43
*** dmellado has quit IRC11:50
*** stevebaker has quit IRC11:51
* ttx waves on Rocky release day11:52
*** alexchadin has quit IRC11:59
zigolennyb: Yes, and I'm done with the release, however, I haven't seen a release for networking-mlnx.12:01
zigoIs it ready?12:01
zigottx: I just hope this one can be fixed before the final release is cut! https://bugs.launchpad.net/nova/+bug/178965412:04
openstackLaunchpad bug 1789654 in OpenStack Compute (nova) rocky "placement allocation_ratio initialized with 0.0" [High,Confirmed]12:04
zigoQuite annoying ...12:04
zigoAppart from that, I'm ready. Please shoot ! :P12:04
ttxThe patch is not merged in master, so I'd say the odds are low that it will make the release. We'll do a stable release soon after12:05
ttxmaybe mention a workaround in release notes12:06
ttxI'll see with smcginnis and dhellmann what our options are12:06
*** udesale has quit IRC12:08
smcginnisttx, zigo: I spoke briefly with mel about that yesterday. The plan is to get a nova stable release out as soon as they can after that merges.12:11
zigoGood, good ! :)12:11
*** udesale has joined #openstack-release12:11
smcginnisttx: Shall we wait for dhellmann to be around, then process the Rocky release?12:13
ttxsmcginnis: I was wondering if we should not include that "known issue" in a release notes of sorts12:14
dhellmanno/12:14
ttxalthough arguably reno makes that a bit more protracted than a wiki edit12:15
smcginnisMorning dhellmann12:15
dhellmannI had final release tagging on my calendar for 1400 utc, did i get that wrong?12:16
dhellmannmaybe that was the press release12:16
smcginnisdhellmann: Oh, I wasn't aware we had set a specific time yet.12:16
ttxpress release would be 15utc ish12:16
smcginnisI suppose an hour should be safe. Maybe we should do it a little before 1400 just to make sure there are no issues before the announcement goes out though.12:17
dhellmannhttp://eavesdrop.openstack.org/meetings/releaseteam/2018/releaseteam.2018-08-24-15.00.log.html#l-11912:17
lennybzigo, we made stable/rocky branch few days ago, since were a bug in devstack/tempest, so we couldn't test it properly.12:17
ttxis it the only thing that needs to be done ?12:17
dhellmannI'm fine with starting now12:17
* dhellmann promises not to get marmalade on the release12:17
*** shardy_lunch is now known as shardy12:18
ttxdhellmann: there are a few steps to do after the patch merges12:18
ttxhttps://releases.openstack.org/reference/process.html#final-release12:18
smcginnisHah, don't remember stating that timeframe. Good thing we have logs. :)12:18
ttxStep 5 should be done shortly before 15utc12:19
dhellmanndo we have the patches we need lined up?12:19
smcginnisI can get those queued up.12:19
dhellmann++12:19
ttxthere was also some rumor of a PR going out at 9 CT instead of 10ct, let me doublecheck12:19
smcginnisOK, how about we wait for 13:00-13:30 ish.12:20
dhellmannit would be good to know fungi was online in case things go wrong12:20
smcginnisThen I can get some breakfast and get those other patches ready.12:20
smcginnisThat was going to be my next step - check in with infra to make sure all lights are green.12:20
ttxoh well, the page is up on the website so I guess any time is good12:21
smcginnisOrly12:21
smcginnisWhich page?12:22
ttxhttps://www.openstack.org/software/ > click on latest release12:22
ttxnothing in the press though yet12:23
smcginniswww still has a not stating latest release is queens.12:23
ttxsmcginnis: ok for 13:00-13:30 ish.12:23
ttxsmcginnis: indeed12:24
smcginnisAndreas is asking about docs team tasks. Have either of you been in contact with the docs team on any task there?12:25
smcginnisOtherwise, infra looks quiet right now.12:25
* smcginnis grabs breakfast before the fun begins12:26
*** alexchadin has joined #openstack-release12:28
*** huats has quit IRC12:30
*** oanson has joined #openstack-release12:30
dhellmannsmcginnis : I haven't been but can check if pkovar is online12:36
*** alexchadin has quit IRC12:38
*** udesale has quit IRC12:40
*** udesale has joined #openstack-release12:53
*** mriedem has joined #openstack-release12:54
*** rosmaita has joined #openstack-release12:55
*** bobh has joined #openstack-release13:04
dhellmannsmcginnis : the doc site update patch is https://review.openstack.org/59815013:08
smcginnisdhellmann: Thanks!13:08
smcginnisdhellmann: Maybe should take WIP out of the commit message.13:09
smcginnisStarting up those other patches, then we can probably proceed.13:09
fungii'll be around at the planned time13:12
fungiassuming the marketing webinar doesn't run long13:12
smcginnisfungi: Thanks13:13
*** alexchadin has joined #openstack-release13:13
*** openstackgerrit has joined #openstack-release13:17
openstackgerritSean McGinnis proposed openstack/releases master: Mark Rocky as released  https://review.openstack.org/59815913:17
openstackgerritSean McGinnis proposed openstack/releases master: Set Stein as current release  https://review.openstack.org/59816013:17
openstackgerritSean McGinnis proposed openstack/releases master: Mark Ocata as extended maintenance  https://review.openstack.org/59816413:23
smcginnisOK, 13:30. And the doc patch just landed.13:26
smcginnisOr is on its way.13:27
dhellmannyeah, approved not merged13:27
smcginnisShould we +W the final release patch?13:27
dhellmannsure. that one is going to take a while to run through.13:28
ttxhomepage is updated13:28
ttxso all set from a web perspective, I think13:28
ttxgo go go13:29
dhellmannwait!13:29
dhellmannwhat's our theme song this time? :-)13:29
ttxboo13:29
ttxApparently it should still be tchoo-tchoo13:29
smcginnis:)13:29
ttxsince latest news on Denver is that the train is back!13:29
smcginnisWhat! :D13:29
dhellmannnot rocky racoon, from the white album?13:29
smcginnisdhellmann: Pick a song. :)13:30
ttxEye of teh Tiger?13:30
ttxThat artist name works well too13:30
dhellmannwfm13:30
ttxhttps://www.youtube.com/watch?v=btPJPFnesV413:30
ttxcue it13:30
dhellmannok, *now* we're ready13:31
ttxwow that group could totally be us13:32
toskyehm, between a rocky package and another, could you please also push few queens backport releases?13:32
smcginnisI'm sure I've seen the video, but man that's funny.13:32
dhellmanntosky : we're not likely to do that today, since it's the final release date for rocky13:32
smcginnistosky: We had a few waiting for stable team review.13:33
smcginnisHopefully soon.13:33
toskyokidoki13:33
smcginnisOK, should we all approve the patch?13:33
dhellmann+113:33
smcginnis313:33
smcginnis213:33
smcginnis113:34
ttx+113:34
* ttx sets that video on loop13:34
smcginnisfungi: Final release patch is going through right now.13:35
*** eumel8 has joined #openstack-release13:35
eumel8o/13:36
dhellmannhi, eumel8!13:36
smcginnisHey eumel8 o/13:36
*** frickler has joined #openstack-release13:36
eumel8here are the station with the drinks?13:37
fungiwoo!13:37
* smcginnis is going with the auto-play of Billie Jean13:37
fungirelease by the late michael jackson13:37
* dhellmann got "one night in bangkok" next13:37
ttxI have INXS's Never tear us apart next13:37
fungipink floyd's "sheep" is playing here13:37
smcginnisAlthough I see one of the other recommendations from YouTube is Livin' on a Prayer. Hopefully that's not a more appropriate soundtrack.13:38
fungi(bootleg live in oakland coliseum, may 1977)13:38
eumel8dhellmann: there was never a Summit in Bangkok ;)13:38
dhellmanneumel8 : yet!13:38
fungiwe had a summit near oakland, at least13:38
dhellmanneumel8 : start from https://www.youtube.com/watch?v=btPJPFnesV4 (official theme song of the release)13:39
smcginnisttx: I realized I was old when I was going to a conference with a younger sales/marketing girl and she had to ask me who the closing party band was because she never heard of INXS.13:39
dhellmannoh, god13:39
dhellmannwe used to play INXS on the college radio station before they were popular enough to show up on the local pop station13:39
ttxI did karaoke that one at fifieldt farewell party13:39
fungiyeah, took aussie rock a while to catch on here in the usa13:39
ttxprobably why Youtube proposed it to me13:40
*** aojea has quit IRC13:40
dhellmannthe doc site update patch is in the post queue now13:40
smcginnisI assume our validate job will take a while on this one.13:41
dhellmannvery likely13:41
eumel8the Eye of the Tiger has a special haircut, maybe it's good for the winter time13:41
ttxwe need to watch that job for timeouts :)13:42
ttx"Everybody hurts" is on now13:43
dhellmannObsession by Animotion13:43
smcginnisLet's hope not. :)13:43
smcginnisIt must think I want the '80's because it's now on to Bon Jovi Livin' On a Prayer.13:44
*** toabctl has joined #openstack-release13:44
smcginniseumel8: You want to see some interesting hair, watch some of those videos. :D13:44
eumel8smcginnis: there are enough recommendations on YouTube ;)13:46
smcginnisSome you can never un-see, so be careful.13:47
dhellmannI Ran, A Flock of Seagulls13:47
ttxNothin compares by Sinead O'Connor13:47
dhellmannthe expectations for videos in the 80's was pretty low13:47
dhellmannit all went into the hair, really13:48
smcginnisToto - Africa. ttx is getting better recommendations though.13:48
* dhellmann would subscribe to ttx's spotify playlist13:48
*** udesale has quit IRC13:49
eumel8but Toto is still alive13:49
*** alexchadin has quit IRC13:50
*** udesale has joined #openstack-release13:50
fungii've got "blood and sand" by pigface going here... not particularly soothing13:51
dhellmannDon't You Want Me, The Human League13:52
fungithe human league is really underrated. they had quite a few decent tracks nobody's ever heard of13:52
ttxBonnie Tyler - Total Eclipse of the Heart13:53
ttxthat video has great meme stills material13:54
*** aojea has joined #openstack-release13:55
dhellmannwe could probably speed up the validation job a bit by checking if the commit being tagged already has a tag13:56
ttxmy eyes hurt13:56
fungibruce lash (prozac for lovers) light jazz cover of the ramones "i wanna be sedated"13:56
dhellmannsome of the steps like checking the readme and packaging won't have changed if there is already a tag13:56
smcginnisfungi: That sounds interesting.13:56
dhellmannHold Me Now by Thompson Twins13:56
fungithe whole prozac for lovers series is amusing13:56
smcginnisdhellmann: Good idea, that could really speed things up.13:57
smcginnisChecking out Proud Mary by prozac for lvoers.13:57
*** pkovar has joined #openstack-release13:58
smcginnisfungi: Interesting transition from pigface.13:58
fungiyup13:58
smcginnisValidate completed!13:58
openstackgerritMerged openstack/releases master: Rocky final releases for cycle-with-milestone projects  https://review.openstack.org/59752913:59
* fungi cheers at ^13:59
dhellmannand away we go13:59
*** Linkid has joined #openstack-release13:59
smcginnisNow the fun part.13:59
ttxdownhill!13:59
fungiright on schedule too13:59
johnsomwahoo13:59
ttxYoussou N'Dour - 7 Seconds ft. Neneh Cherry13:59
smcginnisThose other follow up patches - https://review.openstack.org/#/q/project:openstack/releases+topic:rocky+status:open14:00
*** ykarel is now known as ykarel|away14:00
smcginnisHmm, haven't listened to 7 Seconds in a long time (band not song)14:00
dhellmannpublish-openstack-manuals is still running14:01
eumel8AFS upload14:01
dhellmannDoctor! Doctor! by Thompson Twins14:01
dhellmannoh, the hair14:01
fungihuh, the tag-releases job just got dequeued and requeued14:01
fungiwonder what's up with that14:02
dhellmannweird14:02
fungistreaming the console log this time14:02
ttxDepeche Mode - Enjoy the Silence14:03
*** mlavalle has joined #openstack-release14:03
smcginnisLooks happy so far.14:03
fungiin case if dies again14:03
fungiooh, there's a great 8-bit operators cover of enjoy the silence14:03
fungion a depeche mode tribute album they did14:03
dhellmannSimple Minds - Alive And Kicking14:03
fungicalled "enjoy the science"14:03
smcginnisHah14:04
ttxfungi: that sounds awesome14:04
fungithe 8-bit operators tribute to kraftwerk is really good too14:04
ttxI feel all millenial searching for music on youtube14:04
smcginnisThat sounds like it would be interesting.14:04
fungiwidely varied chiptune covers of popular kraftwerk songs14:04
ttxhttps://www.youtube.com/watch?v=Uo33G-WM774 <-- enjoy the science14:05
fungithe tag job seems to be rolling along fine this time14:05
funginot sure what happened to it the first time14:05
fungiwe're definitely getting tag refs appearing in the release pipeline now14:06
*** ykarel|away has quit IRC14:06
fungi10 tags now and counting14:07
ttxMidnight Oil - Beds Are Burning14:08
smcginnisMidnight Oil - Blue Sky Mine14:09
dhellmannTears For Fears - Shout14:09
mnaseryay14:10
* ttx switches to dhellmann's stream14:10
mnaserthe rocky tag-releases job is running on our cloud14:10
mnaserit would have been nice if it ran in sjc114:10
mnaserthat would have been a nice release-rocky-on-rocky thing14:10
smcginnis:)14:10
ttxmnaser: It's in good hands then14:10
dhellmannoh, that would have been cool14:10
fungii'm sure at least some percentage of the jobs for the tags are running in sjc114:11
mnaserwe're up to 50 nodes now so yeah it's a decent amount14:11
fungii mean, it's a _lot_ of jobs14:11
ttxwe should spot one and tweet it14:11
smcginnisThat might be an interesting thing for Anne to mention.14:11
dhellmann++14:11
fungirelease pipeline is up to 30 concurrent refs now14:11
ttxfungi: if you spot one significant in sjc1, I can tweet about it14:11
smcginnisAnd I'll retweet it.14:12
fungii'll have to see if i can backtrack that info from the scheduler log maybe14:12
* mnaser was looking for nova in release pipeline14:12
ttxPet Shop Boys - It's A Sin14:12
smcginnisU2 - Where the Streets Have No Name14:13
mnasernova just queued up14:13
mnaserlet's see how's the lottery like today14:13
mnasercinder released at our side but not in sjc1 boo14:14
mnaseroh awesome14:15
mnaserneutron is going on sjc114:15
mnaserhttp://zuul.openstack.org/stream.html?uuid=61e31c0feafc4ebfa909e6e30606547a&logfile=console.log14:15
fungigood find!14:15
smcginnisCool14:15
ttxwooho, tweet it is14:15
fungii'm still trying to figure out how to map running job names to nodes in the scheduler log in any meaningful way and coming up short14:15
* mnaser is using the good o'l click and scroll through logs14:16
eumel8docs are live now14:16
mnaser:P14:16
ttxhttps://twitter.com/tcarrez/status/103516933943386931414:16
fungiyeah, the scheduler doesn't log the node names, just their numeric ids14:17
ttxsmcginnis: you'll do the "official" release tweet and I'll retweet you :)14:17
fungiso would have to cross-reference against executor logs or nodepool launcher14:17
smcginnisttx: Sounds like a plan!14:17
dhellmannfungi : maybe we can get the marketing department to file a zuul feature request to make that search easier ;-)14:18
ttxAlphaville - Big in Japan14:18
dhellmannToto - Rosanna14:18
smcginnisPink Floyd - Another Brick In The Wall14:18
smcginnisTagging done.14:18
dhellmannok, now YouTube is just messing with me: The Bangles - Walk Like an Egyptian14:19
openstackgerritzhulingjie proposed openstack/reno master: Migrate the link of bug report button to storyboard  https://review.openstack.org/59817714:19
*** alexchadin has joined #openstack-release14:20
*** huats has joined #openstack-release14:20
ttxYouTube is getting lazy: Alphaville - Forever Young14:21
ttxI might have to force it to play Another Brick in the Wall14:21
smcginnisEnjoying that one yet.14:21
fungitag-releases job finished at 14:16z, count of tag refs enqueued in the release pipeline is fallnig14:22
fungier, falling14:22
smcginnisWhew, that looked like "failing" at first.14:22
fungisucceeding14:22
ttxok, we are back on track14:23
dhellmannAfter The Fire - Der Kommissar14:23
ttxok, back to work, should be push the next patches14:24
ttxshould we*14:24
smcginnisShould be safe to proceed - https://review.openstack.org/#/q/project:openstack/releases+topic:rocky+status:open14:24
smcginnisTwisted Sister - We're Not Gonna Take It :D14:24
dhellmannyes, let's approve https://review.openstack.org/#/c/598159/14:25
ttxhmm14:26
ttxI like to use that one to fix any parallel run conflicts at the end14:26
smcginnisOh, that's a good idea.14:26
fungiso wait until all the tags in the release pipeline are clear?14:26
ttxoh, but they all published already14:26
ttx?14:26
dhellmannyeah, we just had 1 publish job14:27
ttxyeah, probably safe then14:27
dhellmannthe things currently in the release queue do not publish to releases.o.o14:27
fungiahh, right you're building it off the data in the release repo, not from actual tags14:27
ttxOK, I'll push14:27
ttxsmcginnis: leaving the W+1 to you14:28
ttxon https://review.openstack.org/#/c/598159/14:28
smcginnisOn its way14:28
fungiso it may just have dead 404 links to artifacts for a brief window until those also finish14:28
dhellmannright14:28
ttxsure14:28
dhellmannTaco - Puttin' on the Ritz14:28
ttxcan leave with that. We'll run missing-releases to be sure14:28
smcginnisa-ha - Take On Me14:29
ttxR.E.M. - Losing My Religion14:29
* dhellmann switches to smcginnis' playlist14:29
ttxthat video should be great14:30
* ttx switches too14:30
ttxsmcginnis: we'll need https://review.openstack.org/#/c/596369/4 in too14:34
smcginnisttx: Want to send that one through?14:34
dhellmannLipps Inc - Funky Town14:34
ttxDavid Bowie - Heroes14:34
smcginnisTears For Fears - Everybody Wants To Rule the World14:35
* dhellmann has started skipping ahead14:35
dhellmannDire Straits - Money For Nothing14:35
*** ykarel has joined #openstack-release14:35
ttxsmcginnis: sure14:36
openstackgerritMerged openstack/releases master: Mark Rocky as released  https://review.openstack.org/59815914:36
smcginnis9 left in the release queue.14:36
ttxrelease notes links pushed14:37
ttxif we are allowed to skip... Queen - The Show Must Go On14:37
smcginnisMen At Work - Down Under14:38
dimsLOL14:40
dhellmannCyndi Lauper - Time After Time14:41
smcginnisYouTube getting lazy again - Men At Work - Who Can It Be Now14:41
*** alexchadin has quit IRC14:42
ttxTalk Talk - Such A Shame14:43
*** alexchadin has joined #openstack-release14:44
dhellmann4 Non Blondes - What's Up14:45
smcginnisAsia - Heat of the Moment14:45
smcginnisttx, dhellmann: Quick spot check? https://etherpad.openstack.org/p/rocky-email-final14:45
ttx+@14:45
dhellmannsmcginnis : lgtm14:45
ttxsmcginnis: do you have the missing-releases command queued already ? Or should I ?14:46
smcginnisttx: Not yet. Feel free if you're ready to run it.14:46
ttxwaiting foe the last jobs14:46
smcginnisOne job left in the release queue.14:47
smcginnisAnyone have any contact with Anne this morning? Kind of early for her.14:48
smcginnisThe Cure - Friday I'm In Love14:48
ttxI can ping her... do we need her for anything special14:48
smcginnisI just wanted to go the OK from her before I send out the official announcement.14:49
ttxok14:49
ttxI asked her to join us14:49
ttxYT getting all electronic on me with Robert Miles - Children14:50
dhellmannYT is definitely trolling now: Somewhere over the Rainbow - Israel "IZ" Kamakawiwoʻole14:51
ttxAmy Winehouse - Back To Black14:51
smcginnisI saw that one coming on mine and skipped it.14:51
smcginnisHmm, networking-bgpvpn appears to be hung on setting up the venv.14:52
smcginnishttp://zuul.openstack.org/stream.html?uuid=cf5c4d8fd7194b76b56fc072127d96b8&logfile=console.log14:52
* ttx checks taht it is the only one missing14:52
dhellmannmanual override took me to Eurythmics - Sweet Dreams14:52
dhellmannsmcginnis : I wonder what needs to be installed there14:53
dhellmannoh, timeout14:53
smcginnisLazy again... Cure - Just Like Heaven14:53
smcginnisfungi: Might need a re-enqueue.14:53
dhellmannno route to host for log collection14:53
dhellmannlooks like a networking issue14:53
*** ykarel has quit IRC14:54
*** ricolin has joined #openstack-release14:54
fungismcginnis: annabelleb is trying to wrangle/host the marketing webinar starting in 5 minutes14:54
ttxmissing-releases looks confused. Running old-style link checker in the mean time14:54
ttxconfused as in very slow14:55
smcginnisttx: Last time I ran it, it did take some time.14:55
smcginnisSpandau Ballet - True14:56
ttxOasis - Wonderwall14:56
fungii can reenqueue that one but not until it fails (not sure if we got the dequeue rpc client subcommand feature merged yet)14:56
dhellmannwe have a lot of deliverable links to check now, maybe we should add some parallelism to missing-releases14:56
smcginnisfungi: It's still hanging there.14:56
smcginnisNot really sure what it's doing right now.14:57
*** peschk_l has joined #openstack-release14:57
smcginnisOne networking glitch out of all those isn't so bad I guess.14:58
*** annabelleB has joined #openstack-release14:58
dhellmannModern English - I Melt With You14:58
openstackgerritMerged openstack/releases master: Add release notes links for rocky  https://review.openstack.org/59636914:58
ttxsmcginnis: I'd cheat and still say it's released, in time for the offciial PR14:58
annabelleBwho’s irc was disconnected this whole time? this lady’s. *smh* I’m sure y’all had this under control anyways, but hello!14:59
dhellmannI'm running missing-releases and it's going pretty quickly14:59
*** ekcs has joined #openstack-release14:59
dhellmannannabelleB : you missed all of the playlist updates!14:59
ttxdhellmann: I have some ipv6 issues which might be why14:59
dhellmannah, yeah14:59
smcginnisannabelleB: Morning!14:59
dhellmannok, I'll let this keep running and report back14:59
*** pkovar has quit IRC14:59
ttxSunday Bloody Sunday - U214:59
smcginnisannabelleB: OK if I send this out? https://etherpad.openstack.org/p/rocky-email-final14:59
smcginnisannabelleB: Oh, time for the webinar to start.15:00
annabelleByeah!!15:00
* dhellmann can't find the link for the webinary15:01
dhellmannor webinar either15:01
smcginnishttps://zoom.us/j/55180365715:01
dhellmannthanks, smcginnis15:01
* ttx puases music15:02
fungismcginnis: i'll see if i can zuul dequeue it15:02
dhellmann5 errors found15:02
dhellmannmissing tarball https://tarballs.openstack.org/networking-bgpvpn/networking-bgpvpn-9.0.0.tar.gz15:02
dhellmannmissing tarball signature https://tarballs.openstack.org/networking-bgpvpn/networking-bgpvpn-9.0.0.tar.gz.asc15:02
dhellmannmissing tarball signature https://tarballs.openstack.org/neutron-tempest-plugin/neutron_tempest_plugin-0.2.0.tar.gz.asc15:02
dhellmannmissing tarball signature https://tarballs.openstack.org/release-test/openstack-release-test-1.1.0.tar.gz.asc15:02
dhellmannmissing tarball signature https://tarballs.openstack.org/tripleo-ui/tripleo-ui-9.3.0.tar.gz.asc15:02
*** ykarel has joined #openstack-release15:03
smcginnisbgpvpn is OK, that's the one that needs to rerun.15:03
ttxno showstoppers15:03
smcginnistempest-plugin, meh.15:03
smcginnisrelease-test, yeah15:03
dhellmannthose other 4 are signature files15:03
smcginnisDid we determine tripleo-ui doesn't sign?15:04
*** udesale has quit IRC15:04
smcginnisJavascript job or something?15:04
dhellmannIs that a JS project?15:04
smcginnisNot sure.15:04
dhellmannyeah, we fixed that since then15:04
smcginnisYay, it's annabelleB.15:04
smcginnis:)15:04
eumel8smcginnis: it's Project Team Gathering or Project Teams Gathering? in L1915:05
*** zul has quit IRC15:05
smcginnisTeam usually, but I suppose both work.15:06
smcginnisfungi: Looks like that job finally popped.15:06
eumel8smcginnis: there is all time Teams (https://www.openstack.org/ptg) but it's confusing to me15:07
smcginnisttx: Hmm, my announce is awaiting moderator approval. Are you a moderator for announce?15:07
smcginniseumel8: Huh, I never really noticed that.15:08
ttxprobably15:08
ttxlet me check15:08
*** zul has joined #openstack-release15:08
dhellmannttx: I have it15:08
ttxim on it15:08
dhellmannok15:09
dhellmannhmm, there are several others in the queue15:09
ttxyeah, again caught in the blackhole15:09
ttxhmmm maybe we should approve Tony's15:09
dhellmannyeah, probably15:10
fungismcginnis: thanks, looks like it logged at http://logs.openstack.org/44/440fa516b472bc4d3a59d06c5bf571f98d44f1a5/release/release-openstack-python/cf5c4d815:10
ttxand OSSA15:10
ttxI'll discard the other one since it's so old15:10
ttxbut that is kind of sad15:11
fungiyeah, looks like it never uploaded anything anywhere so should be safe to reenqueue. doing that now15:11
dhellmannI wonder if it's safe to un-do the blackholing of moderator emails there15:11
smcginnisWatching it run.15:11
fungidhellmann: i can get you some numbers on what un-blackholing that would look like in a few minutes15:12
dhellmannfungi : no rush15:12
fungisudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/networking-bgpvpn --ref=refs/tags/9.0.0 --newrev=440fa516b472bc4d3a59d06c5bf571f98d44f1a515:12
fungithat look right to everyone?15:12
*** pcaruana has quit IRC15:12
smcginnisfungi: Looks right to me.15:13
ttxsmcginnis: you should tweet before someone steals your thunder15:13
smcginnisDoing it now.15:13
fungireenqueued15:15
fungirelease job is rerunning for it: http://zuul.openstack.org/stream.html?uuid=57d76307dcc145a294f509c02712271d&logfile=console.log15:15
smcginnishttps://twitter.com/SeanTMcGinnis/status/103518436392591769615:16
dhellmannsmcginnis : I'm looking at the failure on https://review.openstack.org/#/c/598164/115:17
dhellmannit looks like we need a real date there15:18
smcginnisOh, hadn't noticed that yet.15:18
smcginnisWe may need to change that for the extended maintenance ones. We can't really declare a date for those.15:19
smcginnisI was wondering if that was going to cause a problem.15:19
dhellmannit passes validation if that date is just left out so let me see if the docs build that way15:20
smcginnisOh, if doc builds fail, that should be an easy update to the table template code.15:21
dhellmannyeah15:21
*** olivier__ has joined #openstack-release15:21
*** olivierb_ has quit IRC15:22
smcginnisfungi: Job failed again. Not a timeout this time.15:23
smcginnisCaching issue?15:23
fungiNo matching distribution found for ceilometer (from networking-odl>=12.0.0->-r /home/zuul/src/git.openstack.org/openstack/networking-bgpvpn/test-requirements.txt (line 24))15:23
ttxdhellmann: http://paste.openstack.org/show/729149/15:23
ttxI got a few more misses15:23
ttx404s15:24
dhellmannoh, I wonder if i ran that against an out of date local repo15:24
dhellmannd'oh15:24
smcginnisI think the script just looks at the latest release.15:24
ttxah ok15:25
ttxso it catches issues in older releass15:25
ttxlike openstacksdk 0.12.015:25
fungithe -ui missing signatures are likely because the javascript jobs were missing the signing role until a couple days ago15:25
*** pkovar has joined #openstack-release15:25
*** ykarel has quit IRC15:25
ttxLet me etherpad that one15:26
ttxso we can analyze them one by one15:26
ttxhttps://etherpad.openstack.org/p/vPBa2EDSdD15:26
*** e0ne has quit IRC15:27
*** ykarel has joined #openstack-release15:29
smcginnisI thought this wasn't allowed - https://github.com/openstack/networking-odl/blob/master/requirements.txt#L2115:30
fungii know15:30
smcginnisSo that looks like what's causing http://logs.openstack.org/44/440fa516b472bc4d3a59d06c5bf571f98d44f1a5/release/release-openstack-python/57d7630/job-output.txt.gz#_2018-08-30_15_19_10_50668415:30
smcginnisNot sure what we can do now at this point/15:30
openstackgerritDoug Hellmann proposed openstack/releases master: Mark Ocata as extended maintenance  https://review.openstack.org/59816415:31
openstackgerritDoug Hellmann proposed openstack/releases master: deal with missing dates for next-phase  https://review.openstack.org/59820715:31
dhellmannsmcginnis : there we go ^^15:31
smcginnisdhellmann: Any idea what we can do about the above?15:32
fungismcginnis: been in since https://review.openstack.org/557370 merged 2018-04-1215:32
funginot sure how it hasn't been noticed before now15:32
dhellmannsmcginnis : the new release job doesn't try to install anything, so we could get them to switch to that15:33
smcginnisdhellmann: tl;dr networking-bgpvpn requires networking-odl. Networking-odl has a URL requirement for ceilometer, not published to pypi. Doesn't appear to work for a dependency of a dependency.15:33
fungidid they have any successful rc builds before this?15:33
dhellmannas far as the siblings thing goes, I don't really understand what the plan is there, from the infra team's side :-/15:33
smcginnisfungi: Odd, they had all milestones and an RC1.15:34
smcginnisRe-enqueue again and see if somehow it works again? Not sure how it would have passed before.15:34
fungiohh... so what if networking-odl is using the new release jobs which don't hit this because they don't install the project, released, then the networking-bgpvpn release tried to use it?15:34
*** bobh has quit IRC15:35
smcginnisHmm, maybe.15:35
fungicould this be the first time networking-bgpvpn has depended on a networking-odl release after april?15:35
smcginnisThe RC1 should have gotten it at least.15:35
dhellmannnetworking-odl uses publish-to-pypi-neutron15:35
dhellmannso not the new job15:35
fungistumped, but still thinking15:36
dhellmannI was suggesting switching networking-bgpvpn to the new job as a way to work around this15:36
dhellmannwhen was that dependency chain established?15:36
smcginnis5 months ago?15:37
smcginnisNo, before that.15:37
dhellmannok, so it's not a question of timing15:37
*** jtomasek has quit IRC15:38
dhellmanncould someone have deleted ceilometer releases from pypi?15:38
*** ykarel_ has joined #openstack-release15:38
smcginnisThis is what's there now - https://pypi.org/project/ceilometer/#history15:39
smcginnis201215:39
*** ykarel has quit IRC15:39
smcginnisNot sure if there's a way to find history of deleted things.15:39
dhellmannunlikely15:40
fungii don't think we ever pushed any newer ceilo releases than that to pypi15:41
funginetworking-odl 13.0.0 was the first release to incorporate that dependency15:42
dhellmannah!15:43
dhellmannthe rc1 wouldn't have been used by pip15:43
dhellmannso this is in fact triggered by the odl release15:43
fungiyeah15:43
dhellmannbecause now there's a new thing for bgpvpn to pull in from pypi15:43
dhellmannand that thing has a bad requirements list15:44
dhellmannok, so that explains it15:44
fungii don't see networking-odl in upper-constraints.txt15:44
fungiso presumably started to be used as soon as it was released15:44
smcginnisSo -odl needs to be fixed and a stable release out, then we can retrigger -bgpvpn?15:44
fungilooks like it15:44
smcginnisThen next question - how can it be fixed if they want to pull in something not on pypi?15:45
dhellmann(a) I'm not sure the ceilometer team wants that code used as a library (b) there is no way to do that if it is not on pypi15:45
dhellmannan alternative fix would be to switch the release job for -bgpvpn15:46
smcginnisAgreed15:46
*** olivier__ has quit IRC15:46
dhellmannbecause the new job doesn't try to install anything15:46
fungilooks like networking-odl should be treated as a library from a release schedule standpoint too if networking-bgpvpn has it as a dependency?15:46
smcginnisAlso agreed.15:46
dhellmannyeah, I have not been able to convince the neutron team to do anything like that in the past15:46
smcginnisdhellmann: Do you know what needs to happen for that job change?15:46
fungiwe would have caught this during the networking-bgpvpn rc builds if networking-odl were released when the other libs were15:47
dhellmannyeah, let me work that patch up15:47
smcginnismlavalle: Are you free?15:47
mlavallesmcginnis: yes15:47
fungi"fixing" this by switching to the new style release jobs still means you can't actually pip install networking-odl (and by proxy networking-bgpvpn)15:48
dhellmannlet me make sure I do the right one: we want to change the release job for networking-bgpvpn, correct?15:48
smcginnismlavalle: We have an issue that didn't surface until processing the final release for networking-bgpvpn.15:48
smcginnismlavalle: It appears they have a requirement on networking-odl which is incorrently referring to a git URL for something not published on pypi.15:48
dhellmannfungi : yeah, that's why I keep objecting to "fixing" this with tox-siblings or similar things in the gate15:48
*** olivierb_ has joined #openstack-release15:48
smcginnismlavalle: So multiple issues here, but main thing is, it appears networking-odl is actually a library and should have been released a few weeks ago. Then we would have identified this issue sooner.15:49
mlavalleok, so it is not really a library15:50
fungiright, tox-siblings isn't a solution to this problem, only a workaround. the solution is to not depend on projects which aren't released to pypi (and/or start releasing more things to pypi)15:50
smcginnismlavalle: Do you know if that can be switched to a library and have the non-client lib release deadline a few weeks ahead of final RCs?15:50
smcginnismlavalle: If it's not a library, is bgpvpn incorrectly using it as one?15:50
mlavalleodl is a self standing Neutron plugin that can be deployed on its own15:50
ttxif it's depended on by something else, we file it under "library"15:50
ttxso the two bits should probably be split15:51
ttxin a lib + two things that consume it15:51
mlavalleI will have have to talk neutworking-bgpvn to see what is going on there15:51
ttxmaybe they just identified interesting code in -odl and decided to import it15:52
dhellmannsmcginnis , fungi , mlavalle : https://review.openstack.org/59821315:52
smcginnismlavalle: There are two issues that need to be addressed.15:53
smcginnismlavalle: One is that this is not allowed - https://github.com/openstack/networking-odl/blob/master/requirements.txt#L2115:54
smcginnismlavalle: The other is that -bgpvpn has a requirement on -odl, effectively making odl a library.15:54
dhellmannreally, we have several things that treat neutron the same way15:54
*** ykarel_ is now known as ykarel15:55
mlavalleso, let me make sure I understand:15:56
mlavalleto unblock the release, we have https://review.openstack.org/#/c/598213/15:57
mlavalleand then we need to fix https://github.com/openstack/networking-odl/blob/master/requirements.txt#L2115:57
mlavalleand see how to handle the dependency of bgpvpn on odl15:57
smcginnismlavalle: Yeah, I think that's it.15:58
mlavalledoes this summarize the situation?15:58
dhellmannyes. any time we have a git URL in a requirements file that's an indication that we have something that is not releasing frequently enough to serve as a library, even though it is being used that way.15:58
dhellmannthis is why oslo does weekly releases of its libraries, for example15:59
* dhellmann needs to find some lunch15:59
mlavalleok.... now is ceilometer going to be in PyPI as suggested in https://github.com/openstack/networking-odl/blob/master/requirements.txt#L21?16:00
*** ykarel has quit IRC16:00
*** pkovar has quit IRC16:01
dhellmannwe'll be switching to pushing all python packages to pypi for stein, so it will be a while16:03
ttxmlavalle: even if it was on Pypi we would rather avoid depending directly on it ?16:03
dhellmannbut you could ask the ceilometer team to switch early16:03
dhellmannwhether the ceilometer team wants it to be used as a library or not is really up to them16:03
ttxdhellmann: one issue is that if people depend on it from a release cycle perspective it becomes a lib16:04
mlavallebecause being used as a library implies releasing frequently16:04
ttximho16:04
smcginnisYeah16:04
dhellmannttx: I agree.16:04
ttxwell, you need to catch integration issues16:04
dhellmannmlavalle : right, that's why you need to talk to the ceilometer team16:04
ttxso that you don't break the people who depend on you on release day16:04
ttxI'd advise to split the code into a proper lib and a proper service16:05
mlavalledhellmann: and also to the odl guys. they have a commitment to bgpvpn and maybe are not even aware of it16:05
dhellmannright16:05
mlavalleok, understood16:05
smcginnisThanks mlavalle16:05
ttx"from ceilometer.network.statistics import driver"16:07
*** zxiiro-away is now known as zxiiro16:09
*** jpich has quit IRC16:10
*** alexchadin has quit IRC16:13
*** ykarel has joined #openstack-release16:14
*** ykarel has quit IRC16:17
*** luizbag has joined #openstack-release16:25
*** ykarel has joined #openstack-release16:25
openstackgerritMatt Riedemann proposed openstack/releases master: Fix neutron rocky release highlight about port bindings  https://review.openstack.org/59822416:26
mriedemmlavalle: ^16:26
mlavallemriedem: thanks, +1ed it16:27
smcginnisfungi, dhellmann: https://review.openstack.org/#/c/598213/ merged, should we try re-enqueuing that bgpvpn release again and see if that took care of it?16:28
*** ykarel_ has joined #openstack-release16:29
fungiwill do16:29
fungiassuming dhellmann is good with it too16:30
*** ykarel has quit IRC16:31
smcginnisProbably out to lunch.16:34
fungii'll roll the dice16:34
fungihave an important e-mail to send and then need to get to lunch/errands myself16:34
fungijust now reran the same enqueue-ref subcommand from before16:34
smcginnisOK, worse case it just fails again.16:35
*** bnemec has quit IRC16:35
ttxOK I'll drop for a couple of hours, looks like you have things under control16:37
smcginnisThanks ttx16:38
smcginnisLooks like it passed.16:38
fungigot further anyway16:39
fungiand done16:39
fungiyep, i think we're good there?16:39
fungihttp://logs.openstack.org/44/440fa516b472bc4d3a59d06c5bf571f98d44f1a5/release/release-openstack-python3/ad3869b16:40
smcginnisIt appears so. At least for the critical parts.16:40
fungithe ara report looks green16:40
* fungi moves on to stirring up trouble on the mailing lists16:40
smcginnisStill has release-openstack-python in the list of jobs, so that will fail. Although now it should see it's already done?16:40
*** rmart04 has quit IRC16:41
* dhellmann returns16:44
openstackgerritMerged openstack/releases master: Fix neutron rocky release highlight about port bindings  https://review.openstack.org/59822416:46
smcginnisdhellmann: Not sure why we still got a release-openstack-python job in there in addition to the release-openstack-python3 one.16:48
dhellmannhmm16:49
smcginnisAnyway, reported failed, but I think everything worked that needed to.16:51
*** ykarel_ has quit IRC16:53
dhellmannsmcginnis : oh, I see why. They have variants that list required-projects for those jobs16:55
dhellmannI'll clean that up16:55
dhellmanneven on my tall monitor that was below the fold in the project file16:55
*** ykarel has joined #openstack-release16:55
*** ykarel has quit IRC16:55
dhellmannsmcginnis : https://review.openstack.org/59823216:56
*** shardy has quit IRC16:56
smcginnisOh, I see.16:57
* ttx is back due to an unexpected meeting16:57
smcginnis:(16:59
ttxsmcginnis: I brought my drink though17:00
smcginnis+1!17:01
*** ykarel has joined #openstack-release17:01
*** dtantsur is now known as dtantsur|afk17:04
*** ykarel_ has joined #openstack-release17:06
*** ykarel has quit IRC17:08
*** mriedem is now known as mriedem_afk17:11
*** aojea has quit IRC17:11
*** annabelleB has quit IRC17:13
*** bnemec has joined #openstack-release17:17
dhellmannsmcginnis : now that the release is done, how do you feel about doing the zuul migration for the release team repos?17:28
*** ykarel_ is now known as ykarel|away17:32
*** annabelleB has joined #openstack-release17:34
*** ykarel|away has quit IRC17:36
*** ricolin has quit IRC17:41
*** dmellado has joined #openstack-release17:44
*** stevebaker has joined #openstack-release17:48
*** electrofelix has quit IRC17:50
*** zxiiro is now known as zxiiro-away17:57
*** annabelleB has quit IRC17:58
*** annabelleB has joined #openstack-release18:06
*** rosmaita has quit IRC18:24
smcginnisdhellmann: Sure, probably a good time to get that out of the way.18:26
smcginnisShould we get this one through soonish? https://review.openstack.org/#/c/598160/18:26
*** bobh has joined #openstack-release18:27
*** annabelleB has quit IRC18:42
dhellmannsmcginnis : we probably should, but we'll want to warn the cycle-trailing teams that we've done it so when they use the new-release tool they want to specify the --series argument18:45
dhellmannI will prepare the zuul migration patches for the release team now18:45
smcginnisOh right. Since we give them two months now, probably not worth waiting for the trailing ones to complete.18:46
dhellmannoh, yesh, I forgot about that18:47
dhellmannI don't want to wait 2 months to change the release jobs18:47
dhellmannI'm going to have to change the goal plan18:47
openstackgerritDoug Hellmann proposed openstack/releases master: import zuul job settings from project-config  https://review.openstack.org/59830618:48
openstackgerritDoug Hellmann proposed openstack/releases master: switch documentation job to new PTI  https://review.openstack.org/59830718:48
openstackgerritDoug Hellmann proposed openstack/releases master: add python 3.6 unit test job  https://review.openstack.org/59830818:48
*** toabctl has quit IRC18:51
openstackgerritDoug Hellmann proposed openstack/releases master: add python 3.6 unit test job  https://review.openstack.org/59830818:56
*** annabelleB has joined #openstack-release18:56
*** annabelleB has quit IRC19:03
*** toabctl has joined #openstack-release19:07
*** mriedem has joined #openstack-release19:11
*** mriedem_afk has quit IRC19:12
openstackgerritMerged openstack/releases master: deal with missing dates for next-phase  https://review.openstack.org/59820719:18
*** olivier__ has joined #openstack-release19:19
*** olivierb_ has quit IRC19:20
*** e0ne has joined #openstack-release19:34
e0nesmcginnis: hi Sean. I'm looking into the releases repo changelog. does release team do releases for all components and PTL has no to do?19:38
smcginnise0ne: Not sure what you mean. If I understand what you are asking right, PTLs just propose the deliverable file patch and merging that takes care of all the release tasks.19:42
smcginnise0ne: At least for official/governed projects.19:42
smcginnise0ne: Non-governed projects need to take care of release tasks themselves.19:43
smcginnise0ne: Does that answer your question? Or am I way off from what you were looking for? :)19:43
e0nesmcginnis: e.g. https://review.openstack.org/#/c/547519/. are you or somebody from release team going to do this for Rocky?19:44
e0nein other words: should I propose a patch to release horizon for rocky cycle?19:44
smcginnise0ne: We do the final release based on the last RC. Horizon was included there - https://review.openstack.org/#/c/597529/2/deliverables/rocky/horizon.yaml19:46
smcginnisWe like to get all PTL's +1 on there, but don't wait for it.19:47
e0nesmcginnis: thank you! now I understand how it works19:48
smcginnis;)19:48
*** olivier__ has quit IRC19:54
*** bobh has quit IRC20:06
*** bobh has joined #openstack-release20:08
dhellmann?ptl I18n20:24
dhellmann:-(20:24
*** mugsie has quit IRC20:25
smcginnisI need to fix the auth stuff. Probably the same for the PTG bot and others.20:28
smcginnisI18n: Frank Kloeker (eumel8)20:28
smcginnis:)20:28
*** e0ne has quit IRC20:29
eumel8mhm?20:33
smcginniseumel8: Sorry, dhellmann was checking on I18n earlier I think.20:33
dhellmanneumel8 : yeah, I was verifying you were still PTL before I emailed you :-)20:34
eumel8yes, I'm still PTL because of the Berlin Summit in Stein. The team decided, it's not my fault ;)20:35
dhellmannhaha20:35
dhellmannthat's the best possible reason20:35
dhellmannanyway, I emailed you the question I had because I assumed you'd be offline by now20:35
eumel8good decision, 11 PM - I should be offline20:37
smcginnisGet to sleep!20:38
*** mugsie has joined #openstack-release20:39
eumel8yeah, good night :)20:42
*** annabelleB has joined #openstack-release20:42
*** harlowja has joined #openstack-release20:43
*** bobh has quit IRC20:46
*** rmart04 has joined #openstack-release20:47
*** annabelleB has quit IRC21:15
*** rmart04 has quit IRC21:16
*** luizbag has quit IRC21:29
*** dmellado has quit IRC21:31
*** dmellado has joined #openstack-release21:31
*** stevebaker has quit IRC21:36
*** v12aml has quit IRC21:38
*** v12aml has joined #openstack-release21:40
*** stevebaker has joined #openstack-release21:41
*** priteau has quit IRC21:47
*** claudiub has quit IRC21:50
*** v12aml has quit IRC21:50
*** v12aml has joined #openstack-release21:51
*** annabelleB has joined #openstack-release21:57
*** mriedem is now known as mriedem_lawnboy22:17
*** tosky has quit IRC22:23
*** lxkong has joined #openstack-release22:24
*** annabelleB has quit IRC22:35
*** annabelleB has joined #openstack-release22:39
*** annabelleB has quit IRC23:07
*** annabelleB has joined #openstack-release23:29
*** mlavalle has quit IRC23:34
*** annabelleB has quit IRC23:51
*** annabelleB has joined #openstack-release23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!