| *** ricolin has joined #openstack-release | 00:57 | |
| *** mriedem_afk has quit IRC | 01:11 | |
| *** ricolin has quit IRC | 01:33 | |
| *** whoami-rajat has joined #openstack-release | 02:02 | |
| *** hongbin has joined #openstack-release | 02:15 | |
| *** ricolin has joined #openstack-release | 02:23 | |
| *** ricolin has quit IRC | 02:33 | |
| openstackgerrit | Mohammed Naser proposed openstack/releases master: governance: release 0.4.0 https://review.openstack.org/643220 | 02:40 |
|---|---|---|
| openstackgerrit | Hongbin Lu proposed openstack/releases master: Release zun-ui 3.0.0 https://review.openstack.org/641914 | 02:46 |
| *** dave-mccowan has joined #openstack-release | 02:49 | |
| *** armstrong has joined #openstack-release | 02:50 | |
| *** armstrong has left #openstack-release | 02:51 | |
| *** armstrong has quit IRC | 02:51 | |
| *** dave-mccowan has quit IRC | 02:54 | |
| openstackgerrit | Hongbin Lu proposed openstack/releases master: Release zun-tempest-plugin 3.0.0 https://review.openstack.org/643223 | 03:00 |
| *** ykarel|away has joined #openstack-release | 03:04 | |
| *** ykarel|away has quit IRC | 03:22 | |
| *** hongbin has quit IRC | 03:36 | |
| *** diablo_rojo has quit IRC | 03:47 | |
| *** ricolin has joined #openstack-release | 04:02 | |
| *** udesale has joined #openstack-release | 04:22 | |
| *** ykarel|away has joined #openstack-release | 04:24 | |
| *** ykarel|away is now known as ykarel | 04:24 | |
| *** lbragstad has quit IRC | 05:00 | |
| *** armax has quit IRC | 05:35 | |
| *** jtomasek has joined #openstack-release | 06:15 | |
| *** witek has joined #openstack-release | 06:43 | |
| *** shardy has quit IRC | 06:44 | |
| *** witek has quit IRC | 06:47 | |
| *** shardy has joined #openstack-release | 06:52 | |
| *** tosky has joined #openstack-release | 07:23 | |
| *** nhbinh_ has quit IRC | 07:24 | |
| *** nhbinh_ has joined #openstack-release | 07:24 | |
| *** witek has joined #openstack-release | 07:30 | |
| *** ifat_afek has joined #openstack-release | 07:44 | |
| *** e0ne has joined #openstack-release | 07:45 | |
| *** tesseract has joined #openstack-release | 08:07 | |
| *** pcaruana has joined #openstack-release | 08:07 | |
| *** hberaud|gone is now known as hberaud | 08:09 | |
| *** pcaruana has quit IRC | 08:11 | |
| *** pcaruana has joined #openstack-release | 08:23 | |
| *** witek has quit IRC | 08:33 | |
| *** dtantsur|afk is now known as dtantsur | 08:59 | |
| *** ykarel is now known as ykarel|lunch | 09:09 | |
| *** jpich has joined #openstack-release | 09:24 | |
| dtantsur | morning folks! what's the current position of releasing anything that already got stable/stein? | 09:27 |
| dtantsur | https://github.com/openstack/rsd-lib/commit/e3a061f4d2bf70e3a2f9411f810d328c065658f1 has just been raised to my attention | 09:27 |
| dtantsur | it makes rsd-lib incompatible with the stein sushy, and I'd like to resolve it by issuing a new release | 09:28 |
| dtantsur | ttx: morning, maybe you're around ^^ | 09:28 |
| openstackgerrit | Dmitry Tantsur proposed openstack/releases master: Release sushy 1.8.1 for Stein https://review.openstack.org/643258 | 09:33 |
| dtantsur | the request in question ^^^ | 09:34 |
| *** ifat_afek has quit IRC | 09:41 | |
| *** e0ne has quit IRC | 09:43 | |
| *** e0ne has joined #openstack-release | 09:46 | |
| *** pcaruana has quit IRC | 09:48 | |
| ttx | dtantsur: looking | 09:52 |
| *** ykarel|lunch has quit IRC | 09:52 | |
| ttx | Bug fix releases should be fine | 09:53 |
| ttx | +2ed, waiting for another reviewer to approve | 09:55 |
| *** ykarel has joined #openstack-release | 09:57 | |
| dtantsur | thanks! | 10:01 |
| openstackgerrit | Merged openstack/releases master: governance: release 0.4.0 https://review.openstack.org/643220 | 10:21 |
| openstackgerrit | Merged openstack/releases master: Release openstackdocstheme 1.29.2 https://review.openstack.org/642908 | 10:21 |
| openstackgerrit | Merged openstack/releases master: Release Puppet OpenStack Stein 3 https://review.openstack.org/642477 | 10:24 |
| openstackgerrit | Merged openstack/releases master: Release solum-dashboard 2.6.0 https://review.openstack.org/642665 | 10:24 |
| openstackgerrit | Merged openstack/releases master: Release solum-tempest-plugin 1.0.0 https://review.openstack.org/642666 | 10:24 |
| openstackgerrit | Merged openstack/releases master: Release murano-tempest-plugin 1.0.0 https://review.openstack.org/642669 | 10:27 |
| *** nhbinh_ has quit IRC | 10:27 | |
| openstackgerrit | Merged openstack/releases master: TripleO stein-3 release https://review.openstack.org/642475 | 10:28 |
| openstackgerrit | Merged openstack/releases master: Release murano-agent 3.6.0 https://review.openstack.org/642670 | 10:28 |
| *** luizbag has joined #openstack-release | 10:30 | |
| openstackgerrit | Merged openstack/releases master: Add stein cycle highlights for ironic https://review.openstack.org/642824 | 10:30 |
| openstackgerrit | Merged openstack/releases master: TripleO: minor release for rocky https://review.openstack.org/637858 | 10:32 |
| openstackgerrit | Thierry Carrez proposed openstack/releases master: Create stable/stein branches for all libraries https://review.openstack.org/643268 | 10:34 |
| ttx | hmm a bunch pf release fails | 10:47 |
| *** udesale has quit IRC | 10:56 | |
| *** udesale has joined #openstack-release | 10:56 | |
| *** jpich has quit IRC | 11:01 | |
| *** jpich has joined #openstack-release | 11:02 | |
| *** binh_ has joined #openstack-release | 11:03 | |
| ttx | We should hold on future releases until that is resolved: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003827.html | 11:04 |
| *** shardy has quit IRC | 11:11 | |
| *** smrcascao has joined #openstack-release | 11:12 | |
| *** udesale has quit IRC | 11:15 | |
| *** udesale has joined #openstack-release | 11:15 | |
| *** zbr has quit IRC | 11:16 | |
| ttx | I posted analysis for all issues on openstack-discuss. I suspect most are linked to a bionic switch /cc: fungi | 11:17 |
| * ttx lunches | 11:17 | |
| *** udesale has quit IRC | 11:27 | |
| *** udesale has joined #openstack-release | 11:28 | |
| *** pcaruana has joined #openstack-release | 11:32 | |
| *** smrcascao has quit IRC | 11:40 | |
| *** dave-mccowan has joined #openstack-release | 11:44 | |
| EmilienM | ttx: hey, not sure what to do yet with the tripleo-ui issue | 11:53 |
| EmilienM | jaosorior: ^ fyi [Release-job-failures] Release of openstack/tripleo-ui failed | 11:54 |
| jaosorior | EmilienM: should we poke folks from the ui team? | 11:54 |
| ttx | EmilienM: it's probably an issue in the job definition that makes it reliant on -xenial hosts | 11:54 |
| ttx | so on our side, not yours | 11:54 |
| ttx | EmilienM: I'll let you know if we need your help :) | 11:55 |
| EmilienM | jaosorior: they have no clue anyway | 11:55 |
| EmilienM | ttx: ok, please ping me or Alex or Juan | 11:56 |
| EmilienM | jaosorior: I think we should retire tripleo-ui soon | 11:57 |
| EmilienM | jaosorior: the time we spend on releasing it isn't worth it | 11:57 |
| EmilienM | ttx: fyi tripleo-ui is retired in our downstream product | 11:57 |
| EmilienM | mainteners re-shifted focus and we also lack of users | 11:58 |
| *** zbr has joined #openstack-release | 11:59 | |
| ttx | ok | 12:13 |
| ttx | might explain why the job is stale | 12:13 |
| *** ifat_afek has joined #openstack-release | 12:14 | |
| jaosorior | EmilienM: ack. now we gotta consider if we retire it now, or for the next release. | 12:17 |
| EmilienM | honestly I would do it now | 12:24 |
| EmilienM | jrist: ^ ? | 12:24 |
| jrist | reading back | 12:25 |
| jrist | lol EmilienM it's true | 12:26 |
| jrist | tripleo-ui is retired from 14 | 12:26 |
| EmilienM | translation: rocky | 12:26 |
| jrist | so yeah... I think we're fine retiring now | 12:26 |
| EmilienM | jrist: and who is doing it? | 12:26 |
| jrist | it was on Beth's plate | 12:27 |
| EmilienM | it would be great if you folks do it, it's your project ;-) | 12:27 |
| jrist | I can submit some patches if needed | 12:27 |
| EmilienM | jrist: https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project | 12:27 |
| openstackgerrit | Tom Barron proposed openstack/releases master: Added Stein ycle-highlights for manila https://review.openstack.org/643311 | 12:28 |
| jrist | EmilienM: I'll push some patches today for that | 12:28 |
| EmilienM | good | 12:28 |
| jrist | I'm not core tho :) will need some eyes | 12:28 |
| *** e0ne has quit IRC | 12:31 | |
| *** hberaud is now known as hberaud|lunch | 12:33 | |
| openstackgerrit | Tom Barron proposed openstack/releases master: Added Stein cycle-highlights for manila https://review.openstack.org/643311 | 12:43 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: look at teams in the branch being scanned https://review.openstack.org/643317 | 12:45 |
| openstackgerrit | Merged openstack/releases master: Put the lists under a directory https://review.openstack.org/642864 | 12:53 |
| *** mriedem has joined #openstack-release | 13:08 | |
| fungi | thanks ttx, i'll see what jumps out at me but if it's a ui project then my first guess is npm | 13:13 |
| ttx | fungi: there are three separate issues, including one with shred. See my emails to list when you get a second :) | 13:14 |
| fungi | on it | 13:14 |
| *** smrcascao has joined #openstack-release | 13:15 | |
| ttx | While I like to blame NPM for everything from Brexit to Trump, the tripleo-ui project fail is due to Ansible task failing with "hostvars['ubuntu-xenial']" is undefined -- so I'm not sure we can blame it on NPM | 13:15 |
| fungi | okay, so looks like the gnupg shred issue is that we try to `shred -u ~/.gnupg/*` and starting in bionic some files in that tree got moved to a subdir. i'll patch that to use find instead i guess | 13:24 |
| ttx | shred really needs a -r | 13:24 |
| fungi | yeah, that'd be nice | 13:25 |
| *** lbragstad has joined #openstack-release | 13:26 | |
| ttx | but then shred is not very efficient on journalled filesystems anyway | 13:26 |
| *** hberaud|lunch is now known as hberaud | 13:30 | |
| *** e0ne has joined #openstack-release | 13:32 | |
| fungi | it's true, more of a belt-and-suspenders approach | 13:32 |
| fungi | anyway, https://review.openstack.org/643328 should be the fix for that | 13:32 |
| ttx | lgtm | 13:36 |
| fungi | i have doubts the rsync one is related to the ubuntu bionic switch | 13:40 |
| fungi | this looks like the executor trying to rsync files off the job node and getting surprised by a new ssh host key, which if it were related to ubuntu-bionic it would be impacting loooooots of jobs. more often when we see that behavior it's because of a rogue vm in the provider of which nova has lost track and so reassigned its ip address to our boot request, then they have an arp fight on the network | 13:42 |
| fungi | and some connections end up at the rogue | 13:42 |
| ttx | so could be transient | 13:43 |
| fungi | yes, my best guess is that build was simply unlucky | 13:44 |
| fungi | i'll dig into the tripleo-ui one shortly; that sounds more complicated | 13:45 |
| *** ricolin has quit IRC | 13:59 | |
| *** armstrong has joined #openstack-release | 14:02 | |
| *** ykarel is now known as ykarel|away | 14:05 | |
| *** ykarel|away has quit IRC | 14:14 | |
| *** mlavalle has joined #openstack-release | 14:19 | |
| *** ifat_afek has quit IRC | 14:23 | |
| fungi | yeah, i think after staring at it a bit i'm going to see if mordred has any ideas where that hostvars reference might be cropping up, though i suspect we need to do nodeset:ubuntu-xenial on release-openstack-javascript at http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n840 for now | 14:24 |
| mordred | I didn't do it | 14:25 |
| fungi | ah! here is is... http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/javascript/release.yaml#n14 | 14:26 |
| fungi | i guess we shouldn't be hard-coding the nodename in the playbook | 14:26 |
| mordred | fungi: wow. yeah | 14:26 |
| mordred | but also ... | 14:26 |
| mordred | ok. yeah. that would be teh ting- the default node_version for install_nodejs should be fine | 14:27 |
| fungi | ahh, even the initial roles list was skipped because we hard-coded the node name there too | 14:27 |
| mordred | so just updating that to ubuntu-bionic should be fine | 14:27 |
| mordred | fungi: it's possible our decision to call the nodes teh same thing as their label might not have been the best life choice | 14:28 |
| fungi | looks like all the other playbooks in there use hosts:all | 14:28 |
| fungi | it's just release.yaml which does hosts:ubuntu-xenial | 14:28 |
| ttx | indeed | 14:29 |
| fungi | so how should the upload-npm role on localhost get the project_ver from the job node's hostvars? i'm not familiar enough with ansible variable handling | 14:30 |
| fungi | i mean, short of just changing the hard-coded node name there | 14:31 |
| mordred | thikning | 14:32 |
| mordred | fungi: we could just re-run version-from-git on localhost and set zuul_work_dir to the src dir location on the executor | 14:35 |
| fungi | i suppose there's no way to do an inline lookup for the name of the (in this case only) hosts entry in the inventory | 14:35 |
| mordred | fungi: we could also maybe set hosts: all and then do include_role: upload-npm delegate_to: localhost | 14:36 |
| mnaser | we're adding support for a new service (mistral) in openstackansible and we want to bring the support ointo stable/rocky (osa doesn't follow stable policy). i'm wondering how i can branch stable/rocky right now? should i make a release repo change? | 14:37 |
| fungi | oh, interesting. and that would implicitly reference the project_ver right? | 14:37 |
| mordred | I think so? | 14:37 |
| *** armax has joined #openstack-release | 14:37 | |
| odyssey4me | mnaser you can just branch it in gerrit as far as I know | 14:37 |
| mnaser | https://review.openstack.org/#/admin/projects/openstack/openstack-ansible-os_mistral,branches | 14:37 |
| fungi | mnaser: you don't already have a stable/rocky branch? | 14:38 |
| mnaser | i cant see anything there | 14:38 |
| mnaser | fungi: no, we made this new 'role' recently | 14:38 |
| mordred | fungi: https://review.openstack.org/643353 Use include_role and delegate_to for upload-npm | 14:38 |
| mordred | fungi: strawman | 14:38 |
| mordred | fungi: I have not done any testing to see if that works | 14:38 |
| fungi | thanks mordred! | 14:38 |
| fungi | mnaser: what odyssey4me is possible if you have create permission for refs/heads in the gerrit acl for that project, but my guess is the release team would prefer a change to openstack/releases if that project is under release management | 14:40 |
| fungi | er, what odyssey4me says | 14:40 |
| mnaser | yeah, so i'm going to guess that we'll have to go through the normal release process, but i'm wondering whats the "path" to branch out much later in this case | 14:40 |
| odyssey4me | ah yeah, so there's likely a formal release process way of doing things | 14:40 |
| mnaser | afaik our roles are already released as _independent so we can just create a branch now .. but i rather check with release team first to hear what they mention | 14:41 |
| *** mlavalle has quit IRC | 14:48 | |
| ttx | mnaser: we have a meeting (just after the TC office hour) we could cover that during it | 14:48 |
| mnaser | ttx: great, thanks, ill try to be there | 14:48 |
| *** mordred has quit IRC | 14:51 | |
| *** luizbag has quit IRC | 14:54 | |
| *** mordred has joined #openstack-release | 14:57 | |
| *** luizbag has joined #openstack-release | 15:08 | |
| *** luizbag has quit IRC | 15:10 | |
| smcginnis | Oh wow, much scrollback. | 15:14 |
| openstackgerrit | Kamil Sambor proposed openstack/releases master: Change 637405 - Merged https://review.openstack.org/643365 | 15:16 |
| *** luizbag has joined #openstack-release | 15:17 | |
| smcginnis | ? | 15:18 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release tripleo-common 10.6.0 https://review.openstack.org/643365 | 15:19 |
| *** mordred has quit IRC | 15:23 | |
| *** mriedem is now known as mriedem_away | 15:24 | |
| *** mordred has joined #openstack-release | 15:30 | |
| ttx | smcginnis: summary is: release jobs are currently broken, fix in progress | 15:35 |
| ttx | smcginnis: how is glass-blowing going ? | 15:36 |
| *** mlavalle has joined #openstack-release | 15:36 | |
| smcginnis | Hah! | 15:40 |
| smcginnis | That's tomorrow. :P | 15:40 |
| ttx | Schedule has it on Monday... Maybe you weren't invited. | 15:40 |
| smcginnis | Monday and Friday to catch the folks who come late or leave early. | 15:41 |
| smcginnis | I couldn't get in until Monday night. | 15:41 |
| smcginnis | And for the record, I chose the horseback riding. | 15:41 |
| *** armstrong has left #openstack-release | 15:42 | |
| ttx | hah! | 15:42 |
| smcginnis | I did get to have a Ridge Vineyards wine last night - https://en.wikipedia.org/wiki/Judgment_of_Paris_(wine) | 15:44 |
| prometheanfire | smcginnis: did paunch get a FFE? used in heat-agents and tripleo | 15:45 |
| prometheanfire | https://review.openstack.org/643275 | 15:45 |
| ttx | smcginnis: that sounds like an anti-French statement | 15:45 |
| smcginnis | ttx: Non! :) | 15:47 |
| smcginnis | prometheanfire: Not that I've seen. | 15:47 |
| smcginnis | Looks like we had a bunch of stuff go through last night. I haven't taken a look through it all yet | 15:48 |
| prometheanfire | smcginnis: k, the triple-o stuff is probably fine, but technically still needs an FFE (and paunch was with that group) | 15:48 |
| smcginnis | jaosorior: ^^ | 15:58 |
| smcginnis | #startmeeting releaseteam | 16:00 |
| ttx | meeting time! | 16:00 |
| openstack | Meeting started Thu Mar 14 16:00:19 2019 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: releaseteam)" | 16:00 | |
| openstack | The meeting name has been set to 'releaseteam' | 16:00 |
| smcginnis | Ping list: smcginnis ttx dhellmann diablo_rojo hberaud evrardjp fungi armstrong | 16:00 |
| ttx | o/ | 16:00 |
| hberaud | o/ | 16:00 |
| smcginnis | #link https://etherpad.openstack.org/p/stein-relmgt-tracking | 16:00 |
| dhellmann | o/ | 16:00 |
| diablo_rojo_phon | o/ | 16:00 |
| fungi | aloha | 16:01 |
| smcginnis | Looks like ttx has us set with a long agenda today (thanks!) so I think we can get going. | 16:01 |
| smcginnis | #topic Standing release automation issues | 16:01 |
| *** openstack changes topic to "Standing release automation issues (Meeting topic: releaseteam)" | 16:01 | |
| ttx | So I approved a bunch of stuff this morning | 16:01 |
| smcginnis | This is related to switching to bionic, right> | 16:02 |
| ttx | and then was assaulted by release-job-failures | 16:02 |
| ttx | smcginnis: partially yes | 16:02 |
| ttx | Three issues | 16:02 |
| fungi | you were our guinea pig! | 16:02 |
| ttx | which I described in several ML posts | 16:02 |
| fungi | can't say i didn't warn you to be on the lookout ;) | 16:02 |
| ttx | First one is solved by https://review.openstack.org/#/c/643328/ | 16:02 |
| ttx | which should merge asap | 16:03 |
| ttx | The governance one is likely transient, we'll reenqueue when https://review.openstack.org/#/c/643328/ lands | 16:03 |
| *** mlavalle has quit IRC | 16:03 | |
| ttx | The tripleo-ui one is due to a hardcoded host mention | 16:03 |
| ttx | fungi: do we ahve a fix in flight for this one? | 16:03 |
| fungi | ttx: yes, i replied to the ml with it | 16:04 |
| ttx | Hmm https://review.openstack.org/#/c/643353/ | 16:04 |
| fungi | let me find that | 16:04 |
| ttx | got it | 16:04 |
| smcginnis | Doesn't look like that one is happy. | 16:04 |
| fungi | eww, look at all the lint | 16:04 |
| *** diablo_rojo has joined #openstack-release | 16:05 | |
| ttx | AttributeError: 'AnsibleUnicode' object has no attribute 'get' eeewwwww | 16:05 |
| ttx | thanks for nothing ansible-lint | 16:05 |
| smcginnis | Can we assume mordred is on that? | 16:06 |
| ttx | maybe it's its way to complain about spaces in the task name | 16:06 |
| ttx | if that is the case I recommend therapy | 16:06 |
| fungi | yeah, i'm pinging him to help figure it out | 16:07 |
| fungi | over in #openstack-infra | 16:07 |
| ttx | anyway, that one is less urgent | 16:07 |
| smcginnis | So assuming that all merges, are we pretty confident that is all the issues? | 16:07 |
| ttx | since it only affects tripleo-ui | 16:07 |
| ttx | which TripleO is not even sure to include in release | 16:07 |
| ttx | yes | 16:07 |
| smcginnis | OK, good. | 16:08 |
| smcginnis | And nothing to reenqueue? | 16:08 |
| ttx | the governance one should be reenqueued | 16:08 |
| fungi | well, it may affect other javascript-based projects trying to release too | 16:08 |
| ttx | openstack/governance 0.4.0 | 16:08 |
| fungi | so getting it ironed out sooner is better | 16:08 |
| fungi | i'll reenqueue the governance tag and make sure we don't see any recurrence but i'm pretty sure that was a random provider issue | 16:09 |
| smcginnis | We have a list of horizon projects on the agenda. I would assume then we shouldn't release any of those for now. | 16:09 |
| ttx | ++ | 16:09 |
| smcginnis | Alright, sounds like everything is in hand. I'll move to the next topic. | 16:10 |
| fungi | when we get a fix for the observed tripleo-ui release failure merged we can try approving one js-based project release and not a large batch initially, just to see what happens | 16:10 |
| smcginnis | #topic Stein branch creation hold | 16:10 |
| *** openstack changes topic to "Stein branch creation hold (Meeting topic: releaseteam)" | 16:10 | |
| smcginnis | fungi: ++ | 16:10 |
| smcginnis | #link https://review.openstack.org/#/c/643268/ | 16:11 |
| ttx | yeah, just a quick point that while the patch has been created, apparently we should wait for tonyb to try something | 16:11 |
| ttx | I think we should do it sooner rather than later though | 16:11 |
| smcginnis | This is related to having the upper-constraints handling updated to simplify that transition, right? | 16:11 |
| ttx | so do we have an ETA on that tonyb-thing? | 16:11 |
| ttx | smcginnis: yes | 16:11 |
| dhellmann | I don't know about an eta | 16:12 |
| smcginnis | Really too bad this is in the middle of his night. | 16:12 |
| smcginnis | I think we're going to have to switch times again. | 16:12 |
| ttx | something he will have to fix as our new fearless leader | 16:12 |
| smcginnis | Or send Tony a heck of a lot of coffee. | 16:12 |
| ttx | the release management diet | 16:12 |
| smcginnis | So nothing for us to do right now, just awareness that some branching things are on hold pending Tony's update, right? | 16:13 |
| ttx | ok, let's make sure we communicate with him later in the day / earlier in his day to see how far it is and if it's worth waiting | 16:13 |
| smcginnis | Does this mean we should not process any branch creation patches until then. (assuming we can release again soon) | 16:14 |
| ttx | hmm, I'd say no.. but I'm not sure.. dhellmann ? | 16:15 |
| dhellmann | this is the change we need: https://review.openstack.org/643387 | 16:15 |
| *** ykarel has joined #openstack-release | 16:16 | |
| smcginnis | dhellmann: So we should merge that before processing any branch requests? | 16:16 |
| ttx | ah. Probably wait then | 16:16 |
| dhellmann | yes, that's the idea | 16:16 |
| smcginnis | OK. | 16:17 |
| dhellmann | I just did that very quickly, so check my spelling in that URL, please :-) | 16:17 |
| ttx | dhellmann: note that we have a bunch that have already gone in | 16:17 |
| smcginnis | I do see it redirects to master for now: https://releases.openstack.org/constraints/upper/stein | 16:17 |
| ttx | (I list them on my commit message) | 16:17 |
| dhellmann | ttx: it won't be the end of the world | 16:17 |
| smcginnis | Do those ones that have branched need to be updated? Or they are fine for now and we eventually get to where we want to down the road? | 16:17 |
| dhellmann | oh, we may need to update that template with the redirects after the opendev change | 16:17 |
| dhellmann | they're fine for now, the patches just point to something that doesn't exist so they can't merge | 16:18 |
| ttx | ok, I think we can safely move to next topic then | 16:18 |
| dhellmann | or rather they can, but they will break local development if they do | 16:18 |
| dhellmann | so they *shouldn't* merge | 16:18 |
| dhellmann | this change was all about removing that window | 16:18 |
| smcginnis | And that's no different than how we've been for at least the last few releases, so that's fine. | 16:19 |
| dhellmann | oh, I wonder if that branch variable is going to contain "stable/stein" not just "stein" | 16:19 |
| smcginnis | We better find out I guess. | 16:20 |
| smcginnis | I'll move on for now though. | 16:20 |
| smcginnis | #topic deliverables with no intermediary releases yet | 16:20 |
| *** openstack changes topic to "deliverables with no intermediary releases yet (Meeting topic: releaseteam)" | 16:20 | |
| dhellmann | yeah | 16:20 |
| ttx | This one is more FYI | 16:20 |
| smcginnis | Do we need to add --type other and --type horizon-plugin to our process doc? | 16:21 |
| ttx | We have a significant set of things that are cycle-with-intermediary and had no release yet | 16:21 |
| ttx | so we have no fall back if they do not produce anything | 16:21 |
| ttx | I guess we could have switched them to cycle-with-rc around FF like the service stuff | 16:22 |
| smcginnis | Maybe add something that at RC1 they we force them if nothing from the team? | 16:22 |
| smcginnis | Those types have always been a little different. I'm never sure if we should treat them as libs or services or what. | 16:22 |
| ttx | but in a lot of cases (like the tempest-plugins) it's just that there is little interest in releasing them more than once (be it through RCs or intermediary releases) | 16:22 |
| smcginnis | Which I guess is why they are called "other" in the first place. :) | 16:22 |
| ttx | I wonder if we should not have one new release type called "automatic" | 16:23 |
| smcginnis | Horizon ones kind of seem like they should be considered client libs though. | 16:23 |
| dhellmann | the reason for making the tempest plugins -with-intermediary is that nothing can consume a pre-release of a python library in our CI system | 16:23 |
| dhellmann | I don't know if anything does consume those from releases, though | 16:23 |
| dhellmann | at least in CI | 16:23 |
| ttx | dhellmann: making them "cycle-automatic" we would just automatically release them on RC weel for example | 16:23 |
| ttx | week* | 16:24 |
| dhellmann | sure, that works | 16:24 |
| fungi | we certainly _could_ consume prereleases in our jobs if we want (there's a pip option to enable them) | 16:24 |
| ttx | note that I would not encourage puting anything other than .. hmm.. "other" into that mode | 16:24 |
| fungi | it's more been a philosophical choice to prefer testing against releases | 16:24 |
| dhellmann | yeah | 16:24 |
| ttx | horizon-plugins should proabbly be made cycle-with-rc | 16:25 |
| smcginnis | "automatic" would make it explicit, but would we still want to give them the option to release as needed? | 16:25 |
| dhellmann | ttx: maybe cycle-automatic is just cycle-with-intermediary && other then? | 16:25 |
| ttx | maybe | 16:25 |
| ttx | smcginnis: sure | 16:25 |
| ttx | although... | 16:25 |
| ttx | I'd say that -automatic things would be re-released at then end | 16:25 |
| ttx | the* | 16:25 |
| dhellmann | we would have to update list-deliverables to let us query for the absence of a value to build the list of intermediary things that need earlier releases, so a new model might be easier | 16:26 |
| smcginnis | If commits merged since last release | 16:26 |
| ttx | like if you do a release around milestone-1 we would still push a new release at the end of the cycle | 16:26 |
| smcginnis | Or we just add to process.rst that we force release c-w-i --type other like we do for the other intermediary deliverables. | 16:26 |
| dhellmann | yeah | 16:26 |
| dhellmann | yeah, that works, too | 16:27 |
| ttx | anyway, that's more for next cycle | 16:27 |
| ttx | Adding a note at the bottom of the tracking doc to remember | 16:27 |
| smcginnis | Yay, defer to the next PTL> :) | 16:27 |
| ttx | For this cycle... I'd say force them at RC1 | 16:27 |
| ttx | or the week after | 16:28 |
| smcginnis | I can add a note to this week's countdown email. | 16:28 |
| ttx | see the task I added to R-2 | 16:28 |
| smcginnis | Great, was going to say that seems like the right timing to me. | 16:28 |
| smcginnis | I will mention it to raise awareness now though. | 16:29 |
| smcginnis | And we would create stable branches with that too? | 16:29 |
| dhellmann | ++ | 16:29 |
| ttx | yes | 16:29 |
| smcginnis | OK, moving on then. | 16:30 |
| smcginnis | #topic mnaser's recent question on the IRC channel | 16:30 |
| *** openstack changes topic to "mnaser's recent question on the IRC channel (Meeting topic: releaseteam)" | 16:30 | |
| ttx | See 14:37 utc and down in the backlog | 16:30 |
| smcginnis | #link http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2019-03-14.log.html#t2019-03-14T14:37:10 | 16:31 |
| ttx | wanting to create a stable/rocky branch on a recently-added deliverable | 16:31 |
| mnaser | Hello | 16:31 |
| ttx | asking how to proceed | 16:31 |
| ttx | mnaser: oh hi sir | 16:31 |
| mnaser | Yep. It’s necessary for us to get it to run and integrate with the rest of the repos | 16:31 |
| smcginnis | It's independent, so I guess just add the branching request there? | 16:32 |
| smcginnis | Odd since it wasn't actually part of that cycle, but these deployment projects are different than the other ones. | 16:32 |
| ttx | is it independent? | 16:32 |
| mnaser | Yep. All our roles are independent indeed so as far as I understood that should be okay? | 16:32 |
| smcginnis | Maybe I read through too fast. | 16:32 |
| dhellmann | yeah, just edit the deliverable file | 16:32 |
| dhellmann | mnaser : do you need to branch the role? or openstack-ansible? | 16:32 |
| mnaser | Yes I believe all our roles are independent. I’m on mobile so I can’t double check | 16:32 |
| mnaser | No, the role: specifically OpenStack-ansible-os_mistral | 16:33 |
| dhellmann | ok, there aren't any branches at all in the deliverable file right now | 16:33 |
| dhellmann | I'm not sure what will happen if you try to just branch one repo | 16:33 |
| dhellmann | so we might need to just do this by hand | 16:33 |
| mnaser | Yep, we added it this cycle.. so it’s a bit tricky | 16:33 |
| ttx | it's not cycle-independent right | 16:33 |
| ttx | ok so | 16:34 |
| ttx | One issue is that in theory we do NOT add deliverable files after a release is done | 16:34 |
| dhellmann | oh, there are *older* deliverable files in the independent directory so I got confused | 16:34 |
| smcginnis | So it isn't independent? I think both statements have been madre. | 16:35 |
| smcginnis | *made | 16:35 |
| dhellmann | smcginnis : it was, but is not now | 16:35 |
| smcginnis | Ah | 16:35 |
| ttx | It's an independent deliverable, as in separate | 16:35 |
| mnaser | Yes. evrardjp worked on making that change | 16:35 |
| ttx | but not cycle-independent model? | 16:35 |
| mnaser | We realized roles don’t really need to be tied in | 16:35 |
| mnaser | Let me verify | 16:35 |
| *** jtomasek has quit IRC | 16:35 | |
| dhellmann | I see deliverables/stein/openstack-ansible-roles.yaml and deliverables/rocky/openstack-ansible-roles.yaml | 16:36 |
| dhellmann | and a couple of individual deliverable in _independent for 2 roles (chrony and redhat-subscription) | 16:36 |
| dhellmann | and deliverables/_independent/openstack-ansible.yaml | 16:36 |
| dhellmann | the change evrardjp made was to stop tagging every role every time there is an OSA release | 16:37 |
| dhellmann | only the main openstack-ansible repo is tagged now | 16:37 |
| dhellmann | they all still need to be branched, though | 16:37 |
| dhellmann | hence the 2 separate deliverable files | 16:37 |
| mnaser | ah, I miss understood then. chrony ans redhat one aren’t ansible ones | 16:37 |
| dhellmann | oh, those may be tripleo | 16:37 |
| mnaser | They are | 16:37 |
| evrardjp | hey | 16:38 |
| mnaser | ok, so within the “release” infrastructure, what we’re doing isn’t necessarily possible? | 16:38 |
| evrardjp | I am in a meeting, sorry I can't join this conversation | 16:38 |
| mnaser | all good :) | 16:38 |
| dhellmann | evrardjp : it's fine, I think we have it | 16:38 |
| smcginnis | We can just blame it all on evrardjp | 16:38 |
| mnaser | I’ll happily take the blame | 16:38 |
| mnaser | It’s 8 degrees Celsius outside. I’m in a happy space. | 16:38 |
| *** e0ne has quit IRC | 16:38 | |
| dhellmann | mnaser : you want to take a new deliverable and create a stable/rocky branch for it, right? | 16:38 |
| dhellmann | and then add that to your rocky release for openstack-ansible? | 16:38 |
| ttx | dhellmann: deliverables/_independent/openstack-ansible.yaml is not branched | 16:38 |
| dhellmann | ttx: right, that's what threw me at first | 16:39 |
| evrardjp | it's been a long time we haven't been independent :) | 16:39 |
| mnaser | dhellmann: ideally. | 16:39 |
| ttx | vi deliverables/rocky/openstack-ansible-roles.yaml | 16:39 |
| ttx | oops | 16:39 |
| mnaser | hah :) | 16:39 |
| dhellmann | I think the thing to do here is to edit deliverables/rocky/openstack-ansible-roles.yaml | 16:39 |
| evrardjp | yeah dhellmann is right, the idea was to still tag osa repo, and stop tagging the rest. | 16:39 |
| evrardjp | so we had to move to a different release file, and defined branchless IIRC | 16:40 |
| evrardjp | sorry, tagless* | 16:40 |
| dhellmann | editing that file to add the new repo and a branch point for it should trigger a new branch in just that repo | 16:40 |
| ttx | ok, so technically you are not adding a new deliverable file, so I think that should work | 16:40 |
| ttx | trying to hole the principles line here | 16:40 |
| ttx | hold even | 16:40 |
| smcginnis | This seems OK. Again, it's a deployment project, so I think it's fine to add something to rocky. | 16:41 |
| dhellmann | I'm willing to be pretty flexible on that for deployment tools | 16:41 |
| evrardjp | I agree with adding the role to osa-roles, if necessary branch manually in gerrit? | 16:41 |
| dhellmann | mnaser: is it clear what you need to do, now? | 16:41 |
| * ttx grabs a drink | 16:41 | |
| dhellmann | evrardjp : we shouldn't need to do anything by hand | 16:41 |
| evrardjp | if necessary | 16:41 |
| evrardjp | hoping it's not | 16:41 |
| smcginnis | ttx: A California wine? | 16:41 |
| ttx | smcginnis: served in a LFOSLS glass | 16:42 |
| smcginnis | lol | 16:42 |
| mnaser | dhellmann: it is and i wanted to get the “everything is alright by release team” sigh off which I seem to have. | 16:42 |
| smcginnis | mnaser: ++ | 16:42 |
| mnaser | I’ll push the change later today. Thank you all. | 16:42 |
| ttx | sigh-off is exact | 16:42 |
| smcginnis | Thanks for bringing it up mnaser | 16:42 |
| dhellmann | yep, +1 from me | 16:42 |
| ttx | It's been a long week, and we are only Thursday | 16:42 |
| smcginnis | Any other topics to cover? | 16:43 |
| mnaser | I need a bigger phone screen. Typing means sigh offs | 16:43 |
| mnaser | Thanks again | 16:43 |
| smcginnis | #topic Open floor | 16:43 |
| *** openstack changes topic to "Open floor (Meeting topic: releaseteam)" | 16:43 | |
| ttx | smcginnis: task review for next week? | 16:43 |
| smcginnis | ttx: Ah, good call. | 16:43 |
| smcginnis | #link https://releases.openstack.org/reference/process.html#rc1-week | 16:44 |
| smcginnis | So we are generating the RC1 release patches? | 16:44 |
| dhellmann | do we want to paste from the process doc into the etherpad? | 16:44 |
| *** ykarel has quit IRC | 16:44 | |
| smcginnis | I'll do that. | 16:44 |
| dhellmann | k | 16:44 |
| dhellmann | some of those devstack-gate instructions are apparently out of date. maybe gmann and fungi can help with those details. | 16:45 |
| smcginnis | I was going to raise that too. | 16:45 |
| smcginnis | We should get the current requirements for that and get process.rst updated. | 16:45 |
| ttx | points 3 and later , we can discuss at next week meeting | 16:45 |
| ttx | since they won't be ready by Thursday morning | 16:46 |
| ttx | I propose to push them to next week meeting agenda | 16:46 |
| fungi | i think the only think devstack-gate needs these days is an entry in the version matrix gile | 16:46 |
| fungi | s/gile/file/ | 16:46 |
| smcginnis | We can move those later, but leaving it there for now to make sure it's seen. | 16:47 |
| ttx | Hmm, point 11 should be moved up | 16:47 |
| ttx | I'll do that | 16:48 |
| gmann | and grenade branch update also | 16:48 |
| gmann | when we cut stable branch | 16:48 |
| fungi | does grenade still rely on devstack-gate for that? | 16:48 |
| gmann | yeah | 16:48 |
| smcginnis | ttx: I do have the cycle-highlight mention in this weeks countdown email, but that could probably use another post of its own. | 16:48 |
| ttx | I'll propose the corresponding process change | 16:48 |
| gmann | zuulv3 job migration is in progress but not merged yet | 16:48 |
| ttx | yeah, around Tuesday-ish | 16:49 |
| ttx | I moved it up as point 2 | 16:49 |
| smcginnis | I'm wondering about our decision to propose the RC1 patches coming from us early in the week. | 16:49 |
| smcginnis | There's usually a crunch for folks getting in those final patches, so us proposing one at the beginning of the week is just going to create extra zuul load that will probably need to get updated with a new hash later anyway. | 16:49 |
| openstackgerrit | Duc Truong proposed openstack/releases master: Add Senlin release highlights https://review.openstack.org/643396 | 16:49 |
| dhellmann | should we wait and do that later in the week? | 16:50 |
| smcginnis | Maybe later on Wednesday? | 16:50 |
| dhellmann | that doesn't give a lot of time for +1 from PTLs | 16:51 |
| smcginnis | I would hope by that point they wouldn't need much time. | 16:51 |
| ttx | done | 16:51 |
| openstackgerrit | Thierry Carrez proposed openstack/releases master: Reorder steps on RC1 week process https://review.openstack.org/643398 | 16:51 |
| fungi | gmann: so http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/roles/test-matrix/files/features.yaml and http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n302 need updating, but that's all for devstack-gate nowadays? | 16:51 |
| ttx | smcginnis: it's a balance | 16:51 |
| ttx | if the RC1s seem to be coming, then waiting is best | 16:52 |
| gmann | fungi: i think so. other part of d-g where legacy jobs rely are static one | 16:52 |
| smcginnis | I suppose we do have more projects now that could be ready already on Monday. | 16:52 |
| ttx | also the announcements tricking down to teh ML are a great reminder | 16:52 |
| ttx | trickling damn | 16:53 |
| smcginnis | I guess that's what we decided and wrote down, so we could just stick with it and see how it goes, then readjust for train if we find it's not optimal. | 16:53 |
| ttx | smcginnis: editing the existing review means less work for ptl too | 16:53 |
| smcginnis | That's true. That could be nice. | 16:53 |
| ttx | hmm.. did we update the ML address for RC announcements ? | 16:54 |
| *** mriedem_away is now known as mriedem | 16:54 | |
| dhellmann | where? | 16:54 |
| ttx | smcginnis: like... one less round for 0.0.0rc1 mistakes | 16:54 |
| smcginnis | Oh, those go to discuss and not announce, right? | 16:54 |
| dhellmann | oh, those go to the -dev/-discuss list | 16:54 |
| ttx | yes² | 16:54 |
| smcginnis | Is that in the job? | 16:54 |
| * ttx is looking | 16:55 | |
| smcginnis | Seems like one of the scripters using codesearch would have gotten that. | 16:55 |
| dhellmann | 94a24d03dc818b4a23d0140863745ef9af590a88 | 16:57 |
| dhellmann | i think that will do it? | 16:57 |
| *** jaosorior has quit IRC | 16:57 | |
| ttx | yes | 16:58 |
| smcginnis | Yep. Great. | 16:58 |
| ttx | i was almost there down in that rabbit hole. | 16:58 |
| ttx | dhellmann: you're fast. It's almost as if you wrote all those things. | 16:58 |
| dhellmann | although I don't see anything at all sending to the release-announce list | 16:58 |
| *** jaosorior has joined #openstack-release | 16:58 | |
| dhellmann | *cough* someone should have updated the comment block at the same time | 16:58 |
| * dhellmann is looking forward to having space in his head for other things | 16:59 | |
| smcginnis | OK, so let's go with generating RC requests early next week as we had previously decided. Anything else we should cover? | 16:59 |
| ttx | sorry for the drill. But then it's always useful to check we still know where to look | 16:59 |
| ttx | smcginnis: nope | 16:59 |
| smcginnis | ++ | 16:59 |
| smcginnis | Nice full hour. Thanks everyone. | 17:00 |
| fungi | thanks smcginnis! | 17:00 |
| smcginnis | #endmeeting | 17:00 |
| *** openstack changes topic to "OpenStack Release Managers office - Come here to discuss how to release OpenStack components - Logged at http://eavesdrop.openstack.org/irclogs/%23openstack-release/" | 17:00 | |
| openstack | Meeting ended Thu Mar 14 17:00:18 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/releaseteam/2019/releaseteam.2019-03-14-16.00.html | 17:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/releaseteam/2019/releaseteam.2019-03-14-16.00.txt | 17:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/releaseteam/2019/releaseteam.2019-03-14-16.00.log.html | 17:00 |
| ttx | Thx smcginnis ! | 17:00 |
| smcginnis | diablo_rojo_phon: Since you've had some experience with it now, would you be intereted in queuing up the RC1 release patches for all cycle-with-rc deliverables early next week? | 17:01 |
| diablo_rojo | smcginnis, yeah I could probably handle that. | 17:02 |
| smcginnis | diablo_rojo: Cool, thanks! | 17:03 |
| diablo_rojo | I'm doing all of them? Not like waiting for PTLs and doing them if they dont? | 17:03 |
| smcginnis | Yeah, I think we just generate them all at the beginning of the week. | 17:04 |
| smcginnis | Then the PTLs can either +1 if they are ready, or block them if they will have an updated sha for us to use later in the week. | 17:04 |
| diablo_rojo | smcginnis, sounds good. Ill generate a list today to make sure I have the right ones if youre willing to sign off on it later :) | 17:05 |
| smcginnis | That'd be great | 17:05 |
| diablo_rojo | smcginnis, sounds good :) Will circle back later. | 17:06 |
| *** udesale has quit IRC | 17:06 | |
| *** tesseract has quit IRC | 17:08 | |
| openstackgerrit | Merged openstack/releases master: Reorder steps on RC1 week process https://review.openstack.org/643398 | 17:17 |
| *** dtantsur is now known as dtantsur|afk | 17:18 | |
| *** jpich has quit IRC | 17:38 | |
| *** e0ne has joined #openstack-release | 17:46 | |
| *** hberaud is now known as hberaud|gone | 17:52 | |
| dhellmann | diablo_rojo : bonus points if you script it ;-) | 18:13 |
| openstackgerrit | Mohammed Naser proposed openstack/releases master: openstack-ansible: add mistral role to rocky https://review.openstack.org/643417 | 18:18 |
| mnaser | dhellmann, smcginnis, ttx ^ as discussed! | 18:19 |
| mnaser | o | 18:20 |
| mnaser | a review already | 18:20 |
| dhellmann | mnaser : commented | 18:20 |
| dhellmann | :-) | 18:20 |
| dhellmann | you caught me as I was flushing my inbox | 18:20 |
| openstackgerrit | Mohammed Naser proposed openstack/releases master: openstack-ansible: add mistral role to rocky https://review.openstack.org/643417 | 18:20 |
| mnaser | voila | 18:20 |
| dhellmann | let's see what the bots have to say | 18:23 |
| *** e0ne has quit IRC | 18:33 | |
| *** notmyname has quit IRC | 18:38 | |
| *** notmyname has joined #openstack-release | 18:39 | |
| *** e0ne has joined #openstack-release | 18:40 | |
| mnaser | dhellmann: sorry to bother ya, just hoping to get this done today -- https://review.openstack.org/#/c/643417/ seems that gates are happy | 18:47 |
| dhellmann | we probably don't need to wait for that branch script fix for this one, since the roles don't worry about upper constraints? | 18:48 |
| dhellmann | mnaser : ^ | 18:48 |
| mnaser | dhellmann: double checking now | 18:49 |
| dhellmann | either way, one repo won't be hard to "fix" and the value it gets won't be wrong | 18:49 |
| dhellmann | we could just fix it on the patch the bot proposes | 18:49 |
| mnaser | dhellmann: looks like we do, even though there's not really a point 'afaik' https://opendev.org/openstack/openstack-ansible-os_mistral/src/branch/master/tox.ini | 18:49 |
| mnaser | yeah, i can make sure we do that | 18:49 |
| dhellmann | ok, approved | 18:49 |
| mnaser | tyvm | 18:49 |
| * dhellmann crosses his fingers that the branch job isn't broken by the bionic update | 18:50 | |
| mnaser | dhellmann: well better break on some small 'rocky' thing than breaking when you're branching all the stein stuff :) | 18:51 |
| mnaser | you're welcome in advance for the testing :P | 18:51 |
| dhellmann | true | 18:51 |
| dhellmann | haha | 18:51 |
| *** gmann is now known as gmann_afk | 18:51 | |
| dhellmann | thanks! | 18:51 |
| *** mgoddard has joined #openstack-release | 18:51 | |
| openstackgerrit | Merged openstack/releases master: openstack-ansible: add mistral role to rocky https://review.openstack.org/643417 | 18:59 |
| openstackgerrit | Graham Hayes proposed openstack/releases master: Add Designate cycle highlights for Stein https://review.openstack.org/643456 | 20:58 |
| *** armax has quit IRC | 21:09 | |
| *** mriedem is now known as mriedem_away | 21:27 | |
| *** whoami-rajat has quit IRC | 21:31 | |
| openstackgerrit | Merged openstack/releases master: Add cinderlib https://review.openstack.org/643133 | 22:00 |
| *** pcaruana has quit IRC | 22:21 | |
| *** e0ne has quit IRC | 22:28 | |
| diablo_rojo | smcginnis, so patches for these Mondayish? http://paste.openstack.org/show/747817/ | 22:30 |
| *** gmann_afk is now known as gmann | 22:40 | |
| *** mriedem_away has quit IRC | 23:23 | |
| fungi | the openstack/tripleo-ui 10.3.0 release jobs finally worked, so it's probably safe to try another release for a js-based project and see how that goes | 23:24 |
| fungi | thanks for bearing with us! | 23:24 |
| *** tosky has quit IRC | 23:26 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!