openstackgerrit | Merged openstack-infra/puppet-accessbot master: Add alumni to accessbot https://review.openstack.org/588115 | 00:07 |
---|---|---|
openstackgerrit | Merged openstack-infra/puppet-accessbot master: Add mlock setting and op_channel option https://review.openstack.org/588134 | 00:07 |
*** efried1 has joined #openstack-infra | 00:16 | |
*** efried has quit IRC | 00:20 | |
*** efried1 is now known as efried | 00:20 | |
*** auristor has quit IRC | 00:31 | |
*** dingyichen has joined #openstack-infra | 00:33 | |
*** auristor has joined #openstack-infra | 00:34 | |
*** xarses has joined #openstack-infra | 01:10 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 01:13 |
*** jtomasek has joined #openstack-infra | 01:13 | |
*** pbourke has quit IRC | 01:15 | |
*** pbourke has joined #openstack-infra | 01:17 | |
*** jtomasek has quit IRC | 01:18 | |
openstackgerrit | Merged openstack-infra/project-config master: Add OpenStack IRC Wall of Fame https://review.openstack.org/588118 | 01:19 |
openstackgerrit | Merged openstack-infra/project-config master: Enforce channel modes with accessbot https://review.openstack.org/588139 | 01:19 |
*** chason has joined #openstack-infra | 01:21 | |
openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: accessbot config: refactor https://review.openstack.org/588999 | 01:31 |
*** kaiokmo has quit IRC | 01:33 | |
*** hongbin has joined #openstack-infra | 01:39 | |
tonyb | I'm seeing the build-wheel-mirror jobs filaing pretty quick in https://review.openstack.org/#/c/588441 looking at the logs it seems like something's up with puppet http://logs.openstack.org/41/588441/1/check/build-wheel-mirror-centos-7/5ff4e54/job-output.txt.gz#_2018-08-05_22_55_55_637217 and | 01:53 |
tonyb | http://logs.openstack.org/41/588441/1/check/build-wheel-mirror-ubuntu-bionic/780ccc6/job-output.txt.gz#_2018-08-05_22_57_20_263688 Is that a know issue? | 01:53 |
openstackgerrit | Ian Wienand proposed openstack-infra/puppet-accessbot master: Small refactor, remove global alumni and mode keys https://review.openstack.org/589001 | 01:55 |
ianw | tonyb: hrm | 02:06 |
ianw | we merged changes to install_puppet iirc | 02:07 |
ianw | no, ok, that's same for a while | 02:08 |
tonyb | ianw: Okay, I really have very little idea how to debug it | 02:09 |
ianw | i think i see what's happened .. | 02:13 |
tonyb | ianw: Thanks fro looking | 02:13 |
ianw | http://logs.openstack.org/41/588441/1/check/build-wheel-mirror-ubuntu-bionic/780ccc6/job-output.txt.gz#_2018-08-05_22_57_10_182524 | 02:18 |
ianw | is where it all starts | 02:18 |
tonyb | ianw: Ahh okay | 02:25 |
tonyb | ianw: that url is a 404 | 02:26 |
ianw | yeah ... and so then we *think* we've installed puppet 3, but really we've installed 5 | 02:27 |
ianw | then it blows up in install_modules.sh | 02:27 |
tonyb | Okay. | 02:29 |
*** psachin has joined #openstack-infra | 02:30 | |
ianw | there is no puppet4 for bionic -> https://tickets.puppetlabs.com/browse/PA-1869?focusedCommentId=558667&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-558667 | 02:31 |
tonyb | ianw: and the modules aren't puppet5 compatible? | 02:35 |
ianw | that's a question ... but the *install* scripts aren't 5 enabled :) | 02:36 |
*** EmilienM has quit IRC | 02:39 | |
*** EmilienM has joined #openstack-infra | 02:39 | |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Support puppet5 for bionic https://review.openstack.org/589007 | 02:53 |
ianw | tonyb: ^ i'll add info in review comments | 02:54 |
tonyb | ianw: Thanks | 02:54 |
*** vivsoni has joined #openstack-infra | 02:56 | |
ianw | tonyb: ok, comment added. i'll look at a role to install afs from ansible | 02:59 |
*** bdodd_ has joined #openstack-infra | 03:08 | |
*** bdodd has quit IRC | 03:09 | |
*** udesale has joined #openstack-infra | 03:52 | |
tonyb | ianw: I reall appreciate you looking at it | 03:55 |
*** d34dh0r53 has quit IRC | 04:00 | |
*** cloudnull has quit IRC | 04:00 | |
*** eglute_s has quit IRC | 04:00 | |
*** cloudnull has joined #openstack-infra | 04:04 | |
*** eglute has joined #openstack-infra | 04:04 | |
*** d34dh0r53 has joined #openstack-infra | 04:04 | |
*** cloudnull has quit IRC | 04:05 | |
*** d34dh0r53 has quit IRC | 04:05 | |
*** eglute has quit IRC | 04:05 | |
*** cloudnull has joined #openstack-infra | 04:05 | |
*** eglute has joined #openstack-infra | 04:06 | |
*** d34dh0r53 has joined #openstack-infra | 04:06 | |
*** hongbin has quit IRC | 04:09 | |
*** ykarel has joined #openstack-infra | 04:20 | |
*** bdodd has joined #openstack-infra | 04:22 | |
*** bdodd_ has quit IRC | 04:24 | |
AJaeger | config-core, would appreciate reviews of https://review.openstack.org/588920 https://review.openstack.org/587631 https://review.openstack.org/583415 https://review.openstack.org/586749 https://review.openstack.org/583849 https://review.openstack.org/580559 https://review.openstack.org/588181 , please | 04:50 |
*** viks_ has joined #openstack-infra | 05:10 | |
*** icey has quit IRC | 05:13 | |
*** icey has joined #openstack-infra | 05:14 | |
*** hwoarang has joined #openstack-infra | 05:28 | |
*** quiquell has joined #openstack-infra | 05:33 | |
*** udesale has quit IRC | 05:34 | |
*** janki has joined #openstack-infra | 05:38 | |
*** lathiat_ is now known as lathiat | 05:44 | |
*** lathiat has quit IRC | 05:44 | |
*** lathiat has joined #openstack-infra | 05:44 | |
*** jtomasek has joined #openstack-infra | 05:58 | |
*** mschuppert has joined #openstack-infra | 05:59 | |
*** AJaeger has quit IRC | 06:01 | |
*** verdurin has quit IRC | 06:01 | |
*** AJaeger has joined #openstack-infra | 06:04 | |
*** nicolasbock has joined #openstack-infra | 06:05 | |
*** verdurin has joined #openstack-infra | 06:05 | |
*** pcaruana has joined #openstack-infra | 06:07 | |
*** udesale has joined #openstack-infra | 06:11 | |
*** chandankumar has joined #openstack-infra | 06:31 | |
*** apetrich has joined #openstack-infra | 06:42 | |
*** fabo_ has quit IRC | 06:51 | |
*** fabo_ has joined #openstack-infra | 06:51 | |
*** fabo_ is now known as fabo | 06:51 | |
*** rcernin has quit IRC | 06:52 | |
*** slaweq has quit IRC | 06:55 | |
*** slaweq has joined #openstack-infra | 06:58 | |
*** ramishra has joined #openstack-infra | 07:03 | |
*** amoralej|off is now known as amoralej | 07:09 | |
*** ajo has joined #openstack-infra | 07:09 | |
*** jhesketh_ has joined #openstack-infra | 07:10 | |
quiquell | Good morning | 07:11 |
quiquell | We have see this error at one of our reviews http://logs.openstack.org/95/583195/18/check/tripleo-ci-centos-7-containers-multinode/769edac/logs/undercloud/var/log/containers/nova/nova-scheduler.log.txt.gz#_2018-08-03_22_57_32_138 | 07:11 |
quiquell | Don't know if it can be caused by a infra issue | 07:11 |
*** jhesketh has quit IRC | 07:11 | |
*** Bhujay has joined #openstack-infra | 07:21 | |
*** ccamacho has joined #openstack-infra | 07:23 | |
*** ginopc has joined #openstack-infra | 07:24 | |
*** zoli is now known as zoli|wfh | 07:25 | |
*** zoli|wfh is now known as zoli | 07:26 | |
*** Bhujay has quit IRC | 07:30 | |
*** Bhujay has joined #openstack-infra | 07:39 | |
*** tosky has joined #openstack-infra | 07:39 | |
*** jpenag is now known as jpena | 07:40 | |
*** Emine has joined #openstack-infra | 07:41 | |
openstackgerrit | Frank Kloeker proposed openstack-infra/project-config master: Setup two additional translation jobs for Rocky https://review.openstack.org/589052 | 07:45 |
eumel8 | ^^ | 07:46 |
eumel8 | AJaeger: any news for the missing translation jobs? | 07:49 |
*** dpawlik has joined #openstack-infra | 07:50 | |
*** ykarel is now known as ykarel|lunch | 07:53 | |
*** rpittau has joined #openstack-infra | 07:53 | |
*** Bhujay has quit IRC | 07:54 | |
*** dingyichen has quit IRC | 07:56 | |
*** dtantsur|afk is now known as dtantsur | 07:59 | |
*** gfidente has joined #openstack-infra | 08:00 | |
quiquell | Do we have a replacement for DEVSTACK_GATE_TIMEOUT at zuulv3 ? | 08:02 |
quiquell | I see zuul.project.timeout | 08:03 |
openstackgerrit | Frode Nordahl proposed openstack-infra/project-config master: Enforce Py3 for `charm-glance` https://review.openstack.org/589067 | 08:07 |
*** Emine has quit IRC | 08:08 | |
*** yamamoto has quit IRC | 08:09 | |
*** jaosorior has joined #openstack-infra | 08:21 | |
*** Emine has joined #openstack-infra | 08:32 | |
*** yamamoto has joined #openstack-infra | 08:43 | |
*** Bhujay has joined #openstack-infra | 08:45 | |
*** olivierbourdon38 has joined #openstack-infra | 09:00 | |
*** ykarel|lunch is now known as ykarel | 09:03 | |
*** yamamoto has quit IRC | 09:15 | |
*** olivierbourdon38 has quit IRC | 09:23 | |
*** panda is now known as panda|rover|off | 09:25 | |
*** olivierbourdon38 has joined #openstack-infra | 09:29 | |
*** apetrich has quit IRC | 09:34 | |
*** Emine has quit IRC | 09:36 | |
*** roman_g has joined #openstack-infra | 09:41 | |
*** adrianc has quit IRC | 09:43 | |
*** yamamoto has joined #openstack-infra | 09:50 | |
ssbarnea | apparently, #oooq channel is not protected by the new spam measure. what do we need to include it in the list of channels that allow only registered users? | 09:51 |
*** Emine has joined #openstack-infra | 09:51 | |
*** Emine has quit IRC | 09:52 | |
*** Emine has joined #openstack-infra | 09:53 | |
quiquell | ssbarnea: Cool | 09:56 |
ssbarnea | i found the file to modify, making a CR now, hopefully it will work | 09:56 |
quiquell | ssbarnea: Puff so painful, please work :-) | 09:58 |
*** vivsoni has quit IRC | 10:00 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Add #oooq channel to accesslist https://review.openstack.org/589103 | 10:00 |
AJaeger | eumel8: no - I cannot help further with those, you need expert help later today | 10:02 |
ssbarnea | quiquell: i am not sure if we want "mask: full_mask" or not. | 10:03 |
AJaeger | ssbarnea: #oooq is not openstack-infra administrated | 10:03 |
AJaeger | Ah, I see your change... | 10:03 |
ssbarnea | AJaeger: please let me know if other magic is needed to make it happen. I checked ChanServ and openstackinfra user is listed as operator on this channel, so it should be able to do its magic, right? | 10:04 |
AJaeger | ssbarnea: if you're op on #oooq, you can do it yourself as well... | 10:04 |
AJaeger | ssbarnea: we run a test on your change, if that passes, all permissions are ok;) | 10:05 |
openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: mirror: Mirror the openSUSE Cloud Master repository https://review.openstack.org/589105 | 10:05 |
ssbarnea | i was not sure it had testing. glad to hear that. how about full_mask option? | 10:06 |
ssbarnea | is it needed or not really? I see very feww channels having it in the config. | 10:07 |
ianw | ssbarnea: have you given openstackinfra chanserver permissions? | 10:09 |
ianw | ssbarnea: see comments in https://review.openstack.org/589001, it might make the mask: option more clear. it's for permissions for chanserv access for users not part of infra. if you just want it like everything else, leave it out | 10:10 |
*** yamamoto has quit IRC | 10:19 | |
*** sambetts_ has joined #openstack-infra | 10:23 | |
*** sambetts_ is now known as sambetts|afk | 10:24 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-subunit2sql master: Add beaker tests https://review.openstack.org/581112 | 10:27 |
ssbarnea | ianw: "access #oooq list" on ChanServ reported: "openstackinfra +AFRefiorstv [modified 16w 5d 18h ago]" which makes me believe it has correct permissions. | 10:28 |
*** vivsoni has joined #openstack-infra | 10:29 | |
*** hughsaunders has quit IRC | 10:32 | |
*** udesale has quit IRC | 10:34 | |
*** hughsaunders has joined #openstack-infra | 10:35 | |
AJaeger | ssbarnea: tests passed, so it's fine ;) | 10:35 |
ssbarnea | lets merge, we got enough spam for one year :D | 10:36 |
AJaeger | ssbarnea: need to wait for another config-core to review https://review.openstack.org/589103 ... | 10:36 |
AJaeger | ssbarnea: is it that bad? Then I'll do it single core ... | 10:37 |
ssbarnea | i guess we can wait a little bit, ianw ^^ | 10:38 |
*** dave-mccowan has joined #openstack-infra | 10:47 | |
ianw | ssbarnea: it *should* work ... the only problem is that we re-run the access bot on config file changes, and it's been getting up towards 30 minutes and puppet kills it | 10:54 |
ianw | so basically, if it's not setup in ... an hour or so, can look further | 10:55 |
*** dave-mccowan has quit IRC | 11:03 | |
openstackgerrit | Merged openstack-infra/project-config master: Add #oooq channel to accesslist https://review.openstack.org/589103 | 11:06 |
*** jpena is now known as jpena|lunch | 11:14 | |
*** zoli is now known as zoli|lunch | 11:22 | |
*** rh-jelabarre has joined #openstack-infra | 11:33 | |
admcleod | hi there, just checking before i do, should i include a test for this: https://review.openstack.org/#/c/589125/ (i assume i can just add one with and one without scriptpath defined to some of the pre-existing tests) | 11:39 |
AJaeger | admcleod: best ask on #openstack-jjb for anything related to jjb (=jenkins job builder) | 11:41 |
admcleod | AJaeger: thanks :} | 11:42 |
*** slagle has joined #openstack-infra | 11:49 | |
*** jcoufal_ has joined #openstack-infra | 11:55 | |
openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add tempest-ansible project under tripleo https://review.openstack.org/589129 | 11:56 |
*** boden has joined #openstack-infra | 11:58 | |
openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add tempest-ansible project under tripleo https://review.openstack.org/589129 | 12:03 |
*** rosmaita has joined #openstack-infra | 12:03 | |
*** radez has joined #openstack-infra | 12:05 | |
*** jcoufal_ has quit IRC | 12:06 | |
*** zoli|lunch is now known as zoli|wfh | 12:06 | |
*** zoli|wfh is now known as zoli | 12:06 | |
*** jcoufal_ has joined #openstack-infra | 12:07 | |
*** e0ne has joined #openstack-infra | 12:08 | |
*** jcoufal_ has quit IRC | 12:10 | |
*** jcoufal_ has joined #openstack-infra | 12:10 | |
*** yamamoto has joined #openstack-infra | 12:11 | |
*** e0ne has quit IRC | 12:13 | |
*** dhill_ has quit IRC | 12:15 | |
*** jpena|lunch is now known as jpena | 12:18 | |
*** vivsoni has quit IRC | 12:21 | |
*** jiapei has joined #openstack-infra | 12:22 | |
*** vivsoni has joined #openstack-infra | 12:22 | |
*** vivsoni has quit IRC | 12:23 | |
*** vivsoni_ has joined #openstack-infra | 12:23 | |
*** quiquell is now known as quiquell|lunch | 12:32 | |
*** dtantsur is now known as dtantsur|brb | 12:36 | |
*** trown|brb is now known as trown | 12:38 | |
*** tpsilva has joined #openstack-infra | 12:39 | |
*** Sigyn has quit IRC | 12:41 | |
*** Sigyn has joined #openstack-infra | 12:42 | |
*** agopi has quit IRC | 12:43 | |
*** hs366 has joined #openstack-infra | 12:47 | |
*** fabo has quit IRC | 12:49 | |
*** yamamoto has quit IRC | 12:50 | |
*** mdrabe has joined #openstack-infra | 12:50 | |
*** dave-mccowan has joined #openstack-infra | 12:52 | |
*** apetrich has joined #openstack-infra | 12:52 | |
*** quiquell|lunch is now known as quiquell|off | 12:57 | |
*** panda|rover|off is now known as panda|rover-ish | 12:58 | |
*** eharney has joined #openstack-infra | 13:00 | |
*** quiquell|off has quit IRC | 13:01 | |
*** dave-mccowan has quit IRC | 13:03 | |
*** kgiusti has joined #openstack-infra | 13:05 | |
*** agopi has joined #openstack-infra | 13:08 | |
*** agopi_ has joined #openstack-infra | 13:08 | |
*** rosmaita has quit IRC | 13:10 | |
*** agopi has quit IRC | 13:12 | |
*** edmondsw has joined #openstack-infra | 13:13 | |
*** jroll has quit IRC | 13:19 | |
*** jroll has joined #openstack-infra | 13:19 | |
*** jcoufal has joined #openstack-infra | 13:21 | |
*** rosmaita has joined #openstack-infra | 13:23 | |
*** janki has quit IRC | 13:24 | |
*** agopi_ is now known as agopi | 13:29 | |
*** jcoufal has quit IRC | 13:30 | |
*** jcoufal_ has quit IRC | 13:30 | |
*** jcoufal has joined #openstack-infra | 13:30 | |
*** Bhujay has quit IRC | 13:31 | |
*** olivierbourdon38 has quit IRC | 13:38 | |
*** mriedem has joined #openstack-infra | 13:41 | |
*** olivierbourdon38 has joined #openstack-infra | 13:41 | |
*** MasterofJOKers has quit IRC | 13:41 | |
*** MasterofJOKers has joined #openstack-infra | 13:43 | |
*** lbragstad has joined #openstack-infra | 13:43 | |
*** olivierbourdon38 has quit IRC | 13:44 | |
*** olivierbourdon38 has joined #openstack-infra | 13:44 | |
*** olivierbourdon38 has quit IRC | 13:46 | |
*** olivierbourdon38 has joined #openstack-infra | 13:47 | |
*** clarkb has joined #openstack-infra | 13:47 | |
*** olivierbourdon38 has quit IRC | 13:47 | |
*** zoli is now known as zoli|afk | 13:51 | |
*** olivierbourdon38 has joined #openstack-infra | 13:59 | |
*** gary_perkins has joined #openstack-infra | 14:04 | |
*** spotz has joined #openstack-infra | 14:08 | |
*** efried1 has joined #openstack-infra | 14:16 | |
*** efried has quit IRC | 14:16 | |
*** efried1 is now known as efried | 14:16 | |
*** Bhujay has joined #openstack-infra | 14:18 | |
*** apetrich has quit IRC | 14:19 | |
*** kklimonda_ has joined #openstack-infra | 14:23 | |
*** Tahvok_ has joined #openstack-infra | 14:23 | |
*** kklimonda has quit IRC | 14:26 | |
*** Tahvok has quit IRC | 14:26 | |
*** zxiiro has quit IRC | 14:26 | |
*** eventingmonkey has quit IRC | 14:26 | |
*** smcginnis[m] has quit IRC | 14:26 | |
*** Tengu has quit IRC | 14:26 | |
*** szaher has quit IRC | 14:26 | |
*** quite has quit IRC | 14:26 | |
*** pfallenop has quit IRC | 14:26 | |
*** dtantsur|brb has quit IRC | 14:26 | |
*** Tahvok_ is now known as Tahvok | 14:26 | |
*** kklimonda_ is now known as kklimonda | 14:26 | |
*** eventingmonkey has joined #openstack-infra | 14:27 | |
*** eharney has quit IRC | 14:28 | |
*** dtantsur has joined #openstack-infra | 14:28 | |
*** Bhujay has quit IRC | 14:29 | |
*** calbers has quit IRC | 14:30 | |
*** dpawlik has quit IRC | 14:31 | |
*** codenix has quit IRC | 14:31 | |
*** calbers has joined #openstack-infra | 14:31 | |
*** zxiiro has joined #openstack-infra | 14:38 | |
*** Tengu has joined #openstack-infra | 14:38 | |
*** zoli|afk is now known as zoli | 14:38 | |
*** zoli is now known as zoli|wfh | 14:39 | |
*** e0ne has joined #openstack-infra | 14:43 | |
*** eglute has quit IRC | 14:45 | |
*** bobh has joined #openstack-infra | 14:45 | |
*** yamamoto has joined #openstack-infra | 14:47 | |
*** eglute has joined #openstack-infra | 14:49 | |
*** jcoufal_ has joined #openstack-infra | 14:50 | |
*** jcoufal_ has quit IRC | 14:50 | |
*** jiapei has quit IRC | 14:51 | |
*** gyee has joined #openstack-infra | 14:52 | |
dhellmann | clarkb : hey, I have ~130 patches ready to go to move the zuul job settings for Infra repos. When do you want those, relative to the PTG and other things the team is working on? | 14:53 |
*** jiapei has joined #openstack-infra | 14:53 | |
AJaeger | config-core, I would appreciate reviews of https://review.openstack.org/588920 https://review.openstack.org/587631 https://review.openstack.org/583415 https://review.openstack.org/586749 https://review.openstack.org/583849 https://review.openstack.org/580559 https://review.openstack.org/588181 , please | 14:54 |
*** dave-mccowan has joined #openstack-infra | 14:58 | |
clarkb | dhellmann: we tend to try avoiding making operational changes (to services directly) when RCs and stuff happen for openstack. That might be a good time to work through some refactoring | 14:59 |
dhellmann | clarkb : yeah, I figured this week would be bad. do you want them at all before the ptg? | 14:59 |
dhellmann | if not I can scrap these and regenerate them later so they're up to date | 14:59 |
*** rpioso|afk is now known as rpioso | 15:00 | |
fungi | scas: i've pushed all the branches and tags from openstack/openstack-chef-repo to openstack/openstack-chef so take a look at let me know if it's what you're expecting | 15:00 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add information to Ref and TimerTrigger repr https://review.openstack.org/589184 | 15:01 |
clarkb | dhellmann: before ptg is probably fine, chances are many of us will pck off reviews like that during quiet times between now and then | 15:01 |
clarkb | dhellmann: using a common topic for the infra change swill likely also help with that | 15:02 |
fungi | change swill is such a great term | 15:02 |
dhellmann | clarkb : I'm proposing them all with python3-first to attach them to the goal | 15:02 |
corvus | eumel8, AJaeger: ^ it's not clear from the logs what's happening with openstack-manuals propose-translation-updates. i've proposed a change to zuul to add more debug info. | 15:03 |
clarkb | dhellmann: we can probably filter most of them with project:^openstack-infra/.* against that topic | 15:03 |
dhellmann | yeah, that would likely work | 15:03 |
AJaeger | corvus: thanks for following up on our discussion from yesterday. | 15:03 |
dhellmann | fungi : you didn't find anything else about why those patches weren't triggering updates to storyboard, did you? | 15:03 |
AJaeger | corvus: it'S also all other repos where we have stable/rocky set up | 15:03 |
corvus | AJaeger: fascinating, that's useful info | 15:04 |
corvus | AJaeger: was july 18 the branch point for rocky? | 15:04 |
scas | fungi: i have some ancillary changes for finalizing the rehoming, but the history looks as i'm expecting. tanks nice man | 15:04 |
corvus | (for manuals) | 15:04 |
AJaeger | corvus: for manuals: We have no rocky | 15:04 |
AJaeger | corvus: not sure when the other repos created their stable branches | 15:05 |
fungi | dhellmann: no, i wasn't in a great position to experiment with that over the weekend. basically the gerrit logs are entirely silent (other than noting the patches were pushed and verified review comments from zuul and fetches from zuul mergers...). normally gerrit logs every interaction it makes with storyboard and i see it logging similar interactions for other reviews both before and after the one | 15:05 |
fungi | you linked | 15:05 |
dhellmann | clarkb : I've been thinking that we want to do some sort of coordination so that while the job setting migrations are in motion we don't change things in project-config to make something obsolete. Do you have any ideas about how to do that? | 15:06 |
dhellmann | fungi : could it have had to do with volume or something? submitting so many patches together? | 15:06 |
clarkb | dhellmann: we could add a temporary rule to the linter thing AJaeger wrote that would fail additive changes to project-config (then remove it when we think the migration is reasonably complete) | 15:07 |
AJaeger | corvus: stable/rocky branches seem to have been created 26th July | 15:07 |
dhellmann | clarkb : would that apply to everyone? or just to a team with the changes in progress? | 15:08 |
clarkb | dhellmann: it would apply to everyone | 15:08 |
dhellmann | ok | 15:09 |
corvus | dhellmann, clarkb: perhaps we could just let the reviewers know? | 15:09 |
dhellmann | yeah, I would be OK with that, too | 15:10 |
clarkb | jillr: on https://review.openstack.org/#/c/583415/6 the new repo uses tripleo.config instead of tripleo-ansible.config for gerrit acl config. Is that intentional? | 15:10 |
corvus | like "no additive changes" seems like a pretty easy thing for reviewers to deal with (and then we don't have to deal with writing exceptions to the linter) | 15:10 |
*** eharney has joined #openstack-infra | 15:10 | |
dhellmann | I had originally thought we would hold off proposing the patch to remove settings from project-config until the others had landed, but maybe we want to propose it so it shows up in the "conflicts with" list in gerrit | 15:10 |
clarkb | corvus: dhellmann ya if we want to keep a running update for which projects are migrating on the infra list that could work too | 15:10 |
dhellmann | where is that list? | 15:11 |
clarkb | dhellmann: sorry, the openstack-infra mailing list | 15:11 |
dhellmann | ah | 15:11 |
fungi | dhellmann: not sure regarding volume being related. an easy test would be to rebase one and see if a comment for it appears in sb (there would normally be a comment for every new patch set mentioning the story in its commit message) | 15:11 |
dhellmann | fungi : I edited a couple based on AJaeger's feedback and they did not | 15:12 |
*** pcaruana has quit IRC | 15:12 | |
dhellmann | it doesn't do any sort of matching to verify that a task for the given project exists, does it? | 15:12 |
dhellmann | even if there's no task id in the message | 15:12 |
clarkb | checking in on packethost we now have non zero error count launching nodes, but the mirror is still happily running | 15:13 |
*** e0ne has quit IRC | 15:14 | |
dhellmann | clarkb , corvus : in the interest of reducing the number of places we have to track status, how about if we use the project-config reviews to indicate that a team has started the migration (a patch to remove their settings with W-1) or is done (a patch to remove their settings without W-1)? | 15:15 |
clarkb | dhellmann: I don't know that we would poll a list of reviews and read through all the diffs to generate that list in our heads | 15:16 |
clarkb | dhellmann: I think most of us tend to read mailing list email, but don't necessarily scan the list of reviews before picking off next review in the queue | 15:16 |
dhellmann | hmm, ok | 15:17 |
*** janki has joined #openstack-infra | 15:17 | |
*** eharney has quit IRC | 15:18 | |
AJaeger | dhellmann: I consider the project-config one nice so that I can check for odd problems, like the moving of legacy jobs | 15:21 |
dhellmann | AJaeger : so you want us to go ahead and propose those patches early? | 15:21 |
*** eharney has joined #openstack-infra | 15:21 | |
AJaeger | dhellmann: yes, that's my personal preference. | 15:22 |
dhellmann | ok | 15:22 |
cmurphy | hey clarkb, i see my gerrit email folder is overflowing, any puppet4 highlights from last week? | 15:22 |
openstackgerrit | Merged openstack-infra/project-config master: Add releasenote job to python-blazarclient https://review.openstack.org/580559 | 15:22 |
openstackgerrit | Merged openstack-infra/project-config master: Move ironic-tempest job for Neutron to "in tree" https://review.openstack.org/588181 | 15:22 |
clarkb | cmurphy: ya, I got a few things merged and mordred helped out with reviews. Uh ask-staging is in the middle of migrating to ask-staging01 so we sort of hung up on being able to test that future parser migraiton | 15:23 |
clarkb | cmurphy: also I left a few comments on some changes let me get links | 15:23 |
cmurphy | clarkb: yeah i noticed i couldn't reach ask-staging.o.o and i wasn't sure if that was expected? | 15:23 |
*** olivierbourdon38 has quit IRC | 15:24 | |
cmurphy | or if some iptables got mangled | 15:24 |
clarkb | cmurphy: I think ianw was working on that, we just need to followup with where it is and possibly help get it properly deployed | 15:24 |
cmurphy | ah ok | 15:25 |
clarkb | cmurphy: https://review.openstack.org/#/c/581335/1 you saw comment there. With the *bot rspec change updates one concern I had was I think many(most/all) will connect to freenode | 15:25 |
clarkb | cmurphy: and we have had to +r our channels due to spamming, so might want to avoid having test bots spam real channels as well | 15:26 |
cmurphy | haha good point | 15:26 |
cmurphy | there is at least one other module where it makes a real connection to a real channel | 15:26 |
*** hamerins has joined #openstack-infra | 15:27 | |
*** jcoufal has quit IRC | 15:27 | |
*** yamamoto has quit IRC | 15:27 | |
clarkb | cmurphy: then the other big change is we (mostly mordred) have deployed bridge.openstack.org to replace puppetmaster.openstack.org. It is a puppetless bastion host that will either do ansible + puppet on remote nodes or just asnible per the new spec | 15:27 |
*** ykarel has quit IRC | 15:28 | |
clarkb | cmurphy: I don't think we need to do anything specific around puppet-4 for this, mostly just an fyi that this is happening. Part of the motivation is that puppetmaster is trusty and we needed a newer host anyway | 15:28 |
cmurphy | sounds good | 15:28 |
fungi | dhellmann: the its-storyboard plug-in for gerrit is _supposed_ to have basically two independent actions it can be called on to perform. one is to act on story footers (leave a comment on the story linking back to the change for review) and on task footers (adjust the state of the task based on the state of the change under review) and as far as i'm aware those are called by totally separate calls | 15:28 |
fungi | into the plug-in so lack of one shouldn't influence the other. we can test that hypothesis by amending the commit message to include a task footer i suppose | 15:28 |
*** jcoufal has joined #openstack-infra | 15:28 | |
*** jcoufal_ has joined #openstack-infra | 15:29 | |
*** jcoufal has quit IRC | 15:29 | |
dhellmann | fungi : done for https://review.openstack.org/588810 | 15:30 |
dhellmann | and the story was indeed updated | 15:31 |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-muranoclient-dsvm-functional-mysql-backend for muranoclient https://review.openstack.org/586749 | 15:31 |
clarkb | cmurphy: we did reorganize the hierdata dirs from groups/ fqdn/ to host-vars/ group-vars/ so that ansible could load them too | 15:31 |
cmurphy | clarkb: just on bridge or on puppetmaster too? | 15:31 |
AJaeger | clarkb: thanks for reviewing! | 15:32 |
clarkb | cmurphy: both | 15:32 |
clarkb | AJaeger: I skipped those that had depends on that were not merged | 15:32 |
cmurphy | clarkb: ah interesting | 15:32 |
AJaeger | clarkb: sorry for including those ;( | 15:32 |
*** jcoufal has joined #openstack-infra | 15:32 | |
*** jcoufal has quit IRC | 15:32 | |
cmurphy | clarkb: how much of bridge is ninja'd versus committed in system-config? i glanced at the recent changes in system-config and the bridge playbooks seem a bit minimal | 15:32 |
cmurphy | mordred: ^ | 15:33 |
clarkb | cmurphy: I think we got the base changes merged that you could spin up a new one | 15:33 |
clarkb | cmurphy: but its still a work in progress until bridge.o.o is ansibling things | 15:33 |
clarkb | cmurphy: there are launch node updates and base playbooks to configure the host | 15:34 |
clarkb | a good exercise would probably be to replace it once we've got the playbooks in a happy spot using only those playbooks though | 15:34 |
cmurphy | ++ | 15:35 |
fungi | dhellmann: i'm starting to wonder if our its rules are incorrect... the comment-on-status-update rule is supposed to trigger an add-standard-comment action on patchset-created but it's possible it's not getting triggered by story footers https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/review.pp#n205 | 15:36 |
mordred | cmurphy, clarkb : yeah it's been being MOSTLY stood up by the patches in that stack | 15:36 |
mordred | there are a few chicken-and-egg issues that are mostly related to the secrets that aren't fully ansibled | 15:36 |
mordred | so - we COULD totally write just a little bit more code to have the current bridge able to set up a new bridge | 15:37 |
fungi | dhellmann: we have two commentlinks rules related to storyboard, "story" (for story tags) and "its-storyboard" (for task tags) https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/review.pp#n205 | 15:37 |
AJaeger | corvus, so for the translation jobs: Merge https://review.openstack.org/589184 and restart zuul would be the next steps, correct? | 15:38 |
mordred | by and large though, I have avoided installing things or setting things up without using the ansible in that stack | 15:38 |
fungi | dhellmann: there may be an implicit interaction there where the commentlink rule has to match the plug-in name, which would explain the current behavior if so (gerrit 2.13 totally changed around the way its rules are configured, so we had to redo this after the last upgrade) | 15:38 |
mordred | cmurphy: https://review.openstack.org/#/c/588326/ is definitely worth a look | 15:38 |
mordred | that's the first stab at making sure the base ansible playbook gets run before we run puppet on nodes so that our integration tests will have all the pieces | 15:39 |
fungi | dhellmann: i'll add to my to do list to experiment with a few possible solutions with review-dev/storyboard-dev | 15:39 |
cmurphy | mordred: neat, i hadn't looked at the open stack yet, only at what was landed | 15:39 |
corvus | AJaeger: yes; at the moment i don't have enough to go on to dig further | 15:40 |
dhellmann | fungi : ok, thanks. I'm not familiar enough with the gerrit side of this to understand the rules in that review.pp file | 15:40 |
fungi | dhellmann: they're pretty magic from my perspective too, unfortunately | 15:40 |
mordred | cmurphy: cool. yeah - the open stack is where most of the fun is ... https://review.openstack.org/#/c/588394 needs to land before we can run more of the playbooks - since the next step would be to run the playbook that does the modules install | 15:41 |
fungi | dhellmann: the documentation for how gerrit its plug-ins work is sparse at best | 15:41 |
* dhellmann nods | 15:41 | |
fungi | even zaro had a tough time explaining how this works, and he wrote the its-storyboard plug-in | 15:42 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add information to Ref and TimerTrigger repr https://review.openstack.org/589184 | 15:42 |
tosky | dear infra, we (tempestconf) are hit by a timeout error on a test which seems to be happen only on ovh nodes, while rax and packethost work | 15:44 |
tosky | compare for example the failed and passed jobs on https://review.openstack.org/#/c/580981/ | 15:44 |
tosky | or the first run and the recheck for https://review.openstack.org/#/c/588593/ | 15:45 |
tosky | does it ring any bell? | 15:45 |
clarkb | tosky: it helps if you can link to specific job logs | 15:46 |
*** ginopc has quit IRC | 15:46 | |
tosky | failures: http://logs.openstack.org/81/580981/7/gate/python-tempestconf-tempest-packstack-admin/d943583/zuul-info/ | 15:47 |
tosky | http://logs.openstack.org/81/580981/7/gate/python-tempestconf-tempest-packstack-admin/b0e0792/zuul-info/ | 15:47 |
tosky | working: http://logs.openstack.org/81/580981/7/check/python-tempestconf-tempest-packstack-admin/27d3863/ | 15:47 |
tosky | and http://logs.openstack.org/81/580981/7/check/python-tempestconf-tempest-packstack-admin/d729479/ | 15:48 |
*** jaosorior has quit IRC | 15:48 | |
fungi | tosky: when you say "timeout error" here it looks like you mean an ssh timeout reaching into a cirros instance started by the test? | 15:49 |
clarkb | tosky: http://logs.openstack.org/81/580981/7/gate/python-tempestconf-tempest-packstack-admin/d943583/job-output.txt.gz#_2018-08-06_14_07_31_997944 looks like metadata service failed to provide the ssh key then ssh authentication failed because the ssh key wasn't configured | 15:49 |
tosky | fungi: yes, that one | 15:49 |
fungi | "SSHTimeout: Connection to the 172.24.4.21 via SSH timed out." | 15:49 |
tosky | the same job works on other providers | 15:49 |
*** jaosorior has joined #openstack-infra | 15:49 | |
tosky | another example from a different review: failure: http://logs.openstack.org/93/588593/1/check/python-tempestconf-tempest-packstack-admin/5040efe/ | 15:51 |
tosky | and working: http://logs.openstack.org/93/588593/1/check/python-tempestconf-tempest-packstack-admin/b8fdc81/ | 15:51 |
fungi | i don't see any possible conflicting addresses/routes on the node where one of those ran, so rules out my first theory | 15:52 |
clarkb | fungi: tosky I don't think this is a failing of the provider, neutron metadata service didn't do its job | 15:52 |
*** Emine has quit IRC | 15:53 | |
dhellmann | clarkb, corvus : how about something like this as an announcement? http://lists.openstack.org/pipermail/openstack-infra/2018-August/006039.html | 15:55 |
*** apetrich has joined #openstack-infra | 15:55 | |
dhellmann | (I'm trying to get this correct so I can write instructions for champions so I don't have to do all of these patches myself) | 15:55 |
tosky | clarkb: but the root cause for that may still related to the provider | 15:56 |
tosky | size of instances, network, etc | 15:56 |
clarkb | tosky: it could be, but I think we need to identify why metadata service failed before we make that leap | 15:56 |
corvus | dhellmann: oh, it's not all at once? i thought it was like "just don't approve changes which add stuff to projects for a while" | 15:57 |
*** dklyle has joined #openstack-infra | 15:57 | |
clarkb | tosky: is this a single node job? if so then networking shouldn't matter its all localhost anyway | 15:57 |
tosky | clarkb: single node job | 15:58 |
dhellmann | corvus : well, I didn't want to assume that all of them would be proposed together all at once | 15:58 |
*** efried has quit IRC | 15:58 | |
dhellmann | corvus : if you want to do it all at once, we could just pick a start date for that freeze | 15:58 |
dhellmann | although I expect proposing the patches to take a few days | 15:59 |
clarkb | tosky: in any case I wouldn't make the jump that this is a provider problem just yet. We need to understand why metadata service failed to provide the ssh public key to the instance | 15:59 |
corvus | dhellmann: AJaeger is the most prolific project-config reviewer, i'm happy with whatever works for him | 16:00 |
jillr | clarkb: no, that is not intended - thank you will update. | 16:01 |
clarkb | dhellmann: yup I think that should work thanks | 16:05 |
dhellmann | I keep anticipating this being a huge number of patches that we're going to need time to absorb. Let me generate them all and see what we actually have. | 16:06 |
openstackgerrit | Jill Rouleau proposed openstack-infra/project-config master: Add Tripleo Ansible repo https://review.openstack.org/583415 | 16:06 |
*** efried has joined #openstack-infra | 16:10 | |
*** ramishra has quit IRC | 16:10 | |
*** jpena is now known as jpena|off | 16:12 | |
*** janki has quit IRC | 16:15 | |
*** bobh has quit IRC | 16:18 | |
*** SamYaple has quit IRC | 16:37 | |
*** SamYaple has joined #openstack-infra | 16:39 | |
*** betherly_ is now known as betherly-afk | 16:40 | |
*** e0ne has joined #openstack-infra | 16:47 | |
fungi | headed out to get some lunch, back soon | 16:50 |
*** evrardjp has quit IRC | 16:51 | |
*** hongbin has joined #openstack-infra | 16:52 | |
AJaeger | infra-root, could you retire stable/newton for tripleo, please? the change for eol tagging has been merged, tonyb has details on what to do in http://lists.openstack.org/pipermail/openstack-dev/2018-July/132346.html . mwhahaha is ready to go... | 16:53 |
AJaeger | infra-root, side effect of this: fixes remaining error in http://zuul.openstack.org/api/config-errors | 16:53 |
*** bobh has joined #openstack-infra | 16:55 | |
*** olivierbourdon38 has joined #openstack-infra | 16:56 | |
clarkb | AJaeger: do you know where that eol_branch scrpt lives? | 16:56 |
openstackgerrit | James E. Blair proposed openstack-infra/puppet-log_processor master: Add severity info to logstash and filter out DEBUG lines https://review.openstack.org/588383 | 16:56 |
corvus | clarkb: ^ that has the severity filter actually filter out debug lines so crm114 (and any future processors) don't see them | 16:57 |
*** apetrich has quit IRC | 16:57 | |
AJaeger | clarkb: good questions. I hope ianw or jhesketh_ know where eol_branch.sh lives | 16:58 |
AJaeger | or tonyb ^ | 16:58 |
openstackgerrit | James E. Blair proposed openstack-infra/puppet-log_processor master: Add severity info to logstash and filter out DEBUG lines https://review.openstack.org/588383 | 16:58 |
AJaeger | clarkb: Running find right now, codesearch did not help ;( | 16:59 |
clarkb | corvus: one comment on ps2, a cleanup that was missed in the refactor | 16:59 |
corvus | on it | 17:00 |
openstackgerrit | James E. Blair proposed openstack-infra/puppet-log_processor master: Add severity info to logstash and filter out DEBUG lines https://review.openstack.org/588383 | 17:00 |
corvus | clarkb: i think that should have all the fixes now :) | 17:00 |
clarkb | yup lgtm | 17:01 |
*** bobh has quit IRC | 17:02 | |
*** bobh has joined #openstack-infra | 17:02 | |
AJaeger | clarkb: https://github.com/openstack-infra/release-tools/commit/39a82df080dda0b3028be623f016354ff0544520#diff-6b15029c01f7d1bce80961f6a7ea121e | 17:02 |
*** amoralej is now known as amoralej|off | 17:03 | |
AJaeger | clarkb: we retired release-tools repo and that repo has the script in it. | 17:03 |
AJaeger | NO idea where we moved it afterwards ;( | 17:03 |
clarkb | interesting, dhellmann may know? | 17:03 |
AJaeger | so, script is https://github.com/openstack-infra/release-tools/blob/0abf645cc36aadce365ba08328e98864272d64ed/eol_branch.sh | 17:03 |
AJaeger | clarkb, dhellmann I found no reviews for it either. dhellmann, smcginnis, what's a good place to move eol_branch.sh to? | 17:05 |
*** hemna_ has joined #openstack-infra | 17:06 | |
smcginnis | Hmm, maybe under openstack/releases in the tools directory? Although that doesn't really look like a "release" thing per se. | 17:07 |
*** bobh has quit IRC | 17:07 | |
AJaeger | It's run by infra at EOL. clarkb, do we have a good place in infra instead? | 17:07 |
clarkb | AJaeger: tonyb normally runs it fwiw, we have escalated his gerrit perms for him to run that script | 17:08 |
AJaeger | clarkb: ah | 17:09 |
dhellmann | AJaeger : you could put it in project-config with the rest of the release tool scripts, since we're going to want a copy there when we can automate this step anyway | 17:10 |
dhellmann | otherwise, in releases/tools like smcginnis suggested would be good | 17:10 |
dhellmann | I think we didn't realize any more of those scripts were necessary, :-/ | 17:10 |
AJaeger | no worries, we found it ;) | 17:10 |
smcginnis | If we can automate it evetually, project-config does sound like a good place then. I think a little more appropriate in the infra domain than the releases domain. | 17:11 |
AJaeger | ok | 17:14 |
corvus | mordred: were you going to drop the "ansible_python_interpreter: python3" setting on bridge from https://review.openstack.org/587985 ? | 17:14 |
corvus | mordred: is 'with_first_found' going away? i actually really like 'with_first_found' -- it makes perfect sense as a short-circuited while loop. the replacement is a lot of boilerplate for the same thing. :| | 17:17 |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Import eol_branch.sh https://review.openstack.org/589236 | 17:19 |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove eol tagging from eol_branch.sh https://review.openstack.org/589237 | 17:19 |
AJaeger | tonyb, smcginnis, dhellmann, clarkb ^ | 17:19 |
*** zoli|wfh is now known as zoli|gone | 17:19 | |
*** zoli|gone is now known as zoli | 17:19 | |
corvus | mordred: related to that, i left a -1 for an important question on https://review.openstack.org/588546 | 17:20 |
clarkb | AJaeger: thanks will take a look shortly. | 17:21 |
corvus | clarkb, mordred: question on https://review.openstack.org/588553 | 17:22 |
AJaeger | clarkb: shall I ask tonyb to coordinate with infra-root? Then I'll send an email out... | 17:22 |
clarkb | AJaeger: yes I think we should dobule check that version of the script is up to date (via tonyb) and then we can run it for tonyb if that is easier or bump tonyb's gerrit perms so that he can run it | 17:23 |
AJaeger | clarkb: ok, will write email in case he does not read here | 17:24 |
corvus | https://review.openstack.org/588626 failed the puppet4-apply test on centos-7 -- is that issue known? | 17:24 |
*** yamamoto has joined #openstack-infra | 17:24 | |
clarkb | corvus: yes it is the ssh keys issue 2018-08-03 17:44:19 +0000 Puppet (err): Failed to apply catalog: Could not parse line "HijxEUeWHhFJDIVRGUf0Ef0nrmBv fungi-openstack-2015" at /root/.ssh/authorized_keys:27 from http://logs.openstack.org/26/588626/1/check/infra-puppet-apply-4-centos-7/ff37fb3/applytest/puppetapplytest27.final.out.FAILED | 17:28 |
clarkb | this is not puppet 4 specific and has happened since we moved to adding root keys with glean on the test images | 17:28 |
corvus | clarkb: i'm not up to speed on that; is there some backlog i should look for? | 17:28 |
clarkb | corvus: https://review.openstack.org/#/c/501887/2 was where I last tried to poke at it. cmurphy was looking at it more recently too iirc, not sure if there was anything new learned | 17:30 |
clarkb | I think ianw had a glean change too /me looks for that | 17:30 |
corvus | clarkb: is this a non-deterministic failure? | 17:30 |
clarkb | corvus: we don't know yet why it happens but it isn't consistent | 17:31 |
clarkb | if you recheck chances are it iwll pass | 17:31 |
clarkb | https://review.openstack.org/#/c/585574/ is the glean change ianw wrote that was related to this | 17:31 |
corvus | clarkb: 501887 failed but the logs have beend deleted :( | 17:31 |
clarkb | corvus: ya :/ | 17:31 |
clarkb | corvus: it also only seems to affect centos | 17:32 |
mordred | corvus: I am not sure for sure if it's going away going away, but I *believe* that is the intent. I think the reason they're wanting to make it go away is that it's not actually a loop but looks like one to people - so apparently it seems like ansible users get confused by it | 17:37 |
mordred | corvus: however, there's definitely a bug in that patch that your -1 points out | 17:37 |
*** dhill_ has joined #openstack-infra | 17:38 | |
*** olivierbourdon38 has quit IRC | 17:40 | |
*** evrardjp has joined #openstack-infra | 17:41 | |
*** vivsoni_ has quit IRC | 17:44 | |
*** vivsoni_ has joined #openstack-infra | 17:44 | |
*** psachin has quit IRC | 17:45 | |
*** bobh has joined #openstack-infra | 17:48 | |
openstackgerrit | Merged openstack-infra/zuul master: Add information to Ref and TimerTrigger repr https://review.openstack.org/589184 | 17:49 |
clarkb | mordred: did you see that openstacksdk for the task wait thing merged? | 17:50 |
clarkb | (I think next step is making a bug fix release?) | 17:50 |
*** jiapei has quit IRC | 17:51 | |
*** dtantsur is now known as dtantsur|afk | 17:53 | |
clarkb | looks like limestone has gone into hibernation mode again. I'm going to restart the nodepool launcher on nl02 to get it going again | 17:57 |
mordred | clarkb: yes - I pushed up a backport of it to stable/rocky and that is winding through the gate right now. I'll push up a release patch as son as it lands | 17:57 |
*** yamamoto has quit IRC | 17:57 | |
clarkb | launcher is restarted | 17:58 |
clarkb | the git sha1 has not changed | 17:59 |
eumel8 | corvus: just to mention that are only 15 periodic jobs for propose-translation-update are running. Since 07/18 and especially the last week: http://status.openstack.org/openstack-health/#/?searchProject=propose-translation-update&duration=P1M&end=2018-08-06T18:00:00.000Z&groupKey=build_name | 18:03 |
eumel8 | looks like a limit | 18:03 |
*** e0ne has quit IRC | 18:05 | |
*** jcoufal_ has quit IRC | 18:05 | |
*** trown is now known as trown|lunch | 18:06 | |
clarkb | followup on the zuul executor kernel updates: they all seem to run a similar number of jobs now, probably a reasonable assumption to infer the kernel did improve the performance | 18:08 |
*** roman_g has quit IRC | 18:09 | |
mordred | clarkb: ++ | 18:09 |
clarkb | eumel8: I don't think we added any pipeline limits like that. Did we happen to remove jobs from that pipeline maybe? | 18:13 |
clarkb | AJaeger: ^ you may know | 18:13 |
*** e0ne has joined #openstack-infra | 18:14 | |
AJaeger | clarkb: we have a limit - let me grab it... | 18:14 |
AJaeger | clarkb: a semaphore to run only 10 jobs, see http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n1216 | 18:15 |
clarkb | oh interesting | 18:15 |
AJaeger | clarkb: but that is supposed to limit the number of *simulantaneous* running jobs | 18:15 |
AJaeger | eumel8: that health report is broken, count http://zuul.openstack.org/builds.html?job_name=propose-translation-update | 18:16 |
AJaeger | eumel8: ok, 16 not 15 for today ;) | 18:17 |
AJaeger | and 16 yesterday | 18:17 |
AJaeger | corvus: ^ | 18:17 |
eumel8 | AJaeger: nice shot, but there are more then 10 in parallel :) | 18:18 |
AJaeger | eumel8: not in parallel, check starting times and lenght of job | 18:19 |
AJaeger | length is around 5 minutes, and we have starting times between 06:00 and 06:10 | 18:19 |
openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Adeed endpoint https://review.openstack.org/589254 | 18:20 |
AJaeger | eumel8: https://review.openstack.org/589184 merged to debug this - now corvus needs to restart zuul to use the change... | 18:20 |
eumel8 | AJaeger: saw it. my expectition is, it works after restart :) | 18:22 |
*** dhill_ has quit IRC | 18:22 | |
AJaeger | ,) | 18:22 |
*** gfidente is now known as gfidente|afk | 18:23 | |
*** bobh has quit IRC | 18:23 | |
*** bobh has joined #openstack-infra | 18:23 | |
*** nicolasbock has quit IRC | 18:23 | |
*** nicolasbock has joined #openstack-infra | 18:24 | |
mordred | clarkb, corvus: btw - I'm focused on an infinite-recursion bug I found in sdk over the weekend - I've finally got it isolated down to a single line reproduction, now I just have to fix it | 18:24 |
mordred | but I'm going to finish fixing it before I push further on bridge | 18:24 |
clarkb | mordred: if that gets us fix for limestone releaesd then I'm happy :) | 18:25 |
mordred | ah - those are unrelated- limestone fix is just waiting on the stable patch to land - I'll get that cut as soon as it does | 18:25 |
clarkb | ah thought you might want to bundle bug fix relases | 18:25 |
mordred | nah. justhave mental state on the recursion problem that I don't want to lose - because, well - recursion problem | 18:29 |
mordred | corvus: also - my original reply on your comment for https://review.openstack.org/#/c/588546 was bogus (I was reading poorly) - I have left a followup comment | 18:34 |
* mordred goes to find sandwich | 18:34 | |
corvus | mordred: ok thanks. i looked in the docs for first_found and didn't see a mention of skip. | 18:48 |
*** numans has quit IRC | 18:54 | |
clarkb | fungi: not sure if you noticed but ianw wrote a letsencrypt spec for us, you probably want to take a look given prior interest https://review.openstack.org/#/c/587283/ | 18:58 |
*** dhill_ has joined #openstack-infra | 18:59 | |
fungi | yep, it's on my reading list | 19:05 |
fungi | thanks! | 19:06 |
*** yamamoto has joined #openstack-infra | 19:17 | |
*** trown|lunch is now known as trown | 19:17 | |
openstackgerrit | Merged openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 19:27 |
*** yamamoto has quit IRC | 19:31 | |
*** yamamoto has joined #openstack-infra | 19:36 | |
*** yamamoto has quit IRC | 19:47 | |
*** jcoufal_ has joined #openstack-infra | 19:49 | |
*** jcoufal_ has joined #openstack-infra | 19:51 | |
*** jcoufal_ has quit IRC | 19:51 | |
*** dbecker has joined #openstack-infra | 20:00 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-subunit2sql master: Fix subunit2sql install guard for idempotency https://review.openstack.org/581335 | 20:05 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-subunit2sql master: Add beaker tests https://review.openstack.org/581112 | 20:05 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-subunit2sql master: Update Gemfile for Zuulv3 https://review.openstack.org/581347 | 20:07 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: Update Gemfile for Zuulv3 https://review.openstack.org/577521 | 20:08 |
*** e0ne has quit IRC | 20:11 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-log_processor master: Add beaker tests https://review.openstack.org/581697 | 20:12 |
*** Guest74859 has joined #openstack-infra | 20:19 | |
*** hamerins has quit IRC | 20:23 | |
*** rtjure has joined #openstack-infra | 20:31 | |
*** kgiusti has left #openstack-infra | 20:31 | |
*** eharney has quit IRC | 20:39 | |
*** trown is now known as trown|outtypewww | 20:49 | |
*** bobh has quit IRC | 20:55 | |
clarkb | cmurphy: https://review.openstack.org/#/c/581112/7 still seems to depends on itself | 20:59 |
cmurphy | bah | 21:00 |
cmurphy | git sucks | 21:00 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-subunit2sql master: Add beaker tests https://review.openstack.org/581112 | 21:01 |
*** rockyg has joined #openstack-infra | 21:06 | |
openstackgerrit | Merged openstack-infra/openstackid-resources master: Adeed endpoint https://review.openstack.org/589254 | 21:12 |
*** edmondsw has quit IRC | 21:15 | |
openstackgerrit | Merged openstack-infra/puppet-subunit2sql master: Update Gemfile for Zuulv3 https://review.openstack.org/581347 | 21:20 |
*** harlowja has joined #openstack-infra | 21:21 | |
openstackgerrit | Merged openstack-infra/openstackid master: Fixed typo on logout from IDP https://review.openstack.org/588402 | 21:22 |
*** hwoarang has quit IRC | 21:24 | |
*** bobh has joined #openstack-infra | 21:28 | |
openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added missing permission to endpoint v2 https://review.openstack.org/589291 | 21:31 |
openstackgerrit | Merged openstack-infra/openstackid-resources master: Added missing permission to endpoint v2 https://review.openstack.org/589291 | 21:32 |
*** bobh has quit IRC | 21:35 | |
*** rockyg has quit IRC | 21:36 | |
*** yamamoto has joined #openstack-infra | 21:41 | |
*** bobh has joined #openstack-infra | 21:47 | |
openstackgerrit | Merged openstack-infra/project-config master: Remove Ansible function jobs from cloud-launcher https://review.openstack.org/588599 | 21:49 |
openstackgerrit | Merged openstack-infra/project-config master: gerrit-dash-creator: revert to readthedocs template https://review.openstack.org/588410 | 21:49 |
*** yamamoto has quit IRC | 21:51 | |
*** slagle has quit IRC | 21:53 | |
openstackgerrit | Merged openstack-infra/project-config master: Switch to publish-openstack-docs-pti for murano https://review.openstack.org/586738 | 21:54 |
*** boden has quit IRC | 21:56 | |
*** boden has joined #openstack-infra | 21:57 | |
*** boden has quit IRC | 21:57 | |
*** boden has joined #openstack-infra | 21:57 | |
*** boden has quit IRC | 21:57 | |
*** boden has joined #openstack-infra | 21:57 | |
*** boden has quit IRC | 21:58 | |
*** boden has joined #openstack-infra | 21:58 | |
*** boden has quit IRC | 21:59 | |
clarkb | jillr: bah http://logs.openstack.org/15/583415/7/check/project-config-gerrit/0785cec/job-output.txt.gz#_2018-08-06_16_17_59_747628 the linter check didn't like that change, we need to remove the acl config line entirely since the default is to use the config file named after the project | 21:59 |
*** boden has joined #openstack-infra | 21:59 | |
*** boden has quit IRC | 21:59 | |
*** boden has joined #openstack-infra | 22:00 | |
*** boden has quit IRC | 22:00 | |
*** boden has joined #openstack-infra | 22:00 | |
*** boden has quit IRC | 22:01 | |
*** boden has joined #openstack-infra | 22:01 | |
*** boden has quit IRC | 22:02 | |
*** boden has joined #openstack-infra | 22:02 | |
*** bobh has quit IRC | 22:02 | |
*** boden has quit IRC | 22:02 | |
*** bobh has joined #openstack-infra | 22:03 | |
*** boden has joined #openstack-infra | 22:03 | |
*** boden has quit IRC | 22:03 | |
*** bobh has quit IRC | 22:07 | |
jillr | clarkb: ack, will do | 22:10 |
openstackgerrit | Merged openstack-infra/project-config master: Add publish-to-pypi for cinder-tempest-plugin https://review.openstack.org/587623 | 22:12 |
openstackgerrit | Jill Rouleau proposed openstack-infra/project-config master: Add Tripleo Ansible repo https://review.openstack.org/583415 | 22:14 |
*** rcernin has joined #openstack-infra | 22:15 | |
openstackgerrit | Merged openstack-infra/project-config master: Enforce Py3 for `charm-glance` https://review.openstack.org/589067 | 22:17 |
openstackgerrit | Merged openstack-infra/project-config master: Setup two additional translation jobs for Rocky https://review.openstack.org/589052 | 22:17 |
corvus | clarkb, fungi: any idea how to get an http password for a gerrit system account (eg, "zuul")? | 22:20 |
clarkb | corvus: it is in the account external id's database table, though I'm not sure if we need to trigger generating that somehow (or if there is a more human friendly way to get that | 22:20 |
corvus | apparently you can now set an "--http-password" on the "gerrit create-account" command.... | 22:21 |
corvus | ooh! | 22:21 |
corvus | https://review.openstack.org/Documentation/cmd-set-account.html | 22:21 |
corvus | set-account has an --http-password option | 22:21 |
clarkb | TIL | 22:22 |
corvus | so i'm going to run "gerrit set-account zuul --http-password $SOMETHING" sound good? | 22:22 |
clarkb | yup | 22:22 |
clarkb | assuming zuul is the username | 22:22 |
corvus | pretty sure it is | 22:22 |
corvus | done | 22:24 |
corvus | adding to hiera now | 22:24 |
*** nicolasbock has quit IRC | 22:25 | |
corvus | committed to puppetmaster and bridge | 22:26 |
fungi | yep, that's what i was going to suggest | 22:26 |
fungi | either set-account --http-password or i think you can do something similar via the rest api | 22:27 |
fungi | https://review.openstack.org/Documentation/rest-api-accounts.html#set-http-password | 22:28 |
fungi | you can either provide a pregenerated password or ask it to generate and return one in the response | 22:29 |
corvus | this is a bit of a one-off, so i find the ssh api much easier to use | 22:29 |
corvus | so i just used pwgen -s | 22:29 |
fungi | yeah, i totally agree | 22:29 |
corvus | (with some suitable number following the -s :) | 22:29 |
fungi | i tend to use the ssh api for most stuff because it's easier than all the setup needed to make rest api calls manually | 22:30 |
corvus | hrm, the zuul docs don't seem to be up to date: https://zuul-ci.org/docs/zuul/admin/drivers/gerrit.html#connection-configuration | 22:30 |
corvus | that should have an entry for 'password' | 22:30 |
fungi | i used the gerrit ssh api just this morning to add and then remove myself from project bootstrappers so i could push content into the resurrected openstack-chef repo | 22:31 |
corvus | for some reason, the failure is only visible in ara: http://logs.openstack.org/91/91c775a1aabf53116da534c27058ea6c62ddeb86/post/publish-zuul-docs/1230b25/ara-report/result/fc2abd92-01f2-4675-b534-3c938a37c0bb/ | 22:32 |
fungi | yeah, the builds status suggests publish-zuul-docs has been failing at least as far back as 2018-07-26 | 22:33 |
corvus | that does not happen in the check/gate docs builds | 22:33 |
corvus | nor does it happen locally | 22:34 |
clarkb | are check and gate using a non warning as error invocation? | 22:34 |
fungi | WARNING: Unexpected return code 1 from command 'zuul --help' | 22:34 |
fungi | neat | 22:35 |
corvus | yeah, same command in build-sphinx-docs | 22:35 |
corvus | http://logs.openstack.org/27/577027/10/gate/build-sphinx-docs/1848b8f/ara-report/result/86e74885-a5bd-42e1-ba13-76ec560f3677/ | 22:36 |
clarkb | hrm what python version is it using I wonder | 22:36 |
fungi | Modified: 15 July 2018, 18.12.41 UTC | 22:36 |
fungi | so it changed at some point after that i guess | 22:37 |
corvus | 7-15 is last success; 7-17 appears to be the first failure | 22:37 |
clarkb | corvus: http://logs.openstack.org/91/91c775a1aabf53116da534c27058ea6c62ddeb86/post/publish-zuul-docs/1230b25/job-output.txt.gz#_2018-08-06_19_31_01_764713 | 22:38 |
clarkb | its failing to build libre2 before running sphinx | 22:38 |
fungi | sphinx 1.7.6 was uploaded july 16, between the last success and the first failure | 22:39 |
corvus | broken: | 22:40 |
corvus | 2018-07-17 01:12:17.176601 | ubuntu-xenial | libre2-1v5 | 22:40 |
corvus | working: | 22:40 |
corvus | 2018-07-15 18:10:05.119819 | ubuntu-xenial | libre2-dev | 22:40 |
corvus | in the bindep install task | 22:40 |
corvus | changed in this commit: https://review.openstack.org/580159 | 22:41 |
clarkb | libre2-dev is installed for profiles compile and test | 22:42 |
clarkb | we must not apply those profiles to the publish job bindep run? | 22:42 |
*** yamamoto has joined #openstack-infra | 22:43 | |
fungi | aha, yep that change added those profiles to it, which becomes a sieve preventing it from running when neither is requested | 22:43 |
corvus | bindep_profile: doc | 22:43 |
corvus | that's in the publish job | 22:43 |
*** agopi has quit IRC | 22:43 | |
fungi | looks like we need to add doc to that line | 22:43 |
corvus | bindep_profile: compile doc | 22:44 |
corvus | that's in the build-sphinx-docs job | 22:44 |
fungi | or include compile in the publish job, that would work | 22:44 |
corvus | hopefully after the artifact fetching rework is done, we can refactor that job to inherit from build-sphinx-docs | 22:47 |
corvus | atm the post-playbook isn't really compatible | 22:47 |
*** yamamoto has quit IRC | 22:47 | |
openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add compile bindep profile to publish-zuul-docs https://review.openstack.org/589301 | 22:48 |
corvus | fungi, clarkb: ^ | 22:48 |
corvus | fungi, clarkb: this is zuul's connection information: http://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/group/zuul-scheduler.yaml | 22:51 |
corvus | is there a way to add in a secret value in there? | 22:51 |
fungi | it gets merged with whatever you put in the groups/zuul-scheduler.yaml file in private hiera | 22:52 |
corvus | i'm going to need to pull that out of hiera and put it in a puppet-language-datastructure | 22:52 |
clarkb | corvus: ya I think you read in the value to a variable, then update a subkey of the dict then use that | 22:52 |
fungi | or at least it's supposed to be | 22:52 |
clarkb | currently we just pass it striaght through but you can modify subkeys iirc | 22:52 |
corvus | oh? a subkey is what i need | 22:52 |
corvus | how does that work? | 22:52 |
corvus | (i need to set zuul_connections[1]['password']) | 22:53 |
clarkb | I think we need to use puppet stdlib, I'm looking | 22:53 |
fungi | aha, you need to merge parts of a key... okay that one's definitely beyond me | 22:53 |
fungi | aside from tracking them in separate keys and then doing some smashing together via puppet or an exec | 22:54 |
clarkb | corvus: https://github.com/puppetlabs/puppetlabs-stdlib#deep_merge should work, I'm reading more to see if there is a better options | 22:55 |
corvus | clarkb: since there's a list in there, that's probably not going to work as-is (i would overwrite the entire gerrit connection) | 22:56 |
corvus | oh i guess i could just merge that hash | 22:56 |
fungi | sort of defeatist but we could simply stick zuul_connections in private hiera | 22:57 |
corvus | still, i'd have to manipulate the list too | 22:57 |
corvus | fungi: yeah, i think the option to beat is "put it in puppet dsl" | 22:57 |
clarkb | there is also a merge | 22:57 |
clarkb | I think either merge or deep_merge are our best bets | 22:58 |
openstackgerrit | Merged openstack-infra/project-config master: Add compile bindep profile to publish-zuul-docs https://review.openstack.org/589301 | 22:59 |
corvus | none of these methods is compatible with the reason this stuff is laid out this way -- to allow direct reuse of our site manifest by others | 23:00 |
corvus | huh, there's a connection_secrets value that puppet-zuul takes | 23:02 |
*** tosky has quit IRC | 23:02 | |
corvus | oh that's clever | 23:02 |
corvus | https://review.openstack.org/482941 | 23:03 |
corvus | that's apparently how we do this :) | 23:03 |
fungi | hah | 23:05 |
fungi | those crafty puppet-zuul authors | 23:06 |
corvus | okay, private hiera updated on puppetmaster and bridge to move the gerrit http password value to zuul_connection_secrets | 23:07 |
corvus | kicking the scheduler | 23:11 |
corvus | okay, password is in place | 23:11 |
corvus | i'd like to restart zuul now to pick up the debug info i added and also test out the http reporting | 23:12 |
*** bobh has joined #openstack-infra | 23:12 | |
clarkb | zuul seems quiet | 23:13 |
clarkb | I'm around for about another 45 minutes to help | 23:13 |
clarkb | then I am making burger | 23:13 |
clarkb | *burgers | 23:13 |
* mordred sticks his head up from the longest day of debugging in the history of mankind | 23:16 | |
mordred | the end result of which was the addition of the line "setattr(instance, self.name, None)" | 23:17 |
mordred | the whole day | 23:17 |
corvus | mordred: you've been helpful here recently -- a patch you wrote a year ago is just what i needed 10 minutes ago :) | 23:17 |
mordred | corvus: yay! | 23:17 |
corvus | or some combination of you and pabelanger | 23:17 |
corvus | does anyone feel like http://zuul.openstack.org/ should have more stuff? | 23:18 |
clarkb | corvus: yes | 23:18 |
corvus | http://zuul.openstack.org/api/info | 23:18 |
ianw | http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 seems to confirm 0 node requests? | 23:19 |
*** hongbin has quit IRC | 23:19 | |
clarkb | it would've just happened bceuase I checked the status page a few minutes ago to see things were relatively idle | 23:19 |
mordred | corvus: that doesn't look great | 23:19 |
corvus | zuul-web is not running | 23:19 |
clarkb | but not that idle | 23:19 |
corvus | i attempted to restart it, maybe it failed | 23:19 |
corvus | the pidfile was left behind | 23:21 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: DNM: test gerrit http reporting https://review.openstack.org/589305 | 23:22 |
corvus | oh hrm, where are my re-enqueue commands? | 23:23 |
corvus | there they are :) | 23:23 |
clarkb | just the pidfile then? did we merge the change that stopped removing the pidfiles in the init scripts? | 23:24 |
corvus | clarkb: i think we did. should be easy enough to start and stop zuul-web a few times and see what happens | 23:24 |
corvus | much easier to fix than zuul-scheduler :) | 23:24 |
clarkb | indeed | 23:24 |
corvus | huzzah! zuul's first gerrit http report -- with inline comments! https://review.openstack.org/589305 | 23:24 |
clarkb | nice | 23:25 |
clarkb | all reporting is over https now? | 23:25 |
corvus | yes, so we should keep an eye out for how that behaves | 23:25 |
corvus | the ssh reporting has been *very* reliable (i believe it uses a semi-persistent connection). i worry about https. but it should attempt 3 times before giving up. | 23:26 |
corvus | #status log zuul now reports to gerrit over HTTPS rather than ssh; please keep an eye out for any issues | 23:27 |
openstackstatus | corvus: finished logging | 23:27 |
corvus | infra-root: ^ | 23:27 |
mordred | corvus: that is the coolest thing I've seen | 23:27 |
clarkb | ya for all the concerns people have had about gerrit's ssh api not being REST it worked really well | 23:27 |
corvus | clarkb: the range highlight seems to have worked | 23:29 |
mordred | corvus: it did! | 23:29 |
clarkb | corvus: yup | 23:29 |
clarkb | it renders a little weird with the diff coloring | 23:29 |
clarkb | there is the random line that ends in green | 23:29 |
clarkb | but that is gerrit's fault not zuul's | 23:29 |
corvus | yeah, with a slight green underline | 23:29 |
mordred | the second comment also renders slightly weird, also gerrit's fault not zuul's | 23:30 |
*** tpsilva has quit IRC | 23:30 | |
mordred | because it chooses to elide some of the lines that are marked with a comment | 23:30 |
corvus | yeah, you have to expand up a lot to actually find zuul-stream-functional | 23:30 |
mordred | yah | 23:30 |
fungi | wowie zowie! | 23:30 |
mordred | but I think that's fine | 23:30 |
corvus | would be nice if it included the whole range | 23:30 |
mordred | because it's just super cool | 23:30 |
mordred | maybe we could submit that as a bug upstream - it might be that having a comment on a line-range that spans outside of the diff section is just a case that hasn't come up | 23:31 |
clarkb | mordred: I'm sure some of it is the size of the highlighted range | 23:32 |
clarkb | I'm guessing most human reviewers pick one a single word or line | 23:32 |
corvus | w00t: a successful merge: https://review.openstack.org/#/c/586888/ | 23:32 |
openstackgerrit | Fatema Khalid Sherif proposed openstack-infra/storyboard-webclient master: Expose ordering criteria in search UI https://review.openstack.org/577870 | 23:32 |
corvus | verified and submit are two separate POST calls | 23:33 |
corvus | so it's good to see that works :) | 23:33 |
mordred | \o/ | 23:33 |
corvus | that's what that looks like: http://paste.openstack.org/show/727449/ | 23:34 |
ianw | corvus / mordred : thoughts before i send out reviews -- would you rather see a "kerberos-client" role and a "openafs-client" role, with the later depending on the prior; or keep it more contained in a openafs-client role? | 23:36 |
*** hemna_ has quit IRC | 23:36 | |
*** rpioso is now known as rpioso|afk | 23:37 | |
corvus | ianw: my instinct is 2 roles, mostly because of how well segregated they are elsewhere | 23:37 |
*** agopi has joined #openstack-infra | 23:38 | |
ianw | corvus: ok, so in terms of laying that out -- does the openafs-client role take kerberos arguments, or we just include the kerberos-client role, and document "you need to setup these parameters"? | 23:38 |
mordred | I think 2 roles is fine as well - although I'm not sure I'd have them depend on each other to start- just having both in the playbook seems easy enough for today | 23:38 |
ianw | or more document "you need to setup extra parameters, see the kerberos-client role" | 23:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!