Saturday, 2017-04-01

*** thorst has quit IRC00:02
*** ijw has joined #openstack-infra00:05
dimsthanks mordred00:05
*** zz_dimtruck is now known as dimtruck00:10
*** ijw has quit IRC00:11
rajinirir-api fails to start, anyone seen this?  [ERROR] /opt/stack/new/ironic/devstack/lib/ironic:1281 ir-api did not start00:12
*** pvaneck has quit IRC00:13
*** Apoorva_ has joined #openstack-infra00:25
*** Apoorva has quit IRC00:28
*** Apoorva_ has quit IRC00:29
*** pramodrj07 has quit IRC00:30
*** MasterOfBugs has quit IRC00:31
*** ijw has joined #openstack-infra00:36
openstackgerritchenhb proposed openstack-infra/project-config master: [rally]Move cinder and heat scenario jobs out non-voting  https://review.openstack.org/45133900:39
*** Sukhdev has quit IRC00:40
openstackgerritMerged openstack-infra/project-config master: Rework golang-dsvm job  https://review.openstack.org/45232300:41
*** mriedem1 has joined #openstack-infra00:43
*** mriedem has quit IRC00:43
*** dimtruck is now known as zz_dimtruck00:46
*** esberglu has joined #openstack-infra00:49
*** Julien-zte has joined #openstack-infra00:51
*** esberglu has quit IRC00:54
*** hongbin has quit IRC00:54
*** zhurong has joined #openstack-infra00:57
*** ijw has quit IRC01:00
*** Sihan_ has joined #openstack-infra01:05
*** john-davidge has joined #openstack-infra01:06
Sihan_Trying to setup the statsd, yet extras.try_import('statsd.statsd') returns None. I check, i have installed the statsd 2.0.101:06
*** adrian_otto has joined #openstack-infra01:09
*** john-davidge has quit IRC01:10
*** liujiong has joined #openstack-infra01:12
*** adrian_otto1 has joined #openstack-infra01:13
fungiSihan_: double-check that it's the _right_ statsd (there's more than one python library by that name). you want the one you would get with `pip install 'statsd>=1.0.0,<3.0'` (matching the version specifier in zuul's requirements.txt file). looks like the latest version on pypi before 3.x is statsd 2.1.201:13
*** adrian_otto has quit IRC01:14
fungibut as long as it's not the "other" python lib called statsd, any version from 1.0.0 through 2.1.2 should work fine01:16
Sihan_2.1.2 does not help, that's why i try 2.0.101:16
Sihan_still getting zuul.Scheduler: Statsd disabled because python statsd package not found01:16
fungiwe have statsd 2.1.2 on our zuul server and i can't import statsd.statsd there either01:19
Sihan_how do we make the statsd work ?01:25
funginot quite sure what's up with that since we do seem to be using statsd successfully01:25
Sihan_hmmm01:25
*** Julien-zte has quit IRC01:27
*** Julien-zte has joined #openstack-infra01:27
fungiit's a bit late at night for me so odds are i'm not thinking straight, but hopefully one of our zuulies in a timezone where it's daylight can remember what the deal is01:28
Sihan_sure, no worries thx01:29
*** ihrachys has joined #openstack-infra01:33
*** Sihan_ has quit IRC01:33
*** Sihan has quit IRC01:35
*** gyee has quit IRC01:37
*** adrian_otto1 has quit IRC01:42
*** askb has quit IRC01:43
*** adrian_otto has joined #openstack-infra01:46
*** zz_dimtruck is now known as dimtruck01:47
*** jerryz has quit IRC01:52
jeblairSihan_ is gone :(01:56
jeblairfor the record, if they show up again, you need to have STATSD_HOST (and maybe STATSD_PORT) exported as environment variables before starting zuul.01:57
jeblairthe statsd module does magic based on those variables01:57
jeblairso import statsd will always work01:57
jeblairbut statsd.statsd will be None unless those env variables are defined01:57
*** thorst has joined #openstack-infra01:59
*** adrian_otto has quit IRC02:12
*** ihrachys has quit IRC02:17
*** thorst has quit IRC02:18
*** adrian_otto has joined #openstack-infra02:19
*** EricGonczer_ has joined #openstack-infra02:21
*** adrian_otto has quit IRC02:21
*** adrian_otto has joined #openstack-infra02:22
*** adrian_otto has quit IRC02:27
*** dharinic has joined #openstack-infra02:27
*** dave-mccowan has quit IRC02:31
*** baoli has joined #openstack-infra02:32
*** EricGonczer_ has quit IRC02:36
*** yanyanhu has joined #openstack-infra02:36
*** yanyanhu has quit IRC02:39
*** mriedem1 has quit IRC02:42
*** yamahata has quit IRC02:44
*** adrian_otto has joined #openstack-infra02:44
*** adrian_otto has quit IRC02:50
*** Hal has joined #openstack-infra03:00
*** dharinic has quit IRC03:03
*** baoli has quit IRC03:04
*** xiaoruiguo has joined #openstack-infra03:05
*** baoli has joined #openstack-infra03:05
*** john-davidge has joined #openstack-infra03:06
*** xiaoruiguo has quit IRC03:08
*** adrian_otto has joined #openstack-infra03:08
*** john-davidge has quit IRC03:11
*** xiaoruiguo has joined #openstack-infra03:11
*** thorst has joined #openstack-infra03:15
*** sekelso has joined #openstack-infra03:15
*** thorst has quit IRC03:19
*** ihrachys has joined #openstack-infra03:21
*** Goneri has joined #openstack-infra03:24
*** baoli has quit IRC03:26
*** adrian_otto has quit IRC03:29
*** beisner has quit IRC03:31
*** gcb has quit IRC03:32
*** Hal has quit IRC03:33
*** beisner has joined #openstack-infra03:34
*** ijw has joined #openstack-infra03:44
*** zhurong has quit IRC03:45
*** sekelso has quit IRC03:46
*** ijw has quit IRC03:48
*** armax has quit IRC03:59
*** adrian_otto has joined #openstack-infra04:04
*** Goneri has quit IRC04:06
*** zhurong has joined #openstack-infra04:07
*** dharinic has joined #openstack-infra04:09
*** marst has quit IRC04:12
*** ihrachys has quit IRC04:13
*** sihan has joined #openstack-infra04:23
*** marst has joined #openstack-infra04:25
mnaserhopefully things are progressing well for infra team :(04:26
*** armax has joined #openstack-infra04:31
*** armax is now known as Guest2563004:31
*** tphummel has quit IRC04:47
*** yee379 has quit IRC04:51
*** zhurong has quit IRC04:51
*** yee379 has joined #openstack-infra04:51
*** Guest25630 has quit IRC04:54
*** liusheng has quit IRC04:56
*** liusheng has joined #openstack-infra05:02
*** zhurong has joined #openstack-infra05:03
*** ihrachys has joined #openstack-infra05:06
*** john-davidge has joined #openstack-infra05:07
*** ihrachys has quit IRC05:10
*** harlowja has quit IRC05:10
*** john-davidge has quit IRC05:11
*** ihrachys has joined #openstack-infra05:13
*** ihrachys has quit IRC05:14
*** jokke_ has quit IRC05:15
*** thorst has joined #openstack-infra05:17
*** thorst has quit IRC05:21
*** psachin has joined #openstack-infra05:25
*** signed8bit_Zzz is now known as signed8b_05:26
*** signed8b_ has quit IRC05:33
ianwthe logs repair is at 87.8%05:39
*** kzaitsev_mb has joined #openstack-infra05:41
*** Julien-zte has quit IRC05:43
*** Julien-zte has joined #openstack-infra05:43
*** harlowja has joined #openstack-infra05:44
*** nadya has joined #openstack-infra05:47
*** nadya has quit IRC05:48
*** Julien-zte has quit IRC05:50
*** Julien-zte has joined #openstack-infra05:51
*** mrunge has quit IRC05:57
*** yamahata has joined #openstack-infra06:10
*** liusheng has quit IRC06:21
*** liusheng has joined #openstack-infra06:21
*** Julien-zte has quit IRC06:28
*** gouthamr has quit IRC06:28
*** Julien-zte has joined #openstack-infra06:28
*** Julien-zte has quit IRC06:35
*** Julien-zte has joined #openstack-infra06:35
*** RuiChen has joined #openstack-infra06:37
*** Julien-zte has quit IRC06:40
*** Julien-zte has joined #openstack-infra06:41
*** harlowja has quit IRC06:45
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: Refactor: block-device filesystem creation, mount and fstab  https://review.openstack.org/44458606:46
*** cshastri has joined #openstack-infra06:49
*** sigmavirus has quit IRC06:52
*** sigmavirus has joined #openstack-infra06:53
*** sigmavirus is now known as Guest7486906:54
*** harlowja has joined #openstack-infra07:03
*** stevebaker has quit IRC07:06
*** slagle has quit IRC07:06
*** cschwede has quit IRC07:06
*** yatin_away has quit IRC07:07
*** yamamoto has joined #openstack-infra07:08
*** john-davidge has joined #openstack-infra07:08
*** john-davidge has quit IRC07:13
*** dharinic has quit IRC07:14
*** thorst has joined #openstack-infra07:18
*** yatin_away has joined #openstack-infra07:19
*** thorst has quit IRC07:22
*** yamamoto has quit IRC07:26
openstackgerritMerged openstack-infra/project-config master: gnocchi: remove old functional jobs  https://review.openstack.org/43789907:28
openstackgerritMerged openstack-infra/project-config master: Skip puppet integration tests for test-requirements.txt  https://review.openstack.org/44983407:28
*** knangia has quit IRC07:31
*** harlowja has quit IRC07:39
openstackgerritDirk Mueller proposed openstack/diskimage-builder master: Fix opensuse-minimal element on non-SUSE hosts  https://review.openstack.org/45080007:53
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** ZZelle has joined #openstack-infra08:02
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: Docker matrix build  https://review.openstack.org/41434708:04
*** ZZelle has quit IRC08:04
*** john-davidge has joined #openstack-infra08:09
*** Julien-zte has quit IRC08:11
*** Julien-zte has joined #openstack-infra08:12
*** john-davidge has quit IRC08:13
*** quite has quit IRC08:14
*** quite has joined #openstack-infra08:14
*** sputnik13 has quit IRC08:15
*** sputnik13 has joined #openstack-infra08:17
*** sputnik13 has quit IRC08:18
*** sputnik13 has joined #openstack-infra08:20
*** thorst has joined #openstack-infra08:20
*** Julien-zte has quit IRC08:24
*** Julien-zte has joined #openstack-infra08:25
*** prateek has joined #openstack-infra08:25
openstackgerritDirk Mueller proposed openstack-infra/nodepool master: Add opensuse 42.2 DIB testing  https://review.openstack.org/45004508:27
*** nadya has joined #openstack-infra08:33
*** nadya has quit IRC08:36
*** thorst has quit IRC08:38
*** psachin has quit IRC08:46
*** psachin has joined #openstack-infra08:47
*** tosky has joined #openstack-infra08:48
*** kzaitsev_mb has quit IRC08:52
dirkpabelanger: I've removed the [wip] from the review just above as it appears to have passed checks last run08:58
*** Julien-zte has quit IRC09:00
*** Julien-zte has joined #openstack-infra09:00
*** xiaoruiguo has quit IRC09:05
*** yamamoto has joined #openstack-infra09:10
*** prateek_ has joined #openstack-infra09:17
*** prateek has quit IRC09:18
*** liusheng has quit IRC09:20
*** liusheng has joined #openstack-infra09:20
*** yamamoto has quit IRC09:25
*** martinkopec has joined #openstack-infra09:29
*** martinkopec has quit IRC09:29
*** jascott1_ has quit IRC09:30
*** thorst has joined #openstack-infra09:35
*** thorst has quit IRC09:39
*** prateek__ has joined #openstack-infra09:39
*** liujiong has quit IRC09:40
*** jkilpatr has quit IRC09:42
*** prateek_ has quit IRC09:43
*** kzaitsev_mb has joined #openstack-infra09:49
*** zhurong has quit IRC09:54
*** jtomasek has quit IRC09:57
*** jascott1_ has joined #openstack-infra10:07
*** Julien-zte has quit IRC10:09
*** john-davidge has joined #openstack-infra10:09
*** jascott1_ has quit IRC10:12
*** john-davidge has quit IRC10:14
*** Satyam2345 has joined #openstack-infra10:25
*** yamamoto has joined #openstack-infra10:26
*** sputnik13 has quit IRC10:26
*** Satyam2345 has left #openstack-infra10:27
*** e0ne has joined #openstack-infra10:27
*** sputnik13 has joined #openstack-infra10:29
*** yamamoto has quit IRC10:32
*** pbourke has quit IRC10:32
*** yamamoto has joined #openstack-infra10:32
*** pbourke has joined #openstack-infra10:34
*** sshnaidm|off has quit IRC10:34
*** jascott1_ has joined #openstack-infra10:50
*** john-davidge has joined #openstack-infra10:50
*** bhavik1 has joined #openstack-infra10:54
ianwok logs recovery is gone10:54
ianwdone10:54
ianwi don't have time to remount it now though10:54
ianwhttp://paste.openstack.org/show/605131/ <- some interesting stats10:55
ianw  236333417 regular files10:55
ianw    82972872 directories10:55
*** jascott1_ has quit IRC10:56
*** prateek__ has quit IRC11:01
*** e0ne has quit IRC11:05
*** yamahata has quit IRC11:11
fungiianw: yeah, per conversation in #openstack-infra-incident i'm already working on that11:14
*** yamamoto has quit IRC11:22
*** kzaitsev_mb has quit IRC11:30
*** thorst has joined #openstack-infra11:37
*** masber has quit IRC11:39
*** adrian_otto has quit IRC11:40
*** Julien-zte has joined #openstack-infra11:40
*** thorst has quit IRC11:41
*** john-davidge has quit IRC11:41
*** bhavik1 has quit IRC11:54
*** yamamoto has joined #openstack-infra12:02
*** jkilpatr has joined #openstack-infra12:06
*** Hal has joined #openstack-infra12:14
*** ccamacho has quit IRC12:21
*** ssbarnea has quit IRC12:25
*** ssbarnea has joined #openstack-infra12:26
*** ssbarnea has quit IRC12:26
*** kzaitsev_mb has joined #openstack-infra12:26
*** kzaitsev_mb has quit IRC12:31
*** ssbarnea has joined #openstack-infra12:32
*** thorst has joined #openstack-infra12:36
*** cshastri has quit IRC12:38
*** ccamacho has joined #openstack-infra12:38
*** nadya has joined #openstack-infra12:42
*** thorst has quit IRC12:45
*** thorst has joined #openstack-infra12:46
*** psachin has quit IRC12:50
*** thorst has quit IRC12:50
*** yanyanhu has joined #openstack-infra12:55
*** mat128 has joined #openstack-infra12:55
*** yanyanhu has quit IRC12:59
*** askb has joined #openstack-infra13:11
*** askb has quit IRC13:11
dmsimardGood morning o/ if there's any cores around, need a final +2 on a trivial jjb patch: https://review.openstack.org/#/c/450847/13:15
*** askb has joined #openstack-infra13:19
*** askb has quit IRC13:19
fungi#status ok The http://logs.openstack.org/ site is back in operation; previous logs as well as any uploaded during the outage should be available again; jobs which failed with POST_FAILURE can also be safely rechecked.13:23
openstackstatusfungi: sending ok13:23
*** baoli has joined #openstack-infra13:23
AJaegerthanks, fungi. Will you reply to openstack-dev mailing or shall I?13:24
fungiAJaeger: if you want to, that would be awesome. i'm still doing some lingering cleanup of the temporary storage13:24
AJaegerok, will do...13:25
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"13:25
-openstackstatus- NOTICE: The http://logs.openstack.org/ site is back in operation; previous logs as well as any uploaded during the outage should be available again; jobs which failed with POST_FAILURE can also be safely rechecked.13:25
fungithanks AJaeger!13:26
*** baoli has quit IRC13:27
* AJaeger has sent the email and goes offline again13:28
* AJaeger wishes everybody a great weekend13:28
openstackstatusfungi: finished sending ok13:29
mordredfungi: woot!13:29
mordredfungi: happy weekend!13:29
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Do not merge: Test ARA 0.12.0rc1  https://review.openstack.org/43996613:38
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Do not merge: Test ARA 0.12.4rc1  https://review.openstack.org/43996613:39
*** EricGonczer_ has joined #openstack-infra13:39
openstackgerritDavid Moreau Simard proposed openstack-infra/tripleo-ci master: Replace "ara generate" by "ara generate html", move report to ara_oooq  https://review.openstack.org/44810813:43
openstackgerritDavid Moreau Simard proposed openstack-infra/tripleo-ci master: Replace "ara generate" by "ara generate html", move report to ara_oooq  https://review.openstack.org/44810813:43
*** thorst has joined #openstack-infra13:47
*** EricGonczer_ has quit IRC13:50
*** thorst has quit IRC13:51
*** john-davidge has joined #openstack-infra13:53
*** john-davidge has quit IRC13:57
*** kzaitsev_mb has joined #openstack-infra14:01
openstackgerritColleen Murphy proposed openstack-infra/puppet-gerrit master: Fix puppet-lint error  https://review.openstack.org/45246114:05
*** nadya has quit IRC14:06
*** wolverineav has joined #openstack-infra14:09
*** marst has quit IRC14:10
*** marst has joined #openstack-infra14:10
pabelangerdirk: okay, I'll take a look and update the patch.14:11
*** marst has quit IRC14:11
*** marst has joined #openstack-infra14:12
*** nadya has joined #openstack-infra14:13
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Add opensuse 42.2 DIB testing  https://review.openstack.org/45004514:15
*** nadya has quit IRC14:15
*** e0ne has joined #openstack-infra14:24
*** e0ne has quit IRC14:25
AJaegerinfra-root, do we have a problem with lists.o.o? I send an email an hour ago to openstack-dev and it's not in the archives or back yet14:26
*** arxcruz|pto has quit IRC14:26
*** e0ne_ has joined #openstack-infra14:26
pabelangerAJaeger: I just a test message last night from jeblair14:27
pabelangersigh14:27
pabelangergot*14:27
AJaegerhttp://lists.openstack.org/pipermail/openstack-dev/2017-April/date.html has one message14:28
jeblairi'll check14:28
mordredoh thank goodness, jeblair is here. (I was just starting to look)14:29
jeblairmore eyes better :)14:29
mordred2 legs bad?14:29
jeblair4 legs good14:29
jeblairdoes anyone know of a message that's missing?14:29
mordredwe agree then14:29
mordredjeblair: AJaeger sent one about an hour ago14:30
AJaegerjeblair: I send one an hour ago as aj@suse.com - in reply to http://lists.openstack.org/pipermail/openstack-dev/2017-March/114831.html14:30
mordred13:28:1614:30
mordredI see an inbound message from you at that time14:31
jeblairmordred: can you paste the queue id?14:31
AJaegergood. But nothing outbound?14:31
mordredjeblair: I was looking in mainlog - but I've got this:14:31
jeblairah 1cuJ4i-0001iI-Gc right?14:31
mordred2017-04-01 13:28:16 1cuJ4i-0001iI-Gc <= aj@suse.com H=prvmx02.microfocus.com [130.57.1.217] P=esmtp S=3124 id=e9e1f4da-7fed-e7ff-c4a1-e8a5e64e9e39@suse.com14:31
mordred2017-04-01 13:38:42 1cuJCz-0001wh-Gt => aj@suse.com R=mailman_verp_router T=mailman_verp_smtp H=prv-mx.provo.novell.com [130.57.1.10] X=TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128 C="250 Ok"14:31
mordredyah14:31
AJaegeryah, looks like me...14:32
*** Julien-zte has quit IRC14:32
jeblairi don't see a corresponding msg in /var/log/mailman/post at that time14:33
mordredwell that's not great14:33
mordredjeblair: the mailman_verp_router is exim passing it on to mailman, right?14:34
*** Julien-zte has joined #openstack-infra14:34
jeblairmordred: mailman_router -> mailman_transport14:34
jeblairmordred: _verp_ is outbound14:34
mordredah14:34
*** mat128 has quit IRC14:35
*** arxcruz|pto has joined #openstack-infra14:35
jeblairmordred: that injects it into mailman with a pipe to /var/lib/mailmain/mail/mailman14:36
jeblairbut spelled right14:36
jeblair/var/lib/mailman/mail/mailman14:37
mordredjeblair: that pipe is only writable by root14:37
jeblairmordred: it's a script14:37
mordredoh14:37
jeblairi meant it pipes it into the script14:37
mordredyes - so it is14:37
mordredthat's ok then14:37
mordred:)14:37
jeblairsorry, unclear :)14:37
mordredif that was a named pipe, it would have bad permissions14:37
jeblairit's setguid, which seems right14:38
jeblairsetgid even14:38
jeblairi found AJaeger's message in /var/lib/mailman/qfiles/shunt/1491053296.674383+8579e4eb6ddd8a31804fd0d51e318edef5ae3a73.pck14:39
jeblairwe have 15 shunted messages14:39
jeblairhttp://paste.openstack.org/show/605136/14:40
jeblairwtf?14:40
jeblairhttps://mail.python.org/pipermail/mailman-developers/2013-November/023408.html and https://mail.python.org/pipermail/mailman-developers/2013-November/023409.html14:41
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add dsvm-nodepool-opensuse-src experimental job  https://review.openstack.org/45246214:41
jeblairii  mailman                          1:2.1.16-2ubuntu0.2                        amd64        Powerful, web-based mailing list manager14:42
*** thorst has joined #openstack-infra14:42
jeblairugh14:42
jeblairwe have upgraded to a broken version of mailman14:42
jeblairi guess ubuntu didn't upgrade mailman14:43
jeblairin trusty14:43
jeblairxenial does have a later version14:43
jeblairi'm inclined to manually patch that file on disk14:44
jeblairand reinject the shunted messages14:44
jeblair(then talk about further upgrading next week)14:44
pabelangerwe could also work to get patch added to trusty-updates with ubuntu devs we know14:45
jeblairya.  it's a pretty bad error.14:45
jeblairwith a simple fix.14:45
jeblairhttps://bugs.launchpad.net/mailman/+bug/125149514:45
openstackLaunchpad bug 1251495 in mailman (Ubuntu Trusty) "Lists with topics enabled can throw unexpected keyword argument 'Delete' exception." [High,Triaged]14:45
*** jkilpatr has quit IRC14:46
pabelangerI can look at the packaging later today and start the process14:46
pabelangerif we decided to manually apply patch14:46
jeblairmordred: ^  manual patch sound like a plan?14:46
*** jkilpatr has joined #openstack-infra14:47
mordredjeblair: yes14:47
jeblairdone14:47
jeblairworking on unshunting now14:47
*** Julien-zte has quit IRC14:50
clarkbthat seems like such a simple thing to backport :(14:51
mordredyah14:52
*** Julien-zte has joined #openstack-infra14:52
mordredSpamapS: ^^ whence you awaken, if you feel so inclined, there is a one-line backport for mailman for trusty-updates that will change it from not working to working14:52
clarkblooks like logs is back too14:53
jeblairokay, i changed the file ande delected the .pyc file14:54
jeblairi'm still getting the same error (though the traceback shows the corrected code)14:54
jeblairwhat am i missing?14:55
clarkbperhaps its laoded in memory of a running process?14:55
*** Julien-z_ has joined #openstack-infra14:55
jeblairclarkb: that's it.  thank you :)14:55
jeblairi was focusing on the fact that i was running a script, but neglected to remember that script is just handing it off to a daemon14:56
*** Julien-zte has quit IRC14:56
jeblairmuch better!14:56
jeblairwe now have 3 shunted messages14:56
jeblair(instead of 15)14:56
jeblairApr 01 14:57:11 2017 (22573) post to openstack-dev from aj@suse.com, size=4895, message-id=<e9e1f4da-7fed-e7ff-c4a1-e8a5e64e9e39@suse.com>, success14:57
*** thorst has quit IRC14:59
jeblairthose 3 shunted message are due to encoding errors.  i don't know if that represents a new class of error or not15:00
*** Julien-z_ has quit IRC15:00
clarkbinteresting. Are they sending eg utf8 with latin1 chars?15:01
jeblairUnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 6: invalid continuation byte15:01
jeblairthat's the error, not sure about the message15:01
clarkbâ is 0xe2 in latin115:03
clarkb(so wouldnt surprise me if that is what is happening due to bad client somehwere)15:06
clarkbbut ya unsure if those wouldve been delivered previously15:06
jeblairhttps://bugs.launchpad.net/mailman/+bug/146275515:07
openstackLaunchpad bug 1462755 in GNU Mailman "qrunner crashes on invalid unicode sequence" [Low,Fix released] - Assigned to Mark Sapiro (msapiro)15:07
jeblairmaybe that ^15:07
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: Validate flavor specification in config  https://review.openstack.org/45187515:07
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: Add ability to select flavor by name or id  https://review.openstack.org/44978415:07
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: Cleanup from config syntax change  https://review.openstack.org/45186815:07
mordredjeblair: wow. that's a great bug15:08
jeblairi think we're probably hitting the debian error mentioned at the end there15:09
mordredyup15:09
jeblairso i think we need to recode the list descriptions15:09
clarkbthe good news is there is a script to fix15:09
mordredI wonder if that's in any way related to our other weird thing of having the language set to ZH_tw or whatever it was set to15:09
mordredlike - clearly i18n support in this upgrade was a bit off15:09
jeblairmordred: maybe in that it's part of the debian upgrade, though i don't think we did anything wrong.15:10
mordredtotally15:10
mordredsorry - that's what I meant - something about language support in the debian upgrade was weird, and that has potentially caused one or more upgrade issues15:11
jeblairya15:11
jeblairhttps://www.msapiro.net/scripts/recode_list15:11
jeblairso i will run that ^15:11
clarkbthat may explain the <?> we got too15:12
mordredjeblair: ++15:12
mordredjeblair: do we need to back something up before running that?15:13
clarkb(as an aside utf8 is clearly the right choice here, just wish they did the conversion more automatically)15:13
mordred++15:13
* mordred reading the script15:13
jeblairmordred: yeah, i'll make a copy of all the config.pck files15:13
mordredkk. cool15:13
clarkblooks like the script runs in noop mode by default (and has a --doit flag to do it)15:14
*** Julien-zte has joined #openstack-infra15:15
*** e0ne_ has quit IRC15:15
jeblairmaybe we should run this on the test server first15:15
clarkb++15:16
jeblairlists-upgrade-test.openstack.org15:16
clarkbalso you can restrict it using hte --language flag to a specific language15:16
jeblairclarkb: i checked the openstack-vi list (one of the ones with an error).  it's preferred language is en.15:16
clarkbgoing off the <?>s I would expect -es -fr and -vi to be the ones needing it15:16
clarkboh interesting15:17
clarkbjeblair: there is a drop down on the list page to switch to vietnamese, so somewhere in there it knows its not just english15:18
jeblairclarkb: yeah, vi is an available language for that list15:18
pabelangerclarkb: just seen this in nodepool job15:19
pabelanger2017-04-01 14:22:39.797364 |   Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='mirror.ord.rax.openstack.org', port=80): Read timed out. (read timeout=60.0)",)': /pypi/simple/gear/15:19
pabelangergoing to check mirrors to make sure proxy cache isn't causing issues15:19
clarkbjeblair: -fr is in french not english so may make a good test case15:20
pabelangerhttp://status.openstack.org/elastic-recheck/#1449136 large spike of pypi mirror disconnections15:20
jeblairclarkb: that was another of the 3 with an error i believe15:21
jeblairhttp://paste.openstack.org/show/605137/15:21
jeblairthat's from the dry run on the test server15:21
jeblairno arguments15:21
clarkbhuh I wonder what in -dev is being changed15:21
clarkbmaybe we want to modify the script to log more of what it is doing (I am currently doing code review on it to make sure it looks sane)15:22
openstackgerritColleen Murphy proposed openstack-infra/infra-specs master: Rename and expand Puppet 4 Preliminary Testing  https://review.openstack.org/44993315:22
jeblairclarkb: i'll do that15:22
*** marst has quit IRC15:22
clarkbI like how the iteration here is done the C way15:23
jeblairinfo15:23
jeblairis the attr it changes on -dev15:23
jeblairinfo Questa lista è dedicata alle discussioni su OpenStack.15:24
*** mriedem has joined #openstack-infra15:24
jeblairso, i guess, a translation of info on the dev list :)15:24
clarkbneat15:25
jeblair(i'm double checking that)15:25
jeblairoh, nope, i'm wrong, that was for the -it list15:26
clarkbits using encode(encoding, errors=replace) to convert. Since latin1 is a strict subset of utf8 we should be fine doing that15:26
jeblairpasswords15:26
jeblairthat's what it's changing on -dev15:26
clarkbI still think thats likely ok since latin1 is subset of utf8 so you'd be able to enter your old password (it will just have to be encoded to utf8 when sent)15:27
jeblairer, nope, i'm still wrong15:27
jeblairi'm really bad at this :)15:27
jeblairtopics is openstack-dev15:28
jeblairdescription, info, passwords for -fr15:28
jeblairpasswords for security (ha!)15:28
jeblairclarkb: yeah, and you can easily get a new password with mm anyway.15:29
clarkb(and by default the script will only convert from latin1 to utf8)15:29
clarkbreading through it it seems to do what is advertises, and other than the non python like iteration I find little to complain about15:29
jeblairclarkb: ok, i'm going to run on the test server with doit now15:29
jeblairit completed without error15:30
jeblairand when i run it again in dry-run, it has nothing to do15:30
fungiooh, seems i got wrapped up in housework and missed the mailman excitement15:30
jeblairfungi: not over yet :)15:30
fungiyeah, just now catching up15:31
clarkbjeblair: http://lists-upgrade-test.openstack.org/cgi-bin/mailman/listinfo/openstack-vi looks better now15:31
jeblairclarkb: yep15:31
jeblairi *just* changed it back to double check that15:31
jeblairbecause i realized i forgot to check before15:31
jeblairbut i'll fix it again15:31
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Add experimental murano functional test gate.  https://review.openstack.org/45199615:32
jeblairre-fixed15:32
jeblairclarkb, mordred, fungi: i'm ready to run it on prod now.  shall i?15:33
clarkbjeblair: I think so, test server was happy and script seems to do what is advertised15:33
fungijeblair: seems safe, i say yea15:33
jeblairdry run looks the same15:34
jeblairwet run complete15:34
jeblairi will unshunt again now15:34
clarkbno more <?> on frontpage15:34
jeblairsudo -u list /var/lib/mailman/bin/unshunt15:34
jeblairbtw ^15:35
jeblair0 shunted messages15:35
pabelangerfor some reason we are doing a full release of mirror.pypi on afs02.dfw.o.o15:35
pabelangerStarting ForwardMulti from 536870932 to 536870932 on afs02.dfw.openstack.org (full release)15:35
jeblairpabelanger: "full release" doesn't mean what you think it means15:36
pabelangerwhich could be related to the pypi failures we are seeing15:36
jeblairpabelanger: it just means "not continuing a previous release"15:36
jeblairpabelanger: it does not mean "full copy of data"15:36
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Fix puppet-lint errors  https://review.openstack.org/45248715:36
clarkbjeblair: and the previous 3 messages were delivered?15:36
pabelangerjeblair: okay, is there a way to debug why that would happen?15:36
jeblair(we *might* be doing that, but there isn't a good way to tell other than to look at how long it's taking)15:36
jeblairpabelanger: i'm saying that "full release" is not a problem15:37
clarkbjeblair: I'm finding that navigating -fr as a non french speaker not as easy as I expected15:37
jeblairpabelanger: "full release" is "perfectly normal release"15:37
pabelangerok15:37
jeblairclarkb: i imagine that's the intent :)15:37
openstackgerritMerged openstack-infra/gerritbot master: Removed unreasonable banning of newer pbr versions  https://review.openstack.org/45192515:37
clarkbjeblair: touché15:37
pabelangerjeblair: perhaps this could be related? http://paste.openstack.org/show/605138/15:38
pabelangerthat was the previous vos release attempt15:38
*** marst has joined #openstack-infra15:38
jeblairclarkb: i don't think they were actual posts; maybe moderated or spam or something15:39
jeblairclarkb: at any rate, they got delivered and generated single-message responses15:39
jeblairApr 01 15:35:08 2017 (22573) <mailman.26.1491033608.20980.openstack-vi@lists.openstack.org> smtp to openstack-vi for 3 recips, completed in 0.147 seconds15:39
jeblairor 3 messages in that case ^15:39
jeblairso probably moderation notice15:39
jeblairclarkb: we successfully delivered spam.  :)15:39
jeblair(to the moderation queue)15:40
clarkbperfect15:40
funginifty15:40
jeblairpabelanger: what's the problem?15:41
fungias for upgrading lists.o.o again, going ahead and planning a move to xenial sounds prudent15:41
pabelangerjeblair: we just got a large spike of pypi mirror disconnecting connections for jobs. I am trying to see if we need to fix anything15:42
pabelangerhttp://status.openstack.org/elastic-recheck/#144913615:42
jeblairhttp://mirror.iad.rax.openstack.org/pypi/simple/packaging/ looks fine to me15:43
pabelangerya, not sure what happend, but a majority of our cloud mirrors had the issue about 10 mins ago15:46
jeblairtransient connectivity issue to rax-dfw?15:48
pabelangerseems so15:49
*** wolverineav has quit IRC15:52
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Add experimental murano functional test gate.  https://review.openstack.org/45199615:53
*** wolverineav has joined #openstack-infra15:55
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Enable multinode topology for slow Patrole tests.  https://review.openstack.org/45027815:56
*** thorst has joined #openstack-infra16:00
*** wolverineav has quit IRC16:02
*** thorst has quit IRC16:05
*** adrian_otto has joined #openstack-infra16:06
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: [WIP] testing keyscan failures  https://review.openstack.org/45248916:13
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Set socket timeout for SSH keyscan  https://review.openstack.org/45147016:14
*** chem_gon` has joined #openstack-infra16:37
*** chem_gone has quit IRC16:39
*** jascott1_ has joined #openstack-infra16:42
*** dimtruck is now known as zz_dimtruck16:45
*** ssbarnea has quit IRC16:46
*** jascott1_ has quit IRC16:47
*** ssbarnea has joined #openstack-infra16:47
*** ssbarnea has quit IRC16:47
*** knangia has joined #openstack-infra16:48
*** ssbarnea has joined #openstack-infra16:49
*** john-davidge has joined #openstack-infra16:53
*** zz_dimtruck is now known as dimtruck16:54
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Fix debian-jessie settings for devstack  https://review.openstack.org/45249116:56
*** john-davidge has quit IRC16:58
*** thorst has joined #openstack-infra17:01
*** adrian_otto has quit IRC17:04
*** thorst has quit IRC17:06
*** dimtruck is now known as zz_dimtruck17:06
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: WIP Fetch server console log if ssh connection fails  https://review.openstack.org/45249417:26
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: WIP Fetch server console log if ssh connection fails  https://review.openstack.org/45249417:30
*** harlowja has joined #openstack-infra17:30
*** harlowja has quit IRC17:34
*** dharinic has joined #openstack-infra17:34
*** mtreinish has quit IRC17:35
*** mtreinish has joined #openstack-infra17:38
*** e0ne has joined #openstack-infra17:40
*** Sukhdev has joined #openstack-infra17:52
*** neiljerram has joined #openstack-infra17:53
*** shu-mutou-AWAY has quit IRC17:53
*** ilpianista has quit IRC17:54
*** jzerebecki has quit IRC17:54
*** e0ne has quit IRC17:56
*** e0ne has joined #openstack-infra17:57
*** adrian_otto has joined #openstack-infra17:59
*** jzerebecki has joined #openstack-infra18:01
*** e0ne has quit IRC18:01
*** e0ne has joined #openstack-infra18:02
*** thorst has joined #openstack-infra18:02
*** e0ne has quit IRC18:02
*** e0ne has joined #openstack-infra18:03
*** e0ne has quit IRC18:03
*** e0ne has joined #openstack-infra18:03
*** e0ne has quit IRC18:04
*** neiljerram has quit IRC18:04
*** e0ne has joined #openstack-infra18:04
*** e0ne has quit IRC18:05
*** e0ne has joined #openstack-infra18:05
*** e0ne has quit IRC18:05
*** adrian_otto has quit IRC18:06
*** ilpianista has joined #openstack-infra18:06
*** thorst has quit IRC18:06
*** benj_ has quit IRC18:11
*** benj_ has joined #openstack-infra18:12
*** yamahata has joined #openstack-infra18:14
*** zz_dimtruck is now known as dimtruck18:25
*** smarcet has joined #openstack-infra18:58
*** benj_ has quit IRC18:59
*** smarcet has quit IRC19:02
*** thorst has joined #openstack-infra19:03
*** gouthamr has joined #openstack-infra19:06
*** thorst has quit IRC19:07
*** wolverineav has joined #openstack-infra19:09
*** benj_ has joined #openstack-infra19:11
*** wolverineav has quit IRC19:14
*** Goneri has joined #openstack-infra19:16
*** Goneri has quit IRC19:18
*** john-davidge has joined #openstack-infra19:20
*** sheeprine has quit IRC19:24
*** john-davidge has quit IRC19:25
*** Sukhdev has quit IRC19:27
*** sheeprine has joined #openstack-infra19:28
*** thorst has joined #openstack-infra19:31
*** thorst has quit IRC19:31
*** kzaitsev_mb has quit IRC19:38
*** sshnaidm|off has joined #openstack-infra19:54
*** slagle has joined #openstack-infra20:04
*** ccamacho has quit IRC20:10
*** wolverineav has joined #openstack-infra20:22
*** wolverineav has quit IRC20:27
*** ccamacho has joined #openstack-infra20:27
*** thorst has joined #openstack-infra20:32
*** eroux has joined #openstack-infra20:36
*** rbuzatu_ has joined #openstack-infra20:45
*** rbuzatu has quit IRC20:49
*** thorst has quit IRC20:51
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: WIP Fetch server console log if ssh connection fails  https://review.openstack.org/45249421:06
*** thorst has joined #openstack-infra21:12
*** thorst has quit IRC21:12
*** gouthamr has quit IRC21:29
*** adrian_otto has joined #openstack-infra21:31
*** Gravious has joined #openstack-infra21:31
*** dharinic has quit IRC21:39
*** Julien-zte has quit IRC21:43
*** Julien-zte has joined #openstack-infra21:45
*** esberglu has joined #openstack-infra21:46
*** john-davidge has joined #openstack-infra21:47
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: WIP Fetch server console log if ssh connection fails  https://review.openstack.org/45249421:49
*** rbuzatu_ has quit IRC21:52
*** john-davidge has quit IRC21:52
*** Julien-zte has quit IRC22:01
*** ccamacho has left #openstack-infra22:11
*** ccamacho has quit IRC22:12
*** rbuzatu has joined #openstack-infra22:12
*** thorst has joined #openstack-infra22:13
*** rbuzatu has quit IRC22:16
*** thorst has quit IRC22:18
openstackgerritSam Yaple proposed openstack-infra/project-config master: Publish LOCI tarball to tarballs.o.o  https://review.openstack.org/45229922:20
*** adrian_otto has quit IRC22:21
openstackgerritMerged openstack-infra/project-config master: Add log publisher for LOCI  https://review.openstack.org/45222922:34
*** dharinic has joined #openstack-infra22:38
*** esberglu has quit IRC22:45
*** esberglu has joined #openstack-infra22:46
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Fetch server console log if ssh connection fails  https://review.openstack.org/45249422:49
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Fetch server console log if ssh connection fails  https://review.openstack.org/45249422:49
*** esberglu has quit IRC22:50
*** ssbarnea has quit IRC22:52
*** ssbarnea has joined #openstack-infra22:53
*** ssbarnea has quit IRC22:53
*** rbuzatu has joined #openstack-infra22:53
*** ssbarnea has joined #openstack-infra22:55
*** rbuzatu has quit IRC22:58
*** neiljerram has joined #openstack-infra23:08
*** esberglu has joined #openstack-infra23:13
*** thorst has joined #openstack-infra23:14
*** gouthamr has joined #openstack-infra23:15
*** thorst has quit IRC23:18
*** jascott1_ has joined #openstack-infra23:31
*** jascott1_ has quit IRC23:35
*** jascott1_ has joined #openstack-infra23:36
*** jascott1_ has quit IRC23:41
*** neiljerram has quit IRC23:50
*** dimtruck is now known as zz_dimtruck23:51
*** asselin has joined #openstack-infra23:52
*** tosky has quit IRC23:54
*** asselin_ has quit IRC23:54
*** rbuzatu has joined #openstack-infra23:55
*** rbuzatu has quit IRC23:59
*** sekelso has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!