Friday, 2017-03-31

*** sekelso has joined #openstack-infra00:00
mordredmnaser: yes, you should00:03
mnaserthere's the one where it fails to find a proper endpoint00:03
*** skelso has quit IRC00:03
mordredI do not like that one00:04
mordredor where your auth info is wrong and it tells you something about not being able to load a plugin00:04
mordredbut it also might be that there is no appropriate endpoint, iirc00:04
* mordred shoudl really file a bug about that next time he notices it00:05
*** neiljerram has quit IRC00:05
pabelangermordred: do you mind a +3
pabelangeradds infracloud mirrors to cacti00:06
*** funzo has joined #openstack-infra00:06
SamYaplemordred: uhh you are noticing now. now is the time maaaan!00:09
*** rfolco has joined #openstack-infra00:11
lifelessfungi: well, we didn't punt, we took existing specification :)00:12
*** funzo has quit IRC00:12
lifelessfungi: what I'm seeing is an invalid version being returned00:13
lifelessby dpkg00:13
jheskethfungi: by the way, if you're still looking for people to help with the on-boarding at the forum/summit, I'd be happy to00:15
*** baoli has quit IRC00:16
pabelangerjhesketh: sign me up too!00:16
*** ramishra has quit IRC00:17
*** baoli has joined #openstack-infra00:17
openstackgerritMerged openstack-infra/bindep master: Bump lower cap of PBR to 1.3
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Remove support for nodepool_id
*** tphummel has joined #openstack-infra00:18
*** baoli has quit IRC00:21
*** jamielennox is now known as jamielennox|away00:25
*** adrian_otto has quit IRC00:26
*** stakeda has joined #openstack-infra00:27
*** funzo has joined #openstack-infra00:28
*** funzo has quit IRC00:30
*** jamielennox|away is now known as jamielennox00:31
pabelangerjhesketh: ianw: if you get a chance today:
jheskethpabelanger: sure00:32
pabelangerjhesketh: ianw: we also enabled proxy caching on our mirrors today:
*** zhurong has joined #openstack-infra00:32
jheskethpabelanger: done00:32
pabelangerso we want to monitor HDD usage over the next few days00:32
pabelangerjhesketh: danke00:32
jheskethyep, we should have things in cacti anyway00:32
ianwpabelanger: yeah, saw some of that in scrollback.  just for rdo?00:33
pabelangerianw: ya, we are experimenting atm. since we only have so much storage on our mirrors. However, we've talked about doing too00:34
pabelangerianw: long term for RDO is to move them into AFS00:34
ianwcool, that's what i thought00:34
pabelangerproxy cache is a short term solution for things00:34
pabelangerianw: I'll be testing that tomorrow00:35
*** tuanluong has joined #openstack-infra00:35
*** MasterOfBugs has quit IRC00:35
openstackgerritMerged openstack-infra/system-config master: Add infracloud mirrors to cacti.o.o
*** thorst has joined #openstack-infra00:38
openstackgerritEric Kao proposed openstack-infra/project-config master: add coverage-jobs to congress
*** thorst has quit IRC00:39
*** rossella_s has quit IRC00:42
*** rossella_s has joined #openstack-infra00:43
*** funzo has joined #openstack-infra00:45
*** jerryz has quit IRC00:47
*** gongysh has joined #openstack-infra00:47
*** zz_dimtruck is now known as dimtruck00:48
*** jerryz has joined #openstack-infra00:48
*** dharinic has joined #openstack-infra00:51
*** armax has quit IRC00:54
*** jerryz has quit IRC00:54
*** yuanying has quit IRC00:58
*** tphummel has quit IRC01:06
*** thorst has joined #openstack-infra01:10
*** yuanying has joined #openstack-infra01:10
*** liujiong has joined #openstack-infra01:10
*** jcoufal has joined #openstack-infra01:11
*** EricGonczer_ has joined #openstack-infra01:11
*** aeng_ has quit IRC01:13
*** dprince has joined #openstack-infra01:15
*** gongysh has quit IRC01:16
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config master: Add functional job for k8s-cloud-provider with sudo privileges
*** jcoufal has quit IRC01:17
SamYapledoes anyone have a list of the current infra apt mirrors?01:21
*** liusheng has joined #openstack-infra01:21
mordredSamYaple: like the regions? or which repos we mirror?01:21
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config master: Add functional job for k8s-cloud-provider with sudo privileges
SamYaplemordred: which repos01:22
mordredSamYaple: will let you browse what we mirror - although you can also install openafs-client on your computer and do ls /afs/
SamYaplemordred: awesome. yea thats what i needed01:23
*** ijw has joined #openstack-infra01:23
SamYapleinteresting naming scheme01:24
SamYaplemordred: supposing i wanted to append to this list, is that system-config? where would the mirroring code for this be since its been over a year since ive looked at it01:25
*** dprince has quit IRC01:26
*** bigjools has joined #openstack-infra01:27
*** thorst has quit IRC01:28
SamYapleoophf. mordred that ceph repo is very out of date01:30
*** rbuzatu has quit IRC01:31
*** thorst has joined #openstack-infra01:37
*** thorst has quit IRC01:38
*** Apoorva has quit IRC01:50
*** john-davidge has joined #openstack-infra01:54
*** oanson has quit IRC01:55
*** oanson has joined #openstack-infra01:56
*** john-davidge has quit IRC01:59
*** rfolco has quit IRC02:00
*** yamahata has quit IRC02:08
*** ijw has quit IRC02:08
*** hongbin has joined #openstack-infra02:20
openstackgerritMerged openstack-infra/tripleo-ci master: Use shorter timeout for update step
openstackgerritzhurong proposed openstack-infra/project-config master: Add install-guide-jobs for murano
*** smcginnis is now known as smcginnis_vacati02:27
*** rbuzatu has joined #openstack-infra02:32
openstackgerritMerged openstack-infra/subunit2sql master: Add run_failure_rate argument to specify values to use in graph
*** rfolco has joined #openstack-infra02:36
*** rfolco has quit IRC02:37
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects
*** thorst has joined #openstack-infra02:39
*** cody-somerville has quit IRC02:43
*** csomerville has joined #openstack-infra02:43
*** rfolco has joined #openstack-infra02:45
*** rfolco has quit IRC02:45
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql master: Fix run_failure_rate title parameter
*** thorst has quit IRC02:56
*** thorst has joined #openstack-infra02:57
*** hamzy has joined #openstack-infra02:58
*** thorst has quit IRC03:01
*** namnh has joined #openstack-infra03:01
*** dimtruck is now known as zz_dimtruck03:03
*** baoli has joined #openstack-infra03:07
*** yuanying has quit IRC03:11
*** yuanying has joined #openstack-infra03:14
*** baoli has quit IRC03:15
*** amotoki has quit IRC03:22
openstackgerritMerged openstack-infra/subunit2sql master: Fix run_failure_rate title parameter
*** arijr has joined #openstack-infra03:26
*** hongbin has quit IRC03:31
*** armax has joined #openstack-infra03:33
*** armax has quit IRC03:36
*** knangia has quit IRC03:41
*** sekelso has quit IRC03:42
*** amotoki has joined #openstack-infra03:50
*** amotoki has quit IRC03:56
*** amotoki has joined #openstack-infra04:00
*** yamamoto has joined #openstack-infra04:01
*** thorst has joined #openstack-infra04:01
*** ihrachys has joined #openstack-infra04:02
*** ijw has joined #openstack-infra04:02
*** Qiming has quit IRC04:04
*** tinwood has quit IRC04:05
*** Qiming has joined #openstack-infra04:06
*** tinwood has joined #openstack-infra04:06
*** thorst has quit IRC04:06
*** zhurong has quit IRC04:09
*** EricGonczer_ has quit IRC04:10
*** tellesnobrega has quit IRC04:12
*** masayukig has quit IRC04:14
*** gouthamr has quit IRC04:14
*** tellesnobrega has joined #openstack-infra04:18
*** ramishra has joined #openstack-infra04:18
*** john-davidge has joined #openstack-infra04:21
*** amotoki has quit IRC04:22
*** armax has joined #openstack-infra04:26
*** john-davidge has quit IRC04:26
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron
*** ihrachys has quit IRC04:30
*** zhurong has joined #openstack-infra04:30
*** psachin has joined #openstack-infra04:41
*** yamamoto has quit IRC04:42
*** yamamoto has joined #openstack-infra04:44
*** nadya has joined #openstack-infra04:52
*** nadya has quit IRC04:54
*** dhajare has joined #openstack-infra04:56
*** thorst has joined #openstack-infra05:02
*** thorst has quit IRC05:07
*** yamamoto has quit IRC05:09
*** frickler has quit IRC05:11
*** frickler has joined #openstack-infra05:12
*** ijw has quit IRC05:15
*** tellesnobrega has quit IRC05:16
*** vikrant has joined #openstack-infra05:17
*** arijr has quit IRC05:20
*** dharinic has quit IRC05:22
*** prateek has joined #openstack-infra05:23
*** rlandy has quit IRC05:24
*** e0ne has joined #openstack-infra05:25
*** e0ne has quit IRC05:26
*** yamamoto has joined #openstack-infra05:28
*** tellesnobrega has joined #openstack-infra05:29
*** zhurong has quit IRC05:41
*** yamamoto has quit IRC05:41
*** Hal has quit IRC05:43
*** rcernin has joined #openstack-infra05:44
*** adrian_otto has joined #openstack-infra05:52
*** yolanda has joined #openstack-infra05:54
yolandahi AJaeger_ around? need reviews on and
*** jaosorior has joined #openstack-infra05:56
*** thorst has joined #openstack-infra06:03
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql master: Disable SmartyPants for docs
*** andreas_s has joined #openstack-infra06:04
*** zhurong has joined #openstack-infra06:05
*** amotoki has joined #openstack-infra06:05
*** amotoki_ has joined #openstack-infra06:08
*** thorst has quit IRC06:08
*** nadya has joined #openstack-infra06:08
*** amotoki has quit IRC06:10
Dinesh_BhorAJaeger_: Hi, Could you please visit this?
*** claudiub has joined #openstack-infra06:21
*** chmouel has left #openstack-infra06:33
*** pgadiya has joined #openstack-infra06:40
*** yamamoto has joined #openstack-infra06:42
*** eroux has joined #openstack-infra06:47
*** markus_z has joined #openstack-infra06:48
yolandahi ianw , thanks. Care reviewing as well?06:49
*** yamamoto has quit IRC06:49
Dinesh_Bhorianw: Hi Ian, It will be great if you visit this:
*** d0ugal has joined #openstack-infra06:52
*** d0ugal has quit IRC06:52
*** d0ugal has joined #openstack-infra06:52
*** d0ugal has quit IRC06:54
*** armax has quit IRC06:56
*** martinkopec has joined #openstack-infra06:57
*** d0ugal has joined #openstack-infra06:58
*** d0ugal has quit IRC06:58
*** d0ugal has joined #openstack-infra06:58
*** florianf has joined #openstack-infra06:58
*** pgadiya has quit IRC07:01
*** thorst has joined #openstack-infra07:04
*** hashar has joined #openstack-infra07:08
*** nadya has quit IRC07:08
*** tesseract has joined #openstack-infra07:08
*** thorst has quit IRC07:09
*** pgadiya has joined #openstack-infra07:10
*** rbuzatu has quit IRC07:12
*** rbuzatu has joined #openstack-infra07:12
*** jpich has joined #openstack-infra07:14
*** matrohon has joined #openstack-infra07:16
*** dharinic has joined #openstack-infra07:16
*** dhajare has quit IRC07:17
*** rbuzatu has quit IRC07:18
openstackgerritDirk Mueller proposed openstack/diskimage-builder master: DNM: give me more debug
*** d0ugal has quit IRC07:22
*** dharinic has quit IRC07:23
*** alexchadin has joined #openstack-infra07:24
*** Qiming has quit IRC07:26
*** d0ugal has joined #openstack-infra07:26
*** d0ugal has quit IRC07:26
*** d0ugal has joined #openstack-infra07:26
*** pcaruana has joined #openstack-infra07:27
gibihi! I see massive amount of POST_FAILUREs in the check queue. Is it already a known issue?07:28
*** Qiming has joined #openstack-infra07:32
*** dhajare has joined #openstack-infra07:33
andreas_sgibi, facing that as well...07:34
*** ralonsoh has joined #openstack-infra07:34
yolandahitting the same, going to take a look07:35
*** ralonsoh_ has joined #openstack-infra07:35
gibithanks for looking into it. If I can help just let me know.07:38
*** ralonsoh has quit IRC07:39
yolanda#status alert Jobs in gate are failing with POST_FAILURE. Infra roots are investigating07:40
openstackstatusyolanda: sending alert07:40
*** dhajare has quit IRC07:40
*** chem has joined #openstack-infra07:42
-openstackstatus- NOTICE: Jobs in gate are failing with POST_FAILURE. Infra roots are investigating07:42
*** ChanServ changes topic to "Jobs in gate are failing with POST_FAILURE. Infra roots are investigating"07:42
yolandaok seems the logs volume is corrupted, going to fix07:43
*** ralonsoh_ is now known as ralonsoh07:44
openstackstatusyolanda: finished sending alert07:46
*** nadya has joined #openstack-infra07:50
*** katkapilatova has quit IRC07:54
*** rcernin has quit IRC07:54
*** dtantsur has quit IRC07:54
*** tesseract has quit IRC07:54
*** katkapilatova has joined #openstack-infra07:55
*** rcernin has joined #openstack-infra07:55
*** matrohon has quit IRC07:55
*** tesseract has joined #openstack-infra07:56
*** matrohon has joined #openstack-infra07:56
*** nadya has quit IRC07:56
*** dhajare has joined #openstack-infra07:57
*** dtantsur has joined #openstack-infra07:57
*** zzzeek has quit IRC08:00
*** nadya has joined #openstack-infra08:03
*** zzzeek has joined #openstack-infra08:04
*** Douhet has quit IRC08:05
*** thorst has joined #openstack-infra08:05
*** nadya has quit IRC08:08
*** Julien-zte has joined #openstack-infra08:09
*** jogo has quit IRC08:13
*** rbuzatu has joined #openstack-infra08:15
openstackgerritRodolfo Alonso Hernandez proposed openstack-infra/bindep master: Update 'pbr' version requirements
*** jogo has joined #openstack-infra08:15
*** vsaienk0 has joined #openstack-infra08:15
vsaienk0AJaeger_:  yolanda please review move ironic-standalone to voting mode, and disable some of ironic jobs...08:16
*** ccamacho has joined #openstack-infra08:17
*** Douhet has joined #openstack-infra08:18
*** rbuzatu has quit IRC08:19
*** rbuzatu has joined #openstack-infra08:19
yolanda#status alert has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed08:20
openstackstatusyolanda: sending alert08:20
ianwyolanda: did you reboot or something?08:21
openstackgerritBogdan Dobrelya proposed openstack-infra/tripleo-ci master: Adapt getthelogs UX for more use cases
-openstackstatus- NOTICE: has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed08:23
*** ChanServ changes topic to " has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed"08:23
*** thorst has quit IRC08:24
*** e0ne has joined #openstack-infra08:24
*** shardy has joined #openstack-infra08:26
*** lucas-afk is now known as lucasagomes08:26
openstackstatusyolanda: finished sending alert08:27
ianwah, it's booting08:28
ianwwe should disable graphical bootup08:28
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Update pip in DLRN venv
*** derekh has joined #openstack-infra08:32
*** dhajare has quit IRC08:35
*** shardy has quit IRC08:37
*** shardy has joined #openstack-infra08:38
*** rossella_s has quit IRC08:42
yolandaianw, it seems to be repairing the logs filesyste, but is taking ages08:43
*** rossella_s has joined #openstack-infra08:43
ianwyeah, it will ... there was a ticket saying something was up with block storage08:44
*** nadya has joined #openstack-infra08:46
*** dhajare has joined #openstack-infra08:49
ttxwe seem to have lost other websites (governance, tarballs, releases) which probably run from trhe same host08:50
ttxif you confirm it's related, I can send a notice so that people know08:51
*** matrohon has quit IRC08:51
*** matrohon has joined #openstack-infra08:52
yolandayes, the static vm cannot boot, as it's having problems with the logs filesystem08:53
yolandattx, however, can you wait a bit? seems the repair finished08:53
ianwyolanda: hmm, sorry, i tried switching virtual terminals (ctl-alt-f1)08:53
ianwi was expecting to see status but it seems blank08:54
ttxah sure08:54
*** ssbarnea has quit IRC08:54
yolandaianw, so not finished yet?08:54
yolandacan you restart the fsck for logs?08:54
ianwyolanda: i dunno ... are you watching the console?08:55
ianwok, so the graphical bit is console 708:56
ianwok, i just made it skip08:57
yolandaok, repairing logs volume08:58
yolandabut at least other services will be up08:58
ianwyolanda: ok, are you running it in a screen?08:58
ianwdoesn't matter, i usually just run in a screen with verbose and it gives you a little clue08:59
ianwas to how long08:59
yolandagoing to do it08:59
*** matrohon has quit IRC08:59
*** ssbarnea has joined #openstack-infra09:02
*** pgadiya has quit IRC09:02
gcbclarkb: yes, we should use released version of oslo libraries,   we test our master branch with peridic jobs like
gcbclarkb: please add me as reviewer in your change09:05
*** dizquierdo has joined #openstack-infra09:06
*** baoli has joined #openstack-infra09:07
*** arxcruz|pto has quit IRC09:11
*** baoli has quit IRC09:12
ianwyolanda: i sent it sigusr1 to start outputting progress09:12
yolandaso slow...09:13
*** dhajare has quit IRC09:14
*** matrohon has joined #openstack-infra09:17
*** pgadiya has joined #openstack-infra09:18
*** annegentle has quit IRC09:18
*** arxcruz|pto has joined #openstack-infra09:19
*** gcb has quit IRC09:21
*** electrofelix has joined #openstack-infra09:21
openstackgerritAnusha Ramineni proposed openstack-infra/project-config master: Enable ironic service in congress gate job
ianwyolanda: did you reboot the server, or was it done automatically?09:25
*** yatinkarel is now known as yatin_away09:26
*** pgadiya has quit IRC09:26
yolandaianw, i did a restart because i was not able to proceed to the fsck. Although lsof/fuser was not showing any process using the volume, fsck was telling the filesystem was in use09:27
ianwok, np, just wondering if rax had done that after their maintenance09:27
*** matrohon_ has joined #openstack-infra09:28
*** peterlisak has quit IRC09:28
*** onovy has quit IRC09:28
openstackgerritMerged openstack-infra/project-config master: Add prep upper constraints macro to bifrost jobs
*** matrohon has quit IRC09:30
*** pgadiya has joined #openstack-infra09:30
*** dhajare has joined #openstack-infra09:31
*** Julien-zte has quit IRC09:31
*** stakeda has quit IRC09:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Add upper constraints limitation of buildimage jobs
*** tosky has joined #openstack-infra09:35
*** psachin has quit IRC09:36
*** yanyanhu has joined #openstack-infra09:37
ianwyolanda: i sent a mail about fiddling the boot config, since nobody else is awake :)09:38
*** peterlisak has joined #openstack-infra09:39
ianwyolanda: one thought is, are we going to be able to mount over /srv/static/logs, or do we basically have to reboot to free that?09:40
yolandaianw, in the past what i did is to copy the content of /srv/static/logs to some temporary directory. Then mount the volume, and copy contents from that temporary directory09:42
*** onovy has joined #openstack-infra09:43
*** sdague has joined #openstack-infra09:44
openstackgerritMerged openstack-infra/project-config master: Add upper constraints limitation of buildimage jobs
*** rossella_s has quit IRC09:46
*** john-davidge has joined #openstack-infra09:47
*** yanyanhu has quit IRC09:48
*** rossella_s has joined #openstack-infra09:48
*** adrian_otto has quit IRC09:53
*** psachin has joined #openstack-infra09:54
ianwyolanda: ok, so we don't have to stop zuul?09:55
yolandano, i don't think so09:55
yolandai won't remove the alert until we have the right volume on place, but things are passing nwo09:56
ianwok, i guess we get enough windows where things aren't copying there09:56
*** Hal has joined #openstack-infra09:58
*** sputnik13 has joined #openstack-infra10:02
*** matrohon_ has quit IRC10:08
*** matrohon_ has joined #openstack-infra10:08
openstackgerrityolanda.robla proposed openstack-infra/project-config master: Exporting UPPER_CONSTRAINTS_FILE on bifrost
*** e0ne has quit IRC10:13
*** Hal has quit IRC10:13
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects
*** amotoki_ has quit IRC10:15
*** Hal has joined #openstack-infra10:16
yolandaAJaeger_, ianw , care reviewing ?10:16
*** e0ne has joined #openstack-infra10:16
*** zhurong has quit IRC10:18
*** liujiong has quit IRC10:19
*** tuanluong has quit IRC10:20
*** nadya has quit IRC10:20
*** thorst has joined #openstack-infra10:21
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects
*** funzo has quit IRC10:23
*** GnomeKing has joined #openstack-infra10:23
*** namnh has quit IRC10:24
*** sambetts|afk is now known as sambetts10:25
*** thorst has quit IRC10:26
*** neiljerram has joined #openstack-infra10:28
*** ralonsoh_ has joined #openstack-infra10:30
*** pbourke has quit IRC10:30
*** pbourke has joined #openstack-infra10:32
*** tellesnobrega has quit IRC10:33
*** ralonsoh has quit IRC10:34
*** martinkopec has quit IRC10:35
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects
*** Julien-zte has joined #openstack-infra10:37
*** hashar is now known as hasharLunch10:37
*** matrohon_ has quit IRC10:38
*** Julien-zte has quit IRC10:39
*** Julien-zte has joined #openstack-infra10:40
*** zhurong has joined #openstack-infra10:41
*** alexchadin has quit IRC10:45
openstackgerritAlfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM Other test for 451445
*** Hal has quit IRC10:48
*** psachin has quit IRC10:50
*** nadya has joined #openstack-infra10:51
*** gcb has joined #openstack-infra10:51
*** psachin has joined #openstack-infra10:52
openstackgerritAlexander Evseev proposed openstack-infra/project-config master: Skip some bindeps for old Ubuntu releases
*** bhavik1 has joined #openstack-infra10:58
*** rhallisey has joined #openstack-infra10:59
dimsianw : fixed the name of the job per your comment - ci is green - thanks!11:01
dimsAJaeger_ : yolanda : one of you peek at that as well? will help get the ball rolling on running kubernetes' e2e tests11:02
openstackgerritMerged openstack-infra/project-config master: Exporting UPPER_CONSTRAINTS_FILE on bifrost
*** mat128 has joined #openstack-infra11:09
*** shardy is now known as shardy_lunch11:14
dimsthanks yolanda frickler11:14
*** funzo has joined #openstack-infra11:17
*** ccamacho is now known as ccamacho|lunch11:17
*** hasharLunch is now known as hashar11:19
*** gcb has quit IRC11:20
*** adarazs is now known as adarazs_lunch11:20
*** jkilpatr has joined #openstack-infra11:21
*** yanyanhu has joined #openstack-infra11:21
*** thorst has joined #openstack-infra11:22
openstackgerrityolanda.robla proposed openstack-infra/project-config master: Fix requirements checkout on buildimage
yolandaianw, AJaeger_ , still around? AJaeger_ , you mentioned we could have conflicts with the checkout of that macro, and we had:
frickleryolanda: ianw: any update on the fsck progress? how large is that volume, btw?11:25
*** yanyanhu has quit IRC11:26
yolandait's over 70%11:26
*** thorst has quit IRC11:27
fricklernice, that sounds like some reasonable progress from 8% :)11:28
yolandanot an uniform progress however, looks slower now11:28
*** lucasagomes is now known as lucas-hungry11:28
*** knangia has joined #openstack-infra11:29
*** zz_dimtruck is now known as dimtruck11:30
*** john-davidge has quit IRC11:32
*** thorst has joined #openstack-infra11:32
*** amotoki has joined #openstack-infra11:32
*** gcb has joined #openstack-infra11:33
*** alexchadin has joined #openstack-infra11:34
*** ldnunes has joined #openstack-infra11:38
*** matrohon has joined #openstack-infra11:38
*** john-davidge has joined #openstack-infra11:38
*** matrohon has quit IRC11:41
*** tellesnobrega has joined #openstack-infra11:42
*** pblaho has quit IRC11:45
*** dimtruck is now known as zz_dimtruck11:46
*** zz_dimtruck is now known as dimtruck11:46
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Switch OVB to use promoted repos from
openstackgerritMerged openstack-infra/tripleo-ci master: Update pip in DLRN venv
*** sekelso has joined #openstack-infra11:55
*** dimtruck is now known as zz_dimtruck11:56
*** mat128 has quit IRC11:56
*** mat128 has joined #openstack-infra11:58
*** skelso has joined #openstack-infra11:58
*** zhurong has quit IRC11:59
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Switch OVB to use promoted repos from
*** sekelso has quit IRC12:00
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Remove unused variables for Upgrades
*** prateek has quit IRC12:01
*** prateek has joined #openstack-infra12:01
*** shardy_lunch is now known as shardy12:03
*** florianf has quit IRC12:07
*** dprince has joined #openstack-infra12:08
mnasermorning folks, thanks for your hard (and late night) work on logs.o.o!12:08
mnaserhope things heal up soon12:08
fungifrickler: that volume is around 12tib12:09
*** aysyd has joined #openstack-infra12:11
pabelangerYa, didn't get a chance to solve our volume maintenance on static.o.o12:11
*** adreznec has quit IRC12:14
fungiwe didn't see an alternative to crossing our fingers and fsck'ing if it broke12:15
fungino room to add another cinder volume because of xen's virtual block device limit12:16
*** adreznec has joined #openstack-infra12:17
fricklerhow about splitting it up? like do log0-9.o.o and make logs.o.o proxy based on the first digit? or is there a specific reason it all need to be on one machine?12:18
*** pblaho has joined #openstack-infra12:18
*** adarazs_lunch is now known as adarazs12:19
fungifrickler: we've talked about some sharding solutions. possible to implement but most of the code complication will be on the write routing side, not routing reads12:19
Zarahi, earlier in #storyboard, witek wanted to add python-monascaclient to the monasca group, and I said I'd check if they needed to do anything more than add 'use-storyboard' to gerrit/projects.yaml12:19
Zarasoo, this is me checking12:19
Zaraas far as I'm aware, there are no bugs filed against it atm12:20
Zaraso they wouldn't need to import anything12:20
*** ccamacho|lunch is now known as ccamacho12:20
Zara is their patch12:20
fungiZara: thanks for the reminder, i meant to look into how i missed setting that in my patch when going through their governance deliverables12:21
Zaraah, great, thanks :)12:21
*** trown|outtypewww is now known as trown12:22
*** florianf has joined #openstack-infra12:22
*** lucas-hungry is now known as lucasagomes12:22
openstackgerrityolanda.robla proposed openstack-infra/project-config master: Fix requirements checkout on buildimage
ZaraI'm looking into sorting out the pynotedb infra a bit more; I got waylaid for a while12:23
yolandahi AJaeger_ , so i guess i missed the "requirements" folder?12:23
ZaraI got as far as making #pynotedb but nothing is officially registered or logged etc yet; not sure where to start with that12:24
*** pgadiya has quit IRC12:24
Zaraoh, and I need to sort out cores for the repo; I think it's just set to infra-root atm.12:27
*** signed8bit_Zzz is now known as signed8b_12:28
*** rook is now known as rook|tower12:31
dhellmanngovernance.o.o is responding extremely slowly for me. Is anyone else seeing an issue?12:32
ttxdhellmann: it's one of the static websites affected by the outage12:32
dhellmannI wasn't aware there was an outage.12:32 is out12:32
ttxor at least was a couple hours ago12:33
fungiyeah, there's a fsck of the logs volume underway, so starving i/o for other sites served from the same volume group in the meantime12:33
dhellmannok, as long as it's not a surprise to anyone12:34
yolandait's on 75% progress12:34
* fungi is not surprised, just always looks this way12:34
*** gordc has joined #openstack-infra12:34
yolandabut since it reached 70% , it became slower12:34
mnaserfsck is usually very unpredictable because the progress depends on how many inodes its going through12:35
dhellmannthat last 20% of a project always takes 80% of the time, right?12:35
mnaserif it reaches some project logs with a looooot of small files it'll bog down12:35
fungiin the past, fsck'ing that volume has taken the better part of a day12:35
fungiit's nothing but tons of small files (compressed logs)12:35
yolandafungi, looks like that. It's running on screen, and looks it won't finish before i'm EOD. Will you take care?12:36
fungiyolanda: happily--thanks for starting it!12:36
*** kashyap has left #openstack-infra12:36
yolandateam effort, over all shifts! :)12:36
fungizaro: yeah, that core group is still empty. give me a few to take care of rerunning our setup script for infra gerrit groups and let me know who you want added12:37
fungier, Zara: ^12:37
fungi(sorry zaro!)12:37
openstackgerritMerged openstack-infra/project-config master: Use StoryBoard for python-monascaclient
*** tphummel has joined #openstack-infra12:38
Zarahahaha, thanks :)12:39
*** chlong has joined #openstack-infra12:40
*** dmsimard|afk is now known as dmsimard12:41
fungicaught up on our review group setup now12:41
*** vicky has joined #openstack-infra12:41
fungiZara: you planning to be an initial core reviewer on that? who else?12:41
*** sshnaidm|off has quit IRC12:42
*** vicky has quit IRC12:43
*** vicky has joined #openstack-infra12:43
fungiZara: and as far as irc channel registration, see
*** vikrant has quit IRC12:44
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add reporter for Federated Message Bus (fedmsg)
Zarafungi: ah, thanks for that link. :) I think initial cores should be me + Gravious (richard ipsum)12:44
*** vikrant has joined #openstack-infra12:45
mordredfungi: ooh, neat - monasca moved to storyboard!12:46
fungiZara: done12:46
fungimordred: yup, i did the migration on monday (or maybe it was a week ago last monday?)12:47
Zarafungi: wonderful, thanks :D12:47
fungimordred: also the interop (defcore, refstack, et al) projects are sb as of the same migration timeframe12:47
Zaraa week ago last monday12:47
*** vicky has quit IRC12:48
Dinesh_BhorHi all, It will be great if someone approves this: It's already having one +2.12:48
*** rlandy has joined #openstack-infra12:48
*** signed8b_ is now known as signed8bit_Zzz12:48
mordredfungi: that's great news!12:49
fungiyolanda: i'm _finally_ attached to that screen session. the server is slower than even i would have expected. i'm going to double-check there's nothing else slowing it down even worse12:50
fungiwell, rather i've finally gotten to the point where i can run screen -x as root anyway (it hasn't returned yet)12:51
fungihuh, there are two screen sessions on there, both started within a minute of each other12:51
mnaserfungi i think something happened which made it worse in the past few minutes..12:52
mnaserlogs.o.o loaded at first and now it's zero response12:52
*** andreas_s has quit IRC12:52
mnaserthis morning i could get the the homepage / directoy listing there, now that doesnt seem to respond12:52
fungiyeah, just running things on the command line takes minutes to get a response12:53
yolandafungi, and is fsck causing it?12:53
fungiyolanda: if i could run commands i might be able to find out :/12:54
fungimy screen -x retry has been hanging for several minutes now12:56
*** flepied has quit IRC12:57
*** Goneri has joined #openstack-infra12:57
*** zz_dimtruck is now known as dimtruck12:58
fungicacti says something there has started using ~4gb of swap13:00
*** gouthamr has joined #openstack-infra13:01
Dinesh_Bhoryolanda: Hi, whenever you get time may I get your attention on this?
fricklermight be the fsck process eating up pretty much memory13:01
fungifrickler: might be all manner of tings. speculation is fun13:02
fungii'm trying to get a second ssh session to it, though i don't have high hopes13:03
funginope, connection reset by peer13:03
fungiin the past i've taken precautions before fsck'ing that volume, like making sure our log grooming cron wasn't running, disabling the jenkins ssh key so new logs couldn't be uploaded, disabling the logs vhost in apache, stopping puppet from updating...13:05
fungianything to free up rsources so that fsck would go faster13:06
*** e0ne has quit IRC13:06
fungiyeah, my "active" ssh session hasn't responded to me for a solid 15 minutes now, and additional attempts to ssh are getting connection resets (probably fighting all the jobs trying to upload logs)13:07
fungimay need to reboot the server, start all over and hope the interrupted fsck doesn't make matters worse13:08
fungiinfra-root: heads up13:08
fungii'm moving to #openstack-infra-incident for this13:08
*** xyang1 has joined #openstack-infra13:09
*** jamesdenton has joined #openstack-infra13:10
*** jamesden_ has joined #openstack-infra13:13
*** jcoufal has joined #openstack-infra13:13
*** esberglu has joined #openstack-infra13:14
*** flepied has joined #openstack-infra13:14
*** baoli has joined #openstack-infra13:14
*** jamesdenton has quit IRC13:14
*** baoli has quit IRC13:14
*** baoli has joined #openstack-infra13:15
*** kgiusti has joined #openstack-infra13:18
*** dave-mccowan has joined #openstack-infra13:18
*** signed8bit_Zzz is now known as signed8b_13:20
*** mdrabe has joined #openstack-infra13:20
andreykurilinclarkb: hi! Did you have a chance to verify my patch for features.yaml ? Maybe we can check it in our gates (there is a check to not apply custom features.yaml to master branch, so nothing critial will be affected)?13:22
sbezverkhello infra team, is it safe to recheck now, or you still chasing POST_FAILURE issue?13:23
sdakesbezverk the bots automatically change the subjects in all registered channels when things are fixed13:24
sdakefungi note - i'd recommend "mosh" which uses UDP for ssh connectivity - it can survive the worst of networking connections13:25
sdakefungi we use it in our labs when someone breaks routing/switching/etc (e.g. when packet loss > 75%)13:26
fungisdake: that's what i use for my personal systems, but isn't much help when the kernel is killing the sshd process13:26
sdakefungi right - just trying to help :)13:26
fungithis isn't a network connectivity issue13:26
fungiit's a server under extreme load and memory pressure issue13:26
sdakefungi i dont mean to interfere - however - if you can somehow get into the box, you might try a firewall on the ports used to upload things13:27
sdakefungi that may relieve pressure13:27
*** wolverineav has joined #openstack-infra13:28
*** AJaeger_ is now known as AJaeger13:28
pabelangerokay, who is an apache RewriteRules master? need some help turning /gem/gems/foobar.gem to /gem/gems/f/fo/foobar.gem currently I am testing with but having issues13:28
*** baoli has quit IRC13:28
openstackgerritMerged openstack-infra/project-config master: Add a DSVM job for golang projects
fungisdake: i already had a session on it but a command i was running was hanging. finally got some responsiveness back and confirmed the wsgi filter we use for log displays is competing with the fsck for available memory, so working on disabling that vhost in apache temporarily13:30
*** tellesnobrega has quit IRC13:31
sdakeits annoying that a journaling filesystem has file corruption - journaling is supposed to prevent that13:32
sdakeanyway on that note i'll leave ya to it- you know way more about this stuf fthen i do :)13:33
*** mariusv has quit IRC13:34
fricklerpabelanger: I'm not a master, but I'm willing to take a look as having that gem mirror working would be really useful to me. I'm about to leave now, though, but if you post what you currently have I'll give it a try later13:35
fungisdake: in this case the filesystem is on a logical volume spread across a dozen cinder volumes, so if we lose network access to one briefly and the filesystem gets remounted read-only by the kernel, we're uncomfortable putting it back writable until we fsck it. unfortunately it's ~12tib of very small files13:37
openstackgerritMerged openstack-infra/project-config master: Fix requirements checkout on buildimage
*** tbarron|out is now known as tbarron13:39
*** dhajare has quit IRC13:39
*** ralonsoh__ has joined #openstack-infra13:41
*** bhavik1 has quit IRC13:42
*** ralonsoh_ has quit IRC13:43
*** nadya has quit IRC13:44
*** rossella_s has quit IRC13:46
*** mrunge- is now known as mrunge13:47
*** rossella_s has joined #openstack-infra13:47
*** baoli has joined #openstack-infra13:47
*** ZZelle has quit IRC13:49
*** dizquierdo has quit IRC13:50
*** ZZelle has joined #openstack-infra13:50
openstackgerritAri proposed openstack-infra/jenkins-job-builder master: Add support for jms_messaging plugin as a trigger
*** ZZelle is now known as Guest1118213:50
*** ihrachys has joined #openstack-infra13:51
*** marst has quit IRC13:52
*** e0ne has joined #openstack-infra13:53
*** alexchadin has quit IRC13:53
*** e0ne has quit IRC13:53
*** mriedem has joined #openstack-infra13:53
*** amoralej is now known as amoralej|lunch13:54
*** Dinesh_Bhor has quit IRC13:55
*** dimtruck is now known as zz_dimtruck13:55
*** zz_dimtruck is now known as dimtruck13:55
*** e0ne has joined #openstack-infra13:56
*** e0ne has quit IRC13:56
*** tphummel has quit IRC13:57
ihrachysis it just me, or links to logs in gerrit are messed up?13:57
*** marst has joined #openstack-infra13:58
AJaegerihrachys: see $TOPIC13:58
ttxihrachys: /topic13:58
*** eharney has joined #openstack-infra13:58
openstackgerritRoman Podoliaka proposed openstack-infra/project-config master: osc-placement: experimental job for running of functional tests
*** amotoki has quit IRC13:59
*** sekelso has joined #openstack-infra14:04
*** skelso has quit IRC14:04
*** tphummel has joined #openstack-infra14:04
*** gcb has quit IRC14:05
fungiihrachys: the log prettifier consumes a ton of memory, so we have to disable the logs.o.o site in apache to free up enough for fsck unfortunately14:06
andreaffungi: is it just me or is getting redirected to ?14:08
*** gcb has joined #openstack-infra14:09
AJaegerandreaf: side effect of  $TOPIC14:09
andreafAJaeger: oh why can't I read :D - thank you14:10
*** asselin has quit IRC14:11
fungiandreaf: disabling the vhost (to reduce load from the log prettifier) is probably causing it to get redirected to the default vhost14:11
*** Guest11182 is now known as ZZelle_14:12
*** jamielennox is now known as jamielennox|away14:16
*** Julien-zte has quit IRC14:20
*** Julien-zte has joined #openstack-infra14:21
*** yamamoto has joined #openstack-infra14:21
openstackgerritAnton Studenov proposed openstack-infra/project-config master: [rally] Add hook for readthedocs
*** brianw has joined #openstack-infra14:24
*** arijr has joined #openstack-infra14:25
*** hashar has quit IRC14:26
*** sekelso has quit IRC14:28
*** arijr has quit IRC14:29
openstackgerritAdam Reznechek proposed openstack-infra/project-config master: Run check-requirements on remaining powervm projects
openstackgerritAri proposed openstack-infra/jenkins-job-builder master: Add support for jms_messaging plugin as a trigger
*** jroll has quit IRC14:32
*** jroll has joined #openstack-infra14:34
*** vikrant has quit IRC14:35
*** amoralej|lunch is now known as amoralej14:36
dmsimardJust a general curiosity - why is the gerrit merge strategy set to "merge if necessary" rather than "rebase if necessary" ?14:36
mordreddmsimard: because rebase if necessary creates a state where git pull may not work as a developer expects14:37
pabelangerand it would slow down things14:37
mordreddmsimard: (we had it set for a while on some of the infra repos as a trial run - and it turns out to confuse folks whose git-fu is lower)14:37
pabelangerif you want clean git logs, you likely want git cherry-pick14:38
*** rbrndt has joined #openstack-infra14:38
dmsimardfair, thanks for explanation :)14:38
mordredthere's another nice side-effect of merge-if-necessary, which is that gpg-signed commits make it into the tree with signatures intact14:39
*** prateek has quit IRC14:39
mordredso you get things like this nice verified badge over on github:
mnaseri should really get around setting up pgp14:40
mnaseri guess today could be the day while we wait for fsck :<14:41
dmsimardyeah I saw something about auto signing commits recently.. I been only signing tags14:41
mordreddmsimard: [commit]14:42
mordred        gpgsign = true14:42
mordredin ~/.gitconfig14:42
mordredand all works nicely14:42
*** askb has quit IRC14:42
mnaser(any recommended choice gpg docs for macos user?)14:42
dmsimardmordred: that's the same thing as doing git config --global :p14:42
*** yolanda has quit IRC14:43
mordredyou don't actually have to set signingkey, fwiw, unless git has a sad14:43
sdakedoes anyoen know if its ok to submit reviews and not cause infra problems on the recovery on logs.oo?14:43
mordredmnaser: sorry - I know nothing about macos)14:43
*** jaypipes has joined #openstack-infra14:43
mnaserim sure most *nix tooling will still work and exists14:44
*** jaypipes is now known as leakypipes14:44
openstackgerritSam Yaple proposed openstack-infra/project-config master: Add log publisher for LOCI
mordredmnaser: I certainly hope so14:45
*** dimtruck is now known as zz_dimtruck14:45
*** zz_dimtruck is now known as dimtruck14:45
*** chem has quit IRC14:46
*** chem has joined #openstack-infra14:46
*** yolanda has joined #openstack-infra14:47
fungisdake: i don't think a handful of people avoiding uploading changes will be much help. just be prepared that you probably won't be able to look at the ci logs for it for a while still (or possibly at all depending on if we run out of room to spool them while waiting for the fsck to finish)14:48
*** sekelso has joined #openstack-infra14:49
SamYapleis the 'link-logs' builder needed for the devstack-logs publisher?14:50
fungiSamYaple: link-logs builder just echoes the url for the logs into the console log14:51
pabelangerlink-logs just shows you the URL of the logs when we upload them14:51
sdakefungi ok thanks14:52
fungiSamYaple: it's mostly in case you're streaming the log console and want to know where the logs will wind up, without having to bounce back through the review and find the url linked there14:52
*** rossella_s has quit IRC14:53
SamYapleoh i see. that is something i do14:53
SamYapleill add that in14:53
openstackgerritSam Yaple proposed openstack-infra/project-config master: Add log publisher for LOCI
*** dimtruck is now known as zz_dimtruck14:54
*** xarses has joined #openstack-infra14:56
*** xarses_ has joined #openstack-infra14:57
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/bindep master: bindep: depends: Add distro family aliases
*** xarses has quit IRC15:01
*** xarses has joined #openstack-infra15:01
*** ijw has joined #openstack-infra15:01
*** xarses_ has quit IRC15:02
*** xarses_ has joined #openstack-infra15:03
*** xarses has quit IRC15:05
*** xarses has joined #openstack-infra15:06
*** xarses has quit IRC15:06
*** xarses has joined #openstack-infra15:07
*** xarses_ has quit IRC15:08
*** rcernin has quit IRC15:08
*** nadya has joined #openstack-infra15:11
*** admcleod is now known as not_adam15:12
*** psachin has quit IRC15:13
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config master: Allow developers to recheck patches with emotion in the check pipeline
*** not_adam is now known as admcleod15:17
*** dolphm has left #openstack-infra15:18
*** jcoufal has quit IRC15:18
*** e0ne has joined #openstack-infra15:18
*** jcoufal has joined #openstack-infra15:19
AJaegerIs it already first of April? ^15:20
*** sekelso has quit IRC15:20
*** zz_dimtruck is now known as dimtruck15:20
AJaegerdmsimard: what timezone are you in?15:20
SamYapleAJaeger: 7 hours away friend15:20
dmsimardAJaeger: git commit --date="Sat Apr 1 14:00 2017 +0100"15:21
dmsimardI cheated because of that intel guy15:21
AJaegerdmsimard: Commit time is different ;)15:21
AJaegerdmsimard: yeah...15:21
SamYaplei say we merge it. its before april15:21
jrollthat is beautiful15:21
*** yamamoto has quit IRC15:21
*** armax has joined #openstack-infra15:22
*** hongbin has joined #openstack-infra15:23
mordredI kind of think we should merge it regardless of the date15:24
SamYaplemordred: agreed. i dont see how this could spiraling out of control at all15:24
dmsimardProbably need to fix the regex if we really want it, it's bound to pick up those unescaped characters :)15:24
sdaguemordred: yeh, I'd be totally ok keeping that as an easter egg15:25
*** yamamoto has joined #openstack-infra15:25
*** sekelso has joined #openstack-infra15:25
sdagueyeh, I don't think that regex would actually work at all :)15:25
AJaegerwhat about removing recheck and reverify completely? ;)15:25
SamYaplewhat about bringing back reverify....15:26
dmsimardOH, this guy ^15:26
* dmsimard misses reverify15:26
*** adrian_otto has joined #openstack-infra15:26
dmsimardwait, let me do a reverify and go through the whole check AND gate pipeline again15:26
SamYapleoh look ive made myself sad15:27
*** jaosorior has quit IRC15:28
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci master: Support container upgrades in multinode OOOQ CI
*** e0ne has quit IRC15:39
mriedemdmsimard: recheck == reverify15:41
dmsimardmriedem: it wasn't always that way15:41
mriedemas in reverify was gate only?15:41
dmsimardreverify used to "recheck" through the gate pipeline only for jobs that failed the gate pipeline15:41
mriedemi believe that was removed for a good reason15:42
mriedemso things had to go back through check15:42
mriedemi'm sure sdague has the details15:42
*** tonytan4ever has joined #openstack-infra15:42
*** apevec has joined #openstack-infra15:43
clarkbfungi: reading up on scrollback it looks like fscking is under control (but slow)? also what is spooling writes (you mentioned it in response to sdake)15:43
apevecrelease.o.o is down15:43
apevecah it's back15:44
apevec( )15:44
fungiapevec: i brought it back up again. we're playing with stopping apache on the server to relieve load during the logs fsck15:44
mnaserapevec i had issues with tarballs.o.o when trying to build images so im putting that aside till fsck's are done15:44
fungiclarkb: the /srv/static volume which is 99gb and normally unused15:44
clarkbalso apparently not in the incident channel15:44
clarkber I'm not in15:44
clarkbmriedem: dmsimard we require check to pass before gating. Which essentially made reverify == recheck15:45
*** pcaruana has quit IRC15:46
dmsimardclarkb: talking about reviews that passed check but failed gate (bad mirror or whatever)15:46
*** tonytan_brb has joined #openstack-infra15:46
dmsimardIt feels wasteful (both in resources and time) to make it go through the check pipeline once more15:46
clarkbdmsimard: I think what we found was it resulted in a significantly more reliable gate pipeline so was worth it15:47
fungidmsimard: yep, pretty impossible to systematically differentiate those from jobs which failed because the change ceased to be valid between when it went through check and (maybe weeks or months later) through the gate15:47
clarkbdmsimard: that and the window scaling of the gate removed the massive gate pipelines and resets that were much more problematic15:47
*** yamamoto has quit IRC15:47
dmsimardfair if the pros outweighs the cons :)15:47
clarkbdmsimard: you have to remember that a single gate failure restarts everything behind it15:48
clarkbdmsimard: so cost of failing in gate is really high compared to run jobs for single change again if it is flaky15:48
fungidmsimard: lots of projects, even big projects, have core reviewers approving changes which have months-old check results15:48
*** tonytan4ever has quit IRC15:48
*** yamamoto has joined #openstack-infra15:48
fungiand so reverify would be a means of reenqueuing those into the gate and causing gate resets over and over instead of just once before it would have to go back through and get a clean check15:49
*** armax has quit IRC15:49
funginot a hypothetical. it absolutely was happening quite a lot15:49
*** jordanP has quit IRC15:49
*** armax has joined #openstack-infra15:50
*** yamamoto has quit IRC15:50
*** yamamoto has joined #openstack-infra15:50
*** yamamoto has quit IRC15:50
*** ralonsoh__ is now known as ralonsoh15:54
markmcclainI think something is broken for posting logs... getting 404 for  also had POST_REVIEW failures earlier too15:54
ihrachysmarkmcclain: see topic15:54
*** Swami has joined #openstack-infra15:55
markmcclainihrachys: ah thanks15:55
*** ramishra has quit IRC15:55
*** arxcruz|pto has quit IRC15:56
*** armax has quit IRC15:57
openstackgerritAri proposed openstack-infra/jenkins-job-builder master: Add support for jms_messaging plugin as a trigger
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add apache rewrite rule for ruby gems
*** armax has joined #openstack-infra15:58
pabelangerfungi: clarkb: okay, it took some testing locally, but ^ is our rewrite rules15:58
pabelangerfor gems15:58
pabelangerif people want to confirm, I'll update my PR with that documentation15:58
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add apache rewrite rule for ruby gems
*** armax has quit IRC15:59
*** rbuzatu_ has joined #openstack-infra16:03
*** rbuzatu has quit IRC16:04
clarkbpabelanger: is the type of corner case I need to reason about16:04
clarkbpabelanger: or do they get mirrored differently?16:05
clarkbyour example url is flat16:05
*** arxcruz|pto has joined #openstack-infra16:05
pabelangerclarkb: yes, they are just mirrored into a flat dir16:05
pabelangergive me a sec and I will get the hash for that16:06
clarkbpabelanger: is that /gems/a-0.1.1.gem on disk?16:06
*** EricGonczer_ has joined #openstack-infra16:06
*** gcb has quit IRC16:07
clarkb(it fails the rewrite rules if hosted at the path I pasted)16:07
pabelangerclarkb: /gems/a/a-/a-0.1.1.gem16:07
clarkber rewrite cond so no rule executed16:07
clarkbok thanks16:07
*** EricGonc_ has joined #openstack-infra16:09
*** gcb has joined #openstack-infra16:09
*** rbuzatu_ has quit IRC16:11
*** EricGonczer_ has quit IRC16:11
clarkbpabelanger: I posted a question (was easier on the review)16:12
pabelangerclarkb: let me test16:12
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add apache rewrite rule for ruby gems
pabelangerclarkb: works16:14
pabelangeralso added 2nd 1/2 of specfiles16:14
*** jpich has quit IRC16:16
clarkbpabelanger: why is the specfile pathing different than gems?16:18
*** derekh has quit IRC16:18
clarkbif they were the same you could do something like gem/(gems|quick)/commonregexinghere16:19
pabelangerclarkb: that is how rubymirror does it16:19
clarkbpabelanger: but didn't you add it to rubymirror?16:19
pabelanger1 sec16:19
pabelangeris my pull request16:19
pabelangeryou can see how I did it16:19
pabelangerbasically, I did easy mode and just hashed the existing directory structure16:19
pabelangerbecause quick seems to have multiple formats16:20
clarkbso quick/Marshal is a fixed part of the path?16:21
clarkband then you only hash the g_spec?16:21
pabelangerthe client will pass that16:21
*** markus_z has quit IRC16:22
clarkbI see16:22
pabelangerhowever, I am not sure what is using marshal things16:22
pabelangerbut was minimal to mirror16:22
clarkband there are different marshall versions so you can't just fix that dir16:22
clarkbin other news did you hear twitter is a main communication tool for infra?16:23
clarkbpabelanger: biggest apache cache dir I see is 431M btw16:24
clarkbpabelanger: if that ends up being reasonably steady state at half a gig I think we are good to go16:24
pabelangerclarkb: I am not sure, maybe EmilienM or mwhahaha know ^16:24
clarkbwe can also reduce cache retention to 8 hours (from 24 hours)16:24
clarkbif we have trouble16:24
pabelangerclarkb: oh, where is that dir?16:24
clarkbpabelanger: /opt/apache_cache16:25
clarkbpabelanger: on each of the mirror hosts16:25
clarkboh thatspecific one?16:25
clarkbsorry its early morning16:25
sdaguedmsimard: what was happening with reverify is people were getting -1s and just reverifying them back into the gate, even though the patch could no longer pass (a requirements issue, or something else merged in front of it)16:25
pabelangerchecking cacti16:25
pabelangerosic already choppy16:26
pabelangermust be apache logs16:26
clarkbya I think its logs16:26
pabelangerclarkb: back to gems, I can hard code marshal path16:26
pabelangerto avoid some regex, if that was what you are asking16:26
clarkbpabelanger: its fine I just wasn't understanding why it was special16:26
mwhahahapabelanger: is the question what uses the Marhsal gem? i'm not sure16:26
clarkband the problem is its an entirely new flat space under each marshall thing16:26
pabelangermwhahaha: ya, or if you need gemspec files for anything16:26
mwhahahawe shouldn't need gemspecs i don't think16:27
*** esberglu has quit IRC16:27
pabelangerclarkb: we could drop the apache rule for gemspec things first16:27
pabelangerand if not needed, I can add flag to disable mirroring that16:27
mwhahahawe just use gems and i don't see a reference to marshal in our stuff16:27
mwhahahaof course we're just using bundler so who knows what it's pulling down16:27
* mwhahaha would have to look16:27
*** esberglu has joined #openstack-infra16:27
clarkbI don't really understand what the purpose of those is? is it so you can install the deps of a thing without isntalling the thing itself?16:28
clarkbor to just get version and dependency info?16:28
*** lucasagomes is now known as lucas-afk16:28
*** rbuzatu has joined #openstack-infra16:29
mwhahahaoh it's for use in not duplicating your deps16:29
clarkbmwhahaha: right but normally it lives in the repo/gem itself iirc16:30
clarkbso wondering what the purpose of histing them under /quick/Marshal.x.y/ is16:30
mwhahahait can be a remote ref but i don't think we need it for our needs16:30
SamYaplewhat is the name of the job that happens in post that creates the tarballs in openstack-server-publish-jobs?16:31
mwhahahasince non of our Gemfiles have a gemspec reference (at least our puppet modules)16:31
clarkbSamYaple: thats a group of jobs that includes the tarball/sdist build docs builds iirc16:32
*** esberglu has quit IRC16:32
SamYapleor is it part of python-jobs16:32
clarkbSamYaple: if you look in python-jobs.yaml at the bottom are the job-groups. The tarball job in python-jobs is a job that updates"branch" tarballs on every merged commit16:34
SamYapleThis is the tarball publishing job ?16:34
clarkbSamYaple: so you get a master tarball and an ocata tarball and so on updated every merge16:34
SamYapleyea thats what im lloking for16:34
SamYaplecool thanks16:34
clarkbSamYaple: the non branch tarball is used for releases and is part of openstack-server-release-jobs16:34
clarkbif you want the per commit branch update then python jobs is what you want looks like16:35
fungiclient release jobs also does tag-based tarballs but also pushes to pypi i think16:35
*** nadya has quit IRC16:35
*** electrofelix has quit IRC16:35
*** apevec has quit IRC16:36
SamYapleusing loci-keystone for an example, it doesn't have python anything going on with it (its Dockerfiles at the moment) so publishing to pypi seems wrong16:36
*** ZZelle has joined #openstack-infra16:36
SamYaplei need to get openstack/loci into a tarball on a per-commit basis (for each branch). it seems weird to use python-jobs if there are no python bits16:37
SamYapleso its a bit of a wierd situation, no?16:37
pabelangerwell, I would hold off on publishing images to tarballs.o.o for now,16:37
SamYapleim not publishing images16:37
pabelangerdepending on how big they are, we are down to our last 50GB16:37
pabelangerand I know images take up more stack then python tarballs16:38
fungiSamYaple: you can just put the tarball job itself in there directly, but you likely don't want our normal tarbal jobs since they expect to be able to tun `tox -e venv python sdist` (and bdist_wheel)16:38
SamYaplepabelanger: the plan is to push 250MB (per branch, which only exists as one branch right now) in total. but that is just to prevent fetching from outside-infra for gate jobs16:39
fungiSamYaple: if you need to make tarballs that are not python sdist tarballs, there might be a job definition available already which does what you need but i don't know so you'll probably need to hunt (or write one)16:39
SamYaplefungi: so 'scp' publisher?16:39
fungiSamYaple: the current {name}-branch-tarball job-template is defined here
*** nadya has joined #openstack-infra16:41
*** marst has quit IRC16:41
fungiit invokes a builder here:
*** chem is now known as chem_gone16:42
*** Apoorva has joined #openstack-infra16:42
*** nadya has quit IRC16:42
fungiwhich runs these scripts:
SamYaplei see.16:43
*** ralonsoh has quit IRC16:44
*** marst has joined #openstack-infra16:44
*** tosky has quit IRC16:45
fungiSamYaple: for comparison, here's a definition for doing something similar with nodejs/npm tarballs:
fungiin the nodejs case, looks like it calls `npm pack` to create a tarball16:46
SamYaplefungi: yea thats what im writing up. using the 'scp' publisher16:46
fungii'm trying to think of a publishing ecosystem where we just call tar directly to create tarballs16:46
fungiwell, the tarball creation doesn't happen in the publisher16:46
SamYapleyea +2 to that16:46
SamYaplefungi: i know, but i can create said tarball16:47
*** kgiusti has quit IRC16:47
fungithe publisher is just what sticks the resulting file(s) somewhere public (logs.o.o, tarballs.o.o, et cetera)16:47
SamYapleyea i get that16:47
SamYapleim good with that16:47
SamYaplethis is mostly just about removing all external-to-infra deps16:48
pabelangeris there a screen on static.o.o for fsck?16:50
clarkbpabelanger: apparently there is, was just mentioned in incident16:50
*** yamamoto has joined #openstack-infra16:51
*** florianf has quit IRC16:51
fungiyeah, root screen session it's running under16:51
jeblairfungi, clarkb: i went to lists.o.o to start doing the prep work we said we would do before the maintenance, and i'm getting this:
fungipabelanger: clarkb: there are apparently two root screen sessions (or were when i first logged in), but the slightly newer of the two is the one i think you want16:52
clarkbjeblair: hrm, I'm guessing the -dev packages are relatively safe to just force downgrade bceause they are headers16:53
fungijeblair: that looks like an unattended upgrade went screwy on a libc update, and then the packages it needs to satisfy that dependency are no longer available. probably solvable with a forced downgrade or upgrade16:54
fungiand yeah, libc6-dev isn't going to be used for much. probably there because we install build-essential or something16:54
jeblairhrm.  do we think that's safe to do outside of the maint window?16:55
jeblairthis is needed in order to do the apt-get autoremove which we expect to save something like 1 hour of time during the maint.16:55
fungiupgrading libc6-dev looks like it should solve that, based on the errors16:55
*** ramishra has joined #openstack-infra16:56
fungii'm taking a look to see if i can figure out what's happened there16:56
openstackgerritSorin Sb├órnea proposed openstack-infra/shade master: Fixed stack_status.split() exception
*** yamamoto has quit IRC16:57
jeblairfungi: i just "dpkg -i libc6-dev..." and i think maybe it's happy now16:58
clarkbjeblair: I'd expect it to be safe (just considering the primary consumer of that is compilations and we won't be doing any of those)16:58
clarkbjeblair: and whatever linkage is already in place happens despite the headers and its working now16:58
fungiyeah, i think there must have been a failed upgrade a while back. apt-cache show libc6-dev says 2.15-0ubuntu10.16 was locally unpacked and marked for installation, but that the current version in the package index for the archive is 2.15-0ubuntu10.1816:58
fungiso `apt-get install libc6-dev` would have probably fixed it too16:59
*** gyee has joined #openstack-infra16:59
jeblairfungi: no, apt-get was throwing a fit16:59
jeblairall similar commands resulted in the pasted error16:59
fungiwhen explicitly telling it to install latest libc6-dev?16:59
fungithat's an unfortunate catch-2217:00
jeblair 1108  apt-get install libc6-dev17:00
jeblairyep, tried that ^17:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add test to validate multi _ heat stack_status
jeblairso i did:17:00
jeblair 1111  apt-get install -d libc6-dev17:00
jeblair 1120  dpkg -i libc6-dev_2.15-0ubuntu10.18_amd64.deb17:00
jeblair(with some cd and ls's inbetween :)17:01
fungioh well, it's showing 2.15-0ubuntu10.18 installed now, so good enough17:01
*** signed8b_ is now known as signed8bit_Zzz17:01
SpamapSif I wanted to show somebody how many jobs the bluebox node provider is running... is there a convenient place? maybe in grafana?17:01
mordredSpamapS: yah - grafana - lemme get you link17:01
jeblairfungi: shall i 'apt-get autoremove' now?17:01
SpamapS ?17:01
SpamapSI just typed 'graf' and that came up in my history17:02
SpamapSso clearly I've been there before :-P17:02
jeblairfungi: oh, hrm, autoremove doesn't want to do anything.17:02
fungiSpamapS: the test nodes graph17:02
SpamapS <-- this one?17:02
mordredSpamapS: yah. Test Nodes correspond to jobs essential17:02
SpamapSK. Just keeping the bean counters happy.17:02
jeblairi count 15 beans right now17:03
mordredbeans make good coffee17:03
jeblairfungi: so, erm, what should we do to get rid of all those kernels?17:03
fungijeblair: so looks like we have 43 kernel packages installed at the moment17:03
AJaegerspeaking about recheck and reverify - should we remove reverify finally? those that remember it, think it only goes through gate...17:04
fungijeblair: latest installed is 3.2.0-80 but uname -a says we're on 3.2.0-7717:04
pabelangerAJaeger: I think we should, maybe a ML post about the removal?17:04
pabelangernot sure how others feel17:04
fungijeblair: i'd probably just apt-get remove all the kernels except the one we're running17:04
*** flepied has quit IRC17:05
AJaegerpabelanger: I agree with the ML...17:05
clarkbfungi: jeblair apt-get autoremove is what I do locally17:05
clarkbbut maybe I trust apt too much?17:05
jeblairclarkb: 0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.17:06
fungiyeah, autoremove isn't seeing these as auto-installed and no longer depended on by any installed package17:07
fungiso i'd just sed the versions out into a package name list, filter out the version we're presently booting, and feed that to an apt-get purge17:08
fungimaybe also filter out the latest version if you want17:08
fungiso we end up with only the currently installed kernel and the latest kernel still present17:09
fungier, currently booted and latest i mean17:09
*** yonglihe has quit IRC17:09
*** armax has joined #openstack-infra17:10
*** eroux has quit IRC17:10
jeblairokay, that's running17:11
*** sambetts is now known as sambetts|afk17:12
fungiit'll still take a while since it'll likely try to regenerate initrds between each package removal, but at least we can do this without services being down17:12
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove 'reverify'
AJaegerMore than 30 months ago we aliased reverify and recheck - time to remove reverify IMHO ^17:14
jeblairfungi: yep.  it's chugging along.17:15
jeblairAJaeger: do people still use it?17:15
AJaegerjeblair: yeah, I sometimes see people using it.17:15
jeblairi agree 30 months is a sufficient deprecation period, but only if we've given notice that it's actually a deprecation period.17:15
*** signed8bit_Zzz is now known as signed8b_17:16
jeblairif not, maybe we want to send an email and wait another week or so.  :)17:16
*** armax has quit IRC17:16
AJaegerI can do that...17:16
*** nadya has joined #openstack-infra17:16
* AJaeger checks archives...17:16
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Use libyaml if possible
*** nadya has quit IRC17:17
*** shardy has quit IRC17:17
*** nadya has joined #openstack-infra17:18
*** rbuzatu_ has joined #openstack-infra17:18
AJaegermmh, seems we did not announce it - ok, will do now...17:18
*** yamahata has joined #openstack-infra17:19
*** ramishra has quit IRC17:20
*** jcoufal_ has joined #openstack-infra17:21
*** rbuzatu has quit IRC17:22
*** jcoufal has quit IRC17:24
_ari_Is there an issue with the Jenkins gerrit I got this for my review -
mnasertopic _ari_17:25
*** neiljerram has quit IRC17:26
*** smarcet has joined #openstack-infra17:28
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove 'reverify'
AJaegermail sent and referenced in my review ^17:30
AJaegerwow, already a +1000 by sdague ;)17:33
*** rcarrillocruz has quit IRC17:33
* JayF may have been putting 'recheck reverify' in everything he wanted to check because he forgot which one worked17:33
AJaegerJayF: 'recheck reverify' works since 'recheck X' works. But 'X reverify' or 'X recheck' does not work.17:35
AJaegerJayF: So, 'no recheck' will do the right thing ;)17:35
*** pvaneck has joined #openstack-infra17:35
*** isviridov_away has quit IRC17:36
*** vaidy has quit IRC17:36
JayFI always did them on separate lines, but I'll remember now17:36
_ari_mnaser: thanks17:36
AJaegerJayF: ah, with separate lines it does work17:36
_ari_mnaser: is there a way to retrigger my review after it is fixed?17:36
*** wolverineav has quit IRC17:36
mnaser_ari_ just type "recheck" in a review17:36
mnaseras a comment17:36
mnaseronce all of this clears out17:36
_ari_mnaser: thanks so much!17:36
mnasernp :>17:36
SamYaplei purpose we change it to 'revericheck'17:37
SamYapleso no more confusion17:37
AJaeger_ari_: item 417:37
*** wolverineav has joined #openstack-infra17:37
mnaser^ probably a much better reference to know working with the CI stuff17:38
*** mdrabe has quit IRC17:38
_ari_AJaeger: thanks I should have read lower down in the developers guide my bad17:38
_ari_I will keep that in mind in the future17:38
openstackgerritOpenStack Proposal Bot proposed openstack-infra/requestsexceptions master: Updated from global requirements
*** jerryz has joined #openstack-infra17:39
*** nadya has quit IRC17:39
*** smarcet has quit IRC17:39
fungiSamYaple: my vote is on recheckify17:39
clarkbjeblair: re lists.o.o it must be happy now?17:39
jeblairclarkb: yeah, it's still (slowly) removing kernels17:40
fungiclarkb: "happily" whittled its way down from 43 kernels to 34 so far anyway17:40
clarkband infra-root if waiting for fsck paint to dry reviewing would be helpful to keep pushing on gerrit upgrade. Don't need to approve it, I can do that (though likely not today as its lists upgrade and logs fsck day)17:40
*** wolverineav has quit IRC17:42
*** felipemonteiro has joined #openstack-infra17:42
clarkbAJaeger: andreykurilin I have +2'd the rally features.yaml change17:43
andreykurilinclarkb: thanks!17:43
*** smarcet has joined #openstack-infra17:45
*** tosky has joined #openstack-infra17:48
sdaguefungi: what about "big bucks, no whammies!"17:48
*** MasterOfBugs has joined #openstack-infra17:48
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Add install-guide-jobs for murano installation guide.
mnaseri was just curious at how far the fsck has gotten if that's okay?17:49
*** e0ne has joined #openstack-infra17:50
*** armax has joined #openstack-infra17:51
jeblairmnaser: 89.4%17:51
mnaserjeblair cool, thanks!17:51
fungisdague: i think we'd be pressing our luck with that one17:52
* fungi lands on a whammy with that pun17:53
SamYapleDoes this mean i have to watch vhs recordings to learn how to make the gate pass?17:54
sdagueSamYaple: yes17:54
sdagueor even better, betamax17:54
SamYaplethat _is_ even better17:54
openstackgerritDan Prince proposed openstack-infra/tripleo-ci master: Update ovb-nonha-oooq to use local registry mirror
fungiSamYaple: bonus if you can work out a way for us to re-theme our ci system based on "bowling for dollars" by next april 117:55
AJaegeralso, if somebody wants to review: cmurphy has a series of puppet changes where I'd like some dib/puppet experts to look at, see
SamYaplefungi: i cant do that, but i can probably make it autoplay Bowling For Soup17:56
AJaegerEmilienM: could you review as well, please? Removes  tripleo-test-cloud-rh217:56
smarcetfungi: afternoon, we will facing a trove access shortage on 13/14 April, due RS DFW dc maintenance, the only way to avoid that, is to move all production database to a new trove instance, we are working on that, but we would need to change some hieras variables for openstackid to point to the new host17:59
smarcetfungi: should i send those new hieras values by email once that we do the move ?17:59
*** baoli has quit IRC18:00
*** baoli has joined #openstack-infra18:01
fungismarcet: yes, you can send them to me and i can update them. or really any infra-root who's around at the time you're ready to switch databases18:01
smarcetfungi: roger that thx :)18:02
clarkbmaybe send them once the new db name is known?18:02
clarkbthen we can stage it18:02
fungialso doable, yes18:02
clarkbalso is rackspace replacing all the hardware?18:02
fungicertainly seems that way18:02
*** jcoufal has joined #openstack-infra18:02
*** Apoorva_ has joined #openstack-infra18:03
openstackgerritMerged openstack-infra/requestsexceptions master: Updated from global requirements
clarkbI'm gonna go grab late breakfast early lunch and be ready for lists things in a couple hours (will be back before that for prep work)18:03
*** jcoufal_ has quit IRC18:04
*** Apoorva has quit IRC18:05
openstackgerritMerged openstack-infra/project-config master: [rally] Add support for custom branching
openstackgerritazat proposed openstack-infra/jenkins-job-builder master: scm: bypass git.submodule.parentCredentials
fungihere's a fun one. did we merge any firewall rule updates for planet.o.o?18:10
fungithe site's down, and iptables -L doesn't show any http open18:10
*** baoli has quit IRC18:10
clarkbwe did change puppet to allow an 01 server18:11
*** baoli has joined #openstack-infra18:11
fungilooks like something has caused us to update the fw rules to start blocking public tcp services there18:11
kfox1111hows logs.oo going?18:11
fungikfox1111: fsck at 89.8% complete18:11
kfox1111cool. thanks.18:11
*** baoli has quit IRC18:12
*** baoli has joined #openstack-infra18:13
fungiclarkb: i see the issue. patch coming18:13
SamYaplefungi: we should just use btrfs. i heard it never has any problems18:13
*** rbuzatu has joined #openstack-infra18:14
EmilienMAJaeger: sure thing18:14
EmilienMbnemec: please review it too if you can
openstackgerritMerged openstack-infra/project-config master: Use job-template check-requirements in masakari projects
kfox1111SamYaple: ;)18:15
kfox1111SamYaple: my brother had a btrfs eat itself. though this was over a year ago. probably almost 2 by now...18:15
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Correct global site manifest regex for planet.o.o
fungiclarkb: mnaser: ianw: ^18:15
fungimnaser: thanks for reporting!18:15
mnasernp :>18:16
*** rbuzatu_ has quit IRC18:17
SamYaplekfox1111: it just happened to me last year. :( i stopped using it, though i wont spread FUD about it18:17
fungiunfortunately planet's probably been offline for some ~44 hours since shortly after the earlier regex merged18:17
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Add experimental murano functional test gate.
AJaegerThe docs team just got the second bug report about failing logs ;(18:18
fungikfox1111: SamYaple: i have a similar story, though the last time i tried using it there still wasn't a fsck for it18:18
kfox1111SamYaple: yeah. I really want btrfs to succedd. I know far to many people that cram zfs into linux. :/18:18
AJaegerfungi, do you want to send an email to openstack-dev or shall I? What's the estimate?18:18
sdaguefungi: once everything is said and done, it would be interesting to figure out why the log analyzer is so memory heavy18:18
kfox1111I'm using btrfs on some of my docker hosts though, with good results.18:18
sdaguebecause it was specifically written as a stream processor, so shouldn't be18:18
*** baoli has quit IRC18:19
fungisdague: agreed, though my guess is that's just the python runtime per request18:19
sdaguefungi: yeh, I wonder if that's one of those things were we could be better with a uwsgi backend or the like18:19
sdaguebut, that's a later conversation18:20
fungisdague: normally it's not an issue, but when the system is already under tons of memory pressure because fsck.ext4 wants nearly 100% of the allocated ram for this server, every little bit helps18:20
sdagueyep, no worries18:20
SamYaplefungi: its a CoW filesystem, there is no fsck!18:20
fungiAJaeger: if you want to send out a note, that's much apprediated!18:20
sdaguefungi: are swift logs no longer a thing?18:20
sdaguewe could also probably simplify the code if that's the case18:20
fungisdague: we were never able to finish working out how to reimplement a browsable file hierarchy on top of swift containers18:20
*** tesseract has quit IRC18:21
*** rbuzatu has quit IRC18:21
*** rbuzatu has joined #openstack-infra18:21
fungiit's not a use case they design for, and so we end up needing to basically create our own posix-ish file allocation table as a separate database18:21
sdagueyeh, that just gets ugly pretty quick18:21
AJaegerfungi, will do...18:22
fungithanks, AJaeger!18:22
notmynamefungi: IIRC it wasn't the browsable file hierarchy (that's possible). it's the metadata on mid-level elements of the generated hierarchy18:22
sdagueok, so I need to go to the shop to get a thing the car looked into, maybe I'll take it as a good time to strip a bunch of that out and propose it to just simplify the code18:22
sdaguewhich might make thinking about the memory thing easier18:22
funginotmyname: right, that was one of the bigger parts. things like updating the last modified time on a directory node when a file under it has been altered/created18:22
notmynamefungi: eg mx(last modified time) of any object with a given prefix18:22
funginotmyname: and being able to sort by that as well18:23
fungiwe were basically reimplementing apache mod_auto_index on top of swift18:24
notmynamefungi: I'd quibble about what was being reimplemented. ;-) we sorta have it. you were wanting FS semantics on object storage18:25
funginotmyname: right18:25
fungiwell, we were wanting a filesystem, but we tried to put it on top of swift because that was an elastic storage option18:25
mnaserdoesnt swift already give you index for pbulic containers18:26
*** jascott1_ has joined #openstack-infra18:26
notmynamemnaser: yeah, it can. not just all the generated metadata (like last mtime) on generated entities. nor does it allow arbitrary sorting18:26
fungimnaser: we were also somewhat hamstrung by the way rackspace cdn's their public swift access, and by wanting to do some postprocessing between the content and the client18:27
mnaserthe only 2 annoyances i can imagine is: writing something that loops and uploads files (a mass uploader rather than an rsync), and clean up18:27
notmynamefungi: might be more options now you've got infra cloud. or if you use another provider18:27
mnaseri know this is a lot of work but throwing ideas18:27
fungiso in the end, we were turning swift containers into a remote filesystem for a separate webserver, which really wasn't playing to its strong suits18:27
mnasernginx has a way to write simple modules in $language_i_cant_remember18:28
mnasernot that i know anything about it but its interesting to know, i know of some companies that use this at big scale -
fungiwell, at this point we have a different elastic storage solution which has a more traditional unix filesystem and are already using that for some other content, so have ideas for how to move our log data to that18:29
notmynamefungi: what are you using?18:29
mnaser(but then if you have something that's working without problems, no need to fix what isnt broken)18:29
notmynamegotta run (technically, I'm on vacation today). happy to pick this up again next week18:30
mordredwell - we have a non-zero amount of work in front of us no matter which thing we do18:30
funginotmyname: sure, any time! enjoy your vacation18:30
mordredand clearly the _current_ thing doesn't make anybody happy18:30
mnasermordred i think logs.o.o was ok chugging along if it wasn't for an issue that you dont control18:31
mnaserthe other options could fail just as much if another issue which you dont control happens too *shrug*18:31
mordredmnaser: yup. totally agree. I think moving forward it's about understanding what our failure mitigation options are18:31
fungimnaser: well, the current backend is also not scalable. the provider limits the size of the volumes we can create and their choice of hypervisor significantly limits the number of volumes we can attach to a single server18:31
fungiand we maxed those out long ago18:32
mordredunfortunately, we're maxed on cinder volumes on the server in rax - so although we got warned that the volume was going to go down, we don't have the headroom to create a new one and migrate the data :(18:32
mordredfungi: ++18:32
mnaseri guess from an operator perspective figuring out how to containing failure to not be as catastophic would be the best next step18:32
fungiwe're going to need to come up with a sharding model anyway, not only for scaling the quantity of data we store but also scaling our ability to age out old data in a timely manner18:33
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual master: Explain OpenStack CI Status
*** Sihan has joined #openstack-infra18:33
mnaseri havent had great experience but running glusterfs on top of a few instances could deliver keep much of the same semantics18:33
AJaegerfungi, what do you think of this ^18:33
SihanHi Does any know how to deploy the zuul status dashboard?18:33
AJaegeror is there a better place to explain that?18:33
Sihanif there is a doc, that would be helpful18:33
mordredmnaser: yah - we've talked some about both ceph and gluster. also about just sharding data across multiple volumes in AFS18:33
mordredeach of those options is just involved enough that full exploration will take a bit of time18:34
clarkbok I am mostly back ish18:34
mnasercephfs could be an interesting option too, afs i dont know much but i think the advantage is big operational experience so a lot of existingknowledge in the infra team18:34
fungiAJaeger: i like the prominent placement, either there or system-config docs would make sense18:34
mnaserAJaeger i think maybe mentioning that if you get POST_FAILURE it means there is likely an issue and to check those sources as well18:34
*** jcoufal_ has joined #openstack-infra18:34
mnasera lot of people dont seem to associate with POST_FAILURE => CI issue rather than "oops did my build break"18:34
mordredmnaser: yah - well, and decoupling filesystem presentation from backend storage18:34
pabelangerclarkb: how do you want to handle the PR on ?18:35
fungiSihan: normally zuul provides it directly (though i think it depends on some javascript bits, which are explained in the install docs)18:35
pabelangerI'm going to as in the ruby channel who is maintaining it18:35
mnasermordred: cant you just, like, put all that data on the cloud? isnt that how it works18:35
mordredmnaser: you have to wave your hands wildly when you say that18:36
fungiSihan: have you tried opening firewall access to port 80 for your zuul server and then just browsing to it via http?18:36
AJaegermnaser: I'll add that...18:36
AJaegerfungi: both?18:36
mnaserand maybe i'm pushing but im not sure how easy this is to do18:36
mnaserbut perhaps the message that gets posted by jenkins18:36
fungiAJaeger: maybe put it in one and refer to that section from the other?18:37
mnaserif it can say something about POST_FAILURE being a CI issue18:37
*** jcoufal has quit IRC18:37
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual master: Explain OpenStack CI Status
AJaegermnaser, fungi: updated ^18:38
*** vaidy has joined #openstack-infra18:38
fungijeblair: looks like the kernel cleanup on lists.o.o just completed?18:39
*** isviridov_away has joined #openstack-infra18:39
clarkbworth noting that sdague was the user we were considering when we switching things to swift and it didn't work :)18:39
clarkbso whatever we decide on hsi input would be valuable too18:39
*** baoli has joined #openstack-infra18:40
*** john-davidge has quit IRC18:40
*** baoli has quit IRC18:40
*** baoli has joined #openstack-infra18:40
*** e0ne has quit IRC18:40
jeblairfungi: yep, i'm removing old kernel header pkgs now for the heck of it18:40
*** e0ne has joined #openstack-infra18:41
*** e0ne has quit IRC18:43
openstackgerritEric Kao proposed openstack-infra/project-config master: add coverage-jobs to congress
jeblairclarkb, fungi: all done.  there are some packages that still want to be upgraded, but they include mailman, so that'll be in step 1 of the maint.18:44
clarkbjeblair: sounds good18:44
fungijeblair: sounds great. thanks!18:44
SihanI didn't see 80 is listened by any process18:44
clarkbfungi: I have approved planet fix18:45
*** e0ne has joined #openstack-infra18:45
*** rbuzatu_ has joined #openstack-infra18:46
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Link to Infra Manual for CI problems
openstackgerritAndreas Jaeger proposed openstack-infra/system-config master: Rework intro
*** rbuzatu has quit IRC18:50
AJaegerfungi, updated project-config, system-config and infra-manual18:50
fungithanks clarkb18:51
*** amoralej is now known as amoralej|off18:54
openstackgerritMerged openstack-infra/system-config master: Correct global site manifest regex for planet.o.o
*** baoli has quit IRC18:59
openstackgerritazat proposed openstack-infra/jenkins-job-builder master: scm: bypass git.submodule.parentCredentials
*** baoli has joined #openstack-infra19:00
*** baoli has quit IRC19:00
*** e0ne has quit IRC19:02
*** baoli has joined #openstack-infra19:05
*** csomerville has quit IRC19:05
*** jascott1_ has quit IRC19:06
AJaegerfungi, are my three changes in case you want to put them on your review queue.19:06
*** chem_gone has quit IRC19:06
* AJaeger considered updating but couldn't come up with something great, so left it out.19:06
*** marst has quit IRC19:07
*** chem_gone has joined #openstack-infra19:07
*** wolverineav has joined #openstack-infra19:09
*** mat128 has quit IRC19:11
*** mat128 has joined #openstack-infra19:12
*** gcb has quit IRC19:12
*** EricGonc_ has quit IRC19:13
*** gcb has joined #openstack-infra19:13
*** mat128 has quit IRC19:16
*** nadya has joined #openstack-infra19:19
openstackgerritMiguel Lavalle proposed openstack-infra/project-config master: Add job to Neutron DVR check queue panel
openstackgerritK Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks
*** nadya has quit IRC19:20
*** jascott1_ has joined #openstack-infra19:22
*** sdague_ has joined #openstack-infra19:23
*** Goneri has quit IRC19:23
*** baoli has quit IRC19:26
*** baoli has joined #openstack-infra19:26
*** baoli has quit IRC19:31
*** wolverineav has quit IRC19:33
openstackgerritSam Yaple proposed openstack-infra/project-config master: Publish LOCI tarball to tarballs.o.o
SamYaplefungi: this is what i came up with ^19:35
*** tonytan_brb has quit IRC19:35
SamYapleshort of creating a stub python project just to publish tarballs, i dont know what to do19:35
SamYaplewhat else rather19:36
*** john-davidge has joined #openstack-infra19:40
pabelangerSamYaple: fungi: left comment19:41
SamYaplepabelanger: oh my mistake. man i need to figure out this reading comprehenssion thing19:42
SamYaplei thought fungi was saying i couldnt do that19:42
pabelangeryou should be fine, but checking backscroll19:42
fungii'm lacking context. still working on logs.o.o situation19:44
fungiand getting ready to be handy during the lists.o.o maintenance19:44
SamYaplefungi: ignore. its fine. thanks for the help19:45
SamYaplepabelanger: yea i just read through the backscroll, i think my mistake was reading 'jobs' as 'publishers' and getting confused19:45
*** john-davidge has quit IRC19:46
pabelangerSamYaple: everything should be fine, just switch out your dir to store the tarball in, and use tarball publisher19:47
pabelangerthat should work19:47
* clarkb tanks up on caffeine before lists.o.o work19:47
SamYaplepabelanger: cool. running checks locally right now then will push new patchset. thanks for all the help19:48
*** sc68cal has left #openstack-infra19:49
*** sdague_ has quit IRC19:51
fungiclarkb: jeblair: i'm here to help with lists.o.o tasks as well if needed, but focusing on logs volume situation as first priority so you can pay closer attention to lists.o.o work19:51
openstackgerritSam Yaple proposed openstack-infra/project-config master: Publish LOCI tarball to tarballs.o.o
jeblairfungi, clarkb: either of you have that etherpad handy?19:54
fungiclarkb: mnaser: ianw: just to close the loop on this one, i've confirmed that seems to be fine now19:54
fungijeblair: was just pulling it up19:54
mnaserfungi cool!19:54
jeblairfungi: thx19:54
jeblairstatus notice will be offline from 20:00 to 23:00 UTC for planned upgrade maintenance19:55
jeblairfungi, clarkb: ^ ?19:55
jeblairi was going to include a link to the ml post... but.... :)19:56
fungijeblair: lgtm19:56
fungioh, i can get you one19:56
jeblairya :)19:56
* fungi just got the joke19:56
fungii bet there's a third-part ml mirror we could link19:56
*** sdague_ has joined #openstack-infra19:56
jeblair#status notice will be offline from 20:00 to 23:00 UTC for planned upgrade maintenance19:56
openstackstatusjeblair: sending notice19:56
funginot finding a good one though, so probably fine19:58
jeblairfungi, clarkb: i'll add lists.o.o to emergency19:58
clarkbjeblair: ok19:58
-openstackstatus- NOTICE: will be offline from 20:00 to 23:00 UTC for planned upgrade maintenance19:58
fungiright on19:58
jeblairdone, and ansible is not running19:58
JayF#success My inbox gets three hours of reprieve thanks to infra! Woohoo maintenance!19:59
fungieven better19:59
fungiJayF: the down-side is that all those messages will just slam your inbox at once when we finish19:59
fungidrink while you can19:59
JayFfungi: shh, you're interrupting the quiet ;)19:59
openstackstatusjeblair: finished sending notice20:00
jeblairi have screen running as root on lists20:00
openstackstatusJayF: Added success to Success page20:00
jeblairi will stop services now20:01
jeblairapt-get upgrade now20:02
* clarkb hops onto the screen20:03
clarkbI see langs getting selected20:03
clarkblooks like mailmain didn't completely stop?20:04
jeblairi apparently stopped it with qfiles present and the package upgrade doesn't like that20:05
jeblairso i'm letting the queue flush (to exim) and then will redo20:05
fungioh, that's just spiffy20:05
clarkbdid you restart exim for that?20:05
clarkbor is that just a file delivery?20:05
* mordred is here, in case he can be helpful20:05
jeblairclarkb: i don't think we need exim running for that20:06
jeblairseems happier this time20:06
clarkbya looks it20:07
jeblairThe following packages have been kept back: landscape-common20:07
jeblairdo we care?20:07
mordredwe should likely delete that from our servers at some point20:07
clarkbthats ubuntus server managment tool iirc20:07
jeblairrunning do-release-upgrade20:07
clarkbso ya don't care20:07
*** rhallisey has quit IRC20:09
fungiright, landscape seems useless for us20:10
*** Qiming has quit IRC20:12
jeblairwe're at the "lots of stuff scrolling by" portion of the upgrade20:14
clarkbcan confirm20:15
* fungi casts "wall of upgrade"20:16
*** Qiming has joined #openstack-infra20:16
SamYapleTIL about do-release-upgrade20:17
SamYapleive always just updated my sources.list...20:17
fungiSamYaple: in debianland, you read the release notes, manye perform a few manual steps outlined there which can't be easily accounted for by package maintscripts, and then update your sources.list and do an upgrade (possibly followed by a reboot) and then a dist-upgrade (probably followed by a reboot)20:19
clarkbjeblair: wants to know what to do with sysctl.conf /me cnosults etherpad20:19
clarkbn for keep current20:19
jeblairclarkb: ya, i decided they were all "keep current version" during the trial run20:19
fungiSamYaple: in ubuntuland they try to write a script that does the stuff you might want from the release notes. makes me a little uneasy, but what are you gonna do (oh, i know, run debian!)20:19
SamYaplefungi: yea i always skipped the release notes too. then i just ended up rekicking20:19
openstackgerritSean Dague proposed openstack-infra/os-loganalyze master: Remove swift support
*** kgiusti has joined #openstack-infra20:20
clarkbjeblair: I see that now thanks20:20
jeblairclarkb: i'm inclined to just follow the script (probably not much has changed in a week), but if you want we can diff.20:20
clarkbjeblair: considering its all keep old I think thats safe20:20
clarkbif we were keeping new it might be worth checking nothing changed20:20
*** ihrachys has quit IRC20:20
clarkbfungi: I solved that by running a rolling distro on my desktop20:21
fungiyeah, not applying any config updates should be safe. we can go back and look at the .dpkg-new versions later if we want20:21
fungiclarkb: me too, (debian/unstable) ;)20:21
*** gouthamr has quit IRC20:22
*** ihrachys has joined #openstack-infra20:22
jeblairaccording to the etherpad, that should be the last question20:25
jeblairwe got the missing site list message last time.  i thought maybe it was due to the hostname mismatch, so i don't know why we got it this time.  hopefully a red herring, but we should be careful (of course) before restarting services.20:27
jeblairclarkb: i guess we should remove the startup links since we have a couple of reboots ahead of us20:29
clarkbjeblair: ok20:29
*** arxcruz|pto has quit IRC20:29
*** EricGonczer_ has joined #openstack-infra20:29
clarkbmaybe do ls for sanity?20:30
clarkbof /etc/rc*20:30
jeblairclarkb: reboot now?20:31
clarkbthats what the script says and the init stuff appears removed20:31
clarkbcan't think of anyting else we should do first20:31
jeblairlooks like there's a workable grub/kernel config20:32
jeblairso i'll reboot now20:32
clarkbya and with the new kernel too20:32
jeblairit's up20:33
jeblairscreen started20:34
clarkbI am attached20:34
jeblairLinux 3.13.0-115-generic #162-Ubuntu SMP Fri Mar 24 16:03:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux20:34
jeblairlooks good20:34
clarkbdo we still need to upgrade again for new kernel?20:34
clarkbseems like thats it20:34
*** ihrachys has quit IRC20:35
jeblairclarkb: right, i'm expecting not.  i'll just run the commands to make me feel better though.20:35
fungialways makes me feel better20:35
*** arxcruz|pto has joined #openstack-infra20:35
jeblairshould we ext4 now?20:35
clarkbI think so20:35
fungimaking excellent time20:36
*** dprince has quit IRC20:36
*** marst has joined #openstack-infra20:36
*** sdague has quit IRC20:36
jeblairoh, i seem to have omitted the remount steps20:37
jeblairi think that's what we did (under the root section)20:37
fungiahh, yep20:37
fungilooks like the pad just goes from stopping daemons to fsck20:38
jeblairi think it's only missing that one command20:38
clarkbthats the doc I remember at least20:38
jeblairok, / is ro20:38
jeblairso much fsck today20:39
clarkbmust be friday20:39
*** sdague_ has quit IRC20:41
*** Apoorva_ has quit IRC20:43
*** Apoorva has joined #openstack-infra20:44
clarkbjeblair: the -v isn't being as verbose as I expected20:44
*** eharney has quit IRC20:45
*** mat128 has joined #openstack-infra20:45
jeblairstrace says that reads and lseeks are happening.20:46
*** gouthamr has joined #openstack-infra20:46
*** sshnaidm|off has joined #openstack-infra20:46
*** trown is now known as trown|outtypewww20:47
*** csomerville has joined #openstack-infra20:48
*** bnemec is now known as beekneemech20:49
*** asselin has joined #openstack-infra20:49
*** rcarrillocruz has joined #openstack-infra20:49
jeblairwe have exceeded the amount of time this took on the test host20:50
fungii wonder if the snapshot booted onto a faster host20:50
jeblairmaybe.  i'm not going to start planning a freakout until it's 2x.20:50
jeblairthere wo go.20:51
jeblairi'll run tune2fs now20:51
jeblairand fsck.ext4 now20:52
jeblairscaling this by the same amount, maybe expect it to finish at 21:20 utc.20:53
*** smarcet has quit IRC20:54
mordredso much fsck20:54
openstackgerritBen Nemec proposed openstack-infra/project-config master: Add coverage check jobs to tripleo python projects
openstackgerritBen Nemec proposed openstack-infra/project-config master: Add py35 job to os-net-config
dmsimardfsck this20:59
dmsimardfsck that20:59
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config master: Rework golang-dsvm job
clarkbone fsck two fsck red fsck blue fsck21:00
dimslol clarkb21:00
*** Qiming has quit IRC21:01
jeblairclarkb: your kids are going to be *weird*.21:01
*** Qiming has joined #openstack-infra21:02
clarkbI know, I am alraedy scheming all the ways I can make them weird. We had rabbit for dinner last easter (should make that a thing). Also I think their first computer will likely be raspi like thing with serial console21:02
fungithe clarklings will keep portland weirdER21:03
*** rbuzatu_ has quit IRC21:03
*** rbuzatu has joined #openstack-infra21:04
openstackgerritMerged openstack-infra/shade master: Fixed stack_status.split() exception
clarkbI just wish it was easier to introduce them to lots of islandy things like breadfruit and sea cucumber and so on21:05
clarkbbecause right now they eat anything I put in front of them21:05
clarkbexcept for spinich21:05
jeblairclarkb: ++ on the raspi.  i dunno if you'se seen/read eben upton's talks on that, but i agree with him that "boot to programming" is key to learning about computers.
clarkbI hadn't but ya I want to force them to learn how things work21:06
clarkband not just jump into facebook21:07
*** asselin_ has joined #openstack-infra21:07
*** kaisers has quit IRC21:07
*** asselin has quit IRC21:07
*** ijw has quit IRC21:10
dimsclarkb : ++++++21:10
dimsclarkb : my younger one takes everything apart :) step 121:11
fungii attribute my and my younger brother's predilection in part to dad thinking it was a waste of money to buy computer games when there was plenty of game source code in the backs of hobby computing magazines (all you had to do was type them in... and usually debug them to find your typos)21:12
*** dimtruck is now known as zz_dimtruck21:13
dimstotally worked :)21:13
jeblairfungi: and a win-win for your dad.21:13
fungii quickly learned to prefer the sources with line-by-line and section-by-section checksums21:14
clarkbI just remember that before I could drive the car I had to know basic things like changing oil, spark plugs, battery, brake line bleeding, air filters, tires, etc. Of course now cars are so complicated I don't even bother with any of them and have someone else do it all :/21:14
*** pramodrj07 has joined #openstack-infra21:15
clarkbalso fixing the bad cpu fan by leaving case open and pointing box fan at it was a good time21:15
*** ldnunes has quit IRC21:15
clarkbfsck is done21:15
fungimine isn't, unfortunately, and won't be for a while21:16
dmsimardclarkb: I totally agree with that statement. Cars are more complicated than OpenStack :)21:16
*** mat128 has quit IRC21:16
jeblairi reckon we reboot again now?21:16
*** sdague has joined #openstack-infra21:16
clarkbjeblair: ya I think thats a good idea21:16
clarkb(and what we ddi last time iirc)21:16
fungimake sure everything starts up correctly21:17
fungino surprises later that way21:17
fungiwell, _fewer_ surprises anyway21:17
jeblairwe do still have services disabled21:17
clarkbdmsimard: it made me sad, but I basically had to trade safety over simplicity :/21:17
clarkbjeblair: ya after reboot next step is turning it all back on again right?21:17
jeblairup and screen21:17
clarkbI'm in21:18
dmsimardclarkb: I think it boils down to interest as well. I could probably learn how to change my breaks and do it myself but I could care less.21:18
fungijeblair: right, i meant the usual system boot proceeding normally (rather than remounting rw and trying to start stuff back up)21:18
jeblairclarkb: i want to start by bringing up apache and seeing if mailman webui is happy, or if there's something funky with the site list21:18
clarkbjeblair: sounds good21:18
jeblairso let's start by changing the default language21:19
jeblairugh, i missed it.  i wonder if de was recently adedd21:20
*** askb has joined #openstack-infra21:21
*** aysyd has quit IRC21:22
mordredit seems like a weird language to have been recently added21:22
clarkbthese steps are not on the etherpad?21:23
fungithis didn't happen in the snapshot upgrade (afaik)21:23
jeblairclarkb: does look chinese to you?21:24
jeblairi was hoping that reconfiguring and re-setting the default language would correct that, but doesn't seem to have.21:24
clarkbbut isn't21:24
jeblairokay, i edited to change DEFAULT_SERVER_LANGUAGE to en21:25
jeblairi'm not sure why dpkg was okay with breaking that earlier but not fixing it now.21:25
jeblairclarkb: yeah, individual lists have their own lang preferences21:25
fungiwas that not previously set?21:25
openstackgerritazat proposed openstack-infra/jenkins-job-builder master: builders: do not pass empty "preload-script"
jeblairfungi: before 2 hours ago?  i don't know.  before i edited it, it was set to zh_TW.21:26
*** sekelso has quit IRC21:27
jeblairat any rate, the mm web ui seems happy now.21:27
*** jmorgan1 has quit IRC21:28
clarkbyes its english now21:28
jeblairanyone want to poke around some more, or should we start up the mm queue runners?21:28
clarkbI apparently don't have all the right font characters for vietnamese :(21:28
*** xarses has quit IRC21:28
*** smarcet has joined #openstack-infra21:28
fungithat would have been a pretty funky default21:28
jeblairclarkb: yeah, i also have a lot of <?>21:28
clarkbjeblair: pretty sure thats a font issue21:28
clarkblgtm otherwise21:29
fungiadmin/mod interfaces seem fine21:29
jeblairi'll start mailman now then21:29
fungimessage archives lgtm21:29
fungiyeah, looks great21:29
jeblairit's quiet, as expected since it has empty queues21:30
jeblairstart exim now?21:30
clarkbthats what will get it to actually do things right?21:31
clarkb(need new mail in for new mail out)21:31
clarkbso ya I think thats next21:31
jeblairclarkb: yep.21:31
*** jcoufal has joined #openstack-infra21:32
jeblairwe should make a real test list on the production server.  :)21:33
*** sdague has quit IRC21:33
*** smarcet has quit IRC21:34
clarkbwe can probably use ifnra list for that this time21:34
*** jcoufal_ has quit IRC21:34
clarkbI just got it21:34
*** Goneri has joined #openstack-infra21:35
funginow mine arrived21:35
fungiworking great!21:36
*** tonytan4ever has joined #openstack-infra21:36
mtreinishjeblair: I got your test message21:37
jeblairmtreinish: it was really just for you21:37
mtreinishheh, that's what I figured :)21:37
*** mat128 has joined #openstack-infra21:38
jeblairfungi, clarkb: the log entries all look good to me21:38
*** tonytan_brb has joined #openstack-infra21:38
*** tonytan4ever has quit IRC21:38
clarkbcool, I wouldn't really know what to look for in them21:39
*** mat128 has quit IRC21:39
fungimaintenance concluded in estimated time (divided by scotty multiplier)21:39
clarkbI guess make sure there aren't any domains that we can't deliver to suddenly (though preserving IP addrs should provent that)21:39
jeblairfungi: that is, of course, exactly how i calculated it.  :)21:39
jeblairi've re-enabled startup scripts21:40
clarkbdo we want to do one final reboot to make sure everything comes back cleanly?21:40
jeblairclarkb: not a bad idea21:40
jeblairrebooting now21:41
jeblairup and screened21:42
clarkbjeblair: host lookups that don't complete == domain is not in dns?21:43
andreykurilinclarkb AJaeger: everything works as expected! thanks21:43
clarkbor perhaps no MX record for domain?21:43
jeblairclarkb: usually yeah21:43
fungiyeah, usual process is look for mx, no mx then look for a/aaaa21:43
clarkbjeblair: is that something we should be cleaning up after, if you don't have working email for domain then we unsub after a month?21:44
clarkbor is cost low enough that we don't care?21:44
jeblairclarkb: should be automatic21:44
clarkbah ok21:44
jeblairclarkb: that should eventually bounce and mailman will drop after 5 bounces by default i think.21:44
*** zz_dimtruck is now known as dimtruck21:44
clarkboh right I remember getting unsubscribed once hence the change to fastmail21:44
jeblairclarkb: for a lot of hosts though, they intermittently have dns resolution problems, so you'll get a bounce, then a message gets through, then another bounce, so they stay on lists.21:45
jeblairclarkb: so there's a sort of fixed overhead that scales with subscribers.  but the bounce processing keeps it from growing without bound.21:46
jeblairi've removed lists.o.o from puppet emergency21:46
*** Sihan has quit IRC21:46
jeblairor ansible emergency21:47
jeblairor whatever21:47
*** xarses has joined #openstack-infra21:47
jeblairstatus notice The upgrade maintenance for has been completed and it is back online.21:48
jeblairclarkb, fungi: ^?21:48
*** xarses_ has joined #openstack-infra21:48
jeblair#status notice The upgrade maintenance for has been completed and it is back online.21:48
openstackstatusjeblair: sending notice21:48
*** xarses_ has quit IRC21:48
clarkbyou can even send a response to the email to prove to everyone it works :)21:49
clarkbthe email thread about scheduling the downtime21:49
*** xarses_ has joined #openstack-infra21:49
jeblairclarkb: i'll leave that to you (or fungi) :)21:50
-openstackstatus- NOTICE: The upgrade maintenance for has been completed and it is back online.21:50
clarkbI can do it21:50
fungiand thanks jeblair for driving a quick and uneventful upgrade!21:51
openstackstatusjeblair: finished sending notice21:52
*** xarses has quit IRC21:52
sbezverkwhat about zuul, is it still undergoing medication process, or it is done too?21:52
*** kzaitsev_mb has joined #openstack-infra21:52
fungisbezverk: still in surgery. there were complications21:52
sbezverkfungi: Oh man, I hope it is not lethal ;)21:53
fungii don't think so, just very lengthy21:53
*** EricGonczer_ has quit IRC21:53
*** ijw has joined #openstack-infra21:53
fungisbezverk: at this point it shouldn't be impacting jobs hopefully, you just can't get to the logs to look at them yet21:53
jeblairsmtp to openstack for 3764 recips, completed in 34.706 seconds21:54
jeblairin case anyone was curious21:54
*** EricGonczer_ has joined #openstack-infra21:54
*** EricGonczer_ has quit IRC21:54
clarkbmail sent21:54
mordredfungi: aren't you glad that long surgery for us does not ACTUALLY involve any surgery?21:56
fungiso glad21:56
clarkbyes I do not like operating rooms21:56
*** xyang1 has quit IRC21:56
jeblairwhen we have a really bad day, people have to run tests on their own computers instead of other computers that run them for free.21:57
pabelangerwow, just got CC'd on PR from salt, back from 201321:57
jeblairthis is different than an actual surgeon's very bad day in many important respects.21:57
mnaser"guess i'll have to do this surgery in my bedroom instead of a hospital facility"21:57
*** ijw has quit IRC21:58
jeblair(also, worth noting that most jobs should actually be running okay since we're dumping logs to a temporary location)21:58
openstackgerritazat proposed openstack-infra/jenkins-job-builder master: builders: do not pass empty "preload-script"
SamYaplepfff like anyone even looks at the logs anyway21:58
SamYaplegreen == LGTM == merge21:58
jeblairSamYaple: indeed, we should be looking at write-only storage21:59
*** Swami has quit IRC21:59
SamYapleyou already have it!21:59
SamYaple| /dev/null21:59
mnaseri think they have read only storage right now.. haa.....21:59
*** leakypipes has quit IRC21:59
jeblairmnaser: we have fsck-only storage i think21:59
clarkbI know you say that in jest but I have read so many logs in the last couple weeks (and on the bright side openstack seems really reliable right now in testing)21:59
*** flepied has joined #openstack-infra22:00
openstackgerritazat proposed openstack-infra/jenkins-job-builder master: scm: bypass git.submodule.parentCredentials
*** rlandy has quit IRC22:00
mnaseri thought logs.o.o was mostly for benchmarking browsers in loading 4mb pages22:00
clarkbmnaser: vim so much better22:00
fungiitym 400mb pages22:00
clarkbI actually need a thing that execs vim to open those files22:00
SamYaplemnaser: chrome wont buffer pages ending with .html, but it will buffer pages ending in .gz22:01
SamYapleif only we pushed console.html.gz initially22:01
SamYapleso much easier to read22:01
clarkbSamYaple: thats partly why we try to keep jobs in smaller console logs22:01
clarkbdevstack for example logs to its own file and os on22:02
mnaserclarkb: we can setup z/os with time sharing option and vim on it and give accounts to all foundation members22:02
clarkbmnaser: well I can just vim
SamYapleclarkb: yea thats the way to do it22:02
clarkbbut need my webbrowser to do that for me when I click on gerrit link rather than copy paste22:02
mnaseri think this is one of the things that kolla is bad at, i improved it by adding per-file-logging for builds but deploys still throw everything into console22:02
SamYapleclarkb: do you think we can modify the console-log publisher to publish both the compressed and uncompressed? and then just come through and clean up the uncompressed in the future?22:03
jeblairclarkb: you could make gertty do that fairly easily i think.22:03
*** jcoufal_ has joined #openstack-infra22:03
clarkbSamYaple: I think we can likely just send the compressed version then our webserver should do the right thing22:03
SamYaplemnaser: we had per image building in kolla initially. people wanted all output to stdout. :(22:03
fungiclarkb: my gertty does that for me via22:03
mnaserSamYaple its not even readable at all with multiple threads22:03
clarkbfungi: nice!22:03
SamYaplemnaser: I KNOW. thats what i said. but i was overruled22:03
mnasernow we do $NCPU build threads + logs per image built... so i guess you were right :p22:04
* SamYaple is taking notes22:04
SamYaplefungi: im going to do that22:04
SamYaplethat sounds fantastiy22:04
fungiclarkb: though BROWSER=lynx works a bit better since you can follow subsequent urls22:04
clarkbfungi: does lynx handle the large files well?22:05
fungiit has for me so far22:05
*** jcoufal has quit IRC22:05
fungidoesn't grind to a halt like ff22:05
jeblairclarkb, fungi: i'm going to terminate screen on lists now.  i don't think watching the matrix is going to tell us anything else.22:05
SamYaplefungi: pretty sure there is an extention to vim to open up vim again with the new webpage if you are running in tmux22:05
clarkbjeblair: ya I detached already22:05
fungiSamYaple: probably22:05
pabelangeremail worked!22:06
SamYaplemnaser: some of the old school kolla folks also raised the complaint (i think jpeeler wasn't a fan of it)22:06
*** jcoufal_ has quit IRC22:06
openstackgerritMerged openstack-infra/shade master: Add test to validate multi _ heat stack_status
clarkbdoes that mean its time to go do yardwork while the sun is up?22:07
fungiit's wet here today, so saving yardwork for sunday when things will have dried out some22:08
pabelangerI get to battle the snow tonight22:08
pabelangerbeen snowing all day22:08
clarkbya today is our nice day then back to rain22:08
bkeroYour place doesn't seem to have gotten the memo about spring.22:08
bkeroclarkb: It's back to nice on Sunday again.22:08
mnaserits snowing here until saturday :(22:09
* mnaser made the mistake of taking winter tires off22:09
clarkbbkero: oh thats new22:10
* bkero has to decide between homebrew on the sunny porch or 45 minute bikeride and tasty Portland beer from a nice neighborhood pub.22:10
pabelangermnaser: yup, same in Ottawa22:10
fungipabelanger: qapla', today is a good day to shovel22:10
*** thorst has quit IRC22:10
clarkbbkero: there is much tasty beer nearer to you if you want to have shorter bikeride ;)22:12
clarkbbkero: I'm sure I can rattle off a giant list22:13
bkeroclarkb: There is. I have one of the best taprooms in Portland 2 blocks from me.22:13
bkeroBut I just gave my road bike a spring refresher. Brakes, gears, chain, tires, etc. I need to go get it out.22:13
*** gordc has quit IRC22:13
bkeroPlus I haven't met my goal of 45 minutes of recorded cardio this week. :)22:13
clarkbeven widmer is better than what you'd get just about anywhere else22:13
clarkband they have shnitzel22:13
clarkb(anywhere else being not portland)22:13
bkerobut picnic tables and beer on a sunny day22:15
clarkbwidmer has outdoor22:15
clarkbI think :P but ya don't go there so much better options22:15
bkeroDo they have beer other than Widmer?22:15
clarkbI think they do have guest taps22:15
clarkbmaybe I'm wrong22:16
*** ijw has joined #openstack-infra22:17
bkeroclarkb: I'm pretty happy with this:
bkeroThe cylinders behind each tap are CO2 chambers, so that when they fill the growlers they keep carbonation and aren't exposed to O2.22:18
bkeroclarkb: I could also bike to Breakside :)22:18
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Use libyaml if possible
*** signed8b_ is now known as signed8bit_Zzz22:21
*** rbrndt has quit IRC22:24
*** rbrndt has joined #openstack-infra22:24
*** rbrndt has quit IRC22:24
*** signed8bit_Zzz is now known as signed8b_22:26
*** jamesden_ has quit IRC22:27
ianwfungi: thakns for fixing planet, need to finish up next week22:27
*** Sihan has joined #openstack-infra22:28
SihanHi, One more question for zuul, do we have any statistics tool for zuul? for example how long does a job wait in the queue or how many patches triggered per day etc.22:29
fungiSihan: zuul has statsd reporting built in, just set up a statsd server and configure to send events there, then you can use something like graphite/grafana to chart whatever you want22:30
fungiwhich is what we do22:30
Sihanthx :)22:31
*** dimtruck is now known as zz_dimtruck22:32
fungiSihan: the graphs (sparklines too) you see at are being generated with calls to but we also have some dashboards on for zuul and (mostly) nodepool which might serve as good examples22:33
*** gouthamr has quit IRC22:34
*** tonytan4ever has joined #openstack-infra22:36
*** ihrachys has joined #openstack-infra22:37
*** tonytan_brb has quit IRC22:38
*** funzo has quit IRC22:41
*** signed8b_ is now known as signed8bit_Zzz22:42
*** adrian_otto has quit IRC22:43
*** Goneri has quit IRC22:44
*** ihrachys has quit IRC22:46
*** zz_dimtruck is now known as dimtruck22:49
*** kgiusti has quit IRC22:49
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_timer_sshkey
openstackgerritMerged openstack-infra/tripleo-ci master: Add gate-tripleo-ci-centos-7-ovb-containers-oooq-nv to cistatus
*** kzaitsev_mb has quit IRC22:54
*** adrian_otto has joined #openstack-infra22:54
*** xarses_ has quit IRC23:03
*** ijw has quit IRC23:04
*** john-davidge has joined #openstack-infra23:05
*** Sukhdev has joined #openstack-infra23:07
*** dimtruck is now known as zz_dimtruck23:08
SukhdevHey Infra experts - I need to kick off a new networking-xxx repo for a new networking driver. Can somebody point me to the process/procedure, please - Last one I did was many years ago - wonder if the same process or different now.23:09
openstackgerritK Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks
*** john-davidge has quit IRC23:10
*** thorst has joined #openstack-infra23:11
*** adrian_otto has quit IRC23:15
*** thorst has quit IRC23:15
*** tonytan4ever has quit IRC23:15
Sukhdevfungi : Thanks a bunch23:15
*** adrian_otto has joined #openstack-infra23:18
dimsmordred : clarkb : one of you still around? easy one - job was just added this morning and won't affect anyone else. please take a look if you get a chance. i know it's too late :)23:19
*** zz_dimtruck is now known as dimtruck23:20
mordreddims: done23:24
*** adrian_otto has quit IRC23:25
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fully qualify project configuration names
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add hostname to TriggerEvent
kfox1111hows logs.o.o looking?23:26
*** dimtruck is now known as zz_dimtruck23:38
*** thorst has joined #openstack-infra23:42
*** ZZelle has quit IRC23:42
*** Julien-zte has quit IRC23:42
*** tosky has quit IRC23:47
fungikfox1111: the fsck originally got started with -p, so aborted around 90% and had to be restarted. we took the opportunity to temporarily redirect log uploads somewhere else with plenty of extra space so we can merge them back in once the main volume is back in service23:47
*** gouthamr has joined #openstack-infra23:50
kfox1111ah. ok. thanks for the info.23:56

Generated by 2.14.0 by Marius Gedminas - find it at!