| *** sekelso has joined #openstack-infra | 00:00 | |
| mordred | mnaser: yes, you should | 00:03 |
|---|---|---|
| mnaser | there's the one where it fails to find a proper endpoint | 00:03 |
| *** skelso has quit IRC | 00:03 | |
| mordred | I do not like that one | 00:04 |
| mordred | or where your auth info is wrong and it tells you something about not being able to load a plugin | 00:04 |
| mordred | but it also might be that there is no appropriate endpoint, iirc | 00:04 |
| * mordred shoudl really file a bug about that next time he notices it | 00:05 | |
| *** neiljerram has quit IRC | 00:05 | |
| pabelanger | mordred: do you mind a +3 https://review.openstack.org/#/c/451863/ | 00:06 |
| pabelanger | adds infracloud mirrors to cacti | 00:06 |
| *** funzo has joined #openstack-infra | 00:06 | |
| SamYaple | mordred: uhh you are noticing now. now is the time maaaan! | 00:09 |
| *** rfolco has joined #openstack-infra | 00:11 | |
| lifeless | fungi: well, we didn't punt, we took existing specification :) | 00:12 |
| *** funzo has quit IRC | 00:12 | |
| lifeless | fungi: what I'm seeing is an invalid version being returned | 00:13 |
| lifeless | by dpkg | 00:13 |
| jhesketh | fungi: by the way, if you're still looking for people to help with the on-boarding at the forum/summit, I'd be happy to | 00:15 |
| *** baoli has quit IRC | 00:16 | |
| pabelanger | jhesketh: sign me up too! | 00:16 |
| *** ramishra has quit IRC | 00:17 | |
| *** baoli has joined #openstack-infra | 00:17 | |
| openstackgerrit | Merged openstack-infra/bindep master: Bump lower cap of PBR to 1.3 https://review.openstack.org/448325 | 00:17 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove support for nodepool_id https://review.openstack.org/451889 | 00:17 |
| *** tphummel has joined #openstack-infra | 00:18 | |
| *** baoli has quit IRC | 00:21 | |
| *** jamielennox is now known as jamielennox|away | 00:25 | |
| *** adrian_otto has quit IRC | 00:26 | |
| *** stakeda has joined #openstack-infra | 00:27 | |
| *** funzo has joined #openstack-infra | 00:28 | |
| *** funzo has quit IRC | 00:30 | |
| *** jamielennox|away is now known as jamielennox | 00:31 | |
| pabelanger | jhesketh: ianw: if you get a chance today: https://review.openstack.org/#/c/451863/ | 00:31 |
| jhesketh | pabelanger: sure | 00:32 |
| pabelanger | jhesketh: ianw: we also enabled proxy caching on our mirrors today: http://mirror.regionone.infracloud-vanilla.openstack.org:8080/rdo/centos7/ | 00:32 |
| *** zhurong has joined #openstack-infra | 00:32 | |
| jhesketh | pabelanger: done | 00:32 |
| pabelanger | so we want to monitor HDD usage over the next few days | 00:32 |
| pabelanger | jhesketh: danke | 00:32 |
| jhesketh | yep, we should have things in cacti anyway | 00:32 |
| ianw | pabelanger: yeah, saw some of that in scrollback. just for rdo? | 00:33 |
| pabelanger | ianw: ya, we are experimenting atm. since we only have so much storage on our mirrors. However, we've talked about doing hub.docker.com too | 00:34 |
| pabelanger | ianw: long term for RDO is to move them into AFS | 00:34 |
| ianw | cool, that's what i thought | 00:34 |
| pabelanger | proxy cache is a short term solution for things | 00:34 |
| pabelanger | ianw: https://review.openstack.org/#/c/451972/ I'll be testing that tomorrow | 00:35 |
| *** tuanluong has joined #openstack-infra | 00:35 | |
| *** MasterOfBugs has quit IRC | 00:35 | |
| ianw | cool | 00:37 |
| openstackgerrit | Merged openstack-infra/system-config master: Add infracloud mirrors to cacti.o.o https://review.openstack.org/451863 | 00:38 |
| *** thorst has joined #openstack-infra | 00:38 | |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: add coverage-jobs to congress https://review.openstack.org/452016 | 00:38 |
| *** thorst has quit IRC | 00:39 | |
| *** rossella_s has quit IRC | 00:42 | |
| *** rossella_s has joined #openstack-infra | 00:43 | |
| *** funzo has joined #openstack-infra | 00:45 | |
| *** jerryz has quit IRC | 00:47 | |
| *** gongysh has joined #openstack-infra | 00:47 | |
| *** zz_dimtruck is now known as dimtruck | 00:48 | |
| *** jerryz has joined #openstack-infra | 00:48 | |
| *** dharinic has joined #openstack-infra | 00:51 | |
| *** armax has quit IRC | 00:54 | |
| *** jerryz has quit IRC | 00:54 | |
| *** yuanying has quit IRC | 00:58 | |
| *** tphummel has quit IRC | 01:06 | |
| *** thorst has joined #openstack-infra | 01:10 | |
| *** yuanying has joined #openstack-infra | 01:10 | |
| *** liujiong has joined #openstack-infra | 01:10 | |
| *** jcoufal has joined #openstack-infra | 01:11 | |
| *** EricGonczer_ has joined #openstack-infra | 01:11 | |
| *** aeng_ has quit IRC | 01:13 | |
| *** dprince has joined #openstack-infra | 01:15 | |
| *** gongysh has quit IRC | 01:16 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Add functional job for k8s-cloud-provider with sudo privileges https://review.openstack.org/451977 | 01:16 |
| *** jcoufal has quit IRC | 01:17 | |
| SamYaple | does anyone have a list of the current infra apt mirrors? | 01:21 |
| *** liusheng has joined #openstack-infra | 01:21 | |
| mordred | SamYaple: like the regions? or which repos we mirror? | 01:21 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Add functional job for k8s-cloud-provider with sudo privileges https://review.openstack.org/451977 | 01:22 |
| SamYaple | mordred: which repos | 01:22 |
| mordred | SamYaple: http://mirror.ord.rax.openstack.org/ will let you browse what we mirror - although you can also install openafs-client on your computer and do ls /afs/openstack.org/mirror/ | 01:22 |
| SamYaple | mordred: awesome. yea thats what i needed | 01:23 |
| *** ijw has joined #openstack-infra | 01:23 | |
| SamYaple | interesting naming scheme | 01:24 |
| SamYaple | mordred: supposing i wanted to append to this list, is that system-config? where would the mirroring code for this be since its been over a year since ive looked at it | 01:25 |
| *** dprince has quit IRC | 01:26 | |
| *** bigjools has joined #openstack-infra | 01:27 | |
| *** thorst has quit IRC | 01:28 | |
| SamYaple | oophf. mordred that ceph repo is very out of date | 01:30 |
| *** rbuzatu has quit IRC | 01:31 | |
| *** thorst has joined #openstack-infra | 01:37 | |
| *** thorst has quit IRC | 01:38 | |
| *** Apoorva has quit IRC | 01:50 | |
| *** john-davidge has joined #openstack-infra | 01:54 | |
| *** oanson has quit IRC | 01:55 | |
| *** oanson has joined #openstack-infra | 01:56 | |
| *** john-davidge has quit IRC | 01:59 | |
| *** rfolco has quit IRC | 02:00 | |
| *** yamahata has quit IRC | 02:08 | |
| *** ijw has quit IRC | 02:08 | |
| *** hongbin has joined #openstack-infra | 02:20 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use shorter timeout for update step https://review.openstack.org/448778 | 02:24 |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Add install-guide-jobs for murano https://review.openstack.org/452025 | 02:24 |
| *** smcginnis is now known as smcginnis_vacati | 02:27 | |
| *** rbuzatu has joined #openstack-infra | 02:32 | |
| openstackgerrit | Merged openstack-infra/subunit2sql master: Add run_failure_rate argument to specify values to use in graph https://review.openstack.org/442147 | 02:35 |
| *** rfolco has joined #openstack-infra | 02:36 | |
| *** rfolco has quit IRC | 02:37 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects https://review.openstack.org/452026 | 02:39 |
| *** thorst has joined #openstack-infra | 02:39 | |
| *** cody-somerville has quit IRC | 02:43 | |
| *** csomerville has joined #openstack-infra | 02:43 | |
| *** rfolco has joined #openstack-infra | 02:45 | |
| *** rfolco has quit IRC | 02:45 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql master: Fix run_failure_rate title parameter https://review.openstack.org/452028 | 02:55 |
| *** thorst has quit IRC | 02:56 | |
| *** thorst has joined #openstack-infra | 02:57 | |
| *** hamzy has joined #openstack-infra | 02:58 | |
| *** thorst has quit IRC | 03:01 | |
| *** namnh has joined #openstack-infra | 03:01 | |
| *** dimtruck is now known as zz_dimtruck | 03:03 | |
| *** baoli has joined #openstack-infra | 03:07 | |
| *** yuanying has quit IRC | 03:11 | |
| *** yuanying has joined #openstack-infra | 03:14 | |
| *** baoli has quit IRC | 03:15 | |
| *** amotoki has quit IRC | 03:22 | |
| openstackgerrit | Merged openstack-infra/subunit2sql master: Fix run_failure_rate title parameter https://review.openstack.org/452028 | 03:25 |
| *** arijr has joined #openstack-infra | 03:26 | |
| *** hongbin has quit IRC | 03:31 | |
| *** armax has joined #openstack-infra | 03:33 | |
| *** armax has quit IRC | 03:36 | |
| *** knangia has quit IRC | 03:41 | |
| *** sekelso has quit IRC | 03:42 | |
| *** amotoki has joined #openstack-infra | 03:50 | |
| *** amotoki has quit IRC | 03:56 | |
| *** amotoki has joined #openstack-infra | 04:00 | |
| *** yamamoto has joined #openstack-infra | 04:01 | |
| *** thorst has joined #openstack-infra | 04:01 | |
| *** ihrachys has joined #openstack-infra | 04:02 | |
| *** ijw has joined #openstack-infra | 04:02 | |
| *** Qiming has quit IRC | 04:04 | |
| *** tinwood has quit IRC | 04:05 | |
| *** Qiming has joined #openstack-infra | 04:06 | |
| *** tinwood has joined #openstack-infra | 04:06 | |
| *** thorst has quit IRC | 04:06 | |
| *** zhurong has quit IRC | 04:09 | |
| *** EricGonczer_ has quit IRC | 04:10 | |
| *** tellesnobrega has quit IRC | 04:12 | |
| *** masayukig has quit IRC | 04:14 | |
| *** gouthamr has quit IRC | 04:14 | |
| *** tellesnobrega has joined #openstack-infra | 04:18 | |
| *** ramishra has joined #openstack-infra | 04:18 | |
| *** john-davidge has joined #openstack-infra | 04:21 | |
| *** amotoki has quit IRC | 04:22 | |
| *** armax has joined #openstack-infra | 04:26 | |
| *** john-davidge has quit IRC | 04:26 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 04:26 |
| *** ihrachys has quit IRC | 04:30 | |
| *** zhurong has joined #openstack-infra | 04:30 | |
| *** psachin has joined #openstack-infra | 04:41 | |
| *** yamamoto has quit IRC | 04:42 | |
| *** yamamoto has joined #openstack-infra | 04:44 | |
| *** nadya has joined #openstack-infra | 04:52 | |
| *** nadya has quit IRC | 04:54 | |
| *** dhajare has joined #openstack-infra | 04:56 | |
| *** thorst has joined #openstack-infra | 05:02 | |
| *** thorst has quit IRC | 05:07 | |
| *** yamamoto has quit IRC | 05:09 | |
| *** frickler has quit IRC | 05:11 | |
| *** frickler has joined #openstack-infra | 05:12 | |
| *** ijw has quit IRC | 05:15 | |
| *** tellesnobrega has quit IRC | 05:16 | |
| *** vikrant has joined #openstack-infra | 05:17 | |
| *** arijr has quit IRC | 05:20 | |
| *** dharinic has quit IRC | 05:22 | |
| *** prateek has joined #openstack-infra | 05:23 | |
| *** rlandy has quit IRC | 05:24 | |
| *** e0ne has joined #openstack-infra | 05:25 | |
| *** e0ne has quit IRC | 05:26 | |
| *** yamamoto has joined #openstack-infra | 05:28 | |
| *** tellesnobrega has joined #openstack-infra | 05:29 | |
| *** zhurong has quit IRC | 05:41 | |
| *** yamamoto has quit IRC | 05:41 | |
| *** Hal has quit IRC | 05:43 | |
| *** rcernin has joined #openstack-infra | 05:44 | |
| *** adrian_otto has joined #openstack-infra | 05:52 | |
| *** yolanda has joined #openstack-infra | 05:54 | |
| yolanda | hi AJaeger_ around? need reviews on https://review.openstack.org/451259 and https://review.openstack.org/451898 | 05:54 |
| *** jaosorior has joined #openstack-infra | 05:56 | |
| *** thorst has joined #openstack-infra | 06:03 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql master: Disable SmartyPants for docs https://review.openstack.org/452048 | 06:03 |
| *** andreas_s has joined #openstack-infra | 06:04 | |
| *** zhurong has joined #openstack-infra | 06:05 | |
| *** amotoki has joined #openstack-infra | 06:05 | |
| *** amotoki_ has joined #openstack-infra | 06:08 | |
| *** thorst has quit IRC | 06:08 | |
| *** nadya has joined #openstack-infra | 06:08 | |
| *** amotoki has quit IRC | 06:10 | |
| Dinesh_Bhor | AJaeger_: Hi, Could you please visit this? https://review.openstack.org/#/c/451668/ | 06:19 |
| *** claudiub has joined #openstack-infra | 06:21 | |
| *** chmouel has left #openstack-infra | 06:33 | |
| *** pgadiya has joined #openstack-infra | 06:40 | |
| *** yamamoto has joined #openstack-infra | 06:42 | |
| *** eroux has joined #openstack-infra | 06:47 | |
| *** markus_z has joined #openstack-infra | 06:48 | |
| yolanda | hi ianw , thanks. Care reviewing https://review.openstack.org/451259 as well? | 06:49 |
| *** yamamoto has quit IRC | 06:49 | |
| Dinesh_Bhor | ianw: Hi Ian, It will be great if you visit this: https://review.openstack.org/#/c/451668/ | 06:51 |
| *** d0ugal has joined #openstack-infra | 06:52 | |
| *** d0ugal has quit IRC | 06:52 | |
| *** d0ugal has joined #openstack-infra | 06:52 | |
| *** d0ugal has quit IRC | 06:54 | |
| *** armax has quit IRC | 06:56 | |
| *** martinkopec has joined #openstack-infra | 06:57 | |
| *** d0ugal has joined #openstack-infra | 06:58 | |
| *** d0ugal has quit IRC | 06:58 | |
| *** d0ugal has joined #openstack-infra | 06:58 | |
| *** florianf has joined #openstack-infra | 06:58 | |
| *** pgadiya has quit IRC | 07:01 | |
| *** thorst has joined #openstack-infra | 07:04 | |
| *** hashar has joined #openstack-infra | 07:08 | |
| *** nadya has quit IRC | 07:08 | |
| *** tesseract has joined #openstack-infra | 07:08 | |
| *** thorst has quit IRC | 07:09 | |
| *** pgadiya has joined #openstack-infra | 07:10 | |
| *** rbuzatu has quit IRC | 07:12 | |
| *** rbuzatu has joined #openstack-infra | 07:12 | |
| *** jpich has joined #openstack-infra | 07:14 | |
| *** matrohon has joined #openstack-infra | 07:16 | |
| *** dharinic has joined #openstack-infra | 07:16 | |
| *** dhajare has quit IRC | 07:17 | |
| *** rbuzatu has quit IRC | 07:18 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: give me more debug https://review.openstack.org/450800 | 07:21 |
| *** d0ugal has quit IRC | 07:22 | |
| *** dharinic has quit IRC | 07:23 | |
| *** alexchadin has joined #openstack-infra | 07:24 | |
| *** Qiming has quit IRC | 07:26 | |
| *** d0ugal has joined #openstack-infra | 07:26 | |
| *** d0ugal has quit IRC | 07:26 | |
| *** d0ugal has joined #openstack-infra | 07:26 | |
| *** pcaruana has joined #openstack-infra | 07:27 | |
| gibi | hi! I see massive amount of POST_FAILUREs in the check queue. Is it already a known issue? | 07:28 |
| *** Qiming has joined #openstack-infra | 07:32 | |
| *** dhajare has joined #openstack-infra | 07:33 | |
| andreas_s | gibi, facing that as well... | 07:34 |
| *** ralonsoh has joined #openstack-infra | 07:34 | |
| yolanda | hitting the same, going to take a look | 07:35 |
| *** ralonsoh_ has joined #openstack-infra | 07:35 | |
| gibi | thanks for looking into it. If I can help just let me know. | 07:38 |
| *** ralonsoh has quit IRC | 07:39 | |
| yolanda | #status alert Jobs in gate are failing with POST_FAILURE. Infra roots are investigating | 07:40 |
| openstackstatus | yolanda: sending alert | 07:40 |
| *** dhajare has quit IRC | 07:40 | |
| *** chem has joined #openstack-infra | 07:42 | |
| -openstackstatus- NOTICE: Jobs in gate are failing with POST_FAILURE. Infra roots are investigating | 07:42 | |
| *** ChanServ changes topic to "Jobs in gate are failing with POST_FAILURE. Infra roots are investigating" | 07:42 | |
| yolanda | ok seems the logs volume is corrupted, going to fix | 07:43 |
| *** ralonsoh_ is now known as ralonsoh | 07:44 | |
| openstackstatus | yolanda: finished sending alert | 07:46 |
| *** nadya has joined #openstack-infra | 07:50 | |
| *** katkapilatova has quit IRC | 07:54 | |
| *** rcernin has quit IRC | 07:54 | |
| *** dtantsur has quit IRC | 07:54 | |
| *** tesseract has quit IRC | 07:54 | |
| *** katkapilatova has joined #openstack-infra | 07:55 | |
| *** rcernin has joined #openstack-infra | 07:55 | |
| *** matrohon has quit IRC | 07:55 | |
| *** tesseract has joined #openstack-infra | 07:56 | |
| *** matrohon has joined #openstack-infra | 07:56 | |
| *** nadya has quit IRC | 07:56 | |
| *** dhajare has joined #openstack-infra | 07:57 | |
| *** dtantsur has joined #openstack-infra | 07:57 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** nadya has joined #openstack-infra | 08:03 | |
| *** zzzeek has joined #openstack-infra | 08:04 | |
| *** Douhet has quit IRC | 08:05 | |
| *** thorst has joined #openstack-infra | 08:05 | |
| *** nadya has quit IRC | 08:08 | |
| *** Julien-zte has joined #openstack-infra | 08:09 | |
| *** jogo has quit IRC | 08:13 | |
| *** rbuzatu has joined #openstack-infra | 08:15 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack-infra/bindep master: Update 'pbr' version requirements https://review.openstack.org/451333 | 08:15 |
| *** jogo has joined #openstack-infra | 08:15 | |
| *** vsaienk0 has joined #openstack-infra | 08:15 | |
| vsaienk0 | AJaeger_: yolanda please review https://review.openstack.org/#/c/450822/ move ironic-standalone to voting mode, and disable some of ironic jobs... | 08:16 |
| *** ccamacho has joined #openstack-infra | 08:17 | |
| *** Douhet has joined #openstack-infra | 08:18 | |
| *** rbuzatu has quit IRC | 08:19 | |
| *** rbuzatu has joined #openstack-infra | 08:19 | |
| yolanda | #status alert logs.openstack.org has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed | 08:20 |
| openstackstatus | yolanda: sending alert | 08:20 |
| ianw | yolanda: did you reboot or something? | 08:21 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Adapt getthelogs UX for more use cases https://review.openstack.org/449552 | 08:22 |
| -openstackstatus- NOTICE: logs.openstack.org has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed | 08:23 | |
| *** ChanServ changes topic to "logs.openstack.org has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed" | 08:23 | |
| *** thorst has quit IRC | 08:24 | |
| *** e0ne has joined #openstack-infra | 08:24 | |
| *** shardy has joined #openstack-infra | 08:26 | |
| *** lucas-afk is now known as lucasagomes | 08:26 | |
| openstackstatus | yolanda: finished sending alert | 08:27 |
| ianw | ah, it's booting | 08:28 |
| ianw | we should disable graphical bootup | 08:28 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Update pip in DLRN venv https://review.openstack.org/438746 | 08:30 |
| *** derekh has joined #openstack-infra | 08:32 | |
| *** dhajare has quit IRC | 08:35 | |
| *** shardy has quit IRC | 08:37 | |
| *** shardy has joined #openstack-infra | 08:38 | |
| *** rossella_s has quit IRC | 08:42 | |
| yolanda | ianw, it seems to be repairing the logs filesyste, but is taking ages | 08:43 |
| *** rossella_s has joined #openstack-infra | 08:43 | |
| ianw | yeah, it will ... there was a ticket saying something was up with block storage | 08:44 |
| *** nadya has joined #openstack-infra | 08:46 | |
| *** dhajare has joined #openstack-infra | 08:49 | |
| ttx | we seem to have lost other websites (governance, tarballs, releases) which probably run from trhe same host | 08:50 |
| ttx | if you confirm it's related, I can send a notice so that people know | 08:51 |
| *** matrohon has quit IRC | 08:51 | |
| *** matrohon has joined #openstack-infra | 08:52 | |
| yolanda | yes, the static vm cannot boot, as it's having problems with the logs filesystem | 08:53 |
| yolanda | ttx, however, can you wait a bit? seems the repair finished | 08:53 |
| ianw | yolanda: hmm, sorry, i tried switching virtual terminals (ctl-alt-f1) | 08:53 |
| ianw | i was expecting to see status but it seems blank | 08:54 |
| ttx | ah sure | 08:54 |
| *** ssbarnea has quit IRC | 08:54 | |
| yolanda | ianw, so not finished yet? | 08:54 |
| yolanda | can you restart the fsck for logs? | 08:54 |
| ianw | yolanda: i dunno ... are you watching the console? | 08:55 |
| yolanda | no | 08:55 |
| ianw | ok, so the graphical bit is console 7 | 08:56 |
| ianw | ok, i just made it skip | 08:57 |
| yolanda | ok, repairing logs volume | 08:58 |
| yolanda | but at least other services will be up | 08:58 |
| ianw | yolanda: ok, are you running it in a screen? | 08:58 |
| yolanda | och | 08:58 |
| yolanda | ouch | 08:58 |
| ianw | doesn't matter, i usually just run in a screen with verbose and it gives you a little clue | 08:59 |
| ianw | as to how long | 08:59 |
| yolanda | going to do it | 08:59 |
| *** matrohon has quit IRC | 08:59 | |
| *** ssbarnea has joined #openstack-infra | 09:02 | |
| *** pgadiya has quit IRC | 09:02 | |
| gcb | clarkb: yes, we should use released version of oslo libraries, we test our master branch with peridic jobs like http://status.openstack.org/openstack-health/#/g/build_name/periodic-nova-py27-with-oslo-master | 09:05 |
| gcb | clarkb: please add me as reviewer in your change | 09:05 |
| *** dizquierdo has joined #openstack-infra | 09:06 | |
| *** baoli has joined #openstack-infra | 09:07 | |
| *** arxcruz|pto has quit IRC | 09:11 | |
| *** baoli has quit IRC | 09:12 | |
| ianw | yolanda: i sent it sigusr1 to start outputting progress | 09:12 |
| ianw | 8% | 09:13 |
| yolanda | so slow... | 09:13 |
| *** dhajare has quit IRC | 09:14 | |
| *** matrohon has joined #openstack-infra | 09:17 | |
| *** pgadiya has joined #openstack-infra | 09:18 | |
| *** annegentle has quit IRC | 09:18 | |
| *** arxcruz|pto has joined #openstack-infra | 09:19 | |
| *** gcb has quit IRC | 09:21 | |
| *** electrofelix has joined #openstack-infra | 09:21 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config master: Enable ironic service in congress gate job https://review.openstack.org/452110 | 09:24 |
| ianw | yolanda: did you reboot the server, or was it done automatically? | 09:25 |
| *** yatinkarel is now known as yatin_away | 09:26 | |
| *** pgadiya has quit IRC | 09:26 | |
| yolanda | ianw, i did a restart because i was not able to proceed to the fsck. Although lsof/fuser was not showing any process using the volume, fsck was telling the filesystem was in use | 09:27 |
| ianw | ok, np, just wondering if rax had done that after their maintenance | 09:27 |
| *** matrohon_ has joined #openstack-infra | 09:28 | |
| *** peterlisak has quit IRC | 09:28 | |
| *** onovy has quit IRC | 09:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add prep upper constraints macro to bifrost jobs https://review.openstack.org/451898 | 09:29 |
| *** matrohon has quit IRC | 09:30 | |
| *** pgadiya has joined #openstack-infra | 09:30 | |
| *** dhajare has joined #openstack-infra | 09:31 | |
| *** Julien-zte has quit IRC | 09:31 | |
| *** stakeda has quit IRC | 09:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add upper constraints limitation of buildimage jobs https://review.openstack.org/451259 | 09:33 |
| *** tosky has joined #openstack-infra | 09:35 | |
| *** psachin has quit IRC | 09:36 | |
| *** yanyanhu has joined #openstack-infra | 09:37 | |
| ianw | yolanda: i sent a mail about fiddling the boot config, since nobody else is awake :) | 09:38 |
| *** peterlisak has joined #openstack-infra | 09:39 | |
| ianw | yolanda: one thought is, are we going to be able to mount over /srv/static/logs, or do we basically have to reboot to free that? | 09:40 |
| yolanda | ianw, in the past what i did is to copy the content of /srv/static/logs to some temporary directory. Then mount the volume, and copy contents from that temporary directory | 09:42 |
| *** onovy has joined #openstack-infra | 09:43 | |
| *** sdague has joined #openstack-infra | 09:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add upper constraints limitation of buildimage jobs https://review.openstack.org/451259 | 09:46 |
| *** rossella_s has quit IRC | 09:46 | |
| *** john-davidge has joined #openstack-infra | 09:47 | |
| *** yanyanhu has quit IRC | 09:48 | |
| *** rossella_s has joined #openstack-infra | 09:48 | |
| *** adrian_otto has quit IRC | 09:53 | |
| *** psachin has joined #openstack-infra | 09:54 | |
| ianw | yolanda: ok, so we don't have to stop zuul? | 09:55 |
| yolanda | no, i don't think so | 09:55 |
| yolanda | i won't remove the alert until we have the right volume on place, but things are passing nwo | 09:56 |
| yolanda | now | 09:56 |
| ianw | ok, i guess we get enough windows where things aren't copying there | 09:56 |
| *** Hal has joined #openstack-infra | 09:58 | |
| *** sputnik13 has joined #openstack-infra | 10:02 | |
| *** matrohon_ has quit IRC | 10:08 | |
| *** matrohon_ has joined #openstack-infra | 10:08 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Exporting UPPER_CONSTRAINTS_FILE on bifrost https://review.openstack.org/452129 | 10:09 |
| *** e0ne has quit IRC | 10:13 | |
| *** Hal has quit IRC | 10:13 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects https://review.openstack.org/452026 | 10:15 |
| *** amotoki_ has quit IRC | 10:15 | |
| *** Hal has joined #openstack-infra | 10:16 | |
| yolanda | AJaeger_, ianw , care reviewing https://review.openstack.org/452129 ? | 10:16 |
| *** e0ne has joined #openstack-infra | 10:16 | |
| *** zhurong has quit IRC | 10:18 | |
| *** liujiong has quit IRC | 10:19 | |
| *** tuanluong has quit IRC | 10:20 | |
| *** nadya has quit IRC | 10:20 | |
| *** thorst has joined #openstack-infra | 10:21 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects https://review.openstack.org/452026 | 10:23 |
| *** funzo has quit IRC | 10:23 | |
| *** GnomeKing has joined #openstack-infra | 10:23 | |
| *** namnh has quit IRC | 10:24 | |
| *** sambetts|afk is now known as sambetts | 10:25 | |
| *** thorst has quit IRC | 10:26 | |
| *** neiljerram has joined #openstack-infra | 10:28 | |
| *** ralonsoh_ has joined #openstack-infra | 10:30 | |
| *** pbourke has quit IRC | 10:30 | |
| *** pbourke has joined #openstack-infra | 10:32 | |
| *** tellesnobrega has quit IRC | 10:33 | |
| *** ralonsoh has quit IRC | 10:34 | |
| *** martinkopec has quit IRC | 10:35 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Add a DSVM job for golang projects https://review.openstack.org/452026 | 10:35 |
| *** Julien-zte has joined #openstack-infra | 10:37 | |
| *** hashar is now known as hasharLunch | 10:37 | |
| *** matrohon_ has quit IRC | 10:38 | |
| *** Julien-zte has quit IRC | 10:39 | |
| *** Julien-zte has joined #openstack-infra | 10:40 | |
| *** zhurong has joined #openstack-infra | 10:41 | |
| *** alexchadin has quit IRC | 10:45 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM Other test for 451445 https://review.openstack.org/451978 | 10:45 |
| *** Hal has quit IRC | 10:48 | |
| *** psachin has quit IRC | 10:50 | |
| *** nadya has joined #openstack-infra | 10:51 | |
| *** gcb has joined #openstack-infra | 10:51 | |
| *** psachin has joined #openstack-infra | 10:52 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/project-config master: Skip some bindeps for old Ubuntu releases https://review.openstack.org/452145 | 10:55 |
| *** bhavik1 has joined #openstack-infra | 10:58 | |
| *** rhallisey has joined #openstack-infra | 10:59 | |
| dims | ianw : fixed the name of the job per your comment - ci is green - https://review.openstack.org/#/c/452026/ thanks! | 11:01 |
| dims | AJaeger_ : yolanda : one of you peek at that as well? will help get the ball rolling on running kubernetes' e2e tests | 11:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Exporting UPPER_CONSTRAINTS_FILE on bifrost https://review.openstack.org/452129 | 11:07 |
| *** mat128 has joined #openstack-infra | 11:09 | |
| *** shardy is now known as shardy_lunch | 11:14 | |
| dims | thanks yolanda frickler | 11:14 |
| *** funzo has joined #openstack-infra | 11:17 | |
| *** ccamacho is now known as ccamacho|lunch | 11:17 | |
| *** hasharLunch is now known as hashar | 11:19 | |
| *** gcb has quit IRC | 11:20 | |
| *** adarazs is now known as adarazs_lunch | 11:20 | |
| *** jkilpatr has joined #openstack-infra | 11:21 | |
| *** yanyanhu has joined #openstack-infra | 11:21 | |
| *** thorst has joined #openstack-infra | 11:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Fix requirements checkout on buildimage https://review.openstack.org/452155 | 11:23 |
| yolanda | ianw, AJaeger_ , still around? AJaeger_ , you mentioned we could have conflicts with the checkout of that macro, and we had: https://review.openstack.org/#/c/452155/1/jenkins/jobs/tripleo.yaml | 11:24 |
| frickler | yolanda: ianw: any update on the fsck progress? how large is that volume, btw? | 11:25 |
| *** yanyanhu has quit IRC | 11:26 | |
| yolanda | it's over 70% | 11:26 |
| *** thorst has quit IRC | 11:27 | |
| frickler | nice, that sounds like some reasonable progress from 8% :) | 11:28 |
| yolanda | not an uniform progress however, looks slower now | 11:28 |
| *** lucasagomes is now known as lucas-hungry | 11:28 | |
| *** knangia has joined #openstack-infra | 11:29 | |
| *** zz_dimtruck is now known as dimtruck | 11:30 | |
| *** john-davidge has quit IRC | 11:32 | |
| *** thorst has joined #openstack-infra | 11:32 | |
| *** amotoki has joined #openstack-infra | 11:32 | |
| *** gcb has joined #openstack-infra | 11:33 | |
| *** alexchadin has joined #openstack-infra | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| *** matrohon has joined #openstack-infra | 11:38 | |
| *** john-davidge has joined #openstack-infra | 11:38 | |
| *** matrohon has quit IRC | 11:41 | |
| *** tellesnobrega has joined #openstack-infra | 11:42 | |
| *** pblaho has quit IRC | 11:45 | |
| *** dimtruck is now known as zz_dimtruck | 11:46 | |
| *** zz_dimtruck is now known as dimtruck | 11:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Switch OVB to use promoted repos from trunk.rdoproject.org https://review.openstack.org/452164 | 11:49 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Update pip in DLRN venv https://review.openstack.org/438746 | 11:51 |
| *** sekelso has joined #openstack-infra | 11:55 | |
| *** dimtruck is now known as zz_dimtruck | 11:56 | |
| *** mat128 has quit IRC | 11:56 | |
| *** mat128 has joined #openstack-infra | 11:58 | |
| *** skelso has joined #openstack-infra | 11:58 | |
| *** zhurong has quit IRC | 11:59 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Switch OVB to use promoted repos from trunk.rdoproject.org https://review.openstack.org/452164 | 11:59 |
| *** sekelso has quit IRC | 12:00 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Remove unused variables for Upgrades https://review.openstack.org/452170 | 12:00 |
| *** prateek has quit IRC | 12:01 | |
| *** prateek has joined #openstack-infra | 12:01 | |
| *** shardy_lunch is now known as shardy | 12:03 | |
| *** florianf has quit IRC | 12:07 | |
| *** dprince has joined #openstack-infra | 12:08 | |
| mnaser | morning folks, thanks for your hard (and late night) work on logs.o.o! | 12:08 |
| mnaser | hope things heal up soon | 12:08 |
| fungi | frickler: that volume is around 12tib | 12:09 |
| *** aysyd has joined #openstack-infra | 12:11 | |
| pabelanger | Ya, didn't get a chance to solve our volume maintenance on static.o.o | 12:11 |
| *** adreznec has quit IRC | 12:14 | |
| fungi | we didn't see an alternative to crossing our fingers and fsck'ing if it broke | 12:15 |
| fungi | no room to add another cinder volume because of xen's virtual block device limit | 12:16 |
| *** adreznec has joined #openstack-infra | 12:17 | |
| frickler | how about splitting it up? like do log0-9.o.o and make logs.o.o proxy based on the first digit? or is there a specific reason it all need to be on one machine? | 12:18 |
| *** pblaho has joined #openstack-infra | 12:18 | |
| *** adarazs_lunch is now known as adarazs | 12:19 | |
| fungi | frickler: we've talked about some sharding solutions. possible to implement but most of the code complication will be on the write routing side, not routing reads | 12:19 |
| Zara | hi, earlier in #storyboard, witek wanted to add python-monascaclient to the monasca group, and I said I'd check if they needed to do anything more than add 'use-storyboard' to gerrit/projects.yaml | 12:19 |
| Zara | soo, this is me checking | 12:19 |
| Zara | as far as I'm aware, there are no bugs filed against it atm | 12:20 |
| Zara | so they wouldn't need to import anything | 12:20 |
| *** ccamacho|lunch is now known as ccamacho | 12:20 | |
| Zara | https://review.openstack.org/#/c/447932/ is their patch | 12:20 |
| fungi | Zara: thanks for the reminder, i meant to look into how i missed setting that in my patch when going through their governance deliverables | 12:21 |
| Zara | ah, great, thanks :) | 12:21 |
| *** trown|outtypewww is now known as trown | 12:22 | |
| *** florianf has joined #openstack-infra | 12:22 | |
| *** lucas-hungry is now known as lucasagomes | 12:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Fix requirements checkout on buildimage https://review.openstack.org/452155 | 12:23 |
| Zara | I'm looking into sorting out the pynotedb infra a bit more; I got waylaid for a while | 12:23 |
| yolanda | hi AJaeger_ , so i guess i missed the "requirements" folder? | 12:23 |
| Zara | I got as far as making #pynotedb but nothing is officially registered or logged etc yet; not sure where to start with that | 12:24 |
| *** pgadiya has quit IRC | 12:24 | |
| Zara | oh, and I need to sort out cores for the repo; I think it's just set to infra-root atm. | 12:27 |
| *** signed8bit_Zzz is now known as signed8b_ | 12:28 | |
| *** rook is now known as rook|tower | 12:31 | |
| dhellmann | governance.o.o is responding extremely slowly for me. Is anyone else seeing an issue? | 12:32 |
| ttx | dhellmann: it's one of the static websites affected by the outage | 12:32 |
| dhellmann | ah | 12:32 |
| dhellmann | I wasn't aware there was an outage. | 12:32 |
| ttx | logs.openstack.org is out | 12:32 |
| dhellmann | ouch | 12:32 |
| ttx | or at least was a couple hours ago | 12:33 |
| fungi | yeah, there's a fsck of the logs volume underway, so starving i/o for other sites served from the same volume group in the meantime | 12:33 |
| dhellmann | ok, as long as it's not a surprise to anyone | 12:34 |
| yolanda | it's on 75% progress | 12:34 |
| * fungi is not surprised, just always looks this way | 12:34 | |
| *** gordc has joined #openstack-infra | 12:34 | |
| yolanda | but since it reached 70% , it became slower | 12:34 |
| mnaser | fsck is usually very unpredictable because the progress depends on how many inodes its going through | 12:35 |
| dhellmann | that last 20% of a project always takes 80% of the time, right? | 12:35 |
| mnaser | if it reaches some project logs with a looooot of small files it'll bog down | 12:35 |
| fungi | in the past, fsck'ing that volume has taken the better part of a day | 12:35 |
| fungi | it's nothing but tons of small files (compressed logs) | 12:35 |
| yolanda | fungi, looks like that. It's running on screen, and looks it won't finish before i'm EOD. Will you take care? | 12:36 |
| fungi | yolanda: happily--thanks for starting it! | 12:36 |
| *** kashyap has left #openstack-infra | 12:36 | |
| yolanda | team effort, over all shifts! :) | 12:36 |
| fungi | zaro: yeah, that core group is still empty. give me a few to take care of rerunning our setup script for infra gerrit groups and let me know who you want added | 12:37 |
| fungi | er, Zara: ^ | 12:37 |
| fungi | (sorry zaro!) | 12:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Use StoryBoard for python-monascaclient https://review.openstack.org/447932 | 12:38 |
| *** tphummel has joined #openstack-infra | 12:38 | |
| Zara | hahaha, thanks :) | 12:39 |
| *** chlong has joined #openstack-infra | 12:40 | |
| *** dmsimard|afk is now known as dmsimard | 12:41 | |
| fungi | caught up on our review group setup now | 12:41 |
| *** vicky has joined #openstack-infra | 12:41 | |
| fungi | Zara: you planning to be an initial core reviewer on that? who else? | 12:41 |
| *** sshnaidm|off has quit IRC | 12:42 | |
| *** vicky has quit IRC | 12:43 | |
| *** vicky has joined #openstack-infra | 12:43 | |
| fungi | Zara: and as far as irc channel registration, see https://docs.openstack.org/infra/system-config/irc.html#channel-requirements | 12:43 |
| *** vikrant has quit IRC | 12:44 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add reporter for Federated Message Bus (fedmsg) https://review.openstack.org/426861 | 12:44 |
| Zara | fungi: ah, thanks for that link. :) I think initial cores should be me + Gravious (richard ipsum) | 12:44 |
| *** vikrant has joined #openstack-infra | 12:45 | |
| mordred | fungi: ooh, neat - monasca moved to storyboard! | 12:46 |
| fungi | Zara: done | 12:46 |
| fungi | mordred: yup, i did the migration on monday (or maybe it was a week ago last monday?) | 12:47 |
| Zara | fungi: wonderful, thanks :D | 12:47 |
| fungi | mordred: also the interop (defcore, refstack, et al) projects are sb as of the same migration timeframe | 12:47 |
| Zara | a week ago last monday | 12:47 |
| Zara | indeed | 12:47 |
| *** vicky has quit IRC | 12:48 | |
| Dinesh_Bhor | Hi all, It will be great if someone approves this: https://review.openstack.org/#/c/451668/ It's already having one +2. | 12:48 |
| *** rlandy has joined #openstack-infra | 12:48 | |
| *** signed8b_ is now known as signed8bit_Zzz | 12:48 | |
| mordred | fungi: that's great news! | 12:49 |
| fungi | yolanda: i'm _finally_ attached to that screen session. the server is slower than even i would have expected. i'm going to double-check there's nothing else slowing it down even worse | 12:50 |
| fungi | well, rather i've finally gotten to the point where i can run screen -x as root anyway (it hasn't returned yet) | 12:51 |
| fungi | huh, there are two screen sessions on there, both started within a minute of each other | 12:51 |
| mnaser | fungi i think something happened which made it worse in the past few minutes.. | 12:52 |
| mnaser | logs.o.o loaded at first and now it's zero response | 12:52 |
| *** andreas_s has quit IRC | 12:52 | |
| mnaser | this morning i could get the the homepage / directoy listing there, now that doesnt seem to respond | 12:52 |
| fungi | yeah, just running things on the command line takes minutes to get a response | 12:53 |
| yolanda | fungi, and is fsck causing it? | 12:53 |
| fungi | yolanda: if i could run commands i might be able to find out :/ | 12:54 |
| fungi | my screen -x retry has been hanging for several minutes now | 12:56 |
| *** flepied has quit IRC | 12:57 | |
| *** Goneri has joined #openstack-infra | 12:57 | |
| *** zz_dimtruck is now known as dimtruck | 12:58 | |
| fungi | cacti says something there has started using ~4gb of swap | 13:00 |
| *** gouthamr has joined #openstack-infra | 13:01 | |
| Dinesh_Bhor | yolanda: Hi, whenever you get time may I get your attention on this? https://review.openstack.org/#/c/451668/ | 13:01 |
| frickler | might be the fsck process eating up pretty much memory | 13:01 |
| fungi | frickler: might be all manner of tings. speculation is fun | 13:02 |
| fungi | i'm trying to get a second ssh session to it, though i don't have high hopes | 13:03 |
| fungi | nope, connection reset by peer | 13:03 |
| fungi | in the past i've taken precautions before fsck'ing that volume, like making sure our log grooming cron wasn't running, disabling the jenkins ssh key so new logs couldn't be uploaded, disabling the logs vhost in apache, stopping puppet from updating... | 13:05 |
| fungi | anything to free up rsources so that fsck would go faster | 13:06 |
| *** e0ne has quit IRC | 13:06 | |
| fungi | yeah, my "active" ssh session hasn't responded to me for a solid 15 minutes now, and additional attempts to ssh are getting connection resets (probably fighting all the jobs trying to upload logs) | 13:07 |
| fungi | may need to reboot the server, start all over and hope the interrupted fsck doesn't make matters worse | 13:08 |
| fungi | infra-root: heads up | 13:08 |
| fungi | i'm moving to #openstack-infra-incident for this | 13:08 |
| *** xyang1 has joined #openstack-infra | 13:09 | |
| *** jamesdenton has joined #openstack-infra | 13:10 | |
| *** jamesden_ has joined #openstack-infra | 13:13 | |
| *** jcoufal has joined #openstack-infra | 13:13 | |
| *** esberglu has joined #openstack-infra | 13:14 | |
| *** flepied has joined #openstack-infra | 13:14 | |
| *** baoli has joined #openstack-infra | 13:14 | |
| *** jamesdenton has quit IRC | 13:14 | |
| *** baoli has quit IRC | 13:14 | |
| *** baoli has joined #openstack-infra | 13:15 | |
| *** kgiusti has joined #openstack-infra | 13:18 | |
| *** dave-mccowan has joined #openstack-infra | 13:18 | |
| *** signed8bit_Zzz is now known as signed8b_ | 13:20 | |
| *** mdrabe has joined #openstack-infra | 13:20 | |
| andreykurilin | clarkb: hi! Did you have a chance to verify my patch for features.yaml ? Maybe we can check it in our gates (there is a check to not apply custom features.yaml to master branch, so nothing critial will be affected)? | 13:22 |
| sbezverk | hello infra team, is it safe to recheck now, or you still chasing POST_FAILURE issue? | 13:23 |
| sdake | sbezverk the bots automatically change the subjects in all registered channels when things are fixed | 13:24 |
| sdake | fungi note - i'd recommend "mosh" which uses UDP for ssh connectivity - it can survive the worst of networking connections | 13:25 |
| sdake | fungi we use it in our labs when someone breaks routing/switching/etc (e.g. when packet loss > 75%) | 13:26 |
| fungi | sdake: that's what i use for my personal systems, but isn't much help when the kernel is killing the sshd process | 13:26 |
| sdake | fungi right - just trying to help :) | 13:26 |
| fungi | this isn't a network connectivity issue | 13:26 |
| fungi | it's a server under extreme load and memory pressure issue | 13:26 |
| sdake | fungi i dont mean to interfere - however - if you can somehow get into the box, you might try a firewall on the ports used to upload things | 13:27 |
| sdake | fungi that may relieve pressure | 13:27 |
| *** wolverineav has joined #openstack-infra | 13:28 | |
| *** AJaeger_ is now known as AJaeger | 13:28 | |
| pabelanger | okay, who is an apache RewriteRules master? need some help turning /gem/gems/foobar.gem to /gem/gems/f/fo/foobar.gem currently I am testing with http://martinmelin.se/rewrite-rule-tester/ but having issues | 13:28 |
| *** baoli has quit IRC | 13:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add a DSVM job for golang projects https://review.openstack.org/452026 | 13:29 |
| fungi | sdake: i already had a session on it but a command i was running was hanging. finally got some responsiveness back and confirmed the wsgi filter we use for log displays is competing with the fsck for available memory, so working on disabling that vhost in apache temporarily | 13:30 |
| *** tellesnobrega has quit IRC | 13:31 | |
| sdake | its annoying that a journaling filesystem has file corruption - journaling is supposed to prevent that | 13:32 |
| sdake | anyway on that note i'll leave ya to it- you know way more about this stuf fthen i do :) | 13:33 |
| *** mariusv has quit IRC | 13:34 | |
| frickler | pabelanger: I'm not a master, but I'm willing to take a look as having that gem mirror working would be really useful to me. I'm about to leave now, though, but if you post what you currently have I'll give it a try later | 13:35 |
| fungi | sdake: in this case the filesystem is on a logical volume spread across a dozen cinder volumes, so if we lose network access to one briefly and the filesystem gets remounted read-only by the kernel, we're uncomfortable putting it back writable until we fsck it. unfortunately it's ~12tib of very small files | 13:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix requirements checkout on buildimage https://review.openstack.org/452155 | 13:38 |
| *** tbarron|out is now known as tbarron | 13:39 | |
| *** dhajare has quit IRC | 13:39 | |
| *** ralonsoh__ has joined #openstack-infra | 13:41 | |
| *** bhavik1 has quit IRC | 13:42 | |
| *** ralonsoh_ has quit IRC | 13:43 | |
| *** nadya has quit IRC | 13:44 | |
| *** rossella_s has quit IRC | 13:46 | |
| *** mrunge- is now known as mrunge | 13:47 | |
| *** rossella_s has joined #openstack-infra | 13:47 | |
| *** baoli has joined #openstack-infra | 13:47 | |
| *** ZZelle has quit IRC | 13:49 | |
| *** dizquierdo has quit IRC | 13:50 | |
| *** ZZelle has joined #openstack-infra | 13:50 | |
| openstackgerrit | Ari proposed openstack-infra/jenkins-job-builder master: Add support for jms_messaging plugin as a trigger https://review.openstack.org/451871 | 13:50 |
| *** ZZelle is now known as Guest11182 | 13:50 | |
| *** ihrachys has joined #openstack-infra | 13:51 | |
| *** marst has quit IRC | 13:52 | |
| *** e0ne has joined #openstack-infra | 13:53 | |
| *** alexchadin has quit IRC | 13:53 | |
| *** e0ne has quit IRC | 13:53 | |
| *** mriedem has joined #openstack-infra | 13:53 | |
| *** amoralej is now known as amoralej|lunch | 13:54 | |
| *** Dinesh_Bhor has quit IRC | 13:55 | |
| *** dimtruck is now known as zz_dimtruck | 13:55 | |
| *** zz_dimtruck is now known as dimtruck | 13:55 | |
| *** e0ne has joined #openstack-infra | 13:56 | |
| *** e0ne has quit IRC | 13:56 | |
| *** tphummel has quit IRC | 13:57 | |
| ihrachys | is it just me, or links to logs in gerrit are messed up? | 13:57 |
| *** marst has joined #openstack-infra | 13:58 | |
| AJaeger | ihrachys: see $TOPIC | 13:58 |
| ttx | ihrachys: /topic | 13:58 |
| *** eharney has joined #openstack-infra | 13:58 | |
| openstackgerrit | Roman Podoliaka proposed openstack-infra/project-config master: osc-placement: experimental job for running of functional tests https://review.openstack.org/452210 | 13:58 |
| ihrachys | sorry | 13:58 |
| *** amotoki has quit IRC | 13:59 | |
| *** sekelso has joined #openstack-infra | 14:04 | |
| *** skelso has quit IRC | 14:04 | |
| *** tphummel has joined #openstack-infra | 14:04 | |
| *** gcb has quit IRC | 14:05 | |
| fungi | ihrachys: the log prettifier consumes a ton of memory, so we have to disable the logs.o.o site in apache to free up enough for fsck unfortunately | 14:06 |
| andreaf | fungi: is it just me or logs.openstack.org is getting redirected to https://docs.openstack.org/infra/system-config ? | 14:08 |
| *** gcb has joined #openstack-infra | 14:09 | |
| AJaeger | andreaf: side effect of $TOPIC | 14:09 |
| andreaf | AJaeger: oh why can't I read :D - thank you | 14:10 |
| *** asselin has quit IRC | 14:11 | |
| fungi | andreaf: disabling the vhost (to reduce load from the log prettifier) is probably causing it to get redirected to the default vhost | 14:11 |
| *** Guest11182 is now known as ZZelle_ | 14:12 | |
| *** jamielennox is now known as jamielennox|away | 14:16 | |
| *** Julien-zte has quit IRC | 14:20 | |
| *** Julien-zte has joined #openstack-infra | 14:21 | |
| *** yamamoto has joined #openstack-infra | 14:21 | |
| openstackgerrit | Anton Studenov proposed openstack-infra/project-config master: [rally] Add hook for readthedocs https://review.openstack.org/452221 | 14:22 |
| *** brianw has joined #openstack-infra | 14:24 | |
| *** arijr has joined #openstack-infra | 14:25 | |
| *** hashar has quit IRC | 14:26 | |
| *** sekelso has quit IRC | 14:28 | |
| *** arijr has quit IRC | 14:29 | |
| openstackgerrit | Adam Reznechek proposed openstack-infra/project-config master: Run check-requirements on remaining powervm projects https://review.openstack.org/452224 | 14:31 |
| openstackgerrit | Ari proposed openstack-infra/jenkins-job-builder master: Add support for jms_messaging plugin as a trigger https://review.openstack.org/451871 | 14:31 |
| *** jroll has quit IRC | 14:32 | |
| *** jroll has joined #openstack-infra | 14:34 | |
| *** vikrant has quit IRC | 14:35 | |
| *** amoralej|lunch is now known as amoralej | 14:36 | |
| dmsimard | Just a general curiosity - why is the gerrit merge strategy set to "merge if necessary" rather than "rebase if necessary" ? | 14:36 |
| mordred | dmsimard: because rebase if necessary creates a state where git pull may not work as a developer expects | 14:37 |
| pabelanger | and it would slow down things | 14:37 |
| mordred | dmsimard: (we had it set for a while on some of the infra repos as a trial run - and it turns out to confuse folks whose git-fu is lower) | 14:37 |
| pabelanger | if you want clean git logs, you likely want git cherry-pick | 14:38 |
| *** rbrndt has joined #openstack-infra | 14:38 | |
| dmsimard | fair, thanks for explanation :) | 14:38 |
| mordred | there's another nice side-effect of merge-if-necessary, which is that gpg-signed commits make it into the tree with signatures intact | 14:39 |
| *** prateek has quit IRC | 14:39 | |
| mordred | so you get things like this nice verified badge over on github: https://github.com/openstack-infra/shade/commit/c1984b636ee9fc6101c28bb57723d2d490b04586 | 14:39 |
| mnaser | i should really get around setting up pgp | 14:40 |
| mnaser | i guess today could be the day while we wait for fsck :< | 14:41 |
| dmsimard | yeah I saw something about auto signing commits recently.. I been only signing tags | 14:41 |
| dmsimard | https://twitter.com/fdevillamil/status/846080928795394048 | 14:41 |
| mordred | dmsimard: [commit] | 14:42 |
| mordred | gpgsign = true | 14:42 |
| mordred | in ~/.gitconfig | 14:42 |
| mordred | and all works nicely | 14:42 |
| *** askb has quit IRC | 14:42 | |
| mnaser | (any recommended choice gpg docs for macos user?) | 14:42 |
| dmsimard | mordred: that's the same thing as doing git config --global :p | 14:42 |
| mordred | yup | 14:42 |
| *** yolanda has quit IRC | 14:43 | |
| mordred | you don't actually have to set signingkey, fwiw, unless git has a sad | 14:43 |
| sdake | does anyoen know if its ok to submit reviews and not cause infra problems on the recovery on logs.oo? | 14:43 |
| mordred | mnaser: sorry - I know nothing about macos) | 14:43 |
| *** jaypipes has joined #openstack-infra | 14:43 | |
| mnaser | im sure most *nix tooling will still work and exists | 14:44 |
| *** jaypipes is now known as leakypipes | 14:44 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Add log publisher for LOCI https://review.openstack.org/452229 | 14:44 |
| mordred | mnaser: I certainly hope so | 14:45 |
| *** dimtruck is now known as zz_dimtruck | 14:45 | |
| *** zz_dimtruck is now known as dimtruck | 14:45 | |
| *** chem has quit IRC | 14:46 | |
| *** chem has joined #openstack-infra | 14:46 | |
| *** yolanda has joined #openstack-infra | 14:47 | |
| fungi | sdake: i don't think a handful of people avoiding uploading changes will be much help. just be prepared that you probably won't be able to look at the ci logs for it for a while still (or possibly at all depending on if we run out of room to spool them while waiting for the fsck to finish) | 14:48 |
| *** sekelso has joined #openstack-infra | 14:49 | |
| SamYaple | is the 'link-logs' builder needed for the devstack-logs publisher? | 14:50 |
| pabelanger | devstack-logs | 14:51 |
| pabelanger | err | 14:51 |
| fungi | SamYaple: link-logs builder just echoes the url for the logs into the console log | 14:51 |
| pabelanger | link-logs just shows you the URL of the logs when we upload them | 14:51 |
| sdake | fungi ok thanks | 14:52 |
| fungi | SamYaple: it's mostly in case you're streaming the log console and want to know where the logs will wind up, without having to bounce back through the review and find the url linked there | 14:52 |
| *** rossella_s has quit IRC | 14:53 | |
| SamYaple | oh i see. that is something i do | 14:53 |
| SamYaple | ill add that in | 14:53 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Add log publisher for LOCI https://review.openstack.org/452229 | 14:54 |
| *** dimtruck is now known as zz_dimtruck | 14:54 | |
| *** xarses has joined #openstack-infra | 14:56 | |
| *** xarses_ has joined #openstack-infra | 14:57 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/bindep master: bindep: depends: Add distro family aliases https://review.openstack.org/450634 | 14:57 |
| *** xarses has quit IRC | 15:01 | |
| *** xarses has joined #openstack-infra | 15:01 | |
| *** ijw has joined #openstack-infra | 15:01 | |
| *** xarses_ has quit IRC | 15:02 | |
| *** xarses_ has joined #openstack-infra | 15:03 | |
| *** xarses has quit IRC | 15:05 | |
| *** xarses has joined #openstack-infra | 15:06 | |
| *** xarses has quit IRC | 15:06 | |
| *** xarses has joined #openstack-infra | 15:07 | |
| *** xarses_ has quit IRC | 15:08 | |
| *** rcernin has quit IRC | 15:08 | |
| *** nadya has joined #openstack-infra | 15:11 | |
| *** admcleod is now known as not_adam | 15:12 | |
| *** psachin has quit IRC | 15:13 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Allow developers to recheck patches with emotion in the check pipeline https://review.openstack.org/452247 | 15:17 |
| *** not_adam is now known as admcleod | 15:17 | |
| *** dolphm has left #openstack-infra | 15:18 | |
| *** jcoufal has quit IRC | 15:18 | |
| *** e0ne has joined #openstack-infra | 15:18 | |
| *** jcoufal has joined #openstack-infra | 15:19 | |
| Zara | :) | 15:20 |
| AJaeger | Is it already first of April? ^ | 15:20 |
| *** sekelso has quit IRC | 15:20 | |
| *** zz_dimtruck is now known as dimtruck | 15:20 | |
| AJaeger | dmsimard: what timezone are you in? | 15:20 |
| SamYaple | AJaeger: 7 hours away friend | 15:20 |
| dmsimard | AJaeger: git commit --date="Sat Apr 1 14:00 2017 +0100" | 15:21 |
| dmsimard | I cheated because of that intel guy | 15:21 |
| AJaeger | dmsimard: Commit time is different ;) | 15:21 |
| AJaeger | dmsimard: yeah... | 15:21 |
| SamYaple | i say we merge it. its before april | 15:21 |
| jroll | that is beautiful | 15:21 |
| *** yamamoto has quit IRC | 15:21 | |
| *** armax has joined #openstack-infra | 15:22 | |
| *** hongbin has joined #openstack-infra | 15:23 | |
| mordred | I kind of think we should merge it regardless of the date | 15:24 |
| SamYaple | mordred: agreed. i dont see how this could spiraling out of control at all | 15:24 |
| dmsimard | Probably need to fix the regex if we really want it, it's bound to pick up those unescaped characters :) | 15:24 |
| sdague | mordred: yeh, I'd be totally ok keeping that as an easter egg | 15:25 |
| *** yamamoto has joined #openstack-infra | 15:25 | |
| *** sekelso has joined #openstack-infra | 15:25 | |
| sdague | yeh, I don't think that regex would actually work at all :) | 15:25 |
| AJaeger | what about removing recheck and reverify completely? ;) | 15:25 |
| SamYaple | what about bringing back reverify.... | 15:26 |
| dmsimard | OH, this guy ^ | 15:26 |
| * dmsimard misses reverify | 15:26 | |
| *** adrian_otto has joined #openstack-infra | 15:26 | |
| dmsimard | wait, let me do a reverify and go through the whole check AND gate pipeline again | 15:26 |
| dmsimard | </friday> | 15:26 |
| SamYaple | oh look ive made myself sad | 15:27 |
| *** jaosorior has quit IRC | 15:28 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Support container upgrades in multinode OOOQ CI https://review.openstack.org/450784 | 15:31 |
| *** e0ne has quit IRC | 15:39 | |
| mriedem | dmsimard: recheck == reverify | 15:41 |
| dmsimard | mriedem: it wasn't always that way | 15:41 |
| mriedem | as in reverify was gate only? | 15:41 |
| dmsimard | reverify used to "recheck" through the gate pipeline only for jobs that failed the gate pipeline | 15:41 |
| dmsimard | yeah | 15:41 |
| mriedem | i believe that was removed for a good reason | 15:42 |
| mriedem | so things had to go back through check | 15:42 |
| mriedem | i'm sure sdague has the details | 15:42 |
| *** tonytan4ever has joined #openstack-infra | 15:42 | |
| *** apevec has joined #openstack-infra | 15:43 | |
| clarkb | fungi: reading up on scrollback it looks like fscking is under control (but slow)? also what is spooling writes (you mentioned it in response to sdake) | 15:43 |
| apevec | release.o.o is down | 15:43 |
| apevec | ah it's back | 15:44 |
| apevec | nm | 15:44 |
| apevec | ( releases.openstack.org ) | 15:44 |
| fungi | apevec: i brought it back up again. we're playing with stopping apache on the server to relieve load during the logs fsck | 15:44 |
| mnaser | apevec i had issues with tarballs.o.o when trying to build images so im putting that aside till fsck's are done | 15:44 |
| fungi | clarkb: the /srv/static volume which is 99gb and normally unused | 15:44 |
| clarkb | also apparently not in the incident channel | 15:44 |
| clarkb | er I'm not in | 15:44 |
| clarkb | mriedem: dmsimard we require check to pass before gating. Which essentially made reverify == recheck | 15:45 |
| *** pcaruana has quit IRC | 15:46 | |
| dmsimard | clarkb: talking about reviews that passed check but failed gate (bad mirror or whatever) | 15:46 |
| *** tonytan_brb has joined #openstack-infra | 15:46 | |
| dmsimard | It feels wasteful (both in resources and time) to make it go through the check pipeline once more | 15:46 |
| clarkb | dmsimard: I think what we found was it resulted in a significantly more reliable gate pipeline so was worth it | 15:47 |
| fungi | dmsimard: yep, pretty impossible to systematically differentiate those from jobs which failed because the change ceased to be valid between when it went through check and (maybe weeks or months later) through the gate | 15:47 |
| clarkb | dmsimard: that and the window scaling of the gate removed the massive gate pipelines and resets that were much more problematic | 15:47 |
| *** yamamoto has quit IRC | 15:47 | |
| dmsimard | fair if the pros outweighs the cons :) | 15:47 |
| clarkb | dmsimard: you have to remember that a single gate failure restarts everything behind it | 15:48 |
| clarkb | dmsimard: so cost of failing in gate is really high compared to run jobs for single change again if it is flaky | 15:48 |
| fungi | dmsimard: lots of projects, even big projects, have core reviewers approving changes which have months-old check results | 15:48 |
| *** tonytan4ever has quit IRC | 15:48 | |
| *** yamamoto has joined #openstack-infra | 15:48 | |
| fungi | and so reverify would be a means of reenqueuing those into the gate and causing gate resets over and over instead of just once before it would have to go back through and get a clean check | 15:49 |
| *** armax has quit IRC | 15:49 | |
| fungi | not a hypothetical. it absolutely was happening quite a lot | 15:49 |
| *** jordanP has quit IRC | 15:49 | |
| *** armax has joined #openstack-infra | 15:50 | |
| *** yamamoto has quit IRC | 15:50 | |
| *** yamamoto has joined #openstack-infra | 15:50 | |
| *** yamamoto has quit IRC | 15:50 | |
| *** ralonsoh__ is now known as ralonsoh | 15:54 | |
| markmcclain | I think something is broken for posting logs... getting 404 for https://docs.openstack.org/infra/system-config/17/452217/1/check/gate-networking-arista-pep8-ubuntu-xenial/77c079d/ also had POST_REVIEW failures earlier too | 15:54 |
| ihrachys | markmcclain: see topic | 15:54 |
| *** Swami has joined #openstack-infra | 15:55 | |
| markmcclain | ihrachys: ah thanks | 15:55 |
| *** ramishra has quit IRC | 15:55 | |
| *** arxcruz|pto has quit IRC | 15:56 | |
| *** armax has quit IRC | 15:57 | |
| openstackgerrit | Ari proposed openstack-infra/jenkins-job-builder master: Add support for jms_messaging plugin as a trigger https://review.openstack.org/451871 | 15:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add apache rewrite rule for ruby gems https://review.openstack.org/452253 | 15:58 |
| *** armax has joined #openstack-infra | 15:58 | |
| pabelanger | fungi: clarkb: okay, it took some testing locally, but ^ is our rewrite rules | 15:58 |
| pabelanger | for gems | 15:58 |
| pabelanger | if people want to confirm, I'll update my PR with that documentation | 15:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add apache rewrite rule for ruby gems https://review.openstack.org/452253 | 15:59 |
| *** armax has quit IRC | 15:59 | |
| *** rbuzatu_ has joined #openstack-infra | 16:03 | |
| *** rbuzatu has quit IRC | 16:04 | |
| clarkb | pabelanger: https://rubygems.org/gems/a/versions/0.1.1 is the type of corner case I need to reason about | 16:04 |
| clarkb | pabelanger: or do they get mirrored differently? | 16:05 |
| clarkb | your example url is flat | 16:05 |
| *** arxcruz|pto has joined #openstack-infra | 16:05 | |
| pabelanger | clarkb: yes, they are just mirrored into a flat dir | 16:05 |
| pabelanger | give me a sec and I will get the hash for that | 16:06 |
| clarkb | pabelanger: is that /gems/a-0.1.1.gem on disk? | 16:06 |
| *** EricGonczer_ has joined #openstack-infra | 16:06 | |
| *** gcb has quit IRC | 16:07 | |
| clarkb | (it fails the rewrite rules if hosted at the path I pasted) | 16:07 |
| pabelanger | clarkb: /gems/a/a-/a-0.1.1.gem | 16:07 |
| clarkb | er rewrite cond so no rule executed | 16:07 |
| clarkb | ok thanks | 16:07 |
| *** EricGonc_ has joined #openstack-infra | 16:09 | |
| *** gcb has joined #openstack-infra | 16:09 | |
| *** rbuzatu_ has quit IRC | 16:11 | |
| *** EricGonczer_ has quit IRC | 16:11 | |
| clarkb | pabelanger: I posted a question (was easier on the review) | 16:12 |
| pabelanger | clarkb: let me test | 16:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add apache rewrite rule for ruby gems https://review.openstack.org/452253 | 16:14 |
| pabelanger | clarkb: works | 16:14 |
| pabelanger | also added 2nd 1/2 of specfiles | 16:14 |
| *** jpich has quit IRC | 16:16 | |
| clarkb | pabelanger: why is the specfile pathing different than gems? | 16:18 |
| *** derekh has quit IRC | 16:18 | |
| clarkb | if they were the same you could do something like gem/(gems|quick)/commonregexinghere | 16:19 |
| pabelanger | clarkb: that is how rubymirror does it | 16:19 |
| clarkb | pabelanger: but didn't you add it to rubymirror? | 16:19 |
| pabelanger | 1 sec | 16:19 |
| pabelanger | https://github.com/rubygems/rubygems-mirror/pull/47 | 16:19 |
| pabelanger | is my pull request | 16:19 |
| pabelanger | you can see how I did it | 16:19 |
| pabelanger | basically, I did easy mode and just hashed the existing directory structure | 16:19 |
| pabelanger | because quick seems to have multiple formats | 16:20 |
| pabelanger | quick/Marshal.#{Gem.marshal_version} | 16:20 |
| clarkb | so quick/Marshal is a fixed part of the path? | 16:21 |
| clarkb | and then you only hash the g_spec? | 16:21 |
| pabelanger | yes | 16:21 |
| pabelanger | the client will pass that | 16:21 |
| pabelanger | correct | 16:21 |
| *** markus_z has quit IRC | 16:22 | |
| clarkb | I see | 16:22 |
| pabelanger | however, I am not sure what is using marshal things | 16:22 |
| pabelanger | but was minimal to mirror | 16:22 |
| clarkb | and there are different marshall versions so you can't just fix that dir | 16:22 |
| clarkb | in other news did you hear twitter is a main communication tool for infra? | 16:23 |
| clarkb | pabelanger: biggest apache cache dir I see is 431M btw | 16:24 |
| clarkb | pabelanger: if that ends up being reasonably steady state at half a gig I think we are good to go | 16:24 |
| pabelanger | clarkb: I am not sure, maybe EmilienM or mwhahaha know ^ | 16:24 |
| clarkb | we can also reduce cache retention to 8 hours (from 24 hours) | 16:24 |
| clarkb | if we have trouble | 16:24 |
| pabelanger | clarkb: oh, where is that dir? | 16:24 |
| clarkb | pabelanger: /opt/apache_cache | 16:25 |
| pabelanger | server? | 16:25 |
| clarkb | pabelanger: on each of the mirror hosts | 16:25 |
| clarkb | oh thatspecific one? | 16:25 |
| clarkb | osic | 16:25 |
| clarkb | sorry its early morning | 16:25 |
| pabelanger | k | 16:25 |
| sdague | dmsimard: what was happening with reverify is people were getting -1s and just reverifying them back into the gate, even though the patch could no longer pass (a requirements issue, or something else merged in front of it) | 16:25 |
| pabelanger | checking cacti | 16:25 |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4941&rra_id=all | 16:25 |
| pabelanger | osic already choppy | 16:26 |
| pabelanger | must be apache logs | 16:26 |
| clarkb | ya I think its logs | 16:26 |
| pabelanger | clarkb: back to gems, I can hard code marshal path | 16:26 |
| pabelanger | to avoid some regex, if that was what you are asking | 16:26 |
| clarkb | pabelanger: its fine I just wasn't understanding why it was special | 16:26 |
| mwhahaha | pabelanger: is the question what uses the Marhsal gem? i'm not sure | 16:26 |
| clarkb | and the problem is its an entirely new flat space under each marshall thing | 16:26 |
| pabelanger | mwhahaha: ya, or if you need gemspec files for anything | 16:26 |
| mwhahaha | we shouldn't need gemspecs i don't think | 16:27 |
| *** esberglu has quit IRC | 16:27 | |
| pabelanger | clarkb: we could drop the apache rule for gemspec things first | 16:27 |
| pabelanger | and if not needed, I can add flag to disable mirroring that | 16:27 |
| mwhahaha | we just use gems and i don't see a reference to marshal in our stuff | 16:27 |
| mwhahaha | of course we're just using bundler so who knows what it's pulling down | 16:27 |
| * mwhahaha would have to look | 16:27 | |
| *** esberglu has joined #openstack-infra | 16:27 | |
| clarkb | I don't really understand what the purpose of those is? is it so you can install the deps of a thing without isntalling the thing itself? | 16:28 |
| clarkb | or to just get version and dependency info? | 16:28 |
| *** lucasagomes is now known as lucas-afk | 16:28 | |
| mwhahaha | probably | 16:29 |
| *** rbuzatu has joined #openstack-infra | 16:29 | |
| mwhahaha | oh it's for use in not duplicating your deps | 16:29 |
| mwhahaha | http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/ | 16:29 |
| clarkb | mwhahaha: right but normally it lives in the repo/gem itself iirc | 16:30 |
| clarkb | so wondering what the purpose of histing them under /quick/Marshal.x.y/ is | 16:30 |
| mwhahaha | it can be a remote ref but i don't think we need it for our needs | 16:30 |
| SamYaple | what is the name of the job that happens in post that creates the tarballs in tarballs.opentack.org? openstack-server-publish-jobs? | 16:31 |
| mwhahaha | since non of our Gemfiles have a gemspec reference (at least our puppet modules) | 16:31 |
| clarkb | SamYaple: thats a group of jobs that includes the tarball/sdist build docs builds iirc | 16:32 |
| *** esberglu has quit IRC | 16:32 | |
| SamYaple | or is it part of python-jobs | 16:32 |
| SamYaple | ok | 16:33 |
| clarkb | SamYaple: if you look in python-jobs.yaml at the bottom are the job-groups. The tarball job in python-jobs is a job that updates"branch" tarballs on every merged commit | 16:34 |
| SamYaple | This is the tarball publishing job https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml#L400-L419 ? | 16:34 |
| clarkb | SamYaple: so you get a master tarball and an ocata tarball and so on updated every merge | 16:34 |
| SamYaple | yea thats what im lloking for | 16:34 |
| SamYaple | looking* | 16:34 |
| SamYaple | cool thanks | 16:34 |
| clarkb | SamYaple: the non branch tarball is used for releases and is part of openstack-server-release-jobs | 16:34 |
| clarkb | if you want the per commit branch update then python jobs is what you want looks like | 16:35 |
| fungi | client release jobs also does tag-based tarballs but also pushes to pypi i think | 16:35 |
| *** nadya has quit IRC | 16:35 | |
| *** electrofelix has quit IRC | 16:35 | |
| *** apevec has quit IRC | 16:36 | |
| SamYaple | using loci-keystone for an example, it doesn't have python anything going on with it (its Dockerfiles at the moment) so publishing to pypi seems wrong | 16:36 |
| *** ZZelle has joined #openstack-infra | 16:36 | |
| SamYaple | i need to get openstack/loci into a tarball on a per-commit basis (for each branch). it seems weird to use python-jobs if there are no python bits | 16:37 |
| SamYaple | so its a bit of a wierd situation, no? | 16:37 |
| pabelanger | well, I would hold off on publishing images to tarballs.o.o for now, | 16:37 |
| SamYaple | im not publishing images | 16:37 |
| pabelanger | depending on how big they are, we are down to our last 50GB | 16:37 |
| pabelanger | and I know images take up more stack then python tarballs | 16:38 |
| pabelanger | k | 16:38 |
| fungi | SamYaple: you can just put the tarball job itself in there directly, but you likely don't want our normal tarbal jobs since they expect to be able to tun `tox -e venv python setup.py sdist` (and bdist_wheel) | 16:38 |
| SamYaple | pabelanger: the plan is to push 250MB (per branch, which only exists as one branch right now) in total. but that is just to prevent fetching from outside-infra for gate jobs | 16:39 |
| fungi | SamYaple: if you need to make tarballs that are not python sdist tarballs, there might be a job definition available already which does what you need but i don't know so you'll probably need to hunt (or write one) | 16:39 |
| SamYaple | fungi: so 'scp' publisher? | 16:39 |
| fungi | SamYaple: the current {name}-branch-tarball job-template is defined here http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n421 | 16:41 |
| *** nadya has joined #openstack-infra | 16:41 | |
| *** marst has quit IRC | 16:41 | |
| fungi | it invokes a builder here: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n724 | 16:42 |
| *** chem is now known as chem_gone | 16:42 | |
| *** Apoorva has joined #openstack-infra | 16:42 | |
| *** nadya has quit IRC | 16:42 | |
| fungi | which runs these scripts: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-tarball.sh http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-wheel.sh | 16:43 |
| SamYaple | i see. | 16:43 |
| *** ralonsoh has quit IRC | 16:44 | |
| *** marst has joined #openstack-infra | 16:44 | |
| *** tosky has quit IRC | 16:45 | |
| fungi | SamYaple: for comparison, here's a definition for doing something similar with nodejs/npm tarballs: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/javascript.yaml#n108 | 16:45 |
| fungi | in the nodejs case, looks like it calls `npm pack` to create a tarball | 16:46 |
| SamYaple | fungi: yea thats what im writing up. using the 'scp' publisher | 16:46 |
| fungi | i'm trying to think of a publishing ecosystem where we just call tar directly to create tarballs | 16:46 |
| fungi | well, the tarball creation doesn't happen in the publisher | 16:46 |
| SamYaple | yea +2 to that | 16:46 |
| SamYaple | fungi: i know, but i can create said tarball | 16:47 |
| *** kgiusti has quit IRC | 16:47 | |
| fungi | the publisher is just what sticks the resulting file(s) somewhere public (logs.o.o, tarballs.o.o, et cetera) | 16:47 |
| SamYaple | yea i get that | 16:47 |
| SamYaple | im good with that | 16:47 |
| SamYaple | this is mostly just about removing all external-to-infra deps | 16:48 |
| pabelanger | is there a screen on static.o.o for fsck? | 16:50 |
| clarkb | pabelanger: apparently there is, was just mentioned in incident | 16:50 |
| *** yamamoto has joined #openstack-infra | 16:51 | |
| *** florianf has quit IRC | 16:51 | |
| fungi | yeah, root screen session it's running under | 16:51 |
| jeblair | fungi, clarkb: i went to lists.o.o to start doing the prep work we said we would do before the maintenance, and i'm getting this: http://paste.openstack.org/show/605094/ | 16:51 |
| fungi | pabelanger: clarkb: there are apparently two root screen sessions (or were when i first logged in), but the slightly newer of the two is the one i think you want | 16:52 |
| clarkb | jeblair: hrm, I'm guessing the -dev packages are relatively safe to just force downgrade bceause they are headers | 16:53 |
| fungi | jeblair: that looks like an unattended upgrade went screwy on a libc update, and then the packages it needs to satisfy that dependency are no longer available. probably solvable with a forced downgrade or upgrade | 16:54 |
| fungi | and yeah, libc6-dev isn't going to be used for much. probably there because we install build-essential or something | 16:54 |
| jeblair | hrm. do we think that's safe to do outside of the maint window? | 16:55 |
| jeblair | this is needed in order to do the apt-get autoremove which we expect to save something like 1 hour of time during the maint. | 16:55 |
| fungi | upgrading libc6-dev looks like it should solve that, based on the errors | 16:55 |
| *** ramishra has joined #openstack-infra | 16:56 | |
| fungi | i'm taking a look to see if i can figure out what's happened there | 16:56 |
| openstackgerrit | Sorin Sbârnea proposed openstack-infra/shade master: Fixed stack_status.split() exception https://review.openstack.org/452260 | 16:57 |
| *** yamamoto has quit IRC | 16:57 | |
| jeblair | fungi: i just "dpkg -i libc6-dev..." and i think maybe it's happy now | 16:58 |
| clarkb | jeblair: I'd expect it to be safe (just considering the primary consumer of that is compilations and we won't be doing any of those) | 16:58 |
| clarkb | jeblair: and whatever linkage is already in place happens despite the headers and its working now | 16:58 |
| fungi | yeah, i think there must have been a failed upgrade a while back. apt-cache show libc6-dev says 2.15-0ubuntu10.16 was locally unpacked and marked for installation, but that the current version in the package index for the archive is 2.15-0ubuntu10.18 | 16:58 |
| fungi | so `apt-get install libc6-dev` would have probably fixed it too | 16:59 |
| *** gyee has joined #openstack-infra | 16:59 | |
| jeblair | fungi: no, apt-get was throwing a fit | 16:59 |
| jeblair | all similar commands resulted in the pasted error | 16:59 |
| fungi | when explicitly telling it to install latest libc6-dev? | 16:59 |
| fungi | that's an unfortunate catch-22 | 17:00 |
| jeblair | 1108 apt-get install libc6-dev | 17:00 |
| jeblair | yep, tried that ^ | 17:00 |
| fungi | guh | 17:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add test to validate multi _ heat stack_status https://review.openstack.org/452261 | 17:00 |
| jeblair | so i did: | 17:00 |
| jeblair | 1111 apt-get install -d libc6-dev | 17:00 |
| jeblair | 1120 dpkg -i libc6-dev_2.15-0ubuntu10.18_amd64.deb | 17:00 |
| jeblair | (with some cd and ls's inbetween :) | 17:01 |
| fungi | oh well, it's showing 2.15-0ubuntu10.18 installed now, so good enough | 17:01 |
| *** signed8b_ is now known as signed8bit_Zzz | 17:01 | |
| SpamapS | if I wanted to show somebody how many jobs the bluebox node provider is running... is there a convenient place? maybe in grafana? | 17:01 |
| mordred | SpamapS: yah - grafana - lemme get you link | 17:01 |
| jeblair | fungi: shall i 'apt-get autoremove' now? | 17:01 |
| SpamapS | http://grafana.openstack.org/dashboard/db/nodepool-blue-box ? | 17:01 |
| mordred | yu[ | 17:01 |
| pabelanger | yup | 17:01 |
| SpamapS | I just typed 'graf' and that came up in my history | 17:02 |
| SpamapS | so clearly I've been there before :-P | 17:02 |
| jeblair | fungi: oh, hrm, autoremove doesn't want to do anything. | 17:02 |
| fungi | SpamapS: the test nodes graph | 17:02 |
| SpamapS | http://grafana.openstack.org/dashboard/db/nodepool-blue-box?panelId=17&fullscreen <-- this one? | 17:02 |
| mordred | SpamapS: yah. Test Nodes correspond to jobs essential | 17:02 |
| mordred | yup | 17:02 |
| SpamapS | K. Just keeping the bean counters happy. | 17:02 |
| jeblair | i count 15 beans right now | 17:03 |
| mordred | beans make good coffee | 17:03 |
| jeblair | fungi: so, erm, what should we do to get rid of all those kernels? | 17:03 |
| fungi | jeblair: so looks like we have 43 kernel packages installed at the moment | 17:03 |
| AJaeger | speaking about recheck and reverify - should we remove reverify finally? those that remember it, think it only goes through gate... | 17:04 |
| fungi | jeblair: latest installed is 3.2.0-80 but uname -a says we're on 3.2.0-77 | 17:04 |
| pabelanger | AJaeger: I think we should, maybe a ML post about the removal? | 17:04 |
| pabelanger | not sure how others feel | 17:04 |
| fungi | jeblair: i'd probably just apt-get remove all the kernels except the one we're running | 17:04 |
| *** flepied has quit IRC | 17:05 | |
| AJaeger | pabelanger: I agree with the ML... | 17:05 |
| clarkb | fungi: jeblair apt-get autoremove is what I do locally | 17:05 |
| clarkb | but maybe I trust apt too much? | 17:05 |
| jeblair | clarkb: 0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded. | 17:06 |
| fungi | yeah, autoremove isn't seeing these as auto-installed and no longer depended on by any installed package | 17:07 |
| clarkb | huh | 17:07 |
| fungi | so i'd just sed the versions out into a package name list, filter out the version we're presently booting, and feed that to an apt-get purge | 17:08 |
| fungi | maybe also filter out the latest version if you want | 17:08 |
| fungi | so we end up with only the currently installed kernel and the latest kernel still present | 17:09 |
| fungi | er, currently booted and latest i mean | 17:09 |
| *** yonglihe has quit IRC | 17:09 | |
| *** armax has joined #openstack-infra | 17:10 | |
| *** eroux has quit IRC | 17:10 | |
| jeblair | okay, that's running | 17:11 |
| *** sambetts is now known as sambetts|afk | 17:12 | |
| fungi | it'll still take a while since it'll likely try to regenerate initrds between each package removal, but at least we can do this without services being down | 17:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove 'reverify' https://review.openstack.org/452264 | 17:13 |
| AJaeger | More than 30 months ago we aliased reverify and recheck - time to remove reverify IMHO ^ | 17:14 |
| jeblair | fungi: yep. it's chugging along. | 17:15 |
| jeblair | AJaeger: do people still use it? | 17:15 |
| AJaeger | jeblair: yeah, I sometimes see people using it. | 17:15 |
| jeblair | i agree 30 months is a sufficient deprecation period, but only if we've given notice that it's actually a deprecation period. | 17:15 |
| *** signed8bit_Zzz is now known as signed8b_ | 17:16 | |
| jeblair | if not, maybe we want to send an email and wait another week or so. :) | 17:16 |
| *** armax has quit IRC | 17:16 | |
| AJaeger | I can do that... | 17:16 |
| *** nadya has joined #openstack-infra | 17:16 | |
| * AJaeger checks archives... | 17:16 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Use libyaml if possible https://review.openstack.org/451113 | 17:16 |
| *** nadya has quit IRC | 17:17 | |
| *** shardy has quit IRC | 17:17 | |
| *** nadya has joined #openstack-infra | 17:18 | |
| *** rbuzatu_ has joined #openstack-infra | 17:18 | |
| AJaeger | mmh, seems we did not announce it - ok, will do now... | 17:18 |
| *** yamahata has joined #openstack-infra | 17:19 | |
| *** ramishra has quit IRC | 17:20 | |
| *** jcoufal_ has joined #openstack-infra | 17:21 | |
| *** rbuzatu has quit IRC | 17:22 | |
| *** jcoufal has quit IRC | 17:24 | |
| _ari_ | Is there an issue with the Jenkins gerrit I got this for my review - https://review.openstack.org/#/c/451871/ | 17:25 |
| mnaser | topic _ari_ | 17:25 |
| *** neiljerram has quit IRC | 17:26 | |
| *** smarcet has joined #openstack-infra | 17:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove 'reverify' https://review.openstack.org/452264 | 17:30 |
| AJaeger | mail sent and referenced in my review ^ | 17:30 |
| AJaeger | wow, already a +1000 by sdague ;) | 17:33 |
| *** rcarrillocruz has quit IRC | 17:33 | |
| * JayF may have been putting 'recheck reverify' in everything he wanted to check because he forgot which one worked | 17:33 | |
| AJaeger | JayF: 'recheck reverify' works since 'recheck X' works. But 'X reverify' or 'X recheck' does not work. | 17:35 |
| AJaeger | JayF: So, 'no recheck' will do the right thing ;) | 17:35 |
| *** pvaneck has joined #openstack-infra | 17:35 | |
| *** isviridov_away has quit IRC | 17:36 | |
| *** vaidy has quit IRC | 17:36 | |
| JayF | I always did them on separate lines, but I'll remember now | 17:36 |
| _ari_ | mnaser: thanks | 17:36 |
| AJaeger | JayF: ah, with separate lines it does work | 17:36 |
| _ari_ | mnaser: is there a way to retrigger my review after it is fixed? | 17:36 |
| *** wolverineav has quit IRC | 17:36 | |
| mnaser | _ari_ just type "recheck" in a review | 17:36 |
| mnaser | as a comment | 17:36 |
| mnaser | once all of this clears out | 17:36 |
| _ari_ | mnaser: thanks so much! | 17:36 |
| mnaser | np :> | 17:36 |
| SamYaple | i purpose we change it to 'revericheck' | 17:37 |
| SamYaple | so no more confusion | 17:37 |
| AJaeger | _ari_: https://docs.openstack.org/infra/manual/developers.html#automated-testing item 4 | 17:37 |
| *** wolverineav has joined #openstack-infra | 17:37 | |
| mnaser | ^ probably a much better reference to know working with the CI stuff | 17:38 |
| *** mdrabe has quit IRC | 17:38 | |
| _ari_ | AJaeger: thanks I should have read lower down in the developers guide my bad | 17:38 |
| _ari_ | I will keep that in mind in the future | 17:38 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/requestsexceptions master: Updated from global requirements https://review.openstack.org/451836 | 17:38 |
| *** jerryz has joined #openstack-infra | 17:39 | |
| *** nadya has quit IRC | 17:39 | |
| *** smarcet has quit IRC | 17:39 | |
| fungi | SamYaple: my vote is on recheckify | 17:39 |
| clarkb | jeblair: re lists.o.o it must be happy now? | 17:39 |
| jeblair | clarkb: yeah, it's still (slowly) removing kernels | 17:40 |
| fungi | clarkb: "happily" whittled its way down from 43 kernels to 34 so far anyway | 17:40 |
| clarkb | and infra-root if waiting for fsck paint to dry reviewing https://review.openstack.org/#/c/449835/ would be helpful to keep pushing on gerrit upgrade. Don't need to approve it, I can do that (though likely not today as its lists upgrade and logs fsck day) | 17:40 |
| *** wolverineav has quit IRC | 17:42 | |
| *** felipemonteiro has joined #openstack-infra | 17:42 | |
| clarkb | AJaeger: andreykurilin I have +2'd the rally features.yaml change | 17:43 |
| andreykurilin | clarkb: thanks! | 17:43 |
| *** smarcet has joined #openstack-infra | 17:45 | |
| *** tosky has joined #openstack-infra | 17:48 | |
| sdague | fungi: what about "big bucks, no whammies!" | 17:48 |
| *** MasterOfBugs has joined #openstack-infra | 17:48 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Add install-guide-jobs for murano installation guide. https://review.openstack.org/452272 | 17:49 |
| mnaser | i was just curious at how far the fsck has gotten if that's okay? | 17:49 |
| *** e0ne has joined #openstack-infra | 17:50 | |
| *** armax has joined #openstack-infra | 17:51 | |
| jeblair | mnaser: 89.4% | 17:51 |
| mnaser | jeblair cool, thanks! | 17:51 |
| fungi | sdague: i think we'd be pressing our luck with that one | 17:52 |
| * fungi lands on a whammy with that pun | 17:53 | |
| sdague | heh | 17:54 |
| SamYaple | Does this mean i have to watch vhs recordings to learn how to make the gate pass? | 17:54 |
| sdague | SamYaple: yes | 17:54 |
| sdague | or even better, betamax | 17:54 |
| SamYaple | that _is_ even better | 17:54 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Update ovb-nonha-oooq to use local registry mirror https://review.openstack.org/452276 | 17:54 |
| fungi | SamYaple: bonus if you can work out a way for us to re-theme our ci system based on "bowling for dollars" by next april 1 | 17:55 |
| AJaeger | also, if somebody wants to review: cmurphy has a series of puppet changes where I'd like some dib/puppet experts to look at, see https://review.openstack.org/#/q/topic:unpuppet-zuul-workers | 17:55 |
| SamYaple | fungi: i cant do that, but i can probably make it autoplay Bowling For Soup | 17:56 |
| AJaeger | EmilienM: could you review https://review.openstack.org/#/c/451115/1/nodepool/nodepool.yaml as well, please? Removes tripleo-test-cloud-rh2 | 17:56 |
| smarcet | fungi: afternoon, we will facing a trove access shortage on 13/14 April, due RS DFW dc maintenance, the only way to avoid that, is to move all production database to a new trove instance, we are working on that, but we would need to change some hieras variables for openstackid to point to the new host | 17:59 |
| smarcet | fungi: should i send those new hieras values by email once that we do the move ? | 17:59 |
| *** baoli has quit IRC | 18:00 | |
| *** baoli has joined #openstack-infra | 18:01 | |
| fungi | smarcet: yes, you can send them to me and i can update them. or really any infra-root who's around at the time you're ready to switch databases | 18:01 |
| smarcet | fungi: roger that thx :) | 18:02 |
| clarkb | maybe send them once the new db name is known? | 18:02 |
| clarkb | then we can stage it | 18:02 |
| fungi | also doable, yes | 18:02 |
| clarkb | also is rackspace replacing all the hardware? | 18:02 |
| fungi | certainly seems that way | 18:02 |
| *** jcoufal has joined #openstack-infra | 18:02 | |
| *** Apoorva_ has joined #openstack-infra | 18:03 | |
| openstackgerrit | Merged openstack-infra/requestsexceptions master: Updated from global requirements https://review.openstack.org/451836 | 18:03 |
| clarkb | I'm gonna go grab late breakfast early lunch and be ready for lists things in a couple hours (will be back before that for prep work) | 18:03 |
| *** jcoufal_ has quit IRC | 18:04 | |
| *** Apoorva has quit IRC | 18:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: [rally] Add support for custom branching https://review.openstack.org/450337 | 18:08 |
| openstackgerrit | azat proposed openstack-infra/jenkins-job-builder master: scm: bypass git.submodule.parentCredentials https://review.openstack.org/452280 | 18:09 |
| fungi | here's a fun one. did we merge any firewall rule updates for planet.o.o? | 18:10 |
| fungi | the site's down, and iptables -L doesn't show any http open | 18:10 |
| *** baoli has quit IRC | 18:10 | |
| clarkb | we did change puppet to allow an 01 server | 18:11 |
| *** baoli has joined #openstack-infra | 18:11 | |
| fungi | looks like something has caused us to update the fw rules to start blocking public tcp services there | 18:11 |
| kfox1111 | hows logs.oo going? | 18:11 |
| fungi | kfox1111: fsck at 89.8% complete | 18:11 |
| kfox1111 | cool. thanks. | 18:11 |
| *** baoli has quit IRC | 18:12 | |
| *** baoli has joined #openstack-infra | 18:13 | |
| fungi | clarkb: i see the issue. patch coming | 18:13 |
| SamYaple | fungi: we should just use btrfs. i heard it never has any problems | 18:13 |
| *** rbuzatu has joined #openstack-infra | 18:14 | |
| EmilienM | AJaeger: sure thing | 18:14 |
| EmilienM | bnemec: please review it too if you can https://review.openstack.org/#/c/451115/ | 18:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Use job-template check-requirements in masakari projects https://review.openstack.org/451668 | 18:15 |
| kfox1111 | SamYaple: ;) | 18:15 |
| kfox1111 | SamYaple: my brother had a btrfs eat itself. though this was over a year ago. probably almost 2 by now... | 18:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Correct global site manifest regex for planet.o.o https://review.openstack.org/452281 | 18:15 |
| fungi | clarkb: mnaser: ianw: ^ | 18:15 |
| fungi | mnaser: thanks for reporting! | 18:15 |
| mnaser | np :> | 18:16 |
| *** rbuzatu_ has quit IRC | 18:17 | |
| SamYaple | kfox1111: it just happened to me last year. :( i stopped using it, though i wont spread FUD about it | 18:17 |
| fungi | unfortunately planet's probably been offline for some ~44 hours since shortly after the earlier regex merged | 18:17 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Add experimental murano functional test gate. https://review.openstack.org/451996 | 18:18 |
| AJaeger | The docs team just got the second bug report about failing logs ;( | 18:18 |
| fungi | kfox1111: SamYaple: i have a similar story, though the last time i tried using it there still wasn't a fsck for it | 18:18 |
| kfox1111 | SamYaple: yeah. I really want btrfs to succedd. I know far to many people that cram zfs into linux. :/ | 18:18 |
| AJaeger | fungi, do you want to send an email to openstack-dev or shall I? What's the estimate? | 18:18 |
| sdague | fungi: once everything is said and done, it would be interesting to figure out why the log analyzer is so memory heavy | 18:18 |
| kfox1111 | I'm using btrfs on some of my docker hosts though, with good results. | 18:18 |
| sdague | because it was specifically written as a stream processor, so shouldn't be | 18:18 |
| *** baoli has quit IRC | 18:19 | |
| fungi | sdague: agreed, though my guess is that's just the python runtime per request | 18:19 |
| sdague | fungi: yeh, I wonder if that's one of those things were we could be better with a uwsgi backend or the like | 18:19 |
| sdague | but, that's a later conversation | 18:20 |
| fungi | sdague: normally it's not an issue, but when the system is already under tons of memory pressure because fsck.ext4 wants nearly 100% of the allocated ram for this server, every little bit helps | 18:20 |
| sdague | yep, no worries | 18:20 |
| SamYaple | fungi: its a CoW filesystem, there is no fsck! | 18:20 |
| fungi | AJaeger: if you want to send out a note, that's much apprediated! | 18:20 |
| sdague | fungi: are swift logs no longer a thing? | 18:20 |
| sdague | we could also probably simplify the code if that's the case | 18:20 |
| fungi | sdague: we were never able to finish working out how to reimplement a browsable file hierarchy on top of swift containers | 18:20 |
| *** tesseract has quit IRC | 18:21 | |
| sdague | ah | 18:21 |
| *** rbuzatu has quit IRC | 18:21 | |
| *** rbuzatu has joined #openstack-infra | 18:21 | |
| fungi | it's not a use case they design for, and so we end up needing to basically create our own posix-ish file allocation table as a separate database | 18:21 |
| sdague | yeh, that just gets ugly pretty quick | 18:21 |
| AJaeger | fungi, will do... | 18:22 |
| fungi | thanks, AJaeger! | 18:22 |
| notmyname | fungi: IIRC it wasn't the browsable file hierarchy (that's possible). it's the metadata on mid-level elements of the generated hierarchy | 18:22 |
| sdague | ok, so I need to go to the shop to get a thing the car looked into, maybe I'll take it as a good time to strip a bunch of that out and propose it to just simplify the code | 18:22 |
| sdague | which might make thinking about the memory thing easier | 18:22 |
| fungi | notmyname: right, that was one of the bigger parts. things like updating the last modified time on a directory node when a file under it has been altered/created | 18:22 |
| notmyname | fungi: eg mx(last modified time) of any object with a given prefix | 18:22 |
| fungi | notmyname: and being able to sort by that as well | 18:23 |
| fungi | we were basically reimplementing apache mod_auto_index on top of swift | 18:24 |
| notmyname | fungi: I'd quibble about what was being reimplemented. ;-) we sorta have it. you were wanting FS semantics on object storage | 18:25 |
| fungi | notmyname: right | 18:25 |
| fungi | well, we were wanting a filesystem, but we tried to put it on top of swift because that was an elastic storage option | 18:25 |
| mnaser | doesnt swift already give you index for pbulic containers | 18:26 |
| mnaser | public* | 18:26 |
| *** jascott1_ has joined #openstack-infra | 18:26 | |
| notmyname | mnaser: yeah, it can. not just all the generated metadata (like last mtime) on generated entities. nor does it allow arbitrary sorting | 18:26 |
| fungi | mnaser: we were also somewhat hamstrung by the way rackspace cdn's their public swift access, and by wanting to do some postprocessing between the content and the client | 18:27 |
| mnaser | the only 2 annoyances i can imagine is: writing something that loops and uploads files (a mass uploader rather than an rsync), and clean up | 18:27 |
| notmyname | fungi: might be more options now you've got infra cloud. or if you use another provider | 18:27 |
| mnaser | i know this is a lot of work but throwing ideas | 18:27 |
| fungi | so in the end, we were turning swift containers into a remote filesystem for a separate webserver, which really wasn't playing to its strong suits | 18:27 |
| mnaser | nginx has a way to write simple modules in $language_i_cant_remember | 18:28 |
| mnaser | lua | 18:28 |
| mnaser | not that i know anything about it but its interesting to know, i know of some companies that use this at big scale - https://github.com/openresty/lua-nginx-module | 18:28 |
| fungi | well, at this point we have a different elastic storage solution which has a more traditional unix filesystem and are already using that for some other content, so have ideas for how to move our log data to that | 18:29 |
| notmyname | fungi: what are you using? | 18:29 |
| fungi | afs | 18:29 |
| mnaser | (but then if you have something that's working without problems, no need to fix what isnt broken) | 18:29 |
| notmyname | gotta run (technically, I'm on vacation today). happy to pick this up again next week | 18:30 |
| mordred | well - we have a non-zero amount of work in front of us no matter which thing we do | 18:30 |
| fungi | notmyname: sure, any time! enjoy your vacation | 18:30 |
| mordred | and clearly the _current_ thing doesn't make anybody happy | 18:30 |
| mnaser | mordred i think logs.o.o was ok chugging along if it wasn't for an issue that you dont control | 18:31 |
| mnaser | the other options could fail just as much if another issue which you dont control happens too *shrug* | 18:31 |
| mordred | mnaser: yup. totally agree. I think moving forward it's about understanding what our failure mitigation options are | 18:31 |
| fungi | mnaser: well, the current backend is also not scalable. the provider limits the size of the volumes we can create and their choice of hypervisor significantly limits the number of volumes we can attach to a single server | 18:31 |
| fungi | and we maxed those out long ago | 18:32 |
| mordred | unfortunately, we're maxed on cinder volumes on the server in rax - so although we got warned that the volume was going to go down, we don't have the headroom to create a new one and migrate the data :( | 18:32 |
| mordred | fungi: ++ | 18:32 |
| mnaser | ouch | 18:32 |
| mnaser | i guess from an operator perspective figuring out how to containing failure to not be as catastophic would be the best next step | 18:32 |
| mordred | ++ | 18:33 |
| fungi | we're going to need to come up with a sharding model anyway, not only for scaling the quantity of data we store but also scaling our ability to age out old data in a timely manner | 18:33 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Explain OpenStack CI Status https://review.openstack.org/452285 | 18:33 |
| *** Sihan has joined #openstack-infra | 18:33 | |
| mnaser | i havent had great experience but running glusterfs on top of a few instances could deliver keep much of the same semantics | 18:33 |
| AJaeger | fungi, what do you think of this ^ | 18:33 |
| Sihan | Hi Does any know how to deploy the zuul status dashboard? | 18:33 |
| AJaeger | or is there a better place to explain that? | 18:33 |
| Sihan | if there is a doc, that would be helpful | 18:33 |
| mordred | mnaser: yah - we've talked some about both ceph and gluster. also about just sharding data across multiple volumes in AFS | 18:33 |
| mordred | each of those options is just involved enough that full exploration will take a bit of time | 18:34 |
| clarkb | ok I am mostly back ish | 18:34 |
| mnaser | cephfs could be an interesting option too, afs i dont know much but i think the advantage is big operational experience so a lot of existingknowledge in the infra team | 18:34 |
| fungi | AJaeger: i like the prominent placement, either there or system-config docs would make sense | 18:34 |
| mnaser | AJaeger i think maybe mentioning that if you get POST_FAILURE it means there is likely an issue and to check those sources as well | 18:34 |
| *** jcoufal_ has joined #openstack-infra | 18:34 | |
| mnaser | a lot of people dont seem to associate with POST_FAILURE => CI issue rather than "oops did my build break" | 18:34 |
| mordred | mnaser: yah - well, and decoupling filesystem presentation from backend storage | 18:34 |
| pabelanger | clarkb: how do you want to handle the PR on https://github.com/rubygems/rubygems-mirror/pull/47 ? | 18:35 |
| fungi | Sihan: normally zuul provides it directly (though i think it depends on some javascript bits, which are explained in the install docs) | 18:35 |
| pabelanger | I'm going to as in the ruby channel who is maintaining it | 18:35 |
| mnaser | mordred: cant you just, like, put all that data on the cloud? isnt that how it works | 18:35 |
| mordred | mnaser: you have to wave your hands wildly when you say that | 18:36 |
| fungi | Sihan: have you tried opening firewall access to port 80 for your zuul server and then just browsing to it via http? | 18:36 |
| AJaeger | mnaser: I'll add that... | 18:36 |
| AJaeger | fungi: both? | 18:36 |
| mnaser | and maybe i'm pushing but im not sure how easy this is to do | 18:36 |
| mnaser | but perhaps the message that gets posted by jenkins | 18:36 |
| fungi | AJaeger: maybe put it in one and refer to that section from the other? | 18:37 |
| mnaser | if it can say something about POST_FAILURE being a CI issue | 18:37 |
| *** jcoufal has quit IRC | 18:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Explain OpenStack CI Status https://review.openstack.org/452285 | 18:38 |
| AJaeger | mnaser, fungi: updated ^ | 18:38 |
| *** vaidy has joined #openstack-infra | 18:38 | |
| fungi | jeblair: looks like the kernel cleanup on lists.o.o just completed? | 18:39 |
| *** isviridov_away has joined #openstack-infra | 18:39 | |
| clarkb | worth noting that sdague was the user we were considering when we switching things to swift and it didn't work :) | 18:39 |
| clarkb | so whatever we decide on hsi input would be valuable too | 18:39 |
| *** baoli has joined #openstack-infra | 18:40 | |
| *** john-davidge has quit IRC | 18:40 | |
| *** baoli has quit IRC | 18:40 | |
| *** baoli has joined #openstack-infra | 18:40 | |
| *** e0ne has quit IRC | 18:40 | |
| jeblair | fungi: yep, i'm removing old kernel header pkgs now for the heck of it | 18:40 |
| *** e0ne has joined #openstack-infra | 18:41 | |
| *** e0ne has quit IRC | 18:43 | |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: add coverage-jobs to congress https://review.openstack.org/452016 | 18:43 |
| jeblair | clarkb, fungi: all done. there are some packages that still want to be upgraded, but they include mailman, so that'll be in step 1 of the maint. | 18:44 |
| clarkb | jeblair: sounds good | 18:44 |
| fungi | jeblair: sounds great. thanks! | 18:44 |
| Sihan | I didn't see 80 is listened by any process | 18:44 |
| clarkb | fungi: I have approved planet fix | 18:45 |
| *** e0ne has joined #openstack-infra | 18:45 | |
| *** rbuzatu_ has joined #openstack-infra | 18:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Link to Infra Manual for CI problems https://review.openstack.org/452287 | 18:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config master: Rework intro https://review.openstack.org/452289 | 18:50 |
| *** rbuzatu has quit IRC | 18:50 | |
| AJaeger | fungi, updated project-config, system-config and infra-manual | 18:50 |
| fungi | thanks clarkb | 18:51 |
| *** amoralej is now known as amoralej|off | 18:54 | |
| openstackgerrit | Merged openstack-infra/system-config master: Correct global site manifest regex for planet.o.o https://review.openstack.org/452281 | 18:54 |
| *** baoli has quit IRC | 18:59 | |
| openstackgerrit | azat proposed openstack-infra/jenkins-job-builder master: scm: bypass git.submodule.parentCredentials https://review.openstack.org/452280 | 18:59 |
| *** baoli has joined #openstack-infra | 19:00 | |
| *** baoli has quit IRC | 19:00 | |
| *** e0ne has quit IRC | 19:02 | |
| *** baoli has joined #openstack-infra | 19:05 | |
| *** csomerville has quit IRC | 19:05 | |
| *** jascott1_ has quit IRC | 19:06 | |
| AJaeger | fungi, https://review.openstack.org/#/q/status:open+++topic:failures are my three changes in case you want to put them on your review queue. | 19:06 |
| *** chem_gone has quit IRC | 19:06 | |
| * AJaeger considered updating status.openstack.org/index.html but couldn't come up with something great, so left it out. | 19:06 | |
| *** marst has quit IRC | 19:07 | |
| *** chem_gone has joined #openstack-infra | 19:07 | |
| *** wolverineav has joined #openstack-infra | 19:09 | |
| *** mat128 has quit IRC | 19:11 | |
| *** mat128 has joined #openstack-infra | 19:12 | |
| *** gcb has quit IRC | 19:12 | |
| *** EricGonc_ has quit IRC | 19:13 | |
| *** gcb has joined #openstack-infra | 19:13 | |
| *** mat128 has quit IRC | 19:16 | |
| *** nadya has joined #openstack-infra | 19:19 | |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/project-config master: Add job to Neutron DVR check queue panel https://review.openstack.org/452294 | 19:19 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 19:20 |
| *** nadya has quit IRC | 19:20 | |
| *** jascott1_ has joined #openstack-infra | 19:22 | |
| *** sdague_ has joined #openstack-infra | 19:23 | |
| *** Goneri has quit IRC | 19:23 | |
| *** baoli has quit IRC | 19:26 | |
| *** baoli has joined #openstack-infra | 19:26 | |
| *** baoli has quit IRC | 19:31 | |
| *** wolverineav has quit IRC | 19:33 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish LOCI tarball to tarballs.o.o https://review.openstack.org/452299 | 19:35 |
| SamYaple | fungi: this is what i came up with ^ | 19:35 |
| *** tonytan_brb has quit IRC | 19:35 | |
| SamYaple | short of creating a stub python project just to publish tarballs, i dont know what to do | 19:35 |
| SamYaple | what else rather | 19:36 |
| *** john-davidge has joined #openstack-infra | 19:40 | |
| pabelanger | SamYaple: fungi: left comment | 19:41 |
| SamYaple | pabelanger: oh my mistake. man i need to figure out this reading comprehenssion thing | 19:42 |
| SamYaple | i thought fungi was saying i couldnt do that | 19:42 |
| pabelanger | you should be fine, but checking backscroll | 19:42 |
| fungi | i'm lacking context. still working on logs.o.o situation | 19:44 |
| fungi | and getting ready to be handy during the lists.o.o maintenance | 19:44 |
| SamYaple | fungi: ignore. its fine. thanks for the help | 19:45 |
| SamYaple | pabelanger: yea i just read through the backscroll, i think my mistake was reading 'jobs' as 'publishers' and getting confused | 19:45 |
| *** john-davidge has quit IRC | 19:46 | |
| pabelanger | SamYaple: everything should be fine, just switch out your dir to store the tarball in, and use tarball publisher | 19:47 |
| pabelanger | that should work | 19:47 |
| * clarkb tanks up on caffeine before lists.o.o work | 19:47 | |
| SamYaple | pabelanger: cool. running checks locally right now then will push new patchset. thanks for all the help | 19:48 |
| *** sc68cal has left #openstack-infra | 19:49 | |
| *** sdague_ has quit IRC | 19:51 | |
| fungi | clarkb: jeblair: i'm here to help with lists.o.o tasks as well if needed, but focusing on logs volume situation as first priority so you can pay closer attention to lists.o.o work | 19:51 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish LOCI tarball to tarballs.o.o https://review.openstack.org/452299 | 19:54 |
| jeblair | fungi, clarkb: either of you have that etherpad handy? | 19:54 |
| fungi | clarkb: mnaser: ianw: just to close the loop on this one, i've confirmed that http://planet.openstack.org/ seems to be fine now | 19:54 |
| fungi | jeblair: was just pulling it up | 19:54 |
| fungi | jeblair: https://etherpad.openstack.org/p/lists.o.o-trusty-upgrade | 19:54 |
| mnaser | fungi cool! | 19:54 |
| jeblair | fungi: thx | 19:54 |
| jeblair | status notice lists.openstack.org will be offline from 20:00 to 23:00 UTC for planned upgrade maintenance | 19:55 |
| jeblair | fungi, clarkb: ^ ? | 19:55 |
| jeblair | i was going to include a link to the ml post... but.... :) | 19:56 |
| fungi | jeblair: lgtm | 19:56 |
| fungi | oh, i can get you one | 19:56 |
| fungi | hah | 19:56 |
| fungi | nm | 19:56 |
| jeblair | ya :) | 19:56 |
| * fungi just got the joke | 19:56 | |
| clarkb | :) | 19:56 |
| clarkb | lgtn | 19:56 |
| fungi | i bet there's a third-part ml mirror we could link | 19:56 |
| *** sdague_ has joined #openstack-infra | 19:56 | |
| jeblair | #status notice lists.openstack.org will be offline from 20:00 to 23:00 UTC for planned upgrade maintenance | 19:56 |
| openstackstatus | jeblair: sending notice | 19:56 |
| fungi | not finding a good one though, so probably fine | 19:58 |
| jeblair | fungi, clarkb: i'll add lists.o.o to emergency | 19:58 |
| clarkb | jeblair: ok | 19:58 |
| -openstackstatus- NOTICE: lists.openstack.org will be offline from 20:00 to 23:00 UTC for planned upgrade maintenance | 19:58 | |
| fungi | right on | 19:58 |
| jeblair | done, and ansible is not running | 19:58 |
| JayF | #success My inbox gets three hours of reprieve thanks to infra! Woohoo maintenance! | 19:59 |
| fungi | even better | 19:59 |
| fungi | JayF: the down-side is that all those messages will just slam your inbox at once when we finish | 19:59 |
| fungi | drink while you can | 19:59 |
| JayF | fungi: shh, you're interrupting the quiet ;) | 19:59 |
| openstackstatus | jeblair: finished sending notice | 20:00 |
| jeblair | i have screen running as root on lists | 20:00 |
| openstackstatus | JayF: Added success to Success page | 20:00 |
| jeblair | i will stop services now | 20:01 |
| jeblair | apt-get upgrade now | 20:02 |
| * clarkb hops onto the screen | 20:03 | |
| clarkb | I see langs getting selected | 20:03 |
| clarkb | looks like mailmain didn't completely stop? | 20:04 |
| jeblair | i apparently stopped it with qfiles present and the package upgrade doesn't like that | 20:05 |
| jeblair | so i'm letting the queue flush (to exim) and then will redo | 20:05 |
| fungi | oh, that's just spiffy | 20:05 |
| clarkb | did you restart exim for that? | 20:05 |
| clarkb | or is that just a file delivery? | 20:05 |
| * mordred is here, in case he can be helpful | 20:05 | |
| jeblair | clarkb: i don't think we need exim running for that | 20:06 |
| jeblair | seems happier this time | 20:06 |
| clarkb | ya looks it | 20:07 |
| jeblair | The following packages have been kept back: landscape-common | 20:07 |
| jeblair | do we care? | 20:07 |
| mordred | nope | 20:07 |
| mordred | we should likely delete that from our servers at some point | 20:07 |
| clarkb | thats ubuntus server managment tool iirc | 20:07 |
| jeblair | running do-release-upgrade | 20:07 |
| clarkb | so ya don't care | 20:07 |
| *** rhallisey has quit IRC | 20:09 | |
| fungi | right, landscape seems useless for us | 20:10 |
| *** Qiming has quit IRC | 20:12 | |
| jeblair | we're at the "lots of stuff scrolling by" portion of the upgrade | 20:14 |
| clarkb | can confirm | 20:15 |
| * fungi casts "wall of upgrade" | 20:16 | |
| *** Qiming has joined #openstack-infra | 20:16 | |
| SamYaple | TIL about do-release-upgrade | 20:17 |
| SamYaple | ive always just updated my sources.list... | 20:17 |
| fungi | SamYaple: in debianland, you read the release notes, manye perform a few manual steps outlined there which can't be easily accounted for by package maintscripts, and then update your sources.list and do an upgrade (possibly followed by a reboot) and then a dist-upgrade (probably followed by a reboot) | 20:19 |
| clarkb | jeblair: wants to know what to do with sysctl.conf /me cnosults etherpad | 20:19 |
| clarkb | n for keep current | 20:19 |
| jeblair | clarkb: ya, i decided they were all "keep current version" during the trial run | 20:19 |
| fungi | SamYaple: in ubuntuland they try to write a script that does the stuff you might want from the release notes. makes me a little uneasy, but what are you gonna do (oh, i know, run debian!) | 20:19 |
| SamYaple | fungi: yea i always skipped the release notes too. then i just ended up rekicking | 20:19 |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze master: Remove swift support https://review.openstack.org/452316 | 20:19 |
| *** kgiusti has joined #openstack-infra | 20:20 | |
| clarkb | jeblair: I see that now thanks | 20:20 |
| jeblair | clarkb: i'm inclined to just follow the script (probably not much has changed in a week), but if you want we can diff. | 20:20 |
| clarkb | jeblair: considering its all keep old I think thats safe | 20:20 |
| jeblair | kk | 20:20 |
| clarkb | if we were keeping new it might be worth checking nothing changed | 20:20 |
| *** ihrachys has quit IRC | 20:20 | |
| clarkb | s/keeping/switching/ | 20:20 |
| clarkb | fungi: I solved that by running a rolling distro on my desktop | 20:21 |
| fungi | yeah, not applying any config updates should be safe. we can go back and look at the .dpkg-new versions later if we want | 20:21 |
| fungi | clarkb: me too, (debian/unstable) ;) | 20:21 |
| *** gouthamr has quit IRC | 20:22 | |
| *** ihrachys has joined #openstack-infra | 20:22 | |
| jeblair | according to the etherpad, that should be the last question | 20:25 |
| jeblair | we got the missing site list message last time. i thought maybe it was due to the hostname mismatch, so i don't know why we got it this time. hopefully a red herring, but we should be careful (of course) before restarting services. | 20:27 |
| jeblair | clarkb: i guess we should remove the startup links since we have a couple of reboots ahead of us | 20:29 |
| clarkb | jeblair: ok | 20:29 |
| *** arxcruz|pto has quit IRC | 20:29 | |
| *** EricGonczer_ has joined #openstack-infra | 20:29 | |
| clarkb | maybe do ls for sanity? | 20:30 |
| clarkb | of /etc/rc* | 20:30 |
| clarkb | lgtm | 20:31 |
| jeblair | clarkb: reboot now? | 20:31 |
| clarkb | thats what the script says and the init stuff appears removed | 20:31 |
| clarkb | can't think of anyting else we should do first | 20:31 |
| jeblair | looks like there's a workable grub/kernel config | 20:32 |
| jeblair | so i'll reboot now | 20:32 |
| clarkb | ya and with the new kernel too | 20:32 |
| jeblair | it's up | 20:33 |
| jeblair | screen started | 20:34 |
| clarkb | I am attached | 20:34 |
| jeblair | Linux lists.openstack.org 3.13.0-115-generic #162-Ubuntu SMP Fri Mar 24 16:03:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | 20:34 |
| jeblair | looks good | 20:34 |
| clarkb | do we still need to upgrade again for new kernel? | 20:34 |
| clarkb | seems like thats it | 20:34 |
| *** ihrachys has quit IRC | 20:35 | |
| jeblair | clarkb: right, i'm expecting not. i'll just run the commands to make me feel better though. | 20:35 |
| jeblair | yup. | 20:35 |
| fungi | always makes me feel better | 20:35 |
| *** arxcruz|pto has joined #openstack-infra | 20:35 | |
| jeblair | should we ext4 now? | 20:35 |
| clarkb | I think so | 20:35 |
| fungi | making excellent time | 20:36 |
| *** dprince has quit IRC | 20:36 | |
| *** marst has joined #openstack-infra | 20:36 | |
| *** sdague has quit IRC | 20:36 | |
| jeblair | oh, i seem to have omitted the remount steps | 20:37 |
| jeblair | https://debian-administration.org/article/643/Migrating_a_live_system_from_ext3_to_ext4_filesystem | 20:37 |
| jeblair | i think that's what we did (under the root section) | 20:37 |
| fungi | ahh, yep | 20:37 |
| fungi | looks like the pad just goes from stopping daemons to fsck | 20:38 |
| jeblair | i think it's only missing that one command | 20:38 |
| clarkb | thats the doc I remember at least | 20:38 |
| jeblair | ok, / is ro | 20:38 |
| jeblair | so much fsck today | 20:39 |
| clarkb | must be friday | 20:39 |
| mordred | fsck-friday | 20:41 |
| *** sdague_ has quit IRC | 20:41 | |
| jeblair | fsckday | 20:41 |
| fungi | fscktastic | 20:42 |
| *** Apoorva_ has quit IRC | 20:43 | |
| *** Apoorva has joined #openstack-infra | 20:44 | |
| clarkb | jeblair: the -v isn't being as verbose as I expected | 20:44 |
| *** eharney has quit IRC | 20:45 | |
| *** mat128 has joined #openstack-infra | 20:45 | |
| jeblair | true. | 20:45 |
| jeblair | strace says that reads and lseeks are happening. | 20:46 |
| *** gouthamr has joined #openstack-infra | 20:46 | |
| *** sshnaidm|off has joined #openstack-infra | 20:46 | |
| *** trown is now known as trown|outtypewww | 20:47 | |
| *** csomerville has joined #openstack-infra | 20:48 | |
| *** bnemec is now known as beekneemech | 20:49 | |
| *** asselin has joined #openstack-infra | 20:49 | |
| *** rcarrillocruz has joined #openstack-infra | 20:49 | |
| jeblair | we have exceeded the amount of time this took on the test host | 20:50 |
| fungi | i wonder if the snapshot booted onto a faster host | 20:50 |
| jeblair | maybe. i'm not going to start planning a freakout until it's 2x. | 20:50 |
| jeblair | there wo go. | 20:51 |
| jeblair | i'll run tune2fs now | 20:51 |
| jeblair | and fsck.ext4 now | 20:52 |
| jeblair | scaling this by the same amount, maybe expect it to finish at 21:20 utc. | 20:53 |
| *** smarcet has quit IRC | 20:54 | |
| mordred | so much fsck | 20:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config master: Add coverage check jobs to tripleo python projects https://review.openstack.org/452321 | 20:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config master: Add py35 job to os-net-config https://review.openstack.org/452322 | 20:54 |
| dmsimard | fsck this | 20:59 |
| dmsimard | fsck that | 20:59 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Rework golang-dsvm job https://review.openstack.org/452323 | 20:59 |
| clarkb | one fsck two fsck red fsck blue fsck | 21:00 |
| dims | lol clarkb | 21:00 |
| *** Qiming has quit IRC | 21:01 | |
| jeblair | clarkb: your kids are going to be *weird*. | 21:01 |
| *** Qiming has joined #openstack-infra | 21:02 | |
| clarkb | I know, I am alraedy scheming all the ways I can make them weird. We had rabbit for dinner last easter (should make that a thing). Also I think their first computer will likely be raspi like thing with serial console | 21:02 |
| fungi | the clarklings will keep portland weirdER | 21:03 |
| *** rbuzatu_ has quit IRC | 21:03 | |
| *** rbuzatu has joined #openstack-infra | 21:04 | |
| openstackgerrit | Merged openstack-infra/shade master: Fixed stack_status.split() exception https://review.openstack.org/452260 | 21:04 |
| clarkb | I just wish it was easier to introduce them to lots of islandy things like breadfruit and sea cucumber and so on | 21:05 |
| clarkb | because right now they eat anything I put in front of them | 21:05 |
| clarkb | except for spinich | 21:05 |
| jeblair | clarkb: ++ on the raspi. i dunno if you'se seen/read eben upton's talks on that, but i agree with him that "boot to programming" is key to learning about computers. https://lwn.net/Articles/543370/ | 21:06 |
| clarkb | I hadn't but ya I want to force them to learn how things work | 21:06 |
| clarkb | and not just jump into facebook | 21:07 |
| *** asselin_ has joined #openstack-infra | 21:07 | |
| *** kaisers has quit IRC | 21:07 | |
| *** asselin has quit IRC | 21:07 | |
| *** ijw has quit IRC | 21:10 | |
| dims | clarkb : ++++++ | 21:10 |
| dims | clarkb : my younger one takes everything apart :) step 1 | 21:11 |
| fungi | i attribute my and my younger brother's predilection in part to dad thinking it was a waste of money to buy computer games when there was plenty of game source code in the backs of hobby computing magazines (all you had to do was type them in... and usually debug them to find your typos) | 21:12 |
| *** dimtruck is now known as zz_dimtruck | 21:13 | |
| dims | totally worked :) | 21:13 |
| jeblair | fungi: and a win-win for your dad. | 21:13 |
| fungi | indeed | 21:13 |
| fungi | i quickly learned to prefer the sources with line-by-line and section-by-section checksums | 21:14 |
| clarkb | I just remember that before I could drive the car I had to know basic things like changing oil, spark plugs, battery, brake line bleeding, air filters, tires, etc. Of course now cars are so complicated I don't even bother with any of them and have someone else do it all :/ | 21:14 |
| *** pramodrj07 has joined #openstack-infra | 21:15 | |
| clarkb | also fixing the bad cpu fan by leaving case open and pointing box fan at it was a good time | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | fsck is done | 21:15 |
| fungi | mine isn't, unfortunately, and won't be for a while | 21:16 |
| jeblair | ack | 21:16 |
| dmsimard | clarkb: I totally agree with that statement. Cars are more complicated than OpenStack :) | 21:16 |
| *** mat128 has quit IRC | 21:16 | |
| jeblair | i reckon we reboot again now? | 21:16 |
| *** sdague has joined #openstack-infra | 21:16 | |
| clarkb | jeblair: ya I think thats a good idea | 21:16 |
| clarkb | (and what we ddi last time iirc) | 21:16 |
| fungi | yeah | 21:17 |
| fungi | make sure everything starts up correctly | 21:17 |
| fungi | no surprises later that way | 21:17 |
| fungi | well, _fewer_ surprises anyway | 21:17 |
| jeblair | we do still have services disabled | 21:17 |
| clarkb | dmsimard: it made me sad, but I basically had to trade safety over simplicity :/ | 21:17 |
| clarkb | jeblair: ya after reboot next step is turning it all back on again right? | 21:17 |
| jeblair | up and screen | 21:17 |
| clarkb | I'm in | 21:18 |
| dmsimard | clarkb: I think it boils down to interest as well. I could probably learn how to change my breaks and do it myself but I could care less. | 21:18 |
| fungi | jeblair: right, i meant the usual system boot proceeding normally (rather than remounting rw and trying to start stuff back up) | 21:18 |
| dmsimard | s/breaks/brakes/ | 21:18 |
| jeblair | clarkb: i want to start by bringing up apache and seeing if mailman webui is happy, or if there's something funky with the site list | 21:18 |
| clarkb | jeblair: sounds good | 21:18 |
| fungi | wfm | 21:19 |
| jeblair | so let's start by changing the default language | 21:19 |
| jeblair | ugh, i missed it. i wonder if de was recently adedd | 21:20 |
| *** askb has joined #openstack-infra | 21:21 | |
| *** aysyd has quit IRC | 21:22 | |
| mordred | it seems like a weird language to have been recently added | 21:22 |
| clarkb | these steps are not on the etherpad? | 21:23 |
| fungi | this didn't happen in the snapshot upgrade (afaik) | 21:23 |
| jeblair | clarkb: does http://lists.openstack.org look chinese to you? | 21:24 |
| clarkb | yes | 21:24 |
| jeblair | i was hoping that reconfiguring and re-setting the default language would correct that, but doesn't seem to have. | 21:24 |
| clarkb | but http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev isn't | 21:24 |
| jeblair | okay, i edited mm_cfg.py to change DEFAULT_SERVER_LANGUAGE to en | 21:25 |
| jeblair | i'm not sure why dpkg was okay with breaking that earlier but not fixing it now. | 21:25 |
| jeblair | clarkb: yeah, individual lists have their own lang preferences | 21:25 |
| fungi | was that not previously set? | 21:25 |
| openstackgerrit | azat proposed openstack-infra/jenkins-job-builder master: builders: do not pass empty "preload-script" https://review.openstack.org/452331 | 21:26 |
| jeblair | fungi: before 2 hours ago? i don't know. before i edited it, it was set to zh_TW. | 21:26 |
| *** sekelso has quit IRC | 21:27 | |
| jeblair | at any rate, the mm web ui seems happy now. | 21:27 |
| *** jmorgan1 has quit IRC | 21:28 | |
| clarkb | yes its english now | 21:28 |
| jeblair | anyone want to poke around some more, or should we start up the mm queue runners? | 21:28 |
| clarkb | I apparently don't have all the right font characters for vietnamese :( | 21:28 |
| *** xarses has quit IRC | 21:28 | |
| *** smarcet has joined #openstack-infra | 21:28 | |
| fungi | that would have been a pretty funky default | 21:28 |
| jeblair | clarkb: yeah, i also have a lot of <?> | 21:28 |
| clarkb | jeblair: pretty sure thats a font issue | 21:28 |
| jeblair | yep | 21:29 |
| clarkb | lgtm otherwise | 21:29 |
| fungi | admin/mod interfaces seem fine | 21:29 |
| jeblair | i'll start mailman now then | 21:29 |
| fungi | message archives lgtm | 21:29 |
| fungi | yeah, looks great | 21:29 |
| jeblair | it's quiet, as expected since it has empty queues | 21:30 |
| jeblair | start exim now? | 21:30 |
| clarkb | thats what will get it to actually do things right? | 21:31 |
| fungi | yup | 21:31 |
| clarkb | (need new mail in for new mail out) | 21:31 |
| clarkb | so ya I think thats next | 21:31 |
| jeblair | clarkb: yep. | 21:31 |
| *** jcoufal has joined #openstack-infra | 21:32 | |
| jeblair | we should make a real test list on the production server. :) | 21:33 |
| *** sdague has quit IRC | 21:33 | |
| *** smarcet has quit IRC | 21:34 | |
| clarkb | we can probably use ifnra list for that this time | 21:34 |
| jeblair | sent | 21:34 |
| *** jcoufal_ has quit IRC | 21:34 | |
| clarkb | I just got it | 21:34 |
| *** Goneri has joined #openstack-infra | 21:35 | |
| fungi | now mine arrived | 21:35 |
| fungi | working great! | 21:36 |
| *** tonytan4ever has joined #openstack-infra | 21:36 | |
| mtreinish | jeblair: I got your test message | 21:37 |
| jeblair | mtreinish: it was really just for you | 21:37 |
| mtreinish | heh, that's what I figured :) | 21:37 |
| *** mat128 has joined #openstack-infra | 21:38 | |
| jeblair | fungi, clarkb: the log entries all look good to me | 21:38 |
| *** tonytan_brb has joined #openstack-infra | 21:38 | |
| *** tonytan4ever has quit IRC | 21:38 | |
| clarkb | cool, I wouldn't really know what to look for in them | 21:39 |
| *** mat128 has quit IRC | 21:39 | |
| fungi | maintenance concluded in estimated time (divided by scotty multiplier) | 21:39 |
| fungi | awesome | 21:39 |
| clarkb | I guess make sure there aren't any domains that we can't deliver to suddenly (though preserving IP addrs should provent that) | 21:39 |
| jeblair | fungi: that is, of course, exactly how i calculated it. :) | 21:39 |
| jeblair | i've re-enabled startup scripts | 21:40 |
| clarkb | do we want to do one final reboot to make sure everything comes back cleanly? | 21:40 |
| jeblair | clarkb: not a bad idea | 21:40 |
| fungi | wfm | 21:41 |
| jeblair | rebooting now | 21:41 |
| jeblair | up and screened | 21:42 |
| clarkb | jeblair: host lookups that don't complete == domain is not in dns? | 21:43 |
| andreykurilin | clarkb AJaeger: https://review.openstack.org/#/c/450490 everything works as expected! thanks | 21:43 |
| clarkb | or perhaps no MX record for domain? | 21:43 |
| jeblair | clarkb: usually yeah | 21:43 |
| fungi | yeah, usual process is look for mx, no mx then look for a/aaaa | 21:43 |
| clarkb | jeblair: is that something we should be cleaning up after, if you don't have working email for domain then we unsub after a month? | 21:44 |
| clarkb | or is cost low enough that we don't care? | 21:44 |
| jeblair | clarkb: should be automatic | 21:44 |
| clarkb | ah ok | 21:44 |
| jeblair | clarkb: that should eventually bounce and mailman will drop after 5 bounces by default i think. | 21:44 |
| fungi | right | 21:44 |
| *** zz_dimtruck is now known as dimtruck | 21:44 | |
| clarkb | oh right I remember getting unsubscribed once hence the change to fastmail | 21:44 |
| jeblair | clarkb: for a lot of hosts though, they intermittently have dns resolution problems, so you'll get a bounce, then a message gets through, then another bounce, so they stay on lists. | 21:45 |
| jeblair | clarkb: so there's a sort of fixed overhead that scales with subscribers. but the bounce processing keeps it from growing without bound. | 21:46 |
| jeblair | i've removed lists.o.o from puppet emergency | 21:46 |
| *** Sihan has quit IRC | 21:46 | |
| jeblair | or ansible emergency | 21:47 |
| jeblair | or whatever | 21:47 |
| *** xarses has joined #openstack-infra | 21:47 | |
| jeblair | status notice The upgrade maintenance for lists.openstack.org has been completed and it is back online. | 21:48 |
| jeblair | clarkb, fungi: ^? | 21:48 |
| *** xarses_ has joined #openstack-infra | 21:48 | |
| fungi | lgtm | 21:48 |
| clarkb | yup | 21:48 |
| jeblair | #status notice The upgrade maintenance for lists.openstack.org has been completed and it is back online. | 21:48 |
| openstackstatus | jeblair: sending notice | 21:48 |
| *** xarses_ has quit IRC | 21:48 | |
| clarkb | you can even send a response to the email to prove to everyone it works :) | 21:49 |
| clarkb | the email thread about scheduling the downtime | 21:49 |
| *** xarses_ has joined #openstack-infra | 21:49 | |
| jeblair | clarkb: i'll leave that to you (or fungi) :) | 21:50 |
| -openstackstatus- NOTICE: The upgrade maintenance for lists.openstack.org has been completed and it is back online. | 21:50 | |
| clarkb | I can do it | 21:50 |
| fungi | thanks! | 21:50 |
| fungi | and thanks jeblair for driving a quick and uneventful upgrade! | 21:51 |
| openstackstatus | jeblair: finished sending notice | 21:52 |
| *** xarses has quit IRC | 21:52 | |
| sbezverk | what about zuul, is it still undergoing medication process, or it is done too? | 21:52 |
| *** kzaitsev_mb has joined #openstack-infra | 21:52 | |
| fungi | sbezverk: logs.openstack.org? still in surgery. there were complications | 21:52 |
| sbezverk | fungi: Oh man, I hope it is not lethal ;) | 21:53 |
| fungi | i don't think so, just very lengthy | 21:53 |
| *** EricGonczer_ has quit IRC | 21:53 | |
| *** ijw has joined #openstack-infra | 21:53 | |
| fungi | sbezverk: at this point it shouldn't be impacting jobs hopefully, you just can't get to the logs to look at them yet | 21:53 |
| jeblair | smtp to openstack for 3764 recips, completed in 34.706 seconds | 21:54 |
| jeblair | in case anyone was curious | 21:54 |
| *** EricGonczer_ has joined #openstack-infra | 21:54 | |
| *** EricGonczer_ has quit IRC | 21:54 | |
| clarkb | mail sent | 21:54 |
| mordred | fungi: aren't you glad that long surgery for us does not ACTUALLY involve any surgery? | 21:56 |
| fungi | so glad | 21:56 |
| clarkb | yes I do not like operating rooms | 21:56 |
| *** xyang1 has quit IRC | 21:56 | |
| jeblair | when we have a really bad day, people have to run tests on their own computers instead of other computers that run them for free. | 21:57 |
| pabelanger | wow, just got CC'd on PR from salt, back from 2013 | 21:57 |
| jeblair | this is different than an actual surgeon's very bad day in many important respects. | 21:57 |
| mnaser | "guess i'll have to do this surgery in my bedroom instead of a hospital facility" | 21:57 |
| *** ijw has quit IRC | 21:58 | |
| jeblair | (also, worth noting that most jobs should actually be running okay since we're dumping logs to a temporary location) | 21:58 |
| openstackgerrit | azat proposed openstack-infra/jenkins-job-builder master: builders: do not pass empty "preload-script" https://review.openstack.org/452331 | 21:58 |
| SamYaple | pfff like anyone even looks at the logs anyway | 21:58 |
| SamYaple | green == LGTM == merge | 21:58 |
| jeblair | SamYaple: indeed, we should be looking at write-only storage | 21:59 |
| *** Swami has quit IRC | 21:59 | |
| SamYaple | you already have it! | 21:59 |
| SamYaple | | /dev/null | 21:59 |
| mnaser | i think they have read only storage right now.. haa..... | 21:59 |
| *** leakypipes has quit IRC | 21:59 | |
| jeblair | mnaser: we have fsck-only storage i think | 21:59 |
| clarkb | I know you say that in jest but I have read so many logs in the last couple weeks (and on the bright side openstack seems really reliable right now in testing) | 21:59 |
| *** flepied has joined #openstack-infra | 22:00 | |
| openstackgerrit | azat proposed openstack-infra/jenkins-job-builder master: scm: bypass git.submodule.parentCredentials https://review.openstack.org/452280 | 22:00 |
| *** rlandy has quit IRC | 22:00 | |
| mnaser | i thought logs.o.o was mostly for benchmarking browsers in loading 4mb pages | 22:00 |
| clarkb | mnaser: vim so much better | 22:00 |
| fungi | itym 400mb pages | 22:00 |
| clarkb | I actually need a thing that execs vim to open those files | 22:00 |
| SamYaple | mnaser: chrome wont buffer pages ending with .html, but it will buffer pages ending in .gz | 22:01 |
| SamYaple | if only we pushed console.html.gz initially | 22:01 |
| SamYaple | so much easier to read | 22:01 |
| clarkb | SamYaple: thats partly why we try to keep jobs in smaller console logs | 22:01 |
| clarkb | devstack for example logs to its own file and os on | 22:02 |
| mnaser | clarkb: we can setup z/os with time sharing option and vim on it and give accounts to all foundation members | 22:02 |
| clarkb | mnaser: well I can just vim http://logs.openstack.org/etc | 22:02 |
| SamYaple | clarkb: yea thats the way to do it | 22:02 |
| clarkb | but need my webbrowser to do that for me when I click on gerrit link rather than copy paste | 22:02 |
| mnaser | i think this is one of the things that kolla is bad at, i improved it by adding per-file-logging for builds but deploys still throw everything into console | 22:02 |
| SamYaple | clarkb: do you think we can modify the console-log publisher to publish both the compressed and uncompressed? and then just come through and clean up the uncompressed in the future? | 22:03 |
| jeblair | clarkb: you could make gertty do that fairly easily i think. | 22:03 |
| *** jcoufal_ has joined #openstack-infra | 22:03 | |
| clarkb | SamYaple: I think we can likely just send the compressed version then our webserver should do the right thing | 22:03 |
| SamYaple | mnaser: we had per image building in kolla initially. people wanted all output to stdout. :( | 22:03 |
| fungi | clarkb: my gertty does that for me via | 22:03 |
| mnaser | SamYaple its not even readable at all with multiple threads | 22:03 |
| fungi | BROWSER=vim | 22:03 |
| clarkb | fungi: nice! | 22:03 |
| SamYaple | mnaser: I KNOW. thats what i said. but i was overruled | 22:03 |
| mnaser | now we do $NCPU build threads + logs per image built... so i guess you were right :p | 22:04 |
| * SamYaple is taking notes | 22:04 | |
| SamYaple | fungi: im going to do that | 22:04 |
| SamYaple | that sounds fantastiy | 22:04 |
| SamYaple | fantastic* | 22:04 |
| fungi | clarkb: though BROWSER=lynx works a bit better since you can follow subsequent urls | 22:04 |
| clarkb | fungi: does lynx handle the large files well? | 22:05 |
| fungi | it has for me so far | 22:05 |
| *** jcoufal has quit IRC | 22:05 | |
| fungi | doesn't grind to a halt like ff | 22:05 |
| jeblair | clarkb, fungi: i'm going to terminate screen on lists now. i don't think watching the matrix is going to tell us anything else. | 22:05 |
| SamYaple | fungi: pretty sure there is an extention to vim to open up vim again with the new webpage if you are running in tmux | 22:05 |
| clarkb | jeblair: ya I detached already | 22:05 |
| fungi | SamYaple: probably | 22:05 |
| pabelanger | email worked! | 22:06 |
| SamYaple | mnaser: some of the old school kolla folks also raised the complaint (i think jpeeler wasn't a fan of it) | 22:06 |
| *** jcoufal_ has quit IRC | 22:06 | |
| openstackgerrit | Merged openstack-infra/shade master: Add test to validate multi _ heat stack_status https://review.openstack.org/452261 | 22:07 |
| clarkb | does that mean its time to go do yardwork while the sun is up? | 22:07 |
| fungi | it's wet here today, so saving yardwork for sunday when things will have dried out some | 22:08 |
| pabelanger | I get to battle the snow tonight | 22:08 |
| pabelanger | been snowing all day | 22:08 |
| clarkb | ya today is our nice day then back to rain | 22:08 |
| bkero | Your place doesn't seem to have gotten the memo about spring. | 22:08 |
| bkero | clarkb: It's back to nice on Sunday again. | 22:08 |
| mnaser | its snowing here until saturday :( | 22:09 |
| * mnaser made the mistake of taking winter tires off | 22:09 | |
| clarkb | bkero: oh thats new | 22:10 |
| * bkero has to decide between homebrew on the sunny porch or 45 minute bikeride and tasty Portland beer from a nice neighborhood pub. | 22:10 | |
| pabelanger | mnaser: yup, same in Ottawa | 22:10 |
| fungi | pabelanger: qapla', today is a good day to shovel | 22:10 |
| *** thorst has quit IRC | 22:10 | |
| pabelanger | indeed | 22:10 |
| clarkb | bkero: there is much tasty beer nearer to you if you want to have shorter bikeride ;) | 22:12 |
| clarkb | bkero: I'm sure I can rattle off a giant list | 22:13 |
| bkero | clarkb: There is. I have one of the best taprooms in Portland 2 blocks from me. | 22:13 |
| bkero | But I just gave my road bike a spring refresher. Brakes, gears, chain, tires, etc. I need to go get it out. | 22:13 |
| clarkb | ah | 22:13 |
| *** gordc has quit IRC | 22:13 | |
| bkero | Plus I haven't met my goal of 45 minutes of recorded cardio this week. :) | 22:13 |
| clarkb | even widmer is better than what you'd get just about anywhere else | 22:13 |
| clarkb | and they have shnitzel | 22:13 |
| clarkb | (anywhere else being not portland) | 22:13 |
| bkero | https://b.zmtcdn.com/data/pictures/6/17050296/ebcb6283b1748e86699b04d268fb9f26.jpg | 22:14 |
| bkero | but picnic tables and beer on a sunny day | 22:15 |
| clarkb | widmer has outdoor | 22:15 |
| clarkb | I think :P but ya don't go there so much better options | 22:15 |
| bkero | Do they have beer other than Widmer? | 22:15 |
| clarkb | I think they do have guest taps | 22:15 |
| clarkb | maybe I'm wrong | 22:16 |
| *** ijw has joined #openstack-infra | 22:17 | |
| bkero | clarkb: I'm pretty happy with this: https://b.zmtcdn.com/data/reviews_photos/d64/92e94cb9e0704adbfe1b0a030046fd64.jpg | 22:17 |
| bkero | The cylinders behind each tap are CO2 chambers, so that when they fill the growlers they keep carbonation and aren't exposed to O2. | 22:18 |
| bkero | clarkb: I could also bike to Breakside :) | 22:18 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Use libyaml if possible https://review.openstack.org/451113 | 22:20 |
| *** signed8b_ is now known as signed8bit_Zzz | 22:21 | |
| *** rbrndt has quit IRC | 22:24 | |
| *** rbrndt has joined #openstack-infra | 22:24 | |
| *** rbrndt has quit IRC | 22:24 | |
| *** signed8bit_Zzz is now known as signed8b_ | 22:26 | |
| *** jamesden_ has quit IRC | 22:27 | |
| ianw | fungi: thakns for fixing planet, need to finish up next week | 22:27 |
| *** Sihan has joined #openstack-infra | 22:28 | |
| Sihan | Hi, One more question for zuul, do we have any statistics tool for zuul? for example how long does a job wait in the queue or how many patches triggered per day etc. | 22:29 |
| fungi | Sihan: zuul has statsd reporting built in, just set up a statsd server and configure to send events there, then you can use something like graphite/grafana to chart whatever you want | 22:30 |
| fungi | which is what we do | 22:30 |
| Sihan | thx :) | 22:31 |
| *** dimtruck is now known as zz_dimtruck | 22:32 | |
| fungi | Sihan: the graphs (sparklines too) you see at http://status.openstack.org/zuul are being generated with calls to http://graphite.openstack.org/ but we also have some dashboards on http://grafana.openstack.org/ for zuul and (mostly) nodepool which might serve as good examples | 22:33 |
| *** gouthamr has quit IRC | 22:34 | |
| *** tonytan4ever has joined #openstack-infra | 22:36 | |
| *** ihrachys has joined #openstack-infra | 22:37 | |
| *** tonytan_brb has quit IRC | 22:38 | |
| *** funzo has quit IRC | 22:41 | |
| *** signed8b_ is now known as signed8bit_Zzz | 22:42 | |
| *** adrian_otto has quit IRC | 22:43 | |
| *** Goneri has quit IRC | 22:44 | |
| *** ihrachys has quit IRC | 22:46 | |
| *** zz_dimtruck is now known as dimtruck | 22:49 | |
| *** kgiusti has quit IRC | 22:49 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_timer_sshkey https://review.openstack.org/450897 | 22:50 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add gate-tripleo-ci-centos-7-ovb-containers-oooq-nv to cistatus https://review.openstack.org/451845 | 22:53 |
| *** kzaitsev_mb has quit IRC | 22:54 | |
| *** adrian_otto has joined #openstack-infra | 22:54 | |
| *** xarses_ has quit IRC | 23:03 | |
| *** ijw has quit IRC | 23:04 | |
| *** john-davidge has joined #openstack-infra | 23:05 | |
| *** Sukhdev has joined #openstack-infra | 23:07 | |
| *** dimtruck is now known as zz_dimtruck | 23:08 | |
| Sukhdev | Hey Infra experts - I need to kick off a new networking-xxx repo for a new networking driver. Can somebody point me to the process/procedure, please - Last one I did was many years ago - wonder if the same process or different now. | 23:09 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 23:09 |
| *** john-davidge has quit IRC | 23:10 | |
| *** thorst has joined #openstack-infra | 23:11 | |
| *** adrian_otto has quit IRC | 23:15 | |
| *** thorst has quit IRC | 23:15 | |
| fungi | Sukhdev: https://docs.openstack.org/infra/manual/creators.html | 23:15 |
| *** tonytan4ever has quit IRC | 23:15 | |
| Sukhdev | fungi : Thanks a bunch | 23:15 |
| *** adrian_otto has joined #openstack-infra | 23:18 | |
| dims | mordred : clarkb : one of you still around? easy one https://review.openstack.org/#/c/452323/ - job was just added this morning and won't affect anyone else. please take a look if you get a chance. i know it's too late :) | 23:19 |
| *** zz_dimtruck is now known as dimtruck | 23:20 | |
| mordred | dims: done | 23:24 |
| *** adrian_otto has quit IRC | 23:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fully qualify project configuration names https://review.openstack.org/451970 | 23:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add hostname to TriggerEvent https://review.openstack.org/452348 | 23:26 |
| kfox1111 | hows logs.o.o looking? | 23:26 |
| *** dimtruck is now known as zz_dimtruck | 23:38 | |
| *** thorst has joined #openstack-infra | 23:42 | |
| *** ZZelle has quit IRC | 23:42 | |
| *** Julien-zte has quit IRC | 23:42 | |
| *** tosky has quit IRC | 23:47 | |
| fungi | kfox1111: the fsck originally got started with -p, so aborted around 90% and had to be restarted. we took the opportunity to temporarily redirect log uploads somewhere else with plenty of extra space so we can merge them back in once the main volume is back in service | 23:47 |
| *** gouthamr has joined #openstack-infra | 23:50 | |
| kfox1111 | ah. ok. thanks for the info. | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
