Thursday, 2016-06-16

*** openstack has joined #openstack-infra05:58
*** amitgandhinz has joined #openstack-infra05:59
*** tkelsey has joined #openstack-infra06:00
*** jyuso1 has quit IRC06:00
*** jyuso1 has joined #openstack-infra06:01
*** alex_xu has quit IRC06:03
*** kzaitsev_mb has quit IRC06:03
*** amitgandhinz has quit IRC06:03
*** tkelsey has quit IRC06:05
*** alex_xu has joined #openstack-infra06:06
*** andreas_s has joined #openstack-infra06:16
*** javeriak has joined #openstack-infra06:17
*** sarob has joined #openstack-infra06:18
*** yolanda has joined #openstack-infra06:18
*** yolanda has quit IRC06:21
*** ramishra has quit IRC06:21
*** sarob has quit IRC06:23
*** yolanda has joined #openstack-infra06:24
*** rcernin has joined #openstack-infra06:24
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/system-config: OpenStack-CI Apt mirror for Ubuntu Cloud Archive
yolandagood morning06:26
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
AJaegergood morning, yolanda !06:26
*** ramishra has joined #openstack-infra06:26
*** javeriak has quit IRC06:31
*** _nadya_ has quit IRC06:31
*** javeriak has joined #openstack-infra06:32
AJaegeryolanda: rax-ord still has old images, do you have a chance to manually upload the current ubuntu-trusty image to it, please?06:33
yolandaAJaeger, sure06:34
*** jcoufal has quit IRC06:36
*** Daisy_ has joined #openstack-infra06:36
AJaegerthanks a lot! I have a failing job due to old jenkins/scripts on that image, and rax-ord seems to be the only provider without that change now.06:37
*** Daisy has quit IRC06:39
*** sree_ has joined #openstack-infra06:40
*** sree_ is now known as Guest2139606:41
yolandaimage uploading06:41
*** aeng has quit IRC06:42
*** sree has quit IRC06:43
*** thorst_ has joined #openstack-infra06:44
*** hichihara has joined #openstack-infra06:45
*** Daisy has joined #openstack-infra06:49
*** salv-orlando has quit IRC06:49
*** salv-orlando has joined #openstack-infra06:50
*** Sukhdev has quit IRC06:51
*** Daisy_ has quit IRC06:51
*** afazekas is now known as afazekas|dentist06:52
*** javeriak has quit IRC06:52
*** thorst_ has quit IRC06:52
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** salv-orl_ has joined #openstack-infra06:55
openstackgerritJames Page proposed openstack-infra/project-config: Add specifications repository for charms
*** salv-orlando has quit IRC06:57
*** kzaitsev_mb has joined #openstack-infra06:59
*** _nadya_ has joined #openstack-infra06:59
*** camunoz has quit IRC06:59
*** Daisy_ has joined #openstack-infra06:59
*** ramishra has quit IRC07:00
*** amitgandhinz has joined #openstack-infra07:00
*** Daisy has quit IRC07:00
*** markvoelker_ has quit IRC07:01
*** javeriak has joined #openstack-infra07:01
*** javeriak has joined #openstack-infra07:01
*** markvoelker has joined #openstack-infra07:01
*** kzaitsev_mb has quit IRC07:03
*** Douhet has quit IRC07:04
*** Daisy_ has quit IRC07:04
*** amitgandhinz has quit IRC07:04
*** mikelk has joined #openstack-infra07:05
*** amoralej|off is now known as amoralej07:05
*** ramishra has joined #openstack-infra07:05
*** sorantis has joined #openstack-infra07:05
*** hdaniel has joined #openstack-infra07:06
*** tesseract has joined #openstack-infra07:09
*** esikachev has joined #openstack-infra07:10
*** tristanC has joined #openstack-infra07:12
*** jamielennox is now known as jamielennox|away07:12
*** jed56 has joined #openstack-infra07:16
*** zeih has joined #openstack-infra07:16
*** pcaruana has joined #openstack-infra07:17
*** vincentll has joined #openstack-infra07:18
*** permalac has quit IRC07:19
*** ifarkas has joined #openstack-infra07:19
*** ramishra has quit IRC07:19
*** ramishra has joined #openstack-infra07:19
*** sarob has joined #openstack-infra07:20
*** matrohon has joined #openstack-infra07:20
*** yaume has joined #openstack-infra07:22
openstackgerritwatanabe isao proposed openstack-infra/zuul: WIP: Improve zuul proxy feature
*** sarob has quit IRC07:25
*** ramishra has quit IRC07:26
*** roxanaghe has joined #openstack-infra07:27
*** ramishra has joined #openstack-infra07:28
*** arxcruz has joined #openstack-infra07:28
openstackgerritMerged openstack-infra/project-config: Add specifications repository for charms
*** shardy has joined #openstack-infra07:30
openstackgerritMerged openstack-infra/project-config: Add stable branch jobs for tripleo-ci
*** roxanaghe has quit IRC07:32
*** ramishra has quit IRC07:32
*** abregman has quit IRC07:32
openstackgerritMerged openstack-infra/project-config: Use https to transfer builds
*** abregman has joined #openstack-infra07:33
*** reduxio has quit IRC07:34
*** ccamacho has joined #openstack-infra07:35
*** jlanoux has joined #openstack-infra07:36
*** ramishra has joined #openstack-infra07:37
*** Guest21396 has quit IRC07:37
*** dizquierdo has joined #openstack-infra07:38
*** praneshp has quit IRC07:39
*** jlanoux_ has joined #openstack-infra07:41
*** jlanoux has quit IRC07:41
openstackgerritMerged openstack-infra/project-config: Added new repo for fuel-plugin-datera-cinder
*** acabot has joined #openstack-infra07:42
openstackgerritMerged openstack-infra/project-config: Shorten the name of the kolla-kubernetes gates
openstackgerritMerged openstack-infra/project-config: Enable python3 jobs for git-upstream
*** ramishra has quit IRC07:46
*** ramishra has joined #openstack-infra07:46
*** salv-orl_ has quit IRC07:49
*** thorst_ has joined #openstack-infra07:49
*** dtardivel has joined #openstack-infra07:49
*** salv-orlando has joined #openstack-infra07:49
*** samP has joined #openstack-infra07:49
AJaegeryolanda: how long does the upload take?07:49
yolandait's still on progress07:49
*** dtantsur|afk is now known as dtantsur07:50
*** Douhet has joined #openstack-infra07:51
*** sree has joined #openstack-infra07:52
*** mhickey has joined #openstack-infra07:53
*** samuelBartel has joined #openstack-infra07:53
*** thorst_ has quit IRC07:56
*** _nadya_ has quit IRC07:57
openstackgerritMerged openstack-infra/project-config: Updated two projects in gerrit/projects.yaml
*** hashar has joined #openstack-infra07:58
*** sarob has joined #openstack-infra07:59
*** ggnel_t has joined #openstack-infra07:59
*** kzaitsev_mb has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** amitgandhinz has joined #openstack-infra08:00
*** acoles_ is now known as acoles08:01
esikachevAJaeger: hi! question about
AJaegeresikachev: yes?08:02
openstackgerritMerged openstack-infra/project-config: Add jobs for Monasca-Analytics
*** tkelsey has joined #openstack-infra08:02
esikachevWhy need remove the project name from job name?08:03
AJaegeresikachev: why do you need it there at all?08:03
AJaegeresikachev: for those kind of integration jobs, we normally use one job and then add it to both. It's the same job, isn't it? Where do you differentiate them?08:04
esikachevyou mean unused var?08:04
*** sarob has quit IRC08:04
*** kzaitsev_mb has quit IRC08:04
esikachevgot it08:04
AJaegeresikachev: you might need to enhance the PROJECTS var to checkout the proper projects.08:05
*** hichihara has quit IRC08:05
AJaegeresikachev: REview the change some more - perhaps you should use the variable ;)08:05
*** sflanigan has quit IRC08:05
*** amitgandhinz has quit IRC08:06
*** jaosorior has quit IRC08:07
*** binbincong has quit IRC08:07
*** jaosorior has joined #openstack-infra08:08
openstackgerritIllia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users
*** permalac has joined #openstack-infra08:09
*** mhickey has quit IRC08:11
*** dizquierdo has quit IRC08:11
*** chem```` has joined #openstack-infra08:12
*** e0ne has joined #openstack-infra08:13
*** chem``` has quit IRC08:14
*** zeih has quit IRC08:14
*** zeih has joined #openstack-infra08:15
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add Rally job against deployment with existing users
*** Julien-zte has joined #openstack-infra08:15
openstackgerritIllia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users
*** Daisy has joined #openstack-infra08:17
*** Daisy_ has joined #openstack-infra08:18
*** Daisy has quit IRC08:18
openstackgerritIllia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users
*** binbincong has joined #openstack-infra08:19
*** zeih has quit IRC08:19
*** abregman has quit IRC08:19
*** chem```` is now known as chem08:20
ikhudoshynAJaeger: <-- fixed that. sry for inconvenience. sometimes i might get dumber in mornings08:21
AJaegerikhudoshyn: I put it to WIP to fix it myself, why did you push one nevertheless? Patchset 5 is fine, I commented on 7 - and will not touch it again08:22
* AJaeger sends some fresh coffee to ikhudoshyn ;)08:22
ikhudoshynAJaeger: thanks08:24
* ikhudoshyn reaches for 'patience' potion08:25
yolandaAJaeger, image finally has been uploaded08:25
*** mhickey has joined #openstack-infra08:26
AJaegeryolanda: yeah, thanks!08:26
ikhudoshynAJaeger: I actually first thought to fix it in the right place yet gerrit's colored diff confused me08:27
*** lucas-afk is now known as lucasagomes08:28
AJaegerikhudoshyn: do you understand my most recend comment?08:28
ikhudoshynyep, I won't do myself anything, just keep waiting08:29
*** BobBall_AWOL is now known as BobBall08:30
*** zeih has joined #openstack-infra08:31
openstackgerritChaoyi Huang proposed openstack-infra/project-config: Add the Tricircle functional test job
*** abregman has joined #openstack-infra08:32
AJaegerikhudoshyn: ok, I'll update in a few minutes...08:33
notmorganlifeless: so, looks like io.FileIO raises the wrong error, instead of an IOError, it raises ValueError when you perform .read(0) on a stream only open for writing. I think we need to add ValueError to the exception here: - there was a bug with python that some cases raised ValueError, looks like io.FileIO has08:33
notmorganmimiced this error case.08:33
notmorganlifeless: i'll look at spinning up a patch and a PR to confirm this.08:33
notmorgan(with tests)08:33
ikhudoshynAJaeger: oh, see, coffee helped, i'm getting smarter back08:33
*** Daisy_ has quit IRC08:34
*** savihou has joined #openstack-infra08:34
notmorganlifeless: fwiw, py3 does the right thing08:34
*** Daisy has joined #openstack-infra08:34
*** jordanP has joined #openstack-infra08:34
*** mhickey has quit IRC08:34
*** dingyichen has quit IRC08:35
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add Rally job against deployment with existing users
*** Daisy has quit IRC08:36
*** Daisy has joined #openstack-infra08:36
AJaegerikhudoshyn: looks now fine to me ;)08:36
openstackgerritSampath Priyankara proposed openstack-infra/project-config: Request to add new project masakari
ikhudoshynAJaeger: great, thanks08:37
*** hferenc has quit IRC08:38
AJaegerinfra-root, panko has a job in the post queue and one in periodic - and I don't see those moving forward. Panko has not merged any change yet - is that confusing our CI system?08:40
*** sree has quit IRC08:42
samPHi!, I (Sampath Priyankara) have just push a patch for request to add new project "masakari". According to the doc, I must wait here.. should I send a mail to openstack-infra ML as well?08:42
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add infrstucture scripts to prepare rh2
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI
*** derekh has joined #openstack-infra08:47
*** hferenc has joined #openstack-infra08:47
AJaegersamP: please have patience - wait can be a couple of days. We will review it in due time...08:48
*** sree has joined #openstack-infra08:48
AJaegersamP: Looking at your change: Please add a paragraph explaining what masakari his and why it should be hosted in the OpenStack infrastructure to your change.08:48
lennybHi, Infra. is there a way not to upgrade pip during ./ or request a specific pip version? I am getting error in pip 1.8.2 ( known issue ) when checking LIBS_FROM_GIT param08:49
*** sree has quit IRC08:49
*** sree has joined #openstack-infra08:49
samPAjaeger: sure, thank you. That explanation in commit msg?08:50
AJaegersamP: yes08:50
samPAJaeger: Sure, I will add. Thank you again.08:51
*** kdas__ has joined #openstack-infra08:51
*** mescanef has joined #openstack-infra08:52
*** Daisy has quit IRC08:53
*** Daisy has joined #openstack-infra08:53
*** mdbooth_ is now known as mdbooth08:53
*** thorst_ has joined #openstack-infra08:54
*** salv-orlando has quit IRC08:55
*** salv-orlando has joined #openstack-infra08:56
*** dizquierdo has joined #openstack-infra08:56
*** Daisy_ has joined #openstack-infra08:56
AJaegeryolanda: thanks for uploading, my test finally passed!08:57
*** Daisy has quit IRC08:58
*** Daisy has joined #openstack-infra08:58
*** Daisy has quit IRC08:58
*** Daisy has joined #openstack-infra08:58
*** roxanaghe has joined #openstack-infra09:00
*** dtantsur is now known as dtantsur|brb09:00
*** kzaitsev_mb has joined #openstack-infra09:00
*** Daisy_ has quit IRC09:01
*** amotoki has quit IRC09:01
*** thorst_ has quit IRC09:01
*** amitgandhinz has joined #openstack-infra09:01
*** andreykurilin has joined #openstack-infra09:02
*** dmk0202 has joined #openstack-infra09:02
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute031.uswest metadata for better tracking
*** kzaitsev_mb has quit IRC09:05
*** amitgandhinz has quit IRC09:06
*** nijaba has quit IRC09:06
*** roxanaghe has quit IRC09:06
*** ifarkas has quit IRC09:06
openstackgerritMerged openstack/diskimage-builder: Add python logger configuration
openstackgerritMerged openstack/diskimage-builder: Introspect logging testing more
*** eezhova has joined #openstack-infra09:09
openstackgerritAlina Nesterova proposed openstack-infra/project-config: Migrated sahara-client-functional-job
*** _nadya_ has joined #openstack-infra09:10
openstackgerritElena Ezhova proposed openstack-infra/project-config: Add a functional job for neutron-lbaas
esikachevAJaeger: take a look, is correct?
openstackgerritTomoki Sekiyama proposed openstack-infra/project-config: Create new functional job for glance_store-cinder
*** nijaba has joined #openstack-infra09:13
*** kdas__ is now known as kushal09:14
*** kushal has joined #openstack-infra09:15
*** sree has quit IRC09:16
openstackgerritSampath Priyankara proposed openstack-infra/project-config: Request to add new project Masakari
*** eezhova has quit IRC09:19
*** kzaitsev_mb has joined #openstack-infra09:20
*** Na3iL has joined #openstack-infra09:20
*** eezhova has joined #openstack-infra09:20
*** zeih has quit IRC09:22
*** ihrachys has joined #openstack-infra09:26
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute088.useast metadata for better tracking
*** electrofelix has joined #openstack-infra09:27
*** mkoderer__ has quit IRC09:28
*** javeriak has quit IRC09:28
*** esikachev has quit IRC09:29
openstackgerritTom Fifield proposed openstack-infra/project-config: Add governance-uc repo
*** markusry has joined #openstack-infra09:31
openstackgerritSampath Priyankara proposed openstack-infra/project-config: Request to add new project Masakari
*** Daisy has quit IRC09:32
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute016.useast metadata for better tracking
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute089.useast metadata for better tracking
*** zeih has joined #openstack-infra09:36
*** tosky has joined #openstack-infra09:36
*** ociuhandu has quit IRC09:39
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute086.useast metadata for better tracking
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Remove pagination for timeline events
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Rework timeline events into a timeline
*** HeOS_ has joined #openstack-infra09:43
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute085.useast metadata for better tracking
*** samP has quit IRC09:45
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute087.useast metadata for better tracking
openstackgerritAndrea Frittoli proposed openstack-infra/project-config: Start using pre-prov creds in dsvm-layer4
*** esikachev has joined #openstack-infra09:47
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute084.useast metadata for better tracking
*** javeriak has joined #openstack-infra09:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute014.useast metadata for better tracking
*** kzaitsev_mb has quit IRC09:54
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute013.useast metadata for better tracking
*** bunting_ is now known as bunting09:55
*** samuelBartel has quit IRC09:56
*** ifarkas has joined #openstack-infra09:57
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute012.useast metadata for better tracking
*** wfoster_ is now known as wfoster09:57
*** tosky has quit IRC09:59
*** thorst_ has joined #openstack-infra09:59
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute015.useast metadata for better tracking
wznoinskanteaya are you around?10:00
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute010.useast metadata for better tracking
*** amitgandhinz has joined #openstack-infra10:02
*** ajmiller_ has joined #openstack-infra10:03
*** ajmiller has quit IRC10:03
*** kushal has quit IRC10:04
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute009.useast metadata for better tracking
*** lvdongbing has quit IRC10:04
*** abregman has quit IRC10:05
*** thorst_ has quit IRC10:06
*** tosky has joined #openstack-infra10:06
*** ajmiller_ has quit IRC10:06
*** ajmiller has joined #openstack-infra10:06
*** amitgandhinz has quit IRC10:06
*** salv-orlando has quit IRC10:09
*** sambetts|afk is now known as sambetts10:09
*** zqfan has joined #openstack-infra10:10
*** _degorenko|afk is now known as degorenko10:11
*** yanyanhu has quit IRC10:11
*** dtantsur|brb is now known as dtantsur10:13
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute011.useast metadata for better tracking
*** zhurong has quit IRC10:14
*** sarob has joined #openstack-infra10:14
*** jamielennox|away is now known as jamielennox10:15
*** javeriak has quit IRC10:16
openstackgerritAlina Nesterova proposed openstack-infra/project-config: Migrated sahara-client-functional-job
*** abregman has joined #openstack-infra10:18
*** sarob has quit IRC10:19
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute017.useast metadata for better tracking
*** javeriak has joined #openstack-infra10:19
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute019.useast metadata for better tracking
*** sorantis has quit IRC10:23
*** ociuhandu has joined #openstack-infra10:25
*** sirushti has quit IRC10:26
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute020.useast metadata for better tracking
*** sorantis has joined #openstack-infra10:26
*** sree has joined #openstack-infra10:29
*** jtomasek_ has joined #openstack-infra10:30
*** mikelk has quit IRC10:32
*** eezhova has quit IRC10:32
*** sirushti has joined #openstack-infra10:33
*** kushal has joined #openstack-infra10:34
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute026.useast metadata for better tracking
*** abregman has quit IRC10:38
openstackgerritJiri Tyr proposed openstack-infra/jenkins-job-builder: Adding property-file into the pipeline publisher
*** sdake has joined #openstack-infra10:40
*** e0ne has quit IRC10:41
*** e0ne has joined #openstack-infra10:41
*** yamamoto has quit IRC10:42
*** sdake_ has joined #openstack-infra10:42
*** esikachev has quit IRC10:42
openstackgerritTom Fifield proposed openstack-infra/project-config: Add governance-uc repo
*** sdake has quit IRC10:45
*** jtyr has joined #openstack-infra10:49
jtyrPlease could somebody review this very useful patch for Jenkins Job Builder? ...
*** esikachev has joined #openstack-infra10:54
*** ccamacho is now known as ccamacho|lunch10:56
*** HeOS has quit IRC10:57
*** kzaitsev_mb has joined #openstack-infra10:57
*** HeOS has joined #openstack-infra10:57
*** jtomasek_ has quit IRC10:59
*** eezhova has joined #openstack-infra10:59
*** kzaitsev_mb has quit IRC11:02
*** YorikSar has quit IRC11:02
*** roxanaghe has joined #openstack-infra11:02
*** amitgandhinz has joined #openstack-infra11:03
*** thorst_ has joined #openstack-infra11:03
*** YorikSar has joined #openstack-infra11:04
*** javeriak has quit IRC11:06
*** roxanaghe has quit IRC11:07
*** salv-orlando has joined #openstack-infra11:07
*** amitgandhinz has quit IRC11:07
*** salv-orlando has quit IRC11:10
*** sdake_ has quit IRC11:10
*** ramishra has quit IRC11:10
*** salv-orlando has joined #openstack-infra11:11
*** thorst_ has quit IRC11:11
*** ramishra has joined #openstack-infra11:16
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Enable Tempest test job also for sahara-tests
*** yamamoto has joined #openstack-infra11:22
*** esikachev has quit IRC11:26
*** YorikSar has quit IRC11:31
*** lucasagomes is now known as lucas-hungry11:31
*** abregman has joined #openstack-infra11:31
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add the ability to permalink to a comment
*** jed56 has quit IRC11:34
*** kaisers1 has joined #openstack-infra11:34
*** jed56 has joined #openstack-infra11:34
*** e0ne has quit IRC11:35
*** weshay has joined #openstack-infra11:37
openstackgerritFlavio Percoco proposed openstack-infra/tripleo-ci: Update Fedora Atomic URL
openstackgerritMerged openstack-infra/project-config: setup release notes job for sahara-dashboard
lifelessnotmorgan: cut out at 'looks like io.FileIO has'11:43
lifelessnotmorgan: saw your patch, would like a test, which needs to say which Python its relevant to so we know when we can drop it11:43
*** kaisers1 has quit IRC11:44
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Make task notes button clearer
*** salv-orlando has quit IRC11:45
*** kaisers1 has joined #openstack-infra11:45
*** salv-orlando has joined #openstack-infra11:46
*** ddieterly has joined #openstack-infra11:47
*** andreykurilin has quit IRC11:47
*** esikachev has joined #openstack-infra11:49
*** Julien-zte has quit IRC11:49
*** ramishra has quit IRC11:52
*** YorikSar has joined #openstack-infra11:52
*** thorst_ has joined #openstack-infra11:53
*** dprince has joined #openstack-infra11:54
*** claudiub has joined #openstack-infra11:55
*** thorst__ has joined #openstack-infra11:55
*** TheJulia_ is now known as TheJulia11:56
eliqiaoAJaeger: hi around? can you help to take a look at ? (zun irc renaming)11:58
*** thorst_ has quit IRC11:58
*** amoralej is now known as amoralej|lunch11:58
lifelessnotmorgan: that said, I thought we had direction specific tests - e.g. write('') vs write(b'') for write streams, so perhaps thats a better and longer term fix11:58
*** maishsk has joined #openstack-infra12:00
AJaegereliqiao: not right now, need to figure out a few bits first...12:02
*** ramishra has joined #openstack-infra12:03
*** roxanaghe has joined #openstack-infra12:03
*** ldnunes has joined #openstack-infra12:03
*** amitgandhinz has joined #openstack-infra12:03
*** shardy has quit IRC12:04
*** sdake has joined #openstack-infra12:05
*** ddieterly is now known as ddieterly[away]12:05
eliqiaoAJaeger: yes? can you tell a bit more?12:06
AJaegereliqiao: I don't have time to look at this right now and need to double check a few bits first12:07
AJaegereliqiao: sorry - hope I manage later today. But there are others to review as well...12:08
*** roxanaghe has quit IRC12:08
*** amitgandhinz has quit IRC12:08
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Enable Tempest test job also for sahara-tests
*** ramishra has quit IRC12:10
eliqiaoAJaeger: oh sorry, not in a hurry, thanks. Just take your time. I can check it tomorrow(my time)12:11
*** maishsk has quit IRC12:12
*** winggundamth_ has joined #openstack-infra12:12
*** ramishra has joined #openstack-infra12:12
*** claudiub has quit IRC12:12
*** rhallisey has joined #openstack-infra12:15
*** sarob has joined #openstack-infra12:15
*** slagle_ is now known as slagle12:16
*** ramishra has quit IRC12:16
*** samuelBartel has joined #openstack-infra12:17
*** claudiub has joined #openstack-infra12:19
*** sarob has quit IRC12:19
*** ramishra has joined #openstack-infra12:21
*** lucas-hungry is now known as lucasagomes12:22
*** jtyr has quit IRC12:23
*** gema has quit IRC12:23
*** rcernin has quit IRC12:24
*** gema has joined #openstack-infra12:25
*** zhurong has joined #openstack-infra12:25
*** dizquierdo has quit IRC12:26
*** ccamacho|lunch is now known as ccamacho12:26
*** kgiusti has joined #openstack-infra12:29
*** maishsk has joined #openstack-infra12:29
*** lvdongbing has joined #openstack-infra12:30
*** trown|outtypewww is now known as trown12:30
*** thiagop has joined #openstack-infra12:30
*** pradk has joined #openstack-infra12:31
*** ddieterly has joined #openstack-infra12:32
*** jcoufal has joined #openstack-infra12:34
openstackgerritTom Fifield proposed openstack-infra/project-config: Add governance-uc repo
*** javeriak has joined #openstack-infra12:34
*** pas-ha has joined #openstack-infra12:39
*** rcernin has joined #openstack-infra12:39
*** pas-ha has left #openstack-infra12:39
*** julim has joined #openstack-infra12:39
*** e0ne has joined #openstack-infra12:40
*** rlandy has joined #openstack-infra12:40
*** zul_ has joined #openstack-infra12:41
*** ddieterly is now known as ddieterly[away]12:41
*** ddieterly[away] has quit IRC12:41
*** yujunz has joined #openstack-infra12:41
*** zul_ has quit IRC12:42
*** javeriak has quit IRC12:42
*** zul_ has joined #openstack-infra12:42
*** zul_ has quit IRC12:43
*** zul_ has joined #openstack-infra12:44
*** zeih has quit IRC12:44
pabelangerinfra-root: looks like zl01 is no longer updating nodepool properly. I am going to graceful stop it for now until we schedule a nodepool reboot12:44
*** eharney has quit IRC12:44
*** edmondsw has joined #openstack-infra12:45
mordredpabelanger: k. is that a behavior we should debug?12:45
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a temporary tripleo cloud "rh2"
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a experimental tripleo job on rh2
pabelangermordred: Yes, going to look at the logs and see what happen. See if anything specific stands out12:46
*** rodrigods has quit IRC12:46
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds API reference publishing jobs for Swift
*** rodrigods has joined #openstack-infra12:46
*** isaacb has joined #openstack-infra12:46
*** baoli has joined #openstack-infra12:47
*** amitgandhinz has joined #openstack-infra12:48
*** amitgandhinz has quit IRC12:48
derekhIn  I'm hoping to at a new (temporary) deployment for tripleo , whats the best way to handover credentials?12:48
*** amitgandhinz has joined #openstack-infra12:49
*** zul_ has quit IRC12:49
*** annegentle has joined #openstack-infra12:49
*** zul has quit IRC12:49
isaacbyolanda: hi12:49
*** zul has joined #openstack-infra12:50
lennybjordanP: regarding , we are facing some difficulties debugging test_router_rescheduling that failed. Do you know who can assist us with it? Thanks12:51
*** winggundamth_ has quit IRC12:51
jordanPlennyb, do you have log of a failure ?12:51
jordanPI mean, a direct http link12:52
*** Julien-zte has joined #openstack-infra12:52
*** baoli_ has joined #openstack-infra12:53
isaacbAJaeger: Hi12:53
*** zeih has joined #openstack-infra12:53
*** rcernin has quit IRC12:54
*** ramishra has quit IRC12:54
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-diskimage_builder: Upodate pip provider to use openstack_pip provider
jordanPlennyb, you sure if you revert that OO patch, the test pass ?12:54
jordanPOr maybe another regression got in12:55
*** jsavak has joined #openstack-infra12:55
*** mikelk has joined #openstack-infra12:55
*** baoli has quit IRC12:55
jordanPnot, that unlikely, nothing else was merged recently12:56
lennybjordanP:  did not check it.12:56
*** Goneri has joined #openstack-infra12:57
jordanPin the log I see "2016-06-16 13:04:11,434 1628 DEBUG    [tempest.common.fixed_network] No valid network provided or created, defaulting to fixed_network_name", was this message present a couple of days ago ?12:58
*** gouthamr has joined #openstack-infra12:58
*** aysyd has joined #openstack-infra12:58
*** abregman is now known as abregman|intervi12:58
lennybjordanP:  sorry, I did check it.   with tempest cf2150a534659964255a9682ecb32c7b78943ea612:58
*** thorst__ is now known as thorst_12:58
*** ramishra has joined #openstack-infra12:58
pabelangermordred: jeblair: at first glance, it looks like nodepool was no longer receiving gearman updates from zl01.  Looks to be the same issue we have then jenkins loose a connection to nodepool.12:58
mordredpabelanger: interesting12:59
lennybjordanP:  I am running CI with temporal tempest git checkout cf2150a534659964255a9682ecb32c7b78943ea612:59
pabelangermordred: jeblair: zl01 was running jobs, nodepool was missing the used event and seeing them only as ready12:59
*** abregman|intervi has quit IRC12:59
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-jenkins: Upodate pip provider to use openstack_pip provider
pabelangerand zl01 was getting more ready nodes, nodepool was sending them just not receiving13:00
pabelangerI won't restart nodepool until jeblair is around to review13:00
isaacbpabelanger: can you assist with ?13:02
isaacbthis cause to fail13:02
isaacbthe script tries to clone non existent "charm-specs" repo13:03
*** eezhova has quit IRC13:03
pabelangerisaacb: where are you seeing the failure? Trying to see how they are related as is for our puppet modules13:03
*** roxanaghe has joined #openstack-infra13:04
*** gouthamr_ has joined #openstack-infra13:04
*** kushal has quit IRC13:04
isaacbpabelanger: I'm trying to create an image.13:04
*** kzaitsev_mb has joined #openstack-infra13:04
*** ramishra has quit IRC13:05
*** amoralej|lunch is now known as amoralej13:05
pabelangerisaacb: with diskimage-builder?13:05
*** winggundamth_ has joined #openstack-infra13:05
*** mescanef has quit IRC13:05
*** ramishra has joined #openstack-infra13:05
isaacbpabelanger: no. snapshot13:05
pabelangerokay, that aside, I can see doesn't exist yet13:05
pabelangerlet me look and see why it failed13:05
isaacbpabelanger: charm-specs is in github13:06
isaacbinstall_modules works with git.openstack.org13:06
*** banix has joined #openstack-infra13:06
pabelangerisaacb: URL?13:06
pabelangerI just looked at our nodepool DIB builds, they failed too13:07
pabelangerso something is wrong13:07
*** rcernin has joined #openstack-infra13:07
*** gouthamr has quit IRC13:07
*** psilvad has joined #openstack-infra13:08
isaacbyou can see that gerrit/projects.yaml point to github as upstream13:08
*** roxanaghe has quit IRC13:08
pabelangerisaacb: right, but it will get imported into openstack/charm-specs, using, that import process failed for some reason13:09
phschwartznibalizer: fungi:
*** e0ne has quit IRC13:09
phschwartznibalizer: fungi: those so far fix my issue. Looking to see if there are others.13:09
rcarrillocruzpabelanger, yolanda :heya, added a bunch of changes for infracloud inventory13:10
*** mdrabe has joined #openstack-infra13:10
yolandahi rcarrillocruz , seems you are having fun today! :)13:10
pabelangerrcarrillocruz: Nice, will review them shortly. Was also going to ping you this morning and see where things were with a controller13:11
*** Hal has joined #openstack-infra13:11
rcarrillocruznot now, but once they move the machines again13:11
yolandagoing to review13:11
rcarrillocruzwe will have to check with networking13:11
*** Hal is now known as Guest5817013:11
*** zeih has quit IRC13:11
rcarrillocruzwe want to have two different clouds, one vanilla another chocolate13:11
*** gouthamr has joined #openstack-infra13:11
rcarrillocruzfor potentially different versions, etc13:11
*** pgadiya has quit IRC13:11
rcarrillocruznot sure if we also want to split them in terms of provisioning13:12
*** e0ne has joined #openstack-infra13:12
rcarrillocruzbefore they were physically split, currently they are not13:12
rcarrillocruzso one baremetal00 dhcp to rule them all13:12
*** zeih has joined #openstack-infra13:12
rcarrillocruzyolanda: which baremetal did we use for east?13:13
rcarrillocruzas there were no config on system-config, i wonder where it is13:13
rcarrillocruzbaremetal00 shows just west machines on the ironic inventory13:13
wznoinskwould anyone know whether there will be any CI talks proposed by infra team for Barcelona?13:13
yolandai think baremetal0113:13
phschwartzyolanda: These are a couple of puppet changes if you don't mind a quick review.
*** psilvad has quit IRC13:14
*** hemanthm_ is now known as hemanthm13:14
rcarrillocruzwznoinsk: pabelanger was about to propose a thread for potential talks for Barcelona13:14
*** psilvad has joined #openstack-infra13:14
rcarrillocruzyolanda: ++, let me check13:14
*** gouthamr_ has quit IRC13:15
Shrewspabelanger: any chance zl01 was running an older version (before jun 8) of zuul?13:15
pabelangerShrews: I don't believe so, we did a gracefully restart last night. So we should have used the latest release at that time.  However, I will confirm13:15
*** eharney has joined #openstack-infra13:17
wznoinskrcarrillocruz: do you discuss these on infra meetings? I'd like to talk about what I'm thinking to submit - 3rd party CI, it may be a good one for collaboration13:17
*** pcarver_ has joined #openstack-infra13:17
*** mescanef has joined #openstack-infra13:17
Shrewspabelanger: don't have the access you do, so can't help you much. was just looking at recent zuul & nodepool changes for possible causes13:17
rcarrillocruzi think we agreed to discuss on the mailing list, but i don't see why you couldn't raise that topic on next meeting13:17
pcarver_Is anybody here familiar with the code that generates the IRC meeting entries on eavesdrop.o.o?13:18
rcarrillocruzif the thread has not gone live by the time, you could talk about it on open discussion too13:18
*** pcarver_ is now known as pcarver13:18
*** zeih has quit IRC13:18
wznoinskok, will see what's on ML and join meeting too, thanks13:18
*** zeih has joined #openstack-infra13:19
*** matt-borland has joined #openstack-infra13:19
*** mriedem has joined #openstack-infra13:19
*** psilvad has quit IRC13:19
pabelangerpuppet-user[8927]: (/Stage[main]/Openstack_project::Puppetmaster/Package[shade]/ensure) change from 1.9.0 to latest failed: Could not get latest version: HTTP-Error: 403 Must access using HTTPS instead of HTTP13:19
pabelangerDid we figure out the fix to ^ last night?13:20
pabelangerpuppet is failing to apply on puppetmaster.o.o which is stopping ansible from running properly13:20
phschwartzpabelanger: I just fixed 2 and we need to fix that one13:20
pabelangerphschwartz: okay, checking13:20
phschwartzpabelanger: there was a change yesterday to pypi where instead of redirecting http to https where it now 403's13:20
*** psilvad has joined #openstack-infra13:21
*** salv-orlando has quit IRC13:21
phschwartzpabelanger: I will put up one for that one, but we need to find the others13:21
*** salv-orlando has joined #openstack-infra13:21
pabelangerphschwartz: what is openstack_pip? first time seeing that13:21
phschwartzpabelanger: it is a provider that fixes the issue that was created by crinkle13:22
pabelangerya, quick google leads me to 30239613:22
anteayawznoinsk: what can I help you with?13:22
*** jcoufal has quit IRC13:23
*** pgadiya has joined #openstack-infra13:23
anteayawznoinsk: I'm here briefly then off running errands this morning13:23
*** bhavik1 has joined #openstack-infra13:23
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: fix the pattern matching in the constraint update logic
mordredphschwartz: thanks13:24
*** sree has quit IRC13:24
pabelangerisaacb: okay, found the issue. Looks to be an issue with our puppet wheel not running properly.  Until fixed, openstack/charm-specs will continue to fail13:25
*** sree has joined #openstack-infra13:25
openstackgerritPhilip Marc Schwartz proposed openstack-infra/system-config: Upodate pip provider to use openstack_pip provider
*** psilvad has quit IRC13:25
*** akshai has joined #openstack-infra13:25
*** pgadiya has quit IRC13:25
phschwartzpabelanger: ^^13:26
phschwartzpabelanger: that is fixing it in openstack_project13:26
*** rfolco has joined #openstack-infra13:26
phschwartzpabelanger: I am sure there will be others in the infra scope, I haven't gone throgh all puppet-* yet13:26
phschwartzmordred: hey, fixing it upstream is good. It is breaking me so why not help others :)13:26
pabelangerphschwartz: great, thanks13:27
*** akshai_ has joined #openstack-infra13:27
phschwartzpabelanger: just need to get those 3 reviews reviewed and merged in.13:27
phschwartzI will keep looking for others13:27
wznoinskanteaya: I was wondering whether you know anyone doing any talks about CI/3rdparty CIs in Barcelona (or plan to do it)13:27
*** raildo-a` is now known as raildo13:27
*** psilvad has joined #openstack-infra13:27
pabelangeryolanda: mordred: do you mind reviewing:
*** vincentll has quit IRC13:28
*** vincentll has joined #openstack-infra13:28
mordredpabelanger: done13:29
pabelangermordred: phschwartz: great, thank you13:29
phschwartzmordred: pabelanger: there might be more. I will alert on the topic if there are.13:30
*** sree has quit IRC13:30
pabelangerphschwartz: ack13:30
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Remove special config for west bifrost host
anteayawznoinsk: I am not aware of anyone13:31
*** akshai has quit IRC13:31
*** ramishra has quit IRC13:31
anteayawznoinsk: I'm guessing someone or multiple someones will put proposals forward, the talk submission process usually has many enteries, but noone has come to me or said anything that I have seen that allows me to say joe shmoe is putting in a talk13:32
*** roxanaghe has joined #openstack-infra13:32
isaacbpabelanger: what about: modules/openstack_project/manifests/template.pp13:32
phschwartzpabelanger: mordred: cloning all puppet-* modules we have in the projects set takes a long time. lol13:33
*** roxanaghe has joined #openstack-infra13:33
mordredphschwartz: right?13:33
wznoinskI'm planning to submit a talk about our journey with 3rdparty CI, I'm looking if anyone would want to join forces on that... I'll ask on the mailing list then13:34
phschwartzmordred: it has been going for over an hour. lol13:34
anteayawznoinsk: good idea13:35
phschwartzwznoinsk: I might be interested. Not 100% sure I will make it yet though. My wife is due any day so I am not sure I want to leave her alone with the 6 year old and a 4 month old as that will still be during the partial sleepless period.13:35
*** mtanino has joined #openstack-infra13:35
*** Na3iL has quit IRC13:36
*** eezhova has joined #openstack-infra13:36
isaacbwznoinsk: for which project of openstack you run the CI?13:36
*** ramishra has joined #openstack-infra13:37
pabelangerisaacb: I am not sure your question about template.pp13:37
wznoinskphschwartz how likely the delivery could be clashing with the prep/summit?13:38
wznoinskisaacb: neutron, nova, networking-ovs-dpdk, networking-odl, devstack  - Intel NFV CI13:38
phschwartzwznoinsk: she is due a couple days from now, aka 4 months old by the summit.13:39
*** zeih has quit IRC13:39
*** eezhova has quit IRC13:39
*** Na3iL has joined #openstack-infra13:39
pabelangerwznoinsk: anteaya: You might consider adding it to . See if any other people are interested in helping out13:40
wznoinskphschwartz: which CI you're running?13:40
*** dizquierdo has joined #openstack-infra13:40
*** ddieterly has joined #openstack-infra13:40
phschwartzwznoinsk: a mixed 3rd party + a full clone of infra for IBM Cloud, along with Rax prior.13:40
pabelangerwznoinsk: anteaya: nothing official by any means but a place for speakers to collaborate13:41
*** furlongm_ is now known as furlongm13:41
yolandasorry, meeting. Going through the review queue now13:42
*** eezhova has joined #openstack-infra13:42
*** inc0 has joined #openstack-infra13:42
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Fix baremetal00 name attribute still referencing west
anteayajust to be clear, I'm not endorsing a talk in third party ci13:43
anteayathose talks tend not to make the roster that I have seen13:43
anteayabut I answered the question posed to me to the best of my ability13:43
wznoinskanteaya you're worried we won't have enough interest in the topic?13:44
anteayathat isn't what I said13:44
*** ddieterly is now known as ddieterly[away]13:44
anteayayou asked me a question and I answered it13:45
anteayaI'm not endorsing anyone's talk proposal13:45
*** _sigmavirus24 is now known as sigmavirus2413:46
*** sigmavirus24 has joined #openstack-infra13:46
*** abregman has joined #openstack-infra13:47
*** ddieterly[away] is now known as ddieterly13:49
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Replace hpuswest name for vanilla
*** rbrndt has joined #openstack-infra13:50
wznoinskanteaya: understood, thanks13:51
anteayawznoinsk: thank you13:52
*** jcoufal has joined #openstack-infra13:52
anteayamy wish for you and for everyone submitting any talk proposal is that you clearly state your intent and may you give a good presentation should you be selected13:52
*** aviau has quit IRC13:54
*** aviau has joined #openstack-infra13:54
*** tpsilva has joined #openstack-infra13:58
*** hongbin has joined #openstack-infra13:58
*** xarses_ has quit IRC13:59
anteayaI'm off doing errands today, back later14:00
*** garyk1 has quit IRC14:01
*** woodster_ has joined #openstack-infra14:01
jeblairpabelanger, mordred: one thing that might be helpful if this happens again is to run a zmq listener process on the zuul server connecting to localhost to see if the problem is that zl is not transmitting or whether nodepool is not listening.14:02
pabelangerjeblair: can we try that now?  We can bring zl01 back online and see what happens14:03
wznoinskpabelanger: I've replied in the above 'infa' etherpad14:03
isaacbCore reviewers: can someone please review:
pabelangerwznoinsk: ack14:03
*** lvdongbing has quit IRC14:03
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Replace hpuseast name for chocolate
jeblairpabelanger: sure14:05
*** abregman has quit IRC14:05
pabelangerokay, bringing zl01 back online14:05
jeblairi'm running a stream listener14:05
*** annegentle has quit IRC14:07
*** nwkarsten has joined #openstack-infra14:07
*** PollyZ has joined #openstack-infra14:07
*** bhavik1 has quit IRC14:07
pabelangerokay, zl01 got a job14:07
*** infra-red has joined #openstack-infra14:08
jeblairit's receiving events14:08
jeblairer sending them rather14:08
fungijeblair: pabelanger: rather than running an extra zmq listener, couldn't you just fire up tcpdump?14:09
jeblairfungi: that would probably work as well14:10
jeblairpabelanger, fungi: nodepool did not get that event14:10
jeblairso it seems to be on the nodepool side14:10
jd__hello guys; could you add myself in panko-core so I can add more team members and merge patches? :)14:10
fungiis each listener in nodepool a separate thread?14:10
*** infra-red has quit IRC14:11
jeblairpabelanger, fungi: i'm running tcpdump on nodepool14:11
fungijd__: no problem--just a sec14:11
jeblairfungi: i believe so (zmq is a bit of a black box about that sort of thing)14:11
fungijd__: i've added you to panko-core and panko-stable-maint14:12
fungijeblair: just wondering whether a thread dump of nodepool would yield anything useful (assuming we could even tell which thread went with which publisher)14:13
jd__thanks fungi!14:13
fungijd__: happy hacking!14:14
fungijd__: oh, also AJaeger pointed out a couple of stuck jobs for that repo14:14
fungijust a sec and i'll see if that's still the case14:14
jd__fungi: yeah I just send a patch to fix those jobs :)14:14
fungioh, perfect!14:14
jd__ makes it all green :)14:14
jd__thanks AJaeger for noticing btw14:14
fungijd__: oh, not failing but actually stuck waiting for a worker14:16
fungipanko-upstream-translation-update (in post) and panko-propose-translation-update (in periodic)14:16
jeblairfungi, pabelanger: tcpdump on nodepool is not seeing traffic from zl0114:16
fungijeblair: have tcpdump running on zl01?14:17
jeblairfungi: not yet14:17
*** zeih has joined #openstack-infra14:17
fungiassuming no firewall or network connectivity issues, it's probably not emitting anything14:17
*** ri0 has joined #openstack-infra14:18
jeblairfungi: agreed; it's likely the zmq bits in nodepool stuck, and not having an active connection14:18
jeblairyep, tcpdump on zl01 shows nothing as well14:18
fungijeblair: ooh, wonder if we have a half-dead socket and there's no keepalive/timeout for it14:18
jeblairfungi: maybe so14:18
*** zz_dimtruck is now known as dimtruck14:18
pabelangerfungi: Ah, right.14:18
funginetstat showing anything for that source/destination?14:18
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-ansible: Update pip provider to use openstack_pip provider
*** abregman has joined #openstack-infra14:19
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-bandersnatch: Update pip provider to use openstack_pip provider
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-gerritbot: Update pip provider to use openstack_pip provider
*** yaume has quit IRC14:19
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-jeepyb: Update pip provider to use openstack_pip provider
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-lodgeit: Update pip provider to use openstack_pip provider
*** daemontool has joined #openstack-infra14:19
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-log_processor: Update pip provider to use openstack_pip provider
jeblairfungi: no, actually14:19
*** yaume has joined #openstack-infra14:19
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-logstash: Update pip provider to use openstack_pip provider
*** winggundamth_ has quit IRC14:20
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-meetbot: Update pip provider to use openstack_pip provider
jeblairso this is probably all on zmq14:20
fungihuh, so really just not connected at all14:20
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-odsreg: Update pip provider to use openstack_pip provider
jeblairit's why i want to get rid of zmq and never touch it again in infra14:20
fungiyes. i totally get that14:20
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-subunit2sql: Update pip provider to use openstack_pip provider
openstackgerritPhilip Marc Schwartz proposed openstack-infra/puppet-zuul: Update pip provider to use openstack_pip provider
pabelangerjeblair: this is what zookeeper is going to replace right?14:21
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: return nodes to nodepool via gearman
*** kaisers_ has joined #openstack-infra14:21
phschwartzpabelanger: mordred: yolanda: fungi:
*** burgerk has joined #openstack-infra14:21
phschwartzabout 13 more reviews for infra related things moving to openstack_pip to fix the pypi issue14:22
jeblairpabelanger: yes, i hope.  we haven't actually agreed on that / landed that spec yet, because we're going to try it out in nodepool first14:22
jeblairpabelanger, fungi, mordred: i'm thinking about doing this as a temporary measure to increase our reliability: Ansible launcher: return nodes to nodepool via gearman
mordredjeblair: looking14:24
mordredphschwartz: thanks! those all look great14:24
fungiokay, this is weird. build:panko-upstream-translation-update and build:panko-propose-translation-update are registered in gearman with no available workers14:24
fungizlstatic01 should be reporting proposal.slave.o.o as available to run them, right?14:25
jeblairfungi: theoretically14:26
fungipresumably it was available at some point because it registered it into gearman14:27
*** zeih has quit IRC14:27
*** jistr is now known as jistr|mtg14:28
*** ifarkas has quit IRC14:28
*** bhavik has joined #openstack-infra14:28
jeblairi'm going to get the worker list from gearman14:28
jeblairnobody else do that because it's huge14:29
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Move gate-manila-buildimage-docker job to check pipeline
*** ri0 has quit IRC14:29
*** amotoki has joined #openstack-infra14:29
jeblairyeah, it is not on proposal's list of jobs14:30
jeblairbut proposal does have many translation-update jobs14:31
fungihowever zlstatic01 does seem to have an up to date jjb config with panko instantiating the translation-jobs group14:32
jeblairjjb seems to want to create that job14:33
jeblairi will point my local zuul-launcher at the config and see if it wants to register it14:33
*** EricGonczer_ has joined #openstack-infra14:34
*** ramishra has quit IRC14:34
jeblairthat's local ^14:36
*** ramishra has joined #openstack-infra14:37
mordredoh - out of date files on zuul launcher14:37
jeblairmordred: which ones?14:37
mordredwe're still recovering from the pip outage causing puppet to not run14:37
fungiis it possible puppet put the jjb configs on teh launcher but didn't exec jjb?14:38
mordredjeblair: the copies of the jjb files on the launcher host I image are not up to date14:38
jeblairmordred: fungi checked the jjb config, and zuul reconfigured at 2016-06-16 11:52:44,95514:38
mordredoh. bother. don't listen to me14:38
*** sarob has joined #openstack-infra14:38
jeblairfungi: we don't need to run jjb, just update the files and reconfigure zuul ^14:38
fungibut yeah, broken puppet because of the pypi behavior change has been lingering in the back of my mind14:38
*** hongbin has quit IRC14:39
fungijeblair: oh, right, zl imports jjb as a lib14:39
jeblairfungi: was this recently added?14:39
fungioldthink dies hard14:39
fungijwbdded yesterday14:39
fungiadded yesterday14:39
jeblairoh... i may have an idea...14:40
jeblairi think we may only re-register jobs if a node is not running a job14:40
jeblairwhich makes sense for a single-use node, but not so much for a long-running one14:41
jeblairthe most recent reconfiguration *was* during a proposal job14:41
*** pcaruana has quit IRC14:41
jeblair2016-06-15 17:39:12,186 DEBUG Received reconfigure request14:42
fungiwould definitely explain it14:42
jeblairthat was the most recent reconfiguration received outside of running a job14:42
*** jamesmcarthur has joined #openstack-infra14:42
*** sarob has quit IRC14:43
notmorganlifeless: ack and i have a test just was super tired at 2am14:43
notmorganlifeless: so, i'll revisit14:43
fungiyeah, i believe that was before the panko patch merged and got puppeted14:43
jeblairfungi: it's possible the registration in gearman came from jenkins.o.o14:44
*** trown is now known as trown|brb14:44
jeblairokay, i think i can fix this fairy quickly14:44
*** kaisers_ has quit IRC14:44
fungimakes sense14:45
*** mixos has joined #openstack-infra14:45
*** gordc has joined #openstack-infra14:45
*** kmartin has joined #openstack-infra14:45
notmorganjeblair: doh. but that makes sense based upon what you're saying14:45
*** xarses_ has joined #openstack-infra14:47
*** roxanaghe has joined #openstack-infra14:47
*** kaisers_ has joined #openstack-infra14:48
*** sarob has joined #openstack-infra14:49
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch back to current-tripleo pre promotion
*** jaosorior has quit IRC14:49
*** mriedem1 has joined #openstack-infra14:50
*** EricGonczer_ has quit IRC14:50
*** bhavik has quit IRC14:50
openstackgerritMário Santos proposed openstack-infra/shade: Add support for changing metadata of compute instances
*** bhavik has joined #openstack-infra14:50
*** mriedem has quit IRC14:51
jeblairfungi: oh, you can probably run 'zuul-launcher reconfigure' if proposal is not running a job right now and the immediate problem will be fixed14:51
jeblairfungi: er, it is running a job14:51
jeblairfungi: but when it goes idle, you could do that14:51
fungiwill do--thanks!14:51
jeblairfungi: "nc 8088" :)14:51
*** roxanaghe has quit IRC14:52
*** trown|brb is now known as trown14:52
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: update registration between jobs
jeblairfungi, notmorgan, mordred: ^14:52
*** jlanoux_ has quit IRC14:53
*** hongbin has joined #openstack-infra14:53
jeblairoh, i want some debug lines in there, one sec14:53
mordredjeblair: looks good in general, pending debug lines14:53
*** sarob has quit IRC14:54
notmorganjeblair: looking14:54
*** sdake has quit IRC14:54
*** bhavik has quit IRC14:55
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: update registration between jobs
jeblairthere we go14:55
jeblairthey also serve as comments :)14:56
*** tphummel has joined #openstack-infra14:56
notmorganjeblair: much better w/ debug14:56
*** salv-orlando has quit IRC14:56
*** dizquierdo has quit IRC14:56
*** ramishra has quit IRC14:57
*** psachin has quit IRC14:57
*** amotoki_ has joined #openstack-infra14:57
notmorganjeblair: lgtm.14:57
*** salv-orlando has joined #openstack-infra14:57
*** mriedem1 is now known as mriedem14:57
*** dtantsur is now known as dtantsur|bbl14:58
*** ramishra has joined #openstack-infra14:59
*** chem has quit IRC14:59
*** zhurong has quit IRC14:59
*** sdake has joined #openstack-infra14:59
*** maishsk has quit IRC15:00
*** salv-orlando has quit IRC15:00
*** chem has joined #openstack-infra15:00
*** amotoki has quit IRC15:00
*** salv-orlando has joined #openstack-infra15:00
*** mixos has quit IRC15:00
*** david-lyle_ is now known as david-lyle15:00
fungiwe seem to have a glut of ready nodes. i'm in a meeting right now but can try to look into it in a bit if nobody beats me to the punch15:02
*** Na3iL has quit IRC15:02
pabelangeroh, we need to take zl01 out of server again15:03
pabelangerI can do that now15:03
pabelangerjeblair: ^ are you finished with tcpdump?15:03
jeblairpabelanger: yep15:03
zigopabelanger: Hi there!15:03
zigopabelanger: Did you have a chance to work on the debian repo thing?15:03
pabelangerzl01 in graceful mode15:03
zigopabelanger: Because it hasn't went far enough, now the TC wants to remove my project from the big tent:
pabelangerzigo: sorry now, been head down working on our zuul-launcher migration15:04
zigopabelanger: So we need to show progress.15:04
zigopabelanger: We *just* need that repo stuff, then I can do the rest alone with a few review from infra.15:04
zigopabelanger: If it's not you, who can I ask?15:04
zigoMonty didn't help at all. :(15:05
*** yujunz has quit IRC15:05
*** cbader_ has quit IRC15:05
odyssey4mezigo what do you need done?15:06
*** amotoki_ has quit IRC15:06
zigoodyssey4me: Packages are currently being published on each merge jobs, like this:
zigoodyssey4me: We need a pickup job that would store these packages into a Debian repo.15:07
pabelangerzigo: Ya, I don't have the bandwidth at the moment.  Apologies for that. My only suggestion at the moment would be to see how to create the repo external to infra at the moment.  Then when bandwidth opens up, get our per project reprepro going again15:07
zigoThen make that debian repo available for other packages to use.15:07
zigopabelanger: We already have that...15:07
pabelangerOr publisher the reprepro bits into static.o.o like we do for our periodic centos-7 afs client15:07
zigoThough there's then no point.15:07
*** markmcd has quit IRC15:08
zigoI'm not sure what that last line means.15:08
odyssey4mezigo ah, so you need to build a repo based on a set of published packages... ?15:08
banixHad a question about a gate job. Cam I add a file to /etc/sudoers.d for a gate job? If yes, how do I go about doing it?15:08
zigoodyssey4me: I need a repo to pickup packages produced after a merge of the packaging-deb git repo, yes.15:09
lennybjordanP:  revert passed
*** kaisers_ has quit IRC15:09
*** kzaitsev_mb has quit IRC15:09
zigoodyssey4me: I've been told to use the AFS of infra, but that's all I know, I have no idea how to integrate with the rest of infra.15:09
*** kaisers_ has joined #openstack-infra15:10
*** markmcd has joined #openstack-infra15:10
*** markmcd1 has joined #openstack-infra15:10
*** mattymo_ is now known as mattymo15:10
odyssey4mezigo hrm, that's a little out of my league at this point... I thought it might just be putting together another repo - if so, I've put together two patches which pabelanger needs to review *hint* ;) and it hasn't been too bad... there's some prior art15:10
jordanPlennyb, ack. Unfortunately I think we can't revert this patch. It's really useful... I feel you have to keep looking for the root cause of the CI failure15:10
zigoodyssey4me: Could you point to me at such a patch?15:11
odyssey4mezigo pabelanger perhaps what may be useful is if we can put a time slot together to understand what needs to be done better and to point at prior art... then perhaps you can put a review together yourself and maybe I can help15:11
zigoThat would be nice.15:12
odyssey4mezigo these are my two new repo mirror requests:
zigoAnything going further would be nice at this point.15:12
odyssey4mezigo just seeing the files that get changed and inspecting them might give you enough to get started15:12
zigoodyssey4me: That public key block is abnormally big.15:13
zigogpg --armor --export-options export-clean,export-minimal --export 0x<KEY-ID> is the way to do it.15:14
zigoWithout the --export-options, then you get such large key.15:14
odyssey4mezigo are they? I quite possibly did something wrong there... I'm happy to receive feedback in the review for improvements :)15:14
odyssey4meah ok, I did an apt-key export15:14
*** ramishra has quit IRC15:14
odyssey4mecan you add that to the review please so that I can pick that change up later15:14
* odyssey4me is technically in a meeting :p15:15
zigoHere's my public key it's a 4096 key as well, see how smaller it is.15:15
zigoAnd I got *A LOT* of signatures on this key, plus a photo, etc.15:15
*** ramishra has joined #openstack-infra15:16
*** piet has joined #openstack-infra15:16
zigoWith the --export-options export-clean,export-minimal, my public key is 51 lines, without, its 388111 lines...15:16
lennybjordanP:  in that case, please approve since that fix is ok. We will see what causes 1 test to fail15:17
openstackgerritMerged openstack-infra/puppet-diskimage_builder: Upodate pip provider to use openstack_pip provider
*** bhavik has joined #openstack-infra15:17
pabelangerodyssey4me: zigo: Right now, we need to bike shed of how this looks in AFS.  I think we agreed to deb-openstack in AFS, with that in mind, we'd basically do the same process as we do with our wheel-build / wheel-release jobs15:17
pabelangerwheel-build is already done by zigo15:18
pabelangerso the -release part is downloading the bits from tarball.o.o, copy them into /afs/.openstack/... then run afs release15:18
*** jistr|mtg is now known as jistr15:19
odyssey4mezigo thanks for the feedback :)15:19
jeblairpabelanger: any reason not to restart nodepool?15:20
*** wojdec has joined #openstack-infra15:21
pabelangerzigo: odyssey4me: The work I was going to do, create new builder based on wheel-build builder (jenkins/jobs/wheel-mirror.yaml), update it to download package bits into right directory, then add afs-release builder to add to AFS15:21
*** EricGonczer_ has joined #openstack-infra15:21
*** maishsk has joined #openstack-infra15:21
pabelangerjeblair: no, nodepool-builder failed so we are fine15:21
pabelangerzigo: odyssey4me: that's basically what is left to do15:22
*** dizquierdo has joined #openstack-infra15:22
wojdecHi Folks. I've hit an issue with the Ostack gerrit system: When submitting a patch I'm told I need to provide contact information. On entering contact information via the dashboard, I get a "Server Error - Cannot Store Information"15:23
wojdecIs this the right forum to raise this, if not where should I report it?15:24
*** ifarkas has joined #openstack-infra15:24
openstackgerritMerged openstack-infra/system-config: Upodate pip provider to use openstack_pip provider
*** ihrachys has quit IRC15:25
*** tphummel has quit IRC15:25
*** ihrachys_ has joined #openstack-infra15:25
zigoI'll have a look to that job.15:25
*** jlanoux has joined #openstack-infra15:25
*** maishsk_ has joined #openstack-infra15:27
*** Swami has joined #openstack-infra15:27
*** maishsk has quit IRC15:28
*** maishsk_ is now known as maishsk15:28
*** Swami has quit IRC15:29
*** andreas_s has quit IRC15:29
*** wojdec has quit IRC15:29
*** hdaniel has quit IRC15:31
*** andreww has joined #openstack-infra15:33
*** andreww has quit IRC15:33
*** xarses_ has quit IRC15:33
fungiwow, people /join, ask questions and then /part again 5 minutes later15:33
*** andreww has joined #openstack-infra15:34
*** openstackgerrit has quit IRC15:34
*** GheRivero has left #openstack-infra15:34
fungibefore anyone has a chance to answer them15:34
*** openstackgerrit has joined #openstack-infra15:34
bkeroI feel that's standard IRC newbie behavior15:34
bkerorule 1) don't ask to ask, rule 2) stick around for an answer15:34
fungiit's standard makes-me-hard-to-care behavior at any rate15:34
fungier, makes -it-hard-for-me-to-care15:35
*** mikelk has quit IRC15:35
*** EricGonczer_ has quit IRC15:35
*** vincentll has quit IRC15:35
odyssey4mepabelanger ah, so perhaps the name decision needs to be something discussed and decided in the next infra meeting?15:35
*** eezhova has quit IRC15:36
clarkbpabelanger: all image builds failed this morning?15:36
odyssey4methen perhaps zigo can try to fathom how to do it on his own, with consultation to yourself and perhaps some others who know how the spaghetti falls :)15:36
fungiclarkb: probably the pip provider issue?15:36
pabelangerclarkb: yes, puppet apply is currently broken15:36
clarkbpabelanger: also I recall questions about ovh images, is it ossible that it is related to the switch to qcow2 over raw?15:36
pabelangerworking on fixing it15:36
pabelangerclarkb: Ohh..... maybe. I think I might know the problem15:37
pabelangerclarkb: I need to check something15:37
*** _nadya_ has quit IRC15:37
pabelangercrinkle: phschwartz: possible bug in openstack_pip?
fungiclarkb: short story is that pypi switched its redirect behavior for http->https, which broke puppet<=3.x pip package provider's ability to scrape the xml api15:37
*** eharney_ has joined #openstack-infra15:38
mordredclarkb: yes. fun is a word in the english language15:38
mordredpabelanger: I agree- looks like a bug15:38
*** Guest58170 has quit IRC15:38
fungiclarkb: there are i hear fixes being proposed upstream in puppet for it, but also crinkle's rewritten pip provider is sane and does not suffer this blight so we're switching everything to that15:38
*** Hal has joined #openstack-infra15:38
clarkbnew house gets internets today and we have started migrating boxes of stuff so likely not to be here long15:39
*** eharney has quit IRC15:39
mordredclarkb: just use containers15:39
*** Hal is now known as Guest4185115:39
clarkbmordred: yes if I had each room of my house deployed as a cloud native containeried application moving would be so easy15:39
greghaynesYa, containers are good for shipping I hear, maybe they work for moving too15:39
fungiboxes are containers, and a house is really just a bigger container to put your smaller containers in15:39
phschwartzpabelanger: hmm, I am looking and to note that didn't error on my review to puppet-diskimage_builder and system-config15:39
clarkbI could just clone all of my furniture, ship them to new location, and have an orchestration engine migrate for me15:39
pabelangerclarkb: no, I think raw images are fine. I looked just now and images uploaded properly and displayed in nodepool correctly today15:40
clarkbpabelanger: you mean qcow2s are ok?15:40
pabelangerclarkb: sorry, yes qcow215:40
mordredphschwartz: perhaps the others didn't use the ensure => parameter?15:40
pabelangerI manually upload ubuntu-trusty yesterday for AJaeger.  Then nodepool did its uploads too15:41
pabelangermordred: I think that is correct15:41
pabelangerphschwartz: mordred: ansible-role-puppet should fail too because we set ensure => ''15:41
phschwartzmordred: yeah, the others were ensure => latest15:42
*** EricGonczer_ has joined #openstack-infra15:42
*** maishsk has quit IRC15:42
nibalizerphschwartz: did you see that failed15:42
phschwartzmordred: so I will need to look at crinkle's code to see if it handles it correctly15:42
*** kzaitsev_mb has joined #openstack-infra15:42
jeblairpabelanger: restarted nodepool15:42
phschwartznibalizer: yeah, that is what we are discussing right now15:43
fungiaha, looks like we have a bunch of stale ready nodes on jenkins05, 06, 07 and zl0115:43
*** eezhova has joined #openstack-infra15:43
jeblairfungi: maybe the nodepool restart will take care of the jenkinses too15:43
phschwartznibalizer: looks like crinkle's openstack_pip provider doesn't support versioned installs only latest.15:43
*** markusry has quit IRC15:43
fungijeblair: entirely possible15:43
jeblairpabelanger, fungi: can you clean up those stale nodes while i go test my zuul patches?15:43
fungijeblair: we also have a bunch of stale delete nodes on the jenkins masters15:44
*** eharney_ is now known as eharney15:44
pabelangerjeblair: sure15:44
fungijeblair: so i'm worried that "cleanup" involving nodepool delete will just mean more stuck deletes15:44
pabelangerfungi: ya, I notices that today too. Think our clouds are having some hiccups this morning15:44
*** shahid_ has quit IRC15:45
*** tphummel has joined #openstack-infra15:45
fungii guess if we delete the jenkins-assigned ones from nodepool's db, then the alien cleanup pass will get rid of them15:45
*** kzaitsev_mb has quit IRC15:46
pabelangerI usually grep for used nodes that are older then 3 hours, then delete them.  But would be curious if alien cleanup does the same thing15:46
*** johnny_ has joined #openstack-infra15:47
fungithe alien cleanup will only remove instances which are not in nodepool's db15:47
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/system-config: OpenStack-CI Apt mirror for MariaDB
fungiit's basically node leak mitigation15:47
*** roxanaghe has joined #openstack-infra15:48
nibalizerso I don't really understand this subsystem but this is an odd bug for sure15:48
nibalizerbecause all the openstack_pip provider is supposed to be doing is just inheriting the pip provider and overwriting latest15:48
fungiactually, the glut of stuck deletes on jenkinses seems to be clearing up now15:49
*** mdrabe has quit IRC15:49
fungipabelanger: were you working on cleaning up stale ready nodes, or shall i?15:50
crinklephschwartz: it inherits from the upstream one and only overrides the latest method15:50
crinkleit otherwise works the same15:50
*** reduxio has joined #openstack-infra15:51
nibalizercrinkle: yah?15:51
nibalizermaybe the capabilities statement doesn't get inherited?15:51
reduxioHello, Can anyone help with third party CI setup?15:51
nibalizerim running a test with just declaring the capabilities in the openstack_pip provider :shrug:15:51
*** ddieterly is now known as ddieterly[away]15:51
crinklenibalizer: oh you said that15:51
crinklenibalizer: uh i'm not sure15:51
*** ddieterly[away] is now known as ddieterly15:51
phschwartzcrinkle: ok, so that is where the issue is. The upstream one checks for ensure to be a value or latest and performs an action based on that. The openstack_pip doesn't ever look at the value from what it looks like15:52
phschwartzcrinkle: nibalizer:
crinklephschwartz: like i said this one is just a subclass of the upstream one15:52
jeblairfungi: i note zmq events from jenkins07 now15:52
*** roxanaghe has quit IRC15:53
*** sorantis has quit IRC15:53
openstackgerritSpencer Krum proposed openstack-infra/puppet-pip: Declare capabilities in openstack_pip provider
nibalizercrinkle: so in one quick test ^^ that does work15:53
crinklephschwartz: the latest method only checks what the latest available is, it doesn't mess with the install method15:53
crinklenibalizer: okay15:54
*** coreyob has quit IRC15:54
nibalizercrinkle: idunno, feels wrong to do it that way?15:54
nibalizeror put another way, it feels like we shouldn't have to do that15:54
fungipabelanger: i'm going to go ahead and delete the stale ready nodes (any in a ready state for >10 minutes)15:54
crinklenibalizer: no i think it makes sense since we have the commands thing there too15:54
nibalizer:shrug: ok15:54
nibalizershould i update and remove the question mark from the commit message?15:55
crinklenibalizer: yes please15:55
mordrednibalizer: I dunno. I like commits that ask questions15:55
*** isaacb has quit IRC15:56
openstackgerritSpencer Krum proposed openstack-infra/puppet-pip: Declare features in openstack_pip provider
*** e0ne has quit IRC15:56
pabelangerfungi: becareful of zl01, they are likely running jobs15:56
pabelangerfungi: okay, looks like zl01 is good15:57
nibalizercrinkle: mordred ^15:57
*** fernnest has joined #openstack-infra15:57
*** andreww is now known as xarses15:57
mordrednibalizer: yup. it has been +Ad15:57
fungipabelanger: oh, were we still not getting zmq events from zl01? i thought that had been resolved15:57
reduxioanteaya: Hi anteaya, i was told you may be able to help me with setting up third party CI, or at least point me in the right direction?15:57
pabelangerfungi: Ya, looks like it stopped correctly before your deletes15:58
pabelangerjeblair: I'll hold off on restart zl01 until you are ready15:58
jeblairpabelanger: i'm ready15:58
*** akshai has joined #openstack-infra15:58
*** Apoorva has joined #openstack-infra15:58
pabelangerzl01 restarted15:59
nibalizerhuh, my gerrit now longer shows the table of votes?15:59
clarkbnibalizer: you are probably on an older patchset then15:59
*** bhavik1 has joined #openstack-infra16:00
nibalizer</3 gerrit16:00
*** bpokorny has joined #openstack-infra16:00
*** tesseract has quit IRC16:00
*** reduxio has quit IRC16:00
*** mriedem1 has joined #openstack-infra16:00
*** kaisers_ has quit IRC16:01
*** dprince has quit IRC16:01
*** akshai_ has quit IRC16:01
*** dprince has joined #openstack-infra16:02
*** mriedem has quit IRC16:02
*** bhavik has quit IRC16:02
*** samuelBartel has quit IRC16:02
*** bhavik1 is now known as bhavik16:02
*** mriedem1 is now known as mriedem16:03
*** bpokorny has quit IRC16:05
*** bpokorny has joined #openstack-infra16:06
*** lucasagomes is now known as lucas-afk16:06
*** gyee has joined #openstack-infra16:06
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a temporary tripleo cloud "rh2"
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a experimental tripleo job on rh2
*** hashar is now known as hasharAway16:07
rcarrillocruznibalizer , clarkb , mordred : when you get a sec,
*** mescanef has quit IRC16:08
rcarrillocruzbeen testing it with logdest, without, with /var/log/ansible-puppet and without16:09
*** javeriak has joined #openstack-infra16:09
*** ansiwen_ is now known as ansiwen16:10
*** akshai_ has joined #openstack-infra16:10
*** roxanaghe has joined #openstack-infra16:10
*** mdrabe has joined #openstack-infra16:11
*** akshai has quit IRC16:13
*** javeriak_ has joined #openstack-infra16:14
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add Zuul as a first-party CI name
jeblairmordred, fungi, pabelanger: ^ i created a Zuul account in gerrit.  this should let us start using that to report on changes16:15
*** kzaitsev_mb has joined #openstack-infra16:15
*** javeriak has quit IRC16:15
*** permalac has quit IRC16:15
clarkbyou don't want to keep "jenkins" around in name if not executables?16:15
fungiclarkb: we could switch back to hudson16:16
pabelangerjeblair: nice16:16
jeblairi think that the time when we stop running jenkins is a good time to make the gerrit account name switch16:16
jeblair(while it would be amusing to keep calling it jenkins, it could be confusing)16:17
*** coreyob has joined #openstack-infra16:17
mordredjeblair: ++16:19
pabelangerI still need to update out dib elements for devstack cache adding support for /home/zuul16:19
pabelangershould dig into that later today16:19
*** johnny_ has quit IRC16:19
*** dprince has quit IRC16:20
*** sdake has quit IRC16:20
*** maishsk has joined #openstack-infra16:21
openstackgerritMerged openstack-infra/puppet-log_processor: Update pip provider to use openstack_pip provider
*** dprince has joined #openstack-infra16:21
nibalizerjeblair: so lets rename it to hudson...16:22
*** ramishra has quit IRC16:22
jeblairnibalizer: fungi already made that joke.  ;)16:22
openstackgerritScott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend
* nibalizer hangs head in shame16:23
* fungi prides himself at being quick with terrible jokes16:23
openstackgerritMerged openstack-infra/puppet-lodgeit: Update pip provider to use openstack_pip provider
openstackgerritMerged openstack-infra/puppet-meetbot: Update pip provider to use openstack_pip provider
jeblairmordred: hrm, i don't think there's a good way to handle the name change for the 'clean-check' gate pipeline requirement.16:24
openstackgerritMerged openstack-infra/puppet-gerritbot: Update pip provider to use openstack_pip provider
*** jsavak has quit IRC16:24
openstackgerritMerged openstack-infra/puppet-subunit2sql: Update pip provider to use openstack_pip provider
mordredjeblair: hrm16:24
mordredjeblair: that sucks a little bit16:24
jeblairmordred: it will require us to either synthetically create a bunch of zuul votes that == jenkins votes in gerrit, or alter zuul to support regexes in usernames16:25
mtreinishjeblair: if you rename the jenkins user that will probably break e-r:
mtreinishthere might be other things out there, which are looking for something similar too16:25
jeblairmtreinish: i'm sure there are16:26
*** ramishra has joined #openstack-infra16:26
*** jsavak has joined #openstack-infra16:27
openstackgerritScott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend
*** cbader has joined #openstack-infra16:28
*** sdake has joined #openstack-infra16:28
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul: Use safe_dump() for Ansible playbooks
*** yamamoto has quit IRC16:29
*** ddieterly is now known as ddieterly[away]16:29
openstackgerritJames E. Blair proposed openstack-infra/elastic-recheck: Support Zuul as a Gerrit user equivalent to Jenkins
jeblairmtreinish: ^16:29
mtreinishjeblair: actually asselin has a patch which makes that a config var:
mtreinishwe probably should just land that and update it in the puppet16:31
*** dtantsur|bbl is now known as dtantsur16:32
mtreinishalthough your patch will have a more graceful transition16:32
jeblairmtreinish: okay.  since that's a binary, we'll probably have a short 'outage' during the transition.  but we can try to land it close to the event16:32
*** esikachev has quit IRC16:32
*** matrohon has quit IRC16:33
*** ramishra has quit IRC16:34
mtreinishjeblair: we can also land your patch first, and switch to asselin's after the rename is done16:34
mtreinishdoesn't bug me either way16:34
*** david-lyle has quit IRC16:35
*** EricGonc_ has joined #openstack-infra16:35
jeblairmtreinish: i +2d asselin's.  that plan sounds like the best approach (no critical timing).  your call.16:36
*** pvaneck has joined #openstack-infra16:36
openstackgerritMerged openstack-infra/puppet-bandersnatch: Update pip provider to use openstack_pip provider
openstackgerritMerged openstack-infra/puppet-jeepyb: Update pip provider to use openstack_pip provider
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch back to current-tripleo pre promotion
*** EricGonczer_ has quit IRC16:37
*** jordanP has quit IRC16:37
mtreinishjeblair: I just fast approved your change, I'll handle rebasing and approving asselin's change after the rename is done16:38
*** yamahata has joined #openstack-infra16:39
*** jlanoux has quit IRC16:39
pabelangerfungi: are you okay adding python3 wheels (329094) back into the mix?  python2 job is working correctly again:
*** ramishra has joined #openstack-infra16:40
*** amitgandhinz has quit IRC16:40
jeblairmtreinish: cool, thanks16:40
pabelangerfungi: just confirming scripts are correct again on 32909416:41
*** amitgandhinz has joined #openstack-infra16:41
fungipabelanger: yeah, my only remaining concern is that they may race and step on each other since for universal wheels both jobs would create a version of the same file16:41
jeblairanyone seen sdague lately?16:41
pabelangerfungi: Right, but since we have only 1 slave atm, is that going to be the case?16:41
fungithough i'm not sure how to tell the python 3 job to only create wheels for non-universal16:42
pabelangerfungi: Oh, I see what you are saying16:42
fungipabelanger: oh, fair point they run on the same server. so worst is that we unnecessarily create a lot of the same wheels multiple times16:42
pabelangerfungi: ya16:42
*** dmk0202 has quit IRC16:42
pabelangerwould be a good optimization to disable the dupe packages between python2 / python316:42
fungithere's also the corner case that supposedly universal wheels created under python 3 don't behave like the ones created under python 2 and that we'd want to prefer the latter over the former16:43
fungii have no idea if that's actually a risk though16:43
pabelangerme neither16:43
pabelangerwheel-build-ubuntu-trusty-amd64-python3 looks correct16:44
pabelangerjust confirmed locally16:44
*** erlon has joined #openstack-infra16:44
fungicool, jobs passed for 330624. +2 from me16:44
pabelangerfungi: 329094 is python3 wheel16:46
fungii've still been waiting for a lull on proposal.slave.o.o but the backlog in post probably means that won't happen for quite a while16:46
*** rossella_s has joined #openstack-infra16:46
fungipabelanger: yeah, i was talking about the static worker job registration fix for zuul16:46
fungithat's in the gate now16:46
pabelangerfungi: Ah16:46
*** EricGonc_ has quit IRC16:46
pabelangersorry for the noise16:46
fungii'll take a look at 329094 shortly16:46
fungineed to heat up some lunch first16:47
*** mixos has joined #openstack-infra16:47
openstackgerritJames E. Blair proposed openstack-infra/zuul: Make pipeline approval username requirement a regular expression
*** eezhova has quit IRC16:47
jeblairmordred, notmorgan: ^16:47
fungipabelanger: since i think the wheelhouse on that node persists between wheel build jobs, i _hope_ that by running the python 2 job first we ensure that the python 3 job doesn't bother building any universal wheels because it finds them already present in the wheelhouse16:48
openstackgerritJames E. Blair proposed openstack-infra/project-config: Change Zuul's username to zuul
jeblairmordred, notmorgan: ^ and that16:49
jeblairfungi: where's the wheelhouse?16:50
*** david-lyle has joined #openstack-infra16:50
*** EricGonczer_ has joined #openstack-infra16:50
openstackgerritMerged openstack-infra/zuul: Ansible launcher: update registration between jobs
jeblairfungi: i updated some long running jobs to rm -rf /home/jenkins/workspace at the beginning of each job16:50
jeblairfungi: i don't know if it was on the wheel builders or not16:51
fungijeblair: oh, no it's not persisted between jobs. we explicitly blow it away each run
fungipabelanger: ^16:51
fungiso much for that idea16:52
*** nwkarsten has quit IRC16:52
fungii wonder if we should reconsider that16:52
fungiit's passed in as WHEEL_DIR=/opt/wheel/workspace16:52
jeblairfungi: it was only on the proposal slave that i do the rm workspace16:52
jeblairfungi: so that should be fine if we want to reconsider that16:53
openstackgerritMerged openstack-infra/puppet-kerberos: Don't have the server install the client
openstackgerritMerged openstack-infra/puppet-logstash: Update pip provider to use openstack_pip provider
fungiwell, even if it were on the proposal slave, we specify a directory outside the job workspace16:53
fungiso clearing the job workspace wouldn't impact the wheel cache16:54
openstackgerritScott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend
openstackgerritMerged openstack-infra/puppet-meetbot: Fix deprecated warnings
*** esikachev has joined #openstack-infra16:54
*** ddieterly[away] is now known as ddieterly16:54
*** ramishra has quit IRC16:55
openstackgerritMerged openstack-infra/puppet-openafs: Increase memory available to network stack on fileservers
notmorganjeblair: oh nice16:55
notmorganjeblair: so we know which is commenting16:55
funginotmorgan: not really. it was always zuul commenting16:55
notmorganfungi: right, but in this case we could make it anything16:56
fungior at least it was after we stopped using the gerrit plugin for jenkins16:56
notmorganfungi: rather than must be jenkins named16:56
notmorganit could be *any* tool as long as the account matches, e.g. having multiple/different tools for different things (in theory)16:56
jeblairnotmorgan: we can currently make it anything, it's just a string match, so we only get to make it one thing.  this lets us make it two things at once.  :)16:56
notmorganjeblair: yeah this fits our needs for sure here :)16:57
fungieasier transition this way16:57
openstackgerritMerged openstack-infra/system-config: Add compute031.uswest metadata for better tracking
mtreinishjeblair: iirc, sdague is going camping over the weekend and said today was prep day16:57
jeblairalso, i think that's just about the last thing in that part of the config that *isn't* a regex16:57
* notmorgan gets caffiene...16:57
mordredjeblair: \o/16:57
jeblairmtreinish: oh fun!  well, there might be some dashboards that need a'fixin when he gets back16:58
mordredjeblair: what email address is it using?16:59
*** derekh has quit IRC16:59
fungiyay! i caught the proposal.slave.o.o idle and reconfigured zl on zlstatic0116:59
jeblairmordred: it does not have a preferred email address (like jenkins).  it does have one though.  why?16:59
fungiand now those jobs have workers16:59
jeblairmordred: (this way, the email address does not show up in the display)16:59
*** PollyZ_ has joined #openstack-infra16:59
fungiin gearman16:59
mordredjeblair: mostly wondering what email will show up on the merge commits17:00
jeblairfungi: cool, sounds like confirmation we're on the right track17:00
jeblairmordred: i put zuul@zuul.openstack.org17:00
mordredcool. I will add that to the robot exclusion list in stackalytics17:00
*** wojdec has joined #openstack-infra17:00
*** mhickey has joined #openstack-infra17:00
jeblairwe might update that to zuul@openstack.org17:00
openstackgerritPaul Belanger proposed openstack-infra/puppet-github: Update pip provider to use openstack_pip provider
mordredI'll add both :)17:00
pabelangerphschwartz: ^17:00
jeblairmordred: i'll go add an alias for zuul, and switch it to zuul@openstack.org17:01
*** EricGonczer_ has quit IRC17:01
*** kushal has joined #openstack-infra17:02
*** PollyZ has quit IRC17:02
*** yamamoto has joined #openstack-infra17:05
odyssey4mepabelanger thanks for the review of - I've replied with a question for clarity when you have a moment17:05
*** xarses has quit IRC17:05
jeblairmordred, pleia2: created alias for infra-root along with a server-side filter17:06
*** xarses has joined #openstack-infra17:06
*** akshai_ has quit IRC17:06
fungijeblair: further confirmation, shortly after workers showed up in gearman for those jobs they ran and cleared out of zuul's queues17:06
openstackgerritMerged openstack-infra/system-config: Add compute016.useast metadata for better tracking
openstackgerritMerged openstack-infra/system-config: Add compute086.useast metadata for better tracking
*** daemontool has quit IRC17:08
mordredjeblair: I've submitted a stackalytics change to add zuul@ and zuul@zuul to the list:
*** trown is now known as trown|lunch17:08
*** PollyZ_ has quit IRC17:08
*** PollyZ has joined #openstack-infra17:08
*** ifarkas has quit IRC17:08
*** salv-orl_ has joined #openstack-infra17:08
asselinmtreinish, jeblair default name 'jenkins' no longer valid for upstream?17:09
pabelangerodyssey4me: replied17:09
jeblairasselin: for now it is, we will be changing it soon17:09
openstackgerritMerged openstack-infra/system-config: Add compute089.useast metadata for better tracking
openstackgerritMerged openstack-infra/system-config: Add compute085.useast metadata for better tracking
jeblairmordred: and updated in gerrit17:10
openstackgerritMerged openstack-infra/system-config: Add compute087.useast metadata for better tracking
openstackgerritMerged openstack-infra/system-config: Add compute084.useast metadata for better tracking
openstackgerritMerged openstack-infra/system-config: Add compute013.useast metadata for better tracking
mtreinishasselin: yeah, they're switching it to 'zuul'17:10
mordredasselin: we've very close to not having any software running named jenkins17:11
pabelangermordred: \o/17:11
fungithis could probably stand a heads-up notice to the -dev ml because i bet a lot of people have e-mail filters and stuff that assume the old name/address17:11
*** roxanaghe has joined #openstack-infra17:11
jeblairfungi: i'm planning on sending one soon17:11
fungii figured!17:11
* fungi notes to adjust his own .forward filter file17:12
*** salv-orlando has quit IRC17:12
jeblairi'm thinking: finish the transition; once no jenkins servers are running, send an announcement (i have one prepped); then we can effect the name change after that17:12
openstackgerritMerged openstack-infra/system-config: Add compute012.useast metadata for better tracking
jeblairwe could do the name change tomorrow during the zuul maintenance, or we could wait longer if we think that's necessary17:13
openstackgerritMerged openstack-infra/system-config: Add compute015.useast metadata for better tracking
openstackgerritMerged openstack-infra/system-config: Add compute009.useast metadata for better tracking
*** Daisy has joined #openstack-infra17:13
openstackgerritMerged openstack-infra/system-config: Add compute010.useast metadata for better tracking
openstackgerritMerged openstack-infra/system-config: Add compute011.useast metadata for better tracking
mordredjeblair: think it's worth it, while we're doing name changes, to make a zuul account in launchpad and replace hudson-openstack?17:13
fungiif pabelanger hasn't written up the announcement for tomorrow yet, we may want to reschedule out a week17:13
openstackgerritMerged openstack-infra/system-config: Add compute017.useast metadata for better tracking
jeblairmordred: zuul doesn't touch launchpad17:14
pabelangerspeaking of that, I have the etherpad here: but we still haven't sent it17:14
openstackgerritMerged openstack-infra/system-config: Add compute019.useast metadata for better tracking
fungiit's getting to be pretty short notice at this point17:14
openstackgerritMerged openstack-infra/system-config: Add compute026.useast metadata for better tracking
openstackgerritMerged openstack-infra/system-config: Add compute020.useast metadata for better tracking
pabelangerya, I left data empty so we can revisit it17:14
openstackgerritMerged openstack-infra/system-config: Fix baremetal00 name attribute still referencing west
mordredjeblair: right. but gerrit does, and it uses the hudson-openstack account to do it - so I guess we could change the account to openstack-gerrit instead of openstack-zuul17:14
fungimordred: i'd rather hold on to hope that we simply won't need an account in launchpad for our automation at some point in the future and leave it alone for now. adjusting membership in everybody's bug groups would be a pain17:14
mordredfungi: good call17:14
fungithat would be a lot of cat herding since i doubt they're all owned by openstack-admins17:15
*** roxanaghe has quit IRC17:16
*** BobBall is now known as BobBall_AWOL17:16
pabelangerfungi: see eitherpad above, but pusing back to next week (R-15) seems to be clear for newton17:16
*** Daisy has quit IRC17:18
jeblairpabelanger: what hosts are we upgrading?17:18
fungizuul and static17:18
*** _ari_ has joined #openstack-infra17:18
pabelangerjeblair: zuul.o.o and static.o.o need to upgraded17:18
jeblairstatic.  that's the one i was forgetting that makes this complex :)17:18
fungistatic is the bigger concern since we need to take everything offline and move 14 cinder volumes17:18
fungiwe could probably do the zuul upgrade fairly quickly/seamlessly by comparison17:19
*** amitgandhinz has quit IRC17:19
jeblairwhich is terrifying since the last time we tried to detach a cinder volume it took 8 hours to fail17:19
fungibut if we can't run any jobs while static.o.o is being moved, then might as well have someone else upgrade zuul at teh same time17:19
_ari_zxiiro: I keep hearing rumors that JJB is going to go away is this true?17:20
pabelangerYa, I would recommend we stop the server over detaching while it is running17:20
jeblair_ari_: it is not true17:20
*** amitgandhinz has joined #openstack-infra17:20
_ari_jeblair: phew17:20
fungijeblair: well, my suggestion to halt the server worked when pabelanger tried it, after we gave up waiting for the fanatics to respond17:20
mordred_ari_: we are eventually going to stop using it in infra - JJB itself has a lovely team of dedicated humans17:20
pabelangerfungi: ++17:20
*** dtantsur is now known as dtantsur|afk17:20
jeblair_ari_: so it will live on with our support as long as they want it17:20
*** sambetts is now known as sambetts|afk17:21
*** rcernin has quit IRC17:21
fungi_ari_: at the moment, jjb is actually being used directly by zuul 2.5 and translated into ansible plays17:21
_ari_jeblair: mordred So I heard there is a plan to replace Jenkins and use Zuul + Ansible17:21
zxiiro_ari_: I hope not, we (the linux foundation) depend on JJB so we plan on keeping it supported17:22
_ari_So what is the need to get rid of Jenkins out of curiosity17:22
fungi_ari_: so it's technically now the "jenkins and zuul job builder" ;)17:22
jeblair_ari_: yes, we are in the process of replacing jenkins with zuul now, with playbooks created by jjb.  in zuulv3, we will stop using jjb and write playbooks directly as job definitions17:22
*** eharney has quit IRC17:23
*** bauzas has quit IRC17:23
pabelanger_ari_: see for an example of zuul-launcher (ansible). Note the _zuul_ansible folder to see out ansible bits17:23
fungibut yeah, we hope jjb continues to have a long life even once we're no longer in need of it for the openstack infrastructure. if anything, the current jjb contributors will have greater freedom and autonomy to take the project in the directions they want without worry for our possibly esoteric use cases17:23
jeblairzxiiro: ^17:23
mordred_ari_: jenkins is complex in support of a ton of features we don't and can't use. it's not designed for our kind of scale and it keeps breaking. we've been working towards moving off of it for about 4 years now, but it's along and involved process, given the central role of the gate17:23
_ari_jeblair: So is the plan to reproduce the functionality that you already get with Jenkins plugins with playsbooks?17:23
mordred_ari_: we do not really use jenkins plugins17:24
*** bauzas has joined #openstack-infra17:24
mordredfor us, jenkins is a remote execution engine that is driven by REST calls17:24
_ari_mordred: ah I see17:24
*** ddieterly is now known as ddieterly[away]17:24
_ari_pabelanger: thanks!17:24
fungiwe basically use jenkins to run scripts on job workers, and collect results/logs17:24
jeblair_ari_: the only plugins we use are the scp and ftp publishers17:24
*** ihrachys_ has quit IRC17:24
zxiiro_ari_: my organizations use many Jenkins plugihns so we can't move away from JJB so we will be providing support it for  even if openstack stops17:25
mordredyay for open source!17:25
pabelangerI really like how JJB has taken on life of its own.17:25
mordredme too17:25
zxiiroone thing that concerns me a bit that came to my attention yesterday was this
_ari_zxiiro: jeblair I used many Jenkins plugins as well and it gives a lot of hooks into different infrastructures like github, Amazon S3,Docker, etc. but I can see if you don't use those why it is not worth while17:26
zxiiroI'd rather see the author try to push those features into JJB proper rather than providing a separate package17:26
*** wojdec1 has joined #openstack-infra17:26
_ari_zxiiro: +117:26
pabelangerzxiiro: ouch17:26
jeblairzxiiro: well, we did design it to use entry points to facilitate that sort of thing17:26
*** PollyZ has quit IRC17:27
*** wojdec has quit IRC17:27
jeblairmostly so that people could have site-local things17:27
*** boltR_ has quit IRC17:27
fungibut yeah, addons make sense if you're only using them internally. at the point where you're going to publish them, might as well sibmit a patch to jjb?17:27
jeblairbut also, for more esoteric plugins17:27
jeblairwe used to not accept jjb support for non-open-sounce jenkins plugins17:28
jeblairi think that slipped when i started reviewing less :)17:28
*** akshai has joined #openstack-infra17:28
_ari_fungi: definitely it is very easy to extend and contribute the plugin feature upstream17:28
*** inc0 has quit IRC17:28
*** eharney has joined #openstack-infra17:28
_ari_zxiiro: one question that has been asked by me and others is where does the pipeline plugin come in to JJB since they no longer store XML on disk for jobs any longer17:29
pabelangereager to see if we are going to have a plugin system for zuulv3+17:29
*** inc0 has joined #openstack-infra17:29
mordredpabelanger: yes17:29
_ari_pabelanger: if you do that aren't you just reinventing what Jenkins already does17:29
mordredpabelanger: it's called "ansible roles"17:29
jeblairpabelanger: ansible17:29
*** johnny_ has joined #openstack-infra17:29
jeblairbest part, we don't have to write it17:29
*** pvaneck has quit IRC17:30
*** ccamacho has left #openstack-infra17:30
zxiiro_ari_: I'm not sure. I haven't had any cycles to look at the pipelining stuff yet17:30
zxiiro_ari_: we currently don't use it though so it's not a priority for us17:30
pabelangerI should have rephrased, interested what our ansible roles for zuulv3 will looking. And if we'll have some sort of community spring up around it17:31
_ari_zxiiro: sure I am able to get the same type of functionality using multijob and matrix plugins17:31
pabelangers/looking/ look like17:31
*** tqtran has joined #openstack-infra17:31
fungipabelanger: i expect a lot of it will be shaped by the ability for projects to have their own job definitions in-repo17:31
pabelangerfungi: Ya, looking forward to what those projects produce17:32
zxiirowill zuulv3 support actual pipeline? we have issues with switching to zuul because we can't create triggers that trigger off of other jobs.17:32
_ari_jeblair: what do you mean you don't have to write it you still have to create a role to do the things Jenkins do right?17:32
zxiirofor example a downstream job is watching an upstream job and triggers if the upstream job passes17:32
phschwartzmordred: when was openstack-infra/ansible-puppet renamed to openstack-infra/ansible-role-puppet?17:32
fungiphschwartz: a couple weeks ago17:32
zxiironot sure what the words to use to describe it but it's like reverse pipelines? i guess?17:32
phschwartzfungi: hmm, seems to have not complained till today17:33
phschwartzfungi: easy enough fix on my side17:33
*** xyang1 has joined #openstack-infra17:33
fungizxiiro: we have dependent jobs, where job y only runs if job x passes17:33
*** tosky has quit IRC17:34
fungizxiiro: that sounds like what you're describing, and it's also been suggested that we could consider inverting the definition so that a job gets marked as depending on some other job rather than a job having a list of dependent jobs17:35
jeblairzxiiro: here's what that looks like today:
zxiirofungi: yeah we have jobs where we don't care that it votes or not. just simply that x job finished (vote happens) and then run a build and not report anything. It's so that downstream projects can see if upstream pushed a new feature that broke them.17:36
jeblairzxiiro: on release, build the tarball, if that succeeds, upload the tarball, if that succeeds, rebuild the trusty wheel mirror, if that succeeds, release the trusty wheel mirror to production17:36
jeblairmordred: that looks like a change number but doesn't appear to be one17:38
*** nwkarsten has joined #openstack-infra17:38
*** nwkarsten has quit IRC17:38
fungiit will be in a couple weeks at our current pace17:38
fungi(or sooner)17:38
*** nwkarsten has joined #openstack-infra17:38
jeblairi can't wait to see what it ends up being! :)17:39
mordredjeblair: it's not a change number17:39
jeblairmordred promises it will be a good one17:39
*** acoles is now known as acoles_17:39
mordredjeblair: its 6 numbers that have no current meaning :)17:39
* fungi guesses it's mordred's yubikey getting stepped on by a cat17:39
jeblairfungi: mordred's cat has opposable thumbs, so probably just actually pressed it17:39
*** dizquierdo has quit IRC17:40
mordredjeblair: ++17:40
*** abregman has quit IRC17:40
*** kzaitsev1mb has joined #openstack-infra17:40
*** kzaitsev_mb has quit IRC17:40
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add install-guide jobs for trove
*** eharney has quit IRC17:46
*** Guest41851 has quit IRC17:47
*** Hal has joined #openstack-infra17:48
*** wojdec1 has left #openstack-infra17:48
*** Hal is now known as Guest8397917:49
* AJaeger has just watched the proposal slave output - cool. thanks, jeblair !17:49
*** rbrndt has quit IRC17:49
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add zl05 - zl07 to nodepool
*** kushal has quit IRC17:52
jeblairwe're going to finish this today :)17:52
pabelangermortal combat style17:52
fungii'll feel a lot more comfortable about the accuracy of our nodepool counts once we're not running a mix of zl and jenkins17:53
jeblairzmq will still be a weak link :(17:53
fungiyep, but it will be a consistent weak link17:54
mordredjeblair: I would like to request that the final patch removing jenkins things has the word "fatality" attached to it17:54
AJaegerjeblair: great!17:54
fungiand easier to orchestrate around in the short term than our current ansible playbook for jenkins restarts17:54
*** jamesmcarthur has quit IRC17:54
AJaegerjeblair: do we really need the same number of zuul-launchers? Or is that something to figure out over time?17:55
andreafd-g change to switch tempest layer4 job to pre-provisioned test accounts: - could I have reviews / comments?17:55
fungiAJaeger: it looked like the load balance/cpu/memory was about right at 1:117:55
jeblairAJaeger: we *think* we need about the same.  however, the zl hosts have 1/4 the ram :)17:55
fungiAJaeger: noting that the zuul-launchers are much smaller servers than the giant flavors we ran our jenkins masters on17:56
fungier, yeah, what jeblair said ;)17:56
AJaegerthanks for explanation17:56
*** vhosakot has joined #openstack-infra17:57
*** PollyZ has joined #openstack-infra17:57
*** SumitNaiksatam has joined #openstack-infra17:58
*** pvaneck has joined #openstack-infra17:58
*** electrofelix has quit IRC17:58
*** mtanino has quit IRC17:59
*** yaume has quit IRC17:59
pabelangeron demand zuul-launchers would be very cool18:00
pabelangerautomated of course18:00
*** baoli_ has quit IRC18:00
*** PollyZ has quit IRC18:00
fungiin/after zuul v318:01
*** dimtruck is now known as zz_dimtruck18:01
fungii wouldn't want to heap any more effort on nice-to-haves in 2.5, this is just our "quick" solution for dropping jenkins18:01
*** vhosakot has quit IRC18:01
fungiany unnecessary effort put into 2.5 saps development resources from 318:02
openstackgerritMerged openstack-infra/puppet-pip: Declare features in openstack_pip provider
*** kzaitsev1mb has quit IRC18:02
pabelangerI think feature requests for v25 are closed :)18:02
*** vhosakot has joined #openstack-infra18:02
pabelangerI'm imagining the count down clock for zuulv25 starts as soon as the last jenkins master goes offline18:03
*** jsavak has quit IRC18:03
*** Sukhdev has joined #openstack-infra18:03
russellbwill there be cake?18:03
*** amoralej is now known as amoralej|off18:04
mordredrussellb: there is always time for cake18:04
*** kmartin has quit IRC18:04
openstackgerritMerged openstack-infra/project-config: Create new functional job for glance_store-cinder
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Add partition image src jobs for agent driver
*** jsavak has joined #openstack-infra18:06
*** rcernin has joined #openstack-infra18:07
*** akshai has quit IRC18:08
*** trown|lunch is now known as trown18:09
*** akshai has joined #openstack-infra18:11
*** mtanino has joined #openstack-infra18:11
*** PollyZ has joined #openstack-infra18:11
openstackgerritMerged openstack-infra/elastic-recheck: Support Zuul as a Gerrit user equivalent to Jenkins
*** eharney has joined #openstack-infra18:14
*** hdaniel has joined #openstack-infra18:14
openstackgerritGhe Rivero proposed openstack-infra/shade: Get user list by domain
oomichifungi: hi, mkoderer has written the detail of our meetup on
oomichifungi: so we can get attendees now :)18:15
*** baoli has joined #openstack-infra18:16
mtreinishoomichi: you might want to put it on the ML18:16
oomichimtreinish: yep, I am searching old mail as a sample now ;)18:16
*** HeOS has quit IRC18:18
fungioomichi: awesome news--thanks!18:18
fungimkoderer: thanks!18:18
*** _nadya_ has joined #openstack-infra18:18
openstackgerritMerged openstack-infra/zuul: Make pipeline approval username requirement a regular expression
fungipabelanger: i approved 329094 but we'll want to keep an eye out for potentially unusual behavior since a lot (most?) of our universal wheels in the mirror will be built under python 3 now instead of 218:22
pabelangerfungi: that is fair18:23
fungii wonder if we could adjust the build script to only copy non-universal wheels when called for python 318:23
*** mriedem1 has joined #openstack-infra18:23
jeblairrussellb: oh yeah, we should have a retirement party for jenkins tomorrow18:23
*** jyuso1 has quit IRC18:24
openstackgerritDong Ma proposed openstack-infra/jenkins-job-builder: Remove the colon after default in docstring
*** mkoderer__ has joined #openstack-infra18:24
fungipabelanger: though that introduces another corner case because of environment markers. module version dependencies may be different under different interpreter versions18:24
*** jyuso1 has joined #openstack-infra18:24
*** ddieterly[away] has quit IRC18:24
fungiso we might end up not building some universal wheels which were needed by versions obtained under python 3 that we didn't need under python 218:24
*** gyee has quit IRC18:25
pabelangerfungi: ya18:25
*** mriedem has quit IRC18:26
*** PollyZ has quit IRC18:27
AJaegercould somebody help me with reviewing the higgins rename change logic, please? I'm only concerned right now on the IRC setup: adds a new channel, then we have to rename channels in system-config - and then we should be able to remove the old channel. Correct?18:27
*** kmartin has joined #openstack-infra18:27
mordredok. that's not a good place for that18:29
AJaegermordred: that change number will take even longer ;)18:29
jeblairmordred is a numbers station18:29
*** mriedem1 is now known as mriedem18:29
fungiindeed. a few more and i'll have the code cracked18:29
* mordred knows how to make integers18:29
*** ddieterly has joined #openstack-infra18:30
*** mixos has quit IRC18:30
Shrewsmordred: i remember my first yubikey18:30
*** nwkarsten has quit IRC18:31
AJaegerfungi, can we add another channel to gerritbot - or do we have to wait for the limit to be changed?18:31
*** maishsk_ has joined #openstack-infra18:31
*** maishsk has quit IRC18:32
*** maishsk_ is now known as maishsk18:32
*** hdaniel has quit IRC18:33
notmorganmordred: lol18:33
notmorganmordred: welcome to my world18:34
*** nwkarsten has joined #openstack-infra18:34
*** nwkarsten has quit IRC18:34
*** baoli has quit IRC18:34
mordrednotmorgan: dude. I'm just following your instructions18:34
*** nwkarsten has joined #openstack-infra18:34
clarkbbte the qa/infra sprint coincides with a gerrit meetup at the same place18:34
*** sarob has joined #openstack-infra18:35
jeblairclarkb: neat18:35
*** sarob has quit IRC18:36
fungiAJaeger: we can add them, just it may not join them all if there are more than 12018:36
jeblairwhat a coincidence18:36
fungitomaw isn't in channel at the moment :/18:36
*** baoli has joined #openstack-infra18:37
AJaegerfungi, if we add them, it will be more than 120. So, I wonder whether we just approve - or wait.18:37
nibalizerI think I'm still not allowed to hack on gerrit :/18:37
*** PollyZ has joined #openstack-infra18:37
nibalizerwhich might actually be a feature18:37
fungiAJaeger: let's wait to add more channels to gerritbot until we can get CHANLIMIT upped for its account18:38
*** rbrndt has joined #openstack-infra18:38
notmorganmordred: and clearly you're doing it right if you found out that issue (finding the right USB port was a challenge for me)18:38
nibalizerfungi: ack18:38
*** sarob has joined #openstack-infra18:38
notmorganmordred: i'm using the one next to the HDMI port on the right (x1c), so i don't put my hands there that often18:38
AJaegerfungi, fine with me.18:39
*** hasharAway is now known as hashar18:39
AJaegerthe higgins rename would be very brief - adding one, rename in system-config, removing the old one. But to do this, we would need to wait for a time without meetings... Or should the rename done differently than 326306 and 330017 ?18:40
AJaeger(I know it's missing the last bit of the rename for project-config)18:40
mordrednotmorgan: I'm also doing that - and I also don't put my hands there- except I do?18:41
*** zz_dimtruck is now known as dimtruck18:41
*** tonytan4ever has joined #openstack-infra18:43
notmorganmordred: it also was really bad when i started18:43
notmorganit gets better as soon as you are used to the thing being there18:44
*** dmk0202 has joined #openstack-infra18:44
openstackgerritMerged openstack-infra/puppet-zuul: Update pip provider to use openstack_pip provider
*** dmk0202 has quit IRC18:45
*** akshai_ has joined #openstack-infra18:45
*** chem is now known as chem|off18:46
fungiAJaeger: the higgins channel move is a net gain of 0, so i wouldn't hold that up, would just suggest a cut-over change and merge that immediately after the channel forward is set18:46
fungishouldn't need a multi-stage series of changes for a channel move18:46
openstackgerritMerged openstack-infra/puppet-github: Update pip provider to use openstack_pip provider
AJaegerfungi, then my guidenance was wrong ;( So, just two moves? Ok, let me update the changes for them since I misguided them...18:47
*** akshai has quit IRC18:47
AJaegerjeblair: your change to add the new zuuls failed - but exit code is 0:
openstackgerritMerged openstack-infra/zuul: Use safe_dump() for Ansible playbooks
jeblair2016-06-16 18:39:27,495 p=7644 u=root |  An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/zuul/ansible/library/'18:48
*** maishsk has quit IRC18:49
jeblairwhy does the ansible log say 'u=root' ?18:49
jeblairzuul-launcher is running as 'zuul' according to ps18:49
*** jamesmcarthur has joined #openstack-infra18:50
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Zun: rename higgins to zun (1)
*** akshai_ has quit IRC18:50
jeblairShrews, mordred: any ideas about the ansible log u=root thing?18:50
mordredjeblair: launcher may be running as zuul. but what user is it running that task as remotely?18:50
*** kmartin has quit IRC18:50
*** sarob_ has joined #openstack-infra18:50
*** xarses has quit IRC18:50
jeblairmordred: jenkins?18:50
*** amit213 has joined #openstack-infra18:51
EmilienMfungi: I have an outstanding request to bring our TripleO CI back to green: can you promote 330661 jobs please?18:51
mordredjeblair: where is that log?18:51
*** cody-somerville has joined #openstack-infra18:51
*** xarses has joined #openstack-infra18:51
*** shamail has joined #openstack-infra18:53
Shrews  says jenkins18:53
mordredjeblair:  I do not see anything in those playbooks which would lead me to believe it should be root18:53
fungiEmilienM: i'm assuming you mean you want it enqueued into the gate, skipping the check pipeline?18:53
EmilienMfungi: yes, we would like it merged asap18:53
*** sarob has quit IRC18:53
Shrewsjeblair: this is where -vvv would be helpful18:54
*** salv-orl_ has quit IRC18:54
EmilienMfungi: thx a ton18:55
jeblairShrews, mordred: when i run locally, i get u=corvus, which is the user my zuul-launcher is running as18:55
*** salv-orlando has joined #openstack-infra18:55
openstackgerritMerged openstack-infra/project-config: Add python3 wheel-builds
jeblairso i think that's supposed to mean u=<user ansible is running as locally>18:55
*** maishsk has joined #openstack-infra18:56
*** baoli has quit IRC18:56
*** _nadya_ has quit IRC18:58
*** e0ne has joined #openstack-infra18:58
*** mhickey has quit IRC18:59
jeblair        user = getpass.getuser()18:59
jeblair        logger = logging.getLogger("p=%s u=%s | " % (mypid, user))18:59
*** yolanda has quit IRC19:00
*** _nadya_ has joined #openstack-infra19:01
*** kzaitsev_mb has joined #openstack-infra19:01
jeblairokay, so we need to pause the migration and solve these two mysteries first:19:02
jeblair1) why does the log say u=root19:02
*** ociuhandu has quit IRC19:02
jeblair2) what caused the missing zuul_log module error19:02
jeblairand i need to grab lunch now.  biab.19:02
Shrewsjeblair: oh, i don't think that's coming from ansible19:04
jeblairoh i have a theory on #219:04
jeblairi think we re-installed zuul *exactly* as that task was running19:04
jeblairapparently re-installing zuul means that file disappears briefly19:05
*** arxcruz has quit IRC19:05
jeblairJun 16 18:39:29 zl04 puppet-user[20639]: (/Stage[main]/Zuul/Exec[install_zuul]) Triggered 'refresh' from 1 events19:05
jeblair2016-06-16 18:39:27,495 p=7644 u=root |  An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/zuul/ansible/library/'19:05
*** jsavak has quit IRC19:05
jeblairShrews: what's not coming from ansible?19:06
*** roaet has joined #openstack-infra19:06
*** hdaniel has joined #openstack-infra19:06
*** xarses_ has joined #openstack-infra19:06
roaethello infra!19:06
Shrewsjeblair: eh, nm19:08
*** EricGonczer_ has joined #openstack-infra19:08
*** xarses has quit IRC19:09
jeblairokay, lunch for real now; if anyone has ideas on #1 they would be greatly appreciated19:09
openstackgerritBrian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin
*** bhavik has quit IRC19:12
*** xarses_ has quit IRC19:12
*** roxanagh_ has joined #openstack-infra19:13
*** EricGonczer_ has quit IRC19:13
*** javeriak has joined #openstack-infra19:14
*** javeriak_ has quit IRC19:15
Shrewsjeblair: mordred: ok, so, the ansible-playbook log (captured via the log_path variable) runs from the local machine (not remote), so we could expect to see the username in that log output to be the zuul-launcher user19:17
*** roxanagh_ has quit IRC19:17
Shrewswhich i'm assuming is root?19:18
Shrewsif that is actually 'zuul', then i have no good answer.19:21
*** e0ne has quit IRC19:22
Shrewsinfra-root: anyone care to see what user zuul-launcher is running as on zl04 ?19:22
mordredShrews: yah - zuul-launcher shows up in ps as running as zuul19:22
Shrewsmordred: on zl04?19:22
mordredzuul      9980     1  0 Jun15 ?        00:12:29 /usr/bin/python /usr/local/bin/zuul-launcher19:22
mordredShrews: yes. that's on zl0419:23
Shrewsmordred: i hate computers19:23
*** _nadya_ has quit IRC19:26
*** mixos has joined #openstack-infra19:27
*** dprince has quit IRC19:27
Shrewsnot sure that the supervisord is relevant, but the env var stuff is19:29
fungioh, fun19:29
*** _ari_ has quit IRC19:30
*** Jeffrey4l has quit IRC19:30
openstackgerritMerged openstack-infra/project-config: Add zl05 - zl07 to nodepool
fungiyeah, so LOGNAME getting carried along into the child after changing effective uid confuses it?19:30
Shrewsfungi: maybe? i can only speculate19:31
*** chem|off` has joined #openstack-infra19:31
Shrewsbut that sounds plausible to me19:31
fungiwell, the initscript is run by root19:31
*** Jeffrey4l has joined #openstack-infra19:31
*** csomerville has joined #openstack-infra19:31
fungiwhich then tells start-stop-daemon to change user when launching the daemon19:32
*** chem|off has quit IRC19:33
*** sarob_ has quit IRC19:34
fungii'm guessing one of the envvars getpass.getuser() queries isn't being reset to match19:34
Shrewsso, i'll change my statement to "i hate python"19:34
*** Guest83979 has quit IRC19:34
Shrewsso, i'll change my statement to "i hate python, too"19:35
fungiyeah, i would have expected (on *nix systems) that it would look at the process uid and then map that via the usual mechanisms to a username19:35
fungimaybe this is an attempt to make fewer calls to the filesystem?19:35
*** cody-somerville has quit IRC19:35
fungiregardless, violates principle of least surprise for me and apparently lots of other people19:36
*** dtardivel has quit IRC19:37
*** ociuhandu has joined #openstack-infra19:37
*** Hal has joined #openstack-infra19:37
*** Hal is now known as Guest1773119:37
*** asselin_ has joined #openstack-infra19:38
*** flepied1 has joined #openstack-infra19:38
*** dimtruck is now known as zz_dimtruck19:39
*** hdaniel has quit IRC19:39
*** asselin__ has quit IRC19:39
*** dmk0202 has joined #openstack-infra19:40
*** EricGonczer_ has joined #openstack-infra19:41
*** flepied has quit IRC19:42
*** PollyZ has quit IRC19:42
*** dmk0202 has quit IRC19:43
*** esikachev has quit IRC19:44
*** nwkarsten has quit IRC19:45
*** EricGonczer_ has quit IRC19:45
openstackgerritMerged openstack-infra/jenkins-job-builder: Remove the colon after default in docstring
*** _nadya_ has joined #openstack-infra19:47
jeblairShrews: aha!19:49
jeblairShrews, fungi, mordred: are we satisfied with that as an explanation for #1, and my theory for #2?19:49
lifelesshmm, gerrit seems to be signing me out after 5m? I thought the bug fix to my config patch was deployed?19:50
hongbinHi, I tried to rename IRC channel from #openstack-higgins to #openstack-zun. I followed this instruction but it turns out I didn't have permission to execute all those commands. Could anyone know how to fix it?19:51
Shrewsjeblair: fairly confident about #1. #2 theory seems highly likely to me19:52
fungilifeless: it's not signing me out quickly. are you sure you don't have different session ids fighting between browser tabs/windows?19:54
fungilifeless: (possibly painful) workaround is to log out of gerrit in all your tabs, then log back in with just one and refresh all the others after that19:55
*** jsavak has joined #openstack-infra19:55
*** hdaniel has joined #openstack-infra19:55
fungiat that point they _should_ (in my experience) all be using the same session19:55
AJaegerlifeless: see I95c20373d572b061e870103a813fef1d25de4b23 - 2m means 2mins, so it was reverted...19:57
AJaegerlifeless: so, it should be at 7 days - but it's less than a day it seems to me...19:57
fungidefinitely doesn't seem to be 5 minutes though19:58
*** tkelsey has quit IRC19:58
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: treat missing metadata as a warning not an error
JayFI can easily reproduce the 5m issue: open multiple review tabs; login to >1 of them19:59
timothyb89does anyone have experience running diskimage-builder behind a proxy? with vars set and the local-config element added I'm still hitting proxy errors from pip at 40-glean19:59
JayFand it sorta spirals down because if it logs you out frequently you're encourages to just keep logging back in so you can vote/comment on apatch19:59
JayFtimothyb89: that might be a good question for the tripleo folks, I know they use that frequently20:00
fungiJayF: yep, it's been a very long-standing flaw with the session design in gerrit20:00
*** salv-orlando has quit IRC20:00
*** salv-orlando has joined #openstack-infra20:01
jeblairfungi, Shrews, mordred: so i think we can continue.  i'm not too worried about #2 -- it will be nice to find a fix for that in the future, but it shouldn't happen too often.  for #1, it would be good to fix that if possible just so we don't look bad, but i don't think we have to block on it.20:01
JayFI will second that it anecdotally seems 'worse' in the recent past20:01
timothyb89JayF: thanks for the suggestion, I'll ask over there!20:01
JayFbut I know how useless of a bugreport that makes ;D20:01
fungijeblair: yeah, #1 seems like we could prove it pretty quickly by echoing some envvars if we're really concerned that's not the cause. for #2 we may want some sort of atomicity solution for replacing the full fileset?20:01
jeblairfungi, Shrews, mordred: maybe we can set LOGNAME or something in our ansible invocation20:01
fungiexport LOGNAME=$USER20:02
*** Apoorva has quit IRC20:02
*** PollyZ has joined #openstack-infra20:03
jeblairoh, the change landed20:03
jeblairthat means nodepool needs a restart20:04
*** ffrank has joined #openstack-infra20:04
fungii'm happy to do that unless you're already working on it20:04
lifelessfungi: oh yeah, this time was tabs20:04
jeblairfungi: on it20:04
lifelessfungi: that said, yeah, the multi day fix ain't in...20:04
fungilifeless: right, i thought someone was going to repropose it with an actually longer (not 2-minute) value, but no idea if that happened yet20:05
jeblairShrews: okay, so we just add that to the env when we invoke ansible and make it say whatever we want :)20:05
jeblairShrews: also, you have a 'bubba' account too?20:06
openstackgerritChristian Fetzer proposed openstack-infra/jenkins-job-builder: Add support for Clang Scan-Build plugin version 1.7
*** shamail has quit IRC20:06
Shrewsjeblair: i want to call it 'cthulu'20:06
Shrewscthulhu... however you spell it20:06
fungii recommend the more traditional spelling, "cthulhu"20:06
Shrewsthat too20:06
fungiyeah, that one20:06
jeblairdo not upset it20:07
fungiktulu also works though20:07
xenogearhi guys, i am trying to update task status on wiki but unable to save? i click on save button and nothing happens.  thoughts?20:07
*** matrohon has joined #openstack-infra20:07
AJaegerxenogear: scroll to the top and answer the question20:07
fungixenogear: did you see the captcha up at the top? you need to answer that (spam mitigation)20:07
*** zz_dimtruck is now known as dimtruck20:07
xenogearoh...must've missed it...let me try again, thx20:07
*** baoli has joined #openstack-infra20:07
xenogear:D thanks! worked20:09
fungijeblair: Shrews: if we wanted to get really pedantic, maybe LOGNAME=`id -un`20:09
*** sarob has joined #openstack-infra20:09
pabelangerreview.o.o looks to be back until control of puppet now20:10
fungi(or whoami if we assume coreutils is installed)20:10
jeblairfungi, Shrews: i figure for this purpose, we can just hardcode zuul :)20:10
jeblairthough, who knows what else might use it20:10
jeblair(presumably nothing important if nothing else is broken)20:11
fungiyep, seems fine to me20:11
*** nwkarsten has joined #openstack-infra20:12
*** nwkarsten has quit IRC20:12
*** sarob has quit IRC20:12
*** jcoufal has quit IRC20:12
*** nwkarsten has joined #openstack-infra20:12
*** sarob has joined #openstack-infra20:13
*** akshai has joined #openstack-infra20:13
*** ddieterly is now known as ddieterly[away]20:14
jeblair#status log zl05 - zl07 are in production; jenkins05 - jenkins07 are in prepare for shutdown mode pending decomissioning20:15
* jeblair chases after statusbot20:16
*** ayoung has quit IRC20:18
jeblairit was running but idle; maybe a dead connection (maybe whatever killed zmq)20:18
jeblairrestarted it20:18
*** openstackstatus has joined #openstack-infra20:19
*** ChanServ sets mode: +v openstackstatus20:19
Shrewsjeblair: we can set the env variables with the 'env' param to Popen(), but that means no other env variables will be inherited. i wonder if that would be ok?20:19
jeblairShrews: we can make a copy of the current env first, then update it with that20:19
jeblair#status log zl05 - zl07 are in production; jenkins05 - jenkins07 are in prepare for shutdown mode pending decomissioning20:19
jeblair(it's still joining)20:19
*** tonytan4ever has quit IRC20:19
Shrewsah, yeah20:19
openstackstatusjeblair: finished logging20:20
*** lucas-afk has quit IRC20:21
*** zul has quit IRC20:21
jeblairwhat's with the ovh quota situation?20:21
jeblairit's basically constant exceptions from that in the nodepool log20:22
*** zul has joined #openstack-infra20:22
*** xarses_ has joined #openstack-infra20:22
pabelangerwe also get it from rackspace too20:22
*** Apoorva has joined #openstack-infra20:22
lifelessmordred: has it been deployed?20:22
lifelessmordred: cause, I'm definitely not getting 7 days of login20:23
*** inc0 has quit IRC20:23
*** dmk0202 has joined #openstack-infra20:23
*** gyee has joined #openstack-infra20:24
*** kgiusti has left #openstack-infra20:24
jeblairwe have 58 servers in ovh-gra1.  the error we are receiving is Forbidden: Quota exceeded for instances: Requested 1, but already used 200 of 200 instances (HTTP 403) (Request-ID: req-a73a12b8-c13d-4fb5-bda0-835a080e932a)20:24
jeblairthat makes no sense.20:25
*** flepied has joined #openstack-infra20:25
lifelessjeblair: quota skew would explain it20:26
mordredlifeless: as in - something on the backend is not updating quota properly?20:27
*** ihrachys has joined #openstack-infra20:27
openstackgerritJames E. Blair proposed openstack-infra/project-config: Remove jenkins05 - jenkins07 from nodepool
*** piet has quit IRC20:28
*** lucasagomes has joined #openstack-infra20:28
*** flepied1 has quit IRC20:29
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling
phschwartznibalizer: looks like we are still getting the failure on ensure => ###20:30
phschwartznibalizer: I am back and awake so I will dig back in20:30
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul: Set LOGNAME env var to fix Ansible logging
*** jsavak has quit IRC20:31
openstackgerritJames E. Blair proposed openstack-infra/system-config: Remove jenkins05 - jenkins07
openstackgerritJames E. Blair proposed openstack-infra/system-config: Decomission jenkins05 - jenkins07
pabelangerphschwartz: It think those failures just need a recheck20:31
jeblairthat's the final jenkins decomissioning ^20:31
*** jsavak has joined #openstack-infra20:32
* rbergeron casually hangs out in case anything fun is happening 20:32
Shrewseveryone flee! it's rbergeron20:33
*** javeriak has quit IRC20:33
phschwartzpabelanger: there were 2 rechecks already, I will do another just to see20:33
rbergeronhey nowwwwwwww20:33
pabelangerphschwartz: ya, we rechecked before nibalizer patch landed20:33
*** _nadya_ has quit IRC20:34
Shrewsjeblair: 330775 should fix the logging stuff20:34
pabelangerjeblair: congrats, awesome work20:34
phschwartzpabelanger: he had a recheck at 2:10 est and it ran at 2:30 est20:35
phschwartzpabelanger: was that still before his patch?20:35
pabelangerphschwartz: I believe so.20:36
mordredjeblair: I +A'd the system-config one and +2'd the other20:36
jeblairrbergeron: the exciting thing right now is watching jenkins run down the clock :)20:36
pabelangerphschwartz: We should have used the Depends-On header on your commits to ensure the patch was in the pipeline before recheck20:36
pabelangerphschwartz: but we didn't20:36
pabelangerit has merged now however20:37
phschwartzpabelanger: I will keep an eye on it20:37
*** _nadya_ has joined #openstack-infra20:38
*** sarob_ has joined #openstack-infra20:38
jeblairi'm a little puzzled by zl05 and zl07 having received 0 nodes, and zl06 only having received 120:39
jeblairokay, 05 just got one20:39
*** esikachev has joined #openstack-infra20:40
Shrewsjeblair: is distribution normally evenly weighted?20:41
*** sarob has quit IRC20:42
jeblairShrews: not even remotely.  i think we probably need to fix that nowish.20:42
Shrewsjeblair: something with geard wakeup algorithm i suspect?20:43
rbergeronjeblair: I am not quite sure I know what you mean (I think i do, but :D) -- but I am content to just watch y'all collaborating on the things.20:43
pabelangerwe restarted nodepool recently right? Trying to account for the reason for the stale nodes20:43
jeblairShrews: yeah, i think i'll add an exponential delay tied to the number of currently held nodes in between noop and grab_job20:43
pabelangermaybe 45mins ago?20:43
jeblairpabelanger: yeah20:43
pabelangerokay, doing some clean up20:43
jeblairpabelanger: at 20:0420:43
openstackgerritMerged openstack-infra/puppet-odsreg: Update pip provider to use openstack_pip provider
pabelangerjeblair: Yup, that lines up with that I am seeing20:44
Shrewsjeblair: maybe just a simple LRU algorithm would work20:44
*** ddieterly[away] is now known as ddieterly20:44
mordredrbergeron: tl;dr - we're currently waiting for the jobs that are running on the last 3 jenkins masters to complete before we turn them off20:45
*** PollyZ has quit IRC20:45
*** hemanthm is now known as hemanthm|afk20:46
*** sarob has joined #openstack-infra20:46
rbergeronmordred: I figured -- I was mostly like, "hm, i'm not sure where to see that" (and thus, "am i insane thinking that's the meaning?")20:46
*** bhavik has joined #openstack-infra20:46
Shrewsi wonder what the real gearman does for wakeup20:46
mordredrbergeron: ah. yah. that's not actually a thing you can effectively watch right now. so 'watch' is metaphorical20:47
*** sarob_ has quit IRC20:49
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Enable storyboard for js-openstack-lib
jeblairShrews: it wakes everyone up at ~the same time.  we could have gear lru or randomize that and it may help, but still, network and host latency may end up dwarfing that.20:50
dmsimardmordred: o/ figured you'd be interested in the UI work we've been doing for ARA
dmsimardstill a WIP of course but better than the "before":
*** ihrachys has quit IRC20:52
*** zul has quit IRC20:52
*** baoli has quit IRC20:53
*** ihrachys has joined #openstack-infra20:53
jeblairi think the older zl servers are gaininng enough workload that they're slowing down enough for the new ones to pick stuff up now.20:54
*** tkelsey has joined #openstack-infra20:54
*** aeng has joined #openstack-infra20:54
jeblairShrews: you can see the disparity here:  (thanks pabelanger!)20:55
*** PollyZ has joined #openstack-infra20:55
mordreddmsimard: ooh. that's nice20:56
dmsimardjeblair: that building node graph is so sexy20:56
dmsimardyou can see there is no jenkins, just zuul now :P20:56
mordredthat's the best picture ever20:57
pabelangerjeblair: I would agree with your statement.20:57
*** rhallisey has quit IRC20:58
*** tkelsey has quit IRC20:59
openstackgerritStephanie Miller proposed openstack/diskimage-builder: Ironic agent kernel should be owned by user building image
*** trown is now known as trown|outtypewww21:00
nibalizerpabelanger: recheck all the things!21:01
nibalizeralso the patch upstream in puppet landed, but I don't know where they are on a release21:01
*** _nadya_ has quit IRC21:01
openstackgerritPaul Belanger proposed openstack-infra/project-config: Remove Jenkins masters from grafana dashboard
Shrewsmordred: let's make t-shirts with that picture21:01
*** _nadya_ has joined #openstack-infra21:02
*** _nadya_ has quit IRC21:02
*** baoli has joined #openstack-infra21:02
*** sarob has quit IRC21:03
*** rfolco has quit IRC21:03
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: delay node assignment under load
jeblairmordred, Shrews: ^21:04
*** sarob has joined #openstack-infra21:04
*** aysyd has quit IRC21:06
mordrednibalizer: are they also patching 3.x?21:07
*** chem|off` has quit IRC21:07
nibalizeryah, already landed21:07
nibalizeri actually only patched 3.x, I think we deterimed 4 is unaffected21:08
EmilienMjeblair: we have some jobs that fail for unknown reason, like
mordrednibalizer: cool21:09
jeblair2016-06-16 20:40:27,502 p=20099 u=zuul |  fatal: [node]: FAILED! => {"changed": false, "failed": true, "msg": "async task did not complete within the requested time"}21:10
jeblairEmilienM: it's a timeout ^21:10
jeblairEmilienM: that's in
pabelangerWow, that's the first time I see a job get killed back of async21:10
jeblairEmilienM: we should find a way to put that in the console log21:10
jeblairpabelanger: yeah... :|21:10
EmilienMjeblair: a timeout ? but Gerrit says 19m 46s21:11
*** thorst_ has quit IRC21:11
pabelangerWe are still getting some watchdog events, but haven't debugged it more21:11
jeblair(but the clue is that it did job complete without task complete)21:11
*** thorst_ has joined #openstack-infra21:11
jeblairEmilienM: so do the timestamps.21:12
EmilienM"async task did not complete within the requested time"21:12
*** psilvad has quit IRC21:12
Shrewsjeblair: so, that looks fine. but if we WERE to manage fairly even distribution, then they're all sleeping unnecessarily for the approximate same amount of time. but it should be fairly short, so maybe that's not too big of an issue21:12
jeblairShrews: very short most of the time, 10 seconds at their design capacity; 40 seconds at 2x their design capacity21:13
EmilienMjeblair: what should I do? recheck?21:14
*** roxanagh_ has joined #openstack-infra21:14
jeblairEmilienM: i'm looking to see if i can find any more info21:14
jeblairEmilienM: in the mean time, yes, recheck please21:14
EmilienMour CI is broken we need this patch landed asap21:15
*** gouthamr has quit IRC21:15
EmilienM(puppet CI this time)21:15
*** krtaylor has quit IRC21:15
Shrewsjeblair: that might be good enough for now21:15
*** ldnunes has quit IRC21:16
jeblairShrews: yeah, it's a quick fix for sure21:16
*** thorst_ has quit IRC21:16
jeblairit doesn't look like we log the timeout value we supply to ansible21:16
pabelangerya, I cannot find it either21:16
openstackgerritMerged openstack-infra/puppet-jenkins: Upodate pip provider to use openstack_pip provider
openstackgerritMerged openstack-infra/zuul: Fix referenced before assignment for BuildCompletedEvent
*** roxanagh_ has quit IRC21:18
jeblairi will add a log line for that21:19
*** eharney has quit IRC21:19
*** ihrachys_ has joined #openstack-infra21:19
lifelessmordred: as in, the backend data is plain wrong. There have been lots and lots of consistency holes in quota /usage tracking in nova21:20
*** baoli has quit IRC21:21
*** esikachev has quit IRC21:21
*** baoli has joined #openstack-infra21:21
*** amitgandhinz has quit IRC21:22
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Log ansible command line
jeblairShrews, pabelanger: ^21:22
*** ihrachys has quit IRC21:22
*** bhavik1 has joined #openstack-infra21:23
*** thorst_ has joined #openstack-infra21:23
*** bhavik has quit IRC21:24
*** bhavik1 is now known as bhavik21:24
pabelangerjeblair: Ya, almost wish ansible would also output the timeout value in the FAILED message you linked above too21:24
pabelangergoing to check ansible and see how to add it21:24
openstackgerritMerged openstack-infra/puppet-ansible: Update pip provider to use openstack_pip provider
*** mdrabe has quit IRC21:25
jeblairpabelanger: yeah, we could also put a zuul_log line at the start of the playbook indicating what the job timeout is21:25
Shrewsjeblair: maybe log it with ' '.join(cmd) instead?21:25
pabelangerjeblair: Ya, that would be nice21:26
jeblairShrews: could -- but seeing the individual components gives me a warm fuzzy21:26
jeblairi don't have to second guess shell parsing21:26
jeblair2016-06-16 14:21:09,567 DEBUG zuul.NodeWorker.test: Ansible post command: ['ansible-playbook', '/tmp/tmpbfY2ke/ansible/post_playbook', '-e', 'success=True', '-v']21:26
jeblairit looks like that ^21:26
jeblair2016-06-16 14:21:01,708 DEBUG zuul.NodeWorker.test: Ansible command: ['ansible-playbook', '/tmp/tmpbfY2ke/ansible/playbook', '-e', 'timeout=600', '-v']21:27
jeblairand that ^21:27
*** rossella_s has quit IRC21:27
Shrewsjeblair: far be it from me to remove your warm fuzzies21:27
* mordred hands Shrews a box of used warm fuzzies21:27
* jeblair curls up with his blanket21:28
*** thorst_ has quit IRC21:28
openstackgerritStephanie Miller proposed openstack/diskimage-builder: Ironic agent kernel should be owned by user building image
jeblairthe last jenkins job just completed.21:30
jeblairfor the record, it was
jeblairand it failed21:30
Shrewshow _ironic_ is that21:31
jeblairthere are some leaked nodes in nodepool, i'm cleaning them up now21:31
*** hdaniel has quit IRC21:31
* Shrews cracks himself up21:31
*** bpokorny_ has joined #openstack-infra21:32
mordreddevananda, jroll: congrats. you snagged the last jenkins job in openstack ever21:32
mordredjeblair: the decomission patch is all yours for the +Aing21:32
jrollit's happening?!21:33
* jroll so excite21:33
Shrewsthe excite is much21:33
*** bpokorny has quit IRC21:34
openstackgerritMerged openstack-infra/zuul: Set LOGNAME env var to fix Ansible logging
asselinare upstream dib images working?21:35
mordredasselin: we're fixing our puppet ... there were issues with pypi changes that phschwartz made a bunch of patches for21:36
mordredI think most of them have landed21:36
pabelangershould be fixed now21:36
*** rcernin has quit IRC21:36
pabelangerShrews: jeblair:
*** dimtruck is now known as zz_dimtruck21:37
openstackgerritKhai Do proposed openstack-infra/project-config: Add a build for the gerrit its-storyboard plugin
*** gouthamr has joined #openstack-infra21:37
*** zz_dimtruck is now known as dimtruck21:37
pabelangerfuture proof21:37
*** dimtruck is now known as zz_dimtruck21:37
*** sarob has quit IRC21:37
fungiit's the unjenkinsing!21:37
*** matrohon has quit IRC21:38
*** dmk0202 has quit IRC21:38
* krotscheck is trying to figure out how to build documentation for javascript projects without littering the project with confusing python build harnesses21:39
*** matt-borland has quit IRC21:40
mordredkrotscheck: I recognize the problem and salute you for diving in21:40
fungikrotscheck: what do you want to build the docs from and to?21:40
*** banix has quit IRC21:40
*** Guest17731 has quit IRC21:40
mtreinishkrotscheck: for devstack and other bash things we gave up trying to do that and just used sphinx21:41
mordredI actually used sphinx to document C++ before I used sphinx to document python21:41
asselinmordred, have you seen this error before?
krotscheckLet's try this again, this time with less bad coffeeshop internet21:42
krotscheckfungi: I would to use as much of the existing infrastructure as possible, with an additional "oh yeah go do API docs on this javascript" piece.21:42
EmilienMthis time, job did nothing21:42
fungikrotscheck: worth noting, there are command-line wrappers around some of what we think of as python doc build tooling (e.g. the rst2html command provided by the python-docutils package)21:42
krotscheckAll the pieces are there, however to use the existing jobs I need a setup.cfg,, tox.ini, etc, and the last thing I want to have is someone looking at our projects and going: Urr... is this python or Javascript?21:43
pabelangerEmilienM: looks like the remote node died21:43
fungikrotscheck: but yeah if you want to reuse a lot of our existing jobs and theming then that does get fiddly21:43
mordredkrotscheck: think stopping at sphinx and leaving out setup.* and tox.ini would be better21:43
mordredkrotscheck: the glue to run sphinx right shouldn't be too much to duplicate in js land21:43
*** thorst_ has joined #openstack-infra21:44
krotscheckfungi, mordred: Ya know, if I do this right, it might even work for python things.21:44
asselinmordred, w/o lines cut off:
mordredkrotscheck: in fact, a generic gulp plugin (or whatnot) to deal with sphinx in javascript repos wouldn't be a terrible thing, right?21:45
*** Goneri has quit IRC21:45
asselinthat failing curl works fine outside of DIB21:45
EmilienMpabelanger: recheck again?21:45
pabelangerEmilienM: jeblair: I would have expected zuul_launcher to error out:
jeblairEmilienM, pabelanger: oh, i wonder if losing connectivity to the remote node could also have manifested as the async timeout?21:45
mordredgreghaynes: ^^ see line from asselin ?21:45
pabelangerchecking logs to see why not21:45
jeblairEmilienM, pabelanger: so maybe both errors are the same, and they are both because we stopped being able to log into the node21:46
*** thorst__ has joined #openstack-infra21:46
jeblairEmilienM: any chance this job could affect zuul's ability to log into the node?21:46
*** burgerk has quit IRC21:46
pabelangerjeblair: right, I think this last case, we lost the SSH connection.  But, I would have expected us to requeue in that case.  Maybe we didn't trap it properly21:46
pabelangerchecking the exit code on ansible-playbooks now21:46
EmilienMjeblair: I don't think so21:47
jeblairpabelanger: we handle ansible not being able to initially ssh in.  this is it failing to do so after the job has run for 20 minutes21:47
pabelanger2016-06-16 21:29:46,148 p=30339 u=root |  fatal: [node]: FAILED! => {"failed": true, "msg": "failed to transfer file to /home/jenkins/.ansible/tmp/ansible-tmp-1466112561.94-113511581390802/source:\nsftp> put /tmp/tmpvMj3p4/ansible/scripts/ /home/jenkins/.ansible/tmp/ansible-tmp-1466112561.94-113511581390802/source\n\nmux_client_request_session: read from master21:48
pabelangerfailed: Broken pipe\r\nWarning: Permanently added '' (RSA) to the list of known hosts.\r\nremote open(\"/home/jenkins/.ansible/tmp/ansible-tmp-1466112561.94-113511581390802/source\"): No such file or directory\r\n"}21:48
pabelangerlet me pastebin21:48
greghaynesasselin: huh, that looks like it is running outside the chroot so that kills most of my theories21:48
pabelangerthat is concerning to me21:48
*** woodster_ has quit IRC21:48
openstackgerritKhai Do proposed openstack-infra/project-config: Add a build for the gerrit its-storyboard plugin
*** thorst_ has quit IRC21:49
greghaynesasselin: are you potentially exporting some proxy settings that only effect dib?21:49
mordredI was just about to ask the same question21:50
asselingreghaynes, yes, we're behind a corp proxy21:50
bknudson_I'm wondering why isn't being picked up so that it can verify?21:50
*** maishsk_ has joined #openstack-infra21:50
*** thorst__ has quit IRC21:50
*** chem has joined #openstack-infra21:50
*** mriedem has quit IRC21:51
*** inc0 has joined #openstack-infra21:51
greghaynesasselin: what you can do that will help debug is prepend a break=after-error to your dib command, that will give you a shell right after it fails21:51
greghaynesthen you can test the command and try and debug what is actually going on21:51
jeblairEmilienM: wait, that job was from 2 days ago?21:51
*** yamahata has quit IRC21:51
greghaynesasselin: but I wonder if maybe youre only setting the proxy for dib or vice versa?21:52
asselingreghaynes, this is via nodepool. do you know how to add that?21:52
*** mriedem has joined #openstack-infra21:52
asselingreghaynes, should be both...can't get out either way w/o it21:52
*** mriedem has quit IRC21:52
mordredasselin: adding that string to the list of elements should work21:52
*** chem is now known as chem|off21:52
mordredoh. wait. poo. no, I'm stupid21:53
pabelangerEmilienM: okay, that is an old job too. back on the 06-14, we've made some changes to zuul-launcher to then.  I would recheck21:53
mordredasselin: please do not listen to me :)21:53
*** maishsk has quit IRC21:53
*** maishsk_ is now known as maishsk21:53
pabelangerEmilienM: so, unsure if you linked the right log. because that job didn't run today21:53
*** jsavak has quit IRC21:53
*** jsavak has joined #openstack-infra21:53
*** zz_dimtruck is now known as dimtruck21:53
greghaynesasselin: youll have to just run it by hand - the command should be somewhere in the nodepool log21:54
asselini need to refresh my memory on running it manually then to reproduce21:54
asselinok will look there21:54
EmilienMthis job is from today21:54
EmilienMjeblair, pabelanger21:54
EmilienMit's still in zuul pipeline, see 33069421:54
EmilienMand I pushed the patch today21:54
EmilienMthere is timestamp issue somewhere21:55
fungigonna go get something to eat, then come back and try to work on artifact signing automation some more21:55
pabelangerEmilienM: You linked the following log: from 06-1421:56
jeblairthe log at does not seem to match that job21:56
EmilienMpabelanger, jeblair: and I confirm, no ssh issue, other jobs seem to work fine21:56
mordredpabelanger: yah - I just walked through to the link from the zuul status page21:56
mordred2016-06-14 16:56:00.056599 [Zuul] Job complete, result: FAILURE21:56
mordred2016-06-16 21:26:28.983831 [Zuul] Job complete, result: FAILURE21:56
mordredthat's at the bottom21:56
EmilienMpabelanger: I found the link on zuul.openstack.org21:56
mordredin the console.log21:56
jeblairso we have a console log with apparently 3 jobs21:56
*** tkelsey has joined #openstack-infra21:56
EmilienMand see timestamp in
EmilienMand then in console.log21:56
*** dimtruck is now known as zz_dimtruck21:57
EmilienMthey are 2 days different o_O21:57
pabelangerit this an NTP issue?21:57
mordredEmilienM: congratulations - something is weird21:57
mordredpabelanger: even if it was - why would we have to FAILURE lines21:57
jeblairand the log from a different job21:57
jeblair2016-06-14 13:48:11.702524 JJB template: gate-{name}-puppet-unit-{puppet_version}-{node}21:58
JayFThat explains what I was about to ask about. I have a job that is showing connectivity issues to in console log, but continues and appears to work21:58
JayF(until it fails for another reason)21:58
jeblairit starts as a puppet-unit21:58
jeblairJayF: can you supply a link?21:58
mordredjeblair: did we maybe get the same node attached to two launchers?21:58
JayFthe failure of the job is unrelated to that (it's an apache restart issue we're already aware of)21:58
*** hashar has quit IRC21:59
jeblairJayF: i don't think that's an error21:59
jeblairJayF: i think that just says "ovh does not have ipv6 connectivity"21:59
JayFjeblair: okay, thanks. I'll keep that in mind if I see that again.21:59
*** somerville32 has joined #openstack-infra21:59
EmilienMdid I break something again?22:00
openstackgerritMerged openstack-infra/zuul: Ansible launcher: Log ansible command line
*** tkelsey has quit IRC22:01
*** thiagop has quit IRC22:01
pabelangerI am trying to find with zuul-launcher we ran the latest job on22:01
jeblairEmilienM, pabelanger, mordred:
*** amotoki has joined #openstack-infra22:03
*** csomerville has quit IRC22:03
*** gouthamr_ has joined #openstack-infra22:04
*** dmk0202 has joined #openstack-infra22:04
pabelangerI think we lost SSH access to the node from ansible-playbook for some reason22:04
*** xyang1 has quit IRC22:04
*** sigmavirus24 is now known as sigmavirus24_22:05
*** mixos has quit IRC22:06
*** xarses_ has quit IRC22:06
*** xarses_ has joined #openstack-infra22:07
*** bpokorny_ has quit IRC22:07
jeblairokay, the two jobs ran on a host with the same ip22:07
*** gouthamr has quit IRC22:07
*** dingyichen has joined #openstack-infra22:07
*** ayoung has joined #openstack-infra22:07
*** bpokorny has joined #openstack-infra22:08
pabelanger2 days apart it seems22:08
jeblairnodepool claims to have deleted the server22:09
pabelangerthat make sense, one is centos-7 the other ubuntu-xenial22:10
jeblairthey have different nova ids22:10
*** akshai has quit IRC22:11
*** baoli has quit IRC22:11
*** bhavik has quit IRC22:12
jeblair was the older job22:12
*** PollyZ has quit IRC22:12
*** baoli has joined #openstack-infra22:12
*** PollyZ has joined #openstack-infra22:12
*** bhavik has joined #openstack-infra22:13
jeblair2016-06-14 13:51:55,142 DEBUG zuul.NodeWorker.centos-7-rax-ord-1656082: Ansible output: fatal: [node]: FAILED! => {"ansible_job_id": "649974018264.27026", "changed": false, "failed": true, "finished": 1, "msg": "could not find job", "started": 1}22:13
jeblairthat's what caused the first job to fail22:13
jeblairwhich is also weird22:13
mordredso ....22:14
*** sarob has joined #openstack-infra22:14
mordredif the original node that was requested to be was managed by a compute node that went down, if cleanup didn't happen properly, it's possible that the compute node came back up and fought the new vm that had claimed that IP for the IP - and because the keys are the same for us, ansible would happily log in to the old vm if it was there for a sec22:15
jeblairmordred: in both instances, we have ansible unable to find things on the host that it expected to find22:16
mordredvms coming back from downed compute nodes due to resource tracker stuff is one of the failure cases in nova that I hear people talk about22:16
jeblairmordred: ansible's non-persistent ssh connections could me more suceptible to that than jenkins22:17
*** weshay has quit IRC22:18
pabelangerso between SSH attempts, a new server came online and claim the IP address between SSH attempts?22:18
mordredor the old server erroenously came back on line for a sec22:18
pabelangeralso, between SSH attempts22:18
*** sarob has quit IRC22:19
mordredit's exceptionally odd enough that unless we see a second instance somewhere, I kinda have a hunch it's going to be a cloud-sun-spot22:20
jeblairi found the 3rd job (the one in the middle)22:20
jeblair2016-06-14 16:59:14,314 DEBUG zuul.NodeWorker.ubuntu-trusty-rax-ord-1665155: Ansible output: fatal: [node]: FAILED! => {"ansible_job_id": "709933771768.1907", "changed": false, "failed": true, "finished": 1, "msg": "could not find job", "started": 1}22:20
jeblairit also failed with a similar error ^22:20
pabelangerIn that case, I think we could make the argument to somehow enable host_key_checking per playbook run some how22:20
pabelangerso we know the IP is for the right server22:20
pabelangerthen clean up when job is finished22:20
mordredwe could chose a known-hosts file just for the job, and set ssh to "auto accept but then die if it changes"22:21
jeblairEmilienM: this seems like a reasonable hypothesis that i'd say go ahead and recheck now if you haven't22:21
*** inc0 has quit IRC22:21
pabelangermordred: ya, that might be safe to do22:21
krotscheckmordred: So, I can write a makefile for sphinx things, but that won't get updates from global-requirements.22:21
jeblairby 'reasonable' i mean 'pull my hair out insane' but...22:21
krotscheckIn fact, anything that doesn't use the python tooling doesn't get useful updates from global requirements.22:22
krotscheckmordred: Any suggestions?22:22
*** PollyZ has quit IRC22:23
mordredkrotscheck: I think you're either going to wind up with a weird python-files-in-javascript-repos thing, or you're going to need to write a few pieces of the glue over again in js as a proper js build plugin thing22:23
mordredthat thing can maybe install sphinx and oslo-sphinx into a build venv and then invoke them on the doc/source dir perhaps?22:24
*** sarob has joined #openstack-infra22:24
jeblairmordred, pabelanger: all 3 instances were in rax-ord, so certainly could have been on the same compute host22:24
mordredkrotscheck: or does that sound too gross?22:24
mordredjeblair: yah22:24
*** PollyZ has joined #openstack-infra22:24
pabelangerjeblair: ya, weird for sure22:24
krotscheckmordred: That's what I'm trying to do via make, but keeping the dependencies up to date is the sticking point22:25
jeblairpabelanger, mordred: i think we should call this a reasonable working hypothesis, see if jroll, JayF, or other rackspace folks find it interesting, and then proceed to protect against (and definitively log errors) by checking the host key22:25
*** rlandy has quit IRC22:25
*** jsavak has quit IRC22:26
JayFjeblair: I haven't been following the whole thing; what's the hypothesis/22:26
mordredkrotscheck: yah - I don't think make is usable for exactly that reason22:26
mordredkrotscheck: we looked at make for the python stuff a few years ago and avoided it in favor of python-based for that exact reason22:26
*** baoli has quit IRC22:27
pabelangerjeblair: mordred: in the mean time, I think we can add [ssh_connection] into our ansible.cfg file, modifying ssh_args to use a per job UserKnownHostsFile, pointing to a file that contains the current host key of the node22:27
pabelangerif that isn't too complicated22:27
jeblairJayF: that a deleted instance (briefly?) came back online while another instance had its same ip address.22:27
*** baoli has joined #openstack-infra22:27
jeblairJayF: mordred hypothesizes this might happen in the case of a host failure...22:28
JayFjeblair: I don't know enough about the virt side of the house to rule it out; but I find it exceptionally unlikely. If you all have access to file tickets, please do so with the instance uuids of the instances in question22:28
jeblairi'm collecting nova ids22:28
JayFjeblair: and I can poke someone to look at the ticket interally22:28
JayFjeblair: because I agree that's ... really strange behavior22:28
*** sarob has quit IRC22:28
mordredit's almost never what you'd want :)22:28
*** vhosakot has quit IRC22:29
*** salv-orlando has quit IRC22:29
krotscheckmordred: Well, in that case I could just use jsdoc and ignore sphinx altogether22:29
JayFYeah; I mean, I want to conjecture but I don't have enough real information to do anything more than theorycraft.22:29
*** jamesmcarthur has quit IRC22:30
*** salv-orlando has joined #openstack-infra22:30
*** camunoz has joined #openstack-infra22:30
jeblairthe info for the three nodes is at the top of
mordredkrotscheck: I think for programmer docs that's not insane - javascript devs are going to expect jsdoc docs, no?22:31
krotscheckmordred: Honestly, they're going to expect stackoverflow22:31
jeblairi'm going to open a ticket now22:31
mordredkrotscheck: they are, aren't they?22:31
krotscheckmordred: Yep.22:31
* krotscheck makes his doc build create stackoverflow22:32
pabelangerI wonder if we could use pretask to run ssh-keyscan delegate_to: so when tasks run, ssh will then find the new know_hosts file22:32
*** ddieterly is now known as ddieterly[away]22:32
*** mriedem has joined #openstack-infra22:32
pabelangerwill test that later tonight22:32
*** ddieterly[away] has quit IRC22:32
*** darrenc is now known as darren_afk22:34
*** salv-orlando has quit IRC22:34
*** salv-orlando has joined #openstack-infra22:35
jeblairJayF: 160616-dfw-000188322:36
JayFjeblair: thanks, I've passed it along.22:38
*** dmk0202 has quit IRC22:39
jeblairpabelanger, mordred: i'm going to send my announcement email now; okay?22:40
mordredjeblair: \o/22:40
*** baoli has quit IRC22:40
openstackgerritDoug Hellmann proposed openstack-dev/pbr: generate the reno cache during sdist if reno is installed
dhellmannmordred, lifeless, ttx: ^^22:41
dhellmannfungi : ^^22:41
mordreddhellmann: oooh22:41
dhellmannit feels a bit hacky, but it seems to work22:41
jeblair#success There is no Jenkins, only Zuul.22:42
openstackstatusjeblair: Added success to Success page22:42
jeblair#status log Jenkins is retired:
openstackstatusjeblair: finished logging22:43
jeblairinfra-root, Shrews, notmorgan, AJaeger, anteaya: ^22:43
jeblairapproving 774 now22:44
*** PollyZ_ has joined #openstack-infra22:45
*** asselin__ has joined #openstack-infra22:45
asselinwow...very cool.22:46
nibalizerjeblair: congratz!22:46
*** asselin_ has quit IRC22:47
EmilienMjeblair: again22:47
* mordred hands everyone most of a slice of cake22:47
jeblairi deleted 2 stuck tripleo nodes22:47
mordredEmilienM: it happened again?22:47
EmilienMzuul says job is complete but it's not, we're still running tempest :(22:48
jeblairEmilienM: this is back to the 2016-06-16 22:41:29,490 p=19835 u=root |  fatal: [node]: FAILED! => {"changed": false, "failed": true, "msg": "async task did not complete within the requested time"}22:48
*** PollyZ has quit IRC22:48
jeblairEmilienM: so the kind of failure that looks more like a timeout -- i don't think we hit the phantom instance issue again22:49
*** nwkarsten has quit IRC22:49
jeblairEmilienM: okay, i think we should recheck that and hold the node so we can poke at it and see if there's a connection issue22:49
bknudson_zuul works with jenkins job builder?22:49
jeblairbknudson_: yes, it reads jjb yaml files and translates them to playbooks22:50
bknudson_ah. interesting22:50
mordredbknudson_: you'll find the generated playbooks along side the console logs for jobs22:50
*** nwkarsten has joined #openstack-infra22:51
bknudson_dsvm jobs22:51
*** rbrndt has quit IRC22:51
jeblairEmilienM: let me know when the job starts after you recheck it and i'll hold the node22:52
mordredbknudson_: all of them. like:
bknudson_I must have been looking in the wrong place22:53
*** darren_afk is now known as darrenc22:55
*** gordc has quit IRC22:55
*** nwkarsten has quit IRC22:56
*** bpokorny_ has joined #openstack-infra22:56
*** ffrank has quit IRC22:56
bknudson_I'm glad I'm not writing this playbook by hand --
bknudson_mode: 365  :)22:57
openstackgerritJames E. Blair proposed openstack-infra/system-config: Re-enable in-progress job url links
jeblairpabelanger, mordred: ^22:58
EmilienMjeblair: done, 33069422:58
*** tkelsey has joined #openstack-infra22:58
bknudson_found it -- -- finally runs pep822:59
mordredjeblair: and until we add it, zl runs just won't have a url while it's running, right?22:59
*** xarses_ has quit IRC22:59
jeblairmordred: they have a url -- telnet://22:59
mordredoh. duh.22:59
*** xarses_ has joined #openstack-infra23:00
jeblairso, granted, this will not immediately work as a link for most people23:00
*** bpokorny has quit IRC23:00
anteayajeblair: thank you23:00
jeblairbut i'll mention that in a followup announcement23:00
anteayajeblair: congratulations23:00
openstackgerritMerged openstack-infra/project-config: Remove jenkins05 - jenkins07 from nodepool
*** mixos has joined #openstack-infra23:01
*** piet has joined #openstack-infra23:01
jeblairanteaya: thanks -- be sure to have some cake at jenkins' retirement party tomorrow23:01
anteayazaro: what is its-base?23:01
anteayajeblair: wooo, sounds delicious23:01
anteayazaro: I didn't configure anything to do with its-base that I recall23:01
*** alaski_ is now known as alaski23:02
EmilienMjeblair: I need to leave a bit, I might come back later -- feel free to play with - and thanks for your help23:02
anteayazaro: where are the instructions to configure its-base?23:03
*** tkelsey has quit IRC23:03
*** PollyZ_ has quit IRC23:03
mordredpabelanger: I should be able to land your console-log last patches now, yeah?23:04
mordredpabelanger: since we no longer need to watch jenkins carefully23:04
anteayazaro: I found but the first 3 steps are just links and all 3 links 404 for me23:04
*** ayoung has quit IRC23:07
jeblairEmilienM: i held the node it just started on23:07
jeblairEmilienM: | 1745771 | rax-ord  | None | ubuntu-xenial | zuul   | zl04    | ubuntu-xenial-rax-ord-1745771 | ubuntu-xenial-rax-ord-1745771 | a5aad02d-8bb0-46de-b777-2f2cfd4e194b | | hold  | 00:00:00:00 |23:07
jeblairEmilienM:  nc 808823:07
*** pradk has quit IRC23:07
*** sarob has joined #openstack-infra23:07
*** edmondsw has quit IRC23:07
jeblairi also have an open ssh connection to that host23:07
openstackgerritMerged openstack-infra/puppet-infracloud: Remove special config for west bifrost host
EmilienMjeblair: ack23:08
mordrednc the log is so cool23:08
*** salv-orl_ has joined #openstack-infra23:08
*** sarob has quit IRC23:09
EmilienMoh see23:09
EmilienMit exited23:09
EmilienMah no, it was staled23:09
jeblairyeah, i think we have ~15 minutes to go based on previous runs23:10
*** salv-orlando has quit IRC23:11
jeblairpabelanger: jenkins05-07 still exist23:12
jeblairpabelanger: which means ansible will run jjb on them for the console-log jjb change23:12
jeblairpabelanger: i'm waiting for 330776 to land23:12
jeblairmordred: oh, you approved those ^23:13
mordredjeblair: I can unapprove real quick23:13
jeblairmordred: might be a good idea23:13
openstackgerritMerged openstack-infra/system-config: Add Zuul as a first-party CI name
mordredjeblair: I dropped a -2 block23:14
mordredjeblair: batch mode in gertty would not let me just remove my +A fwiw23:14
openstackgerritIan Wienand proposed openstack/diskimage-builder: Pre-install pip/virtualenv packages
*** roxanagh_ has joined #openstack-infra23:15
jeblairmordred: hrm23:16
mordredjeblair: I figured blocking the merge was more important than troubleshooting23:16
*** claudiub|2 has joined #openstack-infra23:16
mordredjeblair: when I opened the "vote" box, it did not load that my previous one was W+1 - so when I voted, it was with +0s which it seemingly took to be no-op23:17
*** hongbin has quit IRC23:17
*** asselin has quit IRC23:17
jeblairmordred: makes sense; that case may not be handled well.23:18
*** sarob has joined #openstack-infra23:18
openstackgerritMerged openstack-infra/system-config: Decomission jenkins05 - jenkins07
*** claudiub has quit IRC23:20
*** roxanagh_ has quit IRC23:20
*** ihrachys_ has quit IRC23:20
openstackgerritIan Wienand proposed openstack/diskimage-builder: Pre-install pip/virtualenv packages
*** ihrachys has joined #openstack-infra23:21
jeblairrestarted nodepool after the final config change23:21
*** bhavik has quit IRC23:22
*** amitgandhinz has joined #openstack-infra23:23
*** claudiub has joined #openstack-infra23:23
bknudson_oh, now I know why I didn't see the ansible script is because this just started23:24
*** isaacb has joined #openstack-infra23:24
jeblairbknudson_: we've been phasing it in over the past 2 weeks23:24
bknudson_oh, so just unlucky and picked one that didn't have it23:25
jeblairbut yeah, hit the 100% mark a couple hours ago23:25
mordredjeblair: k. so, it should be safe to re-approve now that 330776 has landed, yeah?23:26
jeblairmordred: after i delete the hosts, after ansible runs23:26
mordredjeblair: there's no jenkins.o.o in site.pp - ansible might run on them, but it shouldn't run jjb?23:27
mordredit should relaly just ensure our ssh keys are still there23:27
*** claudiub|2 has quit IRC23:27
*** amitgandhinz has quit IRC23:27
jeblairmordred: good point23:27
mordred but I'll wait anyway23:27
mordredjust cause23:27
jeblairmordred, pabelanger: 119 nodes are currently involved in testing that patchset :(23:29
EmilienMjeblair: tempest still running /me cross finger23:29
*** sarob has quit IRC23:29
jeblairer, patch series23:29
mordredpabelanger: how about we squash it into one patch23:29
jeblairi'd like us to reinstate the files stanzas for project-config too23:30
mordredwe don't need to phase it in anymore23:30
jeblairmultiple patches can still be handy for dealing with rebases in this sort of thing23:30
mordredyah. but in this case, if I push it up squashed now, it should cancel the other builds, and then do less when we his +A in just a smidge23:31
pabelangermordred: jeblair: Ya, squashing is fine. not that jenkins is gone23:31
* mordred on that23:32
openstackgerritMonty Taylor proposed openstack-infra/project-config: Ensure console-log is our last publisher
openstackgerritJames E. Blair proposed openstack-infra/project-config: Revert "Remove skip test logic for project-config"
openstackgerritMerged openstack-infra/zuul: Ansible launcher: use .txt as logfile extension
pabelangerjeblair: Ya, we'll need to also add other-requirements.txt now, but we can do that in a follow up patch23:34
openstackgerritMerged openstack-infra/system-config: Remove Jenkins documentation; replace with zuul-launcher
openstackgerritMerged openstack-infra/system-config: Remove jenkins playbooks and tools
EmilienMjeblair: it worked /me afk now23:35
fungijeblair: awesome work!23:36
jeblairpabelanger: er, i meant for us to hold off on approving that until we were sure we had consensus23:36
jeblairpabelanger: cause i don't want to have a revert war with other infra-roots23:36
mordredjeblair: should we drop compare-xml ? like, is it valuable anymore?23:36
pabelangerjeblair: Oh23:36
*** sarob has joined #openstack-infra23:36
mordredjeblair, pabelanger: I'm in favor, so count me in23:36
pabelangerjeblair: I read it as, revert, because original patch didn't have consensus23:37
jeblairoh, i guess i could have worded that more clearly23:37
* fungi is catchign up now on how/why we had several jobs in one log23:37
jeblaireither way, it's not so important we can't let this one simmer for a bit23:38
jeblairat least long enough for clarkb to -2 it if he wants ;)23:38
jeblairfungi: yeah. um.  wow.  there's a ticket.23:38
clarkbI have internets at new house now and truck ewnted saturday so next few days are sort of busy23:40
fungimordred: krotscheck: i'd vote for using distro packages of the utilities/libs you need to generate docs and add an other-requirements.txt file listing those as [test] profile entries, so that we'll autoinstall the tools needed to build javascript docs during the job run. relying on distro packages for non-javascript things seems slightly less weird than expecting someone to pip install stuff to build23:41
fungijavascript docs23:41
mordredclarkb: \o/23:42
*** chlong has quit IRC23:42
mordredclarkb: also, while you were out, jeblair finished getting rid of jenkins23:42
*** ayoung has joined #openstack-infra23:44
jeblairit seems like geard thinks it has more workers than it really does23:45
*** piet has quit IRC23:45
jeblairi'm unsure of the cause, but we may want to restart zuul23:45
*** bpokorny_ has quit IRC23:45
jeblairincluding workers for nodes which have been deleted23:46
*** piet has joined #openstack-infra23:46
fungiif the zuul scheduler/geard hasn't been restarted since the last jenkins master went down, we probably ought to do that just so that we're only getting job registrations from zuul-launchers23:46
*** bpokorny has joined #openstack-infra23:46
jeblairoh huh23:47
fungithough in theory they _should_ be identical entries23:47
*** bhavik has joined #openstack-infra23:47
jeblairi think this may be a case where nodepool deleted the node before zuul cleanly shut it down23:47
fungioh that's fun. race?23:47
*** salv-orl_ has quit IRC23:47
jeblairwell, i think that without a keepalive on that connection, geard won't notice it's gone23:47
jeblairoh, but the connection comes from the launcher23:48
jeblairso, erm, i dunno.23:48
*** salv-orlando has joined #openstack-infra23:48
*** xarses_ has quit IRC23:48
*** kzaitsev_mb has quit IRC23:49
openstackgerritMerged openstack-infra/zuul: Ansible launcher: delay node assignment under load
zaroanteaya: its-base is a common code that all its (issue tracking system) plugins depend on.23:51
clarkbmordred: I am happy to report that my fios experience so far has been good. they terminate rj45 not coax which is nice and I can get up to gigabits if I pay lots of money23:51
*** ihrachys has quit IRC23:51
mordredclarkb: I'll be interested to hear if you get intermittent pauses from time to time23:51
mordredclarkb: mine actually seems to be better since I've been back23:51
mordredso it might have just be a point in time thing too23:51
*** Julien-zte has quit IRC23:52
jeblairrestarting zuul23:52
anteayazaro: I don't recall installing and configuring its-base in my local gerrit23:54
anteayazaro: have you a link to any instructions on how I should install and configure?23:54
mordredjeblair: aw crap. we have a bunch of conference talk slides to update now23:54
*** johnny_ has quit IRC23:54
jeblairmordred: i don't plan on speaking at conferences until i have something to say23:54
zaroanteaya: you don’t need to install it.  just installing its-storyboard.jar was all that was needed23:54
anteayazaro: oh, I was hoping that was the thing I did wrong on my installation23:55
anteayazaro: since you call out its-base specificially in the commit message for your patch23:55
jeblair#status log all jenkins servers have been deleted23:56
jeblairmordred: ^23:56
openstackstatusjeblair: finished logging23:56
*** sdake has quit IRC23:57
jeblairand i cleared the ansible inventory cache23:57
zaroanteaya: we need to build its-storyboard and put it on tarballs.o.o so we can install it with puppet23:59
zaroanteaya: building requires building its-base.23:59
zaroso its-base is only required for building not required to run23:59

Generated by 2.14.0 by Marius Gedminas - find it at!