Friday, 2016-04-01

anteayaoh I love that idea00:00
*** salv-orlando has joined #openstack-infra00:00
anteayait is a great idea00:00
anteayait is the reality I find that sometimes comes up short00:00
pleia2clarkb: so, arch with CI? :)00:00
clarkbpleia2: bsaically00:01
*** tsg has joined #openstack-infra00:01
openstackgerritIan Wienand proposed openstack/diskimage-builder: dhcp-all-interfaces depends on dhcp
ianwgreghaynes / prometheanfire : ^ i think that keeps everyone happy00:04
*** salv-orlando has quit IRC00:06
*** fawadkhaliq has quit IRC00:07
*** fawadkhaliq has joined #openstack-infra00:07
prometheanfireianw: we do have an isc dhcp client, but I think I'd rather use dhcpcd as it's simpler00:07
prometheanfireI don't need that script or a custom init script for instance :P00:07
bkeroheh dhcpcd00:08
*** gyee has quit IRC00:08
ianwright, it maps isc-dhcp-client -> dhcpcd right?00:08
prometheanfirebkero: wat :P00:08
prometheanfireianw: nope :P00:08
prometheanfirenet-misc/dhcp -      Description:         ISC Dynamic Host Configuration Protocol (DHCP) client/server00:08
bkerodhcpcd is a different dhcp client implementation00:08
ianwprometheanfire: i mean the package map does that00:08
prometheanfiredhcpcd -      Description:         A fully featured, yet light weight RFC2131 compliant DHCP client00:08
bkeroprometheanfire: Oh nothing. Just giggling at the gentoo reference00:08
prometheanfireoh, sure00:08
ianwmaybe we should call it something else00:08
prometheanfirebkero: iirc, it's a bsd thing :P00:09
*** sflanigan has joined #openstack-infra00:10
*** sflanigan has joined #openstack-infra00:10
bkeroFunny, I don't think it's the default in freebsd00:10
prometheanfireianw: that script doesn't work for us, so I'm happy keeping the mapping to dhcpcd, all the workaround you guys do for ubuntu is funny :P00:10
prometheanfirebkero: he refrences netbsd, dunno00:11
*** ayoung has joined #openstack-infra00:11
*** aeng has quit IRC00:11
prometheanfireianw: I guess the question is weather or not we want to diverge gentoo to have dhcpcd instead of what everyone else has.00:12
openstackgerritIan Wienand proposed openstack/diskimage-builder: Use generic "dhcp-client" name
*** mylu has joined #openstack-infra00:14
ianwok, maybe that makes thing clearer.  l like the idea we just specify "dhcp-client" and let each platform figure it out00:14
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental ubuntu-xenial puppet-apply job
*** mylu has quit IRC00:15
prometheanfireianw: worksforme00:16
prometheanfireianw: did your edit go through00:16
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add ubuntu-xenial to openstack_project::single_use_slave
ianwprometheanfire: which edit?00:17
prometheanfireoh, that was another review00:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental bindep-fallback job for ubuntu-xenial
*** MarkAtwood has quit IRC00:20
*** jaypipes has quit IRC00:21
*** hichihara has joined #openstack-infra00:21
*** mtanino has quit IRC00:21
pabelangerclarkb: ubuntu-xenial uploads happing now00:23
pabelangerhave a few patches up to enable experimental jobs for puppet-apply and bindep00:23
pabelangerwe shouldn't move anything to check / gate until ubuntu xenial is stable release00:23
*** aeng has joined #openstack-infra00:24
prometheanfirepabelanger: are gate checks moving to 16.04 already?00:24
pabelangerprometheanfire: we're just adding the ubuntu-xenial now, how projects use it is another story00:25
clarkbpabelanger: ya I wouldnt switch voting jobs until post release00:25
prometheanfirethought the default was switch and freaking out :P00:25
clarkbjust because another openjdk can happen00:25
pabelangerhonestly, I don't know the migration plan for the projects.  I'm just making sure our infra works on xenial00:25
pabelangerclarkb: ++00:25
pabelangerprometheanfire: no, ubuntu-trusty will be around for a while yet00:25
clarkbpabelanger: prometheanfire we do someehat forcefullu switch the base jobs00:26
pabelangerwe have stable branches to support00:26
clarkbso unittests and devstack will all likely move sometime in may00:26
clarkband yes mitaka will run on trusty00:26
clarkbwe have so many jobs now though... I dont intend on personally proofing each one00:28
pabelangerand ubuntu-xenial on bluebox00:30
clarkbpabelanger an experimental set of jobs for grenade, devstack and devstack gate would probably be good00:30
pabelangerYa, I've had people ask about that already00:30
pabelangerI want to confirm and work first. So jenkins can attach, then setup the devstack experimental job00:31
openstackgerritFei Long Wang proposed openstack/requirements: Add tempest to requirements
pabelangerthen let people hack00:31
clarkbprometheanfire: and fwiw the switch last time was pretty smooth00:32
pabelangermaybe give fedora-24 a try in the morning! cc ianw00:33
clarkbthis time might be harder due to systemd00:33
prometheanfirewe do have a systemd profile though :D00:33
*** Qiming has quit IRC00:34
*** tiswanso has quit IRC00:39
*** mylu has joined #openstack-infra00:39
*** tiswanso has joined #openstack-infra00:39
*** mylu has quit IRC00:41
*** tqtran has quit IRC00:41
*** mylu has joined #openstack-infra00:42
*** _amrith_ is now known as amrith00:42
*** mylu has quit IRC00:44
*** thorst has joined #openstack-infra00:44
*** mylu has joined #openstack-infra00:44
*** tsg has quit IRC00:49
*** moravec has quit IRC00:50
*** jerryz has quit IRC00:50
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Add fix_disk_layout script and builder
*** mylu has quit IRC00:50
*** tsg has joined #openstack-infra00:51
*** thorst has quit IRC00:51
*** mylu has joined #openstack-infra00:52
*** markvoelker_ has quit IRC00:53
*** Daisy has joined #openstack-infra00:54
*** edmondsw has joined #openstack-infra00:54
*** tsg has quit IRC00:56
*** gildub has joined #openstack-infra00:56
*** Daisy_ has joined #openstack-infra00:56
*** Daisy__ has joined #openstack-infra00:58
*** Jeffrey4l has joined #openstack-infra01:00
*** baoli has joined #openstack-infra01:00
*** Swami has quit IRC01:00
*** aeng has quit IRC01:00
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for infra traceroute6 bug 1564688
openstackbug 1564688 in OpenStack-Gate "/tmp/ traceroute6: not found" [Undecided,New]
*** Daisy has quit IRC01:00
*** esikachev has joined #openstack-infra01:02
*** Daisy_ has quit IRC01:02
*** tqtran has joined #openstack-infra01:03
*** mylu has quit IRC01:03
*** mylu has joined #openstack-infra01:04
*** esikachev has quit IRC01:06
*** tqtran has quit IRC01:07
*** mylu has quit IRC01:09
*** pvaneck has quit IRC01:10
openstackgerritMerged openstack-infra/elastic-recheck: Add query for infra traceroute6 bug 1564688
openstackbug 1564688 in OpenStack-Gate "/tmp/ traceroute6: not found" [Undecided,Fix released] - Assigned to Jeremy Stanley (fungi)01:11
*** angdraug has quit IRC01:14
*** mylu has joined #openstack-infra01:15
*** Daisy__ has quit IRC01:16
*** thorst has joined #openstack-infra01:17
*** Jeffrey4l has quit IRC01:17
*** asettle has quit IRC01:19
*** tiswanso has quit IRC01:20
*** tiswanso has joined #openstack-infra01:21
*** mylu has quit IRC01:22
*** thorst has quit IRC01:23
*** mylu has joined #openstack-infra01:24
*** knikolla has quit IRC01:25
*** woodster_ has quit IRC01:27
*** mtanino has joined #openstack-infra01:28
openstackgerritShamail Tahir proposed openstack-infra/irc-meetings: Add Non-ATC Recognition WG Meeting
*** edmondsw has quit IRC01:30
*** Qiming has joined #openstack-infra01:31
*** krtaylor has joined #openstack-infra01:34
*** fawadkhaliq has quit IRC01:40
*** thorst has joined #openstack-infra01:42
*** moravec has joined #openstack-infra01:43
*** thorst has quit IRC01:43
*** zz_dimtruck is now known as dimtruck01:44
*** apoorvad has quit IRC01:44
*** mriedem1 has joined #openstack-infra01:45
*** mriedem has quit IRC01:47
openstackgerritFei Long Wang proposed openstack/requirements: Add tempest to requirements
*** yamahata has quit IRC01:49
*** mylu has quit IRC01:50
*** fawadkhaliq has joined #openstack-infra01:54
*** dimtruck is now known as zz_dimtruck01:55
*** asettle has joined #openstack-infra01:55
*** SumitNaiksatam has joined #openstack-infra01:56
fungiwhat, so jenkins07 still wasn't fixed even after rerunning jjb on it again?01:56
fungiwe seem to have very spotty puppet updates going on01:56
clarkbit claimed to be up to date but didnt really update until I flushed cache01:57
fungigot it. so we still have cache weirdness sounds like01:57
*** baoli has quit IRC01:58
*** baoli has joined #openstack-infra01:58
fungifrom now on i'll double-check a job on every single master i guess, not just spot check a few of them01:58
*** zz_dimtruck is now known as dimtruck01:59
*** Sukhdev has quit IRC01:59
*** asettle has quit IRC02:01
*** tqtran has joined #openstack-infra02:04
*** bpokorny has quit IRC02:06
*** shashank_hegde has quit IRC02:07
*** tqtran has quit IRC02:08
*** tphummel_ has quit IRC02:10
*** achanda has joined #openstack-infra02:14
*** mylu has joined #openstack-infra02:19
*** asettle has joined #openstack-infra02:23
*** akshai_ has quit IRC02:23
*** yamamoto has joined #openstack-infra02:24
*** baoli has quit IRC02:25
*** amitgandhinz has joined #openstack-infra02:27
*** EricGonc_ has quit IRC02:32
*** fawadkhaliq has quit IRC02:33
*** mriedem1 has quit IRC02:38
*** asettle has quit IRC02:39
*** aeng has joined #openstack-infra02:41
*** fawadkhaliq has joined #openstack-infra02:42
*** baoli has joined #openstack-infra02:42
*** amitgandhinz has quit IRC02:43
*** fawadkhaliq has quit IRC02:43
*** asettle has joined #openstack-infra02:45
*** baoli has quit IRC02:45
*** fawadkhaliq has joined #openstack-infra02:45
*** asettle has quit IRC02:50
*** shashank_hegde has joined #openstack-infra02:50
*** yuanying has quit IRC02:51
*** sflanigan is now known as sflanigan_afk02:52
*** fawadkhaliq has quit IRC02:53
*** fawadkhaliq has joined #openstack-infra02:53
*** yamamoto has quit IRC02:54
*** flepied has quit IRC02:54
*** esikachev has joined #openstack-infra03:03
openstackgerritKhai Do proposed openstack-infra/system-config: Change download command to point to git.o.o
*** sflanigan_afk is now known as sflanigan03:07
*** esker has joined #openstack-infra03:07
*** esikachev has quit IRC03:08
*** mylu has quit IRC03:08
*** fawadkhaliq has quit IRC03:08
*** bpokorny has joined #openstack-infra03:09
*** mylu has joined #openstack-infra03:09
*** fawadkhaliq has joined #openstack-infra03:10
*** mylu has quit IRC03:11
*** mylu has joined #openstack-infra03:11
*** fawadkhaliq has quit IRC03:12
*** krotscheck is now known as krotscheck_vaca03:20
*** fawadkhaliq has joined #openstack-infra03:32
*** amitgandhinz has joined #openstack-infra03:34
*** fedexo has joined #openstack-infra03:38
*** links has joined #openstack-infra03:41
*** yamamoto has joined #openstack-infra03:43
*** markvoelker has joined #openstack-infra03:47
*** yuanying has joined #openstack-infra03:49
*** mylu has quit IRC03:54
*** markvoelker_ has joined #openstack-infra03:56
*** markvoelker has quit IRC03:56
*** kushal has quit IRC04:01
*** amitgandhinz has quit IRC04:01
*** achanda has quit IRC04:04
*** bpokorny has quit IRC04:06
*** achanda has joined #openstack-infra04:08
*** mylu has joined #openstack-infra04:10
*** mylu has quit IRC04:10
*** mylu has joined #openstack-infra04:13
*** achanda has quit IRC04:13
*** kushal has joined #openstack-infra04:14
*** asselin__ has joined #openstack-infra04:17
*** asettle has joined #openstack-infra04:17
*** asselin has quit IRC04:18
*** harlowja_at_home has joined #openstack-infra04:19
*** asettle has quit IRC04:21
*** Sukhdev has joined #openstack-infra04:25
*** fawadkhaliq has quit IRC04:26
*** sdake has joined #openstack-infra04:29
*** markvoelker_ has quit IRC04:33
*** markvoelker has joined #openstack-infra04:33
*** dimtruck is now known as zz_dimtruck04:34
*** markvoelker has quit IRC04:38
openstackgerritMerged openstack-infra/project-config: Move periodic-package-afs-centos7 to centos-7
*** mylu has quit IRC04:41
*** mtreinish has quit IRC04:44
*** kengo_sakai has joined #openstack-infra04:44
*** mylu has joined #openstack-infra04:46
*** shashank_hegde has quit IRC04:51
*** mylu has quit IRC04:57
*** rcernin has joined #openstack-infra04:57
*** fawadkhaliq has joined #openstack-infra04:57
*** achanda has joined #openstack-infra05:02
*** maishsk has joined #openstack-infra05:02
*** stevemar_ has quit IRC05:02
*** stevemar has joined #openstack-infra05:02
*** esikachev has joined #openstack-infra05:04
*** mylu has joined #openstack-infra05:05
*** torandu has quit IRC05:06
*** torandu has joined #openstack-infra05:06
*** maishsk has quit IRC05:06
*** esikachev has quit IRC05:08
*** torandu has quit IRC05:08
*** torandu has joined #openstack-infra05:09
*** achanda has quit IRC05:10
*** maishsk has joined #openstack-infra05:11
fricklerdoes exist intentionally or is that some stray artefact?05:12
khappone_Hi, a question for the infra people. What are the usage policies of the openstack etherpad? We're starting a finnish monthly operators video call, and we're wondering if we could use the openstack etherpad for notes?05:15
*** [1]Thelo has joined #openstack-infra05:18
*** harlowja_at_home has quit IRC05:19
*** Thelo has quit IRC05:20
*** [1]Thelo is now known as Thelo05:20
*** tpeoples is now known as tpeoples_vacatio05:23
*** tpeoples_vacatio is now known as tpeoplesvacation05:24
*** davideag_ has joined #openstack-infra05:24
*** jaosorior has joined #openstack-infra05:25
*** davideagnello has quit IRC05:26
*** shashank_hegde has joined #openstack-infra05:28
*** markvoelker has joined #openstack-infra05:29
*** toabctl has left #openstack-infra05:29
*** sdake_ has joined #openstack-infra05:30
*** sdake_ has quit IRC05:31
*** sdake has quit IRC05:32
*** markvoelker has quit IRC05:37
*** yolanda has quit IRC05:39
*** davideag_ has quit IRC05:42
*** asettle has joined #openstack-infra05:43
*** mylu has quit IRC05:43
*** shashank_hegde has quit IRC05:45
*** kzaitsev_mb has joined #openstack-infra05:46
*** mylu has joined #openstack-infra05:48
*** mylu has quit IRC05:48
*** shashank_hegde has joined #openstack-infra05:49
*** mtreinish has joined #openstack-infra05:52
*** kzaitsev_mb has quit IRC05:53
*** ifarkas has quit IRC05:54
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element
*** yamahata has joined #openstack-infra05:58
*** sputnik13net has quit IRC05:58
*** fedexo has quit IRC06:01
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job
AJaegerkhappone_: Is that an OpenStack operator meeting?06:04
*** tqtran has joined #openstack-infra06:06
*** ramishra has quit IRC06:06
*** YorikSar has quit IRC06:06
*** YorikSar has joined #openstack-infra06:08
*** ramishra has joined #openstack-infra06:08
*** tqtran has quit IRC06:10
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** MarkAtwood has joined #openstack-infra06:11
*** mikelk has joined #openstack-infra06:13
*** yolanda has joined #openstack-infra06:15
AJaegerinfra-root, tripleo-quickstart merged yesterday but setup looks broken on,access . I don't see the import from the upstream repo run either. Details at
*** armax has quit IRC06:18
*** MarkAtwood has quit IRC06:20
*** zhurong_ has joined #openstack-infra06:22
*** Sukhdev has quit IRC06:24
*** mikelk has quit IRC06:29
*** achanda has joined #openstack-infra06:30
*** camunoz has quit IRC06:31
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** asettle has quit IRC06:35
*** sdake has joined #openstack-infra06:36
*** fawadkhaliq has quit IRC06:38
*** fawadkhaliq has joined #openstack-infra06:38
*** asettle has joined #openstack-infra06:39
khappone_AJaeger: yes, it's an openstack operators meeting, although in no official capacity, just a bunch of us that decided that common rant sessiosn might be useful06:39
AJaegerkhappone_: I'm not aware of any official policy but if it's an openstack operator meeting, just go ahead. I consider that perfectly fine usage.06:40
khappone_AJaeger: Thanks! Yeah, it seems like the use case would fit06:41
*** akaszuba has joined #openstack-infra06:45
*** jlanoux has joined #openstack-infra06:47
*** rlandy has quit IRC06:52
*** asettle has quit IRC06:52
AJaegerkhappone_: it's community infrastructure and operators - especially finnish ones  ;) - are part of the community06:52
*** gnuoy_ has joined #openstack-infra06:54
*** gnuoy_ has quit IRC06:54
*** shashank_hegde has quit IRC06:55
openstackgerritKoteswara Rao Kelam proposed openstack-infra/project-config: Import from existing repository.
*** asettle has joined #openstack-infra06:57
*** asettle has quit IRC07:02
*** salv-orlando has joined #openstack-infra07:05
*** aeng has quit IRC07:06
*** MarkAtwood has joined #openstack-infra07:07
*** e0ne has joined #openstack-infra07:07
*** MarkAtwood has quit IRC07:11
*** |-paul-| has joined #openstack-infra07:12
*** asettle has joined #openstack-infra07:14
*** asettle has quit IRC07:14
*** gildub has quit IRC07:16
*** dizquierdo has joined #openstack-infra07:16
*** matrohon has joined #openstack-infra07:16
*** jamielennox is now known as jamielennox|away07:17
*** dizquierdo has quit IRC07:18
*** dizquierdo has joined #openstack-infra07:18
*** markvoelker has joined #openstack-infra07:19
*** kzaitsev_mb has joined #openstack-infra07:19
*** zeih has joined #openstack-infra07:20
*** salv-orl_ has joined #openstack-infra07:23
*** mrmartin has joined #openstack-infra07:23
*** kzaitsev_mb has quit IRC07:25
*** pcaruana has joined #openstack-infra07:25
*** markvoelker has quit IRC07:25
*** salv-orlando has quit IRC07:26
*** mrmartin has quit IRC07:29
therveWe're seeing quota issues when running grenade tests in Heat07:30
*** lezbar has joined #openstack-infra07:30
therveDid something change? Is it an issue with a provider?07:30
*** nadya has joined #openstack-infra07:31
*** yaume has joined #openstack-infra07:32
*** nadya has quit IRC07:34
*** e0ne has quit IRC07:35
*** asselin__ has quit IRC07:35
*** e0ne has joined #openstack-infra07:35
*** hichihara has quit IRC07:35
*** dtantsur|afk is now known as dtantsur07:37
*** jcoufal has joined #openstack-infra07:39
*** e0ne has quit IRC07:40
*** yamahata has quit IRC07:42
*** shashank_hegde has joined #openstack-infra07:43
*** e0ne has joined #openstack-infra07:45
*** jaosorior has quit IRC07:46
*** jaosorior has joined #openstack-infra07:46
*** ociuhandu has quit IRC07:51
*** amotoki has joined #openstack-infra07:52
*** e0ne has quit IRC07:54
*** tesseract has joined #openstack-infra07:56
*** fawadkhaliq has quit IRC07:56
*** tesseract is now known as Guest9087707:56
*** e0ne has joined #openstack-infra07:57
*** abregman has joined #openstack-infra07:58
*** dtantsur is now known as dtantsur|bbl07:59
*** arxcruz has joined #openstack-infra08:01
*** ifarkas has joined #openstack-infra08:02
*** yolanda has quit IRC08:04
*** shashank_hegde has quit IRC08:05
*** kzaitsev_mb has joined #openstack-infra08:06
*** jistr has joined #openstack-infra08:07
*** MarkAtwood has joined #openstack-infra08:08
*** mikelk has joined #openstack-infra08:08
*** mrmartin has joined #openstack-infra08:08
*** e0ne has quit IRC08:10
*** MarkAtwood has quit IRC08:12
*** e0ne has joined #openstack-infra08:13
*** cdent has joined #openstack-infra08:16
*** baoli has joined #openstack-infra08:16
*** e0ne has quit IRC08:16
*** kzaitsev_mb has quit IRC08:18
*** sflanigan has quit IRC08:19
*** toabctl has joined #openstack-infra08:20
*** e0ne has joined #openstack-infra08:20
*** baoli has quit IRC08:20
*** kzaitsev_mb has joined #openstack-infra08:21
*** sfinucan has joined #openstack-infra08:22
*** achanda has quit IRC08:22
*** cdent has quit IRC08:24
*** cdent has joined #openstack-infra08:24
*** achanda has joined #openstack-infra08:25
*** e0ne has quit IRC08:26
*** e0ne has joined #openstack-infra08:29
*** cdent has quit IRC08:30
*** e0ne_ has joined #openstack-infra08:31
*** e0ne has quit IRC08:32
*** achanda has quit IRC08:32
*** hashar has joined #openstack-infra08:32
*** cdent has joined #openstack-infra08:32
*** e0ne_ has quit IRC08:33
*** achanda has joined #openstack-infra08:38
*** flepied has joined #openstack-infra08:39
openstackgerritKairat Kushaev proposed openstack/requirements: Bump oslo.context >= 2.2.0 to store roles
*** vgridnev has joined #openstack-infra08:44
*** e0ne has joined #openstack-infra08:48
*** asilenkov has quit IRC08:48
*** katyafervent_awa has quit IRC08:48
*** amit213 has quit IRC08:49
*** abregman is now known as abregman|afk08:51
*** asilenkov has joined #openstack-infra08:51
*** kzaitsev_mb has quit IRC08:51
AJaegertherve: do you have a log file to point to?08:52
*** e0ne has quit IRC08:53
*** kzaitsev_mb has joined #openstack-infra08:53
AJaegertherve: do you have a direct link to the file/line that shows it?08:55
*** gnuoy_ has joined #openstack-infra08:55
therveAJaeger, sure
*** katyafervent_awa has joined #openstack-infra08:56
*** gnuoy_ has quit IRC08:57
therveAt the end of it08:57
AJaegertherve: I looked at the others first ;)08:57
*** achanda has quit IRC08:57
odyssey4megerrit seems to be very slow loading over the last 10 mins or so, is this being experienced by others?08:57
AJaegerinfra-root, could you check that log file by therve - this is bluebox ^08:58
*** kzaitsev_mb has quit IRC08:58
*** e0ne has joined #openstack-infra08:58
AJaegertherve: thanks, hope somebody else can look further. I can't help directly further08:59
therveAJaeger, No problem, thank you08:59
*** MarkAtwood has joined #openstack-infra09:00
therveIt seems fairly localized, not sure it has happened since then09:01
*** mrmartin has quit IRC09:01
*** MarkAtwood has quit IRC09:02
*** MarkAtwood has joined #openstack-infra09:02
*** vgridnev has quit IRC09:04
*** MarkAtwood has quit IRC09:04
*** MarkAtwood has joined #openstack-infra09:04
*** MarkAtwood has quit IRC09:06
*** kelepirci_ has joined #openstack-infra09:06
openstackgerritFerhat Ozkasgarli proposed openstack-infra/project-config: i18n Support for freezer-web-ui
*** vgridnev has joined #openstack-infra09:06
*** shardy has joined #openstack-infra09:10
*** markvoelker has joined #openstack-infra09:11
*** MarkAtwood has joined #openstack-infra09:12
*** sdake has quit IRC09:13
*** MarkAtwood has quit IRC09:14
*** fhubik has joined #openstack-infra09:14
*** esikachev has joined #openstack-infra09:15
*** MarkAtwood has joined #openstack-infra09:16
*** markvoelker has quit IRC09:16
*** MarkAtwood has quit IRC09:18
*** vgridnev has quit IRC09:18
*** hrubi has quit IRC09:19
*** MarkAtwood has joined #openstack-infra09:20
*** MarkAtwood has quit IRC09:21
*** jaosorior has quit IRC09:21
*** MarkAtwood has joined #openstack-infra09:22
*** skolekonov|away is now known as skolekonov09:22
*** MarkAtwood has quit IRC09:23
*** MarkAtwood has joined #openstack-infra09:24
*** YorikSar has quit IRC09:25
*** MarkAtwood has quit IRC09:26
*** MarkAtwood has joined #openstack-infra09:26
*** YorikSar has joined #openstack-infra09:26
*** amotoki has quit IRC09:27
*** MarkAtwood has quit IRC09:27
*** MarkAtwood has joined #openstack-infra09:28
*** MarkAtwood has quit IRC09:29
*** MarkAtwood has joined #openstack-infra09:30
*** MarkAtwo_ has joined #openstack-infra09:32
*** MarkAtwood has quit IRC09:32
*** MarkAtwo_ has quit IRC09:33
*** smcginnis_ has joined #openstack-infra09:34
smcginnis_Not sure if anything can be done about it, but the Cinder meeting logs got cut off and there is no corresponding html summaries:
*** tosky has joined #openstack-infra09:35
*** e0ne has quit IRC09:36
*** MarkAtwood has joined #openstack-infra09:42
*** amotoki has joined #openstack-infra09:42
*** jcoufal has quit IRC09:42
*** MarkAtwood has quit IRC09:43
*** MarkAtwood has joined #openstack-infra09:44
*** ams_ has joined #openstack-infra09:44
*** amotoki has quit IRC09:44
*** hrubi has joined #openstack-infra09:45
*** MarkAtwo_ has joined #openstack-infra09:46
*** amotoki has joined #openstack-infra09:46
*** e0ne has joined #openstack-infra09:47
*** MarkAtwo_ has quit IRC09:47
*** MarkAtwo_ has joined #openstack-infra09:48
*** MarkAtwood has quit IRC09:49
*** MarkAtwo_ has quit IRC09:49
*** MarkAtwood has joined #openstack-infra09:50
*** amotoki has quit IRC09:51
openstackgerritMaish Saidel-Keesing proposed openstack-infra/irc-meetings: Added nonATCrecogniction meeting
*** MarkAtwood has quit IRC09:51
*** MarkAtwood has joined #openstack-infra09:52
*** MarkAtwood has quit IRC09:53
*** MarkAtwood has joined #openstack-infra09:53
*** hrubi_ has joined #openstack-infra09:54
*** Douhet has quit IRC09:54
*** MarkAtwood has quit IRC09:55
*** hrubi has quit IRC09:55
*** Douhet has joined #openstack-infra09:56
*** amotoki has joined #openstack-infra09:56
*** amotoki has quit IRC09:56
*** daemontool has joined #openstack-infra09:57
*** achanda has joined #openstack-infra09:58
*** amotoki has joined #openstack-infra09:59
*** hrubi_ has quit IRC09:59
*** MarkAtwood has joined #openstack-infra09:59
*** hrubi has joined #openstack-infra10:00
*** zeih has quit IRC10:01
*** vgridnev has joined #openstack-infra10:01
*** MarkAtwood has quit IRC10:01
*** MarkAtwo_ has joined #openstack-infra10:01
*** achanda has quit IRC10:02
*** MarkAtwood has joined #openstack-infra10:03
*** MarkAtwo_ has quit IRC10:04
*** MarkAtwood has quit IRC10:05
*** smcginnis_ has quit IRC10:05
*** markvoelker has joined #openstack-infra10:06
*** e0ne has quit IRC10:08
*** ihrachys has joined #openstack-infra10:12
*** markvoelker has quit IRC10:12
*** e0ne has joined #openstack-infra10:12
*** abregman|afk is now known as abregman|now_rea10:12
*** sdague has joined #openstack-infra10:14
*** Qiming has quit IRC10:15
*** zeih has joined #openstack-infra10:15
*** amotoki has quit IRC10:16
*** cdent has quit IRC10:17
*** zeih has quit IRC10:20
*** mrmartin has joined #openstack-infra10:20
*** daemontool_ has joined #openstack-infra10:24
*** jcoufal has joined #openstack-infra10:25
*** nadya has joined #openstack-infra10:26
*** daemontool has quit IRC10:27
*** bauzas is now known as bauwser10:27
*** yamamoto has quit IRC10:30
*** ociuhandu has joined #openstack-infra10:31
*** kong has quit IRC10:36
*** alaski is now known as lascii10:38
*** dtantsur|bbl is now known as dtantsur10:40
*** cdent has joined #openstack-infra10:43
*** amotoki has joined #openstack-infra10:45
*** kzaitsev_mb has joined #openstack-infra10:46
*** e0ne has quit IRC10:47
*** mrmartin has quit IRC10:50
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints
openstackgerritAparna proposed openstack/diskimage-builder: Support to add self-signed certificate in ironic-agent
*** e0ne has joined #openstack-infra10:52
*** sbalukoff has quit IRC10:53
*** aspiers_ is now known as aspiers10:54
*** zeih has joined #openstack-infra10:58
*** trown|outtypewww is now known as trown10:59
*** markvoelker has joined #openstack-infra11:02
*** e0ne has quit IRC11:04
*** markvoelker has quit IRC11:06
*** zeih has quit IRC11:08
*** maishsk_ has joined #openstack-infra11:08
*** e0ne has joined #openstack-infra11:08
*** yamamoto has joined #openstack-infra11:09
*** yamamoto has quit IRC11:09
*** yamamoto has joined #openstack-infra11:09
*** maishsk has quit IRC11:09
*** maishsk_ is now known as maishsk11:09
*** e0ne has quit IRC11:11
*** baoli has joined #openstack-infra11:12
*** hashar has quit IRC11:13
*** baoli has quit IRC11:16
*** tpsilva has joined #openstack-infra11:17
*** hrubi_ has joined #openstack-infra11:20
*** hrubi has quit IRC11:21
*** _degorenko|afk is now known as degorenko11:24
*** hrubi_ has quit IRC11:25
*** ldnunes has joined #openstack-infra11:28
*** weshay has quit IRC11:30
*** achanda has joined #openstack-infra11:30
openstackgerritroman plevka proposed openstack/python-jenkins: Add ssl_verify option to be able to skip server SSL verification
*** e0ne has joined #openstack-infra11:33
*** amotoki has quit IRC11:34
*** thorst has joined #openstack-infra11:35
*** achanda has quit IRC11:35
*** dizquierdo has quit IRC11:36
*** dalgaaf has joined #openstack-infra11:37
daemontool_all, anyone can help with
*** daemontool_ is now known as daemontool11:37
*** abregman|now_rea is now known as abregman11:38
*** abregman is now known as abregman|afk11:41
*** baoli has joined #openstack-infra11:42
*** snikitin has left #openstack-infra11:44
*** baoli_ has joined #openstack-infra11:45
*** nadya has quit IRC11:46
*** zhurong_ has quit IRC11:46
*** arxcruz has quit IRC11:46
*** baoli has quit IRC11:48
*** e0ne has quit IRC11:50
*** Goneri has quit IRC11:50
*** kzaitsev_mb has quit IRC11:50
*** rfolco has joined #openstack-infra11:53
*** e0ne has joined #openstack-infra11:54
*** zeih has joined #openstack-infra11:55
*** kgiusti has joined #openstack-infra11:56
*** e0ne has quit IRC11:57
*** dprince has joined #openstack-infra11:57
*** arxcruz has joined #openstack-infra11:58
*** vhejral has joined #openstack-infra11:59
*** zeih has quit IRC11:59
*** abregman|afk is now known as abregman12:03
*** tiswanso has quit IRC12:05
*** mrmartin has joined #openstack-infra12:05
*** kengo_sakai has quit IRC12:06
*** sambetts has joined #openstack-infra12:06
sambettsAJaeger: QQ, Is the PROJECTS= variable in devstack-gate.yaml order agnostic? E.g. we need the inspector plugin to load after the ironic plugin loads, so do we need to make sure its before Ironic in PROJECTS?12:08
*** e0ne has joined #openstack-infra12:09
*** kzaitsev_mb has joined #openstack-infra12:09
AJaegerdaemontool: that was fixed yesterday already, we had a brief period with broken jobs out. Just add a "recheck" comment.12:10
AJaegersambetts: better ask on #openstack-qa the devstack experts or chck the source docde12:10
*** binbincong has quit IRC12:11
sambettsAJaeger: thanks will do :)12:12
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Fix branch passing into IPA post job
jrollstupid bugfix, easy review12:14
*** mrmartin has quit IRC12:16
AJaegermorning, pabelanger12:16
odyssey4meo/ how's xenial going?12:16
AJaegerjroll: which script uses BRANCH_PATH in your change?12:18
daemontoolAJaeger,  ty12:18
jrollAJaeger: failure here:
AJaegerjroll: my local copy was out of date - found it now...12:19
AJaegerjroll: thanks12:19
jrollAJaeger: np12:19
jrollAJaeger: defaults to master, so didn't see this until it was backported :(12:19
AJaegerjroll: ;) +212:19
pabelangerodyssey4me: ubuntu-xenial was uploaded to clouds last night.  Today, we do some initial testing to make sure our CI is working (eg: jenkins)12:20
*** chem has joined #openstack-infra12:21
odyssey4mepabelanger excellent :) I think what I'd like to do is to have CentOS7 & Xenial run as -nv roles on all our repositories, but only for the master branch - we'll be doing a big push for Xenial support in Newton to get done, and hope that we get some volunteers to complete the CentOS work on top of that in parallel12:24
*** gordc has joined #openstack-infra12:24
AJaegerodyssey4me: let's start with experimental patches first - those we can trigger on demand - until we're sure that xenial works fine everywhere...12:25
AJaegerodyssey4me: an experimental change I would +2 today (if pabelanger is happy with it), an -nv I would wait at least another week...12:25
*** aysyd has joined #openstack-infra12:27
pabelangerRight, so we need to land and this morning12:28
*** SumitNaiksatam has quit IRC12:28
pabelangerand for system-config12:28
odyssey4meAJaeger yup, we're not yet ready to have a go at it anyway as we're rounding out the Mitaka release anyway - but in a week or two we'll start heading in that direction12:28
odyssey4meI was just sharing the intent.12:28
*** nwkarsten has joined #openstack-infra12:29
*** EricGonczer_ has joined #openstack-infra12:30
*** mrmartin has joined #openstack-infra12:30
*** nadya has joined #openstack-infra12:30
*** EricGonc_ has joined #openstack-infra12:31
*** erlon has joined #openstack-infra12:32
*** achanda has joined #openstack-infra12:32
*** |-paul-| has quit IRC12:34
*** EricGonczer_ has quit IRC12:35
AJaegerpabelanger: gave already +4 ;) let's see who can help landing them later...12:35
*** achanda has quit IRC12:37
*** flepied has quit IRC12:39
*** flepied has joined #openstack-infra12:39
*** niska has quit IRC12:40
*** pradk has joined #openstack-infra12:40
*** rlandy has joined #openstack-infra12:40
*** cbader has quit IRC12:41
*** chem has quit IRC12:41
*** devananda has quit IRC12:43
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add scripts to generate website
*** keedya has joined #openstack-infra12:43
*** devananda has joined #openstack-infra12:44
*** flepied has quit IRC12:44
*** annegentle has joined #openstack-infra12:44
*** niska has joined #openstack-infra12:44
*** Qiming has joined #openstack-infra12:46
ZaraHi, so I'm looking at setting up an account in gerrit to read the events stream, to talk to storyboard. I gather I should be using the hudson lp account for this. my question is, how do I do things on the gerrit side? I don't know how to log into gerrit with this account, and I'm not sure I'm the person who should be doing that.12:48
*** vaidy has quit IRC12:49
*** dims_ has quit IRC12:49
*** dims has joined #openstack-infra12:49
Zara(I'm looking at the docs here, for reference, though this isn't a CI system, and should only be *reading* the stream, not posting to gerrit or anything like that. )12:50
*** cbader has joined #openstack-infra12:50
*** hashar has joined #openstack-infra12:52
*** edmondsw has joined #openstack-infra12:53
openstackgerritJulien Danjou proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config
*** vhejral has quit IRC12:57
*** e0ne has quit IRC12:57
openstackgerritVictor Stinner proposed openstack-infra/project-config: Add non-voting gate-trove-python34 check
openstackgerritMerged openstack-infra/storyboard-webclient: Added comments to dashboard.html
openstackgerritsebastian marcet proposed openstack-infra/puppet-redis: improved pesistence model
fungifrickler: is an image one of our root admins made available for someone to download for debugging purposes and (probably) hasn't gotten around to deleting yet12:59
*** akshai has joined #openstack-infra13:00
*** yamamoto has quit IRC13:00
*** mtanino has quit IRC13:01
*** akshai_ has joined #openstack-infra13:02
*** smarcet has joined #openstack-infra13:02
pabelangerAh, I noticed that the other day but forgot to ask the purpose of it13:03
*** eharney has joined #openstack-infra13:03
*** esikachev has quit IRC13:03
pabelangerfungi: to early for a +A on 300257 and 300259 ? puppet-apply and bindep experimental jobs for ubuntu-xenial13:03
*** vaidy has joined #openstack-infra13:04
*** _ari_ has joined #openstack-infra13:05
*** flepied has joined #openstack-infra13:05
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add missing release jobs
*** akshai has quit IRC13:05
*** amotoki has joined #openstack-infra13:06
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on attendees schedule
openstackgerritMerged openstack-infra/openstackid-resources: Fix on attendees schedule
*** tiswanso has joined #openstack-infra13:08
fungiAJaeger: therve: yeah, looks like maybe bumping up against a network quota in devstack? you might have more luck asking in #openstack-qa13:08
AJaegerfungi, pabelanger (from backscroll), tripleo-quickstart merged yesterday but setup looks broken on,access . I don't see the import from the upstream repo run either. Details at
openstackgerritVictor Stinner proposed openstack-infra/project-config: Add non-voting gate-trove-python34 check
fungiAJaeger: yep, saw. i'm digging through the change now looking to see if i can spot a problem13:10
AJaegerthanks, fungi !13:10
pabelangerAJaeger: I am not sure how to debug that. But happy to learn if fungi or other root steps me thru13:11
*** aysyd has quit IRC13:11
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add missing release jobs
fungipabelanger: mostly it'll be something going wrong when ansible/puppet runs manage-projects on review.o.o13:12
*** links has quit IRC13:12
fungipabelanger: most common cause is a missing or syntactically incorrect acl, or an invalid "upstream" clone url, but i'm not immediately seeing that as the case here so going to check syslog13:13
AJaegerfungi: invalid upstream urls should not happen anymore - part of our tests involve cloning the repo13:13
fungiand then possibly try rerunning manage-projects myself with the debug flag if i still don't see what's up13:13
AJaeger(and checking that it has a master branch ;)13:14
fungiAJaeger: yep, or at least there would be a very tight window where they'd have to become invalid between gating and puppeting13:14
*** Goneri has joined #openstack-infra13:14
AJaegerfungi: yeah13:14
*** e0ne has joined #openstack-infra13:15
openstackgerritNiall Bunting proposed openstack/requirements: Don't use 1.3.0 version of cryptography
*** aysyd has joined #openstack-infra13:15
*** vgridnev has quit IRC13:15
fungianother possibility is that manage-projects ran too long for some reason and hit a timeout (average runtimes for it appear to be around 20-22 minutes lately and i think we have puppet set to time that exec out at 30?)13:17
*** weshay has joined #openstack-infra13:17
fungithe change merged yesterday at 18:48 utc and there was a manage_projects exec logged by puppet completing at 19:23 utc after running about 20 minutes. doesn't claim a nonzero return code13:19
pabelangerexec default timeout is 300 seconds: checking if we set it to something else13:19
*** julim_ has quit IRC13:20
*** binbincong has joined #openstack-infra13:20
*** vgridnev has joined #openstack-infra13:21
*** julim has joined #openstack-infra13:22
jrollfungi: easy review I'm blocked on, if you get a moment this morning:
fungiaha, i keep forgetting we started echoing m-p output to a separate log13:22
openstackgerritMerged openstack-infra/project-config: Add designate stable/mitaka to gerritbot
fungiException: Gerrit error executing gerrit replicate openstack/tripleo-quickstart13:22
*** matt-borland has joined #openstack-infra13:23
fungilooks like it may have been a transient error, but because it hit that, it short-circuited the rest of the setup for it13:23
fungii bet if i rerun it, things will work fine13:23
*** salv-orlando has joined #openstack-infra13:23
*** esikachev has joined #openstack-infra13:24
*** dansmith is now known as bizarroman13:25
fungiAJaeger: pabelanger: after rerunning `sudo manage-projects openstack/tripleo-quickstart` the acl seems correctly applied,access13:25
*** bizarroman is now known as bizarrodan13:25
fungifor future reference, when puppet runs manage-projects it gets logged to /var/log/manage_projects.log13:26
*** salv-orl_ has quit IRC13:26
*** bauwser is now known as lawn_bauwer13:27
pabelangerfungi: great13:27
*** dims_ has joined #openstack-infra13:27
*** yamamoto has joined #openstack-infra13:28
AJaegerfungi, and even github is setup at
AJaegerthanks, fungi!13:28
*** mrmartin has quit IRC13:28
*** e0ne has quit IRC13:29
AJaegerso, we can approve - the daily gerrit/projects normalize bot ;)13:29
AJaegerThat's how I noticed...13:29
jrollthanks for the +W fungi :)13:29
*** asselin__ has joined #openstack-infra13:32
*** xyang1 has joined #openstack-infra13:32
*** jistr has quit IRC13:32
fungiclarkb: it looks like you created /var/log/nodepool/image/176634.qcow2 on nodepool back on march 18 for mnaser to inspect. is it safe for me to delete now?13:33
mnaser^ i can say yes13:33
*** openstackgerrit has quit IRC13:33
*** openstackgerrit has joined #openstack-infra13:33
openstackgerritSam Betts proposed openstack-infra/project-config: Add post job for building and uploading tinyipa
fungimnaser: thanks, i was pretty sure i recalled you saying he could delete it, so he likely just got pulled away by some other emergency at that point and forgot. i've removed it now13:34
*** achanda has joined #openstack-infra13:34
openstackgerritMerged openstack-infra/project-config: Fix branch passing into IPA post job
fungijroll: ^ will probably be live in the next 15-30 minutes13:35
*** krtaylor has quit IRC13:35
jrollfungi: yep, I'll keep an eye out13:35
*** chem has joined #openstack-infra13:35
*** salv-orlando has quit IRC13:35
jrollfungi: oh, the update isn't through a post job, is it13:36
*** asselin__ has quit IRC13:36
fungijroll: no, ansible runs from cron every 15 minutes and connects to our servers and updates configuration on them13:37
*** aysyd has quit IRC13:37
fungibut sometimes it takes in excess of 15 minutes to run13:37
jrollright on, thanks13:37
*** achanda has quit IRC13:39
*** zz_dimtruck is now known as dimtruck13:39
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** jsavak has joined #openstack-infra13:39
*** abregman is now known as abregman|afk13:41
*** electrofelix has joined #openstack-infra13:41
*** aysyd has joined #openstack-infra13:41
*** thiagop-afk has joined #openstack-infra13:43
*** thiagop-afk is now known as thiagop13:43
*** annegentle has quit IRC13:46
*** markvoelker has joined #openstack-infra13:47
*** mriedem has joined #openstack-infra13:47
*** Liuqing has joined #openstack-infra13:48
*** e0ne has joined #openstack-infra13:51
*** markvoelker has quit IRC13:51
*** mtanino has joined #openstack-infra13:51
*** lascii is now known as lutf-813:52
*** binbincong has quit IRC13:58
*** aysyd has quit IRC13:58
*** zeih has joined #openstack-infra13:58
dmsimardthe devstack-logs builder is the one we currently use to upload logs for the packstack jobs
dmsimardI don't see anything specific in it that would explain the devstack "explanations and troubleshooting info" that appears when browsing logs13:58
*** Hal has joined #openstack-infra13:58
dmsimardIs there a way to opt-out of that ? devstack-specific info is not super relevant to Packstack jobs as you can imagine.13:58
*** Hal is now known as Guest5425113:58
*** openstack has quit IRC13:58
*** openstack has joined #openstack-infra13:59
*** daemontool has quit IRC13:59
*** zhurong has joined #openstack-infra13:59
*** daemontool_ is now known as daemontool14:01
*** arxcruz has quit IRC14:04
pabelangerinfra-root and project-config-core: if you mind taking a look at: I'd like to do some smoke testing of ubuntu-xenial today if at all possible14:04
openstackgerritVictor Stinner proposed openstack-infra/project-config: Add non-voting gate-trove-python34-db check
*** erlon has quit IRC14:04
*** rbrndt has joined #openstack-infra14:05
*** yamahata has joined #openstack-infra14:05
*** erlon has joined #openstack-infra14:05
*** yaume_ has joined #openstack-infra14:05
*** sdake has joined #openstack-infra14:06
*** e0ne has quit IRC14:07
AJaegerfungi, git  timed just out on rax-dfw here: - is that something you want to know about?14:08
*** sigmavirus24_awa is now known as sigmavirus2414:08
*** annegentle has joined #openstack-infra14:08
*** yaume has quit IRC14:08
*** jsavak has quit IRC14:09
*** jsavak has joined #openstack-infra14:10
mordredpabelanger: will mean that we'll be voting on xenial for infra puppet patches14:10
mordredpabelanger: I think given the context I'm ok with that - but want fungi to weigh in on it14:10
*** tonytan4ever has joined #openstack-infra14:10
pabelangermordred: it should be fine, because of
pabelangerwhich uses the experimental pipeline14:11
mordredI mean, I'm +2 on it - just wanted to make sure fungi expressed an opionion before it +Ad14:13
*** nwkarsten has quit IRC14:13
Zarapabelanger: btw, to close stories in storyboard, easiest way is to mark all tasks as invalid or merged as appropriate; that should automatically move the story from the 'active' list. (I've marked tasks invalid on a couple of old stories you said should be closed)14:13
*** nwkarsten has joined #openstack-infra14:14
pabelangerZara: Ah, cool. Do you mind doing that to one of my stories?  So I can see what an email notification looks like?14:14
*** weshay has quit IRC14:15
*** dimtruck is now known as zz_dimtruck14:15
*** burgerk has joined #openstack-infra14:18
*** yamamoto has quit IRC14:18
*** aysyd has quit IRC14:18
openstackgerritMerged openstack-infra/project-config: Add experimental ubuntu-xenial puppet-apply job
*** nwkarsten has quit IRC14:18
*** amitgandhinz has joined #openstack-infra14:18
*** tiswanso has quit IRC14:18
*** yamamoto has joined #openstack-infra14:20
*** tiswanso has joined #openstack-infra14:20
*** aysyd has joined #openstack-infra14:20
openstackgerritMerged openstack-infra/project-config: Add experimental bindep-fallback job for ubuntu-xenial
*** pradk has quit IRC14:21
*** pradk has joined #openstack-infra14:21
*** cdent has quit IRC14:22
*** zeih has quit IRC14:22
KiallQ - Anyone know of existing projects using the tempest plugin interface?14:23
*** ajmiller has joined #openstack-infra14:24
*** nwkarsten has joined #openstack-infra14:24
fungiAJaeger: it's something we know about, but not something we're able to troubleshoot until we decide where we want to (carefully!) wedge traceroute into our setup14:26
fungiso that we can get better diagnostic data back to rackspace support14:26
Zarapabelanger: I have done, though bear in mind it doesn't subscribe you to a story you create automatically (it probably *should*...), so you'll only get a notification if you manually subscribe. Also that will use an email template for tasks, and those are the ugliest (also on my todo list, yay).14:27
sambettsHi AJaeger, about your comment on the ironic python agent tinyipa job patch, I'm not sure I understand what you mean by use the same setup?14:27
Zarabtw, someone's trying to install storyboard and they're getting an error installing mysql (v5.6, if helpful). this is via an ansible role that we know works. wondering if anyone else has seen any mysql troubles of late?14:28
pabelangeranybody know what is up with jenkins05.o.o? It appears to be stopped14:28
pabelangerchecking logs now14:29
pabelanger# There is insufficient memory for the Java Runtime Environment to continue.14:30
pabelangerguess jenkins05 crashed14:30
*** knikolla has joined #openstack-infra14:31
*** dtardivel has joined #openstack-infra14:32
pabelangerfungi: have a min to walk me through jenkins05?14:33
jeblair_fungi: regarding vexxhost slowness -- i think getting uuids might be the first thing -- it feels like it could even be some misconfigured nova compute host or something14:33
*** jeblair_ is now known as jeblair14:33
fungiZara: meant to answer your earlier questions in scrollback here... the hudson-openstack account in launchpad is what we use to update launchpad bugs/blueprints _from_ gerrit, but i think yesterday's discussion may have gone sideways a bit because i'm not sure how that will be relevant to storyboard14:33
fungiZara: other than we need some account in gerrit to listen on the event stream, and could possibly use that launchpad account to do openid authentication for a gerrit account14:35
fungiassuming event stream listener is the model you're wanting to go with (rather than how we do our current launchpad integration, which is triggered by gerrit hook scripts instead)14:36
openstackgerritPaul Belanger proposed openstack-infra/project-config: [WIP] Example bindep commit
fungii recall the event stream solution was preferred when we were discussing models for this long ago, we're simply using gerrit hook scripts for lp updating because we needed some way to call arbitrary scripts to interface with their already defined api14:37
Zarafungi: That sound like what I was asking after. I've been advised to try an event stream listener, and don't know why it's preferred over hook scripts, but there seems to be a general view that it's easier and I'm inclined to trust that. (this is my first foray into either approach) :D14:38
fungiZara: i don't know that it's _necessarily_ easier, but it does seem like it will be more robust14:39
fungibasically you can look at it as active vs passive models14:39
fungiif storyboard listens on to and triggers actions from the gerrit event stream (like zuul does), then it has more control over what happens and gets to encode more of the logic internally14:40
*** zz_dimtruck is now known as dimtruck14:40
openstackgerritPaul Belanger proposed openstack-infra/project-config: Bring a single ubuntu-xenial node online
Zarafungi: aiui, we want an event stream listener to send information to a script that imports the python-storyboardclient, and that then talks to a specified storyboard instance to update it. so we want a gerrit account for that listener.14:41
pabelangerAJaeger: ^ we're ready to launch a single ubuntu-xenial node14:41
fungiif storyboard presents an action-oriented api and gerrit calls that through triggered hooks, then a lot of the logic ends up bolted on from outside and storyboard ends up with less control over the sequence of actions taken14:41
openstackgerritMerged openstack-infra/system-config: Add ubuntu-xenial to openstack_project::single_use_slave
*** markvoelker has joined #openstack-infra14:41
*** zeih has joined #openstack-infra14:42
*** abregman|afk is now known as abregman14:43
Zarafungi: ah, I'm not actually sure how much control it needs. I suspect the conversations happened before I joined the project. right now we just want to update task statuses from gerrit, as a start... I don't think it has the potential to cause as much chaos as zuul if misconfigured14:43
fungiZara: yep, so anyway there's no specific need for launchpad involvement in the desired case, you just need a gerrit account. you can use basically any gerrit account for testing this (you're just planning to use it to listen to the event stream, not actually leave comments/votes on changes in gerrit), though once we put something into production for, say, the infra team would14:44
fungicreate a fresh service account for that or reuse one it already has for some similar purpose14:44
Zaraahh, okay14:44
*** sigmavirus24 is now known as sigmavirus24_awa14:44
*** sigmavirus24_awa is now known as sigmavirus2414:45
Zaraheh, I'm still fuzzy on how this will be generalised for production, but I think that will become clearer as I work on it.14:45
funginow that said, if you're going to create an account in gerrit to use for this (for testing now, or for use by another non-infra-controlled storyboard instance) you'll need to have _some_ lp account you used to authenticate to gerrit so that you can set your username, ad ssh keys and so on14:45
Zarayeah, I thought that was why hudson was suggested? or was that jumping ahead to the production instance?14:45
*** jistr has joined #openstack-infra14:46
fungithat could just be the personal gerrit account you already had, or you could add a second lp account solely for that purpose14:46
fungifor production use on we'd likely create or reuse an existing service account in gerrit, but we don't generally tie those to launchpad accounts14:46
*** zhurong has quit IRC14:46
*** markvoelker has quit IRC14:47
fungisince we can configure/update them via the gerrit api as admins14:47
*** zeih has quit IRC14:47
*** Liuqing has quit IRC14:47
AJaegersambetts: what I mean is: Both jobs are on the same repo, each is invoked differently and files are placed in different locations. Why can't you put all files in an upload folder - and use the same invocation with just one diffrent parameter?14:47
*** Liuqing has joined #openstack-infra14:48
openstackgerritGraham Hayes proposed openstack-infra/project-config: Enable DB access for Designate unittests
AJaegerpabelanger: you don't need this. min-ready: 0 will allow you starting one.14:49
Zarafungi: okay, so right now it sounds like the simplest thing to do is to just use my personal account to read the events stream, and work on getting the python client to send info from there to my test storyboard instance?14:49
AJaegermin-ready: 1 means: Have always a node ready to launch in the queue.14:49
fungiZara: yep, you don't need anything special to be able to do that. everyone's account has access to listen to the ssh stream-events14:50
*** cdent has joined #openstack-infra14:50
pabelangerAJaeger: but, isn't the issue we haven't registered a ubuntu-xenial job to gearman yet?  And we'll always return NOT_REGISTERED for jobs that need ubuntu-xenial?14:50
Zarafungi: okay, brilliant. I wanted to make sure I'm not breaking any rules, but that sounds fine to me.14:50
fungijeblair: was that an accidental repeat from your input history? or are you still seeing slow behavior in vexxhost? i've been getting uuids to mnaser and he's been running down possible causes for the past couple days14:51
Zarahopefully I'll be able to come back with something more interesting in a little whle14:51
AJaegerpabelanger: let's ask fungi and jeblair about whether "min-ready:0" allows to start jobs - AFAIK it does...14:51
fungipabelanger: sure, i can help you through jenkins0514:51
pabelangerfungi: neat14:51
AJaegerpabelanger: it's just slower since none is busy waiting for us. That's my udnerstanding...14:51
fungipabelanger: i take it you were looking at the jenkins service log and it ran out of memory inside the jvm?14:51
pabelangerfungi: right, I am looking at /tmp/hs_err_pid8695.log on jenkins0514:52
pabelangerwe ran out of ram14:52
fungiAJaeger: pabelanger: yes, min-ready: 0 just means don't build any new nodes unless you see demand for them. it doesn't prevent running anything (nodepool doesn't decide who can run what jobs, it just makes servers)14:52
pabelangerand this server doesn't have a swap14:52
sdaguefungi: ... random friday topic14:53
*** tmorin has joined #openstack-infra14:53
AJaegerpabelanger: so, run "check experimental" ;)14:53
sdaguehas anyone considered mirroring the gerrit event stream to something like mqtt14:53
sdagueso that there could be anonymous read on it?14:53
jeblairfungi: no, that was a response to skimming history, thanks for the update14:53
pabelangerAJaeger: have see 30052614:53
AJaegerpabelanger: with min-ready: -1 nothing would run...14:53
*** mylu has joined #openstack-infra14:53
*** woodster_ has joined #openstack-infra14:54
AJaegerpabelanger: that could also mean that no jenkins had the job so far ;(14:54
fungipabelanger: so two things to do before starting jenkins: 1. make sure all nodes for it are deleted in nodepool, 2. make sure any lingering node definitions are wiped via `sudo rm -rf /var/lib/jenkins/nodes/*`14:54
pabelangerAJaeger: I always though you need a node to connect to jenkins to register the new job names into gearman14:54
pabelangerfungi: Yup I was going to do that14:55
Zarasdague: sounds cool to me; anteaya mentioned there were drawbacks to keeping it visible to everyone, so she might have something interesting to add about it14:55
sdaguebeen playing with mosquitto for homeautomation stuff, and an mqtt with websockets built in might expand the uses for the data stream14:55
sdaguewithout adding load to gerrit14:55
fungisdague: we did ponder some possible ways to proxy gerrit's event stream. the design it has with dedicated worker threads handling stream events for ssh sockets has some inherent scalability issues14:56
fungii don't think we got very far into designing what that would look like though14:56
*** e0ne has joined #openstack-infra14:57
pabelangerAJaeger: 300526 just failed with NOT_REGISTERED14:57
sdagueyeh, understood. I've been reasonably impressed with mosquitto at my scall, and the fact that you can direct websocket connect from js is neat -
pabelangerpretty sure we need to launch a ubuntu-xenial image first otherwise gearman won't register the new job14:57
jeblairfungi: it's a pool of threads, not dedicated, but yes, it does take resources14:57
openstackgerritGraham Hayes proposed openstack-infra/project-config: Enable DB access for Designate unittests
fungijeblair: right, the pool has a limited size was what i meant14:57
sdagueI don't know how it would handle the load of openstack. But the conceptual model is pretty nice.14:57
AJaegerpabelanger: I thought not - but let's see what others say. Happy to learn something new...14:57
sdaguelike I said, random friday thought14:58
fungisdague: i think _ideally_ gerrit would provide some modern stream api which didn't rely on ssh and could be handled more robustly through a web service14:58
sdaguefungi: sure14:58
fungiwhich would remove the last real thing you can only do via ssh14:58
sdagueI would agree14:58
sdaguebut... barring that14:58
fungii wonder if a gerrit plugin could be used to do that14:59
*** nwkarsten has quit IRC14:59
sdaguefungi: possibly, though you'd still be putting some more load on gerrit to do that14:59
sdagueif the events mirror off15:00
sdaguethen the load shifts to another thing15:00
sdaguewhich, with my understanding of gerrit's load, would probably be good15:00
pabelangerfungi: what about the nodes that are attached to jenkins, but in used state. They are safe to stay or have to be deleted?15:00
*** nwkarsten has joined #openstack-infra15:00
AJaegerpabelanger: Have to leave now, will read backscroll later and I'm eager to learn the solution.15:00
pabelangerAJaeger: ack15:00
*** kzaitsev_mb has quit IRC15:00
fungiyeah, just wondering if the gerrit plugin could produce something which apache/whatever could serve to multiple consumers without having to hook in via ssh15:00
*** thorst has quit IRC15:01
*** kzaitsev_mb has joined #openstack-infra15:01
fungipabelanger: definitely delete used nodes from jenkins, those are the most dangerous15:01
pabelangerfungi: okay15:01
*** mrmartin has joined #openstack-infra15:02
fungijenkins will automatically put them back online to run new jobs if it can15:02
*** maishsk has quit IRC15:02
*** jsavak has quit IRC15:02
*** thorst has joined #openstack-infra15:02
*** jsavak has joined #openstack-infra15:03
*** electrofelix has quit IRC15:03
sdaguefungi: yeh, possibly. I know how you could build it with a python glue script in about a day.15:03
sdaguebut how to do it as a gerrit plugin... no real idea15:03
fungisdague: right, that's what we were leaning toward doing, or just using some existing multiplexer even15:04
fungijust thinking more what an upstream wishlist item might be for this so we didn't end up maintaining it indefinitely15:04
*** maishsk has joined #openstack-infra15:05
*** nwkarsten has quit IRC15:05
*** markvoelker has joined #openstack-infra15:05
*** esikache1 has joined #openstack-infra15:05
*** imcsk8 is now known as imcsk8|plane15:05
*** esikachev has quit IRC15:05
pabelangerfungi: thanks15:06
pabelangerjenkins05 back online15:06
pabelangerI'll run JJb shortly15:06
*** Liuqing has quit IRC15:07
pabelangerSo, back to gearman. Using the command clarkb showed me the other day, echo "status" | nc localhost 4730 | grep ubuntu-xenial , I don't see any xenial jobs registered with gearman15:07
pabelangerwhich, and this is where I could be wrong, jobs only register with gearman when a new jenkins slave comes online15:07
*** links has joined #openstack-infra15:07
pabelangerand because ubuntu-xenial is min-ready: 0, it will never launch15:07
daemontoolat your convinience, please review +A only is needed15:08
*** Liuqing has joined #openstack-infra15:09
jeblairpabelanger: that is correct; we put servers at min-ready:1 when we're using them experimentally.15:09
pabelangergreat, so 300530 is the patch then15:09
pabelangerjeblair: thanks15:10
*** yamamoto has quit IRC15:11
*** vgridnev has quit IRC15:11
*** Liuqing has quit IRC15:12
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add Ubuntu binary build and deploy gate for kolla
fungiyep, there's a chicken-and-egg issue with the current design, where nodepool won't build any nodes at min-ready: 0 unless it sees demand (jobs waiting to run on them), but if no node gets built jenkins won't register the jobs for it into gearman, and zuul will instantly fail them as NOT_REGISTERED resulting in no actual demand, so nodepool continues to not build any15:13
jeblairwell, the current design does not involve min-ready:0 -- the design is to always try to have nodes ready15:13
jeblair(the 'pool' in nodepool)15:14
*** abregman is now known as abregman|nb15:14
fungitrue. by "current design" i meant as far as zuul's behavior of reporting NOT_REGISTERED rather than waiting (possibly indefinitely) for the job to be registered15:15
pabelangerand JJB update to date on jenkins05 now15:16
Swansonso when you get cannot merge listed on gerrit how do you resolve that? gill pull --rebase and git review again?  (Trying to avoid screwing this up like I did last time I did this... Would love to find that post-it note..)15:16
*** [1]Thelo has joined #openstack-infra15:18
*** nwkarsten has joined #openstack-infra15:18
fungiSwanson: i generally: git review -d <change_num>; git remote update; git rebase -i origin/master; [resolve merge conflicts and continue]; git review15:20
*** Thelo has quit IRC15:20
*** [1]Thelo is now known as Thelo15:20
Swansonfungi, That sounds really familiar. Thanks for the help!15:20
fungiwe used to have a section on that in but i think it got taken out because people interpreted it to mean that you should rebase every time you update a change, which is really caustic for your reviewers15:20
*** maishsk has quit IRC15:21
*** regXboi has joined #openstack-infra15:21
fungibest to avoid rebasing the change unless there's actually a merge conflict which needs resolving15:21
fungior to get it to pick up a new parent change15:21
fungi(in the event that you're stacking multiple changes for review)15:21
SwansonYep.  I think I just made the same changes in two reviews and didn't stack them.15:22
*** thorst has quit IRC15:22
SwansonThanks again!15:23
*** yamamoto has joined #openstack-infra15:23
*** mrmartin has quit IRC15:24
*** mylu has quit IRC15:24
*** mylu has joined #openstack-infra15:28
*** jsavak has quit IRC15:29
mnaseroh boy15:30
openstackgerritMerged openstack-infra/project-config: Bring a single ubuntu-xenial node online
*** thorst has joined #openstack-infra15:30
*** ajmiller has quit IRC15:30
mnaserall instances had wide open quotas in their respective cgroups15:30
mnasereven though libvirt is configured to set it (including the xml)15:30
mnaserthis explains a bunch of things.15:30
*** e0ne has quit IRC15:31
*** matrohon has quit IRC15:31
*** ajmiller has joined #openstack-infra15:32
*** jsavak has joined #openstack-infra15:33
matt-borlandHello, in the Horizon jobs we've been getting an increased number of errors with our npm-run-* jobs.  We get 403 errors on getting specific resources...e.g.
matt-borlandIt's possible that this is a StrictSSL problem, or could be a proxy issue on the node.15:34
matt-borlandIf anyone has perspective on those kinds of would be much appreciated. :)15:35
*** bpokorny has joined #openstack-infra15:36
clarkbthe test machines shouldnt be behind any proxy15:36
*** markmcd1 has joined #openstack-infra15:36
matt-borlandclarkb, thanks15:36
*** achanda has joined #openstack-infra15:37
*** cdent has quit IRC15:37
clarkblooks like bitbucket is saying go away15:37
clarkbits funny hiw pypi had this same ptoblem forever15:38
*** mylu has quit IRC15:38
clarkband now npm is doomed to repeat its mistake15:38
matt-borland"funny" is a funny word :)15:38
jeblairi believe there's ongoing work to make a local npm mirror15:39
matt-borlandawesome jeblair15:39
*** akaszuba has quit IRC15:39
clarkbthat will probably just mirror the redirect though15:39
*** ociuhandu has quit IRC15:39
clarkbwhich us ehat all the pypi mirror software did and forced us to wrote our own15:40
*** mylu has joined #openstack-infra15:40
jeblairclarkb: oh, you're saying this is a 'package removed' situation?15:40
matt-borlandthe only thing is it is intermittent, so it doesn't always reject.15:40
jeblairi was assuming it was a 'transient hosting failure' situation15:40
matt-borlandyeah, I am guessing so jeblair15:40
clarkbjeblair: its an exterbal package link situation and the package host (bitbucket) is 403ing15:40
*** armax has joined #openstack-infra15:41
jeblairclarkb: ahh, that.  and the original pypi-mirror software.  gotcha.15:41
clarkbso mirroring npm doesnt fix bitbucket15:41
*** mikelk has quit IRC15:41
jeblairclarkb: i assumed our npm mirror would hold content15:41
*** achanda has quit IRC15:42
*** jlanoux has quit IRC15:42
*** fawadkhaliq has joined #openstack-infra15:42
*** LaT0rtue has joined #openstack-infra15:42
clarkbI think it will for the things npm actually hosts15:42
clarkbbut would expect similar behavior to pypi for external link things15:42
*** jlanoux has joined #openstack-infra15:43
jeblair is the most recent version of the infra change to add the npm mirror15:43
jeblairit says it uses
clarkbbecause the view the package repos have taken is you are mirroring what they have which is an external link15:44
*** krtaylor has joined #openstack-infra15:44
matt-borlandyep, interesting15:44
jeblairclarkb, fungi, mordred: what's the current nodepool-shade status?15:44
clarkbit is worth confirming15:44
*** Guest90877 has quit IRC15:44
fungijeblair: there were shade releases yesterday15:45
clarkbjeblair: I reviewed a bunch if stuff that I think needs merge and release. I also have not seen or heard what the tripleo work around is15:45
fungino nodepool restarts yet afaik15:45
*** jsavak has left #openstack-infra15:45
jeblairclarkb: workaround is
fungioh, i guess there was _a_ shade release yesterday, then Shrews pointed out that there was still an outstanding patch which didn't get approved which will need another shade release15:46
*** ajmiller has quit IRC15:46
*** yamamoto has quit IRC15:46
fungibut not sure if that remaining patch was necessary for our use with nodepool15:46
*** roxanaghe has joined #openstack-infra15:46
jeblairfungi, Shrews: was it this one?
Shrewsfungi: i think it was15:47
Shrewsi just rechecked it15:47
Shrewsjeblair: yes15:47
clarkbthats a fun workaround15:47
*** ajmiller has joined #openstack-infra15:47
jeblairit seems to be failing a lot -- are we sure there isn't something to those failures?15:47
*** yaume_ has quit IRC15:47
*** tmorin has left #openstack-infra15:48
fungijeblair: those last failures were my fauly15:48
jeblair"faulty fungi"15:49
fungiwe merged my change to start collecting traceroutes (because rackspace says it needs them to have any hope of diagnosing network issues between ord and dfw when we start using ord again) but i didn't check that traceroute/traceroute6 are actually installed on our new images15:49
jeblairthis is what ubuntu should do after "Z"15:49
timrcSounds like a potential Ubuntu release, jeblair15:49
timrcjeblair: lol15:49
fungiso... many jobs using the net-info job macro began insta-failing15:50
*** tonytan4ever has quit IRC15:50
Shrewsfaulty fungi, jumpy jeblair, moldy mordred15:50
Shrewsjeblair: good idea15:50
jeblairsilly shrews!15:50
fungithe one we really need is cloned clarkb15:50
*** tonytan4ever has joined #openstack-infra15:51
pabelangerfirst issue15:51
pabelangerlack of ubuntu xenial mirror15:51
pabelangerBut, Success that jenkins worked15:51
jeblairShrews, clarkb, fungi: so i think we can restart nodepool now (that hasn't happened, right)?  the change in 299672 will reduce thundering herds, but i think we can suffer through that now to see how things are otherwise.15:52
*** weshay has joined #openstack-infra15:52
*** yolanda has joined #openstack-infra15:53
*** tonytan4ever has quit IRC15:53
pabelangerokay, afk for 25mins15:53
*** tonytan4ever has joined #openstack-infra15:54
openstackgerritMerged openstack-infra/nodepool: Set cache parameters like we do in production
*** links has quit IRC15:55
jeblairclarkb: also that change worked ^ which gives me some confidence in the port caching patch i wrote wednesday15:55
timrcjeblair says "restart nodepool" and pabelanger goes afk.  Smart :P15:55
*** IlyaG has joined #openstack-infra15:56
timrcOh look at the time! I need to go stir my clarkb clone vat.15:56
fungiyeah, it seems like we should be safe for a restart15:56
clarkbjeblair: cool I suppose we could restart and just watch for stampedes15:56
*** ajmiller has quit IRC15:56
clarkbtimrc: those are unlicensed replicas, they have been sold without covers15:56
jeblairyeah, we saw that last time, but it settles out15:57
clarkbjeblair: this is the startup stampede with exclusive blocking locks?15:57
*** ajmiller has joined #openstack-infra15:58
jeblairclarkb: yep16:00
jgriffithanybody know of a reason os-client-config may not be reading my .config/openstack file?16:02
jgriffitherrr... ok, that's a pretty dumb question to ask I guess16:02
mtreinishjgriffith: well you work for a storage company, so hardware corruption?16:02
jgriffithmtreinish: touchet!16:02
clarkbpermissions? also I think openstack is supposed to be a dir then you put clouds.yaml in it16:03
*** Swami has joined #openstack-infra16:03
jgriffithI'm just trying this:
jgriffithrunning as root16:03
jgriffithoh.. what?  clouds.yaml16:03
jgriffithson of a16:03
*** amotoki has quit IRC16:04
* jgriffith is a moron16:04
jgriffithsorry.. and thanks clarkb16:04
jgriffithI just created ~/.config/openstack  NOT ~/.config/openstack/clouds.yaml16:04
jgriffiththat fixed it.. thank you very much16:05
*** hashar has quit IRC16:05
jgriffithmtreinish: not the hardware this time :)16:06
*** apoorvad has joined #openstack-infra16:06
*** trown is now known as trown|lunch16:06
clarkbjgriffith: np16:06
*** austin81 has joined #openstack-infra16:06
mtreinishjgriffith: hah, fair enough :)16:07
*** fhubik has quit IRC16:08
*** roxanaghe has quit IRC16:08
*** ihrachys has quit IRC16:09
fungijgriffith: could have been subtle corruption affecting the filesystem table, which had the effect of renaming ~/.config/openstack/clouds.yaml to ~/.config/openstack (then you can be blameless!)16:11
jgriffithfungi: I like the way you think!!  In fact.. I"m certain that's EXACTLY what happened!16:12
*** esikache1 has quit IRC16:12
*** mriedem is now known as mriedem_lunch16:12
jgriffithfungi: it happens just like that ya know!16:12
*** esikachev has joined #openstack-infra16:12
*** zzzeek has quit IRC16:13
* fungi is a terrible craftsman, always blames his tools16:13
jeblairclarkb, fungi: we should land before restarting nodepool16:13
jgriffithfungi: that saying is bunk!16:13
clarkbjeblair: looking16:13
clarkboh yes I meant to approve that one as part of my reviewing yesterday16:14
*** zzzeek has joined #openstack-infra16:14
openstackgerritGraham Hayes proposed openstack-infra/project-config: Move tox-bandit to check as non voting
clarkbI forgot to do it after I reviewed the change that consumed that16:14
*** sfinucan has quit IRC16:15
*** mylu has quit IRC16:15
*** jcoufal has quit IRC16:15
clarkbthe xenial host that nodepool booted is in rax and both network interfaces look happy16:15
clarkband journalctl works16:16
clarkbsuch excite, this is really going to be the killer feature for dib long term. No longer dependent on providers publishing images16:16
*** nadya has quit IRC16:17
clarkbwant to test on plan9? we can do that16:17
*** jlanoux has quit IRC16:17
pabelangerback, catching up on back scroll16:18
pabelangerclarkb: Yup, worked first time.  Exciting16:19
*** yolanda has quit IRC16:19
clarkb/dev/mapper/main-opt  504G  468G   37G  93% /opt removing devstack-centos7 will help that16:20
pleia2good morning16:20
openstackgerritMerged openstack-infra/system-config: Add server and port caching settings for nodepool
*** jlanoux has joined #openstack-infra16:20
*** roxanaghe has joined #openstack-infra16:21
clarkbpabelanger: the other thing we will need to do before adding xenial to gating jobs is add it to the ubuntu mirror16:22
anteayamorning pleia216:24
anteayaI'm mostly not here today16:24
anteayawas up all night16:24
jeblairdid we roll-out and then roll-back anything for meta-april-fools-day?16:24
clarkbI havne't april fools16:24
*** e0ne has joined #openstack-infra16:24
anteayaI didn't think anyone in here had any time16:25
anteayaneutron seems to be taking the lions share of the jokes this year16:25
clarkbbut if anyone would like to be pranked where should I ship twin 9 month olds? they excel at this16:25
*** salv-orlando has joined #openstack-infra16:25
anteayaha ha ha16:25
anteayado they ship well?16:25
*** jlanoux has quit IRC16:25
clarkbprobably not, they need constant care16:25
pleia2anteaya: ugh, hope the flooding problem is a bit improved at least and you can finally get some rest16:26
anteayayeah, had to get a courier to take that contract16:26
clarkblooks like we need ~45GB overhead to build an image16:26
*** Qiming has quit IRC16:26
anteayapleia2: well the rain has stopped, so I hope the speed of the water coming in starts to slow by this evening16:26
clarkband we are sitting at ~73GB free? so should be ok for now and the removal of devstack-centos7 will clean up about 40GB16:26
anteayaso that's hopeful16:26
pleia2anteaya: good, I hope so :)16:26
anteayathank you16:27
anteayame too16:27
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add ubuntu xenial to reprepro
pabelangerclarkb: I think that is all we need ^16:27
pabelangerjust looked over the code16:27
*** diana_clarke has joined #openstack-infra16:27
*** e0ne has quit IRC16:29
*** esker has quit IRC16:29
*** tsg has joined #openstack-infra16:29
clarkbpabelanger: I think that is correct, everything else is templated to splat out per release16:29
*** roxanaghe has quit IRC16:29
pabelangerclarkb: Ya, that's what I see too16:29
*** e0ne has joined #openstack-infra16:29
*** yolanda has joined #openstack-infra16:30
*** antonym has quit IRC16:31
*** openstackgerrit has quit IRC16:31
pabelangerclarkb: just checked the afs volume too, we are currently at 83GB of 500GB usage16:31
*** rcernin has quit IRC16:32
*** kzaitsev_mb has quit IRC16:32
clarkbso should have plenty of room there16:32
*** tphummel has joined #openstack-infra16:33
NobodyCammorning INFRA folks16:34
NobodyCamis there a good beginner's guide to setting up a Ci system I can point folks to?16:36
*** jsavak has joined #openstack-infra16:36
AJaegerfungi, thanks for explanation about the chicken-egg problem. Now I understand. Sorry, pabelanger !16:36
fungijeblair: well, there is an amusing governance resolution courtesy of russellb, if you haven't seen it yet (300280)16:36
fungibut no, i don't think infra found time for pranks this year16:37
fungisaddened, as this is my favorite annual holiday16:37
*** gyee has joined #openstack-infra16:37
fungiNobodyCam: there is!16:37
anteayagerrit 502'd on me, it has been a while since it did that16:38
pleia2NobodyCam: that's the documentation for our puppet module that does it16:38
jeblairfungi: ha!  thanks russellb!16:38
*** achanda has joined #openstack-infra16:38
NobodyCamawesome thank you :)16:38
fungiNobodyCam: yes, also what pleia2 linked16:38
*** e0ne has quit IRC16:39
anteayaNobodyCam: the link fungi provided is required reading for anyone setting up a third party ci that interacts with our gerrit16:39
russellbjeblair: what did i do16:39
*** amitgandhinz has quit IRC16:39
anteayaNobodyCam: the link pleia2 provided is the documentation for how to set up the code16:39
fungiyep, for some reason i thought the thing i pasted included links to the thing pleia2 pasted, but maybe it doesn't yet16:39
pleia2ah yes, depends on whether you're looking to do it just as a standalone CI for a project somewhere, third party is for interacting with our gerrit for tests in an openstack project that reports to us16:39
fungithough also i think the openstackci documentation/module don't cover setting up a gerrit yet as it was mostly starting from the third-party ci use case. it's at least on the road map to add though if it's not in there yet16:40
pabelanger#success Launched our first ubuntu-xenial job with nodepool!
openstackstatuspabelanger: Added success to Success page16:41
* fungi cheers16:41
jeblairrussellb: you made me laugh :) 30028016:41
russellboh, yay16:41
*** smarcet has quit IRC16:41
clarkbpabelanger: it failed on the mirror :) ok that means we are on the right track adding that in16:42
pabelangerclarkb: indeed16:42
pleia2fungi: yeah, we have the out-dated that includes gerrit stuff, but it is a bit too stale (fortunately it warns as such)16:42
fungipleia2: right, i purposefully didn't suggest that one16:42
*** shardy has quit IRC16:42
pleia2fungi: nods16:42
jeblairthe nodepool config change has landed, i'm going to restart it now16:43
*** fedexo has joined #openstack-infra16:43
*** nwkarsten has quit IRC16:43
pabelangerjeblair: fungi: all that is needed to bring our xenial mirrors online.  Do you mind confirming16:43
pabelangerwhen you have a moment16:43
*** rcarrillocruz has quit IRC16:43
*** nwkarsten has joined #openstack-infra16:43
*** antonym has joined #openstack-infra16:44
*** pcaruana has quit IRC16:44
jeblairfungi, pabelanger: is it easy for one of you to delete leaked bluebox fips?16:44
pabelangersure, I can16:44
jeblairpabelanger: thanks; (cc fungi)16:44
*** openstackgerrit has joined #openstack-infra16:45
*** achanda has quit IRC16:45
pabelangerfungi: jeblair: deleted on bluebox, will check osic too16:46
clarkbosic leaks more slowly16:46
pabelangerosic looks good16:46
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Ironic grenade jobs do not work until after Mitaka
clarkbif at all16:46
*** mmedvede_ has quit IRC16:46
pabelangerya, I see 116:46
*** yamamoto has joined #openstack-infra16:47
*** dtantsur is now known as dtantsur|afk16:47
*** nwkarsten has quit IRC16:48
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's canonicalGitUrl option
fungii haven't seen fip leaks on osic before, fwiw16:49
fungiand we're hoping this restart means the last of the fip leaks on bluebox, yes?16:49
*** roxanaghe has joined #openstack-infra16:50
*** fawadkhaliq has quit IRC16:50
*** edmondsw has quit IRC16:50
fungior at least renders them more transparent and debuggable going forward16:50
jeblair2016-04-01 16:50:12,218 DEBUG nodepool.ProviderManager: Manager ovh-gra1 running task KeypairDelete (queue: 16)16:50
jeblairdoes that seem like something we should be doing ^ ?16:50
*** fawadkhaliq has joined #openstack-infra16:50
clarkbthe change should mean that fips get deleted whenever the server they were attached to is deleted thus preventing leaks16:51
openstackgerritKhai Do proposed openstack-infra/system-config: Change http download command to point to git.o.o
*** kzaitsev_mb has joined #openstack-infra16:51
clarkbjeblair: its definitely something that happens when doing snapshot builds. ovh doesn't have snapshot builds16:51
*** fedexo has quit IRC16:52
pabelangerI think something is up with jenkins03.o.o16:52
*** yamamoto has quit IRC16:52
jeblairclarkb: yeah, and even that should be in the builder now, not the main log16:52
jeblairalso weird -- "KeypairDelete" is the only "Keypair" task i see in the log16:53
*** tsg has quit IRC16:54
openstackgerritMerged openstack-infra/system-config: Add ubuntu xenial to reprepro
pabelangerthread count on jenkins03 looks to be okay, but it is running slow. Load average up to 416:54
pabelangerI think is just crashed16:55
clarkbjeblair: the only keypair code I see is in snapshot builds16:55
*** e0ne has joined #openstack-infra16:55
openstackgerritMerged openstack-infra/project-config: Add non-voting gate-trove-python34-db check
*** jsavak has quit IRC16:55
*** jsavak has joined #openstack-infra16:56
pabelangereep. I think I was the one who crashed jenkins0316:56
pabelangerCurrent thread (0x00007fced001d000):  JavaThread "Handling GET /monitoring from : RequestHandlerThread[#608]" daemon [_thread_in_native, id=20860, stack(0x00007fce96ced000,0x00007fce96dee000)]16:56
clarkbit crashed?16:57
jeblairclarkb: ah, we call delete_keypair in cleanupServer() in nodepool16:57
pabelangerclarkb: Ya, check /tmp/hs_err_pid14369.log on jenkins0316:57
*** bpokorny has quit IRC16:57
*** kzaitsev_mb has quit IRC16:57
pabelangerthat is the 2nd jenkins master today to be down16:57
openstackgerritMerged openstack-infra/project-config: i18n Support for freezer-web-ui
*** tqtran has joined #openstack-infra16:57
pabelangerjenkins05 this morning16:57
clarkbpabelanger: oh thats the I ran out of memory because thread stack ran out of space behavior16:58
clarkbpabelanger: it was probably leaking threads and couldn't make new ones if that was the case16:58
clarkbyou can check that log file and see how many htreads there were16:58
*** bpokorny has joined #openstack-infra16:58
openstackgerritKhai Do proposed openstack-infra/system-config: Change git download command to point to git.o.o
clarkbhrm though in this case it is complaining of sigsegv16:59
openstackgerritMerged openstack-infra/project-config: Add Ironic Inspector basic tempest job
*** bpokorny has quit IRC16:59
jeblairah, i understand the keypair thing.  explanation forthcoming.16:59
*** bpokorny has joined #openstack-infra17:00
clarkband thread count looks reasonable17:00
clarkbso this may be new fun and exciting jenkins brokeness17:00
pabelangerclarkb: okay, I'll start the process of cleaning up nodes. jeblair is it okay to interact with nodepool or should I wait17:00
clarkbor monitoring plugin brokeness17:00
clarkbpabelanger: we might consider clearing out the java melody data on all the servers if we restart them17:00
clarkbsince at least one had problems with that data during hte upgrade17:00
pabelangerclarkb: which folder is that?17:01
*** jsavak has quit IRC17:01
clarkbpabelanger: /var/lib/jenkins/monitoring17:01
*** chem has quit IRC17:02
clarkbsecret.key.not-so-secret is my new favorite file name17:02
jeblairclarkb, mordred:
pabelangerclarkb: okay, will wait until current nodepool testing is over before starting the clean up17:02
*** tosky has quit IRC17:02
jeblairclarkb, mordred: we used to have a conditional around the keypair delete there17:02
*** jsavak has joined #openstack-infra17:02
*** rbrndt has quit IRC17:02
jeblairclarkb, mordred: without that, we're doing an extra API call for every server delete17:02
*** eharney has quit IRC17:02
clarkbjeblair: nice17:02
fungiwhich is hopefully not crippling but would still be good to skip?17:03
clarkbjeblair: so we just need to add back in the check around server key_name?17:03
mordredjeblair: which api call are we making?17:04
jeblairclarkb: ideally, but i don't know what else may have changed wrt to that...(does nodepool still create the keys or did we shade that; i haven't gotten there yet)17:04
jeblairmordred: delete key pair17:04
jeblairKeypairDelete task17:04
mordredoh - the if server['key_name'] == self.provider.keypair yeah?17:05
*** aysyd has quit IRC17:05
mordredthe self.hasExtension conditional doesn't need to be there - but the other one could certainly be there, yeah17:05
jeblairfungi: yeah, we're mostly keeping up (provider task queues are low) so it's not crippling, just extra load, which at nodepool scale is a real thing17:05
jeblairmordred: yep17:06
mordredjeblair: the keypair create logic should be the same17:06
openstackgerritMerged openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's gitHttpUrl option
mordredjeblair: I'm guessing you're on it - let me know if you want me to look at a thing17:07
jeblairmordred: yep, i'm still making sure i have full context, but can write a patch when i am.17:07
mordredI will review it17:07
anteayaaccording to our etherpad we have 4 workrooms, 3 fishbowl and a day of meetup:
mordredalso, happy to hear it's keeping up and otherwise mostly working17:08
anteayaI think the current schedule is missing some sessions:
fungianteaya: what sessions specifically do you think are missing?17:09
*** trown|lunch is now known as trown17:09
anteayawell I see 2 workroom and 2 fishbowl in my browser at that link17:09
anteayaso 2 of the workroom and 1 of the fishbowl17:09
anteayanot sure if meetup is expected to be on that schedule or not17:10
fungidouble-checking. i saw them on the pdf versions but hadn't looked at the final on the www site yet17:10
*** mrmartin has joined #openstack-infra17:10
anteayasorry my fault17:11
anteayathat ws just for wednesday17:11
*** shashank_hegde has joined #openstack-infra17:11
anteayamy fault my fault17:11
fungiyep, we have sessions on wednesday/thursday and then sprint friday17:11
*** yamahata has quit IRC17:11
anteayain the past the project track filter gave you the whole week, not day at a time17:11
anteayasorry about that17:11
diana_clarkeHi folks, where can I see a list of passing and failing runs for a particular job in the check pipeline (before I "recheck" something)?17:12
jeblairmordred: will server['key_name'] exist on a server returned from a cloud without the keypair extension (or whatever is equivalent to that now)?17:12
diana_clarkeI've found the "OpenStack Health" tab for the "gate" pipeline (rather than "check"). My apologies if I'm missing the obvious.17:12
stevemaris there a page where infra has a list of projects for interns/students?17:12
openstackgerritSam Betts proposed openstack-infra/project-config: Add post job for building and uploading tinyipa
anteayastevemar: if the wiki still has a outreachy page17:12
anteayaI'd start there17:12
pleia2stevemar: there are a couple places where folks can start17:12
anteayadiana_clarke: yes health is just on gating17:12
pleia2stevemar: they can have a look at to see if there are any major projects we're working on that are of interest to them17:13
pleia2stevemar: I've also started seeing about doing more tagging of our bugs in storyboard17:13
stevemarpleia2: trying to think of an idea for a full stack developer, i'll take a peak17:13
mordredjeblair: there is no such thing as a cloud without the keypair extension anymore17:14
anteayadiana_clarke: use this and feed in a job name:
fungistevemar: the last outreacy intern we had, we came up with a suitably focused/scoped/bounded project and wrote a spec detailing what needed to be done17:14
anteayadiana_clarke: it will open your browser with a graph from graphite for the job17:14
jeblairmordred: so that entry will always be in the server dict?17:14
pabelangerdiana_clarke: zuul will post the results in review.o.o patchset too for each gate. Be sure to have the 'Toggle CI' button in right way17:14
pabelangerdiana_clarke: sorry, jenkins user17:14
mordredjeblair: should be. but let me emperically check that real quick17:15
pleia2stevemar: ah, for all of openstack, not just infra?17:15
*** e0ne has quit IRC17:15
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Only delete keypairs if needed
fungistevemar: right, if you're looking more for an example to get you started planning/scoping a task, the last one we did for an infra intern was
jeblairmordred: i went aheand and wrote it the 'safe' way, but if you think we shoud make it unsafe to fail hard on an unexpected condition, i can redo ^17:16
mordredjeblair: ok. well, I think shade should ensure the key is always in the dict regardless17:16
mordredwhich ist can do in normalize _servers - but I think the safe way is fine for now17:16
pleia2stevemar: if you are focused on infra if you go to!/search and type in low-hanging-fruit and select the tag looking one, then add openstack-ci and look for the 3 boxes, then type active to find Story Status: Active; that will list all the open infra bugs we at some point thought would be suitable for newcomers (which I'd say means students too)17:16
diana_clarkepabelanger: Yup, but I want to know if it's failing for a bunch of people before I enter "recheck". Normally, I just check other reviews around the same time, but I figure there must be a better way.17:17
diana_clarkeanteaya: Thanks! I'll give that a go.17:17
anteayadiana_clarke: I've found it useful for looking at a single job17:17
*** mrmartin has quit IRC17:18
pabelangerdiana_clarke: Ah, ya. I tend to look at the project view in gerrit and see if there are a bunch of recent -1 on reviews.  If so, dive deeper into gerrit17:18
*** yolanda has quit IRC17:19
*** mrmartin has joined #openstack-infra17:19
*** davideagnello has joined #openstack-infra17:19
*** amitgandhinz has joined #openstack-infra17:19
*** cdent has joined #openstack-infra17:20
mordredjeblair: +217:22
*** eharney has joined #openstack-infra17:22
jeblairwe leaked a bunch of nodes in rax-ord with no metadata17:23
jeblairthis apparently happened a while ago17:23
jeblairmar 2517:23
*** mrmartin has quit IRC17:23
jeblairi will nova delete them17:23
*** mriedem_lunch is now known as mriedem17:24
mordredjeblair: I have verified that we get key_name in the server dict even if we did not launch the node with a keypair17:24
*** imcsk8|plane is now known as imcsk817:24
clarkbI approved the change so if testing is happy with it it should be mergedin around an hour17:25
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's canonicalGitUrl option
mordredjeblair: we do not ensure that key_anme is in the dict in normalize_server though17:25
pabelangerjeblair: if you are deleting nodes, jenkins03 is down any existing nodes can be removed from nodepool17:25
*** jsavak has left #openstack-infra17:26
*** sambetts is now known as sambetts|afk17:26
jeblairpabelanger: done17:27
*** sputnik13 has joined #openstack-infra17:28
zaroclarkb: hey, you have time to take a look at a change to enable puppet code highlighting in gerrit?
anteayaI just submitted
openstackLaunchpad bug 1565018 in openstack-org "Austin Summit Schedule: Project Design Summit Track view please" [Undecided,New]17:29
jeblair2016-04-01 17:28:52,274 ERROR nodepool.NodeLauncher: OpenStackCloudException launching node id: 9278519 in provider: tripleo-test-cloud-rh1 error:17:29
jeblairOpenStackCloudException: unable to create floating IP for net None: Bad floatingip request: Cannot add floating IP to port on subnet f6603146-7ed8-4e09-8e64-f7e17dde8704 which has no gateway_ip17:29
jeblairNeutron server returns request_ids: ['req-f1594bd5-6a79-4869-9973-428694b4cce7']17:29
jeblairmordred, clarkb: ^17:29
jeblairmordred: does that mean that we wanted the second network instead of the first?17:30
clarkbjeblair: in this particular case yes, I do not think that is generally true17:30
pabelangerclarkb: so you are fine with me deleting the content of /var/lib/jenkins/monitoring?17:30
clarkbso the work around is too simple to trick neutron17:30
clarkbpabelanger: ya I moved it aside on the host that broke during the upgrade17:30
clarkbpabelanger: in case we needed or wanted to dig into why it might be causing problems17:30
openstackgerritMerged openstack-infra/shade: Remove conditional blocking on server list
jeblairclarkb: yeah.  i think mordred had a suspicion that nova must have been doing something like 'pick the first' for this to work at all using the nova api... but maybe there's more magic in there than suspected.17:31
clarkbI think when we did it before nova/neutron would be smart enough to find the port htat has a gateway and use that one17:31
jeblairthat would be sufficient magic, it seems17:31
clarkbbut when you explicitly list the port you have to do that yourselves17:31
pabelangerclarkb: okay, purging directory17:32
fungianteaya: apparently non-intuitive but one of the schedule app devs suggested searching like this to get the full-week schedule for a specific team's design summit sessions...
anteayathanks I will try that17:32
fungiseemed to work for me, fwiw17:32
mordredjeblair: ah! oh awesome17:32
mordredthat is sufficient magic - we cna do the same thing17:33
*** nwkarsten has joined #openstack-infra17:33
mordredjeblair: in fact, I can rework my follow up patch to just do that17:33
pabelangerjenkins03.o.o back online, running JJB now17:33
anteayafungi: yes, that is what I want17:33
anteayathank you17:33
*** skolekonov is now known as skolekonov|away17:33
jeblairmordred: do you want to go down that path, or the occ path?17:33
*** mriedem has quit IRC17:33
anteayaI'll append that to my bug17:33
mordredjeblair: I want to go down both paths17:33
*** nadya has joined #openstack-infra17:34
*** mriedem has joined #openstack-infra17:34
mordredjeblair: if we can discover and avoid the need for config, great. but golly what if someone makes two networks with gateway_ips - so we will likely need a way to override the discovery17:35
openstackgerritMonty Taylor proposed openstack-infra/shade: Consume config values for NAT destination
jeblairmordred: yeah.  and to be fair, that is reasonable.  (the general problem is not that openstack allows complexity, but that it does not support simplicity)17:36
mordredjeblair: there is the the occ patch17:36
mordredjeblair: minus tests17:36
*** salv-orlando has quit IRC17:36
anteayafungi: I see you put Infrastructure: in the search field17:36
mordredjeblair: let me start working on a follow up to it that will do the discovery17:36
jeblairmordred: do you think we should rollback nodepool?  cause either solution will require either an occ or shade release, yeah?17:37
mordredsince the structures put into place in that patch are still helpful17:37
mordredjeblair: yes. I do think we should rollback nodepool17:37
jeblairok.  will do now.17:37
mordredbut I'm excited that a) other than that it's working and b) I actually understand a good fix for this now :)17:37
fungianteaya: yep, basically just taking advantage of the fact that all design summit sessions start with the team name followed by a colon, while conference sessions (mostly) do not. they're working on improving this for barcelona a bit apparently but this will suffice for now17:38
*** nwkarsten has quit IRC17:38
jeblairmordred: yeah, the punch-list is getting short17:38
jeblairnodepool has been restarted as pre-shade17:38
*** yamahata has joined #openstack-infra17:39
mordredjeblair: cool. thanks!17:40
*** burgerk has quit IRC17:40
anteayafungi: cool, now that I know how it works, the magical colon17:41
anteayafungi: do they have any objection to me posting about the colon to the -dev list?17:41
anteayaor does anyone else?17:41
mordredjeblair: I think we can test a create_server(auto_ip=True) against the tripleo cloud before we re-try the nodepool restart17:41
fungianteaya: i can't imagine anyone would object to that17:41
anteayaI can't be the only one that wants this17:41
anteayafungi: awesome17:42
* anteaya composes17:42
*** salv-orlando has joined #openstack-infra17:45
clarkbjeblair: looks like there may be image build cleanup necessary due to the restarts :/17:46
anteayamy email draft if anyone cares to review:
*** yamamoto has joined #openstack-infra17:48
jeblairclarkb: hrm.  i thought we fixed that?17:48
clarkbjeblair: I thought so to, but for some reason the building images aren't going away. If we restart the builder they do17:49
clarkbI even thought I wrote a test for this17:49
clarkbmaybe that change never merged?17:49
clarkb it merge17:50
clarkbew have two image tables17:50
clarkbthis cleans up one of them17:50
clarkbneed to do the other17:50
*** nwkarsten has joined #openstack-infra17:51
*** fawadkhaliq has quit IRC17:52
*** fawadkhaliq has joined #openstack-infra17:52
*** markvoelker has quit IRC17:52
*** yamamoto has quit IRC17:52
jeblairclarkb: when that patch was written, it was sufficient.  :)17:52
jeblairbut yeah17:53
*** austin81 has left #openstack-infra17:53
anteayalast call for feedback on an email to the list about how to search the design summit app:
anteayathanks to those who have read so far17:55
*** markvoelker has joined #openstack-infra17:56
*** rbrndt has joined #openstack-infra17:56
fungianteaya: lgtm17:56
anteayafungi: thank you17:57
*** nhandler is now known as StaffUnicorn17:57
*** jistr has quit IRC17:58
pabelangerJJB finished on jenkins03 now17:59
anteayatypo in the subject line, d'oh18:00
*** flepied has quit IRC18:00
mnaserthings should be much improved on our side18:00
*** tsg has joined #openstack-infra18:01
*** asselin__ has joined #openstack-infra18:02
*** harlowja has quit IRC18:02
*** jed56 has quit IRC18:03
*** nadya has quit IRC18:03
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's download protocols
*** piet has joined #openstack-infra18:05
openstackgerritGraham Hayes proposed openstack-infra/project-config: Enable DB access for Designate unittests
*** pvaneck has joined #openstack-infra18:06
*** ybathia has joined #openstack-infra18:07
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's canonicalGitUrl option
*** dtardivel has quit IRC18:08
openstackgerritMerged openstack-infra/nodepool: Only delete keypairs if needed
*** jsavak has joined #openstack-infra18:08
*** bpokorny has quit IRC18:08
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add trailing whitespace to reprepo distributions
pabelangerfungi: clarkb: fix a syntax error in reprepro configuration ^18:09
*** gordc has quit IRC18:09
pabelangerconfirmed it fixes mirror-update18:09
openstackgerritMonty Taylor proposed openstack-infra/shade: Search subnets for gateway_ip to discover NAT dest
*** bpokorny has joined #openstack-infra18:10
mordredjeblair: ^^ that is not the final version of the patch18:10
mordredI just wanted to get it up there18:10
*** asselin__ has quit IRC18:10
mordredI actually want to poke at the occ logic for config a little before we land this18:10
*** smarcet has joined #openstack-infra18:11
*** nihilifer has quit IRC18:12
*** nihilifer has joined #openstack-infra18:14
*** nihilifer has quit IRC18:14
*** nihilifer has joined #openstack-infra18:15
mordredShrews: ^^ fyi - but I've marked the first patch WIP because I want to rework what we did in OCC the other day before we release it18:15
*** piet has quit IRC18:15
mordredShrews: writing this caused me to realize that I _think_ we might be conflating two different issues and want to noodle on it when i'm not streaming through my phonein the passenger seat of a car18:15
jeblairmordred: left some minor comments18:17
* AJaeger signs off and wishes a great weekend...18:17
*** lezbar has quit IRC18:17
anteayaAJaeger: thanks you too18:17
*** daemontool has quit IRC18:18
morganmordred: so i think ZK dogpile stuff is just going t be in dogpile.core18:18
morganmordred: which is nice.18:18
fungihave a great weekend, AJaeger!18:18
morganmordred: since the mutex can be defined there.18:19
fungineed to pop out for groceries before the tourists turn feral. back shortly18:19
*** asselin has joined #openstack-infra18:19
morganmordred: also there is a push to move dogpile.core and dogpile.cache into a single dogpile package because of namespace things (same issue with oslo.*)18:19
EmilienMI don't think this patch deserves a -2, but I might be wrong
morgancc jeblair ^ (re nodepool things and dogpile)18:19
morganEmilienM: it doesn't have a -2.18:20
EmilienMmorgan: it has WIP, indeed but to me it is not a WIP18:20
morganEmilienM: it has a -1 Workflow until mitaka is released.18:20
morganEmilienM: according to the comment.18:21
mordredmorgan: oh good18:21
EmilienMmorgan: I can read it too :-)18:21
morganEmilienM: ;) i know!18:21
EmilienMI just think, (and again, imo), we should not block patches because a release is happening.18:21
morganEmilienM: hm. thats sticky18:21
*** baoli_ has quit IRC18:22
jeblairEmilienM: we have in the past nearly frozen (we called it 'slush') infrastructure around release because of the potential disruption18:22
morganEmilienM: i think the alternative is a stable/<release> or similar branch to apply fixes pre-release to18:22
*** IlyaG has quit IRC18:22
morganand with puppet-* that iirc is much harder18:22
morganjeblair: ++18:22
jeblairEmilienM: because the tarball script is baked into images, if something goes wrong there, it will take quite some time to correct (probably a full day for image rebuilds)18:23
EmilienMjeblair: fair enough, I'll wait :-) I need to be patient ;-)18:23
morganoh this is project-config :P18:23
pabelangerEmilienM: If fungi wants to remove the WIP, I have no objections. I just didn't want it to land and break our when mitaka is being released18:23
morganjeblair: can we make snowcones with the slush?18:23
*** eil397 has joined #openstack-infra18:23
pabelangerrather error on the side of caution18:24
EmilienMpabelanger: np. Just wanted to understand the background and I got it.18:24
jeblairmorgan: yes, but i'd stay away from the yellow flavors18:24
morganjeblair: good idea!18:24
pabelangerEmilienM: Ya, once neutron is officially open, we can land it18:24
pabelangererr, mitaka is release18:24
openstackgerritKhai Do proposed openstack-infra/system-config: Customize Gerrit's download command and schema
Shrewsmordred: i left you some comments. i have to admit, even i'm starting to get befuddled by all of the various network options18:25
morganShrews: sooooo many network options18:26
morgani did a by-hand openstack deploy18:26
morganwas crazy to find all the infos.18:26
Shrewsmorgan: i meant just in shade18:26
*** asselin__ has joined #openstack-infra18:27
mordredShrews: yah.18:28
mordredShrews: I need to rework the config and then do a really clear doc about it18:28
mordredShrews: but I think I understand the landscape well enough to write that doc now18:28
Shrewsmordred: i look forward to reading it with a nice glass of wine and various soft cheeses18:29
mordredShrews: mmm. cheese18:29
Shrewsmordred: or maybe just beer and slim jims18:29
*** asselin has quit IRC18:29
Shrewsyeah... that second thing18:29
*** piet has joined #openstack-infra18:30
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's canonicalGitUrl option
openstackgerritRonald Bradford proposed openstack/requirements: Bump oslo.context to 2.2.0
*** flepied has joined #openstack-infra18:31
*** markvoelker has quit IRC18:31
*** edmondsw has joined #openstack-infra18:32
openstackgerritGraham Hayes proposed openstack-infra/project-config: Enable DB access for Designate unittests
*** kushal has quit IRC18:35
*** baoli has joined #openstack-infra18:35
openstackgerritMerged openstack-infra/shade: Make get_stack fetch a single full stack
*** dalgaaf has quit IRC18:38
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's download protocols
openstackgerritKhai Do proposed openstack-infra/system-config: Customize Gerrit's download command and schema
*** ifarkas has quit IRC18:43
*** keedya has quit IRC18:45
pabelangermordred: have a moment for
*** gordc has joined #openstack-infra18:47
mordredpabelanger: looing18:47
*** shashank_hegde has quit IRC18:49
*** kushal has joined #openstack-infra18:49
*** yamamoto has joined #openstack-infra18:49
*** elo has quit IRC18:49
*** Sukhdev has joined #openstack-infra18:53
*** david-lyle has quit IRC18:53
*** aysyd has joined #openstack-infra18:53
*** markvoelker has joined #openstack-infra18:53
*** markvoelker has quit IRC18:53
*** markvoelker has joined #openstack-infra18:54
*** markvoelker has quit IRC18:54
*** bpokorny has quit IRC18:54
*** yamamoto has quit IRC18:54
*** markvoelker has joined #openstack-infra18:54
openstackgerritMerged openstack-infra/system-config: Add trailing whitespace to reprepo distributions
*** esikachev has quit IRC18:55
*** piet has quit IRC18:56
dstufftclarkb: ;)18:56
clarkb(gerrit doesn't actually require squashed commits ... but everyone seems to interpret it that way)18:57
dstufftyea, it just requires you to review each commit18:57
dstufftwhich leads people to squash anyways I think most of the time18:58
clarkbI dunno I almost never squash18:58
*** piet has joined #openstack-infra18:58
clarkbusually its more due to gate reasons (this won't work unless 3 bugs are fixed beacuse why) situations18:58
*** degorenko is now known as _degorenko|afk18:58
clarkband those tend to be rare18:59
*** david-lyle has joined #openstack-infra18:59
*** Goneri has quit IRC18:59
*** Sukhdev has quit IRC19:00
clarkbjeblair: should I clean up the nodepool db now?19:01
*** piet has quit IRC19:03
*** fawadkhaliq has quit IRC19:05
clarkbpabelanger: also the xenial mirror change merged but I don't see it yet, that might need intervention?19:05
*** matt-borland has quit IRC19:05
clarkbI am not sure how to check on reprepro progress19:06
*** annegentle has quit IRC19:06
*** fawadkhaliq has joined #openstack-infra19:06
*** fawadkhaliq has quit IRC19:07
*** salv-orlando has quit IRC19:07
*** shashank_hegde has joined #openstack-infra19:10
*** tsg has quit IRC19:10
*** annegentle has joined #openstack-infra19:11
*** jsavak has quit IRC19:11
openstackgerritGraham Hayes proposed openstack-infra/project-config: Enable DB access for Designate unittests
*** dimtruck is now known as zz_dimtruck19:12
*** jsavak has joined #openstack-infra19:13
*** fawadkhaliq has joined #openstack-infra19:13
pabelangerclarkb: Ya, watching now.  It is a crontab19:15
pabelangerclarkb: I think mordred wanted it to run every 2 hours right?19:17
pabelangerwhat actually happens is it runs every 1min each 2 hours19:17
pabelangerso, we need to update the crontab19:17
pabelangerfor example:
claytonit appears that our internal nodepool instance got updated and now it's no longer trying to associate FIPs19:18
*** baoli has quit IRC19:19
openstackgerritPaul Belanger proposed openstack-infra/system-config: Run 'reprepro ubuntu' cron once ever 2 hours
pabelangerclayton: there have been some new commit to nodepool using shade. If you want to review the commit history19:21
claytonbeen looking, nothing obvious so far19:21
pabelangerclayton: I _think_ there is some new syntax too for public / floating IP networks19:21
pabelangerbut mordred and jeblair would know more19:21
*** hashar has joined #openstack-infra19:22
mordredclayton: you may want to revert nodepool to afdd58c10a8b85ea1466d32c3a29c0d7a1b0aea919:23
claytontrying that.  it's running the CreateFloatingIPTask, which is a plus19:25
openstackgerritMerged openstack/requirements: Add tempest to requirements
*** nadya has joined #openstack-infra19:27
*** nwkarsten has quit IRC19:27
*** nwkarsten has joined #openstack-infra19:27
*** baoli has joined #openstack-infra19:29
*** xarses has joined #openstack-infra19:30
xarsesHow can I clone node pool images outside of CI? ( has no url so beaker locally doesn't like me19:30
pabelangerclarkb: mirror should be updating in 30mins19:31
xarsesI can use a vagrant image but it returns a different result as the CI instance19:31
*** esikachev has joined #openstack-infra19:32
*** nwkarsten has quit IRC19:32
clarkbpabelanger: I need to fix that cron in my change too19:32
pabelangerxarses: in openstack-infra/project-config there is a build script.  You could do DISTRO=centos-minimal ./tools/build-image.sh19:32
xarsespabelanger: we don't have a URL we can clone it externally from?19:34
*** roxanaghe has quit IRC19:34
xarsespabelanger: I'll give that a shot19:34
xarsespabelanger: sorry, the image that is already being used in CI, I have the project-config repo19:35
*** ybathia has quit IRC19:36
*** harlowja has joined #openstack-infra19:37
pabelangerxarses: right, you need to use diskimage-builder to create the images we use19:37
pabelangerxarses: with the tools/ script does for you19:37
*** esikachev has quit IRC19:38
xarsespabelanger: so the answer is no, we don't have a public URL to clone the current image from, it has to be built by hand?19:38
pabelangerxarses: right, no public images.19:38
*** jsavak has quit IRC19:39
*** auggy is now known as auggy_rodin19:39
clarkbpabelanger: jeblair I will clean up nodepool db situation after lunch19:40
pabelangerclarkb: ack19:40
*** fawadkhaliq has quit IRC19:43
*** baoli has quit IRC19:43
*** fawadkhaliq has joined #openstack-infra19:44
*** fawadkhaliq has quit IRC19:45
*** fawadkhaliq has joined #openstack-infra19:46
*** elo has joined #openstack-infra19:48
*** salv-orlando has joined #openstack-infra19:48
xarsesso where am I supposed to get diskimage-builder from?19:49
*** yamamoto has joined #openstack-infra19:50
fungixarses: if you installed nodepool (did you say you're using nodepool, right?) then diskimage-builder is a prerequisite. we have a puppet-nodepool module you can use to install nodepool and all its dependencies, or you can pip install nodepool and it should pull in diskimage-builder as a dependency for you
fungiif you're not using nodepool, you can still pip install diskimagebuilder19:51
fungier, pip install diskimage-builder19:51
xarsesfungi: as probably derived, I have no clue what I'm doing here19:52
*** Sukhdev has joined #openstack-infra19:52
fungixarses: what is it you ultimately want to use/do?19:52
fungixarses: are you setting up a ci system?19:52
fungiand what kind of configuration management framework are you using (if any)?19:53
xarsesI need an image that I can hand to beaker so that I can diagnose why the CI image fails my project's CI19:53
xarsesfor a specific change19:53
xarsesbasically, I need a live instance I can troubleshoot19:54
fungioh, this may be a question for the #openstack-puppet channel (what change are you diagnosing?)19:54
*** jsavak has joined #openstack-infra19:54
fungixarses: i'm guessing this is some magic being used by the puppet-ceph tests19:54
*** baoli has joined #openstack-infra19:55
*** yamamoto has quit IRC19:55
xarseschanging the puppet service handler from init -> redhat breaks redhat, but it's only happening this way on the CI instance, if I use the companion vagrant image, it doesn't have the same failure19:55
fungixarses: got it. and so you want to be able to run the same job manually on an image similar to what we tested on (our centos-minimal based diskimage-builder images)19:57
*** annegentle has quit IRC19:57
fungifor a moment i thought you were trying to run something which consumes an image, so was off on a completely different tangent19:57
xarsesoh, no the former19:58
fungixarses: in that case pabelanger was on the right track. you want diskimage-builder from somewhere but probably pip install latest release from pypi will get you closest to what we're using19:59
fungiand the script to build an image19:59
clarkbok nodepool is cleaned up20:00
fungijroll was, i think, just doing the same earlier this week and may have some suggestions too20:00
xarsesERROR: Element 'openstack-repos' not found in 'nodepool/elements:/home/andreww/.virtualenvs/fuel/bin/../share/diskimage-builder/elements'20:00
xarses+ IMAGE_ELEMENT=20:00
xarsesI've pip installed diskimage-builder20:01
fungiyeah, that's complaining about one of our dib elements from the project-config repo, in the nodepool/elements directory20:01
fungirather, complaining that it's not finding it20:01
*** annegentle has joined #openstack-infra20:02
xarsesand project-config's test-requirements20:02
pabelangerwhere did you run the command from?20:03
pabelangershould be top-level project-config20:03
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints
*** zz_dimtruck is now known as dimtruck20:03
fungiso like ./tools/build-image.sh20:03
xarsesran it from the tools folder20:03
fungisounds like the script could use a comment noting the cwd it expects20:03
*** apoorvad has quit IRC20:04
lifelessfungi: does my bindep patch feedback make sense to you?20:04
*** fawadkhaliq has quit IRC20:04
pabelangerclarkb: reprepro running now20:04
clarkbpabelanger: kk I should fix my change now before I forget20:04
xarsesyay, no RPM command found...20:04
*** fawadkhaliq has joined #openstack-infra20:05
pabelangerYa, so you need some dependencies too20:05
pabelangerwhat is your host OS?20:05
pabelangerwhat you need at minimum20:06
xarsesmight as well just run this in docker20:06
xarsesya, but that doesn't seem to help the fact that we just tried to call rpm tools20:06
xarses(im building the centos image)20:07
pabelangerYa, you need yum-utils20:07
*** bpokorny has joined #openstack-infra20:07
pabelangerwhich that role is missing20:07
pabelangerneed to fix that20:07
fungilifeless: i haven't had a chance to look yet but hope to shortly20:07
fungiwill let you know20:07
*** jsavak has quit IRC20:07
*** Sukhdev has quit IRC20:08
*** jsavak has joined #openstack-infra20:08
*** roxanaghe has joined #openstack-infra20:09
openstackgerritClark Boylan proposed openstack-infra/system-config: Apt mirror for ceph hammer release
*** angdraug has joined #openstack-infra20:11
openstackgerritClint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate grenade plugins list
pabelangerxarses: what version of debian are you using20:16
pabelangerYa, not surprised it doesn't work20:16
pabelangerI'd use centos7 or ubuntu-trusty as your build host20:16
fungidid yumdownloader get non-backward-compatible changes in newer debian packages, such that it can no longer support centos 7 era rpmdb?20:17
*** sdake has quit IRC20:17
pabelangerI think yumdownloader is deprecated now20:18
pabelangersomething dnf related replaced it20:18
openstackgerritClint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate tempest plugins list
openstackgerritClint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate grenade plugins list
xarsessomething like that20:18
fungiso dib presumably needs updating20:18
fungito conditionally use whatever has superseded yumdownloader when dealing with later rh releases20:19
fungibut still makes me wonder why the newer yum-utils on debian isn't working with centos 7 when the version on ubuntu trusty is20:20
pabelangerRight, we are actually using the host to build the yum cache.  Which needs to be updated to dnf20:20
fungii would expect a debian bug reported against yum-utils for this (not compatible with --releasever=7 centos-release or something like that), definitely not seeing one20:21
fungithough there is a bug open about it missing a depends on python-lzma20:22
fungioh, or maybe just a recommends20:22
fungiSpamapS reported it one year and a day ago20:23
fungidoesn't look like it could be responsible for the issue you're hitting though20:23
xarsessounds like I just need to wrap this all in a container20:23
*** salv-orl_ has joined #openstack-infra20:23
clarkbit may not work in a container20:24
fungiyeah, we _successfully_ (for semi-reasonable definitions of "success") run this daily on ubuntu 14.04 lts20:24
clarkbor at the very least require oyu to punch holes for mounting and things20:24
* xarses sighs20:24
xarsesso again, we can't publish the current image?20:25
clarkbwe could but we don't20:25
clarkbits been a wishlist item needs someone to work onit20:25
fungixarses: we could, though it's not generally usable without modification or introduction of some mechanism to add your authentication credentials to it20:25
*** roxanaghe has quit IRC20:26
fungiright now it has access from our ci system hard-coded into the image, password auth disabled, no other generic accounts, et cetera20:26
clarkbapache to search the files with an index to human names and if we grow a second builder host some sort of aggregation20:26
*** salv-orlando has quit IRC20:26
fungiyeah, i can imagine just pointing apache at the build dir along with a linkfarm we update for human-readable names20:27
clarkbthen there is the issue of formats, do we serve all of them? the smallest one (qcow2) etc20:27
fungibut definitely would need some documentation around how to log into the images, since doing so in their current state is decidedly nontrivial unless you're our ci system20:27
*** dprince has quit IRC20:28
clarkbfungi: you just mount image, add your key, boot20:28
clarkbbut maybe I build too many images now and that is easy to me :)20:28
fungiyeah, modify image while mounted on a loopback is certainly one solution. i was counting that as nontrivial (for typical python developers) but maybe my characterization is overbroad20:28
clarkbfwiw I tend to build the images locally rather than downloda them because I can maintain a cache for dib which is easier on the network bandwidth20:28
clarkbbut I have to trade cpu time and memory to et an imae20:29
*** |-paul-| has joined #openstack-infra20:29
fungiand also you happen to have easy access to an ubuntu trusty system on which to build them20:29
xarsesso this is increasingly sounding like a futile exercise, after I have this image built, It sounds like I won't be able to login anyways20:29
clarkbfungi: it runs in a VM on my HTPC20:29
fungior something similar enough that you're not encountering this particular build bug20:29
clarkbxarses: no, you can you just have to make an additional edit to the image to add your key20:30
fungixarses: when building the image you can tell it to add your credentials with a "devuser" element20:30
fungiwhich is why so far the recommendation has been to recreate rather than download, as you need to jump through hoops to be able to log into it anyway20:30
*** thinrichs has joined #openstack-infra20:30
clarkbI think glean will support metadata injection for ssh keys but that assumes you have a config drive or metadata server20:31
fungiyeah, booting it with vagrant likely won't provide a mechanism for that20:31
fungithough maybe you can do it with a second block device mounted as an additional drive20:32
fungianyway, falls into the "nontrivial" category again20:32
fungii agree it's pretty silly to need to have an ubuntu trusty vm to build a centos 7 image if you're starting from a system which is neither of those things20:33
thinrichsBesides openstack-server-release-jobs, what jobs do I need in zuul for the release team to include Congress in a release?  The release team said I should ask here.20:33
*** sdake has joined #openstack-infra20:33
* xarses goes to lunch20:34
*** roxanaghe has joined #openstack-infra20:35
fungithinrichs: if it's not something being uploaded to pypi, that's probably it? i can't say for sure since we don't set the release team's policies for them. you also need python-jobs to be able to have a tarball job, but you seem to have that already. also maybe you need release-notes-jobs if you're using reno for your release notes?20:36
*** tmorin has joined #openstack-infra20:37
*** IlyaG has joined #openstack-infra20:37
*** markvoelker has quit IRC20:37
*** tmorin has left #openstack-infra20:37
*** apoorvad has joined #openstack-infra20:38
pabelangerxarses: export NODEPOOL_SSH_KEY=`sudo cat /var/lib/nodepool/.ssh/ | awk '{print $2}'`20:38
pabelangerthat will let you login via the jenkins user20:38
pabelangerwell, your path to your public SSH key20:39
*** claudiub has quit IRC20:40
clarkbpabelanger: and then login as the jenkins user?20:41
*** jsavak has quit IRC20:41
fungii think the more general recommendation from dib'ers is to us ethe devuser element20:41
*** jsavak has joined #openstack-infra20:41
clarkbya gives you more control20:41
pabelangerwell, if your using your script20:42
pabelangerotherwise, I would agree devuser element20:42
clarkbmaybe we should just add the devuser element to that script and delete the old ssh injection support20:42
clarkbthen you cna get a devuser thats useless by default or you can configure it and it will be useful20:42
pabelangerI'd be okay with that20:42
fungialso there are some elements you can safely turn off when recreating for local use, particularly the openstack-repos which causes the image to be an order of magnitude larger and lake a ton more time/bandwidth to build20:42
pabelangeras I always forget the export syntax20:42
*** mylu has joined #openstack-infra20:45
*** ybathia has joined #openstack-infra20:45
*** roxanaghe has quit IRC20:46
clarkbmaybe we should just turn that into a doc20:47
clarkbdescribe the elements we use, point out which can be exlcuded to save space (the repos and devstack caching)20:47
fungimight be nice to make at least openstack-repos easy to skip20:47
clarkband an example disk image builder invocation to do something useful20:47
clarkbrather than try and can a thing int oa script20:47
fungiand also how to avoid it adding our accounts to your image ;)20:47
clarkboh I don't mind that20:47
*** annegentle has quit IRC20:48
thinrichsfungi: thanks.  I'll follow up with the release team.20:48
pabelangerwell, it does help when you are trying to get a new image online! :)20:48
*** thinrichs has left #openstack-infra20:48
fungialso i wonder if they come up pointed at our dfw package mirrors20:48
pabelangerHowever, I would much rather use nodepool.yaml then a script20:48
*** annegentle has joined #openstack-infra20:48
clarkbI am saying don't do either20:48
pabelangerYa, default ubuntu mirrors20:48
pabelangercan confirm20:49
clarkband just have a doc that explains how to be self sufficient20:49
fungisimilar to what i did in the devstack-gate readme20:49
*** roxanaghe has joined #openstack-infra20:49
fungia walkthrough essentially, with enough background info on prerequisites/gotchas that most readers won't get irrevocable stuck20:49
*** annegentle has quit IRC20:50
*** ricky1 has joined #openstack-infra20:50
*** annegentle has joined #openstack-infra20:50
*** yamamoto has joined #openstack-infra20:51
clarkbpabelanger: any idea what reprepro is doing? I don't see any xenial logs or other indication it is xenialing yet20:52
clarkbmaybe I just need to be more patient20:53
fungilifeless: read your comments on 200672 and generally agree, though AJaeger has also already replied asking the followup questions i was going to ask20:53
pabelangerclarkb: Ya, I think it is working.20:53
clarkbI should probably go read more manpages20:53
pabelangerbut we should likely enable logging for it20:54
clarkbpabelanger: I think logging is enabled20:54
clarkbpabelanger: there are logs for all the trusty things20:54
clarkbmaybe its determining what needs updating and doesn't have anything ot log yet?20:54
*** ricky1 has quit IRC20:55
*** annegentle has quit IRC20:55
pabelangerclarkb: Ya, we need to add logdir to the options file IIRC20:55
*** yamamoto has quit IRC20:56
*** mriedem1 has joined #openstack-infra20:56
pabelangeror verbose20:56
lifelessfungi: cool, will look at the q's on moday20:57
*** annegentle has joined #openstack-infra20:57
fungilifeless: there's no hurry. i'm answering some of his questions and expounding on others as we speak. and thanks for the input!20:57
clarkbpabelanger: gotcha20:58
*** eharney has quit IRC20:58
*** eharney has joined #openstack-infra20:59
*** mriedem has quit IRC20:59
*** smarcet has quit IRC20:59
*** roxanaghe has quit IRC20:59
*** elo has quit IRC21:00
*** gordc has quit IRC21:02
*** amitgandhinz has quit IRC21:03
pabelangerclarkb: okay, logs enabled but they look to be in /21:03
pabelangershould move them into afs volume21:04
clarkbpabelanger: I would put them in /var/log/reprepro with the other logs21:04
pabelangerthat works too21:04
clarkbodd that they end up in / guessing thats just misconfigured21:04
pabelangerclarkb: ya, it is no long running. Maybe we should run it manually to see what is going on21:06
pabelangermaybe something misconfigured21:06
*** roxanaghe has joined #openstack-infra21:09
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add back logdir for reprepro.pp
pabelangerclarkb: ^21:11
*** trown is now known as trown|outtypewww21:11
*** devananda has quit IRC21:12
*** devananda has joined #openstack-infra21:13
openstackgerritPaul Belanger proposed openstack-infra/system-config: Increase verbosity for reprepro command
pabelangerand verbosity ^21:13
clarkbI went ahead and fast approved the fix for my derp21:13
clarkbI see now that it is used by a template21:13
clarkbwhich I must've missed when cleaning up the parameters21:14
* pabelanger nods21:14
*** hahyes has joined #openstack-infra21:14
*** cdent has quit IRC21:14
hahyeshi new to openstack-infra.. trying to get some answer for jenkins job builder...21:15
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add missing release jobs
*** _ari_ has quit IRC21:15
pabelangerclarkb: Ya, so reprepro finished but didn't pull down xenial. Not sure why atm21:16
pabelangerbut have to afk21:16
openstackgerritMerged openstack-infra/system-config: Change http download command to point to git.o.o
clarkbhahyes: ask away21:16
*** thorst has quit IRC21:17
*** regXboi has quit IRC21:17
*** EricGonc_ has quit IRC21:18
*** rfolco has quit IRC21:21
*** jsavak has quit IRC21:21
*** tmartins has joined #openstack-infra21:22
tmartinshey guys, how can I fill a bug or a request for a feature, for Python Shade?21:22
clarkbtmartins:!/project/760 is the bug tracker for shade21:23
tmartinssounds awesome, thank you!21:24
openstackgerritJulia Kreger proposed openstack-infra/project-config: Correct missing bifrost release setting
*** ayoung is now known as ayoung-afk21:25
*** kakocyan has joined #openstack-infra21:25
*** jcooley_ has quit IRC21:25
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Give project teams access to approve stable/mitaka changes
*** ybathia has quit IRC21:26
kakocyanHi. I am new here and trying to use Jenkins job builder. I need to use folders and also installed dashboard views. How can I specify the jobs under the view and folders?21:26
clarkbI don't know that jjb supports that21:27
clarkbuntil relatively recently the global config for jenkins has included state information about the slaves too making it hard to edit21:28
openstackgerritMerged openstack-infra/system-config: Add back logdir for reprepro.pp
*** mriedem1 is now known as mriedem21:29
*** Sukhdev has joined #openstack-infra21:29
*** akshai_ has quit IRC21:30
*** ayoung-afk is now known as ayoung21:30
openstackgerritMerged openstack-infra/system-config: Increase verbosity for reprepro command
*** ayoung is now known as ayoung_afk21:30
*** Sukhdev has quit IRC21:31
kakocyanI run my local jenkins server and with these plugins, I could inject my job definition(just by specifying job name like 'folder1/childjob'). But when I tried on our central jenkins server, it complained. JenkinsException: Cannot create job[keymaster/Alchemy-Keymaster] because folder for the job does not exist21:31
*** Goneri has joined #openstack-infra21:31
clarkbits possible you could add that to jenkins job builder21:32
clarkbhave it create dirs if necessary21:32
*** ldnunes has quit IRC21:32
openstackgerritDoug Hellmann proposed openstack-infra/project-config: update tag ACLs for all official projects
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Give project teams access to approve stable/mitaka changes
kakocyanActually the dir was already there.  On my local server, it could find and add my job, but not on the main server that I do not have much knowledge and control.21:34
*** baoli has quit IRC21:36
*** kong has joined #openstack-infra21:36
clarkbkakocyan: my reading of that error is that it failed because the dir keymaster did not exist21:36
clarkbkakocyan: it is probably possible to detect that scenario and create the dir first21:36
*** thorst has joined #openstack-infra21:38
*** mriedem has quit IRC21:38
*** nadya has quit IRC21:38
*** annegentle has quit IRC21:39
jeblairi want to use openstackclient to view our quota, but i don't see how...21:39
clarkbjeblair: one sec21:39
jeblair'quota show' wants a 'project' argument...21:39
clarkbhrm I thought there was a server quota show21:40
clarkbbut not seeing it21:40
*** tiswanso has quit IRC21:41
jeblairoh, 'quota show' is an admin method21:41
clarkbyou might need to use nova client to see the nova quota and $other client for that quota21:42
*** thorst has quit IRC21:42
jeblair'limits show --absolute' is a thing21:42
kakocyanOops. we have so many jobs/folders created under that view(keymaster) already. keymaster is not a job. It is a view. But my local server seems to be able to find it. Am I missing something?21:42
clarkbthats an odd thing to make inconsistent21:43
clarkbquota vs limit21:43
clarkbI wonder if osc would be open to making them the same21:43
clarkbkakocyan: on your non local server does keymaster exist?21:43
jeblairclarkb: i recall 'nova absolute-limits'21:43
*** bpokorny has quit IRC21:43
*** daemontool has joined #openstack-infra21:44
clarkbkakocyan: the implication of that error message is that it does not21:44
dtroyerclarkb: agreed, there is a distinction that I don't remember offhand21:44
dtroyerit needs cleaning up21:44
jeblairdtroyer: i'm wondering if 'quota' is used on the admin side and 'limits' is used on the user side?21:44
jeblairdtroyer, clarkb: but yeah, that feels like a distinction without a difference that could probably be cleaned up21:45
clarkbthe other weird one I ran into recently is with security groups21:45
kakocyanYes. On my local server, I created keymaster manually. That is the same with the main server. It was created manually.21:45
clarkbhalf the commands map to nova and half map to neutron21:45
clarkband its sort of not clearwhich do which21:45
dtroyerthose are under active transition21:46
clarkbdtroyer: good to hear21:46
dtroyershould be much better Real Soon Now21:46
clarkbkakocyan: maybe the user doesn't have access to it?21:46
dtroyerthey will detect Neutron in the service catalog21:46
clarkbkakocyan: jjb basically needs full admin rights iirc21:46
jeblairi have a command i want to run against every cloud+region in clouds.yaml21:46
jeblair(i bet you can guess which command)21:47
jeblairis there any neat occ/openstackclient trick to help with that?21:47
dtroyerjeblair: not at present21:47
kakocyanOh. That probably is. I was hoping I can handle with my own id since the view was under my team's control.21:47
dtroyerwe could add something to select the config in interactive mode and pipe stuff in21:47
jeblairdtroyer: or something to output the full list that i could then use in a bash for loop would be cool21:48
dtroyerah, sure.  we may have something already, hang on21:48
kakocyanI will ask the server team if they can let us handle it.  Thanks!21:48
jeblairdtroyer: ooh21:48
*** ayoung_afk is now known as ayoung21:49
dtroyerjust 'configuration show'  adding a list would be easy21:49
*** xyang1 has quit IRC21:49
*** hahyes has quit IRC21:52
*** EricGonczer_ has joined #openstack-infra21:52
*** Sukhdev has joined #openstack-infra21:52
*** Sukhdev has quit IRC21:55
*** sdague has quit IRC21:56
jeblairfor the purposes of what i'm doing, i'm running this command against 23 cloud+region combos21:56
jeblairfirst-world problems, i know21:56
tmartinsclarkb, new story added!!/story/2000537   :-D21:57
*** hashar has quit IRC21:59
*** kakocyan has quit IRC21:59
*** mdenny has joined #openstack-infra21:59
jeblairclarkb, mordred: my survey says we have a limit of 40 server metadata entries (i assume this is a per-server limit) in rax, 20 in internap, and 128 elsewhere.22:00
*** Sukhdev has joined #openstack-infra22:00
*** markvoelker_ has joined #openstack-infra22:01
tmartinsOne more:!/story/2000538   ^_^22:02
*** dimtruck is now known as zz_dimtruck22:03
*** daemontool has quit IRC22:03
*** fawadkhaliq has quit IRC22:05
*** fawadkhaliq has joined #openstack-infra22:05
pvaneckjeblair: Hey, can you initiate the workflow for this: . Want to get the live site updated.22:05
*** Sukhdev has quit IRC22:07
*** ZZelle_ has joined #openstack-infra22:10
*** lutf-8 is now known as alaski22:12
*** sigmavirus24 is now known as sigmavirus24_awa22:13
*** sdake_ has joined #openstack-infra22:13
openstackgerritClark Boylan proposed openstack-infra/nodepool: Cleanup building dib images on startup
*** sdake has quit IRC22:13
clarkbjeblair: ^ I think that will address the problem from earlire today22:13
*** edmondsw has quit IRC22:15
clarkbI should just make a permanent mysql install for nodepool testing22:17
openstackgerritTim Hinrichs proposed openstack-infra/project-config: Add openstack-server-release-jobs to Congress in zuul
*** sdake has joined #openstack-infra22:19
openstackgerritClark Boylan proposed openstack-infra/nodepool: Cleanup building dib images on startup
*** sdake_ has quit IRC22:21
jeblairpvaneck: sure thing22:22
jeblairclarkb: ++22:23
clarkboddly I think the most recent failure is in the fakes not mysql22:24
clarkbit is somehow booting a dib based node without a valid dib image22:24
clarkboh wait I know22:24
*** LaT0rtue has quit IRC22:24
*** Sukhdev has joined #openstack-infra22:25
pvaneckjeblair: thanks!22:25
openstackgerritClark Boylan proposed openstack-infra/nodepool: Cleanup building dib images on startup
openstackgerritMerged openstack-infra/irc-meetings: Add Non-ATC Recognition WG Meeting
*** HeOS has quit IRC22:28
*** achanda has joined #openstack-infra22:29
*** mixos has joined #openstack-infra22:29
*** nwkarsten has joined #openstack-infra22:30
*** aysyd has quit IRC22:31
*** HeOS has joined #openstack-infra22:31
*** mylu has quit IRC22:32
*** mylu has joined #openstack-infra22:33
openstackgerritMerged openstack-infra/puppet-refstack: Make puppet pull releases from Pypi
*** nwkarsten has quit IRC22:35
*** markvoelker_ has quit IRC22:39
*** flepied1 has joined #openstack-infra22:39
openstackgerritIan Cordasco proposed openstack-dev/pbr: package: fix wrong catch in email parsing
*** pradk has quit IRC22:41
*** flepied has quit IRC22:43
*** krtaylor has quit IRC22:44
*** |-paul-| has quit IRC22:44
*** fawadkhaliq has quit IRC22:44
*** fawadkhaliq has joined #openstack-infra22:45
*** mixos has quit IRC22:46
*** rvasilets__ has joined #openstack-infra22:48
*** daemontool has joined #openstack-infra22:51
*** knikolla has quit IRC22:53
*** IlyaG has quit IRC22:54
*** IlyaG has joined #openstack-infra22:54
*** yamamoto has joined #openstack-infra22:55
*** IlyaG has joined #openstack-infra22:55
*** IlyaG has quit IRC22:55
*** krtaylor has joined #openstack-infra22:56
*** keedya has joined #openstack-infra22:58
*** yamamoto has quit IRC23:00
*** rlandy has quit IRC23:02
*** ams_ has quit IRC23:03
*** mylu has quit IRC23:03
*** fawadkhaliq has quit IRC23:04
*** auggy_rodin is now known as auggy23:04
*** fawadkhaliq has joined #openstack-infra23:04
*** rbrndt has quit IRC23:06
*** blogan has quit IRC23:09
*** roxanaghe has quit IRC23:10
*** roxanaghe has joined #openstack-infra23:10
*** mtanino has quit IRC23:13
*** Swami has quit IRC23:13
*** xarses has quit IRC23:16
*** armax has quit IRC23:17
*** daemontool has quit IRC23:17
*** daemontool has joined #openstack-infra23:18
*** armax has joined #openstack-infra23:19
*** trown|outtypewww is now known as trown23:21
*** knikolla has joined #openstack-infra23:22
*** Sukhdev has quit IRC23:23
*** erlon has quit IRC23:27
asselin__zaro, our gerrit stopped e-mail folks. any tips on where to look for debugging it?23:28
clarkbasselin__: do you use our puppet manifests for it? if so you are likely running exim. check the logs for that23:29
asselin__ok thanks23:29
clarkbgerrit will hand off to exim which will then send email23:30
*** angdraug has quit IRC23:34
*** harlowja has quit IRC23:36
*** yamahata has quit IRC23:36
*** LaT0rtue has joined #openstack-infra23:37
*** marcusvrn_ has quit IRC23:37
openstackgerritJulia Kreger proposed openstack-infra/project-config: Fix broken ironic gate jobs
*** daemontool has quit IRC23:38
jeblairpvaneck: refstack.o.o looks dead.  it looks like the npm installation failed due to npm errors23:38
clarkbdid leftpad get deleted again?23:39
pvaneckjeblair: yea, trying to see if the next puppet run will correct it as a new tag ws as just pushed23:40
jeblairclarkb: 403 on phantomjs23:41
clarkbjeblair: same thing as that horizon job23:41
clarkbI think bitbucket is unhappy23:41
jeblairpvaneck: it just ran again; i'm reading logs23:42
*** esker has joined #openstack-infra23:43
*** mixos has joined #openstack-infra23:43
jeblairApr  1 23:38:42 refstack puppet-user[863]: (/Stage[main]/Refstack::App/Exec[npm install]/returns) npm ERR! cb() never called!23:43
jeblairApr  1 23:38:42 refstack puppet-user[863]: (/Stage[main]/Refstack::App/Exec[npm install]/returns) npm ERR! not ok code 023:43
jeblairi have no idea what that means.  i don't see any other errors23:43
jeblairpvaneck: it looks like anything under /assets is 40423:46
pvaneckhmm, on the refstack server, does doing 'npm install' in /opt/refstack yield the same thing?23:46
*** esker has quit IRC23:47
jeblairpvaneck: npm install in that directory as refstack succeeds, and the output doesn't look anything like what was in the logs.23:48
pvaneckcant recreate it locally. not sure why puppet is failing to npm install23:54
jeblairpvaneck: the site is working now...23:54
pvaneckbut it seems like you performing the npm install fixed the issue23:54
jeblairi'm not sure it was that23:55
jeblairthere has been another puppet run, and *that* one performed the sync of the www directory23:55
*** yamamoto has joined #openstack-infra23:56
pvaneckto pick up the changes in /opt/refstack i believe from your manual npm install23:56
jeblairor possibly from the npm install that puppet ran23:57
jeblairhard to say which23:57
*** ajmiller has quit IRC23:58

Generated by 2.14.0 by Marius Gedminas - find it at!