Friday, 2018-06-29

tristanCanyway, i'll give this selenium simple-usage a try today using a simple tox target, we can look after how to properly integrate this00:00
*** Shrews has quit IRC00:26
*** austinsun[m] has joined #zuul00:46
openstackgerritMerged openstack-infra/zuul master: web: fix status page flickering  https://review.openstack.org/57822600:46
*** pwhalen has joined #zuul01:10
*** pwhalen has joined #zuul01:10
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page  https://review.openstack.org/57903901:55
tristanCcorvus: clarkb: initial selenium tests looks like this ^01:55
tristanCsadly i couldn't make the firefox driver works and had to install chrome to get the browser console logs01:57
corvustristanC: can you use chromium?01:58
tristanCprobably, but Chromium isn't officially supported01:59
corvuswe won't be able to use chrome as it's non-free02:00
corvushow important is the logging interface?02:00
tristanCit sounds important to assert no javascript errors appears in the logs02:01
tristanCwe could get away with firefox by using a custom console.log handler in tests that write them to a list02:01
corvusthat might be a workable compromise02:02
tristanCthen we get access using driver.execute_script("return console_logs_list")02:02
tristanCwell 579039 is a first draft, agree that chrome is a blocker, but unfortunately that was the path of least resistance02:03
corvusit looks like a great start -- really easy to follow :)02:03
tristanCnext PS could try adding "click()" action and checks for correct element-id presence in the dom02:04
tristanCalso, Shrews fix for pool's labels works like a charm02:10
tristanChum, wait, now quota calculation seems off, requests are paused while there is plenty of room in the tenant03:01
clarkbtristanC: each pool has its own max server though right? could it be running up against that?03:02
clarkbthe angular updates appear to be working fine this time around03:02
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: status: add pool column to nodepool list output  https://review.openstack.org/57904603:06
tristanCclarkb: indeed, i got confused by some ready-node eating up available pool quota03:08
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: launcher: add pool quota debug and log information  https://review.openstack.org/57904803:12
tristanCAll hail to the angular6 overlord :-)03:14
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page  https://review.openstack.org/57903903:45
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page  https://review.openstack.org/57903904:42
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: DNM: verify selenium test by reproducing the pathname whitelabel bug  https://review.openstack.org/57905204:46
tristanCcorvus: mordred: hopefully the selenium test should catch the bug from 579052 ^04:46
tristanCI switched them back to firefox and make the console.log check for js warning to a noop04:47
*** CrayZee has joined #zuul04:58
*** pcaruana has joined #zuul05:20
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: executor: add executing job INFO log  https://review.openstack.org/57870406:29
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: status: add pool column to nodepool list output  https://review.openstack.org/57904606:32
*** yolanda__ has joined #zuul06:42
*** yolanda__ is now known as yolanda06:43
*** gtema has joined #zuul06:46
gtematobiash: ping07:46
*** jpena|off is now known as jpena07:56
*** jpena has left #zuul07:56
*** jpena has joined #zuul07:57
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: gerrit: use baseurl for change uris lookup  https://review.openstack.org/57908608:02
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: gerrit: use baseurl for change uris lookup  https://review.openstack.org/57908608:02
gtematristianC: do you know any way of consuming environment variables in zuul.conf? I am creating Zuul@Openshift as APB (based on templates shared by Tobias) and want to add/provision sql connection. But I do not want to store pass in configmap. There doesn't seem a way of substituting variables in ConfigMap with Secrets in K8 so far. There is also no way to offload credentials for sqlalchemy.08:04
tobiashgtema: you probably need to create a boot.sh that runs envsubst on the config file08:06
tristanCgtema: as tobiash said, environment variable override is not supported atm08:06
gtematobiash, tristianC: sad. Thanks. boot.sh is not a proper solution, since then configmap update will not be handled correctly08:07
gtemas/handled/detected08:07
tristanCtobiash: could we just add a .format(os.environ) on zuul.conf string access?08:08
tristanCrather .format(**os.environ)08:09
tobiashgtema, tristanC: afaik a zuul.conf change requires a zuul restart08:09
tristanClike that, dburi could be "mysql://{ZUUL_DB_USER}:{ZUUL_DB_PASSWORD}..."08:09
tobiashtristanC: that could be an option, but that still requires the possibility to reload that config during runtime08:10
tobiashmaybe I'm wrong but I thought reconfiguration only re-reads main.yaml08:10
tristanCtobiash: reconfiguration doesn't reload zuul.conf indeed08:10
gtemamaybe, wasn't pointing attention to that08:10
tobiashtristanC, gtema: nodepool does a reload regularly so there it might help08:11
gtemaI think anyway adding env substitution would be helpful also to hide github secrets08:11
gtematobiash, tristianC: since you both are working with Zuul and Openshift, do you have interest in seeing APB of installing zuul on openshift?08:12
gtemawith "full-blown" ;-) s2i08:13
tristanCgtema: environment isn't very secure for handling secrets as it can leaks through /proc/ read access08:13
gtematristianC: storing them in configmap isn't better I suppose08:13
tristanCgtema: sure, a ready to use openshift template for zuul would be handy08:14
gtematristianC: it's not template, it's AnsiblePlaybookBundle, which works like template for provisioning services08:14
gtematristianC: anyway it's not yet ready for public, but will try to publish it in couple of days08:15
gtemas2i's are ready. But the diskimage-builder is a hige challenge (as also tobiash noticed)08:15
gtemacurrent diskimage-builder is not capable of running in container even with basic rpm distros, due to selinux08:16
tobiashgtema: cool08:16
gtemaubuntu is not working out-of-box as well08:16
tobiashgtema: what's the problem with selinux?08:17
tobiashdoesn't privileged solve that?08:17
gtemaruncon -t setfiles_mac_t --08:17
gtemafrom rpm-distro/cleanup.d/99-selinux-fixfiles-restore08:17
gtemain container selinux is disabled, therefore it fails08:17
tobiashgtema: which version of openshift you are using?08:18
gtemaprivileged is solving losetup troubles, but I needed to add additional image flush, since in some cases after empty image is being created and tried to `losetup` it does not exist on fs08:18
gtema3.908:18
tobiashI'm running 3.9 and I thought selinux is supported inside the containers08:19
tobiashI'm running with the overlay2 storage backend08:19
gtemait is "suported". From what I understand it is checked, but the container itself reports it being disabled.08:20
tobiashgtema: ah maybe I'm not facing that issue because I don't have an rpm based image atm08:20
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: config: enable using environ template format for secrets  https://review.openstack.org/57909108:20
gtematobiash: you are using old diskimage-builder from what I see08:20
gtemaand this is present in the newer one08:20
gtemaexactly this step of `runcon`08:20
tobiashoh right, but that was due to failing to install some java package I don't use anymore08:21
tobiashupgrading diskimage builder is on my backlog since some time...08:21
gtematobiash: you might want to wait with that ;-)08:22
gtemaselinux: https://serverfault.com/questions/757606/how-to-enable-selinux-inside-of-a-centos-docker-container/75760908:22
tobiashgtema: thanks for that hint so I should probably try that in the test environment first...08:23
gtemayupp08:24
*** Shrews has joined #zuul08:25
*** Shrews has joined #zuul08:25
*** Shrews has quit IRC08:25
gtematobiash, tristianC: ok. I will further polish APB installation, publish it and present here. If there would be interest I would be happy to move development of it under Openstack from my copmany08:27
gtemathose privileged are however making troubles - you can't really create scc or add scc to sa from APB. So this remains as a manual admin step08:28
tristanCgtema: i'd like to work on making zuul be able to run in un-privileged container, but that needs some more discussions first08:33
gtematristianC: great. Me either08:33
tristanCmy plan is to be able to replace the bubblewrap execution driver by a k8s interface to delegate ansible-playbook execution context to another pod08:40
gtematristianC: this is nice. However we need not to forget, that diskimage-builder is really requiring privileged. And I guess there is also much work to do08:41
tristanCgtema: perhaps the nodepool-builder service could be managed by nodepool-launcher?08:42
gtemahow would this help? launcher itself does not need privileged08:43
gtemait is itself "strage" to create image for VM from container running inside of VM :-|08:43
gtemas/strage/strange08:44
tristanCgtema: e.g. nodepool-launcher launch a builder vm inside each provider and to run disk-image-process inside openstack instead08:46
gtematristian: ah so, very elegant +108:47
*** electrofelix has joined #zuul08:53
gtematobiash: hint - diskimage-builder==2.6.1 does not have problem with selinux (rpm distros). This is last release before the breaking change was introduced. I've opened a bug against diskimage-builder: https://bugs.launchpad.net/diskimage-builder/+bug/177927308:55
openstackLaunchpad bug 1779273 in diskimage-builder "building images in container fails due to selinux fixes" [Undecided,New]08:55
tobiashgtema: thanks08:55
gtemawelcome08:55
*** spsurya has quit IRC09:14
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: executor: add executing job INFO log  https://review.openstack.org/57870409:14
*** jesusaur has quit IRC09:32
*** jesusaur has joined #zuul09:52
*** sshnaidm is now known as sshnaidm|off10:02
*** electrofelix has quit IRC11:02
*** jpena is now known as jpena|lunch11:06
*** gregdek is now known as Guest6296211:19
*** sshnaidm|off has quit IRC11:29
*** nchakrab has joined #zuul11:30
*** bcoca has joined #zuul11:35
*** bcoca has joined #zuul11:35
*** electrofelix has joined #zuul11:40
*** jpena|lunch is now known as jpena12:07
*** hashar has joined #zuul12:41
*** rlandy has joined #zuul12:42
*** Shrews has joined #zuul12:42
*** openstackgerrit has quit IRC12:49
*** bcoca has quit IRC12:50
*** zxiiro is now known as zxiiro-pto13:20
*** hashar has quit IRC13:27
*** snapiri- has joined #zuul13:32
*** elyezer_ has joined #zuul13:32
*** snapiri- has quit IRC13:33
*** snapiri- has joined #zuul13:33
*** CrayZee has quit IRC13:35
*** elyezer has quit IRC13:36
fungitristanC: i've always thought it would be cool if nodepool could handle elastic scaling of zuul executor and merger servers too13:42
fungirather, if zuul+nodepool could do that13:42
mnasermordred: the change is in (i might be late), zuul status page is so clean!14:22
mnaserit's so much easier on my cpu/browser too14:22
mordredmnaser: yay!14:30
*** nchakrab has quit IRC14:42
*** nchakrab has joined #zuul14:43
*** nchakrab has quit IRC14:47
*** openstackgerrit has joined #zuul14:54
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Make websocket streaming more event-driven  https://review.openstack.org/55864614:54
ShrewsTIL that I have no idea how log streaming works today ^^^14:56
ShrewsI also learned that a total VM rebuild at a non-sleeping 4am helps IRC disconnects14:57
mordredShrews: do any of us understand how log streaming works?14:57
Shrewsmordred: none of us will until we recode all of this with C++14:59
corvusi'm really looking forward to porting all the javascript to c++15:00
Shrewstotes15:00
openstackgerritMerged openstack-infra/zuul master: Translate zuulStartStream into typescript  https://review.openstack.org/55861815:05
mordredShrews, corvus: we can compile rust down into webasm ...15:08
*** openstack has quit IRC15:22
*** openstack has joined #zuul15:26
*** ChanServ sets mode: +o openstack15:26
Shrewscorvus: mordred: I think we're gtg with merging https://review.openstack.org/578831 now based on feedback from tristanC15:28
corvus++15:29
*** acozine1 has joined #zuul15:35
*** acozine1 has left #zuul15:36
*** acozine1 has joined #zuul15:38
tobiashmordred: that log streaming reminds me that we should also add a keep alive mechanism to the websocket streaming15:55
mordredtobiash: ++15:55
*** sshnaidm|off has joined #zuul15:55
*** nchakrab has joined #zuul15:56
*** nchakrab has quit IRC16:02
*** tobasco has quit IRC16:06
*** tobasco has joined #zuul16:08
*** weshay has quit IRC16:09
*** Guest19794 has quit IRC16:09
*** mugsie has quit IRC16:09
*** zigo has quit IRC16:09
*** lennyb has quit IRC16:09
clarkbthen compile the C++ to webasm and run it on javscript?16:11
*** weshay has joined #zuul16:14
*** Guest19794 has joined #zuul16:14
*** mugsie has joined #zuul16:14
*** zigo has joined #zuul16:14
*** lennyb has joined #zuul16:14
tobiashmordred: added a question on 55864616:16
*** aspiers[m] has quit IRC16:18
*** austinsun[m] has quit IRC16:18
openstackgerritMerged openstack-infra/nodepool master: Fix for pools with different labels  https://review.openstack.org/57883116:21
mordredtobiash: good question. I have responded - and fixed16:22
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Make websocket streaming more event-driven  https://review.openstack.org/55864616:22
tobiashyay I could help with js stuff :)16:23
*** mattclay has quit IRC16:25
*** mattclay has joined #zuul16:26
*** electrofelix has quit IRC16:29
*** nchakrab has joined #zuul16:36
*** nchakrab has quit IRC16:37
*** pcaruana has quit IRC16:40
*** rlandy is now known as rlandy|brb16:43
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Enable heartbeat on websocket streaming  https://review.openstack.org/57922216:43
tobiashmordred: websocket keepalive might be as easy as this ^16:44
openstackgerritMerged openstack-infra/nodepool master: zk: retry initial zookeeper connection attempts  https://review.openstack.org/57604716:50
*** jpena is now known as jpena|off17:00
*** gtema has quit IRC17:01
*** austinsun[m] has joined #zuul17:01
*** openstack has quit IRC17:11
*** openstack has joined #zuul17:15
*** ChanServ sets mode: +o openstack17:15
*** rlandy|brb is now known as rlandy17:16
openstackgerritMerged openstack-infra/zuul master: Shift log streaming code into StreamComponent  https://review.openstack.org/55861917:17
*** aspiers[m] has joined #zuul17:19
corvustobiash: did you see https://review.openstack.org/579033 ?17:25
tobiashcorvus: yes, cool, but only had a quick look so far17:27
corvustobiash: no rush, it's failing all the jobs and i haven't written tests yet.  but i wondered if you think it'll address the problem sufficiently.17:28
tobiashTIL about git log -L17:36
tobiashI think that could work :)17:36
mordredcorvus: ooh neat17:53
corvusyeah, it looks like if you give it exactly one line, you'll end up with exactly a one line diff for each commit17:54
*** kmalloc is now known as vmalloc18:06
*** vmalloc is now known as kmalloc18:06
*** elyezer_ is now known as elyezer18:09
openstackgerritJeremy Stanley proposed openstack-infra/zuul master: Change "core developer" references  https://review.openstack.org/57924118:26
fungithat's neat. more convenient than pickaxe or iterating over blame18:32
openstackgerritMerged openstack-infra/zuul master: Enable heartbeat on websocket streaming  https://review.openstack.org/57922218:33
fungicorvus: is the use case for that series that a linter job could leave inline comments on each thing it's complaining about?18:35
mordredfungi: s/linter/any/ really - but yeah18:35
fungiwell, i was picking that as a likely example18:35
mordredfungi: I imagine linter jobs would be ... yeah - first likely example18:35
fungii should have said analyzer18:36
mordredfungi: a something18:36
mordredfungi: a something and do a something and leave something for somebody18:36
fungias a more general class of thing with that sort of line-by-line reporting behavior18:36
fungithough i suppose if a job caused a python exception to be raised, the exception and traceback could be inserted as a comment at the line where it was raised18:37
fungior similar for other languages which support line numbers in their debugging18:37
fungiand presumably zuul's configuration validator could be one of the first users of that feature as well i guess, reporting inline on proposed configuration changes where the problem is suspected?18:38
*** elyezer has quit IRC18:39
mordredfungi: yah - I think reporting on those will be cool18:40
fungiafter all, zuul should be a first-class user of its own features ;)18:40
mordred++18:42
*** elyezer has joined #zuul18:49
corvusyes.  the use case is definitely a smurfer that leaves smurfs for smurfs.19:04
fungibest use case EVAR19:08
*** yolanda_ has joined #zuul19:19
*** yolanda__ has joined #zuul19:22
*** yolanda has quit IRC19:23
*** yolanda_ has quit IRC19:24
*** acozine1 has quit IRC19:59
*** SpamapS has quit IRC20:22
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul-jobs master: Ensure iptables-services is installed when persisting firewall config  https://review.openstack.org/57928320:47
dmsimardpabelanger: ^ for some reason that package isn't installed in our RDO image20:47
dmsimardbreaks multi node firewall setup20:48
pabelangerdmsimard: -1, but ya20:49
pabelangerwe can fix it in DIB too20:49
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul-jobs master: Ensure iptables-services is installed when persisting firewall config  https://review.openstack.org/57928320:49
dmsimardfixed20:49
clarkbfrom openstack's perspective you really want to have a firewall configured and ready when the host boots20:50
dmsimardpabelanger: yeah, I was going to send a patch to the dib element but figured I'd fix it here too20:50
clarkbto protect the jobs from themselves20:50
dmsimardclarkb: iptables is up and running, there's just no systemd unit file so doing a "systemctl restart iptables" or "systemctl start iptables" doesn't work20:50
dmsimard¯\_(ツ)_/¯20:50
clarkbdmsimard: how are the rules loaded? the kernel module is probably there just no rules20:50
dmsimardI'm not sure if this image has any rules by default20:52
clarkbya I'm suggesting that you want those :)20:53
clarkbif only to protect jobs from affecting each other on shared network segments20:53
dmsimardyeah good point20:53
pabelangerwhat do we do for single node jobs today20:54
clarkbopenstack bakes a firewall ruleset into the image20:54
pabelangerright, forgot that20:54
dmsimardit's in one of the elements20:54
pabelangerI'd have to check what default SF.io elements do20:55
pabelangerbecause that is where that node (rdo-centos-7) is coming from20:55
clarkb(Updating the playbook seems fine too, we shouldn't assume firewalls will allow the traffic we need for the overlay so we should explicitly allow it ourselves which involves installing a firewll management toolchain)21:01
clarkbbut more generally I would encourage people running CI systems to have test node firewalls to keep jobs from affecting each other as much as possible21:02
pabelangeragree, I think we could create something for base to ensure a basic firewall for all jobs21:14
pabelangerrather then baking in to images21:14
pabelangerbut that is for another day21:14
clarkboh I prefer baking into images because then it is there on boot21:15
clarkbwaiting for a job to run is too late21:15
fungicompletely agree21:29
corvusi think we've got the all clear from clarkb, Shrews, and tristanC for a nodepool release, so i'll tag fe80343 (current master)22:11
corvusas 3.0.222:11
corvusthat look right? (clarkb)22:11
* clarkb double checks the version launchers ar erunning22:11
clarkbfe80343 is what I restarted on lgtm22:12
clarkboh quick question22:12
corvusthe only wonky thing about this upgrade is that it requires a full restart.  but no other action.  i could probably go either way on 3.1.0 vs 3.0.222:12
clarkbwas the zk schema change on a release already?22:12
corvusclarkb: no, that's ^22:13
clarkbya I think because it comes with a zk schema change and requires a full restart 3.1.0 might be appropriate22:13
corvushttps://zuul-ci.org/docs/nodepool/releasenotes.html#upgrade-notes22:13
corvusokay.  i'm convinced.  :)22:13
corvusany objections to 3.1.0 ?22:13
corvushttp://paste.openstack.org/show/724620/22:15
clarkblgtm22:15
corvusnodepool 3.1.0 pushed22:17
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Publish docs on release  https://review.openstack.org/57930022:42
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add a role to return file comments  https://review.openstack.org/57903323:02
*** eikke has quit IRC23:11
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Handle node no longer in pool error  https://review.openstack.org/57930923:14
tristanCcorvus: that's great thanks.23:17
corvustristanC: thanks for your help :)23:19
*** eikke has joined #zuul23:36
*** rlandy has quit IRC23:41
*** SpamapS has joined #zuul23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!