Skip to content

Commit 4e4ad6e

Browse files
dinatale2morrone
dinatale2
authored and
morrone
committedMar 31, 2016
First public prototype
This is the first public prototype of the OpenSFS Buildbot configuration for building Lustre.
0 parents  commit 4e4ad6e

18 files changed

+2413
-0
lines changed
 

‎.gitignore

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
*~
2+
master/gitpoller*
3+
master/*_BUILD_
4+
master/*_TEST_
5+
master/http.*
6+
master/config.py
7+
master/runtests
8+
master/state.sqlite
9+
master/twistd.log*
10+
master/twistd.pid
11+
master/password.py
12+
master/password.pyc
13+
master/public_html/*.rpm

‎README.md

+194
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
# The Lustre Buildbot Configuration
2+
3+
Welcome, this is the buildbot configuration and infrastructure used by the
4+
Lustre Buildbot at http://build.lustre.org. It's used to automate the process
5+
of testing patch sets submitted to Lustre project on
6+
[Gerrit] (http://review.whamcloud.com). If you would like to
7+
contribute to improving our testing infrastructure please open a pull request
8+
against this Github repository.
9+
10+
## Build and Test Strategy
11+
12+
### Patch Sets
13+
14+
The Lustre project relies on Gerrit to track proposed changes. Changes
15+
submitted to Gerrit are called patch sets. Each patch set submitted will
16+
be automatically tested by the buildbot. As you fix the code and push
17+
new changes to Gerrit, each patch set is queued to be tested.
18+
Currently, all patch sets submitted are built using SPL and ZFS version
19+
0.6.5.4.
20+
21+
### Builder Types
22+
23+
When a new patch set is submitted, it is queued up for testing on all
24+
of the available builders. There is currently only one type of builder:
25+
26+
* BUILD: These builders are responsible for verifying that a change
27+
doesn't break the build on a given platform. Every patch set
28+
is built and reported on individually. This helps guarantee that
29+
developers never accidentally break the build.
30+
31+
To maximize coverage builders have been created for most major Linux
32+
distributions. This allows us to catch distribution specific issues and to
33+
verify the build on a wide range of commonly deployed kernels.
34+
35+
Additional builders are maintained to test alternate architectures. If
36+
you're interested in setting up a builder for your distribution or
37+
architecture see the 'Adding a Builder' section below.
38+
39+
No elevated permissions are required for this type of builder. However,
40+
it is assumed that all required development tools and headers are already
41+
installed on the system.
42+
43+
### Build Steps and the `runurl` Utility
44+
45+
The Lustre Buildbot makes extensive use of the `runurl`
46+
utility. This small script takes the URL of a script to execute as its first
47+
argument followed by all arguments to pass to the script. This allows us to
48+
configure a build step which references a trusted URL with the desired script.
49+
This means the logic for a particular build step can be separated from the
50+
`master.cfg` which has some advantages:
51+
52+
* Minimizes the disruption caused by restarting the buildbot to make
53+
changes live. This is only required when modifying the `master.cfg`.
54+
For example, when adding/removing a builder or adding a test suite.
55+
56+
* Build and tests scripts can be run independently making it easy for
57+
developers to locally test proposed changes before submitting them.
58+
59+
* Allows for per-builder customization via the environment.
60+
Each script can optionally source the following files to influence its
61+
behavior.
62+
63+
* `/etc/buildslave` - This file is dynamically generated by the
64+
`bb-bootstrap.sh` script and is run at boot time by the ec2 user data
65+
facility. It includes all the information required to configure and
66+
start a latent buildslave. Most importabtly for scripts this includes
67+
the `BB_NAME` variable which is set to the build slave name.
68+
69+
* Provides a consistent way to trap and handle signals from the buildbot
70+
master. This is particularly helpful when attempting to collect debug
71+
information prior to terminating an instance.
72+
73+
## Configuring the Master
74+
75+
### Important Files
76+
77+
The Lustre Buildbot configuration is broken out in a few different files.
78+
Below is a list of important files and a brief description of what is
79+
contained within them:
80+
81+
* `master/master.cfg` - Core configuration file for the Lustre Buildbot.
82+
83+
* `master/password.py.sample` - Sample credentials file used to create
84+
a `master/password.py` file. This file contains build slave passwords,
85+
various user passwords, and EC2 credentials.
86+
87+
* `master/lustrebuildslave.py` - Contains custom BuilderConfig and
88+
EC2LatentBuildSlave classes used by the Lustre Buildbot. If new types
89+
of build slaves need to be created, please define them in this file.
90+
91+
* `master/lustrefactory.py` - Contains core and supporting functions that
92+
construct BuildFactories for Lustre Buildbot builders. If new types of
93+
build factories need to be created, please define them in this file.
94+
95+
### Credentials
96+
97+
The `master/passwords.py` file contains the credentials required for the
98+
buildbot to interact with ec2. It stores static passwords for
99+
non-ec2 build slaves, the web interface and `buildbot try`. See the
100+
`master/passwords.py.sample` file for a complete description.
101+
102+
### Adding a Builder
103+
104+
The process for adding a new builder varies slightly depending
105+
if it's a standard or latent builder. In all cases the
106+
process begins by adding the new builder and slave to the `master.cfg` file.
107+
One important thing to be aware of is that each builder can potentially have
108+
multiple build slaves.
109+
110+
The first step is to determine what kind of slave your setting up. Both
111+
standard and latent build slaves are supported.
112+
113+
Once you've added your slaves, be sure to add them to the `all_slaves` list
114+
at the bottom of the BUILDSLAVES section in `master.cfg`. Classes have been
115+
provided with suitable default values for slaves. Below are examples on
116+
how to add various types of slaves.
117+
118+
* Linux based m3.large EC2 slave
119+
```
120+
newslaves = [
121+
M3LargeEC2SpotSlave(
122+
name="slavename",
123+
master=bb_master,
124+
url=bb_url,
125+
ami="ami-xxxxxxxx"
126+
),
127+
....
128+
]
129+
```
130+
131+
* Suse based m3.large EC2 slave
132+
```
133+
newslaves = [
134+
M3LargeEC2SuseSpotSlave(
135+
name="slavename",
136+
master=bb_master,
137+
url=bb_url,
138+
ami="ami-xxxxxxxx"
139+
),
140+
....
141+
]
142+
```
143+
144+
* Persistent build slave
145+
```
146+
newslaves = [
147+
BuildSlave(
148+
name="slavename",
149+
password="password"
150+
),
151+
....
152+
]
153+
```
154+
155+
EC2 backed slaves do not require `master` or `url` to be provided.
156+
`master` and/or `url` are only necessary if you are standing up your own Lustre
157+
Buildbot instance.
158+
159+
Now that your build slaves have been added, a builder needs to be created which
160+
owns them. Jump down to the the BUILDERS section and add a
161+
`LustreBuilderConfig` entry to the appropriate list. Each builder must
162+
have a unique `name`. Set the `factory` to `build_factory` for the builder.
163+
Then set `properties` and `tags` options as appropriate.
164+
165+
Finally, you must restart the build master to make it aware of your new
166+
builder. It's generally a good idea to run `buildbot checkconfig` to verify
167+
your changes. Then wait until the buildbot is idle before running
168+
`buildbot restart` in order to avoid killing running builds.
169+
170+
### Updating an EC2 Build Slave to Use a Different AMI
171+
172+
New AMIs for the latest release of a distribution are frequently published for
173+
ec2. These updated AMIs can be used by replacing the current AMI identifier
174+
used by the build slave with the new AMI identifier. All build slaves are
175+
listed in the BUILDSLAVES section of the `master.cfg` file. Remember the
176+
buildbot will need to be restarted to pick up the change.
177+
178+
### Running a Private Master
179+
180+
The official Lustre Buildbot can be accessed by everyone at
181+
http://build.lustre.org/ and it leverages Gerrit's stream-events functionality
182+
to queue changes to test. Developers are encouraged to use this infrastructure
183+
when working on a change. However, this code can be used as a basis for
184+
building a private build and test environment. This may be useful when working
185+
on extending the testing infrastructure itself.
186+
187+
Generally speaking to do this you will need to create a `password.py` file
188+
with your credentials, then list your builders in the `master.cfg` file, and
189+
finally start the builder master. It's assumed you're already familiar with
190+
Amazon ec2 instances and their terminology.
191+
192+
## Licensing
193+
194+
See the [LICENSE](LICENSE) file for license rights and limitations.

‎master/buildbot.tac

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
import os
2+
3+
from twisted.application import service
4+
from buildbot.master import BuildMaster
5+
6+
basedir = '/home/buildbot/lustre-buildbot/master'
7+
rotateLength = 10000000
8+
maxRotatedFiles = 10
9+
configfile = 'master.cfg'
10+
11+
# Default umask for server
12+
umask = None
13+
14+
# if this is a relocatable tac file, get the directory containing the TAC
15+
if basedir == '.':
16+
import os.path
17+
basedir = os.path.abspath(os.path.dirname(__file__))
18+
19+
# note: this line is matched against to check that this is a buildmaster
20+
# directory; do not edit it.
21+
application = service.Application('buildmaster')
22+
from twisted.python.logfile import LogFile
23+
from twisted.python.log import ILogObserver, FileLogObserver
24+
logfile = LogFile.fromFullPath(os.path.join(basedir, "twistd.log"), rotateLength=rotateLength,
25+
maxRotatedFiles=maxRotatedFiles)
26+
application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
27+
28+
m = BuildMaster(basedir, configfile, umask)
29+
m.setServiceParent(application)
30+
m.log_rotation.rotateLength = rotateLength
31+
m.log_rotation.maxRotatedFiles = maxRotatedFiles

‎master/lustrebuildslave.py

+127
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# -*- python -*-
2+
# ex: set syntax=python:
3+
4+
import string
5+
import random
6+
import re
7+
from password import *
8+
from buildbot.plugins import util
9+
from buildbot.buildslave import BuildSlave
10+
from buildbot.buildslave.ec2 import EC2LatentBuildSlave
11+
12+
### BUILDER CLASSES
13+
class LustreBuilderConfig(util.BuilderConfig):
14+
@staticmethod
15+
def nextSlave(builder, slaves):
16+
availableSlave = None
17+
18+
for slave in slaves:
19+
# if we found an idle slave, immediate use this one
20+
if slave.isIdle():
21+
return slave
22+
23+
# hold onto the first slave thats not spun up but free
24+
if availableSlave is None and slave.isAvailable():
25+
availableSlave = slave
26+
27+
# we got here because there was no idle slave
28+
if availableSlave is not None:
29+
return availableSlave
30+
31+
# randomly choose among all our busy slaves
32+
return (random.choice(slaves) if slaves else None)
33+
34+
def __init__(self, mergeRequests=False, nextSlave=None, **kwargs):
35+
if nextSlave is None:
36+
nextSlave = LustreBuilderConfig.nextSlave
37+
38+
util.BuilderConfig.__init__(self, nextSlave=nextSlave,
39+
mergeRequests=mergeRequests, **kwargs)
40+
41+
### BUILD SLAVE CLASSES
42+
class LustreEC2Slave(EC2LatentBuildSlave):
43+
default_user_data = """#!/bin/bash
44+
set -e
45+
set -x
46+
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
47+
48+
# Ensure wget is available for runurl
49+
if ! hash wget 2>/dev/null; then
50+
if hash apt-get 2>/dev/null; then
51+
apt-get --quiet --yes install wget
52+
elif hash dnf 2>/dev/null; then
53+
echo "keepcache=true" >>/etc/dnf/dnf.conf
54+
echo "deltarpm=true" >>/etc/dnf/dnf.conf
55+
echo "fastestmirror=true" >>/etc/dnf/dnf.conf
56+
dnf clean all
57+
dnf --quiet -y install wget
58+
elif hash yum 2>/dev/null; then
59+
yum --quiet -y install wget
60+
else
61+
echo "Unknown package managed cannot install wget"
62+
fi
63+
fi
64+
65+
# Set our bb variables
66+
export BB_MASTER='%s'
67+
export BB_NAME='%s'
68+
export BB_PASSWORD='%s'
69+
export BB_URL='%s'
70+
71+
if [ -z "$BB_URL" ]; then
72+
export BB_URL="https://raw.githubusercontent.com/opensfs/lustre-buildbot-config/master/scripts/"
73+
fi
74+
75+
# Get the runurl utility.
76+
wget -qO/usr/bin/runurl $BB_URL/runurl
77+
chmod 755 /usr/bin/runurl
78+
79+
# Run the bootstrap script
80+
runurl $BB_URL/bb-bootstrap.sh"""
81+
82+
@staticmethod
83+
def pass_generator(size=24, chars=string.ascii_uppercase + string.digits):
84+
return ''.join(random.choice(chars) for _ in range(size))
85+
86+
def __init__(self, name, password=None, master='', url='', instance_type="m3.large",
87+
identifier=ec2_default_access, secret_identifier=ec2_default_secret,
88+
keypair_name=ec2_default_keypair_name, security_name='LustreBuilder',
89+
user_data=None, region="us-west-2", placement="a", max_builds=1,
90+
build_wait_timeout=60 * 30, spot_instance=True, max_spot_price=.08,
91+
price_multiplier=1.25, **kwargs):
92+
93+
self.name = name
94+
95+
tags = kwargs.get('tags')
96+
if not tags or tags is None:
97+
tags={
98+
"ENV" : "DEV",
99+
"Name" : "LustreBuilder",
100+
"ORG" : "COMP",
101+
"OWNER" : "Buildbot Admin <buildbot-admin@lustre.org>",
102+
"PLATFORM" : name,
103+
"PROJECT" : "Lustre",
104+
}
105+
106+
if password is None:
107+
password = LustreEC2Slave.pass_generator()
108+
109+
if user_data is None:
110+
user_data = LustreEC2Slave.default_user_data % (master, name, password, url)
111+
112+
EC2LatentBuildSlave.__init__(
113+
self, name=name, password=password, instance_type=instance_type,
114+
identifier=identifier, secret_identifier=secret_identifier, region=region,
115+
user_data=user_data, keypair_name=keypair_name, security_name=security_name,
116+
max_builds=max_builds, spot_instance=spot_instance, tags=tags,
117+
max_spot_price=max_spot_price, placement=placement,
118+
price_multiplier=price_multiplier, build_wait_timeout=build_wait_timeout,
119+
**kwargs)
120+
121+
class LustreEC2SuseSlave(LustreEC2Slave):
122+
def __init__(self, name, **kwargs):
123+
LustreEC2Slave.__init__(self, name, max_spot_price="0.16",
124+
instance_type="m3.large",
125+
product_description="SUSE Linux (Amazon VPC)",
126+
**kwargs)
127+

‎master/lustrefactory.py

+120
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# -*- python -*-
2+
# ex: set syntax=python:
3+
4+
from buildbot.plugins import util
5+
from buildbot.steps.source.gerrit import Gerrit
6+
from buildbot.steps.source.git import Git
7+
from buildbot.steps.shell import ShellCommand
8+
from buildbot.steps.trigger import Trigger
9+
from buildbot.status.results import SUCCESS
10+
from buildbot.status.results import FAILURE
11+
from buildbot.status.results import SKIPPED
12+
from buildbot.status.results import WARNINGS
13+
14+
def do_step_build(step, name):
15+
props = step.build.getProperties()
16+
if props.hasProperty(name) and props[name] == "yes":
17+
return True
18+
else:
19+
return False
20+
21+
def do_step_zfs(step):
22+
return do_step_build(step, 'buildzfs')
23+
24+
@util.renderer
25+
def dependencyCommand(props):
26+
args = ["runurl"]
27+
bb_url = props.getProperty('bburl')
28+
args.extend([bb_url + "bb-dependencies.sh"])
29+
return args
30+
31+
@util.renderer
32+
def buildzfsCommand(props):
33+
args = ["runurl"]
34+
bb_url = props.getProperty('bburl')
35+
args.extend([bb_url + "bb-build-zfs-pkg.sh"])
36+
37+
spltag = props.getProperty('spltag')
38+
if spltag:
39+
args.extend(["-s", spltag])
40+
41+
zfstag = props.getProperty('zfstag')
42+
if zfstag:
43+
args.extend(["-z", zfstag])
44+
45+
return args
46+
47+
@util.renderer
48+
def buildCommand(props):
49+
args = ["runurl"]
50+
bb_url = props.getProperty('bburl')
51+
52+
style = props.getProperty('buildstyle')
53+
if style == "srpm":
54+
args.extend([bb_url + "bb-build-lustre-srpm.sh"])
55+
elif style == "deb":
56+
args.extend([bb_url + "bb-build-lustre-pkg.sh", "-m", "debs"])
57+
elif style == "rpm":
58+
args.extend([bb_url + "bb-build-lustre-pkg.sh", "-m", "rpms"])
59+
else:
60+
args.extend([bb_url + "bb-build-lustre-pkg.sh"])
61+
62+
with_zfs = props.getProperty('withzfs')
63+
if with_zfs and with_zfs == "yes":
64+
args.extend(["-z"])
65+
66+
with_ldiskfs = props.getProperty('withldiskfs')
67+
if with_ldiskfs and with_ldiskfs == "yes":
68+
args.extend(["-l"])
69+
70+
return args
71+
72+
def getBuildFactory(gerrit_repo, **kwargs):
73+
""" Generates a build factory for a standard lustre builder.
74+
Args:
75+
gerrit_repo (string): Gerrit repo url
76+
Returns:
77+
BuildFactory: Build factory with steps for a standard lustre builder.
78+
"""
79+
bf = util.BuildFactory()
80+
81+
# update dependencies
82+
bf.addStep(ShellCommand(
83+
command=dependencyCommand,
84+
decodeRC={0 : SUCCESS, 1 : FAILURE, 2 : WARNINGS, 3 : SKIPPED },
85+
haltOnFailure=True, logEnviron=False,
86+
description=["installing dependencies"],
87+
descriptionDone=["installed dependencies"]))
88+
89+
# build spl and zfs if necessary
90+
bf.addStep(ShellCommand(
91+
command=buildzfsCommand,
92+
decodeRC={0 : SUCCESS, 1 : FAILURE, 2 : WARNINGS, 3 : SKIPPED },
93+
haltOnFailure=True, logEnviron=False,
94+
doStepIf=do_step_zfs,
95+
hideStepIf=lambda results, s: results==SKIPPED,
96+
description=["building spl and zfs"],
97+
descriptionDone=["built spl and zfs"]))
98+
99+
# Pull the patch from Gerrit
100+
bf.addStep(Gerrit(
101+
repourl=gerrit_repo, workdir="build/lustre",
102+
mode="full", method="clobber", retry=[60,60], timeout=3600,
103+
logEnviron=False, getDescription=True, haltOnFailure=True,
104+
description=["cloning"], descriptionDone=["cloned"]))
105+
106+
# Build Lustre
107+
bf.addStep(ShellCommand(
108+
workdir="build/lustre",
109+
command=buildCommand,
110+
haltOnFailure=True, logEnviron=False,
111+
hideStepIf=lambda results, s: results==SKIPPED,
112+
lazylogfiles=True,
113+
decodeRC={0 : SUCCESS, 1 : FAILURE, 2 : WARNINGS, 3 : SKIPPED },
114+
description=["building lustre"], descriptionDone=["built lustre"]))
115+
116+
# TODO: upload build products
117+
# Primary idea here is to upload the build products to buildbot's public html folder
118+
# what should go in there so far: tarball, srpm (maybe?), and build products (for the testers to download)
119+
120+
return bf

‎master/master.cfg

+335
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,335 @@
1+
# -*- python -*-
2+
# ex: set syntax=python:
3+
4+
from buildbot.plugins import *
5+
from password import *
6+
from twisted.python import log
7+
from lustrebuildslave import *
8+
from lustrefactory import *
9+
from buildbot.status import html
10+
from buildbot.status.web import authz, auth
11+
from buildbot.plugins import status, util
12+
from buildbot.schedulers.triggerable import Triggerable
13+
from buildbot.schedulers.trysched import Try_Userpass
14+
15+
# This is the dictionary that the buildmaster pays attention to. We also use
16+
# a shorter alias to save typing.
17+
c = BuildmasterConfig = {}
18+
19+
####### PROJECT IDENTITY
20+
bb_master_url="build.lustre.org"
21+
bb_master_port="9989"
22+
23+
# Gerrit Configuration
24+
gerrit_url = "review.whamcloud.com"
25+
gerrit_user = "LustreBuildbot"
26+
gerrit_port = 29418
27+
gerrit_project = "fs/lustre-release"
28+
gerrit_branch = "master"
29+
gerrit_id_file = None
30+
gerrit_repo_http = "http://%s/%s.git" % (gerrit_url, gerrit_project)
31+
gerrit_repo = "ssh://%s@%s:%d/%s" % (gerrit_user, gerrit_url, gerrit_port,
32+
gerrit_project)
33+
34+
bb_master = "%s:%s" % (bb_master_url, bb_master_port)
35+
bb_url = "https://raw.githubusercontent.com/opensfs/lustre-buildbot-config/master/scripts/"
36+
bb_slave_port = 9989
37+
bb_web_port = 8010
38+
bb_try_port = 8033
39+
40+
# the 'title' string will appear at the top of this buildbot installation's
41+
# home pages (linked to the 'titleURL').
42+
43+
c['title'] = "Lustre"
44+
c['titleURL'] = "http://" + gerrit_url
45+
46+
# the 'buildbotURL' string should point to the location where the buildbot's
47+
# internal web server is visible. This typically uses the port number set in
48+
# the 'www' entry below, but with an externally-visible host name which the
49+
# buildbot cannot figure out without some help.
50+
51+
c['buildbotURL'] = "http://%s:%s" % (bb_master_url, bb_web_port)
52+
53+
####### FACTORIES
54+
build_factory = getBuildFactory(gerrit_repo_http)
55+
56+
####### BUILDER PROPERTIES
57+
58+
# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
59+
# what steps, and which slaves can execute them. Note that any particular build will
60+
# only take place on one slave.
61+
62+
def merge_dicts(*args):
63+
z = {}
64+
for item in args:
65+
z.update(item)
66+
67+
return z
68+
69+
### Builder properties are broken out into 3 groups. The groups are described below.
70+
### A builders properties must be a combination of 1 property set from each group.
71+
72+
# Global properties are required properties. These properties control the source
73+
# of scripts to execute and which spl and zfs tags to boot strap with.
74+
global_props = {
75+
"bburl" : bb_url,
76+
"spltag" : "spl-0.6.5.4",
77+
"zfstag" : "zfs-0.6.5.4",
78+
}
79+
80+
# This group of properties controls which features to include when compiling lustre.
81+
# Currently, the only options supported are whether or not to build with zfs
82+
# and/or ldiskfs.
83+
with_zfs = {
84+
"withzfs" : "yes",
85+
"withldiskfs" : "no",
86+
}
87+
88+
with_ldiskfs = {
89+
"withzfs" : "no",
90+
"withldiskfs" : "yes",
91+
}
92+
93+
with_zfs_ldiskfs = {
94+
"withzfs" : "yes",
95+
"withldiskfs" : "yes",
96+
}
97+
98+
# This group of properties controls how lustre is built and whether or not zfs must
99+
# be built for a builder. The four types of builds are as follows:
100+
# - srpm: run an autogen, configure, make srpm, and run rpmbuild w/ the resulting srpm
101+
# - simple: a basic autogen, configure, and make
102+
# - deb: autogen, configure, make debs
103+
# - rpm: autogen, configure, make rpms
104+
builder_default_props = {
105+
"buildzfs" : "yes",
106+
"buildstyle" : "srpm",
107+
}
108+
109+
builder_simple_props = {
110+
"buildzfs" : "yes",
111+
"buildstyle" : "simple",
112+
}
113+
114+
# Builder properties are a combination of exactly one set of properties from each
115+
# of the above groups.
116+
debiansys_props = merge_dicts(global_props, with_zfs, builder_simple_props)
117+
susesys_props = merge_dicts(global_props, with_zfs, builder_default_props)
118+
default_props = merge_dicts(global_props, with_zfs_ldiskfs, builder_default_props)
119+
120+
#### BUILDSLAVES
121+
numSlaves = 3 # number of slaves per builder
122+
123+
CentOS_6_7_slaves = [
124+
LustreEC2Slave(
125+
name="CentOS-6.7-x86_64-buildslave%s" % (str(i+1)),
126+
ami="ami-4d5d577d"
127+
) for i in range(0, numSlaves)
128+
]
129+
130+
CentOS_7_2_slaves = [
131+
LustreEC2Slave(
132+
name="CentOS-7.2-x86_64-buildslave%s" % (str(i+1)),
133+
ami="ami-666a7507"
134+
) for i in range(0, numSlaves)
135+
]
136+
137+
SLES_12_1_slaves = [
138+
LustreEC2SuseSlave(
139+
name="SUSE-12.1-x86_64-buildslave%s" % (str(i+1)),
140+
ami="ami-16f61b76"
141+
) for i in range(0, numSlaves)
142+
]
143+
144+
Ubuntu_12_04_slaves = [
145+
LustreEC2Slave(
146+
name="Ubuntu-12.04-x86_64-buildslave%s" % (str(i+1)),
147+
ami="ami-155a6125"
148+
) for i in range(0, numSlaves)
149+
]
150+
151+
Ubuntu_14_04_slaves = [
152+
LustreEC2Slave(
153+
name="Ubuntu-14.04-x86_64-buildslave%s" % (str(i+1)),
154+
ami="ami-5189a661"
155+
) for i in range(0, numSlaves)
156+
]
157+
158+
all_slaves = CentOS_6_7_slaves + CentOS_7_2_slaves + SLES_12_1_slaves + Ubuntu_12_04_slaves + Ubuntu_14_04_slaves
159+
160+
### BUILDERS
161+
builders = [
162+
LustreBuilderConfig(
163+
name="CentOS 6.7 x86_64 (BUILD)",
164+
factory=build_factory,
165+
slavenames=[slave.name for slave in CentOS_6_7_slaves],
166+
tags=["Build"],
167+
properties=default_props,
168+
),
169+
LustreBuilderConfig(
170+
name="CentOS 7.2 x86_64 (BUILD)",
171+
factory=build_factory,
172+
slavenames=[slave.name for slave in CentOS_7_2_slaves],
173+
tags=["Build"],
174+
properties=default_props,
175+
),
176+
LustreBuilderConfig(
177+
name="SLES 12.1 x86_64 (BUILD)",
178+
factory=build_factory,
179+
slavenames=[slave.name for slave in SLES_12_1_slaves],
180+
tags=["Build"],
181+
properties=susesys_props,
182+
),
183+
LustreBuilderConfig(
184+
name="Ubuntu 12.04 x86_64 (BUILD)",
185+
factory=build_factory,
186+
slavenames=[slave.name for slave in Ubuntu_12_04_slaves],
187+
tags=["Build"],
188+
properties=debiansys_props,
189+
),
190+
LustreBuilderConfig(
191+
name="Ubuntu 14.04 x86_64 (BUILD)",
192+
factory=build_factory,
193+
slavenames=[slave.name for slave in Ubuntu_14_04_slaves],
194+
tags=["Build"],
195+
properties=debiansys_props,
196+
),
197+
]
198+
199+
test_builders = []
200+
201+
all_builders = builders + test_builders
202+
203+
c['builders'] = all_builders
204+
205+
# The 'slaves' list defines the set of recognized buildslaves. Each element is
206+
# a BuildSlave object, specifying a unique slave name and password. The same
207+
# slave name and password must be configured on the slave.
208+
c['slaves'] = all_slaves
209+
210+
def prioritizeBuilders(buildmaster, builders):
211+
"""
212+
Called by the buildmaster to prioritize the builders. Returns a sorted
213+
array of builders designed to improve ec2 utilization. Builders with
214+
substantiated, idle slaves are given priority. Followed by builders with no
215+
substantiated slaves. The lowest priority is a builder that is busy.
216+
This helps keep all buildslaves busy while new latent buildslaves are
217+
bootstrapped, a process which can take several minutes.
218+
"""
219+
220+
idle_builders = []
221+
busy_builders = []
222+
avail_builders = []
223+
224+
for b in builders:
225+
idle = False
226+
busy = False
227+
for s in b.slaves:
228+
if s.isIdle():
229+
idle = True
230+
break
231+
232+
if s.isBusy():
233+
busy = True
234+
235+
if idle is True:
236+
if re.search('BUILD', b.name):
237+
idle_builders.insert(0, b)
238+
else:
239+
idle_builders.append(b)
240+
elif busy is True:
241+
if re.search('BUILD', b.name):
242+
busy_builders.insert(0, b)
243+
else:
244+
busy_builders.append(b)
245+
else:
246+
if re.search('BUILD', b.name):
247+
avail_builders.insert(0, b)
248+
else:
249+
avail_builders.append(b)
250+
251+
sorted_builders = idle_builders + avail_builders + busy_builders
252+
253+
log.msg("prioritized %i builder(s): %s" % (len(sorted_builders),
254+
[b.name for b in sorted_builders]))
255+
256+
return sorted_builders
257+
258+
c['prioritizeBuilders'] = prioritizeBuilders
259+
260+
# 'protocols' contains information about protocols which master will use for
261+
# communicating with slaves. You must define at least 'port' option that slaves
262+
# could connect to your master with this protocol.
263+
# 'port' must match the value configured into the buildslaves (with their
264+
# --master option)
265+
c['protocols'] = {'pb': {'port': bb_slave_port}}
266+
267+
####### CHANGESOURCES
268+
269+
# the 'change_source' setting tells the buildmaster how it should find out
270+
# about source code changes. Here we point to the buildbot clone of pyflakes.
271+
272+
# Run a build for each patch set created
273+
c['change_source'] = [
274+
changes.GerritChangeSource(
275+
gerritserver=gerrit_url,
276+
gerritport=gerrit_port,
277+
project=gerrit_project,
278+
branch=gerrit_branch,
279+
username=gerrit_user,
280+
identity_file=gerrit_id_file,
281+
handled_events=["patchset-created"]),
282+
]
283+
284+
####### SCHEDULERS
285+
286+
# Configure the Schedulers, which decide how to react to incoming changes. In this
287+
# case, just kick off a 'runtests' build
288+
289+
c['schedulers'] = [
290+
# Run our build test for each patch submitted to Gerrit
291+
schedulers.AnyBranchScheduler(
292+
name="master-patchset",
293+
change_filter=util.GerritChangeFilter(
294+
project=gerrit_project,
295+
branch_re=(gerrit_branch + "/*"),
296+
eventtype="patchset-created"),
297+
builderNames=[builder.name for builder in builders]),
298+
Try_Userpass(
299+
name="try-scheduler",
300+
port=bb_try_port,
301+
builderNames=[builder.name for builder in builders],
302+
userpass=try_userpass)
303+
]
304+
305+
####### STATUS TARGETS
306+
307+
# 'status' is a list of Status Targets. The results of each build will be
308+
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
309+
# like IRC bots.
310+
311+
c['status'] = []
312+
313+
authz_cfg=util.Authz(
314+
auth=util.BasicAuth(web_userpass),
315+
gracefulShutdown = False,
316+
pingBuilder = False,
317+
pauseSlave = 'auth',
318+
stopBuild = 'auth',
319+
stopAllBuilds = 'auth',
320+
forceBuild = 'auth',
321+
forceAllBuilds = 'auth',
322+
cancelPendingBuild = 'auth',
323+
)
324+
325+
c['status'].append(html.WebStatus(http_port=bb_web_port,
326+
order_console_by_time=True, authz=authz_cfg))
327+
328+
####### DB URL
329+
330+
c['db'] = {
331+
# This specifies what database buildbot uses to store its state. You can leave
332+
# this at its default for all but the largest installations.
333+
'db_url' : "sqlite:///state.sqlite",
334+
}
335+

‎master/password.py.sample

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# -*- python -*-
2+
# ex: set syntax=python:
3+
4+
# Build slaves configured with mkBuildSlave() must have a name and password
5+
# entry. The names must match the names in mkBuildSlave() and the passwords
6+
# should be distributed to the static builders for authentication.
7+
slave_userpass = [
8+
("slave1", "password1"),
9+
("slave2", "password2"),
10+
]
11+
12+
# Web users are authenticated using a basic login and password. From the
13+
# web interface pending builds may be canceled or resubmitted.
14+
web_userpass = [
15+
("user1", "password1"),
16+
("user2", "password2"),
17+
]
18+
19+
# Amazon ec2 credentials which are needed to create latent build slaves.
20+
ec2_default_access = "access"
21+
ec2_default_secret = "secret"
22+
23+
# An Amazon ec2 key pair which should be installed for the default login
24+
# to enable access to the instance. This is optional.
25+
ec2_default_keypair_name = "buildbot"
26+

‎master/public_html/bg_gradient.jpg

1.78 KB
Loading

‎master/public_html/default.css

+640
Large diffs are not rendered by default.

‎master/public_html/favicon.ico

1.12 KB
Binary file not shown.

‎master/public_html/robots.txt

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
User-agent: *
2+
Disallow: /waterfall
3+
Disallow: /builders
4+
Disallow: /changes
5+
Disallow: /buildslaves
6+
Disallow: /schedulers
7+
Disallow: /one_line_per_build
8+
Disallow: /grid
9+
Disallow: /tgrid
10+
Disallow: /json

‎master/templates/README.txt

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
This is the directory to place customized templates for webstatus.
2+
3+
You can find the sources for the templates used in:
4+
buildbot/status/web/templates
5+
6+
You can copy any of those files to this directory, make changes, and buildbot will automatically
7+
use your modified templates.
8+
9+
Also of note is that you will not need to restart/reconfig buildbot master to have these changes take effect.
10+
11+
The syntax of the templates is Jinja2:
12+
http://jinja.pocoo.org/

‎scripts/bb-bootstrap.sh

+302
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,302 @@
1+
#!/bin/bash
2+
3+
# Copyright 2011 Henrik Ingo <henrik.ingo@openlife.cc>
4+
# License = GPLv2 or later
5+
#
6+
# This program is free software; you can redistribute it and/or modify
7+
# it under the terms of the GNU General Public License as published by
8+
# the Free Software Foundation; version 2 or later of the License.
9+
#
10+
# This program is distributed in the hope that it will be useful,
11+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
12+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13+
# GNU General Public License for more details.
14+
#
15+
# You should have received a copy of the GNU General Public License
16+
# along with this program; if not, write to the Free Software
17+
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
18+
19+
set -e
20+
21+
# Check for a local cached configuration.
22+
if test -f /etc/buildslave; then
23+
. /etc/buildslave
24+
fi
25+
26+
# These parameters should be set and exported in the user-data script that
27+
# calls us. If they are not there, we set some defaults but they almost
28+
# certainly will not work.
29+
if test ! "$BB_MASTER"; then
30+
BB_MASTER="build.lustre.org:9989"
31+
fi
32+
if test ! "$BB_NAME"; then
33+
BB_NAME=$(hostname)
34+
fi
35+
if test ! "$BB_PASSWORD"; then
36+
BB_PASSWORD="password"
37+
fi
38+
if test ! "$BB_ADMIN"; then
39+
BB_ADMIN="Automated Lustre BuildBot slave <buildbot-admin@lustre.org>"
40+
fi
41+
if test ! "$BB_DIR"; then
42+
BB_DIR="/var/lib/buildbot/slaves/lustre"
43+
fi
44+
if test ! "$BB_USE_PIP"; then
45+
BB_USE_PIP=0
46+
fi
47+
if test ! "$BB_URL"; then
48+
BB_URL="https://raw.githubusercontent.com/opensfs/lustre-buildbot-config/master/scripts/"
49+
fi
50+
51+
if test ! -f /etc/buildslave; then
52+
echo "BB_MASTER=\"$BB_MASTER\"" > /etc/buildslave
53+
echo "BB_NAME=\"$BB_NAME\"" >> /etc/buildslave
54+
echo "BB_PASSWORD=\"$BB_PASSWORD\"" >> /etc/buildslave
55+
echo "BB_ADMIN=\"$BB_ADMIN\"" >> /etc/buildslave
56+
echo "BB_DIR=\"$BB_DIR\"" >> /etc/buildslave
57+
echo "BB_URL=\"$BB_URL\"" >> /etc/buildslave
58+
fi
59+
60+
BB_PARAMS="${BB_DIR} ${BB_MASTER} ${BB_NAME} ${BB_PASSWORD}"
61+
echo "$0: BB_PARAMS is now $BB_PARAMS"
62+
echo "$0: BB_URL is now $BB_URL"
63+
64+
set -x
65+
66+
# Magic IP address from where to obtain EC2 metadata
67+
METAIP="169.254.169.254"
68+
METAROOT="http://${METAIP}/latest"
69+
# Don't print 404 error documents. Don't print progress information.
70+
CURL="curl --fail --silent"
71+
72+
73+
testbin () {
74+
BIN_PATH="$(which ${1})"
75+
if [ ! -x "${BIN_PATH}" -o -z "${BIN_PATH}" ]; then
76+
return 1
77+
fi
78+
return 0
79+
}
80+
81+
case "$BB_NAME" in
82+
Amazon*)
83+
yum -y install deltarpm gcc python-pip python-devel
84+
easy_install --quiet buildbot-slave
85+
BUILDSLAVE="/usr/local/bin/buildslave"
86+
87+
# User buildbot needs to be added to sudoers and requiretty disabled.
88+
if ! id -u buildbot >/dev/null 2>&1; then
89+
adduser buildbot
90+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
91+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
92+
sed -i.bak '/secure_path/d' /etc/sudoers
93+
fi
94+
;;
95+
96+
CentOS*)
97+
if cat /etc/redhat-release | grep -Eq "6."; then
98+
# The buildbot-slave package isn't available from a common repo.
99+
BUILDSLAVE_URL="http://build.zfsonlinux.org"
100+
BUILDSLAVE_RPM="buildbot-slave-0.8.8-2.el6.noarch.rpm"
101+
yum -y install $BUILDSLAVE_URL/$BUILDSLAVE_RPM
102+
BUILDSLAVE="/usr/bin/buildslave"
103+
else
104+
yum -y install deltarpm gcc python-pip python-devel
105+
easy_install --quiet buildbot-slave
106+
BUILDSLAVE="/usr/bin/buildslave"
107+
fi
108+
109+
# User buildbot needs to be added to sudoers and requiretty disabled.
110+
if ! id -u buildbot >/dev/null 2>&1; then
111+
adduser buildbot
112+
fi
113+
114+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
115+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
116+
sed -i.bak '/secure_path/d' /etc/sudoers
117+
;;
118+
119+
Debian*)
120+
apt-get --yes update
121+
122+
# Relying on the pip version of the buildslave is more portable but
123+
# slower to bootstrap. By default prefer the packaged version.
124+
if test $BB_USE_PIP -ne 0; then
125+
apt-get --yes install gcc curl python-pip python-dev
126+
pip --quiet install buildbot-slave
127+
BUILDSLAVE="/usr/local/bin/buildslave"
128+
else
129+
apt-get --yes install curl buildbot-slave
130+
BUILDSLAVE="/usr/bin/buildslave"
131+
fi
132+
133+
# User buildbot needs to be added to sudoers and requiretty disabled.
134+
if ! id -u buildbot >/dev/null 2>&1; then
135+
adduser --disabled-password --gecos "" buildbot
136+
fi
137+
138+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
139+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
140+
sed -i.bak '/secure_path/d' /etc/sudoers
141+
;;
142+
143+
Fedora*)
144+
# Relying on the pip version of the buildslave is more portable but
145+
# slower to bootstrap. By default prefer the packaged version.
146+
if test $BB_USE_PIP -ne 0; then
147+
dnf -y install gcc python-pip python-devel
148+
easy_install --quiet buildbot-slave
149+
BUILDSLAVE="/usr/bin/buildslave"
150+
else
151+
dnf -y install buildbot-slave
152+
BUILDSLAVE="/usr/bin/buildslave"
153+
fi
154+
155+
# User buildbot needs to be added to sudoers and requiretty disabled.
156+
if ! id -u buildbot >/dev/null 2>&1; then
157+
adduser buildbot
158+
fi
159+
160+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
161+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
162+
sed -i.bak '/secure_path/d' /etc/sudoers
163+
;;
164+
165+
Gentoo*)
166+
emerge-webrsync
167+
emerge app-admin/sudo dev-util/buildbot-slave
168+
BUILDSLAVE="/usr/bin/buildslave"
169+
170+
# User buildbot needs to be added to sudoers and requiretty disabled.
171+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
172+
;;
173+
174+
RHEL*)
175+
yum -y install deltarpm gcc python-pip python-devel
176+
easy_install --quiet buildbot-slave
177+
BUILDSLAVE="/usr/bin/buildslave"
178+
179+
# User buildbot needs to be added to sudoers and requiretty disabled.
180+
if ! id -u buildbot >/dev/null 2>&1; then
181+
adduser buildbot
182+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
183+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
184+
sed -i.bak '/secure_path/d' /etc/sudoers
185+
fi
186+
;;
187+
188+
SUSE*)
189+
# SLES appears to not always register their repos properly.
190+
echo "solver.allowVendorChange = true" >>/etc/zypp/zypp.conf
191+
# while ! zypper --non-interactive up; do sleep 10; done
192+
while ! /usr/sbin/registercloudguest --force-new; do sleep 10; done
193+
# may need to kexec to reload kernel, need to some how do an up?
194+
195+
# Zypper auto-refreshes on boot retry to avoid spurious failures.
196+
zypper --non-interactive install gcc python-devel python-pip
197+
easy_install --quiet buildbot-slave
198+
BUILDSLAVE="/usr/bin/buildslave"
199+
200+
# User buildbot needs to be added to sudoers and requiretty disabled.
201+
if ! id -u buildbot >/dev/null 2>&1; then
202+
groupadd buildbot
203+
useradd buildbot
204+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
205+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
206+
sed -i.bak '/secure_path/d' /etc/sudoers
207+
fi
208+
209+
mkdir "/home/buildbot"
210+
chown "buildbot:" "/home/buildbot/"
211+
;;
212+
213+
OpenSUSE*)
214+
# SLES appears to not always register their repos properly.
215+
echo "solver.allowVendorChange = true" >>/etc/zypp/zypp.conf
216+
# while ! zypper --non-interactive up; do sleep 10; done
217+
while ! /usr/sbin/registercloudguest --force-new; do sleep 10; done
218+
# may need to kexec to reload kernel, need to some how do an up?
219+
220+
# Zypper auto-refreshes on boot retry to avoid spurious failures.
221+
zypper --non-interactive install gcc python-devel python-pip
222+
easy_install --quiet buildbot-slave
223+
BUILDSLAVE="/usr/bin/buildslave"
224+
225+
# User buildbot needs to be added to sudoers and requiretty disabled.
226+
if ! id -u buildbot >/dev/null 2>&1; then
227+
groupadd buildbot
228+
useradd buildbot
229+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
230+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
231+
sed -i.bak '/secure_path/d' /etc/sudoers
232+
fi
233+
234+
mkdir "/home/buildbot"
235+
chown "buildbot:" "/home/buildbot/"
236+
;;
237+
238+
Ubuntu*)
239+
# codename=$(lsb_release -c | awk '{print $2}')
240+
# sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
241+
#deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
242+
#deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
243+
#deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
244+
#deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
245+
#EOF
246+
247+
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
248+
sudo apt-get --yes update
249+
250+
# Relying on the pip version of the buildslave is more portable but
251+
# slower to bootstrap. By default prefer the packaged version.
252+
if test $BB_USE_PIP -ne 0; then
253+
apt-get --yes install gcc python-pip python-dev
254+
pip --quiet install buildbot-slave
255+
BUILDSLAVE="/usr/local/bin/buildslave"
256+
else
257+
apt-get --yes install buildbot-slave
258+
BUILDSLAVE="/usr/bin/buildslave"
259+
fi
260+
261+
# User buildbot needs to be added to sudoers and requiretty disabled.
262+
if ! id -u buildbot >/dev/null 2>&1; then
263+
adduser --disabled-password --gecos "" buildbot
264+
fi
265+
266+
echo "buildbot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
267+
sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
268+
sed -i.bak '/secure_path/d' /etc/sudoers
269+
;;
270+
271+
*)
272+
echo "Unknown distribution, cannot bootstrap $BB_NAME"
273+
;;
274+
esac
275+
276+
# Generic buildslave configuration
277+
if test ! -d $BB_DIR; then
278+
mkdir -p $BB_DIR
279+
chown buildbot.buildbot $BB_DIR
280+
sudo -u buildbot $BUILDSLAVE create-slave --umask=022 --usepty=0 $BB_PARAMS
281+
fi
282+
283+
# Extract some of the EC2 meta-data and make it visible in the buildslave
284+
echo $BB_ADMIN > $BB_DIR/info/admin
285+
$CURL "${METAROOT}/meta-data/public-hostname" > $BB_DIR/info/host
286+
echo >> $BB_DIR/info/host
287+
$CURL "${METAROOT}/meta-data/instance-type" >> $BB_DIR/info/host
288+
echo >> $BB_DIR/info/host
289+
$CURL "${METAROOT}/meta-data/ami-id" >> $BB_DIR/info/host
290+
echo >> $BB_DIR/info/host
291+
$CURL "${METAROOT}/meta-data/instance-id" >> $BB_DIR/info/host
292+
echo >> $BB_DIR/info/host
293+
uname -a >> $BB_DIR/info/host
294+
grep MemTotal /proc/meminfo >> $BB_DIR/info/host
295+
grep 'model name' /proc/cpuinfo >> $BB_DIR/info/host
296+
grep 'processor' /proc/cpuinfo >> $BB_DIR/info/host
297+
298+
# Finally, start it.
299+
sudo -u buildbot $BUILDSLAVE start $BB_DIR
300+
301+
# If all goes well, at this point you should see a buildbot slave joining your
302+
# farm. You can then manage the rest of the work from the buildbot master.

‎scripts/bb-build-lustre-pkg.sh

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/bin/bash
2+
set -x
3+
4+
VERBOSE=0
5+
withzfs="--without-zfs"
6+
withldiskfs="--without-ldiskfs"
7+
MAKE_FLAGS=""
8+
9+
message () {
10+
[ $VERBOSE -eq 1 ] && echo -e "$@"
11+
return 0
12+
}
13+
14+
die () {
15+
echo -e "Error: $@" >&2
16+
exit 1
17+
}
18+
19+
while getopts vlzm: FLAG; do
20+
case "$FLAG" in
21+
v)
22+
VERBOSE=1
23+
;;
24+
z)
25+
withzfs="--with-zfs"
26+
;;
27+
l)
28+
withldiskfs="--with-ldiskfs"
29+
;;
30+
m)
31+
MAKE_FLAGS="$OPTARG"
32+
;;
33+
esac
34+
done
35+
shift $((OPTIND-1))
36+
37+
sh ./autogen.sh || die "Lustre autogen failed"
38+
./configure $withldiskfs $withzfs || die "Lustre configure failed"
39+
make $MAKE_FLAGS -j$(nproc) || die "Building lustre binaries failed"

‎scripts/bb-build-lustre-srpm.sh

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/bin/bash
2+
set -x
3+
4+
VERBOSE=0
5+
withzfs="--without zfs"
6+
withldiskfs="--without ldiskfs"
7+
8+
message () {
9+
[ $VERBOSE -eq 1 ] && echo -e "$@"
10+
return 0
11+
}
12+
13+
die () {
14+
echo -e "Error: $@" >&2
15+
exit 1
16+
}
17+
18+
while getopts vlz FLAG; do
19+
case "$FLAG" in
20+
v)
21+
VERBOSE=1
22+
;;
23+
z)
24+
withzfs="--with zfs"
25+
;;
26+
l)
27+
withldiskfs="--with ldiskfs"
28+
;;
29+
esac
30+
done
31+
shift $((OPTIND-1))
32+
33+
sh autogen.sh || die "Lustre autogen failed"
34+
./configure --enable-dist || die "Lustre configure failed"
35+
make srpm || die "Building lustre srpm failed"
36+
rpmbuild --rebuild $withldiskfs $withzfs lustre*.src.rpm || die "rpmbuild failed"

‎scripts/bb-build-zfs-pkg.sh

+148
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
#!/bin/bash
2+
3+
# Check for a local cached configuration.
4+
if test -f /etc/buildslave; then
5+
. /etc/buildslave
6+
else
7+
echo "Missing configuration /etc/buildslave. Assuming spl and zfs are"
8+
echo "already installed and this is a persistent buildslave."
9+
exit 0
10+
fi
11+
12+
set -x
13+
14+
VERBOSE=0
15+
GIT_SPL=https://github.com/zfsonlinux/spl.git
16+
GIT_ZFS=https://github.com/zfsonlinux/zfs.git
17+
BUILD_ROOT=
18+
SPL_TAG="master"
19+
ZFS_TAG="master"
20+
21+
# Utility functions
22+
message () {
23+
[ $VERBOSE -eq 1 ] && echo -e "$@"
24+
return 0
25+
}
26+
27+
cleanup () {
28+
dir -c
29+
[ -n "${BUILD_ROOT}" ] && rm -Rf ${BUILD_ROOT}
30+
return 0
31+
}
32+
33+
die () {
34+
echo -e "Error: $@" >&2
35+
cleanup
36+
exit 1
37+
}
38+
39+
install_packages () {
40+
SUDO="sudo"
41+
42+
case "$BB_NAME" in
43+
Amazon*)
44+
$SUDO rm *.src.rpm *.noarch.rpm 2>&1
45+
$SUDO yum -y localinstall *.rpm 2>&1
46+
;;
47+
48+
CentOS*)
49+
$SUDO rm *.src.rpm *.noarch.rpm 2>&1
50+
$SUDO yum -y localinstall *.rpm 2>&1
51+
;;
52+
53+
Debian*)
54+
for file in *.deb; do
55+
$SUDO gdebi --quiet --non-interactive $file 2>&1
56+
done
57+
;;
58+
59+
Fedora*)
60+
$SUDO rm *.src.rpm *.noarch.rpm 2>&1
61+
$SUDO dnf -y localinstall *.rpm 2>&1
62+
;;
63+
64+
RHEL*)
65+
$SUDO rm *.src.rpm *.noarch.rpm 2>&1
66+
$SUDO yum -y localinstall *.rpm 2>&1
67+
;;
68+
69+
SUSE*)
70+
$SUDO rm *.src.rpm *.noarch.rpm 2>&1
71+
$SUDO zypper --non-interactive install *.rpm 2>&1
72+
;;
73+
74+
Ubuntu*)
75+
for file in *.deb; do
76+
$SUDO gdebi --quiet --non-interactive $file 2>&1
77+
done
78+
;;
79+
80+
*)
81+
echo "$BB_NAME unknown platform" 2>&1
82+
;;
83+
esac
84+
}
85+
86+
while getopts vs:z: FLAG; do
87+
case "$FLAG" in
88+
v)
89+
VERBOSE=1
90+
;;
91+
z)
92+
ZFS_TAG="$OPTARG"
93+
;;
94+
s)
95+
SPL_TAG="$OPTARG"
96+
;;
97+
esac
98+
done
99+
shift $((OPTIND-1))
100+
101+
if [ -f /etc/buildbot_spl ] && [ -f /etc/buildbot_zfs ]; then
102+
message "spl and zfs already installed. Skipping..."
103+
exit 3
104+
fi
105+
106+
BUILD_ROOT=`mktemp -d`
107+
MAKE_FLAGS="pkg"
108+
CONFIG_OPTIONS=""
109+
110+
# enter build root
111+
pushd ${BUILD_ROOT} &>/dev/null
112+
113+
if [ ! -f /etc/buildbot_spl ]; then
114+
message " == REPO ${GIT_SPL} TAG ${SPL_TAG} =="
115+
git clone ${GIT_SPL} ./spl || die "Failed to get SPL source"
116+
117+
pushd spl &>/dev/null
118+
git checkout ${SPL_TAG} || die "SPL checkout failed"
119+
sh autogen.sh || die "SPL autogen failed"
120+
./configure $CONFIG_OPTIONS || die "SPL configure failed"
121+
make $MAKE_FLAGS || die "SPL make pkg failed"
122+
123+
install_packages
124+
125+
sudo touch /etc/buildbot_spl
126+
popd &>/dev/null
127+
fi
128+
129+
if [ ! -f /etc/buildbot_zfs ]; then
130+
message "== REPO ${GIT_ZFS} TAG ${ZFS_TAG} =="
131+
git clone ${GIT_ZFS} ./zfs || die "Failed to get ZFS source"
132+
pushd zfs &>/dev/null
133+
git checkout ${ZFS_TAG} || die "ZFS checkout failed"
134+
sh autogen.sh || die "ZFS autogen failed"
135+
./configure $CONFIG_OPTIONS || die "ZFS configure failed"
136+
make $MAKE_FLAGS || die "ZFS make rpm failed"
137+
138+
install_packages
139+
140+
sudo touch /etc/buildbot_zfs
141+
popd &>/dev/null
142+
fi
143+
144+
popd &>/dev/null
145+
146+
cleanup
147+
148+
exit 0

‎scripts/bb-dependencies.sh

+180
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
#!/bin/bash
2+
3+
# Check for a local cached configuration.
4+
if test -f /etc/buildslave; then
5+
. /etc/buildslave
6+
else
7+
echo "Missing configuration /etc/buildslave. Assuming dependencies are"
8+
echo "already satisfied and this is a persistent buildslave."
9+
exit 0
10+
fi
11+
12+
set -x
13+
14+
case "$BB_NAME" in
15+
Amazon*)
16+
# Required development packages.
17+
sudo yum -y install kernel-devel-$(uname -r) \
18+
kernel-debuginfo-$(uname -r) \
19+
zlib-devel libuuid-devel libblkid-devel libselinux-devel \
20+
xfsprogs-devel libattr-devel libacl-devel
21+
22+
# Required utilties.
23+
sudo yum -y install git rpm-build wget curl lsscsi parted attr dbench \
24+
watchdog createrepo python python-pip python-docutils
25+
;;
26+
27+
CentOS*)
28+
if cat /etc/redhat-release | grep -Eq "6."; then
29+
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
30+
#sudo yum -y localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
31+
elif cat /etc/redhat-release | grep -Eq "7."; then
32+
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
33+
#sudo yum -y localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm
34+
fi
35+
36+
# Required development tools.
37+
sudo yum -y install gcc make autoconf libtool
38+
39+
# Development packages
40+
sudo yum -y --enablerepo=base-debuginfo install kernel-devel-$(uname -r) \
41+
kernel-debuginfo-$(uname -r) \
42+
zlib-devel libuuid-devel libblkid-devel libselinux-devel \
43+
xfsprogs-devel libattr-devel libacl-devel
44+
45+
# Required utilties.
46+
sudo yum -y install git rpm-build wget curl lsscsi parted attr dbench bc \
47+
watchdog createrepo mock python python-docutils mdadm
48+
49+
# add user to the mock group
50+
sudo usermod -a -G mock buildbot
51+
;;
52+
53+
Debian*)
54+
#sudo apt-get --yes install lsb-release
55+
#sudo wget --quiet http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb
56+
#sudo dpkg -i zfsonlinux_6_all.deb
57+
58+
# Required development tools.
59+
sudo apt-get --yes install build-essential autoconf libtool libtool-bin
60+
61+
# Development packages
62+
sudo apt-get --yes install linux-headers-$(uname -r) \
63+
kernel-debuginfo-$(uname -r) \
64+
zlib1g-dev uuid-dev libblkid-dev libselinux-dev \
65+
xfslibs-dev libattr1-dev libacl1-dev
66+
67+
# Required utilties.
68+
sudo apt-get --yes install git alien fakeroot wget curl bc \
69+
lsscsi parted gdebi attr dbench watchdog createrepo \
70+
python python-pip python-docutils
71+
;;
72+
73+
Fedora*)
74+
#sudo yum install --nogpgcheck http://archive.zfsonlinux.org/fedora/zfs-release$(rpm -E %dist).noarch.rpm
75+
76+
# Required development tools.
77+
sudo dnf -y install gcc autoconf libtool
78+
79+
# Development packages
80+
sudo dnf -y install kernel-devel-$(uname -r)
81+
kernel-debuginfo-$(uname -r) \
82+
zlib-devel libuuid-devel libblkid-devel libselinux-devel \
83+
xfsprogs-devel libattr-devel libacl-devel
84+
85+
# Required utilties.
86+
sudo dnf -y install git rpm-build wget curl lsscsi parted attr dbench \
87+
watchdog createrepo mock python python-pip python-docutils
88+
89+
# add user to the mock group
90+
sudo usermod -a -G mock buildbot
91+
;;
92+
93+
RHEL*)
94+
if cat /etc/redhat-release | grep -Eq "6."; then
95+
EXTRA_REPO="--enablerepo=rhui-REGION-rhel-server-releases-optional"
96+
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
97+
#sudo yum -y localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
98+
elif cat /etc/redhat-release | grep -Eq "7."; then
99+
EXTRA_REPO="--enablerepo=rhui-REGION-rhel-server-optional"
100+
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
101+
#sudo yum -y localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm
102+
else
103+
EXTRA_REPO=""
104+
fi
105+
106+
# Required development tools.
107+
sudo yum -y install gcc autoconf libtool
108+
109+
# Development packages
110+
sudo yum -y $EXTRA_REPO --enablerepo=rhel-debuginfo install kernel-devel-$(uname -r) \
111+
kernel-debuginfo-$(uname -r) \
112+
zlib-devel libuuid-devel libblkid-devel libselinux-devel \
113+
xfsprogs-devel libattr-devel libacl-devel
114+
115+
# Required utilties.
116+
sudo yum -y $EXTRA_REPO install git rpm-build wget curl lsscsi \
117+
parted attr dbench bc watchdog createrepo mock python \
118+
python-pip python-docutils mdadm
119+
120+
# add user to the mock group
121+
sudo usermod -a -G mock buildbot
122+
;;
123+
124+
SUSE*)
125+
# assume SUSE 12 for now
126+
#sudo zypper --non-interactive ar -f http://download.opensuse.org/repositories/filesystems/SLE_12/ OpenSUSE-SLE12
127+
128+
# Required development tools.
129+
sudo zypper --non-interactive install gcc autoconf libtool
130+
131+
# Required utilties.
132+
sudo zypper --non-interactive install git rpm-build wget curl \
133+
lsscsi parted attr createrepo python python-pip python-docutils
134+
135+
# Required development packages.
136+
sudo zypper --non-interactive --plus-content debug install \
137+
kernel-devel kernel-default-debuginfo \
138+
zlib-devel libuuid-devel libblkid-devel libselinux-devel \
139+
xfsprogs-devel libattr-devel libacl-devel kernel-source
140+
;;
141+
142+
OpenSUSE*)
143+
# assume SUSE 12 for now
144+
#sudo zypper --non-interactive ar -f http://download.opensuse.org/repositories/filesystems/SLE_12/ OpenSUSE-SLE12
145+
146+
# Required development tools.
147+
sudo zypper --non-interactive install gcc autoconf libtool
148+
149+
# Required utilties.
150+
sudo zypper --non-interactive install git rpm-build wget curl \
151+
lsscsi parted attr createrepo python python-pip python-docutils
152+
153+
# Required development packages.
154+
sudo zypper --non-interactive --plus-content debug install \
155+
kernel-devel kernel-default-debuginfo \
156+
zlib-devel libuuid-devel libblkid-devel libselinux-devel \
157+
xfsprogs-devel libattr-devel libacl-devel kernel-source
158+
;;
159+
160+
Ubuntu*)
161+
# Required development tools.
162+
sudo apt-get --yes install build-essential autoconf libtool \
163+
module-assistant libreadline-dev dpatch libsnmp-dev quilt
164+
165+
# Required utilties.
166+
sudo apt-get --yes install git alien fakeroot wget curl \
167+
lsscsi parted gdebi attr dbench watchdog \
168+
python python-pip python-docutils
169+
170+
# Required development libraries
171+
sudo apt-get --yes install linux-headers-$(uname -r) \
172+
linux-image-$(uname -r) \
173+
zlib1g-dev uuid-dev libblkid-dev libselinux-dev \
174+
xfslibs-dev libattr1-dev libacl1-dev
175+
;;
176+
177+
*)
178+
echo "$BB_NAME unknown platform"
179+
;;
180+
esac

‎scripts/runurl

+200
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
#!/bin/sh
2+
#
3+
# runurl - Download a URL and run as a program, passing in arguments
4+
#
5+
# Copyright (C) 2009 Eric Hammond <ehammond@thinksome.com>
6+
#
7+
8+
error() { echo "$@" 1>&2; }
9+
fail() { [ $# -eq 0 ] || error "${BNAME}:" "$1"; exit ${2:-1}; }
10+
debug() { [ "$DEBUG" = "0" ] || error "${BNAME}:" "$@"; }
11+
cleanup() { [ -z "${TEMP_D}" -o ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"; }
12+
13+
DEBUG="0"
14+
TEMP_D=""
15+
BNAME=${0##*/}
16+
17+
while [ $# -gt 0 ]; do
18+
case $1 in
19+
-\?|--help) pod2text "${0}"; exit 0;;
20+
-d|--debug) DEBUG=1; shift 1 ;;
21+
-*) fail "Unrecognized option: $1 (try --help)";;
22+
*) url="$1"; shift 1; break ;;
23+
esac
24+
done
25+
26+
[ -n "$url" ] || fail "Missing URL specification (try --help)"
27+
28+
trap cleanup 0
29+
30+
TEMP_D=$(mktemp -d ${TEMPDIR:-/tmp}/${BNAME}.XXXXXX) &&
31+
runfile="${TEMP_D}/runfile" && wgetfile="${TEMP_D}/wget.out" ||
32+
fail "failed to make tempdir"
33+
34+
debug "downloading $url"
35+
36+
wget \
37+
--retry-connrefused \
38+
--tries=20 \
39+
"--output-document=$runfile" \
40+
"--output-file=$wgetfile" \
41+
"$url"
42+
43+
wgetstatus=$?
44+
if [ $wgetstatus != 0 ]; then
45+
cat $wgetfile >&2
46+
fail "wget failed: $wgetstatus" "${wgetstatus}"
47+
fi
48+
49+
chmod 700 "${runfile}" || fail "failed to change perms of ${runfile}"
50+
51+
debug "running"
52+
$runfile "$@"
53+
54+
exitstatus=$?
55+
56+
debug "exit status $exitstatus"
57+
58+
exit $exitstatus
59+
60+
#
61+
# To read the documentation in this file use the command:
62+
#
63+
# perldoc runurl
64+
#
65+
66+
=head1 NAME
67+
68+
runurl - Download a URL and run as a program, passing in arguments
69+
70+
=head1 SYNOPSYS
71+
72+
runurl [I<OPTS>] I<URL> [I<ARGS>]...
73+
74+
=head1 OPTIONS
75+
76+
=over 8
77+
78+
=item B<-?> B<--help>
79+
80+
Debug mode
81+
82+
=item B<-d> B<--debug>
83+
84+
Debug mode
85+
86+
=back
87+
88+
=head1 ARGUMENTS
89+
90+
=over 8
91+
92+
=item B<URL>
93+
94+
The URL of the progran to download and run
95+
96+
=item B<ARGS>
97+
98+
Options and arguments for the downloaded program
99+
100+
=back
101+
102+
=head1 DESCRIPTION
103+
104+
The B<runurl> command is a simple tool that downloads a program (or
105+
script) from the specified URL and runs it.
106+
107+
The first argument to the B<runurl> command is the URL of a script or
108+
program that should be run. Any leading "http://" may be omitted, but
109+
"https://" or "ftp://" and the like must still be specified.
110+
111+
All remaining arguments listed after the URL (including ones which
112+
look like options) are passed verbatim to the program as its own
113+
options and arguments when it is run.
114+
115+
The exit code of B<runurl> is the exit code of the program, unless the
116+
original download of the URL failed, in which case that error is
117+
returned.
118+
119+
=head1 EXAMPLES
120+
121+
If the following content is stored at http://run.alestic.com/demo/hello
122+
123+
#!/bin/bash
124+
echo "hello, $1"
125+
126+
then this command:
127+
128+
runurl run.alestic.com/demo/hello world
129+
130+
will itself output:
131+
132+
hello, world
133+
134+
=head1 CAVEATS
135+
136+
Only run content that you control or completely trust.
137+
138+
Just because you like the content of a URL when you look at it in your
139+
browser does not mean that it will still look like that when B<runurl>
140+
goes to run it. It could change at any point to something that is
141+
broken or even malicious unless it is under your control.
142+
143+
Realize that you are depending on the network for commands to succeed.
144+
If the content is temporarily unavailable or has been moved, then the
145+
B<runurl> command will fail.
146+
147+
=head1 DEPENDENCIES
148+
149+
This program requires that the following already be installed:
150+
151+
wget
152+
153+
=head1 SEE ALSO
154+
155+
The B<runurl> project site:
156+
157+
https://launchpad.net/runurl
158+
159+
Article about using B<runurl> for initial configuration of Amazon EC2
160+
instances:
161+
162+
http://alestic.com/2009/08/runurl
163+
164+
=head1 INSTALLATION
165+
166+
On most Ubuntu releases, the B<runurl> package can be installed
167+
directly from the Alestic.com PPA using the following commands:
168+
169+
code=$(lsb_release -cs)
170+
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu $code main"|
171+
sudo tee /etc/apt/sources.list.d/alestic-ppa.list
172+
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
173+
sudo apt-get update
174+
sudo apt-get install -y runurl
175+
176+
=head1 BUGS
177+
178+
Please report bugs at https://bugs.launchpad.net/runurl
179+
180+
=head1 AUTHOR
181+
182+
Eric Hammond <ehammond@thinksome.com>
183+
184+
=head1 LICENSE
185+
186+
Copyright 2009 Eric Hammond <ehammond@thinksome.com>
187+
188+
Licensed under the Apache License, Version 2.0 (the "License");
189+
you may not use this file except in compliance with the License.
190+
You may obtain a copy of the License at
191+
192+
http://www.apache.org/licenses/LICENSE-2.0
193+
194+
Unless required by applicable law or agreed to in writing, software
195+
distributed under the License is distributed on an "AS IS" BASIS,
196+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
197+
See the License for the specific language governing permissions and
198+
limitations under the License.
199+
200+
=cut

0 commit comments

Comments
 (0)
Please sign in to comment.