The realm of the build czar
    
    Build Czar Duties
    
      The main duties of the Build Czar are summarized as follows
      
        Continuously monitor the builds using the 
          Scoreboard 
      as one of the primary source of information.
      
      Notify developers who broke compilation to fix the errors as soon as possible, 
      ideally by the next day. A red color in the "Compile" column is not at all 
      acceptable - the Build Czar needs to ensure that these problems are identified 
      and fixed in a timely manner. If possible, the Build Czar should let developers 
      know what the source of problems might be. It is quite possible that developers 
      who checked in the code or users who provided the patch may not have resources 
      to investigate the issues, so the Builds Czar's help is essential to keep 
      things moving ahead.
      
      Keep an eye on the tests that are run in every build. Anything abnormal needs 
      to be notified to the right developer. The Build Czar should try helping the 
      developer by providing stack traces (in case of crashes) or other details like 
      printouts with debugging level turned on.
      
        Some tests fail in the daily builds for many reasons like known bugs, transient 
        timeouts etc. Make sure that no new test failures show up. This 
          guy
      knows most of the information. Ask him to help you out with known problems.
      
        Keep an eye on the footprint and 
          performance stats. Any abnormal changes should be brought to the 
        attention of the developer resposible for it or to the 
          devo group.
      
      Keep the builds ticking. Any red on the "Last Finished" column in the 
      Scoreboard should be fixed. The link to the "Build Name" indicates the machine 
      where the build is being run.
      
      The builds don't cover all the possible configurations. If you get a bug report 
      about a compile error in a particular configuration, try setting up a build to 
      make sure that it doesn't show up again if it has been fixed.
      
        Keep an eye on the bugzilla
        
      entries that are registered by users and developers. Decide on the bugs that 
      need to be fixed for the beta and pain developers for an ETA.
    
    
      The document here talks about the powers of a 
      build Czar.
    
    
      The Build Czar is empowered to set up more builds on his own for his 
      convenience. This 
        page has a step by step instructions on how to do that.
    
    
      The build czar can get the build configuration by looking at the config portion 
      of the scoreboard.
    
    Pro-active involvement by the build czar is necessary. Being a pro-active 
      build czar requires monitoring the subversion archive carefully and responding 
      quickly to suspected changes to keep the repo stays stable.
    
    Recipe for Cutting a Beta/Minor Kit
    
      The build czar is also in charge for the release of the beta. Cutting a beta is 
      as simple as cutting butter if things go well. Here is the procedure followed 
      while cutting a beta:
      
        - 
          The whole process takes somewhere between 8-9 hours, it now takes much longer 
          than the original 2 hours for making the release itself where as generating the 
          doxygen documentation now completes in around this 2-3 hours mark.
- 
          I suggest you take advantage of GNU Screen so that even if your SSH session is 
          interrupted, the cutting process can continue. This command is available on 
          both of the two machines we use to cut the release.
          
            - 
              type screento start screen.
- 
              execute commands as normal. Note that Ctrl-A is special in screen, so you need 
              to type Ctrl-A-A to send a Ctrl-A to the shell
- 
              should your session be interrupted, reconnect and type screen -x
- 
              when finished, just type exit twice
 
- 
          Prior to starting this, gather aggregate release notes from all developers. 
          This is usually in the form of an email plea asking for a writeup of 
          significant changes since the last beta. Add these notes to the NEWS files 
          before cutting the release so that all notes are part of the release.
- 
          Checkout a new workspace on anduril.dre.vanderbilt.edu
          - 
            The best place to create the workspace is under /export/anduriltmp/bczar. Don't 
            use the home directory itself, it is an NFS share and not really fast.
          
- 
            Checkout like this:
            
              - 
                svn co --username <your user id> 
                https://svn.dre.vanderbilt.edu/DOC/Middleware/trunk DOC_ROOT
- 
                svn co --username <your user id> 
                https://svn.dre.vanderbilt.edu/DOC/MPC/trunk DOC_ROOT/ACE/MPC
 
- 
          Set $DOC_ROOT to point to the new workspace you checked out.
- 
          Set an environment variable SIGNATURE indicating your full name. This is used 
          to fill the ChangeLog entry.
          - 
            For example,export SIGNATURE="Chris Cleeland"
- 
          Set an environment variable MAILID indicating your mail id. This is used to 
          fill the mail id portion of the ChangeLog entry.
          - 
            For example,export MAILID="cleeland@ociweb.com"
- 
          Change directories to $DOC_ROOT
        
- 
          Tag the release by executing
 ACE/bin/make_release.py --beta --update --tag
 This will only take a couple minutes to complete and once done successfully, 
          you can carry on with BOTH creating the kits and generating the doxygen 
          documentation in parallel. NOTE that--betashould be replaced 
          with--minoror--majoras appropriate.
        
        In the EXTREMELY unlikely event that something goes wrong during the 
          tagging of the repo, the following files must be returned to the state 
        they were in before the release process started and then checked back into SVN:
        
          
            - 
              ACE/ChangeLog
- 
              ACE/PROBLEM-REPORT-FORM
- 
              ACE/VERSION
- 
              ACE/TAO/ChangeLog
- 
              ACE/TAO/PROBLEM-REPORT-FORM
- 
              TAO/VERSION
- 
              CIAO/ChangeLog
- 
              CIAO/PROBLEM-REPORT-FORM
- 
              CIAO/VERSION
- 
              CIAO/ciao/Version.h
- 
              TAO/tao/Version.hace/Version.h
In most cases, a
        svn revert -R *
        from DOC_ROOT will suffice.
    The tag will also need to be removed (both in Middleware and MPC): 
    ACE+TAO+CIAO-X_Y_Z (where X is the ACE Major version number, and Y & Z are the 
    Minor and Beta release numbers of the release that is to be restarted).
      E.g.:
      svn rm 
        https://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z
        svn rm https://svn.dre.vanderbilt.edu/DOC/MPC/tags/ACE+TAO+CIAO-X_Y_Z
      
      
      Note that this only needs to be done if the tagging fails. If 
      kit creation fails, simply restart that process.
      
        Create the kits by executing
        ACE/bin/make_release.py --kit
        This will take somewhere arround 4-8 hours to complete.
        
          - 
            These commands only tags and creates the kits for the software itself, not 
            documentation, this can be started in parrellel with this activity.
          
- 
            The kits end up in /export/anduriltmp/bczar/packages
    To summarize, the following is a transcript of the steps up to this point 
    executing successfully:
    
$ ssh bczar@anduril.dre.vanderbilt.edu
        No default printer
        -bash-3.00$ screen
        -bash-3.00$ cd /export/anduriltmp/bczar
        -bash-3.00$ export DOC_ROOT=$PWD/DOC_ROOT
        -bash-3.00$ export SIGNATURE="Johnny Willemsen"
        -bash-3.00$ export MAILID=jwillemsen@remedy.nl
        -bash-3.00$ svn co https://svn.dre.vanderbilt.edu/DOC/Middleware/trunk DOC_ROOT
        -bash-3.00$ svn co https://svn.dre.vanderbilt.edu/DOC/MPC/trunk 
        DOC_ROOT/ACE/MPC
        -bash-3.00$ cd DOC_ROOT/
        -bash-3.00$ ACE/bin/make_release.py --beta --update --tag
        -bash-3.00$ ACE/bin/make_release.py --kit
      
    
      Feel free to cut and paste with suitable edits.
      
        The packages end up by default under $DOC_ROOT/package-<PID>, you can 
        copy them to the webserver using the following commands. (Note that <PID> 
        needs to be the numerical pid of the process that created the kit, use
        ls -ald
        to determine the correct filename.) At the moment you execute these commands 
        all users can download these packages.
      cp $DOC_ROOT/package-<PID>/ACE* 
        /export/www/download.dre/ACE+TAO-distribution
      
      
        After the repository is tagged you can also start generating the doxygen 
        documentation in parrellel with the kit generation above.
        
          - 
            Login to naboo.dre.vanderbilt.edu as bczar and start a screen session:
 ssh bczar@naboo.dre.vanderbilt.edu
 screen
- 
            After login check that you can,
            
 ssh bczar@download.dre.vanderbilt.edu
 to ensure that this succeeds. If not fix the problem, if ok exit again back to 
            naboo. The make script tries to copy the tar.gz files to the website using ssh.
- 
            cd /web/users/isisbuilds/tmp/ACE_wrappers
 and remove the contents of this directory with
          rm -rf *
          - 
            Update the workspace with the right version tag (replace the X_Y_Z with the ACE 
            version number being released e.g. 5_6_7)
            
 svn co 
              svn://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z/ACE 
              ACE_wrappers
 svn co svn://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z/TAO 
              ACE_wrappers/TAO
 svn co svn://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z/CIAO 
              ACE_wrappers/TAO/CIAO
- 
            Set the needed environment variables using
 export ACE_ROOT=/web/users/isisbuilds/tmp/ACE_wrappers/ACE_wrappers
- 
            Check the doxygen version at the shell by executing the command:
 doxygen --version
- 
            Open up $ACE_ROOT/bin/generate_rel_manpagesin your favorite 
            editor.
 Search for the string 'doxy_version'.
 Check that the version specified in the file is the same as the one you got 
            using the shell command. If it is different change the value of the 
            doxy_version to the one installed on naboo.dre.vanderbilt.edu.
Then exit the editor. 
        Check the file, generate_rel_manpages back into the repository if you have made some 
        changes to it.
        Now you are ready to create documentation
      
cd $ACE_ROOT
        make -f Release manpages
      
      
        While doxygen churns, format a release announcement, including the release 
        notes gathered from developers.
        
          - 
            Let Doug Schmidt review these before 
            you do anything with them.
Make sure the new version is available in Bugzilla.
        - 
          We now have a bczar Bugzilla user (bczar@dre.vanderbilt.edu) with full 
          privileges which points to the bczar user at ISIS. To gain access to this 
          account as a new build czar, you could update the ~/.forward file on one of the 
          ISIS hosts (for example bczar@naboo.dre.vanderbilt.edu) with your own email 
          address (but be aware that if you leave this ~/.forward file in effect, you 
          will get innundated with cron mail messages from all of the ISIS lab build 
          machines, so it is probably best to remove it after obtaining the Bugzilla 
          password). From the "Bugzilla - Main Page" 
          (http://bugzilla.dre.vanderbilt.edu/index.cgi) click on one of the various 
          "LogIn" links, and from this login page you should be able to "Submit Request" 
          to change a forgotten password. If you enter bczar@dre.vanderbilt.edu and click 
          "Submit Request" this will email you the password change link to 
          bczar@dre.vanderbilt.edu, which will then in turn be forwarded to your own 
          email account. Simply copy this link into your browser to set your own bczar 
          password for the next steps.
- 
          here is the description of how to add a new version through Bugzilla.
- 
          go to any Bugzilla "Query" page, you should see a grey box at the bottom. click 
          "Log in" link to log in as bczar@dre.vanderbilt.edu.
- 
          look at the grey box at bottom again. You will see several links following 
          "Edit". Click on the "Product" link.
- 
          you are then at "Select product" page. You should see a list components, i.e., 
          ACE CIAO TAO. Click on the product you want to edit.
- 
          you are then at "Edit product" page. Scroll down a bit, you should see a list 
          of all versions coming with this product. At the very beginning of the list, 
          you should see "Edit versions" link. Click this link.
- 
          you should see a "Add a new version" text box and a "Add" link just above the 
          grey box at the bottom. Click on this link
- 
          you are then at "Add version of [Name of the product]" page.
- 
          you are able to add the new version now.
      
        Back on naboo.dre.vanderbilt.edu once the doxygen generation has finished, update the documentation at www.dre.vanderbilt.edu/Doxygen.
      
        - 
          cd /web/www/Doxygen
- 
          Create a temporary directory. eg. tmp and change Directory to this- 
 mkdir tmp
 cd tmp
- 
          scp file from the download server -
 scp 
          bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution/ACE-html.tar.bz2 .
- 
          Unzip and untar the files - 
 bunzip2 ACE-html.tar.bz2;tar -xvf ACE-html.tar
- back out of the temporary directory
 cd ..
- 
          Create directory 'Current Version No' for example 5.6.7 and change directory into this new one
 mkdir 5.6.7
 cd 5.6.7
- 
          Move contents of the temporary directory's html to this directory -
 mv ../tmp/html .
- 
          Now back our of this directory and remove the already existing soflink to the "Micro" directory -
 cd ..
 rm Micro
- 
          Create softlink "Micro" linking it to new Documentation using -
 ln -s 5.6.7/html 
          Micro
- 
        Remove the directory tmp -
 rm -rf tmp
      
        Back on anduril.dre.vanderbilt.edu where the kit was being generated and once BOTH the kit
        and doxygen generation have finished their work, you should also move the packages to the
        previous versions directory with the appropriate decorators.
        
          - 
            cd /export/www/download.dre/ACE+TAO-distribution
- 
            Modify /export/anduriltmp/bczar/copy_script.shto use the correct ACE version X.Y.Z and run it.
      
        Mail the approved release announcement out to, at minimum the following: ciao-users@list.isis.vanderbilt.edu,
        tao-users@list.isis.vanderbilt.edu, tao-announce@list.isis.vanderbilt.edu,
        ace-users@list.isis.vanderbilt.edu, ace-announce@list.isis.vanderbilt.edu. 
        Do this as yourself (not as bugzilla). N.B.
      You will not be able to post to the users' lists unless you are subscribed to 
      them. Odds are you will not be able to post to the announce lists at all. Ask 
      someone else (like Doug or Johnny) to do this step.
      
        When all cidlc builds are ready with the new version, login to 
        naboo.dre.vanderbilt.edu as bczar and run ./cut_cidlc.sh version-number
         where the version-number is the CIAO release just made. For example
         ./cut_cidlc.sh 0.6.7
If this script is not in its place, then 
        the original is in the bin directory of the distribution.
      
        Update in the autobuild archive the file configs/scoreboard/releases.xml with 
        the made release. This is used by the integrated scoreboard on http://remedy.nl  Remember to do a changelog entry.
      
        Update the ACE_wrappers repo (remember to create a changelog entry, and possiably archive the old changelog to the changelog directory if this has become too long):
        - docs/Download.html to show the new release. Make sure you refer to the 
        previous_versions directory, that way we can exactly track how many people 
        download a specific version.
- 
        etc/index.hml to show the new doxygen package you installed
- bin/diff-builds-and-group-fixed-tests-only.sh to give the correct default old_date for this release.
Update the NEWS, TAO/NEWS, and TAO/CIAO/NEWS files to have a new section for the next release.
    
      Tips to being a Build Czar
    
      1. Trust no one.
      2. Be careful with this guy, he 
      is notorious in breaking builds (and fixing them as well...Rumour has it that 
      it's actually a super-scalar, super-pipelined processor capable of out-of-order 
      execution, in human incarnation).
      3. Don't forgive people who break ACE :-)
      4. If a build hasn't run in a long time (symptoms are a "red" in the Last Run 
      column of the build scoreboard), delete the .disable file in 
      /path/to/build/directory/BUILD_NAME/ by hand.
      5. Think of the group who wrote the scoreboard update script, every time you 
      catch an otherwise not so obvious error with the help of the scoreboard. Tell 
        DEVO group about it.
      6. Send a note to  sysadmin@isis.vanderbilt.edu asking for the repo to be frozen. Provide them a list of names, including yourself and bczar to be granted write permission.
      
      7. Compile once on Win32, Linux and Solaris before cutting a beta.
      8. Trust the release script when making a release. Don't make tar balls by 
      hand. Make sure that the public ftp directories 
      (/project/beguine/ftp/pub/ACE+TAO-distribution and 
      /project/beguine/ftp/pub/ACE+TAO-distribution/diffs) have the right 
      permissions, so that the release script can copy the tar balls.
      9. When making a release, make sure that all the auto_compiles on that machine 
      (deuce.doc.wustl.edu) are stopped. Also make sure that there is enough space in 
      /tmp on that machine.
      10. When all hell breaks loose, don't wait for the nightly builds to monitor 
      improvement. Instead manually start the builds.
      11. Maintain private up-to-date workspaces for problem platforms (read as 
      Solaris).
      12. Don't hesitate to ask for help.
      13. When you get an account to access the svn repo, make sure you are added to 
      the correct groups, for example, 
      gid=100(users),5000(doc),5002(acetaodev),5003(cvs). Otherwise you will have 
      problem to checkout various modules.
      14. Install your public key to the different machines you have frequent access 
      to avoid typing password.
      15. Update this page if you have any more tips for future build czars :-). This 
      page is in svn under ACE_wrappres/docs/bczar/bczar.html
    
    
    
      The Realm of the Build Czar
    
    
    Build Czar Arthur
    Many years have passes since the days of the legendary Build Czar Arthur. His 
      duties were given to him by the mystical Lady of the Lake, who outlined the 
      first responsibilities of the Build Czar.
    
      
      Then bespake the Lady of the Lake,
      And these were the words said shee:
      "I knoweth of thy deeds, thou noble Arthur,
      but thy task hath not finished for thee"
      
      "Thou shalt feitch thy trusty steed,
      And cleanse thy builds againe;
      Then shallt thy ryde hath finnished,
      When new kits released thee cann."
      
      Then bespake him noble Arthur
      And these were the words said he:
      "With what weapons shallt I use,
      To asure these from the devil free?"
      
      Then appeered before noble Arthur,
      Uppon the ground a sacred scroll
      Conjurred by the Lady of the Lake
      Borne of the earth in a roll.
      
      She saies, "Clasp this to thine selfe
      For thee shallt find need for it.
      It shall keep others in the cold,
      Only to be ressurected when thee sees fit."
      
      "Others shall join thy person,
      To ryde with thee in thy quest;
      Thee shallt be thankful of theire help,
      And to alsoe hold them steadfast."
      
      "But if theire talke too lodly rise,
      And causeth much damage to thine cuntry,
      He must come forth, and make proclamation,
      For the next one he shall be."
      
      So hath Arthur to the Lady spoke:
      "For I sweare, and save my othe,
      While enimes and evils I seeke,
      I shall fight against them bothe.