Answers to questions
Posted by rick Wed, 21 Dec 2005 23:20:00 GMT
A commenter to the previous post was kind enough to ask some good questions, so it seems only fair that I make a fair attempt at answering them. I’m taking the liberty of re-ordering things, however. (I had really hoped to update more frequently, but time is scarce at work and at home, during the holiday season, even more scarce!)
What do you guys use for all this? Server platform? Dev platform? Any nix platforms in the mix? Any experience on deving on windows and server on nix?
We’re currently in development, meaning that we don’t have a real honest-to-goodness production system running yet. The closest thing we have to a production system is a machine that does our continuous integration for us, as well as hosting our nightly and milestone build sites. It happens to also run our subversion repository and our trac installation, but that’s coincidental (we assigned DNS names for trac, svn, milestone, and nightly so that we could move them whenever we see fit). It also sports an RT installation for feedback, but that’s coincidental too. That system is running Debian Linux (apt-pinned from stable to unstable, and a 2.6.12 kernel) on an Intel P4 2.8GHz CPU w/ 1GB RAM. It’s running Debian exactly because I set it up and do the admin on it. We do our Subversion work over ssh (svn+ssh://...). The RT instance is running on Apache2 w/ mod_perl. The nightly and milestone app builds are hosted on lighttpd using FCGI to run Rails. Taken together, we have Apache2 fronting the lightppd instances via proxying. That is, if you hit nightly:80 then you hit Apache, which sends you to nightly:8001 behind the scenes.
We have developers currently using the following platforms: Windows XP (2 full-time, 1 DBA doing occasional checkouts and builds), Debian Linux, FreeBSD, and I think some other random Linuxen. While we have AIX systems available in-house noone is chomping at the bit to develop on them, and we haven’t tried to deploy to them. So, yes, we do have some experience doing development on Windows with servers on Unix-ish systems.
When we go to production, we will almost certainly be using Linux for the front-end (i.e., non-database) servers, but there’s a vocal FreeBSD contingent in house, and they make a strong case. If I have any work to do on those servers I would demand that Debian be on them, but I would prefer that we have an actual Manager Of Production and he take the keys away from people like me who’d be more likely to ‘rm -rf’ something than not.
Also highly important to us is protecting client data. The legacy system the company currently runs on is locked up pretty tightly, and any test or development systems are populated with anonymized data. For HIPAA and Sarbanes-Oxley we need to ensure that we continue that tradition with the new system we are writing. Of course, so far we haven’t even purchased the servers that will be used to run the system, so setting up the production environment is cart-before-the-horse. That, and re-tasking developers and data conversion folks away from their jobs to prepare for something we really won’t need to think about for months doesn’t make sense to me. So, for now we don’t have a production system or environment yet. I also can’t argue with waiting as long as possible to buy hardware, and putting Moore’s Law to work on our budget.
What apps do you use? CIA? Switchtower? Trac? SVN Of course. Desktop editors?
We are (obviously) using Subversion (SVN) for version control.
For continuous integration we’re using the simple continuous_builder Rails plugin. We had been using CruiseControl when we were working with Java. While CruiseControl really could be considered platform/language agnostic (ok, at a stretch), it’s really geared for Java projects. I took DamageControl for a spin (I know one of the contributors and saw Aslak at RubyConf—he’s a sharp guy and right on top of the mailing list), but ultimately didn’t stay with it. While CruiseControl is simply monstrous, requiring a beefy machine to get it going, DamageControl is relatively lightweight. I like the idea of being able to use a Rails web interface to manage the various aspects of the project build, to catalog logs, etc.
From reading the mailing list I understood that DamageControl needs to be pulled from SVN to get the newest Rails-based version in a working form. First I hit our corporate proxy (I have no love for this abomination known as “BorderManager”, which would be better replaced, if a proxy is truly necessary, with a fresh install of squid) which made doing an SVN pull a no-go. So I had to do a pull from outside the firewall, tar it up, and suck it down over the web (don’t ask), which is what would be necessary whenever I wanted to svn update in the future. I then set about getting things going and got the web interface up and things purportedly properly initialized. Then I spent about 20 minutes trying to get a project entered in such a way that it would (a) try to do an SVN pull and (b) remember that the project actually existed. Seems I couldn’t on that particular day get both to happen at once for some reason. Then the interface started behaving funny (but which I think was mostly the stylesheet going missing(?)). Then I read more mails on the mailing list with more problems getting things going. I looked at my watch and said to myself, “It’s worth my time to take a look at continuous_builder rather than digging deeper into DamageControl right now.”
So I did, and got continuous_builder set up in half an hour. I spent a little more time looking for a good IRC library for Ruby that would let me post build status into our development group’s IRC channel, but to no avail (as in nothing that would let me write a client really quickly to do that one simple thing). I broke down and wrote a quick Perl script using Net::IRC to enter the room, post a message, and be gone. Seems to work. I’ve considered porting Perl’s Net::IRC straight to Ruby, but that may be a rainy day indeed when I get around to it.
So, back to the questions. On the desktop we are mostly vim/gvim (including one guy on Windows using vim (or is it gvim on Windows?)). The only full IDE we have going is one user using ArachnoIDE on Windows and who seems to like it very well. I’ll have to admit that I’ve been tempted to try development with it or with FreeRIDE (having been impressed with the underlying architecture of FreeRIDE). When we were working with Java we had pretty much standardized on using Eclipse, with the occasional vim hold-outs. We discovered pretty quickly that you have to be either a masochist or about a 3-4 sigma productivity developer to make a strong showing with JBoss/J2EE using a text editor like vim. Once we moved away from Java we realized that we could go back to using text editors again as we didn’t have to manage tens of thousands of lines of XML and thousands of lines of code shortly after coming out of the gate.
We’re not (yet) actually using SwitchTower, but intend to in the future. We’ve currently got a database server running both Oracle and Postgres (SuSE enterprise Linux) for development and testing. We aren’t yet using Migrations either, mostly due to the fact that Migrations was coming along as we were figuring out how to minimize the database dependence in our SQL creation, drop, and load scripts. We took the same general algorithm we were using with our Java environment with Ant and brought it over to our Ruby environment with Rake. We have plenty of constraints, a few triggers, a couple of database-level functions, etc. I now believe I know how we would get the normal table stuff (probably 90% of what we have) to work with Migrations while integrating the oddball stuff into Migrations too. We’ll see how that works. Due to the weird constraints, though, we’ve got a fairly customized fixture-loading/-unloading regime going.
Why not reduce biweekly milestones to weekly?
Mostly because we want to reduce iteration workload variability, and we want to give the user community a chance to inspect a stable milestone application and get some feedback on changes before the milestone is gone too quickly.
By variability I mean that, given a single week for an iteration, it’s likely that someone will be gone for a day or two during that iteration, where over a couple of weeks people are liklely to be here for a good fraction of the iteration, moreso on average as the iteration count increases. We would like to be able to “theme” an iteration from time to time, and we’d like for developers to feel that they’ve been involved in every iteration—that they’ve not missed out on something important. Early, we’re still sharpening tools and learning a bit how we naturally fit together as a team, and how to use the framework and language to a high level of effectiveness. A longer milestone gives a sense that there will be some slack time during the milestone so it’s o.k. to order tasks, make some space, and stop to think about how things work, maybe to play a bit more with the tools and try things out. Experimenting with how one works with the team, how one can accomplish tasks, how one needs to order things to get them done by the milestone all seem to be advantageous for the developers now. We’re also feeling our way into how to estimate developer productivity and task size on the project, so we want to leave a little room for error early in the project as well. On the developer side we may well wish to tighten our milestones down the road.
On the user feedback side, one thing we wish to avoid is the all-too-common moving target syndrome: user pulls up the most recent version of the application and notices there is a problem with it (some bug, some misinterpreted feature/story, something that was ill-conceived, etc.), takes the time to put together succinct feedback, sends it in, gets a message back the next day from the development team saying, “Oh, yeah, we fixed that already.”—with one problem typically replaced the next day by a new problem in the application. We hope that our build and development process helps mitigate whack-a-mole bug hunting, but the perception in the user community that things move too fast to be able to give considered feedback is one that drives users away. We’re trying to fight that perception early by doing two things: (1) making nightly releases so that users can look at today’s version of the application, checking to see how things are evolving in near-real-time. We’re also making milestone releases and keeping them stable for the length of the milestone (two weeks here). Users can spend a while with the milestone system and give well-considered feedback. They can give fast feedback on the nightly system, and check themselves to see if a particular problem has already been fixed.
Some might ask, “well, why not make the milestones shorter but have the history of recent milestones available for users to peruse?” And the answer would be that, actually, that’s how we started out, and we discontinued the trend. First, we found that not only were users not really visiting the historical milestone apps, but that the presence of too many apps was a bit confusing to some. More importantly, we have to have a database running behind every milestone application (as well as the nightly). For a small application with no legacy data this tends to be a low burden—just write a script to populate new databases and run it with every release. We began our conversion process of our legacy data right behind the building of the new application. We have hundreds of thousands of staff & client demographics records alone, many with multiple phone numbers and addresses, plus all the stock data (zip code databases, e.g.)—and the data set will only get larger as we go forward —the client chart is next on the batting list and the client chart data is, um, large. Long story short—with users not digging back into the past that we can tell and data growing as we go forward, it seems wiser to us to save some resources and just keep the most recent milestone and nightly builds available. But, that motivates us to make the milestone builds last for long enough for the users to get a chance to look and play.
How do the clients requests come in to development? Are tickets raised by the client? Is the spec updated? Is the milestone data maintained along with the tickets? Are tickets claimed by devs or assigned? Are time estimates for tickets maintained and analysed?
As for the mechanics, we’re using Trac to manage milestones and tickets. Milestone data is kept along with tickets. We are not currently keeping time estimates for tickets, and we aren’t necessarily going to attempt to analyze time per ticket or task, though we may begin using an abstract “point”-type system for estimating. That remains to be seen. Currently we’re working from a backlog of gathered requirements on the core of the service, built up from experience with the past system, interviews with users in the field, and feedback through other channels (including the RT feedback system). We break the requirements into tickets that are as small as possible without them being laughable. We are currently mostly assigning tickets to developers in an attempt to minimize the tendency toward code ownership (though I think we’re not yet doing well in minimizing that tendency) and educating developers on the various skills they will definitely need through the project, such as writing models, views, controllers, unit tests, functional tests, invoking partials, debugging AJAX, etc. I would like to move over time to a model where developers are picking out tickets more on their own, and I’m watching to see how we might ease into that.
The organization has, prior to this project, had a fairly top-down and mandated structure, which we’ve had quite a bit of success loosening up, though there have been some growing pains. I don’t yet see a wall for us to hit going in this direction, but I’m trying not to change things faster than they appear to be changeable. We’ve gotten pervasive testing, continuous builds, more “agile” processes starting really from zero—but then again that may be the easiest way to start in some cases. We tend to introduce one new thing at a time (for instance we’ve really just started doing RDoc in thie most recent iteration) to keep the learning curve relatively shallow. We then expect the new techniques and processes that work to become part of normal development going forward. That seems to have worked out really well for us.
As for tickets, we have our trac system, which the development team uses. We also have an RT ticket tracking system that’s available for public use throughout the organization, and we’ve publicized it’s value and availability. We publish the completed actions list from trac without developers’ names with every milestone. What we’re shooting for is a no-holds-barred Trac environment where developers can do what they want and need to do to collaborate on the project, as well as a public feedback and ticketing system with progress feedback for the user community. We want to allow the individual developers to not feel “watched” by members of the user community who might be pushing for certain features, or who might want to insist that some developer isn’t working hard enough or fast enough on their pet features. We may relax things a bit down the road, but for now we’re being careful to insulate developers until we can establish a solid relationship with our (LARGE) user community.
That’s a round-about way of saying that we both manage tickets internally, and have users creating tickets. As users put tickets in RT we triage them and bring them into either Trac as real tickets, or into Subversion as part of the design information we use when working out our domain model. As we get further and further along, getting into highly specialized domain areas, we will have a much closer relationship between users and developers. For now, though, we’re getting the battleship up to travelling speed—getting good representatives of the user community, finding time in their schedules, etc.
Well, this has taken (literally) forever to write, so I’m going to finally publish this, whether or not it’s complete… I’ve got a mental backlog of about 5 short articles I want to hit—but I may be writing them from New York (a.k.a. The Holidays).
