Working at Working at WIDE An Internship Reflection Michael Wojcik (2010)
In the spring semester of 2009 I took an internship with the WIDE Research Center. I had been working with WIDE for some time, and the projects I was involved with during the internship were already in progress when the internship began (and continued after it), but that semester saw my most concentrated work with the center.
My work with WIDE fell into three major areas:
- Helping John Monberg with the development environment, process, and implementation of the Our Michigan Ave website, a community resource he began that semester as the primary project for his TC491 class.
- Development and management of the website for the Feminisms & Rhetorics 2009 conference website, including the site itself, the FemRhet conference database, and online applications for proposals, reviews, registration and payment, and conference administration.
- Work on the working environment at WIDE itself, particularly the software development tools and practices. This was an opportunity to consider how my professional experience in software development, and particularly my understanding of how development can be managed, could be translated to the requirements and culture of an academic research center.
This last area, though the most open-ended and arguably least-productive of the three (at least in terms of delivered "products"), is I think the most interesting, and the one I'll have the most to say about. As I want to consider all three, though, I'll expand on each in turn.
Our Michigan Ave
John Monberg started the Our Michigan Ave website at the beginning of 2009, as the major project for his TC491 course. OMA is a community forum for the Michigan Avenue Corridor area, a tax-capturing zone along Michigan Avenue. The site features user-generated content in various areas, particularly for people interested in improvements in the Corridor zone.
In order to build a site that would be more than a class project and could persist after the end of the semester, John designed the project around robust, progressive web development practices adopted from commercial website production. These included agile programming, where small feature teams that included members with assorted skills would work in short iterations on well-defined tasks, distributed source change management with Subversion, and the use of the Ruby on Rails framework as the main technology for the site.
WIDE provided the server space for the first iteration of the site, and since neither John nor the students had ever produced a major RoR-based site before, WIDE also offered significant technical support, mostly in the area of system administration: installing all the necessary technology and handling the initial configuration, and helping diagnose the occasional technical issue.
Jim Ridolfo was the chief sysadmin during the spring semester, but since he was finishing his degree and on the job market he gradually transitioned that position to Matt Katzenberger. In that position Jim and Matt laid most of the groundwork, such as allocating the virtual server for TC491 and installing Debian packages for components John needed such as Ruby.
In turn, I worked closely with John (joined on some occasions by Matt) with bootstrapping his development system. By the beginning of February we had the Subversion repository in place and a skeleton RoR-based website up and running. John's class was able to complete two full iterations of site development, and by the end of the semester the site was live and accruing content.
The OMA project was amazingly successful. A professional team would have done well to produce as feature-rich a site in the same timeframe, and John and his TC491 class (additional content was contributed by his undergraduate technical-writing students, and some community members) were not professional developers. They had no experience with creating a web site using these technologies, or with joint software development. They were part-time developers, with other classes and tasks to work on. And they lacked a dedicated IT organization to provide infrastructure and technical support. I'm very happy to have contributed to the project.
What's more, the project has proven that it can remain viable. Work continued on it in the fall in John's WRA415 (Digital Rhetoric) class, of which I was a member. I'm continuing to work on the site even now.
While I didn't anticipate spending a significant portion of my internship working on OMA, I don't regret it. It turned into an interesting example of how quality software with real utility can be developed by non-specialists in the writing classroom — the topic that John, Matt Penniman, and I will be presenting on at ATTW in March.
Feminisms & Rhetorics
Michigan State hosted the Feminisms & Rhetorics conference in October 2009, and WIDE was extensively involved, particularly on the technology end. I joined the team in late 2008 when we needed to get an online proposal-submission system in place quickly for the 15 December opening of the submission period. The system requirements had not been fully determined at that point, but we knew it included basic user profile and sign-in support, and forms for submitting proposals for individual, panel, roundtable, and alternative presentations. I had to develop both the front end (user interface) and back end (scripts and database), and integrate it into the existing conference website, which at that point contained preliminary information about the conference.
The initial system was thus in place before my internship at WIDE officially began. Since WIDE was hosting and maintaining the site, though, I continued to work on it right through the actual conference itself in October. And because this was a new software system, being developed for users who were approaching their tasks without prior experience, there was a great deal of additional development over that period.
The proposal system was followed by an online review system for ranking proposals, including a mechanism to randomly assign two reviewers to each proposal, and administration features for tracking the process of reviews and the current status of reviewed and unreviewed proposals. Then I added features for conference registration and payment, and additional administration facilities to work with that data.
Following a redesign of the site by Matt Katzenberger in the middle of the spring semester, I took over the static site content as well.
I can say confidently that I underestimated the time and effort that I would end up putting into the FemRhet website (the classic weakness of software developers). And as is also so often the case, many planned features (proxy payments, for example) were never implemented, and there are still some known bugs, particularly in the area of unusual payment situations. A conference is a fixed deadline; it doesn't permit any of the schedule slippage that's typical of most software projects.
However, the site was crucial in making the conference a success, and on the whole I'm pleased with it. Reviewers were quite happy with the review system, and I thought some of the administration tools were quite nice. In the end it achieved its purpose.
I do think it would be useful to consider adapting the FemRhet conference system for other conferences. Significant work would have to be done to make the software generic, rather than specific to this conference, and in an ideal world it would be extensively rewritten for better maintainability and extensibility. It does seem to me, though, that it is something WIDE might extract some additional value from, either by releasing it as an open-source contribution to the community, or actually selling it — perhaps as an online service.
The third component of my internship was the task Bill Hart-Davidson and I discussed when we originally organized it. Bill thought I could draw on my professional software development experience to help WIDE review and improve its processes for creating software. WIDE now has a number of software projects for external clients, as well as ongoing research projects. These efforts mostly share the same core technology, and the same development staff — primarily Mike McLeod, with occasional assistance from Bill and others.
Working on several projects with related products and limited resources usually poses a number of problems. There are issues of scheduling and allocating time and effort, of course. Complex dependencies and unanticipated interruptions (such as unplanned meetings with clients to dicuss new requirements) make it hard to estimate effort and meet deadlines. Questions of scope frequently arise: should a new feature be restricted to the project where it was required, or should it go into other projects? Developers find themselves duplicating effort, when they fix a bug in one version of the application and then have to make similar fixes in other versions.
We hoped, then, that I could work with Mike and Bill to implement tools and procedures that might help WIDE toward goals like:
- reduce duplicated effort by eliminating duplicate code in different flavors of the same software
- provide more internal information about the status and progress of software projects, with as little additional overhead as possible
- collect project information, including requirements, plans, status, history, and issues, in a central, easily-accessible repository
- reduce WIDE's exposure to risks such as data loss due to hardware failure and erroneous changes to the code that break existing features
- make it more feasible to bring on additional developers quickly when resources permit
Of course these are concerns for commercial software developers, so a lot of effort has been put into developing mechanisms to alleviate them. With Bill and Mike, I helped pilot three of them at WIDE, with some success.
Change management with Subversion
My first recommendation was to institute strong change-management protocols for source code, using the Subversion change-management technology. Subversion is an open-source distributed change management system for files. It maintains a repository on a central server, essentially a database of file data plus metadata such as the file's (current) name, location in the file system, and change history. Users have one or more local copies of parts of the repository on their own machines. They can modify files and then commit their changes (with a log message describing the intent of the change) to the repository; they can also update their local copies with the latest data from the repository.
Subversion also has extensive features for merging changes from multiple users, retrieving other versions, creating branches so that multiple versions of files exist in parallel, comparing files and versions, examining the change history, and so on.
Also, Subversion potentially solves two of the biggest problems with web application development:
- Subversion can easily be driven by a CGI script on the server to get the latest version of the code from a branch and deploy it into a directory in the server's hierarchy. That lets deployers easiliy commit code and the deploy it to the server for further testing, demos, production, etc.
- It's also easy to script Subversion so that different branches can be deployed to different URLs on the server, which lets multiple versions run side-by-side. That makes it easy for multiple developers to write and test code simultaneously without interfering with one another.
Since Subversion can greatly simplify the problems of working on several versions of the same application code, using it could have immediate value for WIDE, where we have several applications built around essentially common core code.
More importantly, change management is critical to mature software development, because it's a vital safety net — it's the reset button for mistakes, it effectively backs up source against the failure of a developer's system, it provides access to code when a developer is absent, and so on.
We created a Subversion repository for some of the WIDE sources, and began integrating it into Mike's development tools and processes. This integration is still somewhat awkward and needs to be revisited.
Project planning and tracking with XPlanner
One area where we thought some technological help might be useful was in project planning and tracking. WIDE regularly has a number of fairly long-running software development projects, and historically the WIDE directors and staff have tracked them using a variety of tools and approaches. Since those aren't integrated, there's no single repository of project data.
XPlanner is an open-source web-based software project management system. It was originally designed specifically for projects using Extreme Programming, another agile methodology, but it's suitable for use with Scrum as well. It can be used to track data about a project in several genres: user stories (requirements), defect reports, tasks, iterations, releases, and so on. It records information about task and project status, and can be used to track effort (by having developers include elapsed-time information in status updates). It supports annotations and attachments, and is fairly extensible.
XPlanner provides numerous reports, including analyses and "dashboard" summaries. Used consistently, it accumulates a lot of historical information — useful for things like post-project analysis (especially interesting in an academic and grant-driven environment) and estimating future projects.
I set up an initial XPlanner installation for WIDE, and we did populate it with some projects and tasks. This never really got off the ground, though, as I explain in the recommendations section below.
Agile development with Scrum
Scrum is a software development methodology, part of the agile development movement. (Scrum actually originated outside software development, but it's found wide acceptance there.) As a professional Scrum user at Micro Focus, I feel it would be useful for WIDE as well. Its reliance on self-organizing teams relieves the WIDE directors of unproductive management duties, while the focus on small, well-defined tasks makes it easier to schedule work on multiple projects. And Scrum achieves a lot of process transparency with very little effort, so project tracking is easy and lightweight.
As is typical with agile methodologies, we're trying a phased approach with Scrum, introducing a feature or two at a time and gradually adopting more of the structure into WIDE's software-development processes. We began with the idea of the "daily scrum", a short meeting where feature-team members (ie, the developers) explain what they did since the previous scrum, what they have planned until the next one, and whether they've encountered any impediments to doing what they've committed to.
Because of schedule constraints, we've actually been running scrums less frequently than every day — typically two or three times a week. And rather than having them as face-to-face meetings, we've been conducting them in Google Chat. This has the added advantage of keeping a shared transcript of all of the scrums (more historical data).
Scrum has other key features, such as iterations, user stories, and the backlog, which we should try to begin implementing at WIDE.
Results and recommendations
It's proven difficult to consistently maintain these protocols. That's not surprising. With only two or three people participating (depending on Bill's availability), we have no margin for absences from scrums, for example — if Mike or I has a meeting at the scheduled scrum time, there won't be enough people at the scrum to have a conversation. Between the academic calendar, my job, conferences, etc, there are frequently weeks when it's simply impossible for Mike and I to coordinate even a quick scrum chat.
Similarly, I think everyone finds it difficult to set aside time to try to work with XPlanner. There's little benefit to a project planning and tracking mechanism until everyone involved (including people in "staff" and "management" roles, even when those may be the same people) is reading and updating the information in it, and even then it takes some time to develop a "critical mass" of information. (Typically, a planning tool has to be in use for some months before it has significant historical information that the project participants can't easily remember themselves, for example.)
Subversion is potentially more immediately useful, for the reasons discussed above. Of the three areas, this is the most critical to try to keep continuously active. Used properly, a robust change management system protects developers from outside accidents and from themselves, and it inherently provides some of the project history that's so important both for management tasks (such as planning and review) and simply to save developers time when they need to recall details from some work in the past.
WIDE should continue to use Scrum. Mike and I have recently been having relatively regular scrums (still using Google Chat). We have to commit to this, and then begin to integrate other aspects of Scrum, such as a well-defined backlog, iterations, etc. The benefits of Scrum for dealing with the development situation at WIDE, where there are significantly more projects than developers and management resources are severely constrained, far outweigh the relatively small overhead of following a Scrum protocol. It can be a quite loose and casual version of Scrum — with the small and friendly environment of WIDE, we don't need to worry about time-boxed meetings, official product owners, and so forth. But the three key ingredients of scrum meetings, a backlog, and iterations are, I think, invaluable.
This will require some kind of planning mechanism to keep the backlog, track iterations, and provide a dashboard of some sort so the (already overcommitted) WIDE directors can quickly assess project health. I consider that in the next section.
The experiment with XPlanner was useful, but I think Mike and I agree that XPlanner itself is not an ideal tool for WIDE. While it's relatively easy to use, and as a web application fairly convenient, it's too divorced from everyone's normal work patterns and habits. We could make a conscious effort to update it more frequently, but it would always be an afterthought or second-class task.
Not long ago we discussed the idea of using WIDE's Garden technology as a project planning and tracking system. Though Garden does not have many project-management features per se, its ability to organize information along multiple axes should make it possible to build various project-management views. And it seems like its workstreaming features would be a much lighter-weight mechanism for tracking project progress, since team members could update the project status with a quick note, tweet, etc.
For 2010, I think we should pursue this, perhaps starting a pilot in the second half of the spring sememster.
Subversion (or some other change-management system, but I recommend sticking with what we already have in place and understand) is indispensible. The first priority for the WIDE software development environment should be to refine our use of Subversion.
Mike has had some luck with Subversion, but there are still some issues with integrating it into the tools he regularly uses for development. Also, we need to make it more convenient for Mike and potentially other WIDE developers to test particular code branches.
Mike and I have discussed some areas I can work on in the first quarter of 2010 to make Subversion easier and more useful for him.
Unlike Our Michigan Ave and FemRhet, the Development Environment project had no hard deadlines or milestone deliverables to point to as evidence of success. Its components were at best only partly and provisionally implemented; XPlanner never really saw any serious use, and Subversion and Scrum have been used only intermittently.
Nonetheless, I think this project is crucial to the long-term success of WIDE as a software development organization. Obviously that will only ever be one part of WIDE's mission, and it's conceivable that role will someday diminish in importance. For now, though, software development is a major part of WIDE's contribution to the field and beyond, and a key to WIDE's financial viability as well. That makes WIDE a professional software development center, and it needs to use professional software development practices.
It should be possible to institute practices that make WIDE more effective and more efficient, reducing the supervisory work while producing better-quality software faster.
The greatest obstacle, I believe, is committing to the regular effort to pursue those practices until they become part of our regular work habits. It is natural to defer process improvements in the face of deadlines, meetings, and other time-bound commitments, to say nothing of the distractions of implementing an interesting new feature or tracking down a stubborn bug. (Programming as a field is a textbook case of the conflict between the pleasure and reality principles, to put it in Freudian terms.)
I would urge, then, that everyone at WIDE regularly set aside some time — even a half-hour monthly meeting would probably suffice — to review progress on process improvements and plan for the next set of changes. If I had only one recommendation to make for continuing to improve software development at WIDE, this would be it. I believe WIDE has the skills and dedication to become a state-of-the-art software development organization, within the scope of the Center's mission and projects; but that will require making some room for working specifically toward that goal.
Finally, a few words on how this internship served me as a scholar and student. During 2009 I feel I refined and tightened my academic focus, even while taking coursework in a variety of areas. In this past year I tied a number of disparate projects together into two related threads, which I'll call computer-assisted rhetoric and computational rhetoric. My work at WIDE was relevant to both.
Under the banner of computer-assisted rhetoric I'm including all of the uses of information technology that we normally engage with in the field or practice of rhetoric: everything from "computers and writing" to digital rhetoric to new media to the everyday use of computers and digital communications. Obviously my OMA and FemRhet website work for WIDE fit this category, since those projects revolved around the development of new-media artifacts. And both of those sites are themselves focussed on computer-assisted rhetoric in one form or another; OMA is a forum for user-generated content, and FemRhet existed to enable a rhetoric conference and vet proposals for it.
More broadly, since essentially all of my WIDE work was related to the development of some sort of writing software, either directly or through enabling infrastructure, the internship let me exercise skills in this area and contribute to, and interact with, the community. For all these projects I had to solicit requirements from users and discuss possible solutions, and investigate approaches I might not have chosen for myself.
By computational rhetoric, on the other hand, I mean specifically using information technology to implement rhetorical theories as embedded in algorithms and heuristics. I began working in this area a couple of years ago in my first WIDE-related project, which led to the Estimating Ethos demonstration. Subsequent projects in this area include my Visual Rhetoric Greasemonkey Scripts from 2008 and the more recent Speech Made Visible project.
None of my internship work at WIDE was directly connected to computational rhetoric. However, I'm increasingly convinced that this is a direction which digital rhetoric / techrhet, and cognate fields such as software studies, need to explore. And for that to happen, a solid software development environment is indispensible. Prototypes and proof-of-concept demonstrations are all very well; but the real future of computational rhetoric will have to include powerful, broadly useful, reliable, maintainable applications. Garden, Swarm, and other WIDE software technologies are moving in this direction. I believe the experience I gained with software development environments in academia through WIDE is typical of the work that needs to be done if the potential of computational rhetoric is to be realized.