Monthly Archives: June 2005

The Delphi Development Process: Handling Bugs

In this post I’ll talk about how we handle manage bugs in the product. I think the best way to discuss this would be to simply follow the lifecycle of a bug and try to illustrate how the process works along the way.

A Bug is Born

Well, not really, but you get the idea a QA engineer finds, reproduces, narrows down, prioritizes and finally logs a bug. Our internal bug tracking system (called RAID) which btw, is a Delphi application we’ve maintained/upgraded/improved since before D1 shipped, is the cornerstone of our bug tracking efforts and not only works internally but is also used for logging bugs by Field Testers. RAID is incredibly robust and I’d argue that it has probably been the single most reliable application that our entire team has used since Delphi 1. The server has roughly 200,000 bugs (covering several different products not just Delphi) and runs on a dual processor PIII 500MHz with 512MB of RAM with over 1 million records in the audit table running on Interbase 6.5. RAID would be a great topic to blog on but it’s not something I’ll focus on here. One last remark before I move on, on more than one occasion an R&D engineer has left the Delphi team only to call back later asking if there was any way they could get their hands on RAID for their new company/project. Ok, back on topic, QA fills in the following fields for each bug:

  • Product
  • Version
  • Area
  • Bug Type
  • Bug Serverity
  • Recommendation
  • Description
  • Steps
  • Found by
  • Tester
  • Developer
  • Locale

(I’ll see if I can post a screenshot of RAID)

Many of these fields are automatically filled in as the bug is entered so the QA engineer really just needs to focus on the Type, Severity, Recommendation, Description and Steps. Once the bug is logged the QA engineer may contact the assigned R&D engineer (depending on Severity) and mention the new bug. One of R&D’s core responsibilities is to review their open bugs and set an “Action” field which indicates what the impact of that bug is from an R&D perspective. The Action field corresponds to QA’s Recommendation field giving a final assessment from both sides as to the impact of the bug in question.

Bug Council

Once the bug has been prioritized by QA and R&D it’s up to the R&D engineer to schedule time to work on the bug unless it requires immediate attention. Throughout the development cycle the R&D, QA and Program managers review the bugs logged against the product during “bug councils” where they’ll contact the R&D/QA engineers to discuss (typically) the highest prioritized bugs. Bug Council is where the decisions are made to defer bugs, raise their priority or otherwise draw attention to a specific issue that the team feels should be addressed.

As R&D fixes bugs they are marked as fixed in RAID and checked into the the current branch using StarTeam. Change Requests in StarTeam are created to track which checkins were made to address specific bugs and allows us to better track which checkins were related to bug fixing versus feature work.

Back to QA

Once a bug is marked as fixed the “ball” lands back in QA’s court to verify the fix. If the bug is verified as fixed then the bug is marked as “verified” in RAID. At this point, the bug is considered to be resolved but not yet closed which will occur when QA makes a final regression test pass which usually occurs prior to any major milestone. If the bug doesn’t verify and isn’t truly fixed then it is reopend and the process starts over. Since our team is fairly small QA is usually in close contact with R&D and typically keep each other in sync regarding fixing/verifying bugs.

I’m thinking that I could break up these posts so that they’re smaller and therefore (perhaps) more frequent, thoughts?

Some answers: The Delphi Development Process

Well, thanks to people’s comments I can see that there is quite
a bit of interest surrounding the our Development Process not to mention the
fact that I have more than a few questions to answer. So, let’s get to it:

How about about posting some screenshots of you designing an
average IDE form, with a bit of code showing?

I’ll see what I can do
although I’ll need to ask around a bit before doing that.

Very interesting, particularly this part: “QA’s patented
automated testing framework is written in Delphi” I’ve never heard about a
patented automated testing framework before, and would love to hear some

There is a lot to be said about our testing framework and it is
IMO definately cool technology. Perhaps I can get one of our current QA people
to blog about it and I’ll ask around. I’ve been out of QA since 1999 and I’m
sure there have been many changes to the framework since but if I can’t seem to
generate any interest I’ll certainly consider blogging about it.

A side note to Julian thanking him for his correction, yeah that was
embarrassing. I seriously need a preview button in .Text. To solve that I’m
writing this post using the latest internal build of Delphi and editing using
the HTML designer. That way I can see the stylesheets (including my custom
changes) and edit live right from the Tag Editor at the bottom of the designer.
It works great! I think I’ve found a new way to write posts thanks Julian!

I have a question about the organisation of such a big project:
how was handled the build of Kylix and C++Builder ? Both derive from Delphi but
are slightly differents.

I know the first release of C++Builder was a
separate branch of Development where a few engineers simply took the Delphi code
base and started turning it into C++Builder. Eventually, the cost of maintaining
two trees was too much the two products were integrated into a single codebase
although I’m not exactly sure when C++Builder was merged back into the Delphi
branch. When building the Architect SKU on a developer machine we currently get
Delphi Win32, Delphi .NET, C#Builder and C++Builder all in one shot. Our Linux
products are on separate branches and yes there was lots of merging changes
between the codebases since a lot of the code is the same but significant
portions were different enough to require another branch. I’ll continue to
discuss how development works on a project of this size it’s pretty cool when
you think about it.

acquired a software product called Genitor a while back. That asset was part of
the StarBase acquisition. Do you guys use that tool internally at all?

simple answer the Delphi team does not use Genitor. To be totally honest I went
right to the Borland website looking for it and unless it’s well hidden I can’t
see any information about it. That said, I’m not in product marketing and just
because I’m not familiar with this product shouldn’t reflect at all on the
actual state of the product itself. So, please get in contact with someone
either in marketing or sales for more information.

Steve, do you do code review? How do you do it?

although like most teams I suspect we could be doing better in this area and in
fact I have a number of things I plan on blogging about related to this area so
stay tuned.

How many people are involved in ‘the making of Delphi’?

This is a good question for which I don’t know the answer. Let me
explain… As I mentioned before, the Borland Developer Studio (BDS) is under
construction pretty much around the clock these days given the various locations
of Borland Developers around the world. Having said that I simply don’t know the
total number of people working on the team including R&D, QA, Doc,
Integration. I would say that it’s probably smaller than you’d think. BDS
includes integration with StarTeam, ECO and Together so do they count?

In what language is the delphi compiler (dcc32) written in? I
know the RTL and the IDE are written in delphi, but what about the compiler? Is
it written in delphi, too?

The Delphi compiler is written in C and
developed using C++Builder. At least, that’s what I see up on Danny’s screen
whenever I go into his office. 🙂 Our compiler technology is also leveraged by
our C++ compiler.

“dogfooding” blecchhh!

I agree although I’ve certainly
heard it enough to believe that it’s familar enough nomenclature to be widely
understood. FWIW, you’d hardly ever hear that term here at Borland largely
because it’s not actually necessary. When the tool is written in itself there
really isn’t a requirement to force yourself to use the app you develop because
it’s the only way for you to get work done.

Why should this come as any kind of suprise, that Borland use
Borland tools to generate Borland tools?

For Borland customers it
shouldn’t. In fact, I think most Delphi customers would be shocked if Delphi
weren’t written in Delphi but how about asking the same thing about a VB user.
What I can say is that I’m not surprised that VB isn’t written in VB.

Did you expect them to be using Microsoft tools?

Actually, we do use
MS tools. In fact, we use the C# compiler for a number of pieces of Delphi for
.NET. We don’t really look at it as some sort of war with Microsoft. We use the
tools that allow us to get our job done and thankfully most of the time it’s a
Borland tool.

The interesting question is, are the Delphi developers using
the current alpha/beta version for development of the new version, or the
previous version?

Absolutely. Typically, when a developer finds a stable
build they’ll likely stick with it for anywhere from a few days to a couple of
weeks depending on what sort of work their doing. One benefit of using the
latest build is that you get to use the cool new features that are coming in and
influence how they work and give immediate feedback.

Other things I can think of to cover are:

Whoa, Dan, I’ll
get there but it’s going to take some time but thanks for filling my idea box!

More development related questions

Here are a few more questions that I thought were worth another post.

Are the Standard/Pro/Architect versions of the Delphi source code separated by having
$IFDEF’s in the code?

We use a combination of techniques for handling the
different SKU’s but for the most part IFDEF’s are not how it’s done. The IDE is
broken into numerous Delphi packages (see your Delphi bin directory *.bpl) which
include a set of “core packages” that make up the main part of the IDE.
Additional packages are developed which contain various features and we’re
extremely careful to ensure that packages don’t inadvertantly link to packages
outside of their designated SKU. In addition to separate packages there is also
code that controls when a package is allowed to be used with a particular SKU.

When you build Delphi is there lots of hints and warnings?

There are both hints and warnings generated during the build however,
whether they would be considered “a lot” on a percentage basis I can’t say but I
seriously doubt it. I don’t believe it’s a high percentage just based on the
frequency of checkins I see with comments like “cleaning up hints/warnings” so
I’d say we stay on top of the situation. Of course, some warnings (like
“specific to a platform”) are acceptible so they wouldn’t really count if we had
to actually calculate some sort of percentage. As a team I’d say we take hints
and warnings pretty seriously as should all developers.

Any third-party components used in the Delphi Source?

Yes. I’ll start a list here off the top of my head and add to it later
if/when another R&D engineer points something out (in no particular order):

  • Mike Lischke’s Virtual Treeview
    which is used in (and there may be other places):

    • Project Manager
    • Structure Pane
    • Message view
    • Refactoring Window
  • JclDebug exception
    message dialog
  • Indy which is used in a number of places like the ASP.NET Deployment
  • Microsoft‘s MSHTML
    control for the Welcome page, HTML/ASP.NET designer does that count?
  • W3C’s HTML Tidy

all that comes to mind right now, it’s a pretty short list and at least one
reason is that we want/need to avoid licenses that could get us into trouble.

[Updated: 06/24/2005] Corrected the name of the Jcl exception
stack dialog. Thanks Hallvard!

The Delphi R&D Development Process

I was talking to Dan Miser
awhile back and he mentioned that it might be cool to blog about how the Delphi R&D team actually
develops Delphi itself. I’ve been on the team a long time, well before D1
shipped and I’ve seen a lot of changes to our development process over the
years, some gradual and others not. Right now, as we work on the next release I
think we’re in a very interesting phase of the evolution of our development
process and I’ll write about a few specific areas and mention some of the tools
we’re using. Perhaps a few other R&D folks will chime in and offer some
additional “color commentary”.

Building the IDE

You might wonder, does Borland actually use the tools they produce
for their own development efforts? The answer is a resounding “YES!”. The Delphi
team in particular has always been a huge proponent of what Microsofties call
” which is the act of using what you produce. Not only is the IDE
itself written in Delphi, QA’s patented automated
testing framework is written in Delphi, our Bug Tracking system (RAID) both the
SOAP client and server are written in Delphi (running against an Interbase
server) and prior to StarTeam our version control client was an application
written and maintained by the Delphi R&D team. In fact, there are numerous
utilities used in our build process also written in Delphi as well as countless
little personal utilities that nearly everyone in QA and R&D have developed
for themselves over the years.

Anyway, the product is built using a typical “make” process from the command
line which supports mutliple targets allowing us to build the multitude of SKU‘s. Once built day
to day development work is performed using a “working” build of the IDE and a
convienent “make work” target updates this local “working” build. Currently, the
build takes about 20 minutes for a debug developer build on my 2.4GHz machine
(with 1GB of RAM although that’s not really critical when building from the
command line) and is made up of over 25,000 files.

Delivery to Integration and QA

We have an automated build server which sends out email notifications to
everyone who checked in since the last successful build in the event the build
breaks. Since we now have developers around the globe working on the build it’s
incredibly important to keep it building successfully and any build breaks are
usually tracked down very quickly. Once a the build is successful it is then
promoted to the Integration team where they build the product into a complete
installation for delivery to QA. Most of the time QA tests the product using the
Architect SKU since it has all the “bells and whistles” and as we near certain
milestones they’ll run various SKU tests to ensure that all the different
versions include all the necessary parts.

What’s New and Interesting

To me the “new and interesting” parts of our
development process which is still evolving is how we’re leveraging the
technology that Borland has acquired over the past few years, namely StarTeam
and Caliber. Since the acquisitions we’ve been slowly working both products
(change management and requirements management) and processes into our, let’s
call it, experience laden development process. We’ve made some great strides and
I think we’ve come a long way since Anders Hejlsberg’s Notepad based list of
work items with dashes and “X’s” and Gary Whizin’s spiral notebook.

Using CaliberRM to develop and maintain our list of product requirements has
given us a much better process of scoping product development and assigning work
to individual engineers. I think it’s measurably improved the transparency of
our development process for our management team so that they can more
effectively guide product development. Additionally, CaliberRM has provided a
great place for us to capture requirements for features we don’t have time for
in the current release allowing us to maintain a list of good ideas from one
version to the next. If only we ship a product with all the features we
currently have scoped in Caliber. 🙂

Then there is StarTeam which not only allowed us to free up valuable Delphi
R&D resources used maintaining an old tool but has also given us a bunch of
features we’d all been wanting for years. It’s nice to have a full development
team working on a tool that your team uses everyday. Besides I don’t think the
StarTeam folks could ask for better “dog fooding” of their code so Borland get’s
double benefit from the Delphi team using it.

That’s all for now. Let me know if you find this at all interesting and I’d
be glad to share more about the inner workings of our team

Localized Delphi 2005 Update 2 and Update 3 have signed off

I just got this email message from Chris Pattinson and I know there are many Delphi customers who will be
interested in this information and so I asked Chris permission to post it here:

“It is my pleasure to announce the international team has completed work on
D2005 Update 2 and Update 3. Integration, R&D, QA, and our localization
offices pulled together to sign off over 20 SKU’s 17 days ahead of schedule,
including updating Japanese Trial to Update 2, and German and French to Update

Full steam ahead to Dexter! Except for Leo who will enjoy a very well
deserved vacation to the Philippines for almost a month.

Thanks to everyone for outstanding teamwork, this was an amazing
accomplishment especially given all the distractions in the past couple months!

Best regards,
Chris Pattinson
International Manager, Windows Projects”