07 January 2005

Is beta-testing as effective as it could be?

Interesting to hear the developer's perspectives on beta testing, marketing and the end user. Thanks Rick Stahl and Josh Ledgard. I sense you put alot of time and care into your work. Thanks!

My overall comment: Set your boundaries. If you want to read a blog or publish your comments, you decide "how you react" to the feedback.

There is a firewall. Developers make things, and marketing is about launching the product. Sometimes developers need to stay in the dungeon; then again, sometimes the marketing and beta-testers need to stay out of the dungeon as well. Ref Ref ;-)

  • Comments on Rick Strahl's WebLog:

    Why developers should stay out of marketing, this is good:

    "the computer press will pick this up and immediately flood technical articles with this new and exciting technology"
    -- You would prefer the world not to know about the product? LOL "Awareness of the product" = sales, and cash to continue to pay developers to write code.

    The nature of software development without a firm software baseline:
    "Never is there really time to absorb the current technology fully, expand and get a solid platform. Instead the platform is in constant flux."
    So? The end-user will still buy "what you have," so long as it is marketed.

    Whether the "non-finished"-product is a developer's tool, end-user's don't care. Kind of like me complaining about the plumber's "non-mature-new-hammer." Not my problem. That's why I pay him to fix things and why I pay you to design things.

    A flawed process with a flawed baseline.
    "As I see it there are number of serious flaws in the beta process itself as well."
    One of the funny things about "beta for software" is that it is accepted as "normal product development process." Show me a "beta-Shuttle" or a "beta-Cadillac." Beta = a free ride to avoid accountability; if you do not want to have "customer feedback during beta," don't call it beta. But the customer feedback is still there. Whether you choose to listen to it is another question.

    They have the tools to listen, why aren't they using filters?
    "While a public beta provides potentially many more testing sites, it also increases the signal to noise ratio dramatically with unqualified feedback or feedback that is so obvious or previously handled that it has no bearing."
    There is something called a "filter." If you do not want to hear the comments, you can turn off your RSS-feed. Deal with it.

    The blogs were supposed to pinpoint quality content.
    "adding a public group makes the sample group so big that it will become impossible for Microsoft to actually deal with all incoming requests in anything "
    So, are we saying that despite the "tools to manage IT," there are not tools to manage feedback? Wow! If this "inability to manage feedback" is true, looks like the "blogs are great for feedback"-crowd are blowing smoke.

    In theory, you build a product to solve problems. It appears they're using beta to figure out how to package something that is not linked to a firm user-need. "Feedback provides a better understanding of user scenarios"

    The goal of developing products is to solve problems, not impress. When developers forget the problem they are trying to solve, and get lost in "making the customer happy," developers are going to miss the forest for the trees.
    "There are too many features that are cool on the surface"
    is another way of saying, "We're not solving problems, we're trying to give someone something they already want." That's not innovation, but staying stuck.

    Blogs, they are a good concept.
    "Often times coming up with bug scenarios is not easy and having a public place to discuss these issues can spawn discussion by other members involved in addition to the dedicated testers at Microsoft."
    Good to hear they're talking about feedback, but why stop with "Ask how the consume responds; and shift to discerning what will actually solve the end-user's unstated problem or need.

  • Comments on Josh Ledgard's comments :
    This is the first time I've heard someone say it's a bad thing that we are finally giving people a chance to play with a product early enough to effect the end result
    I didn't read it that way, in fact he was pretty clear on the opposite. Ah, the early hour. The mind does often miss things. ;-)

    Free input

    End-users have end-user perspectives, not "test group" perspectives."
    We made a choice and that choice was to lean on the more public groups because the information was relevant to everyone.
    End user's don't want specs, they want solutions

    This comment appears to miss the point on designing. As an end-user, I do not care about "the spec." I care about solving a problem.
    These customers should have the opportunity to be reading our specs for Orcas soon after they have been written and ever before any code has been produced.
    If you have to give me a spec to understand it, your product is worthless to me. I want it to be self-explanatory and something that is simple. Developers may like specs; end-users like to solve problems.

    Bug management

    Looks like there are poor methods to let all users know "what the reported bugs are." The issue isn't "we have too many bug reports," but that the bug-report-writers have great confidence in their ability to find something that nobody else has found.
    If the firefox team can handle their bug influx from 10 million customers in bugzilla, then I imagine we'll be able to scale to our Visual Studio developers with the right tool. I make no claims that the Feedback Center is perfect today or that it will be tomorrow.
    Perhaps the solution is to teach the public how to quickly discern whether their bug-report has already been provided. Look at Mozilla-Firefox in how they do it.

    Lack of a well-publicized and efficient means to take comments

    What is most surprising is that "beta" isn't a new development cycle in software development. It's not as though "this is a new concept." Yet, today it appears as though some are surprised by the feedback, and have no mechanism to efficiently handle the comments.
    Again, this is not to say that newsgroups and forums do not have a purpose. I just don't believe that purpose is to be a public (or even large private) bug database.
    So, all this time and what are the developers waiting for: Marketing and the end-user to create a system that will communicate end-user comments back to the developer? LOL

    Again, this is a core milestone within software development; that IT as a whole has "matured" to this stage appears that many "lessons learned" and "ideas and solutions" how to expedite and streamline the end-user to developer feedback system have not really been formalized.

    Perhaps this is the same as saying, "We'll take care of that later during the sustainment period..." but it never comes, since personnel get offloaded just as the product launches and manning is descoped at the very time these "lessons learned" and "continuity issues" and "cross talks" could most benefit.

    Overall

    If the "current beta testing process" doesn't appear to work, there's nothing stopping the developers in changing it. That is, you're not waiting for the end-user to calibrate you are you? ;-)

    Rick and Josh, thanks for your comments and perspective. Good luck in your development efforts!
  • Interesting to hear the developer's perspectives on beta testing, marketing and the end user. Thanks Rick Stahl and Josh Ledgard. I sense you put alot of time and care into your work. Thanks!

    My overall comment: Set your boundaries. If you want to read a blog or publish your comments, you decide "how you react" to the feedback.

    There is a firewall. Developers make things, and marketing is about launching the product. Sometimes developers need to stay in the dungeon; then again, sometimes the marketing and beta-testers need to stay out of the dungeon as well. Ref Ref ;-)

  • Comments on Rick Strahl's WebLog:

    Why developers should stay out of marketing, this is good:

    "the computer press will pick this up and immediately flood technical articles with this new and exciting technology"
    -- You would prefer the world not to know about the product? LOL "Awareness of the product" = sales, and cash to continue to pay developers to write code.

    The nature of software development without a firm software baseline:
    "Never is there really time to absorb the current technology fully, expand and get a solid platform. Instead the platform is in constant flux."
    So? The end-user will still buy "what you have," so long as it is marketed.

    Whether the "non-finished"-product is a developer's tool, end-user's don't care. Kind of like me complaining about the plumber's "non-mature-new-hammer." Not my problem. That's why I pay him to fix things and why I pay you to design things.

    A flawed process with a flawed baseline.
    "As I see it there are number of serious flaws in the beta process itself as well."
    One of the funny things about "beta for software" is that it is accepted as "normal product development process." Show me a "beta-Shuttle" or a "beta-Cadillac." Beta = a free ride to avoid accountability; if you do not want to have "customer feedback during beta," don't call it beta. But the customer feedback is still there. Whether you choose to listen to it is another question.

    They have the tools to listen, why aren't they using filters?
    "While a public beta provides potentially many more testing sites, it also increases the signal to noise ratio dramatically with unqualified feedback or feedback that is so obvious or previously handled that it has no bearing."
    There is something called a "filter." If you do not want to hear the comments, you can turn off your RSS-feed. Deal with it.

    The blogs were supposed to pinpoint quality content.
    "adding a public group makes the sample group so big that it will become impossible for Microsoft to actually deal with all incoming requests in anything "
    So, are we saying that despite the "tools to manage IT," there are not tools to manage feedback? Wow! If this "inability to manage feedback" is true, looks like the "blogs are great for feedback"-crowd are blowing smoke.

    In theory, you build a product to solve problems. It appears they're using beta to figure out how to package something that is not linked to a firm user-need. "Feedback provides a better understanding of user scenarios"

    The goal of developing products is to solve problems, not impress. When developers forget the problem they are trying to solve, and get lost in "making the customer happy," developers are going to miss the forest for the trees.
    "There are too many features that are cool on the surface"
    is another way of saying, "We're not solving problems, we're trying to give someone something they already want." That's not innovation, but staying stuck.

    Blogs, they are a good concept.
    "Often times coming up with bug scenarios is not easy and having a public place to discuss these issues can spawn discussion by other members involved in addition to the dedicated testers at Microsoft."
    Good to hear they're talking about feedback, but why stop with "Ask how the consume responds; and shift to discerning what will actually solve the end-user's unstated problem or need.

  • Comments on Josh Ledgard's comments :
    This is the first time I've heard someone say it's a bad thing that we are finally giving people a chance to play with a product early enough to effect the end result
    I didn't read it that way, in fact he was pretty clear on the opposite. Ah, the early hour. The mind does often miss things. ;-)

    Free input

    End-users have end-user perspectives, not "test group" perspectives."
    We made a choice and that choice was to lean on the more public groups because the information was relevant to everyone.
    End user's don't want specs, they want solutions

    This comment appears to miss the point on designing. As an end-user, I do not care about "the spec." I care about solving a problem.
    These customers should have the opportunity to be reading our specs for Orcas soon after they have been written and ever before any code has been produced.
    If you have to give me a spec to understand it, your product is worthless to me. I want it to be self-explanatory and something that is simple. Developers may like specs; end-users like to solve problems.

    Bug management

    Looks like there are poor methods to let all users know "what the reported bugs are." The issue isn't "we have too many bug reports," but that the bug-report-writers have great confidence in their ability to find something that nobody else has found.
    If the firefox team can handle their bug influx from 10 million customers in bugzilla, then I imagine we'll be able to scale to our Visual Studio developers with the right tool. I make no claims that the Feedback Center is perfect today or that it will be tomorrow.
    Perhaps the solution is to teach the public how to quickly discern whether their bug-report has already been provided. Look at Mozilla-Firefox in how they do it.

    Lack of a well-publicized and efficient means to take comments

    What is most surprising is that "beta" isn't a new development cycle in software development. It's not as though "this is a new concept." Yet, today it appears as though some are surprised by the feedback, and have no mechanism to efficiently handle the comments.
    Again, this is not to say that newsgroups and forums do not have a purpose. I just don't believe that purpose is to be a public (or even large private) bug database.
    So, all this time and what are the developers waiting for: Marketing and the end-user to create a system that will communicate end-user comments back to the developer? LOL

    Again, this is a core milestone within software development; that IT as a whole has "matured" to this stage appears that many "lessons learned" and "ideas and solutions" how to expedite and streamline the end-user to developer feedback system have not really been formalized.

    Perhaps this is the same as saying, "We'll take care of that later during the sustainment period..." but it never comes, since personnel get offloaded just as the product launches and manning is descoped at the very time these "lessons learned" and "continuity issues" and "cross talks" could most benefit.

    Overall

    If the "current beta testing process" doesn't appear to work, there's nothing stopping the developers in changing it. That is, you're not waiting for the end-user to calibrate you are you? ;-)

    Rick and Josh, thanks for your comments and perspective. Good luck in your development efforts!
    " />