18 January 2005

Mud's Tests goes on the record: We have no confidence in Bob Wyman's assertions in re "There are no delays in PubSub issuing information"

Creative Commons License

Must cite as "Mud's Tests" This work is licensed under a Creative Commons License.


Ref Ref

Summary

PubSub management has asserted there are no delays in PubSub providing information. They have not made a credible statement.

Users and developers relying on Bob Wyman's assertions do so at their own risk.

There are known interface issues which PubSub has not resolved. Users can get access to information that is in error.

There remain material problems with PubSub. There are factors causing delays. Either Bob Wyman is misleading the world, or he doesn't know the limits of his own design well enough to make such a statement.

Discussion

If you are a blogger or a developer who has had a problem with an XML feed, the problem may not be what you think it is. To date, the "word on the street" is that PubSub has no problems.

Users that "may have a problem with XML feeds" are incorrectly assuming the error lies with non- PubSub platforms. This is incorrect.

Bloggers and developers are creating new platforms to "work with" XML feeds on the incorrect notion that the issue is unrelated to PubSub. This may not be needed.

It appears PubSub's statements may lead some to believe that the "real problem" lies with other platforms. ]

  • Developers, unable to find the error have launched new platforms and aggregators to "better work with XML feeds" that the "current aggregators can't handle."

  • Also, blogs are being told, "There is no problem with PubSub, the problem is with the user. PubSub has no problems. The actual problem lies with the user or another platform."

  • Testers. Moreover, what is also happening is people, who may have problems getting PubSub content, are taking their time to compare PubSub results to other search tools. Bluntly, if you are using Technorati, or you work for someone like

    If you are working on a competing search system because "the subscriptions aren't working out," you might want to stay tuned.

    Apparently, the PubSub CEO doesn't understand his own system and has not adequately demonstrated that he has resolved all issues.

    Bob Wyman appears to have made an assertion that there is no delay. We ask that PubSub demonstrate this.

    Challenge for PubSub management. Show the world that PubSub has:

  • resolved all human-interface issues; resolved all coding problems that might interfere with a user's ability to access information in a timely manner;

  • that all problems associated with "timely delivery of the subscription content" have been completely resolved; and

  • and that all codes that users have access to are correct, valid, workable, and can be integrated into any platform without a problem.

    It appears as though what PubSub is saying is that if the user has a problem with a code, it is due to a user error or a problem with the particular XML-reader and has nothing to do with PubSub.

    Let's examine Bob Wyman's statement recently posted on the Jeremy Zawodny blog:


    Bob Wyman's Claims


    MyStack is an application that I built that consumes and reformats "XML Feeds" from PubSub. It stands outside PubSub and it gets data no faster then any other consumer of PubSub's data. Thus, there can be no difference between the time it takes to get something via MyStack and the time it takes to get it via any other consumer of PubSub XML feeds. There is one exception. The PubSub generated RSS and Atom files are only updated every 15 minutes. This is to discourage people from polling our feeds more often then 15 minutes. With that one exception, all PubSub XML feeds (REST, XMPP, SOAP or telnet) receive new results the instant that they are discovered by PubSub. "Delays", if any, can be measured in milliseconds or seconds at most.

    Admittedly, there will be some delay between when something is posted on a website and when PubSub finds it. If the site pings us, it usually only takes a few seconds or minutes before we fetch the site's feed and process it. However, if a site doesn't ping us, we'll simply scan their feeds on schedule -- 1, 3, or more hours depending on the site and its history. In any case, once PubSub actually find some data, it is matched in milliseconds. We don't store data, we don't index it. There is no delay in our system.

    bob wyman


    ------------------------------------------

    ------------------------------------------


    Wyman's Statements and Mud's Tests Questions


    Wyman: Thus, there can be no difference between the time it takes to get something via MyStack and the time it takes to get it via any other consumer of PubSub XML feeds.

    There is one exception. The PubSub generated RSS and Atom files are only updated every 15 minutes.

    Mud's Tests

    - Is this the only exception?

    - Is it impossible for there to be anything else which gets in the way of PubSub issuing reliable information in a timely manner?

    Wyman: This is to discourage people from polling our feeds more often then 15 minutes. With that one exception, all PubSub XML feeds (REST, XMPP, SOAP or telnet) receive new results the instant that they are discovered by PubSub. "Delays", if any, can be measured in milliseconds or seconds at most.

    Mud's Tests

    - Again, is this the only exception?

    - Is it impossible for there to be any delay [beyond 48 hours]?

    Wyman: Admittedly, there will be some delay between when something is posted on a website and when PubSub finds it.

    Mud's Tests

    - If there is a delay in PubSub issuing information, is it "only related to content" and "nothing to do with PubSub"?

    Wyman: If the site pings us, it usually only takes a few seconds or minutes before we fetch the site's feed and process it. However, if a site doesn't ping us, we'll simply scan their feeds on schedule -- 1, 3, or more hours depending on the site and its history. In any case, once PubSub actually find some data, it is matched in milliseconds.

    Mud's Tests

    - What method does PubSub use to ensure its "method of finding data" is accurate?

    - How are users provided with this information?

    - What guarantee does PubSub have that users are provided the correct information in the "message status reports" and "the information users are provided to access the information"?

    - What testing was done to ensure that all notification methods are accurate, reliable, and deliver content all subsription content in a timely manner?

    Wyman:
    We don't store data, we don't index it.

    Muds' Tests

    - What type of indexing is done on XML feeds stored?

    - What method is used to ensure your platform provides the correct information?
    - How does PubSub ensure that users are given the correct codes?

    - Have there been any reports of problems with the output codes from PubSub?

    Wyman
    There is no delay in our system.

    Mud's tests

    - Are there no reports of any problems with PubSub?

    - What test runs were done to demonstrate that the results are timely?

    - If users were to use the PubSub website, how can they be sure that the information they are given is correct?

    - What method is used to ensure the information users are provided is accurate and usable?

    - Does PubSub have any record of any user providing feedback on inability to generate, find, or get hits in their subscription?

    - Have all bugs associated with these lags been eliminated?

    - Is it PubSub's position that there is absolutely no way that any user could have any lag in getting their results through a PubSub subscription?

    - What guarantee is PubSub issuing: Unqualified and willing to stand by the representations about "there being absolutely no lags" and "there is no problem" and "nothing is wrong"?


    Discussion

    Based on the above questions, I have no confidence that the stress testing of the PubSub account was thorough. Moreover, I have no confidence that PubSub can issue an unqualified statement about the reliability of their platform.

    I will only believe PubSub has a credible platform if I read the following type of unqualified statement:

    "We have done a robust and thorough test of our platform. We have no reports of any delays. All errors are illusory. PubSub remains a reliable platform. Users can use all information from out platform and achieve the desired results. All codes we issue are reliable.

    "PubSub remains a superior product, with no errors that would interfere with any user's ability to get timely and accurate results.

    "There is no need to second guess our platform. All modes of using this platform have been thoroughly tested. There are no errors. Users can use the entire platform and find no error in achieving their results.

    "PubSub has been around for many months, and all bugs related to issuing codes, providing subscription content and ensuring the platform works have been corrected and there is no problem with the PubSub platform.

    "All users, using all means of accessing our data will have no problem getting data, nor will there be any problem getting access to our data and downloading and accessing subscription content."
    Well, let's hear it!

    Concern

    At this juncture, I have read no unqualified statement. I remain skeptical the PubSub platform has been thoroughly tested.

    Moreover, I have yet to read a guarantee that there is no coding problem, and that PubSub has proven beyond all reasonable doubt there all errors are gone, and there is absolutely nothing standing in the way between a user "using PubSub" and "getting timely subscription content."

    At this point I remain concerned there is a reasonable basis to believe that, lacking an unqualified statement in re PubSub's reliability, there remain:

  • error modes which would interfere with the user's ability to access information;

  • unchecked human-interface issues which have not been explored;

  • unresolved human interface issues which are not understood, have not been documented, and these issues were not adequately incorporated into the human-interface testing prior to release from beta; and

  • that valid reports of problems, delays, and interruptions have been ignored without solving the fundamental problem;

  • documented error reports have not been addressed, resolved, and satisfactorily resolved to ensure that users have unfettered access to the subscription information;

  • assertions that PubSub has thoroughly stress tested their website and platform; yet the test plans have failed to adequately ensure that all possible failure modes have been reconciled; and

  • problems getting access to the PubSub system, yet these roadblocks have not been understood, explored, and timely resolved in order to support an unqualified statement about PubSub's reliability.

    Summary

    PubSub has problems providing timely information. They have unresolved human interface issues.

  • The testing done on PubSub has not demonstrated that all failure modes related to "timely providing content" have been resolved.

  • PubSub has failed to remove all human-interface issues related to "timely providing information"

  • PubSub cannot guarantee that its site is providing users with reliable information to get timely access to information.

    Mud's Tests stands by these statements. If you have another view, feel free to state otherwise, demonstrate your tests have been thorough, and it is impossible for PubSub to be related to any problem, issue, failure mode, or problem that would interfere with users getting timely access to their subscription content.

    For those of you who can't read between the lines: Bob Wyman has asserted that "there is no delay" in PubSub providing content.

    He doesn't know the current capabilities of his own platform, nor can he demonstrate that all failure modes have been resolved and eliminated.

    There are material issues impacting PubSub's ability to ensure the end-user has accurate coding information to get timely access to the PubSub-generated content.

    If you are a blogger or developer who is unable to get access to PubSub data, you are not alone. At this juncture, if you have loaded information to your bookmarks, aggregators, or other search tools to find content and are having problems getting timely access to information, you are not alone.

  • Creative Commons License

    Must cite as "Mud's Tests" This work is licensed under a Creative Commons License.


    Ref Ref

    Summary

    PubSub management has asserted there are no delays in PubSub providing information. They have not made a credible statement.

    Users and developers relying on Bob Wyman's assertions do so at their own risk.

    There are known interface issues which PubSub has not resolved. Users can get access to information that is in error.

    There remain material problems with PubSub. There are factors causing delays. Either Bob Wyman is misleading the world, or he doesn't know the limits of his own design well enough to make such a statement.

    Discussion

    If you are a blogger or a developer who has had a problem with an XML feed, the problem may not be what you think it is. To date, the "word on the street" is that PubSub has no problems.

    Users that "may have a problem with XML feeds" are incorrectly assuming the error lies with non- PubSub platforms. This is incorrect.

    Bloggers and developers are creating new platforms to "work with" XML feeds on the incorrect notion that the issue is unrelated to PubSub. This may not be needed.

    It appears PubSub's statements may lead some to believe that the "real problem" lies with other platforms. ]

  • Developers, unable to find the error have launched new platforms and aggregators to "better work with XML feeds" that the "current aggregators can't handle."

  • Also, blogs are being told, "There is no problem with PubSub, the problem is with the user. PubSub has no problems. The actual problem lies with the user or another platform."

  • Testers. Moreover, what is also happening is people, who may have problems getting PubSub content, are taking their time to compare PubSub results to other search tools. Bluntly, if you are using Technorati, or you work for someone like

    If you are working on a competing search system because "the subscriptions aren't working out," you might want to stay tuned.

    Apparently, the PubSub CEO doesn't understand his own system and has not adequately demonstrated that he has resolved all issues.

    Bob Wyman appears to have made an assertion that there is no delay. We ask that PubSub demonstrate this.

    Challenge for PubSub management. Show the world that PubSub has:

  • resolved all human-interface issues; resolved all coding problems that might interfere with a user's ability to access information in a timely manner;

  • that all problems associated with "timely delivery of the subscription content" have been completely resolved; and

  • and that all codes that users have access to are correct, valid, workable, and can be integrated into any platform without a problem.

    It appears as though what PubSub is saying is that if the user has a problem with a code, it is due to a user error or a problem with the particular XML-reader and has nothing to do with PubSub.

    Let's examine Bob Wyman's statement recently posted on the Jeremy Zawodny blog:


    Bob Wyman's Claims


    MyStack is an application that I built that consumes and reformats "XML Feeds" from PubSub. It stands outside PubSub and it gets data no faster then any other consumer of PubSub's data. Thus, there can be no difference between the time it takes to get something via MyStack and the time it takes to get it via any other consumer of PubSub XML feeds. There is one exception. The PubSub generated RSS and Atom files are only updated every 15 minutes. This is to discourage people from polling our feeds more often then 15 minutes. With that one exception, all PubSub XML feeds (REST, XMPP, SOAP or telnet) receive new results the instant that they are discovered by PubSub. "Delays", if any, can be measured in milliseconds or seconds at most.

    Admittedly, there will be some delay between when something is posted on a website and when PubSub finds it. If the site pings us, it usually only takes a few seconds or minutes before we fetch the site's feed and process it. However, if a site doesn't ping us, we'll simply scan their feeds on schedule -- 1, 3, or more hours depending on the site and its history. In any case, once PubSub actually find some data, it is matched in milliseconds. We don't store data, we don't index it. There is no delay in our system.

    bob wyman


    ------------------------------------------

    ------------------------------------------


    Wyman's Statements and Mud's Tests Questions


    Wyman: Thus, there can be no difference between the time it takes to get something via MyStack and the time it takes to get it via any other consumer of PubSub XML feeds.

    There is one exception. The PubSub generated RSS and Atom files are only updated every 15 minutes.

    Mud's Tests

    - Is this the only exception?

    - Is it impossible for there to be anything else which gets in the way of PubSub issuing reliable information in a timely manner?

    Wyman: This is to discourage people from polling our feeds more often then 15 minutes. With that one exception, all PubSub XML feeds (REST, XMPP, SOAP or telnet) receive new results the instant that they are discovered by PubSub. "Delays", if any, can be measured in milliseconds or seconds at most.

    Mud's Tests

    - Again, is this the only exception?

    - Is it impossible for there to be any delay [beyond 48 hours]?

    Wyman: Admittedly, there will be some delay between when something is posted on a website and when PubSub finds it.

    Mud's Tests

    - If there is a delay in PubSub issuing information, is it "only related to content" and "nothing to do with PubSub"?

    Wyman: If the site pings us, it usually only takes a few seconds or minutes before we fetch the site's feed and process it. However, if a site doesn't ping us, we'll simply scan their feeds on schedule -- 1, 3, or more hours depending on the site and its history. In any case, once PubSub actually find some data, it is matched in milliseconds.

    Mud's Tests

    - What method does PubSub use to ensure its "method of finding data" is accurate?

    - How are users provided with this information?

    - What guarantee does PubSub have that users are provided the correct information in the "message status reports" and "the information users are provided to access the information"?

    - What testing was done to ensure that all notification methods are accurate, reliable, and deliver content all subsription content in a timely manner?

    Wyman:
    We don't store data, we don't index it.

    Muds' Tests

    - What type of indexing is done on XML feeds stored?

    - What method is used to ensure your platform provides the correct information?
    - How does PubSub ensure that users are given the correct codes?

    - Have there been any reports of problems with the output codes from PubSub?

    Wyman
    There is no delay in our system.

    Mud's tests

    - Are there no reports of any problems with PubSub?

    - What test runs were done to demonstrate that the results are timely?

    - If users were to use the PubSub website, how can they be sure that the information they are given is correct?

    - What method is used to ensure the information users are provided is accurate and usable?

    - Does PubSub have any record of any user providing feedback on inability to generate, find, or get hits in their subscription?

    - Have all bugs associated with these lags been eliminated?

    - Is it PubSub's position that there is absolutely no way that any user could have any lag in getting their results through a PubSub subscription?

    - What guarantee is PubSub issuing: Unqualified and willing to stand by the representations about "there being absolutely no lags" and "there is no problem" and "nothing is wrong"?


    Discussion

    Based on the above questions, I have no confidence that the stress testing of the PubSub account was thorough. Moreover, I have no confidence that PubSub can issue an unqualified statement about the reliability of their platform.

    I will only believe PubSub has a credible platform if I read the following type of unqualified statement:

    "We have done a robust and thorough test of our platform. We have no reports of any delays. All errors are illusory. PubSub remains a reliable platform. Users can use all information from out platform and achieve the desired results. All codes we issue are reliable.

    "PubSub remains a superior product, with no errors that would interfere with any user's ability to get timely and accurate results.

    "There is no need to second guess our platform. All modes of using this platform have been thoroughly tested. There are no errors. Users can use the entire platform and find no error in achieving their results.

    "PubSub has been around for many months, and all bugs related to issuing codes, providing subscription content and ensuring the platform works have been corrected and there is no problem with the PubSub platform.

    "All users, using all means of accessing our data will have no problem getting data, nor will there be any problem getting access to our data and downloading and accessing subscription content."
    Well, let's hear it!

    Concern

    At this juncture, I have read no unqualified statement. I remain skeptical the PubSub platform has been thoroughly tested.

    Moreover, I have yet to read a guarantee that there is no coding problem, and that PubSub has proven beyond all reasonable doubt there all errors are gone, and there is absolutely nothing standing in the way between a user "using PubSub" and "getting timely subscription content."

    At this point I remain concerned there is a reasonable basis to believe that, lacking an unqualified statement in re PubSub's reliability, there remain:

  • error modes which would interfere with the user's ability to access information;

  • unchecked human-interface issues which have not been explored;

  • unresolved human interface issues which are not understood, have not been documented, and these issues were not adequately incorporated into the human-interface testing prior to release from beta; and

  • that valid reports of problems, delays, and interruptions have been ignored without solving the fundamental problem;

  • documented error reports have not been addressed, resolved, and satisfactorily resolved to ensure that users have unfettered access to the subscription information;

  • assertions that PubSub has thoroughly stress tested their website and platform; yet the test plans have failed to adequately ensure that all possible failure modes have been reconciled; and

  • problems getting access to the PubSub system, yet these roadblocks have not been understood, explored, and timely resolved in order to support an unqualified statement about PubSub's reliability.

    Summary

    PubSub has problems providing timely information. They have unresolved human interface issues.

  • The testing done on PubSub has not demonstrated that all failure modes related to "timely providing content" have been resolved.

  • PubSub has failed to remove all human-interface issues related to "timely providing information"

  • PubSub cannot guarantee that its site is providing users with reliable information to get timely access to information.

    Mud's Tests stands by these statements. If you have another view, feel free to state otherwise, demonstrate your tests have been thorough, and it is impossible for PubSub to be related to any problem, issue, failure mode, or problem that would interfere with users getting timely access to their subscription content.

    For those of you who can't read between the lines: Bob Wyman has asserted that "there is no delay" in PubSub providing content.

    He doesn't know the current capabilities of his own platform, nor can he demonstrate that all failure modes have been resolved and eliminated.

    There are material issues impacting PubSub's ability to ensure the end-user has accurate coding information to get timely access to the PubSub-generated content.

    If you are a blogger or developer who is unable to get access to PubSub data, you are not alone. At this juncture, if you have loaded information to your bookmarks, aggregators, or other search tools to find content and are having problems getting timely access to information, you are not alone.

  • " />