20 January 2005

XML Multi-Channel troubleshooting -- Generic methodology to troubleshoot inconsistent search results stemming from same-XML subscription requests

Creative Commons License

Cite as, "Mud's Tests." This work is licensed under a Creative Commons License.



Disclaimer

This is an OPINION only. If you do not understand that this is an OPINION and not statement of fact, then stop reading.



Note

For purposes of clarification, this speculative piece should be construed only as an informal opinion, not legal advice, nor a statement of fact. Nor should this information be used as a basis to make any specific findings of fact about the nature of the discussed error mode or the specific failure mode that appears to exist.


Overview

The PubSub platform has had apparent internal consistency problem. Part of the platform reports valid subscription results, while other portions of the platform report no content found.

This is problematic in that the inconsistent results stem from common search strings. The purpose of this note is to explore the possible failure modes driving this inconsistent output; explore theories related to the possible explanations for the failure mode; outline a proposed methodology to explore alternatives to the drivers behind this inconsistent reporting; then discuss the implications of these apparent conditions in light of the FeedMesh Concept.

It is hoped that the results of this discussion and speculative piece might assist in pinpointing the actual problem driving the inconsistent outputs from PubSub; and also apply the apparent lessons learned to integration issues in the FeedMesh concept.

FeedMesh

FeedMesh is proposed to be a mechanism that will integrate many platforms and ensure that XML feeds can be timely provided, published, reported, and provided to end-users. This is a gross simplification of the concept.

Yet, we propose that FeedMesh will only succeed if integration issues within a single platform can be resolved. If we are to integrate many platforms, then the lessons learned from an “apparent inability to integrate in a single platform” might serve as useful lessons in approaching the FeedMesh concept.

This is to say, that even if we propose theories and methodologies that do not specifically address the PubSub issue, we might advance the cause of FeedMesh by outlining for the public and developers the apparent ground rules and integration issues related to FeedMesh. In other words, if PubSub’s apparent inconsistent reporting is related to an internal integration issue, the public and developers would benefit if we understand whether this is what is actually occurring, and develop some guiding principles to aid the developers in stress testing the various parts that will be need to be integrated.

In all likelihood the developers have already thought of solutions to the issues raised. However, it is hoped that the larger XML community to include bloggers and content readers might gain a perspective of how things are.

Again, in all likelihood this perspective could be very far from the mark. But could serve as a useful guide for developers and other consultants to calibrate how they approach the end user, and what information they provide and assumptions they make when developing and deploying products for wider use.

I would like to thank PubSub in their being the catalyst for this thinking. I imagine that a solution is very close at hand and the current delays will understandably be soon a thing of the past, water under the XML bridge.

Sections

This note first begins with a discussion of a generic methodology to trouble shoot a recurring error mode; then explores several theories related to the PubSub apparent error mode; then concludes with a discussion of the lessons learned that might be applied to FeedMesh.

In the end, we hope to create a set of metrics that the public might use to evaluate whether FeedMesh is meeting their needs; and also create some common boundaries for what FeedMesh needs to both solve and incorporate during the design and development process.

Generic Methodology

This section of the note outlines a crude and imprecise characterization of the PubSub platform. This discussion should not be used as a basis to specifically evaluate PubSub’s detailed code.

Rather, it is hoped that the “perspective of PubSub” be used only for discussion only in that it outlines a framework and coding steps that in all probability have no relationship to the detailed software codes.

This is to say that we will simply outline generic steps that PubSub appears to do so as to provide a starting point for a discussion of the methodology. Again, this methodology is to service two purposes: To both assist developers in trouble shooting the specific PubSub delay-error-mode; and also in creating a framework other platforms may choose to apply in situations where they are unable to solve what might otherwise be a very simple coding issue.

In short, the intent of this section is to share a number of new perspectives on a situation that appears to have exhausted all available resources and the error mode cannot be reconciled or resolved. This is another way of saying, “We never expect to need to do this, but just in case … we’ve got these ideas that might spark us to think of something new.”


Methodology Overview


This section will outline a methodology to both troubleshoot PubSub and apply this framework to FeedMesh. The methodology hopes to inter alia:

  • Trouble shoot multi-reporting channels.
  • Conduct checks for internal consistency
  • Validate approval checks
  • Ensure “test OK message-status reports” are valid
  • Fill in holes in code commands
  • Identify code-holes not patched, yet still report complete


  • Definitions

    Clearly, the above terms are non-standard. I shall go through the terms to explain their usefulness in the proposed methodology.

    Multi-reporting channels This is a system where parallel reporting efforts occur at the same time, or different times. A system that generates similar outputs, but through different channels and reporting routes. A single platform like PubSub has more than one mechanism to report subscriptions. Thus, it is a multi-reporting channel platform.

    Checks for internal consistency We check for internal consistency in a multi-channel platform to ensure that our processes are consistent. This is to say that when we have multiple different methods to achieve the same results [return and deliver subscription content], that before we provide the end-use with that data, there needs to be an internal check to ensure our various reporting channels are working not only correctly, but that they are reporting the results in the same way.

    When we check for internal consistency, what we are doing is making sure that the search results from search-stream 1 [derived from one reporting channel] are reasonably related to the results from search-stream 2 [derived from a second reporting channel]. This is not to say that the content and outputs need to match; nor is it to say that we have a problem if one search string has slightly different subscription-content as another similar search.

    Rather, the point of the internal consistency check is to allow each section of the multi-reporting channel the opportunity to challenge, check, and compare their operations, tests, and outputs with the other channels. In other words, before issuing a final subscription report, ideally what happens is the various sections of the platform do a check to ensure that the way they are going to present the data, and the results are consistent.

    An error would report if one subscription channel were reporting “no content found” while a second subscription channel was reporting valid content and a full subscription-completion.

    When we check for internal consistency, what we are asking the platform to do is to compare the subscription-results derived from the different methods of request, and check whether the outputs are consistent, not just in content, but in how they were processed, tested, and approved for release.

    Validate approval checks. This term means to ensure that the series of tests used when “checking for internal consistency” are consistently operating in the same manner. In other words, this is a step whereby the platform, before publication and delivery of content, will then check its own approvals to make sure each subscription-provider is doing the same things as each other. Again, if one subscription has had “done to it” different tests than another, then this would indicate something is not performing as expected and further inquiry is needed.

    The purpose of validating the approval checks is to simply ensure that the checks we do on each search string are then aggregated, compared, and each different reporting channel can say, “We all agree that we are all doing the same thing in response to this subscription-request; and we all agree that our steps, tests, and content have arrived at our results in a consistent manner.” This is not a check of the different reporting channel’s content, nor is it a method to say, “Well, this subscription channel reports X, Y, Z content; but the other reporting channel reports A, B, C so there is a problem. Rather the purpose of this step is merely an internal check to ensure that the various reporting channels are checking each other to say, “Yes, we are all doing the same things; and our coding is working as expected; and yes, we are sure that the tests we did in one section of the platform are the same steps done elsewhere.

    What this step does is ensure that if there is a problem with one set of codes with one set of checks applied to one of the subscription channels, that a second set of procedures similarly applied to a different set of reporting channels also are consistently coded. Clearly, this check is not needed if both reporting channels rely on the same end-tests.

    However, in situations like PubSub where it appears as though each reporting channel does different things in response to a given request, it might be useful if there was a mechanism to ensure that the different reporting channels are actually using the same types of code. Specifically, one failure mode that might explain the different results is if one reporting channel within PubSub uses one set of code-procedures that are linked with one type of subscription-request; while another subscription request relies on similar code that is embedded in a subroutine.

    At this point, what we might discern is that if there is one failure mode that is unique to only one reporting channel, then what might be going on is that there is an embedded set of commands and tests related to one reporting channel; and this code has been slightly adjusted and is not exactly matching what the other reporting channel relies on elsewhere.

    Again, this is pure speculation. We make no claims that this is what is happening. We simply use this to illustrate the concept, “Validating approval checks.”

    Test OK messages are valid We now turn to the test result message. This is a term that says, “We’ve done a test, the test reports OK; and the test is based on statusing this type of information located in X-location. In those situations where we assume a test is conducted correctly, we assume the actual data used to test and generate the status message is valid. However, there can be situations where the input data used for the test reports one status that is at odds with real status.

    Suppose we have a prepatory step prior to a test that gathers data; suppose we also have an unknown coding problem that would normally conduct a test on this data and say, “If we have this result, our test status is Y.”

    However, if the problem is that because of a coding problem we have meshed two lines of code, and one part of the test is not done, but the second part still reports the result associated with a “test OK message,” then we’ll never actually conduct a valid test, but our test result will be stored in a location that is processed on the assumption that that test actually executed and this is a valid result.

    With respect to PubSub what could be happening is that there is a test mode that conducts a review of the status of the platform; that test was designed and included in the platform correct. However, there is the possibility that during post-integration testing, an update to that section of the code was updated with a cut and paste, and the new code placed over it had an error. However, when we do the check of that code, the error-mode of the code is outside what we expect; and the test still appears to execute correctly, and there is a result that gets reported, and that result is consistent with what we expect.

    However, here’s the rub. Suppose we have a test-tool that is outside our primary platform. Unknown to our coders, the test-team has created a test-platform based on X, Y, Z assumptions, yet at the same time our actual platform conducts a test that no longer actually complies with those assumptions. This is to say that the requirements that we initially planned for in the specification were one thing; what we designed and created met those specifications; and then what we assumed in the test-tool of “what would happen” did not match exactly with how the error in code actually executed the command.

    In other words, suppose we have two sets of steps: One is the actual code used to deliver the subscription for channel one; at the same time the platform has a second search mode that reports a similar search result, but there is no internal check to makes sure we actually did the searches in the same way. On top of this, suppose there is an error in the code in only one of the channels, while the other reports fine; and at the same time there is not crosscheck between channels to compare the processes and results.

    The next step that would happen during the normal processing is both channels would report their results. Fine. Each has different content. That is expected. But what is not expected is if the test-platform used to evaluate that code is based on one set of assumptions about “how the test will execute”, but one of the reporting channels actually executes something else.

    Normally, we would hope that the test-platform that is outside the PubSub host would catch the inconsistency. That is, if the test-platform is not related to the internal error within PubSub.

    This is another way of saying: If PubSub has two reporting channels that are executing correctly, but one of them executes only part of a test, or the test as designed fails to execute a command, but still reports “OK status,” then what could happen is the test-platform [designed to test based on initial assumptions] could take the results from both and say, “We have valid output” according to the assumptions, but not according to what actually executed.

    In other words, if the test-platform is designed on the basis of assumptions that rely on the code to work on the primary platform, but is not designed to check the inconsistencies between channels, then there would be no signal for the test-platform that the processes were working differently. And if there was an embedded code that was incorrectly reporting status as “OK” [because of a coding problem], there would be nothing that would go in and say, “We’ve got this type of process-test occurring in reporting channel 1, but something else in reporting channel 2” … because the test platform’s code and assumptions also assume that “what it is testing” matches what it was designed to do.

    However, if channel one has an embedded code problem, is only reporting “status OK’ in response to errors, then a similarly designed test-platform would also never think to go back and say, “How did the status-OK message get generated” and “did the processes that actually executed” match across all the channels?

    In my view, what could be the problem with PubSub is the test-platform was designed on the assumption that certain steps would or would not occur; but it does not take into consideration a set of circumstances where “a test that is reporting valid is actually reporting invalid information, but appears to work” It appears there is no test for the external test-platform to ensure that the assumptions the test-platform is using are matching the assumptions to all the processes used in all the reporting channels.

    In other words, suppose we focus on 2 reporting channels. If reporting channel one is executive fine and returning results; and reporting channel two is also reporting fine [but is actually coded wrong, reports a problem, but that problem is reported as OK (for whatever reason)], then there would be no signal to the test platform that there is an internal consistency problem between reporting channels one and two that needs to be explained.

    This would explain why all indications are that the reporting channels are fine: The external test-platform continues to report, “Thins are working as designed.” Which is true. All the channels are reporting as coded, but it appears there is no check by the external test-platform to make sure each individual step is matched across platforms; and that all tests that report “OK” are actually OK. Moreover, it does not appear as though the external test-platform goes back into the PubSub system during operation to sample the various reporting channels and say, “We have one here, and a second here…how do the results compare in terms of test messages; actual steps executed; and the series of steps taken to arrive at the outcome.”

    Clearly, PubSub has a series of steps that generate in the various reporting channels different subscription content. That is not the issue. The issue becomes: How do we explain why a robust, mature, and well acclaimed system can this late in the game have multiple channels reporting different results, yet all indicators are that things are working fine.

    I can think of a couple of options:

  • The external test platform as designed is working as designed: To test the steps. However, the problem could be that the steps are reporting OK, but are actually doing the opposite because of a coding issue whereby one step of the test gets skipped, but the output still reports OK, and the test done on that section of the code similarly makes the same error because the code was copied from the designed platform, and integrated into the test-checks.

  • Another failure mode could be that PubSub does not check across a single subscription and compare the multi-channels. This is to say that the system as designed was always based on the assumption that the code, once tested and deployed, would always execute as designed: Consistently. However, there was no method to ensure that during deployment, data was collected to validate that these initial design assumptions were valid and continuing. In other words: If there was always the assumption that each channel would correctly report status, and that the tests would determine whether the output was valid or not [but there was no method to ensure that a coding error would trigger a signal in situations where a test fails but still reports the desired result, albeit wrong], there would be no reason for anyone to design something that would apparently re-do a function that was already tested as correctly execute. That is a waste of time.

    But let’s go into the netherworld. Suppose we have multi-channels established whereby each section was independently developed, then brought together under one umbrella, and everything tests out OK; but during a critical post-integration phase, a single syntax error occurred, covered up the process-test, and the test that would otherwise report a problem, failed to execute; and at the same time the “returned result” matched the other channel status, so from the external-test platform perspective the common results match the common inputs, so things must be fine.

    This is another way of saying “Two flaws within a single multi-channel cancelled each other out, reported a valid result that the test-platform similarly reported ‘good to go,’ all the while there was no detailed check of the individual commands to say, ‘Yes, each step flawlessly executed and compared with the other channels in all respects.’"

    Rather, because there was no error, there was no need to conduct a detailed check; the detailed check may be linked with the assumption that the detailed check will only be needed if there is a difference. But if there is no reported difference, then the external platform never gets the signal to go in and say, “We need to compare the details.” Again, if everything is reporting fine. There’s no reason to have a detailed step within the external test-platform that is going to compare detailed steps that “should” be the same.

    Again, if we have two errors occurring [coding problem and the test fails to execute; but output is invalid, yet is stored, reported, and interpreted as valid] then there would be no signal for the system to do detailed checks; nor look into the detailed steps; nor ever be prompted to look at the individual steps and compare: What was planned, what actually happened; what results were expected; how was the test results stored; and how did the result get translated into a next step; more broadly because these steps never occurred, there would be no signal to the external test-platform that there was ever an issue.

    Without having the ability to check across the same search string, but compare the different processes of the different channels.

    Fill in holes in code commands This term means finding problems and fixing them. Suppose we have a test step on a one of our channels. And further suppose we have a requirement to do something such as a test. Further suppose that that test, as designed, does the job. Further suppose that we have a problem during post-integration testing, and a quick update is made. Further suppose that there is no post-update check to make sure that our various channels continue to operate as originally designed: They appear to, but there is an embedded coding problem that incorrectly reports “no problem”, and there is no catalyst to look at the details to find the real step that did not uniformly execute.

    When there is a test with a simple command, we expect the test to execute. But if there is a coding error and two failure modes, then this is a leap from where we are, to where we want to go, without actually executing the command as designed-intended; it only produces a result that is meaningless, but desired: “Things are OK.”

    At that point where the test executes incorrectly, but reports “OK” we have a hole. Something was not done.

    Another example is when there is a requirement to execute a command, and we get a favorable report, but the test output is actually incorrect … yet the favorable report fails to trigger the needed detailed check … That is also a hole: We have a problem that is hidden, something should be getting tested, but is not. In short a “hole in the code” is when there is a needed check, but that check does not occur because of an incorrect assumption; or a double-error problem; or a failure of the external-platform to trigger a needed detailed test.

    When we “fill in the holes in the code” what we are doing is looking for these breakdowns, and then re-running the coding effort from square one. We fill in the hole by pretending we are an individual packet of information, and tracing the process millisecond by millisecond. Those holes, when they appear, can be very small. But under the right situations, a slight coding error can create a false report, but the information could be stored incorrectly, thereby signaling “we need not look further,” when in fact, the real problem is: We have a hole that the system, as designed, was not designed to detect on its own, nor given instructions of “what else to check to compare to highlight this disconnect” nor any instructions on how to execute the needed detailed tests to check the different detailed steps.

    Risk code holes not patched yet still report complete This idea is simply one of summarizing the above failure mode. This is another way of saying, “We have a known platform, a desired design, and a specific task…that has a bug, but all the signals are: Things are fine.

    The risk that “code is not patched but still reports OK” can be traced to the multi-error problem; combined with the test-platform that does not look at details when given a message, “everything is fine.”

    In a situation where we have multi-channels reporting the same status [even though they actually executed the command differently] this would contribute to this problem of “we have a hole, but no indication of a hole, so do nothing. Green light. Go.”


    Review


    We’ve gone over some made-up terms to define possible failure modes. We’ve also gone over some hypothetical failure modes within PubSub that would illustrate these concepts. Again, we’re not saying that PubSub is suffering from any one of these failure modes; only that something related to the above situations could be going on.

    The next step is to reconstruct the PubSub platform. Again, we’re not going to focus on what is going on with PubSub, but merely define the general steps that PubSub appears to go through, and then assign to each step a series of questions and failure modes that might spark some additional thinking and exploration.

    Non-Standard Characterization of PubSub

    For purposes of discussion only, I will construct a theoretical model of PubSub. This approach will be useful in both incorporating the above concepts in the failure modes, but more importantly to narrow in on the range of possible failure modes that appear to have eluded understanding.

    Again, to emphasize, the following categories are not intended to represent what PubSub does on the detailed level. Only to service as a generic outline of the things that occur. Clearly, the gaps in this discussion will serve to highlight other areas deserving further study and understanding.

    Six Internal functions and an external oversight function

    The external oversight function is simply the external test-platform. The six section of PubSub will be defined as PubSubX.

    Let us consider the actual functions that PubSub does and compare these with PubSubX. PubSub takes a subscription request, crunches the data, spits out a subscription return.

    PubSubX has more details.


    PubSubX

    Request

    Step 1: Request is the request where search commands are inputted into the system. Step 1 is the conversion phase.

    XML Feed Content

    Step 2: Access -- PubSubX access the XML content, looks into the data available, and searches.

    XML Feed URI

    Step 3: Code -- PubSub X creates an XML feed that will extract from the pool of XML feeds the data matching the request.

    Display

    Step 4: Result -- PubSub X achieves a result and displays it within the PubSubX account. At this juncture, the information has not been presented to the subscriber; rather, PubSubX merely stores the information, does checks on it, and validates that the result meets the objective.

    Conversion

    Step 5: Translate -- PubSubX takes the data that has met the search string goal, and translates it from raw code, into something that is readable.

    Report

    Step 6: Present -- PubSub X then packages the results into something the end-user can read.


    OK: Dig into the details

    At this point of the analysis we will dive into each of the 6 PubSubX sections, propose questions for further analysis, and identify a general framework for analyzing an XML platform.

    Let’s continue the exploration of PubSubX with the hopes of identifying some testing and evaluation to done on each component. We hope to dovetail each step with the above definitions so that developers and management can assess whether such a framework is useful in exploring the error modes.

    Let’s start with each step of our 6-phase construct of PubSubX. Let us also generalize our approach to be one that hoes to identify why there would be multi-channels doing different things: One search channel doing one thing and working as planned; while another is doing something else and providing no result, but is reporting “Operating OK.”

    1. Conversion: Take a valid search and changes it into something to throw at the XML content.

    Step 1 is where we take a valid input, a search term, and converts it into something that will get slapped against the available content. Notice that we are assuming the simple PubSub instructions are followed, and that the search terms across multi-channels are the same. If there are any errors, the error lies with PubSubX, not with the user.

    Let us explore the possible failure modes that occur at this phase of the PubSubX conversion from subscription requests to results. Again, we are focusing exclusively what PubSubX is doing:

  • Is the search terminology as entered correctly converted?

    One failure mode between PubSubX multi-channels is that the different channels could be translating the subscription request and search terms differently. One multi-channel could be correctly converting the search request, while the second channel makes an error, but this error is not caught, and there is no step to check or compare the multi-channels after the search strings have been converted.

  • How is it shown there is a valid request to PubSubX?

    When we take a valid search-subscription request, what we are doing is simply typing in words. PubSubX does the rest. What are transparent to the user are the internal checks. Part of this crunching inside PubSubX.

    But what is done is prior to PubSubX doing its magic is that it must first ensure the term is recognized and is valid.

    Suppose a user enters a valid word, but PubSubX doesn't *’t recognize the word – is the end-user given a proper signal to change the word; or does PubSubX take the information, even though there is a problem, and apply it to the multi-channels, even through there is a problem?

    Have the multi-channels been configured differently so that they process this incoming “invalid search request” differently: One of the channels takes the information crunches it, and produces a valid report: “Nothing.” While the other channel, converts it to something that PubSubX can use, and produces a valid result?

    At this point, the issue becomes to trace how the different multi-channels process the information in parallel, and then compare the results at each step. In my view, it is incorrect to run each search individually through a single channel and then evaluate the merits of those steps on their own. What needs to happen is to run multi-searches along each channel at the same time; then freeze-frame the status of that content, compare the status, and identify what is going on at each phase.

    If we have a valid search going into PubSubX, there must be a system that says, “We have something that is workable-usable”. Thus, there must be a mechanism that does this check; does a test; conducts the test; stores the result; compares that result to what is planned/expected; and then stores the test-result somewhere.

    At this phase, what could be happening in PubSubX is that one of the tests has not executed correctly, but is still reporting a valid result. Also, what could be happening is although the request was actually invalid, there was generated a valid test result. Or what could be happening is that there was an invalid request that is never identified, shown, compared to baseline data; or is never compared with the first step of the other multi-channel.

    Again, a simple solution to this problem would be to introduce a cross-channel checking system that ensures each step is working as planned. If this would slow PubSubX down too much, then one approach would be to do some data sampling using an external test platform that takes a parallel approach to the multi-channel; move away from testing the outputs along a single channel, and focus on each phase of PubSubX to see how the different multi-channels operate.

    Another approach would be to focus on the test platform and evaluate how it was designed with respect to this phase. Again, if the test platform goes linearly from search input to output, but never compares the results, tests, and progress of PubSubX along the data stream, there may be no way that PubSubX would know there is a problem, or that something unusual was happening along one of the channels.

    If the test platform is linear-checking only, but could be enhanced if it took a cross-channel approach at phases, it clearly has to be established how much time, effort would be required to essentially rotate the matrix of analysis 90 degrees from a linear approach to a cross-channel approach.

    If PubSubX plans to be around, the tradeoff at this point becomes how much growth does the platform expect; and will it be cheaper to rotate the matrix of analysis 90-degrees now, and then as content and volume increases it can more quickly find and detect and correct error modes; or are the benefits of having phased-checking meaningless given the actual design of PubSubX.

    Clearly, given PubSub’s development background and life, how PubSub actually works could already match the envisioned solution to address PubSubX’s issues. This is another way of saying: Those guys at PubSub are smart developers, and probably have already though of this and may already doing this, making this moot.

    The question for PubSubX becomes: What mechanism exists within the platform at the conversion phase to ensure when PubSubX converts from a valid request to something PubSubX can work with, how does PubSub x know the search is valid; how is this result demonstrated; is the way the information is presented match the actual status; is there a chance that the presented status as “valid” could be the result of two-errors from an actually incorrect input string; or is there never a search done; or is the result of this phase never established, compared, or demonstrated that the status is something that is baselined, tracked as the search progresses linearly along the channel?

  • Is the search request actually valid?

  • Is the invalid search allowed to enter along one channel, but not the other?

  • Can a valid search in one channel produce output, while a valid search in another channel be converted to something that is invalid, but not known to PubSubX?

  • Do the two channels have different ways of converting the inputted terms into “something else”?

  • Can an invalid request be entered, and each channel convert them differently, but there is no intermediate check to ensure that the conversion was consistent, uniform, and matches?

    Let’s consider the notifications to the end-user at this conversion phase. If there is a problem with a term entered, how is the information presented to the end user to change it?

  • Can an invalid search term be entered, accepted, and no error message provided?

  • Can a term that is invalid get handled differently along the channel?

  • Does each channel work to the search string differently, but all fail to report “we worked on it this way and have an error, but the other channel did something else and found something, so we all agree because one of us worked, we all worked”?

  • Are the channels allowing progress from one channel to override error messages from another channel at an intermediate step?

  • Are error-modes that “should signal a change” not occurring because of a universal accept-command that assumes terms can be converted, but they are later converted differently along different channels during subsequent tests, matching, and publication?

    Step 2: The method PubSubX access information

    PubSubX has a defined method to access the XML information.

    Let’s now consider the same PubSubX model at the second step of the process. At this point, PubSubX takes the converted request and accesses the XML content. The objective of this phase is to access the XML feeds, climb through the architecture, and conduct the search. PubSubX has a very simple task compared to the actual PubSub.

    At this point, we ask several questions related to the possible differences between the two channels and the inputs: Something that PubSubX is doing interferes with the process. One channel is working, the other is not.

    To re-iterate: PubSubX has a valid input. PubSubX is now doing something differently to each multi-channel, and interferes with the ability to consistently provide the same types of output: One channel is doing something the other is not; one is conducting a test, returning a message, or producing a result internally that does not match the other, but the external test-platform does not detect the subtle difference.

  • Does PubSubX have 2 different ways to search based on different parts of PubSubX?

  • Are coded instructions related to one channel unique to that channel?

  • Do all channels dip into a common set of search parameters and instructions?

  • Have instructions and detailed steps been farmed out to different segments of the PubSubX platform based on assumptions about parsing that do not play out?

  • Have instructions for the search for each multi-channel been farmed out, then re-integrated into a common manner, but they do not match the way the test-platform is testing them?

  • Could PubSubX create two different search routines from the same set of input parameters; yet when the results are combined, the differences are not compared with each other, and the overriding positive results from one is assumed to match the results from another, but there is no method using the test-platform to compare the individual steps and ensure that the output and progress is uniform and matching?

  • Does PubSubX have different approaches to searching based on different a different channel selected?

  • Does an internal switch define the search parameters in terms that PubSubX understands, but do not translate into consistent results, so PubSubX makes a default decision, discards the “no return” as a “non-response”, but converts that output as an “still waiting” for one channel; while it interprets for another channel a “no waiting” as a prompt to summarize data, adjust the search, or reorder the priority in the other channel?

  • Are there certain conditions and parameters about what will constitute a “valid hit” or a “valid subscription” for one channel, yet these parameters are assumed to not exist or be relevant with a second channel?

  • Are assumptions embedded in each channel which assume that the channel’s content-display is unique, therefore, this justifies conducting different tests and examinations and conversions at the outset?

  • Are last stages of PubSubX [step 6] driving earlier decisions about the size of the display, which are back-channeling their inputs and parameters into what constitutes an acceptable range of information to identify, search, review and reports as valid?

  • Are there different commands within the tool driving PubSubX to search one way with one part of the tool, but another way with another part of the tool?

  • Is there no intermediate check prior to searching to validate that each channel has a uniform set of assumptions about the nature of the search to be conducted?

    Step 3: XML Feed Code and Commands

    PubSubX now has a new phase. This is the first stage of packaging

    This is where PubSubX has a valid search result, and a completed search routine. This stage of the operation is focused on gathering these valid subscription results into a single data feed.

    The error modes along this step are in how the data is aggregated from the XML pool and pushed into a single data stream prior to holding.

  • Once the PubSubX completes the search, how does the XML feed URI produced… actually link and connect to the reported results?

    There is the possibility that there is a disconnect between what PubSubX is finding, and how PubSubX is actually identifying that “found count”.

    If the reference data used to identify the mailbox or storage location of that data does not match what PubSubX has initially been energized to look for, PubSubX could, in fact, report “we have found something within milliseconds,” but what was found was never retrievable as the identifying code where it was stored does not match the actual code used to retrieve the data from that location.

    The question becomes: What would possibly drive PubSubX to report data from a search quickly, but then lose track of this data; or store it with one code that doesn't *’t match the actual code used to retrieve the information.

    Let’s consider some perplexing possibilities:

    Clearly, earlier problems from earlier phases along the multi-channel could be cascading in a different manner along each multi-channel. Let’s put that aside for the moment and focus exclusively on the possible confusion driving the different results.

    Let’s assume each multi-channel is dong what it is supposed to, but suddenly at this phase, things get really goofy.

    Losing track of data Normally, when something is found, it gets tagged with a reference marker. This reference is then associated with the request. What could be happening is the reference marker has been assumed to be one code, while the needed reference marker is something else.

    What could be happening is that the method of generating the reference marker along one multi-channel is one way; while the reference marker along the reference channel is something else.

    Then, at a crosscheck point, the two confirm “this is our reference marker,” and all the multi-channels agree: We have the same reference marker.

    Small problem. The “reference marker that they all agree on” doesn’t match the subtle difference in the reference marker required for that particular channel.

    Either the reference marker is assigned incorrectly; or it is assigned correctly, but the required reference doesn’t match.

    What would cause the reference marker to not match the required match? Lets consider labeling. If the label on the data [that PubSubX assigns] doesn’t match the actual reading mechanism, then the problem is the reader is looking for something that it will never find.

    But rather than report, “I have nothing and should,” it just keeps saying, “ I have nothing, and that is fine.”

    A “reference marker will also not match the required match” when it gets assigned on the basis of assumptions that are not consistent with the final result-presentation. If there are embedded codes in the reference markers that eliminate content on the assumption that “this platform doesn’t take that information,” it remains to be seen whether this has been a universal command applied to that particular search-string, but not the other.

    In other words, if a particular channel is defined, coded, and established on the assumption that the end-user will not be able to read it, there could be subtle changes in the codes applied to the reference marker to strip out content because of formatting issues; and these universal command-codes are then referenced to make decisions about all content, not just a particular individual hit.

    There may be a universal-command related to a particular viewing medium, that the individual channel should only apply to one of the subscription hits, but the PubSubX is incorrectly applying that standard to all hits. The solution would be to review the individual channel assumptions on “what the end-user is assumed to be able to read,” and then compare those coded parameters with the search parameters of “what constitutes a valid result.”

    This is another way of saying, “Assumptions about the particular viewing mechanism may be hardwired to something that is presumed to be correct; but the end-user is using a viewing device that is not constrained by these assumptions; so, when there is a conflict [between what the design team embedded in the code as a viewing-constraint; and that which is actually used, but not constrained], the default position is to scratch the data for that individual subscription, but the universal command throws everything out.”

    There may be something linked with the presumed display parameters, and those criteria used to “restart the search”. In other words, if a particular assumption is that X-constraints apply, and if they are violated we revert to zero/no-hit; that may be fine for an individual search result. But if that single “zero-out” then becomes a universal command that restarts another clock, it could be clearing out the entire cumulative-saved files on the assumption that “not just the previous hit is invalid, but all hits are invalid.” This would imply that the incorrect PubSubX decrement value is to go to a default value of 0, rather than [what it should do] back-step by one, thereby deleting one result that is not compatible, rather than restarting the search at clock-time 0.

    What could be happening is that as the PubSub subscription is filled with an invalid or ill formatted hit, it says, “Start search over,” clears out the data, and reports, “Starting search.” The external test-platform monitoring these events says, “OK, they’ve found a problem, and recalibrated. Good,” but never stops to ask, “Did you calibrate your search, or have you simply restarted your search at square zero?”

    The channel could very well report back, “We had a problem, we have fixed it and we are starting again.” Little does the test-platform realize: When the channel says, “We’re starting,” it means starting over; while the test platform is incorrectly assuming that the “start over” position is “next one”; and doesn’t realize that the “start over” is actually reverting back to zero.

    This comes at a time when the test-platform doesn’t say, “Hay, you in this channel are getting all these problems, while you doing the same search are having luck. You’re both doing the same search: Why the difference?” Both the channels come back and say, “We’re different.” And the test platform, goes, “Oh, that’s right. Continue.”

    Meanwhile, the same search strings along different channels produce different results: The test platform says, “everything is fine.” And what is actually happening is channel keeps reverting to “continue search,” while it is actually re-starting a search from scratch; while the second channel, never asked to compare its results, continues humming along reporting content.

    If true, it remains to be understood why there is no immediate check across the channels to say, “You should both be getting similar progress and status by now.” It is possible that the philosophy of “different search strings produce different results and that is OK” has been applied to the individual channel, when it should not.

    Perhaps what has happened is each channel [from the same common search string] is permitted to act in different ways, and never reconciled as they should be, because both are created at slightly different times, therefore treated as individual subscriptions/ providing different content, and acting on their own, when they are actually the same or closely matching, regardless the channel used to display the results.

    The question becomes: Should there be a step at this phase that says, “One of the multi-channels is reporting information, but the other mechanisms are not working; we have a problem; we need to back test our processes to find the glitches internally, reconcile those, and then proceed with assigning the correct address code to the content.”

    I suspect this backward testing does not occur; and the external test-platform [which PubSubX users have no visibly to] does not then take both of the multi-channels and compare the outputs to say, “We have different results, we need to change something, or find out why this is happening.”

    In all probability, the external test platform has been established on the assumption that a “valid result in one channel is unrelated to the result from another channel.” Again, let’s not confuse channels from subscription. A single subscription has multiple channels to find that content.

    Let us consider the possibility that the external test platform has been designed on the assumption that each of the channels is different. We remain unclear how the channels are compared: Linearly, or rotated by 90 at each phase of the process. PubSub would have to look into this. Because if the external test platform assumes that each channel is different, [when they should be working the same], there would never be a confirming check to match content from one channel against another; nor would there ever be a confirming check that there’s a problem in how each subscription was tagged.

    My guess is each channel works independently; when, they should actually remain separate as channel-providers, but still be compared to ensure their data remains consistent; or dig into the details when their content is not consistent.

  • What check is done to ensure the searches, as executed for different entry points along the various channels, are consistent between the [a] search done; [b] how archived; and [c] how reported?

    PubSubX may be either linearly analyzed or rotated by 90. Let’ assume that PubSubX has, for this point of the discussion, a milestone approach to checking data.

    It’s trouble enough to check the data along a single subscription. What’s more problematic is when you have multiple channels to access and view that content. The linear problem of a single feed is challenging.

    It remains to be understood what uniform mechanism is done on each search is done along the channel and then archived and reported compared to a different channel from the same search string. If the governing rules defining these internal checks along one channel is slightly different than another, then there is no common link that will tie this channel along a single path.

    Perhaps there is a break in the chain. A hole. One that continues to be reported as “there is no hole,” and the external test-platform says, “You are correct, there is no problem,” when the actual problem is data arrives, has not been tagged correctly, but the “poor tagging or improper tagging” either triggers something to discard it an everything else; or it correctly reports status as invalid, throws the wrong message into the register as “valid” and then this gets stuffed with “too much stuff’; the data is compared, the results don’t match, and then dumped because of an apparent problem with the search.

    This would presume that the back testing is working to clear out the content that is erroneously being reported. And the channel is dumping everything into the “Clear out” box, when it should be retained. Maybe it takes bad information, dumps everything, and never able to register a single positive result.

    Maybe the checks aren’t valid along the string; they tagged incorrectly; or the archiving does exactly the opposite of what was intended because of the “copy and paste” error mode; which in turn, affects the test-platform [as it should not, but does].

    Step 4: Report results

    Data is sent from the pool of XML feeds along the channel into the storage location. This is where PubSubX takes the content collected and stores it. Again, the issue becomes, “Why is one channel collecting, assigning, storing” in one way, while the other channel is not matching the first one?

  • Is the wrong XML feed code applied?

  • Why is there no uniform code applied to all channels?

  • How are the XML feed codes compared?

  • Are codes assigned, created, and stored using dissimilar parameters?

  • Why does one channel have one code, but the other channel [for the same search string] have a different XML feed code applied?

  • Is there a relationship between the codes created for each channel?

  • What does PubSubX do to each channel’s XML feed code differently?

  • Does PubSubX add something to the XML feed code that it shouldn’t add?

  • Does PubSubX remove something from the XML feed code that is needed?

  • Is there a reconciliation across the channels to ensure the same search across different channels produces reliable XML feed codes?

    Perhaps there are some commands that are telling PubSubX to do something with the particular feed, yet different sections of PubSub are differently reporting results.

  • How are different sections of PubSubX differently reporting results? Are the reports differently assigned to each channel; the content stored is assigned based on tagged codes that will never appear because the XML code is different that what is expected?

  • Is there no check done to ensure different entries provide consistent reporting outputs?

  • Are there commands that add things that should not be added; are there things that strip out codes that should not be stripped out?

    Step 5: Convert results from PubSub to output

    This takes the XML-assigned information and prepares it for display. This is where the final packaging occurs. Each section/input goes along a path.

  • Are there any intermediate checks done along the path to validate consistency?

  • After packaging, is some data revealed/added/deleted that should not be?

  • Is content assigned a code or value that mixes it up and creates confusion about “where to actually send the content”?

  • Is there no chance that cross flow of information reports, “No change” but there is actually a change and the comparison check is not done?

    Step 6: Present the report

    One section of PubSubX reports OK, but the other does not.

  • What checks are done on the holes?

  • How are the changes, inconsistencies identified, converted, updated, or validated?

  • Is there an internal check that appears to validate something, but the test validation is not done, or is the validation which tests “there is a hole” changed to report “OK”?


    Summary


    It appears as though what is happening is there are internal checks across different channels and search strings. The intended goal was to test the validity of the individual feed, but the checks do not ensure that each channel is uniformly operating the same with respect to each individual search. It appears the needed checks are not complete; or if complete, are incorrectly reporting status that does not match actual status.

    Either data is added, and no checks done after data is added; or data drops out but reports as complete; or data is identified as problematic, and the entire saved inventory of subscriptions is cleared out.

    There is a hole somewhere which PubSub internal cheeks hasn’t detected. A hole exists, but no actual test has been done to discover, detect, or report that hole because the assumptions of “what is supposed to be going on” do not match how things are actually going.

    Either the hole exists without doing the test; or the test is incorrectly done; or the tests are incorrectly reporting filled when it is actually empty.

    I think there’s a code that tests the validity of the output across channels; but the test is backwards: Falsely reports OK when there is an inconsistency; or the inconstancy is not detected, test and published as a valid result, even though it has nothing.

    I also suspect that there is a box that has a known limitation on the number of parameters allowed. This “box” is related to “what was assumed the user would use to read, display, and access the information.” Perhaps this “box” has parameters defined with specific values; but the actual values are different, not important, or interact in ways that are not as important or more important than originally planned.

    Questions

  • How was it determined the number of codes/size of content was accurate?

  • Why was this constraint chosen?

  • What happens if remove that constraint?

    It remains to be understood what assumptions the codes are using about:

  • The platform people are using to display the information

  • The assumptions on the size, form, acceptance properties of the result;

  • Whether the code platform is designed to work with one set of assumptions, but the end user is actually suing a method of access that is different, or not to e constrained by that assumption designed into the code.

    Theory

    Valid test reports incorrectly because syntax mashed in the instruction. A minor coding error has shoved two sections of the code together. Embedded code has been meshed because of a syntax error so it meshes the test with the check.

    What is needed is a review of the changes to see what was added and where; and look for subtle changes in the code that would permit the test to occur without completing the test, and then take output and store it, but read its output incorrectly.

  • Bad results not blocked

  • Goes to the end, registers content that should show “inconsistency” reports negatively/empty…generating a false positive to continue the search, even though it has actually failed.

    In reality: Test to validate across platforms searches doesn’t report, the problem with the reported code and output internal consistency problem because the test check doesn’t occur as planned/though/reported.

    Theory

    There is a roadblock that PubSub has, but doesn’t realize. PubSub was designed on the assumption of X-parameters; or they don’t test for those parameters; or they don’t check those parameters; or they create execute commands that do 180-degress opposite; comment at one of the test commands has a skip option, does not report a problem, but executes without actually doing the various [a] tests; [b] removals; [c] reviews; [d] reports; or [e] comparison

    Theory

    The very part of the system need to do the subsequent post-deployment checks reported OK, yet when it was not done, and test never executed, interpreted the results as “OK.

    The very code changed and updated was part of the tool to validate the internal inconsistency.

    All stress testing doe prior to the code update. Later used a cut and paste correction. Accidentally removed a minor font with paste. Reported valid on 1 of the channels, but did not validated “test check OK” with subsequent back testing across the channels.

    Theory

    Code change in how it interfaces with test tools. There is a test-tool PubSubX interface issue; because of the interface issue, tests errors falsely report OK and there’s no signal back to the platform to adjust something as designed-intended.

    Theory

    Coding error makes the required alert with a test tool.

    Test tool interface with PubSub: test tool depends on x-parameters; the code-interface issue with PubSub repots valid, even though the opposite: the cross check is not completed; reports OK because empty register [that could never get filled with errors because the test is skipped.

    Theory

    Problem isn’t with PubSub, but with the test tool reporting “OK” despite inconsistency between the channels.

    Theory

    Error in PubSub prevents test platform from reporting the error and test OK.

  • How does the error-mode makes the PubSub test platform integration issue?

    Possible explanation

    The way the executed code update affected the way they designed the interface; when changed would never have revealed the internal check-test-inconsistency because that part of the platform, when changed, would never report an error as expected.

    Review

    We’ve outlined theories. The next step is to generalize the lessons from PubSubX and apply them to FeedMesh. Recall, we have no idea what is causing the inconsistent results across the platform on PubSubX, but for purposes of discussion, let us presume that there will eventually be found a solution to the apparent problem.


    FeedMesh


    For FeedMesh to work, it needs to work with XML and any method to execute using XML. FeedMesh needs to have a single set of commands to internally testing with a single platform and across platforms: Test, execute, validate, and check.

    Move from just XML-standards, and focus on ensuring the platform-procedures;
    Are different platforms correctly identifying and reconciling integration issues?
    The key issue with Feed Mesh will be what method will be used to ensure different XML procedures are:

  • reconciled,
  • tested correctly,
  • identifying inconsistencies; and
  • correctly converting to the proper codes to provide delivery and provide the final report.

    If we have multiple channels that provide the same subscription-streams-feeds, then those duplications need not be eliminated on the assumption that it is redundant; rather, the redundancy can be used to crosscheck the validity of the inputs, and progress along the cycle.

    Having multi-channels forces reconciliation, crosscheck, and force consistency, not just with XML but the commands that are there to execute the objectives. There may be different methods to check, but these need to be brought together and reconciled.

    FeedMesh relies on multiple platforms integration. It needs to work with any approach, not just the standard.

    This is to say that XML and the coding approach should be viewed as two different animals in the same problem.

    The current PubSubX model concludes that there are checks and reviews that are not designed that would be included; or there are functions that are required which have not been incorporated.

    Bluntly, those holes need to be filled. Those checks need to be done. How is FeedMesh going to work with different developers—yell at them as well?

    Not all platforms execute the same.

    These checks, if absent, are needed. We need to make sure that the FeedMesh solution integrate across platforms, not just integrate XML. Different segments should be reporting the same results regardless the methods used in coding and design.

    What is needed is an internal check within the system to make sure the different parts are checking each other. Inconsistencies need to be reported and given visibility.

    Not at the end, but as things progress. Intermediate checks.

    They may already exist. The issue becomes, why are they not ensuring the different channels are working the same way in delivering the same content in the same way.

    When PubSubX gets seriously evaluated, you’ll be well on your way to overcoming a hurdle in implementing FeedMesh: Cross-channel testing based not just on XML standards, but in whether the functions are performing as intended, regardless the different approaches taken to achieve the same results.
  • Creative Commons License

    Cite as, "Mud's Tests." This work is licensed under a Creative Commons License.



    Disclaimer

    This is an OPINION only. If you do not understand that this is an OPINION and not statement of fact, then stop reading.



    Note

    For purposes of clarification, this speculative piece should be construed only as an informal opinion, not legal advice, nor a statement of fact. Nor should this information be used as a basis to make any specific findings of fact about the nature of the discussed error mode or the specific failure mode that appears to exist.


    Overview

    The PubSub platform has had apparent internal consistency problem. Part of the platform reports valid subscription results, while other portions of the platform report no content found.

    This is problematic in that the inconsistent results stem from common search strings. The purpose of this note is to explore the possible failure modes driving this inconsistent output; explore theories related to the possible explanations for the failure mode; outline a proposed methodology to explore alternatives to the drivers behind this inconsistent reporting; then discuss the implications of these apparent conditions in light of the FeedMesh Concept.

    It is hoped that the results of this discussion and speculative piece might assist in pinpointing the actual problem driving the inconsistent outputs from PubSub; and also apply the apparent lessons learned to integration issues in the FeedMesh concept.

    FeedMesh

    FeedMesh is proposed to be a mechanism that will integrate many platforms and ensure that XML feeds can be timely provided, published, reported, and provided to end-users. This is a gross simplification of the concept.

    Yet, we propose that FeedMesh will only succeed if integration issues within a single platform can be resolved. If we are to integrate many platforms, then the lessons learned from an “apparent inability to integrate in a single platform” might serve as useful lessons in approaching the FeedMesh concept.

    This is to say, that even if we propose theories and methodologies that do not specifically address the PubSub issue, we might advance the cause of FeedMesh by outlining for the public and developers the apparent ground rules and integration issues related to FeedMesh. In other words, if PubSub’s apparent inconsistent reporting is related to an internal integration issue, the public and developers would benefit if we understand whether this is what is actually occurring, and develop some guiding principles to aid the developers in stress testing the various parts that will be need to be integrated.

    In all likelihood the developers have already thought of solutions to the issues raised. However, it is hoped that the larger XML community to include bloggers and content readers might gain a perspective of how things are.

    Again, in all likelihood this perspective could be very far from the mark. But could serve as a useful guide for developers and other consultants to calibrate how they approach the end user, and what information they provide and assumptions they make when developing and deploying products for wider use.

    I would like to thank PubSub in their being the catalyst for this thinking. I imagine that a solution is very close at hand and the current delays will understandably be soon a thing of the past, water under the XML bridge.

    Sections

    This note first begins with a discussion of a generic methodology to trouble shoot a recurring error mode; then explores several theories related to the PubSub apparent error mode; then concludes with a discussion of the lessons learned that might be applied to FeedMesh.

    In the end, we hope to create a set of metrics that the public might use to evaluate whether FeedMesh is meeting their needs; and also create some common boundaries for what FeedMesh needs to both solve and incorporate during the design and development process.

    Generic Methodology

    This section of the note outlines a crude and imprecise characterization of the PubSub platform. This discussion should not be used as a basis to specifically evaluate PubSub’s detailed code.

    Rather, it is hoped that the “perspective of PubSub” be used only for discussion only in that it outlines a framework and coding steps that in all probability have no relationship to the detailed software codes.

    This is to say that we will simply outline generic steps that PubSub appears to do so as to provide a starting point for a discussion of the methodology. Again, this methodology is to service two purposes: To both assist developers in trouble shooting the specific PubSub delay-error-mode; and also in creating a framework other platforms may choose to apply in situations where they are unable to solve what might otherwise be a very simple coding issue.

    In short, the intent of this section is to share a number of new perspectives on a situation that appears to have exhausted all available resources and the error mode cannot be reconciled or resolved. This is another way of saying, “We never expect to need to do this, but just in case … we’ve got these ideas that might spark us to think of something new.”


    Methodology Overview


    This section will outline a methodology to both troubleshoot PubSub and apply this framework to FeedMesh. The methodology hopes to inter alia:

  • Trouble shoot multi-reporting channels.
  • Conduct checks for internal consistency
  • Validate approval checks
  • Ensure “test OK message-status reports” are valid
  • Fill in holes in code commands
  • Identify code-holes not patched, yet still report complete


  • Definitions

    Clearly, the above terms are non-standard. I shall go through the terms to explain their usefulness in the proposed methodology.

    Multi-reporting channels This is a system where parallel reporting efforts occur at the same time, or different times. A system that generates similar outputs, but through different channels and reporting routes. A single platform like PubSub has more than one mechanism to report subscriptions. Thus, it is a multi-reporting channel platform.

    Checks for internal consistency We check for internal consistency in a multi-channel platform to ensure that our processes are consistent. This is to say that when we have multiple different methods to achieve the same results [return and deliver subscription content], that before we provide the end-use with that data, there needs to be an internal check to ensure our various reporting channels are working not only correctly, but that they are reporting the results in the same way.

    When we check for internal consistency, what we are doing is making sure that the search results from search-stream 1 [derived from one reporting channel] are reasonably related to the results from search-stream 2 [derived from a second reporting channel]. This is not to say that the content and outputs need to match; nor is it to say that we have a problem if one search string has slightly different subscription-content as another similar search.

    Rather, the point of the internal consistency check is to allow each section of the multi-reporting channel the opportunity to challenge, check, and compare their operations, tests, and outputs with the other channels. In other words, before issuing a final subscription report, ideally what happens is the various sections of the platform do a check to ensure that the way they are going to present the data, and the results are consistent.

    An error would report if one subscription channel were reporting “no content found” while a second subscription channel was reporting valid content and a full subscription-completion.

    When we check for internal consistency, what we are asking the platform to do is to compare the subscription-results derived from the different methods of request, and check whether the outputs are consistent, not just in content, but in how they were processed, tested, and approved for release.

    Validate approval checks. This term means to ensure that the series of tests used when “checking for internal consistency” are consistently operating in the same manner. In other words, this is a step whereby the platform, before publication and delivery of content, will then check its own approvals to make sure each subscription-provider is doing the same things as each other. Again, if one subscription has had “done to it” different tests than another, then this would indicate something is not performing as expected and further inquiry is needed.

    The purpose of validating the approval checks is to simply ensure that the checks we do on each search string are then aggregated, compared, and each different reporting channel can say, “We all agree that we are all doing the same thing in response to this subscription-request; and we all agree that our steps, tests, and content have arrived at our results in a consistent manner.” This is not a check of the different reporting channel’s content, nor is it a method to say, “Well, this subscription channel reports X, Y, Z content; but the other reporting channel reports A, B, C so there is a problem. Rather the purpose of this step is merely an internal check to ensure that the various reporting channels are checking each other to say, “Yes, we are all doing the same things; and our coding is working as expected; and yes, we are sure that the tests we did in one section of the platform are the same steps done elsewhere.

    What this step does is ensure that if there is a problem with one set of codes with one set of checks applied to one of the subscription channels, that a second set of procedures similarly applied to a different set of reporting channels also are consistently coded. Clearly, this check is not needed if both reporting channels rely on the same end-tests.

    However, in situations like PubSub where it appears as though each reporting channel does different things in response to a given request, it might be useful if there was a mechanism to ensure that the different reporting channels are actually using the same types of code. Specifically, one failure mode that might explain the different results is if one reporting channel within PubSub uses one set of code-procedures that are linked with one type of subscription-request; while another subscription request relies on similar code that is embedded in a subroutine.

    At this point, what we might discern is that if there is one failure mode that is unique to only one reporting channel, then what might be going on is that there is an embedded set of commands and tests related to one reporting channel; and this code has been slightly adjusted and is not exactly matching what the other reporting channel relies on elsewhere.

    Again, this is pure speculation. We make no claims that this is what is happening. We simply use this to illustrate the concept, “Validating approval checks.”

    Test OK messages are valid We now turn to the test result message. This is a term that says, “We’ve done a test, the test reports OK; and the test is based on statusing this type of information located in X-location. In those situations where we assume a test is conducted correctly, we assume the actual data used to test and generate the status message is valid. However, there can be situations where the input data used for the test reports one status that is at odds with real status.

    Suppose we have a prepatory step prior to a test that gathers data; suppose we also have an unknown coding problem that would normally conduct a test on this data and say, “If we have this result, our test status is Y.”

    However, if the problem is that because of a coding problem we have meshed two lines of code, and one part of the test is not done, but the second part still reports the result associated with a “test OK message,” then we’ll never actually conduct a valid test, but our test result will be stored in a location that is processed on the assumption that that test actually executed and this is a valid result.

    With respect to PubSub what could be happening is that there is a test mode that conducts a review of the status of the platform; that test was designed and included in the platform correct. However, there is the possibility that during post-integration testing, an update to that section of the code was updated with a cut and paste, and the new code placed over it had an error. However, when we do the check of that code, the error-mode of the code is outside what we expect; and the test still appears to execute correctly, and there is a result that gets reported, and that result is consistent with what we expect.

    However, here’s the rub. Suppose we have a test-tool that is outside our primary platform. Unknown to our coders, the test-team has created a test-platform based on X, Y, Z assumptions, yet at the same time our actual platform conducts a test that no longer actually complies with those assumptions. This is to say that the requirements that we initially planned for in the specification were one thing; what we designed and created met those specifications; and then what we assumed in the test-tool of “what would happen” did not match exactly with how the error in code actually executed the command.

    In other words, suppose we have two sets of steps: One is the actual code used to deliver the subscription for channel one; at the same time the platform has a second search mode that reports a similar search result, but there is no internal check to makes sure we actually did the searches in the same way. On top of this, suppose there is an error in the code in only one of the channels, while the other reports fine; and at the same time there is not crosscheck between channels to compare the processes and results.

    The next step that would happen during the normal processing is both channels would report their results. Fine. Each has different content. That is expected. But what is not expected is if the test-platform used to evaluate that code is based on one set of assumptions about “how the test will execute”, but one of the reporting channels actually executes something else.

    Normally, we would hope that the test-platform that is outside the PubSub host would catch the inconsistency. That is, if the test-platform is not related to the internal error within PubSub.

    This is another way of saying: If PubSub has two reporting channels that are executing correctly, but one of them executes only part of a test, or the test as designed fails to execute a command, but still reports “OK status,” then what could happen is the test-platform [designed to test based on initial assumptions] could take the results from both and say, “We have valid output” according to the assumptions, but not according to what actually executed.

    In other words, if the test-platform is designed on the basis of assumptions that rely on the code to work on the primary platform, but is not designed to check the inconsistencies between channels, then there would be no signal for the test-platform that the processes were working differently. And if there was an embedded code that was incorrectly reporting status as “OK” [because of a coding problem], there would be nothing that would go in and say, “We’ve got this type of process-test occurring in reporting channel 1, but something else in reporting channel 2” … because the test platform’s code and assumptions also assume that “what it is testing” matches what it was designed to do.

    However, if channel one has an embedded code problem, is only reporting “status OK’ in response to errors, then a similarly designed test-platform would also never think to go back and say, “How did the status-OK message get generated” and “did the processes that actually executed” match across all the channels?

    In my view, what could be the problem with PubSub is the test-platform was designed on the assumption that certain steps would or would not occur; but it does not take into consideration a set of circumstances where “a test that is reporting valid is actually reporting invalid information, but appears to work” It appears there is no test for the external test-platform to ensure that the assumptions the test-platform is using are matching the assumptions to all the processes used in all the reporting channels.

    In other words, suppose we focus on 2 reporting channels. If reporting channel one is executive fine and returning results; and reporting channel two is also reporting fine [but is actually coded wrong, reports a problem, but that problem is reported as OK (for whatever reason)], then there would be no signal to the test platform that there is an internal consistency problem between reporting channels one and two that needs to be explained.

    This would explain why all indications are that the reporting channels are fine: The external test-platform continues to report, “Thins are working as designed.” Which is true. All the channels are reporting as coded, but it appears there is no check by the external test-platform to make sure each individual step is matched across platforms; and that all tests that report “OK” are actually OK. Moreover, it does not appear as though the external test-platform goes back into the PubSub system during operation to sample the various reporting channels and say, “We have one here, and a second here…how do the results compare in terms of test messages; actual steps executed; and the series of steps taken to arrive at the outcome.”

    Clearly, PubSub has a series of steps that generate in the various reporting channels different subscription content. That is not the issue. The issue becomes: How do we explain why a robust, mature, and well acclaimed system can this late in the game have multiple channels reporting different results, yet all indicators are that things are working fine.

    I can think of a couple of options:

  • The external test platform as designed is working as designed: To test the steps. However, the problem could be that the steps are reporting OK, but are actually doing the opposite because of a coding issue whereby one step of the test gets skipped, but the output still reports OK, and the test done on that section of the code similarly makes the same error because the code was copied from the designed platform, and integrated into the test-checks.

  • Another failure mode could be that PubSub does not check across a single subscription and compare the multi-channels. This is to say that the system as designed was always based on the assumption that the code, once tested and deployed, would always execute as designed: Consistently. However, there was no method to ensure that during deployment, data was collected to validate that these initial design assumptions were valid and continuing. In other words: If there was always the assumption that each channel would correctly report status, and that the tests would determine whether the output was valid or not [but there was no method to ensure that a coding error would trigger a signal in situations where a test fails but still reports the desired result, albeit wrong], there would be no reason for anyone to design something that would apparently re-do a function that was already tested as correctly execute. That is a waste of time.

    But let’s go into the netherworld. Suppose we have multi-channels established whereby each section was independently developed, then brought together under one umbrella, and everything tests out OK; but during a critical post-integration phase, a single syntax error occurred, covered up the process-test, and the test that would otherwise report a problem, failed to execute; and at the same time the “returned result” matched the other channel status, so from the external-test platform perspective the common results match the common inputs, so things must be fine.

    This is another way of saying “Two flaws within a single multi-channel cancelled each other out, reported a valid result that the test-platform similarly reported ‘good to go,’ all the while there was no detailed check of the individual commands to say, ‘Yes, each step flawlessly executed and compared with the other channels in all respects.’"

    Rather, because there was no error, there was no need to conduct a detailed check; the detailed check may be linked with the assumption that the detailed check will only be needed if there is a difference. But if there is no reported difference, then the external platform never gets the signal to go in and say, “We need to compare the details.” Again, if everything is reporting fine. There’s no reason to have a detailed step within the external test-platform that is going to compare detailed steps that “should” be the same.

    Again, if we have two errors occurring [coding problem and the test fails to execute; but output is invalid, yet is stored, reported, and interpreted as valid] then there would be no signal for the system to do detailed checks; nor look into the detailed steps; nor ever be prompted to look at the individual steps and compare: What was planned, what actually happened; what results were expected; how was the test results stored; and how did the result get translated into a next step; more broadly because these steps never occurred, there would be no signal to the external test-platform that there was ever an issue.

    Without having the ability to check across the same search string, but compare the different processes of the different channels.

    Fill in holes in code commands This term means finding problems and fixing them. Suppose we have a test step on a one of our channels. And further suppose we have a requirement to do something such as a test. Further suppose that that test, as designed, does the job. Further suppose that we have a problem during post-integration testing, and a quick update is made. Further suppose that there is no post-update check to make sure that our various channels continue to operate as originally designed: They appear to, but there is an embedded coding problem that incorrectly reports “no problem”, and there is no catalyst to look at the details to find the real step that did not uniformly execute.

    When there is a test with a simple command, we expect the test to execute. But if there is a coding error and two failure modes, then this is a leap from where we are, to where we want to go, without actually executing the command as designed-intended; it only produces a result that is meaningless, but desired: “Things are OK.”

    At that point where the test executes incorrectly, but reports “OK” we have a hole. Something was not done.

    Another example is when there is a requirement to execute a command, and we get a favorable report, but the test output is actually incorrect … yet the favorable report fails to trigger the needed detailed check … That is also a hole: We have a problem that is hidden, something should be getting tested, but is not. In short a “hole in the code” is when there is a needed check, but that check does not occur because of an incorrect assumption; or a double-error problem; or a failure of the external-platform to trigger a needed detailed test.

    When we “fill in the holes in the code” what we are doing is looking for these breakdowns, and then re-running the coding effort from square one. We fill in the hole by pretending we are an individual packet of information, and tracing the process millisecond by millisecond. Those holes, when they appear, can be very small. But under the right situations, a slight coding error can create a false report, but the information could be stored incorrectly, thereby signaling “we need not look further,” when in fact, the real problem is: We have a hole that the system, as designed, was not designed to detect on its own, nor given instructions of “what else to check to compare to highlight this disconnect” nor any instructions on how to execute the needed detailed tests to check the different detailed steps.

    Risk code holes not patched yet still report complete This idea is simply one of summarizing the above failure mode. This is another way of saying, “We have a known platform, a desired design, and a specific task…that has a bug, but all the signals are: Things are fine.

    The risk that “code is not patched but still reports OK” can be traced to the multi-error problem; combined with the test-platform that does not look at details when given a message, “everything is fine.”

    In a situation where we have multi-channels reporting the same status [even though they actually executed the command differently] this would contribute to this problem of “we have a hole, but no indication of a hole, so do nothing. Green light. Go.”


    Review


    We’ve gone over some made-up terms to define possible failure modes. We’ve also gone over some hypothetical failure modes within PubSub that would illustrate these concepts. Again, we’re not saying that PubSub is suffering from any one of these failure modes; only that something related to the above situations could be going on.

    The next step is to reconstruct the PubSub platform. Again, we’re not going to focus on what is going on with PubSub, but merely define the general steps that PubSub appears to go through, and then assign to each step a series of questions and failure modes that might spark some additional thinking and exploration.

    Non-Standard Characterization of PubSub

    For purposes of discussion only, I will construct a theoretical model of PubSub. This approach will be useful in both incorporating the above concepts in the failure modes, but more importantly to narrow in on the range of possible failure modes that appear to have eluded understanding.

    Again, to emphasize, the following categories are not intended to represent what PubSub does on the detailed level. Only to service as a generic outline of the things that occur. Clearly, the gaps in this discussion will serve to highlight other areas deserving further study and understanding.

    Six Internal functions and an external oversight function

    The external oversight function is simply the external test-platform. The six section of PubSub will be defined as PubSubX.

    Let us consider the actual functions that PubSub does and compare these with PubSubX. PubSub takes a subscription request, crunches the data, spits out a subscription return.

    PubSubX has more details.


    PubSubX

    Request

    Step 1: Request is the request where search commands are inputted into the system. Step 1 is the conversion phase.

    XML Feed Content

    Step 2: Access -- PubSubX access the XML content, looks into the data available, and searches.

    XML Feed URI

    Step 3: Code -- PubSub X creates an XML feed that will extract from the pool of XML feeds the data matching the request.

    Display

    Step 4: Result -- PubSub X achieves a result and displays it within the PubSubX account. At this juncture, the information has not been presented to the subscriber; rather, PubSubX merely stores the information, does checks on it, and validates that the result meets the objective.

    Conversion

    Step 5: Translate -- PubSubX takes the data that has met the search string goal, and translates it from raw code, into something that is readable.

    Report

    Step 6: Present -- PubSub X then packages the results into something the end-user can read.


    OK: Dig into the details

    At this point of the analysis we will dive into each of the 6 PubSubX sections, propose questions for further analysis, and identify a general framework for analyzing an XML platform.

    Let’s continue the exploration of PubSubX with the hopes of identifying some testing and evaluation to done on each component. We hope to dovetail each step with the above definitions so that developers and management can assess whether such a framework is useful in exploring the error modes.

    Let’s start with each step of our 6-phase construct of PubSubX. Let us also generalize our approach to be one that hoes to identify why there would be multi-channels doing different things: One search channel doing one thing and working as planned; while another is doing something else and providing no result, but is reporting “Operating OK.”

    1. Conversion: Take a valid search and changes it into something to throw at the XML content.

    Step 1 is where we take a valid input, a search term, and converts it into something that will get slapped against the available content. Notice that we are assuming the simple PubSub instructions are followed, and that the search terms across multi-channels are the same. If there are any errors, the error lies with PubSubX, not with the user.

    Let us explore the possible failure modes that occur at this phase of the PubSubX conversion from subscription requests to results. Again, we are focusing exclusively what PubSubX is doing:

  • Is the search terminology as entered correctly converted?

    One failure mode between PubSubX multi-channels is that the different channels could be translating the subscription request and search terms differently. One multi-channel could be correctly converting the search request, while the second channel makes an error, but this error is not caught, and there is no step to check or compare the multi-channels after the search strings have been converted.

  • How is it shown there is a valid request to PubSubX?

    When we take a valid search-subscription request, what we are doing is simply typing in words. PubSubX does the rest. What are transparent to the user are the internal checks. Part of this crunching inside PubSubX.

    But what is done is prior to PubSubX doing its magic is that it must first ensure the term is recognized and is valid.

    Suppose a user enters a valid word, but PubSubX doesn't *’t recognize the word – is the end-user given a proper signal to change the word; or does PubSubX take the information, even though there is a problem, and apply it to the multi-channels, even through there is a problem?

    Have the multi-channels been configured differently so that they process this incoming “invalid search request” differently: One of the channels takes the information crunches it, and produces a valid report: “Nothing.” While the other channel, converts it to something that PubSubX can use, and produces a valid result?

    At this point, the issue becomes to trace how the different multi-channels process the information in parallel, and then compare the results at each step. In my view, it is incorrect to run each search individually through a single channel and then evaluate the merits of those steps on their own. What needs to happen is to run multi-searches along each channel at the same time; then freeze-frame the status of that content, compare the status, and identify what is going on at each phase.

    If we have a valid search going into PubSubX, there must be a system that says, “We have something that is workable-usable”. Thus, there must be a mechanism that does this check; does a test; conducts the test; stores the result; compares that result to what is planned/expected; and then stores the test-result somewhere.

    At this phase, what could be happening in PubSubX is that one of the tests has not executed correctly, but is still reporting a valid result. Also, what could be happening is although the request was actually invalid, there was generated a valid test result. Or what could be happening is that there was an invalid request that is never identified, shown, compared to baseline data; or is never compared with the first step of the other multi-channel.

    Again, a simple solution to this problem would be to introduce a cross-channel checking system that ensures each step is working as planned. If this would slow PubSubX down too much, then one approach would be to do some data sampling using an external test platform that takes a parallel approach to the multi-channel; move away from testing the outputs along a single channel, and focus on each phase of PubSubX to see how the different multi-channels operate.

    Another approach would be to focus on the test platform and evaluate how it was designed with respect to this phase. Again, if the test platform goes linearly from search input to output, but never compares the results, tests, and progress of PubSubX along the data stream, there may be no way that PubSubX would know there is a problem, or that something unusual was happening along one of the channels.

    If the test platform is linear-checking only, but could be enhanced if it took a cross-channel approach at phases, it clearly has to be established how much time, effort would be required to essentially rotate the matrix of analysis 90 degrees from a linear approach to a cross-channel approach.

    If PubSubX plans to be around, the tradeoff at this point becomes how much growth does the platform expect; and will it be cheaper to rotate the matrix of analysis 90-degrees now, and then as content and volume increases it can more quickly find and detect and correct error modes; or are the benefits of having phased-checking meaningless given the actual design of PubSubX.

    Clearly, given PubSub’s development background and life, how PubSub actually works could already match the envisioned solution to address PubSubX’s issues. This is another way of saying: Those guys at PubSub are smart developers, and probably have already though of this and may already doing this, making this moot.

    The question for PubSubX becomes: What mechanism exists within the platform at the conversion phase to ensure when PubSubX converts from a valid request to something PubSubX can work with, how does PubSub x know the search is valid; how is this result demonstrated; is the way the information is presented match the actual status; is there a chance that the presented status as “valid” could be the result of two-errors from an actually incorrect input string; or is there never a search done; or is the result of this phase never established, compared, or demonstrated that the status is something that is baselined, tracked as the search progresses linearly along the channel?

  • Is the search request actually valid?

  • Is the invalid search allowed to enter along one channel, but not the other?

  • Can a valid search in one channel produce output, while a valid search in another channel be converted to something that is invalid, but not known to PubSubX?

  • Do the two channels have different ways of converting the inputted terms into “something else”?

  • Can an invalid request be entered, and each channel convert them differently, but there is no intermediate check to ensure that the conversion was consistent, uniform, and matches?

    Let’s consider the notifications to the end-user at this conversion phase. If there is a problem with a term entered, how is the information presented to the end user to change it?

  • Can an invalid search term be entered, accepted, and no error message provided?

  • Can a term that is invalid get handled differently along the channel?

  • Does each channel work to the search string differently, but all fail to report “we worked on it this way and have an error, but the other channel did something else and found something, so we all agree because one of us worked, we all worked”?

  • Are the channels allowing progress from one channel to override error messages from another channel at an intermediate step?

  • Are error-modes that “should signal a change” not occurring because of a universal accept-command that assumes terms can be converted, but they are later converted differently along different channels during subsequent tests, matching, and publication?

    Step 2: The method PubSubX access information

    PubSubX has a defined method to access the XML information.

    Let’s now consider the same PubSubX model at the second step of the process. At this point, PubSubX takes the converted request and accesses the XML content. The objective of this phase is to access the XML feeds, climb through the architecture, and conduct the search. PubSubX has a very simple task compared to the actual PubSub.

    At this point, we ask several questions related to the possible differences between the two channels and the inputs: Something that PubSubX is doing interferes with the process. One channel is working, the other is not.

    To re-iterate: PubSubX has a valid input. PubSubX is now doing something differently to each multi-channel, and interferes with the ability to consistently provide the same types of output: One channel is doing something the other is not; one is conducting a test, returning a message, or producing a result internally that does not match the other, but the external test-platform does not detect the subtle difference.

  • Does PubSubX have 2 different ways to search based on different parts of PubSubX?

  • Are coded instructions related to one channel unique to that channel?

  • Do all channels dip into a common set of search parameters and instructions?

  • Have instructions and detailed steps been farmed out to different segments of the PubSubX platform based on assumptions about parsing that do not play out?

  • Have instructions for the search for each multi-channel been farmed out, then re-integrated into a common manner, but they do not match the way the test-platform is testing them?

  • Could PubSubX create two different search routines from the same set of input parameters; yet when the results are combined, the differences are not compared with each other, and the overriding positive results from one is assumed to match the results from another, but there is no method using the test-platform to compare the individual steps and ensure that the output and progress is uniform and matching?

  • Does PubSubX have different approaches to searching based on different a different channel selected?

  • Does an internal switch define the search parameters in terms that PubSubX understands, but do not translate into consistent results, so PubSubX makes a default decision, discards the “no return” as a “non-response”, but converts that output as an “still waiting” for one channel; while it interprets for another channel a “no waiting” as a prompt to summarize data, adjust the search, or reorder the priority in the other channel?

  • Are there certain conditions and parameters about what will constitute a “valid hit” or a “valid subscription” for one channel, yet these parameters are assumed to not exist or be relevant with a second channel?

  • Are assumptions embedded in each channel which assume that the channel’s content-display is unique, therefore, this justifies conducting different tests and examinations and conversions at the outset?

  • Are last stages of PubSubX [step 6] driving earlier decisions about the size of the display, which are back-channeling their inputs and parameters into what constitutes an acceptable range of information to identify, search, review and reports as valid?

  • Are there different commands within the tool driving PubSubX to search one way with one part of the tool, but another way with another part of the tool?

  • Is there no intermediate check prior to searching to validate that each channel has a uniform set of assumptions about the nature of the search to be conducted?

    Step 3: XML Feed Code and Commands

    PubSubX now has a new phase. This is the first stage of packaging

    This is where PubSubX has a valid search result, and a completed search routine. This stage of the operation is focused on gathering these valid subscription results into a single data feed.

    The error modes along this step are in how the data is aggregated from the XML pool and pushed into a single data stream prior to holding.

  • Once the PubSubX completes the search, how does the XML feed URI produced… actually link and connect to the reported results?

    There is the possibility that there is a disconnect between what PubSubX is finding, and how PubSubX is actually identifying that “found count”.

    If the reference data used to identify the mailbox or storage location of that data does not match what PubSubX has initially been energized to look for, PubSubX could, in fact, report “we have found something within milliseconds,” but what was found was never retrievable as the identifying code where it was stored does not match the actual code used to retrieve the data from that location.

    The question becomes: What would possibly drive PubSubX to report data from a search quickly, but then lose track of this data; or store it with one code that doesn't *’t match the actual code used to retrieve the information.

    Let’s consider some perplexing possibilities:

    Clearly, earlier problems from earlier phases along the multi-channel could be cascading in a different manner along each multi-channel. Let’s put that aside for the moment and focus exclusively on the possible confusion driving the different results.

    Let’s assume each multi-channel is dong what it is supposed to, but suddenly at this phase, things get really goofy.

    Losing track of data Normally, when something is found, it gets tagged with a reference marker. This reference is then associated with the request. What could be happening is the reference marker has been assumed to be one code, while the needed reference marker is something else.

    What could be happening is that the method of generating the reference marker along one multi-channel is one way; while the reference marker along the reference channel is something else.

    Then, at a crosscheck point, the two confirm “this is our reference marker,” and all the multi-channels agree: We have the same reference marker.

    Small problem. The “reference marker that they all agree on” doesn’t match the subtle difference in the reference marker required for that particular channel.

    Either the reference marker is assigned incorrectly; or it is assigned correctly, but the required reference doesn’t match.

    What would cause the reference marker to not match the required match? Lets consider labeling. If the label on the data [that PubSubX assigns] doesn’t match the actual reading mechanism, then the problem is the reader is looking for something that it will never find.

    But rather than report, “I have nothing and should,” it just keeps saying, “ I have nothing, and that is fine.”

    A “reference marker will also not match the required match” when it gets assigned on the basis of assumptions that are not consistent with the final result-presentation. If there are embedded codes in the reference markers that eliminate content on the assumption that “this platform doesn’t take that information,” it remains to be seen whether this has been a universal command applied to that particular search-string, but not the other.

    In other words, if a particular channel is defined, coded, and established on the assumption that the end-user will not be able to read it, there could be subtle changes in the codes applied to the reference marker to strip out content because of formatting issues; and these universal command-codes are then referenced to make decisions about all content, not just a particular individual hit.

    There may be a universal-command related to a particular viewing medium, that the individual channel should only apply to one of the subscription hits, but the PubSubX is incorrectly applying that standard to all hits. The solution would be to review the individual channel assumptions on “what the end-user is assumed to be able to read,” and then compare those coded parameters with the search parameters of “what constitutes a valid result.”

    This is another way of saying, “Assumptions about the particular viewing mechanism may be hardwired to something that is presumed to be correct; but the end-user is using a viewing device that is not constrained by these assumptions; so, when there is a conflict [between what the design team embedded in the code as a viewing-constraint; and that which is actually used, but not constrained], the default position is to scratch the data for that individual subscription, but the universal command throws everything out.”

    There may be something linked with the presumed display parameters, and those criteria used to “restart the search”. In other words, if a particular assumption is that X-constraints apply, and if they are violated we revert to zero/no-hit; that may be fine for an individual search result. But if that single “zero-out” then becomes a universal command that restarts another clock, it could be clearing out the entire cumulative-saved files on the assumption that “not just the previous hit is invalid, but all hits are invalid.” This would imply that the incorrect PubSubX decrement value is to go to a default value of 0, rather than [what it should do] back-step by one, thereby deleting one result that is not compatible, rather than restarting the search at clock-time 0.

    What could be happening is that as the PubSub subscription is filled with an invalid or ill formatted hit, it says, “Start search over,” clears out the data, and reports, “Starting search.” The external test-platform monitoring these events says, “OK, they’ve found a problem, and recalibrated. Good,” but never stops to ask, “Did you calibrate your search, or have you simply restarted your search at square zero?”

    The channel could very well report back, “We had a problem, we have fixed it and we are starting again.” Little does the test-platform realize: When the channel says, “We’re starting,” it means starting over; while the test platform is incorrectly assuming that the “start over” position is “next one”; and doesn’t realize that the “start over” is actually reverting back to zero.

    This comes at a time when the test-platform doesn’t say, “Hay, you in this channel are getting all these problems, while you doing the same search are having luck. You’re both doing the same search: Why the difference?” Both the channels come back and say, “We’re different.” And the test platform, goes, “Oh, that’s right. Continue.”

    Meanwhile, the same search strings along different channels produce different results: The test platform says, “everything is fine.” And what is actually happening is channel keeps reverting to “continue search,” while it is actually re-starting a search from scratch; while the second channel, never asked to compare its results, continues humming along reporting content.

    If true, it remains to be understood why there is no immediate check across the channels to say, “You should both be getting similar progress and status by now.” It is possible that the philosophy of “different search strings produce different results and that is OK” has been applied to the individual channel, when it should not.

    Perhaps what has happened is each channel [from the same common search string] is permitted to act in different ways, and never reconciled as they should be, because both are created at slightly different times, therefore treated as individual subscriptions/ providing different content, and acting on their own, when they are actually the same or closely matching, regardless the channel used to display the results.

    The question becomes: Should there be a step at this phase that says, “One of the multi-channels is reporting information, but the other mechanisms are not working; we have a problem; we need to back test our processes to find the glitches internally, reconcile those, and then proceed with assigning the correct address code to the content.”

    I suspect this backward testing does not occur; and the external test-platform [which PubSubX users have no visibly to] does not then take both of the multi-channels and compare the outputs to say, “We have different results, we need to change something, or find out why this is happening.”

    In all probability, the external test platform has been established on the assumption that a “valid result in one channel is unrelated to the result from another channel.” Again, let’s not confuse channels from subscription. A single subscription has multiple channels to find that content.

    Let us consider the possibility that the external test platform has been designed on the assumption that each of the channels is different. We remain unclear how the channels are compared: Linearly, or rotated by 90 at each phase of the process. PubSub would have to look into this. Because if the external test platform assumes that each channel is different, [when they should be working the same], there would never be a confirming check to match content from one channel against another; nor would there ever be a confirming check that there’s a problem in how each subscription was tagged.

    My guess is each channel works independently; when, they should actually remain separate as channel-providers, but still be compared to ensure their data remains consistent; or dig into the details when their content is not consistent.

  • What check is done to ensure the searches, as executed for different entry points along the various channels, are consistent between the [a] search done; [b] how archived; and [c] how reported?

    PubSubX may be either linearly analyzed or rotated by 90. Let’ assume that PubSubX has, for this point of the discussion, a milestone approach to checking data.

    It’s trouble enough to check the data along a single subscription. What’s more problematic is when you have multiple channels to access and view that content. The linear problem of a single feed is challenging.

    It remains to be understood what uniform mechanism is done on each search is done along the channel and then archived and reported compared to a different channel from the same search string. If the governing rules defining these internal checks along one channel is slightly different than another, then there is no common link that will tie this channel along a single path.

    Perhaps there is a break in the chain. A hole. One that continues to be reported as “there is no hole,” and the external test-platform says, “You are correct, there is no problem,” when the actual problem is data arrives, has not been tagged correctly, but the “poor tagging or improper tagging” either triggers something to discard it an everything else; or it correctly reports status as invalid, throws the wrong message into the register as “valid” and then this gets stuffed with “too much stuff’; the data is compared, the results don’t match, and then dumped because of an apparent problem with the search.

    This would presume that the back testing is working to clear out the content that is erroneously being reported. And the channel is dumping everything into the “Clear out” box, when it should be retained. Maybe it takes bad information, dumps everything, and never able to register a single positive result.

    Maybe the checks aren’t valid along the string; they tagged incorrectly; or the archiving does exactly the opposite of what was intended because of the “copy and paste” error mode; which in turn, affects the test-platform [as it should not, but does].

    Step 4: Report results

    Data is sent from the pool of XML feeds along the channel into the storage location. This is where PubSubX takes the content collected and stores it. Again, the issue becomes, “Why is one channel collecting, assigning, storing” in one way, while the other channel is not matching the first one?

  • Is the wrong XML feed code applied?

  • Why is there no uniform code applied to all channels?

  • How are the XML feed codes compared?

  • Are codes assigned, created, and stored using dissimilar parameters?

  • Why does one channel have one code, but the other channel [for the same search string] have a different XML feed code applied?

  • Is there a relationship between the codes created for each channel?

  • What does PubSubX do to each channel’s XML feed code differently?

  • Does PubSubX add something to the XML feed code that it shouldn’t add?

  • Does PubSubX remove something from the XML feed code that is needed?

  • Is there a reconciliation across the channels to ensure the same search across different channels produces reliable XML feed codes?

    Perhaps there are some commands that are telling PubSubX to do something with the particular feed, yet different sections of PubSub are differently reporting results.

  • How are different sections of PubSubX differently reporting results? Are the reports differently assigned to each channel; the content stored is assigned based on tagged codes that will never appear because the XML code is different that what is expected?

  • Is there no check done to ensure different entries provide consistent reporting outputs?

  • Are there commands that add things that should not be added; are there things that strip out codes that should not be stripped out?

    Step 5: Convert results from PubSub to output

    This takes the XML-assigned information and prepares it for display. This is where the final packaging occurs. Each section/input goes along a path.

  • Are there any intermediate checks done along the path to validate consistency?

  • After packaging, is some data revealed/added/deleted that should not be?

  • Is content assigned a code or value that mixes it up and creates confusion about “where to actually send the content”?

  • Is there no chance that cross flow of information reports, “No change” but there is actually a change and the comparison check is not done?

    Step 6: Present the report

    One section of PubSubX reports OK, but the other does not.

  • What checks are done on the holes?

  • How are the changes, inconsistencies identified, converted, updated, or validated?

  • Is there an internal check that appears to validate something, but the test validation is not done, or is the validation which tests “there is a hole” changed to report “OK”?


    Summary


    It appears as though what is happening is there are internal checks across different channels and search strings. The intended goal was to test the validity of the individual feed, but the checks do not ensure that each channel is uniformly operating the same with respect to each individual search. It appears the needed checks are not complete; or if complete, are incorrectly reporting status that does not match actual status.

    Either data is added, and no checks done after data is added; or data drops out but reports as complete; or data is identified as problematic, and the entire saved inventory of subscriptions is cleared out.

    There is a hole somewhere which PubSub internal cheeks hasn’t detected. A hole exists, but no actual test has been done to discover, detect, or report that hole because the assumptions of “what is supposed to be going on” do not match how things are actually going.

    Either the hole exists without doing the test; or the test is incorrectly done; or the tests are incorrectly reporting filled when it is actually empty.

    I think there’s a code that tests the validity of the output across channels; but the test is backwards: Falsely reports OK when there is an inconsistency; or the inconstancy is not detected, test and published as a valid result, even though it has nothing.

    I also suspect that there is a box that has a known limitation on the number of parameters allowed. This “box” is related to “what was assumed the user would use to read, display, and access the information.” Perhaps this “box” has parameters defined with specific values; but the actual values are different, not important, or interact in ways that are not as important or more important than originally planned.

    Questions

  • How was it determined the number of codes/size of content was accurate?

  • Why was this constraint chosen?

  • What happens if remove that constraint?

    It remains to be understood what assumptions the codes are using about:

  • The platform people are using to display the information

  • The assumptions on the size, form, acceptance properties of the result;

  • Whether the code platform is designed to work with one set of assumptions, but the end user is actually suing a method of access that is different, or not to e constrained by that assumption designed into the code.

    Theory

    Valid test reports incorrectly because syntax mashed in the instruction. A minor coding error has shoved two sections of the code together. Embedded code has been meshed because of a syntax error so it meshes the test with the check.

    What is needed is a review of the changes to see what was added and where; and look for subtle changes in the code that would permit the test to occur without completing the test, and then take output and store it, but read its output incorrectly.

  • Bad results not blocked

  • Goes to the end, registers content that should show “inconsistency” reports negatively/empty…generating a false positive to continue the search, even though it has actually failed.

    In reality: Test to validate across platforms searches doesn’t report, the problem with the reported code and output internal consistency problem because the test check doesn’t occur as planned/though/reported.

    Theory

    There is a roadblock that PubSub has, but doesn’t realize. PubSub was designed on the assumption of X-parameters; or they don’t test for those parameters; or they don’t check those parameters; or they create execute commands that do 180-degress opposite; comment at one of the test commands has a skip option, does not report a problem, but executes without actually doing the various [a] tests; [b] removals; [c] reviews; [d] reports; or [e] comparison

    Theory

    The very part of the system need to do the subsequent post-deployment checks reported OK, yet when it was not done, and test never executed, interpreted the results as “OK.

    The very code changed and updated was part of the tool to validate the internal inconsistency.

    All stress testing doe prior to the code update. Later used a cut and paste correction. Accidentally removed a minor font with paste. Reported valid on 1 of the channels, but did not validated “test check OK” with subsequent back testing across the channels.

    Theory

    Code change in how it interfaces with test tools. There is a test-tool PubSubX interface issue; because of the interface issue, tests errors falsely report OK and there’s no signal back to the platform to adjust something as designed-intended.

    Theory

    Coding error makes the required alert with a test tool.

    Test tool interface with PubSub: test tool depends on x-parameters; the code-interface issue with PubSub repots valid, even though the opposite: the cross check is not completed; reports OK because empty register [that could never get filled with errors because the test is skipped.

    Theory

    Problem isn’t with PubSub, but with the test tool reporting “OK” despite inconsistency between the channels.

    Theory

    Error in PubSub prevents test platform from reporting the error and test OK.

  • How does the error-mode makes the PubSub test platform integration issue?

    Possible explanation

    The way the executed code update affected the way they designed the interface; when changed would never have revealed the internal check-test-inconsistency because that part of the platform, when changed, would never report an error as expected.

    Review

    We’ve outlined theories. The next step is to generalize the lessons from PubSubX and apply them to FeedMesh. Recall, we have no idea what is causing the inconsistent results across the platform on PubSubX, but for purposes of discussion, let us presume that there will eventually be found a solution to the apparent problem.


    FeedMesh


    For FeedMesh to work, it needs to work with XML and any method to execute using XML. FeedMesh needs to have a single set of commands to internally testing with a single platform and across platforms: Test, execute, validate, and check.

    Move from just XML-standards, and focus on ensuring the platform-procedures;
    Are different platforms correctly identifying and reconciling integration issues?
    The key issue with Feed Mesh will be what method will be used to ensure different XML procedures are:

  • reconciled,
  • tested correctly,
  • identifying inconsistencies; and
  • correctly converting to the proper codes to provide delivery and provide the final report.

    If we have multiple channels that provide the same subscription-streams-feeds, then those duplications need not be eliminated on the assumption that it is redundant; rather, the redundancy can be used to crosscheck the validity of the inputs, and progress along the cycle.

    Having multi-channels forces reconciliation, crosscheck, and force consistency, not just with XML but the commands that are there to execute the objectives. There may be different methods to check, but these need to be brought together and reconciled.

    FeedMesh relies on multiple platforms integration. It needs to work with any approach, not just the standard.

    This is to say that XML and the coding approach should be viewed as two different animals in the same problem.

    The current PubSubX model concludes that there are checks and reviews that are not designed that would be included; or there are functions that are required which have not been incorporated.

    Bluntly, those holes need to be filled. Those checks need to be done. How is FeedMesh going to work with different developers—yell at them as well?

    Not all platforms execute the same.

    These checks, if absent, are needed. We need to make sure that the FeedMesh solution integrate across platforms, not just integrate XML. Different segments should be reporting the same results regardless the methods used in coding and design.

    What is needed is an internal check within the system to make sure the different parts are checking each other. Inconsistencies need to be reported and given visibility.

    Not at the end, but as things progress. Intermediate checks.

    They may already exist. The issue becomes, why are they not ensuring the different channels are working the same way in delivering the same content in the same way.

    When PubSubX gets seriously evaluated, you’ll be well on your way to overcoming a hurdle in implementing FeedMesh: Cross-channel testing based not just on XML standards, but in whether the functions are performing as intended, regardless the different approaches taken to achieve the same results.
    " />