03 January 2006

RSS aggregator searching -- Body of knowledge

Using aggregators and RSS for searching is different than a search engine. Self-evidently, RSS usually looks forward in time, while a search engine looks backwards.

[For clarity, I'll simply refer to forward-looking RSS-XML searches as an "RSS search" and a generic search using Yahoo/Google without XML as backward looking. However, in reality a search engine like Google and MSN do have quick ways to convert a search to a retroactive search, but that's outside this discussion.]

Sample things: Things that are missing and could exist

Before I go into the details, I'll illustrate the apparent gap in the existing body of knowledge. For starters, you'll get an idea of the differences in terms. Notice this search:

  • Teaching RSS searching; or

  • XML Searching Body of Knowledge.

    It sounds simple enough, but nobody's used it -- they're probably using other terms, but what? That's where XML Thesaurus Comes in.

    Now, think something more generic: Search tips: Brilliant 85 million hits.

    But does any of it relat to searching with XML; or some nifty tricks on using prospective searches for searches?

    Something more narrow is: RSS Search tips -- but it really doesn't give tips on how to integrate:

  • A. New ideas

  • B. New events

  • C. Potential blogs

  • D. Forward looking outlines

    Also, RSS search tips are geared more toward providers/experts -- publishers and technology users -- not a content-user/searcher-focused, or "how to translate your ideas into effective RSS search requests".

    Remember your user: SOlve problems, not burden them with technology details

    Remember, I -- your user -- don't care about RSS, technology, or the syntax. I want to use RSS/XML searches to solve problems -- assess unfolding situations; know when I am in a poor position to make a decision.

    What I really want is a guide on "how to effectively create RSS searches" -- and I want that methodology to integrate the prospective search feeds with the aggregators.

    While you're reading the following, consider this perspective: "Set aside your blog for the moment, and focus on an outline in your RSS-XML aggregator." The goal of this approach is to make better use of the future information, spend less time blogging, and more effectively manage/organize/structure the incoming information before it arrives, not wade through the mess as it arrives.

    Body of knowledge

    After working with RSS search for over a year, I can see that blogging, search engine searches, and RSS searches overlap, but they are different.

    Specifically, the problem we run into with a prospective search is "our phrases" we may use to describe a future event [ and hopefully future hit] will not necessarily match what others use.

    There's a trick to using RSS feeds and searches. If you have a blog and have "new ideas" about "things going on," rather than blog about things [and producing a rant], you can save yourself alot of time by doing something else.

    That's where the body of knowledge comes in:

    A. How is a prospective RSS-search created differently than a generic Google-Yahoo search;

    B. How can a blogger's "blogging time" [time spent blogging ideas] be reduced;

    C. How can time "spent on blogging" get translated into more effective RSS searches?

    I do not claim to have the answers, but I define the answers as the "RSS Body of knowledge" in terms of "how will people come to use/apply RSS searches" and "how will RSS searches reduce the time spent bloggging."

    In short, the "way we do a Google search" has specific tricks. It's not just in the terms, but in knowing how to quickly translate a search requirement into terms. This is based on "knowing what could be out there".

    The same goes with RSS searches: How to forecast what will "likely be out there" . . . even though the issue, words, or RSS-searches may not exist.

    In other words, why bother to blog, when I can find "what everyone else is saying" that matches an outline I'm developing?

    Rather than blog, it may be more useful to create a template or generic outline, where the RSS feeds can be entered into the aggregator, and the outline -- not the blog -- becomes the means of organizing information.

    As time passes, and the issues get fleshed out, "it would be nice" if this vertical outline, could expand to the right, and create a schedule, whereby the more sophisticated, and in-depth answers to these issues appears: Either in terms of concepts; or in terms of answers to questions earlier posed; or in terms of detailed discussions of the subsequent discussions.

    Think of the GMail e-mail weaving option: You can have the information time-based, so that the answers to original e-mails are linked to the subsequent answers. The Aggregator-model would have the same thing: You would like the original RSS concept, with the idea, with the subsequent answers, and then the detailed information.

    Think of this as an expandable link/answer/response. If I have an original question, issue, or idea, then associate this with a range of answers; I can visually show how the original ideas [not blogged] are graphically-logically linked with the subsequent arguments/discussion/responses.

    Then comes the next phase: Whereby the responses are then compared to the original questions; the arguments/points/discussions are organized into logical arguments; and then the overall schema created.

    The schema for organizing this information may or may not exist; it could be compared to a generic template; and the differences between the emerging pattern and the original template could identify new patterns.

    The point would be to clearly identify a priority to the issues; and then identify "which concepts/arguments/ideas" should be showing up; and then comparing with what is actually showing up. Some of the "actually stuff others are using" may not have as much strength/weight, so it would be nice if there was a way to assign some waiting to the desired search results.

    If our rating goes from 0 to 100, and the "desired hits" are arbitrarily put at 50; but all we're getting are 10s and 20s -- this is a good idea that "our original list of desired hits" is above and beyond what people are talking about.

    At that point, it may be useful for the blogosphere to "learn about these other views" that tend to have more weight, validity, importance, than the apparent lesser valued items at 10-20.

    For those of you who are lost -- that's the point -- RSS searching and blogging are related -- the trick is to know when it is time to listen, how to listen, and how to translate "our ideas" into effective RSS-search terms. This is known as the "body of knowledge" related to RSS-searches: Tricks on how to more effectively create RSS searches based on what we might have blogged.

    Maybe it is emerging, but all of the above is what I've learned in "how to make RSS prospective searches" more useful in identifying trends, gaps, and what is a "value added blog" or something that merely repeats low-value content.

    I'd say more, but why give away the juice on how to do this better than others?

    Wish list

    Using a prospective search tool -- Once the XML feed/subscription is in the aggregator, it would be nice if . . .

    A. There was a flagging system: "Hay, I was supposed to have had a hit/positive result by this X-date, but have nothing. . . what's up?"

    Sometimes even though the RSS-search produces no result, it would be nice to know, "Even though by X-date, we should have had something, we have nothing. . ." -- that it itself is useful to know, and may trigger other search requests, or hunting with other terms.

    B. Some sort of flexible tool that allows one to keep fast notes, notice patterns, but isn't a public blog.

    Something that makes it easy to quickly

    A. Document new information
    B. Rotate the information into new patterns
    C. Translate the new pattern/ideas into a prospective search
    D. Translate the newly-recognized pattern [that has no information, just vague ideas] into something that is a specific request for future content
    E. Translate the RSS-feed into an outline
    F. Adjust the outline into a new pattern
    G. Recognize the value/scoring of the returned content
    H. See which content is or isn't stacking up against the proposed pattern
    I. Adjust the outline so that more attention is put on the valuable searches, that still await content
    J. Rank the value of the content [based on factors I choose, not necessarily what others are using] so that I can see which discussions/arguments/factors [are/are not] getting filled/satisfied/discussed.
    K. Translate the vertical-outline [2-dimensional] into a third-dimension [horizontal, using weight factors related to time, expected responses]

    Next Step

    Each of the templates or checklists for the event could either be manually created, or imported from a generic list of template-checklists. The searches could be assigned to the existing template; or the template could be adjusted based on the content.

    Based on a few indicators, I could import generic checklists/templates, automatically created RSS searches, and wait for the actuals to show up. The approach would identify what was occuring; and compare them with what I/the outlines forecasted.

    Also, simulation data could be imported to track the expected times; and the duration of a particular news-event cycle: Then compare the forecasted times with the actuls -- using that information to forecast other items: Delays, cost overruns, actual time/budget to accomplish an objective.

    If I had a discrete number of indicators, I could automatically create forward searches based on the expectation they are related to a specific checklist, pattern, or cycle. Over time, I would be able to monitor whether the actual events occurred along the expected/forecasted RSS feeds.

    -- This is the end of the content --
  • Using aggregators and RSS for searching is different than a search engine. Self-evidently, RSS usually looks forward in time, while a search engine looks backwards.

    [For clarity, I'll simply refer to forward-looking RSS-XML searches as an "RSS search" and a generic search using Yahoo/Google without XML as backward looking. However, in reality a search engine like Google and MSN do have quick ways to convert a search to a retroactive search, but that's outside this discussion.]

    Sample things: Things that are missing and could exist

    Before I go into the details, I'll illustrate the apparent gap in the existing body of knowledge. For starters, you'll get an idea of the differences in terms. Notice this search:

  • Teaching RSS searching; or

  • XML Searching Body of Knowledge.

    It sounds simple enough, but nobody's used it -- they're probably using other terms, but what? That's where XML Thesaurus Comes in.

    Now, think something more generic: Search tips: Brilliant 85 million hits.

    But does any of it relat to searching with XML; or some nifty tricks on using prospective searches for searches?

    Something more narrow is: RSS Search tips -- but it really doesn't give tips on how to integrate:

  • A. New ideas

  • B. New events

  • C. Potential blogs

  • D. Forward looking outlines

    Also, RSS search tips are geared more toward providers/experts -- publishers and technology users -- not a content-user/searcher-focused, or "how to translate your ideas into effective RSS search requests".

    Remember your user: SOlve problems, not burden them with technology details

    Remember, I -- your user -- don't care about RSS, technology, or the syntax. I want to use RSS/XML searches to solve problems -- assess unfolding situations; know when I am in a poor position to make a decision.

    What I really want is a guide on "how to effectively create RSS searches" -- and I want that methodology to integrate the prospective search feeds with the aggregators.

    While you're reading the following, consider this perspective: "Set aside your blog for the moment, and focus on an outline in your RSS-XML aggregator." The goal of this approach is to make better use of the future information, spend less time blogging, and more effectively manage/organize/structure the incoming information before it arrives, not wade through the mess as it arrives.

    Body of knowledge

    After working with RSS search for over a year, I can see that blogging, search engine searches, and RSS searches overlap, but they are different.

    Specifically, the problem we run into with a prospective search is "our phrases" we may use to describe a future event [ and hopefully future hit] will not necessarily match what others use.

    There's a trick to using RSS feeds and searches. If you have a blog and have "new ideas" about "things going on," rather than blog about things [and producing a rant], you can save yourself alot of time by doing something else.

    That's where the body of knowledge comes in:

    A. How is a prospective RSS-search created differently than a generic Google-Yahoo search;

    B. How can a blogger's "blogging time" [time spent blogging ideas] be reduced;

    C. How can time "spent on blogging" get translated into more effective RSS searches?

    I do not claim to have the answers, but I define the answers as the "RSS Body of knowledge" in terms of "how will people come to use/apply RSS searches" and "how will RSS searches reduce the time spent bloggging."

    In short, the "way we do a Google search" has specific tricks. It's not just in the terms, but in knowing how to quickly translate a search requirement into terms. This is based on "knowing what could be out there".

    The same goes with RSS searches: How to forecast what will "likely be out there" . . . even though the issue, words, or RSS-searches may not exist.

    In other words, why bother to blog, when I can find "what everyone else is saying" that matches an outline I'm developing?

    Rather than blog, it may be more useful to create a template or generic outline, where the RSS feeds can be entered into the aggregator, and the outline -- not the blog -- becomes the means of organizing information.

    As time passes, and the issues get fleshed out, "it would be nice" if this vertical outline, could expand to the right, and create a schedule, whereby the more sophisticated, and in-depth answers to these issues appears: Either in terms of concepts; or in terms of answers to questions earlier posed; or in terms of detailed discussions of the subsequent discussions.

    Think of the GMail e-mail weaving option: You can have the information time-based, so that the answers to original e-mails are linked to the subsequent answers. The Aggregator-model would have the same thing: You would like the original RSS concept, with the idea, with the subsequent answers, and then the detailed information.

    Think of this as an expandable link/answer/response. If I have an original question, issue, or idea, then associate this with a range of answers; I can visually show how the original ideas [not blogged] are graphically-logically linked with the subsequent arguments/discussion/responses.

    Then comes the next phase: Whereby the responses are then compared to the original questions; the arguments/points/discussions are organized into logical arguments; and then the overall schema created.

    The schema for organizing this information may or may not exist; it could be compared to a generic template; and the differences between the emerging pattern and the original template could identify new patterns.

    The point would be to clearly identify a priority to the issues; and then identify "which concepts/arguments/ideas" should be showing up; and then comparing with what is actually showing up. Some of the "actually stuff others are using" may not have as much strength/weight, so it would be nice if there was a way to assign some waiting to the desired search results.

    If our rating goes from 0 to 100, and the "desired hits" are arbitrarily put at 50; but all we're getting are 10s and 20s -- this is a good idea that "our original list of desired hits" is above and beyond what people are talking about.

    At that point, it may be useful for the blogosphere to "learn about these other views" that tend to have more weight, validity, importance, than the apparent lesser valued items at 10-20.

    For those of you who are lost -- that's the point -- RSS searching and blogging are related -- the trick is to know when it is time to listen, how to listen, and how to translate "our ideas" into effective RSS-search terms. This is known as the "body of knowledge" related to RSS-searches: Tricks on how to more effectively create RSS searches based on what we might have blogged.

    Maybe it is emerging, but all of the above is what I've learned in "how to make RSS prospective searches" more useful in identifying trends, gaps, and what is a "value added blog" or something that merely repeats low-value content.

    I'd say more, but why give away the juice on how to do this better than others?

    Wish list

    Using a prospective search tool -- Once the XML feed/subscription is in the aggregator, it would be nice if . . .

    A. There was a flagging system: "Hay, I was supposed to have had a hit/positive result by this X-date, but have nothing. . . what's up?"

    Sometimes even though the RSS-search produces no result, it would be nice to know, "Even though by X-date, we should have had something, we have nothing. . ." -- that it itself is useful to know, and may trigger other search requests, or hunting with other terms.

    B. Some sort of flexible tool that allows one to keep fast notes, notice patterns, but isn't a public blog.

    Something that makes it easy to quickly

    A. Document new information
    B. Rotate the information into new patterns
    C. Translate the new pattern/ideas into a prospective search
    D. Translate the newly-recognized pattern [that has no information, just vague ideas] into something that is a specific request for future content
    E. Translate the RSS-feed into an outline
    F. Adjust the outline into a new pattern
    G. Recognize the value/scoring of the returned content
    H. See which content is or isn't stacking up against the proposed pattern
    I. Adjust the outline so that more attention is put on the valuable searches, that still await content
    J. Rank the value of the content [based on factors I choose, not necessarily what others are using] so that I can see which discussions/arguments/factors [are/are not] getting filled/satisfied/discussed.
    K. Translate the vertical-outline [2-dimensional] into a third-dimension [horizontal, using weight factors related to time, expected responses]

    Next Step

    Each of the templates or checklists for the event could either be manually created, or imported from a generic list of template-checklists. The searches could be assigned to the existing template; or the template could be adjusted based on the content.

    Based on a few indicators, I could import generic checklists/templates, automatically created RSS searches, and wait for the actuals to show up. The approach would identify what was occuring; and compare them with what I/the outlines forecasted.

    Also, simulation data could be imported to track the expected times; and the duration of a particular news-event cycle: Then compare the forecasted times with the actuls -- using that information to forecast other items: Delays, cost overruns, actual time/budget to accomplish an objective.

    If I had a discrete number of indicators, I could automatically create forward searches based on the expectation they are related to a specific checklist, pattern, or cycle. Over time, I would be able to monitor whether the actual events occurred along the expected/forecasted RSS feeds.

    -- This is the end of the content --
    " />