Member update-able Technical Reference website

An interactive technical reference that accepts notes, data, and new information from everyone.

The actual techical data starts with an outline / overview

How to add your own comments, questions, code, etc.. and even new pages(!) to any page.

How to take ownership of a page and edit pages you own.

Possible hosts:

Technical feature / to-do list.

• Make an ADO datasource for a members directory
  • Member contact information (or a link to the Windows or MegaPhone Address book)
    • Collect member information by a standard form, but use two frames or windows and after leaving each field (or when there has been no keypress event for a certain time) use the information entered to start a search for matching contact records which are listed in the second frame or window. The user can select or approve results there and cause them to be copied into the form.
      • On entering the field, display a very quick help screen in the bottom frame. Probably just a few words written directly from the javascript
        • Email: it would actually be cool to do an email extraction from IP address, but that is really not possible.
        • Phone: just a few words about how to enter it and what will happen afterwords. Public phone records can be used to look up name and address.
        • Name: "do you already have an account on this system under a different phone number and/or email address? If so please enter your old phone / email here so we can accuratly update the existing record rather than duplicating your information." then trigger a lookup on the old phone / email as though it was entered above.
      • First, search the local database on the server via members.asp. Return only enough information to uniquely identify the contact to prevent the use of the page to extract members details. E.g. when entering the name, if there are matches in three different states, fill the list with "from <city>, <state>?" Don't release email addresses, phone numbers, and street addresses.

        When a match is found for:

        • Email: Only offer to send an email with the password. Remind the member to log in to change contact info
        • Phone:
        • Name: if it will uniquly ID the member show "from <city>, <state>?" and if selected, fill in City, State, Zip, etc... and the street address without the street number or other numerical information removed. Also maybe not company. The main purpose is to prevent multiple contact records for the same person with different emails and phone numbers. Pulling up the other details from the database improves the chances that they will be submitted exactly as before and will be recognized as an existing record. The street number will be re-entered exactly, where other details like "Ave." vs "Avenue" may not be.
        • Company: same as name, but all information includeing address, city, and state can be listed. This saves people the time of entering company information again, and also prevents duplicate records for the same company and location.
      • Second, member.asp should redirect to a search engine that is appropriate for the type of data entered.
        • For email addresses, search Whois on the domain. Finger would be great, but probably only useful on well managed Unix intranets. The whois will help for work addresses (and names) of any corporate or university users and may even turn up direct contact info for private domain owners. Unless we can tell that it is an ISP, there is no point in asking for conformation, just populate unentered fields.
        • For phone numbers, search anywho or think direct and display the actual result page. Use the frameset onload (which is activated after all the documents in the frame set, i.e. both frames, have loaded) to popup an alert asking if the result is correct and should be transferred or perhaps to enable a "transfer remaining data from below" link or button.
        • For company, same as phone number, perhaps useing a business directory.

        When a secondary match is confirmed, use javascript to parse out the data and transfer it into the form. Don't over-right fields that have already been filled out (err... unless it was filled out by a non-ISP email domain match? maybe conformation should be required for those?)

  • Member Password (encrypted)
    • Store passwords in a cookie on the members browser after they have logged in. Delete the cookie after they have logged out and remind them to log out if they are not useing the reference from a home machine
  • Member user ID number
    • a relativly short id tag that is placed in the HTML as a part of an <A> type tag (maybe <A title="JMN-EFP" ...> for example) around content (<A title="JMN-EFP" name="">) or file link descriptions (A title="JMN-EFP" href="myfile.htm"> submitted by this member and in a <META name="author"> tag (<META name="author" content="JMN-EFP">) in the header of each file posted by the member. Use a comment end code with the end of the content link.
    • develop the ID from the members contact data, maybe initials of name followed by initials of company name or some sort of geek code.
    • need: allow the member to edit and approve the ID (check for duplicates before approving members changes) so that they retain ownership of the ID tag.
    • a second tag or a code at the start of the members tag indicates that a particular content is for publication or is to be kept private. <A title="P-JMN-EFP"... or <META name="author" content="P-JMN-EFP">... Make sure no one ends up with an id that starts with P- before the private code is added.
    • after verifying the user password on login, place the ID in the session variables.
  • need: Add an ASP page to display the members ID's (no names unless oked by member) with links to a form that uses another ASP page to send emails so that the members email address is not given out.
  • need: Add an ASP page that allows members to change password, set options for the display of thier name, company name and email on thier postings ("James Newton of massmind.org says:"). Member.asp provides this EXCEPT for the password.
• When serving web pages,
  • Check for a valid session containing Member ID number.
    • If not found, redirect to a login directory so that a cookie previously placed in the members browser will be appended to the client request. If the cookie is not available, explain the membership requirement and ask for a password. After completeling login, update the cookie, increment the useage count for the file being requested and redirect back to the original page.
  • check the user ID number against the <META name="author"> and deny requests for private content not matching the authors ID. In the denile, allow the member to send a request to the author of the private content via a form and a sendmail script rather than by mailto: so that the authors email is not released.

    intI = Instr(strFile, "<META name='author' content='P-")
    If intI > 0 And Mid(strFile, intI, Len(strID)) <> strID Then
    	Response.Redirect "/techref/private.asp?id=" & strID & "&" & Request.QueryString
    	Response.End
    	End If
    
    

  • Replace all the tags with private member content with comment start codes so that the content is not visable on the browser. It would be visable in the source but this is normally just the file link text or description of the file and the actual content will not be served even if requested due to the <META name="author"> tag in the file.

    strFile = Replace(strFile, "<A title='P-", "<!-- ")
    
    

  • Replace all the comment start codes followed by the current members ID number with <A HREF tags so that that members content again becomes visable.

    strFile = Replace(strFile, "<!-- " & strID, "<A title='" & strID)
    

  • This has been updated to use regular expressions and actually remove all private comments.
• At the bottom of each subject page, add a query.asp?more=<pageurl> offering to search for more information on that subject. Add ?from=<pageurl> to the end of each result link and parse that when serving the result to be used in (need:) a "did this answer your question?" popup which links the search result page back into the origial subject page.
• Allow users to follow / edit / update / comment on / add links  / new pages / upload files / sub-sites. (help file for page edit, add, addpage)
  • Edit content in a text area (HTML raw editing) or if DHTML is available, by a WYSISWG HTML editor in the browser.
  • need: When text is added to an existing page, split it into words and lookup each word with a LIKE '"&strWord&"*' in the database of page titles / post subject lines. If a match is found, check to see if the title matches the words following the first and if so, link that part of the text to the other page.
  • need: Lookup each unlinked word in a dictionary database and place a superscript "?" with a link to a "please correct or define this word" page that offers suggestions for misspellings or a place to enter a new definition.
  • done: Check words in update against spam keywords...
  • need? ...or include a captcha.
  • done: An "email me changes to this page" link. Totally automatic. If the page is edited, it gets mailed to a list of people who are interested. Update.asp now does this. The list of interested people are stored in a list titled "Interested" and they may be indicated by member-id or by actual email address. Each entry in the list is marked as private.
  • need: Add a compact version of the update to a file called new<year><month><day>.htm in the same directory as the updated file. Include a link to the page, the author, and the text. If it was an edit, a difference engine should be used so that only the change is recorded. Offer to send it to users. The major pain here is for pages that were edited. Finding a good, human readable diff has been diff-icult.
    • http://www.openbsd.org/cgi-bin/cvsweb/src/gnu/usr.bin/diff/analyze.c?rev=1.1.1.1&content-type=text/x-cvsweb-markup
    • http://perl.plover.com/diff/
  • need: Log the updates out to a newsletter or email list (members option)
    • http://www.15seconds.com/issue/970406.htm ASP list server
    • http://www.worldwidelearn.com/ezine-publishing.htm free list servers
  • need: Include a java applett or ActiveX control that can edit / display and simulate in a very minimal way, electronic circuits or diagrams.
    • http://www-elec.enst.fr/~dufourd/java.html (cached 20010119122539)
    • http://sunsite.utk.edu/winners_circle/education/ED8N1T2I/ (cached 20010612103238)
    • http://www.particle.kth.se/~fmi/kurs/PhysicsSimulation/subjectIndex.html
    • http://hpv17.infosys.tuwien.ac.at/melange//html/programming.graphics.10.html (dead)
    • http://www.vidyut.com/sunit/JavaOOPDraw.html
    • http://www.15seconds.com/issue/971214.htm Active X controls in ASP pages.
    • http://twiki.org/cgi-bin/view/Wikilearn/EditingATWikiDrawSketch A Java applet that can edit and save SVG standard web vector graphics data.
    • http://math.hws.edu/TMCM/java/xLogicCircuits/index.html Nice little applet that allows one to "iconify" or "expand" circuits and save them to build more complex circuits.
    • http://weblab.sourceforge.net/ Java based applet or stand alone program for circuit drawing combined with a server based simulation engine.
  • need: Also have an optioin to post questions that are listed in a seperate place (always at the very bottom) and automatically post them to the techref member list or to what ever list is most appropriate for the subject area if the poster has a membership in that list (help them set it up if they wish). Add a link to the techref area before the question when sending it so that readers have a good idea of the context and can click to see what info is already present before repeating it. Watch for answers and make a new content entry based on question and answer. May want to edit before posting.
  • need: A database of consultants related to each area with a "would you like to be referred to an expert in this area" if the search results or browsing is not helpful. The consultants pick what pages they want referrals from and maintain their own contact information.
  • done: Add a "<name, company with or without email> says: <BLOCKQUOTE>" to the start of postings and "</BLOCKQUOTE>" to the end at the members option.
  • need: If a member is deleted (for inactivity or at request) any pages they have created should probably be reverted to not-owned unless they are marked private. Same thing with posts where the link to the member page comes up 404.
• Maybe via "control panel" (seperate window?).
  • need: Include some javascript to parse the title and href of each link and display an information table with user-frendly version of: authors experience,  when onMouseOver event occurs. Use onMouseOut to clear.  
  • display document.lastModified or other document information.
  • need: Include some javascript to link to a dictionary site when a word or phrase is selected to provide definitions
  • need: Setup a tool bar with B, I, U, H1,2,3 left right center, bullet, number definition lists etc that inserts the appropriate tag around selected text in the "edit" text area. On timeout, update the document in the main window to show a wysiwyg preview of the new html. Add a save button that posts the new page or what ever to the server. Probably doable in Javascript or Java
• need: Periodically run an ASP page to index all the pages in the local site and update thier last modified dates into a data base (files in techref.mdb) along with a cross reference (links in techref.mdb) that shows what files link to what files. Also update the title into the database to use in autolinking. To improve speed, process only one directory (folder) at a time and add/flag the subdirectorys in the folders table to be processed later. Hint: if the number of files in a folder changes, flag it as out of date. Also, check the last date the folder was indexed and only find/update the files that have changed since that date. Flag each file for link checks on the next run.  The database will also hold (for each file) the date of last upload to the public site, frequency of use, date link list updated,  and an exclude flag for non-postable content. For each folder, the path, date of last index, number of files.
  • Add the <title> of the html document to the file database and use this information for a search engine. Each page holds one concept?
  • link the files in a folder with the index.htm file in that folder or with <folder>s.htm in the parent folder to implement a heyerarchy in the database?
    
• Periodically run an ASP page that picks the least recently checked link then
  • If the link is remote, checks the link.
    • If its ok, cache copy in a directory path made from the url by converting all ":", ".", "?", "=" and "\\" to "/" (e.g. cache / http / www / microsoft / com / dir / subdir / subsubdir / file / parm / value / parm / value) and add/update a "cached <date>" link after the regular link that points to the local copy.
    • If its not available, add/update "down <date>" between the regular link and the cache link. Note: The date on the cache is the last date the page was available.
  • If the link is local,
    • If its ok, no action
    • if its not ok. try to fix it or email the webmaster.
  • Updates the last checked date in the record.
• Now that remote pages are cached, run a search engine through the local copy and report remote url (reassemble from cache location), page of the site where the link was found (lookup in link table) and cached content excerpts.
• Processing Priorety:
  • Are there any files that are known to be out of date and that haven't had thier links indexed? Add / Update links to link table.
  • Are there any directorys that haven't been indexed? Index the directory to the files and dirs tables
  • Are there any files that need to be updated to the remote site? Modify and upload the file to the remote site.
  • Are there any links that haven't been checked / cached? Check and/or cache the link. Update the file and the link table entry
  • Re - index the oldest directory past X hours old
  • Re - check the oldest link past Y hours old

Current problems:

• fixed Images can not be served because only .ASP files are allowed by the server and default.asp seems to kill .GIF files when returning them. I think it may need to MIME encode them. see:
  • http://msdn.microsoft.com/library/periodic/period96/periodic/msj/f1/d3/s25f.htm
  • http://www.learnasp.com/listserv/archives/binaryfiles.asp

    Yep! That did it, now it works, but it is slow so the best thing is to keep images in a seperate (/images) directory which is directly accessable.

• fixed http://204.210.50.240/techref/default.asp?from=/techref/&url=mailto:TakeThisOuTLISTSERV at ~NOSPAM~MITVMA.MIT.EDU?body=SIGNOFF%20PICLIST Worked around by makeing the original HREF =" vice href=" (added a space between the HREF and the =) so that the preprocessor does not change the link to a redirected one. This is slightly incorrect HTML syntax but at least IE4 deals with it. This is done now that we have a asp 404 page.
• fixed Cant pass parameters to asp pages from links in the site. E.g. indexok.asp?n=1 generates an error. Should be ok now that we have an asp 404 page.
• fixed The urls are a bitch. The 404b.htm page has been modified with some JavaScript that redirects to the correct url given something in the standard form but an asp 404 page would be better. Now with the new server license, this works and it is very nice.
  • http://www.asptoday.com/articles/20000120.htm doing 404 with ASP on IIS. Must set type to URL vice file.
  • http://www.php.net/manual/html/install-windows95-nt.html config PWS using regedit
  • http://www.pstruh.cz/help/urlrepl/library.htm
  • http://www.west-wind.com/webconnection/docs/manuallyconfiguringmicrosoftwebservers.htm
  • http://users.neca.com/vmis/filter.htm
  • http://www.15seconds.com/issue/961220.htm ISAPI filter 404 redirector
  • http://www.winmag.com/library/1996/1196/11hd1.htm ISAPI filter sequence of events and debugging.
  • http://www.microsoft.com/technet/IIS/technote/serverop.asp Server Performance Optimization on Microsoft's Web Site

• Can't upload binary files like pictures, etc... There are solutions that allow uploading pictures, but none insure that the pictures will not be porn, stolen, or otherwise unsutable. Tools that allow a user to draw diagrams, charts, or schematics would be more valuable. Pictures can always be emailed to the admin.
  • http://www.pstruh.cz/help/ScptUtl/library.htm

• fixed Web harvesters, offline browsers, and other site rippers just freaking kill the server. Every asp page needs a bit of script to keep track of the last x requesting IP addresses and when more than some number of requests are recieved in a short time, redirect to a "Hey! Slow Down" page. indexok.asp provides that and more. When more than 6 "Slow Downs" have been issued in a few minutes, the user IP is added to a list of blocked users. All asp pages check that list before providing content and redirect to indexok.asp?blocked when found. All asp pages also look for known BOTs (WGET, etc...) and refer to indexok.asp which blocks. See also: Server Failures

See also:

• http://umbraco.org/ An open source, C#.NET CMS.
• http://www.plone.org A very nice CMS. (with a massive security hole)
• http://citeseer.ist.psu.edu/707719.html An interesting multi-user proxy based content management system
• http://kmt.hku.nl/~pieter/SOFT/CGI/wikic.html WikiC is a standalone wikiserver written in the C language.
  It is not a CGI program which runs under a webserver, instead, it is a webserver itself.
  +
• TakeThisOuTbeexcel at ~NOSPAM~hotmail.com refers to

  http://www.futureality.com/page_extractor.asp Page Extractor extracts Email Addresses, Page Links, Image Paths, IP Addresses, Fax Numbers and Phone Numbers as well as custom data extraction with regular expressions. http://www.futureality.com/page_extractor.asp

• http://www.c2.com/cgi/wiki?WikiWikiWeb user editable hypertext db which is presented in an HTML template. Implemented in perl

  Interesting because, there is no membership requirement. Anyone can add anything. Pages can't be deleted if anyother page references them.

• http://www.xanadu.com/ It is, after all, Ted's fault...
• http://www.xanadu.com.au/
• http://www.hyperwave.com/
• http://www.crit.org/ Getting close...
• http://sourceforge.net/projects/webwatcher/ WebWatcher - WebWatcher - a Web-page Update Monitor This program will help you keep an eye on interesting Web-pages. You register a list of URLs you want to monitor, and WebWatcher checks for changes whenever you ask it to, or at given intervals.

James Michael Newton says:

to convert HTML to plain text... or at least strip out all the tags...
join(filter(split(replace(strFile,"<","><"),">"),"<",false),"")

James Michael Newton says:

Add a new action to the update script: "addunder" which has a "heading" parameter. Basically, you do a search (for the heading) and replace (with the heading and the text)

to make a form where part of the text is hidden, use two fields with the same name and recombine them in the script.

James Michael Newton says:

http://www.cs.sunysb.edu/~algorith/files/longest-common-substring.shtml

Redirect from an IP stable URL to your dynamic ISP assigned dialup IP address after registration and a browser hit to a cgi script that updates the redirection to the current value of your dynamic IP address. Putting the browser with a default URL of the update cgi URL in the startup folder causes your machine to connect to the ISP and update your current IP on the redirection server

Notification of site indexing by www.mitsi.com The following link sends an email with the name and IP address of the indexing web robot when the page is indexed:
<A href="http://web.mitsi.com/netresults/email.pl/jamesnewton@geocities.com/techref/www.geocities.com/SiliconValley/Lab/3544/index.htm"> &nbsp; </A>

Questions:

  .