[back]
[Abstract]
[Copyright Notice]
[Contents]
[next]
Smart Cache Manual - Chapter 12
What's new/History of Changes
0.44 8.4.2000
This another very quick bug-fix release. It looks like 'Smoke test' on
Internet is required before releasing, but I do not have even modemoid.
Also my keyboard looks like it will be dead soon.
You can now start to use referer_log without problems. It is a good thing. Nobody want to contribute an icon for Smart Cache?
Bug fixes
- Fixed crashes (from 0.43) or corrupting Referer:
header bug (from 0.37)
when fake_referer is not used and referer_log is used.
- Removed NullPointerException ignoring in main loop.
It there is an error, request will still fails, but you see the exception.
If you see a NPE it is a SC bug and report it.
0.43 6.4.2000
As expected,
0.42 have some serious screw-up in Referer: header handling. This is a
bug fix release. Thanks for all bug reports. It is
a simple: If I do
not know about a bug, I can not fix it. It looks like Smart Cache is
most popular OS/2 software in Germany :)
I have received a working 486 board donated by Monika Schilling.
Java compiler runs much (2-3x) faster, Thanks a lot!
If you never read Bhagavad-Gita or even thinking about purpose
of (your) life, there is a very good starting point. See Interactive Bhagavad-gita, appendix C.
Bug fixes
- TODO list revised to version 2.0.1 and added priority to it.
Nobody wants to donate a working KEYBOARD? :)
- Fixed bug (new in 0.42), which prevents referer_hack from working, it
corrupted every Referer: header instead.
- -import should work under non-Unix OS also (Windows, OS/2).
- corrected some bugs in import, import sometimes fails if object is already in cache.
- import checks if already loaded object in cache is valid.
- File system check must be performed before import.
- Errors are displayed even in load-headers HTTP transaction phase (was
only in connect phase).
- When announcing invalid OBJ reference, SC will print why reference is invalid.
- Removed setDirty() call, used direct variable access -> faster execution of non-JIT JVMs.
- Remote server errors on refresh (RC) are also printed.
- Better error handling when multiple times loading the same URL and
some of loads fails. I hope, that now there are all multiple-load errors
handled correctly.
- Blocked requests are counted as `good` when computing Smart Cache hit
ratio, because they are also handled by Smart Cache.
New features
- New keyword flag_check_interval. How often will Smart Cache
check for presence of immediate_shutdown_flag. Using this type of shutdown
is better than killing the task. See Stopping Smart Cache/Shutdown flag, section 5.3.
This is most requested
feature from Czech users. (Non-Czech users most often requests JunkBusters.)
- New command line switch `-direct`, if used SC will not
use http_proxy.
- New command line switch `-http_proxy`, overrides http_proxy in scache.cnf.
- referer_hack do not longer accept fake URL.
- New keyword 'fake_referer' for setting fake Referer:
HTTP header.
- Import will refuse to import directory with names without '.'
- Data from another Smart Cache can be imported, see Smart Cache command line switches, chapter 7.
Import makes checks if it is really SC data directory (subdirectories must be numbers).
- New keyword 'fake_cookie', used instead wafer.
0.42 2.4.2000
This version was not tested as much as 0.40. If you find some fatal bugs,
send a report and use old 0.40 (see README for list of URLs). 0.40 has
no known fatal bugs.
Bug fixes
- Host names are no longer case-sensitive, when checking in cookies filter or no_proxy.
- Better detection of deleted directories (by hand), when SC is running.
Now it works under Linux better.
- .cacheinfo files are checked for write access.
- Import can works also on read-only filesystems.
- Timestamp is changed after import (by copy), but not after move.
- Import: clashing file name generation corrected.
- Import tries to rename file first (preserving last write date),
if it do not work, it will try to copy it.
- Better error handling when importing.
- Repair now also deletes empty directories.
- Generating of error messages is now faster - green variant. (Creates less Java objects on heap)
- Linger option is more friendly to OS resources.
- Reordered generated headers for avoiding browsers bugs.
- Access.log should have correct timezone now (DST is supported).
- filesystem check is now always performed.
- Non parseable Expire headers are not ignored.
- Distribution classes are no longer hashed by HashJava.
- External mime file can contains # comments.
- No crash on startup with newer beta versions of Kaffe JVM, but GC DO NOT WORK in
Kaffe.
- Request will be processed even if client send bad data format in headers.
- Check if we get negative Content-Length.
- Removed trailing / or \ from cacheroot directory, if user uses it.
(Fixed problems with OS which do no like two // or \\).
- Backslash test is performed in cache data directory, not from global
OS setup (File.separatorChar).
- Do not allow swap_dir1/2 to be < 1.
- CRC32 directory hashing function is simplified (read: faster) and works ok when swap_level1_dirs > swap_level2_dirs (read: brain-dead algorithm was replaced with dumb algorithm).
- -rebalance switch also kills empty directories.
New features
- Much improved this documentation. There will
be a lot of English grammar mistakes. Included in main archive because
Homepage may be unavailabe/outdated.
- Support for special HTTP/1.1 methods: PUT, OPTIONS, DELETE, TRACE.
WWW publishing via HTTP1.1 should work. (I can not test it, I have no such program.)
- Support for SSL/CONNECT gatewaying. These requests are handled
directly and not send to parent proxy server (http_proxy).
If someone needs this, send me mail and I will add security_proxy
in next version.
- New keyword allowconnect for allowed SSL/CONNECT ports.
- New keyword fail_trace for tracing which mask blocked requested
URL.
- New keyword visible_link for setting Smart Cache's homepage
link in generated error messages. (You can direct it to Hobbes for new version checking via Search).
- New keyword mime_types for using external mime table, when
Importing files to Smart Cache, section 6.11.
- New keyword stat_log logs statistics to file.
- New keyword append_via can disable appending "via Smart Cache 0.XX" to your User-Agent: header. If you are not using fake_user_agent this will (somewhat) protect your privacy, because nobody will know, that
you are using Smart Cache.
I hope that this also disables problem with slightly
modified pages without having to use fake_user_agent.
See Troubleshooting, chapter 10.
- Smart Cache repair utility, chapter 8 significantly enhanced.
- More checks when testing cache integrity (checking filesize and date).
- When running GC/repair we will not touch any file, which are less than 6 minutes old (they may be in use).
- -p command line switch. See Smart Cache command line switches, chapter 7.
- Return code 307 is now cacheable.
- Check if cacheroot directory exists and exit if it can not be created.
- You can use hardcoded 1x1 GIF when using ErrorDocument 403 0. Use it
if you can, it is much faster than sending redirect.
- Added bunch of startup quotes from Bhagavad-Gita.
- Load failed error code is 500 instead of 400. I do not know why many
caches uses 400, it can confuse some non-interactive clients.
- Every Pragma header is now considered as no cache request.
- SC will not crash when gets incorrect port number (not a number) in
request.
- New keyword clear_flags_on_start. Should be any flags files deleted when starting?
- Striping Pragma and Expire headers. Discovered by Sergey Levin, when
looking on packets. This helps people, who use this
cache only as front-end cache to browser cache. Smart Cache removes bad
headers and pages will end in browser cache. This kind of use was
discovered and reported by Martin Konicek. He also writes utility for
running Smart Cache in Windows/systray.
- When checking for already matched rule, Smart Cache displays
already matching rule.
- SC always display delta time, when URL is refreshed. If page do
not have Last-Modified date, time of URL receive will be used.
- Partial object downloads are now included in bytes stats.
- Failed object loads are announced with exception info.
- New command line switch '-repair' it calls recursive repair on data
directory.
0.41 8.9.1999 (Limited release)
Bug fixes
- Some typos
- Changed from FREE software to Open Source Software, which is better, because many crippled demo versions claims there are also FREE software. License is still GPL.
- English version of this documentation was checked with ispell -d british.
New features
- GNU Debian/Linux binary distribution. (.deb package) can be built
from Unix sources. Not tested too much and is very primitive.
- Configuration files can be read from alternate directory -cfgdir
command line switch. Needed for better UNIX support.
- Smart cache can now also operate on read-only file systems,
such as CD-ROMS or HPFS drives under Linux.
- Smart cache will test filesystem if is read only.
- Cacheroot can be changed by command line argument -cacheroot
<Directory> [ swap_dirs_1 ] [ swap_dirs_2 ]
- Smart cache can reorder cached files after changing swap_dirs. -rebalance command line option.
repair
has now better built-in MIME table.
repair
has now -q command line switch. Prints less messages.
repair
can read external (Unix style /etc/mime.types format)
MIME table.
- rewrite command line handling. It is possible to use more than one
command line option.
- directive reload_block_files is now obsolete. Changed
files (except
scache.cnf
) are now always reloaded.
- -import <Directory> command line option for
importing data into smart cache directory structure from external
sources. Place downloaded data from web servers (for example with
some web grabber) into some directory for example /tmp/www,
where servers will be stored in subdirectories such as /tmp/www/www.linux.org and import them with java scache -import /tmp/www
0.40 11.3.1999
Bug fixes
- Distribution archive for 0.39 contains classes from 0.38 not 0.39.
- Reloading of block files now works without any side-effect.
New features
- experimental GUI removed. Slow, resource hungry and prevents Smart
Cache from running when no X-server is available. Never again.
- cookies block list can be also loaded from file allows_cookies_to_file
- Warning when config file is not found
0.39 10.3.1999
This document has been revised by Luc Le Blanc, thanks for it.
Bug fixes
- In BUGS file is documented internal Java error. Smart Cache
will report it, send these reports to SUN (not me!).
- Corrected web forwarder, which doesn't works since changes in 0.37.
Why Nobody reports it?
- Socket Linger option (maybe) doesn't works. It's hard to test it
without LAN or WAN.
- Support for Expires: headers generated by IIS 3/4 servers, which
violates HTTP protocol.
New features
- Repair.java program for broken .cacheinfo files (no docs, src only)
has been improved. Use this program for importing files into Smart Cache file system.
- Implemented OUTGOING cookie filer as requested by Klaus Nitsche.
See cookies.txt and Have fun.
- Wafer option (fake cookies)
- Experimental GUI option -gui (unfinished)
- Simple requests and bytes Statistics.
- Refresh time can be also printed in hours
- new keywords pass_file, fail_file. Smart Cache can now
load these directives from external files and when this file is changed,
Smart Cache will detect it and reload it - no need to restart it as before.
0.38 10.2.1999
Bug fixes
- Fixed case-sensitive bug in the regexp engine.
- Another bug gets (at least) documented. Read BUGS.
- Bad content-type header don't crash whole request.
- Default mime type is text/html (like in NutScrape).
- Lord Vishnu is no longer default in fake_agent.
New features
- Repair program for broken .cacheinfo files (no docs, src only)
- password protected documents can be cached
- Smart Cache now displays error (instead of crashing) when
talking to HTTP 0.9 server. (untested)
- fake garbage collector (run via -fakegc). No files gets deleted.
- You may turn on case_sensitive URL mask checking - this considerably speeds up
garbage collection. Profiling GC on my 386/40 was a PAIN! For your information -- It takes about 7 hours.
- Delta time is displayed in days for better readability if greater than 2d.
- Added socket linger option for better support http clients (like my web-grabber).
I didn't test it since I'm more than year unemployed and don't have LAN at home.
0.37 20.12.1998
License has been changed to GPL! Share and Enjoy!
Bug fixes
- Corrected HEAD requests handling. Message body is not sent anymore.
Some stupid web grabbers were too confused. Yes, WGET again.
Because I'm tired with WGET, I started coding my own web grabber.
It's in a very early stage of development (not yet working), but will probably
be the most configurable web grabber ever seen.
- No_proxy now case-insensitive.
- no_proxy localhost will no longer match localhost.xxxx.com
- Log writers are synchronized, so logs are now consistent.
- Some internal optimization tricks based on the Java-Library source.
- Registration keys are no longer needed.
- Last version check and Ad Busters removed. The server side needed for
this stuff is not online.
- Smart Cache crashes less often on bad config file. Sometimes even
prints the number the offending line!
New features
- Support for agent_log and referer_log. Happy logging
and forwarding.
- Directive full_referer_log (0/1). 0 produces a more compact log,
referer is logged only when servers are different.
So you can record your path in cyberspace :)
- no_proxy, fail, cacheonly, nocache can have more than one argument.
Arguments are separated by SPACES.
- Authentification in url ( http://looser:kewl@i.hate.ssh/ ) now works
properly.
- Unknown or unsupported protocols (FTP, gopher) are now forwarded to
your http_proxy (if you have one) and received reply is, of course, cached.
So happy off line FTP sites browsing!
- Default filename (.welcome) can be now redefined. UNIX users may like this.
- When generating filenames, numbers are inserted at beginning, not end -
programs may now still recognize file based on file extension .
- -local command line switch. Forces Smart Cache to run in local mode. Smart
Cache will not try to connect anywhere.
- New garbage collection action '='. File will ALWAYS be
removed if older than X days.
urlmask ftp://* =7
Removes
all files received by FTP when not used in 7 days.
0.36 20.11.1998
Bug fixes
- Due to bug in stupid Wget, I added correct handling of requests with #. No
more multiple downloads with Wget.
- Message cleaned empty dir is now sent to stderr. Less garbage in stdout.
- Some less important internal changes (i already forgot them).
New features
- referer_hack has been extended and can now send USER-specified
URL instead.
- Error_document also supports return code 400 (proxy load failed).
- New feature Fake_user_agent used for making WWW logs more
interesting.
Note: via line is not added to string specified.
0.35 20.9.1998
Bug fixes
- Some internal optimizations for speed in cache engine. GC engine
is still not yet optimized for speed (because I run GC only about twice a
month).
- Support for bunch of new protocols such as hTTP, hTtp, HttP, ...
- Handling errors in multiple downloads of the same object is now
smarter than before. (I wonder if anyone will even notice it).
New features
0.34 12.9.1998
Bug fixes
- Faster in-memory garbage collection.
- Added header Proxy-Connection: Close to generated error
messages. This fixes troubles with old browsers, which don't have proper
handling of Keep-Alive connections.
- Security bug: Password-protected documents were still cached!
New features
- Choice of removing Pragma: no-cache header from clients requests.
See
scache.cnf
for more info (look for remove_pragma_no_cache)
- Customized Error Reply by redirecting to specified URL. Only 403 Error
reply (Forbidden, generated by Fail directive) is now supported.
- Nicer HTML in generated error messages.
- Ad Busters moved to another server, so -faststart switch is no longer needed
for workaround to JVM DNS caching bug.
- Less important init messages are now sent to stderr.
0.33 5.9.1998
Bug fixes
- Workaround for bug in Squid Cache 1.2.X (if Squid is used as http_proxy).
- Improved some messages generated by garbage collector engine.
- Default value of maximal object size increased from 4 to 10 Mb.
- Improved error detection - Smart Cache can now detect when user
of garbage collector removes directory with active file transfers.
New features
- HTTP RC 403 added to group 'temporary remote server errors'.
- By popular demand, size of transfered objects now displayed in
Smart Cache window.
- Refreshed object without last_modified date (delta=0 m) no longer
mentioned in Smart Cache window.
0.32 19.8.1998
Bug fixes
- Fixed bug in garbage collector. GC crashed when there were
fewer objects in cache than gcarraysize.
- Cacheroot directory is now always created on cache startup.
0.31 18.8.1998
Bug fixes
- Fixed some rare cases which created "unreferenced" objects when
object loading failed.
- Fixed bug in GC. Due to changes in 0.30, only 50% "unreferenced"
files were removed.
New features
- Extended directive generate_lastmod. If set to 1, generates Last-modified
header only if object is sent from cache. This
removes problem with both stupid browsers (MSIE, Netscape) and aborted
transfers. If you want to restore the previous operation mode, set it to 2.
- Local filename generating is now significantly faster.
This makes Smart Cache about 10% faster when operating with "normal" URL length
and much faster with very long URLs.
- URL Search string can now be delimited with more chars:
:? = ;
- Smart Cache got TOP 25% Rating from J.A.R.S.
0.30 3.8.1998
Bug fixes
- Last-Modified dates in future no longer confuse the refresh engine.
- In some rare cases, when object load failed, object was deleted
from cache, but not from .cacheinfo files.
- When directory creation fails, Smart Cache no longer creates
file nullnull, when attempting to transfer cacheable object.
New features
- Garbage collection! It's configurable through gc.cnf file.
Run it via java scache -gc and enjoy!
- New directory structure - character ~ is no longer replaced by -.
If you want to convert directories to new format (recommended), use the
-convert030 switch.
0.29 5.7.1998
Bug fixes
- Corrected new error from 0.28 - documents without expire line
don't get cached if pragma_nocache were set to zero.
New features
- Command-line option -faststart. Suppresses latest available version
check and Ad Busters rule set refresh.
- Better last-modified. headers generation, now works even on transient
objects.
- Possibility to send pages from cache if remote server returns internal error.
Good for overloaded servers, such as Geocities.
If you want this, set hide_errors to one.
- English version of docs.
0.28 2.7.1998
Bug fixes
- Handling errors when reading corrupted
.cacheinfo
files.
- Faster saving
.cacheinfo
files.
- Directory memory cache optimized.
- Internal optimalizations for better speed.
- Corrected sended HTTP reply 304 when load failed.
- Ignoring Expires: header in the past.
- Removed filesystem's case sensitivity check.
- Check if is possible to use character '\' in filenames.
New features
- Sending pages which are already deleted on WWW servers (rc=404),
but still existing in cache.
Directive keep_deleted [0/1]. Good for off line browsing
of removed sites.
- Smart Cache can now generate last-modified headers if
document is cacheable and original do not have one.
generate_lastmod [0/1] - it is very good for
web forwarding.
0.27 23.6.1998
- Product license changed from FREEWARE to shareware. Some
features exists only in registered version. Introduced built in web
forwarder.
- Changed syntax for default_refresh_pattern to
reload_age min_age lastmod_factor max_age.
Lastmod_factor is a floating point number and other numbers are times
in minutes. These numbers are used when testing if document needs
to be refreshed. Setting reload_age value > 0 disables reloads for specified time. Setting min_age value > 0 allows caching pages without
Last-Modified header, which makes web browsing much faster.
- Smart Cache has been optimized and now is five times
faster than before. Now is only about 25% slower than Apache 1.0 for OS/2.
- Changed POST method handling - added workaround for some
badly programmed CGI scripts.
- Password protected documents are not cacheable now.
- Smart Cache now beeps, when new version exists.
- Smart Cache will exit (not hangs) when can not bind to specified port.
- Changed handling of redirect - proxycache
now do not sends redirected back to the browser, but remaps the
request and continue working on it - it is necessary for web forwarding.
- Classes are before distributing protected by hashjava utility. This
also make them 10kB smaller.
Registered version
0.26 6.6.1998
- Fixed vsb
[14]
when processing HEAD requests. Smart Cache caches results of this
requests and this overwrites normal pages with zero-sized pages.
- Adding Host: headers to outgoing requests with
port number.
- Corrected file system check - works even if cacheroot directory
does not exists.
- Better resolving of the filename/directory clashes.
- Cache now works better in some rare situations, when
is not possible to create working directory for requested pages.
These pages are still not cached, but are now sended to clients.
- Directive inputtimeout - socket read timeout
for client's request.
- Directive proxyreadtimeout - timeout for reading
reply from remote web server.
- Both time are in seconds.
0.25
- Changed error message Target not set to Host
Unknown
- Hostnames are now no longer case-sensitive.
- Protocol names are no longer case-sensitive.
0.24
- Added support for my AdBusters! web service.
- AdBustersId and AdBustersAuth for AdBusters
login.
- Cache can check if newer version of itself is available.
- Check for unnecessary Fail or Pass mask (already matched
by others).
0.23
- Wildcard '*' support in all directives but
Redirect and No_Proxy
- URL are no longer case sensitive. (Fixes problem with
IBM Alpha Works site)
- Directive NoCaching for defining what URL you do not
want to cache.
- Directive CacheOnly for defining only what are allowed
to gets cached.
- You must use only one of them
- New keyword Pass for marking allowed URLs, which
Fail disables (by mistake).
- Some generic traps for adv. banners installed in default
configuration file.
0.22
- Because many users use this program in commercional environments
for caching entire networks even if this is not allowed by license,
Smart Cache has now hard coded bindaddress to 127.0.0.1.
- Better error messages, displays also thrown Java exception.
0.21
- Corrected URL parser for working with URL containing strange
characters such as '=;%^&'.
- do not create filenames with '|' inside
- If-Modified-Since header is sended as first line
in request.
- If cache assumes that object is in the cache, but
can not be accessed (for example is deleted), request headers will
not be send to server twice.
- Smart Cache do not works with Java 1.0 (only missing class is CRC32).
- Shutdown flag is deleted on the start.
0.20
- New structure of
.cacheinfo
files. Now last access
date is included (needed for future garbage collector).
- Faster exit on shutdown_flag
- Header Server: added to error messages.
- Error messages has no expiration time 5 minutes (like Squid).
- All dates are now in the 'correct' Internet format including
day of the week (some servers tested DOTW on IMS requests!!!)
- Dates with day < 10 now has two digits size as required in RFC.
Java claims to support 'Internet date format standard', but zero on
the beginning was missing.
0.19
- Removed nearly all headers from generated 304 replies as required by
HTTP/1.1 protocol.
- Changed case in generated headers. Some stupid
servers has case sensitive matching on headers.
- More important dates has now correct day of the week.
- Corrected crashing when URL has more than one '?' - (Microsoft's servers).
- Kill unreferenced files tests if does not exist directory
with name of the cached file.
0.18
- When network error is detected when sending (after
sucessfull connect) request to
server, old cached object will be sent.
- Also if network error is detected when reading reply from server,
send old object.
0.17
- If-Modified-Since requests from clients are now
correctly handled in all cases.
Sending bad responses crashed IBM WebExplorer browser (Netscape
was fine) and entire Presentation Manager (OS/2 GUI Interface).
- Cache do not crash if gets date from remote server
in unknown format (expires: headers send by developer.javasoft.com).
0.16
- Check if filesystems allows to create filenames with dot at end only.
(seen on the Hare Krishna World Online).
- Thread synchronization is now much better.
If Smart Cache gets many requests per second for loading the
same object into cache, object will be loaded and lost. (Discovered
by my
webbench
program.
- Removing headers Proxy-Connection and
Connection from client's requests. Some wants to use
persistent connection, which Smart Cache do not support.
- Smart Cache is now smarter when tests for removed
files or directories by user.
0.15
- Cache appends its name to User-Agent header.
- Corrected error (buffering problem) when
processing POST requests.
0.14 23.3.1998
- Cache displays information about objects which are successfully
checked via IMS.
- Better handling of aborted requests. It is not necessary
to wait for remote server reply to be request considered as aborted.
- Shutdown_flag for clean cache shutdown.
- Better support for multiple downloads of the same object.
0.13 - 22.3.1998 First public release on the Net
- When got 404, delete object in the cache
- Cache only `good` error codes from server, not bad, which overwrites
`good` cached object (Squid does this overwriting).
- Temporary files are deleted, when transfer is aborted by network
error.
- Support for case sensitive filesystems with auto detection.
- pragma_no_cache introduced
0.12(Private) - 21.3.1998
- Clients got 304 if they send IMS request and connect fails.
- -killunref introduced.
- Error messages has now last-modified and
Expire: headers like Squid.
- added header Date: to server responses. Some browsers
use this for last-mod check.
- quick_abort introduced
0.11(Private)
- new directives no_proxy, fail, redir
- Better dir cleanup - directories are now deleted up to the dir
hierarchy as much as possible.
- Directories are now cleaned when garbage collected from memory, not
on saving.
0.10(Private) - 20.3.1998
- More intelligent IMS refresh
- Better memory directory garbage collector
- I/O buffer set to 2048 bytes instead 512
- background directory saver set to 10 minutes
- better multi-thread synchronization
19.3.1998
- Core coding is fully completed.
- Let's debug in the real world!
- First 'Smoke test' on the Internet.
- something from www.javaworld.com really gets cached, but due to
bugs in the thread-synchronization many objects gets lost.
17.3.1998
- very basic skeleton of cache appeared
- coded handling of POST requests, but untested.
12.3.1998
[back]
[Abstract]
[Copyright Notice]
[Contents]
[next]
Smart Cache Manual
0.44
Radim Kolar hsn@cybermail.net