Frequent database rebuilds for greylisting

Vernon Schryver vjs@calcite.rhyolite.com
Mon Dec 8 16:28:43 UTC 2003


> From: Gary Mills <mills@cc.UManitoba.CA>

> ...
> constantly.  I can see dbclean running.  When it completes, it becomes
> a zombie.  As soon as the zombie disappears, another dbclean starts up.
> This seems too resource-intensive for my liking.

Which resources are intensively used?  How long do the frequent greylist
dbcleanings take?  A zombie burns little more than a process table
slot, which is trivial compared even to the page tables for a ting
process these days.  Chasing the other hash chains to fix other records
would not be cheap.

> The usual way to delete a record in a database is to mark it deleted,
> and then compress out the marked records later.  Can't dccd do this?
> ...

That is basically what happens.  However as in many other databases,
there are things that need to be cleaned up.

] To enlarge on this idea, is there a timestamp associated with each
] record?  If so, could dccd set the timestamp to some magic value for
] deleted records?  Then, dccd could ignore those records when responding

What happens now is that the target counts of deleted records are
set to zero.

The DCC database does things that are impossible in other databases.
For example, the main database on one of your systems has about 10
million keys for probably 20 million globs of data.  It uses only
450-600 MBytes and achives millisecond reads, additions, and updates.
Any other database you are likely to find would need at least 10 times
the disk storage and be more than 3 orders of magnitude slower.  Even
the 40 times less data for a grey list for your installation would be
a major strain for other schemes.  However, that performance comes at
the cost of equally great restrictions on flexibility.  The DCC database
is not a database as most users think of the word.

I will look for a way to avoid the dbcleaning for ordinary greylist
operations, because I realize that it is impolitic.  However, it
would be nice to know if they are a major technical problem and so
urgent.


Vernon Schryver    vjs@rhyolite.com



More information about the DCC mailing list

Contact vjs@rhyolite.com by mail or use the form.