per at hedeland.org
Wed Mar 16 00:50:02 UTC 2005
First, this is long, very long, and doesn't discuss NTP at all - just
move on if this offends you. Second, I strongly disagree with those that
claim that this discussion is off-topic and should be taken to private
e-mail - meta-discussions about a newsgroup's form and/or charter are
quite relevant (within limits of course), and while there is only a
handful of people that have actually complained about what the mailing
list gateway is doing to the group, per standard Unenet/mailinglist
demographics it can reasonably be assumed that many more are silently
either suffering or simply dropping off.
That being said, I'll also tentatively disagree with those that think
that the influx from the mailing list is lowering the technical quality
of the group. For as long as I can remember, there has been "stupid"
newbie and/or off-topic questions posted directly to the newgroup, and I
don't see the messages coming from the mailing list as having a
perceptly higher ratio of such. Possibly the opposite, as a large
proportion of the messages from the mailing list, for some reason, are
sent by people that regularly provide potentially useful answers to
I can't really buy the argument that the mailing list is needed for
those that don't have access to Usenet though - surely none of the
abovementioned posters are in this category, and in this day and age the
intersection of people that a) have access to unrestricted e-mail, b)
are unable to point a web browser at groups.google.com, and c) have a
need to discuss NTP, must be vanishingly small. I can certainly
appreciate that some people *prefer* the mailing list medium to either
real Usenet or a web-based interface to same, and it's perfectly fine to
provide them with a mail gateway - as long as it doesn't negatively
impact the group.
But in my opinion the non-Usenet-standard form of this gateway is a
severe negative impact on the group, in the way that it fragments and
disconnects discussion threads, leading to redundant posts, and
responses to questions (or vice versa) being hard to find - in short, a
general disorder compared to a properly functioning Usenet newsgroup.
And in my opinion, this is not acceptable. Since Brad has explained that
he doesn't have the technical knowledge to modify the gatewaying
software, I can certainly help out with that, having implemented several
news<->mail gateways without these annoying properties. But there seems
to be some other obstacles - read on...
In article <mailman.18.1110671158.576.questions at lists.ntp.isc.org> Brad
Knowles <brad at stop.mail-abuse.org> writes:
>At 10:01 PM +0000 2005-03-12, Hans Jørgen Jakobsen wrote:
>> On my reader (slrn) the gateway'd articles seem to
>> 1) break threading
> This is the Message-id: issue. This is not going to change. See
This is a most interesting page, since it gives an entirely different
motivation for mailman's behaviour than the one given in the code (which
you earlier - indirectly - referred to). And the motivation given here
is just plain wrong:
1) Unless you assume that all news<->mail gateways use the same
software, or that there is a standard way to rewrite Message-IDs in
news<->mail gatewaying (obviously neither is the case), modifying
Message-IDs in any way greatly *increases* the risk of infinite
loops. It isn't hard to imagine a scenario where two different
gateways each do their own munging of Message-IDs, neither
recognizing the form the other uses, with the result that a message
can circulate forever mail->news->mail->news->...
In contrast, preserving the Message-ID means that the buck in
principle stops the second time someone tries to feed the message
into Usenet - it will be rejected since the message already exists
(for those that don't know it, this is a fundamental property of the
Usenet flood distribution mechanism).
2) There is an abundance of methods to solve the "don't send messages
from the mailing list back to the mailing list when doing news->mail
gatewaying" problem, that don't include munging of Message-IDs. The
most common method is to solve it at the news server side, using the
same mechanisms that normally prevents a news server from even
offering articles back to the peer it received it from (i.e. the Path
header). Another possibility is to have a small local database of
messages that have recently been gatewayed mail->news (using the
unique and unchanging Message-ID as key, of course). And the "add an
extra header" method also works fine with a properly functioning news
3) The "Zawinski algorithm", which is identical to what Wayne Davison
implemented in trn (modulo the use of In-Reply-To) back when
Netscape was at most a gleam in Jamie's eye, can *not* compensate for
the breakage caused by mailman's munging. A posting that doesn't have
*any* Message-ID from a Usenet posting in its References header (the
normal case for a first-level followup from the mailing list to an
original message from the mailing list) can never be properly
Messages that have some Usenet Message-IDs and some non-Usenet in
References can possibly be placed at the right level in the thread,
but not fully linked into it - in particular it's typically not
possible to find the direct parent. This is something you have to
accept for a followup to a private or missing message, but totally
inexcusable when the parent message is sitting right there in the
group - but with a munged Message-ID.
The Subject/date/etc sorting is just a crude workaround "when all
else fails" - it is not threading, but can at most place a message in
the "general area" of related posts. And even this is frequently
defeated by mailman's Subject munging (in particular for the
first-level followup case above - those messages can't even be
"Subject sorted" but ends up totally disconnected from the parent).
Again, this is something that a decent newsreader should implement as
a last resort - but it's totally inexcusable for any software to
systematically force this kind of breakage onto a newsgroup.
 In-Reply-To does not have any defined semantics for Usenet messages
(it would be reasonable to call it "invalid", if it weren't for the fact
that what standards there are specifically "allow" any "undefined"
headers). Of course a newsreader implementor is free to try to take
advantage of it, but it isn't a reasonable requirement. It is however
quite reasonable to expect that mail->news gatewaying software combines
In-Reply-To and References into the only one of them that are relevant
for Usenet, i.e. References, as part of the format conversion it
*should* do (a concept that seems to be unknown to the mailman
developers). And on that topic, a gateway that munges Message-IDs must
*of course* munge References/In-Reply-To in exactly the same way.
In contrast to this, the motivation given in the code was at least
technically correct, even if misguided. It concerned the possibilty of a
message being sent simultaneously to two mailing lists, both gatewayed
to Usenet (though at different places), or of having two different
gateways from a single mailing list into different newsgroups. In these
cases, gatewaying without Message-ID munging typically means that only
the "first" gatewaying of the message is successful, the others will be
rejected since the message already exists on Usenet - and so e.g. the
"cross-mailed" message will only appear in one of the two newsgroups it
"should" be in.
This is a pretty well-known shortcoming of mail->news gatewaying, and
generally accepted given that the occurence is relatively unusual and
that there is no good solution (the non-existent "standard way to
rewrite Message-IDs" could handle it) - certainly way preferable over
causing the type of damage to *all* messages that mailman does.
>> 2) modify Subject header. ie insert "[ntp:questions]", sometimes in
>> several levels
> The gateway does not do this. I've checked the gateway source
>code, and when configured to do so, it goes out of it's way to avoid
>adding the subject prefix to messages being posted to the newsgroup.
>This is done by default, and the gateway we run is configured this
> However, what does not happen in Mailman 2.1.5 is the *removal*
>of all previous examples of the prefix which may have been put in the
>subject line by MUAs.
Sorry, but this is nonsensical. Mailman inserts the string on messages
sent out on the mailing list - none of the newsgroup's business *so
far*. A properly functioning MUA will not modify the Subject on a reply
other than by prefixing with "Re: ", or allowing modifications performed
manually by the user. Ergo, all responses on the mailing list will still
have the string that mailman inserted, and it is still there when
mailman gateways the message into the newsgroup. Saying that "the MUA
put it there" makes about as much sense as saying that the MUA produced
the text of the message - technically correct, but entirely pointless.
> One solution to this problem will be
>incorporated into Mailman 2.1.6, which I am planning on installing as
>soon as it is officially released.
>From the source I looked at, this would be a quick one-liner to fix - no
need to sit around waiting for the next "official release". Removal of
the Message-ID munging might actually require commenting out more than
one line, IIRC. The ability to modify and fix is after all one of the
major advantages with using open-source software.
per at hedeland.org
More information about the questions