2. What should be the nature of the mathematics part of

Wikipedia? I'm asking this because I see it as more or less a replacement (or even better) as things like Eric Weisstein's Mathworld or Foldoc. But there the emphasis is on concise write-ups with formal definitions...

"Concise" is not as valuable here: we have no space limitations, so we should take as much space as is necessary to clearly explain something, give examples, and some background as you suggest. I also think our audience will be more laymen than that of something like a textbook: after all, the audience of a textbook is self-selected math students, while we'll be getting random search hits, so I think it is important for us to explain and link to context in every article.

However, I think that in order to be usable as a mathematical dictionary (should it be?) the short formal definition should be near the beginning (or >maybe even always *at* the beginning) and clearly recognizable as such. So I guess my question is actually if there should be some kind of rule on this.

I think most articles will end up this way: define, then describe. How that's done for each subject will be best determined by experts in that subject, with feedback from the rest of us.

3. A related question is how much redundancy do we want?

Is "reflexive" going to be explained on every page that uses its in its definitions, or do we want one small article that defines it and let all the others link to that (as in Mathworld). If we do write such an article what should be the title? Should it simply be "reflexive" or "reflexive binary relation"? I would say the latter because the term "reflexive" has a higher chance of having other meanings in other contexts.

Redundancy is not bad per se, but linking to separate articles makes it easier to give more detail when needed. Titles should be as clear as necessary--your judgment is as good as anyone's on that matter. There is lots of discussion about these topics on Wikipedia already, and some general consensus about several things, but it would be a bad idea to set up rules to far ahead of time-- we want to see what works first, then edit things based on our experiences. Once we get the content, that's easy to do.

4. What is the current opinion on using HTML 4 special characters?

I'm using 'mozilla' and it handles them fine. Even text-ased 'lynx' tries to represent them with normal characters which results often in a quite readable result. And it would of course be really nice to have things like subset, and, or, forall, element et cetera.

LOTS of discussion exists here. For a first pass, see "Wiki special characters". The symbols on that page should work in most browsers of recent vintage. Mozilla is better than most: IE and NN won't show you "forall", "subset", and some others. A product like TtH would be useful, but would shut out Mac owners and possibly others. Again, use some judgment: if you think you can explain a subject without needing the special characters, then do so; but if you really do need them, go ahead--browsers will catch up. If you can get by with the few extended characters in ISO, then use those directly instead of HTML entities (such as "not", "times", and "middot").

There's lots of discussion about these topics already in Wikipedia, and your contibutions there would be helpful.