• Change the blue sidebar symbolizing recently active topics in the ToC to a bullet/circle/clock icon. Move the topic titles a bit to the right, so there's space for this "recent activity" icon on the left.
• Change the indicator for search results in the ToC to a magnifying glass. Drop the shades of yellow, see below for the numbers.
  
• We now have 4 possible icons on a ToC entry: new activity, search results, my contribs, watchlisted
• Each of these icons would carry several bits of information and give access to several actions:
• Main signaling function: the presence of the icon indicates a certain property of a topic
• Secondary information: the style of the icon (e.g. color) can contain more details on that property; a number overlaid the lower right corner of the icon (similar to Pau's ToC button) indicates how many posts are matching this property.
  A mouseover tooltip could describe the meaning in a few words and/or give further details.
  
• Primary action: a left-click on the icon will scroll the board to that topic and mark the affected posts by a colored sidebar on the left. The "< >" buttons will appear in the top bar and allow jumping between these posts (note this applies to all icons, not just search. Very convenient for flipping through new activity!) The "1 of 42" can be skipped since the total number is shown with the icon, and the running counter is not that interesting anyway (or is it? Probably better to have some kind of temporary 'mark as read' mechanism, but that's another story...)
  To be decided: in which order to jump through the matches. Ascending chronological order would be great for new activity, newest first might be better in the other cases.
  
• There should be some indication which icon is activated that way. One could e.g. remember the good old times when buttons looked three-dimensional and had a very obvious 'pressed' state. To keep the modern look, we could keep the inactive icons flat and only show the active one on a recessed square.
  Clicking any other icon would activate that filter instead of the current one, clicking the same icon again would go back to the unfiltered board (one-click undo without moving the mouse, nice for discoverability).
  
• Secondary action: a right-click (long tap on mobile) will open a context menu, which offers first the primary action again (or not? not sure), then one ore more related actions, and finally an entry "advanced search".
• Each icon would also appear on the top bar, in the same order, and with very similar functions. The difference to the individual topic icons would be
  
• The numbers count matching posts on the entire board, not just one topic.
• The primary action would also reduce the ToC to show only topics with matches.
• The "recent activity" icon (clock):
• Would indicate that there are recently added/changed/removed posts in green/yellow/red ('recently' as in 'since my last visit', this works only if the topic or board is watched)
  
• Reduced to white with outline when there's nothing new (not removed entirely, to still give access to the context menu)
• Secondary actions: mark all as read (set watchlist timestamp to the time the page was loaded, not the time the icon was clicked!), pick a fixed timespan from a list (the latter works without watching).
  
• The "search results" icon (magnifying glass):
• not shown in topic lines unless there are search results
• no secondary actions except for 'advanced search'
• has the search bar next to it in the header line, typing here will press the button
  
• The "my contributions" icon (person symbol)
• Reduced to white with outline when no matches
• Secondary action: show contributions of another user (pick from a dropdown list of all contributors to the topic/board)
• The "watchlist" icon (star)
• Shows if I'm watching the topic / anything on the board (and the number of posts in those topics)
  
• Outlined if not watching
• Actions on topic lines - primary: watch/unwatch topic (this is a bit out-of-line, not sure how to resolve that; maybe ok with tooltip); no secondary actions
  
• Actions on header line - primary: reduce ToC to watched topics; secondary: watch/unwatch board (including different types of watching a board, once available)
• The ToC icon would still open/close the ToC, its number would show the number of topics currently belonging into the ToC, which means the total number of topics on the board if there are no filters active (this gives a quick overview how large the board is, e.g. if one should bother with the filters or just scroll down). Note that this is not redundant with the number in the active filter icon, since those count posts, not topics.
• This should cover everything except the sort order selection, which could go to the advanced search panel, redundantizing the "..." menu. Together with the removal of "v 1 of 42 < > x", this leaves more than enough space for the three new icons in the header line.
• There could be a "help icon" at the very right of the header line, linking to a page that explains all this much better...

I'm not sure if the right-click context menus are discoverable enough, but then, you can do quite a bit without them, and there's a help button.
All the icons and numbers could sound like they're overwhelming for newcomers, but: if you look at an unknown board, not watched, not contributed, not searched, then there are only outlined icons and no numbers. All the information will be added in steps as you become involved.

Phew, that became a longer story than I thought. I hope it makes sense, I'll just throw it up here and see what happens...

Best
Hhhippo