I'm just getting access to the Polymorph spell this level, and I'm curious about beasts with innate capabilities. Swarm of Cranium Rats (CR5) are probably the strongest example:

Innate Spellcasting (Psionics). The swarm’s innate spellcasting ability is Intelligence (spell save DC 13). As long as it has more
than half of its hit points, it can innately cast the following
spells, requiring no components:

At will: command, comprehend languages, detect thoughts

1/day each: confusion, dominate monster

Do you have access to a monster's innate capabilities while you are polymorphed into them? I'm interested in learning the answer for all monster traits in general, but I'm particularly focused on spellcasting.

You can use a beast's innate abilities while Polymorphed (but you probably can't Polymorph into a swarm of beasts)

Polymorph states that:

The target's game statistics, including mental ability scores, are replaced by the statistics of the chosen beast.

And as the guidance on reading monster entries indicates, a monster's game statistics are everything in its stat block, which includes any special actions available to the monster and special traits it has. If a beast has an innate property like spellcasting or darkvision or telepathy, it is available for you to use when polymorphed.

However, you may have the problem in your specific example of becoming a swarm of cranium rats, because Polymorph allows you to assume the form of "a beast", singular - and a swarm is a collection of many beasts, as explored a little in this question about wild shape and swarms.

Polymorph gives you access to all of a creature's abilities in their stat block

The target’s game statistics, including mental ability scores, are replaced by the statistics of the chosen beast. It retains its alignment and personality.

Anything that is in a monster's stat block is considered their statistics (including any innate abilities)(see here for more about what counts as "statistics"). You can use any of them just as the monster can.

A swarm of cranium rats is likely not valid for polymorph

Polymorph works on one singular beast. A swarm is a bit muddy but is in fact described as more than one beast by the rules. Thus, your DM could technically allow it, but it likely is not RAW.

Yes

Polymorph clearly states (emphasis mine):

The creature is limited in the actions it can perform by
the nature of its new form

So, you can use whatever abilities/actions you receive form this new form. Telepathy is an spell, and casting a spell is an action, so you could use that while polymorphed into a swarm of cranium rats.

However, a swarm of cranium rats is a swarm, and polymorph says (emphasis mine):

The new form can be any beast whose challenge rating is equal to or less than the target’s

So you could not polymorph into a swarm of cranium rats.