Bitcoin certainly does have state. The blockchain itself is a representation of the state of Bitcoin at various points in time. Each new block is an update and commitment to the current state in Bitcoin.
The state that blocks updates is really the Unspent Transaction Output Set. The UTXO set is what determines everyone’s “balances”. Only outputs that are in the UTXO set can be validly spent by in a new transaction. If a wallet has an incorrect UTXO set, it could create invalid transactions. If a node has an incorrect UTXO set, it could think a transaction is valid when it is actually invalid.
In addition to the UTXO set, nodes also have to know the hash of the most recent block. This too is state. If nodes or miners do not keep track of this, they could create invalid blocks are mark invalid blocks as valid.
So yes, Bitcoin does have state, and all participants in the network must track it in order to actually be using Bitcoin.