During processing of connected speech, such as sentences or spoken narratives, low-frequency cortical oscillations synchronize in phase with amplitude fluctuations in the broadband speech envelope. The degree of synchronization between brain and envelope oscillations is modulated by higher-order cognitive factors, such as attention and memory, independently of the acoustic signal. This suggests that real-time encoding of speech envelopes reflects more than mere sensory transduction of fluctuations in signal amplitude. Previous research has proposed that neural tracking of the speech envelope reflects the parsing of speech primitives, such as phonemes, syllables, or peaks in the envelope derivative (rapid envelope changes, e.g., Oganian & Chang, 2019). Here, we propose neural coding of temporal speech patterns is better predicted by spectrally-local changes in relative energy within the temporal fine structure–operationalized by cochlear-scaled entropy (CSE). This metric reflects the fact that sensorineural systems respond predominantly to change.