6.5  Continuous Measure of Drowsiness

Using the algorithm described in the previous section we can compute the performance measure on various selections of test data to examine some properties and characteristics of the algorithm.  Although only a single algorithm is being analyzed, several detection algorithms are currently under development - many of which utilize frequencies to 475Hz.  This particular implementation has been chosen for illustration because: (1) it is one of the original detection algorithms developed and therefore has the best understood characteristics; (2) consistent threshold values among different subjects (discussed next).  Recall that the measure as developed increases in magnitude in response to an increase in drowsiness of an individual.

Interpretation of the Continuous Measure Output Plots
The best way to explain the drowsiness measure is by showing several examples of its performance under various conditions and explaining the behavior.  Every output plot has two primary components, the first is the measure itself which is displayed as an amplitude variation over time, and the second are the two horizontal threshold lines spanning the plot.  These threshold lines are drawn on each plot at the same ordinate values (output of the measure) of 12 and 17.  Interestingly, these values were originally derived from analysis of a single subject's test performance using only the instantaneous values of the measure corresponding to the presentation of visual events.  Using the above stated algorithm, it was found for this subject that when the output values were at approximately 12 and below, the subject responded to over 90% of the signals presented, was not asleep and not obviously impaired by excessive drowsiness.  Similarly, when the output values were at approximately 17 and above, over 90% of the events presented were missed (no response), performance failures frequently occurred where the subject tended to either experience micro-sleeps for brief periods of time or fall asleep completely.

The area between the upper and lower threshold lines is considered a gray area regarding the correlation with hits and misses, that is, when the instantaneous values fall between the thresholds, signals may or may not be responded to by the subject.  Especially within this region, the magnitude and sign of the slope of the measure tends to show a correlation with performance such that a large magnitude positive slope passing through the thresholds accounts for more of the missed events, and likewise, a large magnitude negative slope passing back through the threshold accounts for more of the hit events and improving performance.

An interesting feature of these threshold values is that, although developed on the data from a single test subject, the same threshold values, when applied to most of the other individuals, performs just as well at predicting failures as on the original subject.  This may indicate a level of invariance in the measure or some of the pieces which comprise the measure.  But, as with any signal detection problem, the tradeoffs between time-to-detection and false alarms must be balanced.  For all of the remaining examples in this chapter, the fixed threshold values will be used with no adjustment for individual characteristics to facilitate the evaluation of the algorithm's characteristics.


6.5.1  The beginning (first 8.6 minutes of testing)

In Figure 6.5.1, the drowsiness measure was computed during the initial portions of the visual test under the "alert" and "drowsy" conditions for two different test subjects.  The top plot shows data from subject KAL01 and the bottom plot from subject CVJ01.  In each plot, the black traces represent data under "alert" conditions and the red traces from the "drowsy" conditions.  Observe the data from KAL01 on the top trace: the measure on both days has an almost identical magnitude and trend.  There is an unexplained brief excursion of the measure on the alert day at approximately 6.58 minutes into the test.  The fact that the measure is tracking so closely under both conditions might lead us to infer that the subject was not affected by the "drowsy" test conditions during the early portion of their test.  The bottom example shows an elevated measure under the drowsy conditions as compared to the measure on the alert day.  This indicates that the drowsy conditions may have had a more profound effect on the bottom subject than the top subject as indicated by the offset in the measure.  These explanations are reinforced by the fact that the top subject experienced their first failure on the visual test (drowsy conditions) at 100m 48.05s elapsed time-on-task whereas the bottom subject experienced their first failure after only 16m 34.87s elapsed time-on-task.  This would lead us to believe that the bottom subject was more affected by the "drowsy" conditions.

Figure 6.5.1  Start of visual testing

As is evident by most subjects that participated in this test, a consistent pattern emerges in the form of an upward trend in the measure as time-on-task increases.  If the measure increases to the point of failure and the subject is then awakened, the measure immediately drops back to normal levels and the upward trend begins again.  Also, there are times when the measure drops back to a lower level spontaneously in response to changes in the subject's arousal, in these cases, the measure also begins the upward trend again.  This trend is certainly a physiological phenomena; the measure, as computed, has limited memory - a 1.08s (1024 data points) overlap are used in computing the output.

6.5.2  Alert vs. Extreme Drowsiness

Figure 6.5.2 shows the output of the algorithm for the test subject NKS01 on two separate test days.  The top graph shows the measure during the first 18.83 minutes of an "alert" test which is taken in the mid-morning after the subject has had routine sleep the previous night and should be as rested as they normally are.  Notice how the measure stays low and level below the two horizontal threshold lines without the characteristic upward trend.  The bottom plot shows the same subject during a "drowsy test" when the subject has been sleep deprived and the test is taken late at night.  Observe how the measure begins to drift upward as the subject begins to feel the effects of the sleep deprivation.  Performance lapses begin toward the end of the segment and the subject falls asleep entirely several time during the later parts of the test against his will.

Figure 6.5.2  Initial testing period

The performance of this subject is atypical from the rest of the subject group in that the measure remained low and level without the characteristic upward drift starting immediately after the test begins.  Instead, this subject's measure stayed low until about 14.39m into the test when the measure began to increase rapidly (spikes) and performance lapses began to occur.  The overall characteristics of the measure on this particular subject were observably different from the other subjects, even though increases in the measure corresponded with test failures.  This test subject, as a voluntary participant in a sleep apnea screening, was found to have mild obstructive sleep apnea with approximately four episodes per hour.  A detailed analysis of the differences in the performance of the measure will not be discussed at this time, but, it is interesting to note that there may be observable differences in the measure which might serve as an indicator of such a sleep disorder.  We should be aware that these observations pertain to a single test subject and no conclusions can be reached without examining a larger set of subjects and further study.

6.5.3  Eyes Independence

Another feature of this measure of drowsiness is that it is independent of the status of the eyes.  Typically, EEG signals tend to show shifts in their spectral distribution which is dependent upon the status of the eyes.  This has proven to be a nuisance to many researchers attempting to use EEG signals as drowsiness indicators because the status of the eyes can effect the signal and cause ambiguous results (in particular alpha activity is very susceptible to the status of the eyes).  The algorithm developed in this work is not susceptible to fluctuations induced by changes in the status of the eyes and therefore, can be applied in those applications where it may be acceptable for a subject to close their eyes periodically - provided that they remain awake and able to respond to situations as necessary.  A drowsiness measure dependant on eye status (such as blink monitors, etc.) may not be able to distinguish between voluntary eye closure and unintentional closure associated with drowsiness and sleep, and would therefore be ill suited to many real-world applications.  Also, an eyes-based drowsiness indicator would be insensitive to the physiological status (drowsiness) once the eyes have been voluntarily closed and could no longer distinguish between behavioral states.

Prior to each visual test under the "alert" and "drowsy" conditions, a 2 minute segment of data was collected where the subject was instructed to sit quietly with their eyes open (1 minute) and then with their eyes closed (1 minute).  From this data, two segments of 34.49 seconds each (32768 samples) corresponding to the two physical states of the eyes (open and closed) were extracted.  Each 34.49 second segment was chosen from the 60 second segment on the basis of freedom from EEG artifacts (clean).  The two segments were joined together and the drowsiness measure was computed on the resulting data segment.  Data is shown here for two different test subjects for both the "alert" and "drowsy" conditions.

Figure 6.5.3 and Figure 6.5.4 show data for test subject KAL01 on both the "alert" and "drowsy" days, respectively.  The top plot of each figure shows the vertical EOG measurement corresponding to the EEG data being processed below (the occipital EEG channel as usual).  The left half of each plot shows the subject with eyes open and blinking (downward deflection are eyes closing and upward deflection are eyes opening) and the right half shows the subject with their eyes intentionally closed.  Notice how the corresponding measure is insensitive to voluntary changes in the status of the eyes.  Depending on the level of drowsiness of an individual and their individual reactions to that drowsiness, the measure may slowly start to increase after the eyes are closed for a few seconds, this would only typically be observed when the subject is extremely drowsy, and does not typically appear on a well rested subject.  Keep in mind, this upward trend is not induced by the closure of the eyes, instead it is a result of the effect that voluntary eye closure has on the "drowsiness" of the individual.

Figures 6.5.5 and 6.5.6 show data for subject CVJ01 under both "alert" and "drowsy" conditions, respectively.  Again, no significant changes in the behavior of the measure are associated with voluntary changes in the status of the eyes.

 

Figure 6.5.3  Eyes open/closed - "alert" conditions

---

Figure 6.5.4  Eyes open/closed - "drowsy" conditions

---

Figure 6.5.5  Eyes open/closed - "alert" conditions

---

Figure 6.5.6  Eyes open/closed - "drowsy" conditions

6.5.4  Lapses and Performance Failures

When a subject is extremely tired, lapses or brief episodes of sleep begin to appear, sometimes culminating in full-fledged sleep.  These failures are characterized by the drowsiness measure rising above the upper threshold line and cessation of responses to the visual stimulus for extended time periods (as opposed to only one or two missed signals in a row).  The magnitude of the measure is proportional to the intensity of the drowsiness, although the exact nature of the relationship is not completely understood (linear, nonlinear, etc.).  We do know that large magnitudes of the measure (relative to the thresholds) correspond to the initial stages of sleep as demonstrated later.

Lapses and micro-sleeps are a very common phenomena and reported in much of the literature (see Section 2.3)on human performance under drowsy conditions (extremely long time-on-task, sleep deprivation studies, etc.).  And during our visual task, it is not uncommon for subjects to fail completely (succumb to sleep) during their test.

The next set of figures shows examples of various performance failures captured during the laboratory study.  As with the previous plots, these plots will all be displayed with the same threshold values and ordinate range for comparison.  Also included on each of these plots are the corresponding event markers shown previously on the spectrograms which mark the appearance of the visual signals, with solid lines corresponding to hits and dashed lines for misses.

The data shown in Figure 6.5.7 demonstrates a behavioral failure with very rapid onset and exit transitions.  In this figure we see a couple of fast spikes in the measure which cross both thresholds and then returns.  The measure then rises with a very sharp slope (to a peak value of 100.35) while the subject misses a large string of events.  The subject was awakened/alerted when they lost muscle tonus in the neck and their head dropped off to the left side.  The measure immediately returned to normal levels and the subject began responding to the visual signals almost immediately.  After the failure subsided, the computer returned to generating visual stimulus events at random times.  An overall increase in the amplitude of the measure is evident as the test continues and we see that the subject finished this segment by missing a single event and then responding to the next two successive signals without intervention.

It has been observed that the measure is typically not sensitive to single missed events when the subject responds to the proceeding event on their own.  Many times single and multiple missed events are attributed to long eye closures (especially in the latter portions of the test) and momentary inattention.  These are two situations which, although prompted by drowsiness, are not typically distinguished by the current algorithm.  Other researchers such as in Makeig and Inlow (1993) have similarly found that "In particular, isolated errors during periods of low error rate may not be detectable...".

Figure 6.5.7  Performance failure example #1

Figure 6.5.8 shows another typical failure in which the measure rises above the thresholds and the subject misses a long string of visual events.  In this case, the subject was awakened by the test administrator via intercom at which time their measure immediately dropped back to normal levels and responses to the visual signals began again.  This test subject would occasionally miss a signal due to eye closure and the measure appeared insensitive to this phenomena.  During tests, some subjects would try and rest their eyes between visual signals, thinking that they could predict the time span between signals, i.e. a signal would appear, the subject would respond, then the subject may close their eyes for several seconds as a recuperative process.  Many times the single or double missed events occurred during this type of behavior in which the subject was very clearly not asleep.  This eyes resting appeared most often with subjects who wore contact lenses during their tests.  Subjects reported after testing that their contact lenses began to dry out and irritate their eyes because of the long time spent staring intensely at the visual display board.

Figure 6.5.8  Performance failure example #2

In Figure 6.5.9, the subject experienced two failures side-by-side that occurred at the end of their testing session.  During the first of the two failures, the drowsiness measure increased rapidly (to a maximum level of 121.03) and the subject stopped responding to all visual signals as indicated by the event marks.  At about the 14th missed event in the first string of misses the subject's head slowly dropped to the side as muscle tonus was lost in their neck.  As their head began to fall to the side, the drowsiness measure began dropping to a sub-threshold value.  The subject was still unresponsive to the visual signals as their eyes were still closed.  At the time of the final missed event in the string, the test administrator keyed the intercom microphone which produces an audible click.  Typically, the microphone click is of an insufficient noise level as to wake a sleeping subject, but as soon as the microphone was keyed, the subject began responding to the visual signals as presented with no signs of startling.  Video record reveals that the subject did attempt to respond to the last signal shown as a missed event, but the subject apparently did not press the response button properly for the response to register.  The subject then proceeded to respond normally to the two final signals.

After the subject successfully responded to two signals and the computer switched back to randomly appearing events, the measure began to rise again.  The interesting feature of the second string of misses in this figure is that the subject missed all signals with their eyes open and blinking (visible on both the video and vertical EOG).  The subject only began responding to signals after being alerted via keying the intercom microphone switch.  The subject's response to the microphone key is probably a type of conditioned response, all communications with the test subjects during testing is via intercom where all communications are preceded by the brief audio intercom click.  This short failure represents one of the only failure segments captured where the subject had their eyes open and blinking while missing all events.  A drowsiness measure based solely on the status of the eyes would not respond to this situation.

Figure 6.5.9  Performance failure example #3

Finally, Figure 6.5.10 shows a steady increase in the measure that culminates in a failure.  This segment begins with a hit event.  Video and EOG recordings show an interesting series of events during this initial segment.  The subject's eyes were closing on the first missed event (#35), then their eyes opened between events, closed again prior the next event (#36), opened again after the signal, and finally closed for the last two misses (#37 and #38).  The subject opened their eyes again during the first hit event (#39), and continued to respond.  The subject was clearly not asleep but was resting their eyes and periodically checking the display board for the visual signals.  The subject would simply open their eyes periodically to see if there was a signal and was unaware that they had missed several successive events while their eyes were closed.

The measure began another steady increase, during which the subject failed to respond to the string of visual signals (#42-#52).  The subject was alerted verbally by the test administrator to "continue the test and try and keep your eyes open".  Immediately following the verbal alert, the subject began responding to visual signals normally and the measure dropped back to normal levels.

Figure 6.5.10  Performance failure example #4

Short-Duration Lapses
Failures characterized by a rapid increase in the measure for a relatively short time period was also observed during testing, an example is shown in Figure 6.5.11.  There is a single failure shown in this figure around time bin 440 (7.92 min) with an approximate 30 second duration.  Prior to this failure, a similar "spike" in the measure can be observed around time bin 280 (5.05 min) which also lasts for approximately 30 seconds.  But, because there was no visual event presented during the period corresponding with the first observed peak, there was no failure to be observed.  We would expect, however, that if there had been an event displayed during the first episode that the subject would not have responded to the signal as they did later in the test during the failure shown which had nearly identical characteristics.

Figure 6.5.11  Short duration lapse

This brings up a potentially inherent flaw in any drowsiness detection system which relies strictly on behavioral information to train an algorithm (such as using driving simulator data to train an eyes-based measure): it would not have detected the episode around bin 280 because there was no behavioral failure.  Many times, such as when driving, circumstances (and tasks) are "tolerant" of brief failures (although not necessarily for 30 seconds) and the behavior continues undetected.

Figure 6.5.12 shows an example with an uncertain explanation.  The subject misses several consecutive visual signals before the drowsiness measure increases sufficiently to a level which would correlate with this behavior.  Video and EOG analysis shows that the subject's eyes were closed during the single missed event and re-opened for the subsequent events.  Also, the subject's eyes were closing at the beginning of the string of missed events.  From the beginning of the string of missed events, the drowsiness measure begins to rise rapidly, indicating that the subject is extremely drowsy.  Using the information from the video, EOG, and drowsiness measure, we can only infer that the subject closed their eyes and drifted off to sleep.

Figure 6.5.12  Performance failure

6.5.5  Sleep

Upon completion of the visual test under the "drowsy" conditions, the subjects lay down in bed and take a one hour nap.  Because the subjects are extremely tired and fatigued after the visual test, they typically fall asleep very easily.  Throughout the brief nap, EEG data is collected.  The following figures were generated by running the drowsiness tracking algorithm on various segments of this sleep data over several different test subjects.  As with the previous figures, the horizontal threshold lines are drawn in the same locations, but the range of the ordinate was increased to accommodate increased measure amplitude.  The following examples show the performance of the drowsiness measure during known sleep and enable us to compare this output to what we are seeing during the performance tests.  The following sleep examples are being displayed in this section for illustration purposes only (to gauge qualitative performance) and we will not be analyzing the response of the measure with respect to the corresponding sleep stages.

Figure 6.5.13 (kal01 dr:b) shows an example of a sleep segment from the initial stages (laying down) through the first 9.65 minutes in bed.  From the time the subject lays down in bed, they begin their transition toward sleep as indicated by the drowsiness measure.  By the end of this data segment, the measure reaches a maximum value of 84.47 and is significantly above the top threshold value of 17.0 where most subjects fail to respond to the visual signals.

Figure 6.5.13  Sleep onset period

Even more revealing is Figure 6.5.14 (kal01 dr:b and dr:d) which shows the beginning of the visual test under the "drowsy" conditions superimposed over the sleep segment.  It is interesting to note that both sets of data follow the same upward trend during the first few minutes, except the sleep data continues the upward trend while the data from the visual test stays below the threshold lines.  It is also interesting to note that the data taken from the visual test captures the subject with their eyes open while the sleep data was under eyes closed conditions.  This further illustrates the insensitivity of the measure to the status of the eyes (discussed previously in the section discussing/ demonstrating the eyes-independent nature of the measure).

Figure 6.5.14  Sleep onset and initial testing period

Finally, the "alert", "drowsy", and "sleep" segments of data are all superimposed on the same plot where the range of the ordinate is magnified to demonstrate the enormous consistency of the measure on this subject, refer to Figure 6.5.15.  The data during all three conditions follows the same upward trend with an almost identical path. 

Figure 6.5.15  Sleep onset and initial testing periods

It is also interesting to note the individual differences between various subjects and the behavior of the drowsiness measure.  Figure 6.5.16 (cvj01) shows another subject with the "alert", "drowsy" (same as Figure 2), and "sleep" data superimposed on the same scales.  We can see that from the "alert" to the "drowsy" day, the measure has an overall increase in amplitude (relatively constant offset) through the first few minutes of the test.  But, during the sleep portion, the measure begins at an increased amplitude and has a much steeper slope.

Figure 6.5.16  Sleep onset and initial testing periods

As a third example, Figure 6.5.17 (cia01) shows the measure under the same three conditions with similar behavior to the previous example.  The "alert" conditions produce the lowest overall magnitude, with the "drowsy" condition slightly higher, and the "sleep" condition with the highest magnitude.

Figure 6.5.17  Sleep onset and initial testing periods

The response of the drowsiness measure to a subject falling asleep looks very similar to the output of the measure as subject's experience failures on the visual test.  Because we have already shown that the measure is independent of the status of the eyes and the striking similarities between failures on the visual test and the initial stages of sleep, we can be confident that the measure is indeed capturing the expected phenomena: extreme drowsiness and falling asleep.

Extended time sequences (sleep)
Now we can look at the output of the measure for the same three test subjects just shown over the first (approximately) 45 minutes of their sleep session.  During these sessions, the subjects typically move from relaxed wakefulness, through stage 1, stage 2, and delta sleep.  During sleep data collection, the EEG amplification may have to be changed from time-to-time during the course of the data collection as the EEG signal amplitude either becomes too large (saturation) or too small (loss of resolution).  Therefore, small sections of data were omitted from the calculation of the measure corresponding to amplifier adjustments.  These segments (with a small margin of data on either side) were omitted from the plot and appear as missing data to preserve the overall temporal flow of the drowsiness measure.  All data is scaled during processing, so the amplification changes do not effect the output of the measure.

Typically, the drowsiness measure increases over the course of the sleep-onset phase until a plateau is reached where it remains steady as shown in Figure 6.5.18 (kal01).  For this particular example, the subject laid down in bed and did not move at all during the entire sleep session until awakened one hour later.  This example shows the measure during the first 45 minutes of their session.

Figure 6.5.18  Extended sleep sequence

In Figure 6.5.19 (cvj01), the subject does not fall asleep quite as quickly as the previous subject.  This subject experienced a couple of arousals during their sleep-onset phase as indicated by the annotations (the first character of the text marking the appropriate location on the abscissa) and the movement causes the drowsiness measure to decrease, as expected.  The subject quickly resumed their sleep-onset transition after each arousal until eventually reaching a semi-steady level.

Figure 6.5.19  Extended sleep sequence

Finally, Figure 6.5.20 (cia01) shows similar characteristics to the previous examples.  We can observe the subject passing through the initial sleep-onset phase to an initial plateau (around 50) and then to a second plateau (around 125).  The measure dips momentarily at around 18.75 minutes, but video examination of the session did not reveal any apparent movement or arousal in the subject.  The subject then wakes momentarily and the measure again rises upward to the first plateau where it remains through most of the remainder of the segment.

Figure 6.5.20  Extended sleep sequence

<PREVIOUS

NEXT>