TimeBinner

The TimeBinner ObsFunction assigns each observation to a discrete time bin, returning an integer bin number that is consistent across all MPI ranks. Optionally, a bin label timestamp can be written back to the ObsSpace, giving all observations in the same bin an identical timestamp.

This function is registered as an IntObsFunction and must be referenced as IntObsFunction/TimeBinner.

Bin labels and bin windows

Each bin is identified by a bin label timestamp — a whole multiple of the bin interval unit (e.g. a whole hour, midnight UTC). An observation is assigned the label of whichever bin’s window it falls into, where each bin’s window is the inclusive interval:

\[[\text{bin label} + \text{window lower bound},\; \text{bin label} + \text{window upper bound}]\]

For example, with an hour interval and window -PT25M to PT25M, the timestamp 2018-04-17 01:20:02 UTC falls within the window of the 01:00 bin [00:35, 01:25], so it receives the label 2018-04-17 01:00:00 UTC, as does 2018-04-17 00:50:00 UTC. All observations within that window share the same bin number.

Similarly, with a day interval and window -PT6H to PT6H, the same timestamp falls within the midnight bin [2018-04-16 18:00, 2018-04-17 06:00] and receives the label 2018-04-17 00:00:00 UTC.

Bin windows do not need to be symmetric around the bin label. For example, a window of -PT30M to PT0M covers the 30 minutes ending at the hour boundary.

Bin windows do not need to include the bin label timestamp — bin window lower bound can be positive (window starts after the label) or bin window upper bound can be negative (window ends before the label). For example, a window of PT1S to PT1H spans the hour following each label: with an hour interval an observation at 11:00 UTC would be assigned to the bin labelled 10:00 UTC.

Any observation falling outside every bin window is assigned the missing integer value.

Constraints (validated at construction time):

• bin window lower bound ≤ bin window upper bound

• Window width (upper bound − lower bound) < bin interval unit (prevents overlap between adjacent bins)

• |bin window lower bound| ≤ bin interval unit, and |bin window upper bound| ≤ bin interval unit where | . | denotes the absolute value.

Bin numbering

Bin numbers are non-negative integers starting from 0:

• Forward order (default): bin 0 is the bin whose label contains the earliest observation (across all MPI ranks) that falls within a bin window; bin numbers increase with time.

• Reverse chronological order (reverse chronological order: true): bin 0 is the bin whose label contains the latest observation (across all MPI ranks) that falls within a bin window; bin numbers increase going back in time.

Note

When bin window lower bound is positive, a gap exists between each bin label timestamp and the opening of its window. If forward order is being used then any observation that falls within this gap for the first bin (the earliest observations) will recieve a missing bin number, even though a hypothetical “bin −1” window would cover it. For example, with an hour interval, bin window lower bound: PT30M, and bin window upper bound: PT1H20M, an initial observation exactly 00:15 UTC would receive a missing bin number, even though it falls at the hypothetical “bin −1” window [23:30, 00:20] associated with the 23:00 label. See Example D for an illustration of this scenario.

Options

• bin interval unit (required): Granularity of the bins. Accepted values: second, seconds, minute, minutes, hour, hours, day, days.

• bin window lower bound (required): Lower bound of the bin window as an ISO 8601 duration relative to the bin label (e.g., -PT30M for 30 minutes before the label, PT1S for 1 second after). May be negative, zero, or positive.

• bin window upper bound (required): Upper bound of the bin window as an ISO 8601 duration relative to the bin label (e.g., PT0M for exactly the label, PT30M for 30 minutes after). Must be ≥ bin window lower bound.

• input timestamp variable (optional, default: MetaData/dateTime ): The ObsSpace variable containing the timestamps to bin.

• binned timestamp variable (optional): If provided, the bin label timestamp is written to this ObsSpace variable for every observation that falls within a bin window. Observations outside any bin window receive the missing datetime value.

• reverse chronological order (optional, default: false ): If true, bin numbering starts from the latest observation and increases going back in time.

Examples

The examples below use the Variable Assignment filter to compute bin numbers and bin label timestamps for hourly binning of wind observation timestamps.

Example A — window ending at the hour boundary

Observations in the 30-minute window ending at each hour boundary, i.e. with timestamps \(t\) satisfying

\[T - \text{30 min} \leq t \leq T\]

where \(T\) is the bin label timestamp, are assigned to the same bin. Observations outside this window receive missing bin numbers.

- filter: Variable Assignment
  assignments:
    - name: MetaData/binNumbers
      type: int
      function:
        name: IntObsFunction/TimeBinner
        options:
          bin interval unit: hour
          bin window lower bound: -PT30M   # 30 minutes before the hour (inclusive)
          bin window upper bound: PT0M       # up to the hour boundary (inclusive)
          binned timestamp variable: MetaData/binTimestamps  # written for in-window obs only

For the following observation timestamps, the bin numbers and bin label timestamps assigned are:

Observation timestamp	Bin number	Bin label timestamp
2018-04-16T23:50:00Z	0	2018-04-17T00:00:00Z
2018-04-17T00:00:00Z	0	2018-04-17T00:00:00Z
2018-04-17T00:15:00Z	missing	missing
2018-04-17T00:30:00Z	1	2018-04-17T01:00:00Z
2018-04-17T00:35:00Z	1	2018-04-17T01:00:00Z
2018-04-17T01:00:00Z	1	2018-04-17T01:00:00Z

Note that 2018-04-17T00:15:00Z falls between the windows [23:30, 00:00] and [00:30, 01:00] and so receives a missing bin number.

Example B — window spanning the hour boundary

A 59-minute 59-second window centred just before the hour boundary, i.e. with timestamps \(t\) satisfying

\[T - \text{30 min} \leq t \leq T + \text{29 min 59 s},\]

where \(T\) is the bin label timestamp, covers all observations without gaps (since JEDI timestamps have 1-second resolution).

- filter: Variable Assignment
  assignments:
    - name: MetaData/binNumbers
      type: int
      function:
        name: IntObsFunction/TimeBinner
        options:
          bin interval unit: hour
          bin window lower bound: -PT30M
          bin window upper bound: PT29M59S

Using the same observation timestamps as Example A:

Observation timestamp	Bin number	Bin label timestamp
2018-04-16T23:50:00Z	0	2018-04-17T00:00:00Z
2018-04-17T00:00:00Z	0	2018-04-17T00:00:00Z
2018-04-17T00:15:00Z	0	2018-04-17T00:00:00Z
2018-04-17T00:30:00Z	1	2018-04-17T01:00:00Z
2018-04-17T00:35:00Z	1	2018-04-17T01:00:00Z
2018-04-17T01:00:00Z	1	2018-04-17T01:00:00Z

Unlike Example A, 2018-04-17T00:15:00Z now falls within the window [23:30, 00:29:59] of the 00:00 bin and receives bin number 0.

Example C — reverse chronological order

Same window as Example B (\(T - \text{30 min} \leq t \leq T + \text{29 min 59 s}\)), but bin 0 is assigned to the latest observation; earlier observations receive increasing bin numbers.

- filter: Variable Assignment
  assignments:
    - name: MetaData/binNumbers
      type: int
      function:
        name: IntObsFunction/TimeBinner
        options:
          bin interval unit: hour
          bin window lower bound: -PT30M
          bin window upper bound: PT29M59S
          reverse chronological order: true

Using the same observation timestamps, bin 0 is now assigned to the latest in-window observation (2018-04-17T01:00:00Z), so bin numbers increase going backwards in time:

Observation timestamp	Bin number	Bin label timestamp
2018-04-16T23:50:00Z	1	2018-04-17T00:00:00Z
2018-04-17T00:00:00Z	1	2018-04-17T00:00:00Z
2018-04-17T00:15:00Z	1	2018-04-17T00:00:00Z
2018-04-17T00:30:00Z	0	2018-04-17T01:00:00Z
2018-04-17T00:35:00Z	0	2018-04-17T01:00:00Z
2018-04-17T01:00:00Z	0	2018-04-17T01:00:00Z

Note

reverse chronological order only affects bin numbering not the extent of the windows in time. For example, with the same window as Example B, the 2018-04-17T00:00:00Z bin still covers [23:30, 00:29:59] and the 2018-04-17T01:00:00Z bin still covers [00:30, 01:29:59], but the 2018-04-17T00:00:00Z bin is now bin 1 and the 2018-04-17T01:00:00Z bin is bin 0.

Example D — window starting after the bin label

Setting bin window lower bound to a positive value creates a gap between each bin label timestamp and the start of its window. Observations in this gap are not covered by the bin sharing their label — they may be covered by the previous bin’s window, or receive a missing bin number if no previous bin covers them.

In this example the window spans the hour following each label \(T\), excluding the label itself:

\[T + \text{1 second} \leq t \leq T + \text{1 hour}\]

- filter: Variable Assignment
  assignments:
    - name: MetaData/binNumbers
      type: int
      function:
        name: IntObsFunction/TimeBinner
        options:
          bin interval unit: hour
          bin window lower bound: PT1S    # one second after the label
          bin window upper bound: PT1H      # up to one hour after the label (inclusive)

Where a timestamp \(t\) equals a bin label timestamp \(T\), it falls within the upper bound of the previous bin’s window.

In the table below, the second earliest observation (2018-04-16T23:50:00Z) falls within the 2018-04-16T23:00:00Z bin label timestamp’s window [23:00, 23:59], so bin 0 is assigned the 2018-04-16T23:00:00Z label.

Note

An extra timestamp has been added to the start of the hour in this example to illustrate the effect of this gap on the earliest observations.

Observation timestamp	Bin number	Bin label timestamp
2018-04-16T23:00:00Z	missing	missing
2018-04-16T23:50:00Z	0	2018-04-16T23:00:00Z
2018-04-17T00:00:00Z	0	2018-04-16T23:00:00Z
2018-04-17T00:15:00Z	1	2018-04-17T00:00:00Z
2018-04-17T00:30:00Z	1	2018-04-17T00:00:00Z
2018-04-17T00:35:00Z	1	2018-04-17T00:00:00Z
2018-04-17T01:00:00Z	1	2018-04-17T00:00:00Z

The observations at 2018-04-17T00:00:00Z and 2018-04-17T01:00:00Z each fall outside their own bin’s window but are found at the inclusive upper bound of the previous bin’s window, so they are assigned to that previous bin.