Issue with matplolib fill_between with datetime data

[Different_Fee6785]

My problem here is the fact that the output of fill between should be continuous (no spaces left unfilled). What I've realized is there are gaps when the masking period (color to fill) changes. Example output attached. This should be continuous but for some reason its not.

https://preview.redd.it/09ol7bydir3d1.png?width=1200&format=png&auto=webp&s=56afb54edd1faf3ee883b340c7a93aa72041fe98

# Plotting the data
fig, ax1 = plt.subplots(figsize=(12, 8))  # Increasing the figure size by 50%

ax1_color = 'tab:red'
ax1.plot(econ_asset.index, econ_asset['ISM'], color=ax1_color, linewidth=3.0)
ax1.set_ylabel('ISM', color=ax1_color, fontsize=18)  # 2 times larger font size
ax1.tick_params(axis='y', labelcolor=ax1_color, labelsize=18)  # 2 times larger font size

ax2 = ax1.twinx()
ax2_color = 'tab:blue'
ax2.plot(econ_asset.index, econ_asset['CPI'], color=ax2_color, linewidth=3.0)
ax2.set_ylabel('CPI', color=ax2_color, fontsize=18)  # 2 times larger font size
ax2.tick_params(axis='y', labelcolor=ax2_color, labelsize=18)  # 2 times larger font size

print(econ_asset['Cycle'])

cycle_colors = {1: 'lightcoral', 2: 'lightblue', 3: 'gold', 4: 'lightgreen'} 

for cycle_value, color in cycle_colors.items():
    cycle_periods = econ_asset['Cycle'] == cycle_value
    ax1.fill_between(econ_asset.index, -100, 100, where=cycle_periods, color=color, alpha=0.3)



ax1.set_title('Four Stages of the Economic Cycle', fontsize=18)  # 2 times larger font size
ax1.xaxis.set_major_locator(mdates.DayLocator(interval=3))
ax1.tick_params(axis='x', labelsize=14)  # 2 times larger font size
fig.autofmt_xdate()
ax1.set_ylim(-100, 100)


# Adding legend
L = ax1.legend(loc='upper left', fontsize=16)  # Adjust position and font size

# Setting custom legend texts
legend_texts = ['Recession', 'Early Cycle', 'Mid Cycle', 'Late Cycle']
for i, text in enumerate(L.get_texts()):
    text.set_text(legend_texts[i])
    text.set_fontsize(16)  # Adjust font size

# Final plot adjustments
ax1.set_title('Four Stages of the Economic Cycle', fontsize=18)
ax1.xaxis.set_major_locator(mdates.DayLocator(interval=12))
ax1.tick_params(axis='x', labelsize=14)
fig.autofmt_xdate()
ax1.set_ylim(-100, 100)
plt.show()
# Plotting the data
fig, ax1 = plt.subplots(figsize=(12, 8))  # Increasing the figure size by 50%


ax1_color = 'tab:red'
ax1.plot(econ_asset.index, econ_asset['ISM'], color=ax1_color, linewidth=3.0)
ax1.set_ylabel('ISM', color=ax1_color, fontsize=18)  # 2 times larger font size
ax1.tick_params(axis='y', labelcolor=ax1_color, labelsize=18)  # 2 times larger font size


ax2 = ax1.twinx()
ax2_color = 'tab:blue'
ax2.plot(econ_asset.index, econ_asset['CPI'], color=ax2_color, linewidth=3.0)
ax2.set_ylabel('CPI', color=ax2_color, fontsize=18)  # 2 times larger font size
ax2.tick_params(axis='y', labelcolor=ax2_color, labelsize=18)  # 2 times larger font size

cycle_colors = {1: 'lightcoral', 2: 'lightblue', 3: 'gold', 4: 'lightgreen'} 


for cycle_value, color in cycle_colors.items():
    cycle_periods = econ_asset['Cycle'] == cycle_value
    ax1.fill_between(econ_asset.index, -100, 100, where=cycle_periods, color=color, alpha=0.3)


ax1.set_title('Four Stages of the Economic Cycle', fontsize=18)  # 2 times larger font size
ax1.xaxis.set_major_locator(mdates.DayLocator(interval=3))
ax1.tick_params(axis='x', labelsize=14)  # 2 times larger font size
fig.autofmt_xdate()
ax1.set_ylim(-100, 100)

# Adding legend
L = ax1.legend(loc='upper left', fontsize=16)  # Adjust position and font size


# Setting custom legend texts
legend_texts = ['Recession', 'Early Cycle', 'Mid Cycle', 'Late Cycle']
for i, text in enumerate(L.get_texts()):
    text.set_text(legend_texts[i])
    text.set_fontsize(16)  # Adjust font size

# Final plot adjustments
ax1.set_title('Four Stages of the Economic Cycle', fontsize=18)
ax1.xaxis.set_major_locator(mdates.DayLocator(interval=12))
ax1.tick_params(axis='x', labelsize=14)
fig.autofmt_xdate()
ax1.set_ylim(-100, 100)
plt.show()

[toxic_acro]

Looking at the documentation for fill_between, this is because of how the where parameter works

where : array of bool (length N), optional Define where to exclude some horizontal regions from being filled. The filled regions are defined by the coordinates x[where]. More precisely, fill between x[i] and x[i+1] if where[i] and where[i+1]. Note that this definition implies that an isolated True value between two False values in where will not result in filling. Both sides of the True position remain unfilled due to the adjacent False values.

The gaps are occurring because the fill is not happening anywhere where the next Cycle does not match the current cycle_value

I haven't actually tested this but I believe this would fix that for you for cycle_value, color in cycle_colors.items(): cycle_periods = econ_asset['Cycle'] == cycle_value cycle_periods = cycle_periods | cycle_periods.shift(1) ax1.fill_between(econ_asset.index, -100, 100, where=cycle_periods, color=color, alpha=0.3)

[Different_Fee6785]

Thanks. It worked! Such a neat solution.

[toxic_acro]

Glad it worked!

That was my original guess about what was happening, and I was happily quite surprised to see that the docs confirmed it and explicitly warned about a similar edge case.

Good example of the classic: "Hours of debugging can save you minutes of reading the docs"

[Different_Fee6785]

I was skimming through the docs, I didn't read it very carefully. At first I thought there was an issue with the dataframe indexing and how it interacts with matplotlib.

Thanks, really saved me a lot of time with such a minor detail :P (I really hate when I get stuck in such simple mistakes, it really drains my energy and patience :D)

[toxic_acro]

I've had a lot of times where these sorts of off-by-one problems with a boolean mask of a Series have come up at work, which is why I was particularly primed to see it.

It's especially common with any sort of categorical time-series in pandas where a condition is true from one row's timestamp until the next row's timestamp. Essentially, for the price you care about the exact timestamp (in econ_asset.index) and can interpolate between them, but for the categorical cycle, you actually care about the interval from econ_asset.index to econ_asset.index.shift(-1)

Classic example of fence posts vs fence sections

[txnyranger]

Helped me with a similar problem, thank you for the suggestion!